rsync - the ONLY tool you need for copying and backup.
Introduction
rsync - a fast, versatile, remote (and local) file-copying tool
Usages with just one SRC arg and no DEST arg will list the source files instead of copying.
DESCRIPTION
Rsync is a fast and extraordinarily versatile file copying tool. It can copy locally, to/from another host over any remote shell, or to/from a remote rsync daemon. It offers a large number of options that control every aspect of its
behavior and permit very flexible specification of the set of files to be copied. It is famous for its delta-transfer algorithm, which reduces the amount of data sent over the network by sending only the differences between the source files and the existing files in the destination. Rsync is widely used for backups and mirroring and as an improved copy command for everyday use.
Rsync finds files that need to be transferred using a "quick check" algorithm (by default) that looks for files that have changed in size or in last-modified time. Any changes in the other preserved attributes (as requested by options) are made on the destination file directly when the quick check indicates that the file’s data does not need to be updated.
Some of the additional features of rsync are
- support for copying links, devices, owners, groups, and permissions
- exclude and exclude-from options similar to GNU tar
- a CVS exclude mode for ignoring the same files that CVS would ignore
- can use any transparent remote shell, including ssh or rsh
- does not require super-user privileges
- pipelining of file transfers to minimize latency costs
- support for anonymous or authenticated rsync daemons (ideal for mirroring)
Backup to External USB HDD
Initial Backup
rsync -av --stats --progress --exclude-from '/home/terry/Data/rsync/exclude.list' /home/terry /media/27a25959-1dca-4503-be9e-f23b83fa3d6c/backup
exclude.list file (all directories are relative to the source)
$ cat exclude.list .cache/ .config/google-chrome/ .local/share/Trash/ .thumbnails/ Music/ Downloads/tmp/
Output
Number of files: 38817 Number of files transferred: 31567 Total file size: 11128842577 bytes Total transferred file size: 11128828434 bytes Literal data: 11128828434 bytes Matched data: 0 bytes File list size: 1138813 File list generation time: 0.001 seconds File list transfer time: 0.000 seconds Total bytes sent: 11132686553 Total bytes received: 628021 sent 11132686553 bytes received 628021 bytes 23026503.77 bytes/sec total size is 11128842577 speedup is 1.00 rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1070) [sender=3.0.9]
Subsequent backup (delete on destination)
rsync -av --delete --progress --delete --exclude-from '/home/terry/Data/rsync/exclude.list' /home/terry /media/27a25959-1dca-4503-be9e-f23b83fa3d6c/backup
Different --delete options, make your own choice
--del an alias for --delete-during --delete delete extraneous files from dest dirs --delete-before receiver deletes before xfer, not during --delete-during receiver deletes during the transfer --delete-delay find deletions during, delete after --delete-after receiver deletes after transfer, not during --delete-excluded also delete excluded files from dest dirs
Moving a file system (for example /home)
For example, move /home to a separate partition or logical volume
sync -axHAX --progress /oldhome/ /newhome/
options:
- -a, --archive
archive mode; equals -rlptgoD (no -H -A -X) - -x, --one-file-system
don’t cross filesystem boundaries - -H, --hard-links
preserve hard links - -A --acls
preserve ACLs (implies -p) - -X, --xattrs
preserve extended attributes
NOTE: rsync will be used to copy all the data because it knows how to preserve all the attributes, including the extended-attributes which are required for selinux to work when it's enabled. -- System Rescue CD
rsync via SSH
Copy (not sync, DO NOT Delete on destination)
rsync -avz --progress --stats /home/terry terry@terry.im:/home/terry/backup
NOTE: rsync will create a folder under /home/backup/terry on host: terry.im
Trailing slash
A trailing slash on the source changes this behavior to avoid creating an additional directory level at the destination. You can think of a trailing / on a source as meaning "copy the contents of this directory" as opposed to "copy the directory by name", but in both cases the attributes of the containing directory are transferred to the containing directory on the destination. In other words, each of the following commands copies the files in the same way, including their setting of the attributes of /dest/foo:
rsync -av /src/foo /dest rsync -av /src/foo/ /dest/foo
Note also that host and module references don’t require a trailing slash to copy the contents of the default directory.
For example, both of these copy the remote directory’s contents into "/dest":
rsync -av host: /dest rsync -av host::module /dest
Difference between copy and sync
sync deletes on target but copy does NOT.
Copy
Example, copy support.au.oracle.com /opt/backup (folder) to local file system /bea/backup
rsync -av --progress terry@support.au.oracle.com:/opt/backup /bea/backup
NOTE: /bea/backup/backup will be created (without trailing /, means copy the folder)
OR
rsync -av --progress terry@support.au.oracle.com:/opt/backup/ /bea/backup/backup
NOTE: copy the contents inside /opt/backup to local /bea/backup/backup folder
Sync
source: oracle@10.187.46.239:/soa/rsync
destination: local file system /bea/backup
1. synchronize the rsync folder to local /bea/backup folder (will create a new rsync folder under /bea/backup)
rsync -avz --delete --progress oracle@10.187.46.239:/soa/rsync /bea/backup
2. same command with a trailing / (sync the contents inside the folder instead of copying the folder)
rsync -avz --delete --progress oracle@10.187.46.239:/soa/rsync/ /bea/backup
The abovecommand synchronizes all contents inside the /soa/rsync folder to /bea/backup
3. to do the same as 1, change the command to below (trailing slash)
rsync -avz --delete --progress oracle@10.187.46.239:/soa/rsync/ /bea/backup/rsync
rsync over non-standard SSH port
For example, SSH on destination runs on port 2222.
Use --rsh=COMMAND or -e to specify the remote shell to use.
# Use --rsh=COMMAND rsync -avz --progress --stats --rsh='ssh -p 2222' /home/terry/DevOps terry@terry.im:/home/terry # Use -e rsync -avz --progress --stats -e "ssh -p 2222" terry@terry.im:/home/terry/backup/support /home/terry/backup
Reference
man rsync is all you need.