Cool Cmd #3 Rsync
man rsync
DESCRIPTION
Rsync is a fast and extraordinarily versatile file copying tool. It can copy locally, to/from another host over any remote shell, or to/from a remote rsync daemon. It offers a large number of options that control every aspect of its behaviour and permit very flexible specification of the set of files to be copied. It is famous for its delta-transfer algorithm, which reduces the amount of data sent over the network by sending only the differences between the source files and the existing files in the destination. Rsync is widely used for backups and mirroring and as an improved copy command for everyday use.
As seen with dd, copying a whole drive or SD media is a handy tool for cloning one system to another drive including the MBR. If you only want the OS files, rsync can be used to recursively copy an OS file structure, between different mounted drive types, even if it still means the MBR has to be written correctly after, maybe by installing GRUB as a boot loader or using gpart or Gparted to write another correct type.
Rsync is about the best tool for doing incremental backups also, either locally or over network, and encrypted using SSH if required.
There is a great free GUI version for Windows that I use, called Syncback, by 2BrightSparks:
https://www.2brightsparks.com/
For this example, I'm going to clone the directory structure of a working Linux install I have on an external 250GB USB drive to an internal 320GB SATA drive.
A couple of things to remember when copying a working OS, like instantaneous memory processes aren't wanted and won't copy obviously, some of which in Linux are "mounted" in the /proc (process) directory, and network shares should be unmounted also else their contents will be physically copied also. It may be quicker in that case to just run rsync on a required folder by folder basis with a simpler command line, than a more complex one with multiple "--exclude /xxx" folder names.
The MOST important folder to exclude is the new system folder itself that is being copied to, else you get a feedback loop and an infinite copying until the new drive is filled, because the target folder is mounted under the main root / directory!
First, the target drive is prepared as a blank drive, by deleting and reformatting as a new Linux partition, with a swap drive of about 2.5 times memory size, say 5GB (with cfdisk and mkfs.ext4 for example), which is larger than the current OS structure to be copied. I'll assume it is known how to do this.
The basic command structure of rsync is standard, though switch placement is more flexible.
CMD OPTIONS SOURCE_FOLDER DEST_FOLDER e.g.
rsync --dry-run -vr /content /backup
This would run a test copy not actually transfer files - recursively and verbosely, of the files in /content to /backup.
I created a folder called /320GB which is the empty target SATA drive partition, and mounted the physical partition on it:
mount -t ext4 /dev/sdb /320GB
Once mounted, the folder /320GB is the target folder. I need to copy every folder and file that makes up the current working Linux OS, with the exceptions above the target folder itself must not be recopied, and the processes folder /proc is unnecessary as it works for this current system only, and may cause rsync problems. This folder can be added empty to the mirrored directory afterwards for when the new system boots.
The actual command I'm using becomes:
rsync -dry-run -vr / /320GB/ --exclude /320GB/ --exclude /proc/
It is a good habit to always use the -dry-run switch to check the command will do what you want before you run it for real. If all looks right then run without it. Note the -exclude options can be after the destination folder, but the source must always be before the destination obviously so you get the copying in the right direction.
Once the dry run has been done and you are happy it works as you want, go for real:
rsync -vr / /320GB/ --exclude /320GB/ --exclude /proc/
var/www/www2/zones
var/www/www2/grepegsamples_files/
var/www/www2/two-interface_files/
sent 37655478872 bytes received 3163246 bytes 16193782.89 bytes/sec
total size is 37061299540 speedup is 0.98
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1070) [sender=3.0.9]
root@DellLinux:~#
This copied about 37GB of data, and a listing of the original root, then the /320GB folder differences shows:
root@DellLinux:~# ls -l /
total 124
drwxr-xr-x 31 root root 4096 Apr 17 00:04 320GB
drwxr-xr-x 2 root root 4096 Apr 7 20:22 Acer1
drwxr-xr-x 2 root root 4096 Mar 29 02:03 bin
drwxr-xr-x 3 root root 4096 Mar 21 22:02 boot
drwxr-xr-x 14 root root 4096 Mar 22 00:19 debian
drwxr-xr-x 14 root root 3620 Apr 17 00:00 dev
drwxr-xr-x 150 root root 12288 Apr 16 23:56 etc
drwxr-xr-x 3 root root 4096 Mar 21 21:28 home
lrwxrwxrwx 1 root root 30 Mar 21 21:15 initrd.img -> /boot/initrd.img-3.2.0-4-amd64
drwxr-xr-x 17 root root 4096 Apr 8 03:34 lib
drwxr-xr-x 2 root root 4096 Mar 21 21:14 lib64
drwx------ 2 root root 16384 Mar 21 21:13 lost+found
drwxr-xr-x 3 root root 4096 Apr 15 19:02 media
drwxr-xr-x 2 root root 4096 Jan 28 10:53 mnt
drwxr-xr-x 3 root root 4096 Apr 16 18:26 opt
dr-xr-xr-x 128 root root 0 Apr 16 23:56 proc
drwxr-xr-x 2 root root 4096 Mar 31 17:03 Quadra
drwx------ 26 root root 4096 Apr 16 19:56 root
drwxr-xr-x 25 root root 1000 Apr 16 23:58 run
drwxr-xr-x 2 root root 4096 Mar 29 02:03 sbin
drwxr-xr-x 2 root root 4096 Jun 10 2012 selinux
drwxrwxrwx 9 stevee stevee 4096 Apr 12 11:07 share
drwxr-xr-x 2 root root 4096 Mar 21 21:14 srv
drwxr-xr-x 2 root root 4096 Apr 12 16:19 SSDWin7
drwxr-xr-x 2 root root 4096 Mar 24 19:02 Storebird
drwxr-xr-x 13 root root 0 Apr 16 23:56 sys
drwxrwxrwt 10 root root 4096 Apr 17 01:00 tmp
drwxr-xr-x 10 root root 4096 Mar 21 21:14 usr
drwxr-xr-x 12 root root 4096 Mar 23 00:59 var
-rw-r--r-- 1 root root 1276 Apr 13 01:19 varspace.sh
lrwxrwxrwx 1 root root 26 Mar 21 21:15 vmlinuz -> boot/vmlinuz-3.2.0-4-amd64
drwxr-xr-x 2 root root 4096 Apr 11 17:23 W7
root@DellLinux:~#
ls -l /320GB/
total 132
drwxr-xr-x 29 root root 4096 Apr 17 00:02 Acer1
drwx------ 2 root root 4096 Apr 17 00:04 bin
drwx------ 3 root root 4096 Apr 17 00:04 boot
drwx------ 14 root root 4096 Apr 17 00:04 debian
drwx------ 14 root root 4096 Apr 17 00:04 dev
drwx------ 150 root root 12288 Apr 17 00:04 etc
drwx------ 3 root root 4096 Apr 17 00:04 home
drwx------ 17 root root 4096 Apr 17 00:06 lib
drwx------ 2 root root 4096 Apr 17 00:02 lib64
drwx------ 2 root root 16384 Apr 17 00:00 lost+found
drwx------ 3 root root 4096 Apr 17 00:07 media
drwx------ 2 root root 4096 Apr 17 00:02 mnt
drwx------ 3 root root 4096 Apr 17 00:07 opt
drwx------ 2 root root 4096 Apr 17 00:02 Quadra
drwx------ 26 root root 4096 Apr 17 00:07 root
drwx------ 25 root root 4096 Apr 17 00:07 run
drwx------ 2 root root 4096 Apr 17 00:07 sbin
drwx------ 2 root root 4096 Apr 17 00:02 selinux
drwx------ 9 root root 4096 Apr 17 00:15 share
drwx------ 2 root root 4096 Apr 17 00:02 srv
drwx------ 2 root root 4096 Apr 17 00:02 SSDWin7
drwx------ 2 root root 4096 Apr 17 00:04 Storebird
drwx------ 13 root root 4096 Apr 17 00:26 sys
drwx------ 10 root root 4096 Apr 17 00:26 tmp
drwx------ 10 root root 4096 Apr 17 00:26 usr
drwx------ 12 root root 4096 Apr 17 00:40 var
-rw-r--r-- 1 root root 1276 Apr 17 00:04 varspace.sh
drwx------ 2 root root 4096 Apr 17 00:02 W7
Recreating /proc:
mkdir /320GB/proc
and giving it the same permissions:
chmod ugo+rx,ugo-w /320GB/proc/
root@DellLinux:~# ls -al /320GB/proc
dr-xr-xr-x 2 root root 4096 Apr 17 01:07 .
drwxr-xr-x 32 root root 4096 Apr 17 01:07 ..
Ownership is root anyway.
Just to check recursive copy took place:
root@DellLinux:~# ls /320GB/lib
crda libip6tc.so.0.1.0 lsb
discover libipq.so.0.0.0 modprobe.d
firmware libiptc.so.0.0.0 modules
hdparm libnss_mdns4_minimal.so.2 oss-compat
init libnss_mdns4.so.2 security
klibc-2xtYrByCrj5OEwaInv4tMSjej98.so libnss_mdns6_minimal.so.2 systemd
libaudit.so.0.0.0 libnss_mdns6.so.2 terminfo
libauparse.so.0.0.0 libnss_mdns_minimal.so.2 udev
libcryptsetup.so.4.2.0 libnss_mdns.so.2 x86_64-linux-gnu
libip4tc.so.0.1.0 libxtables.so.7.0.0 xtables
That seems ok, so now you have the major task of researching and installing a suitable boot loader like grub2 on the MBR so that the new drive will boot...
Needless to say, I won't be doing a Post on GRUB2 under a ~Cool Command~ heading because it never was cool at all just a complete overly complex headache in my view - particularly for such a useful backup scenario as this...Good Luck with that!
Good How To's seem hard to find also...