{"id":230,"date":"2014-04-17T17:10:23","date_gmt":"2014-04-17T17:10:23","guid":{"rendered":"https:\/\/stevepedwards.today\/DebianAdmin\/?p=230"},"modified":"2014-04-17T17:10:23","modified_gmt":"2014-04-17T17:10:23","slug":"cool-command-3-rsync","status":"publish","type":"post","link":"https:\/\/stevepedwards.today\/DebianAdmin\/cool-command-3-rsync\/","title":{"rendered":"Cool Command #3 Rsync"},"content":{"rendered":"
<\/div>\n

<\/path><\/svg><\/i> \"Loading\"<\/p>\n

<\/div>\n

Cool Cmd #3 Rsync
\n<\/span><\/p>\n

man rsync
\n<\/span><\/p>\n

DESCRIPTION
\n<\/span><\/p>\n

Rsync is a fast and extraordinarily versatile file copying tool. It can copy locally, to\/from another host over any remote shell, or to\/from a remote rsync daemon. It offers a large number of options that control every aspect of its behaviour and permit very flexible specification of the set of files to be copied. It is famous for its delta-transfer algorithm, which reduces the amount of data sent over the network by sending only the differences between the source files and the existing files in the destination. Rsync is widely used for backups and mirroring and as an improved copy command for everyday use.
\n<\/span><\/p>\n

As seen with dd, copying a whole drive or SD media is a handy tool for cloning one system to another drive including the MBR. If you only want the OS files, rsync can be used to recursively copy an OS file structure, between different mounted drive types, even if it still means the MBR has to be written correctly after, maybe by installing GRUB as a boot loader or using gpart or Gparted to write another correct type.
\n<\/span><\/p>\n

Rsync is about the best tool for doing incremental backups also, either locally or over network, and encrypted using SSH if required.
\n<\/span><\/span><\/p>\n

There is a great free GUI version for Windows that I use, called Syncback, by 2BrightSparks:
\n<\/span><\/p>\n

https:\/\/www.2brightsparks.com\/<\/span><\/a>
\n<\/span><\/p>\n

For this example, I'm going to clone the directory structure of a working Linux install I have on an external 250GB USB drive to an internal 320GB SATA drive.
\n<\/span><\/p>\n

A couple of things to remember when copying a working OS, like instantaneous memory processes aren't wanted and won't copy obviously, some of which in Linux are \"mounted\" in the \/proc (process) directory, and network shares should be unmounted also else their contents will be physically copied also. It may be quicker in that case to just run rsync on a required folder by folder basis with a simpler command line, than a more complex one with multiple \"--exclude \/xxx\" folder names.
\n<\/span><\/p>\n

The MOST important folder to exclude is the new system folder itself that is being copied to, else you get a feedback loop and an infinite copying until the new drive is filled, because the target folder is mounted under the main root \/ directory!
\n<\/span><\/p>\n

First, the target drive is prepared as a blank drive, by deleting and reformatting as a new Linux partition, with a swap drive of about 2.5 times memory size, say 5GB (with cfdisk and mkfs.ext4 for example), which is larger than the current OS structure to be copied. I'll assume it is known how to do this.
\n<\/span><\/p>\n

The basic command structure of rsync is standard, though switch placement is more flexible.
\n<\/span><\/p>\n

CMD OPTIONS SOURCE_FOLDER DEST_FOLDER e.g.
\n<\/span><\/p>\n

rsync --dry-run -vr \/content \/backup
\n<\/span><\/p>\n

This would run a test copy not actually transfer files - recursively and verbosely, of the files in \/content to \/backup.
\n<\/span><\/p>\n

I created a folder called \/320GB which is the empty target SATA drive partition, and mounted the physical partition on it:
\n<\/span><\/p>\n

mount -t ext4 \/dev\/sdb \/320GB
\n<\/span><\/p>\n

Once mounted, the folder \/320GB is the target folder. I need to copy every folder and file that makes up the current working Linux OS, with the exceptions above the target folder itself must not be recopied, and the processes folder \/proc is unnecessary as it works for this current system only, and may cause rsync problems. This folder can be added empty to the mirrored directory afterwards for when the new system boots.
\n<\/span><\/p>\n

The actual command I'm using becomes:
\n<\/span><\/p>\n

rsync -dry-run -vr \/ \/320GB\/ --exclude \/320GB\/ --exclude \/proc\/
\n<\/span><\/p>\n

It is a good habit to always use the -dry-run<\/em> switch to check the command will do what you want before you run it for real. If all looks right then run without it. Note the -exclude options can be after the destination folder, but the source must always be before the destination obviously so you get the copying in the right direction.
\n<\/span><\/p>\n

Once the dry run has been done and you are happy it works as you want, go for real:
\n<\/span><\/p>\n

rsync -vr \/ \/320GB\/ --exclude \/320GB\/ --exclude \/proc\/
\n<\/span><\/p>\n

var\/www\/www2\/zones
\n<\/span><\/p>\n

var\/www\/www2\/grepegsamples_files\/
\n<\/span><\/p>\n

var\/www\/www2\/two-interface_files\/
\n<\/span><\/p>\n

sent 37655478872 bytes received 3163246 bytes 16193782.89 bytes\/sec
\n<\/span><\/p>\n

total size is 37061299540 speedup is 0.98
\n<\/span><\/p>\n

rsync error: some files\/attrs were not transferred (see previous errors) (code 23) at main.c(1070) [sender=3.0.9]
\n<\/span><\/p>\n

root@DellLinux:~#<\/span>
\n<\/span><\/span><\/p>\n

This copied about 37GB of data, and a listing of the original root, then the \/320GB folder differences shows:
\n<\/span><\/p>\n

root@DellLinux:~# ls -l \/<\/span><\/p>\n

total 124
\n<\/span><\/p>\n

drwxr-xr-x 31 root root 4096 Apr 17 00:04 320GB
\n<\/span><\/p>\n

drwxr-xr-x 2 root root 4096 Apr 7 20:22 Acer1
\n<\/span><\/p>\n

drwxr-xr-x 2 root root 4096 Mar 29 02:03 bin
\n<\/span><\/p>\n

drwxr-xr-x 3 root root 4096 Mar 21 22:02 boot
\n<\/span><\/p>\n

drwxr-xr-x 14 root root 4096 Mar 22 00:19 debian
\n<\/span><\/p>\n

drwxr-xr-x 14 root root 3620 Apr 17 00:00 dev
\n<\/span><\/p>\n

drwxr-xr-x 150 root root 12288 Apr 16 23:56 etc
\n<\/span><\/p>\n

drwxr-xr-x 3 root root 4096 Mar 21 21:28 home
\n<\/span><\/p>\n

lrwxrwxrwx 1 root root 30 Mar 21 21:15 initrd.img -> \/boot\/initrd.img-3.2.0-4-amd64
\n<\/span><\/p>\n

drwxr-xr-x 17 root root 4096 Apr 8 03:34 lib
\n<\/span><\/p>\n

drwxr-xr-x 2 root root 4096 Mar 21 21:14 lib64
\n<\/span><\/p>\n

drwx------ 2 root root 16384 Mar 21 21:13 lost+found
\n<\/span><\/p>\n

drwxr-xr-x 3 root root 4096 Apr 15 19:02 media
\n<\/span><\/p>\n

drwxr-xr-x 2 root root 4096 Jan 28 10:53 mnt
\n<\/span><\/p>\n

drwxr-xr-x 3 root root 4096 Apr 16 18:26 opt
\n<\/span><\/p>\n

dr-xr-xr-x 128 root root 0 Apr 16 23:56 proc
\n<\/span><\/p>\n

drwxr-xr-x 2 root root 4096 Mar 31 17:03 Quadra
\n<\/span><\/p>\n

drwx------ 26 root root 4096 Apr 16 19:56 root
\n<\/span><\/p>\n

drwxr-xr-x 25 root root 1000 Apr 16 23:58 run
\n<\/span><\/p>\n

drwxr-xr-x 2 root root 4096 Mar 29 02:03 sbin
\n<\/span><\/p>\n

drwxr-xr-x 2 root root 4096 Jun 10 2012 selinux
\n<\/span><\/p>\n

drwxrwxrwx 9 stevee stevee 4096 Apr 12 11:07 share
\n<\/span><\/p>\n

drwxr-xr-x 2 root root 4096 Mar 21 21:14 srv
\n<\/span><\/p>\n

drwxr-xr-x 2 root root 4096 Apr 12 16:19 SSDWin7
\n<\/span><\/p>\n

drwxr-xr-x 2 root root 4096 Mar 24 19:02 Storebird
\n<\/span><\/p>\n

drwxr-xr-x 13 root root 0 Apr 16 23:56 sys
\n<\/span><\/p>\n

drwxrwxrwt 10 root root 4096 Apr 17 01:00 tmp
\n<\/span><\/p>\n

drwxr-xr-x 10 root root 4096 Mar 21 21:14 usr
\n<\/span><\/p>\n

drwxr-xr-x 12 root root 4096 Mar 23 00:59 var
\n<\/span><\/p>\n

-rw-r--r-- 1 root root 1276 Apr 13 01:19 varspace.sh
\n<\/span><\/p>\n

lrwxrwxrwx 1 root root 26 Mar 21 21:15 vmlinuz -> boot\/vmlinuz-3.2.0-4-amd64
\n<\/span><\/p>\n

drwxr-xr-x 2 root root 4096 Apr 11 17:23 W7<\/span><\/p>\n

root@DellLinux:~#
\n<\/span>ls -l \/320GB\/<\/span>
\n<\/span><\/p>\n

total 132
\n<\/span><\/p>\n

drwxr-xr-x 29 root root 4096 Apr 17 00:02 Acer1
\n<\/span><\/p>\n

drwx------ 2 root root 4096 Apr 17 00:04 bin
\n<\/span><\/p>\n

drwx------ 3 root root 4096 Apr 17 00:04 boot
\n<\/span><\/p>\n

drwx------ 14 root root 4096 Apr 17 00:04 debian
\n<\/span><\/p>\n

drwx------ 14 root root 4096 Apr 17 00:04 dev
\n<\/span><\/p>\n

drwx------ 150 root root 12288 Apr 17 00:04 etc
\n<\/span><\/p>\n

drwx------ 3 root root 4096 Apr 17 00:04 home
\n<\/span><\/p>\n

drwx------ 17 root root 4096 Apr 17 00:06 lib
\n<\/span><\/p>\n

drwx------ 2 root root 4096 Apr 17 00:02 lib64
\n<\/span><\/p>\n

drwx------ 2 root root 16384 Apr 17 00:00 lost+found
\n<\/span><\/p>\n

drwx------ 3 root root 4096 Apr 17 00:07 media
\n<\/span><\/p>\n

drwx------ 2 root root 4096 Apr 17 00:02 mnt
\n<\/span><\/p>\n

drwx------ 3 root root 4096 Apr 17 00:07 opt
\n<\/span><\/p>\n

drwx------ 2 root root 4096 Apr 17 00:02 Quadra
\n<\/span><\/p>\n

drwx------ 26 root root 4096 Apr 17 00:07 root
\n<\/span><\/p>\n

drwx------ 25 root root 4096 Apr 17 00:07 run
\n<\/span><\/p>\n

drwx------ 2 root root 4096 Apr 17 00:07 sbin
\n<\/span><\/p>\n

drwx------ 2 root root 4096 Apr 17 00:02 selinux
\n<\/span><\/p>\n

drwx------ 9 root root 4096 Apr 17 00:15 share
\n<\/span><\/p>\n

drwx------ 2 root root 4096 Apr 17 00:02 srv
\n<\/span><\/p>\n

drwx------ 2 root root 4096 Apr 17 00:02 SSDWin7
\n<\/span><\/p>\n

drwx------ 2 root root 4096 Apr 17 00:04 Storebird
\n<\/span><\/p>\n

drwx------ 13 root root 4096 Apr 17 00:26 sys
\n<\/span><\/p>\n

drwx------ 10 root root 4096 Apr 17 00:26 tmp
\n<\/span><\/p>\n

drwx------ 10 root root 4096 Apr 17 00:26 usr
\n<\/span><\/p>\n

drwx------ 12 root root 4096 Apr 17 00:40 var
\n<\/span><\/p>\n

-rw-r--r-- 1 root root 1276 Apr 17 00:04 varspace.sh
\n<\/span><\/p>\n

drwx------ 2 root root 4096 Apr 17 00:02 W7<\/span><\/p>\n

Recreating \/proc:<\/p>\n

mkdir \/320GB\/proc
\n<\/span><\/p>\n

and giving it the same permissions:<\/p>\n

chmod ugo+rx,ugo-w \/320GB\/proc\/
\n<\/span><\/p>\n

root@DellLinux:~# ls -al \/320GB\/proc
\n<\/span><\/p>\n

dr-xr-xr-x 2 root root 4096 Apr 17 01:07 .
\n<\/span><\/p>\n

drwxr-xr-x 32 root root 4096 Apr 17 01:07 ..
\n<\/span><\/p>\n

Ownership is root anyway.<\/p>\n

Just to check recursive copy took place:<\/p>\n

root@DellLinux:~# ls \/320GB\/lib<\/span><\/p>\n

crda libip6tc.so.0.1.0 lsb
\n<\/span><\/p>\n

discover libipq.so.0.0.0 modprobe.d
\n<\/span><\/p>\n

firmware libiptc.so.0.0.0 modules
\n<\/span><\/p>\n

hdparm libnss_mdns4_minimal.so.2 oss-compat
\n<\/span><\/p>\n

init libnss_mdns4.so.2 security
\n<\/span><\/p>\n

klibc-2xtYrByCrj5OEwaInv4tMSjej98.so libnss_mdns6_minimal.so.2 systemd
\n<\/span><\/p>\n

libaudit.so.0.0.0 libnss_mdns6.so.2 terminfo
\n<\/span><\/p>\n

libauparse.so.0.0.0 libnss_mdns_minimal.so.2 udev
\n<\/span><\/p>\n

libcryptsetup.so.4.2.0 libnss_mdns.so.2 x86_64-linux-gnu
\n<\/span><\/p>\n

libip4tc.so.0.1.0 libxtables.so.7.0.0 xtables
\n<\/span><\/p>\n

That seems ok, so now you have the major task of researching and installing a suitable boot loader like grub2 on the MBR so that the new drive will boot...<\/p>\n

Needless to say, I won't be doing a Post on GRUB2 under a ~Cool Command~ heading because it never was cool at all just a complete overly complex headache in my view - particularly for such a useful backup scenario as this...Good Luck with that!<\/p>\n

Good How To's seem hard to find also...<\/p>\n","protected":false},"excerpt":{"rendered":"

<\/div>\n

<\/path><\/svg><\/i> \"Loading\"<\/p>\n

<\/div>\n

Cool Cmd #3 Rsync man rsync DESCRIPTION Rsync is a fast and extraordinarily versatile file copying tool. It can copy locally, to\/from another host over any remote shell, or to\/from a remote rsync daemon. It offers a large number of options that control every aspect of its behaviour and permit very flexible specification of the ...\u00a0 Cool Command #3 Rsync<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[2],"tags":[],"class_list":["post-230","post","type-post","status-publish","format-standard","hentry","category-uncategorized"],"a3_pvc":{"activated":true,"total_views":1,"today_views":0},"_links":{"self":[{"href":"https:\/\/stevepedwards.today\/DebianAdmin\/wp-json\/wp\/v2\/posts\/230","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/stevepedwards.today\/DebianAdmin\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/stevepedwards.today\/DebianAdmin\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/stevepedwards.today\/DebianAdmin\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/stevepedwards.today\/DebianAdmin\/wp-json\/wp\/v2\/comments?post=230"}],"version-history":[{"count":0,"href":"https:\/\/stevepedwards.today\/DebianAdmin\/wp-json\/wp\/v2\/posts\/230\/revisions"}],"wp:attachment":[{"href":"https:\/\/stevepedwards.today\/DebianAdmin\/wp-json\/wp\/v2\/media?parent=230"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/stevepedwards.today\/DebianAdmin\/wp-json\/wp\/v2\/categories?post=230"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/stevepedwards.today\/DebianAdmin\/wp-json\/wp\/v2\/tags?post=230"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}