{"id":146,"date":"2014-04-09T17:41:20","date_gmt":"2014-04-09T17:41:20","guid":{"rendered":"https:\/\/stevepedwards.today\/DebianAdmin\/?p=146"},"modified":"2022-09-24T15:30:41","modified_gmt":"2022-09-24T14:30:41","slug":"cool-command-1-dd-progress","status":"publish","type":"post","link":"https:\/\/stevepedwards.today\/DebianAdmin\/cool-command-1-dd-progress\/","title":{"rendered":"Cool Command #1 &#8211; DD with Progress"},"content":{"rendered":"<div class=\"pvc_clear\"><\/div>\n<p id=\"pvc_stats_146\" class=\"pvc_stats all  \" data-element-id=\"146\" style=\"\"><i class=\"pvc-stats-icon medium\" aria-hidden=\"true\"><svg aria-hidden=\"true\" focusable=\"false\" data-prefix=\"far\" data-icon=\"chart-bar\" role=\"img\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\" viewBox=\"0 0 512 512\" class=\"svg-inline--fa fa-chart-bar fa-w-16 fa-2x\"><path fill=\"currentColor\" d=\"M396.8 352h22.4c6.4 0 12.8-6.4 12.8-12.8V108.8c0-6.4-6.4-12.8-12.8-12.8h-22.4c-6.4 0-12.8 6.4-12.8 12.8v230.4c0 6.4 6.4 12.8 12.8 12.8zm-192 0h22.4c6.4 0 12.8-6.4 12.8-12.8V140.8c0-6.4-6.4-12.8-12.8-12.8h-22.4c-6.4 0-12.8 6.4-12.8 12.8v198.4c0 6.4 6.4 12.8 12.8 12.8zm96 0h22.4c6.4 0 12.8-6.4 12.8-12.8V204.8c0-6.4-6.4-12.8-12.8-12.8h-22.4c-6.4 0-12.8 6.4-12.8 12.8v134.4c0 6.4 6.4 12.8 12.8 12.8zM496 400H48V80c0-8.84-7.16-16-16-16H16C7.16 64 0 71.16 0 80v336c0 17.67 14.33 32 32 32h464c8.84 0 16-7.16 16-16v-16c0-8.84-7.16-16-16-16zm-387.2-48h22.4c6.4 0 12.8-6.4 12.8-12.8v-70.4c0-6.4-6.4-12.8-12.8-12.8h-22.4c-6.4 0-12.8 6.4-12.8 12.8v70.4c0 6.4 6.4 12.8 12.8 12.8z\" class=\"\"><\/path><\/svg><\/i> <img loading=\"lazy\" decoding=\"async\" width=\"16\" height=\"16\" alt=\"Loading\" src=\"https:\/\/stevepedwards.today\/DebianAdmin\/wp-content\/plugins\/page-views-count\/ajax-loader-2x.gif\" border=0 \/><\/p>\n<div class=\"pvc_clear\"><\/div>\n<p>(Amended 18 Sept 2015)<\/p>\n<p><span style=\"color: #0000ff;\">sudo dd if=\/dev\/sdc of=\/dev\/sdb bs=512 conv=sync,noerror status=progress<\/span><\/p>\n<p>For an insight into using the best Block Size values when copying to\/from different media types, and using DD in general, read this link FIRST - see the quotes to get some perspective and optimise your copying time:<\/p>\n<p><a href=\"https:\/\/blog.tdg5.com\/tuning-dd-block-size\/\" target=\"_blank\" rel=\"noopener\">https:\/\/blog.tdg5.com\/tuning-dd-block-size\/<\/a><\/p>\n<p><em>\"Let's start off with a few tests writing out to a HDD:<\/em><\/p>\n<ul>\n<li><em>Reading from \/dev\/zero and writing out to a HDD with the default block size of 512 bytes yields a throughput of 10.9 MB\/s. At that rate, writing 1TB of data would take about 96,200 seconds or just north of <strong>26 hours<\/strong>.<\/em><\/li>\n<li><em>Reading from \/dev\/zero and writing out to a HDD with the Eug-Lug suggested block size of 64K yields a throughput of 108 MB\/s. At that rate, writing 1TB of data would take 9,709 seconds or about <strong>2.7<\/strong> hours to complete. This is a huge improvement, nearly an order of magnitude, over the default block size of 512 bytes.<\/em><\/li>\n<li><em>Reading from \/dev\/zero and writing out to a HDD with a more optimal block size of 512K yields a throughput of 131 MB\/s. At that rate, writing 1TB of data would take about 8,004 seconds or about <strong>2.2 hours<\/strong>. Though not as pronounced a difference, this is even faster than the Eug-Lug suggestion and is more than a full order of magnitude faster than the default block size of 512 bytes.\"<\/em><\/li>\n<li>--<\/li>\n<\/ul>\n<p>You may want to run similar tests on your own kit - say a CD size ISO.iso to and from an ISO.img etc. - at the highest rate you find from the tests below.<\/p>\n<p>As the article states - it's all very dependent on YOUR OS, YOUR memory, YOUR drive types (IDE, SATA, SDRAM USB2, USB3, SSD etc.) and there will be different optimums for each case.<\/p>\n<p>As a rough test for CPU\/mem performance near the lower default of bs=512 (bytes), try\u00a0the lower ranges:<\/p>\n<p>Mint5630 stevee # <span style=\"color: #0000ff;\">dd if=\/dev\/zero of=\/dev\/null bs=512 count=10<\/span><br \/>\n10+0 records in<br \/>\n10+0 records out<br \/>\n5120 bytes (5.1 kB) copied, 0.000136822 s, <span style=\"color: #ff0000;\">37.4 MB\/s<\/span><\/p>\n<p>Mint5630 stevee # <span style=\"color: #0000ff;\">dd if=\/dev\/zero of=\/dev\/null bs=1K count=10<\/span><br \/>\n10+0 records in<br \/>\n10+0 records out<br \/>\n10240 bytes (10 kB) copied, 0.000248849 s, <span style=\"color: #ff0000;\">41.1 MB\/s<\/span><br \/>\nMint5630 stevee # <span style=\"color: #0000ff;\">dd if=\/dev\/zero of=\/dev\/null bs=2K count=10<\/span><br \/>\n10+0 records in<br \/>\n10+0 records out<br \/>\n20480 bytes (20 kB) copied, 0.000159098 s, <span style=\"color: #ff0000;\">129 MB\/s<\/span><br \/>\nMint5630 stevee # <span style=\"color: #0000ff;\">dd if=\/dev\/zero of=\/dev\/null bs=4K count=10<\/span><br \/>\n10+0 records in<br \/>\n10+0 records out<br \/>\n40960 bytes (41 kB) copied, 0.000250597 s, <span style=\"color: #ff0000;\">163 MB\/s<\/span><br \/>\nMint5630 stevee # <span style=\"color: #0000ff;\">dd if=\/dev\/zero of=\/dev\/null bs=8K count=10<\/span><br \/>\n10+0 records in<br \/>\n10+0 records out<br \/>\n81920 bytes (82 kB) copied, 0.000203375 s, <span style=\"color: #ff0000;\">403 MB\/s<\/span><br \/>\nMint5630 stevee # <span style=\"color: #0000ff;\">dd if=\/dev\/zero of=\/dev\/null bs=16K count=10<\/span><br \/>\n10+0 records in<br \/>\n10+0 records out<br \/>\n163840 bytes (164 kB) copied, 0.000283561 s, <span style=\"color: #ff0000;\">578 MB\/s<\/span><br \/>\nMint5630 stevee # <span style=\"color: #0000ff;\"><strong>dd if=\/dev\/zero of=\/dev\/null bs=32K count=10<\/strong><\/span><br \/>\n10+0 records in<br \/>\n10+0 records out<br \/>\n<strong>327680 bytes (328 kB) copied, 0.000219022 s, <span style=\"color: #ff0000;\">1.5 GB\/s<\/span><\/strong><br \/>\nMint5630 stevee # <span style=\"color: #0000ff;\">dd if=\/dev\/zero of=\/dev\/null bs=64K count=10<\/span><br \/>\n10+0 records in<br \/>\n10+0 records out<br \/>\n655360 bytes (655 kB) copied, 0.00045279 s, <span style=\"color: #ff0000;\">1.4 GB\/s<\/span><\/p>\n<p>You see the peak speed is at BS=32K for those tests. It may be even better at larger BS again as below.<\/p>\n<p>This also shows with newer PCs, that the old idea of memory pages of 4k being used for the BS does not necessarily relate optimally, just that binary multiples (of 2k) are probably a better choice than say bs = 3k, but experiment yourself.<\/p>\n<p>If you experiment, you may find that these numbers aren't even consistent for the same block size, when repeated, as it depends what else the OS may be doing, as running the same command over again (UP arrow!) yields very different results, so the best you can infer is the average of say, at least 5 runs of the up arrow, as below! I'd take these averages at about 2.6Gb\/s. The average, not peaks, is what counts when the large file transfer starts, after all.<\/p>\n<p>Mint5630 stevee # <span style=\"color: #0000ff;\">dd if=\/dev\/zero of=\/dev\/null bs=512K count=10<\/span><br \/>\n10+0 records in<br \/>\n10+0 records out<br \/>\n5242880 bytes (5.2 MB) copied, 0.0016244 s, <span style=\"color: #ff0000;\">3.2 GB\/s<\/span><br \/>\nMint5630 stevee # <span style=\"color: #0000ff;\">dd if=\/dev\/zero of=\/dev\/null bs=512K count=10<\/span><br \/>\n10+0 records in<br \/>\n10+0 records out<br \/>\n5242880 bytes (5.2 MB) copied, 0.0019947 s, <span style=\"color: #ff0000;\">2.6 GB\/s<\/span><br \/>\nMint5630 stevee #<span style=\"color: #0000ff;\"> dd if=\/dev\/zero of=\/dev\/null bs=512K count=10<\/span><br \/>\n10+0 records in<br \/>\n10+0 records out<br \/>\n5242880 bytes (5.2 MB) copied, 0.00194826 s,<span style=\"color: #ff0000;\"> 2.7 GB\/s<\/span><br \/>\nMint5630 stevee # <span style=\"color: #0000ff;\">dd if=\/dev\/zero of=\/dev\/null bs=512K count=10<\/span><br \/>\n10+0 records in<br \/>\n10+0 records out<br \/>\n5242880 bytes (5.2 MB) copied, 0.00196851 s, <span style=\"color: #ff0000;\">2.7 GB\/s<\/span><br \/>\nMint5630 stevee # <span style=\"color: #0000ff;\">dd if=\/dev\/zero of=\/dev\/null bs=512K count=10<\/span><br \/>\n10+0 records in<br \/>\n10+0 records out<br \/>\n5242880 bytes (5.2 MB) copied, 0.00204192 s, <span style=\"color: #ff0000;\">2.6 GB\/s<\/span><\/p>\n<p>That above was on dual core Intel T5500 laptop with 2GB RAM, running Mint i686:<\/p>\n<p><span style=\"color: #0000ff;\">uname -a<\/span><br \/>\n<span style=\"color: #ff0000;\">Linux Mint5630 3.16.0-38-generic #52~14.04.1-Ubuntu SMP Fri May 8 09:44:48 UTC 2015 i686 i686 i686 GNU\/Linux<\/span><\/p>\n<p>As you see below, the same 512k BS run on a different, (MORE POWERFUL !) 64 bit dual core PC with 6 GB RAM, but running the \"imperfect\"\u00a0OS for it ideally, gives very different results at less\u00a0speed:<\/p>\n<p>DELLMINT stevee # <span style=\"color: #0000ff;\">uname -a<\/span><br \/>\n<span style=\"color: #ff0000;\">Linux DELLMINT 3.16.0-38-generic #52~14.04.1-Ubuntu SMP Fri May 8 09:43:57 UTC 2015 x86_64 x86_64 x86_64 GNU\/Linux<\/span><\/p>\n<p>The better choice for this PC is BS=512k, as 1M drops off again:<br \/>\nDELLMINT stevee # <span style=\"color: #0000ff;\">dd if=\/dev\/zero of=\/dev\/null bs=512K count=10<\/span><br \/>\n10+0 records in<br \/>\n10+0 records out<br \/>\n5242880 bytes (5.2 MB) copied, 0.00306584 s, <span style=\"color: #ff0000;\">1.7 GB\/s<\/span><br \/>\nDELLMINT stevee # <span style=\"color: #0000ff;\">dd if=\/dev\/zero of=\/dev\/null bs=512K count=10<\/span><br \/>\n10+0 records in<br \/>\n10+0 records out<br \/>\n5242880 bytes (5.2 MB) copied, 0.00307384 s, <span style=\"color: #ff0000;\">1.7 GB\/s<\/span><br \/>\nDELLMINT stevee # <span style=\"color: #0000ff;\">dd if=\/dev\/zero of=\/dev\/null bs=512K count=10<\/span><br \/>\n10+0 records in<br \/>\n10+0 records out<br \/>\n5242880 bytes (5.2 MB) copied, 0.00295928 s,<span style=\"color: #ff0000;\"> 1.8 GB\/s<\/span><br \/>\nDELLMINT stevee # <span style=\"color: #0000ff;\">dd if=\/dev\/zero of=\/dev\/null bs=512K count=10<\/span><br \/>\n10+0 records in<br \/>\n10+0 records out<br \/>\n5242880 bytes (5.2 MB) copied, 0.0030166 s, <span style=\"color: #ff0000;\">1.7 GB\/s<\/span><br \/>\nDELLMINT stevee # <span style=\"color: #0000ff;\">dd if=\/dev\/zero of=\/dev\/null bs=512K count=10<\/span><br \/>\n10+0 records in<br \/>\n10+0 records out<br \/>\n5242880 bytes (5.2 MB) copied, 0.00292228 s, <span style=\"color: #ff0000;\">1.8 GB\/s<\/span><br \/>\nDELLMINT stevee # <span style=\"color: #0000ff;\">dd if=\/dev\/zero of=\/dev\/null bs=512K count=10<\/span><br \/>\n10+0 records in<br \/>\n10+0 records out<br \/>\n5242880 bytes (5.2 MB) copied, 0.00302872 s, <span style=\"color: #ff0000;\">1.7 GB\/s<\/span><\/p>\n<p>So in the real world, what are the results of say cloning a Sandisk USB 3 pendrive of 64GB size, of a Mint OS?<\/p>\n<p>It took about 94 mins\u00a0at bs=1k.<\/p>\n<p>Running it again at bs =512k, as the optimum found for that PC above gave:<\/p>\n<p><span style=\"color: #0000ff;\">dd if=\/dev\/sdh of=\/Quadra\/AMDA8Pen2.img bs=512k<\/span><br \/>\n<span style=\"color: #ff0000;\">120704+0 records in<\/span><br \/>\n<span style=\"color: #ff0000;\"> 120704+0 records out<\/span><br \/>\n<span style=\"color: #ff0000;\"> 63283658752 bytes (63 GB) copied, <strong>4671.51 s<\/strong>, 13.5 MB\/s<\/span><\/p>\n<p>or 4671\/60s = 77mins<\/p>\n<p>which is not that much better anyway, but saved 20 mins from the output, BUT that time is incorrect anyway, as real time for me was 1hr25 or 85 mins.<\/p>\n<p><span style=\"color: #ffffff;\">Another real\u00a0example; I recently copied a 160GB SATA drive to an external USB2 disk that took 30962.4 secs; nearly 9 hrs:<\/span><\/p>\n<p>$ <span style=\"color: #0000ff;\">sudo dd if=\/dev\/sdc of=\/dev\/sdh<\/span><br \/>\n<span style=\"color: #ff0000;\">[sudo] password for stevee: <\/span><br \/>\n<span style=\"color: #ff0000;\">312581808+0 records in<\/span><br \/>\n<span style=\"color: #ff0000;\">312581808+0 records out<\/span><br \/>\n<span style=\"color: #ff0000;\">160041885696 bytes (160 GB) copied, <strong>30962.4 s,<\/strong> 5.2 MB\/s<\/span><\/p>\n<p>-----------<\/p>\n<p>So, back to DD itself<\/p>\n<p><span style=\"color: #0000ff;\">man dd<\/span><\/p>\n<p><span style=\"color: #ff0000;\">NAME<\/span><br \/>\n<span style=\"color: #ff0000;\"> dd - convert and copy a file<\/span><\/p>\n<p><span style=\"color: #ff0000;\">SYNOPSIS<\/span><br \/>\n<span style=\"color: #ff0000;\"> dd [OPERAND]...<\/span><br \/>\n<span style=\"color: #ff0000;\"> dd OPTION<\/span><\/p>\n<p><span style=\"color: #ff0000;\">DESCRIPTION<\/span><br \/>\n<span style=\"color: #ff0000;\"> Copy a file, converting and formatting according to the operands.<\/span><\/p>\n<p><span style=\"color: #ffffff;\">DD is a very effective tool to copy\/clone a device's file system byte for byte - from whatever - a whole drive etc. and so can clone your OS or other file system as backup.\u00a0<\/span><\/p>\n<p>It's default block size is 512 bytes for a good reason, so it can copy a boot sector and would be less prone to copying read errors from the source, as it is copying smaller chunks of data at a time.<\/p>\n<p><span style=\"color: #ffffff;\">Here is the man dd text file:<\/span><br \/>\n<span style=\"color: #339966;\"><a href=\"https:\/\/stevepedwards.today\/DebianAdmin\/wp-content\/uploads\/2014\/04\/manDD.txt\"><span style=\"color: #339966;\">manDD.txt<\/span><\/a><\/span><\/p>\n<p>For DD for Windows, see here:<\/p>\n<p><span style=\"color: #339966;\"><a href=\"https:\/\/uranus.chrysocome.net\/linux\/rawwrite\/dd-old.htm\" target=\"_blank\" rel=\"noopener\"><span style=\"color: #339966;\">https:\/\/uranus.chrysocome.net\/linux\/rawwrite\/dd-old.htm<\/span><\/a><\/span><\/p>\n","protected":false},"excerpt":{"rendered":"<div class=\"pvc_clear\"><\/div>\n<p id=\"pvc_stats_146\" class=\"pvc_stats all  \" data-element-id=\"146\" style=\"\"><i class=\"pvc-stats-icon medium\" aria-hidden=\"true\"><svg aria-hidden=\"true\" focusable=\"false\" data-prefix=\"far\" data-icon=\"chart-bar\" role=\"img\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\" viewBox=\"0 0 512 512\" class=\"svg-inline--fa fa-chart-bar fa-w-16 fa-2x\"><path fill=\"currentColor\" d=\"M396.8 352h22.4c6.4 0 12.8-6.4 12.8-12.8V108.8c0-6.4-6.4-12.8-12.8-12.8h-22.4c-6.4 0-12.8 6.4-12.8 12.8v230.4c0 6.4 6.4 12.8 12.8 12.8zm-192 0h22.4c6.4 0 12.8-6.4 12.8-12.8V140.8c0-6.4-6.4-12.8-12.8-12.8h-22.4c-6.4 0-12.8 6.4-12.8 12.8v198.4c0 6.4 6.4 12.8 12.8 12.8zm96 0h22.4c6.4 0 12.8-6.4 12.8-12.8V204.8c0-6.4-6.4-12.8-12.8-12.8h-22.4c-6.4 0-12.8 6.4-12.8 12.8v134.4c0 6.4 6.4 12.8 12.8 12.8zM496 400H48V80c0-8.84-7.16-16-16-16H16C7.16 64 0 71.16 0 80v336c0 17.67 14.33 32 32 32h464c8.84 0 16-7.16 16-16v-16c0-8.84-7.16-16-16-16zm-387.2-48h22.4c6.4 0 12.8-6.4 12.8-12.8v-70.4c0-6.4-6.4-12.8-12.8-12.8h-22.4c-6.4 0-12.8 6.4-12.8 12.8v70.4c0 6.4 6.4 12.8 12.8 12.8z\" class=\"\"><\/path><\/svg><\/i> <img loading=\"lazy\" decoding=\"async\" width=\"16\" height=\"16\" alt=\"Loading\" src=\"https:\/\/stevepedwards.today\/DebianAdmin\/wp-content\/plugins\/page-views-count\/ajax-loader-2x.gif\" border=0 \/><\/p>\n<div class=\"pvc_clear\"><\/div>\n<p>(Amended 18 Sept 2015) sudo dd if=\/dev\/sdc of=\/dev\/sdb bs=512 conv=sync,noerror status=progress For an insight into using the best Block Size values when copying to\/from different media types, and using DD in general, read this link FIRST - see the quotes to get some perspective and optimise your copying time: https:\/\/blog.tdg5.com\/tuning-dd-block-size\/ \"Let's start off with a <a href=\"https:\/\/stevepedwards.today\/DebianAdmin\/cool-command-1-dd-progress\/\" class=\"more-link\">...<span class=\"screen-reader-text\">\u00a0 Cool Command #1 &#8211; DD with Progress<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[2],"tags":[],"class_list":["post-146","post","type-post","status-publish","format-standard","hentry","category-uncategorized"],"a3_pvc":{"activated":true,"total_views":2,"today_views":0},"_links":{"self":[{"href":"https:\/\/stevepedwards.today\/DebianAdmin\/wp-json\/wp\/v2\/posts\/146","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/stevepedwards.today\/DebianAdmin\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/stevepedwards.today\/DebianAdmin\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/stevepedwards.today\/DebianAdmin\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/stevepedwards.today\/DebianAdmin\/wp-json\/wp\/v2\/comments?post=146"}],"version-history":[{"count":4,"href":"https:\/\/stevepedwards.today\/DebianAdmin\/wp-json\/wp\/v2\/posts\/146\/revisions"}],"predecessor-version":[{"id":8976,"href":"https:\/\/stevepedwards.today\/DebianAdmin\/wp-json\/wp\/v2\/posts\/146\/revisions\/8976"}],"wp:attachment":[{"href":"https:\/\/stevepedwards.today\/DebianAdmin\/wp-json\/wp\/v2\/media?parent=146"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/stevepedwards.today\/DebianAdmin\/wp-json\/wp\/v2\/categories?post=146"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/stevepedwards.today\/DebianAdmin\/wp-json\/wp\/v2\/tags?post=146"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}