{"id":3315,"date":"2016-07-04T02:21:32","date_gmt":"2016-07-04T01:21:32","guid":{"rendered":"https:\/\/stevepedwards.today\/DebianAdmin\/?p=3315"},"modified":"2018-01-28T20:04:01","modified_gmt":"2018-01-28T20:04:01","slug":"experiment-with-pipes-redirection-command-substitution-and-variable-expansion","status":"publish","type":"post","link":"https:\/\/stevepedwards.today\/DebianAdmin\/experiment-with-pipes-redirection-command-substitution-and-variable-expansion\/","title":{"rendered":"Experiment with Pipes, Redirection, Command Substitution and Variable Expansion"},"content":{"rendered":"<div class=\"pvc_clear\"><\/div>\n<p id=\"pvc_stats_3315\" class=\"pvc_stats all  \" data-element-id=\"3315\" style=\"\"><i class=\"pvc-stats-icon medium\" aria-hidden=\"true\"><svg aria-hidden=\"true\" focusable=\"false\" data-prefix=\"far\" data-icon=\"chart-bar\" role=\"img\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\" viewBox=\"0 0 512 512\" class=\"svg-inline--fa fa-chart-bar fa-w-16 fa-2x\"><path fill=\"currentColor\" d=\"M396.8 352h22.4c6.4 0 12.8-6.4 12.8-12.8V108.8c0-6.4-6.4-12.8-12.8-12.8h-22.4c-6.4 0-12.8 6.4-12.8 12.8v230.4c0 6.4 6.4 12.8 12.8 12.8zm-192 0h22.4c6.4 0 12.8-6.4 12.8-12.8V140.8c0-6.4-6.4-12.8-12.8-12.8h-22.4c-6.4 0-12.8 6.4-12.8 12.8v198.4c0 6.4 6.4 12.8 12.8 12.8zm96 0h22.4c6.4 0 12.8-6.4 12.8-12.8V204.8c0-6.4-6.4-12.8-12.8-12.8h-22.4c-6.4 0-12.8 6.4-12.8 12.8v134.4c0 6.4 6.4 12.8 12.8 12.8zM496 400H48V80c0-8.84-7.16-16-16-16H16C7.16 64 0 71.16 0 80v336c0 17.67 14.33 32 32 32h464c8.84 0 16-7.16 16-16v-16c0-8.84-7.16-16-16-16zm-387.2-48h22.4c6.4 0 12.8-6.4 12.8-12.8v-70.4c0-6.4-6.4-12.8-12.8-12.8h-22.4c-6.4 0-12.8 6.4-12.8 12.8v70.4c0 6.4 6.4 12.8 12.8 12.8z\" class=\"\"><\/path><\/svg><\/i> <img loading=\"lazy\" decoding=\"async\" width=\"16\" height=\"16\" alt=\"Loading\" src=\"https:\/\/stevepedwards.today\/DebianAdmin\/wp-content\/plugins\/page-views-count\/ajax-loader-2x.gif\" border=0 \/><\/p>\n<div class=\"pvc_clear\"><\/div>\n<p><iframe loading=\"lazy\" src=\"https:\/\/www.youtube.com\/embed\/e-v0Abq95EU?autoplay=1&amp;version=3&amp;loop=1&amp;playlist=e-v0Abq95EU\" width=\"560\" height=\"315\" frameborder=\"0\" align=\"left\" allowfullscreen=\"allowfullscreen\"><\/iframe><\/p>\n<p>After you read this article, I'll let you think of interesting ways to use the above video example of piping data between terminals into other programs for immediate processing...if the commands are not clear; term 1 creates a fifo pipe in my home dir:<\/p>\n<p><span style=\"color: #0000ff;\">mkfifo fifo_pipe<\/span><\/p>\n<p>term 2 cats that pipe to receive what comes out of that fifo_pipe then pipes that output directly into python :<\/p>\n<p><span style=\"color: #0000ff;\">cat fifo_pipe | python<\/span><\/p>\n<p>A python source file is then streamed\u00a0into term 1, and the program results are run and output in term 2;<\/p>\n<p><span style=\"color: #0000ff;\">cat Downloads\/sieve1.py &gt;&gt; fifo_pipe<\/span><\/p>\n<p>You REALLY have to understand the fundamentals first by playing with these commands yourself; they may not do what you expect. The examples are trial and error-ed to be sure what happens, with reference to Unix:A DB Approach many things have changed from older versions file safety for accidental overwrites for example.<\/p>\n<p>There are 3 main data channels called <i><b>file<\/b><\/i><i><b> <\/b><\/i><i><b>descriptors<\/b><\/i> the shell uses for passing data to and from files, screen and commands: Standard Input, Standard Output and Standard Error, numbered 0,1 and 2 resp.<\/p>\n<p>Certain commands can take input from a file and operate on it in some way. As all data on Unix like systems is a file of some sort containing a byte stream of one form or other (though an empty file has no bytes to stream), this byte stream can be the input to these certain commands, which the command (usually) performs an operation of some sort on that stream, then outputs it (usually) as another byte stream.<\/p>\n<p>For example, cat can read a byte stream from standard input, which is data from the keyboard by default, then outputs it, unchanged, to standard output, which is by default, the terminal.<\/p>\n<p><span style=\"color: #0000ff;\">$ cat <span style=\"color: #ffffff;\">(Rtn)<\/span><\/span><\/p>\n<p><span style=\"color: #0000ff;\">this is the input stream and future output stream<span style=\"color: #ffffff;\">(Rtn)<\/span><\/span><\/p>\n<p><span style=\"color: #ff0000;\">this is the input stream and future output stream <span style=\"color: #ffffff;\">(CrtlD)<\/span><\/span><\/p>\n<p><span style=\"color: #ffffff;\">What is not so obvious at this point, is that the input from the keys are also being split or tee'd off to standard output at the same time whilst being buffered in a memory file somewhere by the shell, so you can read what you are typing, before cat actually gets it to spit it back out again. It is the shell that creates the memory buffer and handles the user input first, (while the cat command itself neither knows nor cares about this data handling) before passing it to the cat command itself via Standard Input (0), that cat then outputs to the terminal via Std Output (1). If input to cat is directed from a non-existent file then the error msg is output to the terminal via Std Error (2). <\/span><\/p>\n<p><span style=\"color: #ffffff;\">The channels used may be shown if numbered appropriately with a chevron and the &amp; in some cases <strong><i>but behaviour like file creation with or without an explicit descriptor varies <\/i><i>depending on the command and redirection type attempted. <\/i><\/strong><\/span><\/p>\n<p><span style=\"color: #ffffff;\">The ls command will list my home directories with no explicit input, as the default input with no switches is all non hidden files and folders from my current home directory:<\/span><\/p>\n<p><span style=\"color: #ffffff;\">$ <span style=\"color: #0000ff;\">ls<\/span><\/span><\/p>\n<p><span style=\"color: #ff0000;\">750GB Dellmint Documents Music Public Videos<\/span><\/p>\n<p><span style=\"color: #ff0000;\">Cprogs Desktop Downloads Pictures Templates<\/span><\/p>\n<p><span style=\"color: #ffffff;\">If Std out is set explicitly, you get the same output as you would expect as the target is the screen:<\/span><\/p>\n<p><span style=\"color: #ffffff;\">$ <span style=\"color: #0000ff;\">ls &gt;&amp;1 <\/span><\/span><\/p>\n<p><span style=\"color: #ff0000;\">750GB Dellmint Documents Music Public Videos<\/span><\/p>\n<p><span style=\"color: #ff0000;\">Cprogs Desktop Downloads Pictures Templates<\/span><\/p>\n<p><span style=\"color: #ffffff;\">You get the same for Stderror as it's default target is also the screen:<\/span><\/p>\n<p><span style=\"color: #ffffff;\">$ <span style=\"color: #0000ff;\">ls &gt;&amp;2<\/span><\/span><\/p>\n<p><span style=\"color: #ff0000;\">750GB Dellmint Documents Music Public Videos<\/span><\/p>\n<p><span style=\"color: #ff0000;\">Cprogs Desktop Downloads Pictures Templates<\/span><\/p>\n<p><span style=\"color: #ffffff;\">The difference between them is that the output from ls is Stout (1) by default, which you are now forcing out of the Stderror (2) channel.<\/span><\/p>\n<p><span style=\"color: #ffffff;\">Similarly, cat can also take stdinput from a file if it exists, and output Stdout (1) to a filename, which it will create if it does not exist already, but will NOT overwrite if it does:<\/span><\/p>\n<p><span style=\"color: #ffffff;\">stevee@AMD ~ $ <span style=\"color: #0000ff;\">cat testfile.txt<\/span><\/span><\/p>\n<p><span style=\"color: #ff0000;\">cat: testfile.txt: No such file or directory<\/span><\/p>\n<p><span style=\"color: #ffffff;\">The file does not exists yet so cat cannot read from it, so it outputs error text via stderror (2) to the screen; but it can create it using a stdout descriptor \"&gt;\" ; an empty file is created to hold the following text when it is typed, then the input data is output to the file after Rtn \u00a0is pressed:<\/span><\/p>\n<p><span style=\"color: #ffffff;\">stevee@AMD ~ $ <span style=\"color: #0000ff;\">cat &gt; testfile.txt<\/span><\/span><\/p>\n<p><span style=\"color: #ff0000;\">this text goes into the file but not to the screen <\/span>(except via the shell)<\/p>\n<p><span style=\"color: #0000ff;\"><span style=\"color: #ffffff;\">Notice you did not see <\/span><span style=\"color: #ffffff;\">duplicate<\/span><span style=\"color: #ffffff;\"> lines as <\/span><span style=\"color: #ffffff;\">above when Rtn hit, because std out was redirected into the newly created file, not the screen, by the right chevron &gt;.<\/span> <\/span><\/p>\n<p><span style=\"color: #ffffff;\">Now cat can open the file as stdin, as it exists:<\/span><\/p>\n<p><span style=\"color: #ffffff;\">stevee@AMD ~ $ <span style=\"color: #0000ff;\">cat testfile.txt<\/span><\/span><\/p>\n<p><span style=\"color: #ff0000;\">this text goes into the file but not to the screen<\/span><\/p>\n<p><span style=\"color: #ffffff;\">Above, the new file contents has gone first via stdin to cat, then via stdout to the screen again.<\/span><\/p>\n<p><span style=\"color: #ffffff;\">Mint does not allow the owner to overwrite his file as was older behaviour:<\/span><\/p>\n<p><span style=\"color: #ffffff;\">stevee@AMD ~ $ <span style=\"color: #0000ff;\">cat &gt; testfile.txt<\/span><\/span><\/p>\n<p><span style=\"color: #ff0000;\">bash: testfile.txt: cannot overwrite existing file<\/span><\/p>\n<p><span style=\"color: #ffffff;\">It is IMPORTANT to realise command order and position, Director type (0,1 or 2) and chevron direction from left to right at this point from a human reading reference e.g. the default stdin to a command from a file, follows in a left to right direction as above but the command is parsed by the shell last\u00a0and then the formerly found file contents fed to it:<\/span><\/p>\n<p><span style=\"color: #ffffff;\">$ <span style=\"color: #0000ff;\">cat testfile.txt<\/span> <\/span><\/p>\n<p><span style=\"color: #ff0000;\">this text goes into the file but not to the screen<\/span><\/p>\n<p><span style=\"color: #ffffff;\">but showing Director type, direction and order of director stream (right to left for stdin) as:<\/span><\/p>\n<p><span style=\"color: #0000ff;\">$ cat 0&lt; testfile.txt <\/span><\/p>\n<p><span style=\"color: #ff0000;\">this text goes into the file but not to the screen<\/span><\/p>\n<p><span style=\"color: #ffffff;\">$ <span style=\"color: #0000ff;\">cat &lt; testfile.txt <\/span><\/span><\/p>\n<p><span style=\"color: #ff0000;\">this text goes into the file but not to the screen<\/span><\/p>\n<p><span style=\"color: #ffffff;\">Commonly, the overall command line placement\u00a0order left to right is cmd, input file, output file: <\/span><\/p>\n<p><span style=\"color: #ffffff;\">stevee@AMD ~ $ <span style=\"color: #0000ff;\">cat &lt; testfile.txt &gt; outfile.txt<\/span><\/span><\/p>\n<p><span style=\"color: #ffffff;\">Above, director streams are right to left stdin, then left to right for stdout. <\/span><\/p>\n<p><span style=\"color: #ffffff;\">$ <span style=\"color: #0000ff;\">cat outfile.txt <\/span><\/span><\/p>\n<p><span style=\"color: #ff0000;\">this text goes into the file but not to the screen<\/span><\/p>\n<p><span style=\"color: #ffffff;\">Likewise, the wordcount command wc, also takes stdin from right to left and outputs left to right, along with the name of the input file:<\/span><\/p>\n<p><span style=\"color: #ffffff;\">$ <span style=\"color: #0000ff;\">wc testfile.txt <\/span><\/span><\/p>\n<p><span style=\"color: #ff0000;\"> 1 11 51 testfile.txt<\/span><\/p>\n<p><span style=\"color: #ffffff;\">$ <span style=\"color: #0000ff;\">wc testfile.txt &gt;&amp;1<\/span><\/span><\/p>\n<p><span style=\"color: #ff0000;\"> 1 11 51 testfile.txt<\/span><\/p>\n<p><span style=\"color: #ffffff;\">BUT, notice the difference if stdin is set explicitly for the input file this time the shell, not the command opens the file, so no filename is given:<\/span><\/p>\n<p><span style=\"color: #0000ff;\">wc 0&lt; testfile.txt <\/span><\/p>\n<p><span style=\"color: #ff0000;\"> 1 11 51<\/span><\/p>\n<p><span style=\"color: #ffffff;\">This is important of you only want the data about the file contents ouput, without the name, as this may become the input for another command. This may be done using a pipe, if the command itself can accept stdout from a prior command as stdin e.g. counting the words in testfile.txt:<\/span><\/p>\n<p><span style=\"color: #ffffff;\">$ <span style=\"color: #0000ff;\">cat testfile.txt <\/span><\/span><\/p>\n<p><span style=\"color: #ff0000;\">this text goes into the file but not to the screen<\/span><\/p>\n<p><span style=\"color: #ffffff;\">stevee@AMD ~ $ <span style=\"color: #0000ff;\">cat testfile.txt | wc -w<\/span><\/span><\/p>\n<p><span style=\"color: #ff0000;\">11<\/span><\/p>\n<p><span style=\"color: #ffffff;\">The cat command can also comply:<\/span><\/p>\n<p><span style=\"color: #ffffff;\">$ <span style=\"color: #0000ff;\">wc -w testfile.txt <\/span><\/span><\/p>\n<p><span style=\"color: #ff0000;\">11 testfile.txt<\/span><\/p>\n<p><span style=\"color: #ffffff;\">stevee@AMD ~ $ <span style=\"color: #0000ff;\">wc -w testfile.txt | cat<\/span><\/span><\/p>\n<p><span style=\"color: #ff0000;\">11 testfile.txt<\/span><\/p>\n<p><span style=\"color: #ffffff;\">As the above output is to stdout, it follows you could keep sending it back to itself via different directors stdout or stderror - so ultimately to the screen:<\/span><\/p>\n<p><span style=\"color: #ffffff;\">$ <span style=\"color: #0000ff;\">wc -w testfile.txt | cat &gt;&amp;1 &gt;&amp;1 &gt;&amp;1<\/span><\/span><\/p>\n<p><span style=\"color: #ff0000;\">11 testfile.txt<\/span><\/p>\n<p><span style=\"color: #ffffff;\">$ <span style=\"color: #0000ff;\">wc -w testfile.txt | cat &gt;&amp;2 &gt;&amp;1 &gt;&amp;2<\/span><\/span><\/p>\n<p><span style=\"color: #ff0000;\">11 testfile.txt<\/span><\/p>\n<p><span style=\"color: #ffffff;\">You may want the output of many commands sent to a file, or many txt files cat'd to one:<\/span><\/p>\n<p><span style=\"color: #ffffff;\"> $ <span style=\"color: #0000ff;\">who;ls;date;cal<\/span><\/span><\/p>\n<p><span style=\"color: #ff0000;\">stevee tty8 2016-07-02 22:59 (:0)<\/span><\/p>\n<p><span style=\"color: #ff0000;\">stevee pts\/1 2016-07-02 23:13 (:0)<\/span><\/p>\n<p><span style=\"color: #ff0000;\">stevee pts\/2 2016-07-03 02:46 (:0)<\/span><\/p>\n<p><span style=\"color: #ff0000;\">RSYNCtest.txt<\/span><\/p>\n<p><span style=\"color: #ff0000;\">Sun Jul 3 02:56:58 BST 2016<\/span><\/p>\n<p><span style=\"color: #ff0000;\"> July 2016 <\/span><\/p>\n<p><span style=\"color: #ff0000;\">Su Mo Tu We Th Fr Sa <\/span><\/p>\n<p><span style=\"color: #ff0000;\"> 1 2 <\/span><\/p>\n<p><span style=\"color: #ff0000;\"> 3 4 5 6 7 8 9 <\/span><\/p>\n<p><span style=\"color: #ff0000;\">10 11 12 13 14 15 16 <\/span><\/p>\n<p><span style=\"color: #ff0000;\">17 18 19 20 21 22 23 <\/span><\/p>\n<p><span style=\"color: #ff0000;\">24 25 26 27 28 29 30 <\/span><\/p>\n<p><span style=\"color: #ff0000;\">31<\/span><\/p>\n<p><span style=\"color: #0000ff;\">who;ls;date;cal &gt; file.txt<\/span><\/p>\n<p><span style=\"color: #ffffff;\">You may create many files, say book chapters to fill then cat to one final:<\/span><\/p>\n<p><span style=\"color: #0000ff;\">touch chapter{1..3}.txt; ls<\/span><\/p>\n<p><span style=\"color: #ff0000;\">chapter1.txt chapter2.txt chapter3.txt<\/span><\/p>\n<p><span style=\"color: #ffffff;\">You can now append to each chapter via the keys (-), without seeing what is already in there though:<\/span><\/p>\n<p><span style=\"color: #0000ff;\">cat - &gt;&gt; chapter1.txt <\/span><\/p>\n<p><span style=\"color: #ff0000;\">this will go into chap 4 soon<span style=\"color: #ffffff;\"> (CtrlD)<\/span><\/span><\/p>\n<p><span style=\"color: #ffffff;\">then append all the chapters to a final page<\/span><\/p>\n<p><span style=\"color: #0000ff;\">cat chapter* &gt; chapter4<\/span><\/p>\n<p><span style=\"color: #0000ff;\">cat chapter4<\/span><\/p>\n<p><span style=\"color: #ff0000;\">this will go into chap 4 soon<\/span><\/p>\n<p><span style=\"color: #ffffff;\">To dump stderror into \/dev\/null or the bit bucket away from screen or script ouput:<\/span><\/p>\n<p><span style=\"color: #0000ff;\">cat abc <\/span><\/p>\n<p><span style=\"color: #ff0000;\">cat: abc: No such file or directory<\/span><\/p>\n<p><span style=\"color: #0000ff;\">cat abc 2&gt; \/dev\/null <\/span><\/p>\n<p><span style=\"color: #ffffff;\">(i.e no stderror shown here)<\/span><\/p>\n<p><span style=\"color: #ffffff;\">You may have overlooked the obvious in the pipe example above, but a pipe does away with the step of creating an intermediary file before the next command can use that for stdin. <\/span><\/p>\n<p><span style=\"color: #ffffff;\">Strip a column from a file using a pipe:<\/span><\/p>\n<p><span style=\"color: #0000ff;\">cat chapter4 | awk '{print $7}'<\/span><\/p>\n<p><span style=\"color: #ffffff;\"><span style=\"color: #ff0000;\">soon<\/span> <\/span><\/p>\n<p><span style=\"color: #ffffff;\">This piping feature was added to Unix in the 70's allowing chains of commands to be strung together to form a programming language of sorts shell scripting which became Windows batch files - where these cmd strings can be saved as an executable file and run if the command pipe is saved in a text file first, given a .sh extension usually for ID, made exe with chmod +x, then run using either of:<\/span><\/p>\n<p><span style=\"color: #ffffff;\">$ <span style=\"color: #0000ff;\">.\/soonpipe.sh <\/span><\/span><\/p>\n<p><span style=\"color: #ff0000;\">soon<\/span><\/p>\n<p><span style=\"color: #ffffff;\">$ <span style=\"color: #0000ff;\">sh soonpipe.sh<\/span> <\/span><\/p>\n<p><span style=\"color: #ff0000;\">soon<\/span><\/p>\n<p><span style=\"color: #ffffff;\">The above is fine when just the output of a command is the desired input to the next, but what if you want the <i><b>result of the argument <\/b><\/i>of the first command to be the next input? The first command has to complete first, before it's result is sent. It depends on the order of precedence the shell operates on special characters: According to my UnixDB book, it's:<\/span><\/p>\n<p><span style=\"color: #ffffff;\"><i><b>Parsing white space; Variables; Command Sub; Redirection; Wildcard file expansion patterns (*?![]); Cmd PATH.<\/b><\/i><\/span><\/p>\n<p>Create 3 files, page1-3; echo a word or phrase into each;<\/p>\n<p><span style=\"color: #0000ff;\"> echo 'once' &gt; page1<\/span><\/p>\n<p><span style=\"color: #0000ff;\">echo 'upon' &gt; page2<\/span><br \/>\n<span style=\"color: #0000ff;\">echo 'a time' &gt; page3<\/span><\/p>\n<p><span style=\"color: #ffffff;\">A wildcard is common in file expansion to simplify a file listing and is evaluated before the command is found from the above list e.g:<\/span><\/p>\n<p><span style=\"color: #ffffff;\">$ <span style=\"color: #0000ff;\">cat page1 page2 page3<\/span><\/span><\/p>\n<p><span style=\"color: #ff0000;\">once<\/span><\/p>\n<p><span style=\"color: #ff0000;\">upon<\/span><\/p>\n<p><span style=\"color: #ff0000;\">a time<\/span><\/p>\n<p><span style=\"color: #ffffff;\">becomes:<\/span><\/p>\n<p><span style=\"color: #ffffff;\">$ <span style=\"color: #0000ff;\">cat page*<\/span><\/span><\/p>\n<p><span style=\"color: #ff0000;\">once<\/span><\/p>\n<p><span style=\"color: #ff0000;\">upon<\/span><\/p>\n<p><span style=\"color: #ff0000;\">a time<\/span><\/p>\n<p><span style=\"color: #ffffff;\">OR:<\/span><\/p>\n<p><span style=\"color: #0000ff;\">cat page[1-3]<\/span><\/p>\n<p><span style=\"color: #ff0000;\">once<\/span><\/p>\n<p><span style=\"color: #ff0000;\">upon<\/span><\/p>\n<p><span style=\"color: #ff0000;\">a time<\/span><\/p>\n<p><span style=\"color: #ffffff;\">OR as ls also lists the files:<\/span><\/p>\n<p><span style=\"color: #ffffff;\">$ <span style=\"color: #0000ff;\">ls page*<\/span><\/span><\/p>\n<p><span style=\"color: #ff0000;\">page1 page2 page3<\/span><\/p>\n<p><span style=\"color: #ffffff;\">The above file list output, if expanded first by creating variables of of file, can become the serial arguments for cat - so the contents are read - not the file names:<\/span><\/p>\n<p><span style=\"color: #ffffff;\">$ <span style=\"color: #0000ff;\">cat $(ls page*)<\/span><\/span><\/p>\n<p><span style=\"color: #ff0000;\">once<\/span><\/p>\n<p><span style=\"color: #ff0000;\">upon<\/span><\/p>\n<p><span style=\"color: #ff0000;\">a time<\/span><\/p>\n<p><span style=\"color: #ffffff;\">BUT if not variables:<\/span><\/p>\n<p><span style=\"color: #0000ff;\">(ls page*)<\/span><br \/>\n<span style=\"color: #ff0000;\">page1 page2 page3<\/span><\/p>\n<p>SO,<\/p>\n<p><span style=\"color: #ffffff;\"><span style=\"color: #0000ff;\">cat 0&lt; (ls page*)<\/span> does not work as you may think:<\/span><\/p>\n<p><span style=\"color: #ff0000;\">bash: syntax error near unexpected token `('<\/span><\/p>\n<p><span style=\"color: #ffffff;\">and neither does this, as it feeds the file's names not their content:<\/span><\/p>\n<p><span style=\"color: #0000ff;\">(ls page*) | cat<\/span><\/p>\n<p><span style=\"color: #ff0000;\">page1<\/span><\/p>\n<p><span style=\"color: #ff0000;\">page2<\/span><\/p>\n<p><span style=\"color: #ff0000;\">page3 <\/span><\/p>\n<p><span style=\"color: #ffffff;\">Now below, you may see how the expansion worked in the PDF eBook Post for pdfunite to cat multiple separate PDFs into a final PDF:<\/span><\/p>\n<p><span style=\"color: #0000ff;\">pdfunite $(ls -tr localhost-*) AllPosts.pdf<\/span><\/p>\n<p><span style=\"color: #ff0000;\"><span style=\"color: #ffffff;\">So what is the $ sign doing in that above example? It makes variables out of the <strong><em>results<\/em><\/strong> of the ls command. <\/span><span style=\"color: #ffffff;\"><i><b><\/b><\/i><\/span><span style=\"color: #ffffff;\"><i><b>All words preceded by a $ are evaluated as variables, unless quoted or escaped<\/b><\/i><\/span><\/span><\/p>\n<p><span style=\"color: #ff0000;\"><span style=\"color: #ffffff;\">You know that<\/span><span style=\"color: #ffffff;\">;<\/span><\/span><\/p>\n<p><span style=\"color: #0000ff;\">ls page*<\/span><\/p>\n<p><span style=\"color: #ff0000;\">page1 page2 page3<\/span><\/p>\n<p><span style=\"color: #ffffff;\">So you can make this operation a variable e.g:<\/span><\/p>\n<p><span style=\"color: #0000ff;\">list=\"ls page*\"<\/span><\/p>\n<p><span style=\"color: #ffffff;\">The contents of this variable is:<\/span><\/p>\n<p><span style=\"color: #0000ff;\">$list<\/span><\/p>\n<p><span style=\"color: #ff0000;\">page1 page2 page3<\/span><\/p>\n<p><span style=\"color: #ffffff;\">This is not just text, it represents the files themselves <i>as variables<\/i>, so that the content of the apostrophes ls and the expansion of page* - becomes the command <i>and<\/i> the list of files to be fed to cat. These, as variables, then becomes the argument for cat to give almost the desired outcome, except for the ls command itself, so not quite what is required due to the stderror part, but the the expansion works:<\/span><\/p>\n<p><span style=\"color: #0000ff;\">cat $list <\/span><\/p>\n<p><span style=\"color: #ff0000;\">cat: ls: No such file or directory<\/span><\/p>\n<p><span style=\"color: #ff0000;\">once<\/span><\/p>\n<p><span style=\"color: #ff0000;\">upon<\/span><\/p>\n<p><span style=\"color: #ff0000;\">a time<\/span><\/p>\n<p><span style=\"color: #ffffff;\">YET it does work using  instead of () as a variable definition container, as it lists ALL files in the directory!! SEEMINGLY w<\/span><span style=\"color: #ffffff;\">eird at first, and not wanted - the page prefix is ignored, listing ALL files present in the directory yet explained above due to * being escaped by the brackets; as ls alone gives this listing:<\/span><\/p>\n<p><span style=\"color: #0000ff;\">list=(ls page*)<\/span><\/p>\n<p><span style=\"color: #0000ff;\">$list<\/span><\/p>\n<p><span style=\"color: #ff0000;\">page1 page2 page3 <b>soonpipe.sh<\/b><\/span><\/p>\n<p><span style=\"color: #ffffff;\">YET: You can view this variable result by listing the variable list and expansion is correct!<\/span><\/p>\n<p><span style=\"color: #0000ff;\">ls $(ls page*)<\/span><\/p>\n<p><span style=\"color: #ff0000;\">page1 page2 page3<\/span><\/p>\n<p><span style=\"color: #ffffff;\">You can see why you need to experiment yourself!<\/span><\/p>\n<p><span style=\"color: #ffffff;\">This variable container output is only correct in context of a command operating on it when fed to cat it expands correctly and gives the result you want which is each file as a separate variable due to the * expansion:<\/span><\/p>\n<p><span style=\"color: #0000ff;\">cat $(ls page*)<\/span><\/p>\n<p><span style=\"color: #ff0000;\">once<\/span><\/p>\n<p><span style=\"color: #ff0000;\">upon<\/span><\/p>\n<p><span style=\"color: #ff0000;\">a time<\/span><\/p>\n<p><span style=\"color: #ffffff;\">This must be why this $(var) format works as stdin for pdfunite also, which seems to only be a pdf specific cat command.<\/span><\/p>\n<p><span style=\"color: #ffffff;\">This also provides the answer to the question of appending port numbers to the -p switch of nmap that I could not find an answer to last year - again the $ creates a variable of the single line, comma delimited\u00a0<a href=\"https:\/\/stevepedwards.today\/DebianAdmin\/wp-content\/uploads\/2015\/08\/BadPortsCommas.txt\">BadPortsCommas.txt<\/a> file and nmap runs it:<\/span><\/p>\n<p><span style=\"color: #0000ff;\">cat\u00a0<a href=\"https:\/\/stevepedwards.today\/DebianAdmin\/wp-content\/uploads\/2015\/08\/BadPortsCommas.txt\">BadPortsCommas.txt<\/a><\/span><\/p>\n<p><span style=\"color: #ff0000;\">...65432,65530,65535<\/span><\/p>\n<p><span style=\"color: #0000ff;\">nmap 127.0.0.1 -p $(cat BadPortsCommas.txt)<\/span><\/p>\n<p><span style=\"color: #ff0000;\">Starting Nmap 6.40 ( https:\/\/nmap.org ) at 2016-07-04 11:20 BST<\/span><br \/>\n<span style=\"color: #ff0000;\">Nmap scan report for localhost (127.0.0.1)<\/span><br \/>\n<span style=\"color: #ff0000;\">Host is up (0.00049s latency).<\/span><br \/>\n<strong><span style=\"color: #ff0000;\">Not shown: 786 closed ports<\/span><\/strong><br \/>\n<span style=\"color: #ff0000;\">PORT STATE SERVICE<\/span><br \/>\n<span style=\"color: #ff0000;\">22\/tcp open ssh<\/span><br \/>\n<span style=\"color: #ff0000;\">25\/tcp open smtp<\/span><br \/>\n<span style=\"color: #ff0000;\">80\/tcp open http<\/span><br \/>\n<span style=\"color: #ff0000;\">139\/tcp open netbios-ssn<\/span><br \/>\n<span style=\"color: #ff0000;\">445\/tcp open microsoft-ds<\/span><\/p>\n<p><span style=\"color: #ff0000;\">Nmap done: 1 IP address (1 host up) scanned in 0.12 seconds<\/span><\/p>\n<p><span style=\"color: #ffffff;\">Clever stuff...<\/span><\/p>\n<p><span style=\"color: #ffffff;\">It is simple enough for two piped commands to be predicted where you know the output of a cmd is what you want as the input for the next left to right, but can become difficult to foresee as more commands are added especially if command substitution is involved as it is not always possible to test for each result in isolation as seen above; y<\/span><span style=\"color: #ffffff;\">ou can't expand <span style=\"color: #0000ff;\">$(ls page*)<\/span> alone for example it has to be found in a command context - this makes building complex pipes, well...complex!<\/span><\/p>\n<p><span style=\"color: #ffffff;\">All you know for a pipeline is that the first command must use stdout and the last in the chain must take stdin for it to hope to work. There is no limit to the amount of cmds in a pipe line. Whether your pipe works for what you intend is another matter. <\/span><i><b><span style=\"color: #ffffff;\">A lot is left to knowledge of filters,\u00a0perseverance\u00a0and imagination.<\/span><\/b><\/i><\/p>\n<p><span style=\"color: #ffffff;\">This one is from my Linux Tutor Course - can you work out what it does as a combination of filters, command substitution, and pipe?: <\/span><\/p>\n<p><span style=\"color: #0000ff;\">awk -F: '{print $0}' \/etc\/group | grep 100.:<\/span><\/p>\n<p><span style=\"color: #ffffff;\">Note that awk is a line oriented programming language in itself, with \"actions\" defined in {}. \u00a0<\/span><\/p>\n<p><span style=\"color: #ffffff;\">{print $0} =\u00a0{print} =\u00a0{print $null} in this case, so\u00a0the whole file (all records, or lines) is printed across all columns (fields), ignoring usual column delimiters such as \" : \".<\/span><\/p>\n<p><span style=\"color: #ffffff;\">From <span style=\"color: #0000ff;\">man awk<\/span>\u00a0, the $ in awk's case does not mean a variable but is an operator for column numbers:\u00a0<span style=\"color: #ff0000;\">$ = Field reference.<\/span><\/span><\/p>\n<p><span style=\"color: #ffffff;\">Try it it's safe because awk - like many other commands that can be used in pipes, does not overwrite any result of operation to the source file. <\/span><\/p>\n<p><span style=\"color: #ffffff;\">But why is there output from this file and \/etc\/passwd but not from \/etc\/shadow?<\/span><\/p>\n<p><span style=\"color: #ffffff;\">What about this...? Ring any bells from the Mint Tutor Course or GUI admin explorations?\u00a0<\/span><\/p>\n<p><span style=\"color: #0000ff;\">awk -F: '{print $1}' \/etc\/group | sort | head -14<\/span><\/p>\n<p><span style=\"color: #ff0000;\">adm<\/span><br \/>\n<span style=\"color: #ff0000;\">audio<\/span><br \/>\n<span style=\"color: #ff0000;\">avahi<\/span><br \/>\n<span style=\"color: #ff0000;\">backup<\/span><br \/>\n<span style=\"color: #ff0000;\">bin<\/span><br \/>\n<span style=\"color: #ff0000;\">cdrom<\/span><br \/>\n<span style=\"color: #ff0000;\">crontab<\/span><br \/>\n<span style=\"color: #ff0000;\">daemon<\/span><br \/>\n<span style=\"color: #ff0000;\">dialout<\/span><br \/>\n<span style=\"color: #ff0000;\">dip<\/span><br \/>\n<span style=\"color: #ff0000;\">disk<\/span><br \/>\n<span style=\"color: #ff0000;\">fax<\/span><br \/>\n<span style=\"color: #ff0000;\">floppy<\/span><br \/>\n<span style=\"color: #ff0000;\">fuse<\/span><\/p>\n<p><a href=\"https:\/\/stevepedwards.today\/DebianAdmin\/wp-content\/uploads\/2016\/07\/grpadmin.png\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter size-medium wp-image-3341\" src=\"https:\/\/stevepedwards.today\/DebianAdmin\/wp-content\/uploads\/2016\/07\/grpadmin.png\" alt=\"grpadmin.png\" width=\"642\" height=\"520\" \/><\/a><\/p>\n<p><span style=\"color: #ffffff;\">Some filters that accept both stdin and stdout so can be used in pipes are:<\/span><\/p>\n<p><span style=\"color: #0000ff;\">pr; head; tail; cut; paste; sort; uniq; nl; awk; sed; grep; join; tr<\/span><\/p>\n<p><span style=\"color: #ffffff;\">Read their man pages for use. e.g:<\/span><\/p>\n<p><span style=\"color: #0000ff;\">ls | pr -3 | head<\/span><\/p>\n<p><a href=\"https:\/\/stevepedwards.today\/DebianAdmin\/wp-content\/uploads\/2016\/07\/prhead.png\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter size-medium wp-image-3331\" src=\"https:\/\/stevepedwards.today\/DebianAdmin\/wp-content\/uploads\/2016\/07\/prhead.png\" alt=\"prhead.png\" width=\"658\" height=\"233\" \/><\/a><\/p>\n<p><span style=\"color: #ffffff;\">Remove the headers and tail formatting blank lines from the pr command:<\/span><\/p>\n<p><span style=\"color: #0000ff;\">ls Documents\/ \u00a0| pr -3 | head | sed -n '1,2!p' | sed -n '2,3!p' | sed -n '5,7!p'<\/span><\/p>\n<p><a href=\"https:\/\/stevepedwards.today\/DebianAdmin\/wp-content\/uploads\/2016\/07\/docssed.png\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter size-medium wp-image-3385\" src=\"https:\/\/stevepedwards.today\/DebianAdmin\/wp-content\/uploads\/2016\/07\/docssed.png\" alt=\"docssed.png\" width=\"834\" height=\"156\" \/><\/a><\/p>\n<p><span style=\"color: #ffffff;\">No doubt every linux admin has a large collection of general and specific pre-defined pipes in his toolkit, tested specifically for his system and duties.<\/span><\/p>\n","protected":false},"excerpt":{"rendered":"<div class=\"pvc_clear\"><\/div>\n<p id=\"pvc_stats_3315\" class=\"pvc_stats all  \" data-element-id=\"3315\" style=\"\"><i class=\"pvc-stats-icon medium\" aria-hidden=\"true\"><svg aria-hidden=\"true\" focusable=\"false\" data-prefix=\"far\" data-icon=\"chart-bar\" role=\"img\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\" viewBox=\"0 0 512 512\" class=\"svg-inline--fa fa-chart-bar fa-w-16 fa-2x\"><path fill=\"currentColor\" d=\"M396.8 352h22.4c6.4 0 12.8-6.4 12.8-12.8V108.8c0-6.4-6.4-12.8-12.8-12.8h-22.4c-6.4 0-12.8 6.4-12.8 12.8v230.4c0 6.4 6.4 12.8 12.8 12.8zm-192 0h22.4c6.4 0 12.8-6.4 12.8-12.8V140.8c0-6.4-6.4-12.8-12.8-12.8h-22.4c-6.4 0-12.8 6.4-12.8 12.8v198.4c0 6.4 6.4 12.8 12.8 12.8zm96 0h22.4c6.4 0 12.8-6.4 12.8-12.8V204.8c0-6.4-6.4-12.8-12.8-12.8h-22.4c-6.4 0-12.8 6.4-12.8 12.8v134.4c0 6.4 6.4 12.8 12.8 12.8zM496 400H48V80c0-8.84-7.16-16-16-16H16C7.16 64 0 71.16 0 80v336c0 17.67 14.33 32 32 32h464c8.84 0 16-7.16 16-16v-16c0-8.84-7.16-16-16-16zm-387.2-48h22.4c6.4 0 12.8-6.4 12.8-12.8v-70.4c0-6.4-6.4-12.8-12.8-12.8h-22.4c-6.4 0-12.8 6.4-12.8 12.8v70.4c0 6.4 6.4 12.8 12.8 12.8z\" class=\"\"><\/path><\/svg><\/i> <img loading=\"lazy\" decoding=\"async\" width=\"16\" height=\"16\" alt=\"Loading\" src=\"https:\/\/stevepedwards.today\/DebianAdmin\/wp-content\/plugins\/page-views-count\/ajax-loader-2x.gif\" border=0 \/><\/p>\n<div class=\"pvc_clear\"><\/div>\n<p>After you read this article, I'll let you think of interesting ways to use the above video example of piping data between terminals into other programs for immediate processing...if the commands are not clear; term 1 creates a fifo pipe in my home dir: mkfifo fifo_pipe term 2 cats that pipe to receive what comes <a href=\"https:\/\/stevepedwards.today\/DebianAdmin\/experiment-with-pipes-redirection-command-substitution-and-variable-expansion\/\" class=\"more-link\">...<span class=\"screen-reader-text\">\u00a0 Experiment with Pipes, Redirection, Command Substitution and Variable Expansion<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[2],"tags":[],"class_list":["post-3315","post","type-post","status-publish","format-standard","hentry","category-uncategorized"],"a3_pvc":{"activated":true,"total_views":1,"today_views":0},"_links":{"self":[{"href":"https:\/\/stevepedwards.today\/DebianAdmin\/wp-json\/wp\/v2\/posts\/3315","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/stevepedwards.today\/DebianAdmin\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/stevepedwards.today\/DebianAdmin\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/stevepedwards.today\/DebianAdmin\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/stevepedwards.today\/DebianAdmin\/wp-json\/wp\/v2\/comments?post=3315"}],"version-history":[{"count":1,"href":"https:\/\/stevepedwards.today\/DebianAdmin\/wp-json\/wp\/v2\/posts\/3315\/revisions"}],"predecessor-version":[{"id":7720,"href":"https:\/\/stevepedwards.today\/DebianAdmin\/wp-json\/wp\/v2\/posts\/3315\/revisions\/7720"}],"wp:attachment":[{"href":"https:\/\/stevepedwards.today\/DebianAdmin\/wp-json\/wp\/v2\/media?parent=3315"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/stevepedwards.today\/DebianAdmin\/wp-json\/wp\/v2\/categories?post=3315"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/stevepedwards.today\/DebianAdmin\/wp-json\/wp\/v2\/tags?post=3315"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}