{"id":2354,"date":"2015-10-01T18:17:18","date_gmt":"2015-10-01T17:17:18","guid":{"rendered":"https:\/\/stevepedwards.today\/DebianAdmin\/?p=2354"},"modified":"2023-11-09T02:42:50","modified_gmt":"2023-11-09T02:42:50","slug":"sed-basics-using-o_reilly-pdf-example-and-unexpected-mind-bending-parsing-behaviour","status":"publish","type":"post","link":"https:\/\/stevepedwards.today\/DebianAdmin\/sed-basics-using-o_reilly-pdf-example-and-unexpected-mind-bending-parsing-behaviour\/","title":{"rendered":"SED Basics using O_Reilly.pdf example, and Unexpected Mind-Bending Parsing Behaviour"},"content":{"rendered":"
<\/div>\n

<\/path><\/svg><\/i> \"Loading\"<\/p>\n

<\/div>\n

The O_Reilly.pdf pdf of this 1990 book refers to POSIX standard changes in SED etc. so may account for different behaviour by my Mint version of grep, sed and awk etc. from their original examples, as I think happened below.<\/p>\n

As an introduction to the ideas of iteration and recursion using SED:<\/p>\n

In simple terms, an <\/span><\/span><\/i><\/span>iterative <\/b><\/span><\/span><\/span>function is one that loops to repeat some part of the code, and a <\/span><\/span><\/span>recursive <\/b><\/span><\/span><\/span>function is one that calls itself again to repeat the code.<\/span><\/span><\/span><\/i><\/p>\n

Experiment with some examples of SED, to try to understand how it works line by line, as a script proceeds, with any changed values in the first line becoming possible targets for change by subsequent script line commands as it's not obvious at first how it works, or to visualise, (especially if you are a crap programmer like me!) like much with grep, sed and awk - depending on what regular expression methods and metacharacters are used.<\/p>\n

Here's the pig, cow, horse example in the book,\u00a0which I think contains omissions re the global function - states:<\/p>\n

Let's look at an example that uses the substitute command. Suppose someone quickly wrote the<\/i><\/p>\n

following script to change \"pig\" to \"cow\" and \"cow\" to \"horse\":<\/i><\/p>\n

s\/pig\/cow\/<\/i><\/p>\n

s\/cow\/horse\/<\/i><\/p>\n

What do you think happened? Try it on a sample file. We'll discuss what happened later, after we look at how sed works.<\/i><\/p>\n

At this point, I guessed the script would change all pig words to cow words, then all cows to horses, (which I assume is what most people would?), so that only the word horse would exist in place of previous pigs and cows. It's not clear what the fictitious scriptwriter actually intended though, from the text.<\/p>\n

Did he want to substitute\u00a0all \"pig\"\u00a0and<\/i>\u00a0\"cow\" words with \"horse\" in the first place or just some? Can SED do that on one line or does it need two lines?<\/p>\n

So, create the script as above:<\/p>\n

cat scriptest.txt<\/span><\/p>\n

s\/pig\/cow\/<\/i><\/span><\/p>\n

s\/cow\/horse\/<\/i><\/span><\/p>\n

and a suitable file to apply it to:<\/p>\n

cat pigcowhorse.txt<\/span><\/p>\n

pig<\/span><\/p>\n

pig cow<\/span><\/p>\n

pig cow horse<\/span><\/p>\n

pig cow horse pig<\/span><\/p>\n

It would seem that each word pig would first change to cow in the process, then all the new cow values would change to horse if you think the first line acts on the whole file before the 2nd<\/sup> line acts, to penultimately change each pig to cow:<\/span><\/p>\n

cow<\/span><\/p>\n

cow cow<\/span><\/p>\n

cow cow horse<\/span><\/p>\n

cow cow horse cow<\/span><\/p>\n

so that ultimately, cows become horses by the incredible way sed may swiftly alter their genetics:<\/span><\/p>\n

horse<\/span><\/p>\n

horse horse<\/span><\/p>\n

horse horse horse<\/span><\/p>\n

horse horse horse horse<\/span><\/p>\n

as the rushed scriptwriter possibly overlooked, from what he desired? <\/span><\/p>\n

If you run this script, it does neither that, nor what the example intends to show, I believe, as some cows don't become horses, and a final pig remains un-mutated:<\/span><\/p>\n

stevee@Mint5630 ~\/Documents\/SEDAWK $ sed -f scriptest.txt pigcowhorse.txt<\/span> <\/span><\/p>\n

horse<\/span><\/p>\n

horse cow<\/span><\/p>\n

horse cow horse<\/span><\/p>\n

horse cow horse pig<\/span><\/p>\n

The authors intended the script to act as explained:<\/span><\/p>\n

As a consequence, any sed command might change the contents of the pattern space for the next<\/i><\/span><\/p>\n

command. The contents of the pattern space are dynamic and do not always match the original input line. That was the problem with the sample script at the beginning of this chapter. The first command would change \"pig\" to \"cow\" as expected. However, when the second command changed \"cow\" to \"horse\" on the same line, it also changed the \"cow\" that had been a \"pig.\" So, where the input file contained pigs and cows, the output file has only horses!<\/i><\/span><\/p>\n

To understand what happens, you need to go in steps, as sed does in the script:<\/span><\/p>\n

sed s\/pig\/cow\/ pigcowhorse.txt (> cows.txt)<\/span><\/span><\/p>\n

cow<\/span><\/p>\n

cow cow<\/span><\/p>\n

cow cow horse<\/span><\/p>\n

cow cow horse pig<\/span><\/p>\n

Now you see why the pig gets left. Sed only acts on the first occurrence on each line.<\/span><\/p>\n

You can guess now what happens the next time round if only the FIRST occurrence of cow gets substituted by horse (s\/cow\/horse\/) and all else left alone, same as the result I got above: <\/span><\/p>\n

horse<\/span><\/p>\n

horse cow<\/span><\/p>\n

horse cow horse<\/span><\/p>\n

horse cow horse pig<\/span><\/p>\n

What the authors forgot possibly due to different sed version behaviour since theirs in 1990 to give us all horses was the global command, g, so that all occurrences of the searched term get substituted on each line:<\/span><\/p>\n

cat scriptest.txt <\/span><\/p>\n

s\/pig\/cow\/g<\/span><\/p>\n

s\/cow\/horse\/g<\/span><\/p>\n

so that ALL pigs become cows, then ALL cows become horses, which when run globally, you do indeed get all horses:<\/span><\/p>\n

sed -f scriptest.txt pigcowhorse.txt<\/span> <\/span><\/p>\n

horse<\/span><\/p>\n

horse horse<\/span><\/p>\n

horse horse horse<\/span><\/p>\n

horse horse horse horse<\/span><\/p>\n

Again, you can run the single sed lines to see the global substitution effect of all occurrences of the searched word on each line. The first pass changes all pigs to cows, the second, all cows to horses:<\/span><\/p>\n

sed s\/pig\/cow\/g pigcowhorse.txt (> cows2.txt)<\/span><\/span><\/p>\n

cow<\/span><\/p>\n

cow cow<\/span><\/p>\n

cow cow horse<\/span><\/p>\n

cow cow horse cow<\/span> <\/span><\/p>\n

sed s\/cow\/horse\/g cows2.txt<\/span> <\/span><\/p>\n

horse<\/span><\/p>\n

horse horse<\/span><\/p>\n

horse horse horse<\/span><\/p>\n

horse horse horse horse<\/span><\/p>\n

So this is what the authors were driving home they just left out the global command aspect unless their 1990 version worked differently:<\/span><\/p>\n

As a consequence, any sed command might change the contents of the pattern space for the next<\/i><\/span><\/p>\n

command. The contents of the pattern space are dynamic and do not always match the original input line. That was the problem with the sample script at the beginning of this chapter. The first command would change \"pig\" to \"cow\" as expected. However, when the second command changed \"cow\" to \"horse\" on the same line, it also changed the \"cow\" that had been a \"pig.\" So, where the input file contained pigs and cows, the output file has only horses!<\/i><\/span><\/p>\n

It seems what they were suggesting, that was desired by the scriptwriter, was to have been achieved by changing the script command order, but not globally either:<\/span><\/p>\n

This mistake is simply a problem of the order of the commands in the script. Reversing the order of the commands - changing \"cow\" into \"horse\" before changing \"pig\" into \"cow\" - does the trick.<\/i><\/span><\/p>\n

s\/cow\/horse\/<\/i><\/span><\/p>\n

s\/pig\/cow\/<\/i><\/span><\/p>\n

This time, if the script is edited as above, (no global option), and run on the original file you would get from:<\/span><\/p>\n

cat pigcowhorse.txt <\/span><\/p>\n

pig<\/span><\/p>\n

pig cow<\/span><\/p>\n

pig cow horse<\/span><\/p>\n

pig cow horse pig<\/span><\/p>\n

to<\/span><\/p>\n

sed -f scriptest.txt pigcowhorse.txt<\/span> <\/span><\/p>\n

cow<\/span><\/p>\n

cow horse<\/span><\/p>\n

cow horse horse<\/span><\/p>\n

cow horse horse pig<\/span><\/p>\n

Again, an unchanged pig left over. <\/span><\/p>\n

Compare that with adding the global g:<\/span><\/p>\n

s\/cow\/horse\/g<\/span><\/p>\n

s\/pig\/cow\/g<\/span><\/p>\n

to get what I think they meant the scriptwriter meant for all along all original<\/b> pigs ONLY get changed to cows, and all original<\/b> cows ONLY get changed to horses!:<\/span><\/p>\n

cat pigcowhorse.txt <\/span><\/p>\n

pig<\/span><\/p>\n

pig cow<\/span><\/p>\n

pig cow horse<\/span><\/p>\n

pig cow horse pig<\/span><\/p>\n

sed -f scriptest.txt pigcowhorse.txt<\/span> <\/span><\/p>\n

cow<\/span><\/p>\n

cow horse<\/span><\/p>\n

cow horse horse<\/span><\/p>\n

cow horse horse cow<\/span><\/p>\n

Only working patiently through these types of examples will you discover errors like this, and think about what is going on, to get a feel for how easy it is to error in the initial thought process of how you think things may work, to how they turn out.<\/span><\/p>\n

The next sed example is a handy one liner for removing blank lines, so I insert 4 in pigcowhorse.txt:<\/span><\/p>\n

cat pigcowhorse.txt<\/span><\/p>\n

pig<\/span><\/p>\n

pig cow<\/span><\/p>\n

pig cow horse<\/span><\/p>\n

pig cow horse pig<\/span><\/p>\n

---<\/span><\/p>\n

From grep, and knowing that the (^) denotes the first char of any\/every line as even blank lines by definition, have to have a null char of some sort to separate it from the next line, so therefore a last character ($) also, by definition, which would be a line feed (LF char) or similar.<\/span><\/p>\n

You can find all empty lines in a file using these special characters together - the first and last chars with NOTHING between them = a blank line. <\/span><\/p>\n

For the 4 blanks in pigcowhorse.txt you get a gap (which I numbered below) in the tty where grep finds only the 4 empty lines:<\/span><\/p>\n

stevee@Mint5630 ~\/Documents\/SEDAWK $ grep ^$ pigcowhorse.txt<\/span> <\/span><\/p>\n

1<\/span><\/p>\n

2<\/span><\/p>\n

3<\/span><\/p>\n

4<\/span><\/p>\n

stevee@Mint5630 ~\/Documents\/SEDAWK $<\/span><\/p>\n

You can use this with sed combined with the d delete option to remove those lines from a file (note the WordPress editor adds extra lines also when pasted - but you realise there is no line space between these in the Terminal?):<\/span><\/p>\n

sed '\/^$\/d' pigcowhorse.txt<\/span> <\/span><\/p>\n

pig<\/span><\/p>\n

pig cow<\/span><\/p>\n

pig cow horse<\/span><\/p>\n

pig cow horse pig<\/span><\/p>\n

--- <\/span><\/p>\n

Extending that regex, you can use grep, as cat, to view any file, using the . period, which denotes any single character; along with the asterisk * wildcard, which denotes none OR any number of any characters. <\/span><\/p>\n

This means the first character of any line (which has to exist for there to be a line in the first place), can be followed by anything or nothing, and be displayed as a search result by grep, i.e. the whole file's contents!<\/span><\/p>\n

grep ^.* pigcowhorse.txt<\/span> <\/span><\/p>\n

pig<\/span><\/p>\n

pig cow<\/span><\/p>\n

pig cow horse<\/span><\/p>\n

pig cow horse pig<\/span><\/p>\n

---<\/span><\/p>\n

These two one-liners are handy to check your understanding of four common special characters (^.*$) which can mean different things to the command line, depending on how they are bounded by each other. For example, if you list the attribs of Documents with the dir option d, you get a single line output from ls:<\/span><\/p>\n

ls -ld Documents\/<\/span><\/p>\n

drwxr-xr-x 4 stevee stevee 4096 Sep 30 10:36 Documents\/<\/span> <\/span><\/p>\n

If you pipe that to grep to search for any single character in that line, you get the whole line as you may expect:<\/span><\/p>\n

ls -ld Documents\/ | grep .<\/span><\/p>\n

drwxr-xr-x 4 stevee stevee 4096 Sep 30 10:36 Documents\/<\/span><\/span><\/p>\n

You may think that grep would still only read the content stream of it's input from ls -ld, if you added the wildcard; <\/span><\/p>\n

ls -ld Documents\/ | grep .*<\/span><\/p>\n

to show none or all characters after any single character (found by the period), but it doesn't do that by a LONG way it lists all<\/b> the characters after any single<\/b> character, found in the actual Documents directory - some 418 files, as if you used grep alone in the first place:<\/span><\/p>\n

ls -ld Documents\/ | grep .* is equal<\/span> to grep .* <\/span><\/p>\n

Why isn't it listing just the single line that defined the Documents line, which has a first character d (from drwx etc.), followed by none or any amount of any other characters i.e. the input stream alone, from ls -ld? <\/span><\/p>\n

It seems the command line parses the . period after grep, and expands as a current directory node, which is also a . period, as seen by using:<\/span><\/p>\n

stevee@AMDA8 ~ $ ls -al<\/span><\/span><\/p>\n

total 140<\/span><\/p>\n

drwxr-xr-x 24 stevee stevee 4096 Oct 1 16:51 .<\/span><\/p>\n

drwxr-xr-x 4 root root 4096 Sep 17 18:46 ..<\/span><\/p>\n

drwxr-xr-x 3 stevee stevee 4096 Sep 18 13:08 .AMD<\/span><\/p>\n

-rw------- 1 stevee stevee 3432 Sep 26 20:21 .bash_history<\/span><\/p>\n

so grep's search now also includes the total current directory (including hidden) contents in (\/home\/stevee), which also includes the Documents directory anyway! Maybe not what you expect?<\/span><\/p>\n

Run it yourself to see. It's only playing with the fundamentals like this, that you will start to think about and start to understand the underlying OS and file system workings, and the tools originally created to manipulate it.<\/span><\/p>\n

The Unix engineers really were smart guys, no doubt.<\/span><\/p>\n

stevee@AMDA8 ~ $ ls -ld Documents\/ | grep .* | wc -l<\/span><\/span><\/p>\n

grep: ..: Is a directory<\/span><\/p>\n

grep: .AMD: Is a directory<\/span><\/p>\n

grep: .cache: Is a directory<\/span><\/p>\n

grep: .cinnamon: Is a directory<\/span><\/p>\n

grep: .config: Is a directory<\/span><\/p>\n

grep: .dbus: Is a directory<\/span><\/p>\n

grep: .gconf: Is a directory<\/span><\/p>\n

grep: .gnome: Is a directory<\/span><\/p>\n

grep: .gnome2: Is a directory<\/span><\/p>\n

grep: .gnome2_private: Is a directory<\/span><\/p>\n

grep: .linuxmint: Is a directory<\/span><\/p>\n

grep: .local: Is a directory<\/span><\/p>\n

grep: .mozilla: Is a directory<\/span><\/p>\n

grep: .pki: Is a directory<\/span><\/p>\n

grep: .ssh: Is a directory<\/span><\/p>\n

412<\/span><\/p>\n

stevee@AMDA8 ~ $<\/span> grep .* | wc -l<\/span><\/span><\/p>\n

grep: ..: Is a directory<\/span><\/p>\n

grep: .AMD: Is a directory<\/span><\/p>\n

grep: .cache: Is a directory<\/span><\/p>\n

grep: .cinnamon: Is a directory<\/span><\/p>\n

grep: .config: Is a directory<\/span><\/p>\n

grep: .dbus: Is a directory<\/span><\/p>\n

grep: .gconf: Is a directory<\/span><\/p>\n

grep: .gnome: Is a directory<\/span><\/p>\n

grep: .gnome2: Is a directory<\/span><\/p>\n

grep: .gnome2_private: Is a directory<\/span><\/p>\n

grep: .linuxmint: Is a directory<\/span><\/p>\n

grep: .local: Is a directory<\/span><\/p>\n

grep: .mozilla: Is a directory<\/span><\/p>\n

grep: .pki: Is a directory<\/span><\/p>\n

grep: .ssh: Is a directory<\/span><\/p>\n

412<\/span> <\/span><\/p>\n","protected":false},"excerpt":{"rendered":"

<\/div>\n

<\/path><\/svg><\/i> \"Loading\"<\/p>\n

<\/div>\n

The O_Reilly.pdf pdf of this 1990 book refers to POSIX standard changes in SED etc. so may account for different behaviour by my Mint version of grep, sed and awk etc. from their original examples, as I think happened below. As an introduction to the ideas of iteration and recursion using SED: In simple terms, ...\u00a0 SED Basics using O_Reilly.pdf example, and Unexpected Mind-Bending Parsing Behaviour<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[2],"tags":[],"class_list":["post-2354","post","type-post","status-publish","format-standard","hentry","category-uncategorized"],"a3_pvc":{"activated":true,"total_views":1,"today_views":0},"_links":{"self":[{"href":"https:\/\/stevepedwards.today\/DebianAdmin\/wp-json\/wp\/v2\/posts\/2354","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/stevepedwards.today\/DebianAdmin\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/stevepedwards.today\/DebianAdmin\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/stevepedwards.today\/DebianAdmin\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/stevepedwards.today\/DebianAdmin\/wp-json\/wp\/v2\/comments?post=2354"}],"version-history":[{"count":3,"href":"https:\/\/stevepedwards.today\/DebianAdmin\/wp-json\/wp\/v2\/posts\/2354\/revisions"}],"predecessor-version":[{"id":10147,"href":"https:\/\/stevepedwards.today\/DebianAdmin\/wp-json\/wp\/v2\/posts\/2354\/revisions\/10147"}],"wp:attachment":[{"href":"https:\/\/stevepedwards.today\/DebianAdmin\/wp-json\/wp\/v2\/media?parent=2354"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/stevepedwards.today\/DebianAdmin\/wp-json\/wp\/v2\/categories?post=2354"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/stevepedwards.today\/DebianAdmin\/wp-json\/wp\/v2\/tags?post=2354"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}