{"id":1941,"date":"2015-08-21T20:57:50","date_gmt":"2015-08-21T19:57:50","guid":{"rendered":"https:\/\/stevepedwards.today\/DebianAdmin\/?p=1941"},"modified":"2023-10-28T21:06:05","modified_gmt":"2023-10-28T20:06:05","slug":"using-awk-and-sed-to-cut-a-column-list-and-for-character-substitution","status":"publish","type":"post","link":"https:\/\/stevepedwards.today\/DebianAdmin\/using-awk-and-sed-to-cut-a-column-list-and-for-character-substitution\/","title":{"rendered":"Using Awk, Sed, Cut and TR To Cut a Column List for Character Substitution and Nmap Bad Ports List"},"content":{"rendered":"
<\/div>\n

<\/path><\/svg><\/i> \"Loading\"<\/p>\n

<\/div>\n

*BEWARE! a \u00a0note on terminal characters before you start: apostrophe pairs can be problematic for copy\/paste operations between different editors, term types, WordPress etc. The common standard is now UTF8, so make sure you set PuTTY correctly using the Translation option:<\/p>\n

\"UTF8Term.jpg\"<\/a><\/p>\n

This may still paste one of the two apostrophes in the wrong \"direction\" and fail for sed and awk command lines - it is easy to miss these visually when they don't translate correctly in a paste! e.g.<\/p>\n

awk `{print $2}`<\/strong> (i.e with backtics that don't show in the HTML!)<\/p>\n

awk '{print $2}'<\/strong><\/p>\n

The first will fail, the second won't (as seen in the Worpress editor but not on the Post page, which translates correctly\u00a0again!). These pastes were from the same source (this page!), into PuTTY then back here again, before PuTTY was set to UTF8!<\/p>\n

-<\/p>\n

I want to separate a list of known Trojan\/virus ports to experiment with an Nmap scan, so needed a quick way to cut out just the port numbers from a column listing of some known dodgy ports I found here:<\/span><\/p>\n

https:\/\/www.simovits.com\/trojans\/trojans.html<\/span><\/a>
\n<\/span><\/p>\n

\"\"
\n<\/span><\/p>\n

As it is in a column format conveniently, linux has some perfect tools to strip these how you want.
\n<\/span><\/p>\n

The site seems down since yesterday, so here's a list to practice with:<\/p>\n

port 2 Death
\nport 20 Senna Spy FTP server
\nport 21 Back Construction, Blade Runner, Doly Trojan, Fore, Invisible FTP, Juggernaut 42 , Larva, MotIv FTP, Net Administrator, Senna Spy FTP server, Traitor 21,
\nWebEx, WinCrash
\nport 22 Shaft
\nport 23 Fire HacKer, Tiny Telnet Server - TTS, Truva Atl
\nport 25 Ajan, Antigen, Email Password Sender - EPS, EPS II, Gip, Gris, Happy99, Hpteam mail, I love you, Kuang2, Magic Horse, MBT (Mail Bombing Trojan),
\nMoscow Email trojan, Naebi, NewApt worm, ProMail trojan, Shtirlitz, Stealth, Tapiras, Terminator, WinPC, WinSpy
\nport 31 Agent 31, Hackers Paradise, Masters Paradise
\nport 41 Deep Throat, Foreplay or Reduced Foreplay
\nport 48 DRAT
\nport 50 DRAT<\/p>\n

OR get the txt file already done here:<\/p>\n

BadPortsBig.txt<\/a><\/p>\n

As I want the port numbers from the 2nd<\/sup> column only at this point, I selected and copied all the text from the browser, and pasted it into a text file on the command line:
\n<\/span><\/p>\n

vi \/badportsBIG.txt<\/span>
\n<\/span><\/p>\n

(If you don't want to do all the examples below in depth, a summary of the commands used to quickly convert the port columns to CSV text, with no duplicates, is on the Notepad page.)<\/p>\n

Press the \"I\" key in vim for text Insert, then the middle mouse button (or both side buttons) to paste all the text in:
\n<\/span><\/p>\n

\"\"
\n<\/span><\/p>\n

Now you just have to strip out columns 1 and 3 which are nicely space delimited, by choosing only column 2 I love linux's ability for this kind of operation:
\n<\/span><\/p>\n

awk '{print $2}' \/badportsBIG.txt
\n<\/span><\/p>\n

This gives you a STDOUT on the screen leaving the original file intact.
\n<\/span><\/p>\n

\"\"
\n<\/span><\/p>\n

Just append this output to a new empty file to keep this data:
\n<\/span><\/p>\n

awk '{print $2}' \/badportsBIG.txt > \/badportsBIGports.txt
\n<\/span><\/p>\n

Now you have a text file with just the port numbers in.
\n<\/span><\/p>\n

From research there appears to be no easy way for Nmap to read port numbers from a file - only hosts\/IP adresses etc. - which I find hard to believe but seems so, as the port numbers have to follow the -p switch, then be comma delimited either before or after the target - in the form e.g.:
\n<\/span><\/p>\n

nmap -p 123,124,125,148<\/em><\/strong> 192.168.1.1
\n<\/span><\/p>\n

(A year later I found the simple answer to that poor assumption of appending file content, using the $(variable) format - linux can manipulate data\u00a0how you want - if\u00a0you know what you're doing!)\u00a0<\/em><\/span><\/strong><\/p>\n

https:\/\/stevepedwards.today\/DebianAdmin\/experiment-with-pipes-redirection-command-substitution-and-variable-expansion\/<\/em><\/strong><\/a><\/span><\/p>\n

This is a pain, because commas now have to be appended between the file's port numbers, and the return delimiters removed so they are all on one big line, so it can be pasted onto Nmap's command line.
\n<\/span><\/p>\n

How do you insert the commas after each port number, then remove the carriage returns\/whitespaces?
\n<\/span><\/p>\n

Note; you have to think like a programmer here about the order you do things. If you removed the white space first, you would be left with one massive number! You can't put the commas in then without severe pain.
\n<\/span><\/p>\n

Research the net as usual or check the UNIX: Database Approach book I mentioned a few Posts back.
\n<\/span><\/p>\n

I originally found that book back in 2001 in New Zealand, as it is big on fundamental programmer's tools like sed, awk, cut, diff and sort for manipulating text at a character level (as databases were written in column and row formats, needing character definitions etc., on the command line only before GUIs back then, as they still can be in SQL\/mysql etc. now, but I never practised with them.
\n<\/span><\/p>\n

Now's the time for a review with an actual real goal at hand.
\n<\/span><\/p>\n

There are some sed and awk examples for commas here:
\n<\/span><\/p>\n

https:\/\/stackoverflow.com\/questions\/8714355\/bash-turning-multi-line-string-into-single-comma-separated<\/span><\/a>
\n<\/span><\/p>\n

Let's try with one on the initial file, as this example also splits off the second column, but does the whole required process in one go, provided cat is used to read my file FIRST, so:
\n<\/span><\/p>\n

awk -vORS=, <\/span>'{ print $2 }'<\/span> file.txt | sed <\/span><\/span>'s\/,$\/\\n\/'<\/span>
\n<\/span><\/span><\/p>\n

becomes:
\n<\/span><\/p>\n

cat \/badportsBIG.txt | awk -vORS=, '{ print $2 }' | sed 's\/,$\/\\n\/'
\n<\/span><\/p>\n

which outputs:
\n<\/span><\/p>\n

\"\"
\n<\/span><\/p>\n

Just what is required to paste the port numbers to nmap. That is so cool!
\n<\/span><\/p>\n

Let's save that STDOUT to a new file:
\n<\/span><\/p>\n

cat \/badportsBIG.txt | awk -vORS=, '{ print $2 }' | sed 's\/,$\/\\n\/' > \/badportsBIGcommas.txt
\n<\/span><\/p>\n

Let's just try nmap with those ports on only one host to start:
\n<\/span><\/p>\n

nmap 192.168.1.1 -p<\/span>
\n0,1,2,5,11,16,17,18,19,20,21,22,23,25,27,28,30,31,37,39,41,44,51,52,53,54,66,69,69,70,79,80,80<\/strong>,81,101,102,103,105,107,109,110,111,113,120,121,123,137,137,138,139,143,146,146,166,170,171,200,201,202,211,212,221,222,230,231,232,285,299,334,335,370,400,401,402,411,420,443,445,455,511,513,514,515,520,555,564,589,600,623,635,650,661,666,667,668,669,680,692,700,777,798,808,831,901,902,903,911,956,991,992,999,1000,1001,1005,1008,1010,1011,1012,1015,1016,1020,1024,1025,1025,1026,1026,1027,1028,1028,1029,1029,1030,1031,1031,1032,1032,1033,1034,1035,1036,1037,1039,1041,1042,1042,1043,1044,1044,1047,1049,1052,1053,1054,1080,1081,1082,1083,1092,1095,1097,1098,1099,1104,1111,1111,1115,1116,1116,1122,1122,1133,1150,1151,1160,1166,1167,1170,1180,1183,1183,1200,1201,1207,1208,1212,1215,1218,1219,1221,1222,1234,1243,1245,1255,1256,1272,1313,1314,1349,1369,1386,1415,1433,1441,1492,1524,1560,1561,1600,1601,1602,1703,1711,1772,1772,1777,1826,1833,1834,1835,1836,1837,1905,1911,1966,1967,1978,1981,1983,1984,1985,1985,1986,1991,1999,2000,2000,2001,2001,2002,2002,2004,2005,2023,2060,2080,2101,2115,2130,2140,2140,2149,2150,2156,2222,2222,2281,2283,2300,2311,2330,2331,2332,2333,2334,2335,2336,2337,2338,2339,2339,2343,2345,2407,2418,2555,2565,2583,2589,2600,2702,2702,2772,2773,2774,2800,2929,2983,2989,3000,3006,3024,3031,3119,3128,3129,3131,3150,3150,3215,3215,3292,3295,3333,3333,3410,3417,3418,3456,3459,3505,3700,3721,3723,3777,3791,3800,3801,3945,3996,3996,3997,3999,4000,4092,4128,4128,4156,4201,4210,4211,4225,4242,4315,4321,4414,4442,4444,4445,4447,4449,4451,4488,4567,4653,4666,4700,4836,5000,5001,5002,5005,5011,5025,5031,5032,5050,5135,5150,5151,5152,5155,5221,5250,5321,5333,5350,5377,5400,5401,5402,5418,5419,5419,5430,5450,5503,5534,5550,5555,5555,5556,5557,5569,5650,5669,5679,5695,5696,5697,5742,5802,5873,5880,5882,5882,5888,5888,5889,5933,6000,6006,6267,6400,6521,6526,6556,6661,6666,6666,6667,6667,6669,6670,6697,6711,6712,6713,6714,6715,6718,6723,6766,6766,6767,6767,6771,6776,6838,6891,6912,6969,6970,7000,7001,7007,7020,7030,7119,7215,7274,7290,7291,7300,7301,7306,7307,7308,7312,7410,7424,7424,7597,7626,7648,7673,7676,7677,7718,7722,7777,7788,7788<\/strong>,7789,7800,7826,7850,7878,7879,7979,7983,8011,8012,8012,8080,8090,8090<\/strong>,8097,8100,8110,8111,8127,8127,8130,8131,8301,8302,8311,8322,8329,8488,8489,8489,8685,8732,8734,8787,8811,8812,8821,8848,8864,8888,9000,9090,9117,9148,9301,9325,9329,9400,9401,9536,9561,9563,9870,9872,9873,9874,9875,9876,9877,9878,9879,9919,9999,10000,10000,10001,10002,10003,10008,10012,10013,10067,10067,10084,10084,10085,10086,10100,10100,10167,10167,10498,10520,10528,10607,10666,10887,10889,11000,11011,11050,11051,11111,11223,11225,11225,11660,11718,11831,11977,11978,11980,12000,12310,12321,12321,12345,12345,12346,12348,12349,12361,12362,12363,12623,12623,12624,12631,12684,12754,12904,13000,13013,13014,13028,13079,13370,13371,13500,13753,14194,14285,14286,14287,14500,14501,14502,14503,15000,15092,15104,15206,15207,15210,15382,15432,15485,15486,15486,15500,15512,15551,15695,15845,15852,16057,16484,16514,16514,16515,16515,16523,16660,16712,16761,16959,17166,17449,17499,17500,17569,17593,17777,18753,19191,19216,20000,20001,20002,20005,20023,20034,20331,20432,20433,21212,21544,21554,21579,21957,22115,22222,22223,22456,22554,22783,22784,22785,23000,23001,23005,23006,23023,23032,23321,23432,23456,23476,23476,23477,23777,24000,24289,25002,25002,25123,25555,25685,25686,25799,25885,25982,26274,26681,27160,27184,27184,27373,27374,27379,27444,27573,27665,28218,28431,28678,29104,29292,29559,29589,29589,29891,29999,30000,30001,30005,30100,30101,30102,30103,30103,30133,30303,30331,30464,30700,30947,31320,31320,31335,31336,31337,31337,31338,31338,31339,31339,31340,31340,31382,31415,31416,31416,31557,31745,31785,31787,31788,31789,31789,31790,31791,31791,31792,31887,32000,32001,32100,32418,32791,33270,33333,33545,33567,33568,33577,33777,33911,34312,34313,34324,34343,34444,34555,35000,35555,35600,36794,37237,37651,38741,38742,40071,40308,40412,40421,40422,40423,40425,40426,41337,41666,43720,44014,44014<\/strong>,44444,44575,44767,44767,45092,45454,45632,45673,46666,46666,47017,47262,47698,47785,47785,47891,48004,48006,48512,49000,49683,49683,49698,50000,50021,50130,50505,50551,50552,50766,50829,50829,51234,51966,52365,52901,53001,54283,54320,54321,55165,55555,55665,55666,56565,57163,57341,57785,58134,58339,59211,60000,60001,60008,60068,60411,60551,60552,60666,61115,61337,61348,61440,61603,61746,61746<\/strong>,61747,61747<\/strong>,61748,61979,62011,63485,64101,65000,65289,65421,65422,65432,65432,65530,65535<\/span><\/p>\n

Starting Nmap 6.40 ( https:\/\/nmap.org<\/a> ) at 2015-08-21 12:52 BST
\n<\/span><\/p>\n

WARNING: Duplicate port number(s) specified. Are you alert enough to be using Nmap? Have some coffee or Jolt(tm).
\n<\/span><\/p>\n

\"\"
\n<\/span><\/p>\n

Ok, what's it being sarcastic about?
\n<\/span><\/p>\n

It's right, the list HAS got many duplicate ports on close inspection, though nmap still works, e.g:
\n<\/span><\/p>\n

80,80,8090,8090,61746,61746 <\/strong>etc.
\n<\/span><\/p>\n

For now though, before searching for more stripping tools, part of the output from the nmap scan for all these ports across 192.168.1.1-254 gave:
\n<\/span><\/p>\n

Nmap scan report for 192.168.1.244
\n<\/span><\/p>\n

Host is up (0.000062s latency).
\n<\/span><\/p>\n

Not shown: 787 closed ports
\n<\/span><\/p>\n

PORT STATE SERVICE
\n<\/span><\/p>\n

22\/tcp open ssh
\n<\/span><\/p>\n

80\/tcp open http
\n<\/span><\/p>\n

139\/tcp open netbios-ssn
\n<\/span><\/p>\n

445\/tcp open microsoft-ds
\n<\/span><\/p>\n

This shows the words closed or filtered in all cases for this scan, (good!) but if you want to find any PCs that have open ports on ANY of these ports (bad!?) for further investigation, then pipe and append grep \"open\" on the nmap line:
\n<\/span><\/p>\n

nmap 192.168.1.1-254 -p 0,1,2 | grep open<\/span>
\n<\/span><\/p>\n

You can then decide what ports are legit on that list and which are suspect from the output e.g:
\n<\/span><\/p>\n

21\/tcp open ftp
\n<\/span><\/p>\n

22\/tcp open ssh
\n<\/span><\/p>\n

23\/tcp open telnet
\n<\/span><\/p>\n

80\/tcp open http
\n<\/span><\/p>\n

443\/tcp open http
\n<\/span><\/p>\n

515\/tcp open printer
\n<\/span><\/p>\n

-----<\/p>\n

So, what tools can remove those duplicates? Uniq!
\n<\/span><\/p>\n

e.g.:<\/p>\n

This leaves the original file unchanged, so you need to pipe the fix to another file e.g.<\/p>\n

cat \/aabbcc.txt<\/span>
\na<\/span>
\na<\/span>
\nb<\/span>
\nb<\/span>
\nc<\/span>
\nc<\/span>
\nd<\/span>
\nd<\/span><\/p>\n

uniq \/aabbcc.txt<\/span>
\na<\/span>
\nb<\/span>
\nc<\/span>
\nd<\/span><\/p>\n

uniq \/aabbcc.txt > \/abcd.txt<\/span><\/p>\n

-------<\/p>\n

Anyway...I don't want to mess up my good original at this point so I'll back it up:
\n<\/span><\/p>\n

cp -v \/badportsBIGcommas.txt \/badportsBIGcommas.bak
\n<\/span><\/p>\n

This sed line is supposed to remove duplicate words, but maybe only if white space delimited so won't work for comma separated text:
\n<\/span><\/p>\n

sed -ri 's\/(.* )1\/1\/g' \/badportsBIGcommas.bak
\n<\/span><\/p>\n

bash: syntax error near unexpected token `('
\n<\/span><\/p>\n

Hmm...doesn't even parse correctly may be due to UTF8 chars in linux or a keymap error for the apostrophes:
\n<\/span><\/p>\n

Maybe it means this:
\n<\/span><\/p>\n

sed -ri 's\/(.* )1\/1\/g'<\/span>
\n<\/strong><\/span><\/span><\/p>\n

as it completed with the new apostrophes. It did not change anything though, seen by looking for known dupes in cat \/badportsBIGcommas.bak e.g. port 80
\n<\/span><\/p>\n

cat \/badportsBIGcommas.bak | grep 80
\n<\/span><\/p>\n

No change as there are 2 consecutive port 80s showing:
\n<\/span><\/p>\n

\"\"
\n<\/span><\/p>\n

Let's check that command works for white space delimited words from his original example:
\n<\/span><\/p>\n

vi \/abc.txt
\n<\/span><\/p>\n

This is is how it works buddy
\nWhat else else<\/strong> you want
\n<\/span><\/p>\n

sed -ri 's\/(.* )1\/1\/g' \/abc.txt
\n<\/span><\/p>\n

This is is how it works buddy
\n<\/span><\/p>\n

What else else<\/strong> you want
\n<\/span><\/p>\n

Nope.
\n<\/span><\/p>\n

This turned out to be another monumental search for such a seemingly simple operation! It may have to be done with a script containing many tools?
\n<\/span><\/p>\n

Best read the info page:
\n<\/span><\/p>\n

info sed<\/span>
\n<\/span><\/span><\/p>\n

It may have been better to remove dupes before the comma addition operation too\u00a6
\n<\/span><\/p>\n

However, it gave a chance to learn some sed format commands from my UNIX:DB book, to see how this s<\/strong>tream ed<\/strong>itor works, and maybe find out why that command above is flawed. Options that can be used with sed include:
\n<\/span><\/p>\n

i \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0inserts before a line
\n<\/em><\/strong><\/span><\/p>\n

a \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0appends after a line
\n<\/em><\/strong><\/span><\/p>\n

c \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0changes line
\n<\/em><\/strong><\/span><\/p>\n

d \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0deletes line
\n<\/em><\/strong><\/span><\/p>\n

p \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0prints line STDOUT
\n<\/em><\/strong><\/span><\/p>\n

q \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0quits after reading up to addressed line
\n<\/em><\/strong><\/span><\/p>\n

r fname \u00a0\u00a0\u00a0\u00a0places content of filename
\n<\/em><\/strong><\/span><\/p>\n

= \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0prints line number addressed
\n<\/em><\/strong><\/span><\/p>\n

s\/s1\/s2\/ \u00a0\u00a0\u00a0\u00a0Subtitutes string s1 by string s2
\n<\/em><\/strong><\/span><\/p>\n

y\/s1\/s2 \u00a0\u00a0\u00a0\u00a0Transforms characters in line by mapping each char in s1 with its counterpart in s2
\n<\/em><\/strong><\/span><\/p>\n

Now you can see what the example above is driving at in that substitution.
\n<\/span><\/p>\n

sed evolved from earlier Unix text editors, ed, then ex, that use an address space context to id lines, so for the 2 line example in \/abc.txt, you can show the whole file like cat would:
\n<\/span><\/p>\n

HPMint stevee # <\/span>sed '2q' \/abc.txt<\/span>
\n<\/span><\/span><\/p>\n

This is is how it works buddy
\n<\/span><\/p>\n

What else else you want<\/span>
\n<\/span><\/span><\/p>\n

Where q is a quit after outputting number of requested lines, or p to print out.
\n<\/span><\/p>\n

This is similar to:
\n<\/span><\/p>\n

HPMint stevee # head -1 \/abc.txt
\n<\/span><\/p>\n

This is is how it works buddy<\/span>
\n<\/span><\/span><\/p>\n

HPMint stevee # tail -1 \/abc.txt
\n<\/span><\/p>\n

What else else you want
\n<\/span><\/p>\n

HPMint stevee # more -1 \/abc.txt
\n<\/span><\/p>\n

This is is how it works buddy<\/span>
\n<\/span><\/span><\/p>\n

--More--(55%)<\/span>
\n<\/span><\/span><\/p>\n

or the initial cursor position for the line number in vim:
\n<\/span><\/p>\n

HPMint stevee # vi +1 \/abc.txt
\n<\/span><\/p>\n

But, a major difference is sed outputs its operation and operants to STDOUT by default, so will show the lines to be operated on, as well as the result, so print them again if the p option is used, so you get line doubling for output e.g:
\n<\/span><\/p>\n

HPMint stevee # sed 'p' \/abc.txt
\n<\/span><\/p>\n

This is is how it works buddy
\n<\/span><\/p>\n

This is is how it works buddy
\n<\/span><\/p>\n

What else else you want
\n<\/span><\/p>\n

What else else you want
\n<\/span><\/p>\n

To negate this, use the -n number switch with no address to id a specific line, so showing the whole file, up to max buffer memory:
\n<\/span><\/p>\n

HPMint stevee # sed -n 'p' \/abc.txt
\n<\/span><\/p>\n

This is is how it works buddy
\n<\/span><\/p>\n

What else else you want<\/span>
\n<\/span><\/span><\/p>\n

The line addressing format shows now in:
\n<\/span><\/p>\n

HPMint stevee # <\/span>sed -n '1p' \/abc.txt<\/span>
\n<\/span><\/span><\/p>\n

This is is how it works buddy<\/span>
\n<\/span><\/span><\/p>\n

HPMint stevee # <\/span>sed -n '2p' \/abc.txt<\/span>
\n<\/span><\/span><\/p>\n

What else else you want
\n<\/span><\/p>\n

sed also responds to the NOT command \"!\" which gives the negative of the above line 1 address, showing line 2 instead or vice versa e.g:
\n<\/span><\/p>\n

HPMint stevee # <\/span>sed -n '1!p' \/abc.txt
\n<\/span><\/span><\/p>\n

What else else you want
\n<\/span><\/p>\n

HPMint stevee # sed -n '2!p' \/abc.txt<\/span>
\n<\/span><\/span><\/p>\n

This is is how it works buddy<\/span>
\n<\/span><\/span><\/p>\n

So, what about the initial example to replace s1 with s2 for the \"else\" word?
\n<\/span><\/p>\n

Can it be understood now from the above research? I still need to know the meaning of the \".\" which I think, from a long time ago, means any single character? So, any single char followed by ANY other combination of none or all chars, because the * is included...?
\n<\/span><\/p>\n

His example was:
\n<\/span><\/p>\n

sed -ri 's\/(.* )1\/1\/g' \/abc.txt<\/span>
\n<\/span><\/span><\/p>\n

From the man page:
\n<\/span><\/p>\n

s\/regexp\/replacement\/
\n<\/span><\/p>\n

Attempt to match regexp against the pattern space. If successful, replace that portion matched with replacement. The replacement may contain the special character & to refer to that portion of the pattern space which matched, and the special escapes \\1 through \\9 to refer to the corresponding matching sub-expressions in the regexp.
\n<\/span><\/p>\n

It looks to me like another \"programmer\" type dude NOT checking his work again...
\n<\/strong><\/span><\/p>\n

The subexpression \\1-9 mentioned there seems to be what he means, but he left out the backslashes! The g is for a global flag so the substitution affects all instances of the character in the addressed line.
\n<\/span><\/p>\n

Now it works, but not exactly, as it seems to have removed a \"d\" from \"buddy\" also, but first, from no change with the missing slashes, to the desired outcome:
\n<\/span><\/p>\n

HPMint stevee # sed -ri 's\/(.*)1\/1\/g' \/abc.txt
\n<\/span><\/p>\n

HPMint stevee # cat \/abc.txt
\n<\/span><\/p>\n

This is is how it works buddy
\n<\/span><\/p>\n

What else else <\/strong>you want<\/span>
\n<\/span><\/span><\/p>\n

HPMint stevee # sed -ri 's\/(.*)\\1\/\\1\/g' \/abc.txt
\n<\/span><\/p>\n

HPMint stevee # cat \/abc.txt
\n<\/span><\/p>\n

This is how it works budy
\n<\/span><\/p>\n

What else <\/strong>you want<\/span>
\n<\/span><\/span><\/p>\n

A missing d also in buddy! I'll just repeat that to see if it does it again, with a ? to show it's my text:
\n<\/span><\/p>\n

HPMint stevee # cat \/abc.txt
\n<\/span><\/p>\n

This is is how it works buddy
\n<\/span><\/p>\n

What else else you want?<\/span>
\n<\/span><\/span><\/p>\n

HPMint stevee # cat \/abc.txt
\n<\/span><\/p>\n

This is how it works budy
\n<\/span><\/p>\n

What else you want?
\n<\/span><\/p>\n

Well, it removed one \"else\" and one \"is\", so I assume the \\1\\1 removes 1 of 2 repeated regular expressions\u00a0in ANY defined string on all lines due to the global \"g\".
\n<\/span><\/p>\n

Seems he could have confined it to a line at a time, say, 2 addressing, from what's been learned so far - ah! like this! (Smartass!)
\n<\/span><\/p>\n

HPMint stevee # sed -ri '2<\/strong>s\/(.*)\\1\/\\1\/g' \/abc.txt
\n<\/span><\/p>\n

HPMint stevee # cat \/abc.txt
\n<\/span><\/p>\n

This is how it works bud<\/strong>dy
\n<\/span><\/p>\n

What else <\/strong>you want?
\n<\/span><\/p>\n

Still, his example was the best I found all day for this type of operation, until reading my book at home, and I have still not found anything this simple to remove numbers in a CSV, single line file, which is what I need here.
\n<\/span><\/p>\n

So, my nostalgic \u00a35 spent on Sumitabha Das' UNIX:DB book has paid off after all. I hope he gives a number substitution example.
\n<\/span><\/p>\n

If I need to change the commas to something else first, to get back to a column view so other tools like \"tr\" may work more easily with lines, another sed example for changing chars in all lines in a file, say a \";\" for a \",\" for the ports file \/badports.csv:
\n<\/span><\/p>\n

\"badportsseparation2.jpg\"
\n<\/span>sed 's\/,\/;\/g' \/badports.csv<\/span>
\n<\/span><\/span><\/p>\n

this changes all commas to semicolons:
\n<\/span><\/p>\n

\"badportsseparation1.jpg\"<\/a><\/p>\n


\n<\/a><\/span>Wish it was that easy for the dupe numbers...
\n<\/span><\/p>\n

So, I should be able to reverse the very first column to commas example above, by replacing all commas (or semicolons now) with newlines (\\n) using that eh?
\n<\/span><\/p>\n

Yep!
\n<\/span><\/p>\n

65432
\n<\/span><\/p>\n

65432
\n<\/span><\/p>\n

65530
\n<\/span><\/p>\n

65535
\n<\/span><\/p>\n

Sorted! As you might say...
\n<\/span><\/p>\n

Now that is back how it was originally in column format, does uniq work\u00a0to remove the dupe numbers? Seems so! My test dupe, <\/span>61746 <\/strong><\/span>from above does not show twice if grepped.
\n<\/span><\/span><\/p>\n

HPMint stevee # <\/span>cat \/badportscolumnreal.txt | uniq | grep 61746<\/span>
\n<\/span><\/span><\/p>\n

61746<\/span>
\n<\/span><\/span><\/p>\n

Lets make the file, finally, with no dupe numbers:
\n<\/span><\/p>\n

cat \/badportscolumnreal.txt | uniq > \/badportscolumn.txt
\n<\/span><\/p>\n

Check for only 1 port 80 too:
\n<\/span><\/p>\n

HPMint stevee # cat \/badportscolumn.txt | grep 80
\n<\/span><\/p>\n

80
\n<\/span><\/p>\n

680
\n<\/span><\/p>\n

Finally, convert this back to a CSV, changing the awk line from above to reflect a single column:
\n<\/span><\/p>\n

\u00a0cat \/badportscolumn.txt | awk -vORS=, '{ print $1 }' | sed 's\/,$\/\\n\/'
\n<\/span><\/p>\n

Note this leaves a trailing comma:
\n<\/span><\/p>\n

65432,65530,65535,<\/span><\/p>\n

Once the extra blank last line I missed in the original file was fixed, there is no comma trail:<\/span><\/p>\n

65432,65530,65535<\/span>
\nHPMint stevee #<\/span> cat \/badportscolumn.txt | awk -vORS=, '{ print $0 }' | sed 's\/,$\/\\n\/'<\/span><\/p>\n

I'm still confused here (it doesn't take much!), as I thought the 2nd arg replaced the first in sed?\u00a0This appears that newlines should be replacing commas here, but they are not and it's working?<\/span><\/p>\n

Hmm - look at which stage does what in order - first cat, which gives the column listing:<\/span><\/p>\n

65432<\/span>
\n65530<\/span>
\n65535<\/span><\/p>\n

HPMint stevee #<\/span> cat \/badportscolumn.txt<\/span><\/p>\n

Then with awk, you get the commas, still with one trailing, even though the original column file last blank line has been fixed\u00a0-\u00a0either $0 or $1 will do:<\/span><\/p>\n

65432,65530,65535,HPMint stevee #<\/span> cat \/badportscolumn.txt | awk -vORS=, '{ print $1 }'<\/span><\/p>\n

With sed added, it removes the\u00a0last comma - so that is where the new line is replacing the final comma ONLY:<\/span><\/p>\n

65000,65289,65421,65422,65432,65530,65535<\/span>
\nHPMint stevee #<\/span> cat \/badportscolumn.txt | awk -vORS=, '{ print $1 }' | sed 's\/,$\/\\n\/'<\/span><\/p>\n

Could sed alone have done this column to commas substitution in the first place? It's not been easy to find how because of the different ways Carriage Returns and Line Feeds are handled, but this example placed a comma at each line end:<\/span><\/p>\n

https:\/\/www.canbike.org\/information-technology\/sed-delete-carriage-returns-and-linefeeds-crlf.html<\/a><\/p>\n

sed 's\/$\/,\/g' \u00a0\/badportscolumn.txt<\/span><\/p>\n

So sed combined with cut can do all required from the start, except remove the final comma:<\/p>\n

cut -d \" \" -f2 BadPortsBig.txt | sed \u00a0's\/$\/,\/g'<\/span><\/p>\n

Note this uses the BASH command line operator \"$\" to signify the \"end of line\" last character (as ^ is the first line char), i.e. just as both<\/p>\n

grep \"^\" \/etc\/passwd<\/span><\/p>\n

grep \"$\" \/etc\/passwd<\/span><\/p>\n

would print any whole file, as every line has to have a first and last character in it, so all lines would be found in both cases!<\/p>\n

Anyway, this\u00a0still leaves the final comma problem. Not a real issue for my current task, as the port list has to be pasted to nmap anyway, as there is no easy way to include the port list after nmap's -p switch.<\/span><\/p>\n

65422,<\/span>
\n65432,<\/span>
\n65530,<\/span>
\n65535,<\/span><\/p>\n

So, for pedantic completeness, the 2nd complex loop example works:<\/span><\/p>\n

where:<\/p>\n

:a  - create a label 'a'\r\nN   - append the next line to the pattern space\r\n$!  - if not the last line\r\nba  - branch (go to) label 'a'\r\ns   - substitute\r\n\/\\n\/     - regex for new line\r\n\/<text>\/ - with text \"<text>\"\r\ng   - global match (as many times as it can)<\/code><\/strong><\/em><\/pre>\n
sed ':a;N;$!ba;s\/\\n\/,\/g' \/badportscolumn.txt<\/span><\/pre>\n
65422,65432,65530,65535<\/span><\/p>\n
In total then, from the first file, to last removed comma, use:<\/p>\n
cut -d \" \" -f2 BadPortsBig.txt | sed ':a;N;$!ba;s\/\\n\/,\/g'<\/span><\/p>\n
The filter\u00a0\u00a0\"tr\" can do the bulk it the most easily from the start.
\n<\/span><\/p>\n
man tr
\n<\/span><\/p>\n
NAME
\n<\/span><\/p>\n
 tr - translate or delete characters
\n<\/span><\/p>\n
SYNOPSIS
\n<\/span><\/p>\n
 tr [OPTION]... SET1 [SET2]
\n<\/span><\/p>\n
DESCRIPTION
\n<\/span><\/p>\n
 Translate, squeeze, and\/or delete characters from standard input, writ
\n<\/span><\/p>\n
 ing to standard output.
\n<\/span><\/p>\n
 -c, -C, --complement
\n<\/span><\/p>\n
 use the complement of SET1
\n<\/span><\/p>\n
 -d, --delete
\n<\/span><\/p>\n
 delete characters in SET1, do not translate
\n<\/span><\/p>\n
 -s, --squeeze-repeats
\n<\/span><\/p>\n
 replace each input sequence of a repeated character that is
\n<\/span><\/p>\n
 listed in SET1 with a single occurrence of that character
\n<\/span><\/p>\n
 -t, --truncate-set1
\n<\/span><\/p>\n
 first truncate SET1 to length of SET2
\n<\/span><\/p>\n
 --help display this help and exit
\n<\/span><\/p>\n
 --version
\n<\/span><\/p>\n
 output version information and exit<\/span><\/p>\n
The format is as simple as can be, with the 2nd expression replacing the first:<\/span><\/p>\n
\u00a0cat \/badportscolumn.txt\u00a0<\/span><\/p>\n
65432
\n65530
\n65535<\/span><\/p>\n
tr '\\n' ',' < \/badportscolumn.txt<\/span><\/p>\n
61747,61748,61979,62011,63485,64101,65000,65289,65421,65422,65432,65530,65535,<\/span><\/p>\n
Note the trailing comma, but how easy is that in one go?!<\/span><\/p>\n
An instant column to CSV file conversion!<\/span><\/p>\n
To save this output as a file you need to feed STDOUT\u00a0into text file as usual e.g :<\/span><\/p>\n
tr '\\n' ',' < \/badportscolumn.txt > \/badportsTR.txt<\/span><\/p>\n
cat \/badportsTR.txt<\/span><\/p>\n
65530,65535,HPMint stevee # cat \/badportsTR.txt<\/span><\/p>\n
Each first and second argument can have multiple chars separated by the | pipe in each apostrophe pair, each mapping to its position counterpart e.g. a to c and b to d, so c replaces a, and d replaces b :<\/span><\/p>\n
 cat \/TRfile.txt<\/span>
\nabaabbaaabbb<\/span>
\ncdccddcccdd<\/span><\/p>\n
tr 'a|b' 'c|d' < \/TRfile.txt<\/span>
\ncdccddcccddd<\/span>
\ncdccddcccddd<\/span><\/p>\n
You see all 2nd arguments - c and d - replaced all first arguments - a and b.\u00a0 <\/span><\/p>\n
Another example from the man page:<\/span><\/p>\n
-d, --delete<\/span>
\n delete characters in SET1, do not translate<\/span><\/p>\n
This should just remove the first line of a's and b's - only for 1 set at a time:<\/span><\/p>\n
cat \/TRfile.txt<\/span>
\nabaabbaaabbb<\/span>
\ncdccddcccddd<\/span><\/p>\n
HPMint stevee # tr -d 'a|b' < \/TRfile.txt<\/span><\/span><\/p>\n
cdccddcccddd<\/span><\/p>\n
or vice versa for 1 set only:<\/span><\/p>\n
tr -d 'c|d' < \/TRfile.txt<\/span>
\nabaabbaaabbb<\/span><\/p>\n
or more chars:<\/span><\/p>\n
HPMint stevee #<\/span>tr -d 'a|b|c' < \/TRfile.txt<\/span><\/p>\n
dddddd<\/span><\/p>\n
SUMMARY - the\u00a0one liner's from start to finish for this problem:<\/strong><\/p>\n
cut -d \" \" -f2 badportsbig.txt | sed ':a;N;$!ba;s\/\\n\/,\/g'<\/span><\/p>\n
awk '{print $2}' badportsbig.txt | sed ':a;N;$!ba;s\/\\n\/,\/g'<\/span><\/p>\n
And to scan the ports:<\/p>\n
awk '{print $2}' badportsbig.txt | uniq > ports.txt<\/span><\/p>\n
 nmap -p $(echo `cat ports.txt`)<\/span><\/p>\n
<\/center>