{"id":4027,"date":"2016-08-13T13:27:14","date_gmt":"2016-08-13T12:27:14","guid":{"rendered":"https:\/\/stevepedwards.today\/DebianAdmin\/?p=4027"},"modified":"2023-10-28T21:59:53","modified_gmt":"2023-10-28T20:59:53","slug":"linux-performance-tools-list-check-utilisation-saturation-errors","status":"publish","type":"post","link":"https:\/\/stevepedwards.today\/DebianAdmin\/linux-performance-tools-list-check-utilisation-saturation-errors\/","title":{"rendered":"Linux Performance Tools List Check Utilisation, Saturation, Errors (USE)"},"content":{"rendered":"
<\/div>\n

<\/path><\/svg><\/i> \"Loading\"<\/p>\n

<\/div>\n

This is a partial list of tools from Brendan's presentations, book and elsewhere it's up to you to learn how to use them, when and why. Many can show rolling O\/P by a 1 sec addition eg mpstat 1<\/span><\/p>\n

sudo apt-get install sysstat\u00a0<\/span><\/p>\n

apt-cache show sysstat<\/span><\/p>\n

Description-en: system performance tools for Linux<\/span>
\n The sysstat package contains the following system performance tools:<\/span>
\n - sar<\/strong>: collects and reports system activity information;<\/span>
\n - iostat<\/strong>: reports CPU utilization and disk I\/O statistics;<\/span>
\n - mpstat<\/strong>: reports global and per-processor statistics;<\/span>
\n - pidstat<\/strong>: reports statistics for Linux tasks (processes);<\/span>
\n - sadf<\/strong>: displays data collected by sar in various formats;<\/span>
\n - nfsiostat<\/strong>: reports I\/O statistics for network filesystems;<\/span>
\n - cifsiostat<\/strong>: reports I\/O statistics for CIFS filesystems.<\/span>
\n .<\/span>
\n The statistics reported by sar deal with I\/O transfer rates,<\/span>
\n paging activity, process-related activities, interrupts,<\/span>
\n network activity, memory and swap space utilization, CPU<\/span>
\n utilization, kernel activities and TTY statistics, among<\/span>
\n others. Both UP and SMP machines are fully supported.<\/span><\/p>\n

uptime\u00a0<\/span><\/p>\n

11:07:33 up 1:33, 3 users, load average: 0.46, 0.59, 0.69\u00a0(1min, 5 min, 15 min)<\/span><\/span><\/p>\n

perl -e 'while(1) {`uptime` =~ \/average: ([\\d.]+)\/; printf(\"% 5s %s\\n\", $1, \"#\" x ($1 * 10)); sleep 3 }'<\/span><\/p>\n

\"perlloadavg.png\"<\/a><\/p>\n

nmon<\/span><\/p>\n

\"nmon.png\"<\/a><\/p>\n

bmon<\/span><\/p>\n

\"bmon.png\"<\/a><\/p>\n

free<\/span><\/p>\n

\"free.png\"<\/a><\/p>\n

sudo atop<\/span><\/p>\n

\"atop.png\"<\/a><\/p>\n

top\/htop<\/span><\/p>\n

\"htop.png\"<\/a><\/p>\n

ps u | head<\/span><\/p>\n

\"ps.png\"<\/a><\/p>\n

netstat -s | head<\/span><\/p>\n

\"netstat.png\"<\/a><\/p>\n

sudo iotop<\/span><\/p>\n

\"iotop.png\"<\/a><\/p>\n

dstat <\/span><\/p>\n

\"dstat.png\"<\/a><\/p>\n

vmstat\u00a0(-sm)<\/span><\/span><\/p>\n

\"vmstat.png\"<\/a><\/p>\n

slabtop<\/span><\/p>\n

\"slabtop.png\"<\/a><\/p>\n

iostat<\/span><\/p>\n

\"iostat.png\"<\/a><\/p>\n

man strace (USE WITH KNOWLEDGE AND CAUTION)<\/span><\/strong><\/span><\/p>\n

DESCRIPTION<\/span><\/p>\n

In the simplest case strace runs the specified command until it exits. It intercepts and records the system calls which are called by a process and the\u00a0<\/span>signals which are received by a process. The name of each system call, its arguments and its return value are printed on standard error or to the file\u00a0<\/span>specified with the -o option.<\/span><\/p>\n

strace is a useful diagnostic, instructional, and debugging tool. System administrators, diagnosticians and trouble-shooters will find it invaluable for\u00a0<\/span>solving problems with programs for which the source is not readily available since they do not need to be recompiled in order to trace them. Students,\u00a0<\/span>hackers and the overly-curious will find that a great deal can be learned about a system and its system calls by tracing even ordinary programs.<\/span><\/p>\n

\"strace.png\"<\/a><\/p>\n

sudo mpstat -P 0,1,2,3<\/span><\/p>\n

\"mpstat0123.png\"<\/a><\/p>\n

nicstat<\/span><\/p>\n

\"nicstat.png\"<\/a><\/p>\n

pidstat | head<\/span><\/p>\n

\"pidstat.png\"<\/a><\/p>\n

sudo lsof | head<\/span><\/p>\n

\"lsof.png\"<\/a><\/p>\n

ss -a | head<\/span><\/p>\n

\"ss.png\"<\/a><\/p>\n

ethtool -i eth0<\/span><\/p>\n

\"ethtool.png\"<\/a><\/p>\n

sudo sar (-BHRrSsw)<\/span><\/span><\/p>\n

\"sar.png\"<\/a><\/p>\n

If you run sar, then a nice addition is sadf as it formats sar data for other use - DB, csv etc. For example, it can show 6 cpu utilizations for each 4 cores which can be grepped nicely for an easy summary:<\/p>\n

sadf -p -P 0,1,2,3 | head -24<\/span><\/p>\n

\"sadf24.png\"<\/a><\/p>\n

Each usage type can be grepped quickly showing 10 min separations for a clear timeline of usage e.g:<\/p>\n

sadf -p -P 0,1,2,3 | grep user<\/span><\/p>\n

\"sadfuser.png\"<\/a>\"sadfsystem.png\"<\/a>These columns could be stripped and graphed quite easily via awk \/ gnuplot etc e.g:<\/p>\n

sadf -p -P 0,1,2,3 | grep user | grep cpu0 | awk '{print $3,$4,$7}' > data1.dat<\/span><\/p>\n

2016-08-17 11:55:01 10.28<\/span>
\n2016-08-17 12:05:01 5.27<\/span>
\n2016-08-17 12:15:01 17.73<\/span>
\n2016-08-17 12:25:01 13.63<\/span>
\n2016-08-17 12:35:01 4.38<\/span>
\n2016-08-17 12:45:01 7.67<\/span>
\n2016-08-17 12:55:01 8.40<\/span>
\n2016-08-17 13:05:01 11.02<\/span>
\n2016-08-17 13:15:01 13.57<\/span>
\n2016-08-17 13:25:01 10.07<\/span>
\n2016-08-17 13:35:01 5.68<\/span>
\n2016-08-17 13:45:01 3.20<\/span>
\n2016-08-17 13:55:01 4.32<\/span>
\n2016-08-17 14:05:01 2.40<\/span><\/p>\n

\"userCPU.png\"<\/a><\/p>\n

cifsiostat<\/span><\/p>\n

\"cifsiostat.png\"<\/a><\/p>\n

Hard drive cached reads:\u00a0sudo hdparm -T \/dev\/sd*<\/span><\/p>\n

\"hdparmTreads.png\"<\/a><\/p>\n

stevee@Dell490 ~ $ sudo hdparm -t \/dev\/sd*<\/span><\/p>\n

\/dev\/sda:<\/span>
\n Timing buffered disk reads: 316 MB in 3.02 seconds = 104.77 MB\/sec<\/span><\/p>\n

\/dev\/sda1:<\/span>
\n Timing buffered disk reads: 100 MB in 0.96 seconds = 104.44 MB\/sec<\/span><\/p>\n

\/dev\/sda2:<\/span>
\n Timing buffered disk reads: 312 MB in 3.00 seconds = 103.97 MB\/sec<\/span><\/p>\n

\/dev\/sda3:<\/span>
\n Timing buffered disk reads: 290 MB in 3.01 seconds = 96.24 MB\/sec<\/span><\/p>\n

\/dev\/sdb:<\/span>
\n Timing buffered disk reads: 222 MB in 3.01 seconds = 73.76 MB\/sec<\/span><\/p>\n

\/dev\/sdb1:<\/span>
\n Timing buffered disk reads: 222 MB in 3.01 seconds = 73.64 MB\/sec<\/span><\/p>\n

\/dev\/sdb2:<\/span>
\n Timing buffered disk reads: read(2097152) returned 1024 bytes<\/span><\/p>\n

\/dev\/sdb5:<\/span>
\n Timing buffered disk reads: 114 MB in 3.04 seconds = 37.45 MB\/sec<\/span><\/p>\n

blkiomon<\/span> EXAMPLES<\/span>
\n To get I\/O statistics for \/dev\/sdw every 10 seconds for a period of one<\/span>
\n hour, use the following command:<\/span><\/p>\n

% blktrace \/dev\/sdw -a issue -a complete -w 3600 -o - | blkiomon -I<\/span>
\n 10 -h -<\/span><\/p>\n

sudo blktrace \/dev\/sda -a issue -a complete -w 20 -o iofile.txt <\/span>
\n=== sda ===<\/span>
\n CPU 0: 0 events, 0 KiB data<\/span>
\n CPU 1: 1 events, 1 KiB data<\/span>
\n Total: 1 events (dropped 0), 1 KiB data<\/span><\/p>\n

Webpage graph<\/p>\n

sudo apt-get install bandwidthd<\/span><\/p>\n

browser address bar: \/var\/lib\/bandwidthd\/htdocs\/index.html<\/p>\n

\"bandwidthd.png\"<\/a><\/p>\n

or add it as an alias in Apache in the virtual host block e.g. remote view it on your apache server \/\/dellmint\/bd<\/p>\n

vi \/etc\/apache2\/sites-available\/000-default.conf<\/span><\/p>\n

Alias \/bd<\/strong> \/var\/lib\/bandwidthd\/htdocs<\/span>
\n<Directory \/var\/lib\/bandwidthd\/htdocs><\/span>
\n Options All<\/span>
\n AllowOverride All<\/span>
\n order allow,deny<\/span>
\n allow from all<\/span>
\n<\/Directory><\/span><\/p>\n

INFO RE SYSTEMTAP!! You may need a development kernel with debugging installed\/enabled! Most tutorials COMPLETELY fail to mention this - and you wonder why the cmd examples don't work!<\/p>\n

see here:\u00a0https:\/\/posulliv.github.io\/2010\/02\/26\/installing-stap\/<\/span><\/a><\/p>\n

\"SystemTap needs to have a debug version of the kernel available. Unfortunately, installing the debug information for a kernel on ubuntu is not a trivial operation to perform. In fact, there is a bug on Launchpad about this issue.\"<\/em><\/strong><\/p>\n

If you want to experiment - use a test PC with a dev kernel...<\/p>\n

https:\/\/www.suse.com\/documentation\/sled11\/singlehtml\/book_sle_tuning\/book_sle_tuning.html#sec.tuning.systemtap.setup<\/a><\/span><\/p>\n

If you HAVE installed just systemtap and\u00a0systemtap-doc, the list of available probes can be seen with:<\/p>\n

man probe::[TAB]<\/span><\/p>\n

Display all 289 possibilities? (y or n)<\/span><\/p>\n

re the above stap kernel info - later, with my std kernel, I did get one of Brendan's stap<\/span> cmds to work to create a flame graph in \/Flamegraph as root <\/strong>(but much beyond this one cmd,\u00a0it seems you need the full dev install a la suse link):<\/p>\n

https:\/\/www.brendangregg.com\/FlameGraphs\/cpuflamegraphs.html#perf<\/span><\/a><\/p>\n

stap -s 32 -D MAXBACKTRACE=100 -D MAXSTRINGLEN=4096 -D MAXMAPENTRIES=10240 \\<\/span>
\n -D MAXACTION=10000 -D STP_OVERLOAD_THRESHOLD=5000000000 --all-modules \\<\/span>
\n -ve 'global s; probe timer.profile { s[backtrace()] <<< 1; } <\/span>
\n probe end { foreach (i in s+) { print_stack(i);<\/span>
\n printf(\"\\t%d\\n\", @count(s[i])); } } probe timer.s(60) { exit(); }' \\<\/span>
\n > out.stap-stacks<\/span>
\n\u00a0.\/stackcollapse-stap.pl out.stap-stacks > out.stap-folded<\/span>
\n\u00a0cat out.stap-folded | .\/flamegraph.pl > stap-kernel.svg<\/span><\/p>\n

\"stapflame.png\"<\/a><\/p>\n

You can\u00a0use perf for\u00a0Brendan's script examples such as his \"perf Hacktogram\"<\/p>\n

https:\/\/www.brendangregg.com\/blog\/2014-07-10\/perf-hacktogram.html<\/span><\/a><\/p>\n

in your Flamegraph directory (previous Post) as root - create this histogram in the terminal:<\/p>\n

\"nethisto.png\"<\/a><\/p>\n

 <\/p>\n","protected":false},"excerpt":{"rendered":"

<\/div>\n

<\/path><\/svg><\/i> \"Loading\"<\/p>\n

<\/div>\n

This is a partial list of tools from Brendan's presentations, book and elsewhere it's up to you to learn how to use them, when and why. Many can show rolling O\/P by a 1 sec addition eg mpstat 1 sudo apt-get install sysstat\u00a0 apt-cache show sysstat Description-en: system performance tools for Linux The sysstat package ...\u00a0 Linux Performance Tools List Check Utilisation, Saturation, Errors (USE)<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[2],"tags":[],"class_list":["post-4027","post","type-post","status-publish","format-standard","hentry","category-uncategorized"],"a3_pvc":{"activated":true,"total_views":2,"today_views":0},"_links":{"self":[{"href":"https:\/\/stevepedwards.today\/DebianAdmin\/wp-json\/wp\/v2\/posts\/4027","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/stevepedwards.today\/DebianAdmin\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/stevepedwards.today\/DebianAdmin\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/stevepedwards.today\/DebianAdmin\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/stevepedwards.today\/DebianAdmin\/wp-json\/wp\/v2\/comments?post=4027"}],"version-history":[{"count":2,"href":"https:\/\/stevepedwards.today\/DebianAdmin\/wp-json\/wp\/v2\/posts\/4027\/revisions"}],"predecessor-version":[{"id":10052,"href":"https:\/\/stevepedwards.today\/DebianAdmin\/wp-json\/wp\/v2\/posts\/4027\/revisions\/10052"}],"wp:attachment":[{"href":"https:\/\/stevepedwards.today\/DebianAdmin\/wp-json\/wp\/v2\/media?parent=4027"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/stevepedwards.today\/DebianAdmin\/wp-json\/wp\/v2\/categories?post=4027"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/stevepedwards.today\/DebianAdmin\/wp-json\/wp\/v2\/tags?post=4027"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}