{"id":3962,"date":"2016-08-11T19:31:02","date_gmt":"2016-08-11T18:31:02","guid":{"rendered":"https:\/\/stevepedwards.today\/DebianAdmin\/?p=3962"},"modified":"2022-09-22T20:29:55","modified_gmt":"2022-09-22T19:29:55","slug":"flame-graphs-by-www-brendangregg-com-intro-to-system-profiling","status":"publish","type":"post","link":"https:\/\/stevepedwards.today\/DebianAdmin\/flame-graphs-by-www-brendangregg-com-intro-to-system-profiling\/","title":{"rendered":"Flame Graphs by www.brendangregg.com &#8211; Intro to System Profiling and Admin Tools"},"content":{"rendered":"<div class=\"pvc_clear\"><\/div>\n<p id=\"pvc_stats_3962\" class=\"pvc_stats all  \" data-element-id=\"3962\" style=\"\"><i class=\"pvc-stats-icon medium\" aria-hidden=\"true\"><svg aria-hidden=\"true\" focusable=\"false\" data-prefix=\"far\" data-icon=\"chart-bar\" role=\"img\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\" viewBox=\"0 0 512 512\" class=\"svg-inline--fa fa-chart-bar fa-w-16 fa-2x\"><path fill=\"currentColor\" d=\"M396.8 352h22.4c6.4 0 12.8-6.4 12.8-12.8V108.8c0-6.4-6.4-12.8-12.8-12.8h-22.4c-6.4 0-12.8 6.4-12.8 12.8v230.4c0 6.4 6.4 12.8 12.8 12.8zm-192 0h22.4c6.4 0 12.8-6.4 12.8-12.8V140.8c0-6.4-6.4-12.8-12.8-12.8h-22.4c-6.4 0-12.8 6.4-12.8 12.8v198.4c0 6.4 6.4 12.8 12.8 12.8zm96 0h22.4c6.4 0 12.8-6.4 12.8-12.8V204.8c0-6.4-6.4-12.8-12.8-12.8h-22.4c-6.4 0-12.8 6.4-12.8 12.8v134.4c0 6.4 6.4 12.8 12.8 12.8zM496 400H48V80c0-8.84-7.16-16-16-16H16C7.16 64 0 71.16 0 80v336c0 17.67 14.33 32 32 32h464c8.84 0 16-7.16 16-16v-16c0-8.84-7.16-16-16-16zm-387.2-48h22.4c6.4 0 12.8-6.4 12.8-12.8v-70.4c0-6.4-6.4-12.8-12.8-12.8h-22.4c-6.4 0-12.8 6.4-12.8 12.8v70.4c0 6.4 6.4 12.8 12.8 12.8z\" class=\"\"><\/path><\/svg><\/i> <img loading=\"lazy\" decoding=\"async\" width=\"16\" height=\"16\" alt=\"Loading\" src=\"https:\/\/stevepedwards.today\/DebianAdmin\/wp-content\/plugins\/page-views-count\/ajax-loader-2x.gif\" border=0 \/><\/p>\n<div class=\"pvc_clear\"><\/div>\n<p><span style=\"color: #0000ff;\"><span style=\"font-size: small;\"><i>Profiling builds a picture of a target that can be studied and understood...by sampling the system state at timed intervals...provides a <\/i><\/span><strong><span style=\"font-size: small;\"><i>coarse<\/i><\/span><\/strong><span style=\"font-size: small;\"><i> view of activity, depending on sampling rate. <\/i><span style=\"color: #ffffff;\">Systems Performance, p30, Brendan Gregg<\/span><\/span><\/span><\/p>\n<p><strong>Flame Graphs<\/strong> are a <strong>stroke of\u00a0genius<\/strong> and <strong>an advanced topic.<\/strong>\u00a0I discovered them only today. They give\u00a0an insight into the inner workings of an OS kernel so prove invaluable for core kernel or user level fault finding or performance profiling for the system itself, or software running on it. It seems a very complex topic to start with (and is for a professional level understanding!),\u00a0but you can learn a lot in a few hours if you watch\u00a0Brendan's videos and get through the first steps to download and run his well thought out and script generous examples, that I have completed below from his site\u00a0- YOU NEED to be ROOT - sudo won't do!:<\/p>\n<p><a href=\"https:\/\/www.brendangregg.com\/FlameGraphs\/cpuflamegraphs.html#perf\"><span style=\"color: #00ff00;\">https:\/\/www.brendangregg.com\/FlameGraphs\/cpuflamegraphs.html#perf<\/span><\/a><br \/>\n<iframe loading=\"lazy\" src=\"https:\/\/www.youtube.com\/embed\/FI2NqdAA4Sc?autoplay=1&amp;version=3&amp;loop=1&amp;playlist=FI2NqdAA4Sc\" width=\"560\" height=\"315\" frameborder=\"0\" align=\"left\" allowfullscreen=\"allowfullscreen\"><\/iframe><\/p>\n<p><strong>Don't be put off by the seeming tech complexity of the info below - I have done the initial research, hit the hurdles and got over\u00a0them for you!<\/strong><\/p>\n<p><a href=\"https:\/\/stevepedwards.today\/DebianAdmin\/wp-content\/uploads\/2016\/08\/perfconflicts.png\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter wp-image-4083\" src=\"https:\/\/stevepedwards.today\/DebianAdmin\/wp-content\/uploads\/2016\/08\/perfconflicts-1024x404.png\" alt=\"perfconflicts.png\" width=\"699\" height=\"275\" \/><\/a><\/p>\n<p>If you can get them created and browser view them, then you can learn the tech side of their value\u00a0by watching Brendan's vid presentations for their overview later. perf is included in linux tools (your kernel version e.g)<\/p>\n<p>(I have dependency problems with kernel -38 on my PCs but not -39 on the laptop).<\/p>\n<p><strong><span style=\"color: #0000ff;\">uname -a<\/span><\/strong><br \/>\n<span style=\"color: #ff0000;\">Linux AMD 3.16.0-39-generic<\/span><\/p>\n<p><strong><span style=\"color: #0000ff;\">apt-get update<\/span><\/strong><\/p>\n<p><strong><span style=\"color: #0000ff;\">apt-get upgrade<\/span><\/strong><\/p>\n<p><strong><span style=\"color: #0000ff;\">apt-get install linux-tools-common<\/span><\/strong><strong><span style=\"color: #0000ff;\">\u00a0linux-tools-generic linux-tools-3.16.0-39-generic\u00a0<\/span><\/strong><\/p>\n<p><strong><span style=\"color: #0000ff;\">apt-get install git<\/span><\/strong><\/p>\n<p><strong><span style=\"color: #0000ff;\">git clone https:\/\/github.com\/brendangregg\/FlameGraph<\/span><\/strong><br \/>\n<span style=\"color: #ff0000;\">Cloning into 'FlameGraph'...<\/span><br \/>\n<span style=\"color: #ff0000;\">remote: Counting objects: 695, done.<\/span><br \/>\n<span style=\"color: #ff0000;\">remote: Total 695 (delta 0), reused 0 (delta 0), pack-reused 695<\/span><br \/>\n<span style=\"color: #ff0000;\">Receiving objects: 100% (695\/695), 1.09 MiB | 463.00 KiB\/s, done.<\/span><br \/>\n<span style=\"color: #ff0000;\">Resolving deltas: 100% (383\/383), done.<\/span><br \/>\n<span style=\"color: #ff0000;\">Checking connectivity... done.<\/span><\/p>\n<p>Now try Brendan's examples:<\/p>\n<p><strong><span style=\"color: #0000ff;\">cd FlameGraph<\/span><\/strong><\/p>\n<p><strong><span style=\"color: #0000ff;\">perf record -F 99 -a -g -- sleep 60 <\/span><\/strong><\/p>\n<p><span style=\"color: #ff0000;\">[ perf record: Woken up 5 times to write data ]<\/span><br \/>\n<span style=\"color: #ff0000;\">[ perf record: Captured and wrote 2.051 MB perf.data (~89589 samples) ]<\/span><\/p>\n<p><strong><span style=\"color: #0000ff;\">perf script | .\/stackcollapse-perf.pl &gt; out.perf-folded <\/span><\/strong><\/p>\n<p><span style=\"color: #ff0000;\">Failed to open \/tmp\/perf-14384.map, continuing without symbols...<\/span><\/p>\n<p><strong><span style=\"color: #0000ff;\">\u00a0.\/flamegraph.pl out.perf-folded &gt; perf-kernel.svg<\/span><\/strong><\/p>\n<p><a href=\"https:\/\/stevepedwards.today\/DebianAdmin\/wp-content\/uploads\/2016\/08\/flamerw.png\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter size-medium wp-image-3966\" src=\"https:\/\/stevepedwards.today\/DebianAdmin\/wp-content\/uploads\/2016\/08\/flamerw.png\" alt=\"flamerw.png\" width=\"1154\" height=\"589\" \/><\/a><\/p>\n<p>This active code SVG graphic is a security risk on WordPress so cannot upload, so you have to create your own below or see the demo folder, to view yours in a browser.<\/p>\n<p><strong><span style=\"color: #0000ff;\">grep -v cpu_idle out.perf-folded | .\/flamegraph.pl &gt; nonidle.svg<\/span><\/strong><\/p>\n<p><strong><span style=\"color: #0000ff;\">grep ext4 out.perf-folded | .\/flamegraph.pl &gt; ext4internals.svg<\/span><\/strong><\/p>\n<p><strong><span style=\"color: #0000ff;\">egrep 'system_call.*sys_(read|write)' out.perf-folded | .\/flamegraph.pl &gt; rw.svg<\/span><\/strong><\/p>\n<p>Reading the actual\u00a0.\/stackcollapse-perf.pl script mentions\u00a0<em>trace<\/em> problems if you have them:<\/p>\n<p><strong><span style=\"color: #0000ff;\">less .\/stackcollapse-perf.pl<\/span><\/strong><\/p>\n<p><strong><em>\"# The output of \"perf script\" should include stack traces. If these are missing<\/em><\/strong><br \/>\n<strong><em># for you, try manually selecting the perf script output; eg:<\/em><\/strong><br \/>\n<strong><em>#<\/em><\/strong><br \/>\n<strong><em># perf script -f comm,pid,tid,cpu,time,event,ip,sym,dso,trace | ...\"<\/em><\/strong><\/p>\n<p>Dir contents now is:<\/p>\n<p><a href=\"https:\/\/stevepedwards.today\/DebianAdmin\/wp-content\/uploads\/2016\/08\/Flamegraphdir-1.png\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter size-medium wp-image-3998\" src=\"https:\/\/stevepedwards.today\/DebianAdmin\/wp-content\/uploads\/2016\/08\/Flamegraphdir-1.png\" alt=\"Flamegraphdir.png\" width=\"658\" height=\"607\" \/><\/a><\/p>\n<p><span style=\"color: #0000ff;\">getfacl FlameGraph\/<\/span><br \/>\n<span style=\"color: #ff0000;\"># file: FlameGraph\/<\/span><br \/>\n<strong><span style=\"color: #ff0000;\"># owner: root<\/span><\/strong><br \/>\n<strong><span style=\"color: #ff0000;\"># group: root<\/span><\/strong><br \/>\n<strong><span style=\"color: #ff0000;\">user::rwx<\/span><\/strong><br \/>\n<span style=\"color: #ff0000;\">group::r-x<\/span><br \/>\n<span style=\"color: #ff0000;\">other::r-x<\/span><\/p>\n<p>Full \"perf stat\" system access is available only for root\u00a0as the Mint kernel is compiled as paranoid level 1 in:<\/p>\n<p><span style=\"color: #0000ff;\"> \u00a0sudo cat \/proc\/sys\/kernel\/perf_event_paranoid<\/span><br \/>\n<span style=\"color: #ff0000;\">1<\/span><\/p>\n<p>If you run as a user, you get:<\/p>\n<p><span style=\"color: #0000ff;\">perf stat -B -ecycles:u,instructions:u -a dd if=\/dev\/zero of=\/dev\/null count=2000000<\/span><br \/>\n<span style=\"color: #ff0000;\">Error:<\/span><br \/>\n<span style=\"color: #ff0000;\">You may not have permission to collect system-wide stats.<\/span><br \/>\n<span style=\"color: #ff0000;\">Consider tweaking \/proc\/sys\/kernel\/perf_event_paranoid:<\/span><br \/>\n<span style=\"color: #ff0000;\"> -1 - Not paranoid at all<\/span><br \/>\n<span style=\"color: #ff0000;\"> 0 - Disallow raw tracepoint access for unpriv<\/span><br \/>\n<strong><span style=\"color: #ff0000;\"> 1 - Disallow cpu events for unpriv<\/span><\/strong><br \/>\n<span style=\"color: #ff0000;\"> 2 - Disallow kernel profiling for unpriv<\/span><\/p>\n<p>So what are Flame Graphs, what do they show and how are they used?<\/p>\n<p>You can see the basis of the idea from the bifurcations in the samples view generated by perf record that first generates data for the perf.data file, which is then read back by perf report here:<\/p>\n<p><span style=\"color: #0000ff;\">perf record -a -g -F 997 sleep 10<\/span><\/p>\n<p><span style=\"color: #0000ff;\">perf report --stdio<\/span><\/p>\n<p><a href=\"https:\/\/stevepedwards.today\/DebianAdmin\/wp-content\/uploads\/2016\/08\/perfrptbifur.png\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter size-medium wp-image-4125\" src=\"https:\/\/stevepedwards.today\/DebianAdmin\/wp-content\/uploads\/2016\/08\/perfrptbifur.png\" alt=\"perfrptbifur.png\" width=\"690\" height=\"459\" \/><\/a><\/p>\n<p><iframe loading=\"lazy\" title=\"LISA13 - Blazing Performance with Flame Graphs\" width=\"1778\" height=\"1000\" src=\"https:\/\/www.youtube.com\/embed\/nZfNehCzGdw?feature=oembed\" frameborder=\"0\" allow=\"accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture\" allowfullscreen><\/iframe><\/p>\n<p>There are more videos to check out on YouTube for this topic and others by Brendan; this pair on Performance Tools are important to Admins:<\/p>\n<p><iframe loading=\"lazy\" title=\"Linux Performance Tools, Brendan Gregg, part 1 of 2\" width=\"1778\" height=\"1000\" src=\"https:\/\/www.youtube.com\/embed\/FJW8nGV4jxY?feature=oembed\" frameborder=\"0\" allow=\"accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture\" allowfullscreen><\/iframe><\/p>\n<p><iframe loading=\"lazy\" title=\"Linux Performance Tools, Brendan Gregg, part 2 of 2\" width=\"1778\" height=\"1000\" src=\"https:\/\/www.youtube.com\/embed\/zrr2nUln9Kk?feature=oembed\" frameborder=\"0\" allow=\"accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture\" allowfullscreen><\/iframe><\/p>\n<p>A list of events for possible analysis can be viewed via:<\/p>\n<p><span style=\"color: #0000ff;\">sudo perf list | wc -l<\/span><br \/>\n<span style=\"color: #ff0000;\">1488<\/span><\/p>\n<p><a href=\"https:\/\/stevepedwards.today\/DebianAdmin\/wp-content\/uploads\/2016\/08\/perlist.png\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter size-medium wp-image-4079\" src=\"https:\/\/stevepedwards.today\/DebianAdmin\/wp-content\/uploads\/2016\/08\/perlist.png\" alt=\"perlist.png\" width=\"690\" height=\"743\" \/><\/a><\/p>\n<p>Examples on Brendan's perf page:\u00a0<a href=\"https:\/\/www.brendangregg.com\/perf.html\"><span style=\"color: #00ff00;\">https:\/\/www.brendangregg.com\/perf.html<\/span><\/a><\/p>\n<p>Now you just have to find a PC\/Server with a performance issue and apply the knowledge! Easy right!? Hmm...make sure you charge a fortune if you do fix it using these means...you deserve it!<\/p>\n<p>Welcome to the mind bending world of hard core OS tech...It's beyond me mostly, as I'm a shit programmer\u00a0but no good reason given the video examples, why anyone\u00a0may not find a given problem with this tool - say a memory leaking program, or one that is too resource heavy from bugs\/bad code. Finding the fault\u00a0is the first step to a fix...<\/p>\n","protected":false},"excerpt":{"rendered":"<div class=\"pvc_clear\"><\/div>\n<p id=\"pvc_stats_3962\" class=\"pvc_stats all  \" data-element-id=\"3962\" style=\"\"><i class=\"pvc-stats-icon medium\" aria-hidden=\"true\"><svg aria-hidden=\"true\" focusable=\"false\" data-prefix=\"far\" data-icon=\"chart-bar\" role=\"img\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\" viewBox=\"0 0 512 512\" class=\"svg-inline--fa fa-chart-bar fa-w-16 fa-2x\"><path fill=\"currentColor\" d=\"M396.8 352h22.4c6.4 0 12.8-6.4 12.8-12.8V108.8c0-6.4-6.4-12.8-12.8-12.8h-22.4c-6.4 0-12.8 6.4-12.8 12.8v230.4c0 6.4 6.4 12.8 12.8 12.8zm-192 0h22.4c6.4 0 12.8-6.4 12.8-12.8V140.8c0-6.4-6.4-12.8-12.8-12.8h-22.4c-6.4 0-12.8 6.4-12.8 12.8v198.4c0 6.4 6.4 12.8 12.8 12.8zm96 0h22.4c6.4 0 12.8-6.4 12.8-12.8V204.8c0-6.4-6.4-12.8-12.8-12.8h-22.4c-6.4 0-12.8 6.4-12.8 12.8v134.4c0 6.4 6.4 12.8 12.8 12.8zM496 400H48V80c0-8.84-7.16-16-16-16H16C7.16 64 0 71.16 0 80v336c0 17.67 14.33 32 32 32h464c8.84 0 16-7.16 16-16v-16c0-8.84-7.16-16-16-16zm-387.2-48h22.4c6.4 0 12.8-6.4 12.8-12.8v-70.4c0-6.4-6.4-12.8-12.8-12.8h-22.4c-6.4 0-12.8 6.4-12.8 12.8v70.4c0 6.4 6.4 12.8 12.8 12.8z\" class=\"\"><\/path><\/svg><\/i> <img loading=\"lazy\" decoding=\"async\" width=\"16\" height=\"16\" alt=\"Loading\" src=\"https:\/\/stevepedwards.today\/DebianAdmin\/wp-content\/plugins\/page-views-count\/ajax-loader-2x.gif\" border=0 \/><\/p>\n<div class=\"pvc_clear\"><\/div>\n<p>Profiling builds a picture of a target that can be studied and understood...by sampling the system state at timed intervals...provides a coarse view of activity, depending on sampling rate. Systems Performance, p30, Brendan Gregg Flame Graphs are a stroke of\u00a0genius and an advanced topic.\u00a0I discovered them only today. They give\u00a0an insight into the inner workings <a href=\"https:\/\/stevepedwards.today\/DebianAdmin\/flame-graphs-by-www-brendangregg-com-intro-to-system-profiling\/\" class=\"more-link\">...<span class=\"screen-reader-text\">\u00a0 Flame Graphs by www.brendangregg.com &#8211; Intro to System Profiling and Admin Tools<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[2],"tags":[],"class_list":["post-3962","post","type-post","status-publish","format-standard","hentry","category-uncategorized"],"a3_pvc":{"activated":true,"total_views":2,"today_views":0},"_links":{"self":[{"href":"https:\/\/stevepedwards.today\/DebianAdmin\/wp-json\/wp\/v2\/posts\/3962","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/stevepedwards.today\/DebianAdmin\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/stevepedwards.today\/DebianAdmin\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/stevepedwards.today\/DebianAdmin\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/stevepedwards.today\/DebianAdmin\/wp-json\/wp\/v2\/comments?post=3962"}],"version-history":[{"count":1,"href":"https:\/\/stevepedwards.today\/DebianAdmin\/wp-json\/wp\/v2\/posts\/3962\/revisions"}],"predecessor-version":[{"id":8942,"href":"https:\/\/stevepedwards.today\/DebianAdmin\/wp-json\/wp\/v2\/posts\/3962\/revisions\/8942"}],"wp:attachment":[{"href":"https:\/\/stevepedwards.today\/DebianAdmin\/wp-json\/wp\/v2\/media?parent=3962"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/stevepedwards.today\/DebianAdmin\/wp-json\/wp\/v2\/categories?post=3962"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/stevepedwards.today\/DebianAdmin\/wp-json\/wp\/v2\/tags?post=3962"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}