introduction to offcpu time flame graphs
TRANSCRIPT
![Page 2: Introduction to offCPU Time Flame Graphs](https://reader034.vdocuments.mx/reader034/viewer/2022042600/58a2bf591a28ab4c028b4cf2/html5/thumbnails/2.jpg)
Classic Flame Graphs are
onCPU time Flame Graphs per se.
![Page 3: Introduction to offCPU Time Flame Graphs](https://reader034.vdocuments.mx/reader034/viewer/2022042600/58a2bf591a28ab4c028b4cf2/html5/thumbnails/3.jpg)
![Page 4: Introduction to offCPU Time Flame Graphs](https://reader034.vdocuments.mx/reader034/viewer/2022042600/58a2bf591a28ab4c028b4cf2/html5/thumbnails/4.jpg)
We are already relying on them to optimizeour Lua WAF & Lua CDN Brain (cfcheck)
![Page 5: Introduction to offCPU Time Flame Graphs](https://reader034.vdocuments.mx/reader034/viewer/2022042600/58a2bf591a28ab4c028b4cf2/html5/thumbnails/5.jpg)
![Page 6: Introduction to offCPU Time Flame Graphs](https://reader034.vdocuments.mx/reader034/viewer/2022042600/58a2bf591a28ab4c028b4cf2/html5/thumbnails/6.jpg)
![Page 7: Introduction to offCPU Time Flame Graphs](https://reader034.vdocuments.mx/reader034/viewer/2022042600/58a2bf591a28ab4c028b4cf2/html5/thumbnails/7.jpg)
I invented offCPU time Flame Graphssomewhere near Lake Tahoe 3 months ago.
![Page 8: Introduction to offCPU Time Flame Graphs](https://reader034.vdocuments.mx/reader034/viewer/2022042600/58a2bf591a28ab4c028b4cf2/html5/thumbnails/8.jpg)
![Page 9: Introduction to offCPU Time Flame Graphs](https://reader034.vdocuments.mx/reader034/viewer/2022042600/58a2bf591a28ab4c028b4cf2/html5/thumbnails/9.jpg)
![Page 10: Introduction to offCPU Time Flame Graphs](https://reader034.vdocuments.mx/reader034/viewer/2022042600/58a2bf591a28ab4c028b4cf2/html5/thumbnails/10.jpg)
I got the inspiration
from Brendan Gregg's blog post"OffCPU Performance Analysis"
![Page 11: Introduction to offCPU Time Flame Graphs](https://reader034.vdocuments.mx/reader034/viewer/2022042600/58a2bf591a28ab4c028b4cf2/html5/thumbnails/11.jpg)
http://dtrace.org/blogs/brendan/2011/07/08/offcpuperformanceanalysis/
![Page 12: Introduction to offCPU Time Flame Graphs](https://reader034.vdocuments.mx/reader034/viewer/2022042600/58a2bf591a28ab4c028b4cf2/html5/thumbnails/12.jpg)
Joshua Dankbaar grabbed me for an online issueright after the company Kitchen Adventure.
![Page 13: Introduction to offCPU Time Flame Graphs](https://reader034.vdocuments.mx/reader034/viewer/2022042600/58a2bf591a28ab4c028b4cf2/html5/thumbnails/13.jpg)
Time to cast a spell over our Linux boxes by systemtap!
![Page 14: Introduction to offCPU Time Flame Graphs](https://reader034.vdocuments.mx/reader034/viewer/2022042600/58a2bf591a28ab4c028b4cf2/html5/thumbnails/14.jpg)
I quickly wrote a macrostyle language extensionnamed stap++ for systemtap with a little bit of Perl.
https://github.com/agentzh/stapxx
![Page 15: Introduction to offCPU Time Flame Graphs](https://reader034.vdocuments.mx/reader034/viewer/2022042600/58a2bf591a28ab4c028b4cf2/html5/thumbnails/15.jpg)
![Page 16: Introduction to offCPU Time Flame Graphs](https://reader034.vdocuments.mx/reader034/viewer/2022042600/58a2bf591a28ab4c028b4cf2/html5/thumbnails/16.jpg)
Nginx workers were badly blocking by somethingin a production box in Ashburn
![Page 17: Introduction to offCPU Time Flame Graphs](https://reader034.vdocuments.mx/reader034/viewer/2022042600/58a2bf591a28ab4c028b4cf2/html5/thumbnails/17.jpg)
/* pseudocode for the nginx event loop */
for (;;)
ret = epoll_wait(...);
/* process new events
and expired timers here... */
![Page 18: Introduction to offCPU Time Flame Graphs](https://reader034.vdocuments.mx/reader034/viewer/2022042600/58a2bf591a28ab4c028b4cf2/html5/thumbnails/18.jpg)
Let's write a simple tool to trace the long blockinglatencies in the Nginx event loop!
$ vim epolllooopblocking.sxx
![Page 19: Introduction to offCPU Time Flame Graphs](https://reader034.vdocuments.mx/reader034/viewer/2022042600/58a2bf591a28ab4c028b4cf2/html5/thumbnails/19.jpg)
#!/usr/bin/env stap++
global begin
probe syscall.epoll_wait.return
if (target() == pid()) begin = gettimeofday_ms()
probe syscall.epoll_wait
if (target() == pid() && begin > 0)
elapsed = gettimeofday_ms() begin
if (elapsed >= $arg_limit :default(200))
printf("[%d] epoll loop blocked for %dms\n",
gettimeofday_s(), elapsed)
![Page 20: Introduction to offCPU Time Flame Graphs](https://reader034.vdocuments.mx/reader034/viewer/2022042600/58a2bf591a28ab4c028b4cf2/html5/thumbnails/20.jpg)
$ ./epollloopblocking.sxx x 22845 arg limit=200
Start tracing 22845...
[1376595038] epoll loop blocked for 208ms
[1376595040] epoll loop blocked for 485ms
[1376595044] epoll loop blocked for 336ms
[1376595049] epoll loop blocked for 734ms
[1376595057] epoll loop blocked for 379ms
[1376595061] epoll loop blocked for 227ms
[1376595062] epoll loop blocked for 212ms
[1376595066] epoll loop blocked for 390ms
![Page 21: Introduction to offCPU Time Flame Graphs](https://reader034.vdocuments.mx/reader034/viewer/2022042600/58a2bf591a28ab4c028b4cf2/html5/thumbnails/21.jpg)
Is it file IO blocking here?
![Page 22: Introduction to offCPU Time Flame Graphs](https://reader034.vdocuments.mx/reader034/viewer/2022042600/58a2bf591a28ab4c028b4cf2/html5/thumbnails/22.jpg)
# add some code to trace file IO latency at the same time...
global vfs_begin
global vfs_latency
probe syscall.rename, syscall.open, syscall.sendfile*,
vfs.read, vfs.write
if (target() == pid()) vfs_begin = gettimeofday_us()
probe syscall.rename.return, syscall.open.return,
syscall.sendfile*.return, vfs.read.return, vfs.write.return
if (target() == pid())
vfs_latency += gettimeofday_us() vfs_begin
![Page 23: Introduction to offCPU Time Flame Graphs](https://reader034.vdocuments.mx/reader034/viewer/2022042600/58a2bf591a28ab4c028b4cf2/html5/thumbnails/23.jpg)
$ ./epollloopblockingvfs.sxx x 22845 arg limit=200
Start tracing 22845...
[1376596251] epoll loop blocked for 364ms (file IO: 19ms)
[1376596266] epoll loop blocked for 288ms (file IO: 0ms)
[1376596270] epoll loop blocked for 1002ms (file IO: 0ms)
[1376596272] epoll loop blocked for 206ms (file IO: 5ms)
[1376596280] epoll loop blocked for 218ms (file IO: 211ms)
[1376596283] epoll loop blocked for 396ms (file IO: 9ms)
![Page 24: Introduction to offCPU Time Flame Graphs](https://reader034.vdocuments.mx/reader034/viewer/2022042600/58a2bf591a28ab4c028b4cf2/html5/thumbnails/24.jpg)
Hmm...seems like file IO isnot the major factor here...
![Page 25: Introduction to offCPU Time Flame Graphs](https://reader034.vdocuments.mx/reader034/viewer/2022042600/58a2bf591a28ab4c028b4cf2/html5/thumbnails/25.jpg)
I suddenly remember my offCPU timeFlame Graph tool created 3 months ago...
![Page 26: Introduction to offCPU Time Flame Graphs](https://reader034.vdocuments.mx/reader034/viewer/2022042600/58a2bf591a28ab4c028b4cf2/html5/thumbnails/26.jpg)
https://github.com/agentzh/nginxsystemtaptoolkit#ngxsamplebtoffcpu
![Page 27: Introduction to offCPU Time Flame Graphs](https://reader034.vdocuments.mx/reader034/viewer/2022042600/58a2bf591a28ab4c028b4cf2/html5/thumbnails/27.jpg)
$ ./ngxsamplebtoffcpu t 10 x 16782 > a.bt
$ stackcollapsestap.pl a.bt > a.cbt
$ flamegraph.pl a.cbt > a.svg
![Page 28: Introduction to offCPU Time Flame Graphs](https://reader034.vdocuments.mx/reader034/viewer/2022042600/58a2bf591a28ab4c028b4cf2/html5/thumbnails/28.jpg)
![Page 29: Introduction to offCPU Time Flame Graphs](https://reader034.vdocuments.mx/reader034/viewer/2022042600/58a2bf591a28ab4c028b4cf2/html5/thumbnails/29.jpg)
![Page 30: Introduction to offCPU Time Flame Graphs](https://reader034.vdocuments.mx/reader034/viewer/2022042600/58a2bf591a28ab4c028b4cf2/html5/thumbnails/30.jpg)
Okay, Nginx was mainly waiting on a lockin an obsolete code path which was added to Nginx
by one of us (long time ago?)
![Page 31: Introduction to offCPU Time Flame Graphs](https://reader034.vdocuments.mx/reader034/viewer/2022042600/58a2bf591a28ab4c028b4cf2/html5/thumbnails/31.jpg)
Let's just remove the guilty code path
from our production system!
![Page 32: Introduction to offCPU Time Flame Graphs](https://reader034.vdocuments.mx/reader034/viewer/2022042600/58a2bf591a28ab4c028b4cf2/html5/thumbnails/32.jpg)
Yay! The number of longrunning requests(longer than 1 second) is almost halved!
![Page 33: Introduction to offCPU Time Flame Graphs](https://reader034.vdocuments.mx/reader034/viewer/2022042600/58a2bf591a28ab4c028b4cf2/html5/thumbnails/33.jpg)
$ ./epollloopblockingvfs.sxx x 16738 arg limit=200
Start tracing 16738...
[1376626387] epoll loop blocked for 456ms (file IO: 455ms)
[1376626388] epoll loop blocked for 207ms (file IO: 206ms)
[1376626396] epoll loop blocked for 364ms (file IO: 363ms)
[1376626402] epoll loop blocked for 350ms (file IO: 349ms)
[1376626414] epoll loop blocked for 309ms (file IO: 309ms)
![Page 34: Introduction to offCPU Time Flame Graphs](https://reader034.vdocuments.mx/reader034/viewer/2022042600/58a2bf591a28ab4c028b4cf2/html5/thumbnails/34.jpg)
![Page 35: Introduction to offCPU Time Flame Graphs](https://reader034.vdocuments.mx/reader034/viewer/2022042600/58a2bf591a28ab4c028b4cf2/html5/thumbnails/35.jpg)
![Page 36: Introduction to offCPU Time Flame Graphs](https://reader034.vdocuments.mx/reader034/viewer/2022042600/58a2bf591a28ab4c028b4cf2/html5/thumbnails/36.jpg)
Okay, now it is file IO that's killing us!
![Page 37: Introduction to offCPU Time Flame Graphs](https://reader034.vdocuments.mx/reader034/viewer/2022042600/58a2bf591a28ab4c028b4cf2/html5/thumbnails/37.jpg)
Let's tune Nginx's open_file_cache configurationsto save the open() system calls.
![Page 38: Introduction to offCPU Time Flame Graphs](https://reader034.vdocuments.mx/reader034/viewer/2022042600/58a2bf591a28ab4c028b4cf2/html5/thumbnails/38.jpg)
But...wait...we have not evenenabled it yet in production...
![Page 39: Introduction to offCPU Time Flame Graphs](https://reader034.vdocuments.mx/reader034/viewer/2022042600/58a2bf591a28ab4c028b4cf2/html5/thumbnails/39.jpg)
# 2520 is the nginx worker process's pid
$ stap++ x 2520 \
e 'probe @pfunc(ngx_open_cached_file)printf("%p\n",$cache);exit()'
0x0
![Page 40: Introduction to offCPU Time Flame Graphs](https://reader034.vdocuments.mx/reader034/viewer/2022042600/58a2bf591a28ab4c028b4cf2/html5/thumbnails/40.jpg)
It is faster and more accurate thanasking Dane to check nginx.conf.
![Page 41: Introduction to offCPU Time Flame Graphs](https://reader034.vdocuments.mx/reader034/viewer/2022042600/58a2bf591a28ab4c028b4cf2/html5/thumbnails/41.jpg)
Let's start by using the sample configurationin Nginx's official documentation.
# file nginx.conf open_file_cache max=1000 inactive=20s;
![Page 42: Introduction to offCPU Time Flame Graphs](https://reader034.vdocuments.mx/reader034/viewer/2022042600/58a2bf591a28ab4c028b4cf2/html5/thumbnails/42.jpg)
Yay! Our online metrics immediately showedeven better numbers!
![Page 43: Introduction to offCPU Time Flame Graphs](https://reader034.vdocuments.mx/reader034/viewer/2022042600/58a2bf591a28ab4c028b4cf2/html5/thumbnails/43.jpg)
What is the cache hit rate then?Can we improve the cache configurations even further?
![Page 44: Introduction to offCPU Time Flame Graphs](https://reader034.vdocuments.mx/reader034/viewer/2022042600/58a2bf591a28ab4c028b4cf2/html5/thumbnails/44.jpg)
#!/usr/bin/env stap++
global misses, total, in_ctx
probe @pfunc(ngx_open_cached_file)
if (pid() == target()) in_ctx = 1 total++
probe @pfunc(ngx_open_cached_file).return
if (pid() == target()) in_ctx = 0
probe @pfunc(ngx_open_and_stat_file)
if (pid() == target() && in_ctx) misses++
probe end
printf("nginx open file cache miss rate: %d%%\n", misses * 100 / total)
![Page 45: Introduction to offCPU Time Flame Graphs](https://reader034.vdocuments.mx/reader034/viewer/2022042600/58a2bf591a28ab4c028b4cf2/html5/thumbnails/45.jpg)
$ ./ngxopenfilecachemisses.sxx x 19642
WARNING: Start tracing process 19642...
Hit CtrlC to end.
C
nginx open file cache miss rate: 91%
![Page 46: Introduction to offCPU Time Flame Graphs](https://reader034.vdocuments.mx/reader034/viewer/2022042600/58a2bf591a28ab4c028b4cf2/html5/thumbnails/46.jpg)
So only 9% ~ 10% cache hit ratefor open_file_cache in our production systems.
![Page 47: Introduction to offCPU Time Flame Graphs](https://reader034.vdocuments.mx/reader034/viewer/2022042600/58a2bf591a28ab4c028b4cf2/html5/thumbnails/47.jpg)
Let's double the cache size! # file nginx.conf open_file_cache max=2000 inactive=180s;
![Page 48: Introduction to offCPU Time Flame Graphs](https://reader034.vdocuments.mx/reader034/viewer/2022042600/58a2bf591a28ab4c028b4cf2/html5/thumbnails/48.jpg)
$ ./ngxopenfilecachemisses.sxx x 7818
WARNING: Start tracing process 7818...
Hit CtrlC to end.
C
nginx open file cache miss rate: 79%
![Page 49: Introduction to offCPU Time Flame Graphs](https://reader034.vdocuments.mx/reader034/viewer/2022042600/58a2bf591a28ab4c028b4cf2/html5/thumbnails/49.jpg)
Yay! The cache hit rate is also doubled!21% Now!
![Page 50: Introduction to offCPU Time Flame Graphs](https://reader034.vdocuments.mx/reader034/viewer/2022042600/58a2bf591a28ab4c028b4cf2/html5/thumbnails/50.jpg)
Lee said, "try 50k!"
![Page 51: Introduction to offCPU Time Flame Graphs](https://reader034.vdocuments.mx/reader034/viewer/2022042600/58a2bf591a28ab4c028b4cf2/html5/thumbnails/51.jpg)
Even a cache size of 20k did not fly.The overall performance was dropping!
![Page 52: Introduction to offCPU Time Flame Graphs](https://reader034.vdocuments.mx/reader034/viewer/2022042600/58a2bf591a28ab4c028b4cf2/html5/thumbnails/52.jpg)
![Page 53: Introduction to offCPU Time Flame Graphs](https://reader034.vdocuments.mx/reader034/viewer/2022042600/58a2bf591a28ab4c028b4cf2/html5/thumbnails/53.jpg)
![Page 54: Introduction to offCPU Time Flame Graphs](https://reader034.vdocuments.mx/reader034/viewer/2022042600/58a2bf591a28ab4c028b4cf2/html5/thumbnails/54.jpg)
So Nginx's open_file_cache is hopelesslywaiting on shm locks
when the cache size is large.
![Page 55: Introduction to offCPU Time Flame Graphs](https://reader034.vdocuments.mx/reader034/viewer/2022042600/58a2bf591a28ab4c028b4cf2/html5/thumbnails/55.jpg)
So Flame Graphs saved us again
![Page 56: Introduction to offCPU Time Flame Graphs](https://reader034.vdocuments.mx/reader034/viewer/2022042600/58a2bf591a28ab4c028b4cf2/html5/thumbnails/56.jpg)
When we are focusing on optimizing one metric,we might introduce new bigger bottleneck
by accident.
![Page 57: Introduction to offCPU Time Flame Graphs](https://reader034.vdocuments.mx/reader034/viewer/2022042600/58a2bf591a28ab4c028b4cf2/html5/thumbnails/57.jpg)
Flame Graphs can always give us
the whole picture.
![Page 58: Introduction to offCPU Time Flame Graphs](https://reader034.vdocuments.mx/reader034/viewer/2022042600/58a2bf591a28ab4c028b4cf2/html5/thumbnails/58.jpg)
Optimizations are also all about balance.
![Page 59: Introduction to offCPU Time Flame Graphs](https://reader034.vdocuments.mx/reader034/viewer/2022042600/58a2bf591a28ab4c028b4cf2/html5/thumbnails/59.jpg)
Nginx's open_file_cache is already a dead end.Let's focus on file IO itself instead.
![Page 60: Introduction to offCPU Time Flame Graphs](https://reader034.vdocuments.mx/reader034/viewer/2022042600/58a2bf591a28ab4c028b4cf2/html5/thumbnails/60.jpg)
$ ./funclatencydistr.sxx x 18243 arg func=syscall.open arg time=20
Start tracing 18243...
Please wait for 20 seconds.
![Page 61: Introduction to offCPU Time Flame Graphs](https://reader034.vdocuments.mx/reader034/viewer/2022042600/58a2bf591a28ab4c028b4cf2/html5/thumbnails/61.jpg)
Distribution of sys_open latencies (in microseconds)
max/avg/min: 565270/2225/5
value | count
8 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ 731
16 |@@@@@@@@@@@@@@ 211
32 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ 510
64 |@@@@ 65
128 | 2
256 |@@@@@@@@@@ 150
512 |@@@@@@@ 119
1024 |@ 21
2048 | 14
4096 | 9
8192 | 10
16384 | 3
32768 | 9
65536 | 4
131072 | 3
262144 | 5
524288 | 1
![Page 62: Introduction to offCPU Time Flame Graphs](https://reader034.vdocuments.mx/reader034/viewer/2022042600/58a2bf591a28ab4c028b4cf2/html5/thumbnails/62.jpg)
Knowing how the latency of individual file IO operationsis distributed, we can trace the details of those "slow samples".
![Page 63: Introduction to offCPU Time Flame Graphs](https://reader034.vdocuments.mx/reader034/viewer/2022042600/58a2bf591a28ab4c028b4cf2/html5/thumbnails/63.jpg)
$ ./slowvfsreads.sxx x 6954 arg limit=100
Start tracing 6954...
Hit CtrlC to end.
[1377049930] latency=481ms dev=sde1 bytes_read=350 err=0 errstr=
[1377049934] latency=497ms dev=sdc1 bytes_read=426 err=0 errstr=
[1377049945] latency=234ms dev=sdf1 bytes_read=519 err=0 errstr=
[1377049947] latency=995ms dev=sdb1 bytes_read=311 err=0 errstr=
[1377049949] latency=208ms dev=sde1 bytes_read=594 err=0 errstr=
[1377049949] latency=430ms dev=sde1 bytes_read=4096 err=0 errstr=
[1377049949] latency=338ms dev=sdd1 bytes_read=402 err=0 errstr=
[1377049950] latency=511ms dev=sdc1 bytes_read=5799 err=0 errstr=
![Page 64: Introduction to offCPU Time Flame Graphs](https://reader034.vdocuments.mx/reader034/viewer/2022042600/58a2bf591a28ab4c028b4cf2/html5/thumbnails/64.jpg)
So the slow samples are distributed evenly among all the disk drives,and the data volumn involved in each call is also quite small.
![Page 65: Introduction to offCPU Time Flame Graphs](https://reader034.vdocuments.mx/reader034/viewer/2022042600/58a2bf591a28ab4c028b4cf2/html5/thumbnails/65.jpg)
Kernellevel offCPU Flame Graphs
![Page 66: Introduction to offCPU Time Flame Graphs](https://reader034.vdocuments.mx/reader034/viewer/2022042600/58a2bf591a28ab4c028b4cf2/html5/thumbnails/66.jpg)
$ ./ngxsamplebtoffcpu p 7635 k t 10 > a.bt
![Page 67: Introduction to offCPU Time Flame Graphs](https://reader034.vdocuments.mx/reader034/viewer/2022042600/58a2bf591a28ab4c028b4cf2/html5/thumbnails/67.jpg)
![Page 68: Introduction to offCPU Time Flame Graphs](https://reader034.vdocuments.mx/reader034/viewer/2022042600/58a2bf591a28ab4c028b4cf2/html5/thumbnails/68.jpg)
![Page 69: Introduction to offCPU Time Flame Graphs](https://reader034.vdocuments.mx/reader034/viewer/2022042600/58a2bf591a28ab4c028b4cf2/html5/thumbnails/69.jpg)
I love Flame Graphs becausethey are one kind of visualizations
that are truly actionable.
![Page 70: Introduction to offCPU Time Flame Graphs](https://reader034.vdocuments.mx/reader034/viewer/2022042600/58a2bf591a28ab4c028b4cf2/html5/thumbnails/70.jpg)
Credits
Thanks Brendan Gregg for inventing Flame Graphs.
Thanks systemtap which was created after dtrace.
Thanks Joshua Dankbaar for walking me through
our production environment.
Thanks Ian Applegate for supporting use of
systemtap in production.
Thanks Dane for pushing everyone onto the same page.
![Page 71: Introduction to offCPU Time Flame Graphs](https://reader034.vdocuments.mx/reader034/viewer/2022042600/58a2bf591a28ab4c028b4cf2/html5/thumbnails/71.jpg)
Systems and systems' laws lay hid in night.God said, "let dtrace be!" and all was light.
![Page 72: Introduction to offCPU Time Flame Graphs](https://reader034.vdocuments.mx/reader034/viewer/2022042600/58a2bf591a28ab4c028b4cf2/html5/thumbnails/72.jpg)
Any questions?
![Page 73: Introduction to offCPU Time Flame Graphs](https://reader034.vdocuments.mx/reader034/viewer/2022042600/58a2bf591a28ab4c028b4cf2/html5/thumbnails/73.jpg)
Int