linux tracing superpowers by eugene pirogov

Linuxtracing

superpowersIevgen Pyrogov @gmile Lviv 2017

Plan1. BSD and

the DTrace era2. Linux and the BPF era

What is full-stack development?

BSD andthe DTrace era

A little of history

I realized at a relatively young age (~19)

that I love debugging.

Brian Cantril

The greatest satisfaction that I have had is nailing a nasty bug —

it's an experience that (for me, anyway) is so visceral as to be nearly primal.

Brian Cantril

2003

Adam Leventhal

Mike ShapiroBryan Cantrill

DTrace

# dtrace -n 'syscall::open*:entry { printf("%s %s",execname,copyinstr(arg0)); }' dtrace: description 'syscall::open*:entry ' matched 2 probes CPU ID FUNCTION:NAME 0 14 open:entry gnome-netstatus- /dev/kstat 0 14 open:entry man /var/ld/ld.config 0 14 open:entry man /lib/libc.so.1 0 14 open:entry man /usr/share/man/man.cf 0 14 open:entry man /usr/share/man/windex 0 14 open:entry man /usr/share/man/man1/ls.1 0 14 open:entry man /usr/share/man/man1/ls.1 0 14 open:entry man /tmp/mpqea4RF 0 14 open:entry sh /var/ld/ld.config 0 14 open:entry sh /lib/libc.so.1 0 14 open:entry neqn /var/ld/ld.config 0 14 open:entry neqn /lib/libc.so.1 0 14 open:entry neqn /usr/share/lib/pub/eqnchar 0 14 open:entry tbl /var/ld/ld.config 0 14 open:entry tbl /lib/libc.so.1 0 14 open:entry tbl /usr/share/man/man1/ls.1 0 14 open:entry nroff /var/ld/ld.config [...]

# dtrace -n 'syscall:::entry { @num[execname] = count(); }' dtrace: description 'syscall:::entry ' matched 228 probes ^C snmpd 1 utmpd 2 inetd 2 nscd 7 svc.startd 11 sendmail 31 poold 133 dtrace 1720

# dtrace -n 'syscall:::entry { @num[probefunc] = count(); }' dtrace: description 'syscall:::entry ' matched 228 probes ^C fstat 1 setcontext 1 lwp_park 1 schedctl 1 mmap 1 sigaction 2 pset 2 lwp_sigmask 2 gtime 3 sysconfig 3 write 4 brk 6 pollsys 7 p_online 558 ioctl 579

$ dtrace -n 'syscall:::entry { @num[pid,execname] = count(); }' dtrace: description 'syscall:::entry ' matched 228 probes ^C 1109 svc.startd 1 4588 svc.startd 2 7 svc.startd 2 3950 svc.startd 2 1626 nscd 2 870 svc.startd 2 82 nscd 6 5011 sendmail 10 6010 poold 74 8707 dtrace 1720

$ dtrace -n 'sysinfo:::readch { @bytes[execname] = sum(arg0); }' dtrace: description 'sysinfo:::readch ' matched 4 probes ^C mozilla-bin 16 gnome-smproxy 64 metacity 64 dsdm 64 wnck-applet 64 xscreensaver 96 gnome-terminal 900 ttymon 5952 Xorg 17544

$ dtrace -n 'proc:::exec-success { trace(curpsinfo->pr_psargs); }' dtrace: description 'proc:::exec-success ' matched 1 probe CPU ID FUNCTION:NAME 0 3297 exec_common:exec-success man ls

0 3297 exec_common:exec-success tbl /usr/share/man/man1/ls.1 0 3297 exec_common:exec-success neqn /usr/share/lib/pub/eqnchar - 0 3297 exec_common:exec-success nroff -u0 -Tlp -man - 0 3297 exec_common:exec-success col -x 0 3297 exec_common:exec-success sh -c more -s /tmp/mpzIaOZF 0 3297 exec_common:exec-success more -s /tmp/mpzIaOZF

dtrace -l dtrace: system integrity protection is on, some features will not be available dtrace: failed to initialize dtrace: DTrace requires additional privileges

Brendan Gregg

One the first guys to use DTrace

outside of SUN

January 27, 2010

ResignationsPetitions and forks

LawsuitResignations

Forks

Forks

Program closures

Linux andthe BPF era

Linuxtracingtools

SystemTap trace strace

sysdig ktap

LTTng

Limitations:

1. Expensive to run

2. Extremely hard to use in production

3. May be unsafe to run

Result: no single consolidated effort in

development

BPF

Berkeley Packet Filter

BPF – is a tiny VM inside the kernel

Initial purpose – to filter TCP packets!

How BPF looked like back then:

l0: ldh [12] l1: jeq #0x800, l3, l2 l2: jeq #0x805, l3, l8 l3: ld [26] l4: jeq #SRC, l4, l8 l5: ld len l6: jlt 0x400, l7, l8 l7: ret #0xffff l8: ret #0

2013

Things startto change

dramatically

Alexei Starovoitov

BPF is brutally hard to write!

Far-far from DTrace one-liners

Brendan Gregg: “BPF is the only language that defeated me!”

We should not write BPF programs

directly

Enter BCC!

BCC

BCC = BPF Compiler

Collection

1. Write BPF program in restricted C

2. Let Python or Lua leverage the rest

1. Write your BPF program in C... inline or in a separate file

2. Write a python script that loads and interacts with your BPF program

3. Attach to kprobes, socket, etc.

4. Read/update maps

5. Configuration, complex calculation/correlations

6. Iterate on above and re-try...in seconds

from bpf import BPF from subprocess import call

prog = """ int hello(void *ctx) { bpf_trace_printk("Hello, World!\\n"); return 0; }; “""

b = BPF(text=prog)

fn = b.load_func("hello", BPF.KPROBE)

BPF.attach_kprobe(fn, “sys_clone")

try: call(["cat", "/sys/kernel/debug/tracing/trace_pipe"]) except KeyboardInterrupt: pass

[root@localhost examples]# ./hello_world.py python-20662 [001] d..1 1138.551706: : Hello, World! tmux-1012 [002] d..1 1139.227627: : Hello, World! tmux-1012 [002] d..1 1139.229636: : Hello, World! byobu-20664 [006] d..1 1139.235396: : Hello, World! byobu-20665 [007] d..1 1139.236660: : Hello, World! byobu-20665 [007] d..1 1139.246109: : Hello, World! ^C

Brenden Blanco

Largest difference between

DTrace and BPF is: ease of use

Simpler tools utilizing BPF are

already being built!

1. BPF backend for perf command

2. ply

#!/usr/bin/env ply

kprobe:SyS_* { $syscalls[func].count() }

3. BPF backend for SystemTap

#!/usr/bin/stap /* * opensnoop.stp Trace file open()s. Basic version of opensnoop. */

probe begin { printf("\n%6s %6s %16s %s\n", "UID", "PID", "COMM", "PATH"); }

probe syscall.open { printf("%6d %6d %16s %s\n", uid(), pid(), execname(), filename); }

Find / fix bcc issues

High-level language

Promotion

Promotion

Education

GUI integration

Examples

# ./execsnoop PCOMM PID RET ARGS bash 15887 0 /usr/bin/man ls preconv 15894 0 /usr/bin/preconv -e UTF-8 man 15896 0 /usr/bin/tbl man 15897 0 /usr/bin/nroff -mandoc -rLL=169n -rLT=169n -Tutf8 man 15898 0 /usr/bin/pager -s nroff 15900 0 /usr/bin/locale charmap nroff 15901 0 /usr/bin/groff -mtty-char -Tutf8 -mandoc -rLL=169n -rLT=169n groff 15902 0 /usr/bin/troff -mtty-char -mandoc -rLL=169n -rLT=169n -Tutf8 groff 15903 0 /usr/bin/grotty

# ./opensnoop PID COMM FD ERR PATH 17326 <...> 7 0 /sys/kernel/debug/tracing/trace_pipe 1576 snmpd 9 0 /proc/net/dev 1576 snmpd 11 0 /proc/net/if_inet6 1576 snmpd 11 0 /proc/sys/net/ipv4/neigh/eth0/retrans_time_ms 1576 snmpd 11 0 /proc/sys/net/ipv6/neigh/eth0/retrans_time_ms 1576 snmpd 11 0 /proc/sys/net/ipv6/conf/eth0/forwarding 1576 snmpd 11 0 /proc/sys/net/ipv6/neigh/eth0/base_reachable_time_ms 1576 snmpd 11 0 /proc/sys/net/ipv4/neigh/lo/retrans_time_ms 1576 snmpd 11 0 /proc/sys/net/ipv6/neigh/lo/retrans_time_ms 1576 snmpd 11 0 /proc/sys/net/ipv6/conf/lo/forwarding 1576 snmpd 11 0 /proc/sys/net/ipv6/neigh/lo/base_reachable_time_ms 1576 snmpd 9 0 /proc/diskstats 1576 snmpd 9 0 /proc/stat 1576 snmpd 9 0 /proc/vmstat 1956 supervise 9 0 supervise/status.new 1956 supervise 9 0 supervise/status.new 17358 run 3 0 /etc/ld.so.cache 17358 run 3 0 /lib/x86_64-linux-gnu/libtinfo.so.5 17358 run 3 0 /lib/x86_64-linux-gnu/libdl.so.2 17358 run 3 0 /lib/x86_64-linux-gnu/libc.so.6 17358 run -1 6 /dev/tty 17358 run 3 0 /proc/meminfo 17358 run 3 0 /etc/nsswitch.conf 17358 run 3 0 /etc/ld.so.cache 17358 run 3 0 /lib/x86_64-linux-gnu/libnss_compat.so.2 17358 run 3 0 /lib/x86_64-linux-gnu/libnsl.so.1 17358 run 3 0 /etc/ld.so.cache 17358 run 3 0 /lib/x86_64-linux-gnu/libnss_nis.so.2 17358 run 3 0 /lib/x86_64-linux-gnu/libnss_files.so.2 17358 run 3 0 /etc/passwd 17358 run 3 0 ./run

# ./bashreadline TIME PID COMMAND 05:28:25 21176 ls -l 05:28:28 21176 date 05:28:35 21176 echo hello world 05:28:43 21176 foo this command failed 05:28:45 21176 df -h 05:29:04 3059 echo another shell 05:29:13 21176 echo first shell again

# ./funccount 'vfs_*' Tracing... Ctrl-C to end. ^C FUNC COUNT vfs_create 1 vfs_rename 1 vfs_fsync_range 2 vfs_lock_file 30 vfs_fstatat 152 vfs_fstat 154 vfs_write 166 vfs_getattr_nosec 262 vfs_getattr 262 vfs_open 264 vfs_read 470 Detaching...

# trace 'sys_execve "%s", arg1' PID COMM FUNC - 4402 bash sys_execve /usr/bin/man 4411 man sys_execve /usr/local/bin/less 4411 man sys_execve /usr/bin/less 4410 man sys_execve /usr/local/bin/nroff 4410 man sys_execve /usr/bin/nroff 4409 man sys_execve /usr/local/bin/tbl 4409 man sys_execve /usr/bin/tbl 4408 man sys_execve /usr/local/bin/preconv 4408 man sys_execve /usr/bin/preconv 4415 nroff sys_execve /usr/bin/locale 4416 nroff sys_execve /usr/bin/groff 4418 groff sys_execve /usr/bin/grotty 4417 groff sys_execve /usr/bin/troff

# trace 'sys_read (arg3 > 20000) "read %d bytes", arg3' PID COMM FUNC - 4490 dd sys_read read 1048576 bytes 4490 dd sys_read read 1048576 bytes 4490 dd sys_read read 1048576 bytes 4490 dd sys_read read 1048576 bytes

trace 'r:bash:readline "%s", retval' PID COMM FUNC - 2740 bash readline echo hi! 2740 bash readline man ls

Bonus track

Thanks!

Questions?