trace kernel code tips
TRANSCRIPT
12/14/14 2/38
Attribution – ShareAlike 3.0
You are free
● to copy, distribute, display, and perform the work
● to make derivative works
● to make commercial use of the work
Under the following conditions
– Attribution. You must give the original author credit.
– Share Alike. If you alter, transform, or build upon this work, you may distribute the resulting work only under a license identical to this one.
For any reuse or distribution, you must make clear to others the license terms of this work.
● Any of these conditions can be waived if you get permission from the copyright holder.
Your fair use and other rights are in no way affected by the above.
License text: http://creativecommons.org/licenses/by-sa/3.0/legalcode
Corrections, suggestions, contributions and translations are welcome!
Rights to Copycopyright © 2014 Viller Hsiao
12/14/14 3/38
Who am I ?
Viller Hsiao
source url: http://goo.gl/n7NAfS
Embeded Linux / RTOS engineer
2007 ~ Present Realtek Corp. 2003 ~ 2007 BenQ/Qisda
12/14/14 4/38
An Embedded Programmer
source url: http://ext.pimg.tw/aabb52043/1367377582-3526768649_n.png
makefile menuconfigmakefile
gcc
armcc
ld objdump
memorymanagement Linker
script
HardwareSpec
uclibc
newlib
glibc
musl
uboot
kbuild
12/14/14 5/38
Topic Today
How to trace C sources environment
We have cross reference tools.ctags, cscope, LXR, source insight ... and so on
BUT
12/14/14 6/38
Difficulties of Tracing C/C++ Sources
● Preprocessor related● Too much #if/#elif/#endif ● Too much level of macros● Configurable included files
12/14/14 7/38
Difficulties of Tracing C/C++ Sources (Cont.)
● Runtime related● Some debugging and error handling stuff● Multi-task environment● Hard to realize HW state change
Different task context
12/14/14 8/38
Topic Today
How to trace C sources environment
Dynamically
Trace code from runtime execution result(Mainly in linux kernel source)
12/14/14 9/38
Tracing from real execution log
Provides clues to trace code
http://www.clker.com/cliparts/c/c/7/p/7/4/earth-hi.pnghttp://goo.gl/4PnwFe
12/14/14 10/38
Agenda Today
● Build environment● Config system● Makefile / kbuild● Toolchain tips
● Runtime trace● Qemu log● Linux ftrace
12/14/14 11/38
menuconfig
● Too much words in menuconfig● Hard to map definition to code world (CONFIG_XXX)
12/14/14 12/38
menuconfig Tips – Search
● Press '/' to search specific configurations
12/14/14 13/38
menuconfig Tips – Search (Cont.)
Dependency
Path in menu
● Search result
Kconfig path
12/14/14 14/38
menuconfig Tips – Result output
● Linux/.config## Automatically generated file; DO NOT EDIT.# Linux/arm 3.14.25 Kernel Configuration#CONFIG_ARM=yCONFIG_SYS_SUPPORTS_APM_EMULATION=y
## General setup#CONFIG_INIT_ENV_ARG_LIMIT=32CONFIG_CROSS_COMPILE=""# CONFIG_COMPILE_TEST is not setCONFIG_LOCALVERSION=""# CONFIG_LOCALVERSION_AUTO is not setCONFIG_HAVE_KERNEL_GZIP=yCONFIG_HAVE_KERNEL_LZ4=yCONFIG_KERNEL_GZIP=y# CONFIG_KERNEL_LZMA is not set# CONFIG_KERNEL_XZ is not set
## Automatically generated file; DO NOT EDIT.# Linux/arm 3.14.25 Kernel Configuration#CONFIG_ARM=yCONFIG_SYS_SUPPORTS_APM_EMULATION=y
## General setup#CONFIG_INIT_ENV_ARG_LIMIT=32CONFIG_CROSS_COMPILE=""# CONFIG_COMPILE_TEST is not setCONFIG_LOCALVERSION=""# CONFIG_LOCALVERSION_AUTO is not setCONFIG_HAVE_KERNEL_GZIP=yCONFIG_HAVE_KERNEL_LZ4=yCONFIG_KERNEL_GZIP=y# CONFIG_KERNEL_LZMA is not set# CONFIG_KERNEL_XZ is not set
12/14/14 15/38
Tips of Tracing makefile
● make V=1● make --dry-run
12/14/14 16/38
Tips of Tracing makefile (Cont.)
● make --debug
12/14/14 17/38
Tips of Tracing Shell Script
● #!/bin/sh “-x”
● Result
#!/bin/sh -x
FOO=something_produced_from_command
if [ x${FOO} != x'' ]; then echo "get foo"else echo "no foo"fi
#!/bin/sh -x
FOO=something_produced_from_command
if [ x${FOO} != x'' ]; then echo "get foo"else echo "no foo"fi
+ FOO=something_produced_from_command+ '[' xsomething_produced_from_command '!=' x ']'+ echo 'get foo'get foo
+ FOO=something_produced_from_command+ '[' xsomething_produced_from_command '!=' x ']'+ echo 'get foo'get foo
12/14/14 18/38
Tips of Tracing Macros/Definitions
● gcc options: “-save-temps” or “-E”
● Use astyle to beauty result
$ astyle -A8 main.i$ astyle -A8 main.i
$ gcc -save-temps -o main main.c
$ lsmain main.c main.i main.o main.s Makefile
$ gcc -save-temps -o main main.c
$ lsmain main.c main.i main.o main.s Makefile
12/14/14 19/38
Tips of Tracing Macros/Definitions (Cont.)
12/14/14 20/38
Tips of Structure Definitions
● Point to Real definition in the headers
# 5 "include/linux/linkage.h" 2# 1 "include/linux/stringify.h" 1# 6 "include/linux/linkage.h" 2# 1 "include/linux/export.h" 1# 26 "include/linux/export.h"
struct kernel_symbol{ unsigned long value; const char *name;};
# 5 "include/linux/linkage.h" 2# 1 "include/linux/stringify.h" 1# 6 "include/linux/linkage.h" 2# 1 "include/linux/export.h" 1# 26 "include/linux/export.h"
struct kernel_symbol{ unsigned long value; const char *name;};
12/14/14 21/38
Kbuild Tips
● Add compile flags for some object
● Remove compile flags for some object
● All subdirectory
CFLAGS_foo.o = -O0 -g -save-tempCFLAGS_foo.o = -O0 -g -save-temp
CFLAGS_REMOVE_foo.o = -OsCFLAGS_REMOVE_foo.o = -Os
subdir-ccflags-y = -save-tempssubdir-ccflags-y = -save-temps
12/14/14 22/38
Qemu Tips
Setup the search rule
● Qemu log (“-d op -D /path/to/qemu.log”)
12/14/14 23/38
Qemu Tips (Cont.)
Example: get first user space entry point
Setup the search rule
● Qemu log + regular expression
12/14/14 24/38
Qemu Tips (Cont.)
● Other tips● Qemu log + disassembly (objdump)
● Qemu + gdb ( + qemu log)– Breakpoint / stack trace– Qemu options
● -s or -gdb tcp::2345: enable gdb port● -S: freeze CPU at startup
12/14/14 25/38
Ftrace – Overview
● Kernel function tracer ( > 2.6.28 ?)● Record execution log of kernel function● Implemented by “-pg”
– Use dynamic ftrace to reduce overhead– Add “-g -O0” to export more information
● Accessing API – debugfs– /path/to/debug/tracing/
mount -t debugfs none /sys/kernel/debugmount -t debugfs none /sys/kernel/debug
12/14/14 26/38
Ftrace – Configs
12/14/14 27/38
Ftrace – Steps
[tracing]$ echo “*mtd*” > set_ftrace_filter
[tracing]$ echo “*blk*” >> set_ftrace_filter
[tracing]$ echo “*:mod:ext3” >> set_ftrace_filter
[tracing]$ echo function > current_tracer
[tracing]$ echo 1 > tracing_on; do_test; echo 0 > tracing_on;
[tracing]$ cat trace
[tracing]$ echo “*mtd*” > set_ftrace_filter
[tracing]$ echo “*blk*” >> set_ftrace_filter
[tracing]$ echo “*:mod:ext3” >> set_ftrace_filter
[tracing]$ echo function > current_tracer
[tracing]$ echo 1 > tracing_on; do_test; echo 0 > tracing_on;
[tracing]$ cat trace
12/14/14 28/38
Ftrace – Available Tracer
[tracing] $ cat available_tracers
function
function_graph nop
[tracing] $ cat available_tracers
function
function_graph nop
12/14/14 29/38
ftrace
● function tracer
12/14/14 30/38
ftrace
● function tracer
12/14/14 31/38
ftrace
● function_graph tracer
12/14/14 32/38
ftrace
● function_graph tracer
Taskswitch
12/14/14 33/38
ftrace
● Better front-end● trace-cmd● kernelshark
12/14/14 34/38
Ftrace for Code Tracing
● Ftrace provides clues of● Per-module view● Top-down view● Procedure excution flow● Tasks/Threads model relationship
12/14/14 35/38
Ftrace – Other tracer
● Trace event● ftrace + tracepoint
● Probe event● ftrace + kprobe
● Other tracer● Stack tracer● Irqoff latency tracer● Syscall tracer
● And so forth … but not the topic today
12/14/14 36/38
Conclusion
● Trace code● Cross reference tool helps● Drill-down trace
– Start from interface functions● Filesystem hooks: open/read/write/ioctl ...● probe/exit● ISR entry● Architecture interface
– Global functions: EXPORT_SYMBOLS● Execution log provide some hints, too.
– Log from build flow– Qemu instruction trace log– Kernel ftrace
12/14/14 37/38
Reference
[1] Building embedded linux demo case
i. Sherif Mousa (Mar 26, 2014), “Building Embedded Linux”
ii. Linaro wiki, “cross-NG”
[2] Linux-3.14 documentation,
i. “kbuild/makefile.txt”
ii. “trace/ftrace.txt”
[3] 池田宗廣，大岩尚宏，島本裕志，竹步晶雄，平松雅巳 (Jan. 2014), “Linux kernel hacks”
[4] man of
i. astyle
ii. qemu
iii. Gnu make
12/14/14 Viller Hsiao
THE END