bridging the gap between hardware and software tracing
DESCRIPTION
For a numbers of years, silicon vendors have been providing hardware tracing facilities to embedded developers. By using these, developers can resolve performance and latency issues more quickly, resulting in shorter time to market. In this talk, we will cover the hardware based tracing facilities offered by various manufacturers and see how they differ from their software counterparts with respect to their instrumentation capabilities, transport mechanisms, output formats, etc. We will also show how joint hardware and software tracing can be used by developers to gain deeper insights in their applications’ behaviour. Finally, we will outline the on-going work within the Linux Trace Toolkit next generation (LTTng) project to enhance hardware tracing support and tracing data visualization.TRANSCRIPT
![Page 1: Bridging the gap between hardware and software tracing](https://reader034.vdocuments.mx/reader034/viewer/2022042714/5549e8c3b4c90512488b4d52/html5/thumbnails/1.jpg)
1
Bridging the gap between hardware and software tracing
Embedded Linux Conference Europe 2013
[email protected] @c_bab
![Page 2: Bridging the gap between hardware and software tracing](https://reader034.vdocuments.mx/reader034/viewer/2022042714/5549e8c3b4c90512488b4d52/html5/thumbnails/2.jpg)
2Source: arm.com [1]
![Page 3: Bridging the gap between hardware and software tracing](https://reader034.vdocuments.mx/reader034/viewer/2022042714/5549e8c3b4c90512488b4d52/html5/thumbnails/3.jpg)
3
ARM Cortex A9
Source: arm.com [1]
![Page 4: Bridging the gap between hardware and software tracing](https://reader034.vdocuments.mx/reader034/viewer/2022042714/5549e8c3b4c90512488b4d52/html5/thumbnails/4.jpg)
4
ARM Cortex A9
PTM
Program Trace Macrocell
Source: arm.com [1]
![Page 5: Bridging the gap between hardware and software tracing](https://reader034.vdocuments.mx/reader034/viewer/2022042714/5549e8c3b4c90512488b4d52/html5/thumbnails/5.jpg)
5
whoami
Christian Babeux, Software Developer, EfficiOS,
Background in embedded and ASIC tools,
Active contributor to the LTTng projects:● lttng-tools, lttng-ust, babeltrace,● CI infra, Website, Twitter.
AUR package maintainer for Arch Linux.
![Page 6: Bridging the gap between hardware and software tracing](https://reader034.vdocuments.mx/reader034/viewer/2022042714/5549e8c3b4c90512488b4d52/html5/thumbnails/6.jpg)
6
Content
What is hardware tracing?,
Why is it useful?,
ARM Coresight & ETM,
Freescale QorIQ & Nexus tracing,
LTTng & hardware tracing.
![Page 7: Bridging the gap between hardware and software tracing](https://reader034.vdocuments.mx/reader034/viewer/2022042714/5549e8c3b4c90512488b4d52/html5/thumbnails/7.jpg)
7
What is hardware tracing?What is hardware tracing?
![Page 8: Bridging the gap between hardware and software tracing](https://reader034.vdocuments.mx/reader034/viewer/2022042714/5549e8c3b4c90512488b4d52/html5/thumbnails/8.jpg)
8
What is hardware tracing?
Hardware component(s) used to traces instructions and data movement of a processing device
Real-time observation
Low intrusiveness
![Page 9: Bridging the gap between hardware and software tracing](https://reader034.vdocuments.mx/reader034/viewer/2022042714/5549e8c3b4c90512488b4d52/html5/thumbnails/9.jpg)
9
What is hardware tracing? Cont.
● External trace
![Page 10: Bridging the gap between hardware and software tracing](https://reader034.vdocuments.mx/reader034/viewer/2022042714/5549e8c3b4c90512488b4d52/html5/thumbnails/10.jpg)
10
What is hardware tracing? Cont.● Pros
– Can accommodate high data bandwidth
– Minimal impact on system performance
● Cons– Trace port not always available
– Custom hardware needed
![Page 11: Bridging the gap between hardware and software tracing](https://reader034.vdocuments.mx/reader034/viewer/2022042714/5549e8c3b4c90512488b4d52/html5/thumbnails/11.jpg)
11
What is hardware tracing? Cont.
● Self-hosted trace
![Page 12: Bridging the gap between hardware and software tracing](https://reader034.vdocuments.mx/reader034/viewer/2022042714/5549e8c3b4c90512488b4d52/html5/thumbnails/12.jpg)
12
What is hardware tracing? Cont.● Pros
– Self-contained, facilities can be used by host OS
– No need for special hardware
● Cons– Limited internal buffer space
– Might impact system performance
![Page 13: Bridging the gap between hardware and software tracing](https://reader034.vdocuments.mx/reader034/viewer/2022042714/5549e8c3b4c90512488b4d52/html5/thumbnails/13.jpg)
13
Hardware tracing support● ARM
– Embedded Trace Macrocell (ETM),
– Program Trace Flow Macrocell (PTM),
– System Trace Macrocell (STM)
● PowerPC– Freescale QorIQ with Nexus tracing
– Branch History Rolling Buffer (BHRB)
● Intel– Intel Processor Tracing (PT)
– Last Branch Record (LBR), Branch Trace Store (BTS)
![Page 14: Bridging the gap between hardware and software tracing](https://reader034.vdocuments.mx/reader034/viewer/2022042714/5549e8c3b4c90512488b4d52/html5/thumbnails/14.jpg)
14
Hardware tracing vs. software tracing● Software tracing:
– Static instrumentation or dynamic code patching
– Can be intrusive
– Can be slow
– Tracepoint level granularity
● Hardware tracing:– Tracing done on hardware
– Instrumentation not required
– Instruction level granularity
![Page 15: Bridging the gap between hardware and software tracing](https://reader034.vdocuments.mx/reader034/viewer/2022042714/5549e8c3b4c90512488b4d52/html5/thumbnails/15.jpg)
15
Why hardware tracing is usefulWhy hardware tracing is useful
![Page 16: Bridging the gap between hardware and software tracing](https://reader034.vdocuments.mx/reader034/viewer/2022042714/5549e8c3b4c90512488b4d52/html5/thumbnails/16.jpg)
16
Why hardware tracing is useful
Profiling– Very fine granularity profiling
Performance measurement
Code coverage
Monitoring– Statistics on application currently running?
![Page 17: Bridging the gap between hardware and software tracing](https://reader034.vdocuments.mx/reader034/viewer/2022042714/5549e8c3b4c90512488b4d52/html5/thumbnails/17.jpg)
17
Why hardware tracing is useful cont.
Snapshot on crash/anomaly– Trace overwrites old data until anomaly detected
Event trigger trace
Hardware-assisted software tracing– Use hardware facility (ringbuffer)
![Page 18: Bridging the gap between hardware and software tracing](https://reader034.vdocuments.mx/reader034/viewer/2022042714/5549e8c3b4c90512488b4d52/html5/thumbnails/18.jpg)
18
ARM Coresight & ETM ARM Coresight & ETM
![Page 19: Bridging the gap between hardware and software tracing](https://reader034.vdocuments.mx/reader034/viewer/2022042714/5549e8c3b4c90512488b4d52/html5/thumbnails/19.jpg)
19
ARM Coresight● Coresight
– Collection of hardware components
– Trace and debug a complete SoC
– Open architecture
● Trace source– Processing elements (CPU, DSP, etc.)
– Buses
– System trace (generated from software)
![Page 20: Bridging the gap between hardware and software tracing](https://reader034.vdocuments.mx/reader034/viewer/2022042714/5549e8c3b4c90512488b4d52/html5/thumbnails/20.jpg)
20
ARM ETM● ETM
– Monitor the core internal bus
– Instructions + data trace
– Hardware filters and triggers
– Trace stream compression
– Traces can be saved in internal buffer (ETB) or shared system memory
![Page 21: Bridging the gap between hardware and software tracing](https://reader034.vdocuments.mx/reader034/viewer/2022042714/5549e8c3b4c90512488b4d52/html5/thumbnails/21.jpg)
21
ARM Coresight
Source: arm.com [2]
![Page 22: Bridging the gap between hardware and software tracing](https://reader034.vdocuments.mx/reader034/viewer/2022042714/5549e8c3b4c90512488b4d52/html5/thumbnails/22.jpg)
22
State of Coresight & ETM in Linux● ETM tracer implementation available in Linux
– Seems to work only on specific hardware
● Coresight support status– Framework patchset proposed by Pratik Pratel [3]
● Trace decoder availability?
![Page 23: Bridging the gap between hardware and software tracing](https://reader034.vdocuments.mx/reader034/viewer/2022042714/5549e8c3b4c90512488b4d52/html5/thumbnails/23.jpg)
23
Interested about Coresight/ETM?
“Hardware Trace in the Kernel” BoF by Pawel Moll from ARM
Today, 4:30 PM in Pentland
![Page 24: Bridging the gap between hardware and software tracing](https://reader034.vdocuments.mx/reader034/viewer/2022042714/5549e8c3b4c90512488b4d52/html5/thumbnails/24.jpg)
24
Freescale QorIQ & Nexus tracingFreescale QorIQ & Nexus tracing
![Page 25: Bridging the gap between hardware and software tracing](https://reader034.vdocuments.mx/reader034/viewer/2022042714/5549e8c3b4c90512488b4d52/html5/thumbnails/25.jpg)
25
Freescale QorIQ● PowerPC based platform targeted for high-performance
communications usage– Multiple e500mc processors,
– DPAA support (packet processing offloading),
– Support the Nexus debugging & tracing standard.
Source: radiolocman [4]
![Page 26: Bridging the gap between hardware and software tracing](https://reader034.vdocuments.mx/reader034/viewer/2022042714/5549e8c3b4c90512488b4d52/html5/thumbnails/26.jpg)
26
Nexus standard● ISO standard for debugging embedded systems (IEEE-
ISTO-5001-2003),
● Designed for low pin count, standard set of connectors (JTAG or Debug port),
● Multiple level of Nexus “compliance”.
![Page 27: Bridging the gap between hardware and software tracing](https://reader034.vdocuments.mx/reader034/viewer/2022042714/5549e8c3b4c90512488b4d52/html5/thumbnails/27.jpg)
27
Nexus standard level● Level 1:
– Run time control only (run, stop, breakpoints, etc.)
– Tracing not supported
● Level 2:– Ownership and program trace
● Level 3:– Data write trace & memory read/write on the fly
● Level 4:– Memory substitution
– Trace triggering via a watchpoint
![Page 28: Bridging the gap between hardware and software tracing](https://reader034.vdocuments.mx/reader034/viewer/2022042714/5549e8c3b4c90512488b4d52/html5/thumbnails/28.jpg)
28
Nexus standard format● Packet based output format,
– Standard defines public messages, vendors can define extensions (TCODES)
– Fixed packet size per message, last packet can be of variable length
– Message can have an optional timestamp
![Page 29: Bridging the gap between hardware and software tracing](https://reader034.vdocuments.mx/reader034/viewer/2022042714/5549e8c3b4c90512488b4d52/html5/thumbnails/29.jpg)
29
Example decoded Nexus message
Message # 328TCODE : 2 Ownership Trace MessageSRC ID : 0 Core0 / CPU0 (Clst0:Core0:Thread0)PID INDEX : 0x02 - Sync PIDPID VALUE : 0x01e01bf59c - LPIDR, MSR[GS], PID/NPIDR (ref:DC1[OTS]), MSR[PR]TIMESTAMP : 1140061 (0x11655d)
Message # 329TCODE : 27 Resource Full MessageSRC ID : 0 Core0 / CPU0 (Clst0:Core0:Thread0)RCODE : 0x8 - Timestamp counterRDATA : 0x00ffffffTIMESTAMP : 0 (0x0)
![Page 30: Bridging the gap between hardware and software tracing](https://reader034.vdocuments.mx/reader034/viewer/2022042714/5549e8c3b4c90512488b4d52/html5/thumbnails/30.jpg)
30
State of Nexus in Linux● Nexus qoriq-debug kernel module,
– Available in Freescale QorIQ SDK Yocto Layer
– Implements a debugfs with memory mapped access to Nexus control register
– cat /sys/kernel/debug/npc/trace_buffer > trace
![Page 31: Bridging the gap between hardware and software tracing](https://reader034.vdocuments.mx/reader034/viewer/2022042714/5549e8c3b4c90512488b4d52/html5/thumbnails/31.jpg)
31
Nexus debugfsroot@p3041ds:~# ls -al /sys/kernel/debug/qoriq-dbg[...]drwxr-xr-x 2 root root 0 Aug 6 20:22 cpu0drwxr-xr-x 2 root root 0 Aug 6 20:22 cpu1drwxr-xr-x 2 root root 0 Aug 6 20:22 cpu2drwxr-xr-x 2 root root 0 Aug 6 20:22 cpu3drwxr-xr-x 2 root root 0 Aug 6 20:22 ddr1drwxr-xr-x 2 root root 0 Aug 6 20:22 dpaa[...]drwxr-xr-x 2 root root 0 Aug 6 20:22 npcnpcdrwxr-xr-x 2 root root 0 Aug 6 20:22 nxc[...]
root@p3041ds:~# ls -al /sys/kernel/debug/qoriq-dbg/cpu0[...]-rw-rw-rw- 1 root root 0 Aug 6 20:22 dc1-rw-rw-rw- 1 root root 0 Aug 6 20:22 dc2-rw-rw-rw- 1 root root 0 Aug 6 20:22 dc4--w--w--w- 1 root root 0 Aug 6 20:22 ddam[...]
![Page 32: Bridging the gap between hardware and software tracing](https://reader034.vdocuments.mx/reader034/viewer/2022042714/5549e8c3b4c90512488b4d52/html5/thumbnails/32.jpg)
32
State of Nexus in Linux● Nexus decoder availability
– Released as part of the babeltrace project
● Open-source software to reconstruct program flow not yet developed
– Integrate such functionality in an IDE ? perf?
![Page 33: Bridging the gap between hardware and software tracing](https://reader034.vdocuments.mx/reader034/viewer/2022042714/5549e8c3b4c90512488b4d52/html5/thumbnails/33.jpg)
33
LTTng & Hardware tracingLTTng & Hardware tracing
![Page 34: Bridging the gap between hardware and software tracing](https://reader034.vdocuments.mx/reader034/viewer/2022042714/5549e8c3b4c90512488b4d52/html5/thumbnails/34.jpg)
34
Project overview
![Page 35: Bridging the gap between hardware and software tracing](https://reader034.vdocuments.mx/reader034/viewer/2022042714/5549e8c3b4c90512488b4d52/html5/thumbnails/35.jpg)
35
Tracers
● lttng-modules: kernel tracer module, compatible with kernels from 2.6.38* to 3.11,
● lttng-ust: user-space tracer, in-process library.
* Kernel tracing is now possible on 2.6.32 to 2.6.37 by backport of 3 Linux Kernel patches [1].
![Page 36: Bridging the gap between hardware and software tracing](https://reader034.vdocuments.mx/reader034/viewer/2022042714/5549e8c3b4c90512488b4d52/html5/thumbnails/36.jpg)
36
Utilities
● lttng-tools: cli utilities and daemons for trace control,
– lttng: cli utility for tracing control,
– lttng-sessiond: tracing registry daemon,
– lttng-consumerd: consume trace data,
– lttng-relayd: network streaming daemon.
![Page 37: Bridging the gap between hardware and software tracing](https://reader034.vdocuments.mx/reader034/viewer/2022042714/5549e8c3b4c90512488b4d52/html5/thumbnails/37.jpg)
37
Viewers
● babeltrace: cli text viewer, trace converter, plugin system,
● lttngtop: ncurse top-like viewer,● Eclipse lttng plugin: front-end for lttng, collect,
visualize and analyze traces, highly extensible.
![Page 38: Bridging the gap between hardware and software tracing](https://reader034.vdocuments.mx/reader034/viewer/2022042714/5549e8c3b4c90512488b4d52/html5/thumbnails/38.jpg)
38
Hardware tracing support
● Initial attempt to support hardware tracing:– Babeltrace Nexus to CTF converter [5]
– Goal: Leverage existing traces visualizer
● Encountered issues:– Traces are not self-contained, need sideband
information
– Internal trace buffer size limitation
– Synchronization with other traces can be tricky
![Page 39: Bridging the gap between hardware and software tracing](https://reader034.vdocuments.mx/reader034/viewer/2022042714/5549e8c3b4c90512488b4d52/html5/thumbnails/39.jpg)
39
Hardware tracing support demo
DEMO
![Page 40: Bridging the gap between hardware and software tracing](https://reader034.vdocuments.mx/reader034/viewer/2022042714/5549e8c3b4c90512488b4d52/html5/thumbnails/40.jpg)
40
Future work
● Decoder/Converter for ARM ETM?
● Control hardware tracing facilities with the lttng-tools command-line?
● Custom views for hardware traces in Eclipse plugin
![Page 41: Bridging the gap between hardware and software tracing](https://reader034.vdocuments.mx/reader034/viewer/2022042714/5549e8c3b4c90512488b4d52/html5/thumbnails/41.jpg)
41
Conclusion
● Availability and usefulness of hardware tracing
● Initial support for self-hosted hardware tracing
● Common abstraction for hardware tracing in the Linux kernel?
![Page 42: Bridging the gap between hardware and software tracing](https://reader034.vdocuments.mx/reader034/viewer/2022042714/5549e8c3b4c90512488b4d52/html5/thumbnails/42.jpg)
42
Questions ?
? lttng.org
@lttng_project
www.efficios.com
![Page 43: Bridging the gap between hardware and software tracing](https://reader034.vdocuments.mx/reader034/viewer/2022042714/5549e8c3b4c90512488b4d52/html5/thumbnails/43.jpg)
43
References
[1] - http://www.arm.com/images/processor/Cortex-A9-osprey.jpg
[2] - http://www.arm.com/images/CoreSight_Diagram_Tiny.jpg
[3] - https://lkml.org/lkml/2012/12/19/331
[4] - http://www.rlocman.ru/i/Image/2010/06/24/2.jpg
[5] - https://github.com/cbab/babeltrace/tree/nexus