pinadx : customizable debugging with dynamic instrumentation

Software & Services Group

1

PinADX: Customizable Debugging with Dynamic Instrumentation

Gregory Lueck, Harish Patil, Cristiano Pereira

Intel Corporation

CGO 2012, San Jose, USA


2

Hypothetical Problem 1

(gdb) runProgram received signal SIGSEGV, Segmentation fault.0x0000000000000000 in ?? ()(gdb) bt#0 0x0000000000000000 in ?? ()#1 0x0000000000000000 in ?? ()

Crash with bad PC and no stack traceCorrupted return address someplace ...Want to stop BEFORE bad “ret” instruction


3

Hypothetical Problem 2

thread stack

?

thread stack

?

thread stack

?

thread stack

?...

Massively threaded applicationHow much stack space needed?At what point does each thread use its max stack?


Traditional Debugger Breakpoints?

4

OriginalApplication

… 1. Debugger catches trap2. Check if “ret” is to good PC3. If yes, resume

How can debugger find all “ret” instructions?Horribly slow to trap on each return

Foo:ret

Bar: …ret

ApplicationIn Debugger

…Foo:trap

Bar: …trap

Overwrite each “ret” with trap ret

ret


5

Dynamic Binary Instrumenation (DBI)Application

if (return to bad PC) Breakpoint()

Applicationif (stack too big) Breakpoint()

Much faster – avoids trap overheadDBI can find all “ret” instructions reliablyGeneral approach – solves stack problem (and others)BUT difficult to integrate with debugger

sub 0x60, %sp…Foo:

retBar: …

ret

Foo:

Bar:

sub 0x10, %sp

…

…Instrumentation


Pin Overview

Tool

JIT compiler

Appl

icat

ion

Code Cache

Trac

es

Tool controls instrumentation(e.g. “if return to bad PC”)

JIT compiler fetches application instructions, calls tool to instrument

Instrumented instructions stored in code cache for efficiency

FetchIn

stru

men

t Optim

izeStore &

execute

6


Original Code Code Cache

JIT Compiler Overview

1’

2

1

6

3

5

2’

6’

Pin

4

3’

5’

Tool inserts instrumentation (e.g. check if return to bad PC)Dynamic

recompilation makes debugging hard

7


8

Process running under Pin

PinADX Architecture

Tool

Pin

Appl

icat

ion Debugger

PinADX presents “pure” view of application. Hides effect of instrumentation and recompilation.

Tool extends debugger via instrumentation.

PinADXcore

GDBor

Microsoft Visual Studio 11

Supports Linux & Windows


9

Rest of the Talk• Introduction / Motivation• Example: Using “Stack Debugger”

extension• Example: Authoring “Stack Debugger”

extension• Implementing PinADX


10

$ gdb ./my-application(gdb) target remote :1234(gdb) break PrintHelloBreakpoint 1 at 0x4004dd: file hw.c, line 13(gdb) contBreakpoint 1, PrintHello () at hw.c:13(gdb) backtrace#0 PrintHello () at hw.c:13#1 main () at hw.c:7(gdb) x/2i $pc=> 0x4004dd <PrintHello+4>: mov $0x4005e8,%edi 0x4004e2 <PrintHello+9>: callq 0x4003b8

Example – Stack Debugger$ pin –appdebug –t stack-debugger.so -- ./my-applicationApplication stopped until continued from debugger.Start GDB, then issue this command at the (gdb) prompt: target remote :1234

Run application under Pin

Debugger connected to Pin

(gdb) contBreakpoint 1, PrintHello () at hw.c:13(gdb) backtrace#0 PrintHello () at hw.c:13#1 main () at hw.c:7(gdb) x/2i $pc=> 0x4004dd <PrintHello+4>: mov $0x4005e8,%edi 0x4004e2 <PrintHello+9>: callq 0x4003b8

(gdb) break PrintHelloBreakpoint 1 at 0x4004dd: file hw.c, line 13


11

(gdb) monitor stackbreak 4000Break when thread uses 4000 stack bytes(gdb) contStopped: Thread uses 4004 bytes of stack(gdb) backtrace#0 0x3f07214445 in _dl_runtime_resolve ()#1 0x00004004e7 in PrintHello () at hw.c:13#2 0x00004004d2 in main () at hw.c:7(gdb) monitor stackbreak 10000Break when thread uses 10000 stack bytes(gdb) break exitBreakpoint 2 at 0x7fffe60f9650(gdb) contBreakpoint 2, 0x7fffe60f9650 in exit ()(gdb) monitor statsMaximum stack usage: 8560 bytes.

Example – Stack DebuggerStop when application uses too much

stack(gdb) monitor stackbreak 4000Break when thread uses 4000 stack bytes(gdb) contStopped: Thread uses 4004 bytes of stack

(gdb) monitor stackbreak 10000Break when thread uses 10000 stack bytes(gdb) break exitBreakpoint 2 at 0x7fffe60f9650(gdb) contBreakpoint 2, 0x7fffe60f9650 in exit ()

(gdb) backtrace#0 0x3f07214445 in _dl_runtime_resolve ()#1 0x00004004e7 in PrintHello () at hw.c:13#2 0x00004004d2 in main () at hw.c:7

(gdb) monitor statsMaximum stack usage: 8560 bytes.


12





13

Stack Debugger – Instrumentation

Thread Start:

[…]sub $0x60, %esp

cmp %esi, %edxjle <L1>

size = StackBase - %esp;if (size > MaxStack) MaxStack = size;if (size > StackLimit) TriggerBreakpoint();

StackBase = %esp;MaxStack = 0;

After each stack-changing instruction

Record initial stack


14

VOID Instruction(INS ins, VOID *){ if (INS_RegWContain(ins, REG_STACK_PTR)) { IPOINT where = (INS_HasFallThrough(ins)) ? IPOINT_AFTER : IPOINT_TAKEN_BRANCH; INS_InsertCall(ins, where, (AFUNPTR)OnStackChange, IARG_REG_VALUE, REG_STACK_PTR, IARG_THREAD_ID, IARG_CONTEXT, IARG_END); }}VOID OnStackChange(ADDRINT sp, THREADID tid, CONTEXT *ctxt){ size_t size = StackBase - sp; if (size > StackMax) StackMax = size; if (size > StackLimit) { ostringstream os; os << "Stopped: Thread uses " << size << " stack bytes."; PIN_ApplicationBreakpoint(ctxt, tid, FALSE, os.str()); }}

Stack Debugger – ImplementationIn

stru

men

tatio

nAn

alys

is

Instrument only instructions that change $SPCall after each instruction


15

int main() { […] PIN_AddDebugInterpreter(HandleDebugCommand, 0);} BOOL HandleDebugCommand(const string &cmd, string *result) { if (cmd == "stats") { ostringstream os; os << "Maximum stack usage: " << StackMax << " bytes.\n"; *result = os.str(); return TRUE; } else if (cmd.find("stackbreak ") == 0) { StackLimit = /* parse limit */; ostringstream os; os << "Break when thread uses " << limit << " stack bytes."; *result = os.str(); return TRUE; } return FALSE; // Unknown command}

Stack Debugger – Implementation


16

Visual Studio IDE Extension


17

Other Debugger Extensions

• Intel Inspector XE Product– Memory Checker– Thread Checker

• Intel SDE: Instruction emulation• Debug from log file (PinPlay, CGO 2010)• Dynamic slicing (Rajiv Gupta, UC

Riverside)• Cmp$im: Cache simulator• Write your own!


18





19

Process running under Pin

PinADX Architecture

Tool

Pin

Appl

icat

ion Debugger

PinADX presents “pure” view of application. Hides effect of instrumentation and recompilation.

Tool extends debugger via instrumentation.

PinADXcore

GDBor

Microsoft Visual Studio 11


20

Communication Details

• Very low level• Symbol processing in debugger• Expression evaluation in debugger• Extension of GDB’s remote debugging protocol

Commands• Read / write registers, memory• Set breakpoints• Continue, single-step, stop

Notifications• Breakpoint triggered• Caught signal• Application exited

DebuggerPinADXcore

Pin


21

Communication Details

Breakpoint alternatives• Insert real INT3 trap instruction• Virtualize inside Pin VMSee paper for details

Commands• Read / write registers, memory• Set breakpoints• Continue, single-step, stop

Notifications• Breakpoint triggered• Caught signal• Application exited

DebuggerPinADXcore

Pin


set breakpoint at 4continue

Code CacheOriginal Code

Breakpoint

1’

2

3

1

4

5

6

breakpoint notification

Execution stops in PinWaits for GDB to continue

BP

2’

3’

PinADXcore Debugger

22


Code CacheOriginal Code

Single Step

1’

2

3

1

4

5

6

step complete notification

Debugger

Execution stops in PinWaits for GDB to continue

do single-stepPinADX

core

23


24

Thanks

• Mark Charney – SDE software emulator• Andria Pazarloglou – Created VS11 GUI plugin• Gregg Miskelly – Microsoft VS11 debugger

architect• Robert Cohn – Father of Pin


25

Summary

• DBI can implement powerful debugger features

• API allows Pin tools to extend debugger easily

• Multi-platform– Linux: GDB– Windows: Microsoft Visual Studio 11 (soon)

• Works with off-the-shelf debuggers

http://pintool.org

pinadx : customizable debugging with dynamic instrumentation

Documents

thread stack

stack problem

max stack

stack tracecorrupted

stack space

gdb x2i

gdb backtrace

gdb runprogram