windows debugging with windbg

35
Low-Level Windows Debugging with WinDbg ARNO HUETTER

Upload: arno-huetter

Post on 21-Feb-2017

3.187 views

Category:

Software


9 download

TRANSCRIPT

Page 1: Windows Debugging with WinDbg

Low-Level Windows Debugging with WinDbgARNO HUETTER

Page 2: Windows Debugging with WinDbg

About the Author

Arno Huetter Arno wrote his first lines of code on a Sinclair ZX80 in 1984.

Over the years, he has been programming in C/C++, Java and C#, and also did quite some database development.

Today he is Development Lead at Dynatrace (APM vendor).

Page 3: Windows Debugging with WinDbg

Dijkstra on Debugging

Page 4: Windows Debugging with WinDbg

The Unsung Hero of Debugging

Page 5: Windows Debugging with WinDbg

WinDbg Core Features WinDbg 1.0 released in 2000 (rewrite of previous WinDBG) Usermode Debugging Kernel/Remote Debugging Post-mortem Debugging Huge set of commands (some quite cryptical) UI (oh well...) Debugging Extensions (.NET: SOS, SOSEX, Psscor2|4, etc) Script Engine 32bit vs. 64bit Version: Use the WinDbg executable matching your

process bitness

Page 6: Windows Debugging with WinDbg

What about Visual Studio Debugger? Visual Studio Debugger?

Easier to use Sufficient for many debugging problems Also supports remote, kernel and post-mortem debugging

WinDbg helps on the really tough cases (and most interesting ones) For example, one might just have a function offset from an eventlog entry (no

symbols), or some bytes from the stack, or a dump from an old binary without symbols, or...

Problem might be not reproducible and only happening in production in border case scenario, or at Windows startup, in a driver, etc. WinDbg is well suited for post-mortem, can also connect over RS232, USB. Lightweight installation, WinDbg even runs from USB stick.

WinDbg Extensions and Scripts provide powerful means, e.g. to analyze heap corruption, etc.

Page 7: Windows Debugging with WinDbg

WinDbg Download Part of Windows SDK/WDK, choose Debugging Tools within SDK

Download Installer https://msdn.microsoft.com/en-us/library/windows/hardware/ff551063

%28v=vs.85%29.aspx

Page 8: Windows Debugging with WinDbg

Tweak your UI

Page 9: Windows Debugging with WinDbg

Set your Symbol Path

Page 10: Windows Debugging with WinDbg

Demo 1: Finding a Memory Leak Enable allocation backtracking (gflags is part of Debugging Tools; be

careful though, backtracking adds considerable runtime overhead)

gflags /i plauscherl.exe +ust

Display heap summary

0:001> !heap -sHeap Flags Reserv Commit Virt Free List UCR Virt Lock Fast (k) (k) (k) (k) length blocks cont. heap -----------------------------------------------------------------------------039c0000 08000002 16420 8520 16364 158 26 5 0 0 LFH03c10000 08001002 60 12 60 3 2 1 0 0 -----------------------------------------------------------------------------

Page 11: Windows Debugging with WinDbg

Finding a Memory Leak Display heap status by entry size

0:001> !heap -stat -h 039c0000 heap @ 039c0000group-by: TOTSIZE max-display: 20 size #blocks total ( %) (percent of total busy bytes) fa0 158 - 14ff00 (97.12) 17d8 1 - 17d8 (0.43) 10d2 1 - 10d2 (0.30) 1000 1 - 1000 (0.29) 20 68 - d00 (0.23)

Page 12: Windows Debugging with WinDbg

Finding a Memory Leak Filter by entry size

0:001> !heap -flt s fa0 _HEAP @ 39c0000 HEAP_ENTRY Size Prev Flags UserPtr UserSize - state 039d1190 01f7 0000 [00] 039d11a8 00fa0 - (busy) 039d2148 01f7 01f7 [00] 039d2160 00fa0 - (busy) 039d3100 01f7 01f7 [00] 039d3118 00fa0 - (busy) 039d40b8 01f7 01f7 [00] 039d40d0 00fa0 - (busy)

Page 13: Windows Debugging with WinDbg

Finding a Memory Leak Inspect entry and get allocation backtrace

0:001> !heap -p -a 039d1190 address 039d1190 found in _HEAP @ 39c0000 HEAP_ENTRY Size Prev Flags UserPtr UserSize - state 039d1190 01f7 0000 [00] 039d11a8 00fa0 - (busy) 77b3cdb3 ntdll!RtlpCallInterceptRoutine+0x00000026 77af71f1 ntdll!RtlAllocateHeap+0x000461d1 73f3f7cb ucrtbase!malloc+0x0000002b d112d plauscherl!operator new+0x0000002c d1027 plauscherl!allocateStuff+0x00000027 d12f6 plauscherl!__scrt_common_main_seh+0x000000ff 77447c04 KERNEL32!BaseThreadInitThunk+0x00000024 77acad1f ntdll!__RtlUserThreadStart+0x0000002f

Page 14: Windows Debugging with WinDbg

Demo 2: Analyzing Crash by Offset We might only have something like this eventlog entry

Faulting application name: plauscherl.exe, version: 0.0.0.0, time stamp: 0x5669ec61Faulting module name: plauscherl.exe, version: 0.0.0.0, time stamp: 0x5669ec61Exception code: 0xc0000005Fault offset: 0x00011a2e

Simply attach WinDbg to .exe or .dll only

windbg -z plauscherl.exe

ModLoad: 00400000 00420000 plauscherl.exe

Page 15: Windows Debugging with WinDbg

Analyzing Crash by Offset Disassemble at offset

0:000> uf 0x00411a2eplauscherl!doSomethingWrong [@ 25]: 25 00411a10 55 push ebp 25 00411a11 8bec mov ebp,esp 25 00411a13 81ecc0000000 sub esp,0C0h 25 00411a19 53 push ebx 25 00411a1a 56 push esi 25 00411a1b 57 push edi 25 00411a1c 8dbd40ffffff lea edi,[ebp-0C0h] 25 00411a22 b930000000 mov ecx,30h 25 00411a27 b8cccccccc mov eax,0CCCCCCCCh 25 00411a2c f3ab rep stos dword ptr es:[edi] 26 00411a2e c7050000000000000000 mov dword ptr ds:[0],0

Page 16: Windows Debugging with WinDbg

Demo 3: Analyzing a Crashdump With dump file: Show stack of selected thread

0:000:x86> k # ChildEBP RetAddr 00 0100f97c 003d19e3 plauscherl!doSomethingWrong+0x1e 01 0100fa50 003d1b86 plauscherl!doSomething+0x23 02 0100fb24 003d238e plauscherl!main+0x76 03 0100fb38 003d21da plauscherl!invoke_main+0x1e 04 0100fb90 003d206d plauscherl!__scrt_common_main_seh+0x15a 05 0100fb98 003d23a8 plauscherl!__scrt_common_main+0xd 06 0100fba0 77447c04 plauscherl!mainCRTStartup+0x8 07 0100fbb4 77acad1f kernel32!BaseThreadInitThunk+0x2408 0100fbfc 77acacea ntdll_77a70000!__RtlUserThreadStart+0x2f09 0100fc0c 00000000 ntdll_77a70000!_RtlUserThreadStart+0x1b

Page 17: Windows Debugging with WinDbg

Analyzing a Crashdump Or: Run full-fledged analysis

0:000:x86> !analyze –vCONTEXT: (.ecxr)eax=cccccccc ebx=7f14b000 ecx=00000000 edx=0ff4f4b8 esi=007efaa8 edi=007ef9cceip=012a151e esp=007ef900 ebp=007ef9cc iopl=0 nv up ei pl nz na pe nccs=0023 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00010206plauscherl!doSomethingWrong+0x1e:012a151e c7050000000000000000 mov dword ptr ds:[0],0 ds:002b:00000000=????????

EXCEPTION_RECORD: (.exr -1)ExceptionAddress: 012a151e (plauscherl!doSomethingWrong+0x0000001e) ExceptionCode: c0000005 (Access violation)Attempt to write to address 00000000

Page 18: Windows Debugging with WinDbg

Analyzing a CrashdumpSTACK_TEXT: 007ef9cc 012a1563 007efb74 007efaa8 7f14b000 plauscherl!doSomethingWrong+0x1e007efaa0 012a160a 00000000 00000000 7f14b000 plauscherl!doSomething+0x23007efb74 012a1bff 00000002 00bb3f98 00bb1f58 plauscherl!main+0x7a007efbc4 012a1a2f 007efbd8 758c919f 7f14b000 plauscherl!__tmainCRTStartup+0x1bf007efbcc 758c919f 7f14b000 007efc1c 77e6a8cb plauscherl!mainCRTStartup+0xf007efbd8 77e6a8cb 7f14b000 b92e2b36 00000000 kernel32!BaseThreadInitThunk+0xe007efc1c 77e6a8a1 ffffffff 77e5f67f 00000000 ntdll!__RtlUserThreadStart+0x20007efc2c 00000000 012a1113 7f14b000 00000000 ntdll!_RtlUserThreadStart+0x1b

FAULTING_SOURCE_FILE: d:\etc\plauscherl\plauscherl\plauscherl.cppFAULTING_SOURCE_LINE_NUMBER: 26

Page 19: Windows Debugging with WinDbg

Demo 4: Analyzing a .NET Deadlock Enter: SOS (Son Of Strike) .NET Debugging Extension

Load SOS (part of every CLR)0:008> .loadby sos clr0:008> .cordll -ve -u -lAutomatically loaded SOS ExtensionCLRDLL: Loaded DLL C:\Windows\Microsoft.NET\Framework\v4.0.30319\mscordacwks.dllCLR DLL status: Loaded DLL C:\Windows\Microsoft.NET\Framework\v4.0.30319\mscordacwks.dll

Memdumps: Always use sos.dll / mscordacwks.dll from identical CLR (version / bitness). Latest WinDbg releases can load matching sos.dll from Microsoft symbol server.

Page 20: Windows Debugging with WinDbg

Analyzing a .NET Deadlock View managed thread

0:004> !clrstack

OS Thread Id: 0x1e70 (4)Child SP IP Call Site056cf7dc 77aaca2c [GCFrame: 056cf7dc] 056cf8b4 77aaca2c [GCFrame: 056cf8b4] 056cf8d0 77aaca2c [HelperMethodFrame_1OBJ: 056cf8d0] System.Threading.Monitor.ReliableEnter(System.Object, Boolean ByRef)056cf94c 7042a287 System.Threading.Monitor.Enter(System.Object, Boolean ByRef)056cf95c 02ca0741 netplauscherl.Program.DoSomethingImportant() 056cf98c 02ca06bb netplauscherl.Program+c.b__4_0() 056cf998 70446274 System.Threading.ThreadHelper.ThreadStart_Context(System.Object)056cf9a4 7042a417 System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)056cfa10 7042a366 System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)056cfa24 7042a321 System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)056cfa3c 704461fc System.Threading.ThreadHelper.ThreadStart()056cfb80 73181396 [GCFrame: 056cfb80] 056cfd64 73181396 [DebuggerU2MCatchHandlerFrame: 056cfd64 ]

Page 21: Windows Debugging with WinDbg

Analyzing a .NET Deadlock Check who owns the locks

0:004> !syncblkIndex SyncBlock MonitorHeld Recursion Owning Thread Info SyncBlock Owner 3 011d3eac 3 1 011d37e8 1e70 4 031f3104 System.Object 4 011d3ee0 3 1 011d51f0 1864 5 031f3110 System.Object-----------------------------Total 4CCW 0RCW 0ComClassFactory 0Free 0

Page 22: Windows Debugging with WinDbg

Debugger Markup Language

Page 23: Windows Debugging with WinDbg

Some WinDbg commands Go

g

Set breakpoint at addressbp [Addr]

Set breakpoint at symbol (pattern)bm SymPattern

Set breakpoint on accessba [r|w|e] [Size] Addr

Page 24: Windows Debugging with WinDbg

More WinDbg commands Display callstack for selected thread

k

Display local variables and parametersdv [/i /t /V /a /n /z] [Pattern]

Display typedt [-n|y] [mod!]Name [-n|y] [Field] [Addr] -abcehioprsv 0:000> dt 0x0100297c MYTYPE1 +0x000 a : 22 +0x004 b : 43 '+' +0x006 c : 0x0 +0x008 d : 0x0 +0x00c gn : [6] 0x0

Page 25: Windows Debugging with WinDbg

More WinDbg commands Display callstacks for all threads

~* k

Dump all registersr

Show process environment block!peb

Display time consumption per thread!runaway

Search memory for patterns -[Flags][b|w|d|q|a|u] Range Pattern

Page 26: Windows Debugging with WinDbg

Selected SOS commands Display managed callstack

!CLRStack [-a] [-l] [-p] [-n]

Display managed / unmanaged callstack for all threads (see: !DumpStack)!EEStack [-short] [-EE]

Display IL associated with managed method!DumpIL <Managed DynamicMethod object> | <MethodDesc pointer>

Heap object statistics!DumpHeap –stat

Display object at specified address!DumpObj [-nofields] <object address>

Page 27: Windows Debugging with WinDbg

Selected SOSEX commands Display deadlocks between SyncBlocks and/or ReaderWriterLocks

!dlk [-d]

Display GC roots for the specified object!mroot <ObjectAddr> [-all]

Print a stack trace of managed and unmanaged frames!mk

Search the managed heap for stringsstrings [ModuleAddress] [Options]

Page 28: Windows Debugging with WinDbg

Selected PSSCOR commands Displays all objects on stack

!DumpStackObjects

Shows detailed info about running HttpRuntime (ASP.NET)!DumpHttpRuntime

Save all .NET assemblies to local disk!SaveAllModules folderName

Dump all exceptions found in heap (incl. callstacks)!dae

Find managed MethodDesc and JITted code address!Name2EE mscorlib.dll System.String.ToString

Page 29: Windows Debugging with WinDbg

For your Toolbox MS DebugDiag

http://blogs.msdn.com/b/debugdiag/ Process-specific rules for crashes,

perfcounters, memory usage, exceptions

Creates full dumps, mini dumps, stack traces

Built-in auto-analysis

Page 30: Windows Debugging with WinDbg

MS DebugDiag

Page 31: Windows Debugging with WinDbg

For your Toolbox SysInternals ProcDump

https://technet.microsoft.com/en-us/sysinternals/dd996900.aspx Commandline tool for dump creation Can be event-triggered, e.g. CPU usage

MS AppVerifier https://msdn.microsoft.com/en-us/library/ms220948(v=vs.90).aspx Runtime verification for native code, e.g. memory corruption

Page 32: Windows Debugging with WinDbg

For your Toolbox VerySleepy

http://www.codersnotes.com/sleepy/ CPU profiler Samples native callstacks, aggregates results over time

Page 33: Windows Debugging with WinDbg

Online Resources Common WinDbg Commands (Thematically Grouped)

http://windbg.info/doc/1-common-cmds.html

SOS.dll (SOS Debugging Extension) https://msdn.microsoft.com/en-us/library/bb190764(v=vs.110).aspx

"If broken it is, fix it you should" - Tess Ferrandez Blog https://blogs.msdn.microsoft.com/tess/

Software Diagnostics Institute http://www.dumpanalysis.org/

Page 34: Windows Debugging with WinDbg

Books Hewardt, Pravat: “Advanced Windows Debugging”

Vostokov, D.: “Memory Dump Analysis”

Page 35: Windows Debugging with WinDbg

Thank you!Twitter: https://twitter.com/ArnoHuBlog: http://arnosoftwaredev.blogspot.com