sql server crash dump analysis a brief tour with windbg and other ugly tools pablo Álvarez doval...
TRANSCRIPT
![Page 1: SQL Server Crash Dump Analysis A brief tour with WinDbg and other ugly tools Pablo Álvarez Doval Debugging & Optimization Team Lead pablod@plainconcepts.com](https://reader035.vdocuments.mx/reader035/viewer/2022062518/56649efb5503460f94c0eb97/html5/thumbnails/1.jpg)
SQL Server Crash Dump Analysis
A brief tour with WinDbg and other ugly tools
Pablo Álvarez DovalDebugging & Optimization Team Lead
![Page 2: SQL Server Crash Dump Analysis A brief tour with WinDbg and other ugly tools Pablo Álvarez Doval Debugging & Optimization Team Lead pablod@plainconcepts.com](https://reader035.vdocuments.mx/reader035/viewer/2022062518/56649efb5503460f94c0eb97/html5/thumbnails/2.jpg)
Who am I?
![Page 3: SQL Server Crash Dump Analysis A brief tour with WinDbg and other ugly tools Pablo Álvarez Doval Debugging & Optimization Team Lead pablod@plainconcepts.com](https://reader035.vdocuments.mx/reader035/viewer/2022062518/56649efb5503460f94c0eb97/html5/thumbnails/3.jpg)
![Page 4: SQL Server Crash Dump Analysis A brief tour with WinDbg and other ugly tools Pablo Álvarez Doval Debugging & Optimization Team Lead pablod@plainconcepts.com](https://reader035.vdocuments.mx/reader035/viewer/2022062518/56649efb5503460f94c0eb97/html5/thumbnails/4.jpg)
![Page 5: SQL Server Crash Dump Analysis A brief tour with WinDbg and other ugly tools Pablo Álvarez Doval Debugging & Optimization Team Lead pablod@plainconcepts.com](https://reader035.vdocuments.mx/reader035/viewer/2022062518/56649efb5503460f94c0eb97/html5/thumbnails/5.jpg)
Session Objectives What is this session about? What isn’t this session about?
![Page 6: SQL Server Crash Dump Analysis A brief tour with WinDbg and other ugly tools Pablo Álvarez Doval Debugging & Optimization Team Lead pablod@plainconcepts.com](https://reader035.vdocuments.mx/reader035/viewer/2022062518/56649efb5503460f94c0eb97/html5/thumbnails/6.jpg)
![Page 7: SQL Server Crash Dump Analysis A brief tour with WinDbg and other ugly tools Pablo Álvarez Doval Debugging & Optimization Team Lead pablod@plainconcepts.com](https://reader035.vdocuments.mx/reader035/viewer/2022062518/56649efb5503460f94c0eb97/html5/thumbnails/7.jpg)
Who are you?
![Page 8: SQL Server Crash Dump Analysis A brief tour with WinDbg and other ugly tools Pablo Álvarez Doval Debugging & Optimization Team Lead pablod@plainconcepts.com](https://reader035.vdocuments.mx/reader035/viewer/2022062518/56649efb5503460f94c0eb97/html5/thumbnails/8.jpg)
Agenda Tools of the Trade Brief Windows Architecture Refresher SQL Server Post-mortem Debugging
Handling SQL Server dumps Analyzing SQL Server dumps
Debugging .NET Applications with SOS
![Page 9: SQL Server Crash Dump Analysis A brief tour with WinDbg and other ugly tools Pablo Álvarez Doval Debugging & Optimization Team Lead pablod@plainconcepts.com](https://reader035.vdocuments.mx/reader035/viewer/2022062518/56649efb5503460f94c0eb97/html5/thumbnails/9.jpg)
Debugging Tools for Windows Free download:
http://www.microsoft.com/whdc/devtools/debugging Updated several times a year Debuggers, extensions, tools and a great help file:
windbg.exe, kd.exe, cdb.exe gflags.exe, tlist.exe, etc debugger.chm
Can be installed via xcopy
![Page 10: SQL Server Crash Dump Analysis A brief tour with WinDbg and other ugly tools Pablo Álvarez Doval Debugging & Optimization Team Lead pablod@plainconcepts.com](https://reader035.vdocuments.mx/reader035/viewer/2022062518/56649efb5503460f94c0eb97/html5/thumbnails/10.jpg)
Demo 0: … is it really so ugly?
![Page 11: SQL Server Crash Dump Analysis A brief tour with WinDbg and other ugly tools Pablo Álvarez Doval Debugging & Optimization Team Lead pablod@plainconcepts.com](https://reader035.vdocuments.mx/reader035/viewer/2022062518/56649efb5503460f94c0eb97/html5/thumbnails/11.jpg)
Thesaurus Just to keep with the forensics analogy:
Corpse Dump file Forensic Lab WinDbg Forensic Scientist You! Gray’s Anathomy Windows Internals 5th Ed.
We are not going to get into details, but we will do a little refresher of some key concepts
![Page 12: SQL Server Crash Dump Analysis A brief tour with WinDbg and other ugly tools Pablo Álvarez Doval Debugging & Optimization Team Lead pablod@plainconcepts.com](https://reader035.vdocuments.mx/reader035/viewer/2022062518/56649efb5503460f94c0eb97/html5/thumbnails/12.jpg)
User mode vs. Kernel mode
User ModeKernel Mode
Hardware Abstraction Layer (HAL)
Device Drivers MicrokernelGraphics Controller
Object Manager
Executive Services
FS
I/O IPC Memory
Processes
Security WMPNP
UNIXLSA Shell
Lsass.exe
Client/Server
csrss.exe
Notepadnotepad.e
xe
Windows on Windows
wowexec.exe
Virtual DOS Machine
ntvdm.exe
Win32
Interix
![Page 13: SQL Server Crash Dump Analysis A brief tour with WinDbg and other ugly tools Pablo Álvarez Doval Debugging & Optimization Team Lead pablod@plainconcepts.com](https://reader035.vdocuments.mx/reader035/viewer/2022062518/56649efb5503460f94c0eb97/html5/thumbnails/13.jpg)
Application, Processes and Threads An application is formed by one or more processes
A process is an in-memory executable, which is made up of one or more threads and its resources
A thread is the basic unit of execution and scheduling in the OS.
![Page 14: SQL Server Crash Dump Analysis A brief tour with WinDbg and other ugly tools Pablo Álvarez Doval Debugging & Optimization Team Lead pablod@plainconcepts.com](https://reader035.vdocuments.mx/reader035/viewer/2022062518/56649efb5503460f94c0eb97/html5/thumbnails/14.jpg)
… is it really worth it?
![Page 15: SQL Server Crash Dump Analysis A brief tour with WinDbg and other ugly tools Pablo Álvarez Doval Debugging & Optimization Team Lead pablod@plainconcepts.com](https://reader035.vdocuments.mx/reader035/viewer/2022062518/56649efb5503460f94c0eb97/html5/thumbnails/15.jpg)
![Page 16: SQL Server Crash Dump Analysis A brief tour with WinDbg and other ugly tools Pablo Álvarez Doval Debugging & Optimization Team Lead pablod@plainconcepts.com](https://reader035.vdocuments.mx/reader035/viewer/2022062518/56649efb5503460f94c0eb97/html5/thumbnails/16.jpg)
Other good reasons…
![Page 17: SQL Server Crash Dump Analysis A brief tour with WinDbg and other ugly tools Pablo Álvarez Doval Debugging & Optimization Team Lead pablod@plainconcepts.com](https://reader035.vdocuments.mx/reader035/viewer/2022062518/56649efb5503460f94c0eb97/html5/thumbnails/17.jpg)
Win32 Virtual Memory Addressing (I)
Kernel
Process 1
Thread 1
Thread 2
Thread n
:
Process 2
Thread 1
Thread 2
Thread n
:
sqlsrv.exe
Thread 1
Thread 2
Thread n
:
Process n
Thread 1
Thread 2
Thread n
:…
4 G
b
2 G
b2
Gb
![Page 18: SQL Server Crash Dump Analysis A brief tour with WinDbg and other ugly tools Pablo Álvarez Doval Debugging & Optimization Team Lead pablod@plainconcepts.com](https://reader035.vdocuments.mx/reader035/viewer/2022062518/56649efb5503460f94c0eb97/html5/thumbnails/18.jpg)
Win32 Virtual Memory Addressing(II)
![Page 19: SQL Server Crash Dump Analysis A brief tour with WinDbg and other ugly tools Pablo Álvarez Doval Debugging & Optimization Team Lead pablod@plainconcepts.com](https://reader035.vdocuments.mx/reader035/viewer/2022062518/56649efb5503460f94c0eb97/html5/thumbnails/19.jpg)
Thread Call Stacks Shows part of the history of the function calls of the
thread Each thread has its own Call Stack i.e:
ntdll!KiFastSystemCallRetUSER32!NtUserGetMessage+0xcnotepad!WinMain+0xe5notepad!WinMainCRTStartup+0x174kernel32!BaseProcessStart+0x23
![Page 20: SQL Server Crash Dump Analysis A brief tour with WinDbg and other ugly tools Pablo Álvarez Doval Debugging & Optimization Team Lead pablod@plainconcepts.com](https://reader035.vdocuments.mx/reader035/viewer/2022062518/56649efb5503460f94c0eb97/html5/thumbnails/20.jpg)
Call Stacks (I) Each thread of the process has its own call stack:
![Page 21: SQL Server Crash Dump Analysis A brief tour with WinDbg and other ugly tools Pablo Álvarez Doval Debugging & Optimization Team Lead pablod@plainconcepts.com](https://reader035.vdocuments.mx/reader035/viewer/2022062518/56649efb5503460f94c0eb97/html5/thumbnails/21.jpg)
Call Stacks (II) Each frame has the following structure:
Frame
Parameters
Return Address
Frame Pointer
Exception Handler
Local Variables
Registros
![Page 22: SQL Server Crash Dump Analysis A brief tour with WinDbg and other ugly tools Pablo Álvarez Doval Debugging & Optimization Team Lead pablod@plainconcepts.com](https://reader035.vdocuments.mx/reader035/viewer/2022062518/56649efb5503460f94c0eb97/html5/thumbnails/22.jpg)
Symbols Symbols make the call stack useful:
Without Symbols:
With Symbols:
kernel32!+136aa
kernel32!CreateFileW+0x35f
![Page 23: SQL Server Crash Dump Analysis A brief tour with WinDbg and other ugly tools Pablo Álvarez Doval Debugging & Optimization Team Lead pablod@plainconcepts.com](https://reader035.vdocuments.mx/reader035/viewer/2022062518/56649efb5503460f94c0eb97/html5/thumbnails/23.jpg)
Symbol formats
Current format: .PDB Old Format: .DBG Retail vs. Debug (Free vs. Checked) builds Private symbols vs. public symbols
![Page 24: SQL Server Crash Dump Analysis A brief tour with WinDbg and other ugly tools Pablo Álvarez Doval Debugging & Optimization Team Lead pablod@plainconcepts.com](https://reader035.vdocuments.mx/reader035/viewer/2022062518/56649efb5503460f94c0eb97/html5/thumbnails/24.jpg)
Symbol Servers Uses the File System as a Symbol’s database:
Organized by name and a unique identifier Folder structure:
\\SymSrv\file_name.pdb\unique_number\____ i.e:
\\Symbols\ntdll.pdb\3B5EDCA52\ntdll.pdb\\Symbols\ntdll.pdb\380FCC4F2\ntdll.pdb
![Page 25: SQL Server Crash Dump Analysis A brief tour with WinDbg and other ugly tools Pablo Álvarez Doval Debugging & Optimization Team Lead pablod@plainconcepts.com](https://reader035.vdocuments.mx/reader035/viewer/2022062518/56649efb5503460f94c0eb97/html5/thumbnails/25.jpg)
Demo 1: Scheduler Non-Yielding
![Page 26: SQL Server Crash Dump Analysis A brief tour with WinDbg and other ugly tools Pablo Álvarez Doval Debugging & Optimization Team Lead pablod@plainconcepts.com](https://reader035.vdocuments.mx/reader035/viewer/2022062518/56649efb5503460f94c0eb97/html5/thumbnails/26.jpg)
Scenario A customer’s SQL Server 2000 is hanging, showing 17883
errors in SQL Server’s ErrorLog
When these errores ocurr, SQL Server automatically triggers the creation of a dump
…
2007-02-12 11:17:14.10 server Error: 17883, Severity: 1, State: 0
2007-02-12 11:17:14.10 server Process 59:0 (834) UMS Context 0x125ABD80 appears to be non-yielding on Scheduler 1.
…
![Page 27: SQL Server Crash Dump Analysis A brief tour with WinDbg and other ugly tools Pablo Álvarez Doval Debugging & Optimization Team Lead pablod@plainconcepts.com](https://reader035.vdocuments.mx/reader035/viewer/2022062518/56649efb5503460f94c0eb97/html5/thumbnails/27.jpg)
Demo 2: DBCC CHECKDB
![Page 28: SQL Server Crash Dump Analysis A brief tour with WinDbg and other ugly tools Pablo Álvarez Doval Debugging & Optimization Team Lead pablod@plainconcepts.com](https://reader035.vdocuments.mx/reader035/viewer/2022062518/56649efb5503460f94c0eb97/html5/thumbnails/28.jpg)
Demo 3: Cluster Resources
![Page 29: SQL Server Crash Dump Analysis A brief tour with WinDbg and other ugly tools Pablo Álvarez Doval Debugging & Optimization Team Lead pablod@plainconcepts.com](https://reader035.vdocuments.mx/reader035/viewer/2022062518/56649efb5503460f94c0eb97/html5/thumbnails/29.jpg)
Managed Debugging with .NET
WinDbg is a native debugger
In order to debug .NET code we need to use debugger extensions: SOS.dll (until framework .NET 3.5) CLR.dll (framework 4.0)
Why all this? Is it worth it?
![Page 30: SQL Server Crash Dump Analysis A brief tour with WinDbg and other ugly tools Pablo Álvarez Doval Debugging & Optimization Team Lead pablod@plainconcepts.com](https://reader035.vdocuments.mx/reader035/viewer/2022062518/56649efb5503460f94c0eb97/html5/thumbnails/30.jpg)
Demo 4: Managed Debugging with SOS
![Page 31: SQL Server Crash Dump Analysis A brief tour with WinDbg and other ugly tools Pablo Álvarez Doval Debugging & Optimization Team Lead pablod@plainconcepts.com](https://reader035.vdocuments.mx/reader035/viewer/2022062518/56649efb5503460f94c0eb97/html5/thumbnails/31.jpg)
Some cool tips… Did we really get to this slide in time?! Well.. enjoy some free tips!
Using SOS from VS.NET Memory dump analysis from inside VS2010
![Page 32: SQL Server Crash Dump Analysis A brief tour with WinDbg and other ugly tools Pablo Álvarez Doval Debugging & Optimization Team Lead pablod@plainconcepts.com](https://reader035.vdocuments.mx/reader035/viewer/2022062518/56649efb5503460f94c0eb97/html5/thumbnails/32.jpg)
Resources [email protected] @Plain Concepts
http://www.geeks.ms/blogs/palvarez http://www.geeks.ms/blogs/rcorral http://www.geeks.ms/blogs/luisguerrero
@MSDN: http://blogs.msdn.com/tess/
Books: Microsoft Windows Internals, 5th Ed.
[Mark E. Russinovich and David A. Solomon]Microsoft Press.
Debugging Applications for Microsoft .NET and Microsoft Windows[John Robbins]Microsoft Press.
![Page 33: SQL Server Crash Dump Analysis A brief tour with WinDbg and other ugly tools Pablo Álvarez Doval Debugging & Optimization Team Lead pablod@plainconcepts.com](https://reader035.vdocuments.mx/reader035/viewer/2022062518/56649efb5503460f94c0eb97/html5/thumbnails/33.jpg)
Any Questions?
Thanks!