Ladebug Kernel Debugging Ladebug Kernel Debugging TutorialTutorial

Bob Lidral

Introduction Introduction

Kinds of kernel debuggingHow to use Ladebug for kernel debuggingNot how to debug a kernel

AgendaAgenda Preparing a Kernel for Debugging Local Kernel Debugging

– Standalone Ladebug– Ladebug With kdbx

Crash Dump Analysis– Standalone Ladebug– Ladebug with kdbx

Remote Kernel Debugging– Without Gateway System– With Gateway System

OverviewOverview

Who– System Engineers– System Administrators

When and Why– Kernel Development and Debugging– Crash Analysis and Troubleshooting– System Configuration and Tuning

Preparing a Kernel for Preparing a Kernel for Debugging (if possible)Debugging (if possible)

Not always possible if debugging a running kernel or a crash dump

Compile without full optimizationDo not strip symbol table information

Local Kernel Debugging Local Kernel Debugging (Standalone)(Standalone)

Debug running kernel on same node as Ladebug session

Must have superuser (root) permissions Enter command:

# ladebug -k /vmunix /dev/mem Some Ladebug commands are not available in this

mode (e.g., cont, next, rerun, run, step, and stop)

Local Kernel Debugging Local Kernel Debugging (with kdbx)(with kdbx)

Debug running kernel on same node as kdbx session

Same restrictions as for standalone LadebugEnter command: # kdbx –dbx /bin/ladebug \ –k /vmunix /dev/memSupports all valid Ladebug commands plus

some kdbx command extensions

Crash Dump Analysis Crash Dump Analysis (Standalone)(Standalone)

Determine cause of a crash or panic May not need superuser (root) permissions Crash dump files are created in

/var/adm/crash Named vmunix.n and vmcore.n where n is

the version number and must match Enter command:

# ladebug -k vmunix.1 vmcore.1

Crash Dump Analysis Crash Dump Analysis (with kdbx)(with kdbx)

Determine cause of a crash or panicSame restrictions as for standalone LadebugEnter command:

# kdbx –dbx /bin/ladebug \ -k vmunix.1 vmcore.1Supports all valid Ladebug commands plus

some kdbx extensions

Remote Kernel DebuggingRemote Kernel Debugging

Run kernel on a remote node under control of Ladebug running on local node

Supports breakpoints in kernel code Requires serial line physical connection between

Test System and either Build or Gateway System– Specific communication port (usually COM1) on Test

System (see documentation)– Communication port may be configurable on Build or

Gateway System

Remote Kernel Debugging Remote Kernel Debugging TerminologyTerminology

Build System – system on which Ladebug executes

Test System – system on which kernel being debugged executes

Gateway System (optional) – between Build System and Test System, connected to– Test System by physical serial line– Build System over network

Remote Kernel Debugging Remote Kernel Debugging System RequirementsSystem Requirements

Test System– Tru64 UNIX Version 2.0 or higher

Verify: % uname –a

– Kernel Debugging Tools subsetVerify: % setld -i | grep -i kernel | \ grep installed

– Kernel Breakpoint Debugger kernel optionVerify: % /sbin/sysconfig -s kdebug

Remote Kernel Debugging Remote Kernel Debugging System RequirementsSystem Requirements

Build System– Tru64 UNIX Version 3.2 or higher– Copy of Test System kernel executable image– Copy of kernel source code (optional, recommended)

Gateway System– Tru64 UNIX Version 2.0 or higher– Kernel Debugging Tools subset

Verify by same methods as for Test System

Serial LineSerial Line

BC16E CableH8571-J DECconnect Passive Adapter

H8571-J DECconnect Passive Adapter

Remote Kernel Debugging Remote Kernel Debugging Without Gateway SystemWithout Gateway System

Specify Build System communication port to use for kdebug by appropriate entry in /etc/remote:– Default (COM1):

kdebug:dv=/dev/tty00:br#9600:pa=none:– Override in order to use as any of Build, Gateway, or

Test System (COM2):

kdebug:dv=/dev/tty01:br#9600:pa=none:

Note: the label “kdebug” must match the value of the Ladebug debugger variable $kdebug_line

kernel

CO

M1

CO

M2

kdebug

Test System

Remote Kernel Debugging Remote Kernel Debugging Without Gateway SystemWithout Gateway System

kdebugd

CO

M1

CO

M2

Ladebug

Build System

Serial Line

Remote Kernel Debugging Remote Kernel Debugging With Console Serial LineWith Console Serial Line

Connect second serial line– Must be connected to COM1 communication

port on Test System– Configurable communication port on Build or

Gateway System

Test System: redirect console input and output at console prompt

>>> set console serial

Remote Kernel Debugging Remote Kernel Debugging With Console Serial LineWith Console Serial Line

Build (or Gateway) System: add entry to /etc/remote for console serial line

tstsys:dv=/dev/tty00:br#9600:pa=none:Build (or Gateway) System: create a new

window for the Test System console and establish connection

% tip tstsys

kernel

CO

M1

CO

M2

kdebug

Test System

Remote Kernel Debugging Remote Kernel Debugging With Console Serial LineWith Console Serial Line

kdebugd

CO

M1

CO

M2

Ladebug

Build System

kdebugSerial Line

ConsoleSerial Line

Remote Kernel Debugging Remote Kernel Debugging With Console Serial LineWith Console Serial Line

Restore Test System console communication to its console:

>>> set console graphicsExit from tip session:

~.

Remote Kernel Debugging Remote Kernel Debugging With Gateway SystemWith Gateway System

Gateway System: specify communication port for kdebug by appropriate entry in /etc/remote:– Default (COM1):

kdebug:dv=/dev/tty00:br#9600:pa=none:– Override in order to use as any of Build, Gateway, or

Test System (COM2):

kdebug:dv=/dev/tty01:br#9600:pa=none:

Remote Kernel Debugging Remote Kernel Debugging With Gateway SystemWith Gateway System

Gateway System: open window to display kdebugd/kdebug communication traffic:– execute tty command to get pathname of

device corresponding to this window– Use this pathname for the value of the Ladebug

debugger variable $kdebug_dbgtty on the Build System

Remote Kernel Debugging Remote Kernel Debugging With Gateway SystemWith Gateway System

Build System: set Ladebug debugger variables describing connections:– $kdebug_host – node name of Gateway System

(default: localhost => no Gateway System)– $kdebug_line – label of /etc/remote entry on

Gateway System for kdebug communication port (default: kdebug)

– $kdebug_dbgtty – terminal window on Gateway System used to display kdebug communication traffic (output of tty command in that window)

(default: null => no Gateway System)

kernel

CO

M1

CO

M2

kdebug

Test System

Remote Kernel Debugging Remote Kernel Debugging With Gateway SystemWith Gateway System

kdebugd

Ladebug

Build System

kdebugdC

OM

1C

OM

2

Gateway System

Serial Line

Network

Remote Kernel Debugging Remote Kernel Debugging StartupStartup

On Build System% ladebug –remote ./vmunix.test

(ladebug) stop in ttyretype

(ladebug) stop in panic

On Test System– If not at console prompt (>>>) # shutdown –h now

Remote Kernel Debugging Remote Kernel Debugging StartupStartup

Then, in either order– On Build System

(ladebug) run– On Test System

>>> boot –flags k

Once started, typing ^R on the Test System will trigger the ttyretype breakpoint. There is no way for Ladebug to gain control except at a breakpoint or other interruption of the kernel.

Where to Get More Where to Get More InformationInformation

Ladebug Debugger Manual– Kernel Debugging

Local Kernel Debugging Crash Dump Analysis

– Remote Kernel Debugging with the kdebug Debugger

Compaq Tru64 Unix Reference Page– ladebug(1)

Other Compaq Tru64 Unix Other Compaq Tru64 Unix ManualsManuals

Kernel Debugging– crashdc– kdbx– kdebug

System AdministrationSystem Configuration and Tuning

Other Compaq Tru64 Unix Other Compaq Tru64 Unix Reference PagesReference Pages

crashdc(8)ikdebug(8)kdbx(8)