Core Dump Analysis

By Mark F. Brown

CC BY-SA

[1]

Preparation You will need an x86 Linux machine You will need developer tools binutils: objdump, readelf, nm gdb gcc

I will be going through all of these examples live.

Presentation is located here: goo.gl/Ocyrk

What is the Core? Core is an archaic name for memory Magnetic-core memory was an early

form of random access memory The term core comes from

conventional transformers whose windings surround a magnetic core[2]

Superseded by semiconductor memory Not to be mixed-up with Processors

(Cores)

What is a Core Dump? Core dump is a recorded state of the

working memory of a process image Process state snapshot is also stored Stored in a standard binary format

e.g. ELF in Linux (We will cover this platform on

x86)Macho in Mac OS X

Why Should I Care? Cross architecture and cross platform

debugging technique Great way to obtain postmortem

information from customers! :-) Good way for OSS developers to get

postmortem information from users. It is hardcore!

How To Trigger a Core Dump Process level core dumps are triggered

via a dumpable signal (asynchronous notification). Terminal control character Trigger them via kill(2) Erroneous behavior which triggers a

dumpable signal. Example dumpable signals SIGILL (illegal instruction) SIGQUIT (keyboard quit) SIGSEGV (invalid memory reference) SIGABT/SIGIOT (abnormal process termination

via: abort(3))

Configure the Environment To enable core dumps you need to

configure the proper resource limits. hard limit can be increased only by root. hard limit can be decreased by any user. soft limit can be set to less then or equal

hard limit by any user. ulimit (sh/bash) manages user resource

limits Configure user/resource limit core

size:

# ulimit -S -c unlimited

# ulimit -S -c

Lets Try Generating a Core 1. # ulimit -S -c unlimited2. # sleep 30003. Press Control + Backslash (^\)4. Terminal will state:

^\Quit (core dumped)5. # ls core* (mileage may vary based on distribution

settings)6. # file

Lets Generate a Core Dump Programmatically

Download file simple_abort.tar.bz at http://goo.gl/1tSgz

getrlimit(2) gets resource limit setrlimit(2) sets resource limit RLIMIT_CORE points to core dumps struct rlimit contains both the hard and

soft limits e.g. setrlimit(RLIMIT_CORE, &limit);

Typical User Space Memory Layout (32-bit x86)

Examining Live Maps* cat /proc//mapsFrom my x86 machine:007e6000-00801000 r-xp 00000000 08:06 261396 /lib/ld-2.11.1.so00801000-00802000 r--p 0001a000 08:06 261396 /lib/ld-2.11.1.so00802000-00803000 rw-p 0001b000 08:06 261396 /lib/ld-2.11.1.so00c99000-00c9a000 r-xp 00000000 00:00 0 [vdso]00cee000-00e41000 r-xp 00000000 08:06 389813 /lib/libc-2.11.1.so00e41000-00e42000 ---p 00153000 08:06 389813 /lib/libc-2.11.1.so00e42000-00e44000 r--p 00153000 08:06 389813 /lib/libc-2.11.1.so00e44000-00e45000 rw-p 00155000 08:06 389813 /lib/libc-2.11.1.so00e45000-00e48000 rw-p 00000000 00:00 0 08048000-0804f000 r-xp 00000000 08:06 142 /bin/sleep0804f000-08050000 r--p 00006000 08:06 142 /bin/sleep08050000-08051000 rw-p 00007000 08:06 142 /bin/sleep097c3000-097e4000 rw-p 00000000 00:00 0 [heap]bf9cc000-bf9e1000 rw-p 00000000 00:00 0 [stack]

Address Space Layout Randomization

Randomizes the base address of: Stack Libraries Heap VDSO (Virtually Dynamic Shared Objects)

See ASLR In Action*1. Check /proc/sys/kernel/randomize_va_page (should be 1 or 2)2. # sleep 3000 &3. # cat /proc//maps > maps1.txt4. # kill %15. # sleep 3000 &6. # cat /proc//maps > maps2.txt7. # kill %18. # sdiff maps1.txt maps2.txt | lessVirtual address mappings dont match for stack, heap,

libraries, vdso!

Object Files Executable, relocatable (.o), and shared

object (.so) files Executable and Linkable Format (ELF) Interpreter manages dynamic linking Interpreter (loader) ld.so/ld-linux.so C Library is libc.so

ELF contents ELF Header: Resides at beginning of file and

provides content information Provides offsets to Program and Section

Headers Program Headers: provides map for segments

in executable Sections Headers: point to sections (used for

linking) Segments are chunks of memory (used to

build a process image) Sections provide information for linking and

relocation

[4]

ELF Layout[5]

Example ELF HeaderELF Header: Magic: 7f 45 4c 46 01 01 01 00 00 00 00 00 00 00 00 00 Class: ELF32 Data: 2's complement, little endian Version: 1 (current) OS/ABI: UNIX - System V ABI Version: 0 Type: EXEC (Executable file) Machine: Intel 80386 Version: 0x1 Entry point address: 0x8048d20 Start of program headers: 52 (bytes into file) Start of section headers: 29104 (bytes into file) Flags: 0x0 Size of this header: 52 (bytes) Size of program headers: 32 (bytes) Number of program headers: 8 Size of section headers: 40 (bytes) Number of section headers: 28 Section header string table index: 27

Useful Program Headers

LOAD: maps code/data to address NOTE: special information DYNAMIC: Dynamic linking information PHDR: Program headers

Program Headers: Type Offset VirtAddr PhysAddr FileSiz MemSiz Flg Align NOTE 0x000234 0x00000000 0x00000000 0x0022c 0x00000 0 LOAD 0x001000 0x00226000 0x00000000 0x00000 0x153000 R E 0x1000

Useful Sections .text: program code .data: data (e.g. initialized local variables) .bss: zeroed out data (statically allocated

variables) .rodata: constant data, immutable strings .init/.fini: start and end code .symtab: symbol table .plt: procedure linkage table (dll functions) .got: global offset table (dll data)

Examining Core* Program Headers One NOTE segment to describe OS and

process data Multiple LOAD segments to describe each

memory map LOAD segments only count in data paged

in from a mmapd file, memory allocated by process, stack, heap, etc

For symbol translation the original ELF executable file is needed.

Core File Address Translation*

[Executable file]Symbol table '.symtab' contains 103 entries: Num: Value Size Type Bind Vis Ndx Name 68: 08048454 96 FUNC GLOBAL DEFAULT 14 main[Core file]Program Headers: Type Offset VirtAddr PhysAddr FileSiz MemSiz Flg Align NOTE 0x000234 0x00000000 0x00000000 0x0022c 0x00000 0 ... LOAD 0x00a000 0x08048000 0x00000000 0x00000 0x01000 R E 0x1000

Core File Address Translation*

Find the symbol main address in core! mains virtual address is 0x0804_8454 Locate the correct LOAD segment (0x0804_8000 len

0x5f4) It is in the first LOAD segment offset file offset 0xa000 Translate address [0x8454-0x8000]+0xa000 = 0xa454 Located in file core offset 0xa454 Remember due to demand paging the data you are

looking for may not be mapped in core, so dont expect everything to be there!

NOTE segment data*CORE notes have magic 45 52 4f 43 CORE (NT_PRSTATUS) contains:

elf_prstatus: /usr/include/linux/elfcore.h signal number: /usr/include/signal.h pid registers

(NT_PRPSINFO) elf_prpsinfo: /usr/include/linux/elfcore.h process state program filename ARG list

(NT_AUXV) auxiliary vector /usr/include/linux/auxvec.h

PLT (Procedure Linkage PLT is a jump table to functions in a

shared library REL.PLT contains the dynamic address

to jump to Loader initializes all REL.PLT entries to

self When entry is called you jump to

loader and loader fixes up to actual destination

This is called lazy loading. It is used to speed up program initialization time

Matching Function Symbols Example*

Given address real address of abort find symbol Check symbol table (.symtab) to see if statically bound If not statically bound build .rel.plt table using executableRelocation section '.rel.plt' at offset 0x3f8 contains 11 entries: Offset Info Type Sym.Value Sym. Name0804a000 00000107 R_386_JUMP_SLOT 00000000 abort

Map PLT offset to core e.g. 0x0804_a000-> core offset 0x0000_b000

Program Headers:Type Offset VirtAddr PhysAddr FileSiz MemSiz Flg AlignNOTE 0x000234 0x00000000 0x00000000 0x0022c 0x00000 0LOAD 0x001000 0x00336000 0x00000000 0x00000 0x1b000 R E 0x1000LOAD 0x006000 0x00540000 0x00000000 0x03000 0x03000 RW 0x1000LOAD 0x009000 0x00c2b000 0x00000000 0x01000 0x01000 R E 0x1000LOAD 0x00a000 0x08048000 0x00000000 0x00000 0x01000 R E 0x1000LOAD 0x00a000 0x08049000 0x00000000 0x01000 0x01000 R 0x1000LOAD 0x00b000 0x0804a000 0x00000000 0x01000 0x01000 RW 0x1000

Reverse Matching Virtual Addresses*

Build a table of .rel.plt targets Match unknown address with .rel.plt

table Validate by using .symtab size as a

boundary checker.

Taking the easy way out* gdb (GNU Debugger) is a great tool for

interpreting core dumps Targeting other platforms is easy

./configure --target=arm-linux Pointing to shared libraries

set solib-absolute-prefix Loading contents

file core-file

Core Dump Tuning Pipe core dump output or core file

name

/proc/sys/kernel/core_pattern Control core output (default 0x3)

/proc//coredump_filter

bit 0 Dump anonymous private mappings.

bit 1 Dump anonymous shared mappings.

bit 2 Dump file-backed private mappings.

bit 3 Dump file-backed shared mappings.

Useful Tools readelf: displays information about elf

files dd: copy or convert data hexedit: view/edits files in hex or ASCII od: dumps files to terminal objdump: dumps object file data

(disassmbler) gcore: generates cores on live

processes

The End

References1. http://en.wikipedia.org/wiki/File:Ferrite_core_memory.jpg2. http://en.wikipedia.org/wiki/Magnetic-core_memory3. The Linux Programming Interface4. http://www.sco.com/developers/devspecs/gabi41.pdf5. http://en.wikipedia.org/wiki/File:Elf-layout--en.svg6. Linux 2.6: Documentation/arm/memory.txt