Core Dump Analysis

By Mark F. Brown
<mark.brown314@gmail.com>
CC BY-SA
[1]
Preparation
You will need an x86 Linux machine
You will need developer tools
binutils: objdump, readelf, nm
gdb
gcc
I will be going through all of these
examples live.
Presentation is located here:
goo.gl/Ocyrk
What is the Core?
Core is an archaic name for memory
Magnetic-core memory was an early
form of random access memory
The term core comes from
conventional transformers whose
windings surround a magnetic core
[2]
Superseded by semiconductor memory
Not to be mixed-up with Processors
(Cores)
What is a Core Dump?
Core dump is a recorded state of the
working memory of a process image
Process state snapshot is also stored
Stored in a standard binary format
e.g.
ELF in Linux (We will cover this platform on
x86)
Macho in Mac OS X
Why Should I Care?
Cross architecture and cross platform
debugging technique
Great way to obtain postmortem
information from customers! :-)
Good way for OSS developers to get
postmortem information from users.
It is hardcore!
How To Trigger a Core Dump
Process level core dumps are triggered
via a dumpable signal (asynchronous
notication).
Terminal control character
Trigger them via kill(2)
Erroneous behavior which triggers a
dumpable signal.
Example dumpable signals
SIGILL (illegal instruction)
SIGQUIT (keyboard quit)
SIGSEGV (invalid memory reference)
SIGABT/SIGIOT (abnormal process termination
via: abort(3))
Congure the Environment
To enable core dumps you need to
congure the proper resource limits.
hard limit can be increased only by root.
hard limit can be decreased by any user.
soft limit can be set to less then or equal
hard limit by any user.
ulimit (sh/bash) manages user resource
limits
Congure user/resource limit core
size:
# ulimit -S -c unlimited
# ulimit -S -c <size in kilobytes>
Lets Try Generating a Core
1. # ulimit -S -c unlimited
2. # sleep 3000
3. Press Control + Backslash (^\)
4. Terminal will state:
^\Quit (core dumped)
5. # ls core* (mileage may vary based on distribution
settings)
6. # le <core le>
Lets Generate a Core Dump
Programmatically
Download le simple_abort.tar.bz at
http://goo.gl/1tSgz
getrlimit(2) gets resource limit
setrlimit(2) sets resource limit
RLIMIT_CORE points to core dumps
struct rlimit contains both the hard and
soft limits
e.g. setrlimit(RLIMIT_CORE, &limit);
Typical User Space Memory
Layout (32-bit x86)
Examining Live Maps*
cat /proc/<PID>/maps
From my x86 machine:
007e6000-00801000 r-xp 00000000 08:06 261396 /lib/ld-2.11.1.so
00801000-00802000 r--p 0001a000 08:06 261396 /lib/ld-2.11.1.so
00802000-00803000 rw-p 0001b000 08:06 261396 /lib/ld-2.11.1.so
00c99000-00c9a000 r-xp 00000000 00:00 0 [vdso]
00cee000-00e41000 r-xp 00000000 08:06 389813 /lib/libc-2.11.1.so
00e41000-00e42000 ---p 00153000 08:06 389813 /lib/libc-2.11.1.so
00e42000-00e44000 r--p 00153000 08:06 389813 /lib/libc-2.11.1.so
00e44000-00e45000 rw-p 00155000 08:06 389813 /lib/libc-2.11.1.so
00e45000-00e48000 rw-p 00000000 00:00 0
08048000-0804f000 r-xp 00000000 08:06 142 /bin/sleep
0804f000-08050000 r--p 00006000 08:06 142 /bin/sleep
08050000-08051000 rw-p 00007000 08:06 142 /bin/sleep
097c3000-097e4000 rw-p 00000000 00:00 0 [heap]

bf9cc000-bf9e1000 rw-p 00000000 00:00 0 [stack]

Address Space Layout
Randomization
Randomizes the base address of:
Stack
Libraries
Heap
VDSO (Virtually Dynamic Shared Objects)
See ASLR In Action*
1. Check /proc/sys/kernel/randomize_va_page
(should be 1 or 2)
2. # sleep 3000 &
3. # cat /proc/<PID>/maps > maps1.txt
4. # kill %1
5. # sleep 3000 &
6. # cat /proc/<PID>/maps > maps2.txt
7. # kill %1
8. # sdi" maps1.txt maps2.txt | less
Virtual address mappings dont match for stack, heap,
libraries, vdso!
Object Files
Executable, relocatable (.o), and shared
object (.so) les
Executable and Linkable Format (ELF)
Interpreter manages dynamic linking
Interpreter (loader) ld.so/ld-linux.so
C Library is libc.so
ELF contents
ELF Header: Resides at beginning of le and
provides content information
Provides o"sets to Program and Section
Headers
Program Headers: provides map for segments
in executable
Sections Headers: point to sections (used for
linking)
Segments are chunks of memory (used to
build a process image)
Sections provide information for linking and
relocation
[4]
ELF Layout
[5]
Example ELF Header
ELF Header:
Magic: 7f 45 4c 46 01 01 01 00 00 00 00 00 00 00 00 00
Class: ELF32
Data: 2's complement, little endian
Version: 1 (current)
OS/ABI: UNIX - System V
ABI Version: 0
Type: EXEC (Executable le)
Machine: Intel 80386
Version: 0x1
Entry point address: 0x8048d20
Start of program headers: 52 (bytes into le)
Start of section headers: 29104 (bytes into le)
Flags: 0x0
Size of this header: 52 (bytes)
Size of program headers: 32 (bytes)
Number of program headers: 8
Size of section headers: 40 (bytes)
Number of section headers: 28
Section header string table index: 27
Useful Program Headers
LOAD: maps code/data to address
NOTE: special information
DYNAMIC: Dynamic linking information
PHDR: Program headers
Program Headers:
Type O"set VirtAddr PhysAddr FileSiz MemSiz Flg Align
NOTE 0x000234 0x00000000 0x00000000 0x0022c 0x00000 0
LOAD 0x001000 0x00226000 0x00000000 0x00000 0x153000 R E 0x1000
Useful Sections
.text: program code
.data: data (e.g. initialized local variables)
.bss: zeroed out data (statically allocated
variables)
.rodata: constant data, immutable strings
.init/.ni: start and end code
.symtab: symbol table
.plt: procedure linkage table (dll functions)
.got: global o"set table (dll data)
Examining Core*
Program Headers
One NOTE segment to describe OS and
process data
Multiple LOAD segments to describe each
memory map
LOAD segments only count in data paged
in from a mmapd le, memory allocated
by process, stack, heap, etc
For symbol translation the original ELF
executable le is needed.
Core File Address
Translation*
[Executable le]
Symbol table '.symtab' contains 103 entries:
Num: Value Size Type Bind Vis Ndx Name
68: 08048454 96 FUNC GLOBAL DEFAULT 14 main
[Core le]
Program Headers:
Type O"set VirtAddr PhysAddr FileSiz MemSiz Flg Align
NOTE 0x000234 0x00000000 0x00000000 0x0022c 0x00000 0
...
LOAD 0x00a000 0x08048000 0x00000000 0x00000 0x01000 R E 0x1000
Core File Address
Translation*
Find the symbol main address in core!
mains virtual address is 0x0804_8454
Locate the correct LOAD segment (0x0804_8000 len
0x5f4)
It is in the rst LOAD segment o"set le o"set 0xa000
Translate address [0x8454-0x8000]+0xa000 = 0xa454
Located in le core o"set 0xa454
Remember due to demand paging the data you are
looking for may not be mapped in core, so dont expect
everything to be there!
NOTE segment data*
CORE notes have magic 45 52 4f 43 CORE
(NT_PRSTATUS) contains:
! elf_prstatus: /usr/include/linux/elfcore.h
! signal number: /usr/include/signal.h
! pid
! registers
(NT_PRPSINFO) elf_prpsinfo: /usr/include/linux/
elfcore.h
! process state
! program lename
! ARG list
(NT_AUXV) auxiliary vector /usr/include/linux/
auxvec.h
PLT (Procedure Linkage
PLT is a jump table to functions in a
shared library
REL.PLT contains the dynamic address
to jump to
Loader initializes all REL.PLT entries to
self
When entry is called you jump to
loader and loader xes up to actual
destination
This is called lazy loading. It is used to
speed up program initialization time
Matching Function Symbols
Example*
Given address real address of abort nd symbol
Check symbol table (.symtab) to see if statically bound
If not statically bound build .rel.plt table using executable
Relocation section '.rel.plt' at offset 0x3f8 contains 11 entries:
Offset Info Type Sym.Value Sym. Name
0804a000 00000107 R_386_JUMP_SLOT 00000000 abort
Map PLT o"set to core e.g. 0x0804_a000-> core o"set
0x0000_b000
Program Headers:
Type Offset VirtAddr PhysAddr FileSiz MemSiz Flg Align
NOTE 0x000234 0x00000000 0x00000000 0x0022c 0x00000 0
LOAD 0x001000 0x00336000 0x00000000 0x00000 0x1b000 R E 0x1000
LOAD 0x006000 0x00540000 0x00000000 0x03000 0x03000 RW 0x1000
LOAD 0x009000 0x00c2b000 0x00000000 0x01000 0x01000 R E 0x1000
LOAD 0x00a000 0x08048000 0x00000000 0x00000 0x01000 R E 0x1000
LOAD 0x00a000 0x08049000 0x00000000 0x01000 0x01000 R 0x1000
LOAD 0x00b000 0x0804a000 0x00000000 0x01000 0x01000 RW 0x1000
Reverse Matching Virtual
Addresses*
Build a table of .rel.plt targets
Match unknown address with .rel.plt
table
Validate by using .symtab size as a
boundary checker.
Taking the easy way out*
gdb (GNU Debugger) is a great tool for
interpreting core dumps
Targeting other platforms is easy
./congure --target=arm-linux
Pointing to shared libraries
set solib-absolute-prex <prex to path>
Loading contents
le <executable>
core-le <core>
Core Dump Tuning
Pipe core dump output or core le
name
/proc/sys/kernel/core_pattern
Control core output (default 0x3)
/proc/<PID>/coredump_lter
bit 0 Dump anonymous private mappings.
bit 1 Dump anonymous shared mappings.
bit 2 Dump le-backed private mappings.
bit 3 Dump le-backed shared mappings.
Useful Tools
readelf: displays information about elf
les
dd: copy or convert data
hexedit: view/edits les in hex or ASCII
od: dumps les to terminal
objdump: dumps object le data
(disassmbler)
gcore: generates cores on live
processes
The End
References
1. http://en.wikipedia.org/wiki/File:Ferrite_core_memory.jpg
2. http://en.wikipedia.org/wiki/Magnetic-core_memory
3. The Linux Programming Interface
4. http://www.sco.com/developers/devspecs/gabi41.pdf
5. http://en.wikipedia.org/wiki/File:Elf-layout--en.svg
6. Linux 2.6: Documentation/arm/memory.txt