Crash Dump Analysis

MFF UK summer semester 2011

Jakub Jerm
Martin Dck
Crash Dump Analysis MFF UK summer semester 2011 2
Motivation
Goal:
Use the right tool when an application or the kernel crashes.
Observation:
printf() is usually not the right tool for satisfying goal.
Crash Dump Analysis MFF UK summer semester 2011 3
Motivation (2)

More observations

A crash might happen in a production system

We cannot alter the binary and run it again

We have to investigate from the record of the memory

layout in the time of the crash (crash dump)

It is not wise to reinvent the wheel

This course is about some well-proven tools and best

practices

Some degree of low-level programming knowledge is

necessary
Crash Dump Analysis MFF UK summer semester 2011 4
Outline

Basic low-level programming knowledge

Processor architectures, assembler, stack, ABI

IA-32, AMD64, SPARC V9

Basic system debugging tools

Solaris, Linux, Windows

Core files, crash dumps

How to analyze them

Basics of gdb, mdb

Crash Dump Analysis MFF UK summer semester 2011 5
Outline (2)

Some common causes of crashes

Memory corruption

Deadlock

Lockup

Tools for dynamic tracing

DTrace

Tools for command line system diagnostics

prstat, pstack, ptree, iostat, vmstat, mpstat, truss

Crash Dump Analysis MFF UK summer semester 2011 6
Agenda

Lectures

Tuesday 10:40 12:10 room S10

Tutorials / Labs

Tuesday 12:20 13:50 room SU1

Physical capacity is limited to 15 students

Please make sure you have your u-lab account working

You can bring your own notebooks

Make sure you have Internet connection

Crash Dump Analysis MFF UK summer semester 2011 7
Agenda (2)

Lecturers

Jakub Jerm (jakub.jermar@oracle.com)

Martin Dck (martin.decky@d3s.mff.cuni.cz)

Other contributors

Vtzslav Btrla (vitezslav.batrla@oracle.com)

Vineeth Pillai
Crash Dump Analysis MFF UK summer semester 2011 8
Agenda (3)

Web

Slides, basic information

http://d3s.mff.cuni.cz/cda

Mailing list

Important announcements, discussion

Please make sure you are enlisted

cda@d3s.mff.cuni.cz

https://d3s.mff.cuni.cz/mailman/listinfo/cda
Crash Dump Analysis MFF UK summer semester 2011 9
Grading

Credit

No lab attendance needed

But strongly recommended

Passing a practical test

E. g. identifying a root cause from a crash dump

At the end of the semester, three possibilities

Exam

Passing a written test

At least three terms, questions will be available

Crash Dump Analysis MFF UK summer semester 2011 10
Expected Knowledge

Basic C language knowledge

At least very basic knowledge of low-level

programming

Principles of computers should suffice

Basic English

For the slides and some lectures

Crash Dump Analysis MFF UK summer semester 2011 11
Resources

Lectures & Labs

Most important hands-on experience

Slides are just an outline

The Book

Frank Hofmann: The Solaris Operating System on

x86 Platforms, Crashdump Analysis, Operating
System Internals
http://opensolaris.org/os/community/documentation/files/book.pdf
Crash Dump Analysis MFF UK summer semester 2011 12
Resources (2)

Other books

Richard McDougall, Jim Mauro, Brendan Gregg:

Solaris Performance and Tools: DTrace and MDB
Techniques for Solaris 10 and OpenSolaris

Dynamic tracing and core dump analysis using mdb

Chris Drake, Kimberley Brown: PANIC! UNIX

System Crash Dump Analysis Handbook

Usefull general reference

Crash Dump Analysis MFF UK summer semester 2011 13
Resources (3)

Other useful resources

NASM Manual, Appendix B: Instruction List

Quick reference of IA-32 and AMD64 instructions

http://www.nasm.us/doc/nasmdocb.html

SPARC Assembly Language Reference Manual,

Appendix E: SPARC V9 Instruction Set

Quick reference of SPARC V9 instructions

http://download.oracle.com/docs/cd/E19963-01/821-1607/sparcv9-15322/index.html
Crash Dump Analysis MFF UK summer semester 2011 14
Disclaimer

Your mileage may vary

Not all systems have the same level of support for

crash dump analysis and observability

We would like to teach the principles without restraining

you to any particular platform

But we also need to be able to show the principles in

action

Therefore we are going to use Solaris most of the time

We are open to your suggestions