Case Study: Linux Kernel Crash Dump (LKCD)

1 Introduction
Linux is increasingly being deployed in enterprise class servers. The rise of linux as a production
kernel is phenomenal considering the fact that linux kernel development is done by a loosely knit
community of developers across the world. Its only in the recent years that the corporate world
has taken notice and is investing in linux in a big way.
Fujitsu and Wipro are involved in contributing to the open source initiatives by undertaking kernel
development projects.
The Linux Kernel Crash Dump project is one among the many kernel development projects
undertaken by the Wipro team for Fujitsu. This project involved enhancing LKCD.

2 Linux Kernel Crash Dump

Most modern enterprise servers address Reliability, Availability and Serviceability (RAS). One of the
key features of RAS is the ability to generate a crash dump after a kernel crash.

2.1

Problem Definition

Six broad areas as shown below were identified for LKCD development. These areas of
design and development were undertaken in two different phases. The scope of design and
development included the following:
IPF Porting
Existing LKCD is compliant with architectures like IA32 and PPC. LKCD enhancement
includes making LKCD compliant with the IA64 platform.
DWARF Compliance of lcrash
The existing lcrash implementation is specific to reading stabs debug information. On IA64,
the default debug information generated by gcc compilers is in dwarf2 format. Hence
existing lcrash implementation cannot analyze dump images if debug information is stored
in dwarf2 format. This activity requires making lcrash compliant with DWARF2. Targeted
kernel version is 2.6.x and hardware platforms are IA32 and IA64.
Compatibility with multiple device
Existing LKCD implementation supports dumping on a single device. Fujitsu has come up
with a patch, which supports dump to multiple devices for kernel version 2.4.17. This
activity requires porting of multiple dump device code from kernel version 2.4.17 to kernel
version 2.6-x for IA64 platform. In addition, this activity shall involve resolution of the
dump-overwriting problem.
Quality Upgradation
Six quality related issues are observed in the LKCD implementation. This activity requires
addressing of these problems. Targeted kernel version is 2.6.x and hardware platform is
IA64.
SCSI polling mode drivers
This involved development of polling based functionality in the existing SCSI drivers
available in the linux kernel (e.g. the fusion driver)

 Stand Alone Dump Design

This involved designing a safe and reliable dumping mechanism for the next generation
linux servers. This dumping mechanism is independent of the crashing system. The stand
alone dumping mechanism is targeted on IA64 platforms.
The above requirements will help in improving the flexibility of the enterprise server
systems. Fujitsu is planning to launch these servers by 2005 in association with Red Hat
and open source community.
Fujitsu evaluated this as a potential project to be outsourced mostly due to the open
source interaction required and their associated limitations with English language.

2.2

Solutions from Wipro

Wipro undertook all the above mentioned requirements and completed them successfully.
The solutions provided involved the following
Patches for LKCD on the IA64 platform
Patches for LKCD/lcrash for ensuring dwarf2 compatibility
Patches for LKCD/MDD
Resolving issues in Quality
Patches for SCSI drivers
Design for the Stand Alone Dumping Mechanism
All the above solutions involved
Prototyping
Multiple design alternatives
Usage of tools for code coverage, performance benchmarking and detecting memory
leaks
Usage Six Sigma methodologies
Interaction with the open source community
Acceptability of designs from the open source community and the client

2.3

Wipro demonstrated a high degree of technical competency to ensure that the patches
were accepted by the open source community.
Wipro met all the timelines and the schedules set in consultation with clients
Fujitsu-Wipro collaboration gave a high degree of visibility to the open source initiatives
of the 2 corporates

2.4

Benefits to client

Measurable Results

Multiple design alternatives proposed for every requirement. This has been commended
by the client.
Recognition and presence in the linux community
No slippage in schedule and timelines
CSAT rating of 5/5