Kdump&Kernelcrashdumpanalysis

Part:1
Introductiontokdump
Part:2
Introductiontokernelcrashdumpanalysis
http://people.redhat.com/gtiwari/fudcon/

IntroductiontoKdump

BulandSingh
RedHat

IntroductiontoKdump
Whatiskexec?
Howdoesitwork?
Whatiskdump?
Howdoesitwork?
Howtoconfigurekdump?
Howtotestkdumpsetup?
Howtoverifykernelcrashdump(vmcore)?
References

Whatiskexec?
kexecisafastbootmechanismthatallowsbootingakernel
fromthecontextofanalreadyrunningkernelwithoutgoing
throughBIOSorpassingthroughthebootloaderstage.

kexecperformsthefunctionofthebootloaderfromwithinthe
kernelandwhenthesystembootsakernelwithkexec,itskips
thebootloaderstage.

Thishastheeffectofreducingthetimerequiredforareboot.

Kexecincludetwocomponents:
UserSpaceTool:/usr/sbin/kexec
oThebinary/usr/sbin/kexecisprovided"kexectools"package.
KernelSystemCall:kexec_load()
oTheLinuxkernelconfiguration"CONFIG_KEXEC"enablesthe
kexecsystemcall.
#grep"CONFIG_KEXEC="/boot/config4.0.4301.fc22.x86_64
CONFIG_KEXEC=y

Useofkexec:
Fastersystemreboots.
BootingwithoutGRUBorLILObootloader.
Savingthedumpofacrashedkernel.

Usingkexec:2Stepprocess
Loadinganewkernelintothephysicalmemory.
#kexecl

Rebootingwiththepreloadedkernel.
#kexece

KexecDesign:

Kexeccommand:
Commandtoloadthekernelinmemory:
#kexecl<kernelimage>initrd=<initrdimage>append=<commandlineoptions>
kernelimage:Pathofthekernelfile.
initrdimage:Pathoftheinitrdimagetobeusedduringboot.
Commandlineoptions:Thecommandlineparametersfornewkernel.
Commandtobootwiththepreloadedkernel:
#kexece
Commandtounloadthekernelwhichwasloadedwiththekexeclcommand.
#kexecu

Kexeccommand:
Commandtoloadakerneltobootatthetimeofkernelpanic.
#kexecp<kernelimage>initrd=<initrdimage>append=<commandlineoptions>
Commandtounloadthekernelwhichwasloadedwiththekexecpcommand.
#kexecup

Commandtocheckthestatusofkexeckernel.
#cat/sys/kernel/kexec_loaded
0kernelisnotloadedinthememory.
1kernelisloadedinthememory.

Demo:

Whatiskdump?
KdumpisareliablekexecbasedcrashdumpingmechanismforLinux.
Kdumpfunctionalityisbrokenmainlyintwocomponents;
oUserspace(kexec)

/usr/sbin/kexec

#grep"CONFIG_KEXEC="/boot/config4.0.4301.fc22.x86_64
CONFIG_KEXEC=y
oKernelspace(kdump)
#grepCRASH_DUMP/boot/config4.0.4301.fc22.x86_64
CONFIG_CRASH_DUMP=y

Whatiskdump?
Kdumpneedtwodifferentkernel:

oStandard(production)kernel.
oCrash(capture)kernel.

Thecrashdumpsarecapturedfromthecontextofafreshlybootedkernel;notfromthecontextofthe
crashedkernel.

Kdumpcanbeconfiguredtodumpeitherdirectlytoadevice,toafile,ortosomeremotelocation
onthenetworkviaNFSorSSH.

Kdumpandkexecarecurrentlysupportedonthex86,x86_64,ppc64,ia64,ands390/s390xarchitectures.

KdumpDesign:

Howdoesitworks?
Wheneverthestandard(production)kernelcrashes,kdumpuseskexectoboot
intoacrash(capture)kernel.
Thiscrash(capture)kernelbootswithverylittleprereservedmemoryand
capturesthedumpimage.
Thestandard(production)kernelreservesasectionofmemorythatthe
crash(capture)kernelusestoboot.
Beawarethatthememoryreservedforthekdumpkernelatboottimecannot
beusedbythestandardkernel.
KexecenablesbootingthecapturekernelwithoutgoingthroughtheBIOS,so
contentsofthefirstkernel'smemoryarepreserved,whichisessentially
thekernelcrashdump.

Whatiskernelcrashdump(vmcore)?
Kernelcrashdumpcapturesthestateofkernelatthemoment
ofpanic.

Kernelcrashdumpisasnapshotofthephysicalmemoryatthe
timeofcrash.

Whoneedskernelcrashdump&why?
Kerneldevelopers:
Toidentifythecauseofkernelpanic.Ithelpsthemtofix
thebug.

Enterprisecustomer:
Topatchtheserverswithabugfixkernelpackageinorderto
avoidthedowntimeinfuture.

KdumpConfigurationfiles:
Thefollowingtwoconfigurationfilesareusedtoconfigurekdump.
/etc/sysconfig/kdump
oConfigurationfileforkexeckernel.
oUsetopassadditionalbootparameterstokexeckernel.

/etc/kdump.conf
oConfigurationfileforkdumpservice.
oUsetosetdumptaget(local/remote).
oUsetosetdefaultactionincasekdumpfailed.

Howtoconfigurekdump?
Installkexectoolspackage.
Reservememoryforkdumpkernel.
Reboot.
Setdumptarget(Local/Remote)forkdump.
Enableandstartkdumpservice.

Demo:

Step:1
Install"kexectools"package.
#dnfinstallkexectools

Verify"kexectools"package.
#rpmqkexectools
kexectools2.0.88.fc22.x86_64

Step:2
Setkernelbootparameter"crashkernel"toreservememoryforcrash/kdumpkernel.

#vi/etc/sysconfig/grub
GRUB_CMDLINE_LINUX="rd.lvm.lv=fedora/swaprd.lvm.lv=fedora/rootrhgbquiet
crashkernel=128M"

Thesizeofcrashkerneldependson;
oMachineArchitecture.
oTotalamountofinstalledsystemmemory.
o[128MB+4bitsforevery4KBpage]

Step:3
Rebuildthegrub2configurationfile.
#grub2mkconfigo/boot/grub2/grub.cfg

Rebootthesystemforchangestotakeeffect.
#reboot

Step:4
Verifythememoryreservedforcrash/kdumpkernel.

#cat/proc/cmdline
BOOT_IMAGE=/vmlinuz4.0.4301.fc22.x86_64root=/dev/mapper/fedorarootro
rd.lvm.lv=fedora/swaprd.lvm.lv=fedora/rootrhgbquietcrashkernel=128M

#cat/sys/kernel/kexec_crash_loaded
1

#cat/sys/kernel/kexec_crash_size
134217728

Step:5
Configurethedumptarget(defaultis/var/crash)
#vi/etc/kdump.conf
path/var/crash

Note:1
Currentlyonlyonedumptargetandpathcanbeconfiguredatonce.

Note:2
Thefreediskspaceonthetargetdumpdevicemustbeequaltothephysical
memoryonthesystem.

/etc/kdump.conf
path<path>
o"path"representsthefilesystempathinwhichvmcorewillbesaved.

path/var/crash

raw<partition>
oUsetosetdumptargettoraw<partition>.
raw/dev/vg/lv_kdump

<fstype><partition>
oUsetosetdumptargettoaspecificpartitionofaspecific
filesystemtype.

ext4/dev/vg/lv_kdump
ext4UUID=031383565e614ab3b58e27507ac41937

/etc/kdump.conf
nfs<nfsmount>
oUsetosetdumptargettoNFSmountpoint.

nfsserver.example.com:/export/tmp

ssh<user@server>
oUsedtosetdumptargetonremotemachine.

sshuser@server.example.com

Step:6
Setthemessagelevelanddumpleveltofilterthepagesfromvmcore.
#vi/etc/kdump.conf
path/var/crash
core_collectormakedumpfilelmessagelevel1d31

/etc/kdump.conf
core_collector<command><options>
oThisallowsyoutospecifythecommandtocopythevmcore.

core_collectormakedumpfilecmessagelevel1d31

c:Usetocompressthevmcorefile.
d:Usetosetthedump_level.
o"dump_level"isusetodecidewhichpagestobefilterfromthevmcorefile.

Step:7
Settheactiontopreformincasedumpingtointendedtargetfails.

#vi/ect/kdump.conf
path/var/crash
core_collectormakedumpfilelmessagelevel1d31
defaultreboot

Note:
Ifnodefaultactionisspecified,"reboot"isassumeddefault.

Step:8
Enablethekdumpservice.
#systemctlenablekdump

Startthekdumpservice.
#systemctlstartkdump.service
#kdumpctlstart

Verifykdumpservicestatus.
#systemctlstatuskdump.service
#kdumpctlstatus

Howtotestkdumpsetup?
Enablemagickey"SysRq"totriggerkernelpanic.
#echo1>/proc/sys/kernel/sysrq

TriggerSysRqpanicevent(c)tocrashthesystem.
#echoc>/proc/sysrqtrigger

Howtoverifykernelcrashdump(vmcore)?
Verify/var/crashdirectoryforvmcoreandvmcoredmesg.txtfile.
#tree/var/crash/
/var/crash/
127.0.0.12015.06.0817:25:32
vmcore
vmcoredmesg.txt
Openkernelcrashdump(vmcore)usingcrashutility.
#crashosreleasevmcore
#crashd1vmcore|egrep"nodename|release|machine|dump_level"

References:
https://www.kernel.org/doc/Documentation/kdump/kdump.txt
https://www.kernel.org/doc/ols/2007/ols2007v1pages167178.pdf
http://people.redhat.com/nhorman/papers/olsslides.pdf
/usr/share/doc/kexectools/kexeckdumphowto.txt

Questions?