8/9/2016

CoolSolutions:RecoveringaLostLVMVolumeDisk
NovellisnowapartofMicroFocus

Products

Services&Support

Partners

LOGIN

Communities

AboutUs

UnitedStates,English

CHANGE

HowtoBuy

RecoveringaLostLVMVolumeDisk
CoolSolutionsHome(New)

NovellCoolSolutions:AppNote
ByJasonRecord

ClassicCoolSolutions
Home

DiggThisSlashdotThis

Authors
CoolBlogs
CoolSolutionsWiki
CoolTools
GetInvolved>
OpenAudio(podcasts)

Posted:27Jul2007

Contents:
Overview
ServerConfiguration
DiskBelongingtoaVolumeGroupRemoved
CorruptedLVMMetaData
DiskPermanentlyRemoved
Conclusion

Overview
LogicalVolumeManagement(LVM)providesahighlevel,flexibleviewofaserver'sdiskstorage.Thoughrobust,problemscan
occur.Thepurposeofthisdocumentistoreviewtherecoveryprocesswhenadiskismissingordamaged,andthenapplythat
processtoplausibleexamples.Whenadiskisaccidentallyremovedordamagedinsomewaythatadverselyaffectsthelogical
volume,thegeneralrecoveryprocessis:
1.Replacethefailedormissingdisk
2.Restorethemissingdisk'sUUID
3.RestoretheLVMmetadata
4.RepairthefilesystemontheLVMdevice
Therecoveryprocesswillbedemonstratedinthreespecificcases:
1.Adiskbelongingtoalogicalvolumegroupisremovedfromtheserver
2.TheLVMmetadataisdamagedorcorrupted
3.Onediskinamultidiskvolumegrouphasbeenpermanentlyremoved
ThisarticlediscusseshowtorestoretheLVMmetadata.Thisisariskyproposition.Ifyourestoreinvalidinformation,youcan
looseallthedataontheLVMdevice.AnimportantpartofLVMrecoveryishavingbackupsofthemetadatatobeginwith,and
knowinghowit'ssupposedtolookwheneverythingisrunningsmoothly.LVMkeepsbackupandarchivecopiesofit'smeta
datain/etc/lvm/backupand/etc/lvm/archive.Backupthesedirectoriesregularly,andbefamiliarwiththeircontents.Youshould
alsomanuallybackuptheLVMmetadatawithvgcfgbackupbeforestartinganymaintenanceprojectsonyourLVMvolumes.
Ifyouareplanningonremovingadiskfromtheserverthatbelongstoavolumegroup,youshouldrefertotheLVMHOWTO
beforedoingso.

ServerConfiguration
Inallthreeexamples,aserverwithSUSELinuxEnterpriseServer10withServicePack1(SLES10SP1)willbeusedwithLVM
version2.Theexampleswilluseavolumegroupcalled"sales"withalinearlogicalvolumecalled"reports".Thelogicalvolume
andit'smountpointareshownbelow.Youwillneedtosubstituteyourmountpointsandvolumenamesasneededtomatch
yourspecificenvironment.

lslvm:~#cat/proc/partitions
majorminor#blocksname
804194304sda
81514048sda1
821052257sda2
831sda3
85248976sda5
816524288sdb
832524288sdc
848524288sdd
lslvm:~#pvcreate/dev/sda5/dev/sd[bd]
Physicalvolume"/dev/sda5"successfullycreated
Physicalvolume"/dev/sdb"successfullycreated
Physicalvolume"/dev/sdc"successfullycreated
Physicalvolume"/dev/sdd"successfullycreated
lslvm:~#vgcreatesales/dev/sda5/dev/sd[bd]
Volumegroup"sales"successfullycreated
lslvm:~#lvcreatenreportsL+1Gsales
Logicalvolume"reports"created
lslvm:~#pvscan
PV/dev/sda5VGsaleslvm2[240.00MB/240.00MBfree]
PV/dev/sdbVGsaleslvm2[508.00MB/0free]
PV/dev/sdcVGsaleslvm2[508.00MB/0free]

https://www.novell.com/coolsolutions/appnote/19386.html

1/6

8/9/2016

CoolSolutions:RecoveringaLostLVMVolumeDisk
PV/dev/sddVGsaleslvm2[508.00MB/500.00MBfree]
Total:4[1.72GB]/inuse:4[1.72GB]/innoVG:0[0]
lslvm:~#vgs
VG#PV#LV#SNAttrVSizeVFree
sales410wzn1.72G740.00M
lslvm:~#lvs
LVVGAttrLSizeOriginSnap%MoveLogCopy%
reportssaleswiao1.00G
lslvm:~#mount|grepsales
/dev/mapper/salesreportson/sales/reportstypeext3(rw)
lslvm:~#dfh/sales/reports
FilesystemSizeUsedAvailUse%Mountedon
/dev/mapper/salesreports
1008M33M925M4%/sales/reports
DiskBelongingtoaVolumeGroupRemoved
Removingadisk,belongingtoalogicalvolumegroup,fromtheservermaysoundabitstrange,butwithStorageArea
Networks(SAN)orfastpacedschedules,ithappens.
Symptom:
Thefirstthingyoumaynoticewhentheserverbootsaremessageslike:

"Couldn'tfindallphysicalvolumesforvolumegroupsales."
"Couldn'tfinddevicewithuuid'56pgEk0zLScKBcz9vJkP65DUBIhwZPSu'."
'Volumegroup"sales"notfound'
Ifyouareautomaticallymounting/dev/sales/reports,thentheserverwillfailtobootandpromptyoutologinasroottofixthe
problem.

1.Typeroot'spassword.
2.Editthe/etc/fstabfile.
3.Commentoutthelinewith/dev/sales/report
4.Reboot
TheLVMsymptomisamissingsalesvolumegroup.Typingcat/proc/partitionsconfirmstheserverismissingoneof
it'sdisks.

lslvm:~#cat/proc/partitions
majorminor#blocksname
804194304sda
81514048sda1
821052257sda2
831sda3
85248976sda5
816524288sdb
832524288sdc
lslvm:~#pvscan

https://www.novell.com/coolsolutions/appnote/19386.html

2/6

8/9/2016

CoolSolutions:RecoveringaLostLVMVolumeDisk
Couldn'tfinddevicewithuuid'56ogEkOzLScKBcz9vJkP65DUBIhwZPSu'.
Couldn'tfinddevicewithuuid'56ogEkOzLScKBcz9vJkP65DUBIhwZPSu'.
Couldn'tfinddevicewithuuid'56ogEkOzLScKBcz9vJkP65DUBIhwZPSu'.
Couldn'tfinddevicewithuuid'56ogEkOzLScKBcz9vJkP65DUBIhwZPSu'.
Couldn'tfinddevicewithuuid'56ogEkOzLScKBcz9vJkP65DUBIhwZPSu'.
Couldn'tfinddevicewithuuid'56ogEkOzLScKBcz9vJkP65DUBIhwZPSu'.
PV/dev/sda5VGsaleslvm2[240.00MB/240.00MBfree]
PV/dev/sdbVGsaleslvm2[508.00MB/0free]
PVunknowndeviceVGsaleslvm2[508.00MB/0free]
PV/dev/sdcVGsaleslvm2[508.00MB/500.00MBfree]
Total:4[1.72GB]/inuse:4[1.72GB]/innoVG:0[0]
Solution:
1.Fortunately,themetadataandfilesystemonthediskthatwas/dev/sdcareintact.
2.Sotherecoveryistojustputthediskback.
3.Reboottheserver.
4.The/etc/init.d/boot.lvmstartscriptwillscanandactivatethevolumegroupatboottime.
5.Don'tforgettouncommentthe/dev/sales/reportsdeviceinthe/etc/fstabfile.
Ifthisproceduredoesnotwork,thenyoumayhavecorruptLVMmetadata.

CorruptedLVMMetaData
TheLVMmetadatadoesnotgetcorruptedveryoftenbutwhenitdoes,thefilesystemontheLVMlogicalvolumeshouldalso
beconsideredunstable.ThegoalistorecovertheLVMvolume,andthencheckfilesystemintegrity.
Symptom1:
Attemptingtoactivatethevolumegroupgivesthefollowing:

lslvm:~#vgchangeaysales
/dev/sdc:Checksumerror
/dev/sdc:Checksumerror
/dev/sdc:Checksumerror
/dev/sdc:Checksumerror
/dev/sdc:Checksumerror
/dev/sdc:Checksumerror
/dev/sdc:Checksumerror
/dev/sdc:Checksumerror
/dev/sdc:Checksumerror
/dev/sdc:Checksumerror
/dev/sdc:Checksumerror
/dev/sdc:Checksumerror
/dev/sdc:Checksumerror
Couldn'treadvolumegroupmetadata.
Volumegroupsalesmetadataisinconsistent
Volumegroupforuuidnotfound:m4Cg2vkBVSGe1qSMNDf63v3fDHqN4uEkmWoTq5TpHpRQwmnAGD18r44OshLdHj05
0logicalvolume(s)involumegroup"sales"nowactive

Thissymptomistheresultofaminorchangeinthemetadata.Infact,onlythreebyteswereoverwritten.Sinceonlyaportionof
themetadatawasdamaged,LVMcancompareit'sinternalchecksumagainstthemetadataonthedeviceandknowit's
wrong.ThereisenoughmetadataforLVMtoknowthatthe"sales"volumegroupanddevicesexit,butareunreadable.

lslvm:~#pvscan
/dev/sdc:Checksumerror
/dev/sdc:Checksumerror
/dev/sdc:Checksumerror
/dev/sdc:Checksumerror
/dev/sdc:Checksumerror
/dev/sdc:Checksumerror
/dev/sdc:Checksumerror
/dev/sdc:Checksumerror
PV/dev/sda5VGsaleslvm2[240.00MB/240.00MBfree]
PV/dev/sdbVGsaleslvm2[508.00MB/0free]
PV/dev/sdcVGsaleslvm2[508.00MB/0free]
PV/dev/sddVGsaleslvm2[508.00MB/500.00MBfree]
Total:4[1.72GB]/inuse:4[1.72GB]/innoVG:0[0]
Noticepvscanshowsalldevicespresentandassociatedwiththesalesvolumegroup.It'snotthedeviceUUIDthatisnot
found,butthevolumegroupUUID.
Solution1:
1.Sincethediskwasneverremoved,leaveitasis.
2.TherewerenodeviceUUIDerrors,sodon'tattempttorestoretheUUIDs.
3.ThisisagoodcandidatetojusttryrestoringtheLVMmetadata.

lslvm:~#vgcfgrestoresales
/dev/sdc:Checksumerror
/dev/sdc:Checksumerror
Restoredvolumegroupsales
lslvm:~#vgchangeaysales

https://www.novell.com/coolsolutions/appnote/19386.html

3/6

8/9/2016

CoolSolutions:RecoveringaLostLVMVolumeDisk
1logicalvolume(s)involumegroup"sales"nowactive
lslvm:~#pvscan
PV/dev/sda5VGsaleslvm2[240.00MB/240.00MBfree]
PV/dev/sdbVGsaleslvm2[508.00MB/0free]
PV/dev/sdcVGsaleslvm2[508.00MB/0free]
PV/dev/sddVGsaleslvm2[508.00MB/500.00MBfree]
Total:4[1.72GB]/inuse:4[1.72GB]/innoVG:0[0]
4.Runafilesystemcheckon/dev/sales/reports.

FollowUs

RequestaCall

18883214272

Print

Feedback

lslvm:~#e2fsck/dev/sales/reports
e2fsck1.38(30Jun2005)
/dev/sales/reports:clean,961/131072files,257431/262144blocks
lslvm:~#mount/dev/sales/reports/sales/reports/
lslvm:~#dfh/sales/reports/
FilesystemSizeUsedAvailUse%Mountedon
/dev/mapper/salesreports
1008M990M0100%/sales/reports
Symptom2:
MinordamagetotheLVMmetadataiseasilyfixedwithvgcfgrestore.Ifthemetadataisgone,orseverelydamaged,thenLVM
willconsiderthatdiskasan"unknowndevice."Ifthevolumegroupcontainsonlyonedisk,thenthevolumegroupandit's
logicalvolumeswillsimplybegone.Inthiscasethesymptomisthesameasifthediskwasaccidentallyremoved,withthe
exceptionofthedevicename.Since/dev/sdcwasnotactuallyremovedfromtheserver,thedevicesarestilllabeledathrough
d.

lslvm:~#pvscan
Couldn'tfinddevicewithuuid'56ogEkOzLScKBcz9vJkP65DUBIhwZPSu'.
Couldn'tfinddevicewithuuid'56ogEkOzLScKBcz9vJkP65DUBIhwZPSu'.
Couldn'tfinddevicewithuuid'56ogEkOzLScKBcz9vJkP65DUBIhwZPSu'.
Couldn'tfinddevicewithuuid'56ogEkOzLScKBcz9vJkP65DUBIhwZPSu'.
Couldn'tfinddevicewithuuid'56ogEkOzLScKBcz9vJkP65DUBIhwZPSu'.
Couldn'tfinddevicewithuuid'56ogEkOzLScKBcz9vJkP65DUBIhwZPSu'.
PV/dev/sda5VGsaleslvm2[240.00MB/240.00MBfree]
PV/dev/sdbVGsaleslvm2[508.00MB/0free]
PVunknowndeviceVGsaleslvm2[508.00MB/0free]
PV/dev/sddVGsaleslvm2[508.00MB/500.00MBfree]
Total:4[1.72GB]/inuse:4[1.72GB]/innoVG:0[0]
Solution2:
1.First,replacethedisk.Mostlikelythediskisalreadythere,justdamaged.
2.SincetheUUIDon/dev/sdcisnotthere,avgcfgrestorewillnotwork.

lslvm:~#vgcfgrestoresales
Couldn'tfinddevicewithuuid'56ogEkOzLScKBcz9vJkP65DUBIhwZPSu'.
Couldn'tfindallphysicalvolumesforvolumegroupsales.
Restorefailed.
3.Comparingtheoutputofcat/proc/partitionsandpvscanshowsthemissingdeviceis/dev/sdc,and
pvscanshowswhichUUIDitneedsforthatdevice.So,copyandpastetheUUIDthatpvscanshowsfor/dev/sdc.

lslvm:~#pvcreateuuid56ogEkOzLScKBcz9vJkP65DUBIhwZPSu/dev/sdc
Physicalvolume"/dev/sdc"successfullycreated
4.RestoretheLVMmetadata

lslvm:~#vgcfgrestoresales
Restoredvolumegroupsales
lslvm:~#vgscan
Readingallphysicalvolumes.Thismaytakeawhile...
Foundvolumegroup"sales"usingmetadatatypelvm2
lslvm:~#vgchangeaysales
1logicalvolume(s)involumegroup"sales"nowactive
5.Runafilesystemcheckon/dev/sales/reports.

lslvm:~#e2fsck/dev/sales/reports
e2fsck1.38(30Jun2005)
/dev/sales/reports:clean,961/131072files,257431/262144blocks
lslvm:~#mount/dev/sales/reports/sales/reports/

https://www.novell.com/coolsolutions/appnote/19386.html

4/6

8/9/2016

CoolSolutions:RecoveringaLostLVMVolumeDisk
lslvm:~#dfh/sales/reports
FilesystemSizeUsedAvailUse%Mountedon
/dev/mapper/salesreports
1008M990M0100%/sales/reports

DiskPermanentlyRemoved
Thisisthemostseverecase.Obviouslyifthediskisgoneandunrecoverable,thedataonthatdiskislikewiseunrecoverable.
Thisisagreattimetofeelgoodknowingyouhaveasolidbackuptorelyon.However,ifthegoodfeelingsaregone,andthere
isnobackup,howdoyourecoverasmuchdataaspossiblefromtheremainingdisksinthevolumegroup?Noattemptwillbe
madetoaddressthedataontheunrecoverablediskthistopicwillbelefttothedatarecoveryexperts.
FollowUs
RequestaCall
18883214272
Print

Feedback

Symptom:
ThesymptomwillbethesameasSymptom2intheCorruptedLVMMetaDatasectionabove.Youwillseeerrorsaboutan
"unknowndevice"andmissingdevicewithUUID.
Solution:
1.Addareplacementdisktotheserver.Makesurethediskisempty.
2.CreatetheLVMmetadataonthenewdiskusingtheolddisk'sUUIDthatpvscandisplays.

lslvm:~#pvcreateuuid56ogEkOzLScKBcz9vJkP65DUBIhwZPSu/dev/sdc
Physicalvolume"/dev/sdc"successfullycreated
3.RestorethebackupcopyoftheLVMmetadataforthesalesvolumegroup.

lslvm:~#vgcfgrestoresales
Restoredvolumegroupsales
lslvm:~#vgscan
Readingallphysicalvolumes.Thismaytakeawhile...
Foundvolumegroup"sales"usingmetadatatypelvm2
lslvm:~#vgchangeaysales
1logicalvolume(s)involumegroup"sales"nowactive
4.Runafilesystemchecktorebuildthefilesystem.

lslvm:~#e2fscky/dev/sales/reports
e2fsck1.38(30Jun2005)
snip
Freeinodescountwrongforgroup#5(16258,counted=16384).
Fix?yes
Freeinodescountwrong(130111,counted=130237).
Fix?yes
/dev/sales/reports:*****FILESYSTEMWASMODIFIED*****
/dev/sales/reports:835/131072files(5.7%noncontiguous),137213/262144blocks
5.Mountthefilesystemandrecoverasmuchdataaspossible.
6.NOTE:Ifthemissingdiskcontainsthebeginningofthefilesystem,thenthefilesystem'ssuperblockwillbemissing.
Youwillneedtorebuildoruseanalternatesuperblock.Restoringafilesystemsuperblockisoutsidethescopeofthis
article,pleaserefertoyourfilesystem'sdocumentation.

Conclusion
LVMbydefaultkeepsbackupcopiesofit'smetadataforallLVMdevices.Thesebackupfilesarestoredin/etc/lvm/backupand
/etc/lvm/archive.Ifadiskisremovedorthemetadatagetsdamagedinsomeway,itcanbeeasilyrestored,ifyouhave
backupsofthemetadata.ThisiswhyitishighlyrecommendedtoneverturnoffLVM'sautobackupfeature.Evenifadiskis
permanentlyremovedfromthevolumegroup,itcanbereconstructed,andoftentimestheremainingdataonthefilesystem
recovered.

AdditionalReferences
SLES10InstallationandAdministrationGuide,7.1LVMConfiguration
http://www.novell.com/documentation/sles10/sles_admin/data/sec_yast2_system_lvm.html
LVMHOWTO
http://www.tldp.org/HOWTO/LVMHOWTO/index.html

Likewhatyousee?
Signupforourweekly
newsletter.

Wanttocontribute?
Itcouldearnyouanano!
Learnmore.

LikeWikis?
JointheCoolSolutions
Wiki.

Interested?

Requestasales
call

NovellCoolSolutions(corporatewebcommunities)areproducedbyWebWiseSolutions.www.webwiseone.com

https://www.novell.com/coolsolutions/appnote/19386.html

5/6

8/9/2016
2016MicroFocus Careers

CoolSolutions:RecoveringaLostLVMVolumeDisk
Legal

FollowUs

https://www.novell.com/coolsolutions/appnote/19386.html

RequestaCall

18883214272

Print

Feedback

6/6