HowYAFFSWorks

CharlesManning
20072010
TableofContents
1Purpose...................................................................................................................................................2
2WhatIsYAFFS?....................................................................................................................................2
3Designandcodingstrategies..................................................................................................................2
4Terminology:YAFFSvsYAFFS1vsYAFFS2....................................................................................3
5Objects...................................................................................................................................................3
6TheYAFFSNANDModel....................................................................................................................4
7HowYAFFS1StoresFiles.....................................................................................................................4
8Garbagecollection.................................................................................................................................7
9YAFFS1Serialnumbers........................................................................................................................8
10YAFFS2NANDmodel........................................................................................................................9
10.1.YAFFS2Extendedtags...............................................................................................................11
11BadblockhandlingNANDerrorhandling........................................................................................11
12RAMstructures..................................................................................................................................12
12.1.yaffs_Object................................................................................................................................13
12.2.ObjectIdlookup..........................................................................................................................13
12.3.Directorystructure.......................................................................................................................14
12.4.Hardlinks....................................................................................................................................15
12.5.Symboliclinkandspecialobject.................................................................................................16
12.6.Fileobject....................................................................................................................................16
12.6.1Tnodetree...........................................................................................................................16
13Howvariousmechanismswork.........................................................................................................17
13.1.BlockandChunkManagement...................................................................................................17
13.1.1BlockStates........................................................................................................................17
13.1.2BlockandChunkAllocation..............................................................................................19
13.1.3Awordaboutwearleveling................................................................................................20
13.1.4yaffs_Tnodeandyaffs_ObjectManagement......................................................................20
13.2.InternalCache..............................................................................................................................21
13.3.Scanning......................................................................................................................................22
13.3.1YAFFS1Scanning..............................................................................................................22
13.3.2YAFFS2Scanning..............................................................................................................22
13.4.Checkpoint...................................................................................................................................23
13.5.ExtendedTagsandPackedTags.................................................................................................24
13.6.Inbandtags..................................................................................................................................24
13.7.SoftDeletion................................................................................................................................25

Page1/25

1 Purpose
Thepurposeofthisdocumentistogiveareasonableexplanationofmostofthecoremechanismsthatmakeyaffswork.As
ofthetimeofwritingthis,theyaffs2codebaseincludesapproximately19,000linesofcodemakingaverydetailed
discussionimpractical.
Thisdocumentshouldserveasafirstportofcallforunderstandingyaffs.Furtherdetailedquestionsanddiscussionsare
welcomedontheyaffsdiscussionlist.
Thisdocumentisupdatedperiodically.Yaffsmechanismdesignchangesfromtimetotimeandthisdocumentwillnot
alwaysreflectthemostuptodatecode.

2 WhatIsYAFFS?
YAFFSstandsforYetAnotherFlashFileSystem,atermcoinedbyCharlesManningin2001whensuggestingthatthe
alreadyclutteredflashfilesystemspacecoulddowithyetanotherofferingthistimeaflashfilesystemdesignedfromthe
grounduptoworkwithNANDflash.
ThefirstreleaseofYAFFSwasdevelopedduringtheendof2001andthebeginningof2002.Bymid2002,YAFFSwas
beingtrialledinvariousproducts.InMay2002,YAFFSwasannouncedandafewmoreprojectsstartedplayingwithit.In
June,portingtootherOSsstarted.InSeptember2002,YAFFSwasannouncedonlinuxdevices.comwhichstartedafar
wideruptakeofYAFFS.
YAFFS1,theoriginalversionofYAFFS,supported512bytepageNANDdeviceswithaSmartMedialikememorylayout.
YAFFS2isafollowuptoYAFFS1extendingYAFFS1tosupportlargeranddifferentdeviceswithdifferentconstraints.
YAFFSishighlyportableandhasbeenusedinmanydifferentproductsandapplicationsasvariedassewingmachines,
pointofsale,phonesandaerospace.YAFFShasbeenusedwithmultipledifferentoperatingsystems.Nativesupportis
availableforLinux,WindowsCEandeCOSwhileyaffsDirectInterfaceprovidesalayerthatcanbeusedinother
applicationssuchasRTOSs.YAFFShasbeenusedwithmultipledifferentCPUsandcompilers.
YAFFShasalsobeenusedasaNORfilesystemandevenasaRAMfilesystem.

3 Designandcodingstrategies
YAFFSisdesignedtoworkinmultipleenvironmentswhichdrivestheneedforportability.Portabilityalsoimprovesthe
debuggingandtestingsinceitiseasiertodevelopanddebugcodeinanapplicationenvironmentthanwithinanOSkernel.
ThisgenerallyallowsYAFFStoprogressfasterwithlessprogrammingresources.Primarystrategiesthatimprove
portabilityinclude:

NoOSspecificfeaturesusedinmaincodebody.

Nocompilerspecificfeaturesusedinthemaincodebody.

AbstracttypesandfunctionsusedtoallowUnicodeorASCIIoperation.

Page2/25

Simplicityisanotherkeygoal.Complexityislimitedtothosesituationsthatprovidesignificantimprovementsto
performanceorutility.Simplicityimprovesrobustnessandtheeaseofintegrationanddevelopment.Primarystrategiesthat
improvesimplicityare:

Singlethreadedmodel.YAFFSislockedonaperpartitionbasisatahighlevel.Thisissimplerthantracking
lowerlevellocking.YaffsDirectInterfaceusesasinglelockforallpartitions.

Logstructuremakesforasimplergarbagecollectorandallocationmethodology.

Abstractionsthatbuildlayeredcodewhichiseasiertounderstandanddebug.

4 Terminology:YAFFSvsYAFFS1vsYAFFS2
BeforeweembarkonourquesttounderstandwhatYAFFSis,aterminologyexplanationisinorder.
TherearetwoversionsofYAFFScode:theoriginalyaffscodebaseandthecurrentyaffs2codebase.Theyaffs2codebase
supportsfunctionalityprovidedbytheyaffscodebaseaswellasbeingextendedwithnewfunctionalitytoprovideextra
modesofoperation,mostimportantlythoserequiredtoworkwithlargerandmoremodernNANDparts.Theyaffs2code
basesupportstheyaffs1modeofoperationthroughabackwardcompatibilitymode.
Unlessexplicitlymentioned,thistextreferstotheyaffs2codebase.
ThistextusesthreetermsYAFFS,YAFFS1andYAFFS2dependingonvariousmodesofoperation.
YAFFS1isasimplermodeofoperationthatusesdeletionmarkerstotrackstate.
YAFFS2isamorecomplexmodeofoperatingthatwasdevelopedtosupportlargerflashtypesthatcannotuseddeletion

markers.

YAFFSmeansoperationscommontoboth.

YAFFS1originallyonlyworkedwith512bytepagedevicesbuthasbeenextendedtosupportlargerflashpagessuchas
IntelM18flashwith1kpages.
YAFFS2wasoriginallydesignedfor1korlargerpagesizesbutcanbeusedwithsmallerpagesbyusinginbandtags.
SincetheYAFFS1modeofoperationissimplertounderstandthantheYAFFS2modeofoperation,YAFFS1willbe
describedfirstandreadersinterestedinYAFFS2operationshouldfirstfullyacquaintthemselveswiththeYAFFS1modeof
operation.Itissuggestedthatthisdocumentbereadsequentiallyratherthanbrowsingit.

5 Objects
InYAFFSspeak,anObjectisanythingthatisstoredinthefilesystem.Theseare:

Regulardatafiles

Directories

Hardlinks

Symboliclinks

Page3/25

Specialobjects(pipes,devicesetc).

AllobjectsareidentifiedbyauniqueintegerobjectId.
InthestandardPOSIXmodel,asusedbyLinux,thereareinodesanddirectoryentries(dentries).
Aninodemaybearegularfile,directory,orspecialfile.
Directoryentriesprovidethenamingmechanismthatisusedtolocateinodes.
UnderPOSIX,eachinodemayhavezero,oneormanydentries:

Onedentryperinodeiscommonlyunderstood.

Multipledentriesperinodeallowthesameinodetobeaccessedviamultiplenames.Thisisachievedthroughhard
links.

Zerodentriesperinodeislessoftenunderstood.Thishappenswhenaninodeisunlinkedwhilethereisstilla
handleopentotheinode.Theunlinkingdeletesthedentrybuttheinodestillexists.Assoonasthehandleisclosed
theinodeisdeletedtoo.

6 TheYAFFSNANDModel
ThememoryinNANDflashisarrangedinpages.Apageistheunitofallocationandprogramming.InYAFFS,theunitof
allocationisthechunk.TypicallyachunkwillbethesameasaNANDpage,butthereisflexibilitytousechunkswhich
maptomultiplepages(eg.AsystemmayhavetwoNANDchipsinparallelrequiring2x512=1024bytechunks).This
distinctiongivesalotofflexibilityinhowthesystemcanbeconfigured.Inthistextthetermpageandchunkare
synonymousunlessstatedotherwise.
Many,typically32to128butasmanyasafewhundred,chunksformablock.Ablockistheunitoferasure.
NANDflashmaybeshippedwithbadblocksandfurtherblocksmaygobadduringtheoperationofthedevice.Thus,
YAFFSisawareofbadblocksandneedstobeabletodetectandmarkbadblocks.
NANDflashalsotypicallyrequirestheuseofsomesortoferrordetectionandcorrectioncode(ECC).YAFFScaneither
useexistingECClogicorprovideitsown.

7 HowYAFFS1StoresFiles
YAFFS1hasamodifiedlogstructure,whileYAFFS2(seebelow)hasatruelogstructure.Atruelogstructuredfilesystem
onlyeverwritessequentially.YAFFS1usesdeletionmarkers,whichbreaksthesequentialwriterule.YAFFS2doesnotuse
deletionmarkersandisthusatruelogstructuredfilesystem.
Insteadofwritingdatainlocationsspecifictothefiles,thefilesystemdataiswrittenintheformofasequentiallog,The
entriesinthelogareallonechunkinsizeandcanholdoneoftwotypesofchunk:

Datachunk:Achunkholdingregulardatafilecontents.

ObjectHeader:Adescriptorforanobject(directory,regulardatafile,hardlink,softlink,specialdescriptor,...).
Thisholdsdetailssuchastheidentifierfortheparentdirectory,objectname,etc.

Page4/25

Eachchunkhastagsassociatedwithit.Thetagscomprisethefollowingimportantfields(othersbeingignoredfornow):

ObjectId:Identifieswhichobjectthechunkbelongsto.

ChunkId:Identifieswhereinthefilethischunkbelongs.AchunkIdofzerosignifiesthatthischunkcontainsan
objectHeader.ChunkId==1signifiesthefirstchunkinthefile(ie.Atfileoffset0),chunkId==2isthenextchunk
andsoon.

DeletionMarker:(YAFFS1only)Showsthatthischunkisnolongerinuse.

ByteCount:Numberofbytesofdataifthisisadatachunk.

SerialNumber:(YAFFS1only)SerialnumberusedtodifferentiatechunkswiththesameobjectIdandchunkId.

Nowlet'sbringthatalltogetherinanexplanation.ThisisasimplifiedexplanationandskipsoverissuessuchasSoft
Deletion(mentionedlaterinthedocument).Forexplanationpurposes,weshallconsiderusageoffictitiousNANDtypethat
has4chunksperblock,startingwithanempty(erased)flash.
Firstwe'llcreateafile.

Block

Chunk ObjectId

ChunkId

Deletion

Comment

Live

ObjectHeaderforthisfile(length0)

500

Nextwewriteafewchunksofdatatothefile.

Block

Chunk ObjectId

ChunkId

Deletion

Comment

500

Live

ObjectHeaderforthisfile(length0)

500

Live

Firstchunkofdata

500

Live

Secondchunkofdata

500

Live

Thirdchunkofdata

Nextweclosethefile.Thiswritersanewobjectheaderforthefile.Noticehowthepreviousobjectheaderisdeleted.

Block

Chunk ObjectId

ChunkId

Deletion

Comment

500

Deleted

Obsoletedobjectheader(length0)

500

Live

Firstchunkofdata

500

Live

Secondchunkofdata

500

Live

Thirdchunkofdata

Page5/25

500

Live

Newobjectheader(lengthn)

Let'snowopenthefileforread/write,overwritepartofthefirstchunkinthefileandclosethefile.Thereplaceddataand
objectheaderchunksbecomedeleted.

Block

Chunk ObjectId

ChunkId

Deletion

Comment

500

Deleted

Obsoleteobjectheader.

500

Deleted

Obsoletefirstchunkofdata

500

Live

Secondchunkofdata

500

Live

Thirdchunkofdata

500

Deleted

Obsoleteobjectheader

500

Live

Newfirstchunkoffile

500

Live

Newobjectheader

Nowlet'sresizethefiletozerobyopeningthefilewithO_TRUNCandclosingthefile.Thiswritesanewobjectheader
withlength0andthecontentsareprunedmakingthedatachunksdeleted.

Block

Chunk ObjectId

ChunkId

Deletion

Comment

500

Deleted

Obsoleteobjectheader.

500

Deleted

Obsoletefirstchunkofdata

500

Deleted

Secondchunkofdata

500

Deleted

Thirdchunkofdata

500

Deleted

Obsoleteobjectheader

500

Deleted

Deletedfirstchunkoffile

500

Delted

Obsoletedobjectheader

500

Live

Newobjectheader(length0).

Notehowallthepagesinblock1arenowmarkedasdeleted.Thismeansthatblock1nolongercontainsusefulinformation
sowecannoweraseblock1andreusethespace.
Wewillnowrenamethefile.Todothiswewriteanewobjectheaderforthefile

Page6/25

Block

Chunk ObjectId

ChunkId

Deleted

Comment

Erased

Erased

Erased

Erased

500

Deleted

Obsoleteobjectheader

500

Deleted

Deletedfirstchunkoffile

500

Deleted

Obsoletedobjectheader

500

Deleted

Obsoleteobjectheader

500

Live

Newobjectheadershowingnewname

Block2nowcontainsonlydeletedchunksandcanbeerasedandreused.
Notehowthetagstellus:
whichchunksbelongtowhichobject,
thepositionofthechunkswithinafile,
whichchunksarecurrent

WiththatinformationwecanrecreatethestateofthefilesregardlessoftheplacementofthechunksinNAND.Thismeans
thatshouldthesystemfail(crash,powerfail,...)atanypointduringthesequencesabove,wearestillabletorecoverthefile
systemstateuptothatpoint.
Notethattherearenofileallocationtablesorsimilarstructuresstoredintheflash.Thisreducestheamountofwritingand
erasing(improvingwriteperformance)whilealsoimprovingrobustness.Corruptionoffileallocationtablesandsimilar
structuresisacommonfailuremechanismforembeddedfilesystems.Thelackofsuchstructuresmakesyaffsmorerobust.
Or,toputitanotherway,youcan'tcorruptwhatyoudon'tstore.

8 Garbagecollection
Aswehaveshownsofar,whenablockismadeuponlyofdeletedchunks,thatblockcanbeerasedandreused.However,
considerwhatwillhappenifwehavemanyblockswhicheachhaveafewcurrentchunksinthem.Clearlywecannoterase
anyofthecurrentblocksorwewilldestroydata.Whatweneedtodoiscopytheusefuldatachunksoffablock,deleting
theoriginalsandallowingtheblocktobeerasedandreused.Thisprocessisreferredtoasgarbagecollection.
TheYAFFSgarbagecollectionprocessworksasfollows:
1.

Findablockworthcollecting(accordingthetheheuristicsbelow).Ifnonefound,thenexit.

2.

Iteratethroughthechunksintheblock.Ifthechunkisinusethencreateanewcopyanddeletetheoriginal.Fixup
theRAMdatastructurestoreflectthechange.

Page7/25

Onceallchunksinthisblockhavebeendeleted,theblockcanbeerasedreadyforreuse.
Notethatalthoughstep2deletestheoriginalchunkwhenitgetscopiedthereisnotmuchpointinactuallyprogrammingthe
deletionmarkerinflashsincewe'reabouttoerasetheblockanyway.Thusyaffsdoesnotprogramthedeletionmarkerin
thesecases.Ifpowerislostpartwaythroughtheoperationwecanstilltellthedifferencebetweentheoriginalandthecopy
bylookingattheserialnumber(notthatitreallymattersanywaysincethetwohavethesamecontentssoeithercanbe
used).
Theheuristicstodeterminewhetherablockisworthcollectingareasfollows:
1.

Iftherearemanyerasedblocksavailable,thenYAFFSdoesnotneedtoworkhard,butwillinsteadonlyattemptto
performgarbagecollectionofblockswithveryfewchunksinuse.Thisistermedpassivegarbagecollection.

2.

If,however,thereareveryfewerasedblocks,YAFFSwillworkmuchhardertorecovermorespaceandwill
garbagecollectblockswithmorechunksinuse.Thisistermedaggressivegarbagecollection.

OK,thepassive/aggressivenamesareabitsilly,buttheywerecontrivedaroundmidnight...givemeabreak!
Ifgarbagecollectionisaggressive,thewholeblockiscollectedinasinglegarbagecollectioncycle.Ifthecollectionis
passivethenthenumberofcopiesisreducedthusspreadingtheeffortovermanygarbagecollectioncycles.Thisisdoneto
reducegarbagecollectionloadandimproveresponsiveness.
Therationalebehindtheaboveheuristicsistodelaygarbagecollectionwhenpossibletoreducetheamountofcollection
thatneedstobeperformed,thusincreasingaveragesystemperformance.Yetthereisaconflictinggoaloftryingtospread
thegarbagecollectionsothatitdoesnotallhappenatthesamecausingfluctuationsinfilesystemthroughput.These
conflictinggoalsmakegarbagetuningquitechallenging.
Allflashfilesystemsneedsomesortofgarbagecollectortoreclaimspace.Thegarbagecollectorisoftenadetermining
factorintheperformanceoftheflashfilesystem.Manyfilesystemshavetodoasignificantamountofeffortinasingle
collectionsweep.YAFFSonlydealswithoneblockatatimewhichconstrainstheamountofeffortinagarbagecollection
cycle,thusreducingthestalltime.
TheYAFFSgarbagecollectionalgorithmisunderreviewwiththegoalofreducingstalltimeandincreasingthroughput.

9 YAFFS1Serialnumbers
InYAFFS1,eachchunkismarkedwitha2bitserialnumberthatisincrementedeverytimeachunkwiththesametags
valueisreplaced(eg.Whenachunkisoverwrittenorwhencopiedbygarbagecollection).
Nowlet'ssupposefirstthatwedidnothaveaserialnumber.

Page8/25

Let'sconsiderarenamewhereanobjectheaderchunkmustbereplacedwithanewonewiththenewname.YAFFScould
dothefollowing:
Deletetheoldchunk.
Writethenewchunk.

Ifthesystemfailsafterthedelete,thesystemcouldendupwithnochunksmatchingtherelevanttagsvalues.Thiscould
causefilestobelost(endupinlost+found).Thus,wehavetowritethenewchunkbeforedeletingtheoldchunk.
Butnowconsiderwhatmighthappenifwedidthis:
Writenewchunk.
Deleteoldchunk.

Ifthesystemfailedafterthewritewenowhavetwochunksthatmatchthetags.Howcanwetellwhichoneiscurrent?
Thecurrentchunkisdeterminedbyinspectingtheserialnumber.Sincetheserialnumberisonlyincrementedbyone
beforetheoldchunkisdeleted,a2bitserialnumberissufficient.Validpairsare:
Old

New

00

01

01

10

10

11

11

00

Thus,byinspectingtheserialnumberattachedtotheoldandnewtagsthecurrentchunkcanbedeterminedandtheold
chunkcanbedeleted.

10 YAFFS2NANDmodel
YAFFS2isanextensionofYAFFSdesignedtofulfillanewsetofobjectivestoworkwithnewerNANDtypesincluding:

Zerooverwrites.YAFFS1onlyeveroverwritesasinglebyteinthespareareaofthechunkstosetthedeletion
marker.MoremodernNANDdevicesarelesstolerantofoverwritesandYAFFS2performsnooverwritesatall.

Sequentialchunkwritingwithinablock.MoremodernNANDreliabilityspecificationstendtoassumesequential
writing.SinceYAFFS2doesnotwritedeletionmarkers,thewritingofchunkswithinablockisstrictlysequential.

SinceYAFFS2performslesswrites,thereispotentialforYAFFS2tohaveimprovedwriteperformance.
YAFFS2cannotuseadeletionmarkerbecausethatwouldviolatethezerooverwritesmandateandalternatemechanisms

Page9/25

areprovidedtodeterminewhichchunksarecurrentandwhicharedeleted.Thesemechanismsare:

SequenceNumber:Aseachblockisallocated,thefilesystem'ssequencenumberisincrementedandeachchunk
intheblockismarkedwiththatsequencenumber.Thesequencenumberthusprovidesawayoforganisingthelog
inchronologicalorder.

ShrinkHeaderMarker:Theshrinkheadermarkerisusedtomarkobjectheadersthatarewrittentoshrinkthesize
ofadatafile.Thisisexplainedmoreintheaccompanyingtext.

Note:TheYAFFS2sequencenumberisnotthesameastheYAFFS1serialnumber!
Note:Westillrefertochunkdeletioninyaffs2.ThechunksaremarkedasdeletedintheRAMdatastructuresusedby
garbagecollectionetc,butthereisnodeletionmarkerwrittentoflash.
ThesequencenumberallowsthechronologicalordertobedeterminedwhichallowsYAFFStodeterminethesequenceof
eventsandrestorethefilesystemstate.Thisisachievedbyscanningbackwardschronologically(ie.Fromthehighest
sequencenumberbackwardstothelowestsequencenumber).Thus:

Sincewe'rescanningbackwards,themostrecentlywrittenandthuscurrentchunkmatchingan
objectId:chunkIdpairwillbeencounteredfirstandallsubsequentmatchingchunksmustbeobsoleteandtreatedas
deleted.

Thefilelengthinobjectheadersareusedtotrimbackresizedfiles.Ifanobjectheaderisencountered,then
subsequentdatachunksthatliebeyondthatfilelengthareclearlyobsoleteandtreatedasdeleted.Notethatboth
currentandobsoleteobjectheadersizesmustbeusedtoobtainanaccuratereconstruction.

Explainingthepurposeoftheshrinkheadermarkerisabitconvoluted.Thepurposeofashrinkheadermarkeristoshow
thatthisobjectheaderindicatesthatafilehasshrunkeninsize,andtopreventthoseobjectheadersfrombeingdeletedby
garbagecollection.Tounderstandthis,itiseasiesttoconsiderwhatwouldhappeniftheshrinkheadermarkersdidnot
exist.
Forsimplificationwe'llassumenogarbagecollectionhappensduringthispartoftheexercise.Itissafetomakethat
assumptionsinceitdoesnotalterthereasoning.
Considerafilethathasbeenthroughthefollowing:
h=open(foo,O_CREAT|O_RDWR,S_IREAD|S_IWRITE);/*createfile/
write(h,data,51024*1024);/*write5MBofdata/
truncate(h,1024*1024);/*truncateto1MB*/
lseek(h,2*1024*1024,SEEK_SET);/*setthefileaccesspositionto2MB*/
write(h,data,1024*1024);/*write1MBofdata*/
close(h);
Thefilewillnowbe3MBinlength,buttherewillbeaholebetween1MBand2MB.AccordingtoPOSIX(andtoaid
securityetc)thatholeshouldalwaysreadbackaszeros.
YAFFS2wouldrepresentthatinNANDwiththefollowingsequenceofchunks:
1.

Objectheaderforfilecreation(filelength=0)

2.

5MBworthofdatachunks(0to5MB)

Page10/25

3.

Objectheaderfortruncation(filelength=1MB)

4.

1MBworthofdatachunks(2MBto3MB)

5.

Objectheaderforclose(filelength=3MB)

Atthisstage,onlythethefollowingdatachunksarecurrent:

Thefirst1MBofdatacreatedinstep2above.

The1MBofdatacreatedinstep4above.

Theobjectheadercreatedinstep5above.

ButYAFFSmustrememberthetruncationthathappenedinstep3,otherwiseYAFFSwillforgetthatthereisahole,
whichwouldcausetheolddatatobevisibleagain.Thus,ifyaffscreatesaholeitfirstwritesashrinkheadertoindicatethe
startoftheholeandaregularobjectheadertomarktheendofthehole.
Theshrinkheadermodifiesthebehaviorofthegarbagecollectortoensurethattheseshrinkheadersarenoteraseduntilitis
safetodoso.Thisunfortunatelyhasanegativeimpactongarbagecollectioninsystemswhichmakeextensiveuseoffiles
withholes.
Shrinkheadersarealsousedtoindicatethatafilehasbeendeletedandthattherecordofthefiledeletionisnotlost.

10.1. YAFFS2Extendedtags
YAFFS2extendsthetagsinobjectheaderswithextrafieldstoimprovemountscanningperformance.Theoperationof
extendedtagsisdescribedlaterinthisdocument.

11 BadblockhandlingNANDerrorhandling
NANDflashisdesignedtobeverylowcostandveryhighdensity.Inordertogethighyields,andthuskeepcostsdown,
NANDflashistypicallyshippedwithafewbadblocksandsomefurtherblocksareexpectedtogobadduringtheuseofthe
device.Thus,anyflashfilesystemwithoutaneffectivebadblockhandlingpolicyisunsuitableforNANDflash.
YAFFS1usestheSmartMediastyleofbadblockmarking.Thischecksthesixthbyte(byte5)ofthesparearea.Inagood
blockthisshouldbe0xFF.Afactorymarkedbadblockshouldbe0x00.IfYAFFS1determinesthatablockhasgonebadit
ismarkedbadwith0x59('Y').UsingadistinctivemarkerallowsYAFFSmarkedbadblockstobedifferentiatedfrom
factorymarkedbadblocks.
YAFFS2modeisdesignedtosupportawiderrangeofdevicesandsparearealayouts.Thus,YAFFS2doesnotdecide
whichbytestomarkbutinsteadcallsdriverfunctionstodeterminewhethertheblockisbad,ortomarkitbad.
So,whendoesYAFFSmarkablockbad?YAFFSwillmarkablockbadifareadorwriteoperationfailsorifthreeECC
errorsaredetected.Ablockthatismarkedbadisretiredfromuse,thusimprovingtherobustnessofthefilesystem.This
policyissuitableforSLCflash.MLCpoliciesareunderdevelopment.
Further,NANDflashcellscansometimesbedisturbed(corrupted)byNANDactivityandchargeloss.Theseerrorsare
correctedbyerrorcorrectioncodes(ECC)whichmaybeimplementedinhardware,softwaredrivers,orwithinYAFFS

Page11/25

itself.Again,anyflashfilesystemlackingeffectiveECChandlingisunsuitableforNANDflash.
YAFFS1modecanuseeitherthebuiltinECCoruseECCprovidedbythedriverorhardware.
SinceYAFFS2modeisdesignedforawiderrangeofdevices,itdoesnotprovideECCinternallybutrequiresthatthedriver
providetheECC.
TheECCcodesuppliedwithYAFFS(yaffs_ecc.c)isthefastestCcodeimplementationofaSmartMediacompatibleECC
algorithmthatweareawareof.Thiscodewillcorrectanysinglebiterrorwithina256bytedatablockanddetect2errors
per256bytedatablock.ThisissufficienterrorcorrectiontoprovideveryhighreliabilityonSLCtypeNANDflash.
Foramorecompletedescriptionoffailurehandling,pleaserefertotheYAFFSErrormitigationdocument.

12 RAMstructures
WhileitistheoreticallypossibletousealogstructuredfilesystemwithveryfewRAMstructures,thiswouldprovide
reduceperformance.Thus,significantRAMdatastructuresarerequiredtostoreenoughinformationtoprovidesufficiently
highperformance.
Beforewejumpin,itisagoodideatofirstfigureoutwhatpurposetheRAMstructuresserve.Themainstructuresare
definedinyaffs_guts.handtheirmainpurposes,are:

Device/partition:Thisisnamedyaffs_Device.Thisisrequiredtoholdinformationrelatingtoayaffspartitionor
mountpoint.Usingthese,ratherthanglobals,allowsYAFFStosimultaneouslysupportmultiplepartitionsand
partitionsofdifferenttypes.Infact,almostotherdatastructuresmentionedherearepartof,oraccessedvia,the
structure.

NANDBlockInformation:Thisisnamedyaffs_BlockInfoandholdsthecurrentstateoftheNANDblocks.Each
yaffs_Devicehasanarrayofthese.

NANDChunkInformation:Thisisabitfieldattachedtotheyaffs_Devicethatholdsthecurrentinusestateofeach
chunkinthesystem.Thereisonebitperchunkinthepartition.

Object:Thisisnamedyaffs_Objectandholdsthestateofanobject(seesection3).Thereisoneoftheseforeach
objectinthefilesystem,whereanobjectisoneofaregularfile,directory,hardlink,symboliclinkorspeciallink.
Therearedifferentvariantsofyaffs_Objectstoreflectthedifferentdatarequiredforthesedifferentobjecttypes.
EachobjectisuniquelyidentifiedbyanobjectId.

Filestructure:Foreachfileobject,YAFFSholdsatreewhichprovidesthemechanismtofindthedatachunksina
file.Thetreeconsistsofnodescalledyaffs_Tnodes(treenodes).

Directorystructure:Thedirectorystructureallowsobjectstobefoundbyname.Thedirectorystructureisbuilt
fromadoublylinkedlistthatisrootedinthedirectoryandbindstogetherthesiblingobjectswithinadirectory.

Objectnumberhashtable:TheobjectnumberhashtableprovidesamechanismtofindanobjectfromitsobjectId.
TheobjectIdishashedtoselectahashbucket.Eachhashbuckethasadoublylinkedlistofobjectsthatbelongin
thishashbucket,aswellasacountoftheobjectsinthebucket.

Cache:YAFFSprovidesaread/writecachethatsignificantlyimprovesperformanceforshortoperations.Thesize
ofthecachecanbesetatruntime.

Page12/25

Theusageofthesestructuresisdescribedinmoredetailbelow.
YAFFSmakessignificantuseofdoublylinkedlists.Thesedatastructuresareveryusefulbecausetheyproviderapid
insertion,removalandtraversal.Theydohowevercomeattheexpenseofrequiringtwopointersratherthanjustone.

12.1. yaffs_Object
Eachobject(file,directory,...)inthesystemisrepresentedbyayaffs_Object.Theyaffs_Object'smainfunctionistostore
mostoftheobjectmetadataandtypespecificinformation.Thismeansthatalmostalltheinformationaboutanobjectcanbe
accessedimmediatelywithoutreadingflash.
Themetadataincludes,amongstothers:
objectId:Thenumberusedtoidentifytheobject.
parent:Apointertotheparentdirectory.Notallobjectshaveaparent.Therootdirectorandspecialdirectoriesfor

unlinkedanddeletedfilesdonothaveparentssothisisNULL.

shortname:Ifthenameisshortenoughtofitinasmallfixedsizedarray(default16charaacters)thenitisstoredhere

otherwiseitmustbefetchedfromflasheachtimeitisrequrested.
type:Typeofobject.
permissions,time,overshipandotherattributes

Dependingontheobjecttype,yaffs_Objectalsostores:
atnodetree,fileextentsetcfordatafiles
adirectorychainfordirectories
anequivalentobjectforhardlinks
astringforsymboliclinks

12.2. ObjectIdlookup
ThepurposeoftheobjectIdlookupmechanismistorapidlyaccessanobjectbyitsdeviceandobjectId.Thisisrequired
duringgarbagecollection,scanningandsimilaroperations.
Eachyaffs_Deviceusesahashtableforthispurpose.Thehashtablehas256buckets(tunable),eachholdingadoubly
linkedlistoftheyaffs_Objectsinthatbucket.
Thehashtable'shashfunctionisjustmaskingtheleastsignificantbitsoftheobjectId.Thisisaverycheapfunctionthat
providesreasonablespread.
YAFFSmakesanattempttoassignobjectIdsinawaythatkeepsthenumberofobjectsineachhashbucketquitelow,thus
preventinganyonehashentrygettingtoolong.

Page13/25

12.3. Directorystructure
Thepurposeofthedirectorystructureistorapidlyaccessanobjectbyname.Thisisrequiredtoperformfileoperations
suchasopening,renamingetc.
TheYAFFSdirectorystructureisformedbyatreeofyaffs_ObjectsoftypeYAFFS_OBJECT_TYPE_DIRECTORY.These
objectshaveadoublylinkednodeofchildren.
Eachyaffs_Objectalsohasadoublylinkedlistnodecalledsiblingwhichchaintogetherthesiblingsinadirectory.
EachYAFFSpartitionhasarootdirectory.
EachYAFFSpartitionalsohassomespecialpurposefakedirectorieswhicharenotstoredinNANDbutareinstead
createdonmounting:

lost+found:Thisdirectoryisusedasaplacetostoreanylostfilepartsthatcannotbeplacedinthenormaldirectory
tree.

Unlinkedanddeleted:Thesearepseudodirectoriesthatarenotinthedirectorytree.Placingobjectsinthese
directoriesgivesthemastatusofbeingunlinkedordeleted.

Eachobjecthasanamethatisusedtolookuptheobjectwithinthedirectory,Thus,eachnamewithinadirectorymustbe
unique.Thisruledoesnotapplytoobjectsintheunlinkedanddeletedpseudodirectoriessinceweneverusenamestolook
upunlinked/deletedfilesandtheremaybemanyfilesofthesamenamethathavebeendeleted.
Nameresolutionisspedupbytwomechanisms:
Shortnamesarestoreddirectlyintheyaffs_Objectsoithattheydon'thavetobeloadedfromflash.
Eachobjecthasanamesum.Thisisasimplehashingofthenamewhichspeedsupthematchingprocessbecauseitis

cheapertocomparetehashthancomparethewholenamestring.

root Sibling

Sibling

Sibling

Children

Sibling

data

Children

Children
e

Sibling

Children

lost+
Sibling
found

Children

Page14/25

Sibling

data

Theabovedirectorystructurerepresentsthefollowingdirectorytree
/

Rootdirectory

/a

Directorycontainingc,d,e

/b

Emptydirectory

/lost+found

Emptydirectory

/a/c

Directory

/a/d

Datafile

/a/e

Datafile

IfaYAFFSpartitionhasbeensomehowcorruptedthenitispossibleforanobjecttobecreatedduringscanningbutwithno
objectheader(whichcontainstheobjectname)beingfound.Theobjectwouldthennothaveaname.Inthesecases,YAFFS
createsapseudonymfromtheobjectnumberoftheformobjnnnnsothattheobjectcanbeplacedinthedirectorystructure.

12.4. Hardlinks
InPOSIXfilesystems,theobjects(inodesinLinuxterminology,vnodeinBSDspeak)andtheobjectnames(dentryin
Linuxterminology)areindependentinthatanobjectcanhavezero,oneormorenames.
Havingfileswithzerolinksiscausedbyconstructssuchas:
h=open(foo,O_CREAT|O_RDWR,S_IREAD|S_IWRITE);/*createfilefoo*/
unlink(foo); /*unlinkitsoitcan'tbefoundinthedirectory*/
/*westillhaveahandletothefilesocanstillaccessitviathehandle*/
read(h,buffer,100);
write(h,buffer,100);
close(h);/*assoonasthehandleisclosedthefileisdeleted*/

YAFFSdoesnotstoreseparateobjectandnamerecordsbecausethatwouldwastespaceandincurextrareading/writing.
Instead,YAFFSisoptimisedforthemosttypicalcaseofonenameperobject.Thatis,YAFFSstoresonenameper
yaffs_Objectandusescheatstoachieveotherbehavior:

Objectsthathavezeronames(ieunlinkedobjects)arehandledbyrenamingtheobjectintoafakedirectorythatis
usedtodesignateunlinkedobjects.

Multiplenamesobjectarehandledbydesignatingoneoftheobjectsthemainobjectandhavingmultiplehardlink
pseudoobjectsthatrefertoit.

Theinodenumber(vnidinBSDspeak)isaperfilesystemuniquenumberthatidentifiestheobject.TheYAFFS
objectidofthemainobjectispresentedtotheuser.Thatis,iftheinodenumberisqueriedfromahardlinkobject
thentheobjectidoftheequivalentobjectisreturned.

Thelinkagebetweenhardlinkpseudoobjectsandtheirequivalentobjectisthroughtwofields:

EachobjecthasahardLinksdoublylinkedlistnodethatlinksthelistofhardlinkstothatobject.

Eachhardlinkhasapointertotheequivalentobject.

Theusageofthisstructureisstraightforwardexceptforoneinterestingcasewhenhardlinksarebeingdeleted.Consider
thefollowingsequenceofevents:
1

#toucha

Createfilea

Page15/25

#lnab

Createahardlinkbwithequivalentobjecta.

#rma

Removea.Theobjectmuststillexistunderthenameb

Instep3,YAFFScannotjustdeletetheequivalentobjectsincethatwouldleaveahanginghardlinkwithnoequivalent
object.
Wealsocannotjustchangethetypeofoneofthehardlinkobjectsbecausethatwouldcausetheobject'sobjectidtochange.
Thiswouldbeunacceptablesinceitwouldbreakanymechanismthatusetheobjectid(ofwhichtherearemanywithinand
outsideofYAFFS).
Thus,ifanobjectisdeletedandithashardlinkstoitthenwedothefollowing:

Selectoneoftheobject'shardlinks.

Renametheobjecttothenameinthehardlink.

Deletethehardlinkpseudoobject.

12.5. Symboliclinkandspecialobject
TheseobjecttypesjuststorevaluesthatareneededtoprovideafullPOSIXfilesystem.
AsymboliclinkobjectsavesasymboliclinkstringthatisusedtoprovidethePOSIXsymbolicstringmechanism.
Aspecialobjectisusedtostoredevicenumbers,pipesandothersimilarspecialobjects.

12.6. Fileobject
Theyaffs_Objectshaveatypeandavariantportionwhichisaunionofdifferentdatarequiredtoholdthestateofdifferent
objecttypes.
ThefileobjectshavetypeYAFFS_OBJECT_TYPE_FILE,andanassociatedfilevariant.Thisstoresthefollowingmain
values:

fileSize:filesize

topLevel:depthoftheTnodetree

top:pointertotopofTnodetree

Themeaningofthefilesizeisquiteobvious.Thisisthecurrentlengthofthefile.Notethatafilemighthaveholesinit
causedbyseekingforwardpasttheextentsofthefile.
ThetopandtopLevelworktogethertocontroltheTnodetrees.

12.6.1 Tnodetree
EachfilehasaTnodetreetoprovidethemappingfromfilepositiontoactualNANDchunkaddress.topisapointertothe
topoftheTnodetree

Page16/25

yaffs_Object

Level2

Level1

Level0

NAND

Note,only4entriesperTnodeareshowntosimplifythediagram.

TheTnodetreeismadeupofTnodes(treenodes)arrangedinlevels,eachofwhichholdseither:

Atlevelsgreaterthan0,aTnodehas8pointerstootherTnodesinthenextleveldown.

Atlevel0,aTnodehas16NANDchunkIdswhichidentifythechunk'slocationinRAM.

Thesenumbersarepowersof2whichmakesforsimplelookupsbyjustapplyingbitmasks.Thecodethattraversesthistree
isinyaffs_FindLevel0Tnode().
Asthefilesizegrows,moreTnodesareaddedandasthelevelsfill,morelevelsareadded.Ifthefileistruncatedorresized
downwards,thenthetreeispruned.Ifachunkinafileisreplaced(eg.dataisoverwrittenorcopiedbygarbagecollection)
thentheTnodetreemustbeupdatedtoreflectthechange.
TnodestypicallymakeupthebulkoftheYAFFS'RAMusage.

13 Howvariousmechanismswork
HavinglookedattheYAFFSdatastructures,wenowconsiderhowsomeofthemechanismsinYAFFSoperates.Thisis
mosteasilydonebywalkingthroughsomeoperationsthatYAFFSperforms.

13.1. BlockandChunkManagement
13.1.1 BlockStates
YAFFStracksthestateofeachblockandchunkinthepartition.Thisinformationisfirstbuiltupduringscanning(or
recoveredfromacheckpoint).Ablockmaybeinoneofthefollowingstates(seedefinitionofyaffs_BlockState):
UNKNOWN

Theblockstateisnotyetknown.

Page17/25

NEEDS_SCANNING Duringprescanningithasbeendeterminedthatthisblockhassomethingonitandneeds
scanning.
SCANNING

Thisblockiscurrentlybeingscanned.

EMPTY

Thisblockhasnothinginit(iserased)andmaybeusedforallocation.Theblockcangettothis
statebybeingerased.

ALLOCATING

Thisistheblockcurrentlybeingusedforallocationbythechunkallocator.

FULL

Allthechunksinthisblockhavebeenallocatedandatleastonechunkcontainsusefulinformation
thathasnotyetbeendeleted.

DIRTY

Allthechunksinthisblockhavebeenallocatedandallhavebeendeleted.Theblockmighthave
beenintheFULLorCOLLECTINGstatebeforethis.Thisblockcannowbeerasedandreturned
toEMPTYstate.

CHECKPOINT

Thisblockhascheckpointdatainit.Whenthecheckpointisinvalidatedthisblockwillbeerased
andreturnedtotheEMPTYstate.

COLLECTING

Thisblockisbeinggarbagecollected.Assoonasalllivedatahasbeeninspectedthisblock
becomesDIRTY.

DEAD

Thisblockhasbeenretiredorwasmarkedbad.Thisisaterminalstate.

Page18/25

UNKNOWN
Regularruntimestates
Prescanning

DEAD

Scanning

NEEDS
SCANNING

EMPTY

SCANNING

CHECKPOINT

ALLOCATING

FULL

Checkpointing

DIRTY
COLLECTING
Garbagecollection

Regularruntimetransitions
Errortransitions
Minortransitions

Theregularruntimestatesare;EMPTY,ALLOCATING,FULL,COLLECTINGandDIRTY.
Thecheckpointstateisusedforcheckpointblocks.
TheDEADstateistheterminalstateforblocksthatwereretiredbecausetheywereeithermarkedbadoranerrorwas
detected.

13.1.2 BlockandChunkAllocation
Anavailablechunkmustbeallocatedbeforeitcanbewritten.Thechunkallocationmechanismisprovidedby
yaffs_AllocateChunk()andisverysimple.Eachpartitionhasacurrentblockthatitisallocatingfrom.Thisistermedthe
allocationblock.Chunksareallocatedsequentiallyfromtheallocationblock.Asthechunksareallocated,blockandchunk
managementinformationisupdated,includingtheblock'spagesInUsecountandthechunkusebitmap.
Whentheblockisfullyallocated,anotheremptyblockisselectedtobecometheallocationblock.Theblocksareselected
bysearchingupwardsfromthepreviousallocationblock.

Page19/25

Previoustoyaffs_guts.cV1.99,yaffsscanningwouldidentifytheallocationblockbeingusedwhenthefilesystemwaslast
shutdownandwouldcontinueallocationfromthatpoint.Thishadthebenefitofcontinuingtousethesameblockand
slightlyreducegarbagecreation.However,doingthiscouldpotentiallycauseproblemsduetoattemptingtowriteablock
thathadbeeninthemiddleofawritewhenpowerwaslost.Thus,fromV1.99onwards,yaffsnowalwaysstartsallocation
onanewblockafteramount.Thisincreasestheamountofgarbage(typicallybyaninsignificantamount)butimproves
robustness.

13.1.3 Awordaboutwearleveling
Sinceflashmemoryblockscanonlyendurealimitednumberofwritesanderases(termedtheflashendurance),aflashfile
systemneedstoensurethatthesamefewblocksarenotoverwrittencontinuouslycausingthoseblockstowearoutwhilethe
otherblocksarehardlyusedatall..Insteaditisimportantthatthefilesystemspreadsoutthewear.
ThisisextremelyimportantinafilesystemsuchasFATwhichusesalimitedsetoflogicalblockstostorethefile
allocationtableentrieswhichgetmodifiedfrequently.IfFATisusedwithastaticlogicaltophysicalblockmappingthen
thefileallocationtableblockswillwearoutfarbeforeotherblockscausingprematuredevicefailure.Thus,forFAT,itis
veryimportantthatthelogicaltophysicalmappinggetsstirredupabittocausethefileallocationtablestobewrittento
differentphysicalblocks,thuslevellingoutthewear.
WearlevellingisfarlessofaconcerninalogstructuredfilesystemsuchasYAFFSsincewealwayswritetotheendofthe
logandthusdon'tkeeponoverwritingthesameblocks.
Therearebasicallytwowaystoachievewearleveling:
useaspecificallywrittensetoffunctionsthatactivelyperformwearleveling.
allowadegreeofwearlevelingtohappenasasideeffectofotheractivity.

YAFFStakesthesecondapproach.
Firstly,beinglogstructured,yaffsinherentlyspreadsoutthewearbyperformingallwritesinsequence.Thatmeansthatwe
don'tsingleoutafewblocksforaspecificpurposeandwritethemmorethanothers.
Secondly,blocksareallocatedseriallyfromtheerasedblocksinthepartition.Thustheprocessoferasing(afterdeletionof
garbagecollection)andthenallocatingblocksdoestendtospreadtheblockuseandprovidesalevelofwearlevelling.
Thus,eventhoughthereisnocodetospecificallyperformwearlevelingitdoeshappenasasideeffectofotheractivity.
ThisstrategyperformswellandhassurvivedmanyacceleratedlifetimetestsinbothsimulationsandonrealNANDbased
products.

13.1.4 yaffs_Tnodeandyaffs_ObjectManagement
yaffs_Tnodesandyaffs_Objectsaredynamicallycreatedandreleasedastheyareneeded.
Theallocationandmanagementcouldbedonebyjustusingthestandardmemoryallocator(malloc,kmallocetc),butthis
couldcauseperformanceproblems,particularlyinembeddedsystemswhichtendtohaverathersimplisticmemory
allocators.
Theallocationanddeletionofmanysmallstructurescancausememoryfragmentationcausingoutofmemoryand
slowdownproblems.
Topreventtheseproblems,YAFFSmanagestheseobjectsinternally.YAFFScreatesmanystructures(100orso)atatime

Page20/25

andthenallocatesthemsinglyfromafreelist.Whenstructuresarereleasedtheyareplacedonthefreelistforreuse.

13.2. InternalCache
ThisdescribesthestandardYAFFSinternalcacheimplementation.Alternativecacheshavebeen
devisedforotherpurposesandatsomefuturepointYAFFSmightbeextendedtosupportpluggable
cachepolicies.
ThemainpurposeoftheinternalcacheistoreduceNANDaccessforbadlybehavedapplications
whichperformmanysmallreadsandwrites.Believeitornot,butthereareapplicationsouttherethat
readorwritefilesafewbytesatatime:
while(...)
write(h,buf,1);
Thissortofbehaviourwouldbeterribleforperformanceandflashmemorylifeifeachreadorwriteoperationresultedina
writetoflash.YAFFSprovidesaninternalcachetomitigateagainstthissortofbehaviour.
TheYAFFSinternalcacheiscalledaShortOpCache(shortoperationcache)andonlycomesintoplayforoperations
whicharenotchunkaligned.ThecachingmechanismisverysimpleandwasprimarilywrittenforlesssophisticatedOSs
thatlacktheirownfilecachinglayer.ThecacheisstillbeneficialinOSslikeLinux,whichprovideacachinglayer,because
itcachesboththereadandwritepath.Thus,whileshortreadswillbeservicedbytheLinuxpagecache,theshortOpCache
doesaddressnonchunkalignedwrites.
Pagealignedreads/writesbypassthecacheunlessthatchunkisalreadyinthecache.
Thecacheisstoredinanarrayofcacheentries.Thenumberofcacheentriesissetintheyaffs_Deviceconfigurationby
dev>nShortOpCachesandthecacheisconfiguredduringinitialisation.Acachesizeofzerodisablesthecache.Thecache
managementalgorithmsaresimplisticandprobablydon'tscalewelltoverylargenumbers.Thismeansthatitisprobably
besttostickwitharound5to20cacheentries.
Eachcacheentryholds:

objectid,chunkid,nbytes(thetaginfoofthechunkbeingcached)

theactualdatabytes

cacheslotstatus(lastusedcounter,dirty,...)

Findinganunusedcacheentryisasimpleoperation.Wejustiteratethroughthecacheentriessearchingforonethatisnot
busy.Ifallcacheentriesarebusythenacachepushoutisperformedandthenweretrythefind.
Wheneveracacheentryisaccessed(readorwritten)thelastusecounterisissettoanincrementingnumber.Thiscounter
valueisusedtodrivetheleastrecentlyused(LRU)algorithm.
Cacheentrieswhichhavebeenmodifiedandnotyetwrittenbacktoflasharemarkedwithadirtyflag.Thisistellsus
whetherweneedtowriteoutthedataduringaflushorpushout.
Apushouthappenswheneverthecacheisfullandweneedtopushanentryoutofthecachetomakespacetoloadanew
entry.TodothisweusetheLRUalgorithmtoselecttheLRUcacheentry.Wethenwriteoutalldirtyentriesforthatfile,in
ascendingorder,andmarktheseentriesasfree.
Wheneverafileisflushed(fileflushorclose),thefile'sdirtyentriesarewrittenout.Atcertaintimes(eg.beforean
unmountoratsync())theentirecacheisflushed.

Page21/25

Thiscachingpolicyisfarfromoptimalbutgivesausefulperformanceimprovementwithverysimplealgorithms(good
bangforbucks).
Apartfromitsnormalcachingfunction,theshortOpCachealsodoesmuchoftheheavyliftingforinbandtagssupportsince
alltheinbandtagsaccessesarenotchunkaligned,theymustallgothroughtheshortOpCache.

13.3. Scanning
Scanningistheprocesswherebyfilesystemstateisrebuiltfromscratch.ThishappensonmountingaYAFFSpartitionifno
checkpointdataisavailableorifthemountingwastoldtoignorethecheckpointdata.
Scanninginvolvesreadingthetagsfromalltheactivechunksinthesystemsocantakequiteawhile,butprobablylesstime
thanyou'dexpect.
ThescanningprocessisdifferentforYAFFS1vsYAFFS2becauseofthedifferencesmentionedbefore.

13.3.1 YAFFS1Scanning
YAFFS1scanningisratherstraightforwardbecauseYAFFS1usesdeletionmarkers.
Allwereallyneedtodoisreadthetagsoffallthechunksinthepartitionanddetermineifthechunkisactive.Orderisnot
importantandblockscouldbereadinanyorder.Ifthedeletionmarkerhasnotbeensetthenitisactiveandweneedtoadd
thechunktothefilesystemstateasfollows:
Ifthechunkisadatachunk(chunkId>0)thenthechunkmustbeaddedtotherelevantfile'stnodetree.Iftheobjectdoes
notexist,thenwecreateitinlost+found,althoughithasnonameandotherfiledetailsyet.
Ifthechunkisanobjectheader(chunkId==0)thentheobjectmustbecreated(oritmightalreadyexistinlost+foundand
weneedtofixupitsdetailsandslotitintothecorrectpartofthedirectorytree.
IfatanytimeduringscanningwefindthatwearereadingasecondchunkwiththesameobjectId:chunkIdasapreviously
encounteredchunkthenthe2bitserialnumberisusedtoresolvethedispute.
Oncethescanningiscompletewestillneedtofixupafewthings:
Hardlinksmustbeconnectedupproperly(itismoreefficienttodeferthisuntilafterthescanninghascompleted).
Also,objectswhichweremarkedasunlinkedmustbedeletedsincetherearenolongerhandlesopentotheseobjects.

13.3.2 YAFFS2Scanning
YAFFS2hasnodeletionmarkerswhichmakesthescanningsignificantlymorecomplex.
Thefirstthingthatneedstobedoneistoprescantheblockstodeterminethesequencenumberfortheblocks.Thelistof
blocksandsequencenumbersisthensortedtomakeachronologicallyorderedlist.
Wenowscanbackwardsthroughtheseblocks(ieinreversechronologicalorder).Thus,thefirstoccurrenceofany
objectId:chunkIdisthemostcurrentoneanditiseasytodeterminewhichischunksareoutofdate.
Wereadthetagsforeachchunkineachblock.
Ifitisadatachunk(ie.chunkId>0)then:

IfachunkwiththesameobjectId:chunkIdhasalreadybeenfoundthenthisisnotthecurrentchunk.
Deletethisone.

Elseiftheobjectismarkedasdeleted,orcorrupted,thendeleteit.

Elseiftheobjectdoesnotyetexistthenthischunkwaswrittensincethelastobjectheaderwriteforthis
objectsothischunkmustbethelastwrittenchunkinthefileandmusthavehappenedjustbeforean

Page22/25

uncleanshutdownwiththefilestillopen.WecancreatethefileandusethechunkIdandnbytestosetthe
fileextents.

Elseiftheobjectdoesexistandthechunkliesbeyondtheobject'sfilescansize(extents)thenitisa
truncatedchunkandcanbedeleted.

Elseitisplacedintothefile'stnodetree.

Ifitisanobjectheader(ie.chunkId==0)then:

Iftheobjectisinthedeleteddirectorythenmarktheobjectasdeletedandsettheobject'sshrinksizeto
zero.

Elseifanobjectheaderhasnotyetbeenfoundforthisobject,thenthisisthemostcurrentobjectheader
andholdsthemostcurrentnameetc.Ifnodatachunkshaveyetbeenfoundforthischunkthenthesizein
theobjectheaderisalsothefileextents.Thefilesizeisusedtosetthefilescansizesinceatthetimeit
waswrittenthefiledidnotextendpastthatpointandwecansafelyignorefurtherdatachunksforthatfile
thatextendpastthatpoint.

Elsejustusethefilesizeasthescansizeifitissmallerthanthecurrentfilescansize.

Elsethisanobsoleteobjectheader.Itisdeleted.

Abitmoreexplanationisprobablyneededtodescribethefilescansize.Thewholepurposeofthisistodeterminedata
chunksthatshouldbeignoredduetothefilebeingtruncated.

13.4. Checkpoint
Mountscanningtakesquitealotoftimeandslowsmounting,thoughperhapsbylessthanyou'dthink.Checkpointingisa
mechanismtospeedthemountingbytakingasnapshotoftheYAFFSruntimestateatunmountorsync()andthen
reconstitutingtheruntimestateonremounting.Thespeedupisdramatic.
Theactualcheckpointmechanismisquitesimple.Astreamofdataiswrittentoasetofblockswhicharemarkedas
holdingcheckpointdataandtheimportantruntimestateiswrittentothestream.
Notallstateneedstobesaved,onlythestateneededtoreconstitutethestructureoftheruntimedata.Forexample,file
metadataneednotbesavedsinceitisveryeasytoloadthatthroughlazyloading.
Thecheckpointstoresdatainthefollowingorder:
startmarker,includingcheckpointformatid.
yaffs_Deviceinformation
blockinformation
chunkflags
objects,includingfilestructure
endmarker
checksum

Thecheckpointvalidityisensuredbythefollowingmechanisms:
thedataisstoredusingwhateverECCmechanismsareinplace.
thestartmarkerincludesacheckpointversionsothatanobsoletecheckpointwillnotbereadifthecheckpointcode

changes.
dataiswrittentothestreamasstructuredrecords,eachwithatypeandsize.

Page23/25

theendmarkermustbereadwhenexpected.
twochecksumsaremaintainedacrossthewholecheckpointdataset.

Ifanychecksfail,thecheckpointisignoredandthestateisrescanned.
Theastutereadermightnoticethatblockstatechangesaswe'rewritingcheckpointblocks,sohowdowemanagetore
constitutetheblockstatescorrectly?Thisisachievedbythefollowing:
Aswewritethecheckpoint,thecheckpointblockallocationselectsEMPTYblocks.Wedon'tchangethestateaswe're

wrtitngthecheckpointsotheirstateiswrittentothecheckpointasEMPTYorCHECKPOINT,itdoesnotreallymatter
which.
Afterreadingthecheckpoint,weknowwhichblockshavebeenusedtostorecheckpointdata.Weupdatethoseblocksto

reflecttheCHECKPOINTstate.

Acheckpointisinvalidatedbyanymodifications(writes,erases,...)thusanymodificationpathalsocheckstoseeifthereis
acheckpointanderasesit.
Theregularyaffsblockallocatorandgarbagecollectionlogicalsoneedstobeawareoftheapproximatesizeofthe
checkpointsothatthesecanensureenoughsparespaceisreservedtostoreacheckpoint.Thereisacalculationthatupdates
thevaluewheneverthenumberofobjectsetcchanges.

13.5. ExtendedTagsandPackedTags
Extendedtagsisanabstractionofawaytostoremoreinformationinthetags,allowingforfasterscantimes.PackedTagsis
aphysicalimplementationofthestorageofextendedtags.
Whenwe'rescanningwereallyonlyneedtobuildupthestructureofthefilesystem.Inotherwords,rebuildthedirectory
relationships,tnodetreesandsuch.Muchofthedetailsuchasfilenames,permissionsetccanbelazyloaded.
Thisrequiredstoringextrainformation,suchastheparentdirectoryandobjecttype,inthetagsofobjectheaders.Buthow
dowedothiswithoutenlargingthetagssize?
ObjectheadersalwayshaveachunkIdofzeroandnbyteshasnomeaningbecauseitisnotadatachunk.Byusingasingle
bittoidentifyobjectheaderswesuddenlyopenupawholelotofspacethatwecanstufffullofusefuldata.Bybeingclever
astowhatdataweuse,wecanpackintheimportantdatainmostcircumstances.
Sometimes,forexampleverylargefiles,thepackingwillnotworkbecausethereareinsufficientbitstoholdthefieldsowe
justwriteregulartagsinstead.That'sOKsincetherecanonlybefewsuchfilesinthesystemsotheperformancereduction
onlyimpactsveryfewfilesandthesystemlevelimpactisunnoticable.Asimple,cheap,strategythatgivesspeedup99%of
thetimeisaveryhandyone.
Otherdetailsarelazyloadedondemand.Thiswillhappenwhentheobjectislookedup(eg.byafileopenorsearchingfor
thefileinthedirectory).

13.6. Inbandtags
Thusfarwehaveconsideredchunkswheretherearesufficientunallocatedbytesinthespare(oroutofband)areatostore
thetags.Thismeansthatthedatapartofthepageisapowerof2insize.Powersof2areveryfasttodealwithwhichis
whymostoperatingsystemsusevariouspowersof2intheirinternals.
Butwhathappensifwedon'thaveenoughspaceinthespareareatostorethetags?Inbandtagstotherescue!Withinband
tagswestorethetagsintheflashdataareaalongsidethedata.Thisgivesusfarmoreflexibilityinthetypesofstoragethat
yaffscansupport,butbreaksthepowerof2alignment.Therearethreemajorpenaltieswithinbandtags:

Duetothelossofchunkalignment,alltransactionsmustpassthroughtheshortOpCache.Thiscostsanextra
memcpy().

Offsettochunk,freespaceandothercalculationsarenolongerjustshiftandmaskoperationsbutinsteadrequire

Page24/25

multiplicationsanddivisionswhichcanrelativelyexpensive.

Readingtagsduringscanningetccaninvolvereadingthewholepage,makingthisslower.Theimpactisminimised
byusingcheckpointing.

Thus,itisbetter,performancewise,toonlyuseinbandtagswhenregulartagswillnotwork.

13.7. SoftDeletion
SoftdeletionisamechanismthatwasaddedtospeedupfiledeletioninYAFFS1mode.
LookingbacktothedescriptionofhowfilesarestoredinYAFFS1,youcanseethatdeletingafileinvolveswritingthe
deletionmarkerofeverydatachunkinthefile.Thiscantakequitesometimeforlargefiles.
Thesoftdeletionmechanismwasaddedtoavoidwritingallthosedeletionmarkers.Insteadofwritingthedeletionmarkers,
softdeletioninsteadtracksthenumberofchunksineachblockthatbelongtodeletedfilesandleavesthecleanuptothe
garbagecollector.
Whenthegarbagecollectorlooksatablockitcanseewhatcurrentchunks(iechunksnotyetdeleted)belongtodeleted
filesanddoesnotcopythose,thuseffectivelydeletingthechunk.Onceallthechunksassociatedwithadeletedfilehave
cleanedupbythegarbagecollectorthefileheaderobjectcanalsobedeleted.
YAFFS2modedoesnotwritedeletionmarkersandthusdoesnotusesoftdeletion.

Page25/25