14/08/2015

Document361468.1

PowerViewisOff

Dashboard

Knowledge

ServiceRequests

Patches&Updates

GiveFeedback...

YouhavebeendirectedtothisdocumentbasedonanIDmatch.

(0)

Sanuj(Available)

ContactUs

361468.1
Alternatively,clickheretosearchonthisphrase.

HugePagesonOracleLinux64bit(DocID361468.1)
InthisDocument

Wasthisdocumenthelpful?
Yes
No

Scope

Details
Introduction

WhyDoYouNeedHugePages?

HowtoConfigure

CheckandValidatetheConfiguration

Troubleshooting

KnownProblemsandLimitations

FurtherReading

References

DocumentDetails
Type:
Status:
LastMajor
Update:
LastUpdate:
Language:

REFERENCE
PUBLISHED
21Jul2015
21Jul2015
English

RelatedProducts

OracleDatabaseEnterprise
Edition
LinuxOS
OracleDatabaseEnterprise
Edition
InformationCenters

InformationCenter:Oracle
Linux[1381248.2]
IndexofOracleDatabase
InformationCenters
[1568043.2]
InformationCenter:Overview
ofDatabaseSecurityProducts
[1548952.2]

PURPOSE

InformationCenter:Overview
DatabaseServer/Client
Installationand
Upgrade/Migration
[1351022.2]

Thisdocumentaimstoprovide.
BasicconfigurationofHugePageson64bitLinux
FundementalreasonstouseHugePagesonLinux
Referencestoknownproblems
ReferencestotechnicalbackgroundonHugePages

APPLIESTO:
LinuxOSVersionEnterpriseLinux4.0toOracleLinux6.0withUnbreakableEnterpriseKernel[2.6.32][ReleaseRHEL4toOL6]
OracleDatabaseEnterpriseEdition
Linuxx8664
OracleLinux
RedHatEnterpriseLinux(RHEL)
SUSELinuxEnterpriseServer(SLES)

Hide

ToBottom

Purpose

Help

DocumentReferences

HugePagesandOracle
Database11gAutomatic
MemoryManagement(AMM)
onLinux[749851.1]

SCOPE
InformationinthisdocumentisusefulforLinuxsystemadministratorsandOracledatabaseadministratorsworkingwithsystem
administrators.

HugePagesonLinux:WhatIt
Is...andWhatItIsNot...
[361323.1]

ThisdocumentcoversinformationaboutLinuxHugePagesfor64bitarchitectures.Formoregenericanduseson32bitandfor
referencespleaseseeDocument361323.1

11.2.0.1/11.2.0.2to11.2.0.3
GridInfrastructureand
DatabaseUpgradeonExadata
DatabaseMachine[1373255.1]

TheconfigurationstepsprovidedhereisprimarilyforOracleLinux.Stillthesameconceptsandconfigurationsshouldapplyto
otherLinuxdistributions.

ShellScripttoCalculate
ValuesRecommendedLinux
HugePages/HugeTLB
Configuration[401749.1]

DETAILS

USE_LARGE_PAGESToEnable
HugePages[1392497.1]

Introduction

ShowMore

HugePagesisafeatureoftheLinuxkernelwhichallowslargerpagestomanagememoryasthealternativetothesmall4KB
pagesize.Foradetailedintroduction,seeDocument361323.1

WhyDoYouNeedHugePages?
HugePagesiscrucialforfasterOracledatabaseperformanceonLinuxifyouhavealargeRAMandSGA.Ifyourcombined
databaseSGAsislarge(likemorethan8GB,canevenbeimportantforsmaller),youwillneedHugePagesconfigured.Notethat
thesizeoftheSGAmatters.AdvantagesofHugePagesare:

LargerPageSizeandLess#ofPages:Defaultpagesizeis4KwhereastheHugeTLBsizeis2048K.Thatmeansthe
systemwouldneedtohandle512timeslesspages.
ReducedPageTableWalking:SinceaHugePagecoversgreatercontiguousvirtualaddressrangethanaregularsized

https://support.oracle.com/epmos/faces/DocumentDisplay?_afrLoop=485535575382755&id=361468.1&_adf.ctrlstate=wpwz66y6b_245

RecentlyViewed

HugePagesonOracleLinux
64bit[361468.1]
UsingMaterializedViews
[106024.1]
WhatareMaterialized
Views?[76673.1]
HowtoIncreasetheNumber
ofOpenFilesonHPUX
[1264834.1]
ADRCIPURGECommandIs
NotCleaningUp

1/4

14/08/2015

Document361468.1

page,aprobabilityofgettingaTLBhitperTLBentrywithHugePagesarehigherthanwithregularpages.Thisreduces
thenumberoftimespagetablesarewalkedtoobtainphysicaladdressfromavirtualaddress.
LessOverheadforMemoryOperations:Onvirtualmemorysystems(anymodernOS)eachmemoryoperationis
actuallytwoabstractmemoryoperations.WithHugePages,sincetherearelessnumberofpagestoworkon,thepossible
bottleneckonpagetableaccessisclearlyavoided.
LessMemoryUsage:FromtheOracleDatabaseperspective,withHugePages,theLinuxkernelwilluselessmemoryto
createpagetablestomaintainvirtualtophysicalmappingsforSGAaddressrange,incomparisontoregularsizepages.
ThismakesmorememorytobeavailableforprocessprivatecomputationsorPGAusage.
NoSwapping:WemustavoidswappingtohappenonLinuxOSatallDocument1295478.1.HugePagesarenot
swappable(whereasregularpagesare).Thereforethereisnopagereplacementmechanismoverhead.HugePagesare
universallyregardedaspinned.
No'kswapd'Operations:kswapdwillgetverybusyifthereisaverylargeareatobepaged(i.e.13millionpagetable
entriesfor50GBmemory)andwilluseanincredibleamountofCPUresource.WhenHugePagesareused,kswapdisnot
involvedinmanagingthem.SeealsoDocument361670.1

NotCleaningUp
CDUMP_{TIMESTAMP}
FoldersInTRACEDirectory
[756140.1]
ShowMore

ThereisageneralmisconceptionwheretheHugePagesisafeaturespecificto32bitLinux.HugePagesisagenericfeature
availabletoallwordsizesandarchitectures.Justthattherearesomespecificswith32bitplatforms.Pleasesee
Document361323.1forfurtherreferences.

HowtoConfigure
Theconfigurationstepsbelowwillguideyoutodoapersistentsystemconfigurationwhereyouwouldneedtodoacomplete
rebootofthesystem.Pleaseplanyouroperationsaccordingly:
Step1:Havethememlockuserlimitsetin/etc/security/limits.conffile.Setthevalue(inKB)slightlysmallerthaninstalledRAM.
e.g.Ifyouhave64GBRAMinstalled,youmayset:
*softmemlock60397977
*hardmemlock60397977

ThereisnoharminsettingthisvaluelargerthanyourSGArequirements.
Theparameterswillbesetbydefaulton:
OracleLinuxwithoraclevalidatedpackage(SeeDocument437743.1)installed.
OracleExadataDBcomputenodes

Step2:RelogontotheOracleproductowneraccount(e.g.'oracle')andcheckthememlocklimit
$ulimitl
60397977

Step3:IfyouhaveOracleDatabase11gorlater,thedefaultdatabasecreatedusestheAutomaticMemoryManagement(AMM)
featurewhichisincompatiblewithHugePages.DisableAMMbeforeproceeding.Todisable,settheinitializationparameters
MEMORY_TARGETandMEMORY_MAX_TARGETto0(zero).PleaseseeDocument749851.1forfurtherinformationincaseyou
encountertheerrorbelow:
ORA00845:MEMORY_TARGETnotsupportedonthissystem

Note:Startingin11.2.0.3,AMMisnolongerconfiguredbydefaultifDBCAdetectsthatthemachinehasmorethan4GBof
RAM.SeeDocument1453227.1fordetails.

Step4:Makesurethatallyourdatabaseinstancesareup(includingASMinstances)astheywouldrunonproduction.Usethe
scripthugepages_settings.shinDocument401749.1tocalculatetherecommendedvalueforthevm.nr_hugepageskernel
parameter.e.g.:
$./hugepages_settings.sh
...
Recommendedsetting:vm.nr_hugepages=1496
$

Note:Youcanalsocalculateapropervaluefortheparameteryourselfbutthatisnotadvisedifyoudonothaveextensive
experiencewithHugePagesandconcepts.
Step5:Editthefile/etc/sysctl.confandsetthevm.nr_hugepagesparameterthere:
...
vm.nr_hugepages=1496
...

https://support.oracle.com/epmos/faces/DocumentDisplay?_afrLoop=485535575382755&id=361468.1&_adf.ctrlstate=wpwz66y6b_245

2/4

14/08/2015

Document361468.1

Thiswillmaketheparametertobesetproperlywitheachreboot.
Step6:Stopallthedatabaseinstancesandreboottheserver
(Althoughsettingscouldhavebeendonedynamicallytheywouldnotbeeffectivetotheextentwerequirebeforeareboot.The
bestpracticeistodoapersistentsystemconfigurationandperformareboottocompletetheconfigurationaspresentedthrough
thestepsabove)

WhatIftheDatabase/SGAConfigurationsChange?
TheperformedconfigurationisbasicallybasedontheRAMinstalledandcombinedsizeofSGAofdatabaseinstancesyouare
running.Basedonthatwhen:
AmountofRAMinstalledfortheLinuxOSchanged
Newdatabaseinstance(s)introduced
SGAsize/configurationchangedforoneormoredatabaseinstances
youshouldreviseyourHugePagesconfigurationtomakeitsuitabletothenewmemoryframework.Ifnotyoumay
experienceoneormoreproblemsbelowonthesystem:
Poordatabaseperformance
Systemrunningoutofmemoryorexcessiveswapping
Databaseinstancescannotbestarted
Crucialsystemservicesfailing

CheckandValidatetheConfiguration
Afterthesystemisrebooted,makesurethatyourdatabaseinstances(includingtheASMinstances)arestarted.Automatic
startupviaOSconfigurationorCRS,ormanualstartup(whichevermethodyouuse)shouldhavebeenperformed.Checkthe
HugePagesstatefrom/proc/meminfo.e.g.:

#grepHugePages/proc/meminfo
HugePages_Total:1496
HugePages_Free:485
HugePages_Rsvd:446
HugePages_Surp:0

Thevaluesintheoutputwillvary.Tomakesurethattheconfigurationisvalid,theHugePages_Freevalueshouldbesmallerthan
HugePages_TotalandthereshouldbesomeHugePages_Rsvd.HugePages_Rsvdcountsfreepagesthatarereservedforuse
(requestedforanSGA,butnottouched/mappedyet).
ThesumofHugepages_FreeandHugePages_RsvdmaybesmallerthanyourtotalcombinedSGAasinstancesallocatepages
dynamicallyandproactivelyasneeded.

Troubleshooting
Someofthecommonproblemsandhowtotroubleshootthemarelistedinthefollowingtable:
Symptom

PossibleCause

TroubleshootingAction

Systemisrunningout
ofmemoryor
swapping

NotenoughHugePagestocovertheSGA(s)andthereforethe
areareservedforHugePagesarewastedwhereSGAsare
allocatedthroughregularpages.

ReviewyourHugePages
configurationtomakesurethatall
SGA(s)arecovered.

Databasesfailtostart

memlocklimitsarenotsetproperly

Makesurethesettingsinlimits.conf
applytodatabaseowneraccount.

Oneofthedatabasefail TheSGAofthespecificdatabasecouldnotfindavailable
tostartwhileanotheris HugePagesandremainingRAMisnotenough.
up

MakesurethattheRAMand
HugePagesareenoughtocoverall
yourdatabaseSGAs

ClusterReadyServices
(CRS)failtostart

HugePagesconfiguredtoolarge(maybelargerthaninstalled
RAM)

MakesurethetotalSGAislessthan
theinstalledRAMandrecalculate
HugePages.

HugePages_Total=
HugePages_Free

HugePagesarenotusedatall.Nodatabaseinstancesareupor DisableAMMandmakesurethatthe
databaseinstancesareup.SeeDoc
usingAMM.
ID1373255.1

Databasestarted
successfullyandthe
performanceisslow

TheSGAofthespecificdatabasecouldnotfindavailable
HugePagesandthereforetheSGAishandledbyregularpages,
whichleadstoslowperformance

MakesurethattheHugePagesare
manyenoughtocoverallyour
databaseSGAs

https://support.oracle.com/epmos/faces/DocumentDisplay?_afrLoop=485535575382755&id=361468.1&_adf.ctrlstate=wpwz66y6b_245

3/4

14/08/2015

Document361468.1

KnownProblemsandLimitations
BelowaresomeoftheknownandrelatedproblemsandlimitationsrelatedtotheHugePagesfeature:
Document749851.1HugePagesandOracleDatabase11gAutomaticMemoryManagement(AMM)onLinux
Document829850.1HugepagesAreNotUsedbyDatabaseBufferCache
Document803238.1OracleNotUtilizingHugepages
Document728063.1SetupHugePagesinanGuestDoesNotWorkwithOracleVM2.1or2.1.1
Document550443.1HugePagesNotReleasedOnOracleRDBMSInstanceShutdownwithRHEL/EL5Update1(5.1)
Document860350.1/proc/meminfoDoesNotProvideHugePagesInformationonOracleEnterpriseLinux(OEL5)
Document1557478.1ALERT:DisableTransparentHugePagesonSLES11,RHEL6,OEL6andUEK2Kernels

FurtherReading
Tobeabletodoadvanced/manualconfigurationswithHugePagesyouneedtounderstandtheimplementationandtheory
behindtheconcept.Youmayreadthefollowingforfurtherinformation:
Document361323.1HugePagesonLinux:WhatItIs...andWhatItIsNot...
Document401749.1ShellScripttoCalculateValuesRecommendedLinuxHugePages/HugeTLBConfiguration
Document397568.1LinuxIA64exampleofallocating48GBSGAusinghugepages

REFERENCES
NOTE:749851.1HugePagesandOracleDatabase11gAutomaticMemoryManagement(AMM)onLinux
NOTE:361323.1HugePagesonLinux:WhatItIs...andWhatItIsNot...
NOTE:1373255.111.2.0.1/11.2.0.2to11.2.0.3GridInfrastructureandDatabaseUpgradeonExadataDatabaseMachine
NOTE:401749.1ShellScripttoCalculateValuesRecommendedLinuxHugePages/HugeTLBConfiguration
NOTE:1392497.1USE_LARGE_PAGESToEnableHugePages
NOTE:361670.1SlowPerformancewithHighCPUUsageon64bitLinuxwithLargeSGA
NOTE:1557478.1ALERT:DisableTransparentHugePagesonSLES11,RHEL6,OL6andUEK2Kernels
Didn'tfindwhatyouarelookingfor?

AskinCommunity...

Related
Products
OracleDatabaseProducts>OracleDatabaseSuite>OracleDatabase>OracleDatabaseEnterpriseEdition>RDBMS
OracleLinuxandVirtualization>OracleLinux>OperatingSystem>LinuxOS>LinuxKernel>MemoryManagement
OracleDatabaseProducts>OracleDatabaseSuite>OracleDatabase>OracleDatabaseEnterpriseEdition>RDBMS

Keywords
HUGEPAGES HUGETLB LARGEPAGE MEMORY_TARGET PAGESIZE PERFORMANCE RAM VLM

Translations
English Source

Japanese

BacktoTop

Copyright(c)2015,Oracle.Allrightsreserved.

LegalNoticesandTermsofUse

PrivacyStatement

https://support.oracle.com/epmos/faces/DocumentDisplay?_afrLoop=485535575382755&id=361468.1&_adf.ctrlstate=wpwz66y6b_245

4/4