ORACLE.EXEHangsWithIncreaseinNonPagedPoolMemory.

BEQConnectionmayFail
WithORA12560(DocID866622.1)

InthisDocument

Symptoms
Cause
Solution

APPLIESTO:

OracleNetServicesVersion8.1.7.0to10.2.0.4.0[Release8.1.7to10.2]
OracleDatabaseEnterpriseEditionVersion8.1.7.0to10.2.0.4[Release8.1.7to10.2]
z*OBSOLETE:MicrosoftWindows
MicrosoftWindows

SYMPTOMS

Oracledatabaseinstancewillhangafterseveralhours.

TheBEQ(bequeath)connectionfailswith:

ERROR:
ORA12560:TNS:protoadaptererror
ERROR:ORA12500:TNS:listenerfailedtostartadedicatedserverprocess

Theremoteconnectionsfailwith:

TNS12500:TNS:listenerfailedtostartadedicatedserverprocess
TNS12540:TNS:internallimitrestrictionexceeded
TNS12560:TNS:protocoladaptererror
TNS00510:Internallimitrestrictionexceeded
32bitWindowsError:8:Execformaterror

Youmayseethebelowsymptoms:

1]TheSQL>count(*)fromv$processesshowsthecountbelowthelimitdefinedintheparameterfile.
2]Thememoryusageoforacle.exeiswellbelowthe32bitlimitation.
3]MonitoringtheORACLE.EXEprocessusingprocessexplorer/TaskManagershowsanabnormalnonpagedpool
countwhichkeepsonincreasing.(IntermsofMB)
4]UsuallythenonpagedpoolcountforanORACLE.EXEwouldbelessthananMB.Youmightseethenonpagedpool
countforotherinstance(oracle.exe)onthesamemachinelessthan1MB.
5]Manyhandlespersist(theprocessexploreroutputshowninthescreenshotbelow)thuskeepingnonpagedpoolat
ahighlevel.


Evenusingorastackutilitytochangethememoryusageoftheoracleprocesswillnotpreventthedatabase
hang.Usingtheorastackmightdelaytheoccurenceoftheproblembuttheinstancewilleventuallyhangoncethe
nonpagedpoolisfull.

ForExample:

C:\>orastackoracle.exe500000(everybackgroundprocesswillnowuse500Kinsteadof1Mb)
C:\>orastacktnslsnr.exe500000(everyprocessconnectedthroughSQL*Netwillnowuse500Kinsteadof1Mb)
C:\>orastacksqlplus.exe500000(everysqlplusconnectionwillnowuse500Kinsteadof1Mb)

DriversusetheNonPagedPoolformanyoftheirrequirementsbecausetheycanbeaccessedatanyinterruptRequest
Level(IRQL).TheIRQLdefinesthehardwarepriorityatwhichaprocessoroperatesatanygiventime.TheNonPaged
Poolcannotbepagedout.

WithintheNonPagedpool,youwouldfindhandlesthatareusedbyapplicationsintheusermodespaceaswellas
KernelModedrivers(typicallyendingina.sysfileextension).

whathappenstoasystemwhenthesePoolResourcesgetdepleted?Someofthemostcommonsymptomsexhibited
are:

(i)Themachinebecomessluggish
(ii)Userscannolongerlogontothemachine
(iii)Consoleaccessissluggish
(iv)Userscannotconnecttofilesharesorothersharedresources
(v)Systemhangsincludingtheconsoleitselfbeingunresponsive

Symptomssuchasthisareusuallythefirstindicationthatthereissomethingcausinganissuewiththemachine.

IftheNonPagedPoolonaserverhasbecomedepleted,themachinewillloganEventintheSystemLogasshown
below:

EventID2019
EventType:Error
EventSource:Srv
EventCategory:None
EventID:2019
Description:TheserverwasunabletoallocatefromthesystemNonPagedpoolbecausethepoolwasempty.

Thefollowingerrorstypicallyareanindicationthatthereisnofreeaddressspaceontheserver:

(i)Outofmemoryerrors(ORA4030)
(ii)Connectionspawningerrors(ORA12500)
(iii)Failuretospawnaprocesssuchasapqslave.

CAUSE

Thefollowingarethepossiblecausesforthisproblem:

1]Afaultyapplicationwhichiscausingthehandlestoincrease.

2]Afirewallexistbetweentheapplicationandthedatabaseserver.Theapplicatonontheremoteclient/serveris
failingtocontinuetheconnectionthroughthefirewall.Connectionsreachthedatabasebutdonotseemtoreturnand
hencehungsessionslingerasorphanedhandles.

SOLUTION

Thefollowingactionmayhelpyoutofixthisproblem:

1]Stoptheapplicationconnectingtothisdatabaseandmonitortheoracleprocessforthehandlecountandthe
Nonpagedpoolsize.Ifthehandlecountandthenonpagedpoolsizedoesnotincreasedrasticallythenwecan
confirmthattheproblemiswiththeapplication.

2]Checktoseeifthefirewallexitsbetweentheclientapplicationandthedatabasesever.Involveyour
Network/Firewalladminandenusrethatthereisa2waycommunicatonbetweentheapplicationandthedatabase
server.