KnowledgeBase

PoornetworkperformanceorhighnetworklatencyonWindowsvirtualmachines(2008925)
Symptoms
AvirtualmachinewithonevirtualCPUandahighCPUload,oravirtualmachinewithtwoormorevirtualCPUsingeneralandaWindows2003/2008/7guestoperatingsystem,mayexperiencethesesymptoms:
Poornetworkperformanceand/orhighpingresponsetimes:
Whenreceivingnetworktraffic(regardlessoftheamountofdataandtype)
WhileunderhighCPUload,orsharingCPUresourceswithhighlyutilizedvirtualmachines
Observedthroughputmaydecreaseto512kB/sonGigabitEthernet.Timeoutsandconnectivitydisruptionsmayalsobeobserved.
Pingrepliesmaytakeupto20seconds.
Sensitiveserviceslikedatabaseserversmayperformpoorlyortimeout.
Thenumberofvirtualandphysicalnetworkcardshasnoeffectonthisissue.
Thisissueoccurswithdifferentvirtualnetworkadaptertypes(E1000,VMXNET2andVMXNET3).
Measuredperformanceresults(generatedwithtoolslikeiperf)mayworsenwhenaddingmorevirtualCPUstothevirtualmachine.

Cause
Therearethreepossiblecausesforthisissue:
Powerplan
OnWindows2008and2008R2,thepowerplanissettoBalancedbydefault.MicrosofthasobservedandconfirmedthatchangingthepowerplanfromBalancedtoHighPerformancemayincreaseoverall
performance.Formoreinformation,seeGeneralGuidelinesforImprovingOperatingSystemPerformance(http://msdn.microsoft.com/enus/library/ee377075.aspx).Aggressivepowersavingplanscanadverselyaffect
performance,especiallywithlatencysensitiveapplicationslikewebanddatabaseservers.
HighCPUreadytime(alsoreferredas"%RDY"or"%RDYtime")
Note:Thisinformationissimplifiedanditsonlypurposeistoillustratethecauseofthedescribedissue.Itshouldnotbereferencedoutsideofthecontextofthisdocument.Althoughtheexampleweusehere
issufficienttodescribethecauseofthestatedissue,itdoesnotclaimtobetechnicallycorrectineverydetailduetothecomplexityoftheCPUschedulingprocessforvirtualmachines.
Thisexampleisbasedontheseassumptions:
AnESXi5.0hostwithahyperthreadingenabledquadcoreCPU,resultingin4physicaland8logicalCPUs
AWindows2008R2virtualmachinewith4virtualCPUs
ThreeWindows2003virtualmachineswith2virtualCPUseach
Inthisconfiguration,theESXihostexposes8logicalCPUsas10virtualCPUstothevirtualmachines.Inotherwords,theESXihostisovercommitted.Dependingontheutilizationofthevirtualmachines,
theESXihostwillnotbeabletoprovideallvirtualmachineswiththerequestedCPUtime,thustheperformanceofthevirtualmachineswillbeasexpected.However,iftheloadonmultiplevirtualmachines
increases,theESXihosthastodecidewhichvirtualmachinewillbeservedfirstwiththecurrentlyavailableCPUtime.ItisimportanttonotethattheESXihostwillservemulticorevirtualmachinesonly
whenitisabletoserveallthevirtualCPUsfromtheparticularvirtualmachineatonce.OtherwiseavirtualmachinewithalowernumberofvirtualCPUswillbeservedfirst.AlthoughthisishowtheCPU
schedulerissupposedtowork,itcanalsoleadtosituationswherecertainvirtualmachineshavetowaitforanunreasonableamountoftimefortherequestedCPUtime.Insuchcasesyoucanobservea
degradedoverallperformanceandincreasedresponsetimes.FormoreinformationaboutCPUschedulingandthemeaningoftheCPU%RDYtime,seetheVMwareTechnicalPaper,Performance
TroubleshootingforvSphere4.1(http://www.vmware.com/resources/techresources/10179).
ReceiveSideScaling(RSS)
RSSisamechanismwhichallowsthenetworkdrivertospreadincomingTCPtrafficacrossmultipleCPUs,resultinginincreasedmulticoreefficiencyandprocessorcacheutilization.Ifthedriverorthe
operatingsystemisnotcapableofusingRSS,orifRSSisdisabled,allincomingnetworktrafficishandledbyonlyoneCPU.Inthissituation,asingleCPUcanbethebottleneckforthenetworkwhileother
CPUsmightremainidle.
Note:TomakeuseoftheRSSmechanism,thehardwareversionofthevirtualmachinemustbe7orhigher,thevirtualnetworkcardmustbesettoVMXNET3,andtheguestoperatingsystemmustbecapable
andconfiguredproperly.Onsomesystemsithastobeenabledmanually.TheseoperatingsystemsarecapableofusingRSS:
Windows2003SP2(enabledbydefault)
Windows2008(enabledbydefault)
Windows2008R2(enabledbydefault)
WindowsServer2012(enabledbydefault)
Linux2.6.37andnewer(enabledbydefault)
ForfurtherinformationonMicrosoftRSS,seeReceiveSideScaling(RSS)(http://technet.microsoft.com/enus/library/hh997036.aspx).FormoreinformationonLinuxReceiveSideScaling,seeRSSandmultiqueue
supportinLinuxdriverforVMXNET3(2020567)(/selfservice/search.do?cmd=displayKC&docType=kc&docTypeID=DT_KB_1_1&externalId=2020567).

Resolution
Beforeproceedingwiththesesteps,ensurethat:

therearenoproblemsinyouexternalinfrastructurelikefaultyhardwareorpossiblemisconfigurations(commonconfigurationproblemsareIPconflicts,unintendedtrafficshaping,misconfiguredtrunkand
EtherChannelports)
thenetworkisnotcongested
thenetworktheESXihostisonisstableandperformsasexpected
thevirtualmachinesareconfiguredwiththeVMXNET3networkadapter
thehardwaredriversandfirmwareversionsarerecent
theBIOSisrecentandconfiguredappropriately
thevirtualmachineisrunningthelatestversionofVMwareTools(theycontainthedriversforthevirtualhardware)
anysecuritysoftwarelikeintrusiondetection/preventionsystemsorpacketinspectorshaveenoughresourcesavailableandareconfiguredcorrectly(checkthelogsforincorrectlyfilteredtrafficordropped
packets)
Afteryouhaveconfirmedthatyourinfrastructureishealthyandallcomponentsareconfiguredcorrectly,checkthepowersavingconfiguration.ForvirtualmachineswithmorethanonevirtualCPU,alsocheckifhigh
CPU%RDYtimeshaveanegativeimpactonthesevirtualmachines.
ThefinalstepistochecktheRSSsettings.ChangingtheRSSsettingsshouldonlybedonebytrainednetworkadministrators.VMwarealsorecommendsconfirmingthatallrelevantapplications(includingthe
operatingsystem)supportchangestotheRSSconfiguration.

Powerplan
Toensurethatthesystemtakesadvantageoftheavailableresources,itisimportanttodisableallpowersavingfeatureswhileinvestigatingperformanceissues.Ifthepowersavingconfigurationappearstoberelatedto
theperformanceproblems,acustomizedpowerplanbasedontheperformanceandpowersavingrequirementsshouldbecreated.Ifyouareunsureaboutwhichpowersavingconfigurationisrecommendedforyour
system,engageyourhardwarevendor.
ToadjustthepowerplansettingsonaWindows2008Server:
1. ClickStart,typepowercfg.cpl,andpressEnter.
2. EnsurethattheHighperformanceoptionisselected.
Note:Steps3through6areoptional.
3. ClickChangeplansettings.
4. ClickChangeadvancedpowersettings.
5. Toenableaccesstoallsettings,clickChangesettingsthatarecurrentlyunavailable.
6. Browsetheavailablesettingsandadjustasnecessary.
7. ClickOKtoconfirmandcloseallwindows.
Note:Somechangesmightrequirearebootoftheguestsystem.
CheckingCPU%RDYtimes
TodetermineifavirtualmachineisimpactedbyhighCPU%RDYtimes,useoneofthesemethods:
CountallvirtualCPUsonaparticularhostorcluster,anddividebythenumberoflogicalCPUs.Aresultofoneorhighermeansthatthehostorclusterisovercommittedandshouldbeinvestigated.Valuesof
fourorhigherareconsideredoverloadedandmustbeinvestigatedimmediately.
Notes:
Theintentofthismethodistoquicklydetermineifahostisovercommitted,ratherthandeterminingifitisnot.VMwarerecommendsusingesxtoptoobservedetailedhostperformance.
Althoughhyperthreadingdoublesthenumberoflogicalprocessors,itcannotprovidethesameperformanceastwophysicalprocessorcores.Ifitislikelythatthehostisovercommitted,calculate
usingthenumberofphysicalCPUs,ratherthanlogicalCPUs.
TheesxtopcommanddisplaysthevaluesfortheCPU%RDYtimewhenrunonthehostwiththeaffectedvirtualmachines.Formoreinformationonhowtouseandinterprettheoutputofesxtop,seethe
WorldStatisticssectionintheInterpretingesxtopStatistics(http://communities.vmware.com/docs/DOC9279)Communitiesdocument.Youcanalsorunresxtop,whichisprovidedwiththevSphereManagement
Assistant(vMA).
Thevmsupportcommandprovidesthecapabilitytocreateperformancesnapshots.Formoreinformation,seeCollectingperformancesnapshotsusingvmsupport(1967)(/selfservice/search.do?
cmd=displayKC&docType=kc&docTypeID=DT_KB_1_1&externalId=1967).
Torelieveanovercommittedhost,useoneofthesemethods:
Movetheaffectedvirtualmachinetoahostwithmoreavailableresources
Moveothervirtualmachinesoffthehost
DecreasethenumberofvirtualCPUsontheaffectedvirtualmachine
Note:ChangingtheCPUcountmightnotbesupportedbytheguestoperatingsystem.Formoreinformation,contacttheoperatingsystemvendor.

EnablingandconfiguringReceiveSideScaling(RSS)
BeforeenablingRSS:
EnsurethatthehardwareversionofthevirtualmachineissettoVersion7orhigher.Formoreinformation,seeVirtualmachinehardwareversions(1003746)(/selfservice/search.do?
cmd=displayKC&docType=kc&docTypeID=DT_KB_1_1&externalId=1003746).
EnsurethatthevirtualnetworkadapterissettoVMXNET3andthattheoperatingsystemissupportedbythisadapter.Formoreinformation,seeChoosinganetworkadapterforyourvirtualmachine
(1001805)(/selfservice/search.do?cmd=displayKC&docType=kc&docTypeID=DT_KB_1_1&externalId=1001805).
EnsurethatRSSisenabledintheguestoperatingsystem.ToverifythisinaWindowsguestoperatingsystem,openacommandpromptandrunthecommand:
netshinttcpshowglobal
TheoutputindicateswhetherReceiveSideScalingStateisenabledornot.
EnsurethatthenetworkadapterinthevirtualmachineisconfiguredtouseRSS.ToverifythisinaWindowsguestoperatingsystem:
1. OpentheDeviceManager,navigatetoNetworkadapters,andrightclicktheadapteryouwishtoenableRSSon.
2. InthePropertieswindow,clicktheAdvancedtab,thenclickRSSinthelistontheleftside.
3. ChangetheValuetoEnabledandclickOKtoclosethewindow.Arebootmightbenecessaryforthechangestotakeeffect.

Note:Enabling/disablingtheRSSfeatureinterruptsthenetworkconnectionontheadapterforseveralseconds.Ifyouareaccessingthesystemviaaremotedesktopsession,ensurethatyoucanaccess
thesysteminanotherwayincaseanissueoccursthatcausesthenetworkconnectiontonotreturn.
Bydefault,WindowsusesuptofourCPUsforRSS.
FormoreinformationonhowtoconfigureCPUaffinitiesorthenumberofCPUsusedforRSS,seetheMSDNarticleSettingtheRSSCPUConfiguration(http://msdn.microsoft.com/enus/library/ff570796%28v=VS.85%29.aspx)
.
FormoreinformationonenablingordisablingRSS,seetheMicrosoftSupportArticleInformationabouttheTCPChimneyOffload,ReceiveSideScaling,andNetworkDirectMemoryAccessfeaturesin
WindowsServer2008(http://support.microsoft.com/kb/951037).
Note:TheprecedinglinkswerecorrectasofDecember10,2013.Ifyoufindalinkisbroken,providefeedbackandaVMwareemployeewillupdatethelink.
Note:TSO(TCPSegmentationOffload)isafeatureofsomeNICsthatoffloadsthepacketizationofdatafromtheCPUtotheNIC.TSOissupportedbytheE1000,EnhancedVMXNET,andVMXNET3virtual
networkadapters(butnotbythenormalVMXNETadapter).InESXi,TSOisenabledbydefaultintheVMkernel,butissupportedinvirtualmachinesonlywhentheyareusingtheVMXNET3device,theEnhanced
VMXNETdevice,ortheE1000device.TSOcanimproveperformanceeveniftheunderlyinghardwaredoesnotsupportTSO.

AdditionalInformation
Forrelatedinformation,see:
PerformanceBestPracticesforVMwarevSphere5.0(http://www.vmware.com/pdf/Perf_Best_Practices_vSphere5.0.pdf)
ReceiveSideScaling(http://technet.microsoft.com/enus/network/dd277646)(MicrosoftTechNetarticle)
ReceiveSideScaling(http://msdn.microsoft.com/enus/library/ff567236%28v=VS.85%29.aspx)(MSDNLibraryarticle)
DegradedoverallperformanceonWindowsServer2008R2(http://support.microsoft.com/kb/2207548)(Microsoftsupportarticle)
Windows2008/2008R2defaultPowerPlanof"Balanced"canincreaselatencyandreducethroughput(http://web.archive.org/web/20130411021814/http://windowsazurecat.com/2011/06/windows20082008r2defaultpowerplanofbalancedcan
increaselatencyandreducethroughput/)(archivedcopyofMicrosoftCustomerAdvisoryTeamarticle)
Note:TheprecedinglinkswerecorrectasofFebruary5,2014.Ifyoufindalinkisbroken,providefeedbackandaVMwareemployeewillupdatethelink.
Ifyouobservedroppedpacketsinesxtop,seeTheoutputofesxtopshowsdroppedreceivepacketsatthevirtualswitch(1010071)(/selfservice/search.do?cmd=displayKC&docType=kc&docTypeID=DT_KB_1_1&externalId=1010071).

SeeAlso
Choosinganetworkadapterforyourvirtualmachine(1001805)(http://kb.vmware.com/kb/1001805)
Virtualmachinehardwareversions(1003746)(http://kb.vmware.com/kb/1003746)
TroubleshootingnetworkperformanceissuesinavSphereenvironment(1004087)(http://kb.vmware.com/kb/1004087)
Theoutputofesxtopshowsdroppedreceivepacketsatthevirtualswitch(1010071)(http://kb.vmware.com/kb/1010071)
CollectingperformancesnapshotsusingvmsupportinESXandESXi(1967)(http://kb.vmware.com/kb/1967)
RSSandmultiqueuesupportinLinuxdriverforVMXNET3(2020567)(http://kb.vmware.com/kb/2020567)
Windows(2093679)(http://kb.vmware.com/kb/2093679)
Windows(2097316)(http://kb.vmware.com/kb/2097316)

UpdateHistory
11/14/2013AddedESXi5.5toProducts
12/10/2013AddedMicrosoftlinktoenableordisableRSS

RequestaProductFeature
TorequestanewproductfeatureortoprovidefeedbackonaVMwareproduct,pleasevisittheRequestaProductFeature(http://www.vmware.com/contact/contactus.html?department=prod_request)page.
Permalinkto:PoornetworkperformanceorhighnetworklatencyonWindowsvirtualmachines(http://kb.vmware.com/kb/2008925)
Readourblog(http://blogs.vmware.com/kb/)

WatchKBTV(http://blogs.vmware.com/kbtv/)

Followus(http://www.twitter.com/vmwarekb)

request.html)

1(javascript:void(rate(1,this)))
2(javascript:void(rate(2,this)))
3(javascript:void(rate(3,this)))
4(javascript:void(rate(4,this)))
5(javascript:void(rate(5,this)))
39Ratings

Actions
BookmarkDocument(javascript:addBookmark(actionBoxUtil.docURL,actionBoxUtil.docTitle))
PrintDocument(javascript:window.print())
SubscribetoDocument(/selfservice/microsites/microsite.do?cmd=displayKC&docType=kc&docTypeID=DT_KB_1_1&externalId=2008925&format=rss)
(http://www.addthis.com/bookmark.php?v=250&username=xa4b5f42f36e60a29e)

KB:2008925
Updated:Apr25,2014
Categories:
Howto
Troubleshooting
Languages:
English
Product(s):
VMwareESX

RequestNewContent(http://www.vmware.com/landing_pages/knowledgebasecontent

VMwareESXi
ProductVersion(s):
VMwareESX3.5.x
VMwareESX4.0.x
VMwareESX4.1.x
VMwareESXi3.5.xEmbedded
VMwareESXi3.5.xInstallable
VMwareESXi4.0.xEmbedded
VMwareESXi4.0.xInstallable
VMwareESXi4.1.xEmbedded
VMwareESXi4.1.xInstallable
VMwareESXi5.0.x
VMwareESXi5.1.x
VMwareESXi5.5.x