6/12/2017 AIXforSystemAdministrators

AIX for System Administrators

PracticalGuidetoAIX(andLinuxonPower)

HOME FSLVM GENERAL HMC NETWORKS NIM PERFORMANCE STORAGEBACKUP UPD.INSTALL POWERHA POWERVM EXTRA

CLICKADTOSUPPORT:
IOSTATFCSTAT:

IOPS(I/Opersecond)foradiskislimitedbyqueue_depth/(averageIOservicetime).Assumingaqueue_depthof3,andanaverageIO SEARCHONTHISBLOG:
servicetimeof10ms,thisequalsto300IOPSforthehdisk.Andformanyapplicationsthismaynotbeenoughthroughput.
Search
IOisqueuedateachlayer,wheretravels:
filesystemfilesystembuffer
hdiskqueue_depths
adapternum_cmd_elems

ABOUT
AdapterI/O:

TherearenoadapterstatsinAIX.Theyarederivedfromthediskstats.Theadapterbusy%issimplythesumofthediskbusy%.
FSLVM
Soiftheadapterbusy%is,forexample,350%thenyouhave3.5disksbusyonthatadapter.Oritcouldbe7disksat50%busyor14
disksat25%or.... FSFilesystem
LVLogicalVolume
Thereisnowaytodeterminetheadapterbusyandinfactitisnotclearwhatitwouldreallymean.Theadapterhasadedicatedonboard PVPhysicalVolume
CPUthatisalwaysbusy(probablynorealOS)andwedon'trunnmonoftheseadapterCPUstofindoutwhattheyarereallydoing.
VGVolumeGroup

MirrorPool
SnapshotFS
IOSTAT: SnapshotVG/LV

iostatsprinttheSystemthroughputreportsinceboot
iostatdhdisk15displayonly1diskstatistics(hdisk1) GENERAL
iostata5showsadapterstatisticsaswell
AIXHistory
Backup

iostatDRTl52shows2statisticsin5secondsinterval Commands
Dextendedreport CPUProcesses
Rresestmin.maxvaluesateachinterval CrontabAt
Taddstimestamp DateTime
llonglistingmode Devices
DumpCore

Disks:xfersreadwrite ErrptDiagAlog
queue Firmware
IOAIO,DIO,CIO
MemoryPag.Space
%tmbpstpsbreadbwrtnrpsavgmaxtimefailwpsavgmaxtimefailavgminmaxavgavg ODM
serv
Printing
actservservoutsservservoutstimetimetimewqszsqsz
SRC
qfull
hdisk413.9212.2K13.9156.7K55.5K9.614.940.4004.40.61.1000.00.00.00.00.0 StartupShutdown
0.0 Syslogd
hdisk520.1246.4K16.5204.8K41.6K12.416.249.8004.20.61.3000.00.00.00.00.0 SystemKernel
0.0 Tunables
hdisk619.7282.4K17.1257.9K24.5K14.713.637.3002.40.61.1000.00.00.00.00.0
UserAuthentication
0.0
UserGroup
hdisk718.7300.3K20.3215.4K84.9K13.113.939.7007.20.75.6000.00.00.00.00.0
0.0 UserLogin
UserPassword

xfers(transfers): HMC

ASMI
%tmact:percentoftimethedevicewasactive(wecanseeifdiskloadisbalancedcorrectlyornot,1usedheavilyothersnot)
DPO
bps:amountofdatatransferred(readorwritten)persecond(defaultisinbytespersecond)
tps:numberoftransferspersecond HMCBasicsConsole
(TransferisanI/Orequest,andmultiplelogicalrequestscanbecombinedinto1I/Orequestwithdifferentsize) HMCCLI
bread:amountofdatareadpersecond(defaultisinbytespersecond) HMC/P7Install
bwrtn:amountofdatawrittenpersecond(defaultisinbytespersecond) HMCUpd./Upgr.
ISDBasics
%tm_act:
RMC
TheOShasasensor,regularilyaskingthediskifitisbusyornot.Whenthedisksanswershalfofthetimes"I'mbusy",thenthe"%
tm_act"willbe50%.Ifthediskanswerseverytime"I'mbusy"thentm_actwillbe100%,etc..Adiskanswerswith"busy",whenthereare
requestedoperationsnotyetfulfilled,readorwrite.Ifmanyverysmallrequestsgotothediskthechanceofthesensoraskingexactly
NETWORK
whenonesuchoperationisstillopengoesupmuchmoresothantherealactivityofthedisk.
BasicsDevices,
So,"100%busy"doesnotnecessarilymeanthediskisattheedgeofitstrasnferbandwidth.Itcouldmeaneitherthatbecausethedisk Routing
isgettingrelativelyfewbutbigrequests(example:streamI/O)butitcouldalsomeanthatthediskisgettingalotofrequestswhich
arerelativelysmallsothatthediskisoccupiedmostofthetime,butnotusingitscompletetransferbandwith. BasicsProtocol,
Tofindoutwhichisthecaseanalysethecorresponding"bread"and"bwrtn"columnfromiostat. Subnet
BasicsVlan
Commands
DSHPSSH
read/write:
Eth.Chan.

rps/wps:numberofread/writetransferspersecond. EthernetAdapter
avgserv:averageservicetimeperread/writetransfer(defaultisinmilliseconds) IVEHEA
timeouts:numberofread/writetimeoutspersecond Netcd
fail:numberoffailedread/writerequestspersecond NFS
RSHRCP

Sendmail
SSHSCP
queue(waitqueue):
SSHX11
avgtime:averagetimespentinthewaitqueue(waitingtogetsenttothedisk,thedisk'squeueisfull)(defaultisin TelnetFTP

http://aix4admins.blogspot.in/2013/07/iostatfcstatiopsiopersecondfor.html 1/4
6/12/2017 AIXforSystemAdministrators
avgtime:averagetimespentinthewaitqueue(waitingtogetsenttothedisk,thedisk'squeueisfull)(defaultisin TelnetFTP
milliseconds)
avgwqsz:averagewaitqueuesize(waitingtobesenttothedisk)
avgsqsz:averageservicequeuesize(thiscan'texceedqueue_depthforthedisk) NIM
sqfull:numberoftimestheservicequeuebecomesfullpersecond(thatis,thediskisnotacceptinganymoreservicerequests) Basics
BootpdTftpd

Client(Install)
Fromtheapplication'spointofview,thelengthoftimetodoanIOisitsservicetimeplusthetimeitwaitsinthehdiskwaitqueue. Client(Machines)
Timespentinthequeueindicatesincreasingqueue_depthmayimproveperformance,forcorrecttuningcheckmaximumnumbersaswell. LPPSource
Ifavgwqszisoften>0,thenincreasequeue_depth MKSYSB
Ifsqfullinthefirstreportishigh,thenincreasequeue_depth Nimadm
SPOT
Whenyouincreasethequeue_depths(somoreIOaresenttothedisk),theIOservicetimesarelikelytoincrease,butthroughputwill
alsoincrease.IfIOservicetimesstartapproachingthedisktimeoutvalue,thenyou'resubmittingmoreIOsthanthediskcanhandle:
PERFORMANCE
#lsattrElhdisk400|greptimeout
rw_timeout40READ/WRITEtimeoutvalueTrue Basics
CPUGeneral
Ifread/writemaxservicetimesare30or60secondsorclosetowhattheread/writetimeoutvalueissetto,thislikelyindicatesa CPUVirtualization
commandtimeoutorsometypeoferrorrecoverythediskdriverexperienced.
I/OLVM
I/Oiostat,fcstat

Netw.General
Agoodgeneralrulefortuningqueue_depths,istotuneuntiltheseratesareachieved: Netw.SEA,vEth.
averagereadservicetime~15ms RAMGeneral
withwritecache,writeaverage~2ms(writestypicallygotowritecachefirst) RAMsvmon
RAMVMM
Typicallyforlargedisksubsystemsthataren'toverloaded,IOservicetimeswillaveragearound510ms.Whensmallrandomreadsstart
Toolstopas,nmon
averaginggreaterthan15ms,thisindicatesthestorageisgettingbusy.
Toolsvmstat

STORAGEBACKUP
Fortuning,wecansetupthesecategories:
Adapter
1.We'refillingupthequeuesandIOsarewaitinginthehdiskoradapterdrivers BasicsSAN
2.We'renotfillingupthequeues,andIOservicetimesaregood
BasicsSettings
3.We'renotfillingupthequeues,andIOservicetimesarepoor
EMC
4.We'renotfillingupthequeues,andwe'resendingIOstothestoragefasterthanitcanhandleanditlosestheIOs
Hitachi
#2:wewanttoreachthis HPEVASSA
#3:indicatesbottleneckbeyondhdisk(probablyinadapter,SANfabricoratstorageboxside) MPIO
#4:shouldbeavoided(ifstoragelosesIOs,atthehostIOwilltimeout,withrecovercodeitwillberesubmitted,inthemeantimeappl. SDD
iswaitingforthisIO) SDDPCM
TapeDevices

Generalruleofthumb,if%tm_actgreaterthan70%,thanprobablybettertomigratesomethingtootherdisksaswell.
UPDATEINSTALL
Movingdatatolessbusydrivescanobviouslyhelpeasethisburden.Generallyspeaking,themoredrivesthatyourdatahits,thebetter.
Basics
%iowait:percentageoftimetheCPUisidleANDthereisatleastoneI/Oinprogress(allCPUsaveragedtogether) Commands
(HighI/OwaitdoesnotmeandefinitelyI/Obottleneck.ZeroI/OwaitdosnotmeanthereisnoI/Pbotleneck.) IFIX
%iowait>25%systemisprobablyI/Obound.

POWERHA

Appl.Monitor
Basics
BuildConfigure
CAA
FCSTAT:
Clverify
CommandsCases

#fcstatfcs0 Configtoolong
DARESnapshot
FIBRECHANNELSTATISTICSREPORT:fcs0 DiskHeartbeat
... NetworkTopology
WorldWideNodeName:0x20000000C9F170E8<WWPNnumber StorageVG,NFS
WorldWidePortName:0x10000000C9F170E8
...
PortSpeed(supported):8GBIT<FCadapterspeed POWERVM
PortSpeed(running):8GBIT
... Basics
ErrorFrames:0<bothofthemaffectIO,whenframesaredamagedor Commands
DumpedFrames:0<framesarediscarded AMEAMS

... CDROMDVDROM
FCSCSIAdapterDriverInformation LPM
NoDMAResourceCount:37512<IOsqueuedattheadapterduetolackofresources(max_xfer_size) NIBLA
NoAdapterElementsCount:104848<numberoftimessinceboot,anIOwastemporarilyblockedduetoaninadequatenum_cmd_elems
NPIV
value
NoCommandResourceCount:13915968<sameasabove,nonzerovaluesindicatethatincreasingnum_cmd_elemsmayhelpimproveIO SEA
servicetimes SharedStoragePool
... SRIOV
vNIC
VIOSupdate/upgrade
#lsattrElfcs0|egrep'xfer|num|dma'
Virt.Eth.Ad.
lg_term_dma0x800000LongtermDMATrue
Virt.Proc.Ent.Cap.
max_xfer_size0x100000MaximumTransferSizeTrue
num_cmd_elems2048MaximumnumberofCOMMANDStoqueuetotheadapterTrue VSCSI
VSCSIStor.Pool
Whenthedefaultvalueisused(max_xfer_size=0x100000)thememoryareais16MBinsize.Whensettingthisattributetoanyother
allowablevalue(say0x200000)thenthememoryareais128MBinsize.AtAIX6.1TL2orlaterachangewasmadeforvirtualFCadapters
sotheDMAmemoryareaisalways128MBevenwiththedefaultmax_xfer_size.ThismemoryareaisaDMAmemoryarea,butitisdifferent LINUX
thantheDMAmemoryareacontrolledbythelg_term_dmaattribute(whichisusedforIOcontrol).Thedefaultvalueforlg_term_dmaof
Install
0x800000isusuallyadequate.

SoforheavyIOandespeciallyforlargeIOs(suchasforbackups)it'srecommendedtosetmax_xfer_size=0x200000. +EXTRAS
Likethehdiskqueue_depthattribute,changingthenum_cmd_elemsvaluerequiresstoppinguseoftheresourcesorareboot.
+others
alt_disk

Labels:PERFORMANCE awksed
bash
citrix

11comments: gpfsspectrumscale
java
kdb
Anonymous August26,2013at6:00AM
ksh
HI linuxinstall
http://aix4admins.blogspot.in/2013/07/iostatfcstatiopsiopersecondfor.html 2/4
6/12/2017 AIXforSystemAdministrators
HI linuxinstall
locale
CouldyouexplainGPFSconcept
multibos
Reply
perl
profiles
Replies
rbac

HabibKhilji December11,2013at10:58PM Rsh

samba
Checkthefollowinglinkandalsogoogle.
http://www03.ibm.com/systems/software/gpfs/ screen
script

Reply sudo
syslogng
vi

RajeshMohan December5,2013at12:45PM

Hi,
Canyoupleasehelpmetofindhowmanyethcards&fccardsihaveineachvioserverbasedonbelowinfo?

VioS1:
ent0Available02004PortGigabitEthernetPCIExpressAdapter(e414571614102004)
ent1Available02014PortGigabitEthernetPCIExpressAdapter(e414571614102004)
ent2Available02024PortGigabitEthernetPCIExpressAdapter(e414571614102004)
ent3Available02034PortGigabitEthernetPCIExpressAdapter(e414571614102004)

fcs0Available03008GbPCIExpressDualPortFCAdapter(df1000f114108a03)
fcs1Available03018GbPCIExpressDualPortFCAdapter(df1000f114108a03)

VioS2:
fcs0Available01008GbPCIExpressDualPortFCAdapter(df1000f114108a03)
fcs1Available01018GbPCIExpressDualPortFCAdapter(df1000f114108a03)

Reply

Replies

NaunihalSingh December30,2013at8:18AM

You'vefollowing

VIOSServer1:Youhave1QuadportEthernetCardand1DualportFCCard

VIOSServer2:Youhaveonly1DualportFCCard

Reply

PhuongPhmMinh September16,2014at3:18AM

Greatblog.Likethisblog.Thankyousomuch^^^

Reply

Kapalin October7,2014at7:47AM

Nicelyexplained,Thanksalotforthisinformation

Reply

DAC April6,2015at6:07PM

Hello,verygoodarticle...Infact,Ijustsolvemyproblems...
Onenoteon"max_xfer_size":Ifyou'reusingNPIV,YOUHAVETOCHANGETHEVALUEONTHEVIOSSERVERFIRST,thenreboot(orrmdevRlandcfgmgr),
andthen,changeitontheClientLPAR.Ifyoufailtodothisinthisorder,youmayfindyourselfwith"cfgmgr"errorsorLED554ifyouboot
theLPAR.

Thanks,Chris'sAIXBlog(https://www.ibm.com/developerworks/community/blogs/cgaix/entry/multibos_saved_my_bacon1?lang=es).

Reply

Replies

aix April7,2015at9:29AM

Hi,thanksforyourvaluableinfo.

Durga September29,2015at10:33AM

Hi,

Ihavequery,I'mhaving2VIOS&8LPAR'sinframe

2VIOSupdatingthelg_term_dma,max_xfer_size&rebooting

7LPAR'supdatingthelg_term_dma,max_xfer_size&rebooting

1LPARNOTUPDATINGthevaluesoflg_term_dma,max_xfer_size

So,theupdatedvaluesofVIOSwillnoteffecttheLPAR,whichwasnotupdated.

Thanks&Regards,

Reply

AttilaZsmboki April8,2015at11:08AM

Verygooddeepdiveinfos.Thankyou.

Reply

Replies

aix April9,2015at12:42PM

szvesen:)

Reply

http://aix4admins.blogspot.in/2013/07/iostatfcstatiopsiopersecondfor.html 3/4
6/12/2017 AIXforSystemAdministrators

NewerPost Home OlderPost

Subscribeto:PostComments(Atom)

aix4admins.blogspot.com(2015)Unauthorizeduseofthismaterialisstrictlyprohibited..ThemeimagesbyStorman.PoweredbyBlogger.

http://aix4admins.blogspot.in/2013/07/iostatfcstatiopsiopersecondfor.html 4/4