Monitoring Linux IO

Monitoring IO performance
using iostat & pt-diskstats

MySQL Conference & Expo 2013
Ben Mildren
MySQL Team Technical Lead
About Me
Ben Mildren
Team Technical Lead at Pythian
Over 10 years experience as a production

DBA.
Experience of MySQL, SQL Server, and

MongoDB.
Email: mildren@pythian.com
LinkedIn: benmildren
Twitter: @productiondba
Slideshare: www.slideshare.net/benmildren
Why Pythian?
Recognized Leader:
Global industry-leader in remote database administration services and
consulting for Oracle, Oracle Applications, MySQL and Microsoft SQL Server
Work with over 250 multinational companies such as Forbes.com, Fox Sports,
Nordion and Western Union to help manage their complex IT deployments
Expertise:
Pythians data experts are the elite in their field. We have the highest
concentration of Oracle ACEs on staff10 including 2 ACE Directorsand 2
Microsoft MVPs.
Pythian holds 7 Specializations under Oracle Platinum Partner program,
including Oracle Exadata, Oracle GoldenGate & Oracle RAC
Global Reach & Scalability:

Around the clock global remote support for DBA and consulting, systems
administration, special projects or emergency response
Why the interest in IO monitoring?

LatencyComparisonNumbers
L1cachereference0.5ns
Branchmispredict5ns
L2cachereference7ns14xL1cache
Mutexlock/unlock25ns
Mainmemoryreference100ns20xL2cache,200xL1cache
Compress1KbyteswithZippy3,000ns
Send1Kbytesover1Gbpsnetwork10,000ns0.01ms
Read4KrandomlyfromSSD*150,000ns0.15ms
Read1MBsequentiallyfrommemory250,000ns0.25ms
Roundtripwithinsamedatacenter500,000ns0.5ms
Read1MBsequentiallyfromSSD*1,000,000ns1ms4Xmemory
Diskseek10,000,000ns10ms20xdatacenterroundtrip
Read1MBsequentiallyfromdisk20,000,000ns20ms80xmemory,20XSSD
SendpacketCA>Netherlands>CA150,000,000ns150ms
*Assuming~1GB/secSSD
Credit
ByJeffDean:http://research.google.com/people/jeff/
OriginallybyPeterNorvig:http://norvig.com/21days.html#answers
(https://gist.github.com/jboner/2841832)
Monitoring IO on Linux
There are a number of tools available to monitor IO on Linux. This

presentation looks at two tools, iostat and pt-diskstats. These tools
look to provide an overview of block devices iops, throughput and
latency.
You can dive deeper:
Tools such as iotop and atop can be used to expose process
level IO performance by gathering data from /proc/[process]/io.
blktrace can be used with blkparse, either independently or by
using btrace. Output can also be analysed using seekwatcher.
Tools such as bonnie/bonnie++, iometer, iozone, and ORION
can be used to benchmark a block device.
iostat
iostat is part of the sysstat utilities which is maintained by Sebastien

Godard.
sysstat is open source software written in C, and is available under
the GNU General Public License, version 2.
Other utilities in the sysstat package include mpstat, pidstat, sar,
nfsiostat, and cifsiostat.
sysstat is likely available from your favourite repo, but the latest
version can be found here:
http://sebastien.godard.pagesperso-orange.fr/download.html
If you have an old (or very old) version installed, it is recommended
you install the latest stable version.
iostat
At the time of writing the current stable version is 10.0.5.

Note version 10 removes support for kernels older than 2.6.
The version you have installed can be found with -V option.
[ben@lab~]$iostatV
sysstatversion10.0.3
(C)SebastienGodard(sysstat<at>orange.fr)
iostat
By default iostat produces two reports; the CPU Utilization report

and the Device Utilization report.
The default invocation shows the statistics since system start up.
[ben@lab~]$iostat
Linux3.8.4102.fc17.x86_64(lab.mysqlhome)
04/10/2013
_x86_64_
avgcpu:%user%nice%system%iowait%steal%idle
17.562.933.622.420.0073.47
Device:tpskB_read/skB_wrtn/skB_readkB_wrtn
sda8.51100.1744.62890987396917
dm04.8778.766.8070056560488
dm17.4020.7437.45184505333128
dm20.140.190.3716933300
(2CPU)
iostat
Inclusion of the CPU and Device Utilization reports can be

controlled with the -c and -d options.
The Network Filesystem report (-n) was deprecated in version 10
and replaced with the nfsiostat and cifsiostat utilities.
[ben@lab~]$iostatc
04/10/2013
_x86_64_
(2CPU)
_x86_64_
(2CPU)
17.012.663.692.350.0074.29
[ben@lab~]$iostatd
04/10/2013
Device:tpskB_read/skB_wrtn/skB_readkB_wrtn
sda8.2894.6045.60926915446773
dm04.6575.006.3473486162104
dm17.2819.0038.75186129379644
dm20.150.170.5117015024
iostat
The default Device Utilization report can be replaced with extended

statistics using the -x option.
[ben@lab~]$iostatcx|[ben@lab~]$iostatx
04/10/2013
_x86_64_
(2CPU)
16.232.273.672.200.0075.63
Device:rrqm/swrqm/sr/sw/srkB/swkB/savgrqszavgquszawaitr_awaitw_awaitsvctm%util
sda1.342.673.334.3981.1545.4432.810.5166.4129.5994.307.776.00
dm00.000.003.240.8264.105.7234.380.3791.2935.84309.174.942.01
dm10.000.001.345.7716.5439.1415.660.3143.2032.9545.596.424.56
dm20.000.000.050.100.150.589.560.0168.8126.9689.1416.180.25
[ben@lab~]$iostatdx
04/10/2013
_x86_64_
(2CPU)
sda1.342.673.324.3981.0945.4232.800.5166.4029.5994.277.776.00
dm00.000.003.240.8264.055.7134.380.3791.2935.84309.174.942.00
dm10.000.001.345.7716.5339.1215.660.3143.2032.9545.586.424.56
dm20.000.000.050.100.150.589.560.0168.7526.9689.0016.210.25
10
iostat
Whilst reviewing the stats since since system start up can be useful,
more often you will want to review current activity.
Current activity can be displayed by specifying an interval measured
in seconds.
Output will continue until interrupted or a specified count has been
reached.
[ben@lab~]$iostatdx[interval][count]
[ben@lab~]$iostatdx3
The above command will display extended device statistics since system start up in the first
report, and the deltas for the last 3 seconds in subsequent reports until interrupted.
The above command will display extended device statistics since system start up in the first
report, and the deltas for the last 3 seconds in two further reports.
11
iostat
04/10/2013
_x86_64_
(2CPU)
sda1.162.662.894.2770.3744.6632.100.4664.3729.5587.967.945.69
dm00.000.002.800.7855.535.2333.940.3290.1235.83286.385.031.80
dm10.000.001.175.6814.3938.7315.500.2942.1232.8144.056.384.37
dm20.000.000.040.110.130.7010.680.0163.3026.9677.3415.470.24
sda0.004.670.001.000.0021.3342.670.0223.330.0023.3323.332.33
dm00.000.000.000.000.000.000.000.000.000.000.000.000.00
dm10.000.000.005.330.0021.338.000.035.250.005.254.382.33
dm20.000.000.000.000.000.000.000.000.000.000.000.000.00
sda0.001.670.009.000.0041.339.190.4550.440.0050.446.525.87
dm00.000.000.000.000.000.000.000.000.000.000.000.000.00
dm10.000.000.008.330.0033.338.000.4654.920.0054.922.522.10
dm20.000.000.002.000.008.008.000.0945.500.0045.5018.833.77
12
iostat
13
Since version 7.1.3 the report can be made more readable by

including the registered device mapper names using the -N option.
Using the -N can skew the columns on each line, so it can be useful
to also specify the -h option to keep the report easily readable.
Adding -k or -m will specify kB / mB per second respectively.
(If the POSIXLY_CORRECT environment variable is NULL the data
read / written is displayed in kB by default)
Adding the -t option will include a timestamp with each report.
Adding the -z option will exclude any inactive devices from the
individual reports.
In version 10.1.3 (currently development version), adding the -y
option suppresses the first report showing statistics since system
start up.
iostat
[ben@lab~]$iostatdxNhtz32
04/10/2013
_x86_64_
(2CPU)
04/10/201311:37:08AM
sda
1.072.632.674.1765.0443.6031.760.4363.1729.5484.728.065.52
vg_proddbalv_root
0.000.002.590.7451.364.9433.800.3089.5135.81276.285.091.70
vg_proddbalv_home
0.000.001.085.5813.2737.9315.360.2841.3232.8242.976.394.26
vg_proddbalv_tmp
0.000.000.040.110.120.7311.040.0160.8426.9672.7015.280.23
04/10/201311:37:11AM
-t (timestamp reported with sample)
sda
0.003.330.003.330.0034.6720.800.0928.100.0028.1016.705.57
vg_proddbalv_home
0.000.000.006.000.0034.6711.560.1016.000.0016.009.285.57
-z (report only includes active devices for this sample)

-N (registered device mapper name)
-h (columns stay aligned even with long device names)
14
iostat
In version 10.0.5, the devices in the Device Utilization can be

grouped using the -g option.
It is possible to specify multiple groups by supplying the -g option
multiple times.
It's probably easier not to use the -N option, as the group devices
named would have to reference the registered device mapper
names.
Using the -T option will display the group totals only.
15
iostat
[ben@lab~]$iostatV
sysstatversion10.0.5
(C)SebastienGodard(sysstat<at>orange.fr)
[ben@lab~]$iostatdxgMyLVMdm0dm1dm2gOthersda
04/10/2013
_x86_64_
(2CPU)
dm00.000.002.280.7645.534.7333.150.2480.1232.12224.365.081.54
dm10.000.000.865.4010.5737.4515.360.2539.9632.6041.136.404.00
dm20.000.000.030.160.091.0211.630.0149.6426.9654.0014.570.28
MyLVM0.000.003.166.3156.1943.2020.970.5053.0132.2063.446.151.94
sda0.832.592.364.0756.4243.2030.960.3757.7326.6775.738.145.24
Other0.832.592.364.0756.4243.2030.960.3757.7326.6775.738.145.24
[ben@lab~]$iostatdxTgMyLVMdm0dm1dm2gOthersda
04/10/2013
_x86_64_
(2CPU)
MyLVM0.000.003.166.3156.1843.2020.970.5053.0132.2063.446.141.94
Other0.832.592.364.0856.4143.2030.960.3757.7326.6775.728.145.24
16
iostat
17
The Device Utilization report can be produced on a partition level

using the -p option.
Historically partition statistics were restricted when moving from the
2.4 kernel to 2.6 kernel, however the enhanced statistics were made
available again from the 2.6.25 kernel.
The -p option was mutually exclusive to the -x option up unto
version 8.1.8
The partition focused report, can be limited to specific devices but
not specific partitions.
Currently the group options (-g and -T) don't work well with the
partition focused report.
iostat
[ben@lab~]$iostatpdx
04/10/2013
_x86_64_
(2CPU)
sda5.404.9411.359.14398.2687.9947.462.33113.7556.49184.876.2312.76
sda10.030.000.230.000.930.008.080.0015.8815.7660.0015.870.36
sda20.020.000.170.000.780.008.930.0010.7810.780.0010.780.19
sda35.364.9410.848.19396.1287.9950.902.29120.1158.50201.695.219.92
dm00.000.009.892.11311.619.3753.512.86238.35119.45796.494.745.69
dm10.000.006.2111.3483.3178.0918.390.9051.5624.0466.645.199.10
dm20.000.000.250.130.940.537.680.0136.0727.5351.9911.560.44
[ben@lab~]$iostatpsdadx
04/10/2013
_x86_64_
(2CPU)
sda2.684.746.529.47235.6090.1140.731.4590.4850.74117.866.9211.07
sda10.010.000.110.000.450.008.080.0015.8815.7660.0015.870.18
sda20.010.000.080.000.370.008.930.0010.7810.780.0010.780.09
sda32.664.746.288.40234.5690.1144.231.4095.4552.19127.775.227.67
18
pt-diskstats
pt-diskstats is part of the percona toolkit.

pt-diskstats is open source software written in Perl, and is available
under the GNU General Public License, version 2.
Other utilities in the percona toolkit include pt-stalk, pt-tablechecksum, pt-table-sync, pt-query-digest, and pt-summary.
Percona toolkit can be downloaded from
http://www.percona.com/software/percona-toolkit/
or more simply from the command line:
wgetpercona.com/get/perconatoolkit.tar.gz
Tools can also be downloaded individually:

wgetpercona.com/get/TOOL
e.g. wgetpercona.com/get/ptdiskstats
19
pt-diskstats
At the time of writing the current stable version of percona toolkit is

version 2.1.1.
Full documentation of pt-diskstats can be found here:
http://www.percona.com/doc/percona-toolkit/2.1/pt-diskstats.html
The version of pt-diskstats you have installed can be found with
--version option.
[ben@lab~]$./ptdiskstatsversion
ptdiskstats2.2.1
20
pt-diskstats
By default pt-diskstats produces useful stats reporting current

device activity. Inactive devices are hidden, but subsequently
added if they become active.
The default invocation shows the statistics for the last interval (which
by default is 1 second), and will continue until interrupted.
Similar to: iostat -dxy 1
[ben@lab~]$./ptdiskstats
#tsdevicerd_srd_avkbrd_mb_srd_mrgrd_cncrd_rtwr_swr_avkbwr_mb_swr_mrgwr_cncwr_rtbusyin_prgio_sqtimestime
1.0sda0.00.00.00%0.00.01.08.00.050%0.00.04%11.05.817.5
1.0sda30.00.00.00%0.00.01.08.00.050%0.00.00%01.00.00.0
1.0dm20.00.00.00%0.00.03.04.00.00%0.00.04%13.02.911.7
1.0sda1.04.00.00%0.172.013.04.00.132%0.211.125%014.00.712.3
1.0sda31.04.00.00%0.172.07.07.40.146%0.01.49%08.00.06.4
1.0dm11.04.00.00%0.172.016.03.00.00%0.18.821%017.00.412.1
1.0dm20.00.00.00%0.00.00.00.00.00%0.10.04%00.00.00.0
1.0sda0.00.00.00%0.00.01.028.00.086%0.00.04%11.00.65.0
1.0sda30.00.00.00%0.00.01.028.00.086%0.00.00%01.00.00.0
1.0dm10.00.00.00%0.00.00.00.00.00%0.00.00%00.00.00.0
1.0dm20.00.00.00%0.00.00.00.00.00%0.00.00%00.00.00.0
21
pt-diskstats
The interval can be adjusted using the --interval option

The number of samples can be limited using the --iterations option.
Similar to: iostat -dxy 3 3
[ben@lab~]$./ptdiskstatsinterval3iterations3
3.0sda0.00.00.00%0.00.01.33.00.00%0.037.24%01.39.428.2
3.0sda30.00.00.00%0.00.01.04.00.00%0.022.72%01.00.022.7
3.0dm10.00.00.00%0.00.00.00.00.00%0.00.02%00.00.00.0
3.0dm20.00.00.00%0.00.00.74.00.00%0.030.02%00.70.030.0
3.0sda0.00.00.00%0.00.01.36.00.050%0.010.14%21.30.714.6
3.0sda30.00.00.00%0.00.01.08.00.057%0.01.11%11.00.86.3
3.0dm00.00.00.00%0.00.00.74.00.00%0.00.01%20.70.018.0
3.0dm10.00.00.00%0.00.01.34.00.00%0.00.01%11.31.89.0
3.0dm20.00.00.00%0.00.01.04.00.00%0.027.03%01.00.027.0
3.0sda0.00.00.00%0.00.02.33.40.00%0.143.45%02.326.420.0
3.0sda30.00.00.00%0.00.01.74.80.00%0.133.43%01.712.120.6
3.0dm00.00.00.00%0.00.00.34.00.00%0.1191.03%00.3211.092.0
3.0dm10.00.00.00%0.00.00.00.00.00%0.00.03%00.00.00.0
3.0dm20.00.00.00%0.00.00.74.00.00%0.023.52%00.70.023.5
22
pt-diskstats
23
Displayed columns can be restricted using perl regex with the

--columns-regex option.
Displayed devices can be restricted using perl regex with the
--devices-regex option.
Adding the --show-timestamps option will include a timestamp with
each report.
The output can be grouped by sample or by disk using the --groupby option.
Samples (of /proc/diskstats) can be saved for later analysis using
the --save-samples option.
Contrary to the documentation --version-check is enabled by
default, if you wear a tin foil hat, disable using --noversion-check.
The interlude: A primer in Linux IO
Read http://kernel.dk/blk-mq.pdf for recent development
24
A primer in Linux IO
Applications in the user space make requests to the VFS.

The request is passed to the block layer if the page is not in the
page cache or the request is made using direct IO.
Requests in the block layer can be split if the request is across
multiple devices, or remapped from a device partition to the
underlying block device.
Requests are handled by the IO scheduler on a per block device
basis. Dependent on the scheduler, the request could be merged to
the front or back of existing requests (all schedulers), or sorted in
the request queue (cfq & deadline). The anticipatory scheduler was
removed from the kernel in version 2.6.33.
Statistics are calculated at the block layer.
25
A primer in Linux IO
To understand what is acceptable performance there's no substitute

to understanding the block device hardware.
A block device is an abstraction, potentially it could be a 10 disk
array with a hardware RAID controller, it could be a LUN exposed
from a SAN, even if it's a single disk, the expected disk performance
could vary greatly dependent on it's specification.
Expected IOPs, latency and throughput can be gathered via
benchmarks or estimated using calculations:
http://www.techish.net/hardware/iops-calculator-and-raid-calculators-estimators/
http://www.wmarow.com/strcalc/
As a rough estimate you can expect:

75-100 iops from a 7200 rpm disk
125-150 iops from 10k rpm disk
175-200 iops from 15k rpm disk
1000's iops from SSD (++++++++++)
26
Block layer disk statistics
27
From the 2.6 kernel, statistics are held for all block devices and
partitions in /proc/diskstats.
/proc/diskstats lists the block devices major number, minor number,
and name as well as a statistic set of 11 counters.
Prior to 2.6.25 the statistic set of partitions was only made up of 4
counters and the counters weren't consistent with the underlying
block device statistics.
Statistics are also held for individual devices and partitions in sysfs.
/sys/block/[dev]/stat holds the statistic set for the device.
/sys/block/[dev]/[partition]/stat holds the statistic set for the device
partition.
/proc/diskstats (2.6.25+)
[ben@lab~]$cat/proc/diskstats
70loop000000000000
71loop100000000000
72loop200000000000
73loop300000000000
74loop400000000000
75loop500000000000
76loop600000000000
77loop700000000000
80sda447831547022573021210711859995422418089246087675010877637298349
81sda1463163417664642041062156465
82sda226731213641460000040534146
83sda343885152762249646119736973520542241808920557562006545526772954
110sr000000000000
2530dm042736017962261391325154140187656319936603040014590697
2531dm116476044921853048211370701572032454903308382175079524
2532dm257403410154733747049232185560061399201034
28
/sys/block/[dev]/stat
[ben@lab~]$cat/sys/block/sda/stat
450001547022622941214926903535675019062766249938011331197464821
[ben@lab~]$cat/sys/block/sda/sda1/stat
463163417664642041062156465
26731213641460000040534146
44102152762254638120158477382567961907296571531406792746916857
29
Block layer disk statistics

Field 1 read_IOs: Total number of reads completed (requests)
Field 2 read_merges: Total number of reads merged (requests)
Field 3 read_sectors: Total number of sectors read (sectors)
Field 4 read_ticks: Total time spent reading (milliseconds)
Field 5 write_IOs: Total number of writes completed (requests)
Field 6 write_merges: Total number of writes merged (requests)
Field 7 write_sectors: Total number of sectors written (sectors)
Field 8 write_ticks: Total time spent writing (milliseconds)
Field 9 in_flight: The number of I/Os currently in flight. It does not include I/O
requests that are in the queue but not yet issued to the device driver. (requests)
Field 10 io_ticks: This value counts the time during which the device has had I/O
requests queued. (milliseconds)
Field 11 time_in_queue: The number of I/Os in progress (field 9) times the number
of milliseconds spent doing I/O since the last update of this field. (milliseconds)
30
iostat extended statistics

04/10/2013
_x86_64_
(2CPU)
sda1.162.662.894.2770.3744.6632.100.4664.3729.5587.967.945.69
dm00.000.002.800.7855.535.2333.940.3290.1235.83286.385.031.80
dm10.000.001.175.6814.3938.7315.500.2942.1232.8144.056.384.37
dm20.000.000.040.110.130.7010.680.0163.3026.9677.3415.470.24
sda0.004.670.001.000.0021.3342.670.0223.330.0023.3323.332.33
dm00.000.000.000.000.000.000.000.000.000.000.000.000.00
dm10.000.000.005.330.0021.338.000.035.250.005.254.382.33
dm20.000.000.000.000.000.000.000.000.000.000.000.000.00
sda0.001.670.009.000.0041.339.190.4550.440.0050.446.525.87
dm00.000.000.000.000.000.000.000.000.000.000.000.000.00
dm10.000.000.008.330.0033.338.000.4654.920.0054.922.522.10
dm20.000.000.002.000.008.008.000.0945.500.0045.5018.833.77
31

04/10/2013
_x86_64_
(2CPU)
sda1.162.662.894.2770.3744.6632.100.4664.3729.5587.967.945.69
dm00.000.002.800.7855.535.2333.940.3290.1235.83286.385.031.80
dm10.000.001.175.6814.3938.7315.500.2942.1232.8144.056.384.37
dm20.000.000.040.110.130.7010.680.0163.3026.9677.3415.470.24
sda0.004.670.001.000.0021.3342.670.0223.330.0023.3323.332.33
dm00.000.000.000.000.000.000.000.000.000.000.000.000.00
dm10.000.000.005.330.0021.338.000.035.250.005.254.382.33
dm20.000.000.000.000.000.000.000.000.000.000.000.000.00
sda0.001.670.009.000.0041.339.190.4550.440.0050.446.525.87
dm00.000.000.000.000.000.000.000.000.000.000.000.000.00
dm10.000.000.008.330.0033.338.000.4654.920.0054.922.522.10
dm20.000.000.002.000.008.008.000.0945.500.0045.5018.833.77
32
iostat - extended statistics
rrqm/s(requests)
delta[read_merges(f2)]/interval
wrqm/s(requests)
delta[write_merges(f6)]/interval
r/s(requests)
delta[read_IOs(f1)]/interval
w/s(requests)
delta[write_IOs(f5)]/interval
33

04/10/2013
_x86_64_
(2CPU)
sda1.162.662.894.2770.3744.6632.100.4664.3729.5587.967.945.69
dm00.000.002.800.7855.535.2333.940.3290.1235.83286.385.031.80
dm10.000.001.175.6814.3938.7315.500.2942.1232.8144.056.384.37
dm20.000.000.040.110.130.7010.680.0163.3026.9677.3415.470.24
sda0.004.670.001.000.0021.3342.670.0223.330.0023.3323.332.33
dm00.000.000.000.000.000.000.000.000.000.000.000.000.00
dm10.000.000.005.330.0021.338.000.035.250.005.254.382.33
dm20.000.000.000.000.000.000.000.000.000.000.000.000.00
sda0.001.670.009.000.0041.339.190.4550.440.0050.446.525.87
dm00.000.000.000.000.000.000.000.000.000.000.000.000.00
dm10.000.000.008.330.0033.338.000.4654.920.0054.922.522.10
dm20.000.000.002.000.008.008.000.0945.500.0045.5018.833.77
34
rkB/s(sectors|kB|mB)
(delta[read_sectors(f3)]/interval)/conversionfactor
wkB/s(sectors|kB|mB)
(delta[write_sectors(f7)]/interval)/conversionfactor
avgrqsz(sectors)
delta[read_sectors(f3)+write_sectors(f7)]/
delta[read_IOs(f1)+write_IOs(f5)]
(or0.0ifnoIO)
35

04/10/2013
_x86_64_
(2CPU)
sda1.162.662.894.2770.3744.6632.100.4664.3729.5587.967.945.69
dm00.000.002.800.7855.535.2333.940.3290.1235.83286.385.031.80
dm10.000.001.175.6814.3938.7315.500.2942.1232.8144.056.384.37
dm20.000.000.040.110.130.7010.680.0163.3026.9677.3415.470.24
sda0.004.670.001.000.0021.3342.670.0223.330.0023.3323.332.33
dm00.000.000.000.000.000.000.000.000.000.000.000.000.00
dm10.000.000.005.330.0021.338.000.035.250.005.254.382.33
dm20.000.000.000.000.000.000.000.000.000.000.000.000.00
sda0.001.670.009.000.0041.339.190.4550.440.0050.446.525.87
dm00.000.000.000.000.000.000.000.000.000.000.000.000.00
dm10.000.000.008.330.0033.338.000.4654.920.0054.922.522.10
dm20.000.000.002.000.008.008.000.0945.500.0045.5018.833.77
36
avgqusz(requests)
(delta[time_in_queue(f11)]/interval)/1000.0
await(milliseconds)
delta[read_ticks(f4)+write_ticks(f8)]/
delta[read_IOs(f1)+write_IOs(f5)]
(or0.0ifnoIO)
37

04/10/2013
_x86_64_
(2CPU)
sda1.162.662.894.2770.3744.6632.100.4664.3729.5587.967.945.69
dm00.000.002.800.7855.535.2333.940.3290.1235.83286.385.031.80
dm10.000.001.175.6814.3938.7315.500.2942.1232.8144.056.384.37
dm20.000.000.040.110.130.7010.680.0163.3026.9677.3415.470.24
sda0.004.670.001.000.0021.3342.670.0223.330.0023.3323.332.33
dm00.000.000.000.000.000.000.000.000.000.000.000.000.00
dm10.000.000.005.330.0021.338.000.035.250.005.254.382.33
dm20.000.000.000.000.000.000.000.000.000.000.000.000.00
sda0.001.670.009.000.0041.339.190.4550.440.0050.446.525.87
dm00.000.000.000.000.000.000.000.000.000.000.000.000.00
dm10.000.000.008.330.0033.338.000.4654.920.0054.922.522.10
dm20.000.000.002.000.008.008.000.0945.500.0045.5018.833.77
38
r_await(milliseconds)
delta[read_ticks(f4)]/delta[read_IOs(f1)]
(or0.0ifnoreadIOs)
w_await(milliseconds)
delta[write_ticks(f8)]/delta[write_IOs(f5)]
(or0.0ifnowriteIOs)
39

04/10/2013
_x86_64_
(2CPU)
sda1.162.662.894.2770.3744.6632.100.4664.3729.5587.967.945.69
dm00.000.002.800.7855.535.2333.940.3290.1235.83286.385.031.80
dm10.000.001.175.6814.3938.7315.500.2942.1232.8144.056.384.37
dm20.000.000.040.110.130.7010.680.0163.3026.9677.3415.470.24
sda0.004.670.001.000.0021.3342.670.0223.330.0023.3323.332.33
dm00.000.000.000.000.000.000.000.000.000.000.000.000.00
dm10.000.000.005.330.0021.338.000.035.250.005.254.382.33
dm20.000.000.000.000.000.000.000.000.000.000.000.000.00
sda0.001.670.009.000.0041.339.190.4550.440.0050.446.525.87
dm00.000.000.000.000.000.000.000.000.000.000.000.000.00
dm10.000.000.008.330.0033.338.000.4654.920.0054.922.522.10
dm20.000.000.002.000.008.008.000.0945.500.0045.5018.833.77
40
svctm(milliseconds)
((delta[read_IOs(f1)+write_IOs(f5)]*HZ)/interval)/
(delta[IO_ticks(f10)]/interval)
(or0.0iftput=0)
*HZ=tickspersecond,(1000onmostsystems).
**Thisfieldwillberemovedinafuturesysstatversion.
%util(percent)
((delta[IO_ticks(f10)]/interval)/10)/devices
*devices=1orthenumberofdevicesinthegroup(g
option).
41
pt-diskstats
3.0sda0.00.00.00%0.00.01.33.00.00%0.037.24%01.39.428.2
3.0sda30.00.00.00%0.00.01.04.00.00%0.022.72%01.00.022.7
3.0dm10.00.00.00%0.00.00.00.00.00%0.00.02%00.00.00.0
3.0dm20.00.00.00%0.00.00.74.00.00%0.030.02%00.70.030.0
3.0sda0.00.00.00%0.00.01.36.00.050%0.010.14%21.30.714.6
3.0sda30.00.00.00%0.00.01.08.00.057%0.01.11%11.00.86.3
3.0dm00.00.00.00%0.00.00.74.00.00%0.00.01%20.70.018.0
3.0dm10.00.00.00%0.00.01.34.00.00%0.00.01%11.31.89.0
3.0dm20.00.00.00%0.00.01.04.00.00%0.027.03%01.00.027.0
3.0sda0.00.00.00%0.00.02.33.40.00%0.143.45%02.326.420.0
3.0sda30.00.00.00%0.00.01.74.80.00%0.133.43%01.712.120.6
3.0dm00.00.00.00%0.00.00.34.00.00%0.1191.03%00.3211.092.0
3.0dm10.00.00.00%0.00.00.00.00.00%0.00.03%00.00.00.0
3.0dm20.00.00.00%0.00.00.74.00.00%0.023.52%00.70.023.5
42
pt-diskstats
3.0sda0.00.00.00%0.00.01.33.00.00%0.037.24%01.39.428.2
3.0sda30.00.00.00%0.00.01.04.00.00%0.022.72%01.00.022.7
3.0dm10.00.00.00%0.00.00.00.00.00%0.00.02%00.00.00.0
3.0dm20.00.00.00%0.00.00.74.00.00%0.030.02%00.70.030.0
3.0sda0.00.00.00%0.00.01.36.00.050%0.010.14%21.30.714.6
3.0sda30.00.00.00%0.00.01.08.00.057%0.01.11%11.00.86.3
3.0dm00.00.00.00%0.00.00.74.00.00%0.00.01%20.70.018.0
3.0dm10.00.00.00%0.00.01.34.00.00%0.00.01%11.31.89.0
3.0dm20.00.00.00%0.00.01.04.00.00%0.027.03%01.00.027.0
3.0sda0.00.00.00%0.00.02.33.40.00%0.143.45%02.326.420.0
3.0sda30.00.00.00%0.00.01.74.80.00%0.133.43%01.712.120.6
3.0dm00.00.00.00%0.00.00.34.00.00%0.1191.03%00.3211.092.0
3.0dm10.00.00.00%0.00.00.00.00.00%0.00.03%00.00.00.0
3.0dm20.00.00.00%0.00.00.74.00.00%0.023.52%00.70.023.5
43
pt-diskstats
rd_s(requests)Comparabletoiostatr/s
delta[read_IOs(f1)]/interval
rd_avkb(kB)Similartoiostatavgrqszbutisolatesreads
conversionfactor*delta[read_sectors(f3)]/
delta[read_IOs(f1)]
(Conversionfactor=2=documentationbug)
rd_mb_s(mB)ComparabletoiostatrmB/s
conversionfactor*delta[read_sectors(f3)]/interval
rd_mrg(percent)Similartoiostatrrqm/sexpressedas%
100*delta[read_merges(f2)]/
(delta[read_merges(f2)]+delta[read_IOs(f1)])
44
pt-diskstats
3.0sda0.00.00.00%0.00.01.33.00.00%0.037.24%01.39.428.2
3.0sda30.00.00.00%0.00.01.04.00.00%0.022.72%01.00.022.7
3.0dm10.00.00.00%0.00.00.00.00.00%0.00.02%00.00.00.0
3.0dm20.00.00.00%0.00.00.74.00.00%0.030.02%00.70.030.0
3.0sda0.00.00.00%0.00.01.36.00.050%0.010.14%21.30.714.6
3.0sda30.00.00.00%0.00.01.08.00.057%0.01.11%11.00.86.3
3.0dm00.00.00.00%0.00.00.74.00.00%0.00.01%20.70.018.0
3.0dm10.00.00.00%0.00.01.34.00.00%0.00.01%11.31.89.0
3.0dm20.00.00.00%0.00.01.04.00.00%0.027.03%01.00.027.0
3.0sda0.00.00.00%0.00.02.33.40.00%0.143.45%02.326.420.0
3.0sda30.00.00.00%0.00.01.74.80.00%0.133.43%01.712.120.6
3.0dm00.00.00.00%0.00.00.34.00.00%0.1191.03%00.3211.092.0
3.0dm10.00.00.00%0.00.00.00.00.00%0.00.03%00.00.00.0
3.0dm20.00.00.00%0.00.00.74.00.00%0.023.52%00.70.023.5
45
pt-diskstats
rd_cnc(Little'slaw)iostathasnoequivalent
delta[read_ticks(f4)]/interval/1000/devicesingroup
rd_rt(milliseconds)Differstoiostatr_await
delta[read_ticks(f4)]/
(delta[read_IOs(f1)]+delta[read_merges(f2)])
46
pt-diskstats
3.0sda0.00.00.00%0.00.01.33.00.00%0.037.24%01.39.428.2
3.0sda30.00.00.00%0.00.01.04.00.00%0.022.72%01.00.022.7
3.0dm10.00.00.00%0.00.00.00.00.00%0.00.02%00.00.00.0
3.0dm20.00.00.00%0.00.00.74.00.00%0.030.02%00.70.030.0
3.0sda0.00.00.00%0.00.01.36.00.050%0.010.14%21.30.714.6
3.0sda30.00.00.00%0.00.01.08.00.057%0.01.11%11.00.86.3
3.0dm00.00.00.00%0.00.00.74.00.00%0.00.01%20.70.018.0
3.0dm10.00.00.00%0.00.01.34.00.00%0.00.01%11.31.89.0
3.0dm20.00.00.00%0.00.01.04.00.00%0.027.03%01.00.027.0
3.0sda0.00.00.00%0.00.02.33.40.00%0.143.45%02.326.420.0
3.0sda30.00.00.00%0.00.01.74.80.00%0.133.43%01.712.120.6
3.0dm00.00.00.00%0.00.00.34.00.00%0.1191.03%00.3211.092.0
3.0dm10.00.00.00%0.00.00.00.00.00%0.00.03%00.00.00.0
3.0dm20.00.00.00%0.00.00.74.00.00%0.023.52%00.70.023.5
47
pt-diskstats
wr_s(requests)Comparabletoiostatw/s
delta[write_IOs(f5)]/interval
wr_avkb(kB)Similartoiostatavgrqszbutisolateswrites
conversionfactor*delta[write_sectors(f7)]/
delta[write_IOs(f5)]
wr_mb_s(mB)ComparabletoiostatwmB/s
conversionfactor*delta[write_sectors(f7)]/interval
wr_mrg(percent)Similartoiostatwrqm/sexpressedas%
100*delta[write_merges(f6)]/
(delta[write_merges(f6)]+delta[write_IOs(f5)])
48
pt-diskstats
3.0sda0.00.00.00%0.00.01.33.00.00%0.037.24%01.39.428.2
3.0sda30.00.00.00%0.00.01.04.00.00%0.022.72%01.00.022.7
3.0dm10.00.00.00%0.00.00.00.00.00%0.00.02%00.00.00.0
3.0dm20.00.00.00%0.00.00.74.00.00%0.030.02%00.70.030.0
3.0sda0.00.00.00%0.00.01.36.00.050%0.010.14%21.30.714.6
3.0sda30.00.00.00%0.00.01.08.00.057%0.01.11%11.00.86.3
3.0dm00.00.00.00%0.00.00.74.00.00%0.00.01%20.70.018.0
3.0dm10.00.00.00%0.00.01.34.00.00%0.00.01%11.31.89.0
3.0dm20.00.00.00%0.00.01.04.00.00%0.027.03%01.00.027.0
3.0sda0.00.00.00%0.00.02.33.40.00%0.143.45%02.326.420.0
3.0sda30.00.00.00%0.00.01.74.80.00%0.133.43%01.712.120.6
3.0dm00.00.00.00%0.00.00.34.00.00%0.1191.03%00.3211.092.0
3.0dm10.00.00.00%0.00.00.00.00.00%0.00.03%00.00.00.0
3.0dm20.00.00.00%0.00.00.74.00.00%0.023.52%00.70.023.5
49
pt-diskstats
wr_cnc(Little'slaw)iostathasnoequivalent
delta[write_ticks(f8)]/interval/1000/devicesingroup
wr_rt(milliseconds)Differstoiostatw_await
delta[write_ticks(f8)]/
(delta[write_IOs(f5)]+delta[write_merges(f6)])
50
pt-diskstats
3.0sda0.00.00.00%0.00.01.33.00.00%0.037.24%01.39.428.2
3.0sda30.00.00.00%0.00.01.04.00.00%0.022.72%01.00.022.7
3.0dm10.00.00.00%0.00.00.00.00.00%0.00.02%00.00.00.0
3.0dm20.00.00.00%0.00.00.74.00.00%0.030.02%00.70.030.0
3.0sda0.00.00.00%0.00.01.36.00.050%0.010.14%21.30.714.6
3.0sda30.00.00.00%0.00.01.08.00.057%0.01.11%11.00.86.3
3.0dm00.00.00.00%0.00.00.74.00.00%0.00.01%20.70.018.0
3.0dm10.00.00.00%0.00.01.34.00.00%0.00.01%11.31.89.0
3.0dm20.00.00.00%0.00.01.04.00.00%0.027.03%01.00.027.0
3.0sda0.00.00.00%0.00.02.33.40.00%0.143.45%02.326.420.0
3.0sda30.00.00.00%0.00.01.74.80.00%0.133.43%01.712.120.6
3.0dm00.00.00.00%0.00.00.34.00.00%0.1191.03%00.3211.092.0
3.0dm10.00.00.00%0.00.00.00.00.00%0.00.03%00.00.00.0
3.0dm20.00.00.00%0.00.00.74.00.00%0.023.52%00.70.023.5
51
pt-diskstats
busy(percent)Comparabletoiostat%util
100*delta[io_ticks(f10)]/(1000*interval)
in_prg(requests)iostathasnoequivalent#BIOs
in_flight(f9)
ios_s(requests)Comparabletoiostatr/s+w/s
(delta[readIOs(f1)]+delta[write_IOs(f5)])/interval
52
pt-diskstats
3.0sda0.00.00.00%0.00.01.33.00.00%0.037.24%01.39.428.2
3.0sda30.00.00.00%0.00.01.04.00.00%0.022.72%01.00.022.7
3.0dm10.00.00.00%0.00.00.00.00.00%0.00.02%00.00.00.0
3.0dm20.00.00.00%0.00.00.74.00.00%0.030.02%00.70.030.0
3.0sda0.00.00.00%0.00.01.36.00.050%0.010.14%21.30.714.6
3.0sda30.00.00.00%0.00.01.08.00.057%0.01.11%11.00.86.3
3.0dm00.00.00.00%0.00.00.74.00.00%0.00.01%20.70.018.0
3.0dm10.00.00.00%0.00.01.34.00.00%0.00.01%11.31.89.0
3.0dm20.00.00.00%0.00.01.04.00.00%0.027.03%01.00.027.0
3.0sda0.00.00.00%0.00.02.33.40.00%0.143.45%02.326.420.0
3.0sda30.00.00.00%0.00.01.74.80.00%0.133.43%01.712.120.6
3.0dm00.00.00.00%0.00.00.34.00.00%0.1191.03%00.3211.092.0
3.0dm10.00.00.00%0.00.00.00.00.00%0.00.03%00.00.00.0
3.0dm20.00.00.00%0.00.00.74.00.00%0.023.52%00.70.023.5
53
pt-diskstats
qtime(milliseconds)!iostatavgqusz
delta[time_in_queue(f11)]/
(delta[read_IOs(f1)+read_merges(f2)+write_IOs(f5)+
write_merges(f6)]+delta[in_flight(f9)])
delta[io_ticks(f10)]/
write_merges(f6)])
stime(milliseconds)!iostatsvctm
delta[io_ticks(f10)]/
write_merges(f6)])
54
An example using iostat

[ben.mildren@316403db7~]$iostatdx5
Linux2.6.18194.17.1.el5(xxxx)
04/25/2013
....
sda0.004.600.0016.800.0085.6010.190.063.290.003.290.060.10
sdb7607.005.20253.6023.4031467.20114.40228.030.863.092.934.862.6874.10
dm00.000.000.000.400.001.608.000.004.000.004.002.000.08
dm10.000.000.0028.200.00112.808.000.114.070.004.070.040.12
dm20.000.00374.800.0031441.600.00167.780.792.112.110.001.9874.18
dm30.000.007860.400.4031441.601.608.0023.212.952.954.000.0974.16
dm40.000.000.000.000.000.000.000.000.000.000.000.000.00
....
55
RAID Controller CLIs
Adaptec
arcconf?
Dell
omreport?
HPSmartArray
hpacucli
Type"help"foralistofsupportedcommands.
Type"exit"toclosetheconsole.
LSI
MegaCli?
Oracle
raidconfig?
56
Inthiscaseit'saDellserverandcontroller
omreportstoragecontroller
ControllerPERC6/iIntegrated(Embedded)
Controllers
ID:0
Status:Ok
Name:PERC6/iIntegrated
...
State:Ready
FirmwareVersion:6.2.00013
...
DriverVersion:00.00.04.17RH1
...
NumberofConnectors:2
...
CacheMemorySize:256MB
...
57
/dev/sdbisRAID5SASHDDs
[root@316402db6srvadmin]#omreportstoragevdisk
ListofVirtualDisksintheSystem
...
ID:1
Status:Ok
...
Layout:RAID5
Size:836.63GB(898319253504bytes)
DeviceName:/dev/sdb
BusProtocol:SAS
Media:HDD
ReadPolicy:NoReadAhead
WritePolicy:WriteBack
...
StripeElementSize:64KB
DiskCachePolicy:Disabled
58
/dev/sdbisRAID5SASHDDs
[root@316402db6srvadmin]#omreportstoragepdiskcontroller=0vdisk=1
ListofPhysicalDisksonControllerPERC6/iIntegrated(Embedded)
ControllerPERC6/iIntegrated(Embedded)
ID:0:0:2
...
State:Online
FailurePredicted:No
...
BusProtocol:SAS
Media:HDD
...
Capacity:136.13GB(146163105792bytes)
...
VendorID:DELL
ProductID:ST3146356SS
...
59
Calculating IOPS
ST3146356SS - Cheetah 15K.6 SAS

(Googled - http://www.datasheets.pl/hard_drives/ST3450856SS.pdf)
Disk rotation speed
15k rpm
Avg rotational latency 2.0ms = (1 / (15000/60)) * 0.5 * 1000

Avg read access time 3.4ms
Avg write access time 3.9ms
Looking at omreport storage vdisk

Layout
RAID-5
Counting disks in omreport storage pdisk controller=0 vdisk=1

# Disks
60
Calculating IOPS
http://www.techrepublic.com/blog/datacenter/calculate-iops-in-a-storage-array/2182
To calculate the IOPS range, use this formula: Average IOPS: Divide 1 by the sum
of the average latency in ms and the average seek time in ms (1 / (average latency
in ms + average seek time in ms).
Using gathered data:

1 / (2ms + ((3.4ms +3.9ms)/2))
1 / (0.002 + ((0.0034+0.0039)/2)) = 177 iops (Single disk)
This is inline with the earlier rough estimate of 175-200 iops from 15k rpm disk!
Avg read/write access time are the only measures that should differ per model,
rotational latency should be constant dependent on the disk rotation speed, so the
estimates should be reasonable.
61
Calculating multi disk array IOPS
http://www.techrepublic.com/blog/datacenter/calculate-iops-in-a-storage-array/2182
(Total Workload IOPS * Percentage of workload that is read operations) + (Total
Workload IOPS * Percentage of workload that is write operations * RAID IO
Penalty
Using gathered data:

((4 * 177) * 0.9) + (((4 * 177) * 0.1) / 4) = 654.9 iops
This is doesn't account for the controller cache, so should be a worst case limit.
Picture credit also http://www.techrepublic.com/blog/datacenter/calculate-iops-in-a-storage-array/2182
62
Calculating throughput
Formula to calculate maximum throughput

MB/s = IOPS * KB per IO / 1024
Using gathered data for IOPs, and estimates for KB per IO:
654.9 * 8 / 1024 = 5.1 mb/s
654.9 * 16 / 1024 = 10.2 mb/s
654.9 * 64 / 1024 = 40.9 mb/s
654.9 * 128 / 1024 = 81.9 mb/s
Where KB per IO fits on this scale depends on how random vs sequential the IO is.
63

Linux2.6.18194.17.1.el5(xxxx)
04/25/2013
....
sda0.004.600.0016.800.0085.6010.190.063.290.003.290.060.10
sdb7607.005.20253.6023.4031467.20114.40228.030.863.092.934.862.6874.10
dm00.000.000.000.400.001.608.000.004.000.004.002.000.08
dm10.000.000.0028.200.00112.808.000.114.070.004.070.040.12
dm20.000.00374.800.0031441.600.00167.780.792.112.110.001.9874.18
dm30.000.007860.400.4031441.601.608.0023.212.952.954.000.0974.16
dm40.000.000.000.000.000.000.000.000.000.000.000.000.00
....
Observations:
~275 IOPs well below estimated limit of ~650 IOPs
Average queue size (avgqu-sz) is low.
At ~30mb/s (rkB/s + wkB/s), throughput will not be saturating the controller,
rearranging the throughput formula we can see IOPs are about 114kb per IO which
leans to a more sequential workload.
Latency (read/write average wait time) is inline with drive manufacturer
expectations.
64

Linux2.6.18194.17.1.el5(xxxx)
04/25/2013
....
sda0.004.600.0016.800.0085.6010.190.063.290.003.290.060.10
sdb7607.005.20253.6023.4031467.20114.40228.030.863.092.934.862.6874.10
dm00.000.000.000.400.001.608.000.004.000.004.002.000.08
dm10.000.000.0028.200.00112.808.000.114.070.004.070.040.12
dm20.000.00374.800.0031441.600.00167.780.792.112.110.001.9874.18
dm30.000.007860.400.4031441.601.608.0023.212.952.954.000.0974.16
dm40.000.000.000.000.000.000.000.000.000.000.000.000.00
....
Observations:
Average request size (avgrq-sz) is ~114k (matches throughput calc), so the load is
probably not being generated by MySQL.
High number of merges, not a bad thing, but might want to investigate why..
(In this case they were caused by an LVM Snapshot being read during a backup).
65
Thank you Q&A

To contact us
sales@pythian.com
1-877-PYTHIAN
To follow us
http://www.pythian.com/blog
http://www.facebook.com/pages/The-Pythian-Group/163902527671
@pythian
http://www.linkedin.com/company/pythian
66

Monitoring Linux IO

Caricato da

Informazioni sul documento

Descrizione originale:

Copyright

Formati disponibili

Condividi questo documento

Condividi o incorpora il documento

Opzioni di condivisione

Hai trovato utile questo documento?

Questo contenuto è inappropriato?

Copyright:

Formati disponibili

Monitoring Linux IO

Caricato da

Copyright:

Formati disponibili

Monitoring IO performance

using iostat & pt-diskstats

Team Technical Lead at Pythian

Over 10 years experience as a production

Experience of MySQL, SQL Server, and

Global Reach & Scalability:

Why the interest in IO monitoring?

There are a number of tools available to monitor IO on Linux. This

iostat is part of the sysstat utilities which is maintained by Sebastien

At the time of writing the current stable version is 10.0.5.

By default iostat produces two reports; the CPU Utilization report

Inclusion of the CPU and Device Utilization reports can be

The default Device Utilization report can be replaced with extended

Since version 7.1.3 the report can be made more readable by

-t (timestamp reported with sample)

-z (report only includes active devices for this sample)

In version 10.0.5, the devices in the Device Utilization can be

The Device Utilization report can be produced on a partition level

pt-diskstats is part of the percona toolkit.

Tools can also be downloaded individually:

At the time of writing the current stable version of percona toolkit is

By default pt-diskstats produces useful stats reporting current

The interval can be adjusted using the --interval option

Displayed columns can be restricted using perl regex with the

The interlude: A primer in Linux IO

Read http://kernel.dk/blk-mq.pdf for recent development

Applications in the user space make requests to the VFS.

To understand what is acceptable performance there's no substitute

As a rough estimate you can expect:

Block layer disk statistics

Block layer disk statistics

iostat extended statistics

iostat extended statistics

iostat - extended statistics

iostat extended statistics

iostat - extended statistics

iostat extended statistics

iostat - extended statistics

iostat extended statistics

iostat - extended statistics

iostat extended statistics

iostat - extended statistics

An example using iostat

RAID Controller CLIs

RAID Controller CLIs

RAID Controller CLIs

RAID Controller CLIs

ST3146356SS - Cheetah 15K.6 SAS

Avg rotational latency 2.0ms = (1 / (15000/60)) * 0.5 * 1000

Looking at omreport storage vdisk

Counting disks in omreport storage pdisk controller=0 vdisk=1

Using gathered data:

Calculating multi disk array IOPS

Using gathered data:

Picture credit also http://www.techrepublic.com/blog/datacenter/calculate-iops-in-a-storage-array/2182

Formula to calculate maximum throughput

An example using iostat

An example using iostat

Thank you Q&A

Potrebbero piacerti anche