Lesson 2: Manage and Administer - Administering Volume Manager - Text

Page 1 of 4

LESSON 2: MANAGE AND ADMINISTER - ADMINISTERING VOLUME MANAGER

PRINT DOCUMENT

Lesson 2 Administering Volume Manager

Welcome to Lesson 2, Administering Volume Manager.
Lesson topics and objectives
In this lesson, we'll explore performance monitoring and tuning with Storage Foundation. We'll also explore changing the volume
layout and managing volume tasks.
Introduction to performance monitoring with SF
Let's start with performance monitoring and tuning and show you some benchmarking tools that you could use for this.
Storage performance analysis process
The first step of performance tuning is you need to identify that you have a problem currently with your performance. The problem
could be on a disk, a volume or a set of disks inside a volume. Then what you need to understand is what level of performance do
you want to get if you make some changes on those objects and then obviously what changes do you need to make to get to the
level that you want. So you need to understand the application workload, that's really the first part of deciding and identifying where
you want to get to, a good performance level. What size of application are you running, what size of IO is it sending to that object,
how often is it sending it and what type of IO is it. Is it random, is it sequential, is it reads/writes, is it all of the above? What type of
application are you running? So we have an example here in the slide of an email server with a small I/O size, random I/O, probably
mostly writes but could be also mixed, read and write mix. Once we know what kind of application workload we are running, we are
going to go down the complete I/O path from the application to disk. So we are going to trace where that I/O is going from the
application into the file system, to the volume, to the subobjects and ultimately to the disks. We want to get characteristics for each
component, that refers to where do we want to be at the end of the process. What performance level or SLA or guideline or baseline
do we want to get to? Before we determine that we've done our job and the performance is that much better. We also have to be
able to measure the performance all along the process.
Using performance analysis tools
That's done using the available tools that I'm going to show you in the next few slides. Ever since the first Volume Manager version
was released, we've had two tools that were specific to Volume Manager, vxstat and vxtrace. A little bit later along the evolution of
the product, we included a new tool called vxbench which now is an old tool, but now you have three tools - vxstat, vxtrace and
vxbench. They are used for very different things. As we discussed in an earlier lesson, vxbench is used to actually simulate an
application I/O, different types of I/O; whereas with vxstat and vxtrace, they don't send any type of I/O at all nor do they simulate
I/O. They report back on I/O that's already happening. So in case you are wondering, gee, could I use vxbench to simulate the I/O
and then run vxstat or vxtrace to report on it, the answer is absolutely. In addition to those tools, we also have other tools that you
could download for free from the web notably PostMark for file system tuning and benchmarking. And for volume and device level
benchmarking and tuning you can do, you can run other tools that are typically included with native OSs like dd and tar. And you can
even write your own C programs and use those.
The vxstat utility example: Load balancing and volume layout
So I want to explore in more detail some of the volume manager based tools that I just mentioned and we'll start with vxstat. vxstat
is lot like the Solaris-based iostat command. It'll give you the number of operations, the blocks that are written and the average time
in milliseconds that each operation took to complete. So in this example, we are looking at throughput in operations in blocks of time
for three different volumes. The first volume is striped across two columns. Now we said in earlier lessons that a striped volume
typically provides better read and write performance overall than a concatenated volume or nonstripe volume and that's definitely
true. But as you add more columns to the volume or you test the same kind of benchmark on volumes with more columns, you are
going to see a notable performance increase. So if you look at the total I/Os and throughput and time for some of the examples at
the bottom of the slide, you notice the average time, especially the average write time is going to be less over all the different I/Os
and less for each I/O than it is in the first example. This kind of proves that volumes that are striped across more columns can and
probably will improve your overall performance. We don't want to forget however that adding too many columns to a striped volume
can infringe upon redundancy for that volume. So for example if somebody takes this to the extreme level and wants to add 20
columns to a striped volume, that's going to be great for performance, but they also have to remember that any failure of any one of
those disks representing each column fails, the whole volume will go down. So remember the volumes are not necessarily mirrored as
well as striped.
vxtrace: Analyzing the I/O profile
If we have an I/O that's either not completing successfully or that's causing a bottleneck in the volume somewhere and we want to
take a closer look at the I/O that's happening to and from that object specifically then we use this program right here vxtrace. vxtrace

http://symantecpartners.vportal.net/media/symantecpartners/media/_generated/transcripts/t... 8/23/2011

Lesson 2: Manage and Administer - Administering Volume Manager - Text

Page 2 of 4

analyses the I/O from start to finish. And there is usually a lot of output that happens as a result of running a vxtrace. Now on your
slide, you have quite a bit of output here and a lot of things are happening at the same time. So I'm going to try to break this down
easily for you. First thing we want to look at is if you look at the left most column, you have a control number starting with 30147.
This can also be thought of as a sequence number which helps vxtrace order its specific output. If you look at the middle part of the
output, you see a number and you see two letters OP, that stands for an operation code number. So, what you do is you look for an
operation code number that matches the number in the left most column as the sequence number and you can see and identify and
trace that I/O from start to finish using that operations number and matching it back. So, we have here is, we have a control number,
the first one is 30147, process ID 54 which is at the end of that output. It's the start of a read operation on block 115648 which
happens to be the volume block where the I/O starts. We also have another control number of 30148 immediately after that which is
part of operation 30147, so this might be simultaneous or concurrent I/O, but it's length 32 sectors. If you see the LEN at the end of
the second line, that stands for the length of the actual I/O which in this case is 32 sectors or 16k. While that I/O is in process,
another I/O starts, indicated by control number 30149 on the lefthand side. You see the start of that I/O. It's on block 86976, it's
another read, also length 32 sectors, but this one's sent by process ID 57. So it's definitely a different process, it's sending this new
read into the volume. We now see some other activity in what looks like the same disk related to a third operation, but now we start
to see after that we see end-end. So there is an end on the fifth line and there is an end on the sixth line. The whole point of vxtrace
is to analyze the I/O from start to finish. If you see an I/O with a starting point for a process with no ending point that means that
I/O is still hung up in the kernel or it never finished. And there there could be a lot of reasons why those things could happen. But if
the I/O does end, you will at least know that it completed successfully and is not still hung up in the kernel, so that's probably a good
thing to figure out first and foremost to see if your I/O is being hung up or not. If it's not being hung up, if it's completed
successfully, you should see an endpoint with the same operations number as the original start for that I/O. So, it looks like on lines 5
and 6, we have an endpoint for those two I/Os that I just discussed because they are both related to operation 30147 and they have
length 32 and the time shows up as well. The time cannot be calculated unless an I/O completes successfully. So in this case, it does
and the time here for each I/O is 1 millisecond. So the time is in milliseconds. Now what's happening after that point is other
processes are going to start to send more I/Os. Now using vxtrace, you can also analyze the type of I/O that's happening meaning
whether it's sequential or random. It's pretty easy to see whether it's read or write. You can see a lot of reads in these outputs, so
they are all reads. But it's a little bit more difficult to see whether it's a sequential read or a random read. So you have to do a little
bit of detective work here. The best way to do this is look at the block designations for especially the starting block designations for
the I/O. The first one is a block 115648, the next one is at block 86976. The next one is at block 86976 again. You have another one
at block 151232, another one at 29472. That seems pretty random to me, so this looks like random read I/O hitting that same disk
and same volume. If you still have a performance problem at least you'll know that it most likely is the disk performing because it's
being used by three different processes at the same time and it could be the disk is really the bottleneck. Now the question remains,
could any one of those processes be run on another part of the volume or different disk in the volume. I would bet if you re-laid out
this volume and changed it to a stripe volume, if it's not already, or if it is already, add some more columns to it like the previous
examples we discussed in this lesson you would get better performance out of that volume and your I/Os will complete faster and
more efficiently. This is the kind of thing you want to look for.
Selecting a stripe unit size
For random I/O, if you decide you want to do something like that and either add or expand or change the type of volume in the
striped columns. If that takes care of the problem, then fine. If it doesn't take care of the problem, but you still have a striped volume
you're playing with, it might be a good idea to change the striped unit size. Now if you have random I/O and the stripe unit size is too
small, you are going to be squeezing the I/O and forcing it to go to more than one column at the same time. Now in this case, and it
depends on the type of I/O. In this particular case, that's not a good thing because random I/O as you just saw may hit different
parts of the volume at several different times in the volume. So it's the opposite for random I/O to get good performance. So instead
of squeezing the striped unit size too thin, you want to expand the size of the striped unit and try to satisfy each I/O with a single
disk or single striped column. With sequential I/O, it's the opposite. You want to squeeze it in, force it to more columns and have
more spindles operating on the same I/O at the same time, because sequential I/O is a lot more predictable and inline than random
I/O. Now, bear in mind these are not hard and fast written stone rules and regulations, these are just guidelines. It all depends on
what application you are running and what kind of storage you have underneath it. So after you run your benchmarking operation
and you know the change that you want to make, you run something like this command at the bottom of your slide here, a vxassist
relayout, and make the change and then rerun the benchmark and hopefully you will have verified that you did the right thing.
Possible layout changes
These are some other suggestions of what you can change inside a volume to increase performance based on the type of I/O that's
happening on the volume.
Changing the volume layout
So let's talk more about how to change the volume layout.
Changing the volume layout
Changing the volume layout is probably the single most I/O intensive operation in Volume Manager that you can do to a volume
manager object. It affects everything in the volume, not just usually a plex or a subdisk but everything, every object in the volume is
affected by this. It does not change the placement of user data, or change the state of user data when you run it and you can always
run it while the volume is mounted and being accessed by the file system. Here are some examples of changing the volume layout in
your slide. You can change concatenated volumes to striped volumes. You can change RAID5 to striped mirror for better write

http://symantecpartners.vportal.net/media/symantecpartners/media/_generated/transcripts/t... 8/23/2011

Lesson 2: Manage and Administer - Administering Volume Manager - Text

Page 3 of 4

performance and avoid the parity bottleneck. You can change the stipe unit size, you can reduce or add columns to get the
performance you want, and you can change nonlayer to layered volumes or layered to nonlayered volumes as well.
Online relayout notes
If you decide you don't like what's going on, you can actually pause a relayout and reverse the relayout back to its original state of
any volume during that process. These are the things you cannot do during online volume relayout. You can't create a snapshot
during a relayout, you can't add or delete a mirror during a relayout. You can't do multiple relayouts at the same time and the reason
for that is it's just simply too I/O intensive on the driver and the daemon. And you cannot perform a relayout on a volume with what's
called a sparse plex.
Changing the layout: VEA
So one relayout at a time without adding any mirrors or deleting any mirrors and controlled environments. Make sure you don't have
a lot of I/O hitting the volume from the outside from the user space while you are doing the relayout or that's just going to cause a
lot of contention both sides. So same kind of thing as defragmenting a file system, you try to do it at an opportune time when there
is very little if any I/O activity going on. One of the ways you can do this is through the VEA. There's an option called change layout.
What you are going to say here is you are going to tell the VEA what you want to change the volume to. It knows what it is right
now, you tell it what you want it to be. So, in this case, we are taking a datavol01 and we are changing it to a concatenated layout.
You can also do things like retain volume size at completion, which is important for RAID5 volumes. You can also set a temporary
space size because Volume Manager will use free space elsewhere on another disk inside the same disk group to manage the
relayout. Volume Manager copies objects, it never cuts and pastes subdisks or plexes. It always copies and pastes subdisks and
plexes. So it needs a place on other disks to do those copies and make them before it recopies the data from those free disks back
into the original volume. So it will use a certain amount of temp space by default. If none exists, it will give you a message cannot do
the relayout because it doesn't have enough space. But the more space or temp space that you give it now if you have it, the faster
relayout operation will go. So it's a good idea to use as much space as you can affordably and make the relayout happen as fast as
you can without using any other contention on any other processes or applications. Once you get the relayout done, you have a new
volume and you can open up more application I/O to it.
Changing the layout: VEA
This is a good operation to do in the VEA because it gives you a task status window that looks like this with the name of the volume,
the initial layout and the layout that you are relaying it out to. So, in this case, this might be a different example here, we have an
initial concatenated volume. We are changing it to a layered stripe-mirrored volume with two columns with a 128 sector or 64K stripe
unit size. You notice on the right hand side we have the option to pause the relayout or actually abort the relayout. So the relayout
will keep running until you are 100% complete unless you pause it or abort it. If you pause it, you will see the continue button open
up or ungray and you'll be able to hit the continue button and the relayout will continue right from where you left off pausing it. And
all throughout this process, the volume will remain online.
Changing the layout: CLI
If you run this command from the command line, the command line is a little bit more picky about what you are doing to the volume
with regards to what syntax you need to use in the command. For example, vxassist relayout will only work for volumes we are
changing the stripe unit size or adding or removing columns from RAID5 striped/mirrored or striped volume. Everything else needs to
use vxassist convert.
vxassist relayout
Now here are some examples of vxassist relayout, and you can see each example in this slide is either going to change to a stripe
layout, change the number of columns in the volume or change the stripe unit size in the volume. And that's no coincidence because
these are the only types of operations that relayout will let you do from the command line.
vxassist convert
When you come to vxassist convert, you can see you can change layer to nonlayered and back and forth and you can actually change
the type of volume to a totally different type with vxassist convert. So the bottom line here is vxassist convert is a more extreme
change than vxassist relayout and if you don't want to have to deal with so much syntax and specificity you can go back to the VEA
and do the command that way through the VEA and make your life a lot easier. In addition to that, you can go to the command line
in the VEA once the command has been finished and you can actually see the Volume Manager commands that were run as a result
of you clicking the VEA objects. So you can see the commands and then save the commands and then you can remember that way.
Managing volume tasks
So, here's an option to manage the volume task.
Managing volume tasks: CLI
Time you are doing an operation in Volume Manager that involves Volume Manager objects, even if it's a very fast operation like

http://symantecpartners.vportal.net/media/symantecpartners/media/_generated/transcripts/t... 8/23/2011

Lesson 2: Manage and Administer - Administering Volume Manager - Text

Page 4 of 4

creating a volume for the first time, there is a task that's associated and there may be child tasks that are also associated with that
object creation or management. The task always has a task ID, but it doesn't have a task tag. A task tag is something that the
administrator would add to the task and basically tag it to the ID so that if you wanted to pause the task, continue the task or abort
the task, you could refer to the task by its tag name as opposed to its ID.
Managing volume tasks: VEA
The command log file is what I just referred to when I said you could do things through the VEA and see the corresponding
command line command show up in a log. So you'll see the veacmdlog file on the root disk here is going to have that information.
You'll also see information about tasks and task errors if there are any errors that come up. You'll also see information about paused,
continued and aborted tasks. In the VEA itself, you'll see information about the task itself and you'll actually be able to right click on a
task and do these things that you see in a slide if you want to operate on that task specifically. You also see an operation called
throttle task here. The throttle task is good if you want to slow down the rate of I/O that Volume Manager uses internally to finish the
task and allow more latitude to any user I/O that's hitting that volume object at the same time. So throttle task slows the task down
and allows more I/O to happen more often from the user side. So, that may improve the performance of any user applications or any
file system that's using that volume while you are trying to run the task on it. By default, no task is throttled and they all get
simultaneous and alternating access between any user I/O and internal I/O.
vxtask list
The vxtask list command will give you information specifically about the task ID and any tag if there is a tag on that task. If the task
running has a corresponding parent task; for example, if you created a volume and now you are creating a mirrored volume and you
are synchronizing the mirrors, you look at one of the mirror tasks and you may see a parent task ID of the volume creation on vxtask
list in the PTID field. Under progress and the PCT fields, you can see the percentage completed of the task and the progress indicates
where in that object's block cycle is that task right now. So in other words, if you were to divide the 11976 into 20480, you'll probably
get about 58%. So, the task is just more than half finished and it doesn't have too far to go hopefully. So, the progress indicates the
offsets, starting and ending point of the block where that task lies and the affected object that's being run on is at the end of that
line, in this case, it's myvol. You can also see under the state field whether a task is running, paused or aborting.
vxtask monitor
vxtask monitor is a realtime rolling command which allows you to see a task line by line as the completions are happening. So, it'll go
from 0 to 100% completion and it will just basically tail itself, it's kind of like tail-f command in UNIX. And it keeps giving you output
as it goes. You can choose to output that task in long format. You can print a certain number of counter tasks and then have it quit.
You can print the task operations for a certain amount of time and after that many seconds has lapsed, have that quit. Or you can
print a waiting message after an interval with no activity if there is nothing going on in the task and you want to know if there is
nothing going on. Any of these options have been available in the product for many versions and years, but they are all available for
any tasks you're running in Volume Manager. Most of the time, this command and this command vxtask is used for things like
recovering volumes. Mirroring volumes, setting up RAID5 volumes or covering RAID5 volumes and snapshoting. Here are some
examples of pausing, resuming and aborting a task.
vxtask abort|pause|resume
Note that the vxtask resume is pretty much the same thing as the vxtask continue button from the VEA. So the VEA continue button
and the command line resume command are essentially the same thing.
vxrelayout
Sometimes you'll find that the VEA calls vxrelayout under the hood which is a very similar command to vxassist relayout. The
difference being that vxrelayout gives you different information in the output like at the bottom of your slide here. You can also run
this command from the command line like we're are doing here. In this case, we are taking the 5 column striped volume and
converting it to or actually relaying it out to a 6 column striped volume. So we are adding one column in the volume and it's 58%
completed. So, this is just an additional command that you can use to see how much progress you are making during that relayout,
and that concludes this lesson, thank you very much.
Lesson summary
and that concludes this lesson, thank you very much.

http://symantecpartners.vportal.net/media/symantecpartners/media/_generated/transcripts/t... 8/23/2011