Documenti di Didattica
Documenti di Professioni
Documenti di Cultura
Summary: Get more information on the virtualization capabilities of the IBM® POWER5™
servers. Follow along as Nigel Griffiths illustrates how to set up and use the IBM Virtual I/O
Server (VIO Server).
Tags for this article: griffiths, guide, i/o, lpar, nigel, power5, powervm, server, setting, up...
more tags
Show articles and other content related to my search: ibm vio server
Introduction
This article describes the virtualization capabilities of the IBM® POWER5™ servers, provides
examples that apply equally to both pSeries® p5 and eServer™ OpenPower™ systems, and
shows how to set up and use the IBM Virtual I/O Server (VIO Server). The VIO Server is
currently based on a subset of AIX® 5.3 that includes additional packages and services. It comes
with the IBM optional packages called the Advanced POWER Virtualization (APV) for pSeries
p5 machines or the Advanced OpenPower Virtualization (AOPV) packages for OpenPower
machines. As these two versions have identical features and functions, I reference them as VIO
Server throughout this article. The VIO Server provides a complete environment for the VIO
Server, full support from IBM, and higher levels of performance.
Virtualization is a hot topic in the computing industry, with many widely different technologies
and solutions being recommended, developed, and used. The POWER5-based machines have
inherited the know how from the IBM mainframes to provide opportunities for a significant
reduction in operating costs for complex environments. Unlike software solutions available from
other vendors, the POWER5 implementation uses advanced processor features, firmware (also
known as Hypervisor), and hardware features to create efficient and flexible virtualization
capabilities. Uniquely, these capabilities are offered from the top to the bottom of the server
range -- a powerful 64-way SMP (Symmetric multiprocessor) machine down to a two-way, desk-
side system. The key to this virtualization is the VIO Server.
This article:
Explains the VIO Server concepts and how it works between logical partitions (LPARs)
for disk access and networks.
Covers the advantages of using a VIO Server and typical usage scenarios.
Shows, by example, how to set up the IBM VIO server and VIO clients.
Back to top
pSeries servers from IBM have, since October 2001, allowed a machine to be divided into
LPARs, with each LPAR running a different OS image -- effectively a server within a server.
You can achieve this by logically splitting up a large machine into smaller units with CPU,
memory, and PCI adapter slot allocations.
The new POWER5 machines (pSeries p5 and OpenPower servers) can also run an LPAR with
less than one whole CPU -- up to ten LPARs per CPU. So, for example, on a four CPU machine,
20 LPARs can easily be running. With each LPAR needing a minimum of one SCSI adapter for
disk I/O and one Ethernet adapter for networking, the example of 20 LPARs would require the
server to have at least 40 PCI adapters. This is where the VIO Server helps.
The VIO Server owns real PCI adapters (Ethernet, SCSI, or SAN), but lets other LPARs share
them remotely using the built-in Hypervisor services. These other LPARs are called Virtual I/O
client partitions (VIO client). And because they don't need real physical disks or real physical
Ethernet adapters to run, they can be created quickly and cheaply.
Both APV and AOPV versions of the VIO Server are special-purpose, single-function
appliances and are not intended to run general applications.
The Linux VIO Server for pSeries p5 or OpenPower hardware first became available
with the SUSE SLES 9 distribution. Unlike the VIO Server, this is just a copy of the
Linux operating system. This means it can run other central services such as NFS,
network installation, DNS, an Apache Web site, or Samba services. Some care should be
taken that these functions do not interfere with the performance of the VIO Server
service. This software is also available on the Debian Linux for POWER distribution.
There are different implementations for VIO clients. Actually, these are just the regular operating
systems, but they include the device drivers for running as a VIO client.
This article covers the VIO Server and the AIX and Linux VIO clients.
The VIO Server provides a virtual SCSI disk service, as shown in Figure 1 below.
With the VIO Server, data is moved directly between the VIO client partition and the real disk
device using Remote DMA (RDMA) protocols. In the SUSE VIO Server implementation, data
moves from the VIO client to the VIO Server, and thence to the disk. This is "double buffering".
Figure 1 shows a single VIO Server providing virtual SCSI services to multiple VIO client
partitions. Each VIO client operates as if it had a dedicated SCSI device but, in fact, each client
device is a real disk partition (logical disk partition) on the VIO Server. Alternatively, on the
VIO Server, it could use a complete disk (hdisk). The VIO Server and VIO client communicate
using the internal pSeries Hypervisor firmware (PHYP) feature, which efficiently allows disk I/O
requests to be transferred between the LPARs using a message-passing protocol.
In Figure 1 above, the VIO Server has a few disks that could be SCSI or fiber channel storage
area network (SAN) disks. The disk subsystem hardware or a RAID5 SCSI adapter can provide
data protection. The VIO clients use the VIO client device driver just as they would a regular
local device disk to communicate with the matching server VIO device driver. Then the VIO
Server actually does the disk transfers on behalf of the VIO client. There is a strict client/server
relationship between the VIO client and the VIO Server.
Virtual Ethernet
The LPARs in the machine can use the virtual Ethernet switch service (in the Hypervisor) in a
number of different ways.
Back to top
You can use a VIO Server in any number of scenarios. Below are five typical examples that
would make good use of a VIO Server.
The VIO Server is used for "bits and bobs" LPAR like test, development, training,
practice, new application trials, and so on. Typically, VIO clients might have a couple of
four GB to eight GB virtual SCSI disks and one or two virtual Ethernets.
In Figure 6, three large production LPARs are running (they would have dedicated disks
and Ethernet) with a few extra small VIO clients and one VIO Server on the machine
using spare capacity. This "spare" capacity could be demanded by the production LPARs
during peaks in their workload.
The VIO Server has one or two CPU(s), possible RAID 5 SCSI disks, or SAN disks.
Typically, clients have one or more four GB virtual SCSI disks each and might have
different groups of LPARs around a different Virtual Ethernet.
Figure 7 shows dozens of VIO clients with a medium-sized VIO Server supporting them
on what might be several disk packs.
Serious I/O setup only once (to reduce setup and management)
The VIO Server has SAN disks connected by two to four Fibre Channel adapters and two
Ethernet adapters to run Ether channel for redundancy and additional bandwidth. The
VIO Server has load balancing and failover, but VIO clients have a much simpler disk
and Ethernet setup. The VIO Server could have one to three CPU(s), but the VIO clients
are larger, too.
For example, one to eight CPU(s) run quite large applications. Typically, VIO clients
could have hundreds of GB of virtual SCSI disks and many Virtual Ethernets. This
complex setup is not covered in this article. Figure 8 shows two regular LPARs (it would
have dedicated disks) and a fully configured VIO Server (large) with multiple paths to
disks and Ethernet. This is supporting some large VIO client LPARs.
The counter argument is that VIO Server is only running a few device drivers. Devices
drivers are extremely reliable. Also, anything that would crash one VIO server could also
crash the second one. Figure 9 shows that instead of using local physical device drivers,
the VIO client uses the virtual resource device drivers to communicate with the VIO
Server, which does the real I/O. Except the virtual VIO Server device drivers and the
physical resource device drivers, there is very little code running on the VIO Server.
Little can go wrong on the VIO Server side.
I'm not going to cover duplicated VIO Servers. Further details are in the Advanced POWER
Virtualization on IBM p5 Servers redbook (Download Adobe Reader).
Back to top
Prerequisites
This section describes the software, hardware, skills, and type of network you'll need.
Software
For a pSeries p5 machine, the software is included in the APV feature. This runs AIX and
Linux VIO clients.
For an OpenPower machine, the software is included in the AOPV feature. This will only
run the Linux VIO clients (AIX does not run on these machines).
Hardware
You'll need:
Skills
This article doesn't show you a screen-by-screen level of detail and each input field. It's assumed
you already understand:
Though the IBM VIO Server is based on AIX, it's quite different. You only use the VIO Server
commands; some standard AIX features are not present (for example, smitty) and some typical
AIX setups are not allowed, such as LVM- (Logical Volume Management) level mirroring of
VIO client logical volumes. Don't think of the VIO Server as AIX, but AIX experience can be
useful in understanding the terms used.
Basic AIX systems administration such as installing from an mksysb image, configuring
networks, AIX style volume groups, and logical volumes terms.
If you intend to use a Linux VIO client, then you need:
o Basic Linux systems administration such as installing an RPM (rpm -Uvh
<package>.rpm), configuring an Ethernet network adapter (ifconfig eth0
<ipaddress> mask 255.255.255.0), and managing a filesystem (mount
/dev/sda5 /mnt). These tasks are identical to working on the Intel platform, and
there are many books, training courses, and Internet material covering the regular
system administration commands and tasks.
o How to install SuSE Linux in either text mode (on a dumb/ASCII screen) or with
a VNC (Virtual Network Computing) session. Once you've installed Linux a
couple of times, this becomes a routine task. For the VNC install, the boot prompt
extra command is vnc=1 password=abc123. There are six characters for the
password. The system also prompts you for the other details.
Hardware Management Console (HMC):
o How to install the HMC hardware and software
o How to set it up (It's assumed this has already been done.)
o How to use the HMC to create and start a simple LPAR and its profiles
The pSeries p5 and OpenPower range of machines internals are like the names of the
adapter positions. For example, the Tn names for internal adapters and Cn for real
adapters in a PCI slot. You need to create the VIO Server LPAR with the right SCSI
disks and Ethernet resources on the HMC, with and without the CD. Above n is the
number of the slot; details are in the hardware manuals, redbooks, or on the large sticker
on the outside of the machine covers.
Network
The VIO Server must be able to communicate directly with the HMC for advanced functions and
error reporting. Since this is easily forgotten, I recommend a network like the one in Figure 10.
Many sites also have other dedicated networks in addition to the above. For example, a network
for remote backup or a network dedicated for systems administration in addition to the networks
in Figure 11 below.
Back to top
This section covers the steps and three extra common tasks to get started with the VIO Server.
Please note that it takes longer to describe some steps than to actually implement!
Back to top
Figure 11 shows the SUSE VIO Server in an LPAR and two VIO client LPARs that are going to
be set up for this article. It's a logical diagram of the example setup, which is explained in the
rest of this article.
Figure 11. The SUSE SLES 9 Server LPAR
Ethernet
For simplicity, the VIO client LPARs are given Ethernet IP addresses within the address range of
the regular physical Ethernet network in this computer room. The VIO Server bridges between
physical and virtual networks, meaning that the client LPARs will appear like any other
computer to users. This is the most likely option to be implemented and hides the Virtual
Ethernet network completely from users in order to allow simple access to the client LPARs.
Disks
For the disks, let's use the internal SCSI adapter in the VIO Server and one disk. The first client's
(Client X) virtual disk connects to a logical volume (disk partition) on the VIO Server. The
second client's (Client Y) virtual disk is supported by a whole disk on the VIO Server. This
shows all of the common types of setup -- the SEA network and disk partitions, plus allocating a
whole disk. In practice, most people use a logical volume.
Back to top
First, do some planning of the VIO Server and client logical partitions. Experience has shown
that just creating LPARs without some planning causes problems and can waste a lot of time.
Table 1 shows the planning I've done for this example, which is an OpenPower 720. Except for
the references to PCI slots like C3, T6, and T14, which are machine dependant, the references
could be for any pSeries p5 or OpenPower machine. In this example, the VIO client logical
partitions are going to be Linux, but equally they could be running AIX. Notes are included
where AIX VIO clients would be different.
Table 1. Planning
VIO Server Client A Client B
Hostname op34 op36 op37
Ethernet adapter C3 bridging Virtual Virtual
IP address 9.137.62.34 9.137.62.36 9.137.62.37
Virtual LAN ID (port) 1 1 1
Mask 255.255.255.0 255.255.255.0 255.255.255.0
Gateway 9.137.62.1 9.137.62.1 9.137.62.1
DNS 9.137.62.2 9.137.62.2 9.137.62.2
CD adapter T6 for install only T6 for install only T6 for install only
SCSI adapter T14 Virtual Virtual
Disk size hdisk0 is 36 GB 4 GB
hdisk1 is 36 GB for client 73 GB
Y
Device on VIO Server lv00 hdisk1
Virtual SCSI adapters Slot 3 for Client X Slot 3 to server slot 3
Slot 4 for Client Y Slot 3 to server slot 4
Profile names Normal Normal Normal
Normal with CD Normal with CD Normal with CD
CPU values:
Dedicated/shared CPU Shared Shared Shared
CPU desired 0.4 0.3 0.3
CPU min 0.2 0.1 0.1
CPU max 1 2 2
Virtual processors 1 2 2
Memory values:
Memory 512 MB 2048 MB 256 MB
Back to top
Next, you need to create the VIO Server LPAR. You do this on the HMC and create a special
VIO Server LPAR, but initially with no extra virtualization features. You'll add the virtual
features later. The only feature that is different from a regular Linux LPAR is the LPAR Partition
Environment feature on the first panel of the Create Logical Partition Wizard. Here you must not
select the AIX or Linux option, but must select VIO Server. See Figure 12 below.
Create the LPAR and the first profile as above, using the details in Table 1. (This article assumes
you are familiar with the HMC and creating LPARs. If you are not, see Resources for documents
that describe how to create LPARs.) I call the LPAR profile that is normally used with a name of
"Normal". Further hints:
A VIO Server LPAR can use dedicated CPUs, which is a good idea if you have plenty of
CPUs or are expecting to do lots of I/O for many VIO client LPARs; it avoids any delay
in starting the I/O on the real adapters. Dedicated CPUs are running the VIO Server all
the time.
A VIO Server LPAR can use shared CPUs, which is a good idea if you don't have whole
CPUs that can be assigned. This also means unused CPU cycles are given back to the
shared pool for other LPARs to use. If the machine becomes heavily loaded, it can
introduce tiny delays in starting the I/O on real adapters. Shared CPU partitions are time-
sliced onto the CPU, along with other LPARs. Setting the VIO Server partition to
Uncapped and with a high weight is generally a good idea.
A simple CPU rule of thumb: Assign at least ten percent of those CPUs to the VIO Server
for CPUs that are going to be used for the VIO Server and client partitions. For example,
for five CPUs in the shared pool being used for both VIO Server and VIO clients,
allocate 0.5 of a CPU to the VIO Server.
A simple memory rule of thumb: Use 512 MB of memory.
It's recommended to have an LPAR profile with the adapter connected to the CD drive
included to make installing the IBM VIO Server from CD straightforward. Copy the
Normal profile and rename it Normal with CD. Then change the new profile properties
to include the CD SCSI adapter. This will be used to initially boot the LPAR with a
DVD/CD drive for installing the VIO Server.
If this is a new machine and you are the only user, installations go much faster if you
assign the LPAR a whole CPU or more. If the LPAR is going to be assigned less than this
in production, it can always be reduced later, but this simple trick might save you ten
minutes per LPAR installation.
Back to top
Now install the VIO Server into this partition as a recover from mksysb image methods. AIX
systems administration experts will be familiar with this. The basic steps are:
On the HMC in the Activate LPAR dialog, boot the LPAR into the System Management
Services (SMS) menu by selecting both Open a Terminal and the Advanced button and
then Boot Mode = SMS. Once in the SMS menus, choose the Boot Options and select
Install/Boot. Then choose List all Devices and carefully select the CD-ROM drive,
Normal Boot, and Yes to leave the SMS menus.
Read the instructions carefully and, if free, elect to install the VIO Server on hdisk0.
(This is assumed in the rest of the article.)
Warning: If you want to use a whole disk for your VIO client, then you need to make
sure the recovery of the VIO Server mksysb is not spread across all the disks.
Assuming you now have the VIO Server up and running, you need can add and set up the
VIO Server virtualization features.
Important hints
If you have the DLPAR change software installed and working, it's possible to dynamically add
Virtual Ethernet and virtual SCSI. In practice, I recommend you shutdown your VIO Server and
VIO client LPARs during this initial setup to make sure it works the first time. If you set up
DLPAR later on, you can then experiment, but remember DLPAR changes also have to be
implemented identically in your LPAR profile -- if you want the same configuration next time,
you need to reboot your LPAR.
In this article, I take a simple and ultra-safe approach to shut down the VIO Server, making
changes to the VIO Server profile and restarting it to avoid any confusion and complications. So
on the VIO Server, use shutdown.
If you make changes to an LPAR profile, the LPAR must be shutdown and then restarted from
the HMC to pick up those changes. If you use shutdown restart in the VIO Server LPAR, then
you'll have only the same resources that were available when the LPAR was previously started
from the HMC. You need to completely stop the LPAR to get the new resources.
Back to top
On the HMC, you can now define the Virtual Ethernet. First shutdown the VIO Server LPAR (as
root run: shutdown). On the HMC, change your Normal profile properties by right-clicking the
Profile and selecting Properties. You also need to select the VIO tab. Select Ethernet at the
bottom and click Create. By default, this will be allocated to slot number 2 and a Port virtual
LAN ID of 1. Any LPAR with the same Port virtual LAN ID will be able to communicate with
each other. This is going to be set up as SEA. To do this, you want to log into the VIO Server
over the network and set these two options:
If you want different Virtual Ethernet LANs so that different groups of LPARS can communicate
with each other, all you need to do is have different Port virtual LAN ID numbers. These
complex configurations are not covered in this article.
In Figure 13, you should see the VIO Server in the lower half and VIO client in the top half. It
shows that if the Port virtual LAN IDs are the same, then the LPARs can communicate. It also
shows the additional settings for the VIO Server. (The trunk is selected and IEEE 802.1Q is not
selected.) These additional settings are really for the bridging feature, as Virtual Ethernet does
not really have a client/server relationship -- all LPARs are equal on the network.
Figure 13 also shows the Virtual Ethernet settings. At the bottom is the VIO Server (or any
LPAR that will be doing the bridging to the real Ethernet adapter) and at the top is the VIO
client, or any LPAR that only uses the Virtual Ethernet.
On the other Virtual Ethernet LPARs, you can use the ifconfig command to set up your
network just as you would any network. If it's AIX, you can use smitty or websm. If it's SUSE,
the YAST tool can be used. It finds the Virtual Ethernet adapter just like any other regardless of
the tool selection. Figure 14 shows a SUSE example.
Figure 14. Non-bridging Virtual Ethernet LPARs
Back to top
On the HMC, you can now define the two different virtual SCSI devices. These two types of
virtual disks (a logical volume or a whole hdisk) appear identical on the HMC; only on the actual
VIO Server LPAR are they set up any differently.
If not done already, shutdown the VIO Server LPAR (as root run: shutdown).
On the HMC, select the VIO Server and change your Normal profile properties by right-
clicking the Profile and selecting Properties. You also need to select the Virtual I/O
tab.
Select SCSI at the bottom and click Create.
This is the VIO Server, so select Adapter Type: Server.
Select Any Remote Partition and Slot can connect. Ideally, this should name the
specific LPAR and slot to eliminate the risk of the wrong connection between server and
client, but at this point you have not created the client partition, so you can't name it yet.
This is fixed up later; see Clean up the HMC section for details.
Select OK.
Do this a second time for the second virtual SCSI.
The client LPARs are going to use the VIO Server slots 3 and 4. Any more SCSI adapters are
optional in this example. In practice, the writer typically sets up a handful of extra virtual
devices, so they can be used in the future without stopping the VIO Server or having to do
dynamic changes. Unused virtual adapters cost very little, so it's not a waste.
In Figure 15, you have the eventual configuration showing how the VIO Server at the bottom
and client at the top both explicitly refer to each other to eliminate errors. You'll reach this
configuration in Step 11. I have not covered it yet, but the client LPAR is, of course, shown here
too.
Back to top
This might be obvious, but you don't need real adapters for your disks or Ethernet connection
because you're going to use virtual resources for these.
I recommend you install the client LPAR using CD because it's simple, so you'll want to have the
CD SCSI adapter within your LPAR. Once installed, this can be removed from the LPAR
profile.
It's recommended that you create two identical LPAR profiles -- one with and one without the
CD. Once installed, the writer uses NFS to remotely mount a filesystem containing the AIX and
Linux CDs, so you don't need the CD drive from then on to install additional LPP or RPM
packages.
Add the Virtual Ethernet adapter on the VIO screen with the same Port virtual LAN ID, which is
1 in this example, but do not select the trunk or IEEE 802.1Q compatible adapter options.
Don't forget, you have two client LPARs to create with the two different SCSI Remote Partition
virtual slot numbers but the same Virtual Ethernet Port virtual LAN ID.
Back to top
Now that you've created the client LPARs, you can go back to the VIO Server LPAR and
connect up the virtual SCSI adapters explicitly to their virtual client LPARs and slots. This
ensures that only the right client LPAR connects to the right virtual SCSI disk. It's a safety
precaution and worth doing.
On the HMC, highlight the VIO Server LPAR profile and bring up its properties. In the VIO
tab, select each Server SCSI resource and the Properties button. You also need to set:
In this example, you have two virtual SCSI adapters on the VIO Server to "clean up". This is
very easy to get wrong, and this is why I planned it in advance (see Table 1).
Back to top
You now have all the connections set up for the VIO Server and virtual clients, but still have to
connect the virtual SCSI disk to a piece of real disk space and the virtual and real Ethernets using
the SEA. This is done on the VIO Server only as follows. First, start the VIO Server LPAR again
from the HMC.
Once the VIO Server is running and assuming no network is set up, you need to find out the
names of the virtual resources you have to work with using:
$ lsdev -virtual
name status description
nt2 Available Virtual I/O Ethernet Adapter (l-lan)
vhost0 Available Virtual SCSI Server Adapter
vhost1 Available Virtual SCSI Server Adapter
vhost2 Available Virtual SCSI Server Adapter
vhost3 Defined Virtual SCSI Server Adapter
vsa0 Available LPAR Virtual Serial Adapter
clientY Available Virtual Target Device - Logical Volume
clientZ Available Virtual Target Device - Logical Volume
If your real Ethernet adapter has more than one port, it can be confusing since your Virtual
Ethernet will have a higher number. In the case of a two-port Ethernet card, the Virtual Ethernet
name might be en2, as en1 can be the second port on the real adapter. Also, make sure you plug
in the Ethernet cable into the right port.
On your machine, the resource names might be slightly different, so be careful in following this
example.
Create the SEA between the real and Virtual Ethernets with (Note: Do not type the arrow or
comments when you try this.) :
And created the SEA with a name of ent3. Take a look with:
$ lsdev -dev en3
name status description
ent3 Available Shared Ethernet Adapter
Now program the TCP/IP details on the SEA adapter. This command is used instead of smitty
(which is not available on the VIO Server) or ifconfig (not allowed for the SEA). You'll, of
course, have to decide your own hostname and address.
Once the VIO Server is running and before the virtual clients are started, you need to create the
disk space and connect it to the virtual SCSI resource that the VIO client will try to attach to.
Here you see the VIO Server is using the first disk and the others are currently unused. Next,
take a look at the free space on that first disk and the primary volume group called rootvg:
$ lsvg
rootvg
$ lsvg rootvg
VOLUME GROUP: rootvg VG IDENTIFIER: 00c033ea00004c000000010104ffa3fc
VG STATE: active PP SIZE: 128 megabyte(s)
VG PERMISSION: read/write TOTAL PPs: 271 (34688 megabytes)
MAX LVs: 256 FREE PPs: 151(18328 megabytes)
LVs: 14 USED PPs: 120(15636 megabytes)
OPEN LVs: 12 QUORUM: 2
TOTAL PVs: 1 VG DESCRIPTORS: 2
STALE PVs: 0 STALE PPs: 0
ACTIVE PVs: 1 AUTO ON: yes
MAX PPs per VG: 32512
MAX PPs per PV: 1016 MAX PVs: 32
LTG size (Dynamic): 256 kilobyte(s) AUTO SYNC: no
HOT SPARE: no BB POLICY: relocatable
$ lsvg -pv rootvg
rootvg:
PV_NAME PV STATE TOTAL PPs FREE PPs FREE DISTRIBUTION
Hdisk0 active 271 55 22..09..00..00..24
$
$ lspv hdisk0
PHYSICAL VOLUME: hdisk0 VOLUME GROUP: rootvg
PV IDENTIFIER: 00c033eaf709961e VG IDENTIFIER
00c033ea00004c000000010104ffa3fc
PV STATE: active
STALE PARTITIONS: 0 ALLOCATABLE: yes
PP SIZE: 128 megabyte(s) LOGICAL VOLUMES: 14
TOTAL PPs: 271 (34688 megabytes) VG DESCRIPTORS: 2
FREE PPs: 151(19328 megabytes) HOT SPARE: no
USED PPs: 120 (15636 megabytes) MAX REQUEST: 256 kilobytes
FREE DISTRIBUTION: 22..09..00..00..24
USED DISTRIBUTION: 33..45..54..54..30
$
The lv00 confirms the name used to create the logical volume.
To connect this to the VIO client resource, the name of the resource (here it's clientx) is used to
make it very clear which partition is using it, but you could use any suitable name:
You can now start the VIO Client X and find its virtual SCSI resources. In this example, you've
defined just one logical volume for this VIO client, but many logical volumes could be used. I
recommend you keep them to a minimum to make the configuration simpler.
In the above section, you found that hdisk1 was unused. This disk will be used to support the
VIO Client Y. The disk must not be in a volume group. In the lspv command above, there is no
logical volume group name next to the disk, so it's not in a volume group. To connect this disk to
the virtual SCSI disk for VIO Client Y, try:
$ lsdev -virtual
name status description
ent2 Available Virtual I/O Ethernet Adapter (l-lan)
vhost0 Available Virtual SCSI Server Adapter
vhost1 Available Virtual SCSI Server Adapter
...
vsa0 Available LPAR Virtual Serial Adapter
clientx Available Virtual Target Device - Logical Volume
clienty Available Virtual Target Device - Logical Volume
ent3 Available Shared Ethernet Adapter
$
$ lsdev -dev clientx -attr
attribute value description user_settable
LogicalUnitAddr 0x8100000000000000 Logical Unit Address False
aix_tdev lv00 Target Device Name False
$ lsdev -dev ent3 -attr
attribute value description
pvid 3 PVID to use for the SEA device
pvid_adapter ent2 Default virtual adapter to use for non-VLAN-tagged
real_adapter ent0 Physical adapter associated with the SEA
thread 0 Thread mode enabled (1) or disabled (0)
virt_adapters ent2 List of virtual adapters associated with the SEA
Back to top
Now you can start up your VIO clients and install them. This can be AIX (but only if the
hardware is pSeries p5 and not OpenPower), SUSES SLES 9, Red Hat 3 update 3 onwards, or
Debian. They should find both the:
Virtual Ethernet, which will be named a Virtual Ethernet and behave like a real physical
adapter.
Virtual SCSI disk, which is presented just like an SCSI disk, but it will only be the size of
the underlying disk partition or disk.
These should install just like a regular real Ethernet and SCSI disk.
For AIX, this installation should be just like a normal AIX partition.
For Linux, here are some additional notes. Once running, the Virtual Ethernet looks and behaves
like a very fast one GB real adapter:
clienta:~ # ifconfig
eth0 Link encap:Ethernet HWaddr AE:38:00:00:D0:02
inet addr:9.137.62.178 Bcast:9.137.62.255 Mask:255.255.255.0
inet6 addr: fe80::ac38:ff:fe00:d002/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:1075 errors:0 dropped:0 overruns:0 frame:0
TX packets:350 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:113566 (110.9 Kb) TX bytes:40940 (39.9 Kb)
Interrupt:184
Once running, you can see the virtual SCSI disk is being treated just like a regular disk:
clienta:~ # fdisk -l
Disk /dev/sda: 4194 MB, 4194304000 bytes
130 heads, 62 sectors/track, 1016 cylinders
Units = cylinders of 8060 * 512 = 4126720 bytes
Device Boot Start End Blocks Id System
/dev/sda1 * 1 1 3999 41 PPC PReP Boot
/dev/sda2 6 132 511810 82 Linux swap
/dev/sda3 133 1016 3562520 83 Linux
And you'll find the IBM virtual SCSI client kernel module installed (see ibmveth and ibmvscsic
below):
clienta:~ # lsmod
Module Size Used by
evdev 31416 0
joydev 31520 0
st 72688 0
ipv6 478560 93
sg 74176 0
usbcore 183644 1
ibmveth 44536 0
subfs 30168 2
dm_mod 108224 0
ibmvscsic 43072 2
sr_mod 44380 0
sd_mod 43792 3
scsi_mod 192024 5 st,sg,ibmvscsic,sr_mod,sd_mod
Network installation might be trickier, as you might have to activate the devices drivers for the
installation tools to find the virtual adapters. Network installations are not covered in this article.
Depending on which release you use, the installer might give you, with a series of menus,
options to install the IBM virtual SCSI client and Virtual Ethernet drivers before you start an
installation properly. Later releases fully understand and install the device drivers for these
virtual resources without manual intervention.
Back to top
Once you've created your client LPAR and set it up the way you like, you should consider
backing up the operating system images. Backups are a large subject for which many books have
been written. There are many backup solutions from both commercial applications and freely
available tools in the AIX and Linux world. For AIX, IBM has the Tivoli® Storage Manager
product for remote backups. For Linux, one of the popular freely available tools is Amanda
(Advanced Maryland Automatic Network Disk Archiver), which provides remote backup with
disk caching and tape library management. There is also a Linux "Backup and Recovery How
To" on the Internet for more information.
This article just covers the special considerations for VIO Servers and VIO clients. Backups are
important for at least these three reasons, and these apply to VIO systems, too:
HMC
The HMC data includes definitions of the LPAR physical resources such as CPU, memory, PCI
slots, and definitions of the LPAR virtual resources (such as the connections between VIO
Server and VIO clients).
See Effective System Management Using the IBM Hardware Management Console for pSeries,
SG24-7038-00.
If the HMC fails, the data is still held in the Service Processor (FSP) and can be read by a
replacement or recovered HMC. It's vital that the configuration details are available in case of a
disaster. HMC backups are documented in manuals, InfoCenter help files, and IBM Redbooks.
It's also recommended that details of the LPARs are documented on paper. For example,
something similar to the planning table used to create the LPARs in this article.
VIO Server
The VIO Server itself needs to be backed up. There is the backupios command for saving the
rootvg volume group to either a tape, filesystem, CD, or DVD. The other volume group structure
(not the contents) can be saved and restored with the savevgstruct and restorevgstrct
commands. To save the contents of other volume groups (not rootvg), you'll have to make other
arrangements such as using the oem_setup_env command, dd command, savevg command, tar
command, or other backup solutions.
If the VIO Server is purely being used for virtual I/O, then you need to back up:
The details of the client logical volumes or the details of hdisks. Details include number
of logical volumes, their size, the disk layout, which clients, and their use.
The contents of the client logical volumes or hdisks.
To recover the VIO Server, you can simply reinstall the original install image, which is a mksysb
image in much the same way as you installed the VIO Server in the first place.
There are different approaches to backing up the VIO client images from the VIO Server. First,
note that you have the option of doing hot or cold backups:
Hot backup
A hot backup is while the VIO client is running. This is dangerous and is not
recommended.
Cold backup
A cold backup is the only sensible way to back up the client from the VIO Server. This is
simply a matter of shutting down the VIO client. For Linux as root on the client, try:
shutdown -fh now, and for AIX shutdown -Fh.
For the logical volume method, you have to use the dd command to copy the logical
volume images to a file, tape, NFS, and so on.
For the whole hdisk method, the large size probably means a tape is the best option.
The cp command is not a good idea, since it copies a file using small blocks and is very
inefficient and slow. A better command is dd with a large block size. For example, 64 KB
blocks use the bs=64k option. To copy a logical volume, try:
Alternatively, you can back up straight to a tape drive using a command like tar or
backup. Some machines support a writeable DVD device that can also be used as a
backup medium.
Because AIX and the VIO Server can perform DLPAR changes of PCI slots, a single tape
drive and its associated SCSI adapter can be moved to the VIO Server for the backup
period, and then removed so it can be used in other LPARs.
Recovery of a VIO client involves getting the disk image back in the right place and
starting the VIO client LPAR again.
VIO client
The VIO client can be used to back up its own data just like a regular LPAR running AIX or
Linux. With an AIX VIO client, the best backup method is the mksysb command.
It's unlikely that VIO client LPARs will have its own tape, since the purpose of a VIO client is to
share physical resources to reduce hardware requirements. As with the VIO Server, the DLPAR
changes of PCI slots can be used to temporarily introduce a tape driver to the client and then for
it to back up its own data. Automating this process can be hard to coordinate between lots of
client LPARs, but it can be done using scripts on a central machine. Some machines support a
writeable DVD device that can also be used as a backup medium.
A second option is for the client to use another LPAR (possibly the VIO Server) or another
machine to save the data using either a:
In all three cases, the high speed of the Virtual Ethernet can boost backup performance.
Recovery using these methods can be harder work. With AIX, you can simply recover the
mksysb image. With Linux, you might have to reinstall from the original CD-ROMs and then
overwrite the running Linux with the backup.
Back to top
Another option is to create a copy of the VIO client SCSI disk and use it as the virtual SCSI disk
for a new VIO client LPAR.
For the logical volume, create another logical volume of identical size and use the dd command
to create a copy of the original logical volume. Be careful with the header structure of the logical
volume; AIX can keep some information in the first block.
For the whole disk, you would need a disk of identical size and then copy from the original to the
new partition using the dd command.
You need to check that the cloned LPAR is not using the same Ethernet IP addresses as the
original. Alternatively, you can clone the original LPAR before putting it on the network.
Back to top
For AIX 5.3, VIO clients have the DLPAR and RAS features already installed.
For the Linux VIO clients, these need to be added as described below.
After installing Linux, it's strongly recommended that you install the IBM packages for both
DLPARs (this works for physical and virtual resources) and the daemons and tools to increase
RAS. This ensures that you get the expected reliability from your pSeries p5 or OpenPower
machine. These RPMs put the LPAR in touch with the HMC for dynamic changes, reporting
problems, and provides tools to use on the LPAR, too.
The tools can be downloaded from Service and productivity tools for Linux on POWER. At this
Web site, select the right version of Linux tab then the HMC-Managed option to find the list.
Below is the list of RPMs at the time of writing this article, but the version numbers might have
since been updated since.
1. src-1.2.2.1-0.ppc.rpm
2. rsct.core.utils-2.3.4.2-0.ppc.rpm
3. rsct.core-2.3.4.2-0.ppc.rpm
4. csm.core-1.4.0.3-79.ppc.rpm
5. csm.client-1.4.0.3-79.ppc.rpm
6. devices.chrp.base.ServiceRM-2.2.0.0-1.ppc.rpm
7. DynamicRM-1.1-2.ppc.rpm
8. rpa-dlpar-1.0-11.ppc.rpm
9. rpa-pci-hotplug-1.0-8.ppc.rpm
10. librtas-1.1-12.ppc64.rpm
A prerequisite is the rdistcommand. For SUSE Linux, this is on the Linux SuSE SLES 9 CD3
and file name rdist-6.1.5-792.1.ppc.rpm.
Download these RPMs and install or update existing packages. You should find you can now do
DLPAR changes. In addition, Linux can report problems to the HMC, which is used to forward
problems to IBM (if set up) and used by hardware maintenance staff for diagnosis and
correction.
A similar procedure is available for Red Hat and other Linux versions. See the above Web site
for details about the packages you need to install.
Back to top
Conclusion
This article described the virtualization capabilities of IBM POWER5 servers and, through
examples that apply equally to both pSeries p5 and eServer OpenPower systems, showed how to
set up and use the VIO Server.
Virtualization is a hot topic in the computing industry. The POWER5-based machines provide
opportunities for a significant reduction in operating costs for complex environments. Unlike
software solutions available from other vendors, the POWER5 implementation uses advanced
processor features (Hypervisor) and hardware features to create efficient and flexible
virtualization capabilities.
Acknowledgements
The author wishes to thank Dave Williams and Stephen Atkins, both of IBM UK, for reviewing
and improving this article.
Resources
Get a full list of Linux on Power applications and more on the way.
Visit the Virtual Innovation Center for Hardware for AIX development support. This is
the primary source for all pSeries AIX development.
Want more? The developerWorks AIX and UNIX zone hosts hundreds of informative
articles and introductory, intermediate, and advanced tutorials on the eServer brand.
The IBM developerWorks team hosts hundreds of technical briefings around the world
which you can attend at no charge.
Nigel Griffiths is the EMEA Linux on POWER Hands-On Technical Advocate and EMEA p5
Virtualization Technical Focus Group Leader. He specializes in virtualization, performance,
sizing, and tools. Nigel is also a certified consulting member of the IT-Specialist profession. You
can contact Nigel at nag@uk.ibm.com.
Comments
Thank you for your comment. The problem with the images in the article has now been fixed.
Posted by jamaya on 24 March 2010
i cannot see pictures 12 to 15, instead there is this message: "XML error: The image is not
displayed because the width is greater than the maximum of 580 pixels. Please decrease the
image width."
Add comment
Back to top
Close [x]
What's this?
This little timesaver lets you update your My developerWorks profile with just one click! The
general subject of this content (AIX and UNIX, Information Management, Lotus, Rational,
Tivoli, WebSphere, Java, Linux, Open source, SOA and Web services, Web development, or
XML) will be added to the interests section of your profile, if it's not there already. You only
need to be logged in to My developerWorks.
And what's the point of adding your interests to your profile? That's how you find other users
with the same interests as yours, and see what they're reading and contributing to the community.
Your interests also help us recommend relevant developerWorks content to you.
Close [x]
What's this?
Removing this interest does not alter your profile, but rather removes this piece of content from a
list of all content for which you've indicated interest. In a future enhancement to My
developerWorks, you'll be able to see a record of that content.
static.content.url=http://www.ibm.com/developerworks/js/artrating/
SITE_ID=1
Zone=AIX and UNIX
ArticleID=87326
ArticleTitle=POWER5 virtualization: How to set up the IBM Virtual I/O Server
publish-date=06292005
author1-email=nag@uk.ibm.com
author1-email-cc=mmccrary@us.ibm.com
url=http://www.ibm.com/developerworks/aix/library/au-aix-vioserver-v2/index.html
Table of contents
Introduction
What is a VIO Server?
Why use the VIO Server?
Prerequisites
Getting started
1. Logical diagram
2. Planning your setup
3. Creating the VIO Server LPAR
4. Installing the VIO Server
5. Defining the VIO Server -- Virtual Ethernet
6. Defining the VIO Server -- virtual SCSI
7. Creating the VIO client LPARs
8. Cleaning up the HMC
9. VIO Server -- preparing for the clients
10. VIO client LPAR installations
11. Backing up a VIO Server and VIO client
12. Cloning a client
13. Linux DLPARs and RAS
Conclusion
Resources
About the author
Comments
My developerWorks community
Interact, share, and communicate with developers worldwide.
My Home
Profiles
Groups
Blogs
Bookmarks
Activities
Files
Wikis
Forums
Podcasts
My developerWorks overview
Tags
Use the search field to find all types of content in My developerWorks with that tag. Popular
tags shows the top tags for this particular content zone (for example, Java technology, Linux,
WebSphere). My tags shows your tags for this particular content zone (for example, Java
technology, Linux, WebSphere).
Popular tags
admin (7)
administrator (8)
administrators (9)
aix (513)
aix_and_unix (224)
application_d... (41)
arpan_sen (7)
articles (11)
audit (7)
authentication (10)
bash (9)
boost (10)
c (13)
c++ (6)
command_line (10)
commands (21)
cpu (9)
debugging (6)
disk (8)
dw (21)
file_system (6)
green (6)
hacmp (8)
ibm (14)
ibm_nas (6)
installation (9)
io (7)
java (12)
kdc (10)
ken_milberg (11)
kerberos (25)
kernel (7)
ldap (24)
linux (50)
lpar (14)
martin_brown (7)
martin_streich... (12)
memory (18)
migration (15)
nas (17)
network (16)
nfs (12)
nmon (11)
open_source (20)
operating_sys... (20)
optimizing (8)
p (8)
perf (8)
performance (44)
porting (8)
power (13)
power6 (7)
programming (7)
python (23)
scripting (12)
security (33)
shell (13)
solaris (10)
speaking_unix (31)
ssh (14)
storage (7)
sysadmin (9)
system (14)
system_admini... (88)
system_p (32)
systems_admin (14)
test (11)
testing (10)
tips (12)
toolkit (15)
tools (36)
tools_and_ut... (96)
troubleshootin... (8)
tuning (24)
unit (9)
unix (206)
vio (11)
vios (9)
virtualizatio... (16)
wpar (15)
My tags
To access My Tags, please sign in
End of My tags
MoreLess
admin (7)
administrator (8)
administrators (9)
aix (513)
aix_and_unix (224)
application_d... (41)
arpan_sen (7)
articles (11)
audit (7)
authentication (10)
bash (9)
boost (10)
c (13)
c++ (6)
command_line (10)
commands (21)
cpu (9)
debugging (6)
disk (8)
dw (21)
file_system (6)
green (6)
hacmp (8)
ibm (14)
ibm_nas (6)
installation (9)
io (7)
java (12)
kdc (10)
ken_milberg (11)
kerberos (25)
kernel (7)
ldap (24)
linux (50)
lpar (14)
martin_brown (7)
martin_streich... (12)
memory (18)
migration (15)
nas (17)
network (16)
nfs (12)
nmon (11)
open_source (20)
operating_sys... (20)
optimizing (8)
p (8)
perf (8)
performance (44)
porting (8)
power (13)
power6 (7)
programming (7)
python (23)
scripting (12)
security (33)
shell (13)
solaris (10)
speaking_unix (31)
ssh (14)
storage (7)
sysadmin (9)
system (14)
system_admini... (88)
system_p (32)
systems_admin (14)
test (11)
testing (10)
tips (12)
toolkit (15)
tools (36)
tools_and_ut... (96)
troubleshootin... (8)
tuning (24)
unit (9)
unix (206)
vio (11)
vios (9)
virtualizatio... (16)
wpar (15)
admin 7
administrator 8
administrators 9
aix 513
aix_and_unix 224
application_d... 41
arpan_sen 7
articles 11
audit 7
authentication 10
Special offers
About IBM
Privacy
Contact
Terms of use
IBM Feeds
Jobs