AXP02 AIX5L On System I HA Solutions

International Technical Support Organization
Leveraging System i
for Linux and AIX HA and DR
Yessong Johng
International Technical Support Organization
IBM Rochester
© 2007 IBM Corporation. All rights reserved.

IBM STG Technical Conference
Agenda
AIX Topics
• Factors that shape the AIX 5L high availability (HA) solution
• Two types of HA solutions for AIX 5L on System i
• Decision flowchart
• HACMP introduction
• Simple HACMP cluster with shared disk
• HACMP/XD
• HACMP/XD for GLVM
• Integration with i5/OS for HA
• Summary and comparison
Linux Topics
• i5/OS Based Disaster Recovery Solution
• HA: Keeping Linux running even when the hosting partition goes down
© 2007 IBM Corporation

Factors that Shape the AIX 5L HA Solution

Type of HA required
• Same data center? Local and remote sites?
• RTO and RPO
¾ How long an outage can I tolerate?
¾ How much data can I lose?
• HA only or HA + DR?
• How is DR defined? Is 50 mi a sufficient distance, or is 1000 mi?
• Data availability only or data availability with automatic app restart?
Existing environment
• Storage type
¾ Most often SAN
¾ Internal SCSI and virtual storage also used
• Staff skills
¾ Familiar with HACMP?
¾ New to AIX 5L?
Core business applications and databases

• Must application state be preserved? Can transactions be restarted?
• Can the application or database provide clustering or HA?

Identifying Pain Points in Existing Solutions

CIO/management
• HA? Does our business have any? (Found in SMB, not large
enterprises)
• High cost (including annual maintenance)
• Long recovery time, effect on business operations
• Additional staff needed to maintain
Technical staff
• Lack of understanding, solution configured by 3rd party
• Significant manual intervention (long hours) needed in case of failover
• Too complex, different solutions for different servers/workloads
• Difficult to obtain support from vendor
Users
• Unable to recover recent files
• HA practices affecting core business operations

Effect of HA Element on Complete Solution

HA frequently an afterthought, especially in new consolidation
solutions
• Happens most frequently in SMB space – funds and hardware are
limited
• Focus is on other important areas – application support, hardware
configuration
HA approach can influence:

• System I/O configuration and, therefore, total system cost
• Choice of storage (SAN provides most flexibility, but can be
costlier)
• Application or database performance
• Additional software purchases
• Staff training needs

Standard and System i-specific Solutions
Standard AIX 5L HA System i-specific

Solution Option
Data center HA HACMP + SAN Dual VIO from i5/OS
Remote HA HACMP/XD VIO + Copy Services

(SAN)
Remote HA HACMP for GLVM VIO + XSM
(SCSI)
Positioning Lead-with Solution in HA integration solution
most cases for some environments
HACMP will be the preferred approach for most clients – why?

• Proven product, references (on System i and System p)
• Abundant documentation (Web sites, Redbooks)
• Clear support channel

Basic Decision Flowchart
Criteria used:
• Hardware type
• Local HA or remote HA/DR
HACMP Overview
Scalable from 2 to 32 server nodes
Automatic failure detection and
faster recovery
Near-continuous data availability
(less than 10 seconds for disk
Server Workload Server recovery)
A Fallover B
Unlimited distance clusters

Shared Disk Dynamic cluster reconfiguration
Automatic configuration discovery
HACMP Clustered file system support
Cluster single point-of-control
Local Cluster
Configurable recovery policies and
options

Simple HACMP Cluster

Service IP
address
WAN/Internet
Client
Network Network
Switch Switch
Primary and secondary node
• Primary node owns the “shared” disk
and service IP address
• On failover, secondary node takes Primary Secondary
over the “shared” disk and service IP Node Node
address
• Similar to i5/OS clustering with
switched disk
External storage required for “shared” Fibre Switch Fibre Switch

disk
• Disk visible to both nodes, but active FAStT900
External storage FAStT/DS4000
only in primary node DS6000/DS8000
• Connectivity through a switch

required

HACMP Networking
Network Network
Switch Switch
Network Network
Switch Switch
Service1
9.180.20.1 198.168.2.1 Primary_boot2 198.168.2.2
9.180.20.12
198.168.1.1
Primary_boot1 Secondary_boot1 Secondary_boot2
9.180.20.11 9.180.20.21 9.180.20.22
198.168.1.2
Primary Secondary
Node Node

HACMP/XD
HACMP/Extended Distance adds Disaster Recovery
• Data replication provided by DS Copy Services (Metro or Global Mirror)
• HACMP automates heartbeating and switchover/failover
• Allows for a common HA strategy if i5/OS is using Copy Services for
replication
WAN
Server Server
A B
DS Metro Mirror
DS8000 1 Hardware-based DS8000 2
Data Mirroring

HACMP/XD for GLVM (formerly HAGEO)

Data replication is provided by AIX 5L itself, by using remote physical
volumes (RPVs) in LVM: local and remote volumes are mirrored
HACMP automates heartbeating and switchover/failover
Disk units can be internal or on external storage
WAN
Server Server
A B
Disk-based replication
by using remote
volumes
AIX 5L AIX 5L
LVM LVM
Managing HACMP
System Management (C-SPOC)
Move cursor to desired item and press Enter.
Manage HACMP Services

HACMP Communication Interface Management
HACMP Resource Group and Application Management
HACMP Log Viewing and Management
HACMP Security and Users Management
HACMP Logical Volume Management
HACMP Concurrent Logical Volume Management
HACMP Physical Volume Management
Open a SMIT Session on a Node
C-SPOC (Cluster Single Point of Control) is integrated with SMIT upon HACMP installation
Go directly to this menu with ‘smitty cl_admin’

HACMP Is Supported on System i

5765-F62: IBM High Availability Cluster Multi-processing
(HACMP) for AIX 5L, Version 5.4:
• Same product for System i and System p
• HACMP/XD and HACMP/XD for GLVM are supported
• Micro-partitioning is supported
¾ Micro-partitioning for AIX 5L does not require the Advanced
POWER Virtualization feature on System i
• Does not support virtual storage from i5/OS
• AIX 5L on System i uses the same adapters as on System p to
connect to FAStT or ESS/DS6000/DS8000
¾ 4-Gbit/s or 2-Gbit/s

AIX 5L HA Integration with i5/OS

Data replication is provided by i5/OS Geographic Mirroring
AIX 5L uses virtual storage, located in an IASP
Example shows integrated AIX 5L and iSCSI environment

AIX 5L HA Integration with i5/OS

Data replication is provided by System Storage Copy Services
AIX 5L uses virtual storage, located in an IASP
Example shows integrated AIX 5L and iSCSI environment

Local HA with Virtual Storage

AIX 5L partitions can
i5/OS LPAR Linux/AIX 5L LPAR i5/OS LPAR access virtual disk from two
or more i5/OS partitions
Two virtual disks of equal
Virtual SCSI size from separate i5/OS
partitions allow mirroring of
AIX 5L system disk
Two Virtual SCSI
OS-level Mirroring server/client adapter pairs
required, one for each i5/OS
partition providing storage
Mirroring accomplished
Virtual Ethernet with LVM within AIX 5L
AIX 5L partition becomes
highly available, able to
withstand failure of either
Two VIO Servers can be used in place of two i5/OS host i5/OS partition
partitions

Hardware Configuration for HA Solutions

Challenge: Which Fibre adapter to use?
• If HACMP is involved, must I use System p Fibre cards?
• Are the same 4-Gbit/s Fibre cards available on System i?
Solution: Consult the “AIX 5L Facts and Features”
• http://www-03.ibm.com/servers/eserver/iseries/aix/getting_started.html, Storage Interfaces section
Challenge: How many Fibre adapters?

• Am I accounting for MPIO for the AIX 5L partitions?
• Does the hardware configuration contain enough slots and exp. units?
Solution: Use the System Planning Tool (SPT)
Challenge: How many network adapters?

• Using 2 Ethernet cards for heartbeat is recommended (does not always happen)
• Can Virtual Ethernet between LPARs be used?
Solution: Use the System Planning Tool (SPT)

Hardware Configuration: SPT Example
Take advantage of #5790 exp. units when connecting to SAN

Greater flexibility: no IOP per Fibre adapter required
Summary and Comparison

Local HACMP/XD HACMP/XD XSM + VIO + Metro Mirror Dual Local
HACMP for GLVM IASP + VIO + IASP VIO
Differentiators Leading solution Dual-site HA Dual-site HA Single HA plan Single HA plan Transparent
for single-site to AIX 5L
Ext storage SCSI or SAN i5/OS replication Ext storage
HA
replication replication Leverage VIO
LVM replication
Local/remote Local Remote Remote Remote Remote Local
Separate LPP? Yes Yes Yes No Yes* No
App restart? Yes Yes Yes OS restart OS restart Not needed
IPAT? Yes Yes Yes Not needed Not needed Not needed
Type of SAN SAN SAN or SCSI SAN or SCSI SAN SAN or SCSI
storage
AIX 5L only? Yes Yes Yes AIX 5L, i5/OS, AIX 5L, i5/OS, AIX 5L, Linux
Linux, Windows Linux, Windows
Provides DR? No Possibly Yes Yes Possibly No
* This HA solution uses the Copy Services for System i offering for simplified management of external
storage replication.

Linux

i5/OS Based Disaster Recovery Solution:

Treating the storage space as a DR vehicle
When storage has been virtualized, the Virtual Disk
represents the entire Linux installation.
The IFS objects that represents the Virtual Disk (typically
located at /qfpnwsstg/<storage space name>) can be saved
with the SAV command.
• Backup of the entire Network Storage Space (i.e. Virtual Disk),
an IFS object, represents the entire Linux instance
¾ Operating system, programs, and data files
¾ Provides a complete backup of the Linux instance
A saved storage space can be restored on a new system and
the Linux instance can be brought back on line.
This method provides a good foundation for Disaster
Recovery solutions.


How to create a backup
Step 1: Vary off the Network Server that the virtual disk is
attached to
• The Network Server cannot be active while the storage space is saved
• This will take the Linux instance down and it will be unavailable to the users
Step 2: Unlink the storage space from the Network Server

• This is to avoid the storage space being linked to the Network Server when it is restored
Step 3: Write the storage space out to a savefile

Step 4: Relink the storage space to back to the production
network server
Step 5: Vary on the Network Server
Step 6: Write the savefile from step 3 out to tape


How to recover from a disaster using backup
Step 1: Create a Linux partition on the restoration system
• Ensure that the partition has a virtual SCSI adapter connected to the i5/OS
partition that the storage space will be restored on
Step 2: Create a Network Server on the restoration system
• Ensure that the Network Server references the new partition for its hardware
resources
Step 3: Restore the savefile that contains the storage space
Step 4: Restore the storage space from the savefile
Step 5: Link the restored storage space to the Network Server
(ADDNWSSTGL)
Step 6: Vary on the Network Server
At this point your Linux environment should be back-up and

running


Additional Items to Consider
When a storage space is saved the linkage parameter (i.e., what
Network Server(s) it is linked to) is saved as well.
When the storage space is restored, an attempt will be made to

automatically link it to the Network Server (if a network server
by that name exists)
So, to make the recovery process even easier, you should

ensure that the new Network Server has the same name that
was used previously and that if multiple storage spaces are
being restored they are restored in the order they were linked to
the Network Server in.

Complementing Linux Based Solution

Linux centric file level solution utilizing:
• find
¾ to locate files that meet a backup criteria, e.g. files modified since last backup
• tar
¾ to build an archive of those files
• gzip
¾ to compress that archive to save space
• cpio
¾ to write the resulting archive out to a save set
Advantages of this solution
• Linux can continue to run when files are backed up
• Size of save sets can significantly smaller
• Use standard Linux utilities requiring little i5/OS admin support
¾ Scripts are implemented within Linux that will perform the actual backup of files to
the IFS

High Availability
Keeping Linux Running when the Hosting Partition is Down

High-Availability
Keeping Linux running when the hosting partition goes down
Linux partitions can access

virtual disk from two or
more i5/OS partitions. i5/OS LPAR Linux LPAR i5/OS LPAR
Two virtual disks of equal

size from separate i5/OS Virtual SCSI
partitions allow mirroring of
the Linux system disk
Two Virtual SCSI
server/client adapter pairs OS-level Mirroring
required, one for each i5/OS
partition providing storage
Mirroring accomplished
with OS tools within Linux Virtual Ethernet
Linux partition becomes

highly available, able to
withstand failure of either
host i5/OS partition.

Introduction to RAID
RAID (Redundant Array of Independent Disks)
Introduced first around 1988
Allows users to access multiple independent disks like one

disk
• Many different types of RAID
The file system sits on top of RAID subsystem
RAID may be implemented by software or hardware

RAID 0
A striping technique
Allows for very fast data
access
Provides no fault tolerance
Generally implemented
with at least 2 disks
May also be used with 1
disk to take advantage of a
cache controller

RAID 1
Also known as disk mirroring

Improved read performance
Write performance comparable
to single disk
Excellent fault tolerance
Requires at least 2 disks
May be implemented at the
device, controller, or bus level

RAID 5
Stripes data across drives
Utilizes a rotating parity array
• Allows for overlapping read and
write operations
• I/O isn’t blocked on a single
parity disk
Parity data can be utilized to
rebuild the array
Can lose one disk in the set and
still access all the data in the set
Loss of second disk at the same
time results in loss of all data in
the set

RAID 10
The data is stripped for

performance (RAID 0) and
mirrored for protection
(RAID 1)
No parity data
Excellent fault tolerance


Virtual Disk and RAID

IFS with parity protection enabled provides RAID 5
protection
• Provides virtual disk fault tolerance
Two i5/OS partitions may be used to provide virtual disks
• Software RAID 1 (mirroring) can combine the two virtual disks
The result is a highly available Linux pPartition with
excellent fault tolerance
• i5/OS updates
• Data dackup
• Disk failures

Highly Available Linux with RAID 1
i5/OS i5/OS Linux

Host 1 Host 2
P P P
/dev/md0
/dev/sdb1
/dev/sda4
Virtual LAN

Configuring VDASD from Two i5/OS Servers
Give the power controlling partition the

higher slot number
• Ensures power controlling disks
are last
Prevents STMF RAM disk from
interfering with RAID configuration
The New Partition Profile
The partition profile

now shows two client
SCSI adapters
• Make sure both are
marked as required
Remember to activate
the profile once from the
HMC

Installing the Partition to RAID
Select to ‘Manually
Partition with Disk
Druid’
RHEL4’s Disk Druid
sees both dDisks
• The disk from the
secondary partition is
first
• Disk from the power
controlling partition is
second.
RHEL4 RAID Configuration
Create partitions of type

RAID
• The RAID partitions should
be of similar size
Need a PReP boot partition
on the first disk
Also need a /boot partition
Swap created on second
disk to keep space
utilization consistent
between disks
Swap could be on the RAID
set

RHEL4 RAID Configuration (cont.)
Clicking the ‘RAID’

button brings up the
‘RAID Options’
screen
Chose to create a
RAID device
This will associate
the RAID partitions
with a usable RAID
device

Create a RAID1 Device
Creates a RAID1 device

using both Virtual Disks
Set the mount point to
be ‘/’
The file system will be
formatted to ext3

Disk Druid Shows the Device and Mount
Answer subsequent
prompts and begin
installation
The root partition
will be installed to
the RAID device

Linux Runs with One Virtual Disk
i5/OS i5/OS Linux

Host 1 Host 2
P P P
Virtual LAN

Checking Raid Functionality
/proc/mdstat reports current

RAID state
Upon failure - messages show up

on the Linux console and in
/var/log/messages
/proc/mdstat shows sda3 failed

and only 1 of 2 disks are active

Monitoring with mdadm

The mdadm program can e-mail administrators after a
failure
‘mdadm –monitor –scan –mail=<e-mail address>’

Disk Recovery
Use the fdisk command to

ensure the disk is accessible
If the partitions has been

rebooted you may need to rescan
the SCSI bus
Use mdadm to remove the

failed disk

Add the Repaired Disk
Use the mdadm

command to add the
repaired disk back into
the RAID array
mdadm –add <RAID
Device> <RAID
Partition>

Monitoring Disk Recovery

Multiple ways to monitor
recovery
• mdadm –D <RAID Device>
• watch tail /proc/mdstat

Trademarks
The following are trademarks of the International Business Machines Corporation in the United States and/or other countries. For a complete list of IBM Trademarks, see
www.ibm.com/legal/copytrade.shtml: i5/OS, i5/OS logo, AIX, AIX 5L, BladeCenter,Blue Gene, DB2, e-business logo, eServer, IBM, IBM Logo, Infoprint,IntelliStation, System i ,
pSeries, OpenPower, POWER5, POWER5+, Power Architecture, TotalStorage, Websphere, xSeries, z/OS, zSeries
The following are trademarks or registered trademarks of other companies:

Java and all Java based trademarks and logos are trademarks of Sun Microsystems, Inc., in the United States and other countries or both
Microsoft, Windows,Windows NT and the Windows logo are registered trademarks of Microsoft Corporation in the United States, other countries, or both.
Intel, Intel logo, Intel Inside, Intel Inside logo, Intel Centrino, Intel Centrino logo, Celeron, Intel Xeon, Intel SpeedStep, Itanium, and Pentium are trademarks or registered
trademarks
of Intel Corporation or its subsidiaries in the United States and other countries.
UNIX is a registered trademark of The Open Group in the United States and other countries or both.
Linux is a trademark of Linus Torvalds in the United States, other countries, or both.
Other company, product, or service names may be trademarks or service marks of others.
NOTES:
Any performance data contained in this document was determined in a controlled environment. Actual results may vary significantly and are dependent on many factors
including system hardware configuration and software design and configuration. Some measurements quoted in this document may have been made on development-level
systems. There is no guarantee these measurements will be the same on generally-available systems. Users of this document should verify the applicable data for their
specific environment.
IBM hardware products are manufactured from new parts, or new and serviceable used parts. Regardless, our warranty terms apply.
Information is provided “AS IS” without warranty of any kind.
All customer examples cited or described in this presentation are presented as illustrations of the manner in which some customers have used IBM products and the results
they may have achieved. Actual environmental costs and performance characteristics will vary depending on individual customer configurations and conditions.
This publication was produced in the United States. IBM may not offer the products, services or features discussed in this document in other countries, and the information
may be subject to change without notice. Consult your local IBM business contact for information on the product or services available in your area.
All statements regarding IBM's future direction and intent are subject to change or withdrawal without notice, and represent goals and objectives only.
Information about non-IBM products is obtained from the manufacturers of those products or their published announcements. IBM has not tested those products and cannot
confirm the performance, compatibility, or any other claims related to non-IBM products. Questions on the capabilities of non-IBM products should be addressed to the
suppliers of those products.
Prices are suggested US list prices and are subject to change without notice. Starting price may not include a hard drive, operating system or other features. Contact your
IBM representative or Business Partner for the most current pricing in your geography.
Any proposed use of claims in this presentation outside of the United States must be reviewed by local IBM country counsel prior to such use.
The information could include technical inaccuracies or typographical errors. Changes are periodically made to the information herein; these changes will be incorporated in
new editions of the publication. IBM may make improvements and/or changes in the product(s) and/or the program(s) described in this publication at any time without notice.
Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites. The
materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk.
IBM may have patents or pending patent applications covering subject matter in this document. The furnishing of this document does not give you any license to these
patents. Send license inquires, in writing, to IBM Director of Licensing, IBM Corporation, New Castle Drive, Armonk, NY 10504-1785 USA.

AXP02 AIX5L On System I HA Solutions

Caricato da

Informazioni sul documento

Titolo originale

Copyright

Formati disponibili

Condividi questo documento

Condividi o incorpora il documento

Opzioni di condivisione

Hai trovato utile questo documento?

Questo contenuto è inappropriato?

Copyright:

Formati disponibili

AXP02 AIX5L On System I HA Solutions

Caricato da

Copyright:

Formati disponibili

International Technical Support Organization

© 2007 IBM Corporation. All rights reserved.

© 2007 IBM Corporation

Factors that Shape the AIX 5L HA Solution

 Core business applications and databases

© 2007 IBM Corporation

Identifying Pain Points in Existing Solutions

© 2007 IBM Corporation

Effect of HA Element on Complete Solution

 HA approach can influence:

© 2007 IBM Corporation

Standard and System i-specific Solutions

Standard AIX 5L HA System i-specific

Data center HA HACMP + SAN Dual VIO from i5/OS

Remote HA HACMP/XD VIO + Copy Services

 HACMP will be the preferred approach for most clients – why?

© 2007 IBM Corporation

Basic Decision Flowchart

 Unlimited distance clusters

© 2007 IBM Corporation

Simple HACMP Cluster

 External storage required for “shared” Fibre Switch Fibre Switch

• Connectivity through a switch

© 2007 IBM Corporation

© 2007 IBM Corporation

© 2007 IBM Corporation

HACMP/XD for GLVM (formerly HAGEO)

Move cursor to desired item and press Enter.

Manage HACMP Services

Open a SMIT Session on a Node

© 2007 IBM Corporation

HACMP Is Supported on System i

© 2007 IBM Corporation

AIX 5L HA Integration with i5/OS

© 2007 IBM Corporation

AIX 5L HA Integration with i5/OS

© 2007 IBM Corporation

Local HA with Virtual Storage

© 2007 IBM Corporation

Hardware Configuration for HA Solutions

 Challenge: How many Fibre adapters?

 Challenge: How many network adapters?

© 2007 IBM Corporation

Hardware Configuration: SPT Example

 Take advantage of #5790 exp. units when connecting to SAN

Summary and Comparison

Separate LPP? Yes Yes Yes No Yes* No

App restart? Yes Yes Yes OS restart OS restart Not needed

© 2007 IBM Corporation

© 2007 IBM Corporation

i5/OS Based Disaster Recovery Solution:

© 2007 IBM Corporation

i5/OS Based Disaster Recovery Solution:

 Step 2: Unlink the storage space from the Network Server

 Step 3: Write the storage space out to a savefile

© 2007 IBM Corporation

i5/OS Based Disaster Recovery Solution:

 At this point your Linux environment should be back-up and

© 2007 IBM Corporation

i5/OS Based Disaster Recovery Solution:

 When the storage space is restored, an attempt will be made to

 So, to make the recovery process even easier, you should

© 2007 IBM Corporation

Core business applications and databases

HA approach can influence:

HACMP will be the preferred approach for most clients – why?

Unlimited distance clusters

External storage required for “shared” Fibre Switch Fibre Switch

Challenge: How many Fibre adapters?

Challenge: How many network adapters?

Take advantage of #5790 exp. units when connecting to SAN

Step 2: Unlink the storage space from the Network Server

Step 3: Write the storage space out to a savefile

At this point your Linux environment should be back-up and

When the storage space is restored, an attempt will be made to

So, to make the recovery process even easier, you should

Linux partitions can access

Two virtual disks of equal

Linux partition becomes

Introduced first around 1988

Allows users to access multiple independent disks like one

The file system sits on top of RAID subsystem

RAID may be implemented by software or hardware

Also known as disk mirroring

The data is stripped for

Give the power controlling partition the

The partition profile

Create partitions of type

Clicking the ‘RAID’

Creates a RAID1 device

Upon failure - messages show up

If the partitions has been

Use the mdadm