HACMP High Availability

Testing and Updates

November, 2005

2005 Mt Xia Technical Consulting Group - All Rights Reserved.

Introduction
Scope
HACMP Testing Scenarios
Operating System Updates
HACMP Updates
Database and Application Updates

HACMP Update

2005 Mt Xia Technical Consulting Group - All Rights

Scope
Scope
This presentation covers failover and failback testing in an
example HACMP environment.
Also described here are update methods that need to be
considered for support and maintenance of the cluster.
This presentation is not designed as a step-by-step
procedure for these processes, only as a guideline for
development and implementation of update methodologies.

HACMP Update

2005 Mt Xia Technical Consulting Group - All Rights

Service Outages
Updates to the system microcode, operating system, HACMP,
applications and databases will require periodic service outages in
order to keep the systems up-to-date and current.
The time required to perform updates can vary widely depending upon
the methodology chosen to perform the updates.
The service outages may be classified as follows:
Significant
Moderate
Minimal

HACMP Update

2005 Mt Xia Technical Consulting Group - All Rights

Significant Service Outage

Significant service outages to business functions can be experienced
by implementing an update methodology as follows:
Stop applications
Stop database
Update operating system
Update system microcode
Update adapter microcode
Update HACMP
Update database
Update Applications
Restart HACMP
Test Failover/Failback

HACMP Update

2005 Mt Xia Technical Consulting Group - All Rights

Moderate Service Outage

Moderate service outages to business functions can be experienced
by implementing an update methodology as follows:
Create alternate clone of rootvg on hot-standby node
Update alternate clone
Reboot hot-standby from alternate clone
Create alternate clone of rootvg on active node
Update alternate clone
Reboot active node from alternate clone
Synchronize HACMP between nodes
Restart HACMP on active node
Restart HACMP on hot-standby node

HACMP Update

2005 Mt Xia Technical Consulting Group - All Rights

Minimal Service Outage

Minimal service outages to business functions can be
experienced by implementing an update methodology as
follows:
Update hot-standby node
Failover to hot-standby node
Update original node
Synchronize HACMP between nodes

HACMP Update

2005 Mt Xia Technical Consulting Group - All Rights

Update Steps
Steps to perform during an update/upgrade process:
Update AIX operating system
Update system microcode
Update adapter microcode
Update HACMP
Update database
Update applications
Synchronize HACMP
Test Failover/Failback

HACMP Update

2005 Mt Xia Technical Consulting Group - All Rights

HACMP Testing Scenarios

Failover/Failback
Automated
Manual (desired)

HACMP Update

2005 Mt Xia Technical Consulting Group - All Rights

HACMP Automated Testing

smitty hacmp
System Management
Automated Testing
Provides automated testing of basic HACMP functions
Some automated tests will always fail because of the cluster
configuration
Results can be difficult to interpret

HACMP Update

2005 Mt Xia Technical Consulting Group - All Rights

HACMP Manual Testing

Manual Failover/Failback testing
Provides targeted testing of specific environment and/or
components
Results are easier to interpret
smitty hacmp
System Management
Start HACMP
clinfo
Stop HACMP

HACMP Update

2005 Mt Xia Technical Consulting Group - All Rights

HACMP Policies
Any changes to the startup environment of a clustered machine
requires failover testing
Enterprise wide unique identifiers shall be used for all shared
resources
Enterprise wide unique user name, group name, UID, and GID number
will be used
Regularly scheduled HACMP verification for each cluster.
Regularly scheduled audit of HACMP patch levels

HACMP Update

2005 Mt Xia Technical Consulting Group - All Rights

HACMP Guidelines
Oracle database's will normally be implemented in a non-concurrent
access cluster
Application servers will normally be implemented in 2-way cascading
clusters
In virtualized environment, HACMP will be implemented in a 1-way
cascading cluster with failover to a standby node built with minimal
resources

HACMP Update

2005 Mt Xia Technical Consulting Group - All Rights

HACMP Standards
Machine Naming
IP addresses, names, aliases
VG, Major Numbers, LV, JFS Log, and Mount Points
Cluster Names
Resource Group Names
Application Server Names
Application start and stop script names
User names and UID numbers
Group names and GID numbers
HACMP filesets

HACMP Update

2005 Mt Xia Technical Consulting Group - All Rights

HACMP Procedures
SAN Storage requests
DNS requests
Network cabling
TSM backups
rootvg backups
Disaster Recovery
Failover Testing
LVM Modifications
Change Control
User Modifications

HACMP Update

2005 Mt Xia Technical Consulting Group - All Rights