Using Serviceguard Extension For RAC

Using Serviceguard Extension for RAC
Version A.11.20
HP Part Number: 5900-1887

Published: August 2011
Legal Notices
Copyright 2011 Hewlett-Packard Development Company, L.P.
Confidential computer software. Valid license from HP required for possession, use, or copying. Consistent with FAR 12.211 and 12.212, Commercial
Computer Software, Computer Software Documentation, and Technical Data for Commercial Items are licensed to the U.S. Government under
vendors standard commercial license.
The information contained herein is subject to change without notice. The only warranties for HP products and services are set forth in the express
warranty statements a-ccompanying such products and services. Nothing herein should be construed as constituting an additional warranty. HP
shall not be liable for technical or editorial errors or omissions contained herein.
Oracle is a registered trademark of Oracle Corporation.
UNIX is a registered trademark in the United States and other countries, licensed exclusively through The Open Group.
VERITAS is a registered trademark of VERITAS Software Corporation.
VERITAS File System is a trademark of VERITAS Software Corporation.
Contents
Advantages of using SGeRAC.........................................................................8
User Guide Overview....................................................................................9
Where to find Documentation on the Web......................................................11
1 Introduction to Serviceguard Extension for RAC............................................12
What is a Serviceguard Extension for RAC Cluster? ...................................................................12
Group Membership............................................................................................................13
Using Packages in a Cluster ...............................................................................................13
Serviceguard Extension for RAC Architecture..............................................................................14
Group Membership Daemon...............................................................................................14
Overview of SGeRAC and Cluster File System (CFS)/Cluster Volume Manager (CVM).....................14
Package Dependencies.......................................................................................................15
Storage Configuration Options............................................................................................15
About Veritas CFS and CVM from Symantec..........................................................................15
Overview of SGeRAC and Oracle 10g, 11gR1, and 11gR2 RAC...................................................16
Overview of SGeRAC Cluster Interconnect Subnet Monitoring ......................................................17
How Cluster Interconnect Subnet Works................................................................................17
Configuring Packages for Oracle RAC Instances.........................................................................18
Configuring Packages for Oracle Listeners..................................................................................18
Node Failure.........................................................................................................................19
Larger Clusters ......................................................................................................................20
Up to Four Nodes with SCSI Storage....................................................................................20
Point-to-Point Connections to Storage Devices ........................................................................21
Extended Distance Cluster Using Serviceguard Extension for RAC.................................................22
GMS Authorization.................................................................................................................22
Overview of Serviceguard Manager.........................................................................................23
Starting Serviceguard Manager...........................................................................................23
Monitoring Clusters with Serviceguard Manager....................................................................23
Administering Clusters with Serviceguard Manager................................................................23
Configuring Clusters with Serviceguard Manager...................................................................24
2 Serviceguard Configuration for Oracle 10g, 11gR1, or 11gR2 RAC.................25

Interface Areas.......................................................................................................................25
Group Membership API (NMAPI2).......................................................................................25
SGeRAC Detection.............................................................................................................25
Cluster Timeouts.................................................................................................................25
Serviceguard Cluster Timeout..........................................................................................25
CSS Timeout.................................................................................................................26
RAC IMR Timeout..........................................................................................................26
Oracle Cluster Software......................................................................................................26
Automated Oracle Cluster Software Startup and Shutdown.................................................26
Monitoring...................................................................................................................26
Allowed Characters for Oracle 10g/11gR1/11gR2 RAC Cluster Names...............................26
Shared Storage.................................................................................................................26
Multipathing ................................................................................................................27
OCR and Vote Device....................................................................................................27
Mirroring and Resilvering...............................................................................................27
Shared Storage Activation..............................................................................................27
Listener.............................................................................................................................27
Automated Startup and Shutdown...................................................................................27
Manual Startup and Shutdown........................................................................................28
Contents
Network Monitoring...........................................................................................................28
SGeRAC Heartbeat Network..........................................................................................28
CSS Heartbeat Network.................................................................................................28
RAC Cluster Interconnect................................................................................................28
Public Client Access.......................................................................................................28
RAC Instances........................................................................................................................28
Automated Startup and Shutdown........................................................................................28
Manual Startup and Shutdown............................................................................................29
Shared Storage.................................................................................................................29
Network Planning for Cluster Communication.............................................................................29
Planning Storage for Oracle Cluster Software.............................................................................30
Planning Storage for Oracle 10g/11gR1/11gR2 RAC..................................................................30
Volume Planning with SLVM................................................................................................31
Storage Planning with CFS..................................................................................................31
Volume Planning with CVM.................................................................................................31
Installing Serviceguard Extension for RAC .................................................................................33
Veritas Cluster Volume Manager (CVM) and Cluster File System (CFS)...........................................33
Veritas Storage Management Products.......................................................................................33
About Multipathing............................................................................................................33
About Device Special Files.......................................................................................................33
About Cluster-wide Device Special Files (cDSFs).....................................................................34
Configuration File Parameters...................................................................................................35
Cluster Communication Network Monitoring..............................................................................35
Single Network for Cluster Communications..........................................................................36
Alternate ConfigurationFast Reconfiguration with Low Node Member Timeout.........................37
Alternate ConfigurationMultiple RAC Databases.................................................................38
Guidelines for Changing Cluster Parameters..........................................................................39
When Cluster Interconnect Subnet Monitoring is used........................................................39
When Cluster Interconnect Subnet Monitoring is not Used..................................................39
Limitations of Cluster Communication Network Monitor...........................................................40
Cluster Interconnect Monitoring Restrictions.......................................................................40
Creating a Storage Infrastructure with LVM.................................................................................40
Building Volume Groups for RAC on Mirrored Disks...............................................................41
Creating Volume Groups and Logical Volumes .................................................................41
Selecting Disks for the Volume Group..........................................................................41
Creating Physical Volumes.........................................................................................41
Creating a Volume Group with PVG-Strict Mirroring......................................................42
Building Mirrored Logical Volumes for RAC with LVM Commands.............................................42
Creating Mirrored Logical Volumes for RAC Redo Logs and Control Files..............................42
Creating Mirrored Logical Volumes for RAC Data Files.......................................................43
Creating RAC Volume Groups on Disk Arrays .......................................................................44
Creating Logical Volumes for RAC on Disk Arrays..................................................................45
Oracle Demo Database Files ..............................................................................................45
Displaying the Logical Volume Infrastructure ..............................................................................46
Exporting the Logical Volume Infrastructure ...........................................................................46
Exporting with LVM Commands ......................................................................................46
Installing Oracle Real Application Clusters.................................................................................47
Creating a Storage Infrastructure with CFS.................................................................................47
Creating an SGeRAC Cluster with CFS for Oracle 11gR1 or 11gR2...........................................48
Initializing the Veritas Volume Manager................................................................................48
Deleting CFS from the Cluster..............................................................................................51
Creating a Storage Infrastructure with CVM................................................................................52
Initializing the Veritas Volume Manager................................................................................52
Using CVM 5.x or later......................................................................................................53
Preparing the Cluster and the System Multi-node Package for use with CVM 5.x or later.........53
4
Contents
Mirror Detachment Policies with CVM..........................................................................55

Using CVM 5.x.................................................................................................................55
Preparing the Cluster for Use with CVM 5.x......................................................................55
Starting the Cluster and Identifying the Master Node....................................................56
Converting Disks from LVM to CVM............................................................................56
Initializing Disks for CVM..........................................................................................56
Creating Disk Groups for RAC...................................................................................56
Creating Volumes...................................................................................................................57
Mirror Detachment Policies with CVM...................................................................................57
Oracle Demo Database Files ...................................................................................................57
Adding Disk Groups to the Cluster Configuration........................................................................58
Prerequisites for Oracle 10g, 11gR1, or 11gR2 (Sample Installation)...............................................59
Installing Oracle 10g, 11gR1, or 11gR2 Cluster Software..............................................................62
Installing on Local File System..............................................................................................62
Installing Oracle 10g/11gR1/11gR2 RAC Binaries......................................................................62
Installing RAC Binaries on a Local File System........................................................................62
Installing RAC Binaries on Cluster File System........................................................................63
Creating a RAC Demo Database..............................................................................................63
Creating a RAC Demo Database on SLVM or CVM................................................................63
Creating a RAC Demo Database on CFS..............................................................................64
Verifying Oracle Disk Manager is Configured............................................................................64
Configuring Oracle to Use Oracle Disk Manager Library.............................................................65
Verify that Oracle Disk Manager is Running...............................................................................65
Configuring Oracle to Stop Using Oracle Disk Manager Library...................................................66
Using Serviceguard Packages to Synchronize with Oracle 10g/11gR1/11gR2 RAC.........................67
Preparing Oracle Cluster Software for Serviceguard Packages.................................................67
Configure Serviceguard Packages........................................................................................67
3 Support of Oracle RAC ASM with SGeRAC.................................................69

Introduction............................................................................................................................69
SG/SGeRAC Support for ASM on HP-UX 11i v2.........................................................................69
Overview..........................................................................................................................69
Why ASM over SLVM?.......................................................................................................69
Configuring SLVM Volume Groups for ASM Disk Groups.........................................................70
SG/SGeRAC Support for ASM on HP-UX 11i v3.........................................................................73
Overview..........................................................................................................................73
ASM over SLVM................................................................................................................73
Configuring SLVM Volume Groups for ASM Disk Groups.........................................................74
Sample Command Sequence for Configuring SLVM Volume Groups....................................75
ASM over Raw disk............................................................................................................76
Configure Raw Disks/Disk Array Logical Units for ASM Disk Group..........................................77
Additional Hints on ASM Integration with SGeRAC.....................................................................77
Consider using the MNP/Simple Dependency-based SGeRAC Toolkits Framework....................77
ASM Halt is needed to ensure disconnect of ASM from SLVM Volume Groups............................77
ASM may require Modified Backup/Restore Procedures..........................................................78
Installation, Configuration, Support, and Troubleshooting........................................................78
Additional Documentation on the Web and Scripts.................................................................79
4 SGeRAC Toolkit for Oracle RAC 10g or later...............................................80

Introduction ...........................................................................................................................80
Background...........................................................................................................................80
Coordinating the Oracle RAC/Serviceguard Extension for RAC stack.......................................80
Serviceguard/Serviceguard Extension for RAC multi-node packages and simple package
dependencies ...................................................................................................................82
Why use multi-node packages/simple package dependencies for Oracle RAC integration...............83
Serviceguard Extension for RAC Toolkit operation ......................................................................84
Contents
Startup and shutdown of the combined Oracle RAC-SGeRAC stack .........................................85

How Serviceguard Extension for RAC starts, stops and checks Oracle Clusterware ....................86
How Serviceguard Extension for RAC Mounts, dismounts and checks ASM disk groups...............86
How Serviceguard Extension for RAC Toolkit starts, stops, and checks the RAC database
instance............................................................................................................................87
How Serviceguard Extension for RAC Toolkit interacts with storage management subsystems........87
Use Case 1: Oracle Clusterware storage and database storage in SLVM/CVM ....................87
Use Case 2: Oracle Clusterware storage and database storage in CFS ...............................88
Use case 3: Database storage in ASM over SLVM.............................................................88
How to create a Serviceguard Extension for RAC modular package..........................................89
How Serviceguard Extension for RAC Toolkit maintenance mode works....................................89
Use Case 1: Performing maintenance with Oracle Clusterware............................................89
Use Case 2: performing maintenance with ASM disk groups..............................................89
Use Case 3: performing maintenance with Oracle RAC database instance...........................90
Serviceguard Extension for RAC Toolkit internal file structure.........................................................90
Support for the SGeRAC Toolkit................................................................................................92
Conclusion..........................................................................................................................112
Additional Documentation on the Web....................................................................................112
5 Maintenance.........................................................................................113
Reviewing Cluster and Package States with the cmviewcl Command............................................113
Types of Cluster and Package States...................................................................................113
Examples of Cluster and Package States.........................................................................113
Types of Cluster and Package States..............................................................................115
Cluster Status .............................................................................................................116
Node Status and State ................................................................................................116
Package Status and State ............................................................................................116
Package Switching Attributes........................................................................................117
Status of Group Membership........................................................................................117
Service Status ............................................................................................................117
Network Status...........................................................................................................118
Failover and Failback Policies.......................................................................................118
Examples of Cluster and Package States .............................................................................118
Normal Running Status................................................................................................118
Quorum Server Status..................................................................................................119
CVM Package Status...................................................................................................119
Status After Moving the Package to Another Node...........................................................120
Status After Package Switching is Enabled......................................................................121
Status After Halting a Node.........................................................................................121
Viewing Data on Unowned Packages.............................................................................121
Checking the Cluster Configuration and Components................................................................122
Checking Cluster Components...........................................................................................123
Setting up Periodic Cluster Verification................................................................................125
Example....................................................................................................................125
Limitations.......................................................................................................................126
Online Reconfiguration..........................................................................................................126
Online Node Addition and Deletion...................................................................................126
Managing the Shared Storage...............................................................................................127
Making LVM Volume Groups Shareable..............................................................................127
Making a Volume Group Unshareable ..........................................................................128
Activating an LVM Volume Group in Shared Mode...............................................................128
Deactivating a Shared Volume Group ...........................................................................128
Making Offline Changes to Shared Volume Groups.............................................................128
Adding Additional Shared LVM Volume Groups ..................................................................130
Changing the CVM Storage Configuration .........................................................................130
6
Contents
Removing Serviceguard Extension for RAC from a System..........................................................130

Monitoring Hardware ...........................................................................................................131
Using Event Monitoring Service..........................................................................................131
Using EMS Hardware Monitors..........................................................................................131
Adding Disk Hardware .........................................................................................................131
Replacing Disks....................................................................................................................132
Replacing a Mechanism in a Disk Array Configured with LVM...............................................132
Replacing a Mechanism in an HA Enclosure Configured with Exclusive LVM............................132
Online Replacement of a Mechanism in an HA Enclosure Configured with Shared LVM (SLVM)...133
Offline Replacement of a Mechanism in an HA Enclosure Configured with Shared LVM (SLVM)...134
Replacing a Lock Disk.......................................................................................................134
Online Hardware Maintenance with Inline SCSI Terminator ..................................................134
Replacement of I/O Cards.....................................................................................................136
Replacement of LAN Cards....................................................................................................136
Offline Replacement.........................................................................................................136
Online Replacement.........................................................................................................136
After Replacing the Card..................................................................................................136
Monitoring RAC Instances......................................................................................................137
6 Troubleshooting......................................................................................138
A Software Upgrades ...............................................................................139
Rolling Software Upgrades....................................................................................................139
Upgrading Serviceguard to SGeRAC cluster........................................................................140
Upgrading from an existing SGeRAC A.11.19 cluster to HP-UX 11i v3 1109
HA-OE/DC-OE...........................................................................................................140
Upgrading from an existing Serviceguard A.11.19 cluster to HP-UX 11i v3 1109 HA-OE/DC-OE
along with SGeRAC....................................................................................................140
Upgrading from an existing Serviceguard A.11.19 cluster to HP-UX 11i v3 1109
HA-OE/DC-OE along with SGeRAC (Alternative approach).........................................141
Upgrading from Serviceguard A.11.18 cluster to HP-UX 11i v3 1109 HA-OE/DC-OE along
with SGeRAC.............................................................................................................141
Upgrading from Serviceguard A.11.20 cluster to HP-UX 11i v3 1109 HA-OE/DC-OE along
with SGeRAC.............................................................................................................142
Steps for Rolling Upgrades ...............................................................................................142
Keeping Kernels Consistent...........................................................................................143
Example of Rolling Upgrade .............................................................................................143
Step 1. ......................................................................................................................144
Step 2. .....................................................................................................................144
Step 3. .....................................................................................................................145
Step 4. .....................................................................................................................146
Step 5. .....................................................................................................................146
Limitations of Rolling Upgrades .........................................................................................147
Non-Rolling Software Upgrades.............................................................................................148
Limitations of Non-Rolling Upgrades ..................................................................................148
Migrating an SGeRAC Cluster with Cold Install....................................................................148
Upgrade Using DRD.............................................................................................................149
Rolling Upgrade Using DRD..............................................................................................149
Non-Rolling Upgrade Using DRD.......................................................................................149
Restrictions for DRD Upgrades...........................................................................................149
B Blank Planning Worksheets......................................................................151

LVM Volume Group and Physical Volume Worksheet.................................................................151
Oracle Logical Volume Worksheet..........................................................................................151
Index.......................................................................................................153
Contents
Advantages of using SGeRAC

HP Serviceguard Extension for RAC (SGeRAC) amplifies the availability and simplifies the
management of Oracle Real Application Cluster (RAC). SGeRAC allows you to integrate Oracle
RAC into a Serviceguard cluster while also easily managing the dependency between Oracle
Clusterware and Oracle RAC with a full range of storage management options.
Built on the functionality of HP Serviceguard, SGeRAC uses a networked group of HP Integrity
servers to create highly available clusters that support Oracle RAC. Integrated tightly, these two
products provide the best features of HP enterprise clusters and Oracle relational database servers.
You gain high availability, data integrity, flexibility, and scalability in your Oracle RAC
environmentall of which are essential to protect your infrastructure and enhance business uptime.
Business Outcomes
Serviceguard Extension for RAC ensures that you do not have a single point of failure within the
entire Oracle RAC cluster. Not only that, the unique architecture of SGeRAC leverages the full
aggregate processing power of up to 16 nodes to access the database. This can increase the
overall throughput for query-intensive applications, that generate random reads and writes to very
large databases, and applications that access separate partitions of the database. SGeRAC even
provides extra protection for database integrity.
SGeRAC works together with Oracle Clusterware to provide an improved high-availability solution
that can address your concerns over unplanned downtime. The availability of your applications is
improved with rapid automatic detection and recovery from a wide range of hardware, software,
network, and storage errors. SGeRAC also provides a superior level of protection by enabling you
to withstand multiple node failures. Serviceguard solutions are integrated with HP Matrix Operating
Environment for HP-UX so that you can maintain service level objectives during planned and
unplanned downtime.
The tight integration between SGeRAC and Oracle RAC also enables faster detection and failover,
and you can even achieve nearly continuous application availability by implementing a fully-disaster
tolerant solution.
SGeRAC uses advanced clustering mechanisms and robust volume managers to eliminate data
corruption and preserve data integrity. Youll also benefit from data integrity protection among
nodes, both inside and outside of the cluster. Flexibility is paramount, so SGeRAC supports four
storage management options for Oracle RAC: Cluster file system (CFS), Shared Logical Volume
Manager (SLVM), Cluster Volume Manager (CVM) and Automated Storage Management (ASM)
on SLVM and raw volumes.
User Guide Overview

This user guide covers how to use Serviceguard Extension for RAC (Oracle Real Application Cluster)
to configure Serviceguard clusters for use with Oracle Real Application Cluster software, on HP
High Availability clusters running the HP-UX operating system.
Chapter 1 Introduction to Serviceguard Extension for RAC

Describes a Serviceguard cluster and provides a roadmap for using this guide. This chapter
should be used as a supplement to Chapters 13 of the Managing Serviceguard users guide.
Chapter 2 Serviceguard Configuration for Oracle 10g, 11gR1, or 11gR2 RAC

Describes the additional steps you need to take to use Serviceguard with Real Application
Clusters when configuring Oracle 10g/11gR1/11gR2 RAC. This chapter should be used as
a supplement to Chapters 46 of the Managing Serviceguard users guide.
Chapter 3 Oracle Automatic Storage Management (ASM)

Discusses the use of the Oracle 10g database server feature called Automatic Storage
Management (ASM) in configurations of HP Serviceguard for single database instance failover.
Chapter 4 SGeRAC Toolkit for Oracle RAC 10g or later

Discusses how Serviceguard Extension for RAC Toolkit enables a new framework for the
integration of Oracle 10g Release 2 (10.2.0.1) or later version of Real Application Clusters
(Oracle RAC1) with HP Serviceguard Extension for Real Application Clusters A.11.17 or later
(SGeRAC2).
Chapter 5 Maintenance
Describes tools and techniques necessary for ongoing cluster operation. This chapter should
be used as a supplement to Chapters 78 of the Managing Serviceguard users guide.
Chapter 6 Troubleshooting
Lists where to find troubleshooting information.
Appendix A Software Upgrades

Describes rolling, non-rolling, and migration with cold install upgrade procedures for SGeRAC
clusters.
Appendix B Blank Planning Worksheets

Contains planning worksheets for LVM, and Oracle Logical Volume.
To view the following documents, go to www.hp.com/go/hpux-core-docs and www.hp.com/

go/hpux-serviceguard-docs.
VERITAS Volume Manager Administrators Guide (includes a glossary of VERITAS terminology).
VERITAS Volume Manager Storage Administrator Administrators Guide.
VERITAS Volume Manager Reference Guide.
VERITAS Volume Manager Migration Guide.
VERITAS Volume Manager for HP-UX Release Notes.
VERITAS Storage Foundation for Oracle RAC. HP Serviceguard Storage Management Suite
Configuration Guide Extracts.
VERITAS Storage Foundation for Oracle RAC. HP Serviceguard Storage Management Suite
Administration Guide Extracts.
If you will be using Veritas Cluster Volume Manager (CVM) and Veritas Cluster File System (CFS)
from Symantec with Serviceguard refer to the HP Serviceguard Storage Management Suite Version
A.03.01 for HP-UX 11i v3 Release Notes.
These release notes describe suite bundles for the integration of HP Serviceguard A.11.20 on
HP-UX 11i v3 with Symantecs Veritas Storage Foundation.
Problem Reporting
If you have any problems with the software or documentation, please contact your local
Hewlett-Packard Sales Office or Customer Service Center.
Typographical Conventions
10
audit(5)
An HP-UX manpage. audit is the name and 5 is the section in the HP-UX
Reference. On the web and on the Instant Information CD, it may be a hot link
to the manpage itself. From the HP-UX command line, you can enter man
audit or man 5 audit to view the manpage. See man(1).
ComputerOut
Text displayed by the computer.
UserInput
Commands and other text that you type.
Command
A command name or qualified command phrase.
Variable
The name of a variable that you may replace in a command or function or

information in a display that represents several possible values.
[ ]
The contents are optional in formats and command descriptions. If the contents
are a list separated by |, you must choose one of the items.
{ }
The contents are required in formats and command descriptions. If the contents
are a list separated by |, you must choose one of the items.
...
The preceding element may be repeated an arbitrary number of times.
Separates items in a list of choices.
Where to find Documentation on the Web
SGeRAC Documentation
Go to www.hp.com/go/hpux-serviceguard-docs, and then click HP Serviceguard
Extension for RAC.
Related Documentation
Go to www.hp.com/go/hpux-serviceguard-docs, www.hp.com/go/
hpux-core-docs, and www.hp.com/go/hpux-ha-monitoring-docs.
The following documents contain additional useful information:
Clusters for High Availability: a Primer of HP Solutions. Hewlett-Packard Professional

Books: Prentice Hall PTR, 2001 (ISBN 0-13-089355-2)
Serviceguard Extension for RAC Version A.11.20 Release Notes
HP Serviceguard Version A.11.20 Release Notes
Managing Serviceguard Eighteenth Edition
HP Serviceguard Storage Management Suite Version A.02.01 for HP-UX 11i v3 Release
Notes
Using High Availability Monitors
Using the Event Monitoring Service
Using Advanced Tape Services
Managing Systems and Workgroups
Managing Serviceguard NFS
HP-UX System Administrator's Guide: Logical Volume Management HP-UX 11i Version
3
11
1 Introduction to Serviceguard Extension for RAC

Serviceguard Extension for RAC (SGeRAC) enables the Oracle Real Application Cluster (RAC),
formerly known as Oracle Parallel Server RDBMS, to run on HP high availability clusters under the
HP-UX operating system. This chapter introduces Serviceguard Extension for RAC and shows where
to find different kinds of information in this book. The following topics are presented:
What is a Serviceguard Extension for RAC Cluster? (page 12)
Serviceguard Extension for RAC Architecture (page 14)
Overview of SGeRAC and Cluster File System (CFS)/Cluster Volume Manager (CVM) (page
14)
Overview of SGeRAC and Oracle 10g, 11gR1, and 11gR2 RAC (page 16)
Overview of SGeRAC Cluster Interconnect Subnet Monitoring (page 17)
Node Failure (page 19)
Larger Clusters (page 20)
Extended Distance Cluster Using Serviceguard Extension for RAC (page 22)
What is a Serviceguard Extension for RAC Cluster?

A high availability cluster is a grouping of HP servers having sufficient redundancy of software
and hardware components that a single point of failure will not disrupt the availability of computer
services. High availability clusters configured with Oracle Real Application Cluster software are
known as RAC clusters. Figure 1 shows the basic configuration of a RAC cluster on HP-UX.
Figure 1 Overview of Oracle RAC Configuration on HP-UX
In the figure, two loosely coupled systems (each one known as a node) are running separate
instances of Oracle software that read data from and write data to a shared set of disks. Clients
connect to one node or the other via LAN.
With RAC on HP-UX, you can maintain a single database image that is accessed by the HP servers
in parallel and gain added processing power without the need to administer separate databases.
12
Introduction to Serviceguard Extension for RAC
When properly configured, Serviceguard Extension for RAC provides a highly available database
that continues to operate even if one hardware component fails.
Group Membership
Group membership allows multiple instances of RAC to run on each node. Related processes are
configured into groups. Groups allow processes in different instances to choose which other
processes to interact with. This allows the support of multiple databases within one RAC cluster.
A Group Membership Service (GMS) component provides a process monitoring facility to monitor
group membership status. GMS is provided by the cmgmsd daemon, which is an HP component
installed with Serviceguard Extension for RAC.
Figure 2 shows how group membership works. Nodes 1 through 4 of the cluster share the Sales
database, but only Nodes 3 and 4 share the HR database. There is one instance of RAC each on
Node 1 and Node 2, and two instances of RAC each on Node 3 and Node 4. The RAC processes
accessing the Sales database constitute one group, and the RAC processes accessing the HR
database constitute another group.
Figure 2 Group Membership Services
Using Packages in a Cluster

To make other important applications highly available (in addition to the Oracle Real Application
Cluster), you can configure your RAC cluster to use packages. Packages group applications and
services together. In the event of a service, node, or network failure, Serviceguard Extension for
RAC can automatically transfer control of all system resources in a designated package to another
node within the cluster, allowing your applications to remain available with minimal interruption.
There are failover packages, system multi-node packages, and multi-node packages:
Failover package. The typical high-availability package is a failover package. It usually is

configured to run on several nodes in the cluster, and runs on one at a time. If a service, node,
network, or other package resource fails on the node where it is running, Serviceguard can
automatically transfer control of the package to another cluster node, allowing services to
remain available with minimal interruption.
System multi-node packages. There are also packages that run on several cluster nodes at
once, and do not fail over. These are called system multi-node packages and multi-node
packages. As of Serviceguard Extension for RAC A.11.18, the non-failover packages that
What is a Serviceguard Extension for RAC Cluster?
13
are supported are those specified by Hewlett-Packard, and you can create your own multi-node
packages. For example, the packages HP supplies for use with the Veritas Cluster Volume
Manager (CVM) and the Veritas Cluster File (CFS) System (on HP-UX releases that support
Veritas CFS and CVM. Also, see About Veritas CFS and CVM from Symantec (page 15)).
Multi-node package. A system multi-node package must run on all nodes that are active in
the cluster. If it fails on one active node, that node halts. A multi-node package can be
configured to run on one or more cluster nodes. It is considered UP as long as it is running
on any of its configured nodes.
NOTE: In RAC clusters, you create packages to start and stop Oracle Clusterware and RAC itself
as well as to run applications that access the database instances. For details on the use of packages
with Oracle Clusterware and RAC, refer to RAC Instances (page 28)
Serviceguard Extension for RAC Architecture

This chapter discusses the main software components used by Serviceguard Extension for RAC.
The following is a list of components:
Oracle Components
Serviceguard Extension for RAC Components
Custom Oracle Database

Group Membership Services (RAC)
Serviceguard Components
Package Manager
Cluster Manager
Network Manager
Operating System
Volume Manager Software
HP-UX Kernel
Group Membership Daemon

In addition to the Serviceguard daemon processes mentioned in Chapter 3 of the Managing
Serviceguard users guide, there is another daemon that is used by Oracle to enable communication
with Serviceguard Extension for RAC:
cmgmsdGroup Membership Daemon for RAC 10g or later
This HP daemon provides group membership services for Oracle Real Application Cluster 10 g or
later. Group membership allows multiple Oracle instances to run on the same cluster node. GMS
is illustrated in Figure 2 (page 13).
Overview of SGeRAC and Cluster File System (CFS)/Cluster Volume

Manager (CVM)
SGeRAC supports Veritas Cluster File System (CFS)/Cluster Volume Manager (CVM) from Symantec
through Serviceguard. CFS and CVM are not supported on all versions of HP-UX (on HP-UX releases
that support Veritas CFS and CVM). For more information, see About Veritas CFS and CVM from
Symantec (page 15)).
For information on configuring CFS and CVM with Serviceguard, refer to the latest edition of the
Managing Serviceguard users guide at www.hp.com/go/hpux-serviceguard-docs >
HP Serviceguard.
14
Package Dependencies
When CFS is used as shared storage, the application and software using the CFS storage should
be configured to start and stop using Serviceguard packages. These application packages should
be configured with a package dependency on the underlying multi-node packages, which manages
the CFS and CVM storage reserves.
Configuring the application to be start/stop through Serviceguard package is to ensure the
synchronization of storage activation/deactivation and application startup/shutdown.
With CVM configurations using multi-node packages, CVM shared storage should be configured
in Serviceguard packages with package dependencies.
Refer to the latest edition of the Managing Serviceguard users guide for detailed information on
multi-node packages.
Storage Configuration Options

CFS provides SGeRAC with additional options, such as improved manageability. When planning
a RAC cluster, application software could be installed once and be visible by all cluster nodes. A
central location is available to store runtime logs, for example, RAC alert logs.
Oracle RAC data files can be created on a CFS, allowing the database administrator or Oracle
software to create additional data files without the need of root system administrator privileges.
The archive area can be on a CFS. Oracle instances on any cluster node can access the archive
area when database recovery requires the archive logs.
About Veritas CFS and CVM from Symantec

Veritas Cluster File System (CFS) and Cluster Volume Manager (CVM) are supported on some, but
not all current releases of HP-UX. Check the latest Release Notes for your version of Serviceguard
for up-to-date information at www.hp.com/go/hpux-serviceguard-docs > HP
Serviceguard Extension for RAC.
CAUTION: Once you create the disk group and mount point packages, you must administer the
cluster with CFS commands, including cfsdgadm, cfsmntadm, cfsmount, and cfsumount.
You must not use the HP-UX mount or umount command to provide or remove access to a shared
file system in a CFS environment. Using these HP-UX commands under these circumstances is not
supported. Use cfsmount and cfsumount instead.
If you use the HP-UX mount and umount commands, serious problems could occur, such as writing
to the local file system instead of the cluster file system. Non-CFS commands could cause conflicts
with subsequent CFS command operations on the file system or the Serviceguard packages, and
will not create an appropriate multi-node package, which means cluster packages will not be
aware of file system changes.
IMPORTANT: To avoid cluster wide panics and/or database failures, Storage Foundation for
Oracle RAC (SFRAC/SFCFS for RAC) installations using Cluster Volume Manager (CVM) shared
disk groups must have a dgfailpolicy of "leave". For additional details, read the latest version of
this technote at
www.symantec.com/business/support/index?page=content&id=TECH144672
Overview of SGeRAC and Cluster File System (CFS)/Cluster Volume Manager (CVM)
15
NOTE: Beginning with HP-UX 11i v3 1109 HA-OE/DC-OE, SGeRAC is included as a licensed
bundle at no additional cost. To install SGeRAC A.11.20 on your system during 1109
HA-OE/DC-OE installation, you must select T1907BA (SGeRAC) in the Software tab.
If you are using any one of the bundles namely, T2771DB, T2773DB, T2774DB, T2775DB,
T8684DB, T8694DB, T8685DB, T8695DB, T2774EB, T8684EB, or T8694EB) and if you select
SGeRAC during upgrade from HP Serviceguard Storage Management Suite (SMS) cluster, then
this configuration is supported only if Oracle RAC is deployed over SLVM. This happens because
these Serviceguard SMS bundles either contain only VxVM which cannot be used for Oracle RAC
or they are not Oracle specific bundles.
If SGeRAC is not installed as a part of HP-UX 11i v3 1109 HA-OE/DC-OE, it is automatically
installed during the installation of Serviceguard SMS bundles (T2777DB, T8687DB, T8697DB,
T2777EB, T8687EB, T8697EB).
If SGeRAC is installed with HP-UX 11i v3 1109 HA-OE/DC-OE, then depending upon the available
version of Serviceguard and SGeRAC in the SMS bundle, the installation of Serviceguard SMS
may succeed or fail.
The table below discusses the different installation scenarios for Serviceguard SMS bundles:
SMS bundle
Impact on Serviceguard SMS installation
Install SMS bundle 7 which has the same version of Serviceguard Continues the installation of Serviceguard SMS by
and SGeRAC version available on the 1109 HA-OE/DC-OE.
skipping the installation of Serviceguard and
SGeRAC.
Install Serviceguard SMS bundle 7 which has a higher version
of Serviceguard and SGeRAC version available on the 1109
HA-OE/DC-OE.
Installation of Serviceguard SMS follows installation

of a higher version of Serviceguard and SGeRAC.
Install SMS bundle 7 which has a lower version of Serviceguard Installation of Serviceguard SMS fails indicating that
and SGeRAC version available on the 1109 HA-OE/DC-OE.
higher Serviceguard version is already available on
the system.
Overview of SGeRAC and Oracle 10g, 11gR1, and 11gR2 RAC

Starting with Oracle 10g RAC, Oracle has bundled its own cluster software. The initial release is
called Oracle Cluster Ready Service (CRS). CRS is used both as a generic term referring to the
Oracle Cluster Software and as a specific term referring to a component within the Oracle Cluster
Software. At subsequent release, Oracle generic CRS is renamed to Oracle Clusterware. Oracle
Clusterware is the generic term referring to the Oracle Cluster Software.
The Oracle Cluster Software includes the following components: Cluster Synchronization Services
(CSS), Cluster Ready Service (CRS), and Event Management (EVM).
CSS manages the Oracle cluster membership and provides its own group membership service to
RAC instances. When installed on an SGeRAC cluster, CSS utilizes the group membership service
provided by SGeRAC.
CRS manages Oracle's cluster resources based on configurationincluding start, stop, and monitor,
and failover of the resources.
EVM publishes events generated by CRS and may run scripts when certain events occur.
When installed on an SGeRAC cluster, both the Oracle Cluster Software and RAC can continue
to rely on the shared storage capability, networking monitoring, as well as other capabilities
provided through Serviceguard and SGeRAC.
16
Oracle 10g/11gR1/11gR2 RAC uses the following two subnets for cluster communication purposes:
CSS Heartbeat Network (CSS-HB)Oracle Clusterware running on the various nodes of the
cluster communicate among themselves using this network.
RAC Cluster Interconnect Network (RAC-IC)Database instances of a database use this

network to communicate among themselves.
NOTE: In this document, the generic terms CRS and Oracle Clusterware will subsequently be
referred to as Oracle Cluster Software. The use of the term CRS will still be used when referring
to a sub-component of Oracle Cluster Software.
For more detailed information on Oracle 10g/11gR1/11gR2 RAC, refer to Chapter 2:
Serviceguard Configuration for Oracle 10g, 11gR1, or 11gR2 RAC.
Overview of SGeRAC Cluster Interconnect Subnet Monitoring

In SGeRAC, the Cluster Interconnect Subnet Monitoring feature is used to monitor cluster
communication subnets. This feature requires the use of a package configuration parameter known
as the CLUSTER_INTERCONNECT_SUBNET. It can be set up to monitor certain subnets used by
applications that are configured as Serviceguard multi-node packages.
The following describes the characteristics of subnets that can be monitored by using the
CLUSTER_INTERCONNECT_SUBNET parameter:
A subnet used only for the communications among instances of an application configured as
a multi-node package.
A subnet whose health does not matter if there is only one instance of an application (package)
running in the cluster. The instance is able to provide services to its clients regardless of whether
the subnet is up or down on the node where the only instance of the package is running.
A failure of the cluster interconnect subnet on all nodes of the cluster, where the multi-node package
is running, is handled by bringing down all but one instance of the multi-node package.
In certain 10g/11gR1/11gR2 RAC configurations, this parameter can be used to monitor Oracle
Cluster Synchronization Services (CSS-HB) and/or RAC cluster interconnect subnet when Oracle
Clusterware and RAC Database instances are configured as Serviceguard multi-node packages.
Cluster Interconnect Subnet Monitoring provides the following benefits:
Better availability by detecting and resolving RAC-IC subnet failures quickly in certain
configurations. (For example, in configurations where Oracle CSS-HB subnet and the RAC-IC
subnet are not the same. This configuration helps avoid one RAC database (RAC-IC) traffic
from interfering with another.)
Assists in providing services (on one node) when Oracle CSS-HB/RAC-IC subnet fails on all
nodes.
Facilitates fast and reliable reconfiguration with use of multiple SG-HB networks.
Allows separation of SG-HB and Oracle RAC-IC traffic (recommended when RAC-IC traffic
may interfere with SG-HB traffic).
How Cluster Interconnect Subnet Works

The CLUSTER_INTERCONNECT_SUBNET parameter works similar to the existing SUBNET package
configuration parameter. The most notable difference is in the failure handling of the subnets
monitored using these individual parameters.
While the failure of subnets monitored using SUBNET package configuration parameter is handled
by halting the instance of the package on the node where the subnet has failed, the failure of
subnets monitored using CLUSTER_INTERCONNECT_SUBNET package configuration parameter
is handled by ensuring at least one instance of the multi-node package remains running in the
cluster.
Overview of SGeRAC Cluster Interconnect Subnet Monitoring
17
For example, when a multi-node package (pkgA) is configured to run on all nodes of the cluster,
and configured to monitor a subnet (SubnetA) using the CLUSTER_INTERCONNECT_SUBNET
parameter:
If more than one instance of pkgA is running in the cluster and SubnetA fails on one of the
nodes where the instance of pkgA is running, the failure is handled by halting the instance of
pkgA on the node where the subnet has failed.
If pkgA is running on only one node of the cluster and SubnetA fails on that node, pkgA will
continue to run on that node after the failure.
If pkgA runs on all nodes of the cluster and SubnetA fails on all nodes of the cluster, the
failure is handled by halting all but one instance of pkgA. Where the instance of pkgA will
be left running is randomly determined.
The following describes the behavior of cluster interconnect subnet monitoring feature under the
following scenarios:
For a multi-node package with CLUSTER_INTERCONNECT_SUBNET configured, upon an

explicit request to start the package on a node, no attempt to start the package instance on
that node will be made if the subnet is not up on that node.
If a multi-node package with CLUSTER_INTERCONNECT_SUBNET configured is running on

only one node of the cluster, with the subnet that it monitors is down on all nodes of the cluster,
the restoration of the subnet on any other node does not affect the running instance. No attempt
will be made automatically to start the package instance on the restored node. An attempt to
start the instance of the package on the restored node will be made, if the user explicitly tries
to start the package instance.
If a multi-node package with CLUSTER_INTERCONNECT_SUBNET is running on more than

one node of the cluster, upon a failure of the subnet on all nodes where the package is running,
all but one package instance will be halted one by one. Additionally, if
NODE_FAIL_FAST_ENABLED is set to YES for such a package, all but one node will be
brought down one by one. For example, one node will be brought down, cluster reformation
will happen, another node will be brought down, and cluster reformation will happen until
one node remains in the cluster.
NOTE: The CLUSTER_INTERCONNECT_SUBNET parameter is available only for use with
multi-node packages.
For more information on the Cluster Interconnect Subnet Monitoring feature, refer to chapter 2,
section Cluster Communication Network Monitoring (page 35). This section describes various
network configurations for cluster communications in SGeRAC/10g or 11gR1/11gR2 RAC cluster,
and how the package configuration parameter CLUSTER_INTERCONNECT_SUBNET can be used
to recover from Oracle Cluster Communications network failures.
Configuring Packages for Oracle RAC Instances

Oracle instances can be configured as packages with a single node in their node list.
NOTE: Packages that start and halt Oracle instances (called instance packages) do not fail over
from one node to another, they are single-node packages. You should include only one NODE_NAME
in the package ASCII configuration file. The AUTO_RUN setting in the package configuration file
will determine whether the RAC instance will start up as the node joins the cluster. Your cluster
may include RAC and non-RAC packages in the same cluster.
Configuring Packages for Oracle Listeners

Oracle listeners can be configured as packages within the cluster (called listener packages). Each
node with a RAC instance can be configured with a listener package. Listener packages are
18
configured to automatically fail over from the original node to an adoptive node. When the original
node is restored, the listener package automatically fails back to the original node.
In the listener package ASCII configuration file, the FAILBACK_POLICY is set to AUTOMATIC.
The SUBNET is a set of monitored subnets. The package can be set to automatically startup with
the AUTO_RUN setting.
Each RAC instance can be configured to be registered with listeners that are assigned to handle
client connections. The listener package script is configured to add the package IP address and
start the listener on the node.
For example, on a two-node cluster with one database, each node can have one RAC instance
and one listener package. Oracle clients can be configured to connect to either package IP address
(or corresponding hostname) using Oracle Net Services. When a node failure occurs, existing
client connection to the package IP address will be reset after the listener package fails over and
adds the package IP address. For subsequent connections for clients configured with basic failover,
clients would connect to the next available listener package's IP address and listener.
Node Failure
RAC cluster configuration is designed so that in the event of a node failure, another node with a
separate instance of Oracle can continue processing transactions. Figure 3 shows a typical cluster
with instances running on both nodes.
Figure 3 Before Node Failure
Figure 4 shows the condition where node 1 has failed and Package 1 has been transferred to
node 2. Oracle instance 1 is no longer operating, but it does not fail over to node 2. The IP address
for package 1 was transferred to node 2 along with the package. Package 1 continues to be
available and is now running on node 2. Also, node 2 can now access both the Package 1 disk
and Package 2 disk. Oracle instance 2 now handles all database access, since instance 1 has
gone down.
Node Failure
19
Figure 4 After Node Failure
In the above figure, pkg1 and pkg2 are not instance packages. They are shown to illustrate the
movement of the packages.
Larger Clusters
Serviceguard Extension for RAC supports clusters of up to 16 nodes. The actual cluster size is
limited by the type of storage and the type of volume manager used.
Up to Four Nodes with SCSI Storage

You can configure up to four nodes using a shared F/W SCSI bus; for more than four nodes,
FibreChannel must be used. An example of a four-node RAC cluster appears in the following figure.
20
Figure 5 Four-Node RAC Cluster
In this type of configuration, each node runs a separate instance of RAC and may run one or more
high availability packages as well.
The figure shows a dual Ethernet configuration with all four nodes connected to a disk array (the
details of the connections depend on the type of disk array). In addition, each node has a mirrored
root disk (R and R). Nodes may have multiple connections to the same array using alternate links
(PV links) to take advantage of the array's use of RAID levels for data protection.
Point-to-Point Connections to Storage Devices

Some storage devices allow point-to-point connection to a large number of host nodes without
using a shared SCSI bus. An example is shown in Figure 6, a cluster consisting of eight nodes
with a FibreChannel interconnect. (Client connection is provided through Ethernet.) The nodes
access shared data on an HP StorageWorks EVA or XP series or EMC disk array configured with
16 I/O ports. Each node is connected to the array using two separate Fibre channels configured
with PV Links. Each channel is a dedicated busthere is no daisy-chaining.
Larger Clusters
21
Figure 6 Eight-Node Cluster with EVA, XP or EMC Disk Array
FibreChannel switched configurations also are supported using either an arbitrated loop or fabric
login topology. For additional information about supported cluster configurations, refer to the HP
9000 Servers Configuration Guide, available through your HP representative.
Extended Distance Cluster Using Serviceguard Extension for RAC

Basic Serviceguard clusters are usually configured in a single data center, often in a single room,
to provide protection against failures in CPUs, interface cards, and software. Extended Serviceguard
clusters are specialized cluster configurations, which allow a single cluster to extend across two
or three separate data centers for increased disaster tolerance. Depending on the type of links
employed, distances of up to 100 km between data centers can be achieved.
Refer to Chapter 2 of the Understanding and Designing Serviceguard Disaster Tolerant Architectures
users guide, which discusses several types of extended distance cluster configurations that use
basic Serviceguard technology with software mirroring (using MirrorDisk/UX or CVM) and Fibre
Channel.
GMS Authorization
SGeRAC includes the Group Membership Service (GMS) authorization feature, which allows only
the listed users to access the GMS. By default, this feature is disabled. To enable this feature,
uncomment the variable GMS_USER[0] and add as many as users as you need.
Use the following steps to enable the GMS authorization (If Oracle RAC is already installed):
1. If Oracle RAC database instance and Oracle Clusterware are running, shut them down on
all nodes.
2. Halt the Serviceguard cluster.
3. Edit /etc/opt/nmapi/nmutils.conf to add all Oracle users on all nodes.
GMS_USER[0]=<oracle1>
GMS_USER[1]=<oracle2>
...
GMS_USER[n-1]=<oraclen>
22
4.
5.
Restart the Serviceguard cluster.

Restart Oracle Clusterware (for Oracle 10g, 11gR1, and 11gR2) and Oracle RAC database
instance on all nodes.
Use the following steps to disable the GMS authorization:
1.
2.
3.
4.
5.
If Oracle RAC database instance and Oracle Clusterware (for Oracle 10g, 11gR1, and
11gR2) are running, shut them down on all nodes.
Halt the Serviceguard cluster.
Edit /etc/opt/nmapi/nmutils.conf and comment the GMS_USER[] settings on all
nodes.
Restart the Serviceguard cluster.
Restart Oracle Clusterware (for Oracle 10g, 11gR1, and 11gR2) and Oracle RAC database
instance on all nodes.
Overview of Serviceguard Manager

NOTE: For more-detailed information, see the section on Serviceguard Manager in the latest
version of the Serviceguard Release Notes. Check the Serviceguard/SGeRAC/SMS/Serviceguard
Manager Plug-in Compatibility and Feature Matrix and the latest Release Notes for up-to-date
information about Serviceguard Manager compatibility. You can find both documents at
www.hp.com/go/ hpux-serviceguard-docs > HP Serviceguard.
Serviceguard Manager is the graphical user interface for Serviceguard. It is available as a plug-in
to the System Management Homepage (SMH). SMH is a web-based graphical user interface (GUI)
that replaces SAM as the system administration GUI as of HP-UX 11i v3 (you can still run the SAM
terminal interface).
You can use Serviceguard Manager to monitor, administer, and configure Serviceguard clusters.
You can see properties, status, and alerts of clusters, nodes, and packages.
You can do administrative tasks such as run or halt clusters, cluster nodes, and packages.
You can create or modify a cluster and its packages.
Starting Serviceguard Manager

To start the Serviceguard Manager plug-in in your web browser from the System Management
Homepage, click on the link to Serviceguard Cluster or a particular cluster. Then select a
cluster, node, or package, and use the drop-down menus below Serviceguard Manager to navigate
to a task.
Use Serviceguard Managers built-in help to guide you through the tasks; this manual will tell you
if a task can be done in Serviceguard Manager but does not duplicate the help.
Monitoring Clusters with Serviceguard Manager

From the main page of Serviceguard Manager, you can see status and alerts for the cluster, nodes,
and packages. You can also drill down to see the configuration and alerts of the cluster, nodes,
and packages.
Administering Clusters with Serviceguard Manager

Serviceguard Manager allows you administer clusters, nodes, and packages if access control
policies permit:
Cluster: halt, run
Cluster nodes: halt, run
Package: halt, run, move from one node to another, reset node- and package-switching flags
Overview of Serviceguard Manager
23
Configuring Clusters with Serviceguard Manager

You can configure clusters and packages in Serviceguard Manager. You must have root (UID=0)
access to the cluster nodes.
24
2 Serviceguard Configuration for Oracle 10g, 11gR1, or

11gR2 RAC
This chapter shows the additional planning and configuration that is needed to use Oracle Real
Application Clusters 10g/11gR1/11gR2 with Serviceguard. The following topics are presented:
Interface Areas (page 25)
Oracle Cluster Software (page 26)
Planning Storage for Oracle Cluster Software (page 30)
Planning Storage for Oracle 10g/11gR1/11gR2 RAC (page 30)
Installing Serviceguard Extension for RAC (page 33)
Installing Oracle Real Application Clusters (page 47)
Cluster Communication Network Monitoring (page 35)
Creating a Storage Infrastructure with LVM (page 40)
Creating a Storage Infrastructure with CFS (page 47)
Creating a Storage Infrastructure with CVM (page 52)
Installing Oracle 10g, 11gR1, or 11gR2 Cluster Software (page 62)
Interface Areas
This section documents interface areas where there is expected interaction between SGeRAC,
Oracle 10g/11gR1/11gR2 Cluster Software, and RAC.
Group Membership API (NMAPI2)

The NMAPI2 client links with the SGeRAC provided NMAPI2 library for group membership service.
The group membership is layered on top of the SGeRAC cluster membership where all the primary
group members are processes within cluster nodes. Cluster membership has node names and group
membership has process names. Upon an SGeRAC group membership change, SGeRAC delivers
the new group membership to other members of the same group.
SGeRAC Detection
When Oracle 10g/11gR1/11gR2 Cluster Software is installed on an SGeRAC cluster, Oracle
Cluster Software detects the existence of SGeRAC, and CSS uses SGeRAC group membership.
Cluster Timeouts
SGeRAC periodically check heartbeat member timeouts to determine when any SGeRAC cluster
member has failed or when any cluster member is unable to communicate with the other cluster
members. CSS uses a similar mechanism for CSS memberships. Each RAC instance group
membership also has a member timeout mechanism, which triggers Instance Membership Recovery
(IMR).
NOTE:
HP and Oracle support SGeRAC to provide group membership to CSS.
Serviceguard Cluster Timeout

The Serviceguard cluster heartbeat timeout is set according to user requirements for availability.
The Serviceguard cluster reconfiguration time is determined by the cluster timeout, configuration,
the reconfiguration algorithm, and activities during reconfiguration.
Interface Areas
25
CSS Timeout
When SGeRAC is on the same cluster as Oracle Cluster Software, the CSS timeout is set to a
default value of 600 seconds (10 minutes) at Oracle software installation.
This timeout is configurable with Oracle tools and should not be changed without ensuring that
the CSS timeout allows enough time for Serviceguard Extension for RAC (SGeRAC) reconfiguration
and to allow multipath (if configured) reconfiguration to complete.
On a single point of failure, for example a node failure, Serviceguard reconfigures first and SGeRAC
delivers the new group membership to CSS via NMAPI2. If there is a change in group membership,
SGeRAC updates the members of the new membership. After receiving the new group membership,
CSS initiates its own recovery action as needed, and propagates the new group membership to
the RAC instances.
RAC IMR Timeout

RAC instance IMR timeout is configurable. RAC IMR expects group membership changes to occur
within this time or IMR will begin evicting group members. The IMR timeout must be above the
SGeRAC reconfiguration time and adhere to any Oracle-specified relation to CSS reconfiguration
time.
NOTE:
later.
IMR timeout as a configurable parameter has been deprecated in Oracle 11gR1 and
Oracle Cluster Software

Oracle Cluster Software should be started after activating its required shared storage resources.
Shared storage resources can be activated after SGeRAC completes startup. Oracle Cluster Software
should not activate any shared storage. Similarly, for halting SGeRAC at run level 3 and removing
shared storage resources from Oracle Cluster Software, Oracle Cluster Software must be halted
first.
Automated Oracle Cluster Software Startup and Shutdown

The preferred mechanism that allows Serviceguard to notify Oracle Cluster Software to start and
to request Oracle Cluster Software to shutdown is the use of Serviceguard packages using SGeRAC
toolkit.
Monitoring
Oracle Cluster Software daemon monitoring is performed through programs initiated by the HP-UX
init process. SGeRAC monitors Oracle Cluster Software to the extent that CSS is a NMAPI2 group
membership client and group member. SGeRAC provides group membership notification to the
remaining group members when CSS enters and leaves the group membership.
Allowed Characters for Oracle 10g/11gR1/11gR2 RAC Cluster Names

Oracle Clusterware uses SGeRAC Group Membership Service to get the node status, it registers
a group in SGeRAC cmgmsd daemon with Oracle cluster name. Since the valid group name in
SGeRAC only allows alpha letter (a-z, A-Z), decimal digital (0-9), underscore (_), dollar sign ($)
and number sign(#), only these characters are allowed in Oracle 10g/11gR1/11gR2 Clusterware
cluster name.
Shared Storage
SGeRAC supports shared storage using HP Shared Logical Volume Manager (SLVM), Cluster File
System (CFS), Cluster Volume Manager (CVM), and ASM (ASM/SLVM in 11i v2/v3 and ASM
over raw disks in 11i v3). CFS and CVM are not supported on all versions of HP-UX (on HP-UX
releases that support them). See About Veritas CFS and CVM from Symantec (page 15).
26
Serviceguard Configuration for Oracle 10g, 11gR1, or 11gR2 RAC
The file /var/opt/oracle/oravg.conf must not be present so Oracle Cluster Software will
not activate or deactivate any shared storage.
Multipathing
Multipathing is automatically configured in HP-UX 11i v3 (this is often called native multipathing).
Multipathing is supported through either SLVM pvlinks or CVM Dynamic Multipath (DMP). In some
configurations, SLVM or CVM does not need to be configured for multipath as the multipath is
provided by the storage array. Since Oracle Cluster Software checks availability of the shared
device for the vote disk through periodic monitoring, the multipath detection and failover time must
be less than CRS's timeout specified by the Cluster Synchronization Service (CSS) MISSCOUNT.
On SGeRAC configurations, the CSS MISSCOUNT value is set to 600 seconds. Multipath failover
time is typically between 30 to 120 seconds. For information on Multipathing and HP-UX 11i v3,
see About Multipathing (page 33).
OCR and Vote Device

Shared storage for the OCR and Vote device should be on supported shared storage volume
managers with multipath configured and with either the correct multipath failover time or CSS
timeout.
Mirroring and Resilvering

On node and cluster wide failures, when SLVM mirroring is used and Oracle resilvering is available,
the recommendation for the logical volume mirror recovery policy is set to either full mirror
resynchronization (NOMWC) or fast resynchronization (MWC) for control and redo files and no
mirror resynchronization (NONE) for the datafiles since Oracle would perform resilvering on the
datafiles based on the redo log.
NOTE: If Oracle resilvering is not available, the mirror recovery policy should be set to either
full mirror resynchronization (NOMWC) or fast resynchronization (MWC) of all control, redo, and
datafiles.
Contact Oracle to determine if your version of Oracle RAC allows resilvering and to appropriately
configure the mirror consistency recovery policy for your logical volumes.
For more information on using NOMWC and MWC, refer to the HP-UX System Administrator's
Guide: Logical Volume Management HP-UX 11i Version 3 at www.hp.com/go/hpux-core-docs
> HP-UX 11i V3.
Shared Storage Activation

Depending on your version of Oracle Cluster Software, the default configuration for activation of
the shared storage for Oracle Cluster Software may be controlled by the /var/opt/oracle/
oravg.conf file. For the default configuration where the shared storage is activated by SGeRAC
before starting Oracle Cluster Software or RAC instance, the oravg.conf file should not be
present.
Listener
Automated Startup and Shutdown
CRS can be configured to automatically start, monitor, restart, and halt listeners.
If CRS is not configured to start the listener automatically at Oracle Cluster Software startup, the
listener startup can be automated with supported commands, such as srvctl and lsnrctl,
through scripts or SGeRAC packages. If the SGeRAC package is configured to start the listener,
the SGeRAC package would contain the virtual IP address required by the listener.
Interface Areas
27
Manual Startup and Shutdown

Manual listener startup and shutdown is supported through the following commands: srvctl and
lsnrctl.
Network Monitoring
SGeRAC cluster provides network monitoring. For networks that are redundant and monitored by
Serviceguard cluster, Serviceguard cluster provides local failover capability between local network
interfaces (LAN) that is transparent to applications utilizing User Datagram Protocol (UDP) and
Transport Control Protocol (TCP).
Virtual IP addresses (floating or package IP address) in Serviceguard provide remote failover
capability of network connection endpoints between cluster nodes, and transparent local failover
capability of network connection endpoints between redundant local network interfaces.
NOTE: Serviceguard cannot be responsible for networks or connection endpoints that it is not
configured to monitor.
SGeRAC Heartbeat Network

Serviceguard supports multiple heartbeat networks, private or public. Serviceguard heartbeat
network can be configured as a single network connection with redundant LAN or multiple
connections with multiple LANs (single or redundant).
CSS Heartbeat Network

The CSS IP addresses for peer communications are fixed IP addresses. When CSS heartbeats are
on a single network connection it does not support multiple heartbeat networks. To protect against
a network single point of failure, the CSS heartbeat network should be configured with redundant
physical networks under SGeRAC monitoring. Since SGeRAC does not support heartbeat over
Hyperfabric (HF) networks, the preferred configuration is for CSS and Serviceguard to share the
same cluster interconnect.
RAC Cluster Interconnect

Each set of RAC instances maintains peer communications on a single connection and may not
support multiple connections on HP-UX with SGeRAC. To protect against a network single point of
failure, the RAC cluster interconnect should be configured with redundant networks under
Serviceguard monitoring and for Serviceguard to take action (either a local failover or an instance
package shutdown, or both) if the RAC cluster interconnect fails. Serviceguard does not monitor
Hyperfabric networks directly (integration of Serviceguard and HF/EMS monitor is supported).
Public Client Access

When the client connection endpoint (virtual or floating IP address) is configured using Serviceguard
packages, Serviceguard provides monitoring, local failover, and remote failover capabilities.
When Serviceguard packages are not used, Serviceguard does not provide monitor or failover
support.
RAC Instances
Automated Startup and Shutdown
CRS can be configured to automatically start, monitor, restart, and halt RAC instances. If CRS is
not configured to automatically start the RAC instance at Oracle Cluster Software startup, the RAC
instance startup can be automated through scripts using supported commandssuch as srvctl
or sqlplus, in an SGeRAC package to start and halt RAC instances.
NOTE:
28
srvctl and sqlplus are Oracle commands.
Manual Startup and Shutdown

Manual RAC instance startup and shutdown is supported through the following commands: srvctl
or sqlplus.
Shared Storage
It is expected the shared storage is available when the RAC instance is started. Since the RAC
instance expects the shared storage to be available, ensure the shared storage is activated. For
SLVM, the shared volume groups must be activated and for CVM, the disk group must be activated.
For CFS, the cluster file system must be mounted.
Oracle Cluster Software requires shared storage for the Oracle Cluster Registry (OCR) and a vote
device. Automatic Storage Management (ASM) can not be used for the OCR and vote device in
prior Oracle 11gR2 versions since these files must be accessible before Oracle Cluster Software
starts.
For Oracle 10g, the minimum required size for each copy of the OCR is 100 MB, and for each
vote disk it is 20 MB. For Oracle 11gR2, the minimum required size for each copy of the OCR is
300 MB, and for each vote disk it is 300 MB.
The Oracle OCR and vote device can be created on supported shared storage, including SLVM
logical volumes, CVM raw volumes, and CFS file systems. Oracle 11gR2 supports OCR and vote
device on ASM over SLVM, ASM over raw device files, and Cluster File System (CFS).
Network Planning for Cluster Communication

The network for cluster communication is used for private internal cluster communications. Although
one network adapter per node is sufficient for the private network, it is recommended that a
minimum of two network adapters for each network to be used for higher availability.
NOTE: Starting with SGeRAC A.11.20, Oracle Grid Infrastructure 11.2.0.2 HAIP feature can
be configured and used in an SGeRAC cluster. All recommended network configuration of SGeRAC
cluster are not supported when using HAIP. For information about most recommended network
configuration of SGeRAC, see Cluster Communication Network Monitoring (page 35) section.
The following are the major categories of inter-cluster traffic in an SGeRAC environment:
Serviceguard HeartBeat (SG-HB)Serviceguard heartbeat and communications traffic. SG-HB

is supported over single or multiple subnet networks.
Cluster Synchronization Services Heartbeat (CSS-HB)CSS heartbeat traffic and communications

traffic for Oracle Clusterware. CSS-HB uses a single logical connection over a single subnet
network.
RAC Interconnect (RAC-IC)RAC instance peer-to-peer traffic and communications for Global
Cache Service (GCS) and Global Enqueue Service (GES), formally Cache Fusion (CF) and
Distributed Lock Manager (DLM). Network HA is provided by the HP-UX platform (Serviceguard
or Auto Port Aggregation (APA)).
Automatic Storage Management Interconnect (ASM-IC) (only when using ASM, Automatic
Storage Management)ASM instance peer-to-peer traffic. When it exists, ASM-IC should be
on the same network as CSS-HB. Network HA is required either through Serviceguard failover
or HP APA .
Global Atomic Broadcast/Link Level Traffic (GAB/LLT) (only when using CFS/CVM)Symantec
cluster heartbeat and communications traffic. GAB/LLT communicates over link level protocol
Network Planning for Cluster Communication
29
(DLPI) and supported over Serviceguard heartbeat subnet networks, including primary and
standby links.
Highly available virtual IP (HAIP) (only when using Oracle Grid Infrastructure 11.2.0.2) IP
addresses, which Oracle Database and Oracle ASM instances use to ensure highly available
and load balanced across the provided set of cluster interconnect interfaces.
The most common network configuration is to have all interconnect traffic for cluster
communications to go on a single heartbeat network that is redundant so that Serviceguard
monitors the network and resolves interconnect failures by cluster reconfiguration.
The following are situations when it is not possible to place all interconnect traffic on a single
network:
RAC GCS (cache fusion) traffic may be very high, so an additional dedicated heartbeat
network for Serviceguard needs to be configured.
Some networks, such as Infiniband, are not supported by CFS/CVM, so the CSS-HB/RAC-IC
traffic may need to be on a separate network that is different from SG-HB network.
Certain configurations for fast re-configurations requires a dual Serviceguard heartbeat network,
and CSS-HB/RAC-IC does not support multiple networks for HA purposes.
In a multiple database configuration, RAC-IC traffic of one database may interfere with RAC-IC
traffic of another database; therefore, the RAC-IC traffic of databases may need to be
separated.
In the above cases, you will see a longer time to recover some network failures beyond those
protected by primary and standby, unless Serviceguard is configured to monitor the network.
A failure of CSS-HB/RAC-IC network in such configuration does not force Serviceguard to reform
the cluster. If Serviceguard is not configured to monitor the network, Oracle will take at least CSS
misscount time interval to resolve the network failure. The default value of CSS misscount in SGeRAC
configurations is 600 seconds.
To avoid longer recovery times, manage Oracle Clusterware and RAC-DB instances using
Serviceguard multi-node packages. In addition, configure the CLUSTER_INTERCONNECT_SUBNET
package configuration parameter (as done with a standard SUBNET package configuration
parameter) in the respective multi-node packages to monitor the CSS-HB/RAC-IC networks.
Planning Storage for Oracle Cluster Software

Oracle Cluster Software requires shared storage for the Oracle Cluster Registry (OCR) and a vote
device. Automatic Storage Management cannot be used for the OCR and vote device in prior
Oracle 11gR2 versions because these files must be accessible before Oracle Cluster Software
starts.
For Oracle 10g, the minimum required size for each copy of the OCR is 100 MB and for each
vote disk it is 20 MB. For Oracle 11gR1, the minimum required size for each copy of the OCR is
300 MB, and for each vote disk it is also 300 MB.
The Oracle OCR and vote device can be created on supported shared storage, including SLVM
logical volumes, CVM raw volumes, and CFS file systems (on HP-UX releases that support Veritas
CFS and CVM). See About Veritas CFS and CVM from Symantec (page 15). Oracle 11gR2
supports OCR and vote device on ASM over SLVM, ASM over raw device files, and Cluster File
System (CFS).
Planning Storage for Oracle 10g/11gR1/11gR2 RAC

NOTE: The Oracle 11gR2 OUI allows only ASM over SLVM, ASM over raw device files, Cluster
File System for Clusterware files, and Database files.
30
Volume Planning with SLVM

Storage capacity for the Oracle database must be provided in the form of logical volumes located
in shared volume groups. The Oracle software requires at least two log files for each Oracle
instance, several Oracle control files and data files for the database itself. For all these files,
Serviceguard Extension for RAC uses HP-UX raw logical volumes located in volume groups that
are shared between the nodes in the cluster. High availability is achieved by using high availability
disk arrays in RAID modes. The logical units of storage on the arrays are accessed from each node
through multiple physical volume links (PV links, also known as alternate links), which provide
redundant paths to each unit of storage. Fill out a Logical Volume worksheet to provide logical
volume names for logical volumes that you will create with the lvcreate command. The Oracle
DBA and the HP-UX system administrator should prepare this worksheet together. Create entries
for shared volumes only. For each logical volume, enter the full pathname of the raw logical volume
device file. Be sure to include the desired size in MB. Following is a sample worksheet filled out.
However, this sample is only representative. For different versions of the Oracle database, the size
of files are different. Refer to Appendix B: Blank Planning Worksheets, for samples of blank
worksheets. Make as many copies as you need. Fill out the worksheet and keep it for future
reference.
Storage Planning with CFS

With CFS, the database software, database files (control, redo, data files), and archive logs may
reside on a cluster file system visible by all nodes. Also, the OCR and vote device can reside on
CFS directories.
The following software needs to be installed in order to use this configuration:
SGeRAC
CFS
CFS and CVM are not supported on all versions of HP-UX (on HP-UX releases that support them.
See About Veritas CFS and CVM from Symantec (page 15)).
will not create an appropriate multi-node package, which means cluster packages will not be
aware of file system changes.
NOTE: For specific CFS Serviceguard Storage Management Suite product information refer to
your version of the HP Serviceguard Storage Management Suite Release Notes.
Volume Planning with CVM

Storage capacity for the Oracle database must be provided in the form of volumes located in
shared disk groups. The Oracle software requires at least two log files for each Oracle instance,
several Oracle control files, and data files for the database itself. For all these files, Serviceguard
Extension for RAC uses HP-UX raw volumes, which are located in disk groups that are shared
between the nodes in the cluster. High availability is achieved by using high availability disk arrays
in RAID nodes. The logical units of storage on the arrays are accessed from each node through
multiple physical volume links via DMP (Dynamic Multipathing) that provides redundant paths to
each unit of storage.
Planning Storage for Oracle 10g/11gR1/11gR2 RAC
31
Fill out the Veritas Volume worksheet to provide volume names for volumes that you will create
using the Veritas utilities. The Oracle DBA and the HP-UX system administrator should prepare this
worksheet together. Create entries for shared volumes only. For each volume, enter the full pathname
of the raw volume device file. Be sure to include the desired size in MB. Following are sample
worksheets filled out. Refer to Appendix B: Blank Planning Worksheets, for samples of blank
worksheets. Make as many copies as you need. Fill out the worksheet and keep it for future
reference.
ORACLE LOGICAL VOLUME WORKSHEET FOR LVM Page ___ of ____
===============================================================================
RAW LOGICAL VOLUME NAME SIZE (MB)
Oracle Cluster Registry: _____/dev/vg_rac/rora_ocr_____100___ (once per cluster)
Oracle Cluster Vote Disk: ____/dev/vg_rac/rora_vote_____20___ (once per cluster)
Oracle Control File: _____/dev/vg_rac/ropsctl1.ctl______110______
Oracle Control File 2: ___/dev/vg_rac/ropsctl2.ctl______110______
Oracle Control File 3: ___/dev/vg_rac/ropsctl3.ctl______110______
Instance 1 Redo Log 1: ___/dev/vg_rac/rops1log1.log_____120______
Instance 1 Redo Log 2: ___/dev/vg_rac/rops1log2.log_____120_______
Instance 1 Redo Log 3: ___/dev/vg_rac/rops1log3.log_____120_______
Instance 1 Redo Log: __________________________________________________
Instance 2 Redo Log 1: ___/dev/vg_rac/rops2log1.log____120________
Instance 2 Redo Log: _________________________________________________
Data: System ___/dev/vg_rac/ropssystem.dbf___500__________
Data: Sysaux ___/dev/vg_rac/ropssysaux.dbf___800__________
Data: Temp ___/dev/vg_rac/ropstemp.dbf______250_______
Data: Users ___/dev/vg_rac/ropsusers.dbf_____120_________
Data: User data ___/dev/vg_rac/ropsdata1.dbf_200__________
Data: User data ___/dev/vg_rac/ropsdata2.dbf__200__________
Data: User data ___/dev/vg_rac/ropsdata3.dbf__200__________
Parameter: spfile1 ___/dev/vg_rac/ropsspfile1.ora __5_____
Password: ______/dev/vg_rac/rpwdfile.ora__5_______
Instance 1 undotbs1: /dev/vg_rac/ropsundotbs1.dbf___500___
Instance 2 undotbs2: /dev/vg_rac/ropsundotbs2.dbf___500___
Data: example1__/dev/vg_rac/ropsexample1.dbf__________160____
ORACLE LOGICAL VOLUME WORKSHEET FOR CVM

Page ___ of ____
===============================================================================
RAW VOLUME NAME
SIZE (MB)
Oracle Cluster Registry: _____/dev/vx/rdsk/ops_dg/ora_ocr_____100___ (once per cluster)
Oracle Cluster Vote Disk: ____/dev/vx/rdsk/ops_dg/ora_vote_____20___ (once per cluster)
Oracle Control File: _____/dev/vx/rdsk/ops_dg/opsctl1.ctl______110______
Oracle Control File 2: ___/dev/vx/rdsk/ops_dg/opsctl2.ctl______110______
Oracle Control File 3: ___/dev/vx/rdsk/ops_dg/opsctl3.ctl______110______
Instance 1 Redo Log 1: ___/dev/vx/rdsk/ops_dg/ops1log1.log_____120______
Instance 1 Redo Log 2: ___/dev/vx/rdsk/ops_dg/ops1log2.log_____120______
Instance 1 Redo Log 3: ___/dev/vx/rdsk/ops_dg/ops1log3.log_____120_______
Instance 2 Redo Log 1: ___/dev/vx/rdsk/ops_dg/ops2log1.log____120________
Instance 2 Redo Log: _________________________________________________
Data: System ___/dev/vx/rdsk/ops_dg/opssystem.dbf___500__________
Data: Sysaux ___/dev/vx/rdsk/ops_dg/opssysaux.dbf___800__________
Data: Temp ___/dev/vx/rdsk/ops_dg/opstemp.dbf______250_______
Data: Users ___/dev/vx/rdsk/ops_dg/opsusers.dbf_____120_________
Data: User data ___/dev/vx/rdsk/ops_dg/opsdata1.dbf_200__________
Data: User data ___/dev/vx/rdsk/ops_dg/opsdata2.dbf__200__________
Data: User data ___/dev/vx/rdsk/ops_dg/opsdata3.dbf__200__________
Parameter: spfile1 ___/dev/vx/rdsk/ops_dg/opsspfile1.ora __5_____
Password: ______/dev/vx/rdsk/ops_dg/pwdfile.ora__5_______
Instance 1 undotbs1: /dev/vx/rdsk/ops_dg/opsundotbs1.dbf___500___
Instance 2 undotbs2: /dev/vx/rdsk/ops_dg/opsundotbs2.dbf___500___
Data: example1__/dev/vx/rdsk/ops_dg/opsexample1.dbf__________160____
32
Installing Serviceguard Extension for RAC

Installing Serviceguard Extension for RAC includes updating the software and rebuilding the kernel
to support high availability cluster operation for Oracle Real Application Clusters.
Prior to installing Serviceguard Extension for RAC, the following must be installed:
Correct version of HP-UX
Correct version of Serviceguard
To install Serviceguard Extension for RAC, use the following steps for each node:
NOTE: All nodes in the cluster must be either SGeRAC nodes or Serviceguard nodes. For the
up-to-date version compatibility for Serviceguard and HP-UX, see the SGeRAC release notes for
your version.
1.
2.
3.
4.
Mount the distribution media in the tape drive, CD, or DVD reader.
Run Software Distributor, using the swinstall command.
Specify the correct input device.
Choose the following bundle from the displayed list:
Serviceguard Extension for RAC
5.
After choosing the bundle, select OK to install the software.
Veritas Cluster Volume Manager (CVM) and Cluster File System (CFS)
CVM (and CFS Cluster File System) are supported on some, but not all current releases of HP-UX.
See the latest release notes for your version of Serviceguard at
www.hp.com/go/hpux-serviceguard-docs.
NOTE: The HP-UX 11i v3 I/O subsystem provides multipathing and load balancing by default.
This is often referred to as native multipathing. When CVM is installed on HP-UX 11i v3, DMP is
the only supported multipathing solution, and is enabled by default.
Veritas Storage Management Products

About Multipathing
Multipathing is automatically configured in HP-UX 11i v3 (this is often called native multipathing),
or in some cases can be configured with third-party software such as EMC Powerpath.
NOTE: The HP-UX 11i v3 I/O subsystem provides multipathing and load balancing by default.
This is often referred to as native multipathing. When CVM is installed on HP-UX 11i v3, DMP is
the only supported multipathing solution, and is enabled by default.
For more information about multipathing in HP-UX 11i v3, see the white paper HP-UX 11i v3 Native
Multipathing for Mass Storage, and the Logical Volume Management volume of the HP-UX System
Administrators Guide at www.hp.com/go/hpux-core-docs > HP-UX 11i v3.
About Device Special Files

HP-UX releases up to and including 11i v2 use a naming convention for device files that encodes
their hardware path. For example, a device file named /dev/dsk/c3t15d0 would indicate SCSI
controller instance 3, SCSI target 15, and SCSI LUN 0. HP-UX 11i v3 introduces a new nomenclature
for device files, known as agile addressing (also called persistent LUN binding). Under the agile
addressing convention, the hardware path name is no longer encoded in a storage devices
nameeach device file name reflects a unique instance number, for example /dev/
[r]disk/disk3, that does not need to change when the hardware path changes. Agile addressing
is the default on new 11i v3 installations, but the I/O subsystem still recognizes the pre-11i v3
Installing Serviceguard Extension for RAC
33
nomenclature. You are not required to migrate to agile addressing when you upgrade to 11i v3,
though you should seriously consider its advantages. It is possible, though not a best practice, to
have legacy DSFs on some nodes and agile addressing on othersthis allows you to migrate the
names on different nodes at different times, if necessary.
NOTE:
The examples in this document use legacy naming conventions.
About Cluster-wide Device Special Files (cDSFs)

Under agile addressing on HP-UX 11i v3, each device has a unique identifier as seen from a given
host; this identifier is reflected in the name of the Device Special File (DSF).
Because DSF names may be duplicated between one host and other, it is possible for different
storage devices to have the same name on different nodes in a cluster, and for the same piece of
storage to be addressed by different names. Cluster-wide device files (cDSFs), available as of the
September 2010 HP-UX Fusion Release, ensure that each storage device used by the cluster has
a unique device file name.
IMPORTANT: Check the latest version of the release notes (at the address given in the preface
to this manual) for information about Serviceguard support for cDSFs.
HP recommends that you use cDSFs for the storage devices in the cluster because this makes it
simpler to deploy and maintain a cluster, and removes a potential source of configuration errors.
For more information, see Creating Cluster-wide Device Special Files (cDSFs) in the Managing
Serviceguard, Eighteenth Edition manual at www.hp.com/go/hpux-serviceguard-docs >
HP Serviceguard .
Points to Note
cDSFs can be created for any group of nodes that you specify, provided that Serviceguard
A.11.20 is installed on each node.
Normally, the group should comprise the entire cluster.
cDSFs apply only to shared storage; they will not be generated for local storage, such as root,
boot, and swap devices.
Once you have created cDSFs for the cluster, HP-UX automatically creates new cDSFs when
you add shared storage.
HP recommends that you do not mix cDSFs with persistent (or legacy DSFs) in a volume group,
and you cannot use cmpreparestg (1m) on a volume group in which they are mixed.
For more information about cmpreparestg, see About Easy Deployment in the Managing
Serviceguard, Nineteenth Edition manual at www.hp.com/go/hpux-serviceguard-docs
> HP Serviceguard .
Where cDSFs Reside

cDSFs reside in two new HP-UX directories, /dev/cdisk for cluster-wide block device files and
/dev/rcdisk for cluster-wide character devicefiles. Persistent DSFs that are not cDSFs continue
to reside in /dev/disk and /dev/rdisk, and legacy DSFs (DSFs using the naming convention
that was standard before HPUX 11i v3) in /dev/dsk and /dev/rdsk. It is possible that a
storage device on an 11i v3 system could be addressed by DSFs of all three types of device
but if you are using cDSFs, you should ensure that you use them exclusively as far as possible.
NOTE: Software that assumes DSFs reside only in /dev/disk and /dev/rdisk will not find
cDSFs and may not work properly as a result.
34
Limitations of cDSFs
cDSFs are supported only within a single cluster; you cannot define a cDSF group that crosses
cluster boundaries.
A node can belong to only one cDSF group.
cDSFs are not supported by CVM, CFS, or any other application that assumes DSFs reside
only in /dev/disk and /dev/rdisk.
cDSFs do not support disk partitions.

Such partitions can be addressed by a device file using the agile addressing scheme, but not
by a cDSF.
For more information about Cluster-wide Device Special Files (cDSFs), see the Managing
Serviceguard, Eighteenth Edition manual at www.hp.com/go/hpux-serviceguard-docs >
HP Serviceguard .
Configuration File Parameters

You need to code specific entries for all the storage groups that you want to use in an Oracle RAC
configuration. If you are using LVM, the OPS_VOLUME_GROUP parameter is included in the cluster
ASCII file. If you are using Veritas CVM, the STORAGE_GROUP parameter is included in the package
ASCII file. Details are as follows:
OPS_VOLUME_GROUP
The name of an LVM volume group whose disks are attached to at least
two nodes in the cluster. The disks will be accessed by more than one
node at a time using SLVM with concurrency control provided by Oracle
RAC. Such disks are considered cluster aware.
Volume groups listed under this parameter are marked for activation in
shared mode. The entry can contain up to 40 characters.
STORAGE_GROUP
This parameter is used for CVM disk groups. Enter the names of all the
CVM disk groups the package will use.
In the ASCII package configuration file, this parameter is called
STORAGE_GROUP.
Unlike LVM volume groups, CVM disk groups are not entered in the cluster configuration file, they
are entered in the package configuration file.
NOTE: CVM 5.x or later with CFS does not use the STORAGE_GROUP parameter because the
disk group activation is performed by the multi-node package. CVM 5.x and later without CFS
uses the STORAGE_GROUP parameter in the ASCII package configuration file in order to activate
the disk group (on HP-UX releases that support Veritas CFS and CVM). See About Veritas CFS
and CVM from Symantec (page 15).
Do not enter the names of LVM volume groups in the package ASCII configuration file.
Cluster Communication Network Monitoring

This section describes the various network configurations for cluster communications in
SGeRAC/10g/11gR1/11gR2 RAC cluster, and how the package configuration parameter
CLUSTER_INTERCONNECT_SUBNET can be used to recover from Oracle cluster communications
network failures in certain configurations.
Support for the Oracle Grid Infrastructure 11.2.0.2 HAIP with SGeRAC A.11.20
The following network configuration requirements must be met to use the Oracle Grid Infrastructure
11.2.0.2 HAIP feature in an SGeRAC cluster:
1. Any networks in the SGeRAC A.11.20 cluster configured for heartbeat must also be configured
as Oracle cluster interconnect network in the Oracle Grid Infrastructure 11.2.0.2. Similarly
Configuration File Parameters
35
any network configured for Oracle cluster interconnect must also be configured as SGeRAC
A.11.20 heartbeat network.
NOTE:
Do not configure Serviceguard heartbeat and Oracle cluster interconnect in mutually
exclusive networks.
2.
Serviceguard standby interfaces must not be configured for the networks used for Serviceguard
heartbeat and Oracle cluster interconnect.
For Oracle Grid Infrastructure 11.2.0.2 HAIP feature to work properly in an SGeRAC cluster and
to have a resilient network configured for Serviceguard heartbeat and Oracle cluster interconnect,
use one of the following network configurations:
1. Configure one or more networks, each using one HP Auto Port Aggregation (APA) interface
for Oracle cluster interconnect and Serviceguard heartbeat. The APA interfaces must contain
two or more physical network interfaces in hot standby mode or in LAN monitor mode.
NOTE: APA interfaces on all the cluster nodes in the same network must use the same APA
mode (LAN monitor or hot standby). Since these monitoring and failover functions work
differently and do not communicate with each other, unexpected failover can occur.
2.
Configure two or more networks on different interfaces without any standby interface for both
Oracle cluster interconnect and Serviceguard heartbeat.
Single Network for Cluster Communications

The single network configuration is the most common configuration for network installations. In this
configuration, there is sufficient bandwidth for all cluster communications traffic to go through one
network. If there are multiple databases, all database traffic goes through the same network.
As shown in Figure 2-1, both CSS-HB and SG-HB are on the same network. The primary and
standby interface pair protects against a single network interface failure. Serviceguard monitors
the network and will perform a local LAN failover (transparent to CSS and RAC) if the primary
interface fails.
NOTE: A package with the CLUSTER_INTERCONNECT_SUBNET parameter is available for both
Modular and Legacy packages. A package with this parameter can be configured only when all
nodes of the cluster are running SGeRAC version A.11.18 or higher. For more information, see
the latest edition of the Managing Serviceguard Eighteenth Edition user guide at
www.hp.com/go/hpux-serviceguard-docs > HP Serviceguard.
Figure 7 Single Network for Cluster Communications
36
Alternate ConfigurationFast Reconfiguration with Low Node Member Timeout

High RAC-IC traffic may interfere with SG-HB traffic and cause unnecessary member timeout if
Serviceguard cluster configuration parameter MEMBER_TIMEOUT is low. If MEMBER_TIMEOUT
cannot be increased, use of an additional network dedicated for SG-HB alone avoids unnecessary
member timeouts when RAC-IC traffic is high. This configuration is for a cluster with two or more
nodes that have high RAC-IC traffic and/or need faster failover (a low value for Serviceguard
configuration parameter MEMBER_TIMEOUT).
NOTE: Starting with Serviceguard A.11.19, the faster failover capability is in core Serviceguard.
This configuration can be used for faster failover.
Figure 8 SG-HB/RAC-IC Traffic Separation
Each primary and standby pair protects against a single failure. With the SG-HB on more than
one subnet, a single subnet failure will not trigger a Serviceguard reconfiguration. If the subnet
with CSS-HB fails, unless subnet monitoring is used, CSS will resolve the interconnect subnet failure
with a CSS cluster reconfiguration. It will wait for the CSS misscount time interval before handling
the CSS-HB subnet failure (by bringing down the node on which the CSS-HB subnet has failed).
The default value of CSS misscount in SGeRAC configurations is 600 seconds.
As shown in Figure 8, CLUSTER_INTERCONNECT_SUBNET can be used in conjunction with the
NODE_FAIL_FAST_ENABLED package configuration parameter to monitor the CSS-HB network.
A failure of CSS-HB subnet on a node should be handled by bringing down that node. Therefore,
set NODE_FAIL_FAST_ENABLED to YES for the package monitoring the CSS-HB subnet. If the
monitored subnet fails, the failure of the CSS-HB subnet on a node will bring down the instance of
the multi-node package and the node where the subnet has failed (When Oracle Clusterware is
configured as a multi-node package and CLUSTER_INTERCONNECT_SUBNET is used to monitor
the CSS-HB subnet).
A failure of CSS-HB subnet on all nodes will result in the multi-node package failing on the nodes
one by one (resulting in that node going down), and one instance of the multi-node package and
node will remain providing services to the clients.
Use a separate package to monitor only the CSS-HB subnet and have Oracle Clusterware multi-node
package depend on the package monitoring the CSS-HB subnet. The NODE_FAIL_FAST_ENABLED
parameter is set to NO for the Oracle Clusterware package, and is set to YES for the package
37
monitoring CSS-HB subnet (Oracle Cluster Interconnect Subnet Package as shown in the package
configuration parameters examples below).
NOTE: Do not configure CLUSTER_INTERCONNECT_SUBNET in the RAC Instance package due
to the RAC-IC network being the same as CSS-HB network.
The following is an example of the relevant package configuration parameters:
Oracle Clusterware Package:
PACKAGE_NAME
PACKAGE_TYPE
LOCAL_LAN_FAILOVER_ALLOWED
NODE_FAIL_FAST_ENABLED
DEPENDENCY_NAME
DEPENDENCY_CONDITION
DEPENDENCY_LOCATION
CRS_PACKAGE
MULTI_NODE
YES
NO
CI-PACKAGE
CI-PACKAGE=UP
SAME_NODE
Oracle Cluster Interconnect Subnet Package: Package to monitor the CSS-HB subnet
PACKAGE_NAME
CI-PACKAGE
PACKAGE_TYPE
MULTI_NODE
YES
YES
CLUSTER_INTERCONNECT_SUBNET
192.168.1.0
NOTE: For information on guidelines to change certain Oracle Clusterware and Serviceguard
cluster configuration parameters, see Guidelines for Changing Cluster Parameters (page 39).
Alternate ConfigurationMultiple RAC Databases

When there are multiple independent RAC databases on the same cluster, and if there is insufficient
bandwidth over a single network, a separate network can be used for different database interconnect
traffic. This will avoid RAC-IC traffic of one database from interfering with that of another.
Figure 9 RAC/RAC-IC Traffic SeparationMultiple Database Configuration
As shown in Figure 9, each primary and standby pair protects against a single failure. If the subnet
with SG-HB (lan1/lan2) fails, Serviceguard will resolve the subnet failure with a Serviceguard
cluster reconfiguration. If the 192.168.2.0 subnet (lan3 and lan4) fails, Oracle instance membership
recovery (IMR) will resolve the interconnect failure subnet, unless Serviceguard subnet monitoring
is used. Oracle will wait for IMR time interval prior to resolving the subnet failure. In SGeRAC
configurations, default value of IMR time interval may be as high as seventeen minutes.
CLUSTER_INTERCONNECT_SUBNET can be configured for RAC instance MNP to monitor the
RAC-IC subnet that is different from CSS-HB subnet. The parameter file (SPFILE or PFILE) for
RAC instances must have cluster_interconnects parameter defined, to hold IP address in
38
the appropriate subnet if the RAC Instances use a RAC-IC network different from CSS-HB network.
No special subnet monitoring is needed for CSS-HB Subnet because Serviceguard monitors the
subnet (heartbeat) and will handle failures of the subnet.
The database instances that use 192.168.2.0 must have cluster_interconnects defined in
their SPFILE or PFILE as follows:
orcl1.cluster_interconnects=192.168.2.1
orcl2.cluster_interconnects=192.168.2.2
NOTE: Do not configure CLUSTER_INTERCONNECT_SUBNET in the RAC instance package if

the RAC-IC network is the same as CSS-HB network.
The following is an example of the relevant package configuration parameters:
Oracle RAC Instance Package
PACKAGE_NAME
PACKAGE_TYPE
CLUSTER_INTERCONNECT_SUBNET
RAC_PACKAGE
MULTI_NODE
YES
NO
192.168.2.0
NOTE: For information on guidelines to change certain Oracle Clusterware and Serviceguard
cluster configuration parameters, see Guidelines for Changing Cluster Parameters (page 39)
Guidelines for Changing Cluster Parameters

This section describes the guidelines for changing the default values of certain Oracle clusterware
and Serviceguard cluster configuration parameters.
The guidelines vary depending on whether clustering interconnect subnet monitoring is used to
monitor the CSS HB subnet or not.
When Cluster Interconnect Subnet Monitoring is used

The Cluster Interconnect Subnet Monitoring feature is used to monitor the CSS-HB network.
If the Serviceguard cluster configuration parameter MEMBER_TIMEOUT is changed, then it is
necessary to also change the Oracle Clusterware parameter CSS miscount.
To change the CSS miscount, use the following guidelines:
The CSS miscount parameter should be greater than the following:
For SLVM: (number of nodes 1) times (F + SLVM timeout) + 15 seconds
For CVM/CFS: (two * number of nodes 1) times F + 15 seconds
When both SLVM and CVM/CFS are used, then take the max of the above two calculations.
When Cluster Interconnect Subnet Monitoring is not Used

The Cluster Interconnect Monitoring is not used to monitor the CSS HB subnet.
If the Serviceguard cluster configuration parameter MEMBER_TIMEOUT is changed, then it is
necessary to also change the Oracle Clusterware parameter CSS miscount.
To change the CSS miscount, use the following guidelines:
The CSS miscount parameter should be greater than the following:
For SLVM: F + SLVM timeout + 15 seconds
For CVM/CFS: 3 times F + 15 seconds
When both SLVM and CVM/CFS are used, then take the max of the above two calculations.
39
NOTE:
1. The F represents the Serviceguard failover time as given by the
max_reformation_duration field of cmviewcl v f line output.
2. SLVM timeout is documented in the whitepaper, LVM link and Node Failure Recovery Time.
Limitations of Cluster Communication Network Monitor

In Oracle Grid Infrastructure 11.2.0.2, using HAIP feature, Oracle allows to use a subnet for
Oracle Clusterware (CSS-HB) and RAC interconnect (RAC-IC) communication which is not configured
in SGeRAC cluster. So beginning with Oracle Grid Infrastructure 11.2.0.2 version,
CLUSTER_INTERCONNECT_SUBNET parameter must not be used in SGeRAC A.11.20 toolkit
Oracle Clusterware Multi-node and Oracle RAC database Multi-node packages.
CLUSTER_INTERCONNECT_SUBNET requires the subnet to be configured within the SGeRAC
cluster.
The Cluster Interconnect Monitoring feature does not coordinate with any feature handling subnet
failures (including self). The failure handling of multiple subnet failures may result in a loss of
services, for example:
A double switch failure resulting in the simultaneous failure of CSS-HB subnet and SG-HB
subnet on all nodes of a two-node cluster. (Assuming the CSS-HB subnet is different from SG-HB
subnet). Serviceguard may choose to retain one node while the failure handling of interconnect
subnets might choose to retain the other node to handle CSS-HB network failure. As a result,
both nodes will go down.
NOTE: To reduce the risk of failure of multiple subnets simultaneously, each subnet must
have its own networking infrastructure (including networking switches).
A double switch failure resulting in the simultaneous failure of CSS-HB subnet and RAC-IC
network on all nodes may result in loss of services (Assuming the CSS-HB subnet is different
from RAC-IC network). The failure handling of interconnect subnets might choose to retain one
node for CSS-HB subnet failures and to retain RAC instance on some other node for RAC-IC
subnet failures. Eventually, the database instance will not run on any node as the database
instance is dependent on clusterware to run on that node.
Cluster Interconnect Monitoring Restrictions

In addition to the above limitations, the Cluster Interconnect Monitoring feature has the following
restriction:
Cluster Lock device/Quorum Server/Lock Lun must be configured in the cluster.
Creating a Storage Infrastructure with LVM

In addition to configuring the cluster, you create the appropriate logical volume infrastructure to
provide access to data from different nodes. This is done with Logical Volume Manager (LVM) or
Veritas Cluster Volume Manager (CVM). LVM configuration is done before cluster configuration,
and CVM configuration is done after cluster configuration.
This section describes how to create LVM volume groups for use with Oracle data. Before configuring
the cluster, you create the appropriate logical volume infrastructure to provide access to data from
different nodes. This is done with Logical Volume Manager. Separate procedures are given for
the following:
40
Building Volume Groups for RAC on Mirrored Disks
Building Mirrored Logical Volumes for RAC with LVM Commands
Creating RAC Volume Groups on Disk Arrays
Creating Logical Volumes for RAC on Disk Arrays
The Event Monitoring Service HA Disk Monitor provides the capability to monitor the health of
LVM disks. If you intend to use this monitor for your mirrored disks, you should configure them in
physical volume groups. For more information, refer to the manual Using HA Monitors.
NOTE: When using LVM version 2.x, the volume groups are supported with Serviceguard. The
steps shown in the following section are for configuring the volume groups in Serviceguard clusters
LVM version 1.0.
For more information on using and configuring LVM version 2.x, see the HP-UX 11i Version 3:
HP-UX System Administrator's Guide: Logical Volume Management located at www.hp.com/go/
hpux-core-docs > HP-UX 11i v3.
For LVM version 2.x compatibility requirements see the Serviceguard/SGeRAC/SMS/Serviceguard
Mgr Plug-in Compatibility and Feature Matrix at www.hp.com/go/hpux-serviceguard-docs
> HP Serviceguard Extension for RAC.
NOTE: For more information, see the Serviceguard Version A.11.20 Release Notes at
www.hp.com/go/hpux-serviceguard-docs > HP Serviceguard Extension for
RAC.
Building Volume Groups for RAC on Mirrored Disks

The procedure described in this section uses physical volume groups for mirroring of individual
disks to ensure that each logical volume is mirrored to a disk on a different I/O bus. This kind of
arrangement is known as PVG-strict mirroring. It is assumed that your disk hardware is already
configured so that the disk can be used as a mirror copy, which is connected to each node on a
different bus other than the bus that is used for the other (primary) copy.
Creating Volume Groups and Logical Volumes

If your volume groups have not been set up, use the procedure in the next sections. If you have
already done LVM configuration, skip ahead to the section Installing Oracle Real Application
Clusters (page 47).
Selecting Disks for the Volume Group
Obtain a list of the disks on both nodes and identify which device files are used for the same disk
on both. Use the following command on each node to list available disks as they are known to
each system:
# lssf /dev/dsk/*
In the following examples, we use /dev/rdsk/c1t2d0 and /dev/rdsk/c0t2d0, which are

the device names for the same disks on both ftsys9 and ftsys10. In the event that the device
file names are different on the different nodes, make a careful note of the correspondences.
Creating Physical Volumes
On the configuration node (ftsys9), use the pvcreate command to define disks as physical
volumes. This only needs to be done on the configuration node. Use the following commands to
create two physical volumes for the sample configuration:
# pvcreate -f /dev/rdsk/c1t2d0
41
Creating a Volume Group with PVG-Strict Mirroring

Use the following steps to build a volume group on the configuration node (ftsys9). Later, the
same volume group will be created on other nodes.
1. Set up the group directory for vgops:
# mkdir /dev/vg_rac
2.
Create a control file named group in the directory /dev/vg_rac, as follows:

# mknod /dev/vg_rac/group c 64 0xhh0000
The major number is always 64, and the hexadecimal minor number has the form
0xhh0000
where hh must be unique to the volume group you are creating. Use the next hexadecimal
number that is available on your system, after the volume groups that are already configured.
Use the following command to display a list of existing volume groups:
# ls -l /dev/*/group
3.
Create the volume group and add physical volumes to it with the following commands:
# vgcreate -g bus0 /dev/vg_rac /dev/dsk/c1t2d0
# vgextend -g bus1 /dev/vg_rac /dev/dsk/c0t2d0
The first command creates the volume group and adds a physical volume to it in a physical
volume group called bus0. The second command adds the second drive to the volume group,
locating it in a different physical volume group named bus1. The use of physical volume
groups allows the use of PVG-strict mirroring of disks and PV links.
4.
Repeat this procedure for additional volume groups.
Building Mirrored Logical Volumes for RAC with LVM Commands

After you create volume groups and define physical volumes for use in them, you define mirrored
logical volumes for data, logs, and control files. It is recommended that you use a shell script to
issue the commands described in the next sections. The commands you use for creating logical
volumes vary slightly depending on whether you are creating logical volumes for RAC redo log
files or for use with Oracle data.
Creating Mirrored Logical Volumes for RAC Redo Logs and Control Files
Create logical volumes for use as redo log and control files by selecting mirror consistency recovery.
Use the same options as in the following example:
# lvcreate -m 1 -M n -c y -s g -n redo1.log -L 408 /dev/vg_rac
-m 1Specifies single mirroring.
-M nEnsures that mirror write cache recovery is set off.
-c yMirror consistency recovery is enabled.
-s gMirroring is PVG-strict. It occurs between different physical volume groups.
-n redo1.logSpecify the name of the logical volume.
-L 28allocates 28 megabytes.
NOTE: Use the -c y options for both redo logs and control files. These options allow the redo
log files to be resynchronized by SLVM following a system crash before Oracle recovery proceeds.
If these options are not set correctly, you may not be able to continue with database recovery.
If the command is successful, the system will display messages like the following:
42
Logical volume /dev/vg_rac/redo1.log has been successfully created

with character device /dev/vg_rac/rredo1.log
Logical volume /dev/vg_rac/redo1.log has been successfully extended
NOTE: With LVM 2.1 and above, mirror write cache (MWC) recovery can be set to ON for
RAC Redo Logs and Control Files volumes. Example:
# lvcreate -m 1 -M y -s g -n redo1.log -L 408 /dev/vg_rac
NOTE: The character device file name (also called the raw logical volume name) is used by the
Oracle DBA in building the RAC database.
Creating Mirrored Logical Volumes for RAC Data Files

Following a system crash, the mirrored logical volumes need to be resynchronized, which is known
as resilvering.
On node and cluster-wide failures, when SLVM mirroring is used and Oracle resilvering is available,
the recommendation is no mirror resynchronization (NONE) for the datafiles because Oracle would
perform resilvering on the datafiles based on the redo log.
If Oracle resilvering is not available, the mirror recovery policy should be set to either full mirror
resynchronization (NOMWC) or fast resynchronization (MWC) . For more information on using
NOMWC and MWC, refer to the HP-UX System Administrator's Guide: Logical Volume Management
HP-UX 11i Version 3 on www.hp.com/go/hpux-core-docs > HP-UX 11i v3.
In the case Oracle resilvering is not available, create logical volumes for use as Oracle data
files by using the same options as in the following:
Example for NOMWC:
# lvcreate -m 1 -M n -c y -s g -n system.dbf -L 408 /dev/vg_rac
-M nEnsures that mirror write cache recovery is set to OFF.
-c yMirror consistency recovery is enabled.
-n system.dbfSpecify the name of the logical volume.
-L 408Allocates 408 megabytes.
Example for MWC

# lvcreate -m 1 -M y -s g -n system.dbf -L 408 dev/vg_rac
The -M y option ensures that mirror write cache recovery is set ON.
NOTE: Contact Oracle to determine if your version of Oracle RAC allows resilvering and to
appropriately configure the mirror consistency recovery policy for your logical volumes.
In the case Oracle resilvering is available, create logical volumes for use as Oracle data files
by using the same options as in the following example:
# lvcreate -m 1 -M n -c n -s g -n system.dbf -L 408 /dev/vg_rac
-M nEnsures that mirror write cache recovery is set to OFF.
-c nMirror consistency recovery is disabled.
-n system.dbfSpecify the name of the logical volume.
-L 408Allocates 408 megabytes.
If the command is successful, the system will display messages like the following:
43
Logical volume /dev/vg_rac/system.dbf has been successfully created

with character device /dev/vg_rac/rsystem.dbf
Logical volume /dev/vg_rac/system.dbf has been successfully extended
NOTE: The character device file name (also called the raw logical volume name) is used by the
Oracle DBA in building the OPS database.
Creating RAC Volume Groups on Disk Arrays

The procedure described in this section assumes that you are using RAID-protected disk arrays and
LVMs physical volume links (PV links) to define redundant data paths from each node in the cluster
to every logical unit on the array.
On your disk arrays, you should use redundant I/O channels from each node, connecting them
to separate controllers on the array. Then you can define alternate links to the LUNs or logical
disks you have defined on the array. If you are using SAM, choose the type of disk array you wish
to configure, and follow the menus to define alternate links. If you are using LVM commands,
specify the links on the command line.
The following example shows how to configure alternate links using LVM commands. The following
disk configuration is assumed:
8/0.15.0
8/0.15.1
8/0.15.2
8/0.15.3
8/0.15.4
8/0.15.5
/dev/dsk/c0t15d0
/dev/dsk/c0t15d1
/dev/dsk/c0t15d2
/dev/dsk/c0t15d3
/dev/dsk/c0t15d4
/dev/dsk/c0t15d5
/*
/*
/*
/*
/*
/*
I/O
I/O
I/O
I/O
I/O
I/O
Channel
Channel
Channel
Channel
Channel
Channel
0
0
0
0
0
0
(8/0)
(8/0)
(8/0)
(8/0)
(8/0)
(8/0)
10/0.3.0
10/0.3.1
10/0.3.2
10/0.3.3
10/0.3.4
10/0.3.5
/dev/dsk/c1t3d0
/dev/dsk/c1t3d1
/dev/dsk/c1t3d2
/dev/dsk/c1t3d3
/dev/dsk/c1t3d4
/dev/dsk/c1t3d5
/*
/*
/*
/*
/*
/*
I/O
I/O
I/O
I/O
I/O
I/O
Channel
Channel
Channel
Channel
Channel
Channel
1
1
1
1
1
1
(10/0)
(10/0)
(10/0)
(10/0)
(10/0)
(10/0)
SCSI
SCSI
SCSI
SCSI
SCSI
SCSI
SCSI
SCSI
SCSI
SCSI
SCSI
SCSI
address
address
address
address
address
address
address
address
address
address
address
address
15
15
15
15
15
15
3
3
3
3
3
3
LUN
LUN
LUN
LUN
LUN
LUN
0
1
2
3
4
5
*/
*/
*/
*/
*/
*/
LUN
LUN
LUN
LUN
LUN
LUN
0
1
2
3
4
5
*/
*/
*/
*/
*/
*/
Assume that the disk array has been configured, and that both the following device files appear
for the same LUN (logical disk) when you run the ioscan command:
/dev/dsk/c0t15d0
/dev/dsk/c1t3d0
Use the following procedure to configure a volume group for this logical disk:
1. Set up the group directory for vg_rac:
# mkdir /dev/vg_rac
2.
Create a control file named group in the directory /dev/vg_rac, as follows:

The major number is always 64, and the hexadecimal minor number has the form
0xhh0000
number that is available on your system, after the volume groups that are already configured.
Use the following command to display a list of existing volume groups:
3.
Use the pvcreate command on one of the device files associated with the LUN to define the
LUN to LVM as a physical volume.
44
It is only necessary to do this with one of the device file names for the LUN. The -f option is
only necessary if the physical volume was previously used in some other volume group.
4.
Use the following to create the volume group with the two links:
# vgcreate /dev/vg_rac /dev/dsk/c0t15d0 /dev/dsk/c1t3d0
LVM will now recognize the I/O channel represented by/dev/dsk/c0t15d0 as the primary link
to the disk. If the primary link fails, LVM will automatically switch to the alternate I/O channel
represented by /dev/dsk/c1t3d0. Use the vgextend command to add additional disks to the
volume group, specifying the appropriate physical volume name for each PV link.
Repeat the entire procedure for each distinct volume group you wish to create. For ease of system
administration, you may wish to use different volume groups to separate logs from data and control
files.
NOTE: The default maximum number of volume groups in HP-UX version 2.0 is 10. If you intend
to create enough new volume groups that the total exceeds 10, you must increase the maxvgs
system parameter and then reboot the system. Use the kctune utility to change kernel parameter
area, then choose Configurable Parameters, maxvgs appears on the list.
Creating Logical Volumes for RAC on Disk Arrays

After you create volume groups and add PV links to them, you define logical volumes for data,
logs, and control files. The following are some examples:
#
#
#
#
lvcreate
lvcreate
lvcreate
lvcreate
-n
-n
-n
-n
ops1log1.log -L 4 /dev/vg_rac
opsctl1.ctl -L 4 /dev/vg_rac
system.dbf -L 28 /dev/vg_rac
opsdata1.dbf -L 1000 /dev/vg_rac
Oracle Demo Database Files

The following set of files is required for the Oracle demo database that can be created during the
installation process.
Table 1 Required Oracle File Names for Demo Database
Logical Volume Name
LV Size
(MB)
Raw Logical Volume Path Name
Oracle File
Size (MB)*
opsctl1.ctl
118
/dev/vg_rac/ropsctl1.ctl
110
opsctl2.ctl
118
110
opsctl3.ctl
118
110
ops1log1.log
128
/dev/vg_rac/rops1log1.log
120
ops1log2.log
128
120
ops1log3.log
128
120
ops2log1.log
128
120
ops2log2.log
128
120
ops2log3.log
128
120
opssystem.dbf
408
/dev/vg_rac/ropssystem.dbf
400
opssysaux.dbf
808
/dev/vg_rac/ropssysaux.dbf
800
opstemp.dbf
258
/dev/vg_rac/ropstemp.dbf
250
opsusers.dbf
128
/dev/vg_rac/ropsusers.dbf
120
opsdata1.dbf
208
/dev/vg_rac/ropsdata1.dbf
200
45
Table 1 Required Oracle File Names for Demo Database (continued)

Logical Volume Name
LV Size
(MB)
Raw Logical Volume Path Name
Oracle File
Size (MB)*
opsdata2.dbf
208
200
opsdata3.dbf
208
200
opsspfile1.ora
/dev/vg_rac/ropsspfile1.ora
pwdfile.ora
/dev/vg_rac/rpwdfile.ora
opsundotbs1.dbf
508
/dev/vg_rac/ropsundotbs1.log
500
opsundotbs2.dbf
508
/dev/vg_rac/ropsundotbs2.log
500
example1.dbf
168
/dev/vg_rac/ropsexample1.dbf
160
The size of the logical volume is larger than the Oracle file size because Oracle needs extra space
to allocate a header in addition to the file's actual data capacity.
Create these files if you wish to build the demo database. The three logical volumes at the bottom
of the table are included as additional data files, that you can create as needed, supplying the
appropriate sizes. If your naming conventions require, you can include the Oracle SID and/or the
database name to distinguish files for different instances and different databases. If you are using
the ORACLE_BASE directory structure, create symbolic links to the ORACLE_BASE files from the
appropriate directory. Example:
# ln -s /dev/vg_rac/ropsctl1.ctl/u01/ORACLE/db001/ctrl01_1.ctl
After creating these files, set the owner to oracle and the group to dba with a file mode of 660.
The logical volumes are now available on the primary node, and the raw logical volume names
can now be used by the Oracle DBA.
Displaying the Logical Volume Infrastructure

To display the volume group, use the vgdisplay command:
# vgdisplay -v /dev/vg_rac
Exporting the Logical Volume Infrastructure

Before the Oracle volume groups can be shared, their configuration data must be exported to other
nodes in the cluster. This is done either in Serviceguard Manager or by using HP-UX commands,
as shown in the following sections.
All volume and file system related functions are in the Disks and File Systems (fsweb) tool that is
also a plugin for the HP-UX System Management Homepage (HP SMH). It can be launched from
the HP SMH. Serviceguard Manager provides a link to fsweb tool and launches the fsweb tool for
all disk and file system (LVM) related configurations. For CVM/CFS related configurations, the
fsweb tool launches the Veritas Enterprise Administrator (VEA) tool.
NOTE: Serviceguard Manager is the graphical user interface for Serviceguard. It is available as
a plug-in to the System Management Homepage (SMH). SMH is a web-based graphical user
interface (GUI) that replaces SAM as the system administration GUI as of HP-UX 11i v3 (but, you
can run the SAM terminal interface). See Using SAM in the latest edition of the Managing
Serviceguard users guide.
Exporting with LVM Commands

Use the following commands to set up the same volume group on another cluster node. In this
example, the commands set up a new volume group on a system known as ftsys10. This volume
group holds the same physical volume that was created on a configuration node known as ftsys9.
To set up the volume group on ftsys10 (and other nodes), use the following steps:
46
1.
On ftsys9, copy the mapping of the volume group to a specified file.

# vgexport -s -p -m /tmp/vg_rac.map
2.
/dev/vg_rac
Still on ftsys9, copy the map file to ftsys10 (and to additional nodes as necessary.)
# rcp /tmp/vg_rac.map ftsys10:/tmp/vg_rac.map
3.
On ftsys10 (and other nodes, as necessary), create the volume group directory and the
control file named group.
# mkdir /dev/vg_rac
For the group file, the major number is always 64, and the hexadecimal minor number has
the form
0xhh0000
where hh must be unique to the volume group you are creating. If possible, use the same
number as on ftsys9. Use the following command to display a list of existing volume groups:
4.
Import the volume group data using the map file from node ftsys9. On node ftsys10 (and
other nodes, as necessary), enter:
# vgimport -s -m /tmp/vg_rac.map /dev/vg_rac
Installing Oracle Real Application Clusters

NOTE: Some versions of Oracle RAC requires installation of additional software. Refer to your
version of Oracle for specific requirements.
Before installing the Oracle Real Application Cluster software, make sure the storage cluster is
running. Login as the oracle user on one node and then use the Oracle installer to install Oracle
software and to build the correct Oracle runtime executables. When executables are installed to
a local file system on each node, the Oracle installer copies the executables to the other nodes in
the cluster.
For details on Oracle installation, refer to the Oracle installation documentation. As part of this
installation, the Oracle installer installs the executables and optionally, the Oracle installer can
build an Oracle demo database on the primary node. The demo database files can be the character
(raw) device files names for the logical volumes create earlier.
For a demo database on SLVM or CVM, create logical volumes as shown in Table 1: Required
Oracle File Names for Demo Database . As the installer prompts for the database file names,
either the pathnames of the raw logical volumes instead of using the defaults. If you do not wish
to install the demo database, select install software only.
Creating a Storage Infrastructure with CFS

In addition to configuring the cluster, you create the appropriate logical volume infrastructure to
provide access to data from different nodes. This is done with Logical Volume Manager (LVM) or
Veritas Cluster Volume Manager (CVM).
You can also use a mixture of volume types, depending on your needs. LVM configuration is done
before cluster configuration, and CVM configuration is done after cluster configuration. (Note for
HP-UX releases that support Veritas CFS and CVM. See About Veritas CFS and CVM from
Symantec (page 15))
This section has information about configuring a cluster that uses the Veritas cluster file system
(CFS) with Veritas Cluster Volume Manager (CVM) 5.x or later. The next section (Creating a
Storage Infrastructure with CVM (page 52)) has information about configuring the Veritas Cluster
Volume Manager (CVM) with other filesystems, not CFS.
Installing Oracle Real Application Clusters
47
For more information, refer to your version of the Serviceguard Extension for RAC Release Notes
and HP Serviceguard Storage Management Suite Release Notes located at
www.hp.com/go/hpux-serviceguard-docs.
will not create an appropriate multi-node packagecluster packages will not be aware of file
system changes.
Creating an SGeRAC Cluster with CFS for Oracle 11gR1 or 11gR2

The following software needs to be pre-installed in order to use this configuration.
NOTE: If you are installing CFS 4.1 or higher version with either the Storage Management Suite
(SMS) bundle or Mission Critical Operating Environment (MCOE), High Availability Operating
Environment (HAOE), or Data Center Operating Environment (DCOE), use the appropriate product
number as described in the HP Serviceguard Storage Management Suite Release Notes.
With CFS, the database software and database files (control, redo, data files), and archive logs
may reside on a cluster file system that is visible by all nodes.
In the example below, both the Oracle RAC software and datafiles reside on CFS. There is a single
Oracle home. Three CFS file systems are created for Oracle home, Oracle datafiles, and for the
Oracle Cluster Registry (OCR) and vote device. The Oracle Cluster Software home is on a local
file system.
/cfs/mnt1 for Oracle Base and Home
/cfs/mnt2 for Oracle datafiles
/cfs/mnt3 - for OCR and Vote device
Initializing the Veritas Volume Manager

Use the following steps to create a two node SGeRAC cluster with CFS and Oracle:
1. Initialize the Veritas Volume Manager.
NOTE:
2.
CVM 5.x or later does not require rootdg.
Create the cluster ASCII file:

# cd /etc/cmcluster
# cmquerycl -C clm.asc -n ever3a -n ever3b
Edit the cluster file.
3.
Create the cluster:

# cmapplyconf -C clm.asc
4.
Start the cluster:

# cmruncl
# cmviewcl
The following output will be displayed:
CLUSTER
ever3_cluster
48
STATUS
up
NODE
ever3a
ever3b
5.
STATUS
up
up
STATE
running
running
Configure the Cluster Volume Manager (CVM).

Configure the system multi-node package, SG-CFS-pkg, to configure and start the CVM/CFS
stack. The SG-CFS-pkg does not restrict heartbeat subnets to a single subnet and supports
multiple subnets.
# cfscluster config -s
CVM is now configured Starting CVM
When CVM starts up, it selects a master node. From this node, you must issue the disk group
configuration commands. To determine the master node, issue the following command from
each node in the cluster:
# vxdctl -c mode
mode: enabled: cluster active SLAVEmaster: ever3b
or
mode: enabled: cluster active MASTERmaster: ever3b
6.
Converting disks from LVM to CVM.

You can use the vxvmconvert utility to convert LVM volume groups into CVM disk groups.
Before you can do this, the volume group must be deactivatedany package that uses the
volume group must be halted. This procedure is described in the lastest edition of the Managing
Serviceguard user guide.
7.
Initializing disks for CVM/CFS.

You need to initialize the physical disks that will be employed in CVM disk groups. If a physical
disk has been previously used with LVM, you should use the pvremove command to delete
the LVM header data from all the disks in the volume group (if you have not previously used
the disk with LVM, this is not necessary).
To initialize a disk for CVM, log on to the master node, then use the vxdiskadm program to
initialize multiple disks, or use the vxdisksetup command to initialize one disk at a time,
as in the following example:
# /etc/vx/bin/vxdisksetup -i c4t4d0
8.
Create the disk group for RAC.

Use the vxdg command to create disk groups. Use the -s option to specify shared mode, as
in the following example:
# vxdg -s init cfsdg1 c4t4d0
9.
Create the disk group multi-node package. Use the following command to add the disk group
to the cluster:
# cfsdgadm add cfsdg1 all=sw
Package name SG-CFS-DG-1 was generated to control the resource
shared disk group cfsdg1 is associated with the cluster.
10. Activate the disk group.

# cfsdgadm activate cfsdg1
49
11. Creating volumes and adding a cluster filesystem.

# vxassist -g cfsdg1 make vol1 10240m
#vxassist -g cfsdg1 make vol2 10240m
# vxassist -g cfsdg1 make vol3 600m
# newfs -F vxfs /dev/vx/rdsk/cfsdg1/vol1
version 7 layout
10485760 sectors, 10485760 blocks of size 1024, log size 16384 blocks
largefiles supported
version 7 layout
version 7 layout
12. Configure mount point.
# cfsmntadm add cfsdg1 vol1 /cfs/mnt1 all=rw
Package name SG-CFS-MP-1 was generated to control the resource.
Mount point /cfs/mnt1 was associated with the cluster.
#cfsmntadm add cfsdg1 vol2 /cfs/mnt2 all=rw
Mount point /cfs/mnt2 was associated with the cluster.
# cfsmntadm add cfsdg1 vol3 /cfs/mnt3 all=rw
Mount point /cfs/mnt3 that was associated with the cluster.
NOTE: The diskgroup and mount point multi-node packages (SG-CFS-DG_ID# and
SG-CFS-MP_ID#) do not monitor the health of the disk group and mount point. They check
that the application packages that depend on them have access to the disk groups and mount
points. If the dependent application package loses access and cannot read and write to the
disk, it will fail. However, the DG or MP multi-node package will not fail.
13. Mount cluster filesystem.
# cfsmount /cfs/mnt1
50
14. Check CFS mount points.

# bdf | grep cfs
/dev/vx/dsk/cfsdg1/vol1
10485760
36455 9796224
0% /cfs/mnt1
10485760
36455 9796224
0% /cfs/mnt2
614400 17653 559458 3% /cfs/mnt3
15. View the configuration.

# cmviewcl
CLUSTER
ever3_cluster
NODE
ever3a
ever3b
STATUS
up
STATUS
up
up
STATE
running
running
MULTI_NODE_PACKAGES
PACKAGE
SG-CFS-pkg
SG-CFS-DG-1
SG-CFS-MP-1
SG-CFS-MP-2
SG-CFS-MP-3
STATUS
up
up
up
up
up
STATE
running
running
running
running
running
AUTO_RUN
enabled
enabled
enabled
enabled
enabled
SYSTEM
yes
no
no
no
no
system changes.
Deleting CFS from the Cluster

Halt the applications that are using CFS file systems.
1. Unmount CFS mount points.
# cfsumount /cfs/mnt1
2.
Delete mount point multi-node package.

# cfsmntadm delete /cfs/mnt1
The following output will be generated:
Mount point /cfs/mnt1 was disassociated from the cluster
Mount point /cfs/mnt2 was disassociated from the cluster
51

Mount point /cfs/mnt3 was disassociated from the cluster Cleaning
up resource controlling shared disk group cfsdg1 Shared disk group
cfsdg1 was disassociated from the cluster.
NOTE:
3.
The disk group package is deleted if there is no dependency.
Delete disk group multi-node package.

# cfsdgadm delete cfsdg1
Shared disk group cfsdg1 was disassociated from the cluster.
NOTE: cfsmntadm delete also deletes the disk group if there is no dependent package.
To ensure the disk group deletion is complete, use the above command to delete the disk
group package.
4.
De-configure CVM.
# cfscluster stop
Stopping CVM...CVM is stopped
# cfscluster unconfig
CVM is now unconfigured
Creating a Storage Infrastructure with CVM

In addition to configuring the cluster, you need to create the appropriate logical volume infrastructure
to provide access to data from different nodes.
This is done with Logical Volume Manager (LVM) or Veritas Cluster Volume Manager (CVM). LVM
configuration is done before cluster configuration, and CVM configuration is done after cluster
configuration (on HP-UX releases that support Veritas CFS and CVM). See About Veritas CFS and
CVM from Symantec (page 15).
This section shows how to configure storage using the Veritas Cluster Volume Manager (CVM).
The examples show how to configure RAC disk groups. Also, you can create CVM disk groups
for non-RAC use. For more information, including details about configuration of plexes (mirrors),
multipathing, and RAID, refer to the HP-UX documentation for the Veritas Volume Manager.
Initializing the Veritas Volume Manager

If you are about to create disk groups for the first time, you need to initialize the Volume Manager.
This is done by creating a disk group known as rootdg that contains at least one disk. Use the
following command after installing CVM on each node:
# vxinstall
This displays a menu-driven program that steps you through the CVM initialization sequence. From
the main menu, choose the Custom option, and specify the disk you wish to include in rootdg.
IMPORTANT: Creating a rootdg disk group is only necessary the first time you use the Volume
Manager. CVM 5.x or later does not require a rootdg.
52
Using CVM 5.x or later

This section has information on how to set up the cluster and the system multi-node package with
CVMwithout the CFS filesystem, on HP-UX releases that support them. See About Veritas CFS
and CVM from Symantec (page 15).
Preparing the Cluster and the System Multi-node Package for use with CVM 5.x or later
The following steps describe how to prepare the cluster and the system multi-node package with
CVM 5.x or later only.
1. Create the cluster file.
# cd /etc/cmcluster
# cmquerycl -C clm.asc -n ever3a -n ever3b
Edit the cluster file.
2.
Create the cluster.

# cmapplyconf -C clm.asc
Start the cluster.

# cmruncl
# cmviewcl
CLUSTER
STATUS
ever3_cluster
up
NODE
ever3a
ever3b
3.
STATUS
up
up
STATE
running
running
Configure the Cluster Volume Manager (CVM).

To configure and start the CVM stack, you need to configure the system multi-node package,
SG-CFS-pkg. The SG-CFS-pkg does not restrict heartbeat subnets to a single subnet and
supports multiple subnets.
Use the cmapplyconf command:
# cmapplyconf -P /etc/cmcluster/cfs/SG-CFS-pkg.conf
# cmrunpkg SG-CFS-pkg
When CVM starts, it selects a master node. From this node, you must issue the disk group
configuration commands. To determine the master node, issue the following command from
each node in the cluster:
# vxdctl -c mode
mode: enabled: cluster active SLAVEmaster: ever3b
or
mode: enabled: cluster active MASTERmaster: ever3b
Converting disks from LVM to CVM.

Use the vxvmconvert utility to convert LVM volume groups into CVM disk groups. Before
you can do this, the volume group must be deactivated, which means that any package
53
that uses the volume group must be halted. This procedure is described in the Managing
Serviceguard Eighteenth Edition user guide Appendix G.
Initializing disks for CVM.

It is necessary to initialize the physical disks that will be employed in CVM disk groups.
If a physical disk has been previously used with LVM, you should use the pvremove
command to delete the LVM header data from all the disks in the volume group (this is
not necessary if you have not previously used the disk with LVM).
To initialize a disk for CVM, log on to the master node, then use the vxdiskadm program
to initialize multiple disks, or use the vxdisksetup command to initialize one disk at a
time, as in the following example:
# /etc/vx/bin/vxdisksetup -i c4t4d0
Create the disk group for RAC.

Use the vxdg command to create disk groups. Use the -s option to specify shared mode,
as in the following example:
# vxdg -s init ops_dg c4t4d0
4.
Creating volumes and adding a cluster filesystem.

# vxassist -g ops_dg make vol1 10240m
#vxassist -g ops_dg make vol2 10240m
# vxassist -g ops_dg make vol3 300m
5.
View the configuration.

# cmviewcl
CLUSTER
ever3_cluster
NODE
ever3a
ever3b
STATUS
up
STATUS
up
up
STATE
running
running
MULTI_NODE_PACKAGES
PACKAGE
SG-CFS-pkg
54
STATUS
up
STATE
running
AUTO_RUN
enabled
SYSTEM
yes
IMPORTANT: After creating these files, use the vxedit command to change the ownership of
the raw volume files to oracle and the group membership to dba, and to change the permissions
to 660. Example:
# cd /dev/vx/rdsk/ops_dg
# vxedit -g ops_dg set user=oracle *
# vxedit -g ops_dg set group=dba *
# vxedit -g ops_dg set mode=660 *
system changes.
Mirror Detachment Policies with CVM
The required CVM disk mirror detachment policy is "global"as soon as one node cannot see a
specific mirror copy (plex), all nodes cannot see it as well. The alternate policy is "local"if one
node cannot see a specific mirror copy, then CVM will deactivate access to the volume for that
node only.
This policy can be reset on a disk group basis by using the vxedit command, as follows:
# vxedit set diskdetpolicy=global <DiskGroupName>
NOTE: The specific commands for creating mirrored and multipath storage using CVM are
described in the HP-UX documentation for the Veritas Volume Manager.
Using CVM 5.x

This section has information on how to prepare the cluster and the system multi-node package with
CVM 5.x (on HP-UX releases that support them). See About Veritas CFS and CVM from Symantec
(page 15).
Preparing the Cluster for Use with CVM 5.x

To use the Veritas Cluster Volume Manager (CVM) version 5.x, the cluster must be running with a
special CVM package. Before you create disk groups, the cluster must be configured and running.
NOTE:
Cluster configuration is described in the previous section.
To prepare the cluster for CVM disk group configuration, you need to ensure that only one heartbeat
subnet is configured. Then, use the following command, which creates the special package that
communicates cluster information to CVM:
# cmapplyconf -P /etc/cmcluster/cvm/VxVM-CVM-pkg.conf
WARNING!
The above file should never be edited.
55
After the above command completes, start the cluster and create disk groups for shared use as
described in the following sections.
Starting the Cluster and Identifying the Master Node
Run the cluster to activate the special CVM package:
# cmruncl
After the cluster is started, it will run with a special system multi-node package named
VxVM-CVM-pkg that is on all nodes. This package is shown in the following output of the cmviewcl
-v command:
CLUSTER
bowls
STATUS
up
NODE
spare
split
strike
STATUS
up
up
up
STATE
running
running
running
SYSTEM_MULTI_NODE_PACKAGES:
PACKAGE
STATUS
VxVM-CVM-pkg up
STATE
running
When CVM starts up, it selects a master node. From this node, you must issue the disk group
configuration commands. To determine the master node, issue the following command from each
node in the cluster:
# vxdctl -c mode
One node will identify itself as the master. Create disk groups from this node.
Converting Disks from LVM to CVM
You can use the vxvmconvert utility to convert LVM volume groups into CVM disk groups. Before
you can do this, the volume group must be deactivatedany package that uses the volume group
must be halted. This procedure is described in the latest edition of the Managing Serviceguard
user guide, Appendix G.
Initializing Disks for CVM
You need to initialize the physical disks that will be employed in CVM disk groups. If a physical
disk has been previously used with LVM, you should use the pvremove command to delete the
LVM header data from all the disks in the volume group (this is not necessary if you have not
previously used the disk with LVM).
To initialize a disk for CVM, log on to the master node, then use the vxdiskadm program to
initialize multiple disks, or use the vxdisksetup command to initialize one disk at a time, as in
the following example:
# /usr/lib/vxvm/bin/vxdisksetup -i /dev/dsk/c0t3d2
Creating Disk Groups for RAC
Use the vxdg command to create disk groups. Use the -s option to specify shared mode, as in the
following example:
# vxdg -s init ops_dg c0t3d2
Verify the configuration with the following command:
# vxdg list
NAME
rootdg
ops_dg
56
STATE
enabled
enabled,shared
ID
971995699.1025.node1
972078742.1084.node2
Creating Volumes
Use the vxassist command to create logical volumes. The following is an example:
# vxassist -g ops_dg make log_files 1024m
This command creates a 1024MB volume named log_files in a disk group named ops_dg.
The volume can be referenced with the block device file /dev/vx/dsk/ops_dg/log_files
or the raw (character) device file /dev/vx/rdsk/ops_dg/log_files.
Verify the configuration with the following command:
# vxdg list
IMPORTANT: After creating these files, use the vxedit command to change the ownership of
the raw volume files to oracle and the group membership to dba, and to change the permissions
to 660. Example:
# cd /dev/vx/rdsk/ops_dg
# vxedit -g ops_dg set user=oracle *
# vxedit -g ops_dg set group=dba *
# vxedit -g ops_dg set mode=660 *
Mirror Detachment Policies with CVM

The required CVM disk mirror detachment policy is globalas soon as one node cannot see a
specific mirror copy (plex), all nodes cannot see it as well. The alternate policy is localif one
node cannot see a specific mirror copy, then CVM will deactivate access to the volume for that
node only.
This policy can be re-set on a disk group basis by using the vxedit command, as follows:
# vxedit set diskdetpolicy=global <DiskGroupName>
NOTE: The specific commands for creating mirrored and multipath storage using CVM are
described in the HP-UX documentation for the Veritas Volume Manager.
Oracle Demo Database Files

The following set of volumes is required for the Oracle demo database which you can create during
the installation process.
Table 2 Required Oracle File Names for Demo Database
Volume Name
Size (MB) Raw Device File Name
Oracle File Size

(MB)
opsctl1.ctl
118
/dev/vx/rdsk/ops_dg/opsctl1.ctl
110
opsctl2.ctl
118
110
opsctl3.ctl
118
110
ops1log1.log
128
/dev/vx/rdsk/ops_dg/ops1log1.log
120
ops1log2.log
128
120
ops1log3.log
128
120
ops2log1.log
128
120
ops2log2.log
128
120
Creating Volumes
57
Table 2 Required Oracle File Names for Demo Database (continued)

Volume Name
Size (MB) Raw Device File Name
Oracle File Size

(MB)
ops2log3.log
128
120
opssystem.dbf
508
/dev/vx/rdsk/ops_dg/opssystem.dbf
500
opssysaux.dbf
808
/dev/vx/rdsk/ops_dg/opssysaux.dbf
800
opstemp.dbf
258
/dev/vx/rdsk/ops_dg/opstemp.dbf
250
opsusers.dbf
128
/dev/vx/rdsk/ops_dg/opsusers.dbf
120
opsdata1.dbf
208
/dev/vx/rdsk/ops_dg/opsdata1.dbf
200
opsdata2.dbf
208
200
opsdata3.dbf
208
200
opsspfile1.ora
508
/dev/vx/rdsk/ops_dg/opsspfile1.ora
500
opspwdfile.ora
508
/dev/vx/rdsk/ops_dg/opspwdfile.ora
500
opsundotbs1.dbf
508
/dev/vx/rdsk/ops_dg/opsundotbs1.dbf
500
opsundotbs2.dbf
508
/dev/vx/rdsk/ops_dg/opsundotbs2.dbf
500
opsexmple1.dbf
168
/dev/vx/rdsk/ops_dg/opsexample1.dbf
160
Create these files if you wish to build the demo database. The three logical volumes at the bottom
of the table are included as additional data files that you can create as needed, supplying the
appropriate sizes. If your naming conventions require, you can include the Oracle SID and/or the
database name to distinguish files for different instances and different databases. If you are using
the ORACLE_BASE directory structure, create symbolic links to the ORACLE_BASE files from the
appropriate directory.
Example:
# ln -s /dev/vx/rdsk/ops_dg/opsctl1.ctl \
/u01/ORACLE/db001/ctrl01_1.ctl
Example:
1. Create an ASCII file, and define the path for each database object.
control1=/u01/ORACLE/db001/ctrl01_1.ctl
2.
Set the following environment variable where filename is the name of the ASCII file created.
# export DBCA_RAW_CONFIG=<full path>/filename
Adding Disk Groups to the Cluster Configuration

For CVM 5.x or later, if the multi-node package was configured for disk group activation, the
application package should be configured with package dependency to ensure the CVM disk
group is active.
For CVM 5.x or later (without using multi-node package) after creating units of CVM storage, you
need to specify the disk groups in each package configuration ASCII file. Use one STORAGE_GROUP
parameter for each CVM disk group the package will use. You also need to identify the CVM disk
groups, file systems, logical volumes, and mount options in the package control script (on HP-UX
releases that support them). See About Veritas CFS and CVM from Symantec (page 15).
For more detailed information on the package configuration process, refer to the Managing
Serviceguard Eighteenth Edition users guide.
58
Prerequisites for Oracle 10g, 11gR1, or 11gR2 (Sample Installation)

The following sample steps prepare an SGeRAC cluster for Oracle 10g, 11gR1, or 11gR2. Refer
to the Oracle documentation for Oracle installation details.
1. Create inventory groups on each node.
Create the Oracle inventory group if one does not exist, create the OSDBA group, and create
the Operator Group (optional).
# groupadd oinstall
# groupadd dba
# groupadd oper
2.
Create Oracle user on each node.

# useradd -u 203 -g oinstall -G dba,oper oracle
3.
Change the password on each node.

# passwd oracle
4.
Create symbolic links.

Required if Motif 2.1 Development Environment Package is not installed.
# ln -s /usr/lib/libX11.3 /usr/lib/libX11.sl
#ln -s /usr/lib/libXIE.2 /usr/lib/libXIE.sl
# ln -s /usr/lib/libXext.3 /usr/lib/libXext.sl
# ln -s /usr/lib/libXhp11.3 /usr/lib/Xhp11.sl
# ln -s /usr/lib/libXi.3 /usr/lib/libXi.sl
# ln -s /usr/lib/libXm.4 /usr/lib/libXm.sl
# ln -s /usr/lib/libXp.2 /usr/lib/libXp.sl
# ln -s /usr/lib/libXt.3 /usr/lib/libXt.sl
# ln -s /usr/lib/libXtst.2 /usr/lib/libXtst.sl
5.
6.
Enable remote access (ssh) for Oracle user on all nodes.

Create file system for Oracle directories.
In the following samples, /mnt/app is a mounted file system for Oracle software. Assume
there is a private disk c4t5d0 at 18 GB size on all nodes. Create the local file system on
each node.
# umask 022
# pvcreate /dev/rdsk/c4t5d0
# mkdir /dev/vg01
# mknod /dev/vg01/group c 64 0x010000
# vgcreate /dev/vg01 /dev/dsk/c4t5d0
# lvcreate -L 16000 /dev/vg01
# newfs -F vxfs /dev/vg01/rlvol1
# mkdir -p /mnt/app
# mount /dev/vg01/lvol1 /mnt/app
# chmod 775 /mnt/app
7.
Create Oracle cluster software home directory.

For installing Oracle cluster software on local file system, create the directories on each node.
# mkdir -p /mnt/app/crs/oracle/product/10.2.0/crs
59
# chown -R oracle:oinstall /mnt/app/crs/oracle/product/10.2.0/crs

# chmod -R 775 /mnt/app/crs/oracle/product/10.2.0/crs
8.
Create Oracle base directory (for RAC binaries on local file system).
If installing RAC binaries on local file system, create the oracle base directory on each node.
# mkdir -p /mnt/app/oracle
# chown -R oracle:oinstall /mnt/app/oracle
# chmod -R 775 /mnt/app/oracle
# usermod -d /mnt/app/oracle oracle
9.
Create Oracle base directory (for RAC binaries on cluster file system).
If installing RAC binaries on Cluster File System, create the oracle base directory once, because
this is a CFS directory visible by all nodes. The CFS file system used is /cfs/mnt1.
# mkdir -p /cfs/mnt1/oracle
# chown -R oracle:oinstall /cfs/mnt1/oracle
# chmod -R 775 /cfs/mnt1/oracle
# chmod 775 /cfs
# chmod 775 /cfs/mnt1
Modify oracle user to use new home directory on each node.
# usermod -d /cfs/mnt1/oracle oracle
10. Prepare shared storage on SLVM.

This section assumes the OCR, vote device, and database files are created on SLVM volume
group vg_rac.
NOTE: Step 10 is only applicable up to Oracle 11gR1. If you prefer to use SLVM storage
for Oracle 11gR2, see Oracle Database 11g Release 2 Real Application Clusters with
SLVM/RAW on HP-UX Installation Cookbook http://h18006.www1.hp.com/storage/pdfs/
4AA2-7668ENW.pdf.
a.
Change permission of Shared Logical Volume group.

This section assumes the OCR, vote device, and database files are created on SLVM
volume group vg_rac.
# chmod 755 /dev/vg_rac
b.
Change permission and ownership of Oracle cluster software vote device and database
files.
# chown oracle:oinstall /dev/vg_rac/r*
# chmod 660 /dev/vg_rac/r*
c.
Change Permission of OCR device.

# chown root:oinstall /dev/vg_rac/rora_ocr
# chmod 640 /dev/vg_rac/rora_ocr
d.
Create raw device mapping file for Oracle Database Configuration Assistant.
In this example, the database name is ver10
# ORACLE_BASE=/mnt/app/oracle; export ORACLE_BASE
# mkdir -p $ORACLE_BASE/oradata/ver10
# chown -R oracle:oinstall $ORACLE_BASE/oradata
# chmod -R 755 $ORACLE_BASE/oradata
60
The following is a sample of the mapping file for DBCA:

system=/dev/vg_rac/ropssystem.dbf
sysaux=/dev/vg_rac/ropssysaux.dbf
undotbs1=/dev/vg_rac/ropsundotbs01.dbf
undotbs2=/dev/vg_rac/ropsundotbs02.dbf
example=/dev/vg_rac/ropsexample1.dbf
users=/dev/vg_rac/ropsusers.dbf
redo1_1=/dev/vg_rac/rops1log1.log
control1=/dev/vg_rac/ropsctl1.ctl
temp=/dev/vg_rac/ropstmp.dbf
spfile=/dev/vg_rac/ropsspfile1.ora
In this sample, create the DBCA mapping file and place at:
/mnt/app/oracle/oradata/ver10/ver10_raw.conf.
11. Prepare shared storage on CFS.
This section assumes the OCR, Vote device, and database files are created on CFS directories.
The OCR and vote device reside on /cfs/mnt3 and the demo database files reside on
/cfs/mnt2.
a. Create OCR and vote device directories on CFS.
Create OCR and vote device directories on Cluster File System. Run commands only on
one node.
# chmod 775 /cfs
# cd /cfs/mnt3
# mkdir OCR
# chmod 755 OCR
# mkdir VOTE
# chmod 755 VOTE
# chown -R oracle:oinstall /cfs/mnt3
b.
Create directory for Oracle demo database on CFS.

Create the CFS directory to store Oracle database files. Run commands only on one
node.
# chmod 775 /cfs
# cd /cfs/mnt2
# mkdir oradata
# chown oracle:oinstall oradata
# chmod 775 oradata
61
NOTE: The volume groups are supported with Serviceguard. The steps shown in the
following section are for configuring the volume groups in Serviceguard clusters LVM
version 1.0.
For more information on using and configuring LVM version 2.x, see the HP-UX System
Administrator's Guide: Logical Volume Management located at www.hp.com/go/
hpux-core-docs > HP-UX 11i v3.
Installing Oracle 10g, 11gR1, or 11gR2 Cluster Software

The following sample steps for an SGeRAC cluster for Oracle 10g, 11gR1, or 11gR2. Refer to the
Oracle documentation for Oracle installation details.
Installing on Local File System

Logon as a oracle user:
$ export DISPLAY={display}:0.0
$ cd <10g/11gR1/11gR2 Cluster Software disk directory>
$ ./runInstaller
Use following guidelines when installing on a local file system:
1. Specify CRS HOME as /mnt/app/crs/oracle/product/<oracle version>/crs.
This is a local file system.
2. If you are using SLVM for OCR, specify OCR Location as /dev/vg_rac/rora_ocr.
If you are using CFS for OCR, specify the OCR Location as /cfs/mnt3/OCR/ocr_file.
3.
If you are using SLVM for the vote device, specify the Vote Disk Location as /dev/vg_rac/
rora_vote.
If you are using CFS for the vote device, specify the Vote Disk Location as /cfs/mnt3/VOTE/
vote_file.
NOTE: During Oracle 10g/11gR1/11gR2 cluster configuration, Oracle gives the default
cluster name crs. This default name can be changed, using the combination of the following
characters: a-z, A-Z, 0-9, _, $and #.
4.
5.
When prompted, run orainstRoot.sh on each node.

When prompted, run root.sh on each node.
Installing Oracle 10g/11gR1/11gR2 RAC Binaries

Refer to the Oracle documentation for Oracle installation details.
Installing RAC Binaries on a Local File System

$ export ORACLE BASE=/mnt/app/oracle
$ cd <10g/11g RAC installation disk directory>
$ ./runInstaller
1. In this example, the path to ORACLE_HOME is on a local file system /mnt/app/oracle/
product/<oracle version>/db_1.
2. Select installation for database software only.
3. When prompted, run root.sh on each node.
62
Installing RAC Binaries on Cluster File System

$ export ORACLE BASE=/cfs/mnt1/oracle
$ cd <10g/11g RAC installation disk directory>
$ ./runInstaller
1. In this example, the path to ORACLE_HOME is located on a CFS directory /cfs/mnt1/
oracle/product/<oracle version>/db_1.
2. Select installation for database software only.
3. When prompted, run root.sh on each node.
Creating a RAC Demo Database

This section demonstrates the steps for creating a demo database with datafiles on raw volumes
with SLVM or CVM, or with Cluster File System (on HP-UX releases that support them. See About
Veritas CFS and CVM from Symantec (page 15)).
Before setting up Listeners using netca, verify that listeners are already configured as part of
Clusterware installation or not. If listeners are configured, then there is no need of configuring them
again unless there is a specific requirement.
Creating a RAC Demo Database on SLVM or CVM

Export environment variables for oracle user:
export ORACLE_BASE=/mnt/app/oracle
export DBCA_RAW_CONFIG=/mnt/app/oracle/oradata/ver10/ver10_raw.conf
export ORACLE_HOME=$ORACLE_BASE/product/<oracle version>/db_1
export ORA_CRS_HOME=/mnt/app/crs/oracle/product/<oracle version>/crs
LD_LIBRARY_PATH=$ORACLE_HOME/lib:/lib:/usr/lib:$ORACLE_HOME/rdbms/lib
SHLIB_PATH=$ORACLE_HOME/lib32:$ORACLE_HOME/rdbms/lib32
export LD_LIBRARY_PATH SHLIB_PATH
export PATH=$PATH:$ORACLE_HOME/bin:$ORA_CRS_HOME/bin:/usr/local/bin:
CLASSPATH=$ORACLE_HOME/jre:$ORACLE_HOME/jlib:$ORACLE_HOME/rdbms/jlib:$ORACLE_HOME/network/jlib
export CLASSPATH
export DISPLAY={display}:0.0
NOTE: The following procedure is only applicable up to Oracle 11gR1. If you prefer to use
SLVM storage for Oracle 11gR2, see Oracle Database 11g Release 2 Real Application Clusters
with SLVM/RAW on HP-UX Installation Cookbook http://h18006.www1.hp.com/storage/pdfs/
4AA2-7668ENW.pdf.
1.
Set up listeners with Oracle network configuration assistant (If the listeners are not configured.).
Use the Oracle network configuration assistant to configure the listeners with the following
command:
$ netca
2.
Create database with Oracle database configuration assistant.

Use the Oracle database configuration assistant to create the database with the following
command:
$ dbca
a. In this sample, the database name and SID prefix are ver10.
b. Select the storage option for raw devices.
Creating a RAC Demo Database
63
Creating a RAC Demo Database on CFS

Export environment variables for oracle user:
export ORACLE_BASE=/cfs/mnt1/oracle
export ORACLE_HOME=$ORACLE_BASE/product/<oracle version>/db_1
export ORA_CRS_HOME=/mnt/app/crs/oracle/product/<oracle version>/crs
LD_LIBRARY_PATH=$ORACLE_HOME/lib:/lib:/usr/lib:$ORACLE_HOME/rdbms/lib
SHLIB_PATH=$ORACLE_HOME/lib32:$ORACLE_HOME/rdbms/lib32
export LD_LIBRARY_PATH SHLIB_PATH
export \ PATH=$PATH:$ORACLE_HOME/bin:$ORA_CRS_HOME/bin:/usr/local/bin:
CLASSPATH=$ORACLE_HOME/jre:$ORACLE_HOME/jlib:$ORACLE_HOME/rdbms/jlib:$ORACLE_HOME/network/jlib
export CLASSPATH
export DISPLAY={display}:0.0
1.
Set up listeners with Oracle network configuration assistant (if the listeners are not configured).
Use the Oracle network configuration assistant to configure the listeners with the following
command:
$ netca
2.
Create database with database configuration assistant.

Use the Oracle database configuration assistant to create the database with the following
command:
$ dbca
Use
a.
b.
c.
following guidelines when installing on a local file system:

In this sample, the database name and SID prefix are ver10.
Select the storage option for cluster file system.
Enter /cfs/mnt2/oradata as the common directory.
Verifying Oracle Disk Manager is Configured

NOTE:
1.
The following steps are specific to CFS 4.1 or later.
Check the license for CFS 4.1 or later.

#/opt/VRTS/bin/vxlictest -n "VERITAS Storage Foundation for Oracle"
-f "ODM"
output: ODM feature is licensed
2.
Check that the VRTSodm package is installed:

#swlist VRTSodm
output for CFS 4.1: VRTSodm 4.1m VERITAS Oracle Disk Manager
VRTSodm.ODM-KRN 4.1m VERITAS ODM kernel files VRTSodm.ODM-MAN 4.1m
VERITAS ODM manual pages VRTSodm.ODM-RUN 4.1m VERITAS ODM commands
output for CFS 5.0: VRTSodm 5.0.01.01 Veritas Oracle Disk Manager
VRTSodm.ODM-KRN 5.0.01.01 Veritas ODM kernel files VRTSodm.ODM-MAN
5.0.01.01 Veritas ODM manual pages VRTSodm.ODM-RUN 5.0.01.01 Veritas
ODM commandsVRTSodm 5.0.31.5 Veritas Oracle Disk Manager
VRTSodm.ODM-KRN 5.0.31.5 Veritas ODM kernel files VRTSodm.ODM-MAN
5.0.31.5 Veritas ODM manual pages VRTSodm.ODM-RUN 5.0.31.5 Veritas
ODM commands
3.
64
Check that libodm.sl is present
IMPORTANT: Beginning with HP Serviceguard Storage Management Suite (SMS) Version

A.04.00.00, /opt/VRTSodm/lib/libodm.sl is changed to /opt/VRTSodm/lib/
libodm.so on itanium(IA) architecture.
If you are using itanium architecture and HP Serviceguard SMS Version A.04.00.00 or later,
then you must use /opt/VRTSodm/lib/libodm.so file instead of /opt/VRTSodm/lib/
libodm.sl file.
#ll -L /opt/VRTSodm/lib/libodm.sl
output: -r-xr-xr-x 1 root sys 94872 Aug 25 2009 /opt/VRTSodm/lib/libodm.sl
Configuring Oracle to Use Oracle Disk Manager Library

NOTE:
1.
2.
3.
Login as a Oracle user.

Shutdown database.
Link the Oracle Disk Manager library into Oracle home.
For Oracle 10g on HP 9000 Systems:
$ rm ${ORACLE_HOME}/lib/libodm10.sl
$ ln -s /opt/VRTSodm/lib/libodm.sl ${ORACLE_HOME}/lib/libodm10.sl
For Oracle 10g on Integrity Systems:
$ rm ${ORACLE_HOME}/lib/libodm10.so
$ ln -s /opt/VRTSodm/lib/libodm.sl ${ORACLE_HOME}/lib/libodm10.so
$ rm ${ORACLE_HOME}/lib/libodm11.sl
$ ln -s /opt/VRTSodm/lib/libodm.sl ${ORACLE_HOME}/lib/libodm11.sl
$ rm ${ORACLE_HOME}/lib/libodm11.so
$ ln -s /opt/VRTSodm/lib/libodm.sl ${ORACLE_HOME}/lib/libodm11.so
4.
Start Oracle database.
Verify that Oracle Disk Manager is Running

NOTE:
1.
2.
Start the cluster and Oracle database (if not already started).
Check that the Oracle instance is using the Oracle Disk Manager function:
# cat /dev/odm/stats
abort:
cancel:
commit:
create:
delete:
identify:
io:
reidentify:
resize:
unidentify:
mname:
vxctl:
vxvers:
0
0
18
18
0
349
12350590
78
0
203
0
0
10
Configuring Oracle to Use Oracle Disk Manager Library
65
io req:
io calls:
comp req:
comp calls:
io mor cmp:
io zro cmp:
cl receive:
cl ident:
cl reserve:
cl delete:
cl resize:
cl same op:
cl opt idn:
cl opt rsv:
**********:
3.
9102431
6911030
73480659
5439560
461063
2330
66145
18
8
1
0
0
0
332
17
Verify that the Oracle disk manager is loaded:

# kcmodule -P state odm
Output: state loaded
4.
In the alert log, verify the Oracle instance is running. The log should contain output similar to
the following:
For CFS 4.1:
Oracle instance running with ODM: VERITAS 4.1 ODM Library, Version
1.1
For CFS 5.0.1:
Oracle instance running with ODM: VERITAS 5.1 ODM Library, Version
1.0
For CFS 5.1 SP1:
Oracle instance running with ODM: Veritas 5.1 ODM Library, Version
2.0
Configuring Oracle to Stop Using Oracle Disk Manager Library

NOTE:
1.
2.
3.
Login as a Oracle user.

Shutdown database.
Change directories:
$ cd ${ORACLE_HOME}/lib
4.
Remove the file linked to the ODM library.

$ rm libodm10.sl
$ ln -s ${ORACLE_HOME}/lib/libodmd10.sl ${ORACLE_HOME}/lib/libodm10.sl
$ rm libodm10.so
$ ln -s ${ORACLE_HOME}/lib/libodmd10.so ${ORACLE_HOME}/lib/libodm10.so
$ rm libodm11.sl
$ ln -s ${ORACLE_HOME}/lib/libodmd11.sl ${ORACLE_HOME}/lib/libodm11.sl
66
$ rm libodm11.so
$ ln -s ${ORACLE_HOME}/lib/libodmd11.so ${ORACLE_HOME}/lib/libodm11.so
5.
Restart the database.
Using Serviceguard Packages to Synchronize with Oracle

10g/11gR1/11gR2 RAC
It is recommended to start and stop Oracle Cluster Software in a Serviceguard packagethe
Oracle Cluster Software will start after SGeRAC is started, and will stop before SGeRAC is halted.
Serviceguard packages should also be used to synchronize storage activation and deactivation
with Oracle Cluster Software and RAC instances.
Preparing Oracle Cluster Software for Serviceguard Packages

CRS starts all RAC instances by default. You must disable CRS from starting up RAC instances
when you configure your RAC instances in SGeRAC packages.
On each node of the cluster, disable the automatic startup of the Oracle Clusterware at boot
time.
Login as root and enter:
: $ORA_CRS_HOME/bin/crsctl disable crs
(Check CRS logs or check for Oracle processes, ps -ef | grep ocssd.bin)
On one node of the cluster, disable the Oracle RAC database and instances from being started
automatically by the Oracle Clusterware.
Login as the Oracle administrator and run the following command to set the database
management policy to manual.
For Oracle 10g:
: $ORACLE_HOME/bin/srvctl modify database -d <dbname> -y manual
For Oracle 11g:
: $ORACLE_HOME/bin/srvctl modify database -d <dbname> -y MANUAL
Configure Serviceguard Packages

A Serviceguard package is needed for each node to start and stop Oracle Cluster Software.
Storage activation (SLVM).

When the Oracle Cluster Software required storage is configured on SLVM volume groups,
the Serviceguard package should be configured to activate and deactivate the required storage
in the package control script.
As an example, modify the control script to activate the volume group in shared mode and
set VG in the package control script for SLVM volume groups.
VG[0]=vg_rac
Storage Activation (CVM)

When the Oracle Cluster Software required storage is configured on CVM disk groups, the
Serviceguard package should be configured to activate and deactivate the required storage
in the package configuration file and control script.
In the package configuration file, specify the disk group with the STORAGE_GROUP key word.
In the package controls script, specify the disk group with the CVM_DG variable.
As an example, in the package configuration file should have the following:
STORAGE_GROUP ops_dg
Using Serviceguard Packages to Synchronize with Oracle 10g/11gR1/11gR2 RAC
67
Modify the package control script to set the CVM disk group to activate for shared write
and to specify the disk group.
CVM_DG[0]=ops_dg
Storage Activation (CFS)

When the Oracle Cluster Software required storage is configured on a Cluster File System
(CFS), the Serviceguard package should be configured to depend on the CFS multi-node
package through package dependency. With package dependency, the Serviceguard package
that starts Oracle Cluster Software will not run until its dependent CFS multi-node package is
up and will halt before the CFS multi-node package is halted.
Setup dependency conditions in Serviceguard package configuration file (SAMPLE).
DEPENDENCY_NAME
DEPENDENCY_LOCATION
mp1
SG-CFS-MP-1=UP
SAME_NODE
DEPENDENCY_NAME
DEPENDENCY_LOCATION
mp2
SG-CFS-MP-2=UP
SAME_NODE
DEPENDENCY_NAME
DEPENDENCY_LOCATION
mp3
SG-CFS-MP-3=UP
SAME_NODE
Start and stop Oracle Cluster Software.

In the Serviceguard package control script, configure the Oracle Cluster Software start in the
customer_defined_run_cmds function.
For 10g 10.1.0.4 or later:
/sbin/init.d/init.crs start
<CRS HOME>/bin/crsctl start crs
In the Serviceguard package control script, configure the Oracle Cluster Software stop in the
customer_defined_halt_cmds function.
/sbin/init.d/init.crs stop
<CRS HOME>/bin/crsctl stop crs
When stopping Oracle Cluster Software in a Serviceguard package, it may be necessary to
verify that the Oracle processes have stopped and exited before deactivating storage or
halting CFS multi-node package. The verification can be done with a script that loops and
checks for the successful stop message in the Oracle Cluster Software logs or the existences
of Oracle processes that needed to be stopped, specifically the CSS daemon (ocssd.bin).
For example, this script could be called by the Serviceguard package control script after the
command to halt Oracle Cluster Software and before storage deactivation.
68
3 Support of Oracle RAC ASM with SGeRAC

Introduction
This chapter discusses the use of the Oracle 10g Release 2 (10g R2) and11g Release 1 (11g R1)
database server feature called Automatic Storage Management (ASM) in configurations of HP
Serviceguard Extension for Real Application Clusters (SGeRAC).
We begin with a brief review of ASMfunctionality, pros, cons, and method of operation. Then,
we look in detail at how we configure ASM with SGeRAC (version A.11.17 or later is required).
SG/SGeRAC Support for ASM on HP-UX 11i v2

Overview
Support for ASM with SG/SGeRAC commences with SGeRAC A.11.17. However, please note
that it has been possible to set up Oracle databases using ASM on HP-UX from the time Oracle
10g became available on HP-UX, using the following configurations that do not require
SG/SGeRAC:
Non-clustered single instance Oracle
Oracle single instance and RAC databases running in a pure Oracle clusterware environment
The following requirements/restrictions apply to SG/SGeRAC (A.11.17 or later) support of ASM

(and are summarized in Table 3):
Table 3 ASM support with SG/SGeRAC A.11.17 or later
Supported
Not Supported
Oracle RAC database
Oracle single instance database, with or without Oracle

clusterware
SLVM Logical Volumes as ASM Disk Group Members
Raw disks/LUs or CVM Logical Volumes as ASM Disk

Group Members
ASM configurations on SGeRAC using SGeRAC A.11.17

or later, and Oracle 10g R2 RAC
ASM configurations on SGeRAC using SGeRAC 11.16,

Oracle 10g R1 RAC or HP-UX 11i v1
ASM configurations on SGeRAC using SGeRAC A.11.17

or later, and Oracle 11g R1
Single instance Oracle database with ASM managed

storage running in Serviceguard cluster
Maximum SGeRAC cluster size for ASM configurations is

16
Single instance Oracle database with ASM-managed

storage in SGeRAC configuration wherein SGeRAC
provides node membership to Oracle Clusterware
SGeRAC Toolkit framework based on Multi-Node Package

(MNP) /Simple Package Dependency to manage ASM
configurations on SGeRAC up to a maximum cluster size
of 16
Why ASM over SLVM?

As mentioned above, we require ASM disk group members in SGeRAC A.11.17 (or later)
configurations to be raw logical volumes managed by SLVM.
The main reason for this requirement is to expedite ASM support with SGeRAC. We leverage
existing HP-UX capabilities to provide multipathing for SLVM logical volumes, using either the PV
Links feature, or separate products such as HP StorageWorks Secure Path that provide multipathing
Introduction
69
for specific types of disk arrays. Other advantages of the "ASM-over-SLVM" configuration are as
follows:
ASM-over-SLVM ensures that the HP-UX devices used for disk group members will have the
same names (the names of logical volumes in SLVM volume groups) on all nodes, easing ASM
configuration.
ASM-over-SLVM protects ASM data against inadvertent overwrites from nodes inside/outside
the cluster. If the ASM disk group members are raw disks, there is no protection currently
preventing these disks from being incorporated into LVM or VxVM volume/disk groups.
The disadvantages of the ASM-over-SLVM configuration are as follows:
Additional configuration and management tasks are imposed by the extra layer of volume
management (administration of volume groups, logical volumes, physical volumes).
There is a small performance impact from the extra layer of volume management.
SLVM has some restrictions in the area of online reconfiguration, the impact of which will be
examined later in this chapter.
Configuring SLVM Volume Groups for ASM Disk Groups

Ideally, ASM disk group members should be raw disks or array logical units, but due to the lack
of built-in multipathing we require them to be SLVM raw logical volumes in SGeRAC configurations.
But we would still like these logical volumes presented to ASM to resemble raw disks, as far as
possible.
Hence, each SLVM logical volume (LV) used as a member of an ASM disk group is required to be
laid out to occupy the usable space, in contiguous fashion, of exactly one single physical volume
(PV). This implies the following about LV:
Contiguous
Not striped or mirrored
Does not span multiple PVs
Does not share a PV with other LVs
The idea is that ASM provides the mirroring, striping, slicing, and dicing functionality as needed
and SLVM supplies the multipathing functionality not provided by ASM. Figure 13 indicates this
1-1 mapping between SLVM PVs and LVs used as ASM disk group members.
Further, the default retry behavior of SLVM could result in an I/O operation on an SLVM LV taking
an indefinitely long period of time. This behavior could impede ASM retry and rebalance
capabilities; hence, a finite timeout must be configured for each SLVM LV. For example, the timeout
could be configured to the value (total number of physical paths to the PV * PV timeout), providing
enough time for SLVM to try all available paths, if needed.4
The PVs used in an ASM disk group can be organized into SLVM volume groups as desired by
the customer. In the example shown in Figure 13, for each ASM disk group, the PVs corresponding
to its members are organized into a separate SLVM volume group.
70
Support of Oracle RAC ASM with SGeRAC
Figure 10 1-1 mapping between SLVM logical and physical volumes for ASM configuration
If the LVM patch PHKL_36745 (or equivalent) is installed in the cluster, a timeout equal to (2* PV
timeout) will suffice to try all paths.
The SLVM volume groups are marked as shared volume groups and exported across the SGeRAC
cluster using standard SGeRAC procedures. As noted above, multiple physical paths to each
physical volume should be configured using the LVM PV Links feature or a separate multipathing
product such as HP StorageWorks Secure Path.
Please note that, for the case in which the SLVM PVs being used by ASM are disk array LUs, the
requirements in this section do not place any constraints on the configuration of the LUs. The LUs
may be configured with striping, mirroring and other characteristics at the array level, following
guidelines provided by Oracle and the array provider for use by ASM.
Sample Command Sequence for Configuring SLVM Volume Groups

This section includes an example of a command sequence that can be used to prepare SLVM
Logical Volumes for use by ASM to meet the requirements specified above.
The scenario for our example is that we are preparing a new volume group named vgora_asm
with two PVs, each with two physical paths. The physical paths for the first PV are /dev/dsk/
c9t0d1 and /dev/dsk/c10t0d1 and those for the second PV are /dev/dsk/c9t0d2 and
/dev/dsk/c10t0d2.
1. Create the volume group with the two PVs, incorporating the two physical paths for each
(choosing hh to be the next hexadecimal number that is available on the system, after the
volume groups that are already configured) :
#
#
#
#
#
#
pvcreate -f /dev/dsk/c9t0d1
pvcreate -f /dev/dsk/c9t0d2
mkdir /dev/vgora_asm
mknod /dev/vgora_asm/group c 64 0xhh0000
vgcreate /dev/vgora_asm /dev/dsk/c9t0d1
vgextend /dev/vgora_asm /dev/dsk/c9t0d2
71
# vgextend /dev/vgora_asm /dev/dsk/c10t0d1

# vgextend /dev/vgora_asm /dev/dsk/c10t0d2
2.
For each of the two PVs, create a corresponding LV.
Create an LV of zero length.
Mark the LV as contiguous.
Extend each LV to the maximum size possible on that PV (the number of extents available
in a PV can be determined via vgdisplay -v <vgname>).
Configure LV timeouts, based on the PV timeout and number of physical paths, as

described in the previous section. If a PV timeout has been explicitly set, its value can be
displayed via pvdisplay -v. If not, pvdisplay will show a value of default, indicating
that the timeout is determined by the underlying disk driver. For SCSI, in HP-UX 11i v2,
the default timeout is 30 seconds.
Null out the initial part of each LV to ensure ASM accepts the LV as an ASM disk group
member.5 Note that we are zeroing out the LV data area, not its metadata. It is the ASM
metadata that is being cleared.
See Oracle Metalink Doc ID: Note:268481.1 RE-CREATING ASM INSTANCES AND
DISKGROUPS at https://metalink.oracle.com/ (Oracle MetaLink account required)
#
#
#
#
#
#
#
#
#
#
#
#
#
3.
lvcreate -n lvol1 vgora_asm

lvchange -C y /dev/vgora_asm/lvol1
Assume vgdisplay shows each PV has 2900 extents in our example
lvextend -l 2900 /dev/vgora_asm/lvol1 /dev/dsk/c9t0d1
lvextend -l 2900 /dev/vgora_asm/lvol2 /dev/dsk/c9t0d2
Assume a PV timeout of 30 seconds.
There are 2 paths to each PV, so the LV timeout value is 60 seconds
lvchange -t 60 /dev/vgora_asm/lvol1
dd if=/dev/zero of=/dev/vgora_asm/rlvol1 bs=8192 count=12800
Export the volume group across the SGeRAC cluster and mark it as shared, as specified by
SGeRAC documentation. Assign the right set of ownerships and access rights to the raw logical
volumes on each node as required by Oracle (oracle:dba and 0660, respectively).
We can now use the raw logical volume device names as disk group members when configuring
ASM disk groups using the Oracle database management utilities. There are a number of ways
of doing this described in Oracle ASM documentation, including the dbca database creation
wizard as well as sqlplus.
The same command sequence, with some modifications, can be used for adding new disks to an
already existing volume group that is being used by ASM to store one or more RAC databases.
If the database(s) should be up and running during the operation, we use the Single Node Online
volume Reconfiguration (SNOR) feature of SLVM.
Step 1 of the above sequence is modified as follows:
72
First, deactivate the volume group vgora_asm on all nodes but one, say node A. This requires
prior shutdown of the database(s) using ASM-managed storage and ASM itself, on all nodes
but node A. See the section ASM Halt is needed to ensure disconnect of ASM from SLVM
Volume Groups to understand why it is not adequate to shut down only the database(s) using
the volume group to be reconfigured, and why we must shut down ASM itself and therefore
all database(s) using ASM-managed storage, on all nodes but node A.
Next, on node A, switch the volume group to exclusive mode, using SNOR.
Initialize the disks to be added with pvcreate, and then extend the volume group with
vgextend.
Step 2 remains the same. Logical volumes are prepared for the new disks in the same way.
In step 3, switch the volume group back to shared mode, using SNOR, and export the VG across
the cluster, ensuring that the right ownership and access rights are assigned to the raw logical
volumes. Activate the volume group, and restart ASM and the database(s) using ASM-managed
storage on all nodes (they are already active on node A).

Overview
Support for ASM with SG/SGeRAC commences with SGeRAC A.11.17.01 on HP-UX 11i v3.
SGeRAC support ASM where the member disk groups are either logical volumes managed by HP
Shared Logical Volume Manager (SLVM) or raw disks/disk array logical units(LUs).
A new I/O infrastructure that enables the native built-in multipathing functionality is introduced in
HPUX 11i v3. This feature offers users a continuous I/O access to a LUN or disk if any of the paths
fails. This feature is enabled in the operating system by default. In addition, new DSF(device special
file) format is introduced in this operating system. An example of new DSF is /dev/disk/disk1,
compare to the legacy DSF, /dev/rdsk/cxtydz.
Please note that it has been possible to set up Oracle databases using ASM on HP-UX from the
time Oracle 10g became available on HP-UX, using the following configurations that do not require
SG/SGeRAC:
Non-clustered single instance Oracle.
Oracle single instance and RAC databases running in a pure Oracle clusterware environment.
The following requirements/restrictions apply to SG/SGeRAC (A.11.17.01 or later) support
of ASM (summarized in Table 4):
Table 4 ASM support with SG/SGeRAC A.11.17.01 or later

Supported
Not Supported
Oracle RAC database
Oracle single instance database, with or without Oracle

clusterware
SLVM Logical Volumes as ASM Disk Group Members
CVM Logical Volumes as ASM Disk Group Members
ASM configurations on SGeRAC using SGeRAC

A.11.17.01 or later, and Oracle 10g R2 RAC
Single instance Oracle database with ASM managed

storage running in Serviceguard cluster
ASM configurations on SGeRAC using SGeRAC

A.11.17.01 or later, and 11g R1 RAC
Single instance Oracle database with ASM-managed

storage in SGeRAC configuration wherein SGeRAC
provides node membership to Oracle Clusterware
Maximum SGeRAC cluster size for ASM configurations is

16
SGeRAC Toolkit3 framework based on Multi-Node Package
(MNP) /Simple Package Dependency to manage ASM
configurations on SGeRAC up to a maximum cluster size
of 16
Raw disks/disk array LUs as ASM Disk Group Members
ASM over SLVM

SGeRAC support for ASM-over-SLVM continues in HPUX 11i v3. SGeRAC 11.17.01 or later is
required for ASM-over-SLVM configurations on HP-UX 11i v3.
73
The advantages of the "ASM-over-SLVM" configuration are as follows:
ASM-over-SLVM ensures that the HP-UX devices used for disk group members will have the
same names (the names of logical volumes in SLVM volume groups) on all nodes, easing ASM
configuration.
ASM-over-SLVM protects ASM data against inadvertent overwrites from nodes inside/outside
the cluster. If the ASM disk group members are raw disks, there is no protection currently
preventing these disks from being incorporated into VxVM volume/disk groups.
The disadvantages of the ASM-over-SLVM configuration are as follows:
Additional configuration and management tasks are imposed by the extra layer of volume
management (administration of volume groups, logical volumes, physical volumes).
There is a small performance impact from the extra layer of volume management.
SLVM has some restrictions in the area of online reconfiguration, the impact of which will be
examined later in this chapter.
Configuring SLVM Volume Groups for ASM Disk Groups

We would like the logical volumes presented to ASM to resemble raw disks, as far as possible.
Hence, each SLVM logical volume (LV) used as a member of an ASM disk group is required to be
laid out to occupy the usable space, in contiguous fashion, of exactly one single physical volume
(PV). This implies the following about LV:
Contiguous
Not be striped or mirrored
Does not span multiple PVs
Does not share a PV with other LVs
The idea is that ASM provides the mirroring, striping, slicing, and dicing functionality as needed
and SLVM supplies the multipathing functionality not provided by ASM. Figure 14 indicates this
1-1 mapping between SLVM PVs and LVs used as ASM disk group members.
Further, the default retry behavior of SLVM could result in an I/O operation on an SLVM LV taking
an indefinitely long period of time. This behavior could impede ASM retry and rebalance
capabilities; hence a finite timeout must be configured for each SLVM LV. For example, the timeout
could be configured to the value (total number of physical paths to the PV * PV timeout), providing
enough time for SLVM to try all available paths, if needed.
The PVs used in an ASM disk group can be organized into SLVM volume groups as desired by
the customer. In the example shown in Figure 14, for each ASM disk group, the PVs corresponding
to its members are organized into a separate SLVM volume group.
74
Figure 11 1-1 mapping between SLVM logical and physical volumes for ASM configuration
The SLVM volume groups are marked as shared volume groups and exported across the SGeRAC
cluster using standard SGeRAC procedures.
Please note that, for the case in which the SLVM PVs being used by ASM are disk array LUs, the
requirements in this section do not place any constraints on the configuration of the LUs. The LUs
may be configured with striping, mirroring, and other characteristics at the array level, following
guidelines provided by Oracle and the array provider for use by ASM.
Sample Command Sequence for Configuring SLVM Volume Groups

In this section, we provide an example of a command sequence that can be used to prepare SLVM
Logical Volumes for use by ASM. The example below demonstrates how volume group is created
using new DSF format. HP-UX will automatically use the redundant path for the volume group in
the background.
The scenario for our example is that we are preparing a new volume group named vgora_asm.
The physical path for the first PV is /dev/rdisk/disk1 and the second PV is /dev/rdisk/
disk2.
1. Create the volume group with the two PVs (choosing hh to be the next hexadecimal number
that is available on the system, after the volume groups that are already configured):
#
#
#
#
#
#
2.
pvcreate -f /dev/rdisk/disk1
pvcreate -f /dev/rdisk/disk2
mkdir /dev/vgora_asm
mknod /dev/vgora_asm/group c 64 0xhh0000
vgcreate /dev/vgora_asm /dev/disk/disk1
vgextend /dev/vgora_asm /dev/disk/disk2
For each of the two PVs, create a corresponding LV
Create an LV of zero length
Mark the LV as contiguous

75
Extend each LV to the maximum size possible on that PV (the number of extents available
in a PV can be determined via vgdisplay -v <vgname>)
Configure LV timeouts, based on the PV timeout and number of physical paths, as

described in the previous section. If a PV timeout has been explicitly set, its value can be
displayed via pvdisplay -v. If not, pvdisplay will show a value of default, indicating
that the timeout is determined by the underlying disk driver. For SCSI, in HP-UX 11i v3,
the default timeout is 30 seconds.
Null out the initial part of each LV to ensure ASM accepts the LV as an ASM disk group
member. Note that we are zeroing out the LV data area, not its metadata. It is the ASM
metadata that is being cleared.
#
#
#
#
#
#
#
#
#
#
#
#
#
3.

Assume vgdisplay shows each PV has 2900 extents in our example
lvextend -l 2900 /dev/vgora_asm/lvol1 /dev/disk/disk1
lvextend -l 2900 /dev/vgora_asm/lvol2 /dev/disk/disk2
Assume a PV timeout of 30 seconds.
There are 2 paths to each PV, so the LV timeout value is 60 seconds
Export the volume group across the SGeRAC cluster and mark it as shared, as specified by
SGeRAC documentation. Assign the right set of ownerships and access rights to the raw logical
volumes on each node as required by Oracle (oracle:dba and 0660, respectively).
We can now use the raw logical volume device names as disk group members when configuring
ASM disk groups using the Oracle database management utilities. There are a number of ways
of doing this described in Oracle ASM documentation, including the dbca database creation
wizard as well as sqlplus.
The same command sequence, with some modifications, can be used for adding new disks to an
already existing volume group that is being used by ASM to store one or more RAC databases.
If the database(s) should be up and running during the operation, we use the Single Node Online
volume Reconfiguration (SNOR) feature of SLVM.
Step 1 of the above sequence is modified as follows:
First, deactivate the volume group vgora_asm on all nodes but one, say node A. This requires
prior shutdown of the database(s) using ASM-managed storage and ASM itself, on all nodes
but node A. See the section to understand why it is not adequate to shut down only the
database(s) using the volume group to be reconfigured, and why we must shut down ASM
itself and therefore all database(s) using ASM-managed storage, on all nodes but node A.
Next, on node A, switch the volume group to exclusive mode, using SNOR.
Initialize the disks to be added with pvcreate, and then extend the volume group with
vgextend.
Step 2 remains the same. Logical volumes are prepared for the new disks in the same way.
In step 3, switch the volume group back to shared mode, using SNOR, and export the VG across
the cluster, ensuring that the right ownership and access rights are assigned to the raw logical
volumes. Activate the volume group, and restart ASM and the database(s) using ASM-managed
storage on all nodes (they are already active on node A).
ASM over Raw disk

As mentioned above, a new I/O infrastructure that enables the native built-in multipathing
functionality is included in HP-UX 11i v3. This feature offers users a continuous I/O access to a
LUN or disk if any of the paths fails. This newly added functionality enables SGeRAC (11.17.01
76
or later) to support ASM on raw disks/disk array LUs. In HP-UX 11i v3, new DSF is introduced.
SGeRAC will support the DSF format that ASM support with the restriction that native multipathing
feature is enabled.
The advantages for ASM-over-raw are as follows:
There is a small performance improvement from one less layer of volume management.
Online disk management (adding disks, deleting disks) is supported with ASM-over-raw.
The disadvantages for ASM-over-raw are as follows:
Might not see the HP-UX devices (raw disks/disk array LUs) used for disk group members as
the same names on all nodes.
There is no protection to prevent the raw disks from being incorporated into VxVM volume/disk
groups.
Configure Raw Disks/Disk Array Logical Units for ASM Disk Group
Oracle provides instructions on how to configure disks for ASM where the member disks are raw
logical volume. The instructions to configure raw disks/disk LUs are the following:
For Oracle 10g R2, please refer to Oracle Database Installation Guide 10g Release
2 for hpux Itanium , Chapter 2, Preinstallation Tasks, section Preparing Disk Group
for an Automatic Storage Management Installation.
For 11g R1, please refers to Oracle Clusterware Installation Guide 11g Release 1
(11.) for HP-UX, Chapter 5, Configuring Oracle Real Application Clusters Storage,
section Configuring Disks for Automatic Storage Management.
Then, these raw devices can be used as disk group members to configure ASM disk group members
using Oracle database management utilities.
Additional Hints on ASM Integration with SGeRAC

This section includes some pointers that may be useful when deploying ASM in an SGeRAC
environment.
Consider using the MNP/Simple Dependency-based SGeRAC Toolkits Framework

The SGeRAC Toolkit which provides a framework to integrate Oracle 10g R2 RAC or 11g R1 RAC
with SGeRAC and is based on the SGeRAC A.11.17 multi-node package and simple package
dependency features provides a uniform, intuitive and easy-to-manage method to co-ordinate
between SGeRAC and Oracle Clusterware and manage all the storage options supported by
SGeRAC, including ASM-over-SLVM and ASM-over-raw devices.
ASM Halt is needed to ensure disconnect of ASM from SLVM Volume Groups
This section is specific to ASM-over-SLVM only.
When an ASM disk group is dismounted on a node in the SGeRAC cluster, there is no guarantee
that processes in the ASM instance on that node and client processes of the ASM instance will
close their open file descriptors for the raw volumes underlying the members of that ASM disk
group.
Consider a configuration in which there are multiple RAC databases using ASM to manage their
storage in an SGeRAC cluster. Assume each database stores its data in its own exclusive set of
ASM disk groups.
If we shut down the database instance for a specific RAC database on a node, and then dismount
its ASM disk groups on that node, some Oracle processes may still hold open file descriptors to
the underlying raw logical volumes. Hence, an attempt at this point to deactivate the corresponding
SLVM volume group(s) on the node may fail. The only way to ensure success of the deactivation
77
of the volume groups is to first shut down the ASM instance and its clients (including all databases
that use ASM based storage) on that node.
The major implications of this behavior include the following:
Many SGeRAC customers use SGeRAC packages to start and shut down Oracle RAC instances.
In the startup and shutdown sequences, the package scripts activate and deactivate the SLVM
volume groups used by the instance.
For the ASM environment, it is not appropriate to include SLVM volume group activation and
deactivation in the database instance package control script, since the deactivation may fail.
In the SGeRAC Toolkit that is MNP/Simple Dependency-based, the SLVM volume groups
underlying ASM disk groups are managed instead from the package control script for Oracle
Clusterware.
When there are multiple RAC databases using ASM to manage their storage in the cluster,
online reconfiguration of SLVM volume groups for one database will impact the others, even
if each database is configured with its own set of ASM disk groups and underlying SLVM
volume groups.
An SLVM volume group is reconfigured online, using the SLVM SNOR feature. This procedure
requires the volume group to be deactivated on all nodes but one. This in turn requires shutting
down the ASM instance and all client database instances on all nodes but one.
However, note that many storage reconfiguration operations can be confined to the ASM
layer. For example, if a new file has to be created in a disk group, there will be no SLVM
operation required, if the disk group has adequate free space. Adding disks to, and deleting
disks from, ASM disk groups may require SLVM reconfiguration (online disk addition is
discussed in the section).
ASM, that the corresponding logical volume be created at the same time, to avoid the potential
impact of a future online SLVM reconfiguration operation to create the logical volume. Note that,
when adding one or more physical volumes in an SLVM configuration, one can avoid an online
SLVM reconfiguration operation by creating a new volume group for the physical volume(s).
ASM may require Modified Backup/Restore Procedures

Backup/restore procedures for databases stored in ASM must make use of Oracle Recovery
Manager (RMAN), per Oracle documentation. Customers wishing to deploy ASM should ensure
that their backup/restore tools/scripts/procedures are compatible with this ASM requirement.
Installation, Configuration, Support, and Troubleshooting

Oracle ASM is part of the Oracle database server installation (both 10g R2 and 11g R1). No
additional software from HP is required to operate ASM in SGeRAC environment. Information on
how to use the optional MNP/Simple Dependency based SGeRAC Toolkit is separately documented.
Oracle ASM and ASM disk groups may be configured at the time of creating a database or as a
separate step. These tasks are carried out using Oracle database management tools, for example,
the database creation wizard dbca. In either case, ensure that SLVM volume groups and raw
disks/LUs have been prepared and activated prior to ASM disk group configuration or
reconfiguration, as discussed in this section. For SLVM, use the names of raw LVs contained in the
SLVM volume groups when Oracle commands and utilities require the specification of the names
of raw HP-UX devices for storing ASM disk groups.
For troubleshooting, it may be useful to look at subdirectories for database and ASM instances
under $ORACLE_BASE/admin/, where log and trace files for these instances are deposited.
Oracle clusterware deposits log and trace information regarding its management of ASM and
database instances under $ORACLE_HOME/log/<hostname>/racg.
78
Additional Documentation on the Web and Scripts
Oracle Clusterware Installation Guide 11g Release 1 (11.1) for HP-UX at www.oracle.com/
pls/db111/portal.portal_db?selected=11&frame= HP-UX Installation Guides Clusterware
Installation Guide for HP-UX
ASM related sections in Oracle Manuals
Oracle Database Administrator's Guide 10g R2 (10.2) at www.oracle.com/pls/db102/

portal.portal_db?selected=4 Guides Administrator's Guide
Oracle Database Oracle Clusterware and Oracle Real Application Clusters

Administration and Deployment Guide10g R2 (10.2) at www.oracle.com/pls/db102/
portal.portal_db?selected=4 High Availability Oracle Clusterware and Oracle Real
Application Clusters Administration and Deployment Guide
Oracle Database Administrator's Guide 11g R1 at www.oracle.com/pls/db111/

portal.portal_db?selected=4&frame= Supporting Documentation Administrator's
Guide
www.oracle.com/technology/products/database/asm/index.html
www.oracle.com/pls/db111/portal.portal_db?selected=4&frame= Storage Storage

Administrators Guide
79
4 SGeRAC Toolkit for Oracle RAC 10g or later

Introduction
This chapter discusses how Serviceguard Extension for RAC Toolkit enables a new framework for
the integration of Oracle 10g Release 2 (10.2.0.1) or later version of Real Application Clusters
(Oracle RAC1) with HP Serviceguard Extension for Real Application Clusters A.11.17 or later
(SGeRAC2).
SGeRAC Toolkit leverages the multi-node package and simple package dependency features
introduced by HP Serviceguard (SG) A.11.17 to provide a uniform, easy-to-manage and intuitive
method for coordinating the operation of the combined software stack across the full range of
storage management options supported by SGeRAC.
As background to the discussion of the Toolkit, the following is reviewed:
The coordination issues, pertaining to the combined stack, that the toolkit addresses.
The features in SGeRAC that enable the framework.
The discussion of SGeRAC begins with the reasons why the Toolkit uses the SG/SGeRAC multi-node
package and simple package dependency features. Next, there is an outline of the flow of control
during startup and shutdown of the combined stack using SGeRAC. This is followed by a description
of how the Toolkit interacts both with Oracle RAC and with the storage management subsystems.
Then, the SGeRAC internal file structure is discussed. Lastly, the SGeRAC benefits are listed.
HP recommends to use SGeRAC toolkit for the following reasons:
The SGeRAC Toolkit ensures the following:
RAC database will not run unless the Oracle CRS is running.
CRS package will not try to come up before it's required dependency package like SG CFS
mount point and disk group packages comes up. That is how SGeRAC toolkit enforces the
proper dependencies.
The storage needed for the operation of Oracle Clusterware is activated before Oracle
Clusterware processes are started.
The storage needed for the operation of a RAC database instance is activated before the RAC
database instance is started.
Oracle Clusterware and the RAC database instance are halted before deactivating the storage
needed by these two entities, while halting these packages or SGeRAC nodes.
Background
Coordinating the Oracle RAC/Serviceguard Extension for RAC stack
The Oracle 10g and later database server offers a built-in feature called Oracle Clusterware which
builds highly available RAC and single instance databases in clustered configurations. Since the
release of Oracle 10g, HP has recommended a combined SGeRAC/Oracle Clusterware
configuration for RAC deployments on HP-UX 11i.3 In this combined environment, the responsibilities
of SGeRAC include the following:
80
Provide cluster membership information to the Oracle Clusterware CSS (Cluster Synchronization
Service) daemon.
Provide clustered storage to meet the needs of Oracle Clusterware and RAC database instances.
The Oracle Clusterware quorum voting and registry devices can be configured as shared raw
logical volumes managed by SGeRAC using Shared Logical Volume Manager (SLVM) or
Cluster Volume Manager (CVM) or, beginning with SGeRAC A.11.17, as shared files managed
by SGeRAC using the Cluster File System (CFS). Beginning with Oracle 11gR2 release, Oracle
SGeRAC Toolkit for Oracle RAC 10g or later
Clusterware voting and registry devices can also be configured using oracle ASM (Automatic
Storage Management) disk groups. The members of disk groups are configured as raw devices
(on HP-UX 11i v3). Oracle 11gR2 is supported only on HP-UX 11i v3 (11.31) with SGeRAC
A.11.19 or later.
The RAC database files can be configured as shared raw logical volumes managed by SGeRAC
using SLVM or CVM. Beginning with SGeRAC A.11.17, the RAC database files may be
configured as shared files managed by SGeRAC using CFS. Also, beginning with Oracle 10g
R2 and SGeRAC A.11.17, the RAC database files may also be configured as files in Oracle
ASM (Automatic Storage Management) Disk Groups. The members of the ASM Disk Groups
are configured as raw devices.
The responsibilities of Oracle Clusterware in this combined environment include the following:
Management of the database and associated resources (database instances, services, virtual
IP addresses (VIPs), listeners, etc.).
Management of Oracle ASM instances, if configured.
All pieces of the combined stack must start up and shut down in the proper sequence and we need
to be able to automate the startup and shutdown sequences, if desired. In particular, the storage
needed for the operation of Oracle Clusterware must be activated up before the Oracle Clusterware
processes are started, and the storage needed for the operation of a RAC database instance must
be activated before the instance is started. On shutdown, the sequence is reversedOracle
Clusterware and the RAC database instance must be halted before deactivating the storage needed
by these two entities.
Traditionally, in the SG and SGeRAC environment, these ordering requirements have been met
using a package to encapsulate the startup and shutdown of an application as well as the startup
and shutdown of storage needed by that application. In SG and SGeRAC, a different model is
introduced for the case where the storage needs of an application are met by using a CFS. Here
the CFS is started up and shutdown in a separate package from the one that starts up and shuts
down the application. Beginning with patches PHSS_40885/PHSS_40886 SGeRAC has introduced
new MNP in to existing SGeRAC toolkit namely ASMDG MNP. In SGeRAC, a different model is
recommended for the case where the storage needs of an application are met by using Oracle
ASM. Here ASM disk groups are mounted, dismounted, and monitored using a separate MNP
package. It also activates and deactivates the shared volume groups needed by the ASM disk
groups if ASM is configured over SLVM. The ordering requirement is met by using the SGeRAC
feature of simple package dependencies, discussed later in this chapter.
Can we manage the storage needs of Oracle Clusterware and RAC database instances in Oracle
RAC, using SGeRAC packages in the ways just discussed? Starting in Oracle 10.1.0.4, Oracle
made the following improvements in coordination between Oracle Clusterware and platform
clusterware, enabling such use of SGeRAC packages.
Support for on-demand startup and shutdown of Oracle Clusterware and RAC database instances
In addition to starting up and shutting down Oracle Clusterware automatically as HP-UX 11i
is taken up to init level 3 and taken down to a lower level respectively, Oracle Clusterware
can be start up and shut down on demand.
To disable the automatic startup of Oracle Clusterware on entering init level 3, we use the
crsctl disable crs command. Oracle Clusterware may thereafter be started up and
shut down on demand using the commands crsctl start crs and crsctl stop crs
respectively.
In addition to starting up and shutting down the RAC database instance automatically as
Oracle Clusterware itself is started up and shut down, we can start up and shut down the RAC
database instance on demand.
To disable the automatic startup of the RAC database instance with the startup of Oracle
Clusterware, we follow the procedures described by Oracle to remove auto-start for the
instance. The RAC database instance may thereafter be started up and shut down on demand
Background
81
by, for example, using the command srvctl start instance... and srvctl stop
instance... respectively.
NOTE: The above mentioned steps are the mandatory prerequisite steps to be performed before
you configure SGeRAC toolkit for CRS, ASMDG (if storage is ASM/SLVM), and RAC MNPs.
Support for invocation of Oracle Clusterware commands from customer-developed scripts
This includes invocation of such commands from SGeRAC package control scripts or module scripts;
therefore, SGeRAC packages can invoke commands to start up and shutdown Oracle Clusterware
and/or RAC database instances.
With these improvements, it became possible, using SGeRAC packages, to meet the sequencing
requirements mentioned above for the startup and shutdown of Oracle Clusterware and RAC
database instances with respect to the SGeRAC-managed storage used by these entities.
Serviceguard/Serviceguard Extension for RAC multi-node packages and simple

package dependencies
The new features in SG/SGeRAC that enable the framework provided by are multi-node packages
(MNPs) and simple package dependencies.
Configuring an SGeRAC package via the PACKAGE_TYPE attribute to be of type MULTI-NODE
allows the package to run on more than one node at once. MNPs have the following features:
Each instance of an MNP behaves as a normal package but does not failover.
FAILOVER_POLICY and FAILBACK_POLICY configuration attributes cannot be specified

for an MNP.
Relocatable IP addresses cannot be configured for MNPs.
Failures of package components such as services, EMS resources or subnets, will cause the
MNP to be halted only on the node on which the failure occurred. Unlike failover packages,
when an instance of an MNP fails, only node switching is set to disabled, rather than the
global AUTO_RUN.
SG/SGeRAC Package Manager commands have been enhanced to manage MNPs:
cmviewcl has been enhanced to show the overall package status as a summary of the
status of package instances. If not all the configured instances of an MNP are running,
a qualifier is added to the STATUS field of the form (<running>/<configured>) where
<running> indicates the number of instances running and <configured> indicates
the total number of configured instances. The status of each instance can also be displayed.
cmrunpkg/cmhaltpkg have been enhanced to allow multiple -n parameters to be

specified to indicate the nodes to which the command is to be applied. Using these
commands, the instances of the MNP can be started and stopped independently. MNPs
can be started automatically by setting AUTO_RUN to YES.
Simple package dependencies are used to describe the dependency relationship between packages.
To configure package A with a simple package dependency on package B, we set the following
three attributes of package A:
82
DEPENDENCY_NAME: Each dependency must have a unique name within the package which
is used to identify it.
DEPENDENCY_CONDITION: Expression describing what must be true for the dependency to

be satisfied. The syntax is: <package name> = UP, where <package name> is the package
being depended on (B in this case).
DEPENDENCY_LOCATION: Describes where the condition must be satisfied. For SGeRAC, the
only possible value for this attribute is SAME_NODE.
Simple package dependencies have the following features/restrictions:
cmrunpkg will fail if the user attempts to start a package that has a dependency on another
package that is not running. The package manager will not attempt to start a package if its
dependencies are not met. If multiple packages are specified to cmrunpkg, they will be
started in dependency order. If the AUTO_RUN attribute is set to YES, the package manager
will start the packages automatically in dependency order.
cmhaltpkg will fail if the user attempts to halt a package that has another package depending
on it that is still running. If multiple packages are specified to cmhaltpkg, they will be halted
in dependency order. During cmhaltcl or cmhaltnode, the package manager will halt
packages in dependency order.
The output of cmviewcl shows the current state of each dependency on each node where
the package is configured.
A failover or multi-node package may define dependencies on multiple multi-node packages.

Multiple failover or multi-node packages may depend on a multi-node package. Multi-level
dependencies can exist; for example, A depends on B which in turn depends on C, etc.
If A depends on B and B fails, A (the appropriate instance, if A is of type multi-node) is halted.
Why use multi-node packages/simple package dependencies for Oracle

RAC integration
As mentioned previously, we want to use packages to manage the storage for Oracle Clusterware
and RAC database instances. To make the case for the use of the SGeRAC package manager
enhancements, we need separate SGeRAC packages for Oracle Clusterware and the RAC database
instances.
If we have a package only for Oracle Clusterware, then the storage for each RAC database in
the cluster as well as for Oracle Clusterware must be managed out of the Oracle Clusterware
package. This approach could potentially create both a single point of failure as well as a
management bottleneck.
For example, if the storage for one database fails to start up on a given node (for example, because
an SLVM volume group containing part of the database fails to start up), the Oracle Clusterware
package will fail to start up Oracle Clusterware, and consequently none of the RAC database
instances will start up on that node. For some types of failure (for example, the SLVM volume group
is corrupt), the failure to start up the databases could be cluster-wide.
Another example: if we are in a CFS environment with a CFS for each RAC database, then we
need packages for each of the databases to properly model the dependency of a database instance
on its CFS. We cannot do this if there is only a package for Oracle Clusterware. Setting up the
dependency as between Oracle Clusterware and each CFS will result in inappropriate system
behavior (for example, after shutting down a RAC database instance on a node, we will not be
able to shut down its CFS on that node without shutting down Oracle Clusterware.)
If ASM over SLVM or ASM over raw disk is used as storage for oracle RAC database, he can
configure ASMDG MNP. This is a new enhancement to SGeRAC toolkit. This MNP decouples ASM
Disk Group management from OC MNP Package and creates an independent MNP package for
ASM DG management.
The VG for ASM Diskgroup will be specified in this new ASM DG Package configuration file
instead of OC Package configuration file. This package will be responsible for activating the VG
which contains the diskgroup for database and mounting and unmounting of the diskgroup. This
package will also monitor the state of diskgroup.
Configuring this package is mandatory only in metrocluster environment. However, it is only a
recommendation not mandatory in non metrocluster environment. You can configure CRS and RAC
MNP as suggested by the old toolkit.
Why use multi-node packages/simple package dependencies for Oracle RAC integration
83
An example of a bottleneck created if we only have a package for Oracle Clusterware is this: if
we concentrate all storage management in the Oracle Clusterware package, then any time there
is a change in the storage configuration for one database (for example, an SLVM volume group
is added), we would have to modify the Oracle Clusterware package.
These are the main arguments in favor of having separate packages for Oracle Clusterware and
each RAC database. But then the question arises: how do we ensure that these packages start and
halt in the proper order? Prior to version A.11.17, SG/SGeRAC did not provide a mechanism for
package ordering.
Two new features are introduced in SG/SGeRAC that help us solve this problem: MNPs and simple
package dependencies. The combination of MNPs and simple package dependencies is a very
good fit for our problem of coordinating Oracle RAC and SGeRAC. We configure Oracle
Clusterware as one MNP and each RAC database as another MNP and we set up the database
MNPs to depend on the Oracle Clusterware MNP. This is the core concept of SGeRAC.
Both Oracle Clusterware and the RAC database are multi-instance applications well suited to being
configured as MNPs. Further, the use of MNPs reduces the total package count and simplifies
SGeRAC package configuration and administration. Simple package dependencies enable us to
enforce the correct start/stop order between the Oracle Clusterware MNP and RAC database
MNPs.
Serviceguard Extension for RAC Toolkit operation

In this section, we first discuss how SGeRAC organizes the flow of control through the combined
stack. Then we deal with the interactions of the toolkit, first with Oracle RAC, and then with the
storage management subsystems.
Figure 15 shows how the toolkit enables SGeRAC and Oracle Clusterware to work in harmony
and ensure proper sequencing between themselves and the resources they manage.
It shows Oracle Clusterware and the resources it manages, via its Cluster Ready Services (CRS)
component, in red and the SGeRAC package manager (PM) in blue, as well as the resources it
manages.
Each resource manager starts up and shuts down the resources it manages, honoring the
dependencies that it recognizes. The dependencies are shown in red or blue depending on whether
they are Oracle Clusterware dependencies or SGeRAC dependencies.
The storage packages (Oracle Clusterware Storage and Database Storage) and associated
dependencies only exist for CFS, and the ASM instance and associated dependencies only exist
when ASM is being used.
SGeRAC provides scripts for Oracle Clusterware MNP, ASMDG MNP, and RAC DB instance
MNP. The Oracle Clusterware MNP starts, stops, and checks the status of the Oracle Clusterware.
The ASMDG MNP mounts dismounts and monitors the status of ASM Disk groups. The RAC DB
instance MNP starts, stops, and checks the status of the RAC instance. Oracle Clusterware and
RAC instance auto-start have to be disabled (using Oracle specified commands and procedures,
as previously discussed in this chapter).
84
Figure 12 Resources managed by SGeRAC and Oracle Clusterware and their dependencies
Startup and shutdown of the combined Oracle RAC-SGeRAC stack

The combined stack is brought up in proper order by cmrunnode or cmruncl as follows.
1. SGeRAC starts up.
2. The SGeRAC package manager starts up Oracle Clusterware via the Oracle Clusterware
MNP, ensuring that the storage needed is made available first. This requires prior startup of
disk group and mount point MNPs in the case of the storage needed by Oracle Clusterware
being managed by CFS.
3. On startup, Oracle Clusterware brings up the resources that it manages, that have been set
to auto-start within the Oracle Clusterware registry. It brings up the node applications (this is
an Oracle Clusterware term indicating resources configured per cluster node, such as the
Virtual Internet Protocol address (VIP), Oracle listener, etc.) and the ASM instance if configured.
Oracle Clusterware does not automatically bring up the RAC database instances (and the
Oracle services dependent on them) since we have turned off auto-start for the instances.
4. After the start of CRS, SGeRAC mounts the ASM disk groups which are configured for RAC
databases via ASMDG MNP packages.
5. Lastly, SGeRAC starts up the RAC database instance via the RAC database instance MNP,
ensuring that the storage needed by it is made available first (this requires prior startup of
disk group and mount point MNPs or ASM DG MNPs depending the kind of storage configured
for RAC databases, either CFS or ASM disk groups respectively). It can do this since the
SGeRAC dependency on the Oracle Clusterware MNP being up is met. So now the RAC
database instances and dependent Oracle services start up.
The combined stack is brought down in proper order by cmhaltnode or cmhaltcl as follows.
First, SGeRAC shuts down the RAC database via the RAC database instance MNP, followed by
shutdown of the storage needed by it. This requires subsequent shutdown of mount point and disk
group MNPs or ASMDG MNPs that are dependent on the particular RAC MNP in the case of the
storage needed by the RAC database being managed by CFS or ASM disk groups respectively.
85
Next, SGeRAC package manager shuts down Oracle Clusterware via the Oracle Clusterware
MNP, followed by the storage needed by Oracle Clusterware (this requires subsequent shutdown
of mount point and disk group MNPs in the case of the storage needed by Oracle Clusterware
being managed by CFS). It can do this since the dependent RAC database instance MNP is already
down. Before shutting itself down, Oracle Clusterware shuts down the ASM instance if configured,
and then the node applications. Lastly, SGeRAC itself shuts down.
Note that the stack can be brought up or down manually, package by package, by using
cmrunpkg/cmhaltpkg in the proper dependency order. To disable (partially or wholly) automatic
startup of the stack when a node joins the cluster, the AUTO_RUN attribute should be set to NO on
the packages that should not automatically be started.
How Serviceguard Extension for RAC starts, stops and checks Oracle Clusterware
Having discussed how the toolkit manages the overall control flow of the combined stack during
startup and shutdown, we will now discuss how the toolkit interacts with Oracle Clusterware and
RAC database instances. We begin with the toolkit interaction with Oracle Clusterware.
The MNP for Oracle Clusterware provides start and stop functions for Oracle Clusterware and has
a service for checking the status of Oracle Clusterware.
The start function starts Oracle Clusterware using crsctl start crs. To ensure successful
startup of Oracle Clusterware, the function, every 10 seconds, runs crsctl check until the
command output indicates that the CSS, CRS, and EVM daemons are healthy. If Oracle Clusterware
does not start up successfully, the start function will execute the loop until the package start timer
expires, causing SGeRAC to fail the instance of the Oracle Clusterware MNP on that node.
The stop function stops Oracle Clusterware using crsctl stop crs. Then, every 10 seconds,
it runs ps until the command output indicates that the processes called evmd.bin, crsd.bin,
and ocssd.bin no longer exist.
The check function runs ps to determine process id of the process called ocssd.bin. Then, in a
continuous loop driven by a configurable timer, it uses kill -s 0 to check if this process exists.
The other daemons are restarted by Oracle Clusterware, so they are not checked.
When Oracle Clusterware MNP is in maintenance mode, the check function pauses the Oracle
Clusterware health checking. Otherwise, if the check function finds that the process has died, it
means that Oracle Clusterware has either failed or been inappropriately shut downwithout using
cmhaltpkg. The service that invokes the function fails at this point and the SGeRAC package
manager fails the corresponding Oracle Clusterware MNP instance.
How Serviceguard Extension for RAC Mounts, dismounts and checks ASM disk groups
We discuss the toolkit interaction with the ASM disk groups.
The MNP for the ASM diskgroups that are needed by RAC database provides mount and dismount
functions for the ASM diskgroups and has a service for checking the status of those ASM diskgroups
whether they are mounted or not.
The start function executes su to the Oracle software owner user id. It then determines the ASM
instance id on the current node for the specified diskgroup using crsctl status resource
ora.asm. It is stored in variable and used for future references. Then it mounts the ASM disk
groups mentioned in that ASMDG MNP by connecting to ASM instance using sqlplus.
The stop function executes su to the Oracle software owner user id. It unmounts the ASM diskgroups
which are specified in that ASMDG MNP by connecting to ASM instance via sqlplus.
The check function determines the status of the ASM disk groups that are mentioned in ASMDG
MNP. When ASMDG MNP is in maintenance mode, the ASM diskgroup status checking is paused.
Otherwise, in a continuous loop driven by a configurable timer, the check function monitors the
status of the ASM diskgroups mentioned in that ASMDG MNP. If one or more ASM diskgroup is
in a dismounted state, the check function will report failurethe ASM diskgroup is dismounted
without using cmhaltpkg. The service that invokes the function fails at this point and the SGeRAC
86
package manager fails the corresponding ASMDG MNP and the RAC MNP that is dependent on
ASMDG MNP.
How Serviceguard Extension for RAC Toolkit starts, stops, and checks the RAC
database instance
Next, the toolkit interaction with the RAC database is discussed.
The MNP for the RAC database instance provides start and stop functions for the RAC database
instance and has a service for checking the status of the RAC database instance.
The start function executes su to the Oracle software owner user id. It then determines the Oracle
instance id8 on the current node for the specified database using srvctl status database.
Then it starts the corresponding RAC database instance using srvctl start instance. If an Oracle
Clusterware placement error occurs, indicating that CRS is not ready to start the instance, the
function sleeps for 2 minutes and then retries. At most 3 attempts are made to start the instance.
The stop function executes su to the Oracle software owner user id. It then determines the Oracle
instance id on the current node for the specified database using srvctl status database.
Then it stops the corresponding Oracle RAC instance using srvctl stop instance. If the user
configurable parameter STOP_MODE is abort and Oracle RAC Instance is not halted by srvctl
command within ORA_SHUTDOWN_TIMEOUT seconds, the Oracle RAC instance is terminated via
killing its background processes.
The check function executes ps and crs_stat commands to determine the health of RAC instance.
When the Oracle database instance MNP is in maintenance mode, the RAC instance health
checking is paused. Otherwise, in a continuous loop driven by a configurable timer, the check
function runs ps to check the number of the monitored RAC instance background processes. If one
or more RAC background processes are gone and crs_stat command shows Oracle Clusterware
has not restarted the Oracle RAC instance, the function will report the RAC instance as down. This
means that the RAC instance failed or has been inappropriately shut down without using
cmhaltpkg. The service that invokes the function fails at this point and the SGeRAC package
manager fails the corresponding RAC database MNP instance.
How Serviceguard Extension for RAC Toolkit interacts with storage management
subsystems
The core concept of the Toolkit, namely, configuring an MNP for Oracle Clusterware and for each
RAC database and configuring a dependency of each RAC database MNP on the Oracle
Clusterware MNP holds true across the following storage management options supported by
SGeRAC: SLVM, CVM, ASM over raw device (on HP-UX 11i v3) and CFS. The above dependency
may not hold well if ASM over SLVM is used as a storage option for RAC databases. Beginning
with the SGeRAC A.11.19 patches, PHSS_40885 (11i v2) and PHSS_40886 (11i v3), SGeRAC
toolkit introduces a new ASMDG MNP package to decouple ASM disk group management from
OC MNP. In previous toolkit versions, RAC database shared volume groups used for ASM disk
groups were defined in the OC MNP. However, the Storage management option deployed will
have some impact on the configuration of the toolkit.
Use Case 1: Oracle Clusterware storage and database storage in SLVM/CVM

Figure 13 Use Case 1 Setup
In this case, Oracle Clusterware quorum and voting disk and RAC database files are stored in raw
logical volumes managed by SLVM or CVM. The management of SLVM or CVM storage for Oracle
Clusterware and database is specified in the package configuration of the respective MNPs.
87
Use Case 2: Oracle Clusterware storage and database storage in CFS

In this case, Oracle Clusterware quorum and registry device data is stored in files in a CFS. Oracle
database files are also stored in a CFS.
For each CFS used by Oracle Clusterware for its quorum and registry device data, there will be
a dependency configured from the Oracle Clusterware MNP to the mount point MNP corresponding
to that CFS. The mount point MNP has a dependency on the CFS system MNP (SMNP).
Similarly, for each CFS used by the RAC database for database files, there will be a dependency
configured from the RAC database MNP to the mount point MNP corresponding to that CFS. Only
the Oracle Clusterware and RAC DB Instance MNPs, and their dependencies, in this use case are
to be configured and managed as described in this chapter. The rest of the multi-node packages
shown for this use case are created via the CFS subsystem. Configuration and administration
procedures for these MNPs are specified in the SGeRAC user guide.
Use case 3: Database storage in ASM over SLVM

The above diagram can be considered as one use case. Here we have one Oracle Clusterware
MNP, three ASMDG MNP, and four RAC database MNP. All the ASMDG MNPs should be made
88
dependent on Oracle Clusterware MNP. Disk groups that are exclusively used by a RAC DB should
be managed in separate ASM DG MNP. If different RAC Database uses different ASM Disk groups
then those, ASM DGs should not be configured in a single ASMDG MNP. As RAC DB Instance
MNP 3 and RAC DB Instance MNP 4 use completely different ASM diskgroups, they are made
dependent on their respective ASMDG MNP(ASMDG MNP 2, ASMDG MNP 3). However, If two
RAC DB use same set of ASM Disk groups, then those ASM DG can be configured in a single
ASMDG MNP. Then, both the RAC MNP is made dependent on the ASMDG MNP. RAC DB
Instance MNP 1, RAC DB Instance MNP 2 makes use of same set of ASM diskgroups so both
MNPs are made dependent on ASMDG MNP 1.
How to create a Serviceguard Extension for RAC modular package

In SGeRAC A.11.19, SGeRAC introduces modular package format. With this feature, the user
can use the single SGeRAC package interface to manage SGeRAC Toolkit to simplify Toolkit
management. For more information see, Support for the SGeRAC Toolkit (page 92) section.
How Serviceguard Extension for RAC Toolkit maintenance mode works

As of SGeRAC A.11.18, SGeRAC Toolkit supports the maintenance mode, so the user can maintain
Oracle Clusterware and Oracle database instance without halting Oracle Clusterware MNP and
Oracle database instance MNP. Beginning with the SGeRAC A.11.19 patches, PHSS_40885
(11i v2) and PHSS_40886 (11i v3), SGeRAC toolkit also supports the maintenance mode of
ASMDG MNPthe user can maintain ASM DG included in ASMDG MNP without halting the
MNP. The MAINTENANCE_FLAG parameter is specified in the respective toolkit configuration files.
Use Case 1: Performing maintenance with Oracle Clusterware

Oracle Clusterware MNP, ASMDG MNP (if configured) and the Oracle database instance MNP
go into maintenance mode.
1. Make sure the MAINTENANCE_FLAG parameter for Oracle Clusterware MNP, is set to yes
when these packages are created. If not, shutdown the MNPs first, set the MAINTENANCE_FLAG
to yes, and then restart MNPs.
2. On the maintenance node, create a debug file called oc.debug in the Oracle Clusterware
MNP working directory. All the three MNPs on this node will go into maintenance mode. The
maintenance mode message will appear in the Toolkit package log files, e.g. OC MNP
pausing Oracle Clusterware checking and entering maintenance mode.
3. The user can maintain the Oracle Clusterware, ASM disk groups, and Oracle Database
instance on that node while Toolkit package is still running.
4. After the maintenance work is completed, the user can remove the created oc.debug in step
2 to bring the Toolkit package out of maintenance mode and resume normal monitoring by
Serviceguard. The Toolkit in maintenance mode message will appear in OC MNP package
log files, e.g. Starting Oracle Clusterware checking again after maintenance.
Use Case 2: performing maintenance with ASM disk groups

Only the Oracle ASMDG MNP goes into maintenance mode.
1. Make sure the MAINTENANCE_FLAG parameter for ASMDG MNP is set to yes when this
package is created. If not, shutdown this MNP first, set the MAINTENANCE_FLAG to yes, and
then restart the MNP.
2. On the maintenance node, create a debug file called asm_dg.debug file in the Oracle
ASMDG MNP working directory. The Oracle ASMDG MNP on this node will go into
maintenance mode. The maintenance mode message will appear in ASMDG MNP package
log files, e.g. ASM DG MNP pausing ASM DG instance checking and entering maintenance
mode.
89
3.
4.
The user can maintain the Oracle ASM disk groups on that node while Oracle ASMDG MNP
package is still running.
After the maintenance work is completed, the user can remove the created asm_dg.debug
in step 2 to bring the Oracle ASMDG MNP package out of maintenance mode to resume
normal monitoring by Serviceguard. The maintenance mode message will appear in the Oracle
database instance package log files, e.g. Starting ASM DG MNP checking again after
maintenance.
Use Case 3: performing maintenance with Oracle RAC database instance

1.
2.
3.
4.
Make sure the MAINTENANCE_FLAG parameter for Oracle database instance MNP is set to
yes when this packages is created. If not, shutdown this first, set the MAINTENANCE_FLAG to
yes, and then restart MNP.
On the maintenance node, create a debug file called rac.debug file in the Oracle database
instance MNP working directory. The Oracle database instance MNP on this node will go
into maintenance mode. The maintenance mode message will appear in Oracle database
instance package log files, e.g. RAC MNP pausing RAC instance checking and entering
maintenance mode.
The user can maintain the Oracle database instance on that node while Oracle database
instance package is still running.
After the maintenance work is completed, the user can remove the created rac.debug in
step 2 to bring the Oracle database instance package out of maintenance mode to resume
normal monitoring by Serviceguard. The maintenance mode message will appear in the Oracle
database instance package log files, e.g. Starting RAC MNP checking again after
maintenance.
Serviceguard Extension for RAC Toolkit internal file structure

There is a set of files in SGeRAC that deal with SGeRAC specific configuration and logic, and a
different set of files that deal with Oracle Clusterware, ASMDG MNP, and RAC specific logic,
with a bridge in between.
On the SGeRAC-specific side is the MNP ASCII configuration file and the control script (for legacy
packages), or module script (for modular packages). The ASCII configuration file parameters are
stored in the SGeRAC configuration database, at cmapplyconf time, and are used by the package
manager in its actions on behalf of this package. The control script or module script invokes the
Oracle Clusterware specific functions for start, stop, and check through the bridge script.
On the Oracle Clusterware specific side, there is a configuration file (oc.conf) that is sourced
by the start, stop and check script files (oc.sh and oc.check). The bridge script
(toolkit_oc.sh) allows the start, stop, and check calls to remain unaffected by changes in the
Oracle Clusterware specific scripts. A similar set of files deals with RAC instance specific logic.
Figure 19 shows the internal file structure of the toolkit for Oracle Clusterware. Figure 20 shows
the similar internal file structure of the toolkit for the ASMDG MNP. Figure 21 shows the similar
internal file structure of the toolkit for the RAC DB instance.
90
Figure 16 Internal structure of SGeRAC for Oracle Clusterware
Figure 17 Internal structure of SGeRAC for ASMDG MNP
Serviceguard Extension for RAC Toolkit internal file structure
91
Figure 18 Internal structure of SGeRAC for RAC DB instance
Support for the SGeRAC Toolkit

NOTE: The content in this section was part of SGeRAC Toolkit README file till SGeRAC A.11.20
patch PHSS_41590. From SGeRAC A.11.20 patch PHSS_41642 onwards the README content
has been moved to this Administration guide.
CONTENTS:
A. Overview
B. SGeRAC Toolkit Required Software
C. SGeRAC Toolkit File Structure
D. SGeRAC Toolkit Files
E. SGeRAC Toolkit Configuration
E-1 Package Configuration File Parameters
E-2 Toolkit Configuration File Parameters
F. SGeRAC Toolkit Configuration Procedures
G. SGeRAC Toolkit Administration
H. SGeRAC Toolkit Supported Configurations
I. SGeRAC Toolkit Q & A
J. SGeRAC Toolkit Limitation/Restriction
K. SGeRAC Toolkit Legacy Package to Modular Package Migration
L. Migration of Legacy CFS Disk group and Mount point Packages to Modular
CFS Disk group and Mount point Packages (CFS DG-MP).
M. SGeRAC Toolkit Adding new ASMDG MNP Package to the existing
configured OC MNP and RAC MNP
N. SGeRAC Toolkit Package Cleanup
O. SGeRAC Toolkit Whitepaper Link
PREFACE:
This README file describes the SGeRAC Toolkit which enables integration of
Oracle Real Application Clusters (RAC) with HP Serviceguard Extension for
Real Application Clusters (SGeRAC). This document covers the Toolkit file
structure, files, configuration parameters and procedures, administration,
supported configurations, known problem and workaround, and the support
restrictions of this version of Toolkit. A link to a whitepaper on the same
topic is also included.
This document assumes that its readers are already familiar with Serviceguard
(SG), SGeRAC, and Oracle RAC, including installation and configuration
procedures.
This version of SGeRAC Toolkit supports Oracle 10g Release 2, 11g Release 1
and 11g Release 2 versions/revision of RAC only.
A. Overview
Oracle 10g and later database server software offers a built-in feature called Oracle
Clusterware for building highly available RAC and single instance databases
in clustered configurations. Since the release of Oracle 10g, HP has
recommended a combined SGeRAC-Oracle Clusterware configuration for RAC
deployments on HP-UX. In the combined stack, SGeRAC provides cluster
92
membership information to the Oracle Clusterware and provides clustered

storage to meet the needs of Oracle Clusterware and RAC database instances.
The Oracle Clusterware manages the database and associated resources (e.g.
database instances, services, virtual IP addresses, listeners, etc), and ASM
instances if configured.
The startup and shutdown of the various components in the combined SGeRAC-Oracle
Clusterware configuration must be coordinated in the proper sequence. The storage
needed for the operation of Oracle Clusterware must be activated before the
Oracle Clusterware processes are started, and the storage needed for the operation
of a RAC database instance must be activated before the RAC database instance
is started. On shutdown, the sequence is reversed. Oracle Clusterware and the RAC
database instance must be halted before deactivating the storage needed by these
two entities.
The SGeRAC Toolkit provides the mechanism to coordinate the ordering of the
pieces in the combined SGeRAC/Oracle RAC stack and the dependency between the
Oracle Clusterware and Oracle RAC instances. Also, the Toolkit offers a
uniform, easy-to-manage and intuitive method to coordinate the operation
across the full range of storage management options supported by SGeRAC.
The SGeRAC Toolkit leverages multi-node package (MNP) and package dependency
features supported in SG/SGeRAC A.11.17 or later. In the SGeRAC Toolkit,
scripts for configuring three MNPs - Oracle Clusterware MNP (OC MNP), RAC
database instance MNP (RAC MNP) and Oracle Automatic Storage Management (ASM)
Diskgroup MNP (ASMDG MNP) are provided. The "OC MNP" allows SGeRAC to
start/stop and monitor Oracle Clusterware processes. The "RAC MNP" allows
SGeRAC to start/stop and monitor processes of a RAC database instance. The
"ASMDG MNP" allows mounting/unmounting and monitoring of the ASM Disk groups.
Since Oracle Clusterware must be running before any RAC database instances
can be started, RAC MNP packages are required to define a 'same node up'
package dependency on the OC MNP. When using ASM for storage management,
the ASM DG MNP must have a package dependency on the OC MNP and the
corresponding RAC MNP packages must have a "same node up" dependency on
the ASM DG MNP.
In SGeRAC, the Oracle Clusterware storage (Oracle Cluster Registry
and voting disk) can be configured as:
- Raw logical volumes managed by Shared Logical Volume Manager (SLVM)
or Cluster Volume Manager (CVM)
- Files managed by Cluster File System (CFS)
- Oracle Automatic Storage management (ASM) Disk Groups (from Oracle 11g R2)
The members of the ASM Disk Groups can be either shared raw logical volumes
managed by SLVM (ASM over SLVM) or HP-UX raw disks ( ASM over HP-UX raw disks)
In SGeRAC, the RAC database files can be configured as
- Raw logical volumes managed by Shared Logical Volume Manager (SLVM)
or Cluster Volume Manager (CVM)
- Files managed by Cluster File System (CFS)
- Oracle Automatic Storage management (ASM) Disk Groups (from Oracle 11g R2)
The members of the ASM Disk Groups can be either shared raw logical volumes
managed by SLVM (ASM over SLVM) or HP-UX raw disks ( ASM over HP-UX raw disks)
Dependency structure while using different storage management options

1. Dependency structure in the case of SLVM
----------------------------|
|
|
|
|
|
|
|
|
OC-MNP
|<--------| RAC-MNP
|
|
|
|
|
|
|
|
|
----------------------------The SLVM Volume group used for Oracle Clusterware storage are configured in the
OC-MNP package.
The SLVM Volume group used for RAC database storage are configured in the RAC-MNP
package.
2. Dependency structure in the case of CFS
----------------------------|
|
|
|
|
|
|
|
|
OC-MNP
|<--------| RAC-MNP
|
|
|
|
|
|
|
|
|
----------------------------|
|
|
|
|
|
|
|
V
V
----------------------------|
|
|
|
|
|
|
|
| CFS-MP1-MNP |
| CFS-MP2-MNP |
|
|
|
|
| Holds OCR & |
| Holds data |
| VOTE files |
| base files |
93
----------------------------|
|
|
|
|
|
V
V
----------------------------|
|
|
|
|
|
|
|
| CFS-DG1-MNP |
| CFS-DG2-MNP |
|
|
|
|
|
|
|
|
----------------------------|
|
|
|
|
|
V
V
--------------------------------------|
|
|
|
|
SG-CFS-pkg
|
|
|
|
|
---------------------------------------
3. Dependency structure in the case of ASM over SLVM and ASM over HP-UX raw disks.
--------------|
|
|
|
| RAC-MNP
|
|
|
|
|
--------------|
|
|
V
--------------|
|
|
|
| ASMDG-MNP |
|
|
|
|
--------------|
|
|
V
--------------|
|
|
|
|
OC-MNP
|
|
|
|
|
--------------In case of ASM over SLVM
The SLVM Volume groups used for Oracle Clusterware storage are configured in the
OC-MNP package.
The SLVM Volume groups used for RAC database storage are configured in the ASMDG MNP
package.
In case of ASM over HP-UX raw disks
Do not specify any HP-UX raw disks information either in OC-MNP package or in ASMDG
MNP package.
B. SGeRAC Toolkit Required Software

To configure and run this version of SGeRAC Toolkit, the following software
is required:
-
HP-UX 11i v2 or HP-UX 11i v3

SG and SGeRAC A.11.19 or later
Oracle 10g R2 RAC or 11g R1 RAC or 11g R2 RAC
CFS or CVM 5.0.1, or CFS or CVM 5.1 SP1 (if CVM/CFS is used for storage management)
The Oracle patches p7225720 and p7330611 must be installed, if you want
to use the ASMDG MNP feature. An additional patch, p6967375 must be
installed if you are using RAC 11gR1 on a PA-RISC system. In the case
of 11gR2 installations no Oracle patches are required.
NOTE: Oracle patches p7225720 and p7330611 are not availabe on Oracle RAC 10.2.0.4
for IA. Upgrade to Oracle RAC 10.2.0.5 to use ASMDG MNP feature on 10.2.0.4 on IA.
C. SGeRAC Toolkit File Structure
From SG/SGeRAC version A.11.19 and later, the SGeRAC Toolkit uses Modular
Packages to implement OC MNP, ASMDG MNP and RAC MNP.
94
After installation of SGeRAC, the SGeRAC Toolkit module Attribute Definition Files (ADF)
reside under the /etc/cmcluster/modules/sgerac directory and the module
scripts reside under the /etc/cmcluster/scripts/sgerac directory.
The SGeRAC Toolkit files reside under /opt/cmcluster/SGeRAC/toolkit. This
directory contains three subdirectories crsp, asmp and racp. Subdirectory
crsp contains the Toolkit scripts for OC MNP. Subdirectory racp contains
the Toolkit scripts for RAC MNP. Subdirectory asmp contains the Toolkit
scripts for ASMDG MNP.
The following files are installed during the SGeRAC Toolkit installation:
/opt/cmcluster/SGeRAC/toolkit/README
/opt/cmcluster/SGeRAC/toolkit/crsp/toolkit_oc.sh
/opt/cmcluster/SGeRAC/toolkit/crsp/oc.conf
/opt/cmcluster/SGeRAC/toolkit/crsp/oc.sh
/opt/cmcluster/SGeRAC/toolkit/crsp/oc.check
/opt/cmcluster/SGeRAC/toolkit/racp/toolkit_dbi.sh
/opt/cmcluster/SGeRAC/toolkit/racp/rac_dbi.conf
/opt/cmcluster/SGeRAC/toolkit/racp/rac_dbi.sh
/opt/cmcluster/SGeRAC/toolkit/racp/rac_dbi.check
/opt/cmcluster/SGeRAC/toolkit/asmp/toolkit_asmdg.sh
/opt/cmcluster/SGeRAC/toolkit/asmp/asm_dg.conf
/opt/cmcluster/SGeRAC/toolkit/asmp/asm_dg.sh
/opt/cmcluster/SGeRAC/toolkit/asmp/asm_dg.check
/etc/cmcluster/modules/sgerac/erac_tk_oc
/etc/cmcluster/modules/sgerac/erac_tk_oc.1
/etc/cmcluster/modules/sgerac/erac_tk_rac
/etc/cmcluster/modules/sgerac/erac_tk_rac.1
/etc/cmcluster/modules/sgerac/erac_tk_asmdg
/etc/cmcluster/modules/sgerac/erac_tk_asmdg.1
/etc/cmcluster/scripts/sgerac/erac_tk_oc.sh
/etc/cmcluster/scripts/sgerac/erac_tk_rac.sh
/etc/cmcluster/scripts/sgerac/erac_tk_asmdg.sh
/etc/cmcluster/scripts/sgerac/oc_gen.sh
/etc/cmcluster/scripts/sgerac/rac_gen.sh
/etc/cmcluster/scripts/sgerac/asmdg_gen.sh
D. SGeRAC Toolkit Files

The SGeRAC Toolkit files under /etc/cmcluster/modules/sgerac are the
module ADF files:
erac_tk_oc.1 - Toolkit OC MNP module ADF file.
erac_tk_oc
- A symbolic link to the latest version of OC MNP module
file (e.g. erac_tk_oc.1).
erac_tk_rac.1 - Toolkit RAC MNP module ADF file.
erac_tk_rac
- A symbolic link to the latest version of RAC MNP module
file (e.g. erac_tk_rac.1).
erac_tk_asmdg.1 - Toolkit ASMDG MNP module ADF file.
erac_tk_asmdg
- A symbolic link to the latest version of ASMDG MNP module
file (e.g. erac_tk_asmdg.1).
The files under /etc/cmcluster/scripts/sgerac are the module script
files:
erac_tk_oc.sh
- Toolkit OC MNP module script file.
erac_tk_rac.sh
- Toolkit RAC MNP module script file.
erac_tk_asmdg.sh - Toolkit ASMDG MNP module script file.
oc_gen.sh
- script to get Toolkit parameters from the SG
configuration database and generate the Toolkit configuration
file oc.conf at OC MNP configuration time. It is called by the
OC MNP module script only.
rac_gen.sh - script to get Toolkit parameters from the SG configuration
database and generate the Toolkit configuration file
rac_dbi.conf at RAC MNP configuration time. It is called by
the RAC MNP module script only.
asmdg_gen.sh - script to get Toolkit parameters from the SG
configuration database and generate the Toolkit configuration
file asm_dg.conf at ASMDG MNP configuration time. It is called
by the ASMDG MNP module script only.
The files and subdirectories under /opt/cmcluster/SGeRAC/toolkit directory are:
README
crsp
racp
asmp
this file
subdirectory for OC MNP
subdirectory for RAC MNP
subdirectory for ASMDG MNP
The files under /opt/cmcluster/SGeRAC/toolkit/crsp are for the OC MNP:

toolkit_oc.sh - The entry point to the toolkit entity. It is an interface
between the OC MNP package control script and the oc.* files
listed below.
oc.conf
- The Toolkit configuration file that contains a list of

pre-defined variables users must customize for use with
their Oracle Clusterware. This is a configuration file
95
which is read by the Toolkit script oc.sh.

oc.check
oc.sh
- Toolkit monitor script that checks if the Oracle Clusterware

is running.
- Toolkit script to start, stop, and check the Oracle Clusterware.
The files under /opt/cmcluster/SGeRAC/toolkit/racp are for the RAC MNP:

toolkit_dbi.sh - The entry point to the Toolkit entity. It is an
interface between the RAC database instance MNP package
control script and the rac_dbi.* files listed below.
rac_dbi.conf - The Toolkit configuration file that contains a list of
their Oracle RAC database instances. This is a configuration
file which is read by the Toolkit script rac_dbi.sh.
rac_dbi.check - Toolkit monitor script that checks if the RAC instance is
running.
rac_dbi.sh - Toolkit script to start, stop, and check the RAC instance.
The files under /opt/cmcluster/SGeRAC/toolkit/asmp are for the ASMDG MNP:
toolkit_asmdg.sh - The entry point to the Toolkit entity. It is an
interface between the ASMDG MNP package
control script and the asm_dg.* files listed below.
asm_dg.conf - The Toolkit configuration file that contains a list of
their Oracle ASMDG. This is a configuration
file which is read by the Toolkit script asm_dg.sh.
asm_dg.check - Toolkit monitor script that checks if the ASM disk group
is mounted or not.
asm_dg.sh - Toolkit script to mount, unmount and check the status ASM disk group.
E. SGeRAC Toolkit Configuration

The OC MNP supports storage management using SLVM, CVM, CFS, ASM ( For 11gR2 or later
versions of RAC only) and ASM over SLVM (For 11gR2 or later versions of RAC only).
The RAC MNP supports storage management using SLVM, CVM, CFS, ASM or ASM over SLVM.
Users should follow the storage configuration procedures required by SLVM, CVM, CFS,
ASM or ASM over SLVM respectively to configure the storage for Oracle Clusterware
(OCR and voting disk) and RAC database (control files, data files, redo-logs, etc).
The HP manual "Using Serviceguard Extension for RAC", found on
http://www.hp.com/go/hpux-serviceguard-docs -> Serviceguard Extension for RAC,
contains detailed instructions on this topic.
Certain package parameters and some specific parameter values are needed when
configuring the Toolkit MNPs. In addition, some Toolkit specific
configurations are needed in the Toolkit configuration file. The following
sections contain the parameters and values required by the SGeRAC Toolkit.
E-1. Package Configuration File Parameters:
The SGeRAC Toolkit for SGeRAC A.11.19 and later supports both modular and legacy
package formats, these two different formats are described as below:
E-1-1: Modular package configuration file parameters:
For modular packages, it is not necessary to create a package script file,
and the package configuration file template can be created by running the
Serviceguard command "cmmakepkg -m sg/multi_node_all -m
<toolkit module name> [-t <config file>]".
For the OC MNP:
----------package_name
Set to any name desired for the OC MNP.
package_type
Set by default to multi_node
package_description
Set by default to "SGeRAC Toolkit Oracle Clusterware package"
node_name
Specify the names for the nodes that the OC MNP will run on.
auto_run
Set to yes or no depending on whether the OC MNP is to be started on
cluster join or on demand.
local_lan_failover_allowed
Set by default to yes to allow cluster to switch LANs locally in the event
of a failure.
node_fail_fast_enabled
Set by default to no.
96
run_script_timeout, halt_script_timeout
Default value is 600 seconds for a 4 node cluster. This value is suggested
as an initial value. It may need to be tuned for your environment.
script_log_file
Set by default to "$SGRUN/log/$SG_PACKAGE.log"
TKIT_DIR
Set to the OC MNP working directory. After the cmapplyconf command, the OC
MNP configuration file oc.conf will be created in this directory. If the
oc.conf file already exists in the directory then all the configuration
parameters will be overwritten and the original oc.conf file will be backed
up in oc.conf.old.
TKIT_SCRIPT_DIR
Set to the OC MNP script files directory. The default value is the OC
MNP script files installation directory:
/opt/cmcluster/SGeRAC/toolkit/crsp.
ORA_CRS_HOME, CHECK_INTERVAL, MAINTENANCE_FLAG
These parameters can be set in the cmmakepkg command if the Toolkit
configuration file is given with the -t option. Otherwise set them based on
the description in E-2 to fit your Oracle environment.
vgchange_cmd, vg
When SLVM or ASM over SLVM is used for Oracle Clusterwaer storage, specify
the corresponding SLVM Volume Group names and set activation to shared mode.
- set vgchange_cmd to "vgchange -a s"
- specify the name(s) of the Shared Volume Groups in vg[0], vg[1]....
Note:
Do not specify SLVM Volume groups used by RAC databases here.

When using ASM over SLVM for RAC database storage, specify the SLVM
Volume Group of member disks in the ASM DG MNP package configuration
file.
cvm_activation_cmd, cvm_dg
If CVM is used for Oracle Clusterware storage management and the CVM disk
group activation and deactivation are to be handled in the package control
file:
- set cvm_activation_cmd to
"vxdg -g \$Disk group set activation=sharedwrite"
- specify the name(s) of the CVM Disk Groups in cvm_dg[0], cvm_dg[1]...
cluster_interconnect_subnet
Refer to the SGeRAC manual for the steps to configure monitoring of the
Oracle Clusterware heartbeat subnet.
service_name
Set by default to crsp_monitor
service_cmd
Set by default to "$SGCONF/scripts/sgerac/erac_tk_oc.sh oc_check"
service_restart
Set by default to none
service_fail_fast_enabled
Set by default to no
service_halt_timeout
Set by default to 300
dependency_name, dependency_condition, dependency_location
If CVM or CFS is used for managing the storage of the Oracle Clusterware,
and Serviceguard Disk Group (DG) MNP and Mount Point (MP) MNP are used to
handle the disk group and file system mount point, configure a dependency
for the corresponding DG MNP (for CVM) or MP MNP (for CFS).
For example, for the package using CVM:
DEPENDENCY_NAME
DG-MNP-name
DG-MNP-PKG=UP
DEPENDENCY_LOCATION
SAME_NODE
For the package using CFS:
DEPENDENCY_NAME
DEPENDENCY_LOCATION
MP-MNP-name
MP-MNP-PKG=UP
SAME_NODE
Note: For modular style CFS DG-MP package, as a dependency modular style CFS
DG-MP MNP must be mentioned in OC MNP configuration file.
For the ASMDG MNP:
-----------Note: If ASM over SLVM is being used for the RAC database, it is recommended
to use the new ASMDG package to manage the ASM disk group.
package_name
Set to any name desired for the ASMDG MNP.
package_type
Set by default to multi_node.
97
package_description
Set by default to "SGeRAC Toolkit Oracle ASMDG package"
node_name
Specify the names for the nodes that the ASMDG MNP will run on.
auto_run
Set to yes or no depending on whether the ASMDG MNP is to be started on
Set by default to yes to allow the cluster to switch LANs locally in the
event of a failure.
script_log_file
TKIT_DIR
Set to the ASMDG MNP working directory. After
ASMDG MNP configuration file asm_dg.conf will
If The asm_dg.conf file already exists in the
configuration parameters will be overwritten,
file will be backed up in asm_dg.conf.old.
the cmapplyconf command, the

be created in this directory.
directory then all the
and the original asm_dg.conf
TKIT_SCRIPT_DIR
Set to the ASMDG MNP script files directory. The default value is the ASMDG
MNP script files installation directory:
/opt/cmcluster/SGeRAC/toolkit/asmp
ORACLE_HOME, CHECK_INTERVAL...MAINTENANCE_FLAG
configuration file is given with -t option. Otherwise set them based on the
description in E-2 to fit your Oracle environment.
ORA_CRS_HOME, OC_TKIT_DIR
It is not required to set these values, the cmapplyconf command will
automatically set them at package configuration time based on the setting
in the OC MNP. Set by default to "<set by cmapplyconf>". Refer to E-2 for
the descriptions.
vgchange_cmd, vg
- set vgchange_cmd to "vgchange -a s". When using ASM over HP-UX raw disks, ignore this step.
- specify the name(s) of the Shared Volume Groups in vg[0], vg[1]...
When using ASM over HP-UX raw disks, ignore this step.
service_name
Set by default to asmdg_monitor, if multiple ASMDG MNPs are configured, you
need to set a different service_name for each ASMDG MNP.
service_cmd
Set by default to "$SGCONF/scripts/sgerac/erac_tk_asmdg.sh asmdg_check"
service_restart
Default value is 300
Configure a dependency on the OC MNP.
For example,
DEPENDENCY_NAME
DEPENDENCY_LOCATION
OC-MNP-name
OC-MNP-PKG=UP
SAME_NODE
For the RAC MNP:

-----------package_name
Set to any name desired for the RAC MNP.
package_type
Set by default to multi_node.
package_description
Set by default to "SGeRAC Toolkit RAC Instance package"
node_name
Specify the names for the nodes that the RAC MNP will run on.
auto_run
98
Set to yes or no depending on whether the RAC MNP is to be started on

Set by default to yes to allow the cluster to switch LANs locally in the
event of a failure.
script_log_file
TKIT_DIR
Set to the RAC MNP working directory. After the cmapplyconf command, the
RAC MNP configuration file rac_dbi.conf will be created in this directory.
If the rac_dbi.conf file already exists in the directory then all the
configuration parameters will be overwritten, and the original rac_dbi.conf
file will be backed up in rac_dbi.conf.old.
TKIT_SCRIPT_DIR
Set to the RAC MNP script files directory. The default value is the RAC MNP
script files installation directory:
/opt/cmcluster/SGeRAC/toolkit/racp
ORACLE_HOME, CHECK_INTERVAL...MAINTENANCE_FLAG
configuration file is given with -t option. Otherwise set them based on the
description in E-2 to fit your Oracle environment.
ORA_CRS_HOME, OC_TKIT_DIR
It is not necessary to set the values, the cmapplyconf command will
automatically set them at package configuration time based on the setting in
the OC MNP. Set by default to "<set by cmapplyconf>". Refer to E-2 for the
descriptions.
vgchange_cmd, vg
If SLVM logical volumes are used for the Oracle RAC database storage
management, in the package control file:
- set vgchange_cmd to "vgchange -a s"
- specify the name(s) of the Shared Volume Groups in vg[0], vg[1]...
cvm_activation_cmd, cvm_dg
If CVM is used for the RAC database storage management and the CVM disk
script:
- set cvm_activation_cmd to
- specify the name(s) of the CVM Disk Groups in cvm_dg[0],
cvm_dg[1]...
cluster_interconnect_subnet
Set this parameter if you want to monitor the Oracle RAC interconnect
network.
service_name
Set by default to racp_monitor, if multiple RAC MNPs are configured, need
to set different service_name for each RAC MNP.
service_cmd
Set by default to "$SGCONF/scripts/sgerac/erac_tk_rac.sh rac_check"
service_restart
Default value is 300
Configure a dependency on the OC MNP. If CVM or CFS is used for managing
the database files, and Serviceguard Disk Group (DG) MNP and Mount Point
(MP) MNP are used to handle the disk group and file system mount point,
add one more dependency for the corresponding DG MNP (for CVM) or MP MNP
(for CFS).
DEPENDENCY_NAME
OC-MNP-name
OC-MNP-PKG=UP
DEPENDENCY_LOCATION
SAME_NODE
DEPENDENCY_NAME
DEPENDENCY_LOCATION
DEPENDENCY_NAME
DG-MNP-name
DG-MNP-PKG=UP
SAME_NODE
OC-MNP-name
99
DEPENDENCY_LOCATION
OC-MNP-PKG=UP
SAME_NODE
DEPENDENCY_NAME
MP-MNP-name
MP-MNP-PKG=UP
DEPENDENCY_LOCATION
SAME_NODE
Note: For modular style CFS DG-MP package, as a dependency OC MNP and
modular style CFS DG-MP MNP must be mentioned in RAC MNP
configuration file. When ASMDG package is configured:
DEPENDENCY_NAME
DEPENDENCY_LOCATION
ASMDG-MNP-name
ASMDG-MNP-PKG=UP
SAME_NODE
Note: When ASMDG MNP is configured, make sure you configure the dependency
on the ASMDG MNP which is managing the disk group of the current RAC MNP.
Since ASMDG MNP is already configured with a dependency on OC MNP, there
is no need to configure a dependency on OC MNP for this RAC MNP.
E-1-2. Legacy Package Configuration File Parameters:
For legacy packages, the package configuration file template can be created
by running the Serviceguard command "cmmakepkg -p".
For the OC MNP:
----------PACKAGE_NAME
Set to any name desired for the OC MNP.
PACKAGE_TYPE
Set to MULTI_NODE.
FAILOVER_POLICY, FAILBACK_POLICY
Comment out.
NODE_NAME
Specify the names for the nodes that the OC MNP will run on.
AUTO_RUN
Set to YES or NO depending on whether the OC MNP is to be started on
Set by default to YES to allow cluster to switch LANs locally in the event
of a failure.
Set by default to NO.
RUN_SCRIPT, HALT_SCRIPT
Set to the package control script.
RUN_SCRIPT_TIMEOUT, HALT_SCRIPT_TIMEOUT
STORAGE_GROUP
If the Oracle Clusterware registry and vote devices are stored in a CVM
disk group, specify it using this parameter.
DEPENDENCY_NAME, DEPENDENCY_CONDITION, DEPENDENCY_LOCATION
If CVM or CFS is used for managing the storage of the Oracle Clusterware,
and Serviceguard Disk Group (DG) MNP and Mount Point (MP) MNP are used to
handle the disk group and file system mount point, configure a dependency
for the corresponding DG MNP (for CVM) or MP MNP (for CFS).
DEPENDENCY_NAME
DG-MNP-name
DG-MNP-PKG=UP
DEPENDENCY_LOCATION
SAME_NODE
DEPENDENCY_NAME
DEPENDENCY_LOCATION
MP-MNP-name
MP-MNP-PKG=UP
SAME_NODE
SERVICE_NAME
Specify a single SERVICE_NAME, corresponding to the service definition in
the control script. This service invokes Toolkit script "toolkit_oc.sh
check".
SERVICE_HALT_TIMEOUT
For the ASMDG MNP:

-----------Note: If ASM over SLVM is being used for RAC database, it is recommended to
use the new ASM DG package to manage the ASM disk group.
PACKAGE_NAME
Set to any name desired for the ASMDG MNP.
PACKAGE_TYPE
Set to MULTI_NODE.
Comment out.
NODE_NAME
Specify the names for the nodes that the ASMDG MNP will run on.
AUTO_RUN
Set to YES or NO depending on whether the ASMDG MNP is to be started on
of a failure.
Configure a dependency on the OC MNP.
DEPENDENCY_NAME
OC-MNP-name
OC-MNP-PKG=UP
DEPENDENCY_LOCATION
SAME_NODE
SERVICE_NAME
the control script. This service invokes Toolkit script "toolkit_asmdg.sh
check".
For the RAC MNP:

-----------PACKAGE_NAME
Set to any name desired for the RAC MNP.
PACKAGE_TYPE
Set to MULTI_NODE.
Comment out.
NODE_NAME
Specify the names for the nodes that the RAC MNP will run on.
AUTO_RUN
Set to YES or NO depending on whether the RAC MNP is to be started on
of a failure.
STORAGE_GROUP
If the database is stored in a CVM disk group, specify it using this
parameter.
Configure a dependency on the OC MNP. If CVM or CFS is used for managing
the database files, and Serviceguard Disk Group (DG) MNP and Mount Point
(MP) MNP are used to handle the disk group and file system mount point,
add one more dependency for the corresponding DG MNP (for CVM) or MP MNP
(for CFS).
For example, for a package using CVM:
DEPENDENCY_NAME
OC-MNP-name
OC-MNP-PKG=UP
DEPENDENCY_LOCATION
SAME_NODE
101
DEPENDENCY_NAME
DEPENDENCY_LOCATION
DG-MNP-name
DG-MNP-PKG=UP
SAME_NODE
For a package using CFS:

DEPENDENCY_NAME
DEPENDENCY_LOCATION
OC-MNP-name
OC-MNP-PKG=UP
SAME_NODE
DEPENDENCY_NAME
DEPENDENCY_LOCATION
MP-MNP-name
MP-MNP-PKG=UP
SAME_NODE
When ASMDG package is configured:

DEPENDENCY_NAME
DEPENDENCY_LOCATION
ASMDG-MNP-name
ASMDG-MNP-PKG=UP
SAME_NODE
Note: When ASMDG MNP is configured, make sure you configure the dependency
on the ASMDG MNP which is managing the disk group of the current RAC MNP.
Since ASMDG MNP is already configured with a dependency on OC MNP, there
is no need to configure a dependency on OC MNP for this RAC MNP.
SERVICE_NAME
the control script. This service invokes Toolkit script "toolkit_dbi.sh
check".
E-1-3. The Legacy Package Control Script Parameters

The package control script template can be created by running the Serviceguard
command "cmmakepkg -s" for a legacy package.
For the OC MNP:
----------When SLVM or ASM over SLVM is used for Oracle Clusterwaer storage, specify
the corresponding SLVM Volume Group names and set activation to shared mode.
- set VGCHANGE to "vgchange -a s"
- specify the name(s) of the Shared Volume Groups in vg[0], vg[1]....
Note:
Do not specify SLVM Volume groups used by RAC databases here.

When using ASM over SLVM for RAC database storage, specify the SLVM
Volume Group of member disks in the ASM DG MNP package configuration
file.
If CVM is used for Oracle Clusterware storage management and the CVM disk
file:
- set CVM_ACTIVATION_CMD to
- specify the name(s) of the CVM Disk Groups in CVM_DG[0], CVM_DG[1]...
Configure one package service:
- set SERVICE_NAME[0] to the name of service specified in the ASCII
configuration file
- set SERVICE_CMD[0] to "<OC MNP working directory>/toolkit_oc.sh
check"
- set SERVICE_RESTART[0] to ""
In the function customer_defined_run_cmds:
- start Oracle Clusterware using the command:
<OC MNP working directory>/toolkit_oc.sh start
In the function customer_defined_halt_cmds:
- stop Oracle Clusterware using the command:
<OC MNP working directory>/toolkit_oc.sh stop
For the RAC MNP:

-----------If SLVM volumes are used for the Oracle RAC database storage management,
in the package control file:
- set VGCHANGE to "vgchange -a s"
- specify the name(s) of the Shared Volume Groups in VG[0], VG[1]...
If CVM is used for the RAC database storage management and the CVM disk group
activation and deactivation are to be handled in the package control file:
- set CVM_ACTIVATION_CMD to
"vxdg -g \$DiskGroup set activation=sharedwrite"
- specify the name(s) of the CVM Disk Groups in CVM_DG[0], CVM_DG[1]...
configuration file
- set SERVICE_CMD[0]
to "<RAC MNP working directory>/toolkit_dbi.sh check"
- start the RAC instance using the command:
<RAC MNP working directory>/toolkit_dbi.sh start
- stop the RAC instance using the command:
<RAC MNP working directory>/toolkit_dbi.sh stop
For the ASMDG MNP:

------------ set VGCHANGE to "vgchange -a s" .
- specify the name(s) of the ASM Shared Volume Groups in VG[0], VG[1]...
configuration file
- set SERVICE_CMD[0]
to "<ASMDG MNP working directory>/toolkit_asmdg.sh check"
- mount the ASM disk group using the command:
<ASMDG MNP working directory>/toolkit_asmdg.sh start
- unmount the ASM disk group using the command:
<ASMDG MNP working directory>/toolkit_asmdg.sh stop
E-2. Toolkit Configuration File Parameters

The SGeRAC Toolkit configuration file
/opt/cmcluster/SGeRAC/toolkit/crsp/oc.conf
(for the OC MNP), /opt/cmcluster/SGeRAC/toolkit/racp/rac_dbi.conf (for the
RAC MNP) and /opt/cmcluster/SGeRAC/toolkit/asmp/asm_dg.conf (for the ASM MNP)
can be edited and tested before configuring the OC MNP and RAC MNP.
For the OC MNP (oc.conf):
--------------------ORA_CRS_HOME
Oracle Clusterware home directory.
CHECK_INTERVAL
The time interval in seconds (default 60) between consecutive checks of
Oracle Clusterware status by the MNP.
MAINTENANCE_FLAG
OC MNP maintenance mode: yes or no(default). This variable will enable or
disable maintenance mode for the OC MNP. When the OC MNP MAINTENANCE_FLAG
is "yes", the MAINTENANCE_FLAG in the RAC MNP must be set to "yes" as well
and other values are not suppported.
When the OC MNP needs to be maintained on one node, this parameter must be
set to yes and a file "<OC MNP working directory>/oc.debug" needs to be
created manually on that node. During this maintenance period the Oracle
Clusterware process checking is paused. Even if Oracle Clusterware is
brought down on the local node, the OC MNP on that node will not be halted.
When the OC MNP is in maintenance mode and the RAC MNP maintenance mode is
enabled, the corresponding RAC MNP on the same node will be in maintenance
mode as well regardless of the presence of its maintenance debug file.
To continue monitoring and come back from the maintenance mode on the local
node, you must remove the "oc.debug" file on that node. It is the
user's responsibility to ensure that the Oracle Clusterware is properly
running after the maintenance phase.
For the ASMDG MNP (asm_dg.conf):

--------------------------ORA_CRS_HOME
ORACLE_HOME
Oracle database home directory.
ORACLE_USER
HP-UX user name corresponding to the owner of Oracle software.
ASM_HOME
The home directory of Oracle where ASM is installed.
KILL_ASM_FOREGROUNDS
This parameter specifies if any ASM foreground processes that have file
descriptors open on the dismounted disk group volumes need to be killed or
Support for the SGeRAC Toolkit 103
not. This parameter can be set to either Yes or No(default Yes)

ASM_DISK GROUP
ASM Disk groups used by the database instance
ASM_VOLUME_GROUP:
Volume groups used in the ASM disk groups for this database instance.
CHECK_INTERVAL
Time interval in seconds (default 60) between consecutive checks of ASM
disk group status by the MNP.
MAINTENANCE_FLAG
ASMDG MNP maintenance mode: yes or no(default). This variable will enable
or disable maintenance mode for the ASMDG MNP. When the ASMDG MNP needs to
be maintained on one node, then this parameter must be set to yes and a
file "<ASMDG MNP working directory>/asm_dg.debug" must be created manually
on that node. During this maintenance period Oracle ASM disk group checking
is paused. Even if the Oracle ASM disk group is brought down on the local
node, the ASMDG MNP on that node will not be halted. To continue monitoring
and come back from the maintenance mode on one node, you must remove the
"asm_dg.debug" file on that node. It is the user's responsibility to ensure
that the Oracle ASM disk group is mounted properly after the maintenance
phase.
OC_TKIT_DIR
Set to OC MNP working directory. When the MAINTENANCE_FLAG is set to yes,
ASMDG MNP uses this parameter to check the OC MNP's maintenance status: If
the OC MNP MAINTENANCE_FLAG is set to yes and the oc.debug file is in the
OC_TKIT_DIR directory, the ASMDG MNP knows the OC MNP on the same node is
in maintenance mode. In this case, because of the dependency on the OC MNP,
the ASMDG MNP will go into maintenance mode as well regardless of the
presence of its debug file.
If MAINTENANCE_FLAG is no, OC_TKIT_DIR is not required, and ASMDG MNP will
not check the OC MNP maintenance status.
For the RAC MNP (rac_dbi.conf):

--------------------------ORACLE_HOME
Oracle database home directory.
ORA_CRS_HOME
ORACLE_USER
HP-UX user name corresponding to the owner of Oracle software.
ORACLE_DBNAME
The database name.
START_MODE
Mode of instance start: mount, nomount or open (default).
STOP_MODE
Mode of instance stop: normal, transactional, immediate or abort (default).
If the RAC MNP fails, this Toolkit will use the "abort" option to halt the
RAC instance for a quick package shutdown regardless of the STOP_MODE value.
CHECK_INTERVAL
Time interval in seconds (default 60) between consecutive checks of RAC
instance status by the MNP.
ORA_RESTART_TIMEOUT
Time interval in seconds (default 180) for the Toolkit script to wait for
Oracle to restart an instance which is terminated prematurely before
exiting the package.
ORA_SHUTDOWN_TIMEOUT
Time interval in seconds (default 120) for the Toolkit script to wait for
the Oracle abort to complete before killing the Oracle background
processes. The ORA_SHUTDOWN_TIMEOUT variable is used to protect against
a worst-case scenario where a hung database prevents the package halt
script from completing. The value of ORA_SHUTDOWN_TIMEOUT must be less
than the HALT_SCRIPT_TIMEOUT value set in the rac package configuration
file.
MAINTENANCE_FLAG
RAC MNP maintenance mode: yes or no (default). This variable will enable or
disable maintenance mode for the RAC MNP.
When the RAC MNP needs to be maintained on one node, this parameter must
be set to yes and a file "<RAC MNP working directory>/rac.debug" needs to
be created manually on that node. During this maintenance period the Oracle
RAC instance's process checking is paused. Even if the Oracle RAC instance
is brought down on the local node, the RAC MNP on that node will not be
halted.
To continue monitoring and come back from the maintenance mode on one
node, you should remove the "rac.debug" file on that node. It is the
user's responsibility to ensure that the Oracle RAC instance is properly
running after the maintenance phase.
OC_TKIT_DIR
Set to the OC MNP working directory. When MAINTENANCE_FLAG is yes, the RAC
MNP uses this parameter to check the OC MNP maintenance status: If the OC
MNP MAINTENANCE_FLAG is set to yes and oc.debug is in the OC_TKIT_DIR
directory, the RAC MNP knows the OC MNP on the same node is in maintenance
mode. In this case, because of the dependency on the OC MNP, the RAC MNP
will go into maintenance mode as well regardless of the presence of its
debug file.
If the MAINTENANCE_FLAG is set to no, OC_TKIT_DIR is not required, and the
RAC MNP will not check the OC MNP's maintenance status.
ASM_TKIT_DIR
Note: this parameter should be set only if the new ASM DG MNP is being used.
Set this to the ASM MNP working directory. When the MAINTENANCE_FLAG is
yes, it is used to check the ASMDG MNP maintenance status.
F. SGeRAC Toolkit configuration Procedures

To use the SGeRAC Toolkit in an environment running with SGeRAC and Oracle 10g
R2 or later version/revision of RAC, you must disable the automatic start
feature for the Oracle Clusterware and the RAC instances which are configured
and managed in the Toolkit MNP. Since both modular and legacy package formats
are supported in SGeRAC A.11.19 and later, the guidelines for configuring and using the
SGeRAC Toolkit in the combined stack environment are described in several
steps.
The sequence for the modular package setup are F-1, F-2, F-4, F-5, F-8 and
F-10. The sequence for the legacy package setup are F-1, F-3, F-4, F-9 and
F-10.
The steps are as follows:
F-1. SGeRAC Toolkit pre-setup [For both modular and legacy packages]
1. Install and configure the SGeRAC cluster. Refer to the HP manual
"Managing Serviceguard" for more details.
2. Prepare storage for Oracle Clusterware and RAC database. Refer to HP
manual "Using Serviceguard Extension for RAC" for more details. For the
configuration of ASM over SLVM logical volumes, refer to HP whitepaper
"Support of Oracle RAC ASM with SGeRAC" for more information.
3. Install and configure Oracle Clusterware and RAC database instance. Refer
to the Oracle Real Application Cluster Installation documents and "Oracle
Real Application Clusters Administrator's Guide" for more details.
4. On each node of the cluster, disable the automatic startup of the Oracle
Clusterware at boot time.
Login as root and enter:
: $ORA_CRS_HOME/bin/crsctl disable crs
5. On one node of the cluster, disable the Oracle RAC database and instances
from being started automatically by the Oracle Clusterware.
Login as the Oracle administrator and run the following command to set the
database management policy to manual.
For Oracle 10g:
: $ORACLE_HOME/bin/srvctl modify database -d <dbname> -y manual
For Oracle 11g:
: $ORACLE_HOME/bin/srvctl modify database -d <dbname> -y MANUAL
F-2. OC MNP creation procedures [For Modular Package]:
1. On one node of the cluster, create an OC MNP working directory under
/etc/cmcluster.
The following step requires root privilege.
: mkdir /etc/cmcluster/OCMNP-Dir
2. Go to step 3 if you don't want to test the oc.conf file before configuring
OC MNP. Otherwise copy the OC MNP configuration file located in the
Toolkit directory /opt/cmcluster/SGeRAC/toolkit/crsp to the OC MNP working
directory. Then edit and test the configuration file oc.conf on one node
based on the description in E-2 to fit your Oracle environment.
All the following steps require root privilege.
: cd /etc/cmcluster/YourOwn-OCMNP-Dir
: cp /opt/cmcluster/SGeRAC/toolkit/crsp/oc.conf .
3. Generate the OC MNP configuration file and edit the file based on the
description in E-1-1. Then configure the OC MNP.
If oc.conf is configured and tested in step 2, use the following command
to create the package configuration file:
: cmmakepkg -m sg/multi_node_all -m sgerac/erac_tk_oc -t oc.conf
pkgConfigFile.
Otherwise, create the package configuration file and set the Oracle
Clusterware parameters in this file directly:

: cmmakepkg -m sg/multi_node_all -m sgerac/erac_tk_oc pkgConfigFile
Edit the package template files based on the description in E-1-1.
4. Now apply the package configuration file:
: cmapplyconf -P pkgConfigFile
F-3. OC MNP creation procedures [For Legacy packages]:

1. On one node of the cluster, create an OC MNP working directory under
/etc/cmcluster and copy the files in the Toolkit directory
/opt/cmcluster/SGeRAC/toolkit/crsp.
All the following steps require root privilege.
: mkdir /etc/cmcluster/YourOwn-OCMNP-Dir
: cd /etc/cmcluster/OCMNP-Dir
: cp /opt/cmcluster/SGeRAC/toolkit/crsp/* .
2. Edit the configuration file oc.conf based on the description in E-2 to
suit your Oracle environment.
3. Generate the package configuration file and control script for the OC MNP.
: cmmakepkg -p pkgConfigFile
: cmmakepkg -s pkgControlScript
Edit the package template files based on the description in E-1-2 and E-1-3
and make sure that the pkgControlScript is executable.
4. Create the OC MNP working directory on all other participating nodes.
Distribute the edited package configuration file, package control script,
and the Toolkit scripts created in steps 2 and 3 to all nodes. Then apply
the package configuration file from one node:
F-4. OC MNP startup procedures [For both Modular and Legacy Packages]:
1. On each node of the cluster, halt the Oracle Clusterware if it is running.
: $ORA_CRS_HOME/bin/crsctl stop crs
2. On one node of the cluster, start the OC MNP via cmrunpkg.
: cmrunpkg <OCMNP-package-name>
Use cmviewcl to check the package status.
OC MNP configured in the cluster.
There should be only one
3. After the package is up and running, verify that the Oracle Clusterware is
running on each node of the cluster.
On each node, enter:
: $ORA_CRS_HOME/bin/crsctl check crs
For
CSS
CRS
EVM
Oracle 10g, messages like the following should be seen:

appears healthy
appears healthy
appears healthy
For Oracle 11g R1, messages like the following should be seen:
Cluster Synchronization Services appears healthy
Cluster Ready Services appears healthy
Event Manager appears healthy
For Oracle 11g R2, messages like the following should be seen:
CRS-4638: Oracle High Availability Services is online
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
The RAC instances should not be running.
Note: Steps F-5 to F-7 are required only if you are using ASM over SLVM for RAC
database and if you are planning to use the ASMDG package to manage your ASM
disk group.
F-5. ASMDG creation procedures [For Modular Packages]:

1. On one node of the cluster, create an ASMDG MNP working directory under
/etc/cmcluster.
: mkdir /etc/cmcluster/ASMDGMNP-Dir
2. Go to step 3 if you don't want to test the asm_dg.conf before configuring
the ASMDG MNP. Otherwise copy the ASMDG MNP configuration file located in
the Toolkit directory /opt/cmcluster/SGeRAC/toolkit/asmp to the ASMDG MNP
working directory. Then edit and test the configuration file asm_dg.conf
based on the description in E-2 to suit your Oracle environment.
: cd /etc/cmcluster/ASMDGMNP-Dir
: cp /opt/cmcluster/SGeRAC/toolkit/asmp/asm_dg.conf .
3. Generate the package configuration file for the ASMDG MNP and edit the
file based on the description in E-1. Then configure ASMDG MNP.
If asm_dg.conf is configured and tested in step 2, use the following
command to create the package configuration file:
: cmmakepkg -m sg/multi_node_all -m sgerac/erac_tk_asmdg -t asm_dg.conf
pkgConfigFile
: cmmakepkg -m sg/multi_node_all -m sgerac/erac_tk_asmdg pkgConfigFile
Edit the package template files based on the description in E-1.
F-6. ASMDG MNP creation procedures [For Legacy Packages]:

1. On one node of the cluster, create the ASMDG MNP working directory
under /etc/cmcluster and copy over the files in the Toolkit
directory /opt/cmcluster/SGeRAC/toolkit/asmp.
: mkdir /etc/cmcluster/YourOwn-ASMDGMNP-Dir
: cd /etc/cmcluster/YourOwn-ASMDGMNP-Dir
: cp /opt/cmcluster/SGeRAC/toolkit/asmp/* .
2. Edit the configuration file asm_dg.conf based on the description in E-2
to suit your Oracle environment.
3. Generate the package configuration file and control scripts for the
ASMDG MNP.
Edit the package template files based on the description in E-1-2 and
E-1-3, and make sure that the pkgControlScript is executable.
4. Create the ASMDG MNP working directory on all other participating nodes.
Distribute the edited package configuration file, package control
script, and the Toolkit scripts created in steps 2 and 3 to all nodes. Now
apply the package configuration file from one node :
F-7. Oracle ASMDG MNP startup procedures: [For both Modular and Legacy packages]
1. On one node of the cluster, start the ASMDG MNP via cmrunpkg.
: cmrunpkg Your-ASMDGMNP-Name
2. After the package is up and running, verify that the ASM disk group is
mounted.
On one node of the cluster, login as the Oracle administrator and enter:
:$ORACLE_HOME/bin/asmcmd lsdg
Messages like the following should be seen:
MOUNTED -
NORMAL - <DG_NAME>
NOTE: To configure another ASMDG MNP package to manage the ASM disk group
used by a different RAC Database, repeat the steps in F-6 and F-7.
F-8. RAC MNP creation procedures [For Modular Package]:

1. On one node of the cluster, create a RAC MNP working directory under
/etc/cmcluster.
: mkdir /etc/cmcluster/YourOwn-RACMNP-Dir
2. Go to step 3 if you don't want to test the rac_dbi.conf before configuring
the RAC MNP. Otherwise copy the RAC MNP configuration file located in the
Toolkit directory /opt/cmcluster/SGeRAC/toolkit/racp to the RAC MNP
working directory. Then edit and test the configuration file rac_dbi.conf
based on the description in E-2 to suit your Oracle environment.
: cd /etc/cmcluster/YourOwn-RACMNP-Dir
: cp /opt/cmcluster/SGeRAC/toolkit/racp/rac_dbi.conf .
3. Generate the package configuration file for the RAC MNP.
If rac_dbi.conf is configured and tested in step 2, use the following
command to create the package configuration file:
: cmmakepkg -m sg/multi_node_all -m sgerac/erac_tk_rac -t rac_dbi.conf
pkgConfigFile
: cmmakepkg -m sg/multi_node_all -m sgerac/erac_tk_rac pkgConfigFile
Edit the package template files based on the description in E-1.

F-9. RAC MNP creation procedures [For Legacy Packages]:

1. On one node of the cluster, create a RAC MNP working directory
under /etc/cmcluster and copy over the files in the Toolkit
directory /opt/cmcluster/SGeRAC/toolkit/racp.
: mkdir /etc/cmcluster/YourOwn-RACMNP-Dir
: cd /etc/cmcluster/YourOwn-RACMNP-Dir
: cp /opt/cmcluster/SGeRAC/toolkit/racp/* .
2. Edit the configuration file rac_dbi.conf based on the description in E-2
to suit your Oracle environment.
3. Generate the package configuration file and control scripts for the RAC
MNP.
Edit the package template files based on the description in E-1-2 and
E-1-3, and make sure that the pkgControlScript is executable.
4. Create the RAC MNP working directory on all other participating nodes.
Distribute the edited package configuration file, package control
script, and the Toolkit scripts created in steps 2 and 3 to all nodes. Now
apply the package configuration file from one node :
F-10 Oracle RAC MNP startup procedures: [For both Modular and Legacy packages]
1. On one node of the cluster, start the RAC MNP via cmrunpkg.
: cmrunpkg Your-RACMNP-Name
2. After the package is up and running, verify that the RAC instance is
running.
On one node of the cluster, login as the Oracle administrator and enter:
:$ORACLE_HOME/bin/srvctl status instance -d $databaseName -i $instanceName
Messages like the following should be seen:
Instance <InstanceName> is running on node <NodeName>
3. If more than one RAC database is configured in the cluster and the RAC
instances are to be managed in the RAC MNP, repeat the steps in F-9 and
steps 1 and 2 in F-10 for each RAC database.
G. SGeRAC Toolkit Administration

The Toolkit OC MNP and RAC MNP and ASMDG MNP are administrated using
Serviceguard package commands. To start the MNP when the SG/SGeRAC cluster
is up and running, use cmrunpkg or cmmodpkg to start all instances, or use
cmrunpkg or cmmodpkg with the option "-n nodeName" to start a single instance
on a specified node. To halt the MNP, use cmhaltpkg to halt all instances, or
use cmhaltpkg with the option "-n nodeName" to halt a single instance on a
specified node. If the package configuration parameter AUTO_RUN is set to yes,
the MNP will be started automatically when the cluster starts up or when the node
joins the cluster. After the Oracle Clusterware and RAC instances are started in
the MNP packages, users should not manage (halt or restart) Oracle Clusterware and
RAC instances outside of the package, otherwise the package could fail.
If a node is added to the cluster and will run a RAC instance, the OC MNP
and ASMDG MNP and the corresponding RAC MNP must be modified and reapplied
to add the node into the package configuration. If a node running the RAC
instance is deleted from the cluster, the OC MNP and ASMDG MNP and the
corresponding RAC MNP must be modified and reapplied to delete the node
from the package configuration.
If TKIT_SCRIPT_DIR is not set to default value, copy the new OC MNP, ASMDG MNP
and RAC MNP scripts to the TKIT_SCRIPT_DIR directory from the respective
location of /opt/cmcluster/SGeRAC/toolkit/ when SGeRAC upgrade is done or
when an SGeRAC patch is installed.
If a node running the OC MNP and RAC MNP is to be shutdown via command
/etc/shutdown, users should run "cmhaltnode -f" first to cleanly halt the
cluster before issuing the shutdown command.
If the Oracle Clusterware or RAC software is being upgraded then it is
recommended to bring down the OC MNP, ASMDG MNP (if configured) and RAC MNP
packages and start Oracle Clusterware and Database using Oracle commands
crsctl and srvctl. After completion of the upgrade, update (if there are any
changes) the configuration files of the above mentioned packages as needed
and restart the packages.
H. SGeRAC Toolkit Supported Configurations

This version of Toolkit supports the following configurations. The maximum
number of nodes supported in SGeRAC Toolkit depends on the number of nodes
supported by SGeRAC with each storage management configuration. Refer to
"Number of nodes supported in SGeRAC for SLVM, CVM and CFS Matrix" posted on
http://www.hp.com/go/hpux-serviceguard-docs -> Serviceguard Extension for RAC.
-----------------------------------------------------------------Toolkit Multi-Node package | Storage Management

| HP-UX version
-----------------------------------------------------------------Oracle Clusterware
| SLVM raw
| 11i v3
Oracle Clusterware
| ASM over HP-UX raw
| 11i v3
disks (For 11gR2 or later)
Oracle Clusterware
| ASM over SLVM logical | 11i v3
volumes (For 11gR2 or later)
Oracle Clusterware
| CVM raw(version 5.0.1)| 11i v3
Oracle Clusterware
| CFS(version 5..1 SP1) | 11i v3
RAC Instance
| CFS(version 5.1 SP1)) | 11i v3
RAC Instance
| ASM over SLVM logical | 11i v2
| volumes
RAC Instance
| ASM over HP-UX raw
| 11i v3
disks
------------------------------------------------------------------
I. SGeRAC Toolkit Q & A

There are two configuration scenarios in this version of SGeRAC Toolkit.
Question 1:
In some configurations, the customer may not want to configure a RAC MNP
package, but instead wants Oracle Clusterware to control the startup of the
Oracle RAC instances, is this supported by the SGeRAC Toolkit?
Answer 1:
Yes, in this case, the customer can follow the setup steps in section F with
exception of steps 5, 13-18. and configure only the OC MNP using the SGeRAC
Toolkit, then the Istartup of OC MNP will bring up the Oracle RAC instance
also.
Question 2:
How to start a user defined Oracle Service in an SGeRAC Toolkit environment?
Answer 2:
Oracle automatically creates one database service when the database is
created. For many installations, using the default service is sufficient and
the default service is always started and does not require extra steps to
start. If a user needs more flexibility in the management of the workload
using the database, and creates some additional services, these services can
be started using the SGeRAC Toolkit in 2 ways:
1. The user can configure the OC MNP only and let the Oracle Clusterware
bring up the Oracle Services.
2. If both the OC MNP and RAC MNP are configured and the user has configured
additional services, the user is responsible for starting those additional
services.
J. SGeRAC Toolkit Limitation/Restriction

There are some restrictions and limitations in this release of SGeRAC Toolkit.
1. The SGeRAC Toolkit RAC MNP supports ASM over SLVM and ASM over HP-UX raw
disks/disk array logical units in SGeRAC environment. The Toolkit does
not support single instance using ASM over SLVM in the cluster.
2. The SGeRAC Toolkit OC MNP and RAC MNP do not restart the Oracle Clusterware
or RAC instances. After the RAC MNP is started and the corresponding
database instances are up and running, the RAC instances are under Oracle
Clusterware control.
3. Disk groups that are exclusively used by a RAC database should be managed
in separate ASMDG MNP. If two RAC databases use different ASM disk groups
then those ASM disk groups should not be configured in a single ASMDG MNP.
However, if two RAC databases use the same set of ASM disk groups then
those ASM disk groups can be configured in a single ASMDG MNP and then
both RAC MNP can be configured to have dependencies on the ASMDG MNP.
K. SGeRAC Toolkit Legacy Package to Modular Package Migration

For SGeRAC Toolkit A.11.17 and A.11.18 users, the migration from the SGeRAC
Toolkit legacy packages to modular packages can be done with the following
steps:
1. After backing up the legacy package directory, the user can do a rolling
upgrade one node at a time or can halt the entire cluster to upgrade to
SGeRAC A.11.19 or later version.
2. Shutdown the legacy RAC MNP if a RAC MNP is running.

: cmhaltpkg <legacy RAC MNP>
3. Create a new RAC MNP package working directory on one node, then cd to the
new package directory.
: cd /etc/cmcluster/RACMNP-Dir
4. Use the cmmigratepkg command to migrate the legacy RAC MNP to modular
format.
: cmmigratepkg -p <legacy RAC MNP> -s -o <RAC MNP output>
5. Create a new modular RAC MNP ascii file.
: cmmakepkg -i <RAC MNP output> -m sg/multi_node_all -m <toolkit module>
-t <RAC MNP configuration file> <new RAC MNP ascii file>
6. Edit the new RAC MNP ascii file, set the Toolkit parameters, refer to
section E-2 to verify and configure new modular RAC MNP parameters.
7. Now apply the package configuration file.
: cmapplyconf -P <new RAC MNP ascii file>
8. Shutdown the legacy OC MNP if the OC MNP is running.
: cmhaltpkg <legacy OC MNP>
9. Create a new OC MNP package working directory on one node, then cd to the
new package directory.
: cd /etc/cmcluster/OCMNP-Dir
10. Use the cmmigratepkg command to migrate the legacy OC MNP to modular
format.
: cmmigratepkg -p <legacy OC MNP> -s -o <OC MNP output>
11. Create a new modular OC MNP ascii file.
: cmmakepkg -i <OC MNP output> -m sg/multi_node_all -m <toolkit module>
-t <OC MNP configuration file> <new OC MNP ascii file>
12. Edit the new OC MNP ascii file, set the Toolkit parameters, refer to
section E-2 to verify and configure the new modular OC MNP parameters.
13. Now apply the package configuration file
: cmapplyconf -P <new OC MNP ascii file>
14. You may now start the new OC MNP and RAC MNP in the cluster using the
cmrunpkg or cmmodpkg e commands.
15. After verifying the successful operation of the new OC MNP and RAC MNP
packages, you may delete the legacy packages using the cmdeleteconf
command.
L. Migration of Legacy CFS Disk group and Mount point Packages to Modular
CFS Disk group and Mount point Packages(CFS DG-MP).
Beginning with the SG A.11.20 patch PHSS_41628 and SG CFS A.11.20 patch PHSS_41674,
new modular CFS Disk group and Mount point feature has been introduced. It will
allow to consolidate all disk group and mount point packages for an application
into a single modular package. To migration from legacy CFS Disk group and Mount
point MNPs to modular CFS Disk group and Mount point MNPs (CFS DG-MP) can be done
with the following steps:
1. Create modular CFS DG-MP MNP for Oracle Clusterware storage
: cmmakepkg -m sg/cfs_all /etc/cmcluster/OC-DGMP/OC-DGMP.ascii
2. Edit the OC-DGMP.ascii file with package name,Diskgroup and mount point
and other required package information
For example
cvm_disk_group
< DiskGroup used for Oracle Clusterware storage >
cvm_activation_mode
"< node1 > =sw < node2 > =sw"
cfs_mount_point
< Mount Point location for CRS>
cfs_volume
< CFS volume >
cfs_mount_options
"< node1 > =cluster < node2 > =cluster"
cfs_primary_policy
""
3. Create modular CFS DG-MP MNP for RAC database storage
: cmmakepkg -m sg/cfs_all /etc/cmcluster/RAC-DGMP/RAC-DGMP.ascii
4. Edit the RAC-DGMP.ascii file with package name,Diskgroup and mount point
and other required package information
For example
cvm_disk_group
< DiskGroup used for RAC database storage >
cvm_activation_mode
"< node1 > =sw < node2 > =sw"
cfs_mount_point
< Mount Point location RAC>
cfs_volume
< CFS volume >
cfs_mount_options
"< node1 > =cluster < node2 > =cluster"
cfs_primary_policy
""
5. Take a backup of OC MNP configuration file.
: cmgetconf -p < OC MNP > < backup OC MNP package configuration file >
6. Edit the backup of OC MNP ascii file, remove the existing dependency on
legacy MP and DG package and add modular CFS DG-MP package for Oracle
Clusterware storage as a dependency.
For example
110
OC MNP ascii file with legacy style DG and MP package:

DEPENDENCY_NAME
DG-MNP-name
DG-MNP-PKG=UP
DEPENDENCY_LOCATION
SAME_NODE
DEPENDENCY_NAME
DEPENDENCY_LOCATION
DG-MNP-name
DG-MNP-PKG=UP
SAME_NODE
OC MNP ascii file with modular style CFS DG-MP package :

DEPENDENCY_NAME
OC-DGMP-name
OC-DGMP-PKG=UP
DEPENDENCY_LOCATION
SAME_NODE
7. Take backup of all RAC MNP configuration file.

: cmgetconf -p < RAC MNP> < backup RAC MNP package configuration file >
8. Edit the backup of RAC MNP ascii file, remove the existing dependency on
legacy MP and DG package and add modular CFS DG-MP package for Oracle RAC
database storage as a dependency.
For example
RAC MNP ascii file with legacy style CFS DG and MP package:
DEPENDENCY_NAME
DG-MNP-name
DG-MNP-PKG=UP
DEPENDENCY_LOCATION
SAME_NODE
DEPENDENCY_NAME
DEPENDENCY_LOCATION
DG-MNP-name
DG-MNP-PKG=UP
SAME_NODE
RAC MNP ascii file with modular style CFS DG-MP package :
DEPENDENCY_NAME
RAC-DGMP-name
RAC-DGMP-PKG=UP
DEPENDENCY_LOCATION
SAME_NODE
9. Shutdown all the RAC MNP, if RAC MNPs is running.
: cmhaltpkg < RAC MNP 1 > < RAC MNP 2 > ...
10. Shutdown the OC MNP, if the OC MNP is running.
: cmhaltpkg < OC MNP >
11. Shutdown the legacy Disk group (SG-CFS-DG-id#) and Mount point MNPs (SG-MP-id#)
: cmhaltpkg < SG-MP-id# > < SG-CFS-DG-id# >
12. Delete the OC MNP and all the RAC MNPs from cluster
: cmdeleteconf -p < RAC MNP >
: cmdeleteconf -p < OC MNP >
13. Delete all legacy style Disk group MNPs and Mount Point MNPs from cluster
: cmdeleteconf -p < legacy MP MNP >
: cmdeleteconf -p < legacy DG MNP >
14. Apply and run both modular CFS DG-MP packages for Oracle Clusterware and RAC
database storage created in step number [1] and [3]
: cmapplyconf -P < OC-DGMP-MNP configuration file >
: cmapplyconf -P < RAC-DGMP-MNP configuration file >
: cmrunpkg < OC-DGMP-MNP > < RAC-DGMP-MNP >
15. Apply the updated OC MNP configuration file which was modified in step number [6]
: cmapplyconf -P < backup OC MNP configuration file >
16. Apply the updated RAC MNP configuration file which was modified in step number [8]
: cmapplyconf -P < backup RAC MNP configuration file>
17. You may now start the OC MNP and RAC MNP in the cluster using the
: cmrunpkg < OC MNP > < RAC MNP >
M. SGeRAC Toolkit Adding new ASMDG MNP Package to the existing

configured OC MNP and RAC MNP
For ASM over SLVM:
1. Halt all the RAC MNPs.
2. Halt the OC MNP.
3. Remove all SLVM volume groups which are used for RAC Databases from the
OC MNP configuration file or from OC MNP control script(in the case of a legacy OC MNP)
except for the SLVM volume group used for the OCR/VOTING Disk.
4. Run cmapplyconf on the OC MNP configuration file.
5. Start the OC MNP.
6. Follow the instructions in the section E of this README and configure new ASMDG MNP package
and run cmapplyconf on the ASMDG MNP configuration file.
7. Start ASMDG MNP.
8. Edit the RAC MNP configuration file, and add a dependency on its corresponding ASMDG MNP.
9. Run cmapplyconf on the RAC MNP configuration file.
10.Start the RAC MNP.
ASM over HP-UX raw disks (HP-UX 11i v3 only):
1. Halt all the RAC MNPs.
2. Following the instructions in the section E of this README, configure the new ASMDG MNP
and run cmapplyconf on the ASMDG MNP.
111
3.
4.
5.
6.
Start the ASMDG MNP.

Edit the RAC MNP configuration script, and add a dependency on its corresponding ASMDG MNP.
Run cmapplyconf on the RAC MNP configuration file.
Start the RAC MNP.
N. SGeRAC Toolkit Package Cleanup

1. Shutdown the RAC MNP.
: cmhaltpkg <RAC MNP name>
2. Delete the RAC MNP configuration.
: cmdeleteconf -p <RAC MNP name>
3. Remove the RAC MNP working directory on all nodes
: rm -rf /etc/cmcluster/RACMNP-Dir
NOTE: If more than one
repeated for all
Step number 4 to
If no ASMDG MNPs
RAC MNPs are configured, step number 1 and 3 should be

the RAC MNPs.
6 are required only if you have configured ASMDG MNP.
are configured, please proceed to step number 7
4. Shutdown the ASMDG MNP.

: cmhaltpkg <ASMDG MNP name>
5. Delete the ASMDG MNP configuration.
: cmdeleteconf -p <ASMDG MNP name>
6. Remove the ASMDG MNP working directory on all nodes
: rm -rf /etc/cmcluster/ASMDGMNP-Dir
NOTE: If more than one ASMDG MNPs are configured, step number 4 to 6 should be
repeated for all the ASMDG MNPs.
7. Shutdown the OC MNP.
: cmhaltpkg <OC MNP name>
8. Delete the OC MNP configuration.
: cmdeleteconf -p <OC MNP name>
9. Remove the OC MNP package working directory on all nodes.
: rm -rf /etc/cmcluster/OCMNP-Dir
O. SGeRAC Toolkit Whitepaper Link
HP has published a whitepaper "Use of Serviceguard Extension For RAC Toolkit
with Oracle RAC 10g Release 2 or later" that contains the SGeRAC Toolkit
background and operation information. This whitepaper is posted on
http://www.hp.com/go/hpux-serviceguard-docs -> Serviceguard Extension for RAC.
Please note that this whitepaper has not been updated to include
documentation for the new ASMDG MNP.
There is another white paper "Support of Oracle RAC ASM with SGeRAC" that contains
details on ASM features and how to configure ASM with SGeRAC. This whitepaper is posted
on http://www.hp.com/go/hpux-serviceguard-docs -> Serviceguard Extension for RAC.
Conclusion
Using SGeRAC Toolkit with multi-node packages and simple package dependencies provides a
uniform, intuitive, and easy-to-manage method to perform the following:
Co-ordinate between SGeRAC and Oracle Clusterware
Manage all the storage options supported by SGeRAC -CFS, SLVM, CVM , ASM over SLVM
and ASM over raw device (on HP-UX 11i v3)
Although the concepts of resource dependency and resource aggregation delivered by SGeRAC
with the multi-node package and simple package dependency features are present in some form
or other in other clusterware productsincluding Oracle Clusterwarethe framework provided
by SGeRAC is unique due to the high level of multi-vendor (Oracle, Symantec, HP) and multi-storage
platform (CFS, SLVM, CVM, ASM over SLVM, ASM over raw device) integration it offers.
Additional Documentation on the Web

www.hp.com/go/serviceguardsolutions
www.hp.com/go/sgerac
112
5 Maintenance
This chapter includes information about carrying out routine maintenance on a Real Application
Cluster configuration. Starting with version SGeRAC A.11.17, all log messages from cmgmsd log
to /var/adm/syslog/syslog.log by default. As presented here, these tasks differ in some
details from the similar tasks described in the Managing Serviceguard documentation.
Tasks include:
Reviewing Cluster and Package States with the cmviewcl Command (page 113)
Checking the Cluster Configuration and Components (page 122)
Online Reconfiguration (page 126)
Managing the Shared Storage (page 127)
Removing Serviceguard Extension for RAC from a System (page 130)
Monitoring Hardware (page 131)
Adding Disk Hardware (page 131)
Replacing Disks (page 132)
Replacement of I/O Cards (page 136)
Replacement of LAN Cards (page 136)
Reviewing Cluster and Package States with the cmviewcl Command

A cluster or its component nodes may be in several different states at different points in time. Status
information for clusters, packages, and other cluster elements is shown in the output of the cmviewcl
command and in some displays in Serviceguard Manager. This section explains the meaning of
many of the common conditions the cluster or package may be in.
Information about cluster status is stored in the status database that is maintained on each individual
node in the cluster. You can display information contained in this database by issuing the cmviewcl
command:
# cmviewcl -v
The command when issued with the -v option displays information about the whole cluster. See
the man page for a detailed description of other cmviewcl options.
TIP: Some commands take longer to complete in large configurations. In particular, you can
expect Serviceguards CPU utilization to increase during cmviewcl -v as the number of packages
and services increases.
You can also specify that the output should be formatted as it was in a specific earlier release by
using the -r option indicating the release format you wish. Example:
# cmviewcl -r A.11.16
See the man page for a detailed description of other cmviewcl options.
Types of Cluster and Package States

A cluster or its component nodes may be in several different states at different points in time. The
following sections describe many of the common conditions the cluster or package may be in.
Examples of Cluster and Package States

The following is an example of the output generated shown for the cmviewcl command:
113
CLUSTER
cluster_mo
NODE
minie
STATUS
up
STATUS
up
STATE
running
Quorum_Server_Status:
NAME
STATUS
white
up
STATE
running
Network_Parameters:
INTERFACE
STATUS
PRIMARY
up
PRIMARY
up
STANDBY
up
NODE
mo
PATH
0/0/0/0
0/8/0/0/4/0
0/8/0/0/6/0
STATUS
up
NAME
lan0
lan1
lan3
STATE
running
Quorum_Server_Status:
NAME
STATUS
white
up
STATE
running
Network_Parameters:
INTERFACE
STATUS
PRIMARY
up
PRIMARY
up
STANDBY
up
PATH
0/0/0/0
0/8/0/0/4/0
0/8/0/0/6/0
NAME
lan0
lan1
lan3
MULTI_NODE_PACKAGES
PACKAGE
SG-CFS-pkg
NODE_NAME
minie
STATUS
up
STATUS
up
Script_Parameters:
ITEM
STATUS
Service
up
Service
up
Service
up
Service
up
Service
up
NODE_NAME
mo
NODE_NAME
minie
MAX_RESTARTS
0
5
5
0
0
Maintenance
SYSTEM
yes
RESTARTS
0
0
0
0
0
NAME
SG-CFS-vxconfigd
SG-CFS-sgcvmd
SG-CFS-vxfsckd
SG-CFS-cmvxd
SG-CFS-cmvxpingd
SWITCHING
enabled
MAX_RESTARTS
0
5
5
0
0
STATUS
up
STATUS
up
Dependency_Parameters:
DEPENDENCY_NAME
SG-CFS-pkg
114
AUTO_RUN
enabled
SWITCHING
enabled
STATUS
up
Script_Parameters:
ITEM
STATUS
Service
up
Service
up
Service
up
Service
up
Service
up
PACKAGE
SG-CFS-DG-1
STATE
running
STATE
running
STATE
running
SATISFIED
yes
RESTARTS
0
0
0
0
0
NAME
SG-CFS-vxconfigd
SG-CFS-sgcvmd
SG-CFS-vxfsckd
SG-CFS-cmvxd
SG-CFS-cmvxpingd
AUTO_RUN
enabled
SWITCHING
enabled
SYSTEM
no
NODE_NAME
mo
STATUS
up
DEPENDENCY_NAME
SG-CFS-pkg
PACKAGE
SG-CFS-MP-1
NODE_NAME
minie
STATUS
up
STATUS
up
DEPENDENCY_NAME
SG-CFS-DG-1
NODE_NAME
mo
STATUS
up
DEPENDENCY_NAME
SG-CFS-DG-1
PACKAGE
SG-CFS-MP-2
NODE_NAME
minie
STATUS
up
STATUS
up
DEPENDENCY_NAME
SG-CFS-DG-1
NODE_NAME
mo
STATUS
up
DEPENDENCY_NAME
SG-CFS-DG-1
PACKAGE
SG-CFS-MP-3
NODE_NAME
minie
STATUS
up
STATUS
up
DEPENDENCY_NAME
SG-CFS-DG-1
NODE_NAME
mo
STATUS
up
DEPENDENCY_NAME
SG-CFS-DG-1
STATE
running
SWITCHING
enabled
SATISFIED
yes
STATE
running
STATE
running
AUTO_RUN
enabled
SYSTEM
no
SWITCHING
enabled
SATISFIED
yes
STATE
running
SWITCHING
enabled
SATISFIED
yes
STATE
running
STATE
running
AUTO_RUN
enabled
SYSTEM
no
SWITCHING
enabled
SATISFIED
yes
STATE
running
SWITCHING
enabled
SATISFIED
yes
STATE
running
STATE
running
AUTO_RUN
enabled
SYSTEM
no
SWITCHING
enabled
SATISFIED
yes
STATE
running
SWITCHING
enabled
SATISFIED
yes
Types of Cluster and Package States

A cluster or its component nodes may be in several different states at different points in time. The
following sections describe many of the common conditions the cluster or package may be in.
115
Cluster Status
The status of a cluster may be one of the following:
Up. At least one node has a running cluster daemon, and reconfiguration is not taking place.
Down. No cluster daemons are running on any cluster node.
Starting. The cluster is in the process of determining its active membership. At least one
cluster daemon is running.
Unknown. The node on which the cmviewcl command is issued cannot communicate with
other nodes in the cluster.
Node Status and State

The status of a node is either up (active as a member of the cluster) or down (inactive in the cluster),
depending on whether its cluster daemon is running or not. Note that a node might be down from
the cluster perspective, but still up and running HP-UX.
Serviceguard Manager will display the alerts and status icon for the cluster, node, and package
health.
A node may also be in one of the following states:
Failed. A node never sees itself in this state. Other active members of the cluster will see a
node in this state if that node was in an active cluster, but is no longer, and is not halted.
Reforming. A node is in this state when the cluster is re-forming. The node is currently running
the protocols which ensure that all nodes agree to the new membership of an active cluster.
If agreement is reached, the status database is updated to reflect the new cluster membership.
Running. A node in this state has completed all required activity for the last re-formation and
is operating normally.
Halted. A node never sees itself in this state. Other nodes will see it in this state after the
node has gracefully left the active cluster, for instance with a cmhaltnode command.
Unknown. A node never sees itself in this state. Other nodes assign a node this state if it has
never been an active cluster member.
Package Status and State

The status of a package can be one of the following:
Up. The package control script is active.
Down. The package control script is not active.
Unknown.
A system multi-node package is up when it is running on all the active cluster nodes. A multi-node
package is up if it is running on any of its configured nodes.
The state of the package can be one of the following:
116
Starting. The start instructions in the control script are being run.
Running. Services are active and being monitored.
Halting. The halt instructions in the control script are being run.
Maintenance
Package Switching Attributes

Packages also have the following switching attributes:
Package Switching. Enabledthe package can switch to another node in the event of
failure.
Switching Enabled for a Node. Enabledthe package can switch to the referenced
node. Disabledthe package cannot switch to the specified node until the node is enabled
for the package using the cmmodpkg command.
Every package is marked Enabled or Disabled for each node that is either a primary or
adoptive node for the package.
For multi-node packages, node switching Disabled means the package cannot start on that
node.
Status of Group Membership

The state of the cluster for Oracle RAC is one of the following:
Up. Services are active and being monitored. The membership appears in the output of
cmviewcl -l group.
Down. The cluster is halted and GMS services have been stopped. The membership does not
appear in the output of the cmviewcl -l group.
The following is an example of the group membership output shown in the cmviewcl command:
# cmviewcl -l group
GROUP
DGop
DBOP
DAALL_DB
IGOPALL
MEMBER
1
0
1
0
0
1
2
1
PID
10394
10499
10501
10396
10396
10501
10423
10528
MEMBER_NODE
comanche
chinook
comanche
chinook
comanche
chinook
comanche
chinook
where the cmviewcl output values are the following:
GROUPThe name of a configured group.
MEMBERThe ID number of a member of a group.
PIDThe Process ID of the group member.
MEMBER_NODEThe Node on which the group member is running.
Service Status
Services have only status, as follows:
UpThe service is being monitored.
DownThe service is not running. It may have halted or failed.
UninitializedThe service is included in the package configuration, but it was not started
with a run command in the control script.
Unknown.
117
Network Status
The network interfaces have only status, as follows:
Up.
Down.
UnknownWhether the interface is up or down cannot be determined. This can happen

when the cluster is down. A standby interface has this status.
NOTE:
Serial Line Status has been de-supported as of Serviceguard A.11.18.
Failover and Failback Policies

Packages can be configured with one of two values for the FAILOVER_POLICY parameter:
CONFIGURED_NODEthe package fails over to the next node in the node list in the package
configuration file.
MIN_PACKAGE_NODEthe package fails over to the node in the cluster with the fewest running
packages on it.
Packages can also be configured with one of two values for the FAILBACK_POLICY parameter:
AUTOMATICa package following a failover returns to its primary node when the primary
node becomes available again.
MANUALa package following a failover must be moved back to its original node by a system
administrator.
Failover and failback policies are displayed in the output of the cmviewcl -v command.
Examples of Cluster and Package States

The following sample output from the cmviewcl -v command shows status for the cluster in the
sample configuration.
Normal Running Status

Everything is running normallyboth nodes in a two-node cluster are running, and each Oracle
RAC instance package is running as well. The only packages running are Oracle RAC instance
packages.
CLUSTER
example
NODE
ftsys9
STATUS
up
STATUS
up
STATE
running
Network_Parameters:
INTERFACE
STATUS
PRIMARY
up
STANDBY
up
PATH
56/36.1
60/6
NAME
lan0
lan1
PACKAGE
ops_pkg1
STATE
running
AUTO_RUN
disabled
STATUS
up
NODE
ftsys9
Policy_Parameters:
POLICY_NAME
CONFIGURED_VALUE
Start
configured_node
Failback
manual
Node_Switching_Parameters:
NODE_TYPE
STATUS
SWITCHING
Primary
up
enabled
NODE
118
Maintenance
STATUS
STATE
NAME
ftsys9
(current)
ftsys10
up
running
Network_Parameters:
INTERFACE
STATUS
PRIMARY
up
STANDBY
up
PATH
28.1
32.1
NAME
lan0
lan1
PACKAGE
ops_pkg2
STATE
running
AUTO_RUN
disabled
STATUS
up
NODE
ftsys10
Policy_Parameters:
POLICY_NAME
CONFIGURED_VALUE
Start
configured_node
Failback
manual
NODE_TYPE
STATUS
SWITCHING
Primary
up
enabled
Alternate
up
enabled
NAME
ftsys10
ftsys9
(current)
Quorum Server Status

If the cluster is using a quorum server for tie-breaking services, the display shows the server name,
state and status following the entry for each node, as in the following excerpt from the output of
cmviewcl -v:
CLUSTER
example
NODE
ftsys9
STATUS
up
STATUS
up
STATE
running
Quorum Server Status:

NAME
STATUS
lp-qs
up
...
NODE
ftsys10
STATUS
up
STATE
running
STATE
running
Quorum Server Status:

NAME
STATUS
lp-qs
up
STATE
running
CVM Package Status

If the cluster is using the VERITAS Cluster Volume Manager for disk storage, the system multi-node
package CVM-VxVM-pkg must be running on all active nodes for applications to be able to access
CVM disk groups. This package is shown in the following output of the cmviewcl command:
CLUSTER
example
STATUS
up
NODE
ftsys8
ftsys9
STATUS
down
up
STATE
halted
running
PACKAGE
STATUS
VxVM-CVM-pkg up
STATE
running
When you use the -v option, the display shows the system multi-node package associated with
each active node in the cluster, as in the following:
119
PACKAGE
STATUS
VxVM-CVM-pkg up
NODE
ftsys8
STATE
running
STATUS
down
NODE
STATUS
ftsys9
up
Script_Parameters:
ITEM
STATUS
Service
up
STATE
halted
STATE
running
MAX_RESTARTS
0
RESTARTS
0
NAME
VxVM-CVM-pkg.srv
Status After Moving the Package to Another Node

After issuing the following command:
# cmrunpkg -n ftsys9 pkg2
the output of the cmviewcl -v command is as follows:

CLUSTER
example
NODE
ftsys9
STATUS
up
STATUS
up
STATE
running
Network_Parameters:
INTERFACE
STATUS
PRIMARY
up
STANDBY
up
PATH
56/36.1
60/6
NAME
lan0
lan1
PACKAGE
pkg1
STATE
running
AUTO_RUN
enabled
STATUS
up
NODE
ftsys9
Policy_Parameters:
POLICY_NAME
CONFIGURED_VALUE
Failover
min_package_node
Failback
manual
Script_Parameters:
ITEM
STATUS
Service
up
Subnet
up
Resource
up
MAX_RESTARTS
0
0
NODE_TYPE
STATUS SWITCHING
Primary
up
enabled
Alternate
up
enabled
PACKAGE
pkg2
STATUS
up
STATE
running
RESTARTS
NAME
0
service1
0
15.13.168.0
/example/float
NAME
ftsys9
ftsys10
(current)
AUTO_RUN
disabled
NODE
ftsys9
Policy_Parameters:
POLICY_NAME
CONFIGURED_VALUE
Failover
min_package_node
Failback
manual
Script_Parameters:
ITEM
STATUS
NAME
MAX_RESTARTS
Service
up
service2.1
0
Subnet
up
15.13.168.0
0
NODE_TYPE
STATUS
SWITCHING
Primary
up
enabled
120 Maintenance
NAME
ftsys10
RESTARTS
0
0
Alternate
NODE
ftsys10
up
STATUS
up
enabled
ftsys9
(current)
STATE
running
Network_Parameters:
INTERFACE
STATUS
PRIMARY
up
STANDBY
up
PATH
28.1
32.1
NAME
lan0
lan1
Now pkg2 is running on node ftsys9. Note that it is still disabled from switching.
Status After Package Switching is Enabled

The following command changes package status back to Package Switching Enabled:
# cmmodpkg -e pkg2
The output of the cmviewcl command is now as follows:

CLUSTER
example
NODE
ftsys9
STATUS
up
STATUS
up
PACKAGE
pkg1
pkg2
NODE
ftsys10
STATUS
up
up
STATUS
up
STATE
running
STATE
running
running
AUTO_RUN
enabled
enabled
NODE
ftsys9
ftsys9
STATE
running
Both packages are now running on ftsys9 and pkg2 is enabled for switching. Ftsys10 is
running the daemon and no packages are running on ftsys10.
Status After Halting a Node

After halting ftsys10, with the following command:
# cmhaltnode
ftsys10
the output of cmviewcl is as follows on ftsys9:

CLUSTER
example
NODE
ftsys9
STATUS
up
STATUS
up
PACKAGE
pkg1
pkg2
NODE
ftsys10
STATUS
up
up
STATUS
down
STATE
running
STATE
running
running
AUTO_RUN
enabled
enabled
NODE
ftsys9
ftsys9
STATE
halted
This output is seen on both ftsys9 and ftsys10.
Viewing Data on Unowned Packages

The following example shows packages that are currently unownednot running on any configured
node. Information on monitored resources is provided for each node. This information allows you
to identify the cause of a failure and decide where to start the package up again.
UNOWNED_PACKAGES
PACKAGE
PKG3
STATUS
down
STATE
halted
AUTO_RUN
enabled
NODE
unowned
121
Policy_Parameters:
POLICY_NAME
CONFIGURED_VALUE
Failover
min_package_node
Failback
automatic
Script_Parameters:
ITEM
STATUS
Resource
up
Subnet
up
Resource
up
Subnet
up
Resource
up
Subnet
up
Resource
up
Subnet
up
NODE_NAME
manx
manx
burmese
burmese
tabby
tabby
persian
persian
NAME
/resource/random
192.8.15.0
/resource/random
192.8.15.0
/resource/random
192.8.15.0
/resource/random
192.8.15.0
NODE_TYPE
STATUS
SWITCHING
Primary
up
enabled
Alternate
up
enabled
Alternate
up
enabled
Alternate
up
enabled
NAME
manx
burmese
tabby
persian
Checking the Cluster Configuration and Components

Serviceguard provides tools that allow you to check the soundness of the cluster configuration, and
the health of its components. In past releases, much of this was done by cmcheckconf (1m)
and/or cmapplyconf (1m) and could be done only when you were changing the configuration
of the cluster or packages.
As of Serviceguard A.11.20, these commands perform additional checks, and a new command,
cmcompare (1m) allows you to compare the contents and characteristics of cluster-wide files to
make sure they are consistent. In addition, you can check configuration of the cluster and all of its
packages at any time by running cmcheckconf (1m) without arguments (or with -v; see below).
These checks help you to ensure that packages will start up and fail over successfully.
Specifically, the following capabilities are new as of Serviceguard A.11.20.
122
Maintenance
NOTE:
All of the checks below are performed when you run cmcheckconf without any arguments
(or with only -v, with or without -k or -K). cmcheckconf validates the current cluster and
package configuration, including external scripts and pre-scripts for modular packages, and
runs cmcompare to check file consistency across nodes. (This new version of the command
also performs all of the checks that were done in previous releases.) See Checking Cluster
Components (page 123) for details.
You may want to set up a cron (1m) job to run cmcheckconf regularly. See Setting up
Periodic Cluster Verification (page 125).
These new checks are not done for legacy packages. For information about legacy and
modular packages.
LVM volume groups:
Check that each volume group contains the same physical volumes on each node.
Check that each node has a working physical connection to the physical volumes.
Check that volume groups used in modular packages are cluster-aware.
LVM logical volumes
Check that file systems have been built on the logical volumes identified by the fs_name
parameter in the cluster's packages.
File consistency:
Check that files including the following are consistent across all nodes.
/etc/hosts (must contain all IP addresses configured into the cluster)
/etc/nsswitch.conf
/etc/services
package control scripts for legacy packages (if you specify them)
/etc/cmcluster/cmclfiles2check
/etc/cmcluster/cmignoretypes.conf
/etc/cmcluster/cmknowncmds
/etc/cmcluster/cmnotdisk.conf
user-created files (if you specify them)
Checking Cluster Components

The following table shows each component that you can check, the command or tool to use, and
where to find further information.
NOTE:
The table includes all the checks available as of A.11.20, not just the new ones.
Table 5 Verifying Cluster Components

Component (Context)
Tool or Command; More Information
Comments
Volume groups (cluster)
cmcheckconf (1m), cmapplyconf

(1m)
Checked for the following:

Existence
Availability on each node
123
Table 5 Verifying Cluster Components (continued)

Component (Context)
Comments
Same physical volumes on each
node
Physical volumes connected on
each node
Volume groups (package)

(1m)
Checked only on nodes configured

to run the package.
LVM logical volumes (package)

(1m)
Checked for modular packages

only, as part of package validation
(cmcheckconf -P).
LVM physical volumes (package)

(1m)

(cmcheckconf -P).
LANs (cluster)
cmviewcl (1m), cmcheckconf (1m), Standby status is checked for each

single NIC; cmviewcl also
cmapplyconf (1m)
reports APA status.
cmviewcl flags offline standby in
an existing cluster, but not for a
cluster that is being created.
Quorum Server (cluster)

(1m).
Commands check that the quorum

server, if used, is running and all
nodes are authorized to access it.
And, if more than one IP address
is specified, that the quorum server
is reachable from all nodes
through both the IP addresses.
Lock disk(s) (cluster)

(1m)
Commands check that the disk(s)

are accessible from all nodes, and
that the lock volume group(s) are
activated on at least one node.
Lock LUN (cluster)

(1m)
Commands check that all nodes

are be accessing the same
physical device and that the lock
LUN device file is a block device
file.
File consistency (cluster)
cmcheckconf (1m), cmcompare (1m). To check file consistency across all

nodes in the cluster, do the
IMPORTANT: See the manpage for
following:
differences in return codes from
1. Customize /etc/cmcluster/
cmcheckconf without options versus
cmfiles2check
cmcheckconf -C.
2. Distribute it to all nodes using
cmsysnc (1m)
3. run cmcheckconf -C
For a subset of nodes, or to check
only specific characteristics such
as ownership, content, etc., use
cmcompare (1m).
124
Mount points (package)

(1m)
Commands check that the

mount-point directories specified
in the package configuration file
exist on all nodes that can run the
package.
Service commands (package)

(1m)
Commands check that files

specified by service commands
Maintenance
Table 5 Verifying Cluster Components (continued)

Component (Context)
Comments
exist and are executable. Service
commands whose paths are nested
within an unmounted shared
filesystem are not checked.
IP addresses (cluster)

(1m)
Commands check that all IP

addresses configured into the
cluster are in each node's /etc/
hosts.
Package IP addresses (package)

(1m)
File systems (package)

(1m)
For LVM only, commands check

that file systems have been built on
the logical volumes identified by
the fs_name parameter.
EMS resources (package)

(1m)
Commands check that configured

resources are available on each
node that can run the package.
External scripts and pre-scripts

(modular package)

(1m)
A non-zero return value from any

script will cause the commands to
fail.
SGeRAC Toolkit files consistency

(package)

(1m)

(cmcheckconf -P).
Cluster Interconnect subnet (package) cmcheckconf (1m), cmapplyconf

(1m)

(cmcheckconf -P).
Setting up Periodic Cluster Verification

You can use cron (1m) to run cluster verification at a fixed interval. Specify the commands to
run in a crontab file (see crontab (1)).
NOTE: The job must run on one of the nodes in the cluster. Because only the root user can run
cluster verification, and cron (1m) sets the jobs user and group IDs to those of the user who
submitted the job, you must edit the file /var/spool/cron/crontabs/root as the root user.
Example
The short script that follows runs cluster verification and sends an email to admin@hp.com when
verification fails.
#!/bin/sh
cmcheckconf -v >/tmp/cmcheckconf.output
if (( $? != 0 ))
then
mailx -s "Cluster verification failed" admin@hp.com 2>&1 </tmp/cmcheckconf.output
fi
To run this script from cron, you would create the following entry in /var/spool/cron/
crontabs/root:
0 8,20 * * * verification.sh
See the cron (1m) manpage for more information.
125
Limitations
Serviceguard does not check the following conditions:
Access Control Policies properly configured
File systems configured to mount automatically on boot (that is, Serviceguard does not check
/etc/fstab)
Shared volume groups configured to activate on boot
Volume group major and minor numbers unique
Redundant storage paths functioning properly
Kernel parameters and driver configurations consistent across nodes
Mount point overlaps (such that one file system is obscured when another is mounted)
Unreachable DNS server
Consistency of settings in .rhosts and /var/admin/inetd.sec
Consistency across cluster of major and minor numbers device-file numbers
Nested mount points
Staleness of mirror copies
Online Reconfiguration
The online reconfiguration feature provides a method to make configuration changes online to a
Serviceguard Extension for RAC (SGeRAC) cluster. Specifically, this provides the ability to add
and/or delete nodes from a running SGeRAC Cluster, and to reconfigure SLVM Volume Group
(VG) while it is being accessed by only one node.
Online Node Addition and Deletion

Online node addition enables the addition or deletion of nodes in an SGeRAC cluster to or from
another running cluster. Node(s) can be added and/or deleted by changing the cluster
configuration. This is done by editing the cluster specification file and re-applying the configuration
to the already running cluster. For deleting online node(s), the node(s) needs to be halted before
deleting them from the cluster.
Use the following steps for adding a node using online node reconfiguration.
1. Export the mapfile for the volume groups that needs to be visible in the new node ( vgexport
-s -m mapfile -p <sharedvg>).
2. Copy the mapfile to the new node.
3. Import the volume groups into the new node (vgimport -s -m mapfile <sharedvg>).
4. Add node to the cluster onlineedit the cluster configuration file to add the node details and
run cmapplyconf.
After adding a new node to an SGeRAC cluster, where SGeRAC Oracle Clusterware-MNP
or ASMDG-MNP, or RAC-MNP packages are already configured. Follow these steps before
issuing the cmrunnode command:
a. For Oracle Clusterware multi-node packages:
1) Create a directory for Oracle clusterware toolkit configuration file. This directory
must be the same as the one which is created on the existing cluster nodes.
For example; mkdir /etc/cmcluster/<YourOwn-OC-multi-node package-Dir/
2)
b.
126
Maintenance
Copy oc.conf from Oracle clusterware multi node package directory from other
nodes to the Oracle Clusterware multi node package directory of new node.
For Oracle RAC multi-node packages:
1)
Create a directory for RAC toolkit configuration file. This directory should be same
as the one which is created on the existing cluster nodes.
For example; mkdir /etc/cmcluster/<YourOwn-RACmulti-node package-Dir/
2)
Copy the rac_dbi.conf from RAC multi node package directory from other nodes
to the the RAC multi node package directory of new node.
Step b1 and b2 must be repeated for all the RAC MNP packages in the cluster.
c.
For oracle ASMDG multi-node packages

1) Create a directory for ASMDG toolkit configuration file. This directory should be
same as the one which is created on the existing cluster nodes.
For example; mkdir /etc/cmcluster/<YourOwn-ASMDGmulti-node package-Dir/
2)
Copy asm_dg.conf from ASMDG multi node package directory from other nodes
to the the ASMDG multi node package directory of new node.
Step c1 and c2 must be repeated for all the ASMDG MNP packages in the cluster.
5.
Make the new node join the cluster (cmrunnode) and run the services.
Use
1.
2.
3.
the following steps for deleting a node using online node reconfiguration:
Halt the node in the cluster by running cmhaltnode.
Edit the cluster configuration file to delete a node(s).
Run cmapplyconf.
Managing the Shared Storage

The following describes the confirmation process for Single Node Online volume Re-Configuration,
in addition to procedures for configuring LVM version 1.0. There are differences between LVM
version 1 and version 2.
NOTE:
When using LVM, the volume groups are supported with Serviceguard.
For more information on using and configuring LVM version 2.x, see the HP-UX 11i Version 3:
HP-UX System Administrator's Guide: Logical Volume Management located at www.hp.com/go/
hpux-core-docs > HP-UX 11iv3.
For LVM version 2 compatibility requirements, see the Serviceguard/SGeRAC/SMS/Serviceguard
Mgr Plug-in Compatibility and Feature Matrix at www.hp.com/go/hpux-serviceguard-docs >
HP Serviceguard.
NOTE: For more information, see the Serviceguard Version A.11.20 Release Notes at
www.hp.com/go/hpux-serviceguard-docs > HP Serviceguard.
Making LVM Volume Groups Shareable

Normally, volume groups are marked to be activated in shared mode when they are listed with
the OPS_VOLUME_GROUP parameter in the cluster configuration file or in Serviceguard Manager
that occurs when the configuration is applied. However, in some cases you may want to manually
make a volume group sharable. For example, if you wish to add a new shared volume group
without shutting down the cluster, you can use the manual method to do it online. When convenient,
it's a good practice to bring down the cluster and reconfigure it to include the new volume group.
1. Use the vgchange command on each node to ensure that the volume group to be shared is
currently inactive on all nodes. Example:
# vgchange -a n /dev/vg_rac
2.
On the configuration node, use the vgchange command to make the volume group shareable
by members of the cluster:
127
# vgchange -S y -c y /dev/vg_rac
This command is issued from the configuration node only, and the cluster must be running on
all nodes for the command to succeed. Note that both the -S and the -c options are specified.
The -S y option makes the volume group shareable, and the -c y option causes the cluster
ID to be written out to all the disks in the volume group. This command specifies the cluster
that a node must be a part of to obtain shared access to the volume group.
Making a Volume Group Unshareable

Use the following steps to unmark a previously marked shared volume group:
1. Remove the volume group name from the ASCII cluster configuration file.
2. Enter the following command:
# vgchange -S n -c n /dev/volumegroup
The above example marks the volume group as non-shared, and not associated with a cluster.
Activating an LVM Volume Group in Shared Mode

Activation and deactivation of shared volume groups is normally done through a control script. If
you need to perform activation from the command line, you can issue the following command from
each node to activate the volume group in shared mode. (The node where you first enter the
command becomes the server node.)
# vgchange -a s -p /dev/vg_rac
The following message is displayed:

Activated volume group in shared mode.
This node is the Server.
When the same command is entered on the second node, the following message is displayed:
Activated volume group in shared mode.
This node is a Client.
NOTE: Do not share volume groups that are not part of the RAC configuration unless shared
access is controlled.
Deactivating a Shared Volume Group

Issue the following command from each node to deactivate the shared volume group:
Remember that volume groups remain shareable even when nodes enter and leave the cluster.
NOTE: If you wish to change the capacity of a volume group at a later time, you must deactivate
and unshare the volume group first. If you add disks, you must specify the appropriate physical
volume group name and make sure the /etc/lvmpvg file is correctly updated on both nodes.
Making Offline Changes to Shared Volume Groups

You may need to change the volume group configuration of RAC shared logical volumes to add
capacity to the data files or to add log files. No configuration changes are allowed on shared
LVM volume groups while they are activated. The volume group must be deactivated first on all
nodes, and marked as non-shareable.
To learn more about an additional method, Single Node Online volume Re-Configuration (SNOR),
please see the white paper SLVM Online Volume Re-configuration at www.hp.com/go/
hpux-core-docs > HP-UX 11i v3 > White papers.
Use the following procedure (examples assume the volume group is being shared by node 1 and
node 2, and they use the volume group vg_rac):
128
Maintenance
1.
2.
Ensure that the Oracle RAC database is not active on either node.
From node 2, use the vgchange command to deactivate the volume group:
3.
From node 2, use the vgexport command to export the volume group:
# vgexport -m /tmp/vg_rac.map.old /dev/vg_rac
This dissociates the volume group from node 2.

4.
From node 1, use the vgchange command to deactivate the volume group:
5.
Use the vgchange command to mark the volume group as unshareable:

# vgchange -S n -c n /dev/vg_rac
6.
Prior to making configuration changes, activate the volume group in normal (non-shared)
mode:
# vgchange -a y /dev/vg_rac
7.
8.
Use normal LVM commands to make the needed changes. Be sure to set the raw logical volume
device file's owner to oracle and group to dba with a mode of 660.
Next, from node 1, deactivate the volume group:
9.
Use the vgexport command with the options shown in the example to create a new map
file:
# vgexport -p -m /tmp/vg_rac.map /dev/vg_rac
Make a copy of /etc/lvmpvg in /tmp/lvmpvg, then copy the file to /tmp/lvmpvg on

node 2. Copy the file /tmp/vg_rac.map to node 2.
10. Use the following command to make the volume group shareable by the entire cluster again:
# vgchange -S y -c y /dev/vg_rac
11. On node 2, issue the following command:

# mkdir /dev/vg_rac
12. Create a control file named group in the directory /dev/vg_rac, as in the following:
The major number is always 64, and the hexadecimal minor number has the format:
0xhh0000
number that is available on your system after the volume groups that are already configured.
13. Use the vgimport command, specifying the map file you copied from the configuration node.
In the following example, the vgimport command is issued on the second node for the same
volume group that was modified on the first node:
# vgimport -v -m /tmp/vg_rac.map /dev/vg_rac /dev/dsk/c0t2d0/dev/dsk/c1t2d0
14. Activate the volume group in shared mode by issuing the following command on both nodes:
# vgchange -a s -p /dev/vg_rac
Skip this step if you use a package control script to activate and deactivate the shared volume
group as a part of RAC startup and shutdown.
129
Adding Additional Shared LVM Volume Groups

To add capacity or to organize your disk resources for ease of management, you may wish to
create additional shared volume groups for your Oracle RAC databases. If you decide to use
additional shared volume groups, they must conform to the following rules:
Volume groups should include different PV links to each logical unit on the disk array.
Volume group names must be the same on all nodes in the cluster.
Logical volume names must be the same on all nodes in the cluster.
If you are adding or removing shared LVM volume groups, make sure that you modify the cluster
configuration file and any package control script that activates and deactivates the shared LVM
volume groups.
Changing the CVM Storage Configuration

To add new CVM disk groups, the cluster must be running.
If you are creating new CVM disk groups, be sure to determine the master node on which to do
the creation by using the following command:
# vxdctl -c mode
One node will identify itself as the master. Create disk groups from this node.
Similarly, you can delete CVM disk groups provided they are not being used by a cluster node at
the time.
NOTE: For CVM without CFS, if you are adding a disk group to the cluster configuration, make
sure you also modify any package or create the package control script that imports and deports
this disk group. If you are adding a CVM disk group, be sure to add the STORAGE_GROUP entry
for the disk group to the package ASCII file.
For CVM with CFS, if you are adding a disk group to the cluster configuration, make sure you also
create the corresponding multi-node package. If you are adding a CVM disk group, be sure to
add to the packages the necessary package dependency that depend on the CVM disk group.
If you are removing a disk group from the cluster configuration, make sure that you also modify
or delete any package control script that imports and deports this disk group. If you are removing
a CVM disk group, be sure to remove the STORAGE_GROUP entries for the disk group from the
package ASCII file.
When removing a disk group that is activated and deactivated through a multi-node package,
make sure to modify or remove any configured package dependencies to the multi-node package.
Removing Serviceguard Extension for RAC from a System

If you wish to remove a node from Serviceguard Extension for RAC operation, use the swremove
command to delete the software. Note the following:
The cluster service should not be running on the node from which you will be deleting
Serviceguard Extension for RAC.
The node from which you are deleting Serviceguard Extension for RAC should not be in the
cluster configuration.
If you are removing Serviceguard Extension for RAC from more than one node, swremove
should be issued on one node at a time.
NOTE: After removing Serviceguard Extension for RAC, your cluster will still have Serviceguard
installed. For information about removing Serviceguard, refer to the Managing Serviceguard user
guide for your version of the product.
130 Maintenance
Monitoring Hardware
Good standard practice in handling a high-availability system includes careful fault monitoring so
as to prevent failures if possible, or at least to react to them swiftly when they occur.
The following should be monitored for errors or warnings of all kinds.
Disks
CPUs
Memory
LAN cards
Power sources
All cables
Disk interface cards
Some monitoring can be done through simple physical inspection, but for the most comprehensive
monitoring, you should examine the system log file (/var/adm/syslog/syslog.log) periodically
for reports on all configured HA devices. The presence of errors relating to a device will show the
need for maintenance.
Using Event Monitoring Service

Event Monitoring Service (EMS) allows you to configure monitors of specific devices and system
resources. You can direct alerts to an administrative workstation where operators can be notified
of further action in case of a problem. For example, you could configure a disk monitor to report
when a mirror was lost from a mirrored volume group being used in a non-RAC package.
For additional information, refer to www.hp.com/go/hpux-ha-monitoring-docs > HP Event
Monitoring Service.
Using EMS Hardware Monitors

A set of hardware monitors is available for monitoring and reporting on memory, CPU, and many
other system values. For additional information, refer to Online Diagnostics (EMS and STM)
Administrator's Guide at www.hp.com/go/hpux-diagnostics-docs > Diagnostics and Monitoring
Tools.
Adding Disk Hardware

As your system expands, you may need to add disk hardware. This also means modifying the
logical volume structure. Use the following general procedure:
1. Halt packages.
2. Ensure that the Oracle database is not active on either node.
3. Deactivate and mark as unshareable any shared volume groups.
4. Halt the cluster.
5. Deactivate automatic cluster startup.
6. Shutdown and power off system before installing new hardware.
7. Install the new disk hardware with connections on all nodes.
8. Reboot all nodes.
9. On the configuration node, add the new physical volumes to existing volume groups, or create
new volume groups as needed.
10. Start up the cluster.
11. Make the volume groups shareable, then import each shareable volume group onto the other
nodes in the cluster.
12. Activate the volume groups in shared mode on all nodes.
Monitoring Hardware
131
13. Start up the Oracle RAC instances on all nodes.

14. Activate automatic cluster startup.
NOTE: As you add new disks to the system, update the planning worksheets (described in
Appendix B: Blank Planning Worksheets), so as to record the exact configuration you are using.
Replacing Disks
The procedure for replacing a faulty disk mechanism depends on the type of disk configuration
you are using and on the type of Volume Manager software. For a description of replacement
procedures using CVM, refer to the chapter on Administering Hot-Relocation in the VERITAS
Volume Manager Administrators Guide. Additional information is found in the VERITAS Volume
Manager Troubleshooting Guide.
The following sections describe how to replace disks that are configured with LVM. Separate
descriptions are provided for replacing a disk in an array and replacing a disk in a high-availability
enclosure.
Replacing a Mechanism in a Disk Array Configured with LVM

With any HA disk array configured in RAID 1 or RAID 5, refer to the arrays documentation for
instruction on how to replace a faulty mechanism. After the replacement, the device itself
automatically rebuilds the missing data on the new disk. No LVM activity is needed. This process
is known as hot swapping the disk.
NOTE: If your LVM installation requires online replacement of disk mechanisms, the use of disk
arrays may be required, because software mirroring of JBODs with MirrorDisk/UX does not permit
hot swapping for disks that are activated in shared mode.
Replacing a Mechanism in an HA Enclosure Configured with Exclusive LVM

Non-Oracle data that is used by packages may be configured in volume groups that use exclusive
(one-node-at-a-time) activation. If you are using exclusive activation and software mirroring with
MirrorDisk/UX and the mirrored disks are mounted in a high-availability disk enclosure, you can
use the following steps to hot plug a disk mechanism.
1. Identify the physical volume name of the failed disk and the name of the volume group in
which it was configured. In the following examples, the volume group name is shown as/dev/
vg_sg01 and the physical volume name is shown as/dev/c2t3d0. Substitute the volume
group and physical volume names that are correct for your system.
2. Identify the names of any logical volumes that have extents defined on the failed physical
volume.
3. On the node where the volume group is currently activated, use the following command for
each logical volume that has extents on the failed physical volume:
# lvreduce -m 0 /dev/vg_sg01/lvolname /dev/dsk/c2t3d0
4.
5.
Remove the failed disk and insert a new one. The new disk will have the same HP-UX device
name as the old one.
On the node from where you issued the lvreduce command, issue the following command
to restore the volume group configuration data to the newly inserted disk:
# vgcfgrestore /dev/vg_sg01 /dev/dsk/c2t3d0
132
Maintenance
6.
Issue the following command to extend the logical volume to the newly inserted disk:
# lvextend -m 1 /dev/vg_sg01 /dev/dsk/c2t3d0
7.
Finally, use the lvsync command for each logical volume that has extents on the failed
physical volume. This synchronizes the extents of the new disk with the extents of the other
mirror.
# lvsync /dev/vg_sg01/lvolname
Online Replacement of a Mechanism in an HA Enclosure Configured with Shared

LVM (SLVM)
If you are using software mirroring for shared concurrent activation of Oracle RAC data with
MirrorDisk/UX and the mirrored disks are mounted in a high-availability disk enclosure, use the
following LVM command options to change/replace disks via OLR (On Line Replacement).
NOTE: This procedure supports either LVM or SLVM VG and is online (activated) and uses an
online disk replacement mechanism. It is supported for SLVM; however, the PV being replaced
must be detached from each node. For example, running pvchange -a N /dev/dsk/ from
one node only detaches the disk from that node's perspective.
1.
Detach the target PV by using one of the following commands on each node of the cluster:
# pvchange -a n [pv path]
Use the pvchange command -a n [pv path] to detach only one path or replace a disk
if the primary disk path is not performing well and you want to disable the path. The pvchange
-a n command detaches the single specified PV Link (device path). (If the path was the path
in use, LVM will switch to any alternate PV Link that is still available.)
OR
# pvchange -a N [pv path]
Alternatively, use the pvchange -a N [pv path] command to detach a disk (all paths to
the disk) and close it. Use this to allow diagnostics or replace a multi-ported disk.
NOTE: If the volume group is mirrored, applications can continue accessing data on mirror
copies after the commands above. If the volume is not mirrored, then any access attempts to
the device may hang indefinitely or time out. This depends upon the LV timeout value configured
for the logical volume.
2.
Replace new disk.

The new disk size needs to be of equal or greater size. This is required whether or not the
disk replacement is online or offline.
3.
Restore the LVM header to the new disk using the following command:
# vgcfgrestore -n [vg name] [pv raw path]
It is only necessary to perform the vgcfgrestore operation once from any node on the
cluster.
4.
Attach PV or Activate the VG from each node of the cluster using the following commands:
#pvchange -a y [pv path]
OR
# vgchange -a [y|e|s] [vg name]
The PV must be detached from all nodes and must be attached from each of the nodes to
make it usable. Alternatively, you can reactivate the VG from each of the nodes. (This command
cannot attach all the paths to the PV, therefore each PV link has to be attached as well.)
Replacing Disks
133
NOTE: After executing one of the commands above, any I/O queued for the device will
restart. If the device replaced in step #2 was a mirror copy, then it will begin the
resynchronization process that may take a significant amount of time to complete. The progress
of the resynchronization process can be observed using the vgdisplay(1M),
lvdisplay(1M) or pvdisplay(1M) commands.
Offline Replacement of a Mechanism in an HA Enclosure Configured with Shared

LVM (SLVM)
Hot plugging of disks is not supported for Oracle RAC data that is configured in volume groups
with Shared LVM (SLVM). If you need this capability, you should use disk arrays for your Oracle
RAC data.
If you are using software mirroring for shared concurrent activation of Oracle RAC data with
MirrorDisk/UX and the mirrored disks are mounted in a high-availability disk enclosure, use the
following steps to carry out offline replacement:
1.
2.
Make a note of the physical volume name of the failed mechanism (for example, /dev/dsk/
c2t3d0).
Deactivate the volume group on all nodes of the cluster:
# vgchange -a n vg_rac
3.
4.
Replace the bad disk mechanism with a good one.

From one node, initialize the volume group information on the good mechanism using
vgcfgrestore(1M), specifying the name of the volume group and the name of the physical
volume that is being replaced:
# vgcfgrestore /dev/vg_rac /dev/dsk/c2t3d0
5.
Activate the volume group on one node in exclusive mode, then deactivate the volume group:
# vgchange -a e vg_rac
This will synchronize the stale logical volume mirrors. This step can be time-consuming,
depending on hardware characteristics and the amount of data.
6.
Deactivate the volume group:

# vgchange -a n vg_rac
7.
Activate the volume group on all the nodes in shared mode using vgchange - a s:
# vgchange -a s vg_rac
Replacing a Lock Disk

Replacing a failed lock disk mechanism is the same as replacing a data disk. If you are using a
dedicated lock disk (one with no user data on it), then you need to issue only one LVM command:
# vgcfgrestore /dev/vg_lock /dev/dsk/c2t1d0
After doing this, wait at least an hour, then review the syslog file for a message showing that the
lock disk is healthy again.
Online Hardware Maintenance with Inline SCSI Terminator

Serviceguard allows online SCSI disk controller hardware repairs to all cluster nodes if you use
HPs inline terminator (C2980A) on nodes connected to the end of the shared FW/SCSI bus. The
inline terminator cable is a 0.5 meter extension cable with the terminator on the male end that
connects to the controller card for an external bus. The inline terminator is used instead of the
termination pack that is attached to the controller card. The inline terminator makes it possible to
physically disconnect the node from the end of the F/W SCSI bus without breaking the bus's
termination. (Nodes attached to the middle of a bus using a Y cable also can be detached from
134
Maintenance
the bus without harm.) When using inline terminators and Y cables, ensure that all orange-socketed
termination packs are removed from the controller cards.
NOTE: You cannot use inline terminators with internal FW/SCSI buses on D and K series systems,
and you cannot use the inline terminator with single-ended SCSI buses. You must not use an inline
terminator to connect a node to a Y cable.
Figure 19 shows a three-node cluster with two F/W SCSI buses. The solid line and the dotted line
represent different buses, both of which have inline terminators attached to nodes 1 and 3. Y
cables are also shown attached to node 2.
Figure 19 F/W SCSI Buses with Inline Terminators
The use of inline SCSI terminators allows you to do hardware maintenance on a given node by
temporarily moving its packages to another node and then halting the original node while its
hardware is serviced. Following the replacement, the packages can be moved back to the original
node.
Use the following procedure to disconnect a node that is attached to the bus with an inline SCSI
terminator or with a Y cable:
1. Move any packages on the node that requires maintenance to a different node.
2. Halt the node that requires maintenance. The cluster will reform, and activity will continue on
other nodes. Packages on the halted node will switch to other available nodes if they are
configured to switch.
3. Disconnect the power to the node.
4. Disconnect the node from the inline terminator cable or Y cable if necessary. The other nodes
accessing the bus will encounter no problems as long as the inline terminator or Y cable
remains connected to the bus.
5. Replace or upgrade hardware on the node, as needed.
6. Reconnect the node to the inline terminator cable or Y cable if necessary.
7. Reconnect power and reboot the node. If AUTOSTART_CMCLD is set to 1 in the /etc/
rc.config.d/cmcluster file, the node will rejoin the cluster.
8. If necessary, move packages back to the node from their alternate locations and restart them.
Replacing Disks
135
Replacement of I/O Cards

After an I/O card failure, you can replace the card using the following steps. It is not necessary
to bring the cluster down to do this if you are using SCSI inline terminators or Y cables at each
node.
1. Halt the node by using Serviceguard Manager or the cmhaltnode command. Packages
should fail over normally to other nodes.
2. Remove the I/O cable from the card. With SCSI inline terminators, this can be done without
affecting the disks or other nodes on the bus.
3. Using SAM, select the option to do an online replacement of an I/O card.
4. Remove the defective I/O card.
5. Install the new card. The new card must be exactly the same card type, and it must be installed
in the same slot as the card you removed.
6. In SAM, select the option to attach the new I/O card.
7. Add the node back into the cluster by using Serviceguard Manager or the cmrunnode
command.
The Administration menu of Serviceguard Manager contains all the administration operations.
Replacement of LAN Cards

If you have a LAN card failure that requires the LAN card to be replaced, you can replace it online
or offline, depending on the type of hardware and operating system you are running. It is not
necessary to bring the cluster down to do this.
Offline Replacement
The following steps show how to replace a LAN card offline. These steps apply to HP-UX 11i:
1.
2.
3.
4.
5.
6.
Halt the node by using the cmhaltnode command.

Shut down the system using /etc/shutdown, then power down the system.
Remove the defective LAN card.
Install the new LAN card. The new card must be exactly the same card type, and it must be
installed in the same slot as the card you removed.
Power up the system.
If necessary, add the node back into the cluster by using the cmrunnode command. (You
can omit this step if the node is configured to join the cluster automatically.)
Online Replacement
If your system hardware supports hotswap I/O cards, and if the system is running HP-UX 11i
(B.11.11 or later), you have the option of replacing the defective LAN card online. This will
significantly improve the overall availability of the system. To do this, follow the steps provided in
the section How to Online Replace (OLR) a PCI Card Using SAM in the document Configuring
HP-UX for Peripherals. The OLR procedure also requires that the new card must be exactly the same
card type as the card you removed to avoid improper operation of the network driver. Serviceguard
will automatically recover the LAN card once it has been replaced and reconnected to the network.
After Replacing the Card

After the online or offline replacement of LAN cards has been done, Serviceguard will detect that
the MAC address (LLA) of the card has changed from the value stored in the cluster binary
configuration file, and it will notify the other nodes in the cluster of the new MAC address. The
cluster will operate normally after this.
It is also recommended that you update the new MAC address in the cluster binary configuration
file by re-applying the cluster configuration. Use the following steps for online reconfiguration:
136
Maintenance
1.
Use the cmgetconf command to obtain a fresh ASCII configuration file, as follows:
# cmgetconf config.ascii
2.
Use the cmapplyconf command to apply the configuration and copy the new binary file to
all cluster nodes:
# cmapplyconf -C config.ascii
This procedure updates the binary file with the new MAC address and thus avoids data inconsistency
between the outputs of the cmviewconcl and lanscan commands.
Monitoring RAC Instances

The DB Provider provides the capability to monitor RAC databases. RBA (Role Based Access)
enables a non-root user to have the capability to monitor RAC instances using Serviceguard
Manager.
Monitoring RAC Instances
137
6 Troubleshooting
Go to www.hp.com/go/hpux-serviceguard-docs, and then click HP Serviceguard . In the
User Guide section, click on the latest Managing Serviceguard manual and see the Troubleshooting
your Cluster chapter.
NOTE:
138
Troubleshooting
All messages from cmgmsd log to /var/adm/syslog/syslog.log by default.
A Software Upgrades
Serviceguard Extension for RAC (SGeRAC) software upgrades can be done in the two following
ways:
rolling upgrade
non-rolling upgrade
Instead of an upgrade, moving to a new version can be done with:
migration with cold install
Rolling upgrade is a feature of SGeRAC that allows you to perform a software upgrade on a given
node without bringing down the entire cluster. SGeRAC supports rolling upgrades on version
A.11.15 and later, and requires all nodes to be running on the same operating system revision
and architecture.
During rolling upgrade the nodes can run on mixed version of HP-UX. Rolling upgrades are not
intended as a means of using mixed release of HP-UX with in the same cluster. HP recommends to
upgrade all the cluster nodes to the new release level at the earliest.
Non-rolling upgrade allows you to perform a software upgrade from any previous revision to any
higher revision or between operating system versions but requires halting the entire cluster.
The rolling and non-rolling upgrade processes can also be used any time one system needs to be
taken offline for hardware maintenance or patch installations. Until the upgrade process is complete
on all nodes, you cannot change the cluster configuration files, and you will not be able to use
any of the new features of the Serviceguard/SGeRAC release.
There may be circumstances when, instead of doing an upgrade, you prefer to do a migration
with cold install. The cold install process erases the preexisting operating system and data, and
then installs the new operating system and software. After a cold install, you must restore the data.
The advantage of migrating with a cold install is that the software can be installed without regard
for the software currently on the system or concern for cleaning up old software.
A significant factor when deciding to either do an upgrade or cold install is overall system downtime.
A rolling upgrade will cause the least downtime. This is because only one node in the cluster is
down at any one time. A non-rolling upgrade may require more down time, because the entire
cluster has to be brought down during the upgrade process.
One advantage of both rolling and non-rolling upgrades versus cold install is that upgrades retain
the preexisting operating system, software, and data. Conversely, the cold install process erases
the preexisting systemyou must reinstall the operating system, software, and data. For these
reasons, a cold install may require more downtime.
The sections in this appendix are as follows:
Rolling Software Upgrades (page 139)
Steps for Rolling Upgrades (page 142)
Example of Rolling Upgrade (page 143)
Limitations of Rolling Upgrades (page 147)
Non-Rolling Software Upgrades (page 148)
Limitations of Non-Rolling Upgrades (page 148)
Migrating an SGeRAC Cluster with Cold Install (page 148)
Rolling Software Upgrades

SGeRAC version A.11.15 and later allow you to roll forward to any higher revision provided all
of the following conditions are met.
The upgrade must be done on systems of the same architecture (HP 9000 or Integrity Servers).
All nodes in the cluster must be running on the same version of HP-UX.
139
Each node must be running a version of HP-UX that supports the new SGeRAC version.
Each node must be running a version of Serviceguard that supports the new SGeRAC version.
For more information on support, compatibility, and features for SGeRAC, refer to the Serviceguard
Compatibility and Feature Matrix, located at www.hp.com/go/hpux-serviceguard-docs
> HP Serviceguard Extension for RAC.
Upgrading Serviceguard to SGeRAC cluster

Beginning with HP-UX 11i v3 1109 HA-OE/DC-OE, SGeRAC is included as a licensed bundle at
no additional cost. To install SGeRAC A.11.20 on your system during OE installation, you must
select T1907BA (SGeRAC) in the Software tab.
The following sections discuss the different scenarios for including SGeRAC while upgrading from
an earlier OE to HP-UX 11i v3 1109 HA-OE/DC-OE in a Serviceguard environment.
Upgrading from an existing SGeRAC A.11.19 cluster to HP-UX 11i v3 1109 HA-OE/DC-OE
To upgrade from an existing Serviceguard and SGeRAC A.11.19 deployment to HP-UX 11i v3
1109 HA-OE/DC-OE, you must select the licensed SGeRAC A.11.20 bundle available in the
HP-UX 11i v3 1109 HA-OE/DC-OE media.
NOTE: You must select the product SGeRAC T1907BA when upgrading to HP-UX 11i v3 1109
HA-OE/DC-OE, otherwise Serviceguard A.11.20 installation will fail.
To perform rolling upgrade for an SGeRAC cluster with Oracle RAC configured, the HP-UX OS
version must be the same, that is, either 11i v2 or 11i v3. If you upgrade from 11i v2 to 11iv3,
you can perform an offline upgrade or rolling upgrade. For more information see, Non-Rolling
Software Upgrades (page 148) section and Steps for Rolling Upgrades (page 142).
NOTE: For all the scenarios discussed in the following sections, HP recommends that the method
used to upgrade from Serviceguard A.11.19 to Serviceguard A.11.20 and SGeRAC A.11.20
must be an offline upgrade. Using Dynamic Root Disk (DRD) utilities can significantly reduce the
planned maintenance time to perform the upgrade.
If you want to perform a rolling upgrade that includes SGeRAC, then you must follow the procedures
described in the scenario Upgrading from an existing Serviceguard A.11.19 cluster to HP-UX
11i v3 1109 HA-OE/DC-OE along with SGeRAC (page 140) onwards.
To perform an offline upgrade
1. Halt the cluster. Use the following command:
cmhaltcl -f
2.
3.
Upgrade all the nodes in the cluster to the new HP-UX release 1109 and select SGeRAC
A.11.20 during the upgrade.
Restart the cluster using the following command:
cmruncl
along with SGeRAC
To upgrade from Serviceguard A.11.9 cluster to HP-UX 11i v3 1109 HA-OE/DC-OE along with
SGeRAC:
1. Install Serviceguard A.11.19 patch PHSS_42216 on all the cluster nodes before upgrading
from Serviceguard A.11.19 to HP-UX 11i v3 1109 HA-OE/DC-OE, Serviceguard and SGeRAC
A.11.20. Otherwise, upgraded Serviceguard and SGeRAC A.11.20 node will not be able
to join the existing 11.19 cluster.
2. Select the licensed SGeRAC A.11.20 bundle. To upgrade the 1109 HA-OE/DC-OE:
On the first node:
a. Halt the node using the following command:
140 Software Upgrades
cmhaltnode <node_name>
b.
3.
Select the SGeRAC bundle T1907BA while upgrading the node to HP-UX 11i v3 1109
HA-OE/DC-OE
After upgrading to HP-UX 11i v3 1109 HA-OE/DC-OE, Serviceguard, and SGeRAC A.11.20,
you must install Serviceguard A.11.20 patch PHSS_42137 on the upgraded node. This patch
allows an upgraded Serviceguard and SGeRAC A.11.20 node to join the existing Serviceguard
A.11.19 cluster.
NOTE: If Serviceguard A.11.20 patch PHSS_42137 is not installed on the upgraded
Serviceguard and SGeRAC A.11.20 node, it cannot join the existing A.11.19 cluster.
4.
Run the following command to join the upgraded Serviceguard and SGeRAC A.11.20 node
to the existing Serviceguard A.11.19:
cmrunnode <node_name>
NOTE: Please make sure that Serviceguard Extension for RAC is installed on all the nodes
in the cluster and all nodes are up and running before attempting to deploy Oracle RAC in
this cluster.
5.
Repeat steps 2 to 4 for all the nodes in the cluster to complete the upgrade.
along with SGeRAC (Alternative approach)
This is an alternative approach for upgrading from Serviceguard A.11.19 to Serviceguard and
SGeRAC A.11.20
1. Perform a rolling upgrade of Serviceguard A.11.19 to HP-UX 11i v3 1109 HA-OE/DC-OE
with Serviceguard A.11.20 without selecting the SGeRAC licensed bundle.
2. Install the Serviceguard A.11.20 patch PHSS_42137 on all cluster nodes.
Serviceguard A.11.20 node, it cannot join the existing Serviceguard A.11.20 cluster when
you install SGeRAC on that node.
3.
Use the following steps to perform a rolling upgrade of Serviceguard A.11.20 to include
SGeRAC A.11.20 on each node:
On the first node:
4.
b. Install SGeRAC bundle T1907BA available on the 1109 HA-OE/DC-OE media.

Run the following command to join the upgraded node to the cluster:
The upgraded Serviceguard and SGeRAC A.11.20 node will be able to join the existing
Serviceguard A.11.20 cluster.
NOTE: Please make sure that Serviceguard Extension for RAC is installed on all the nodes
in the cluster and all nodes are up and running before attempting to deploy Oracle RAC in
this cluster.
5.
Repeat steps 3 and 4 for all the nodes in the cluster to complete the upgrade.
Upgrading from Serviceguard A.11.18 cluster to HP-UX 11i v3 1109 HA-OE/DC-OE along with
SGeRAC
To upgrade from Serviceguard A.11.18 to HP-UX 11i v3 1109 HA-OE/DC-OE:
141
1.
2.
Perform a rolling upgrade to Serviceguard A.11.19.

Perform a rolling or offline upgrade to HP-UX 11i v3 1109 HA-OE/DC-OE with SGeRAC.
Refer the procedures described in the scenario Upgrading from an existing Serviceguard
A.11.19 cluster to HP-UX 11i v3 1109 HA-OE/DC-OE along with SGeRAC (page 140) to
perform rolling upgrade from Serviceguard A.11.19 to HP-UX 11i v3 1109 HA-OE/DC-OE
with SGeRAC.
NOTE: Using DRD utilities can significantly reduce the planned maintenance time to perform
this upgrade.
Upgrading from Serviceguard A.11.20 cluster to HP-UX 11i v3 1109 HA-OE/DC-OE along with
SGeRAC
To upgrade from Serviceguard A.11.20 to HP-UX 11i v3 1109 HA-OE/DC-OE:
1. Install Serviceguard A.11.20 patch PHSS_42137 on all the cluster nodes before you start
upgrading to HP-UX 11i v3 1109 HA-OE/DC-OE.
Serviceguard A.11.20 node, when you install SGeRAC on a node, it will not be able to join
the existing Serviceguard A.11.20 cluster.
2.
Select the licensed SGeRAC A.11.20 bundle to upgrade each node to HP-UX 11i v3 1109
HA-OE/DC-OE. To upgrade to HP-UX 11i v3 1109 HA-OE/DC-OE:
On the first node:
b.
Select the SGeRAC bundle T1907BA while upgrading the node to HP-UX 11i v3 1109
HA-OE/DC-OE.
NOTE: In this state, if you try the commands like cmcheckconf and cmapplyconf on the
cluster configuration file, no Error or Warning message related to the missing SGeRAC software
are displayed.
3.
Run the following command to join the upgraded node to the cluster:
NOTE: Please make sure that Serviceguard Extension for RAC is installed on all nodes in
the cluster and all nodes are up and running before attempting to deploy Oracle RAC in this
cluster.
4.
Repeat steps 2 to 3 for all then nodes in the cluster to complete the upgrade.
Steps for Rolling Upgrades

Use the following steps when performing a rolling SGeRAC software upgrade:
1. Halt Oracle (RAC, Clusterware) software on the local node (if running).
2. Halt Serviceguard/SGeRAC on the local node by issuing the Serviceguard cmhaltnode
command.
3. Edit the /etc/rc.config.d/cmcluster file to include the following line:
AUTOSTART_CMCLD = 0
4.
5.
Upgrade the HP-UX OS (if required), Serviceguard, and SGeRAC to the new release (SGeRAC
requires the compatible version of Serviceguard and OS). For more information on how to
upgrade HP-UX, see HP-UX Installation and Update Guide for the target version of HP-UX.
Edit the /etc/rc.config.d/cmcluster file, on the local node, to include the following
line:
AUTOSTART_CMCLD = 1
142
Software Upgrades
NOTE: It is optional to set this parameter to 1. If you want the node to join the cluster at
boot time, set this parameter to 1, otherwise set it to 0.
6.
7.
8.
Restart the cluster on the upgraded node (if desired). You can do this in Serviceguard Manager,
or from the command line, issue the Serviceguard cmrunnode command.
Start Oracle (Clusterware, RAC) software on the local node.
Repeat steps 1-7 on the other nodes, one node at a time until all nodes have been upgraded.
NOTE: Be sure to plan sufficient system capacity to allow moving the packages from node
to node during the upgrade process, to maintain optimum performance.
If a cluster fails before the rolling upgrade is complete (perhaps because of a catastrophic power
failure), the cluster could be restarted by entering the cmruncl command from a node that has
been upgraded to the latest revision of the software.
NOTE: HP recommends you to upgrade the Oracle RAC software either before SG/SGeRAC
rolling upgrade or after SG/SGeRAC rolling upgrade, if you are upgrading Oracle RAC software
along with SG/SGeRAC.
Halt the SGeRAC toolkit packages before upgrading Oracle RAC. For more information on Oracle
RAC software upgrade, see Oracle documentation.
Keeping Kernels Consistent

If you change kernel parameters or perform network tuning with ndd as part of doing a rolling
upgrade, be sure to change the parameters to the same values on all nodes that can run the same
packages in a failover scenario. The ndd command allows the examination and modification of
several tunable parameters that affect networking operation and behavior.
Example of Rolling Upgrade

The following example shows a simple rolling upgrade on two nodes, each running standard
Serviceguard and RAC instance packages, as shown in Figure 20. (This and the following figures
show the starting point of the upgrade as SGeRAC A.11.15 for illustration only. A roll to SGeRAC
version A.11.16 is shown.)
SGeRAC rolling upgrade requires the same operating system version on all nodes. However,
during rolling upgrade the nodes can run on mixed version of HP-UX. The example assumes all
nodes are running HP-UX 11i v2. For your systems, substitute the actual release numbers of your
rolling upgrade path.
NOTE: While you are performing a rolling upgrade, warning messages may appear while the
node is determining what version of software is running. This is a normal occurrence and not a
cause for concern.
143
Figure 20 Running Cluster Before Rolling Upgrade
Step 1.
1.
2.
Halt Oracle (RAC, Clusterware) software on node 1.

Halt node 1. This will cause the nodes packages to start up on an adoptive node. You can
do this in Serviceguard Manager, or from the command line issue the following:
# cmhaltnode -f node1
This will cause the failover package to be halted cleanly and moved to node 2. The Serviceguard
daemon on node 1 is halted, and the result is shown in Figure 21.
Figure 21 Running Cluster with Packages Moved to Node 2
Step 2.
Upgrade node 1 and install the new version of Serviceguard and SGeRAC (A.11.16), as shown
in Figure 22.
144 Software Upgrades
NOTE: If you install Serviceguard and SGeRAC separately, Serviceguard must be installed before
installing SGeRAC.
Figure 22 Node 1 Upgraded to SG/SGeRAC 11.16
Step 3.
1.
If you prefer, restart the cluster on the upgraded node (node 1). You can do this in Serviceguard
Manager, or from the command line issue the following:
# cmrunnode node1
2.
3.
At this point, different versions of the Serviceguard daemon (cmcld) are running on the two
nodes, as shown in Figure 23.
Start Oracle (Clusterware, RAC) software on node 1.
Figure 23 Node 1 Rejoining the Cluster
145
Step 4.
1.
2.
Halt Oracle (RAC, Clusterware) software on node 2.

Halt node 2. You can do this in Serviceguard Manager, or from the command line issue the
following:
# cmhaltnode -f node2
This causes both packages to move to node 1. See Figure A-5.

3.
4.
Upgrade node 2 to Serviceguard and SGeRAC (A.11.16) as shown in Figure A-5.

When upgrading is finished, enter the following command on node 2 to restart the cluster on
node 2:
# cmrunnode node2
5.
Start Oracle (Clusterware, RAC) software on node 2.
Figure 24 Running Cluster with Packages Moved to Node 1
Step 5.
Move PKG2 back to its original node. Use the following commands:
# cmhaltpkg pkg2
# cmrunpkg -n node2 pkg2
# cmmodpkg -e pkg2
The cmmodpkg command re-enables switching of the package that is disabled by the cmhaltpkg
command. The final running cluster is shown in Figure 25.
146
Software Upgrades
Figure 25 Running Cluster After Upgrades
Limitations of Rolling Upgrades

The following limitations apply to rolling upgrades:
During a rolling upgrade, you should issue Serviceguard/SGeRAC commands (other than
cmrunnode and cmhaltnode) only on a node containing the latest revision of the software.
Performing tasks on a node containing an earlier revision of the software will not work or will
cause inconsistent results.
You cannot modify the cluster or package configuration until the upgrade is complete. Also,
you cannot modify the hardware configurationincluding the clusters network
configurationduring a rolling upgrade. This means that you must upgrade all nodes to the
new release before you can modify the configuration file and copy it to all nodes.
The new features of the Serviceguard/SGeRAC release may not work until all nodes have
been upgraded.
Binary configuration files may be incompatible between releases of Serviceguard/SGeRAC.

Do not manually copy configuration files between nodes.
Within a Serviceguard/SGeRAC cluster, no more than two versions of Serviceguard and

SGeRAC can be running while the rolling upgrade is in progress.
You can perform a rolling upgrade only on a configuration that has not been modified since
the last time the cluster was started.
Rolling upgrades are not intended as a means of using mixed releases of HP-UX, Serviceguard,
or SGeRAC within the same cluster. HP recommends to upgrade all the cluster nodes to the
new release level at the earliest.
You can upgrade OS during rolling upgrade only if your cluster uses SLVM over Raw disks,
ASM over Raw disks, or ASM over SLVM configuration for the shared storage.
For more information see, http://www.hp.com/go/hpux-serviceguard-docs > HP
Serviceguard manual.
147
For more information on support, compatibility, and features for SGeRAC, refer to the
Serviceguard and Serviceguard Extension for RAC Compatibility and Feature Matrix, located
at www.hp.com/go/hpux-serviceguard-docs > HP Serviceguard Extension for RAC .
You cannot delete Serviceguard/SGeRAC software (via swremove) from a node while the
cluster is in the process of a rolling upgrade.
Non-Rolling Software Upgrades

A non-rolling upgrade allows you to perform a software upgrade from any previous revision to
any higher revision or between operating system versions. For example, you may do a non-rolling
upgrade from SGeRAC A.11.14 on HP-UX 11i v1 to A.11.16 on HP-UX 11i v2, given both are
running the same architecture.
The cluster cannot be running during a non-rolling upgrade, therefore it is necessary to halt the
entire cluster in order to perform the upgrade.
Use the following steps for a non-rolling software upgrade:
1. Halt Oracle (RAC, Clusterware) software on all nodes in the cluster.
2. Halt all nodes in the cluster.
# cmhaltcl -f
3.
4.
5.
If necessary, upgrade all the nodes in the cluster to the new HP-UX release.
Upgrade all the nodes in the cluster to the new Serviceguard/SGeRAC release.
Restart the cluster. Use the following command:
# cmruncl
6.
7.
If necessary, upgrade all the nodes in the cluster to the new Oracle (RAC, CRS, Clusterware)
software release.
Restart Oracle (RAC, Clusterware) software on all nodes in the cluster and configure the
Serviceguard/SGeRAC packages and Oracle as needed.
Limitations of Non-Rolling Upgrades

The following limitations apply to non-rolling upgrades:
Binary configuration files may be incompatible between releases of Serviceguard. Do not

manually copy configuration files between nodes.
It is necessary to halt the entire cluster when performing a non-rolling upgrade.
Migrating an SGeRAC Cluster with Cold Install

There may be circumstances when you prefer a cold install of the HP-UX operating system rather
than an upgrade. The cold install process erases the preexisting operating system and data, and
then installs the new operating system and software. After the cold install, you must then restore
the data.
CAUTION: The cold install process erases the preexisting software, operating system, and data.
If you want to retain any existing software, make sure to back up that software before migrating.
Use the following process as a checklist to prepare the migration.
1. Back up the required data, including databases, user and application data, volume group
configurations, etc.
2. Halt cluster applications, including RAC, and then halt the cluster.
3. Do a cold install of the HP-UX operating system. For more information on the cold install
process, see the HP-UX Installation and Update Guide located at www.hp.com/go/
hpux-core-docs > By OS Release > Setup and install - general.
4. Install additional required software that did not come with your version of HP-UX OE.
5. Install a Serviceguard/SGeRAC version that is compatible with the new HP-UX operating
system version. For more information on support, compatibility, and features for SGeRAC,
148
Software Upgrades
refer to the Serviceguard Compatibility and Feature Matrix, located at www.hp.com/go/

hpux-serviceguard-docs > HP Serviceguard Extension for RAC.
6. Recreate any user accounts needed for the cluster applications.
7. Recreate the network and storage configurations (Set up stationary IP addresses and create
LVM volume groups and/or CVM disk groups required for the cluster).
8. Recreate the SGeRAC cluster.
9. Restart the cluster.
10. Reinstall the cluster applications, such as RAC.
11. Restore the data.
Upgrade Using DRD

DRD stands for Dynamic Root Disk. Using a Dynamic Root Disk on HP-UX 11i v3 allows you to
perform the update on a clone of the root disk, then halt the node and reboot it from the updated
clone root disk.
You can obtain the DRD software free from www.software.hp.com search for DynRootDisk.
For more information, go to HP's Dynamic Root Disk Information Library at www.hp.com/go/drd.
IMPORTANT: Use the clone disk only on the system that it was created. Serviceguard does not
support booting from a clone disk made on another system (sometimes referred to as DRD
re-hosting).
Rolling Upgrade Using DRD

A rolling upgrade using DRD is like a rolling upgrade, but is even less disruptive because each
node is down for a shorter time. It is also very safeif something goes wrong you can roll back
to the original (pre-upgrade) state by rebooting from the original disk.
This method is the least disruptive, but you need to make sure your cluster is eligible. See Restrictions
for DRD Upgrades (page 149).
If, after reading and understanding the restrictions, you decide to perform a rolling upgrade using
DRD, follow the instructions under Performing a Rolling Upgrade Using DRD in Appendix D of
the latest edition of Managing Serviceguard, at www.hp.com/go/hpux-serviceguard-docs > HP
Serviceguard.
Non-Rolling Upgrade Using DRD

In a non-rolling upgrade with DRD, you clone each node's root disk, apply the upgrade to the
clone, halt the cluster, and then reboot each node from its updated clone root disk.
This method involves much less cluster down time than a conventional non-rolling upgrade, and is
particularly safe because the nodes can be quickly rolled back to their original (pre-upgrade) root
disks. But you must make sure your cluster is eligible. See Restrictions for DRD Upgrades (page 149).
Restrictions for DRD Upgrades
Before you proceed, read the sections Upgrading from an Earlier Serviceguard Release and
Rolling Upgrade in the latest version of the release notes for A.11.20.
Serviceguard A.11.20 is supported on HPUX 11i v3 only. For more information, see HP-UX
11i v3 Installation and Update Guide at www.hp.com/go/ hpux-core-docs > HP-UX 11i
v3.
You can perform a rolling upgrade from A.11.19 to a later release, or from an earlier release
to A.11.19, but you cannot do a rolling upgrade from a pre-A.11.19 release to a post-A.11.19
release.
This is because A.11.19 is the only version of Serviceguard that will allow both the older
version of the cluster manager and the new version (introduced in A.11.19) to coexist during
a rolling upgrade.
If you are upgrading from a pre-A.11.19 release: Start by reading Upgrading from an Earlier
Serviceguard Release and Rolling Upgrade in the release notes. Then, if you decide to upgrade
Upgrade Using DRD
149
to A.11.19 in preparation for a rolling upgrade to A.11.20, continue with the following
subsection that provides information on upgrading to A.11.19.
150
Software Upgrades
B Blank Planning Worksheets

This appendix reprints blank planning worksheets used in preparing the RAC cluster. You can
duplicate any of these worksheets that you find useful and fill them in as a part of the planning
process.
LVM Volume Group and Physical Volume Worksheet

VG and PHYSICAL VOLUME WORKSHEET
Page ___ of ____
==========================================================================
Volume Group Name: ______________________________________________________
PV Link 1
PV Link2
Physical Volume Name:_____________________________________________________
Physical Volume Name: ____________________________________________________
Volume Group Name: _______________________________________________________
PV Link 1
PV Link2
Physical Volume Name: _____________________________________________________

Physical Volume Name:______________________________________________________
Oracle Logical Volume Worksheet

NAME
SIZE
Oracle Control File 1:
_____________________________________________________
_____________________________________________________
_____________________________________________________
Instance 1 Redo Log 1:
_____________________________________________________
_____________________________________________________
_____________________________________________________
Instance 1 Redo Log:
_____________________________________________________
LVM Volume Group and Physical Volume Worksheet
151
152
_____________________________________________________
_____________________________________________________
_____________________________________________________
_____________________________________________________
_____________________________________________________
_____________________________________________________
Data: System
_____________________________________________________
Data: Rollback
_____________________________________________________
Data: Temp
_____________________________________________________
Data: Users
_____________________________________________________
Data: Tools
_____________________________________________________
Blank Planning Worksheets
Index
A
activation of volume groups
in shared mode, 128
administration
cluster and package states, 113
array
replacing a faulty mechanism, 132, 133, 134
B
building a cluster
CVM infrastructure, 52
building an RAC cluster
displaying the logical volume infrastructure, 46
logical volume infrastructure, 40
building logical volumes
for RAC, 45
C
CFS, 47, 51
cluster
state, 118
status options, 116
Cluster Communication Network Monitoring, 35
cluster volume group
creating physical volumes, 41
creating a storage infrastructure, 47
CVM
creating a storage infrastructure, 52
use of the CVM-pkg, 55
D
deactivation of volume groups, 128
deciding when and where to run packages, 18
deleting from the cluster, 51
deleting nodes while the cluster is running, 130
demo database
files, 45, 57
disk
choosing for volume groups, 41
disk arrays
creating logical volumes, 44
disk storage
creating the infrastructure with CVM, 52
disks
replacing, 132
Dynamic Root Disk (DRD), 140
E
eight-node cluster with disk array
figure, 22
EMS
for preventive monitoring, 131
enclosure for disks
replacing a faulty mechanism, 132
Event Monitoring Service
in troubleshooting, 131
exporting
shared volume group data, 46
exporting files
LVM commands, 46
F
figures
eight-node cluster with EMC disk array, 22
node 1 rejoining the cluster, 145
node 1 upgraded to HP-UX 111.00, 145
running cluster after upgrades, 147
running cluster before rolling upgrade, 144
running cluster with packages moved to node 1, 146
running cluster with packages moved to node 2, 144
H
HAIP, 35
hardware
adding disks, 131
monitoring, 131
heartbeat subnet address
parameter in cluster manager configuration, 35
high availability cluster
defined, 12
I
in-line terminator
permitting online hardware maintenance, 134
installing
Oracle RAC, 47
installing software
Serviceguard Extension for RAC, 33
IP address
switching, 19
L
lock disk
replacing a faulty mechanism, 134
logical volumes
blank planning worksheet, 151
creating, 45
creating for a cluster, 42, 56, 57
creating the infrastructure, 40
disk arrays, 44
filled in planning worksheet, 31
lssf
using to obtain a list of disks, 41
LVM
creating on disk arrays, 44
LVM commands
exporting files, 46
M
maintaining a RAC cluster, 113
maintenance
153
adding disk hardware, 131

making changes to shared volume groups, 128
monitoring hardware, 131
N
network
status, 118
node
halting status, 121
in an RAC cluster, 12
status and state, 116
non-rolling upgrade
DRD, 149
O
online hardware maintenance
by means of in-line SCSI terminators, 134
Online node addition and deletion, 126
Online reconfiguration, 126
opsctl.ctl
Oracle demo database files, 45, 57
opslog.log
Oracle
demo database files, 45, 57
Oracle 10 RAC
installing binaries, 62
Oracle 10g/11gR2 RAC
introducing, 25
Oracle Disk Manager
configuring, 64
Oracle RAC
installing, 47
Oracle10g
installing, 62
P
package
basic concepts, 13
moving status, 120
state, 118
status and state, 116
switching status, 121
package configuration
service name parameter, 35
packages
deciding where and when to run, 18
physical volumes
creating for clusters, 41
filled in planning worksheet, 151
planning
worksheets for logical volume planning, 31
worksheets for physical volume planning, 151
planning worksheets
blanks, 151
point to point connections to storage devices, 21
PVG-strict mirroring
creating volume groups with, 42
154 Index
R
RAC
overview of configuration, 12
status, 117
RAC cluster
defined, 12
removing Serviceguard Extension for RAC from a system,,
130
replacing disks, 132
rollback.dbf
Oracle demo database files, 45, 46, 58
rolling software upgrades
example, 143
steps, 142
rolling upgrade
DRD, 149
limitations, 147, 148
S
service
status, 117
service name
parameter in package configuration, 35
SERVICE_NAME
parameter in package configuration, 35
Serviceguard Extension for RAC
installing, 33
introducing, 12
shared mode
activation of volume groups, 128
deactivation of volume groups, 128
shared volume groups
making volume groups shareable, 127
sharing volume groups, 46
SLVM
making volume groups shareable, 127
state
cluster, 118
node, 116
of cluster and package, 113
package, 116, 118
status
cluster, 116
halting node, 121
moving package, 120
network, 118
node, 116
normal running RAC, 119
of cluster and package, 113
package, 116
RAC, 117
service, 117
switching package, 121
Storage Management Suite (SMS) , 16
switching IP addresses, 19
system multi-node package
used with CVM, 55
system.dbf
T
temp.dbf
troubleshooting
monitoring hardware, 131
replacing disks, 132
U
upgrade
DRD, 149
upgrade restrictions
DRD, 149
V
volume group
creating for a cluster, 42
creating physical volumes for clusters, 41
volume groups
adding shared volume groups, 130
displaying for RAC, 46
exporting to other nodes, 46
making changes to shared volume groups, 128
making shareable, 127
making unshareable, 128
W
worksheet
logical volume planning, 31
worksheets
physical volume planning, 151
worksheets for planning
blanks, 151
155

Using Serviceguard Extension For RAC

Caricato da

Informazioni sul documento

Titolo originale

Copyright

Formati disponibili

Condividi questo documento

Condividi o incorpora il documento

Opzioni di condivisione

Hai trovato utile questo documento?

Questo contenuto è inappropriato?

Copyright:

Formati disponibili

Using Serviceguard Extension For RAC

Caricato da

Copyright:

Formati disponibili

Using Serviceguard Extension for RAC

HP Part Number: 5900-1887

2 Serviceguard Configuration for Oracle 10g, 11gR1, or 11gR2 RAC.................25

Mirror Detachment Policies with CVM..........................................................................55

3 Support of Oracle RAC ASM with SGeRAC.................................................69

4 SGeRAC Toolkit for Oracle RAC 10g or later...............................................80

Startup and shutdown of the combined Oracle RAC-SGeRAC stack .........................................85

Removing Serviceguard Extension for RAC from a System..........................................................130

B Blank Planning Worksheets......................................................................151

Advantages of using SGeRAC

User Guide Overview

Chapter 1 Introduction to Serviceguard Extension for RAC

Chapter 2 Serviceguard Configuration for Oracle 10g, 11gR1, or 11gR2 RAC

Chapter 3 Oracle Automatic Storage Management (ASM)

Chapter 4 SGeRAC Toolkit for Oracle RAC 10g or later

Appendix A Software Upgrades

Appendix B Blank Planning Worksheets

To view the following documents, go to www.hp.com/go/hpux-core-docs and www.hp.com/

VERITAS Volume Manager Administrators Guide (includes a glossary of VERITAS terminology).

VERITAS Volume Manager Storage Administrator Administrators Guide.

VERITAS Volume Manager Reference Guide.

VERITAS Volume Manager Migration Guide.

VERITAS Volume Manager for HP-UX Release Notes.

Text displayed by the computer.

Commands and other text that you type.

A command name or qualified command phrase.

The name of a variable that you may replace in a command or function or

The preceding element may be repeated an arbitrary number of times.

Separates items in a list of choices.

Where to find Documentation on the Web

Clusters for High Availability: a Primer of HP Solutions. Hewlett-Packard Professional

Serviceguard Extension for RAC Version A.11.20 Release Notes

HP Serviceguard Version A.11.20 Release Notes

Managing Serviceguard Eighteenth Edition

Using High Availability Monitors

Using the Event Monitoring Service

Using Advanced Tape Services

Managing Systems and Workgroups

Managing Serviceguard NFS

1 Introduction to Serviceguard Extension for RAC

What is a Serviceguard Extension for RAC Cluster? (page 12)

Serviceguard Extension for RAC Architecture (page 14)

Overview of SGeRAC Cluster Interconnect Subnet Monitoring (page 17)

Node Failure (page 19)

Larger Clusters (page 20)

What is a Serviceguard Extension for RAC Cluster?

Introduction to Serviceguard Extension for RAC

Using Packages in a Cluster

Failover package. The typical high-availability package is a failover package. It usually is

Serviceguard Extension for RAC Architecture

Serviceguard Extension for RAC Components

Custom Oracle Database

Volume Manager Software

Group Membership Daemon

cmgmsdGroup Membership Daemon for RAC 10g or later

Overview of SGeRAC and Cluster File System (CFS)/Cluster Volume

Introduction to Serviceguard Extension for RAC

Storage Configuration Options

About Veritas CFS and CVM from Symantec

Impact on Serviceguard SMS installation

Installation of Serviceguard SMS follows installation

Overview of SGeRAC and Oracle 10g, 11gR1, and 11gR2 RAC

Introduction to Serviceguard Extension for RAC