Installation RAC10GR2 QFS

Document: Installation_RAC10GR2_QFS.
odt
Author: Michael Ahrens, SUN Microsystems GmbH
SUN Microsystems
GmbH
Oracle RAC on Sun Cluster and QFS
Author: Michael Ahrens

SUN Microsystems GmbH
Created 07.06.2006 12:00:00
Last change: 09/12/2006 10:31:28 vorm.
Version: 0.1
Page 1 of 90
Document: Installation_RAC10GR2_QFS.odt
Table of Contents
Table of Contents......................................................................................................................................................2
Document Info...........................................................................................................................................................4
Content..............................................................................................................................................................4
Hardware Installation................................................................................................................................................5
Architecture.......................................................................................................................................................5
Drawing.............................................................................................................................................................5
Solaris Installation.....................................................................................................................................................6
Solaris 10 u1......................................................................................................................................................6
Setup slvm.........................................................................................................................................................6
EIS-CD..............................................................................................................................................................7
QFS 4.5..............................................................................................................................................................7
Cluster Installation....................................................................................................................................................8
SunCluster 3.1U4..............................................................................................................................................8
Setup IPMP groups on all nodes for all public network interfaces.................................................................11
Modify NVRAM parameter:...........................................................................................................................12
Create Users and Groups.................................................................................................................................12
Check SSH daemon.........................................................................................................................................12
Setup SSH for oracle.......................................................................................................................................12
Setup userenv for oracle..................................................................................................................................13
Install the RAC/OPS Service from the CD.....................................................................................................14
Install udlm......................................................................................................................................................14
Workaround Bug.............................................................................................................................................15
Set kernel parameters......................................................................................................................................17
Create cluster resourcegroup for rac...............................................................................................................17
Create multi-owner disk set.............................................................................................................................18
Configure QFS software on all nodes.............................................................................................................19
Create global mountpoint and vfstab entries on all nodes..............................................................................20
Create cluster resourcegroup for qfs...............................................................................................................20
Last Systemanalyse.........................................................................................................................................22
Prepare Oracle Installation......................................................................................................................................23
Share staging area............................................................................................................................................23
Mount staging area..........................................................................................................................................23
Perform cluster check......................................................................................................................................23
Install Clusterware 10.2.0.1.....................................................................................................................................24
Run installer from ora-rac-3............................................................................................................................24
Running vipca.................................................................................................................................................34
Check Cluster Status.......................................................................................................................................40
Install crsstat.sh...............................................................................................................................................41
Install Database Home 10.2.0.1..............................................................................................................................42
Install Management Agent......................................................................................................................................51
Install Patchset 10.2.0.2 on CRS Home..................................................................................................................60
Stop all oracle processes.................................................................................................................................60
Install binaries.................................................................................................................................................60
Restart Custerware..........................................................................................................................................65
Install Patchset 10.2.0.2 on DB Home....................................................................................................................66
Install binaries.................................................................................................................................................66
Install Interim Patch 5117016 on DB Home...........................................................................................................71
Install with opatch...........................................................................................................................................71
Create Database ZDA..............................................................................................................................................74
Run DBCA......................................................................................................................................................74
Cluster Optimiziation..............................................................................................................................................78
Apply Brownout Patches.................................................................................................................................78
Set the CSS misscount to match the service level...........................................................................................78
Run CSS in Real Time....................................................................................................................................78
Set the check_interval for VIP addresses........................................................................................................78
Optimize VIP Failover............................................................................................................................................80
Adjusting Listener Configuration....................................................................................................................80
Page 2 of 90
Addendum...............................................................................................................................................................82
Metalink Note 284752.1..................................................................................................................................82
Metalink Note 294430.1..................................................................................................................................83
Metalink Note 294430.1..................................................................................................................................89
Page 3 of 90
Document Info
Content
This is an installation summary of the installation and patch process of the RAC environment. It is
based on the official procuct documentation provided by SUN and Oracle and cannot replace those doc-
uments. The OS and Clusterinstallation was strongly based on EIS done.
Page 4 of 90
Hardware Installation
Architecture
The cluster is based on two SF440 with 4 CPU's, 16GB RAM and 4 internal Disks. The internaldisks
are mirrored with SLVM. Two disks are used for the OS and the other two are used for Oracle. Oracle
will not be installed on shared disks because of the rolling upgrade capability. The shared disks are luns
from a SAN and they are mirrored with SLVM/OBAN, filesystem is QFS 4.5. The clusterinterconnect
is realized with four switched gigabitethernet connections.
Drawing
IPMP Group IPMP Group

ipmp0 ipmp0
4 x Interconnect
Oracle RAC Cluster Oracle RAC Cluster

Node: ora-rac-3 MpxIO MpxIO Node: ora-rac-4
Storage A Storage B
Page 5 of 90
Solaris Installation
Solaris 10 u1
Solaris 10 u1 SUNWCXall is installed.
• Do not forget the 512Mb (SC3.1 strikly recommended) partition for /globaldevices
Mirrored by SLVM, on every clusternode another metadevicename for this filesystem must be used.
ora-rac-3 d30
ora-rac-4 d35
• For SVM reserve slice 7 with 32MB (EIS recommendation) for metaDB/replicas.
• Ensure that the SAN Foundation Suite packages are installed and patched. (SAN is necessary for fibre
server/storage)
change the home of root

#mkdir /root
#vi /etc/passwd
root:x:0:0:Super-User:/root:/sbin/sh
logout,login
Setup slvm
#cat /etc/lvm/md.tab
mddb1 -c 3 /dev/dsk/c3t0d0s7
d10 -m d11
d11 1 1 /dev/dsk/c3t0d0s0
d20 -m d21
d30 -m d31
Page 6 of 90
EIS-CD
Setup EIS environment

cd /cdrom/eis-dvd/sun/install
./setup-standard.sh
Install actual Solaris 10 patches

cd /cdrom/eis-dvd/patch/10
unpack-patches
cd /tmp/10
./install-patches
Analyse basic systemstatus

explorer
cd /opt/SUNWexplo/output/explorer*
/cdrom/eis-dvd/sun/tools/explominer/explominer
see the outputfile ExploMiner_EIS_Patchlist_nobody of the explominer and in-

stall all neccessary patches.
QFS 4.5
cd /tmp/qfs_packages
pkgadd -d .
choose both qfs packages
install Patch: 122807-04
Page 7 of 90
Cluster Installation
SunCluster 3.1U4
Configure /etc/inet/hosts.
127.0.0.1 localhost
192.168.10.238 ora-rac-4-1
192.168.10.237 ora-rac-4-2
192.168.10.215 ora-rac-3
172.16.193.1 ora-rac-3-priv
192.168.10.223 ora-rac-3-vip
192.168.10.217 ora-rac-4 loghost ora-rac-4.
172.16.193.2 ora-rac-4-priv
192.168.10.239 ora-rac-4-vip
Additionally for Solaris10 configure /etc/inet/ipnodes

Enter all cluster nodes and all the logical Hosts.
Do not configure cluster nodes as routers!
Set up file /etc/defaultrouter scinstall will touch file /etc/notrouter per default.
defaultrouter = 192.168.10.254
When using supported network adapters which use the *ce* driver for private transport, insert into file
/etc/system:
set ce:ce_taskq_disable=1
To enable MpxIO on Solaris 10 start
stmsboot -e before SunCluster will be installed !!

Do not run this command if SunCluster is already installed.
For safety reasons clean the devices with:
devfsadm -C on all nodes.
Notice: For use of VLANs for the cluster transport:

• one separated VLAN per cluster transport,
• no network traffic should be possible between the different cluster transport VLANs,
• VLANs should behave like separated physical segments.
Page 8 of 90
Configure
#ndd -set /dev/ip ip_strict_dst_multihoming 0
You should do this in a startscript. You can use S68nettune from EIS-DVD.
See also SRDB 73036
Copy the SunCluster Software from the EIS-DVD to the disk and bunzip and untar this file.
/cdrom/eisdvd/suninternal/progs/SunCluster/3.1_U4/suncluster_sc31u4_sparc_EIS
.tar.bz2
Edit XML-File and remove the Explorer entries, because within the software is a very old version of the
explorer.
vi ./suncluster_sc31u4_sparc/Solaris_sparc/.install/config/PPXMLS/SunCluster.xml
Remove following entries:
<forward_dependency id="Explorer" base_point_product_id="Explorer"
topology_category="Host-Based" subcomp_refs="Explorer">
<description>"Explorer is required for NC to be installed"</description>
<lowest_compat_version_noupgrade>
<versionInfo versionNo="4.2"/>
</lowest_compat_version_noupgrade>
</forward_dependency>
Explorer,
Use the Java ES installer for package installation.

(Optional): To use the GUI of the installer
be sure that the DISPLAY is set correctly.
# ./installer
do NOT install JES Agents at this time !
Choose 'Configure Later'
'Configure Now' does not work this time!
Install/Establish SunCluster 3.1U4 on ora-rac-3 using scinstall.
Select menu 1)
Interactive Q+A
(Partition for /globaldevices required.)
If you not install package SUNWescom disable the scsymon service:
# svcadm -v disable /system/cluster/scsymon-srv
Page 9 of 90
Install Sun Cluster patches: Note the structure under .../sun2/patch/SunCluster3.x/ ..3.1 (containing
sub-directories, 8, 9, 10, 9_x86, 10_x86) ..agents (containing sub-directories common, 8, 9, 10,
10_x86)
the following core patch is required for SC3.1U4:
• 120500-04 Sun Cluster 3.1 Core Patch for Solaris 10
Page 10 of 90
Setup IPMP groups on all nodes for all public network interfaces.
At least you need one test IP for each network adapter and one production IP for each IPMP group. e.g:
IPMP group active-active with interfaces ce0 and ce4 with one physical IP.
ora-rac-3
Entry of /etc/hostname.ce0 :
ora-rac-3-1 netmask + broadcast + group ipmp0 up \
addif ora-rac-3 netmask + broadcast + deprecated -failover up
ora-rac-3-2 netmask + broadcast + group ipmp0 deprecated -failover up
ora-rac-4
ora-rac-4-1 netmask + broadcast + group ipmp0 up \
addif ora-rac-3 netmask + broadcast + deprecated -failover up
ora-rac-4-2 netmask + broadcast + group ipmp0 deprecated -failover up
If the defaultrouter is NOT 100% available please read infodocs

70060 and 70062.
Notes:
• Do not use test IP for normal applications
• Test IP for all adapters in the same IPMP group must belong to a single IP subnet
• You need an addtional IP for each logical host.
• In the /etc/default/mpathd file, the value of

TRACK_INTERFACES_ONLY_WITH_GROUPS must be yes. (default)
Common actions for filesystems and boot devices:

Set localonly flag for the did device of the root mirror disk:
• make sure only local node in node list:
# scconf -pvv | grep dsk/d<N>
• if other nodes in node list, remove them:
# scconf -r -D name=dsk/d<N>,nodelist=<other_phy_host>
• set localonly on root mirror disk:
# scconf -c -D name=dsk/d<N>,localonly=true
NOTE: To find out the did device use scdidadm -l command!

Enter global filesystems into /etc/vfstab on all nodes. Recommendation:
mount archivelog destinaton FS as an QFS Filesystem for each instance not as an global filesystem.
Page 11 of 90
See later in this document for qfs. Create all mountpoints on all nodes.
Add global and logging option in /etc/vfstab if you use UFS.
Modify NVRAM parameter:

Set boot-device & diag-device to both sides of the mirror.
Suggested naming convention:

• Primary: rootdisk
• Secondary: rootmirror
Create Users and Groups

To be done on all node of the cluster
/usr/sbin/groupadd -g 61000 oinstall
/usr/sbin/groupadd -g 61001 dba
/usr/sbin/useradd -u 61000 -g oinstall -G dba -s /usr/bin/bash -d
/export/home/oracle oracle
mkdir -p /export/home/oracle
chown -R oracle:oinstall /export/home/oracle
passwd oracle
> oracle
> oracle
Check SSH daemon

ps -ef | grep sshd
Setup SSH for oracle

Note: This steps are different for any node in the cluster. Do complete the steps on the indicated nodes.
Steps to be done on ora-rac-3 and ora-rac-4 :
su - oracle
mkdir ~/.ssh
chmod 700 ~/.ssh
/usr/bin/ssh-keygen -t rsa
/usr/bin/ssh-keygen -t dsa
Steps to be done host on ora-rac-3:

cd .ssh
cat id_dsa.pub > authorized_keys
cat id_rsa.pub >> authorized_keys
scp authorized_keys ora-rac-4:/export/home/oracle/.ssh/authorized_keys
Page 12 of 90

cat id_dsa.pub >> authorized_keys
cat id_rsa.pub >> authorized_keys

scp ora-rac-4:/export/home/oracle/.ssh/authorized_keys authorized_keys
Steps to be done on ora-rac-3 and ora-rac-4:

ssh ora-rac-3 date
ssh ora-rac-4 date
exec /usr/bin/ssh-agent $SHELL
/usr/bin/ssh-add
Setup userenv for oracle

# add to ~/.profile
. ~/set_oracle_env.sh
PS1='${LOGNAME}:${ORACLE_SID}> '
export PS1
# Set Environment for oracle
#!/usr/bin/bash
clear
echo "Choose your Oracle Home and hit RETURN"
echo "Enter 1 to use Oracle Home DB"
echo "Enter 2 to use Oracle Home Agent"
echo "Enter 3 to use Oracle Home CRS"
echo "Hit any other key to skip Oracle Home selection"
read answer
if [[ $answer = 1 ]]
then
ORACLE_HOME=/oracle/product/db ;
ORACLE_SID=ZDA1
elif [[ $answer = 2 ]]
then
ORACLE_HOME=/oracle/product/agent/agent10g ;
elif [[ $answer = 3 ]]
then
ORACLE_HOME=/oracle/product/crs ;
else
return
fi
export PATH=$ORACLE_HOME/bin:/oracle/Opatches/OPatch:$PATH
ORACLE_BASE=/oracle
export ORACLE_BASE
Page 13 of 90
export ORACLE_SID
export ORACLE_HOME
EDITOR=vi
export EDITOR
if [ -t 0 ]; then
stty intr ^C
fi
# stty erase ^?
alias sql='sqlplus / as sysdba'
alias crsi='/oracle/product/crs/bin/crs_stat -t'
alias xxx=". ~/set_oracle_env.sh"
alias oh='cd $ORACLE_HOME'
alias al='vi /oracle/admin/ZDA/bdump/alert_${ORACLE_SID}.log'
echo "Tip: Enter xxx to change your current setting"
Install the RAC/OPS Service from the CD

Via pkgadd on all nodes. You always need SUNWscucm SUNWudlm SUNWudlmr.
• For SVM you require SUNWscmd.
• SUNWschwr is needed for Shared QFS for Hardware Raid Support.
Install udlm
Remark: Don't use the version from the EIS DVD, for Oracle 10G the asynchronous version is re-
quired, on the EIS DVD is only the synchronous version available. See chapter „Prepair Oracle Installa-
tion“ for the Installationmedias.
#cd /install/oracle/stage/cl/racpatch
#uncompress ORCLudlm.tar.Z;tar xf ORCLudlm.tar
#pkgadd -d .
#The following packages are available:
1 ORCLudlm Oracle UNIX Distributed Lock Manager
(sparc) Dev Release 06/11/04, 64bit 3.3.4.8 reentrant, async libskgxn2.so
Select package(s) you wish to process (or 'all' to process

all packages). (default: all) [?,??,q]:
Page 14 of 90
Workaround Bug
The OUI may fail if it does not find ssh and scp binaries in /usr/local/bin.
mkdir -p /usr/local/bin
ln -s /usr/bin/ssh /usr/local/bin/ssh
ln -s /usr/bin/scp /usr/local/bin/scp
If you use Solaris Secure Shell (sshd) with enabled X11 forwarding and udlm you should read Sun Al-
ert 57428.
Configure 'udlm.port' and 'udlm.num_ports' in /opt/SUNWudlm/etc/udlm.conf
# Modified by rac_udlm_init on Tue May 9 09:14:16 CEST 2006
#
# ident "@(#)udlm.conf.template 1.5 03/04/15 SMI"
#
# Copyright 1999-2003 Sun Microsystems, Inc. All rights reserved.
# Use is subject to license terms.
#
# This file has been created by SUNW.rac_udlm resource type
# callback programs.
#
# Please do not edit this file.
#
udlm.oracle.config.file : /etc/opt/SUNWcluster/conf/udlm.conf
udlm.schedclass : RT
udlm.schedpriority : 11
udlm.port : 7000
udlm.num_ports : 32
udlm.failfastmode : panic
udlm.start_timeout : 100
udlm.step1_timeout : 100
udlm.abort_timeout : 325
#
udlm.resource_name : rac_udlm_rs
udlm.resource_group_name : rac_rg
#
# Cluster.number.nodes should be obtained dynamically

#
Page 15 of 90
udlm.nodes : ${cluster.number.nodes}
#
#
udlm.clustname : boschpoc
udlm.server.0.hname : 172.16.193.1
udlm.server.1.hname : 172.16.193.2
udlm.server.2.hname : ${cluster.node.2.hahost}
#
# Relate CMM steps to UDLM steps
#
# Format:
# ucmm.timeout.<stepname>.<component order> : <value>
#
#
ucmm.timeout.cmmstep2.05 : ${udlm.step1_timeout}
ucmm.timeout.cmmstart.05 : ${udlm.start_timeout}
ucmm.timeout.cmmabort.05 : ${udlm.abort_timeout}
ucmm.timeout.cmmstop.05 : ${udlm.stop_timeout}
Page 16 of 90
Set kernel parameters
* Oracle settings *
set noexec_user_stack=1
set rlim_fd_cur=8192
• end of Oracle settings *
Create cluster resourcegroup for rac

# scrgadm -a -t SUNW.rac_framework
# scrgadm -a -t SUNW.rac_udlm
# scrgadm -a -t SUNW.rac_svm
# scrgadm -a -g rac_rg -y maximum_primaries=2 -y desired_primaries=2 \
-y nodelist=ora-rac-3,ora-rac-4
# scrgadm -a -j rac_framework_rs -g rac_rg -t SUNW.rac_framework
# scrgadm -a -j rac_udlm_rs -g rac_rg -t SUNW.rac_udlm \
-y resource_dependencies=rac_framework_rs -x port=7000
# scrgadm -a -j rac_svm_rs -g rac_rg -t SUNW.rac_svm \
-y resource_dependencies=rac_framework_rs
Increase svm_step4_timeout if using SVM and RAC is managed by the RGM.

# scrgadm -c -j rac_svm_rs -x svm_step4_timeout=360
The value 360 is an example!
See FIN I1139-1. Determine the best value for your configuration!
This is necessary when your SVM shared disk group setup is large or complex.
Switch the Resource Group online.

• # scswitch -Z -g rac_rg
Page 17 of 90
Create multi-owner disk set
#metaset -s <setname> -M -a -h <nodelist>

NOTE (only SC3.1U4): If you get the message
metaset: host1: rac-ds: node host2 is not in membership list.
you should add on both nodes to /var/run/nodelist
1 ora-rac-3 172.16.193.1
2 ora-rac-4 172.16.193.2
Add drives (diddevices) to the metaset
#metaset -s racset -a /dev/did/dsk/d4

#metaset -s racset -a /dev/did/dsk/d5
root@ora-rac-3 # cat /etc/lvm/md.tab | grep racset

racset/d40 -m racset/d41 racset/d42
racset/d41 1 1 /dev/did/dsk/d5s0
Page 18 of 90
Configure QFS software on all nodes.

Laying out partitions
• Edit /etc/opt/SUNWsamfs/mcf
# Equipment Eq Eq Family Device Additional
# Identifier Ord Type Set State Parameters
#----------- --- ---- ------ ------ ----------
#
sharedrac 20 ma sharedrac on shared
/dev/md/racset/dsk/d50 21 mm sharedrac on
/dev/md/racset/dsk/d40 22 mr sharedrac on
sharedfs 30 ma sharedfs on shared
/dev/md/racset/dsk/d70 31 mm sharedfs on
/dev/md/racset/dsk/d60 32 mr sharedfs on
• Edit /etc/opt/SUNWsamfs/hosts.sharedrac
ora-rac-3 clusternode1-priv 1 - server
ora-rac-4 clusternode2-priv 2 -
• Edit /etc/opt/SUNWsamfs/hosts.sharedfs
ora-rac-3 clusternode1-priv 1 -
ora-rac-4 clusternode2-priv 2 - server
• Edit /etc/opt/SUNWsamfs/samfs.cmd
notrace
fs = sharedrac
nosam
forcedirectio
sync_meta=1
mh_write
qwrite
nstreams=1024
stripe=1
rdlease=300
wrlease=300
aplease=300
fs = sharedfs
sync_meta=1
nstreams=1024
stripe=1
Make the new configuration on both nodes available.
Page 19 of 90
#samd config
Use sam-fsd command for validation
Create a QFS filesystem (only 1 node).

#sammkfs -S sharedrac
#sammkfs -S sharedfs
Create global mountpoint and vfstab entries on all nodes

mkdir -p /oradata
mkdir -p /oraarch
chown root:other /oradata /oraarch
vi /etc/vfstab
sharedrac - /oradata samfs - no shared
sharedfs - /oraarch samfs - no shared
#mount sharedrac
#mount sharedfs
Create cluster resourcegroup for qfs

Because of a timingproblem between SLVM/SC and qfs implement a hack to be shure that metaset is
available befor activating qfs. See bug id: 6222648
# cd /opt/SUNWsamfs/sc/bin
# vi scqfs_boot.hacked
#!/bin/ksh
my_host=`uname -n`
while true;
do
scstat -g | grep rac_svm | grep $my_host | read r rn h status statmsg
if [ $status = "Online" ]; then break;
else sleep 10; fi
done
exec /opt/SUNWsamfs/sc/bin/scqfs_init $*
# chmod 755 scqfs_boot.hacked
# mv scqfs_boot scqfs_boot_org
# ln -s scqfs_boot.hacked scqfs_boot
# scrgadm -a -t SUNW.qfs
Page 20 of 90
# scrgadm -a -g qfs_rac_rg -h ora-rac-3,ora-rac-4

# scrgadm -a -g qfs_rac_rg -t SUNW.qfs -j qfs_rac_rs \
-x QFSFileSystem=/oradata
# scswitch -Z -g qfs_rac_rg
# scrgadm -a -g qfs_arch_rg -h ora-rac-4,ora-rac-3
# scrgadm -a -g qfs_arch_rg -t SUNW.qfs -j qfs_arch_rs -x \
QFSFileSystem=/oraarch
# scswitch -Z -g qfs_arch_rg
If you are not able to register SUNW.qfs then create the link:
# cd /usr/cluster/lib/rgm/rtreg
# ln -s /opt/SUNWsamfs/sc/etc/SUNW.qfs
Set ownership and permissions:

chown oracle:dba /oradata /oraarch
chmod 755 /oradata /oraarch
Page 21 of 90
Last Systemanalyse
explorer
cd /opt/SUNWexplo/output/explorer*
/cdrom/eis-dvd/sun/tools/explominer/explominer
see the outputfile ExploMiner_EIS_Patchlist_nobody of the explominer and in-

stall all neccessary patches.
Page 22 of 90
Prepare Oracle Installation
Share staging area

All Oracle software is located on Host ora-poc2:/oracle/install. The directory is shared as /install/oracle
and can be mounted on every node if needed.
Share stage directory:
chown -R oracle:oinstall /oracle
share -rw /oracle/install
Mount staging area

In order to install oracle software, the staging area must be mounted on the instaation node of the clus-
ter.
mkdir /install
mount ora-poc2:/oracle/install /install
Perform cluster check

As oracle:
/install/10gR2/clusterware/Disk1/runcluvfy.sh stage -pre crsinst -n ora-rac-
3,ora-rac-4 -verbose > ~/cluvfy.log
Page 23 of 90
Install Clusterware 10.2.0.1
Run installer from ora-rac-3

Login as oracle:
DISPLAY=192.168.10.118:0.0
export DISPLAY
/install/10gR2/clusterware/Disk1/runInstaller
Page 24 of 90
Page 25 of 90
Page 26 of 90
Page 27 of 90
Page 28 of 90
Page 29 of 90
Page 30 of 90
Page 31 of 90
Page 32 of 90
Page 33 of 90
The given interface(s), "ce0" is not public. Public interfaces should be used
to configure virtual IPs.
Workaround: Run VIPCA manually
Running vipca
>> ora-rac-3 as root
/oracle/product/db/bin/vipca
Page 34 of 90
Page 35 of 90
Page 36 of 90
Page 37 of 90
Page 38 of 90
Page 39 of 90
Check Cluster Status

ORACLE_HOME=/oracle/product/crs
export ORACLE_HOME
PATH=$ORACLE_HOME/bin:$PATH
export PATH
$ crs_stat -t
Name Type Target State Host
-----------------------------------------------------------
ora....SM1.asm application ONLINE ONLINE ora-rac-3
ora....-1.lsnr application ONLINE ONLINE ora-rac-3
ora....c-1.gsd application OFFLINE OFFLINE
ora....c-1.ons application ONLINE ONLINE ora-rac-3
ora....c-1.vip application ONLINE ONLINE ora-rac-3
ora....SM2.asm application ONLINE ONLINE ora-rac-4
ora....-2.lsnr application ONLINE ONLINE ora-rac-4
ora....c-2.gsd application OFFLINE OFFLINE
ora....c-2.ons application ONLINE ONLINE ora-rac-4
ora....c-2.vip application ONLINE ONLINE ora-rac-4
Page 40 of 90
Install crsstat.sh
Because of the unreadable output of „crs_stat -t“ setup a wrapper for crs_stat.
mkdir /oracle/tools
vi /oracle/tools/crsstat.sh
#!/usr/bin/ksh
#
# Sample 10g CRS resource status query script
#
# Description:
# - Returns formatted version of crs_stat -t, in tabular
# format, with the complete rsc names and filtering keywords
# - The argument, $RSC_KEY, is optional and if passed to the script, will
# limit the output to HA resources whose names match $RSC_KEY.
# Requirements:
# - $CRS_HOME should be set in your environment
RSC_KEY=$1
QSTAT=-u
AWK=/usr/xpg4/bin/awk
# Table header:echo ""
$AWK \
'BEGIN {printf "%-45s %-10s %-18s\n", "HA Resource", "Target", "State";
printf "%-45s %-10s %-18s\n", "-----------", "------", "-----";}'
# Table body:
$CRS_HOME/bin/crs_stat $QSTAT | $AWK \
'BEGIN { FS="="; state = 0; }
$1~/NAME/ && $2~/'$RSC_KEY'/ {appname = $2; state=1};
state == 0 {next;}
$1~/TARGET/ && state == 1 {apptarget = $2; state=2;}
$1~/STATE/ && state == 2 {appstate = $2; state=3;}
state == 3 {printf "%-45s %-10s %-18s\n", appname, apptarget, appstate; state=0;}'
Output of crsstat.sh
HA Resource Target State

----------- ------ -----
ora.ora-rac-3.gsd ONLINE ONLINE on ora-rac-3
ora.ora-rac-3.ons ONLINE ONLINE on ora-rac-3
ora.ora-rac-3.vip ONLINE ONLINE on ora-rac-3
ora.ora-rac-4.gsd ONLINE ONLINE on ora-rac-4
ora.ora-rac-4.ons ONLINE ONLINE on ora-rac-4
ora.ora-rac-4.vip ONLINE ONLINE on ora-rac-4
Page 41 of 90
Install Database Home 10.2.0.1

/install/10gR2/database/Disk1/runInstaller
Page 42 of 90
Page 43 of 90
Page 44 of 90
Page 45 of 90
Page 46 of 90
Page 47 of 90
Page 48 of 90
Page 49 of 90
Page 50 of 90
Install Management Agent

/install/10gR2/gridcontrol/DVD/runInstaller
Page 51 of 90
Page 52 of 90
Page 53 of 90
Page 54 of 90
Page 55 of 90
Password: oracle1
Page 56 of 90
Page 57 of 90
Page 58 of 90
Page 59 of 90
Install Patchset 10.2.0.2 on CRS Home
Stop all oracle processes

As oracle:
srvctl stop nodeapps -n ora-rac-3
srvctl stop nodeapps -n ora-rac-4
As root on every node:
/etc/init.d/init.crs stop
If the processes won't stop, disable crs and reboot the host.
/etc/init.d/init.crs disable
init 6
Install binaries
Login as oracle on ora-rac-3
Set and export DISPLAY
Run Oracle Installer

/install/10gR2/patches/p4547817/Disk1/runInstaller
Page 60 of 90
Page 61 of 90
Page 62 of 90
Page 63 of 90
Page 64 of 90
Restart Custerware
As root (on every node)
/etc/init.d/init.crs start
If the cluster processes have been disabled before, run

/etc/init.d/init.crs enable
Page 65 of 90
Install Patchset 10.2.0.2 on DB Home
Install binaries
./runInstaller
Page 66 of 90
Page 67 of 90
Page 68 of 90
Page 69 of 90
Page 70 of 90
Install Interim Patch 5117016 on DB Home
Install with opatch

>> ora-rac-3 as user oracle
#cd /install/10gR2/5117016
#opatch apply -local
Page 71 of 90
>> ora-rac-4 as user oracle

#cd /install/10gR2/5117016
#opatch apply -local
Page 72 of 90
Change /etc/init.d/init.crsd:
>> ora-rac-3 and ora-rac-4 as user root
from: exec $ORA_CRS_HOME/bin/crsd $BOOTFLAG $*
to : exec newtask $ORA_CRS_HOME/bin/crsd $BOOTFLAG $*
Setup system parameters on both Nodes:

>> ora-rac-3 and ora-rac-4 as user root
#projadd -p 61002 -c "Oracle RAC" -U oracle \
-K 'project.max-sem-ids=(priv,1024,deny)' \
-K 'process.max-sem-nsems=(priv,2048,deny)' \
-K 'project.max-shm-ids=(priv,256,deny)' \
-K 'project.max-shm-memory=(priv,18446744073709551615,deny)' \
MAH
cat >> /etc/user_attr
oracle::::project=RAC
<CTRL>-d
Check all by reboot

>> ora-rac-3 as user root
#scshutdown -g0 -y

ok boot

ok boot
Page 73 of 90
Create Database ZDA
Run DBCA
>> ora-rac-3
/oracle/product/db/bin/dbca
Page 74 of 90
Page 75 of 90
Page 76 of 90
Password: oracle
Page 77 of 90
Cluster Optimiziation
Overview and best practices for setting up a "zero brownout" configuration to achieve maximum high
availability. More info in note 369101.1.
Note: Patch 5024639 is not yet available and skipped therefore.
Apply Brownout Patches

This patches are installed by using OPatch on DB and ASM Home:
5186679 (This became patch 4960705)
5186674 (This became patch 5016142)
5127434
5127472
5127482
5176453
Set the CSS misscount to match the service level

1. Shut down the clusterware stack on all nodes but one
2. On the remaining node, run as root:

$ORA_CRS_HOME/bin/crsctl set css misscount 5
$ORA_CRS_HOME/bin/crsctl set css disktimeout 200
3. Reboot the nodes for this to take affect.
4. Confirm the new values for the CSS parameters

$ORA_CRS_HOME/bin/crsctl get css misscount
$ORA_CRS_HOME/bin/crsctl get css disktimeout
Run CSS in Real Time

1. On each node as root use, crsctl set css priority 4
2. Restart each node in series, using crsctl stop crs followed by crsctl start crs
Set the check_interval for VIP addresses

>> ora-ra-3 as root
1. Generate the resource profile for the VIP resource
cd $ORA_CRS_HOME/crs/profile
crs_stat -p ora.ora-rac-3.vip > ora. ora-rac-3.vip.cap
crs_stat -p ora.ora-rac-4.vip > ora. ora-rac-4.vip.cap
(current value of check_interval = 60 (default))
Page 78 of 90
2. Edit ora.<hostname>.vip.cap and set the parameter CHECK_INTERVAL to 5

vi ora.ora-rac-3.vip.cap
vi ora.ora-rac-4.vip.cap
3. Update the resource profile as the root user:

crs_register -u ora.ora-rac-3.vip
crs_register -u ora.ora-rac-4.vip
Note: The check interval should be long enough to sustain the refresh time for the VIP.
Page 79 of 90
Optimize VIP Failover
Adjusting Listener Configuration

Make the local IP-address the first listening entry in the listener.ora.
More infos in note 3589757 and 339107.1
These bugs are based on oracle 10.1 but are not resolved by 10.2
>> All nodes
node ora-rac-3. Before:
LISTENER_ORA-RAC-3 =
(DESCRIPTION_LIST =
(DESCRIPTION =
(ADDRESS =
(PROTOCOL = TCP)
(HOST = ora-rac-3-vip)
(PORT = 1521)
(IP = FIRST))
(ADDRESS =
(PROTOCOL = TCP)
(HOST = 192.168.10.215)
(PORT = 1521)
(IP = FIRST))
)
)
[...]
After:
(DESCRIPTION_LIST =
(DESCRIPTION =
(ADDRESS =
(PROTOCOL = TCP)
(HOST = 192.168.10.215)
(PORT = 1521)
(IP = FIRST))
(ADDRESS =
(PROTOCOL = TCP)
(PORT = 1521)
(IP = FIRST))
)
)
[...]
Page 80 of 90
node ora-rac-4. Before:

(DESCRIPTION_LIST =
(DESCRIPTION =
(ADDRESS =
(PROTOCOL = TCP)
(PORT = 1521)
(IP = FIRST))
(ADDRESS =
(PROTOCOL = TCP)
(HOST = 192.168.10.217)
(PORT = 1521)
(IP = FIRST))
)
)
[...]
After:
(DESCRIPTION_LIST =
(DESCRIPTION =
(ADDRESS =
(PROTOCOL = TCP)
(HOST = 192.168.10.217)
(PORT = 1521)
(IP = FIRST))
(ADDRESS =
(PROTOCOL = TCP)
(PORT = 1521)
(IP = FIRST))
)
)
[...]
Page 81 of 90
Addendum
Metalink Note 284752.1
Page 82 of 90
Page 83 of 90
Page 84 of 90
Page 85 of 90
Page 86 of 90
Page 87 of 90
Page 88 of 90
Page 89 of 90
Page 90 of 90

Installation RAC10GR2 QFS

Caricato da

Informazioni sul documento

Titolo originale

Copyright

Formati disponibili

Condividi questo documento

Condividi o incorpora il documento

Opzioni di condivisione

Hai trovato utile questo documento?

Questo contenuto è inappropriato?

Copyright:

Formati disponibili

Installation RAC10GR2 QFS

Caricato da

Copyright:

Formati disponibili

Document: Installation_RAC10GR2_QFS.

Author: Michael Ahrens

IPMP Group IPMP Group

Oracle RAC Cluster Oracle RAC Cluster

change the home of root

Setup EIS environment

Install actual Solaris 10 patches

Analyse basic systemstatus

see the outputfile ExploMiner_EIS_Patchlist_nobody of the explominer and in-

Additionally for Solaris10 configure /etc/inet/ipnodes

To enable MpxIO on Solaris 10 start

stmsboot -e before SunCluster will be installed !!

Notice: For use of VLANs for the cluster transport:

#ndd -set /dev/ip ip_strict_dst_multihoming 0

Use the Java ES installer for package installation.

If you not install package SUNWescom disable the scsymon service:

# svcadm -v disable /system/cluster/scsymon-srv

the following core patch is required for SC3.1U4:

• 120500-04 Sun Cluster 3.1 Core Patch for Solaris 10

If the defaultrouter is NOT 100% available please read infodocs

• In the /etc/default/mpathd file, the value of

Common actions for filesystems and boot devices:

NOTE: To find out the did device use scdidadm -l command!

Modify NVRAM parameter:

Suggested naming convention:

Create Users and Groups

Check SSH daemon

Setup SSH for oracle

Steps to be done host on ora-rac-3:

Steps to be done host on ora-rac-4:

Steps to be done host on ora-rac-3:

Steps to be done on ora-rac-3 and ora-rac-4:

Setup userenv for oracle

Install the RAC/OPS Service from the CD

Select package(s) you wish to process (or 'all' to process

# Cluster.number.nodes should be obtained dynamically

Set kernel parameters

Create cluster resourcegroup for rac

Increase svm_step4_timeout if using SVM and RAC is managed by the RGM.

Switch the Resource Group online.

Create multi-owner disk set

#metaset -s <setname> -M -a -h <nodelist>

#metaset -s racset -a /dev/did/dsk/d4

root@ora-rac-3 # cat /etc/lvm/md.tab | grep racset

Configure QFS software on all nodes.

Make the new configuration on both nodes available.

Create a QFS filesystem (only 1 node).

Create global mountpoint and vfstab entries on all nodes

Create cluster resourcegroup for qfs

# scrgadm -a -g qfs_rac_rg -h ora-rac-3,ora-rac-4

Set ownership and permissions:

see the outputfile ExploMiner_EIS_Patchlist_nobody of the explominer and in-

Prepare Oracle Installation

Share staging area

Mount staging area

Perform cluster check

Install Clusterware 10.2.0.1

Run installer from ora-rac-3

Workaround: Run VIPCA manually

Check Cluster Status

HA Resource Target State

Install Database Home 10.2.0.1

Run installer from ora-rac-3