Sr. No.

Description

User

Check all the Cluster Services daemon

oracle

Check configuration of Voting Disk and Oracle Cluster

Registry

root

Take backup of OCR and voting disks (no need to take

backup of voting disk manually with dd command as oracle
automatically takes every 4 hours)

root

4
5

Run dbupgdiag.sql and save the output for reference

Shutdown the database

Shutdown the cluster

oracle
oracle
root
oracle

Take backup of Grid and Oracle home

root

8
9
10
11
12
13
14

Take backup of oraInventory

Start crs
Start cluster
Check all cluster resources , it should show DB as up and
running
Configure SSH passwordless authenticaltion between each
RAC node
Unzip 11.2.0.4 patchset - 3rd zip file is for grid.
Create 4 empty folders i.e. grid_2 and db_2 on both the
nodes, give permission and change ownership

15

Make sure we have sufficient space available in /tmp

location

16

Verify cluster components

17

Remove timeouts configured for Putty sessions

18

Cluster freeze and verification steps

oracle
root
root
oracle
oracle
oracle
root
oracle
oracle /
root
oracle

19

Unset Oracle environment variables

oracle

20

Collect orachk output to check the system readiness and

save it for reference

oracle

PRE-CHECKS
Command
ps -ef|grep d.bin
ps -ef|grep d.bin| wc -l
cd /opt/app/grid/oracle/product/11.2.0/grid_1/bin
./ocrcheck
./crsctl query css votedisk
./ocrconfig -showbackup
./ocrconfig -backuploc /opt/app/oracle
./ocrconfig -manualbackup
SQL> @?/rdbms/admin/dbupgdiag.sql
./srvctl stop database -d uc1asvt
./crsctl stop cluster -all
./crsctl stop crs (on both the nodes)
cd /opt/app/oracle/product/11.2.0
cp -r db_1 db_11.2.0.3
cd /opt/app/grid/oracle/product/11.2.0
cp -r grid_1 grid_11.2.0.3
cat /etc/oraInst.loc
cp -r /opt/app/oracle/product/oraInventory
/opt/app/oracle/product/oraInventory_bkp
./crsctl start crs
./crsctl start cluster -all
./crsctl stat res -t

unzip p13390677_112040_Linux-x86-64_3of7.zip
mkdir grid_2
chmod 775 grid_2
chown root:dba grid_2
mkdir db_2

./cluvfy comp software -n all

./cluvfy comp software -n all -verbose
./cluvfy stage -pre crsinst -n [nodelist] -verbose
With root access, we set the below options in the global SSHd
configuration file (sshd_config) and restarted the SSHd service:
ClientAliveInterval 30
TCPKeepAlive yes
ClientAliveCountMax 99999
/etc/init.d/sshd restart
#
#
#
#
#

haconf -makerw
hagrp -freeze cvm -persistent
hastatus -sum
haconf -dump -makero
haclus -display |grep -i Read

unset
unset
unset
unset
unset
unset

ORACLE_BASE
ORACLE_HOME
ORACLE_SID
ORA_CRS_HOME
ORA_NLS10
TNS_ADMIN

./orachk -u -o pre

Expected Output / Key Notes

It would list all the daemons - main 4 are ohasd,crsd,evmd and cssd
10
All files currently storing OCR
All files currently storing Voting Disks
It will show all auto backups
Take a manual backup
It will show the recently taken manual backup of OCR and Vote disk
Refer Metalink Doc ID 556610.1 for more details
It will do a graceful shutdown of database.
It will shutdown the crs daemon and all cluster services across nodes
Take backup of Oracle Home
Take backup of Grid Home
inst_group=dba
inventory_loc=/opt/app/oracle/product/oraInventory
Take backup of Inventory
It will start ohasd and other daemons
It will start remaining daemons in both the nodes
Database should be open and stable
Refer Metalink Doc ID 300548.1 for steps to configure SSH
On Node1 only.
On node 1 and node 2 both
On node 1 and node 2 both
Involve Infra team if required
Run these commands from Grid Home. Make sure all the failures are resolved. Warnings
can be ignored selectively.

Test this throughly to avoid any issues with ssh client terminating in the middle of the
upgrade.

The steps mentioned are applicable for Veritas cluster. These are mandatory to avoid
known issues while performing upgrade. Refer Metalink Doc ID 1360034.1 for more
details.

Make sure all the failures are resolved. Warnings can be ignored selectively. Refer
Metalink Doc ID 1457357.1 for details.

IM
Sr. No.

Description

User

Invoke ./runInstaller from "grid" folder (created in the

location where 11.2.0.4 binaries were unzipped) to install
and upgrade current Grid Home on 11.2.0.3 version

oracle

Apply Grid PSU locally on both nodes

oracle

Run rootupgrade.sh on both the nodes after Grid PSU is

applied on all the nodes successfully

Invoke ./runInstaller from "database" folder (created in the

location where 11.2.0.4 binaries were unzipped) to install
and upgrade current Database Home on 11.2.0.3 version

Run root.sh on both the nodes

Apply Database PSU locally on both nodes

Run root.sh on both the nodes after Databse PSU is applied

on all the nodes successfully

Perform Database upgrade Pre-Checks

oracle

Unset & set variables as mentioned

oracle

10

root

oracle

root
oracle

root

Invoke dbua to upgrade the database from old Oracle Home

oracle
to the new one installed

IMPLEMENTATION
Command

Sample command:
/opt/app/grid/oracle/product/11.2.0/grid_2/OPatch/opatch apply
/opt/app/oracle/product/11.2.0.4_Patches/19955028/19769489 -oh
/opt/app/grid/oracle/product/11.2.0/grid_2 -local
sh /opt/app/grid/product/11.2.0/grid_2/rootupgrade.sh

sh /opt/app/oracle/product/11.2.0/db_2/root.sh
Sample command:
/opt/app/oracle/product/11.2.0/db_2/OPatch/opatch apply
/opt/app/oracle/product/11.2.0.4_Patches/19955028/19769489 -oh
/opt/app/oracle/product/11.2.0/db_2 -local
sh /opt/app/grid/product/11.2.0/grid_2/rootupgrade.sh

Unset below environment variables:

unset
unset
unset
unset
unset
unset

ORACLE_BASE
ORACLE_HOME
ORACLE_SID
ORA_CRS_HOME
ORA_NLS10
TNS_ADMIN

Set environment variables as below:

export
export
export
export

ORACLE_SID=INSTANCE_NAME
ORACLE_HOME=/opt/app/oracle/product/11.2.0/db_2
PATH=$ORACLE_HOME/bin:$PATH
LD_LIBRARY_PATH=$ORACLE_HOME/lib

Expected Output / Key Notes

Do NOT run rootupgrade.sh script when you get the pop up to do so.

Refer Metalink Doc ID 1410202.1 for reference

Execute rootupgrade.sh as real ROOT user and not using "sudo su - root"

On node 1 only

Execute root.sh as real ROOT user and not using "sudo su - root"
Refer Metalink Doc ID 1410202.1 for reference

Do NOT run root.sh script when you get the pop up to do so.

Disable FRA before upgrading the database. You can enable it back post upgrade is
completed successfully. We did this as we were facing some errors during upgrade.

Sr. No.

Description
Run the orachk and dbupgdiag.sql again and save the
output
Take backup of OCR and voting disks (no need to take
backup of voting disk manually with dd command as oracle
automatically take every 4 hours)
Shutdown the database

Shutdown the cluster

Take backup of new Grid and Oracle Home

1
2

User
oracle
root
oracle
root
oracle
root

Take backup of oraInventory

7
8

Start crs
Start cluster
Check all cluster resources , it should show DB as up and
running
Run catbundle.sql psu apply
Check opatch lsinventory on both the servers for all 4
homes

9
10
11

oracle
root
root
oracle
sys
oracle

12

Verify the applied patch from dba_registry_history

sys

13

Registry components validity check

sys

14

Gather dictionary stats, using any one of the command

sys

15

Gather the fixed objects stats

sys

POST-CHECKS
Command
./orachk -u -o post
./ocrconfig -showbackup
./ocrconfig -backuploc /opt/app/oracle
./ocrconfig -manualbackup
./srvctl stop database -d uc1asvt
./crsctl stop cluster -all
./crsctl stop crs (on both the nodes)
cd /opt/app/oracle/product/11.2.0
cp -r db_2 db_11.2.0.4
cd /opt/app/grid/oracle/product/11.2.0
cp -r grid_2 grid_11.2.0.4
cat /etc/oraInst.loc
cp -r /opt/app/oracle/product/oraInventory
/opt/app/oracle/product/oraInventory_bkp
./crsctl start crs
./crsctl start cluster -all
./crsctl stat res -t
SQL> @?/rdbms/admin/catbundle.sql psu apply
$ORACLE_HOME/Opatch/opatch lsinventory
select action_time,action,version,comments from dba_registry_history;
col COMP_NAME format a50
set lines 500
select COMP_ID, COMP_NAME, VERSION, STATUS from dba_registry;
SQL> EXEC DBMS_STATS.GATHER_SCHEMA_STATS ('SYS');
SQL> exec DBMS_STATS.GATHER_DATABASE_STATS
(gather_sys=>TRUE);
SQL> EXEC DBMS_STATS.GATHER_DICTIONARY_STATS;
EXEC DBMS_STATS.GATHER_FIXED_OBJECTS_STATS;

Expected Output / Key Notes

Check the score and see if it can be make better by using the recommendations. Verify
all the db components are valid.
It will show all auto backups
Take a manual backup
It will show the recently taken manual backup of OCR and Vote disk
It will do a graceful shutdown of database.
It will shutdown the crs daemon and all cluster services across nodes
Take backup of oracle home
Take backup of grid home
inst_group=dba
inventory_loc=/opt/app/oracle/product/oraInventory

It will start ohasd and other daemons

It will start remaining daemons in both the nodes
Database should be open and stable
On any one node from each environment.
On all the nodes (both Grid and Oracle)
It would show the time when patch was applied with the comments

All the components should be valid. Any invalid components should be fixed.

Refer Metalink Doc ID 457926.1 for details.

Gather fixed objects stats if the load is heavy and if the system is busy. For more details
when to gather fixed object stats, read following note:
Document:798257.1GATHER_FIXED_OBJECTS_STATS Considerations