New features in Oracle 9i/10g/11g RAC

Oracle Real Application Clusters New features

Oracle 9i RAC
OPS (Oracle Parallel Server) was renamed as RAC
CFS (Cluster File System) was supported
OCFS (Oracle Cluster File System) for Linux and Windows
watchdog timer replaced by hangcheck timer

Oracle 10g R1 RAC

Cluster Manager replaced by CRS
ASM introduced
Concept of Services expanded
ocrcheck introduced
ocrdump introduced
AWR was instance specific

Oracle 10g R2 RAC

CRS was renamed as Clusterware
asmcmd introduced
CLUVFY introduced
OCR and Voting disks can be mirrored
Can use FAN/FCF with TAF for OCI and ODP.NET

Oracle 11g R1 RAC

Oracle 11g RAC parallel upgrades - Oracle 11g have rolling upgrade features whereby
RAC database can be upgraded without any downtime.
Hot patching - Zero downtime patch application.
Oracle RAC load balancing advisor - Starting from 10g R2 we have RAC load balancing
advisor utility. 11g RAC load balancing advisor is only available with clients who
use .NET, ODBC, or the Oracle Call Interface (OCI).
ADDM for RAC - Oracle has incorporated RAC into the automatic database diagnostic
monitor, for cross-node advisories. The script addmrpt.sql run give report for
single instance, will not report all instances in RAC, this is known as instance
ADDM. But using the new package DBMS_ADDM, we can generate report for all instances
of RAC, this known as database ADDM.
Optimized RAC cache fusion protocols - moves on from the general cache fusion
protocols in 10g to deal with specific scenarios where the protocols could be
further optimized.
Oracle 11g RAC Grid provisioning - The Oracle grid control provisioning pack allows
us to "blow-out" a RAC node without the time-consuming install, using a pre-
installed "footprint".

Oracle 11g R2 RAC

We can store everything on the ASM. We can store OCR & voting files also on the
ASM.
ASMCA
Single Client Access Name (SCAN) - eliminates the need to change tns entry when
nodes are added to or removed from the Cluster. RAC instances register to SCAN
listeners as remote listeners. SCAN is fully qualified name. Oracle recommends
assigning 3 addresses to SCAN, which create three SCAN listeners.
Clusterware components: crfmond, crflogd, GIPCD.
AWR is consolidated for the database.
11g Release 2 Real Application Cluster (RAC) has server pooling technologies so
it�s easier to provision and manage database grids. This update is geared toward
dynamically adjusting servers as corporations manage the ebb and flow between data
requirements for datawarehousing and applications.
By default, LOAD_BALANCE is ON.
GSD (Global Service Deamon), gsdctl introduced.
GPnP profile.
Cluster information in an XML profile.
Oracle RAC OneNode is a new option that makes it easier to consolidate databases
that aren�t mission critical, but need redundancy.
raconeinit - to convert database to RacOneNode.
raconefix - to fix RacOneNode database in case of failure.
racone2rac - to convert RacOneNode back to RAC.
Oracle Restart - the feature of Oracle Grid Infrastructure's High Availability
Services (HAS) to manage associated listeners, ASM instances and Oracle instances.
Oracle Omotion - Oracle 11g release2 RAC introduces new feature called Oracle
Omotion, an online migration utility. This Omotion utility will relocate the
instance from one node to another, whenever instance failure happens.
Omotion utility uses Database Area Network (DAN) to move Oracle instances. Database
Area Network (DAN) technology helps seamless database relocation without losing
transactions.
Cluster Time Synchronization Service (CTSS) is a new feature in Oracle 11g R2 RAC,
which is used to synchronize time across the nodes of the cluster. CTSS will be
replacement of NTP protocol.
Grid Naming Service (GNS) is a new service introduced in Oracle RAC 11g R2. With
GNS, Oracle Clusterware (CRS) can manage Dynamic Host Configuration Protocol (DHCP)
and DNS services for the dynamic node registration and configuration.
Cluster interconnect: Used for data blocks, locks, messages, and SCN numbers.
Oracle Local Registry (OLR) - From Oracle 11gR2 "Oracle Local Registry (OLR)"
something new as part of Oracle Clusterware. OLR is node�s local repository,
similar to OCR (but local) and is managed by OHASD. It pertains data of local node
only and is not shared among other nodes.
Multicasting is introduced in 11gR2 for private interconnect traffic.
I/O fencing prevents updates by failed instances, and detecting failure and
preventing split brain in cluster. When a cluster node fails, the failed node needs
to be fenced off from all the shared disk devices or diskgroups. This methodology
is called I/O Fencing, sometimes called Disk Fencing or failure fencing.
Re-bootless node fencing (restart)? - instead of fast re-booting the node, a
graceful shutdown of the stack is attempted.
Clusterware log directories: acfs*
HAIP (IC VIP).
Redundant interconnects: NIC bonding, HAIP.
RAC background processes: DBRM � Database Resource Manager, PING � Response time
agent.
Virtual Oracle 11g RAC cluster - Oracle 11g RAC supports virtualization.

Virtual IP (VIP) in RAC

What is (use of) Virtual IP (VIP) in Oracle Real Application Clusters (RAC)?

When installing Oracle 10g/11g R1 RAC, three network interfaces(IPs) are required
for each node in the RAC cluster, they are:

Public Interface: Used for normal network communications to the node

Private Interface: Used as the cluster interconnect
Virtual (Public) Interface: Used for failover and RAC management

When installing Oracle 11g R2 RAC, we need one more network interface(IP) is
required for each node in the RAC cluster.

SCAN Interface (IP): Single Client Access Name (SCAN) is a new Oracle Real
Application Clusters (RAC) 11g Release 2 feature, that provides a single name for
clients to access an Oracle Database running in a cluster. The benefit is clients
using SCAN do not need to change if you add or remove nodes in the cluster.

When a client connects to a tns-alias, it uses a TCP connection to an IP address,

defined in the tnsnames.ora file. When using RAC, we define multiple addresses in
our tns-alias, to be able to failover when an IP address, listener or instance is
unavailable. TCP timeouts can differ from platform to platform or implementation to
implementation. This makes it difficult to predict the failover time.

Oracle 10g Cluster Ready Services enables databases to use a Virtual IP address to
configure the listener ON. This feature is to assure that oracle clients quickly
failover when a node fails. In Oracle Database 10g RAC, the use of a virtual IP
address to mask the individual IPO addresses of the clustered nodes is required.
The virtual IP addresses are used to simplify failover and are automatically
managed by CRS.

To create a Virtual IP (VIP) address, the Virtual IP Configuration Assistant

(VIPCA) is called from the root.sh script of a RAC install, which then configures
the virtual IP addresses for each node specified during the installation process.
In order to be able to run VIPCA, there must be unused public IP addresses
available for each node that has been configured in the /etc/hosts file.

One public IP address for each node to use for its Virtual IP address for client
connections and for connection failover. This IP address is in addition to the
operating system managed public host IP address that is already assigned to the
node by the operating system. This public Virtual IP must be associated with the
same interface name on every node that is a part of the cluster. The IP addresses
that are used for all of the nodes that are part of a cluster must be from the same
subnet. The host names for the VIP addresses must be registered with the domain
name server (DNS). The Virtual IP address should not be in use at the time of the
installation because this is a Virtual IP address that Oracle manages internally to
the RAC processes. This virtual IP address does not require a separate NIC. The
VIPs should be registered in the DNS. The VIP addresses must be on the same subnet
as the public host network addresses. Each Virtual IP (VIP) configured requires an
unused and resolvable IP address.

Using virtual IP we can save our TCP/IP timeout problem because Oracle notification
service (ONS) maintains communication between each nodes and listeners. Once ONS
found any listener down or node down, it will notify another nodes and listeners.
While new connection is trying to establish connection to failure node or listener,
virtual IP of failure node automatically divert to surviving node and session will
be establishing in another surviving node. This process doesn't wait for TCP/IP
timeout event. Due to this new connection gets faster session establishment to
another surviving nodes/listener.
Virtual IP (VIP) is for fast connection establishment in failover dictation. Still
we can use physical IP address in Oracle 10g in listener if we have no worry for
failover timing. We can change default TCP/IP timeout using operating system
utilities/commands and kept smaller. But taking advantage of VIP (Virtual IP
address) in Oracle 10g RAC database is advisable.

Commands related to VIP

#srvctl start nodeapps -n node_name -- Starts GSD, VIP, listener and ONS
#srvctl start nodeapps -n rac99

#srvctl stop nodeapps -n node_name [-r] -- Stops GSD, VIP, listener and ONS
#srvctl stop nodeapps -n lnx03
#srvctl config nodeapps -n node_name [-a] [-g] [-o] [-s] [-l]
-a Display VIP configuration
#srvctl config nodeapps -n node33 -a

#srvctl modify nodeapps -n node_name [-A new_vip_address]

#srvctl modify nodeapps -n lnx06 -A 10.50.99.43/255.255.252.0/eth0

srvctl modify nodeapps [-n node_name -A new_vip_address] [-S subnet/netmask[/if1[|

if2|...]] [-m multicast_ip_address] [-p multicast_port_number] [-e
eons_listen_port] [-l ons_local_port] [-r ons_remote_port] [-t host[:port]
[,host:port,...]] [-v]
srvctl modify nodeapps -n node1 -A 100.200.300.40/255.255.255.0/eth0

srvctl add vip -n node_name -A {name|ip}/netmask[/if1[if2|...]] [-k network_number]

[-v]
#srvctl add vip -n node96 -A 192.124.16.96/255.255.255.0 -k 2

srvctl remove vip -i "vip_name_list" [-f] [-y] [-v]

srvctl remove vip -i "vip1,vip2,vip3" -f -y -v

srvctl start vip {-n node_name|-i vip_name} [-v]

srvctl start vip -i dev1-vip -v

srvctl stop vip {-n node_name|-i vip_name} [-r] [-v]

srvctl stop vip -n node1 -v

srvctl status vip {-n node_name|-i vip_name}

srvctl status vip -i node1-vip

srvctl enable vip -i vip_name [-v]

srvctl enable vip -i prod_vip -v

srvctl disable vip -i vip_name [-v]

srvctl disable vip -i vip3 -v

srvctl config vip {-n node_name|-i vip_name}

srvctl config vip -n devnode2

srvctl getenv vip -i vip_name [-t "name_list"] [-v]

srvctl getenv vip -i node1-vip

srvctl setenv vip -i vip_name {-t "name=val[,name=val,...]" | -T "name=val"}

srvctl setenv vip -i dev1-vip -t LANG=en

srvctl unsetenv vip -i vip_name -t "name_list" [-v]

srvctl unsetenv vip -i myvip -t CLASSPATH

#olsnodes -i ==> will show VIPs of the nodes.

node1 168.192.1.1
node2 168.192.2.1
node3 168.192.3.1
node4 168.192.4.1

$vipca - VIP Configuration Assistant

$vipca -nodelist node1,node3

Background processes in RAC

Oracle Real Application Clusters (RAC) Specific Background Processes
Oracle RAC is composed of two or more database instances. They are composed of
memory structures and background processes same as the single instance database.

Oracle RAC instances are composed of following background processes:

ACMS � Atomic Control file to Memory Service
GTX0-j � Global Transaction Process
LMON � Global Enqueue Service Monitor
LMD � Global Enqueue Service Daemon
LMS � Global Cache Service Process
LCK0 � Instance Enqueue Process
DIAG � Diagnosability Daemon
RMSn � RAC Management Processes
RSMN � Remote Slave Monitor
DBRM � Database Resource Manager (from 11g R2)
PING � Response Time Agent (from 11g R2)
LMHB � Global Cache/Enqueue Service Heartbeat Monitor
RCBG � Result Cache BackGround Process

These processes spawned for supporting the multi-instance coordination.

ACMS (from Oracle 11g) Atomic Control file Memory Service

In an Oracle RAC environment ACMS background process is an agent that ensures a
distributed SGA memory update(ie) SGA updates are globally committed on success or
globally aborted in event of a failure.

GTX0-j (from Oracle 11g) Global Transaction Process

The process provides transparent support for XA global transactions in a RAC
environment. The database auto tunes the number of these processes based on the
workload of XA global transactions.

LMON Global Enqueue Service Monitor (Lock monitor)

The LMON monitors the entire cluster to manage the global enqueues and the
resources and performs global enqueue recovery operations. LMON manages instance
and process failures and the associated recovery for the Global Cache Service (GCS)
and Global Enqueue Service (GES). In particular, LMON handles the part of recovery
associated with global resources. LMON provided services are also known as cluster
group services (CGS). Lock monitor manages global locks and resources. It handles
the redistribution of instance locks whenever instances are started or shutdown.
Lock monitor also recovers instance lock information prior to the instance recovery
process. Lock monitor co-ordinates with the Process Monitor (PMON) to recover dead
processes that hold instance locks.

LMDx Global Enqueue Service Daemon

The LMD is the lock agent process that manages enqueue manager service requests for
Global Cache Service enqueues to control access to global enqueues and resources.
This process manages incoming remote resource requests within each instance. The
LMD process also handles deadlock detection and remote enqueue requests. Remote
resource requests are the requests originating from another instance. LMDn
processes manage instance locks that are used to share resources between instances.
LMDn processes also handle deadlock detection and remote lock requests.

LMSx Global Cache Service Processes

The LMSx are the processes that handle remote Global Cache Service (GCS) messages.
Real Application Clusters software provides for up to 10 Global Cache Service
Processes. The number of LMSx varies depending on the amount of messaging traffic
among nodes in the cluster.

This process maintains statuses of datafiles and each cached block by recording
information in a Global Resource Directory(GRD). This process also controls the
flow of messages to remote instances and manages global data block access and
transmits block images between the buffer caches of different instances. This
processing is a part of cache fusion feature.

The LMSx handles the acquisition interrupt and blocking interrupt requests from the
remote instances for Global Cache Service resources. For cross-instance consistent
read requests, the LMSx will create a consistent read version of the block and send
it to the requesting instance. The LMSx also controls the flow of messages to
remote instances.

The LMSn processes handle the blocking interrupts from the remote instance for the
Global Cache Service resources by:
Managing the resource requests and cross-instance call operations for the shared
resources.
Building a list of invalid lock elements and validating the lock elements during
recovery.
Handling the global lock deadlock detection and Monitoring for the lock conversion
timeouts.

LCKx Instance Enqueue process

This process manages the global enqueue requests and the cross-instance broadcast.
Workload is automatically shared and balanced when there are multiple Global Cache
Service Processes (LMSx). This process is called as instance enqueue process. This
process manages non-cache fusion resource requests such as library and row cache
requests. The instance locks that are used to share resources between instances are
held by the lock processes.

DIAG Diagnosability Daemon

Monitors the health of the instance and captures the data for instance process
failures.

RMSn RAC Management Service

This process is called as Oracle RAC Management Service/Process. These processes
perform manageability tasks for Oracle RAC. Tasks include creation of resources
related Oracle RAC when new instances are added to the cluster.

RSMN Remote Slave Monitor

This process is called as Remote Slave Monitor. This process manages background
slave process creation and communication on remote instances. This is a background
slave process. This process performs tasks on behalf of a coordinating process
running in another instance.

LMHB Global Cache/Enqueue Service Heartbeat Monitor

LMHB monitors the heartbeat of LMON, LMD, and LMSn processes to ensure they are
running normally without blocking or spinning. LMBH trace reports low memory, swap
space problem and system average load.

Oracle RAC instances use two processes GES(Global Enqueue Service), GCS(Global
Cache Service) that enable cache fusion. The GES and GCS maintain records of the
statuses of each datafile and each cached block using global resource directory
(GRD). This process is referred to as cache fusion and helps in data integrity.

Oracle RAC is composed of two or more instances. When a block of data is read from
datafile by an instance within the cluster and another instance is in need of the
same block, it is easy to get the block image from the instance which has the block
in its SGA rather than reading from the disk. To enable inter instance
communication Oracle RAC makes use of interconnects. The Global Enqueue
Service(GES) monitors and Instance enqueue process manages the cache fusion.
clscfg commands in Oracle RAC
CLSCFG: -- Oracle cluster configuration tool, in Oracle Real Application
Cluster(RAC)

clscfg -help or clscfg -h

clscfg -install -- creates a new configuration
clscfg -add -- adds a node to the configuration
clscfg -delete -- deletes a node from the configuration
clscfg -upgrade -- upgrades an existing configuration
clscfg -downgrade -- downgrades an existing configuration
clscfg -local -- creates a special single-node configuration for ASM
clscfg -concepts -- brief listing of terminology used in the other modes
clscfg -trace -- may be used in conjunction with any mode above for tracing

cluster name of Oracle RAC cluster

How to find out cluster name of Oracle RAC cluster ?

The cluster name is case-insensitive, must be unique across environment, must be no

more than 15 characters in length, must be alphanumeric and may contain hyphens
(-). Underscore characters (_) are not allowed. But installation script/OUI will
not fail, even if you provide more than 15 characters in cluster name.

1) cemutlo utility can be used to find out Oracle RAC Cluster name.

$GRID_HOME/bin/cemutlo -n

e.g.:
/u01/app/11.2/grid/bin/cemutlo -n

Below command will give number of characters in Oracle RAC Cluster name.
/u01/app/11.2/grid/bin/cemutlo -n|wc -c

2) olsnodes utility can be used to find out Oracle RAC Cluster name.
olsnodes -c

crs_stat commands in Oracle RAC

CRS_STAT: -- deprecated in Oracle Clusterware 11g release 2

crs_stat [resource_name [...]] [-v] [-l] [-q] [-c cluster_node]

crs_stat [resource_name [...]] -t [-v] [-q] [-c cluster_node]
crs_stat -p [resource_name [...]] [-q]
crs_stat [-a] resource_name -g
crs_stat [-a] resource_name -r [-c cluster_node]
crs_stat -f [resource_name [...]] [-q] [-c cluster_node]
crs_stat -ls resource_name

crs_stat or crs_stat -u or crs_stat -l

crs_stat -t -- tabular format
crs_stat -v -- verbose
crs_stat -p -- more details
AUTO_START = 0 restore
= 1 always restart
= 2 never restart
RESTART_ATTEMPTS = n -- How many times Clusterware should attempt to start the
resource?

crs_stat -ls
crs_stat -t/v/p resource_name or crs_stat -t/v/p|grep keyword -- will show the
entries with that keyword
crs_stat -p lnx02
crs_stat -v | grep eg6245

crs_stat -p resource_name > filename.cap -- generate resource profile file

This crs_stat command was replaced with crsctl command (options).

crsctl commands in Oracle RAC

crsctl - Cluster Ready Service Control

$crsctl -- to get help

$crsctl query crs activeversion
$crsctl query crs softwareversion [node_name]

#crsctl start crs

#crsctl stop crs
(or)
#/etc/init.d/init.crs start
#/etc/init.d/init.crs stop

#crsctl enable crs

#crsctl disable crs
(or)
#/etc/init.d/init.crs enable
#/etc/init.d/init.crs disable

$crsctl check crs

$crsctl check cluster [-node node_name] -- Oracle RAC 11g command, checks the
viability of CSS across nodes
#crsctl start cluster -n HostName -- 11g R2
#crsctl stop cluster -n HostName -- 11g R2
#crsctl stop cluster -all -- 11g R2

$crsctl check cssd

$crsctl check crsd
$crsctl check evmd
$crsctl check oprocd
$crsctl check ctss

#/etc/init.d/init.cssd stop
#/etc/init.d/init.cssd start

#/etc/rc.d/init.d/init.evmd
#/etc/rc.d/init.d/init.cssd
#/etc/rc.d/init.d/init.crsd

#mv /etc/rc3.d/S96init.cssd /etc/rc3.d/_S96init.cssd -- to stop cssd from

autostarting after reboot

#crsctl check css votedisk

#crsctl query css votedisk -- lists the voting disks used by CSS

#crsctl add css votedisk PATH

#crsctl add css votedisk PATH -force -- if Clusterware is not running
#crsctl delete css votedisk PATH
#crsctl delete css votedisk PATH -force -- if Clusterware is not running
#crsctl set css parameter_name value -- set parameters on OCR
#crsctl set css misscount 100
#crsctl unset css parameter_name -- sets CSS parameter to its default
#crsctl unset css misscount
#crsctl get css parameter_name -- gets the value of a CSS parameter
#crsctl get css disktimeout
#crsctl get css misscount
#crsctl get css reboottime

#crsctl start resources -- starts Clusterware resources

./crsctl start resource ora.DATA.dg
#crsctl stop resources -- stops Clusterware resources

$crsctl status resource

$crsctl status resource -t
$crsctl stat resource -t

$crsctl lsmodules crs -- lists CRS modules that can be used for debugging
CRSUI
CRSCOMM
CRSRTI
CRSMAIN
CRSPLACE
CRSAPP
CRSRES
CRSCOMM
CRSOCR
CRSTIMER
CRSEVT
CRSD
CLUCLS
CSSCLNT
COMMCRS
COMMNS

$crsctl lsmodules css -- lists CSS modules that can be used for debugging
CSSD
COMMCRS
COMMNS

$crsctl lsmodules evm -- lists EVM modules that can be used for debugging
EVMD
EVMDMAIN
EVMCOMM
EVMEVT
EVMAPP
EVMAGENT
CRSOCR
CLUCLS
CSSCLNT
COMMCRS
COMMNS

$crsctl start has (HAS - High Availability Services)

$crsctl stop has
$crsctl check has

OCR Modules -- cannot be listed with crsctl lsmodules command

OCRAPI
OCRCLI
OCRSRV
OCRMAS
OCRMSG
OCRCAC
OCRRAW
OCRUTL
OCROSD

#crsctl debug statedump crs -- dumps state info for crs objects
#crsctl debug statedump css -- dumps state info for css objects
#crsctl debug statedump evm -- dumps state info for evm objects

#crsctl debug log crs [module:level]{,module:level} ...

-- Turns on debugging for CRS
#crsctl debug log crs CRSEVT:5,CRSAPP:5,CRSTIMER:5,CRSRES:5,CRSRTI:1,CRSCOMM:2
#crsctl debug log css [module:level]{,module:level} ...
-- Turns on debugging for CSS
#crsctl debug log css CSSD:1
#crsctl debug log evm [module:level]{,module:level} ...
-- Turns on debugging for EVM
#crsctl debug log evm EVMCOMM:1

#crsctl debug trace crs -- dumps CRS in-memory tracing cache

#crsctl debug trace css -- dumps CSS in-memory tracing cache
#crsctl debug trace evm -- dumps EVM in-memory tracing cache

#crsctl debug log res resource_name:level -- turns on debugging for resources

#crsctl debug log res "ora.lnx04.vip:1"

#crsctl trace all_the_above_commands -- tracing by adding a "trace" argument.

#crsctl trace check css

#crsctl backup -h
#crsctl backup css votedisk

Here is the list of the options for CRSCTL in 11gR2:

crsctl add - add a resource, type or other entity
crsctl backup - back up voting disk for CSS
crsctl check - check a service, resource or other entity
crsctl config - output autostart configuration
crsctl debug - obtain or modify debug state
crsctl delete - delete a resource, type or other entity
crsctl disable - disable autostart
crsctl discover - discover DHCP server
crsctl enable - enable autostart
crsctl get - get an entity value
crsctl getperm - get entity permissions
crsctl lsmodules - list debug modules
crsctl modify - modify a resource, type or other entity
crsctl query - query service state
crsctl pin - Pin the nodes in the nodelist
crsctl relocate - relocate a resource, server or other entity
crsctl replace - replaces the location of voting files
crsctl release - release a DHCP lease
crsctl request - request a DHCP lease
crsctl setperm - set entity permissions
 crsctl set - set an entity value
crsctl start - start a resource, server or other entity
crsctl status - get status of a resource or other entity
crsctl stop - stop a resource, server or other entity
crsctl unpin - unpin the nodes in the nodelist
crsctl unset - unset a entity value, restoring its default

crsctl add resource resource_name -type resource_type [-file file_path | -attr

"attribute_name=attribute_value,attribute_name=attribute_value,..."] [-i] [-f]
crsctl add resource r1 -type test_type1 -attr "PATH_NAME=/tmp/r1.txt"
crsctl add resource app.appvip -type app.appvip.type -attr "RESTART_ATTEMPTS=2,
START_TIMEOUT=100,STOP_TIMEOUT=100,CHECK_INTERVAL=10,USR_ORA_VIP=172.16.0.0,
START_DEPENDENCIES=hard(ora.net1.network)pullup(ora.net1.network),
STOP_DEPENDENCIES=hard(ora.net1.network)"
crsctl add type type_name -basetype base_type_name {-attr "ATTRIBUTE=attribute_name
| -file file_path,TYPE={string | int} [,DEFAULT_VALUE=default_value]
[,FLAGS=[READONLY][|REQUIRED]]"}
crsctl add type test_type1 -basetype cluster_resource -attr
"ATTRIBUTE=FOO,TYPE=integer,DEFAULT_VALUE=0"
crsctl add crs administrator -u user_name [-f]
crsctl add crs administrator -u scott
crsctl add css votedisk path_to_voting_disk [path_to_voting_disk ...] [-purge]
crsctl add css votedisk /stor/grid/ -purge
crsctl add serverpool server_pool_name {-file file_path | -attr
"attr_name=attr_value[,attr_name=attr_value[,...]]"} [-i] [-f]
crsctl add serverpool testsp -attr "MAX_SIZE=5"
crsctl add serverpool sp1 -file /tmp/sp1_attr

crsctl check cluster [-all | [-n server_name [...]]

crsctl check cluster -all
crsctl check crs
crsctl check css
crsctl check ctss -- Cluster Time Synchronization services
crsctl check evm
crsctl check resource {resource_name [...] | -w "filter" } [-n node_name] [-k
cardinality_id] [-d degree_id] }
crsctl check resource appsvip

crsctl config crs

crsctl delete crs administrator -u user_name [-f]

crsctl delete crs administrator -u scott
crsctl delete resource resource_name [-i] [-f]
crsctl delete resource myResource
crsctl delete type type_name [-i]
crsctl delete type app.appvip.type
crsctl delete css votedisk voting_disk_GUID [voting_disk_GUID [...]]
crsctl delete css votedisk 61f4273ca8b34fd0bfadc2531605581d
crsctl delete node -n node_name
crsctl delete node -n node06
crsctl delete serverpool server_pool_name [server_pool_name [...]] [-i]
crsctl delete serverpool sp1

crsctl disable crs

crsctl discover dhcp -clientid clientid [-port port]

crsctl discover dhcp -clientid dsmjk252clr-dtmk01-vip
crsctl enable crs

crsctl get hostname

crsctl get clientid dhcp -cluname cluster_name -viptype vip_type [-vip
vip_res_name] [-n node_name]
crsctl get clientid dhcp -cluname dsmjk252clr -viptype HOSTVIP -n tmjk01
crsctl get css parameter
crsctl get css disktimeout
crsctl get css ipmiaddr
crsctl get nodename
crsctl getperm resource resource_name [ {-u user_name | -g group_name} ]
crsctl getperm resource app.appvip
crsctl getperm resource app.appvip -u oracle
crsctl getperm resource app.appvip -g dba
crsctl getperm type resource_type [-u user_name] | [-g group_name]
crsctl getperm type app.appvip.type
crsctl getperm serverpool server_pool_name [-u user_name | -g group_name]
crsctl getperm serverpool sp1

crsctl lsmodules {mdns | gpnp | css | crf | crs | ctss | evm | gipc}
crsctl lsmodules evm
mdns: Multicast domain name server
gpnp: Grid Plug and Play service
css: Cluster Synchronization Services
crf: Cluster Health Monitor
crs: Cluster Ready Services
ctss: Cluster Time Synchronization Service
evm: Event Manager
gipc: Grid Interprocess Communication

crsctl modify resource resource_name -attr "attribute_name=attribute_value" [-i] [-

f] [-delete]
crsctl modify resource appsvip -attr USR_ORA_VIP=10.1.220.17 -i
crsctl modify type type_name -attr "ATTRIBUTE=attribute_name,TYPE={string | int}
[,DEFAULT_VALUE=default_value [,FLAGS=[READONLY][| REQUIRED]]" [-i] [-f]]
crsctl modify type myType.type -attr "ATTRIBUTE=FOO,DEFAULT_VALUE=0
ATTRIBUTE=BAR,DEFAULT_VALUE=baz"
crsctl modify serverpool server_pool_name -attr "attr_name=attr_value
[,attr_name=attr_value[, ...]]" [-i] [-f]
crsctl modify serverpool sp1 -attr "MAX_SIZE=7"

crsctl pin css -n node_name [ node_name [..]]

crsctl pin css -n node2

crsctl query crs administrator

crsctl query crs activeversion
crsctl query crs releaseversion
crsctl query crs softwareversion node_name
crsctl query css ipmiconfig
crsctl query css ipmidevice
crsctl query css votedisk
crsctl query dns {-servers | -name name [-dnsserver DNS_server_address] [-port
port] [-attempts number_of_attempts] [-timeout timeout_in_seconds] [-v]}

crsctl release dhcp -clientid clientid [-port port]

crsctl release dhcp -clientid spmjk662clr-spmjk03-vip

crsctl relocate resource {resource_name | resource_name | -all -s source_server |

-w "filter"} [-n destination_server] [-k cid] [-env "env1=val1,env2=val2,..."] [-i]
[-f]
crsctl relocate resource myResource1 -s node1 -n node3
crsctl relocate server server_name [...] -c server_pool_name [-i] [-f]
crsctl relocate server node6 node7 -c sp1

crsctl replace discoverystring 'absolute_path[,...]'

crsctl replace discoverystring "/oracle/css1/*,/oracle/css2/*"
crsctl replace votedisk [+asm_disk_group | path_to_voting_disk [...]]
crsctl replace votedisk +diskgroup1
crsctl replace votedisk /mnt/nfs/disk1 /mnt/nfs/disk2

crsctl request dhcp -clientid clientid [-port port]

crsctl request dhcp -clientid tmj0462clr-tmjk01-vip

crsctl set css parameter value

crsctl set css ipmiaddr ip_address
crsctl set css ipmiaddr 192.0.2.244
crsctl set css ipmiadmin ipmi_administrator_name
crsctl set css ipmiadmin scott

crsctl set log {[crs | css | evm "component_name=log_level, [...]"] |

[all=log_level]}
crsctl set log crs "CRSRTI=1,CRSCOMM=2"
crsctl set log evm all=2
crsctl set log res "myResource1=3"
crsctl set {log | trace} module_name "component:debugging_level
[,component:debugging_level][,...]"
crsctl set log crs "CRSRTI:1,CRSCOMM:2"
crsctl set log crs "CRSRTI:1,CRSCOMM:2,OCRSRV:4"
crsctl set log evm "EVMCOMM:1"
crsctl set log res "resname:1"
crsctl set log res "resource_name=debugging_level"
crsctl set log res "ora.node1.vip:1"
crsctl set log crs "CRSRTI:1,CRSCOMM:2" -nodelist node1,node2
crsctl set trace "component_name=tracing_level,..."
crsctl set trace "css=3"

crsctl setperm resource resource_name {-u acl_string | -x acl_string | -o user_name

| -g group_name}
crsctl setperm resource myResource -u user:scott:rwx
crsctl setperm type resource_type_name {-u acl_string | -x acl_string | -o
user_name | -g group_name}
crsctl setperm type resType -u user:scott:rwx
crsctl setperm serverpool server_pool_name {-u acl_string | -x acl_string | -o
user_name | -g group_name}
crsctl setperm serverpool sp3 -u user:scott.tiger:rwx

crsctl start cluster [-all | -n server_name [...]]

crsctl start cluster -n node1 node2
crsctl start crs
crsctl start ip -A {IP_name | IP_address}/netmask/interface_name
crsctl start ip -A 192.168.29.220/255.255.252.0/eth0
crsctl start resource {resource_name [...] | -w filter | -all} [-n server_name] [-k
cid] [-d did] [-env "env1=val1,env2=val2,..."] [-i] [-f]
crsctl start resource myResource -n server1
crsctl start testdns [-address address [-port port]] [-once] [-v]
crsctl start testdns -address 192.168.29.218 -port 63 -v

crsctl status resource {resource_name [...] | -w "filter"} [-p | -v [-e]] | [-f |

-l | -g] [[-k cid | -n server_name] [-d did]] | [-s -k cid [-d did]] [-t]
crsctl status resource ora.stai14.vip
crsctl stat res -w "TYPE = ora.scan_listner.type"
crsctl status type resource_type_name [...]] [-g] [-p] [-f]
crsctl status type ora.network.type
crsctl status ip -A {IP_name | IP_address}
crsctl status server [-p | -v | -f]
crsctl status server { server_name [...] | -w "filter"} [-g | -p | -v | -f]
crsctl status server node2 -f
crsctl status serverpool [-p | -v | -f]
crsctl status serverpool [server_pool_name [...]] [-w] [-g | -p | -v | -f]
crsctl status serverpool sp1 -f
crsctl status serverpool
crsctl status serverpool -p
crsctl status serverpool -w "MAX_SIZE > 1"
crsctl status testdns [-server DNS_server_address] [-port port] [-v]

crsctl stop cluster [-all | -n server_name [...]] [-f]

crsctl stop cluster -n node1
crsctl stop crs [-f]
crsctl stop crs
crsctl stop resource {resource_name [...] | -w "filter" | -all} [-n server_name] [-
k cid] [-d did] [-env "env1=val1,env2=val2,..."] [-i] [-f]
crsctl stop resource -n node1 -k 2
crsctl stop ip -A {IP_name | IP_address}/interface_name
crsctl stop ip -A MyIP.domain.com/eth0
crsctl stop testdns [-address address [-port port]] [-domain GNS_domain] [-v]

crsctl unpin css -n node_name [node_name [...exit]]

crsctl unpin css -n node1 node4

crsctl unset css parameter

crsctl unset css reboottime
crsctl unset css ipmiconfig

HAS (High Availability Service)

crsctl check has
crsctl config has
crsctl disable has
crsctl enable has
crsctl query has releaseversion
crsctl query has softwareversion
crsctl start has
crsctl stop has [-f]

ocrcheck commands in Oracle RAC

OCRCHECK: -- Displays health of OCR (Oracle Cluster Registry).

$ocrcheck -help or ocrcheck -h

$ocrcheck
Version : 2
Total space (kbytes) : 262144
Used space (kbytes) : 16256
Available space (kbytes) : 245888
ID : 1918913332
Device/File Name : /dev/raw/raw1 Device/File integrity check succeeded
Device/File Name : /dev/raw/raw2 Device/File integrity check succeeded
Cluster registry integrity check succeeded

#ocrcheck -local
Status of Oracle Cluster Registry is as follows :
Version : 3
Total space (kbytes) : 262132
Used space (kbytes) : 9200
Available space (kbytes) : 252932
ID : 604793089
Device/File Name : /u02/crs/cdata/localhost/lnx6.olr Device/File integrity check
succeeded
Local OCR integrity check succeeded

$ocrcheck -local -config

Log file will be $ORACLE_HOME/log/node_name/client/ocrcheck_pid.log

Debugging can be controlled through $ORA_CRS_HOME/srvm/admin/ocrlog.ini

How do I identify the OCR file location?

#ocrcheck

ocrconfig commands in Oracle RAC

OCRCONFIG: -- OCR (Oracle Cluster Registry) CONFIGuration tool
#ocrconfig -help or ocrconfig -h

#ocrconfig -showbackup [auto|manual]

-- default location is $ORA_CRS_HOME/cdata/cluster_name
#ocrconfig -showbackup
#ocrconfig -backuploc dir_name -- change OCR autobackup directory location
#ocrconfig -backuploc /u02/backups

#ocrconfig -manualbackup -- Oracle RAC 11g command, to perform OCR backup

manually
#ocrconfig -restore backup_file.ocr -- recovering from autobackup file
#ocrconfig -restore /u02/backups/backup00.ocr

#ocrconfig -export file_name.dmp [-s online] -- exports OCR content to a file

#ocrconfig -export /tmp/ocr_exp
#ocrconfig -import file_name.dmp
-- recover OCR logically, must be done on all nodes
#ocrconfig -import /tmp/ocr_exp

#ocrconfig -replace ocr [file_name] -- adding/replacing an OCR file

#ocrconfig -replace ocrmirror [file_name]
#ocrconfig -replace ocrmirror /data/ocr/ocrfile2

#ocrconfig -repair ocr file_name

#ocrconfig -repair ocrmirror file_name
#ocrconfig -repair -replace current_OCR_location -replacement target_OCR_location

#ocrconfig -upgrade [user [group]] -- upgrades OCR

#ocrconfig -downgrade [-version version_string] -- downgrades OCR

#ocrconfig -overwrite

#ocrconfig �local �import file_name

#ocrconfig �local �manualbackup
#ocrconfig -local -backuploc new_olr_backup_path
#ocrconfig -local -restore file_name
#ocrconfig -local -restore $GRID_HOME/cdata/host06/backup_20171106_02946.olr
--restoring OLR file

#ocrconfig -add +new_disk_group

#ocrconfig -delete +unused_disk_group

#ocrconfig -add file_location

#ocrconfig -add /dev/sdd1
#ocrconfig -delete old_storage_location
#ocrconfig -delete /dev/raw/raw2

Log file will be $ORACLE_HOME/log/node_name/client/ocrconfig_pid.log

Debugging can be controlled through $ORA_CRS_HOME/srvm/admin/ocrlog.ini

How to take backup of OCR file?

#ocrconfig -manualbackup
#ocrconfig -export file_name.dmp

How to recover OCR file?

#ocrconfig -restore backup_file.ocr
#ocrconfig -import file_name.dmp

ocrdump commands in Oracle RAC

OCRDUMP: -- dumps OCR (Oracle Cluster Registry) contents to a file

#ocrdump -help or ocrdump -h

#ocrdump [file_name|-stdout] [-backupfile backup_filename] [-keyname key_name] [-

xml] [-noheader]

#ocrdump -- default filename is OCRDUMPFILE

#ocrdump MYFILE
#ocrdump ${HOST}_OCRDUMP
#ocrdump -backupfile my_file
#ocrdump -stdout -keyname SYSTEM
#ocrdump -stdout -xml
$ocrdump -local olr.lst --> Normal Text Format
$ocrdump -local -xml olr_xml.lst --> XML format
$ocrdump -local -backupfile olr_backup_file_name

Log file will be $ORACLE_HOME/log/node_name/client/ocrdump_pid.log

Debugging can be controlled through $ORA_CRS_HOME/srvm/admin/ocrlog.ini

How to take backup of OCR file?

#ocrdump -backupfile my_file

oifcfg commands in Oracle RAC

OIFCFG: -- Oracle Interface Configuration tool
A command line tool for both single instance Oracle databases and RAC databases
that enables us to allocate and deallocate network interfaces to components, direct
components to use specific network interfaces, and retrieve component configuration
information.

oifcfg [-help] -- will give help

$ ./oifcfg -help
$ ./oifcfg
oifcfg iflist [-p [-n]]
-p includes description of the subnet, -n includes subnet mask

oifcfg iflist -- display a list of current subnets

etho 147.43.1.60
ethl 192.168.1.150

oifcfg iflist -p -n
etho 147.43.1.60 UNKNOWN 255.255.255.0 (public interfaces are UNKNOWN)
ethl 192.168.1.150 PRIVATE 255.255.255.0

oifcfg getif [-node node_name|-global] [-if if_name[/subnet] [-type

{cluster_interconnect|public|storage}]]
-- To display a list of networks

oifcfg getif
eth1 192.168.1.150 global cluster_interconnect
eth0 192.168.0.150 global public

oifcfg setif {-node node_name|-global} {if_name/subnet:{cluster_interconnect|

public|storage}}...
oifcfg setif -global eth0/10.50.99.0:public
oifcfg setif -global eth0/172.19.141.0:cluster_interconnect

oifcfg delif [-node node_name|-global] [if_name[/subnet]]

oifcfg delif -global
oifcfg delif -global eth0
oifcfg delif -global eth1/172.21.65.0

olsnodes commands in Oracle RAC

OLSNODES

Provides the list of nodes and other information for all nodes participating in the
cluster

#olsnodes [node_name] [-g] [-i] [-l] [-n] [-p] [-s] [-t] [-v]
node_name -- displays information for the particular node

g -- more details
i -- with VIP
l -- local node name
n -- with node number
p -- private interconnect
s -- status of the node (ACTIVE or INACTIVE)
t -- type of the node (PINNED or UNPINNED)
v -- verbose

How to find out the nodes in Oracle RAC cluster?

#olsnodes -- will list the nodes in the cluster

#olsnodes -n

#olsnodes node44 -v

#olsnodes -n -p -i
node1-pub 1 node1-prv node1-vip
node2-pub 2 node2-prv node2-vip

#olsnodes -i
node1 178.192.1.1
node2 178.192.2.1
node3 178.192.3.1
node4 178.192.4.1

#olsnodes -c

How to find out the nodes in cluster (or) how to find out the master node?

#olsnodes - Which ever displayed first, is the master node of the cluster.

Oracle RAC Interview Questions/FAQs Part1

Oracle DBA RAC Interview Questions Part1

1. What is the use of RAC?

2. What are the prerequisites for RAC setup?

3. What are Oracle Clusterware/Daemon processes and what they do?

Ans:
ohasd, orarootagent, oragent, ocssd, crsd, ctsd, evmd,

4. What are the special background processes for RAC (or) what is difference in
stand-alone database & RAC database background processes?
DIAG, LCKn, LMD, LMSn, LMON

5. What are structural changes in 11g R2 RAC?

Ans:
http://satya-racdba.blogspot.com/2010/07/new-features-in-9i-10g-11g-rac.html
Grid & ASM are on one home,
Voting disk & ocrfile can be on the ASM,
SCAN,
By using srvctl, we can mange diskgroups, home, ons, eons, filesystem, srvpool,
server, scan, scan_listener, gns, vip, oc4j,
GSD

6. What are the new features in 11g (R2) RAC?

Ans:
http://satya-racdba.blogspot.com/2010/07/new-features-in-9i-10g-11g-rac.html
Grid & ASM are on one home,
Voting disk & ocrfile can be on the ASM,
SCAN,
By using srvctl, we can mange diskgroups, home, ons, eons, filesystem, srvpool,
server, scan, scan_listener, gns, vip, oc4j,
GSD

7. What is cache fusion?

Ans:
Transferring of data between RAC instances by using private network. Cache Fusion
is the remote memory mapping of Oracle buffers, shared between the caches of
participating nodes in the cluster. When a block of data is read from datafile by
an instance within the cluster and another instance is in need of the same block,
it is easy to get the block image from the instance which has the block in its SGA
rather than reading from the disk.

8. What is the purpose of Private Interconnect?

Ans:
Clusterware uses the private interconnect for cluster synchronization (network
heartbeat) and daemon communication between the clustered nodes. This communication
is based on the TCP protocol. RAC uses the interconnect for cache fusion (UDP/RDS)
and inter-process communication (TCP).

9. What are the Clusterware components?

Ans:
Voting Disk - Oracle RAC uses the voting disk to manage cluster membership by way
of a health check and arbitrates cluster ownership among the instances in case of
network failures. The voting disk must reside on shared disk.

Oracle Cluster Registry (OCR) - Maintains cluster configuration information as well

as configuration information about any cluster database within the cluster. The OCR
must reside on shared disk that is accessible by all of the nodes in your cluster.
The daemon OCSSd manages the configuration info in OCR and maintains the changes to
cluster in the registry.

Virtual IP (VIP) - When a node fails, the VIP associated with it is automatically
failed over to some other node and new node re-arps the world indicating a new MAC
address for the IP. Subsequent packets sent to the VIP go to the new node, which
will send error RST packets back to the clients. This results in the clients
getting errors immediately.

crsd � Cluster Resource Services Daemon

cssd � Cluster Synchronization Services Daemon
evmd � Event Manager Daemon

10. What is OCR file?

Ans:
RAC configuration information repository that manages information about the cluster
node list and instance-to-node mapping information. The OCR also manages
information about Oracle Clusterware resource profiles for customized applications.
Maintains cluster configuration information as well as configuration information
about any cluster database within the cluster. The OCR must reside on shared disk
that is accessible by all of the nodes in your cluster. The daemon OCSSd manages
the configuration info in OCR and maintains the changes to cluster in the registry.

11. How to take backup of OCR file?

Ans:
#ocrconfig -manualbackup
#ocrconfig -export file_name.dmp
#ocrdump -backupfile my_file
$cp -p -R /u01/app/crs/cdata /u02/crs_backup/ocrbackup/RAC1

12. How to recover OCR file?

Ans:
#ocrconfig -restore backup_file.ocr
#ocrconfig -import file_name.dmp

13. What is local OCR (OLR)?

Ans:
/etc/oracle/local.ocr
/var/opt/oracle/local.ocr

14. How to check backup (location) of OCR files?

Ans:
#ocrconfig �showbackup

15. How do you identify the OCR file location?

Ans:
check /var/opt/oracle/ocr.loc or /etc/ocr.loc
# ocrcheck

16. Who will manage OCR files?

Ans:
cssd will manage OCR.

17. Who will take backup of OCR files?

Ans:
crsd will take backup.

18. What is Voting file/disk and how many files should be there?
Ans:
Voting Disk File is a file on the shared cluster system or a shared raw device
file. Oracle Clusterware uses the voting disk to determine which instances are
members of a cluster. Voting disk is akin to the quorum disk, which helps to avoid
the split-brain syndrome. Oracle RAC uses the voting disk to manage cluster
membership by way of a health check and arbitrates cluster ownership among the
instances in case of network failures. The voting disk must reside on shared disk.

Number of voting files must be odd i.e. 1, 3, 5, 7, 9, 11, 13, 15(max) etc.

19. How to take backup of voting file?

Ans:
dd if=/u02/ocfs2/vote/VDFile_0 of=$ORACLE_BASE/bkp/vd/VDFile_0
crsctl backup css votedisk -- from 11g R2

20. How do I identify the voting disk location?

Ans:
# crsctl query css votedisk

21. If voting disk/OCR file got corrupted and don�t have backups, how to get them?
Ans:
We have to install Clusterware.

Oracle DBA RAC Interview Questions Part2

Oracle RAC Interview Questions/FAQs Part2

22. What is split brain syndrome?

Ans:
Will arise when two or more instances attempt to control a cluster database. In a
two-node environment, one instance attempts to manage updates simultaneously while
the other instance attempts to manage updates.

23. What are various IPs used in RAC? Or How may IPs we need in RAC?
Ans:
Public IP, Private IP, Virtual IP, SCAN IP

24. What is the use of virtual IP?

Ans:
When a node fails, the VIP associated with it is automatically failed over to some
other node and new node re-arps the world indicating a new MAC address for the IP.
Subsequent packets sent to the VIP go to the new node, which will send error RST
packets back to the clients. This results in the clients getting errors
immediately.

Without using VIPs or FAN, clients connected to a node that died will often wait
for a TCP timeout period (which can be up to 10 min) before getting an error. As a
result, you don't really have a good HA solution without using VIPs.

25. What is the use of SCAN IP (SCAN name) and will it provide load balancing?
Ans:
Single Client Access Name (SCAN) is a new Oracle Real Application Clusters (RAC)
11g Release 2, feature that provides a single name for clients to access an Oracle
Database running in a cluster. The benefit is clients using SCAN do not need to
change if you add or remove nodes in the cluster.

26. How many SCAN listeners will be running?

Ans:
Three SCAN listeners only.

27. What is FAN?

Ans:
Applications can use Fast Application Notification (FAN) to enable rapid failure
detection, balancing of connection pools after failures, and re-balancing of
connection pools when failed components are repaired. The FAN process uses system
events that Oracle publishes when cluster servers become unreachable or if network
interfaces fail.

28. What is FCF?

Ans:
Fast Connection Failover provides high availability to FAN integrated clients, such
as clients that use JDBC, OCI, or ODP.NET. If you configure the client to use fast
connection failover, then the client automatically subscribes to FAN events and can
react to database UP and DOWN events. In response, Oracle gives the client a
connection to an active instance that provides the requested database service.

29. What is TAF and TAF policies?

Ans:
Transparent Application Failover (TAF) - A runtime failover for high availability
environments, such as Real Application Clusters and Oracle Real Application
Clusters Guard, TAF refers to the failover and re-establishment of application-to-
service connections. It enables client applications to automatically reconnect to
the database if the connection fails, and optionally resume a SELECT statement that
was in progress. This reconnect happens automatically from within the Oracle Call
Interface (OCI) library.

30. How will you upgrade RAC database?

31. What are rolling patches and how to apply?

32. How to add/remove a node?

33. What are nodeapps?

Ans:
VIP, listener, ONS, GSD

34. What is gsd (Global Service Daemon)?

35. How to do load balancing in RAC?

36. What are the uses of services? How to find out the services in cluster?
Ans:
Applications should use the services to connect to the Oracle database. Services
define rules and characteristics (unique name, workload balancing, failover
options, and high availability) to control how users and applications connect to
database instances.

37. How to find out the nodes in cluster (or) how to find out the master node?
Ans:
To find out which is the master node, use any one of the below.
i) # olsnodes -- Which ever displayed first, is the master node of the cluster.
ii) select MASTER_NODE from v$ges_resource;
iii) check ocssd.log file and search for "master node number".
iv) oclumon manage -get master (in Oracle RAC 12c)
v) ocrconfig -manualbackup

38. How to know the public IPs, private IPs, VIPs in RAC?
Ans:
# olsnodes -n -p -i
node1-pub 1 node1-prv node1-vip
node2-pub 2 node2-prv node2-vip

39. What utility is used to start DB/instance?

Ans:
srvctl start database �d database_name
srvctl start instance �d database_name �i instance_name

40. How can you shutdown single instance?

Ans:
Change cluster_database=false
srvctl stop instance �d database_name �i instance_name

Oracle RAC Interview Questions/FAQs Part3

Oracle RAC DBA Interview Questions Part3

41. What is HAS (High Availability Service) and the commands?

Ans:
HAS includes ASM & database instance and listeners.
crsctl check has
crsctl config has
crsctl disable has
crsctl enable has
crsctl query has releaseversion
crsctl query has softwareversion
crsctl start has
crsctl stop has [-f]

42. How many nodes are supported in a RAC Database?

Ans:
10g Release 2, support 100 nodes in a cluster using Oracle Clusterware, and 100
instances in a RAC database.

43. What is fencing?

Ans:
I/O fencing prevents updates by failed instances, and detecting failure and
preventing split brain in cluster. When a cluster node fails, the failed node needs
to be fenced off from all the shared disk devices or diskgroups. This methodology
is called I/O Fencing, sometimes called Disk Fencing or failure fencing.

44. Why Clusterware installed in root (why not in oracle)?

45. What are the wait events in RAC?
Ans:
http://satya-racdba.blogspot.com/2012/10/wait-events-in-oracle-rac-wait-events.html

gc buffer busy
gc buffer busy acquire
gc current request
gc cr request
gc cr failure
gc current block lost
gc cr block lost
gc current block corrupt
gc cr block corrupt
gc current block busy
gc cr block busy
gc current block congested
gc cr block congested.
gc current block 2-way
gc cr block 2-way
gc current block 3-way
gc cr block 3-way
(even if we have n number of nodes, there can be only 3-way wait event)
gc current grant 2-way
gc cr grant 2-way
gc current grant busy
gc current grant congested
gc cr grant congested
gc cr multi block read
gc current multi block request
gc cr multi block request
gc cr block build time
gc current block flush time
gc cr block flush time
gc current block send time
gc cr block send time
gc current block pin time
gc domain validation
gc current retry
ges inquiry response
gcs log flush sync

46. What is the difference between cr block and cur (current) block?

47. What are the initialization parameters that must have same value for every
instance in an Oracle RAC database?
Ans:
http://satya-racdba.blogspot.com/2012/09/init-parameters-in-oracle-rac.html

ACTIVE_INSTANCE_COUNT
ARCHIVE_LAG_TARGET
COMPATIBLE
CLUSTER_DATABASE
CLUSTER_DATABASE_INSTANCE
CONTROL_FILES
DB_BLOCK_SIZE
DB_DOMAIN
DB_FILES
DB_NAME
DB_RECOVERY_FILE_DEST
DB_RECOVERY_FILE_DEST_SIZE
DB_UNIQUE_NAME
INSTANCE_TYPE
PARALLEL_MAX_SERVERS
REMOTE_LOGIN_PASSWORD_FILE
UNDO_MANAGEMENT

48. What are the new features in Oracle RAC 12c?

Ans:
http://satya-racdba.blogspot.com/2012/10/new-features-in-oracle-rac-12c.html

49. How is the Cluster Health Monitor different from OSWatcher?

50. Why we need user equivalency? What happen if we remove it between node 3,5 in a
Production of 6 node RAC cluster?

51. What is the purpose of OLR?

With the OCR files being placed in ASM and accessible only when the cluster
processes have started up and ASM instance is up and running. However without the
OCR file the clusterware processes cannot start up.

To resolve this, a copy of the Oracle Local Registry(OLR) registry is now created
on each node during the initial installation which stores node specific information
and allows us to get around the problem.

Oracle Local Repository (OLR) contains information that allows the cluster
processes to be started up with the OCR being in the ASM storage system. Since the
ASM file system is unavailable until the Grid processes are started up a local copy
of the contents of the OCR is required which is stored in the OLR.

OLR stores the information on the clusterware configuration, version information,

and GpnP wallets. OHASD process mostly manages this file.

52. What is difference is between relocating the SCAN using srvctl relocate scan
and SCAN_LISTENER by using srvctl relocate scan_listener command?

53. What is the difference between Crash recovery and Instance recovery?
When an instance crashes in a single node database on startup a crash recovery
takes place. In a RAC environment the same recovery for an instance is performed by
the surviving nodes called Instance recovery.

54. Why is the interconnect used for?

It is a private network which is used to ship data blocks from one instance to
another for cache fusion. The physical data blocks as well as data dictionary
blocks are shared across this interconnect.

55. How do you determine what protocol is being used for Interconnect traffic?
One of the ways is to look at the database alert log for the time period when the
database was started up.

56. What files components in RAC must reside on shared storage?

Spfiles, ControlFiles, Datafiles and Redolog files should be created on shared
storage.

SCAN in Oracle RAC

Single Client Access Name (SCAN)

Oracle RAC 11g release 2 introduces the Single Client Access Name (SCAN), which
provides a single name for clients to access Oracle Databases running in a cluster
and simplify the database connection strings that an Oracle Client uses to connect.

Without SCAN, till Oracle Clusterware 11g release 1, TNSNAMES has 1 entry per node.

RACDB =
(DESCRIPTION =
(ADDRESS = (PROTOCOL = TCP)(HOST = node1-vip)(PORT = 1521))
(ADDRESS = (PROTOCOL = TCP)(HOST = node2-vip)(PORT = 1521))
. . .
(ADDRESS = (PROTOCOL = TCP)(HOST = nodeN-vip)(PORT = 1521))
(CONNECT_DATA =
. . . ))

With every cluster change, like node addition or removal, all client TNSNAMES need
to be changed.
With SCAN, from Oracle Clusterware 11g release2, only 1 entry per cluster is used,
regardless of the number of nodes.

RACDB =
(DESCRIPTION =
(ADDRESS = (PROTOCOL = TCP)(HOST = SCANname)(PORT = 1521))
(CONNECT_DATA =
. . . ))

Single Client Access Name (SCAN) - eliminates the need to change TNSNAMES entry
when nodes are added to or removed from the Cluster. RAC instances register to SCAN
listeners as remote listeners. Oracle recommends assigning 3 addresses to SCAN,
which will create 3 SCAN listeners, though the cluster has got dozens of nodes.
SCAN is a domain name registered to at least one and up to three IP addresses,
either in DNS (Domain Name Service) or GNS (Grid Naming Service). The SCAN must
resolve to at least one address on the public network. For high availability and
scalability, Oracle recommends configuring the SCAN to resolve to three addresses.

SCAN components in a cluster:

SCAN Name
SCAN IPs (3)
SCAN Listeners (3)

Public network addresses are used to provide services to clients. If clients are
connecting to the Single Client Access Name (SCAN) addresses, then we may need to
change public and virtual IP addresses as we add or remove nodes from the cluster,
but you do not need to update clients with new cluster addresses. SCANs function
like a cluster alias. SCANs are resolved on any node in the cluster, so unlike a
VIP address for a node, clients connecting to the SCAN no longer require updated
VIP addresses as nodes are added to or removed from the cluster. Because the SCAN
addresses resolve to the cluster, rather than to a node address in the cluster,
nodes can be added to or removed from the cluster without affecting the SCAN
address configuration.

Oracle Database 11g release 2 (11.2) instances only register with SCAN listeners as
remote listeners. Databases register with SCAN listeners as remote listeners, and
also continue to register with all node listeners. It's highly recommended that the
clients are Oracle 11g R2 or later version clients, to allow them to fully take
advantage of the failover with the SCAN settings.

Because of the Oracle Clusterware installation requirement that we provide a SCAN

name during installation, if we resolved at least one IP address using the
server /etc/hosts file to bypass the installation requirement, but we do not have
the infrastructure required for SCAN, after the installation, we can ignore the
SCAN and connect to the databases in the cluster using VIPs. Oracle does not
support removing the SCAN address. Oracle recommends that we should not configure
SCAN IP addresses in the hosts file. But if we use the hosts file to resolve SCAN
name, we can have only one SCAN IP address.

Each cluster will have 3 SCAN-Listeners, combined with a SCAN-VIP defined as

cluster resources. The root.sh script automatically configures the Single Client
Access Name (SCAN) VIPs and SCAN listeners. When we run root.sh on the first node,
all 3 scan IPs are plugged on to the first node�s public interface. The SCAN
VIP/LISTENER combination will failover to another node in the cluster, if the
current node fails. SCAN VIP addresses must be on the same subnet as virtual IP
addresses and public IP addresses.

Cluster Resources
--------------------------------------------
ora.LISTENER_SCAN1.lsnr 1 ONLINE ONLINE node1
ora.LISTENER_SCAN2.lsnr 1 ONLINE ONLINE node2
ora.LISTENER_SCAN3.lsnr 1 ONLINE ONLINE node3

Domain Name Service (DNS)

The SCAN is a fully qualified name (hostname+domain) that is configured to resolve
to all the addresses allocated for the SCAN. The addresses resolve using Round
Robin DNS either on the DNS server, or within the cluster in a GNS configuration.
SCAN listeners can run on any node in the cluster. SCANs provide location
independence for the databases, so that client configuration does not have to
depend on which nodes run a particular database.

To use the DNS method for defining SCAN, the network administrator must create a
single name that resolves to 3 separate IP addresses using round-robin algorithms.
Regardless of how many systems are part of cluster, Oracle recommends that 3 IP
addresses are configured to allow for failover and load-balancing. SCAN Name needs
to be registered in DNS and resolved using round-robin algorithm. To test the
round-robin algorithm, perform nslookup on the SCAN Name. The DNS server should
return those 3 set of IP Address in different order each time.

It is important that the IP addresses are on the same subnet as the public network
for the server. The other two requirements are that the name (not including the
domain suffix) are 15 characters or less in length and that the name can be
resolved without using the domain suffix. Also, the IP addresses should not be
specifically assigned to any of the nodes in the cluster.

Grid Naming Service (GNS)

Grid Naming Service (GNS) is a new feature introduced in Oracle RAC 11g R2.

As part of the Grid Infrastructure installation process, optionally we can also

provide Grid Naming Service (GNS) information, which includes the GNS Sub Domain
Name and the GNS VIP Address.

Using GNS assumes that a DHCP server is running on the public network with enough
available addresses to assign the required IP addresses and the SCAN VIP. Only one
static IP address is required to be configured and it should be in the DNS domain.

Oracle Clusterware 11g release 2 (11.2) supports the use of Dynamic Host
Configuration Protocol (DHCP) for the VIP addresses and the SCAN address, but not
the public address. DHCP provides dynamic configuration of the host's IP address,
but it does not provide an optimal method of producing names that are useful to
external clients. When using GNS and DHCP, Oracle Clusterware configures the VIP
addresses for the SCAN name that is provided during cluster configuration. The node
VIP and the three SCAN VIPs are obtained from the DHCP server when using GNS. If a
new server joins the cluster, then Oracle Clusterware dynamically obtains the
required VIP address from the DHCP server, updates the cluster resource, and makes
the server accessible through GNS.

Summary of points about SCAN:

SCAN Name will represent the cluster in the network.
SCAN used by clients to connect to any database in the cluster.
SCAN is a GSD resource, which is managed by CRS.
SCAN provides a single domain name (via DNS), allowing end-users to address a RAC
cluster as-if it were a single IP address.
Removes the requirement to change the client connection if cluster changes.
Load balances across the instances providing a service. SCAN listener selects least
loaded node.
SCAN requires a DNS entry or GNS to be used.
SCAN is an alternative to the transparent application failover (TAF) for automatic
load balancing.
Provides failover between instances.
SCAN VIP/LISTENER will failover to another node in the cluster.
Clients do not require VIP information.
Must resolve to at least one address on the public network.
SCAN provides location independence for the databases.
Allow clients to use EZConnect or JDBC connections.
SCAN Listener would forward the request to local listener that�s running on VIPs.
Each cluster will have 3 SCAN listeners, each having a SCAN VIP defined as cluster
resources.
Instance registers with local listener on its node. Database �REMOTE_LISTENER�
registers instances with all SCAN listeners.
SCAN listeners on each node in the cluster, which are not replacements for a
regular listener.
PMON process of each instance registers the database services with the default
listener on the local node and with each SCAN listener, which is specified by the
REMOTE_LISTENER database parameter.
Client queries DNS to resolve to SCAN.

SCAN (Single Client Access Name) commands

cluvfy comp scan -> Use the cluvfy comp scan component verification command to
check the Single Client Access Name (SCAN) configuration.

crsctl status resource -w "TYPE = ora.scan_listner.type"

SCAN:
srvctl add scan -n scan_name [-k network_number] [-S subnet/netmask[/if1[|
if2|...]]]
#srvctl add scan -n scan.mycluster.example.com -- Adds new SCAN information for a
cluster

srvctl remove scan [-f] -- Removes SCAN information

srvctl remove scan
srvctl remove scan -f

srvctl start scan [-i ordinal_number] [-n node_name]

srvctl start scan
srvctl start scan -i 1 -n node1

srvctl stop scan [-i ordinal_number] [-f]

srvctl stop scan
srvctl stop scan -i 1
srvctl status scan [-i ordinal_number]
srvctl status scan
srvctl status scan -i 1

srvctl enable scan [-i ordinal_number]

srvctl enable scan
srvctl enable scan -i 1

srvctl disable scan [-i ordinal_number]

srvctl disable scan
srvctl disable scan -i 3

srvctl config scan [-i ordinal_number] -- Shows the current SCAN configuration
srvctl config scan
srvctl config scan -i 2

srvctl modify scan -n scan_name

srvctl modify scan -n scan1
-- Modifies SCAN information (used when changing SCAN to DNS after initially
using /etc/hosts)

srvctl relocate scan -i ordinal_number [-n node_name]

srvctl relocate scan -i 2 -n node2

ordinal_number=1,2,3 --> this refers to the SCAN VIPs

Scan_listener:
srvctl add scan_listener [-l lsnr_name_prefix] [-s] [-p "[TCP:]port_list[/IPC:key]
[/NMP:pipe_name][/TCPS:s_port] [/SDP:port]"]
#srvctl add scan_listener -l myscanlistener -- Adds a new SCAN listener for a
cluster on the default port of 1521
#srvctl add scan_listener -p 1565 -- Adds a new SCAN listener on a
different port

srvctl remove scan_listener [-f]

srvctl remove scan_listener -- Removes the SCAN listener
srvctl remove scan_listener -f

srvctl start scan_listener [-n node_name] [-i ordinal_number]

srvctl start scan_listener
srvctl start scan_listener -i 1

srvctl stop scan_listener [-i ordinal_number] [-f]

srvctl stop scan_listener
srvctl stop scan_listener -i 3

srvctl status scan_listener [-i ordinal_number]

srvctl status scan_listener
srvctl status scan_listener -i 1

srvctl enable scan_listener [-i ordinal_number]

srvctl enable scan_listener
srvctl enable scan_listener -i 2

srvctl disable scan_listener [-i ordinal_number]

srvctl disable scan_listener
srvctl disable scan_listener -i 1
srvctl config scan_listener [-i ordinal_number] -- Shows the existence and port
numbers for the SCAN listeners
srvctl config scan_listener
srvctl config scan_listener -i 3

srvctl modify scan_listener {-p [TCP:]port[/IPC:key][/NMP:pipe_name] [/TCPS:s_port]

[/SDP:port] | -u }
srvctl modify scan_listener �u -- Modifies the SCAN listener information to match
the new SCAN VIP information
srvctl modify scan_listener -p 1521

srvctl relocate scan_listener -i ordinal_number [-n node_name]

srvctl relocate scan_listener -i 1

ordinal_number=1,2,3

GNS (Grid Naming Service):

srvctl add gns -i ip_address -d domain
srvctl add gns -i 192.124.16.96 -d cluster.mycompany.com

srvctl remove gns [-f]

srvctl remove gns

srvctl start gns [-l log_level] [-n node_name]

srvctl start gns

srvctl stop gns [-n node_name [-v] [-f]

srvctl stop gns

srvctl status gns [-n node_name]

srvctl status gns

srvctl enable gns [-n node_name]

srvctl enable gns

srvctl disable gns [-n node_name]

srvctl disable gns -n devnode2

srvctl config gns [-a] [-d] [-k] [-m] [-n node_name] [-p] [-s] [-V] [-q name] [-l]
[-v]
srvctl config gns -n lnx03

srvctl modify gns [-i ip_address] [-d domain]

srvctl modify gns -i 192.100.010.007

srvctl relocate gns [-n node_name]

srvctl relocate gns -n node2