Understanding Oracle RAC Internals

1/20/2012
Copyright 2011, Oracle and/or its affiliates. All rights reserved.
Understanding Oracle RAC Internals

Markus Michalewicz (Markus.Michalewicz@oracle.com)
Senior Principal Product Manager Oracle RAC and Oracle RAC One Node
2
1/20/2012
Safe Harbor Statement
The following is intended to outline our general product direction. It is intended for
information purposes only, and may not be incorporated into any contract. It is not a
commitment to deliver any material, code, or functionality, and should not be relied upon
in making purchasing decisions. The development, release, and timing of any features
or functionality described for Oracles products remains at the sole discretion of Oracle.
Agenda
The Oracle RAC Architecture
VIPs and Networks
Listeners and SCAN
and Services
Client Connectivity
Node Membership
The Interconnect
Installation and Upgrade
1/20/2012

Typical hardware architecture
Network
Client
Public Lan
Node 1
Node 2
Node N-1
Node N
Interconnect
with switch
SAN switch
Shared Storage

Typical software deployment 11g Release 2
Oracle Real Application Clusters (RAC)

Oracle Grid Infrastructure
OS
OS
OS
OS
Shared Storage
1/20/2012

Typical software deployment 11g Release 2
Oracle Real Application Clusters (RAC)

Oracle Automatic Storage Management (ASM)
Oracle Clusterware
OS
OS
OS
OS
Oracle ASM managed

Shared Storage
7

Oracle RAC 11g Release 2 per node layer
Cloud File System
(aka ASM Cluster File
File System - ACFS)
DB Instance
Oracle RAC
ASM Instance

OS
OS
HA Framework
Node
Membership
OS
For all other files

11.2.0.3: For Backup Sets
Oracle Clusterware files
Voting Files (disks)
Cluster Registry (OCR)
Database Files
Redo / Archive Logs
Oracle ASM managed

Shared Storage
8
Control Files
etc.
1/20/2012

Oracle Grid Infrastructure 11g Release 2 process overview
ASM Instance

OS
OS
HA Framework
Node
Membership
My Oracle Support (MOS)
OS
Note 1053147.1 - 11gR2 Clusterware
and Grid Home - What You Need to Know
Note 1050908.1 - How to Troubleshoot
Grid Infrastructure Startup Issues
9

Local and Cluster Resources (11.2.0.2 list)
LISTENER_SCAN1
LISTENER_SCAN2
LISTENER_SCAN3
ora.oc4j
If database is deployed:
ora.SCAN1.VIP
ora.SCAN2.VIP
ora.SCAN3.VIP
ora.cvu
ora.Dbname.db
ora.node1.vip
ora.node2.vip
ora.node3.vip
ora.node4.vip
ora.asm
ora.asm
ora.asm
ora.asm
ora.dgname.dg
ora.dgname.dg
ora.dgname.dg
ora.dgname.dg
ora.listener.lsnr
ora.listener.lsnr
ora.listener.lsnr
ora.listener.lsnr
ora.ons
ora.ons
ora.ons
ora.ons
ora.gsd
ora.gsd
ora.gsd
ora.gsd
ora.net1.network
ora.net1.network
ora.net1.network
ora.net1.network
ora.registry.acfs
ora.registry.acfs
ora.registry.acfs
ora.registry.acfs
ora.Dbname.Srvcname.svc
Command to list:
crsctl stat res -t

OS
10
OS
OS
OS
1/20/2012
VIPs and Networks
11
VIPs and Networks

Oracle VIPs
SCAN and node VIPs are cluster resources
SCAN VIPs can float in the cluster
rac1
Node VIPs per default run on specific nodes

All VIPs on the public network (SCAN, node,
user or application VIPs) depend on the Oracle
network resource.
The network resource monitors
the public network interface(s).
IOCTL based checks vs. default gateway ping
Default gateway checks as backup
rac2
LISTENER_SCAN1
LISTENER_SCAN2
ora.SCAN1.VIP
ora.SCAN2.VIP
ora.rac1.vip
ora.rac2.vip
ora.listener.lsnr
ora.listener.lsnr
ora.net1.network
ora.net1.network

OS
OS
racgvip for backward compatibility only
12
1/20/2012
VIPs and Networks

Node VIP Details
1
2
3
[GRID]> crsctl status res ora.rac1.vip -p

NAME=ora.rac1.vip
TYPE=ora.cluster_vip_net1.type
...
AGENT_FILENAME=%CRS_HOME%/bin/orarootagent%CRS_EXE_SUFFIX%
...
rac1
CHECK_INTERVAL=1
CHECK_TIMEOUT=30
...
ora.rac1.vip
START_DEPENDENCIES=hard(ora.net1.network)
pullup(ora.net1.network)
ora.listener.lsnr
...
STOP_DEPENDENCIES=hard(ora.net1.network)
...
ora.net1.network
USR_ORA_VIP=rac1-vip
VERSION=11.2.0.2.0
Oracle Grid
rac2
ora.rac2.vip
ora.listener.lsnr
ora.net1.network
Infrastructure
OS
13
OS
VIPs and Networks

The network resource
The Oracle network resource monitors the
public network interface(s) (e.g. eth0).
The default is network number 1 (net1)
There is one network resource per subnet:
rac2
[GRID]> srvctl config network
Network exists:
1/192.168.0.0/255.255.255.0/eth0, type static
ora.rac1.vip
ora.rac2.vip
ora.listener.lsnr
ora.listener.lsnr
ora.net1.network
ora.net1.network
New in 11.2.0.2:
Direct management of the network resource
srvctl verb network
Network type support (static, dhcp, mixed)
14
rac1

OS
OS
1/20/2012
VIPs and Networks

How to enable multiple networks in the cluster?
1
[GRID]> srvctl add network h

Adds a network configuration to the Oracle Clusterware.
Usage: srvctl add network
-k <net_num>
network number
(default number is 1)
-S <subnet>/<netmask>/[if1[|if2...]]
NET address spec for network
-w <network_type>
The network type

(static, dhcp, mixed)
-h
Print usage
-v
Verbose output
ora.net1.network
ora.net1.network
ora.net2.network
ora.net2.network
For example:
OS
srvctl add network -k 2 -S 10.1.1.0/255.255.255.0
15
OS
VIPs and Networks

How to enable multiple networks in the cluster?
2
[GRID]> srvctl add vip -h

Adds a VIP to the Oracle Clusterware.
Usage: srvctl add vip
-n <node_name>
Node name
-A <vip_name|ip>/<netmask>/[if1[|if2...]]
VIP address spec for node applications
-k <net_num>
-h
-v
network number
(default number is 1)
ora.rac2.vip
ora.rac2-vip2.vip
Print usage
ora.net1.network
ora.net1.network
Verbose output
ora.net2.network
ora.net2.network
For example:
srvctl add vip -n rac1 -k 2 -A rac1-vip2/255.255.255.0
srvctl add vip -n rac2 -k 2 -A rac2-vip2/255.255.255.0
16
ora.rac1.vip
ora.rac1-vip2.vip

OS
OS
1/20/2012
VIPs and Networks

Application / user and node VIPs
1
[GRID]> crsctl stat res ora.rac1-vip2.vip -p

NAME=ora.rac1-vip2.vip
TYPE=ora.cluster_vip_net2.type
...
HOSTING_MEMBERS=rac1
This is a node vip!
...
START_DEPENDENCIES=hard(ora.net2.network)
pullup(ora.net2.network)
...
STOP_DEPENDENCIES=hard(ora.net2.network)
...
ora.rac1.vip
ora.rac1-vip2.vip
ora.net1.network
ora.net1.network
ora.net2.network
ora.net2.network
USR_ORA_VIP=rac1-vip2
OS
VERSION=11.2.0.2.0
17
ora.rac2.vip
ora.rac2-vip2.vip
OS
VIPs and Networks

2
[GRID]> appvipcfg create

-network=1
-ip 192.168.0.66
-vipname=myVIP
-user=root
This is an application vip!

[GRID]> appvipcfg create
-network=2
-ip 10.1.1.88
-vipname=myNewVIP
-user=root
ora.rac1.vip
ora.rac1-vip2.vip
myVIP
ora.rac2.vip
ora.rac2-vip2.vip
myNewVIP
ora.net1.network
ora.net1.network
ora.net2.network
ora.net2.network

OS
18
OS
1/20/2012
VIPs and Networks

3
[GRID]> crsctl stat res myVIP -p

NAME=myVIP
TYPE=app.appvip_net1.type
...
HOSTING_MEMBERS=rac1
...
This is an application vip!
ora.rac1.vip
ora.rac1-vip2.vip
myVIP
RESTART_ATTEMPTS=0
ora.rac2.vip
ora.rac2-vip2.vip
myNewVIP
...
USR_ORA_VIP=192.168.0.66
ora.net1.network
ora.net1.network
ora.net2.network
ora.net2.network
VERSION=11.2.0.2.0

You can modify it as needed!
19
OS
OS
Listeners and SCAN
20
10
1/20/2012
Listeners
Listeners and dependencies
In Oracle RAC 11g Release 2 there have been
three major changes regarding listeners:
1.
2.
3.
rac2
LISTENER_SCAN2
ora.SCAN1.VIP
ora.SCAN2.VIP
ora.rac1.vip
ora.rac2.vip
Listeners are managed by an Oracle agent
SCAN Listeners were introduced
ora.listener.lsnr
ora.listener.lsnr
ora.net1.network
ora.net1.network
Listeners per default now run from the

Oracle Grid Infrastructure (GRID) home
A listener depends on a VIP

A listener depends indirectly on a network
21
rac1
LISTENER_SCAN1

OS
OS
Listeners
The default LISTENER
[GRID]> srvctl config listener
Name: LISTENER
Network: 1, Owner: oracle Grid Software Owner
Home: <CRS home>
rac1
End points: TCP:1521
rac2
ora.LISTENER.lsnr
ora.LISTENER.lsnr
ora.net1.network
ora.net1.network

OS
22
OS
11
1/20/2012
Listeners
The default LISTENER FAQ
Can you define another port? - YES
rac1
Can you define another owner? YES

Can you define another home? YES, if really needed.
Can I add another listener on another port? - YES
rac2
ora.LISTENER.lsnr
ora.LISTENER.lsnr
ora.net1.network
ora.net1.network

OS
23
OS

1
[GRID]> srvctl modify listener -l LISTENER -p TCP:1545

Name: LISTENER
rac1
Network: 1, Owner: oracle
rac2
Home: <CRS home>

Check: <CRS HOME>/network/admin/endpoints_listener.ora before:
LISTENER_RAC1=(DESCRIPTION=(ADDRESS_LIST=
(ADDRESS=(PROTOCOL=TCP)(HOST=rac1-vip)
(PORT=1521))(ADDRESS=(PROTOCOL=TCP)
(HOST=192.168.0.51)(PORT=1521)(IP=FIRST))))
# line added by Agent
24
ora.LISTENER.lsnr
ora.LISTENER.lsnr
ora.net1.network
ora.net1.network

OS
OS
12
1/20/2012

2
[GRID]> srvctl modify listener -l LISTENER -p TCP:1545

Name: LISTENER
rac1
rac2
Home: <CRS home>

Check: <CRS HOME>/network/admin/endpoints_listener.ora after restart:
LISTENER_RAC1=(DESCRIPTION=(ADDRESS_LIST=
(ADDRESS=(PROTOCOL=TCP)(HOST=rac1-vip)
(PORT=1545))(ADDRESS=(PROTOCOL=TCP)
(HOST=192.168.0.51)(PORT=1545)(IP=FIRST))))
25
ora.LISTENER.lsnr
ora.LISTENER.lsnr
ora.net1.network
ora.net1.network

OS
OS

Can you define another owner? - YES
[GRID]>
srvctl modify listener -l LISTENER -u root
[GRID]>
srvctl config listener
You need to be root

Name: LISTENER
rac1
Network: 1, Owner: root For demonstration

purposes only
rac2
Home: <CRS home>

ora.LISTENER.lsnr
ora.LISTENER.lsnr
ora.net1.network
ora.net1.network

OS
26
OS
13
1/20/2012

Can you define another home? - YES
[GRID]> srvctl modify listener -l LISTENER
-o /u01/app/oracle/product/11.2.0/dbhome_1

Name: LISTENER
rac1
rac2

Home: /u01/app/oracle/product/11.2.0/dbhome_1
This may not be required.
ora.LISTENER.lsnr
ora.LISTENER.lsnr
ora.net1.network
ora.net1.network
Instead, create a new listener from a separate home, if needed

and use the SRVCTL command as shown, but try using the GI home.
27
OS
OS
Listeners
Can I add another listener on another port? - YES
Name: LISTENER
Home: <CRS home>
rac1
rac2
Name: LISTENER2011
Home: <CRS home>
MOS Note 220970.1 - RAC: Frequently Asked Questions
How to use SCAN and node listeners with different ports?
28
ora.LISTENER.lsnr
ora.LISTENER.lsnr
ora.LISTENERK2.lsnr
ora.LISTENERK2.lsnr
ora.net1.network
ora.net1.network

OS
OS
Add additional node listeners to the system using different ports
14
1/20/2012
Listeners
Can I add another listener on another network? - YES
[GRID]> srvctl add listener -l ListenerK2 -p 1544 -k 2
Name: LISTENER
rac1
Home: <CRS home>

Name: LISTENERK2
ora.LISTENER.lsnr
ora.LISTENER.lsnr
Ora.LISTENERK2.lsnr
ora.LISTENERK2.lsnr
ora.net1.network
ora.net1.network
ora.net2.network
ora.net2.network
Home: <CRS home>

MOS Note 1063571.1 - How to Configure A Second

Listener on a Separate Network in 11.2 Grid Infrastructure
29
rac2

OS
OS
Listeners
Remember its just another listener
1

Name: JUSTALISTENER
Network: 1, Owner: root
Home: <CRS home>
rac2
LISTENER_SCAN1
LISTENER_SCAN2
...
ora.SCAN1.VIP
ora.SCAN2.VIP
ora.rac1.vip
ora.rac2.vip
ora.LISTENER.lsnr
ora.LISTENER.lsnr
ora.net1.network
ora.net1.network
[GRID]> vi /u01/app/11.2.0/grid/network/admin/listener.ora
JUSTALISTENER=(DESCRIPTION=(ADDRESS_LIST=
(ADDRESS=(PROTOCOL=IPC)(KEY=JUSTALISTENER))))
[GRID]> vi /u01/app/11.2.0/grid/network/admin/endpoints_listener.ora
JUSTALISTENER_RAC1=(DESCRIPTION=(ADDRESS_LIST=
(ADDRESS=(PROTOCOL=TCP)(HOST=rac1-vip)(PORT=1522))
(ADDRESS=(PROTOCOL=TCP)(HOST=192.168.0.51)(PORT=1522)(IP=FIRST))))
30
rac1

OS
OS
15
1/20/2012
Listeners
Remember its just another listener
2
(ADDRESS=(PROTOCOL=IPC)(RATE_LIMIT=10) For demonstration purposes only
(KEY=JUSTALISTENER)))) # line added by Agent
...
rac1
[GRID]> srvctl stop listener -l JustAListener
LISTENER_SCAN2
ora.SCAN1.VIP
ora.SCAN2.VIP
ora.rac1.vip
ora.rac2.vip
...
ora.LISTENER.lsnr
ora.LISTENER.lsnr
ora.net1.network
ora.net1.network
[GRID]> srvctl start listener -l JustAListener

(ADDRESS=(PROTOCOL=IPC)(RATE_LIMIT=10)
(KEY=JUSTALISTENER)))) # line added by Agent
31
rac2
LISTENER_SCAN1

OS
OS
SCAN
The basics and overview
In Oracle RAC 11g Release 2
SCAN Listeners were introduced
SCAN
There is one SCAN setup per cluster
Read about SCAN:
http://www.oracle.com/technetwork/database/
clustering/overview/scan-129069.pdf
MOS Note 887522.1
11gR2 Grid Infrastructure Single Client
Access Name (SCAN) Explained
SCAN consists of a bundle containing:
32
1.
SCAN IP
2.
Associated (dependency) SCAN Listener
rac2
rac1
rac3
LISTENER_SCAN1
LISTENER_SCAN2
LISTENER_SCAN3
ora.SCAN1.VIP
ora.SCAN2.VIP
ora.SCAN3.VIP
ora.LISTENER.lsnr
ora.LISTENER.lsnr
ora.LISTENER.lsnr
ora.net1.network
ora.net1.network
ora.net1.network

OS
OS
OS
16
1/20/2012
SCAN
The SCAN bundle
1 3 SCANs are the default for HA and LB
SCAN
Regardless of number of nodes

You can define less or more, if really needed.
SCAN listener resources run with
an active dispersion dependency
If you use more nodes in the cluster than
SCAN listeners are defined, no node should
run more than one SCAN bundle at a time
If you use less nodes in the cluster than
SCAN listeners are defined, there will be
nodes running more than one SCAN bundle
at a time.
SCAN VIP moves with the listener, if possible.
33
rac2
rac1
rac3
LISTENER_SCAN1
LISTENER_SCAN2
LISTENER_SCAN3
ora.SCAN1.VIP
ora.SCAN2.VIP
ora.SCAN3.VIP
ora.LISTENER.lsnr
ora.LISTENER.lsnr
ora.LISTENER.lsnr
ora.net1.network
ora.net1.network
ora.net1.network

OS
OS
OS
SCAN
The SCAN bundle
2
[GRID]> srvctl config scan

SCAN name: cluster1, Network: 1/192.168.0.0/255.255.255.0/eth0
SCAN VIP name: scan1, IP: /cluster1.us.oracle.com/192.168.0.41
rac1
rac2
[GRID]> srvctl modify scan
LISTENER_SCAN1
Modifies the SCAN name.
ora.SCAN1.VIP
ora.LISTENER.lsnr
ora.LISTENER.lsnr
ora.net1.network
ora.net1.network
Usage: srvctl modify scan

-n <scan_name>
Domain name qualified SCAN name
-h
Print usage
http://www.oracle.com/technetwork/database/
clustering/overview/scan-129069.pdf page 2
34
SCAN

OS
OS
17
1/20/2012
SCAN
The SCAN bundle
3
[GRID]> srvctl add scan -h

Adds a SCAN VIP to the Oracle Clusterware.
Usage: srvctl add scan -n <scan_name>
-n <scan_name>
Domain name qualified SCAN name
-k <net_num>
network number (default number is 1)
SCAN
rac1
rac2
LISTENER_SCAN1
-S <subnet>/<netmask>/[if1[|if2...]]
NET address spec for network
ora.SCAN1.VIP
-h
ora.LISTENER.lsnr
ora.LISTENER.lsnr
ora.net1.network
ora.net1.network
Print usage
Note: SCAN can only operate on one network / in one subnet.

OS
35
OS
SCAN
The SCAN FAQ
1 From MOS Note 220970.1
RAC: Frequently Asked Questions
SCAN
With Oracle RAC 11g Release 2 using SCAN is the default.

As with other listeners, there is no direct communication
between the node (listeners) and the SCAN listeners.
Listeners are only aware of the instances and services served,
since the instances (PMON) register themselves and the services
they host with the listeners.
The instances use the LOCAL and REMOTE Listener
parameters to know with which listeners to register.
Listeners used for a client connection to Oracle RAC should be
managed by Oracle Clusterware and should be listening on an
Oracle managed VIP.
36
rac1
rac2
LISTENER_SCAN1
ora.SCAN1.VIP
ora.LISTENER.lsnr
ora.LISTENER.lsnr
ora.net1.network
ora.net1.network

OS
OS
18
1/20/2012
SCAN
The SCAN FAQ
2 Can you define another port? - YES
See MOS Note 220970.1
Use srvctl modify scan_listener p <newPort>
Can you define another owner? NO

Can you define another home? NO
Can you have more than one node listener with SCAN? YES
Can the SCAN and the node listener ports differ? - YES
See also MOS Note 972500.1 How to
Modify SCAN Setting or SCAN Listener Port after Installation
37
SCAN
rac1
rac2
LISTENER_SCAN1
ora.SCAN1.VIP
ora.LISTENER.lsnr
ora.LISTENER.lsnr
ora.net1.network
ora.net1.network

OS
OS
Listeners and Instance Registration

Listener changes may require instance changes
See MOS Note 220970.1
Whenever you change an instance parameter configuration

a change to the LOCAL_LISTENER or REMOTE_LISTENER
per database (instance) may be required.
Note that using server-side TNSANMES aliases to
resolve the listener configuration is not recommended.
The use of the TNSNAMES connector string is the default:
local_listener = (DESCRIPTION= (ADDRESS_LIST=
(ADDRESS=(PROTOCOL=TCP)(HOST=192.168.0.61)(PORT=2011))
))'
38
SCAN
rac1
rac2
DB Instance
DB Instance
LISTENER_SCAN1
ora.SCAN1.VIP
ora.LISTENER.lsnr
ora.LISTENER.lsnr

OS
OS
19
1/20/2012

EXAMPLE
1 Taken from MOS Note 220970.1 RAC: FAQ
srvctl add listener -l LISTENER2011 -p 2011
srvctl start listener -l LISTENER2011
srvctl status listener -l LISTENER2011
lsnrctl status LISTENER2011
SCAN
rac1
rac2
DB Instance
DB Instance
LISTENER_SCAN1
ora.SCAN1.VIP
ora.LISTENER.lsnr
ora.LISTENER.lsnr
Name: LISTENER2011
Home: <CRS home>
OS
OS

39

EXAMPLE
SCAN
rac1
rac2
New node listener does not serve any database instance

You must alter the LOCAL_LISTENER parameters
for each instance accordingly:
DB Instance
DB Instance
LISTENER_SCAN1
ora.SCAN1.VIP
alter system set local_listener =

(DESCRIPTION= (ADDRESS_LIST=
ora.LISTENER.lsnr
ora.LISTENER.lsnr
(ADDRESS=(PROTOCOL=TCP)(HOST=192.168.0.61)(PORT=2011))))
' scope=BOTH SID='OCRL1';

alter system register;
OS
40
OS
20
1/20/2012

EXAMPLE
SCAN
rac1
rac2
Note: Use spaces between the various segments of the command

Reason: the database agent in Oracle Clusterware currently
determines whether the local_listener or remote_listener have
been manually set by a string comparison operation.
If the string looks unaltered it will reset these parameters
with the default values that it determines on instance startup.
Slight modifications are therefore required. ER 11772838
has been filed to allow for a more convenient mechanism.
41
DB Instance
DB Instance
LISTENER_SCAN1
ora.SCAN1.VIP
ora.LISTENER.lsnr
ora.LISTENER.lsnr

OS
OS

Using listeners in different subnets / networks
SCAN can only operate in one network / subnet
If listeners in different subnets are used,
use LISTENER_NETWORKS:
http://download.oracle.com/docs/cd/E11882_01
/server.112/e17110/initparams115.htm#REFRN10313
Note: Listeners specified by the LISTENER_NETWORKS

parameter should not be used in the LOCAL_LISTENER and
REMOTE_LISTENER parameters. Otherwise, cross registration
will happen and connections will be redirected cross networks.
For more information, see also

MOS Note 1063571.1 How to Configure A Second
Listener on a Separate Network in 11.2 Grid Infrastructure
42
rac1
rac2
DB Instance
DB Instance
LISTENER_SCAN1
ora.SCAN1.VIP
ora.LISTENER.lsnr
ora.LISTENER.lsnr
Ora.LISTENERK2.lsnr
ora.LISTENERK2.lsnr
ora.net1.network
ora.net1.network
ora.net2.network
ora.net2.network

OS
OS
21
1/20/2012
Listeners, SCAN and Services
43
Listeners and Service Registration

A summary
Services depend on a database
Per default, the database instance will register
all services it hosts with all listeners it registers with.
SCAN
rac1
Services operate within the following boundaries:

Network (indirectly via dependency)
Server Pools (in policy-managed database environments)
Currently there is no support for service failover
Between Server Pools

Between networks
44
rac2
DB Instance
DB Instance
ora.Dbname.db
ora.Dbname.db
ora.LISTENER.lsnr
ora.LISTENER.lsnr

OS
OS
22
1/20/2012
Services and Dependencies

Dependency on DB and Server Pool
1
[GRID]> srvctl config service -d ORCL -s MyService

Service name: MyService
Service is enabled
SCAN
Server pool: ORCL

rac1
Cardinality: 1
Disconnect: false
Service role: PRIMARY
Management policy: AUTOMATIC
...
Preferred instances: ORCL_1
Available instances:
rac2
DB Instance
DB Instance
ora.ORCL.db
ora.ORCL.db
ora.ORCL.MyService.svc
ora.LISTENER.lsnr
ora.LISTENER.lsnr

OS
45
OS
Services and Dependencies

Start / Stop dependencies
2
[GRID]> crsctl stat res ora.orcl.myservice.svc -p

...
DEFAULT_TEMPLATE=PROPERTY(RESOURCE_CLASS=service)
PROPERTY(SERVICE_NAME=%GEN_SERVICE_NAME%)
PROPERTY(DB_UNIQUE_NAME=CONCAT(PARSE(%NAME%, .,
2), STAT(ora.orcl.db, USR_ORA_DOMAIN), .))
ELEMENT(INSTANCE_NAME=STAT(ora.orcl.db,
GEN_USR_ORA_INST_NAME))
...
START_DEPENDENCIES=hard(ora.orcl.db,type:ora.cluster
_vip_net1.type) weak(type:ora.listener.type)
pullup(type:ora.cluster_vip_net1.type)
pullup:always(ora.orcl.db)
dispersion(type:ora.service.type)
STOP_DEPENDENCIES=hard(intermediate:ora.orcl.db,inte
rmediate:type:ora.cluster_vip_net1.type)
SCAN
rac1
rac2
DB Instance
DB Instance
ora.ORCL.db
ora.Dbname.db
ora.ORCL.MyService.svc
ora.LISTENER.lsnr
ora.LISTENER.lsnr

OS
OS
...
46
23
1/20/2012
Client Connectivity
47
Client Connectivity
Direct or indirect connect
Connect Time Load Balancing (CTLB)
Connect Time Connection Failover (CTCF)
Runtime Connection Load Balancing (RTLB)
Runtime Connection Failover (RTCF)
BATCH
Production
SCAN
Email
48
Connection
Pool
24
1/20/2012
Client Connectivity
Connect Time Connection Failover
jdbc:oracle:thin:@MySCAN:1521/Email
PMRAC =
(DESCRIPTION =
(FAILOVER=ON)
(ADDRESS = (PROTOCOL = TCP)(HOST = MySCAN)(PORT = 1521))
(CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = Email)))
Email
49
MySCAN
BATCH
Production
Connection
Pool
Client Connectivity
Runtime Time Connection Failover
PMRAC =
(DESCRIPTION =
(FAILOVER=ON)
(CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = Email)
...))
Email
50
MySCAN
BATCH
Production
Connection
Pool
25
1/20/2012
Client Connectivity
Runtime Time Connection Failover
PMRAC =
(DESCRIPTION =
(FAILOVER=ON)
(CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = Email)
(FAILOVER_MODE= (TYPE=select)(METHOD=basic)(RETRIES=180)(DELAY=5))))
BATCH
Production
51
MySCAN
Email
Connection
Pool
Client Connectivity
More information
If problems occur, see:
Note 975457.1 How to Troubleshoot Connectivity Issues with 11gR2 SCAN Name
For more advanced configurations, see:
Note 1306927.1 Using the TNS_ADMIN variable and changing the default port
number of all Listeners in an 11.2 RAC for an 11.2, 11.1, and 10.2 Database
Email
52
??
?
MySCAN
BATCH
Production
Connection
Pool
26
1/20/2012
Client Connectivity
Two ways to protect the client
1.
Transparent Application Failover (TAF)
2.
Fast Application Notification (FAN)
Tries to make the client unaware of a failure

Provides means of CTCF and RTCF
Allows for pure selects (reads) to continue
Write transactions need to be re-issued
The Application needs to be TAF aware
FAN wants to inform clients ASAP

Client can react to failure asap
Expects clients to re-connect on failure (FCF)
Sends messages about changes in the cluster
Email
53
MySCAN
BATCH
Production
Connection
Pool
Client Connectivity and Service Definition

Define settings on the server
HA (and LB) settings
[GRID]> srvctl config service

-d ORCL -s MyService
can be defined per service
Service name: MyService
Clients connecting to the service will
...
adhere to the settings considering the

client used.
DTP transaction: false

AQ HA notifications: false
Failover type: NONE
BATCH
Production
MySCAN
Failover method: NONE

Email
TAF failover retries: 0

TAF failover delay: 0
Connection Load Balancing Goal: LONG
Runtime Load Balancing Goal: NONE
TAF policy specification: BASIC
54
27
1/20/2012
Client Connectivity
Use a FAN aware connection pool
1
If a connection pool is used
The clients (users) get a physical

connection to the connection pool
The connection pool creates a physical
connection to the database
It is a direct client to the database
Internally the pool maintains logical connections
BATCH
Production
Connection Pool
MySCAN
Email
55
Client Connectivity
Use a FAN aware connection pool
2
The connection pool
Invalidates connections to one instance

Re-establishes new logical connections
May create new physical connections

Prevent new clients to be misrouted
The application needs to handle the

transaction failure that might have occurred.
BATCH
Production
Connection Pool
MySCAN
Email
56
28
1/20/2012
Client Connectivity
The Load Balancing (LB) cases
On the Client Side
On the Server Side
Email
57
MySCAN
BATCH
Production
Connection
Pool
Client Connectivity
Connect Time Load Balancing (CTLB) on the client side
PMRAC =
(DESCRIPTION =
(FAILOVER=ON)(LOAD_BALANCE=ON)
(CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = Email)))
Email
58
MySCAN
BATCH
Production
Connection
Pool
29
1/20/2012
Client Connectivity
Connect Time Load Balancing (CTLB) on the server side
Traditionally, PMON dynamically registers the services to the specified listeners with:
Service names for each running instance of the database and instance names for the DB
The listener is updated with the load information for every instance and node as follows:
1-Minute OS Node Load Average all 30 secs.
Number of Connections to Each Instance
Number of Connections to Each Dispatcher

BATCH
Production
MySCAN
Email
Connection
Pool
?
59
Client Connectivity
Use FAN for the Load Balancing cases
Connect Time Connection Failover (CTCF)
Runtime Connection Failover (RTCF)
30% connections
RAC
Database
Im busy
Instance1
10% connections
Im very busy
Im idle
60% connections
60
Instance2
Instance3
30
1/20/2012
Client Connectivity
Use FAN for the Load Balancing cases
Also via AQ (Advanced Queuing) based notifications
Background is always the Load Balancing Advisory
RAC
Database
30% connections
Im busy
Instance1
MySCAN
10% connections
For more information, see:

Oracle Real Application
Clusters Administration and
Deployment Guide 11g Release 2:
5 Introduction to Automatic Workload Management
61
Im very busy
Im idle
Instance2
60% connections
Instance3
Node Membership
62
31
1/20/2012
Oracle Clusterware Architecture

Node Membership Processes and Basics
Public Lan
Public Lan
Private Lan /
Interconnect
Main processes involved:

CSSD (ora.cssd)
Oracle Clusterware
CSSDMONITOR
was: oprocd
now: ora.cssdmonitor
CSSD
CSSD
SAN
Network
63
CSSD
Voting
Disk
SAN
Network

What does CSSD do?
Monitors nodes using 2 communication channels:
Private Interconnect Network Heartbeat
Voting Disk based communication Disk Heartbeat
Evicts (forcibly removes nodes from a
cluster) nodes dependent on heartbeat

feedback (failures)
Oracle Clusterware
CSSD
Ping
CSSD
Ping
64
32
1/20/2012

Interconnect basics network heartbeat
Each node in the cluster is pinged every second
Nodes must respond in css_misscount time (defaults to 30 secs.)
Reducing the css_misscount time is generally not supported
Network heartbeat failures
Ping
will lead to node evictions
CSSD-log:
[date / time] [CSSD][1111902528]
clssnmPollingThread: node mynodename
(5) at 75% heartbeat fatal, removal
in 6.770 seconds
65
CSSD
CSSD

Voting Disk basics disk heartbeat
1 Each node in the cluster pings (r/w) the Voting Disk(s) every second
Nodes must receive a response in (long / short) diskTimeout time
IF I/O errors indicate clear accessibility problems timeout is irrelevant
Disk heartbeat failures
will lead to node evictions
CSSD-log:
[CSSD] [1115699552] >TRACE:
clssnmReadDskHeartbeat:
node(2) is down. rcfg(1) wrtcnt(1)
LATS(63436584) Disk lastSeqNo(1)
CSSD
CSSD
Ping
66
33
1/20/2012

Voting Disk basics Structure
2 Voting Disks contain dynamic and static data:
Dynamic data: disk heartbeat logging
Static data: information about the nodes in the cluster
With 11.2.0.1 Voting Disks got an identity:
E.g. Voting Disk serial number: [GRID]> crsctl query css votedisk
1.
2 1212f9d6e85c4ff7bf80cc9e3f533cc1 (/dev/sdd5) [DATA]
Node information
Disk Heartbeat Logging
Voting Disks must therefore not be copied using dd or cp anymore

67

Voting Disk basics Simple Majority rule
3 Oracle supports redundant Voting Disks for disk failure protection
Simple Majority Rule applies:
Each node must see the simple majority of configured Voting Disks
at all times in order not to be evicted (to remain in the cluster)

trunc(n/2+1) with n=number of voting disks
configured and n>=1

CSSD
CSSD
Ping
68
34
1/20/2012

Simple Majority rule in extended clusters
http://www.oracle.com/goto/rac
Using standard NFS to support
a third voting file for extended

cluster configurations (PDF)
CSSD
CSSD
Same principles apply

Voting Disks are just
geographically dispersed
69

Voting Disks in Oracle ASM does not change their usage
[GRID]> crsctl query css votedisk
1.
2 1212f9d6e85c4ff7bf80cc9e3f533cc1 (/dev/sdd5) [DATA]
2.
2 aafab95f9ef84f03bf6e26adc2a3b0e8 (/dev/sde5) [DATA]
3.
2 28dd4128f4a74f73bf8653dabd88c737 (/dev/sdd6) [DATA]
Located 3 voting disk(s).
Oracle ASM auto creates 1/3/5 Voting Files

Voting Disks reside in one diskgroup only
Based on Ext/Normal/High redundancy
and on Failure Groups in the Disk Group
Per default there is one failure group per disk
ASM will enforce the required number of disks
New failure group type: Quorum Failgroup
70
35
1/20/2012

Oracle Cluster Registry (OCR) placement in Oracle ASM
The OCR is managed like a datafile in ASM (new type)
71
It adheres completely to the redundancy settings for the diskgroup (DG)

There can be more than one OCR location in more than one DG (DG:OCR 1:1)
Recommendation is 2 OCR locations, 1 in DATA, 1 in FRA for example

Backup of Clusteware files is fully automatic (11.2+)
Clusterware Files (managed in ASM) enables fully Automatic Backups:
The Voting Disks are backed up into the OCR
Any configuration change in the cluster (e.g. node
addition) triggers a new backup of the Voting Files.
A single, failed Voting Disks is restored by ASM
automatically within a Disk Group no action required
Note: Do not use DD to back up the Voting Disks anymore!
The OCR is backed up automatically every 4 hours
Manual Backups can be taken as required
ONLY IF all Voting Disks are corrupted or failed
AND (all copies of) the OCR are also corrupted or unavailable
THEN manual interference would be required the rest is automatic.
72
36
1/20/2012
Fencing Basics
Why are nodes evicted?
Evicting (fencing) nodes is a preventive measure (its a good thing)!
Nodes are evicted to prevent consequences of a split brain:
Shared data must not be written by independently operating nodes
The easiest way to prevent this is to forcibly remove a node from the cluster
CSSD
73
CSSD
Fencing Basics
How are nodes evicted? STONITH
Once it is determined that a node needs to be evicted,
A kill request is sent to the respective node(s)
Using all (remaining) communication channels
A node (CSSD) is requested to kill itself STONITH like
STONITH foresees that a remote node kills the node to be evicted
CSSD
74
CSSD
37
1/20/2012
Fencing Basics
EXAMPLE: Network heartbeat failure
The network heartbeat between nodes has failed
It is determined which nodes can still talk to each other
A kill request is sent to the node(s) to be evicted
Using all (remaining) communication channels Voting Disk(s)

A node is requested to kill itself; executer: typically CSSD
CSSD
CSSD
2
75
Fencing Basics
What happens, if CSSD is stuck?
A node is requested to kill itself
BUT CSSD is stuck or sick (does not execute) e.g.:
CSSD failed for some reason
CSSD is not scheduled within a certain margin
See also: MOS note

1050693.1 Troubleshooting 11.2
Clusterware Node
Evictions (Reboots)
OCSSDMONITOR (was: oprocd) will take over and execute
CSSD
2
CSSDmonitor
CSSD
CSSD
76
38
1/20/2012
Fencing Basics
How can nodes be evicted?
Oracle Clusterware 11.2.0.1 and later supports IPMI (optional)
Intelligent Platform Management Interface (IPMI) drivers required
IPMI allows remote-shutdown of nodes using additional hardware
A Baseboard Management Controller (BMC) per cluster node is required
CSSD
77
CSSD
Fencing Basics
EXAMPLE: IPMI based eviction on heartbeat failure
The network heartbeat between the nodes has failed
It is determined which nodes can still talk to each other
IPMI is used to remotely shutdown the node to be evicted
CSSD
78
39
1/20/2012
Fencing Basics
Which node gets evicted?
Voting Disks and heartbeat communication is used to determine the node
In a 2 node cluster, the node with the lowest node number should survive
In a n-node cluster, the biggest sub-cluster should survive (votes based)
CSSD
79
CSSD
Fencing Basics
Cluster members can escalate a kill request
Cluster members (e.g Oracle RAC instances) can request
Oracle Clusterware to kill a specific member of the cluster

Oracle Clusterware will then attempt to kill the requested member
Oracle RAC
DB Inst. 1
Oracle RAC
DB Inst. 2
Oracle Clusterware
Inst. 1:
kill inst. 2
80
40
1/20/2012
Fencing Basics
If the requested member kill is unsuccessful, a node eviction escalation can be issued,
which leads to the eviction of the node, on which the particular member currently resides
Oracle RAC
DB Inst. 1
Oracle RAC
DB Inst. 2
Oracle Clusterware
Inst. 1:
kill inst. 2
81
Fencing Basics
Oracle RAC
DB Inst. 1
Oracle RAC
DB Inst. 2
Oracle Clusterware
Inst. 1:
kill inst. 2
82
41
1/20/2012
Fencing Basics
Oracle RAC
DB Inst. 1
Oracle Clusterware
83
Re-Bootless Node Fencing

With 11.2.0.2 onwards, fencing may not mean re-boot
Until Oracle Clusterware 11.2.0.2, fencing meant re-boot
With Oracle Clusterware 11.2.0.2, re-boots will be seen less, because:
Re-boots affect applications that might run an a node, but are not protected
Customer requirement: prevent a reboot, just stop the cluster implemented...
App X
RAC DB
Inst. 1
CSSD
84
App Y
RAC DB
Inst. 2
Oracle Clusterware
CSSD
42
1/20/2012

How it works
With Oracle Clusterware 11.2.0.2, re-boots will be seen less:
Instead of fast re-booting the node, a graceful shutdown of the stack is attempted
It starts with a failure e.g. network heartbeat or interconnect failure
App X
RAC DB
Inst. 2
RAC DB
Inst. 1
CSSD
85
App Y
Oracle Clusterware
CSSD

How it works
Then IO issuing processes are killed; it is made sure that no IO process remains
For a RAC DB mainly the log writer and the database writer are of concern
App X
App Y
RAC DB
Inst. 1
CSSD
86
Oracle Clusterware
CSSD
43
1/20/2012

How it works
Once all IO issuing processes are killed, remaining processes are stopped
IF the check for a successful kill of the IO processes, fails reboot
App X
App Y
RAC DB
Inst. 1
CSSD
87
Oracle Clusterware
CSSD

How it works
Once all remaining processes are stopped, the stack stops itself with a restart flag
App X
App Y
RAC DB
Inst. 1
Oracle
Clusterware
CSSD
88
OHASD
44
1/20/2012

How it works
OHASD will finally attempt to restart the stack after the graceful shutdown
App X
App Y
RAC DB
Inst. 1
OHASD
Oracle
Clusterware
CSSD
89

EXCEPTIONS
With Oracle Clusterware 11.2.0.2, re-boots will be seen less, unless:
IF the check for a successful kill of the IO processes fails reboot

IF CSSD gets killed during the operation reboot
IF cssdmonitor (oprocd replacement) is not scheduled reboot
IF the stack cannot be shutdown in short_disk_timeout-seconds reboot
App X
RAC DB
Inst. 1
CSSD
90
App Y
RAC DB
Inst. 2
Oracle Clusterware
CSSD
45
1/20/2012
The Interconnect
91
The Interconnect
Heartbeat and memory channel between instances
Network
Public Lan
Node 1
Node 2
Node N-1
Node N
Client
Interconnect
with switch
SAN switch
92
46
1/20/2012
The Interconnect
Redundant Interconnect Usage
1 Redundant Interconnect Usage can be used as a bonding alternative
It works for private networks only; the nodeVIPs use a different approach
It enables HA and Load Balancing for up to 4 NICs per server (on Linux / Unix)
It can be used by Oracle Databases 11.2.0.2 and Oracle Clusterware 11.2.0.2
It uses so called HAIPs that are assigned to the private networks on the server
The HAIPs will be used by the database and ASM instances and processes
Node 1
93
Node 2
HAIP1
HAIP3
HAIP2
HAIP4
The Interconnect
Redundant Interconnect Usage
2 A multiple listening endpoint approach is used
The HAIPs are taken from the link-local (Linux / Unix) IP range (169.254.0.0)
To find the communication partners, multicasting on the interconnect is required
With 11.2.0.3 Broadcast is a fallback alternative (BUG 10411721)
Multicasting is still required on the public lan for MDNS for example.
Details in My Oracle Support (MOS) Note with Doc ID 1212703.1:
11.2.0.2 Grid Infrastructure Install or Upgrade may fail due to Multicasting
Node 1
94
Node 2
HAIP1
HAIP3
HAIP2
HAIP4
47
1/20/2012
The Interconnect
Redundant Interconnect Usage and the HAIPs
If a network interface fails, the assigned HAIP is failed over to a remaining one.
Redundant Interconnect Usage allows having networks in different subnet
You can either have one subnet for all networks or a different one for each
You can also use VLANs with the interconnect. For more information see:
Note 1210883.1 - 11gR2 Grid Infrastructure Redundant Interconnect and ora.cluster_interconnect.haip
Note 220970.1 - RAC: Frequently Asked Questions - How to use VLANs in Oracle RAC? AND
Are there any issues for the interconnect when sharing the same switch as the public network by using VLAN to separate the network?
Node 1
HAIP1
95
Node 2
HAIP1
HAIP3
HAIP2
HAIP4
HAIP3
Installation and Upgrade
96
48
1/20/2012
Installation
Patch Sets are also more than BUG fixe with 11.2.0.x
Upgrade + New Features
Upgrade + Full Installation
And (always) Out-of-place
Patch uses new Oracle Home
First patch set: 11.2.0.2

11.2.0.3 works the same
Less new features
97
Installation
Installation is meant to be simpler with 11g Release 2
Oracle Universal Installer (OUI)
is the main installation instrument.
Installation tips and tricks in:
MOS note 810394.1 (generic)
See:
RAC Platform Specific Starter Kits and Best Practices
Step by Step for various versions available.
See also:
RAC Platform Generic Load Testing and System Test Plan Outline
For an Oracle VM based deployment see:
98
Oracle RAC OVM Templates on OTN
Details in MOS note 1185244.1 Pre-Built Grid

Infrastructure and Oracle RAC Templates For Oracle VM
49
1/20/2012
Upgrade and Update

Upgrade to 11g Release 2 (11.2.0.3) recommended
The Upgrade to 11g Release 2 from
previous versions is generally recommended
Things to consider:
MOS note 1312225.1 - Things to Consider

Before Upgrading to 11.2.0.2 Grid Infrastructure
MOS note 1320966.1 - Things to Consider

Before Upgrade to 11.2.0.2 Database Performance
MOS note 1363369.1

Things to Consider Before Upgrading to 11.2.0.3
Grid Infrastructure/ASM
99
10
0
50

Understanding Oracle RAC Internals

Caricato da

Informazioni sul documento

Copyright

Formati disponibili

Condividi questo documento

Condividi o incorpora il documento

Opzioni di condivisione

Hai trovato utile questo documento?

Questo contenuto è inappropriato?

Copyright:

Formati disponibili

Understanding Oracle RAC Internals

Caricato da

Copyright:

Formati disponibili

1/20/2012

Copyright 2011, Oracle and/or its affiliates. All rights reserved.