Sei sulla pagina 1di 88

Under the Hood of Oracle Clusterware 2.

0: Grid Infrastructure
Alex Gorbachev 29 August, 2012

Agenda
Place of Grid Infrastructure in Oracle RAC Node membership and evictions Clusterware architecture & startup sequence Resources Management and troubleshooting

2009/2010 Pythian

Agenda
th Th
High

Need to memorize

le

ss

or yo e yo u u ne u ed nd to ers m tan em d, or ize

Low Shallow

Understanding
2009/2010 Pythian

In-depth

Single Instance Oracle Database


APP

Query/DML/DDL

SERVER

INSTANCE Memory (SGA, PGA) Processes (PMON, SMON, LGWR and etc. + multiple shadow processes)

Read/write
Datales Controlles redo logs ashback logs, change tracking and etc...

Database

2012 Pythian

Single Instance Oracle Database


APP

SERVER

INSTANCE

Database

2012 Pythian

Oracle RAC Database


APP

SERVER 1

INSTANCE 1

Database

2012 Pythian

Oracle RAC Database


APP

SERVER 1

SERVER 2

INSTANCE 1

INSTANCE 2

Database

2012 Pythian

Oracle RAC Database


APP

SERVER 1

SERVER 2

SERVER 3

INSTANCE 1

INSTANCE 2

INSTANCE 3

Database

2012 Pythian

Oracle RAC Database


APP

SERVER 1

SERVER 2

SERVER 3

INSTANCE 1

INSTANCE 2

INSTANCE 3

Database

2012 Pythian

Oracle RAC Database


APP

SERVER 1

SERVER 2

SERVER 3

INSTANCE 1

INSTANCE 2

INSTANCE 3

Database

2012 Pythian

RAC looks simple. Eh?

2012 Pythian

Role of Grid Infrastructure


OS OS OS

VIP Listener
Service

VIP Listener
Service

VIP Listener
Service

Instance ASM Grid Infrastr. ASM

Instance ASM

Instance

Grid Infrastr.

Grid Infrastr.

interconnect storage access

OCR Shared storage

Voting disk

2009/2010 Pythian

Role of Grid Infrastructure


OS OS OS

VIP Listener
Service

VIP Listener
Service

VIP Listener
Service

Instance ASM Grid Infrastr. ASM

Instance ASM

Instance

Grid Infrastr.

Grid Infrastr.

interconnect storage access

OCR Shared storage

Voting disk

2009/2010 Pythian

OS

OS

Clusterware

Clusterware

CSSD
OPROCD interconnect

CSSD
OPROCD

2009/2010 Pythian

OS

OS

Clusterware

Clusterware

CSSD
OPROCD interconnect

CSSD
OPROCD

2009/2010 Pythian

OS

OS

Clusterware

Clusterware

CSSD
OPROCD interconnect

CSSD
OPROCD

2009/2010 Pythian

OS Clusterware

OS Clusterware

CSSD
OPROCD interconnect

CSSD
OPROCD

Voting disk

2009/2010 Pythian

OS Clusterware

OS Clusterware

CSSD
OPROCD interconnect

CSSD
OPROCD

Voting disk

2009/2010 Pythian

OS

OS Clusterware

Shoot The Other Node In The Head


Clusterware
9

CSSD
interconnect

CSSD
OPROCD

OPROCD

Voting disk

2009/2010 Pythian

OS Clusterware

OS Clusterware

VIP

RACG EVMD CRSD

CSSD
OPROCD interconnect

CSSD
OPROCD

Voting disk

10

2009/2010 Pythian

OS Clusterware

OS Clusterware

CSSD
OPROCD interconnect

CSSD
OPROCD

Voting disk

11

2009/2010 Pythian

OS Clusterware

OS Clusterware

CSSD
OPROCD interconnect

CSSD
OPROCD

Voting disk

11

2009/2010 Pythian

OS Clusterware

OS Clusterware

CSSD
OPROCD interconnect

CSSD
OPROCD

Voting disk

11

2009/2010 Pythian

OS Clusterware

CSSD
interconnect

CSSD
OPROCD

Voting disk

11

2009/2010 Pythian

OS Clusterware

Ask The Other Node To Reboot Itself


11

CSSD
interconnect

CSSD
OPROCD

Voting disk

(c) known quote


2009/2010 Pythian

11gR2 Grid Infrastructure: CSSD attempts graceful shutdown

12

2009/2010 Pythian

OS Clusterware

OS Clusterware

CS SD

CSSD
interconnect OPROCD

OPROCD

Voting disk

13

2009/2010 Pythian

OS Clusterware

OS Clusterware

CSSD Monitor/Agent

CS SD

CSSD
interconnect OPROCD

OPROCD

Voting disk

13

2009/2010 Pythian

OS Clusterware

CSSD Monitor/Agent

CSSD
interconnect OPROCD

Voting disk

13

2009/2010 Pythian

OS Clusterware

CSSD interconnect

CSSD
OPROCD

Voting disk

14

2009/2010 Pythian

OS Clusterware

CSSD interconnect

CSSD
OPROCD

Voting disk

14

2009/2010 Pythian

OS Clusterware

CSSD interconnect

CSSD
OPROCD

Voting disk

14

2009/2010 Pythian

OS Clusterware

CSSD interconnect

CSSD
OPROCD

OPROCD/CSSD Mon

Voting disk

14

2009/2010 Pythian

OS Clusterware

CSSD OPROCD/CSSD Mon


interconnect OPROCD

Voting disk

14

2009/2010 Pythian

OS Clusterware

OS Clusterware

CSSD
interconnect OPROCD

CSSD
OPROCD

Voting disk

15

2009/2010 Pythian

OS Clusterware

OS Clusterware

CSSD
interconnect OPROCD

CSSD
OPROCD

Voting disk

15

2009/2010 Pythian

OS Clusterware

OS Clusterware

CSSD
interconnect OPROCD

CSSD
OPROCD

Voting disk

15

2009/2010 Pythian

OS Clusterware

CSSD
interconnect

CSSD
OPROCD

Voting disk

15

2009/2010 Pythian

OS Clusterware

OS Clusterware

CSSD
interconnect OPROCD

CSSD
OPROCD

Voting disk

16

2009/2010 Pythian

OS Clusterware

OS Clusterware

CSSD
interconnect OPROCD

CSSD
OPROCD

16

2009/2010 Pythian

OS Clusterware

OS Clusterware

CSSD
interconnect OPROCD

CSSD
OPROCD

16

2009/2010 Pythian

CSSD
interconnect

CSSD

16

2009/2010 Pythian

11gR2 Grid Infrastructure: CSSD attempts graceful shutdown

17

2009/2010 Pythian

OS

OS

Clusterware LMON Instance

Clusterware Instance

CSSD
OPROCD interconnect

CSSD
OPROCD

18

2009/2010 Pythian

OS

OS

Clusterware LMON Instance

Clusterware Instance

member kill

CSSD
OPROCD interconnect

CSSD
OPROCD

18

2009/2010 Pythian

OS

OS

Clusterware LMON Instance

Clusterware Instance

member kill

CSSD
OPROCD interconnect

CSSD
OPROCD

18

2009/2010 Pythian

OS

OS

Clusterware LMON Instance

Clusterware Instance

member kill

CSSD
OPROCD interconnect

CSSD
OPROCD

18

2009/2010 Pythian

OS

OS

Clusterware LMON Instance

Clusterware Instance

member kill

CSSD
OPROCD interconnect

CSSD
OPROCD

18

2009/2010 Pythian

OS

OS

Clusterware LMON Instance

Clusterware Instance

member kill

CSSD
OPROCD interconnect

CSSD
OPROCD

Eviction by escalation of a member kill

18

2009/2010 Pythian

OS

Clusterware LMON Instance

member kill

CSSD
OPROCD interconnect

Eviction by escalation of a member kill

18

2009/2010 Pythian

OS Clusterware

OS Clusterware

CSSD
OPROCD interconnect

CSSD
OPROCD

Voting disk

19

2009/2010 Pythian

OS Clusterware

OS Clusterware

CSSD
OPROCD interconnect

CSSD
OPROCD

11gR2 Intelligent Platform Management Interface


19

Voting disk

2009/2010 Pythian

OS Clusterware

OS Clusterware

CSSD
OPROCD interconnect

CSSD
OPROCD

11gR2 Intelligent Platform Management Interface


19

Voting disk

2009/2010 Pythian

OS Clusterware

OS Clusterware

CSSD
OPROCD interconnect

CSSD
OPROCD

11gR2 Intelligent Platform Management Interface


19

Voting disk

2009/2010 Pythian

OS Clusterware

OS Clusterware

CSSD
OPROCD interconnect

CSSD
OPROCD

Exadata Fencing
20

Voting disk

2009/2010 Pythian

Grid Infrastructure Startup

10g / 11gR1
Linux

11gR2
Linux

& UNIX inittab

& UNIX inittab

init.cssd init.evmd init.crsd


Linux

init.ohasd run

& UNIX init.d

init.crs

Linux

& UNIX init.d

ohasd start
Windows

Windows

Services

Services

21

2009/2010 Pythian

Startup in Linux & Unix


[root@cheese2 ~]# ps -fe | grep 'init\.' | grep -v grep

root

4283

0 02:52 ?

00:00:00 /bin/sh /etc/init.d/init.ohasd run

[root@cheese2 ~]# tail -1 /etc/inittab

h1:35:respawn:/etc/init.d/init.ohasd run >/dev/null 2>&1 </dev/null

[root@cheese2 ~]# ls -l /etc/rc3.d/*ohasd* lrwxrwxrwx 1 root root 17 Sep 15 02:02 /etc/rc3.d/K15ohasd -> /etc/init.d/ohasd lrwxrwxrwx 1 root root 17 Sep 15 02:02 /etc/rc3.d/S96ohasd -> /etc/init.d/ohasd

22

2009/2010 Pythian

/etc/oracle/scls_scr/{host}/root/cssrun

Pre 11gR2 Clusterware Startup


/etc/oracle/scls_scr/{host}/root/crsstart enable disable

init.crs start

init.cssd autostart
init.cssd oprodc init.cssd oclsomon init.cssd oclsvmon init.cssd daemon oprocd oclsomon.bin oclsvmon.bin ocssd.bin

init.cssd fatal init.evmd run init.crsd run evmd.bin

crsd.bin
t

23

2009/2010 Pythian

Grid Infrastructure Startup

24

2009/2010 Pythian

Grid Infrastructure Startup

init.ohasd run t

24

2009/2010 Pythian

Grid Infrastructure Startup


/etc/oracle/scls_scr/{host}/root/ohasdrun

init.ohasd run t

24

2009/2010 Pythian

Grid Infrastructure Startup


/etc/oracle/scls_scr/{host}/root/ohasdrun

init.ohasd run t

24

2009/2010 Pythian

Grid Infrastructure Startup


/etc/oracle/scls_scr/{host}/root/ohasdrun

/etc/init.d/ohasd

init.ohasd run t

24

2009/2010 Pythian

Grid Infrastructure Startup


/etc/oracle/scls_scr/{host}/root/ohasdrun

/etc/init.d/ohasd

/etc/oracle/scls_scr/{host}/root/ohasdstr enable disable

init.ohasd run t

24

2009/2010 Pythian

Grid Infrastructure Startup


/etc/oracle/scls_scr/{host}/root/ohasdrun

/etc/init.d/ohasd

/etc/oracle/scls_scr/{host}/root/ohasdstr enable disable

init.ohasd run t

24

2009/2010 Pythian

Grid Infrastructure Startup


/etc/oracle/scls_scr/{host}/root/ohasdrun

init.ohasd run t

24

2009/2010 Pythian

Grid Infrastructure Startup


/etc/oracle/scls_scr/{host}/root/ohasdrun

OHAS

init.ohasd run t

24

2009/2010 Pythian

Grid Infrastructure Startup


/etc/oracle/scls_scr/{host}/root/ohasdrun

Ora Agent

OHAS

init.ohasd run t

24

2009/2010 Pythian

Grid Infrastructure Startup


/etc/oracle/scls_scr/{host}/root/ohasdrun

Ora Agent

OHAS

Root Agent

init.ohasd run t

24

2009/2010 Pythian

Grid Infrastructure Startup


/etc/oracle/scls_scr/{host}/root/ohasdrun

Ora Agent

OHAS

Root Agent CSSD Agent

init.ohasd run t

24

2009/2010 Pythian

Grid Infrastructure Startup


/etc/oracle/scls_scr/{host}/root/ohasdrun

Ora Agent

OHAS

Root Agent CSSD Agent

init.ohasd run

CSSD Monitor t

24

2009/2010 Pythian

Grid Infrastructure Startup


/etc/oracle/scls_scr/{host}/root/ohasdrun

Ora Agent

OHAS

Root Agent CSSD Agent CSS

init.ohasd run

CSSD Monitor t

24

2009/2010 Pythian

Grid Infrastructure Startup


/etc/oracle/scls_scr/{host}/root/ohasdrun

Ora Agent

ACFS Drivers CTSS Disk Monitor OHAS Root Agent CSSD Agent init.ohasd run CSSD Monitor t CRS

CSS

24

2009/2010 Pythian

Grid Infrastructure Startup


/etc/oracle/scls_scr/{host}/root/ohasdrun ASM EVM GPnP GIPC Ora Agent MDNS

ACFS Drivers CTSS Disk Monitor OHAS Root Agent CSSD Agent init.ohasd run CSSD Monitor t CRS

CSS

24

2009/2010 Pythian

Grid Infrastructure Startup


/etc/oracle/scls_scr/{host}/root/ohasdrun ASM EVM GPnP GIPC Ora Agent MDNS CRS Root Agent

ACFS Drivers CTSS Disk Monitor OHAS Root Agent CSSD Agent init.ohasd run CSSD Monitor CRS

VIP SCAN IP Network GNS ACFS Registry

CSS

24

2009/2010 Pythian

Grid Infrastructure Startup


/etc/oracle/scls_scr/{host}/root/ohasdrun ASM EVM GPnP GIPC Ora Agent MDNS CRS Root Agent

ACFS Drivers CTSS Disk Monitor OHAS Root Agent CSSD Agent init.ohasd run CSSD Monitor CRS CRS Ora Agent

VIP SCAN IP Network GNS ACFS Registry

CSS

Database Instance Listener Services Diskgroups ONS eONS SCAN Listener


t

24

2009/2010 Pythian

Grid Infrastructure Startup (static slide)


ASM EVM GPnP GIPC Ora Agent MDNS CRS Root Agent

ACFS Drivers CTSS Disk Monitor OHAS Root Agent CSSD Agent init.ohasd run CSSD Monitor CRS CRS Ora Agent

VIP SCAN IP Network GNS ACFS Registry

CSS

Database Instance Listener Services Diskgroups ONS eONS SCAN Listener


t

25

2009/2010 Pythian

Grid Infrastructure Log Files


$GRID_HOME/log/{hostname}/ alert<host>.log ohasd crsd cssd agent/ohasd/oraagent_oracle agent/ohasd/oracssdagent_root agent/ohasd/oracssdmonitor_root agent/ohasd/orarootagent_root agent/crsd/oraagent_oracle agent/crsd/orarootagent_root ctssd diskmon gipcd gnsd gpnpd mdnsd racg

26

2009/2010 Pythian

Oracle Cluster Registry


Repository

data

for all shared configuration

Except OCR location itself

OCR

is accessed mostly read-only is written only by CRS - Oracle Local Registry (OLR)

Every component reads OCR only from a single OCR master node managed by ohasd

OCR

11gR2

27

2009/2010 Pythian

DEMO Interconnect Failure


Simulate

with ifconfig eth1 down on node 2 Both nodes notice the loss Racing to evict each other

from voting disk => 2 equal sub-clusters (cohorts) survives the one with the lowest leader #

leader is the node with lowest # in sub-cluster

Winner

evicts another node

Setting kill-block in voting disk

11gR2

new feature: CSSD does clean restart

28

2009/2010 Pythian

DEMO Voting Disk Failure


Simulate

server CSSD detects voting disk IO stale

with storage interface down or on NFS

disktimeout setting - 200 seconds default

CSSD

starts eviction 11gR2 new feature: CSSD does clean restart

29

2009/2010 Pythian

DEMO CSSD is not healthy


Simulate

using kill -STOP <cssd.bin pid> (and try kill -9) Another node observes NHB loss

After misscount seconds => attempt eviction

but CSSD is frozen and cant commit suicide

CSSD

Monitor detects CSSD timeout

Commit suicide

30

2009/2010 Pythian

Homework Host sick - CPU stalled


Used

to simulate by pausing OPROCD

kill -STOP <oprocd pid> sleep 1 or 2 kill -CONT <oprocd pid>

OPROCD

is now a thread in CSSD Monitor

kill -STOP {cssdmonitor.bin} ; sleep 1 ; kill -CONT {cssdmonitor.bin}

Doesnt

produce any visible results! Is there still OPROCD?

31

2009/2010 Pythian

DEMO Startup troubleshooting


Break

before starting up

Interconnect, voting disk, Grid Home missing

Check

processes using ps -fe | grep init Check syslog (/var/log/messages) boot sequence Clusterware log files

if *.bin processes are running already

crsctl

check crs crsctl status resource -t -init


32 2009/2010 Pythian

11gR2 Cluster Resources


Introduced

resource type

APPLICATION was the only supported resource type in 11gR1 Resource types using resource agents

Cluster

and local resources Sophisticated resources dependencies (stop & start)


hard weak attraction pullup dispersion

33

2009/2010 Pythian

Grid Infrastructure Processes

ora.cluster_interconnect.haip ASM

CRS Managed Resources

OHAS Internal Resources

EVM GPnP GIPC CRS Root Agent

Ora Agent

MDNS

ACFS Drivers CTSS Disk Monitor OHAS Root Agent CSSD Agent init.ohasd run CSSD Monitor CRS


CRS Ora Agent

VIP SCAN IP Network GNS ACFS Registry

CSS

Database Instance Listener Services Diskgroups ONS eONS SCAN Listener

34

2009/2010 Pythian

Troubleshooting Something Down


OHASD

up? OHASD Agents up? Internal resources up? CRSD Agents up? Managed resources up?

Listener & VIP Database & ASM instance Services

Have

the nodes rebooted? Have resources re-started?


35 2009/2010 Pythian

11gR2 Grid Infrastructure References


Oracle Clusterware Administration and Deployment Guide MOS 1053147.1

11gR2 Clusterware and Grid Home - What You Need to Know

MOS 1050908.1

How to Troubleshoot Grid Infrastructure Startup Issues

MOS 1053970.1

Troubleshooting 11.2 Grid Infastructure Installation Root.sh Issues

MOS 1050693.1

Troubleshooting 11.2 Clusterware Node Evictions (Reboots)

MOS 942166.1

How to Proceed from Failed 11gR2 Grid Infrastructure Installation

36

2009/2010 Pythian

Potrebbero piacerti anche