Oracle(R) 10gR2 Real Application Cluster Installation Using Sun Cluster Software and Sun StorageTek(R) 000 Series

!AS Appliance
AS"# $isk "roducts %roup Sun &icros'ste(s) Inc*

Sun &icros'ste(s) Inc*

+ 200 Sun &icros'ste(s) Inc*) ,1 0 !etwork Circle) Santa Clara) CA - 0 , USA All rig.ts reser/ed* T.is product or docu(ent is protected 0' cop'rig.t and distri0uted under licenses restricting its use) cop'ing) distri0ution) and deco(pilation* !o part of t.is product or docu(ent (a' 0e reproduced in an' for( 0' an' (eans wit.out prior written aut.ori1ation of Sun and its licensors) if an'* T.ird2part' software) including font tec.nolog') is cop'rig.ted and licensed fro( Sun suppliers* "arts of t.e product (a' 0e deri/ed fro( 3erkele' 3S$ appliances) licensed fro( t.e Uni/ersit' of California* Sun) Sun &icros'ste(s) Sun Stor#dge) t.e Sun logo) are trade(arks) registered trade(arks) or ser/ice (arks of Sun &icros'ste(s) Inc* in t.e U*S* and ot.er countries* U!I4 is a registered trade(ark in t.e United States and ot.er countries) e5clusi/el' licensed t.roug. 46Open Co(pan') 7td* 8indows is a registered trade(ark of &icrosoft Corporation in t.e United States and ot.er countries* All S"ARC trade(arks are used under license and are trade(arks or registered trade(arks of S"ARC International) Inc* in t.e U*S* and ot.er countries* "roducts 0earing S"ARC trade(arks are 0ased upon an arc.itecture de/eloped 0' Sun &icros'ste(s) Inc* T.e O"#! 7OO9 and Sun:s %rap.ical User Interface was de/eloped 0' Sun &icros'ste(s) Inc* for its users and licensees* Sun acknowledges t.e pioneering efforts of 4ero5 in researc.ing and de/eloping t.e concept of /isual or grap.ical user interfaces for t.e co(puter industr'* Sun .olds a non2e5clusi/e license fro( 4ero5 to t.e 4ero5 %rap.ical User Interface) w.ic. license also co/ers Sun:s licensees w.o i(ple(ent O"#! 7OO9 %UIs and ot.erwise co(pl' wit. Sun:s written license agree(ents* R#STRICT#$ RI%;TS< Use) duplication) or disclosure 0' t.e U*S* %o/ern(ent is su0=ect to restrictions of >AR 2*22?21,(g)(2)(@6A?) and >AR 2*22?21-A?)) or $>AR 2 2*22?2?01 (0)(@6- ) and $>AR 22?*?2022B(a)* $OCU&#!TATIO! IS "ROCI$#$ DAS ISE A!$ A77 #4"R#SS OR I&"7I#$ CO!$ITIO!S) R#"R#S#!TATIO!S A!$ 8ARRA!TI#S) I!C7U$I!% A!F I&"7I#$ 8ARRA!TF O> &#RC;A!TA3I7ITF) >IT!#SS >OR A "ARTICU7AR "UR"OS# OR !O!2I!>RI!%#&#!T) AR# $ISC7AI&#$) #4C#"T TO T;# #4T#!T T;AT SUC; $ISC7AI&#RS ;#7$ TO 3# 7#%A77F

Sun &icros'ste(s) Inc*

Table of Contents
Introduction*************************************************************************************************************************************************************1 Sun StorageTek 000 Series !AS Appliance for Oracle*****************************************************************************************1 Sun Cluster for Oracle RAC************************************************************************************************************************************1 Supported Configuration*****************************************************************************************************************************************2 Sun Cluster Guoru( $e/ice(G$) ***************************************************************************************************************************B SCSI Reser/ation *************************************************************************************************************************************************B Internet SCSI (iSCSI)**********************************************************************************************************************************************, Oracle RAC wit. Sun Cluster I(ple(entation o/er/iew******************************************************************************************, Stage 1 < ;ost Configurations *******************************************************************************************************************************@ Stage 2a < >ile Colu(e Configuration on StorageTek !AS Appliance*********************************************************************? Stage 20 < iSCSI 7U! Configuration on StorageTek !AS and Cluster !odes********************************************************A Stage B < Sun Cluster B*1 U , setup************************************************************************************************************************Stage , < Installing Oracle RAC >ra(ework for Sun Cluster*********************************************************************************10 Stage < Oracle #n/iron(ent Setup*********************************************************************************************************************11

Stage @ < Oracle 10gR2 Clusterware Installation***************************************************************************************************1B Stage ? < Oracle RAC $ata0ase 0inaries Installation********************************************************************************************1, $ata0ase Creation************************************************************************************************************************************************1 Reference*************************************************************************************************************************************************************1@ Appendi5***************************************************************************************************************************************************************1?

Oracle 10gR2 RAC wit. Sun Cluster on StorageTek !AS Appliance

1 Introduction

Sun &icros'ste(s) Inc*

Introduction
T.is tec.nical report descri0es t.e procedure to i(ple(ent a two2node Oracle10gR2(R) Real Application Cluster (RAC) wit. Sun Cluster Software in a Solaris T&10 U2 Operating en/iron(ent using Sun StorageTek(R) 000 !AS appliance for storage* T.e intended audience are s'ste( and6or data0ase ad(inistrators* T.e reader of t.is docu(ent is assu(ed to .a/e a fair understanding of Solaris) Sun Cluster software) Oracle Clusterware) iSCSI and !>S protocols* !ote < T.e ter( Sun StorageTek NAS Appliance is s'non'(ousl' used wit. Sun StorageTek NAS 5320 Appliance.

Sun StorageTek 000 Series !AS Appliance for Oracle

So(e of t.e ke' ad/antages 0' i(ple(enting Oracle RAC using Sun StorageTek 000 Series !AS Appliance are < 1* All t.e different t'pes of Oracle files suc. as 0inaries) data files) cluster files) 0ackup sets etc*) can 0e stored on t.e !AS Appliance and can 0e s.ared across (ultiple nodes using !>S protocol* 2* T.ere is no reHuire(ent for additional /olu(e (anage(ent and file (anage(ent* It is >ile s'ste( 0ased storage w.ic. are !>S s.ared and /er' eas' to (anage* B* Controller 0ased RAI$2 data protection* ,* 3usiness Continuit' protection features suc. as C.eckpoint Software (Snaps.ot) and >ile Replicator (Re(ote (irroring) pro/ide wa's to protect t.e data fro( failures* 3ot. t.ese features are certified to work wit. Oracle10g6-i data0ases* * Oracle RAC i(ple(entations on Sun StorageTek !AS appliance is certified 0' Oracle* @* $'na(ic addition of space to t.e file /olu(e wit.out an' down ti(e for Oracle RAC*

Sun Cluster for Oracle RAC

Sun Cluster is part of t.e Solaris #nterprise S'ste(* Sun Cluster software offers t.e pre(ier a/aila0ilit' platfor( for i(pro/ing t.e predicta0ilit' and resilience of 0usiness critical applications I suc. as Oracle data0ases in a clustered en/iron(ent* Sun Cluster fra(ework for Oracle RAC pro/ides t.e enterprise wit. increased (anagea0ilit' for RAC deplo'(ents on 0ot. S"ARC and 4@, platfor(s* T.e following are so(e of t.e ad/antages w.ile i(ple(enting Oracle RAC wit.in Sun Cluster fra(ework < Network Interconnects & node synchronization 3etter fault (anage(ent and a/aila0ilit' due to tig.ter integration wit. Solaris kernel* ;eart 0eats are (ore relia0le and tuna0le* 8it. redundant pri/ate interconnect (up to @)) offers 0etter co((unication and fault tolerance 0' doing transparent fail2o/er of traffic fro( failed link(s) to t.e working link(s)* T.is is configured as part of Sun Cluster installation) aided 0' t.e auto2disco/er' capa0ilit' in t.e installation tools* !etwork interface na(es could 0e different fro( node to node) allowing fle5i0ilit' of using different !ICs (odel or place(ent of !ICs in ser/er e5pansion slots) etc* At runti(e) Sun Cluster auto(aticall' plu(0 up t.e interconnect interfaces and assign network address and su0net attri0utes* T.e interconnects can 0e d'na(icall' (odifia0le and new interconnects also can 0e added in real2ti(e wit. no down ti(e to Oracle RAC* Onl' one interface na(e clprivnet0 w.ic. is configured 0' Sun Cluster will 0e used 0' t.e Oracle CRS for internode co((unication* T.is pseudo interface is unifor( across t.e nodes* Cac.e fusion traffic is striped o/er t.ese (ultiple interconnects resulting in .ig.er network t.roug.put* Oracle 10gR2 RAC wit. Sun Cluster on StorageTek !AS Appliance

2 Sun Cluster for Oracle RAC

Sun &icros'ste(s) Inc*

Sun ClusterJs R$T (Relia0le $atagra( Transport) along wit. Solaris RS& (Re(ote S.ared &e(or') on a speciali1ed interconnect .ardware SCI (Scala0le Co.erent Interface) pro/ides an opti(i1ed I"C transport for Oracle RAC wit. 0etter t.roug.put and latenc' in co(parison wit. Oracle RACJs default U$"6I" transport la'er* Sun Cluster takes care of nodes ti(e s'nc.roni1ation during installation and keeps t.e ti(e across t.e node s'nc.roni1ed using !etwork Ti(e protocol w.ic. is critical for Oracle Clusterware operations*

I/O Fencing T.e Guoru( de/ice (anage(ent 0' Sun Cluster w.ic. is a for( of fencing is acti/ated on a two node configuration to detect split20rain and also t.e .ealt. of participating nodes of a cluster* If a node is dee(ed un.ealt.') t.e node not onl' e/icted fro( cluster (e(0ers.ip) 0ut no writes are allowed fro( t.e node to t.e s.ared storage to ensure data integrit'* Sun Cluster ena0les fail2fast 0e.a/ior in t.e kernel I6O (odule so t.at on t.e failed node) upon recei/ing error fro( t.e storage) t.e dri/er panics t.e node*

Supported Configuration
Configuration Operating S'ste( Solaris 10 Update 2 Solaris 10 6 Solaris 10 U 1 wit. "atc.11-0-0220 (Sparc) 6 11-0-1220 (5A@) or a0o/e Cluster Configuration Ser/er configuration Two node cluster ( S"ARC 6 4@, ) Sparc 6 4@, @, 0it At least 2%3 of (e(or' !etwork interface ( !ode interconnects ) Sun Cluster Oracle Clusterware 6 $ata0ase Storage %ig# ( 1000 &0it ) B*1 U , 6 B*2 10*1*0*, or a0o/e 10*2*0*2 or a0o/e Sun StorageTek 000 series !AS Appliance StorageTek !AS OS /ersion ,*20 or a0o/e RAI$2 Configuration "rotocol Sun Cluster Guoru( de/ice !>S/B iSCSI 7U! of Sun StorageTek 000 !AS (or) A s.ared >C storage (direct attac.ed 6 SA!) Value

Oracle 10gR2 RAC wit. Sun Cluster on StorageTek !AS Appliance

B Sun Cluster Guoru( $e/ice(G$)

Sun &icros'ste(s) Inc*

Sun Cluster Guoru( $e/ice(G$)

In Sun Cluster) t.e (ec.anis( t.at deter(ines node participation is known as a quorum* #ac. cluster node is counted towards a Huoru( /ote* >or t.e cluster to 0e operational) t.e /ote reHuire(ent is !62 K 1 w.ere ! is t.e node count* In a two2node cluster) in order for t.e cluster to sur/i/e) it reHuires at least 2 /otes* 8it.out a t.ird Huoru() in t.e e/ent of one of t.e cluster node is down) t.e entire cluster would 0e down* In order to a/oid t.at condition) a s.ared storage de/ice is identified and configured as t.e Huoru( de/ice* T.e de/ice (ust 0e a s.ared disk t.at can 0e accessed 0' 0ot. t.e nodes of t.e cluster* Sun Cluster configurations use Huoru( de/ices to (aintain data and resource integrit' 0' pre/enting a(nesia and split20rain pro0le(s w.en t.e cluster node atte(pts to =oin t.e cluster*
!ode2a 1 !ode20 1

1 Guoru( $e/ice

T.e reHuire(ents <

Two2node cluster (ust .a/e a Huoru( de/ice* >or ot.er topologies) it is optional* Configure odd nu(0er of Huoru( de/ices* To ensure t.e Huoru( de/ices to .a/e .a/e co(pletel' independent failure pat.wa's* T.ere is no real si1e reHuire(ent for t.e Huoru( de/ice* It is also possi0le to assign t.e de/ice w.ic. could contain user data as t.e Huoru( de/ice*

To i(ple(ent a two2node Sun Cluster wit. StorageTek !AS Appliance) t.e following (et.ods can 0e used* 1* Use iSCSI 7un fro( StorageTek !AS appliance as s.ared 0lock de/ice for Huoru( de/ice* 2* Use s.ared disk fro( direct attac.ed >C storage (or) SA! 7un for Huoru( de/ice* B* Using Sun Cluster B*2 /ersion) a Guoru( Ser/er can 0e configured* In t.is docu(ent) t.e procedure for (et.od (1) w.ic. uses iSCSI as t.e Huoru( de/ice is e5plained* If t.e user c.oose to use a non2iSCSI lun as G$) t.e sa(e procedure can 0e followed e5cluding Stage220 w.ic. descri0es iSCSI configuration*

SCSI Reser/ation
Sun Cluster uses t.e SCSI reser/ation (ec.anis( in con=unction wit. Huoru( principle for data integrit' protection* 8.en t.e cluster is for(ed) one node takes responsi0ilit' of t.e Huoru( de/ice and is tagged as t.e owner* T.e ot.er node is tagged as capa0le of 0eco(ing owner* T.e ke's are written to t.e Huoru( de/ice 0' 0ot. t.e nodes and so) t.e nodes are allowed to 0e part of t.e cluster* If a node lea/es t.e cluster) t.e ot.er node clears t.e ke's t.at 0elong to t.e node t.at left t.e cluster* T.e node is fenced and no I6O can .appen fro( t.e node* 8.en t.at node re=oins t.e cluster) t.e ke's are re2registered and ena0led access* Sun Cluster uses e(ulated "%R (persistent group reser/ation) for SCSI22 de/ices* >or cluster nodes wit. (ore t.an two nodes) t.e s.ared storage de/ice (ust 0e SCSI2B capa0le*

Oracle 10gR2 RAC wit. Sun Cluster on StorageTek !AS Appliance

, Internet SCSI (iSCSI)

Sun &icros'ste(s) Inc*

Internet SCSI (iSCSI)

iSCSI is a protocol t.at ena0les transport of 0lock data o/er I" network* T.is protocol doesnJt reHuire an' special network infrastructure t.at is t'picall' reHuired for 0lock de/ice access fro( de/ices suc. as >i0re c.annel* Sun StorageTek !AS appliance .as t.e capa0ilit' to pro/ide 0lock le/el access for t.e file /olu(e* T.e iSCSI luns are created and granted access to t.e nodes in t.e StorageTek !AS appliance* After t.at) t.e iSCSI configuration .as to 0e done at t.e nodes* T.e iSCSI luns are disco/ered 0' t.e nodes eit.er using static or dynamic (target) (et.od* In static (et.od) de/ice is disco/ered 0' pro/iding t.e iSCSI na(e w.ic. is gi/en 0' t.e !AS appliance for t.at lun* In target disco/er' (et.od) all t.e iSCSI luns t.at are created in t.e !AS appliance can 0e accessed 0' t.e node*

Oracle RAC wit. Sun Cluster I(ple(entation o/er/iew

>or t.is proof of concept) StorageTek !AS B20 Appliance is used to store all Oracle files including Oracle data0ase files) OCR) Coting and Oracle 0inaries* >or Huoru( de/ice) iSCSI lun fro( StorageTek !AS is used*Sun Cluster 0inaries are stored local to t.e nodes* In our test en/iron(ent) a fres. install of Solaris 10 U 2 was done on 0ot. t.e cluster nodes* A slig.tl' different procedure .a/e to 0e followed for t.e 4A@ and Sparc arc.itectures* T.ose are e5plained w.ere/er applica0le* Note : T.e steps pro/ided .ere are to 0e considered onl' as a guideline and not to 0e su0stituted for t.e install procedures pro/ided 0' t.e indi/idual co(ponents* >or details) please refer to t.e installation docu(ents for Sun Cluster B*1) Oracle RAC agent for Sun Cluster B*1) Oracle Clusterware) Oracle $ata0ase) StorageTek B20 !AS Appliance and Solaris operating s'ste(* Also) /erif' t.e co(pati0ilit' (atri5 0etween Solaris /ersions) Oracle /ersions and Sun Cluster /ersions for dependencies* T.e steps and procedures e5plained in t.is docu(ent are 0ased on Sun Cluster B*1 Update , /ersion* >or Sun Cluster B*2 /ersion) please refer to t.e docu(entation for t.at steps need to 0e followed*

Oracle 10gR2 RAC wit. Sun Cluster on StorageTek !AS Appliance

Stage 1 < ;ost Configurations

Sun &icros'ste(s) Inc*

Oracle 10gR2 RAC wit. Sun Cluster on StorageTek !AS Appliance

@ Stage 1 < ;ost Configurations

Sun &icros'ste(s) Inc*

Stage 1 < ;ost Configurations

T.e following steps are to 0e perfor(ed on 0ot. t.e nodes*

hysical configurations
1* 2* B* ,* Two cluster nodes wit. si(ilar arc.itecture (4A@ or Sparc)* One 00&3 partition for glo0alde/ice configuration on eac. node* At least two pri/ate interconnects (%ig#) 0etween t.e( /ia gig2# switc.* Sun StorageTek !AS appliance wit. two %ig2# network interface for data pat.* T.e t.ird !IC is used for (anage(ent*

!n"iron#ent configurations
1* #na0le telnet sessions and /erif' < # svcadm -v enable svc:/network/telnet:default # svcs -a | grep telnet 2* #na0le rlogin6rs. sessions L /erif' < # inetadm -e svc:/network/login:rlogin # inetadm -e svc:/network/shell:default # inetadm -e svc:/network/rexec:default # inetadm | egrep rlog B* Set t.e following para(eters in etc system file* 8it. Solaris 10) a lot of s.ared (e(or' and se(ap.ore settings are o0soleted* T.e s!mma" /alue is to specif' t.e (a5i(u( s.ared (e(or' t.at can 0e allocated for t.e Oracle instance* It is reco((ended to .a/e t.is /alue to at least 0M of t.e p.'sical (e(or'* set noexec_user_stack=1 set shmsys:shminfo_shmmax=4294967295 ,* 8it. Solaris 10) t.e (ultipat. software ( Sun Traffic &anager) is 0uilt2in t.e kernel itself* Cerif' w.et.er t.e (ultipat. option in kernel drv #p.con# file* If t.e mpxio-disable=yes) t.en c.ange t.e /alue to no. * At least two p.'sical %iga0it #t.ernet connections are reHuired 0etween t.e cluster nodes for pri/ate interconnects to 0e used 0' 0ot. Oracle and Sun Cluster* T.ere is no need to configure pri/ate .ostna(es* As long as t.e p.'sical connection is a/aila0le) Sun Cluster plu(0s t.e interface up and assigns I" addresses and na(es clprivnet$%&. #dit etc !osts and add t.e cluster nodes and CI" nodes* # Internet host table # 127.0.0.1 localhost 10.8.54.54 node-a loghost 10.8.54.57 node-b # VIP Nodes 10.8.54.80 10.8.54.81

# Primary Cluster node # Secondary Cluster Node

node-a-vip node-b-vip

Oracle 10gR2 RAC wit. Sun Cluster on StorageTek !AS Appliance

? Stage 2a < >ile Colu(e Configuration on StorageTek !AS Appliance

Sun &icros'ste(s) Inc*

Stage 2a < >ile Colu(e Configuration on StorageTek !AS Appliance

It is reco((ended to install t.e latest StorageTek !AS O6S /ersion ( at t.is ti(e of writing) ,*20 ) and an' patc.es I if a/aila0le fro( .ttp<66sunsol/e*co(* Configure network interface for StorageTek !AS wit. at least two %iga0it !IC interfaces* It is reco((ended to connect eac. interface to a different switc. for .ig. a/aila0ilit'* T.e role for t.ose interfaces (ust 0e 'rimary* It is also reco((ended to configure one %iga0it interface for (ile )eplicator 2 w.ic. is a pre(ier feature for re(ote replication w.ic. replicates /olu(es to anot.er StorageTek !AS Appliance* T.e role for t.at interface (ust 0e N&irrorO* Create one RAI$ group ( 7U!) using t.e dri/es fro( a tra'* #ac. RAI$ group stripes t.e data across t.e dri/es t.at were c.osen*

Create a 7U! using 1, 5 1,@%3 dri/es*

Create t.ree partitions ( w.ic. also creates file /olu(e )) for Oracle 0inaries) data0ase files and Clusterware files* It is also possi0le to use sa(e file /olu(e for data0ase and clusterware files I since t.e' .a/e si(ilar !>S (ount options* a) >or $ata0ase wit. 200%3 1* Create 1%3 file /olu(e (partition) oradata 2* Create one 200%3 seg(ent seg*. B* Attac. t.at seg(ent to t.e file /olu(e oradata* 0 ) >or Oracle 0inaries ) create 20%3 file/olu(e oracle+in. c) >or OCR and Coting Clusterware files) e"ternal redundancy is used wit. one cop' of OCR and Coting disk * Create 2%3 file/olu(e cluster,are.

>ro( t.e %UI (enu2P Uni5 Configuration 2P Configure !>S 2P add t.e .osts and create .ost group 0' adding t.e two nodes into a group* Host group : scracnas Hosts : node-a, node-b T.e alternate wa' is to use t.e N%eneralO .ost group w.ic. grants access of t.ese /olu(es to an' node*

Configure !>S e5port for t.e file /olu(es fro( %UI (enu 2P click on file /olu(e 2P rig.t click and c.oose properties* 1* Access 2* &ap root user B* ;ost < )ead -rite < root < C.oose .ost /roup and fro( drop2down) c.oose t.e scracnas *

Oracle 10gR2 RAC wit. Sun Cluster on StorageTek !AS Appliance

A Stage 20 < iSCSI 7U! Configuration on StorageTek !AS and Cluster !odes

Sun &icros'ste(s) Inc*

Stage 20 < iSCSI 7U! Configuration on StorageTek !AS and Cluster !odes
T.e following is t.e procedure to create and use t.e iSCSI lun as Huoru( de/ice* >or t.is proof2of2concept) t.e iSCSI de/ice is going to 0e used purel' for Huoru( purposes and not for storing an' data* !o iS!S ser/er is configured* On Sun StorageTek !AS Appliance< 1* Using we0 0ased UI) point t.e UR7 to !AS appliance and) create file /olu(e (iscsiqd) wit. 10&3 si1e* 2* %o to ISCSI Configuration screen and add t.e nodes in t.e Access list to allow access to t.e cluster nodes* ( add node0a and node0+ ) B* %o to Configure iSCSI lun screen and add a new iSCSI lun >ro( t.e drop2down /olu(e list* C.oose t.e file/olu(e iscsiqd* C.oose si1e s(aller t.an t.e file/olu(e si1e ( &3 ) After t.is step) t.e !AS 0o5 adds t.e iSCSI lun and assigns na(e* iqn.1986-03.com.sun:01:000e0c9f0afe.44D4C0E4.iscsiqd On 0ot. t.e cluster nodes < 1* Only for $olaris %& and $olaris %& '(date % < It is strongl' reco((ended to update t.e iSCSI dri/ers to t.e latest patc. for proper functioning* O0tain t.e latest S1N-iscsir and S1N-iscsiu patc.es ( w.ic. can 0e o0tained fro( .ttp<66sunsol/e*sun*co( )* As at ti(e of t.is writing) t.e following patc.es need to 0e applied < 11-0-0220 (Sparc) 11-0-1220(4@,) # patchadd -d 119090-20 PKGINST: SUNWiscsir NAME: Sun iSCSI Device Driver (root) CATEGORY: system ARCH: sparc VERSION: 11.10.0,REV=2005.01.04.14.31 BASEDIR: / VENDOR: Sun Microsystems, Inc. DESC: Sun iSCSI Device Driver NOT! : No i$C$I dri"er (atches are re)uired for $olaris %& '(date *+ 2* #na0le t.e static disco/er' (ode* It is reco((ended to use static disco/er' (et.od* #iscsiadm modify discovery --static enable B* Add t.e iSCSI static configuration* >or e5a(ple) # iscsiadm add static-config \ iqn.1986-03.com.sun:01:000e0c9f0afe.44D4C0E4.iscsiqd,10.8.11.250 ,* $isco/er t.e iSCSI de/ice and run #ormat co((and to /erif' w.et.er t.e de/ice is seen 0' t.e .ost* # devfsadm -i iscsi # format
Searching for disks...done

Oracle 10gR2 RAC wit. Sun Cluster on StorageTek !AS Appliance

- Stage 20 < iSCSI 7U! Configuration on StorageTek !AS and Cluster !odes

Sun &icros'ste(s) Inc*

AVAILABLE DISK SELECTIONS: 0. c0t0d0 <SUN72G cyl 14087 alt 2 hd 24 sec 424> /pci@1c,600000/scsi@2/sd@0,0 1. c0t1d0 <SUN72G cyl 14087 alt 2 hd 24 sec 424> /pci@1c,600000/scsi@2/sd@1,0 2. c2t6080020FFF9E6D400000000300000000d0 <SUN-StorageTekNAS-4.11 cyl 2248 alt 2 hd 32 sec 256> 6scsiQ/.ci6ssdRg@0A0020fff-e@d,00000000B00000000

Note < Refer to Solaris $ocu(entation (or) (an page of iscsiadm co((and for /arious options t.at can 0e used to disco/er) /iew and configure iSCSI luns*

Stage B < Sun Cluster B*1 U , setup

!OT# < "lease refer to Sun Cluster Installation &anual for detailed step 0' step procedure to install Sun cluster Software* T.e following are t.e 0road o/er/iew of steps t.at are done < 1* On 0ot. t.e nodes) create 00&3 partition* T.en create U>S files'ste( on t.at partition and (ount using t.e (ountpoint glo+aldevices* # newfs /dev/rdsk/c0t1d0s1 # mkdir /globaldevices # mount /globaldevices &ake t.e following entr' into etc v#sta+*
/dev/dsk/c0t1d0s1 /dev/rdsk/c0t1d0s1 /globaldevices ufs 2 yes

2* For ,-. bases syste#s only < $o t.e following on 0ot. t.e nodes* a* &ake a cop' of t.e file as 0ackup* # cp /boot/solaris/filelist.ramdisk filelist.ramdisk.orig 0* Add t.e following entr' in t.e file* # echo etc/cluster/nodeid >> /boot/solaris/filelist.ramdisk
Sun Cluster Installsteps on node-a:

1* Install t.e sun cluster software* >ro( t.e installation C$ or location) run t.e co((and scinstall co((and to initiate t.e installation* # cd <CD mount> /sparc/stack/Solaris_sparc/Product/sun_cluster/Solaris_10/Tools # ./scinstall 2* C.oose to install ) 2nly t!is node at t!is time option B* "ro/ide t.e cluster na(e* >or e5a(ple) scracnas

,* After t.e co(pletion of t.e steps) t.e s'ste( re0oots* Oracle 10gR2 RAC wit. Sun Cluster on StorageTek !AS Appliance

10 Stage B < Sun Cluster B*1 U , setup

Sun &icros'ste(s) Inc*

* 8.en t.e s'ste( co(es 0ack up) run usr cluster +in scsetup and perfor( t.e following < a) Add t.e second node infor(ation* T.is is to aut.ori1e t.e ot.er node to 0e added to t.is cluster* 0) Add t.e transport ca0les and ena0le t.e( (if t.e pri/ate interconnects are directl' connected 0etween t.e .osts instead of connection /ia transport =unction)*
Sun Cluster installsteps on node-b:

1* Install t.e Sun Cluster software 0' issuing scinstall co((and* # cd <CD mount> /sparc/stack/Solaris_sparc/Product/sun_cluster/Solaris_10/Tools # ./scinstall 2* C.oose NAdd node to t!e e"isting clusterO option* B* "ro/ide t.e cluster na(e scracnas) t.e pri(ar' cluster node na(e node0a) t.e interconnect infor(ation* Sun Cluster aut.enticates t.e infor(ation) installs t.e 0inaries and re0oots t.e node* /uoru# de"ice configuration fro# any of the cluster node+ For e0a#(le1 fro# node2a : 1* Once 0ot. t.e s'ste(s are up) run scdidam co((and to find out t.e s.ared storage to 0e used as Guoru( de/ice*
[root@node-a]# scdidadm -L 1 node-a:/dev/rdsk/c0t0d0 /dev/did/rdsk/d1 2 node-a:/dev/rdsk/c0t1d0 /dev/did/rdsk/d2 3 node-a:/dev/rdsk/c2t6080020FFF9E6D400000000300000000d0 /dev/did/rdsk/d3 3 node-b:/dev/rdsk/c3t6080020FFF9E6D400000000300000000d0 /dev/did/rdsk/d3 4 node-b:/dev/rdsk/c0t0d0 /dev/did/rdsk/d4 5 node-b:/dev/rdsk/c1t0d0 /dev/did/rdsk/d5

Sun Cluster assigns 6de/6did6rdsk6dS nu(0ers for all t.e de/ices t.e cluster node sees* !ote t.at 6de/6did6rdsk6dB is assigned to t.e sa(e iSCSI 88! seen 0' 0ot. t.e nodes I t.oug. eac. node assigned uniHue controller nu(0er* 2* Using usr cluster +in scsetup co((and) C.oose to add quorum device and add d3* B* 8it. t.at step) t.e 0asic infrastructure for Sun Cluster B*1 Update , is read'* !OT# < &ake sure t.at t.e de/ices t.at are 0eing used are not s.ared wit. ot.er cluster en/iron(ents*

Stage , < Installing Oracle RAC >ra(ework for Sun Cluster

On both the nodes: 1* For $ 34C 5ased syste#s only < a) Sun Cluster d'na(ic lock (anager packages (U$7&) are reHuired to 0e added 0efore we install t.e Oracle RAC fra(ework*
# cd <CD mount>/ agents/components/SunCluster_Oracle_RAC_FRAMEWORK_3.1/ Solaris_10/Packages # pkgadd -d . SUNWscor SUNWscucm SUNWudlm SUNWudlmr

Oracle 10gR2 RAC wit. Sun Cluster on StorageTek !AS Appliance

11 Stage , < Installing Oracle RAC >ra(ework for Sun Cluster

Sun &icros'ste(s) Inc*

!ote < SU!8=scor and SU!8cscor are for Tapanese and c.inese c.aracter support* Add t.ose if necessar'* 0) Oracle RAC patc.es (ust 0e added at t.is ti(e* T.e patc. is a/aila0le wit. t.e Oracle Clusterware C$* T.is package is pro/ided 0' Oracle to interface wit. t.e /endor clusterware product* # cd <Oracle 10gR2 Clusterware CD mount >/racpatch # cp ORCLudlm.tar /tmp # cd /tmp # tar -xvf ./ORCLudlm # pkgadd -d . ORCLudlm 2* For ,-. 5ased syste#s only < a) Add Oracle RAC >ra(ework agent* !ote t.at t.ere is no need to add U$7& packages*
# cd <CD mount>/ agents/components/SunCluster_Oracle_RAC_FRAMEWORK_3.1/ Solaris_10/Packages # pkgadd -d . SUNWscor SUNWscucm

B* Add ;ardware Raid agent if t.e RAI$ is (anaged e5ternall' I suc. as Sun StorageTek !AS Appliance*
# cd <sparc/agents/components/SunCluster_Oracle_RAC_HWRAID_3.1/Solaris_10/Packages # pkgadd -d . SUNWschwr

,* >ro( one of t.e cluster node) run t.e scsetup co((and to add Sun 3luster support #or 2racle )A3 data ser/ice w.ic. t.en proceeds to add t.e rac0#rame,ork0rg resource group) place t.e( in (anaged state and t.en 0ring t.e( online* # scsetup * Cerif' t.e Sun Cluster Setup and /erif' t.e status of resources rac4#rame,ork) rac4udlm(for Sparc)) rac4!,raid are Online* (refer to Appendi5) # scstat -g

Stage

< Oracle #n/iron(ent Setup

T.e following are done to setup t.e en/iron(ent for t.e Oracle user on 0ot. t.e cluster nodes* &ake sure t.at sa(e groupid and userid is pro/ided on 0ot. nodes* "lease refer to t.e Oracle docu(entation for details*

Ty(e of Files ORAC7#Q;O&# L ORAQCRSQ;O&# OCR6Coting 7ocation

6ount (oint 6oracle6products6d0610*2*0 6oracle6oracrs610*2*0 6clusterware6ocr6ocr*d0f 6clusterware6/oting6/oting*d0f 6oradata

7ocation !AS (!>S) 6 7ocal !AS (!>S)

File Volu#e in N3$ oracle0in clusterware

RAC $ata0ase >iles

!AS (!>S)

oradata

On eac. node) (ake t.e directories and assign owners.ip and access rig.ts* # mkdir /orahome Oracle 10gR2 RAC wit. Sun Cluster on StorageTek !AS Appliance

12 Stage

< Oracle #n/iron(ent Setup

Sun &icros'ste(s) Inc*

# mkdir /oracle # mkdir /clusterware # groupadd -g 2000001 oinstall # groupadd -g 2000002 dba # useradd -u 5000001 -d /orahome -g "oinstall" -G "dba" -m -s /bin/ksh oracle # chown -R oracle:oinstall /orahome /oracle /clusterware # chmod -R 775 /orahome /oracle /clusterware # passwd oracle < Provide same password on both nodes> &ake entries in etc v#sta+ so t.at t.e /olu(es can 0e (ounted auto(aticall' upon s'ste( re0oots* &ake sure t.at t.e rsi1e and wsi1e are atleast B29* $irect I6O (ust 0e used on OCR) Coting and data0ase files* $irect I6O s.ould not 0e used for Oracle 0inaries (ount point* As oracle user) (ake sure to add t.e .ost na(es in 5 .r!osts * Ot.erwise) rs. (a' fail and causes N1ser 6quivalence (ailedO during Oracle:s runcluv#y co((and* 6ount O(tions : Ty(e of File Oracle Clusterware files (Coting and OCR) L Oracle $ata0ase >iles Oracle Clusterware 0inaries) Oracle $ata0ase 0inaries NF$ 6ount O(tions rw)0g).ard)nointr)rsi1eUB2?@A)wsi1eUB2?@A)protoUtcp) /ersUB)noac)forcedirectio rw)0g).ard)nointr)rsi1eUB2?@A)wsi1eUB2?@A)protoUtcp) /ersUB)noac)suid

8escri(tions for the NF$ #ount o(tions : Mount Options hard / soft Description If N.ardO (ounted) t.e Oracle ser/er does not cras. if t.e underl'ing file s'ste( is una/aila0le (eit.er due to planned or unplanned down ti(e for !AS appliance)* If NsoftO (ounted) t.en t.e Oracle ser/er (a' cras. if t.e data0ase /olu(es I especiall' for control and redo log /olu(es are una/aila0le* It is strongl' reco((ended to use N.ardO (ount for Oracle data0ases* T.e (ode of file s'ste( access* read2write or read2onl'* T.is option instructs t.e Oracle $ata0ase ser/er to .onor t.e set2uid 0it on files (ounted at t.e (ount point* If 'ou are placing an' of t.e Oracle 0inaries onto t.e StorageTek !AS Appliance t.en t.is setting (ust 0e used* T.is option (akes t.e connection interrupti0le* 8.en co(0ined wit. t.e N.ardO option) t.is para(eter ena0les t.e ad(inistrator to interrupt t.e application t.at is .olding t.e !>S connection* T.is option ena0les $irect I6O for all files under t.e (ount point and

rw / ro suid

intr

forcedirectio

Oracle 10gR2 RAC wit. Sun Cluster on StorageTek !AS Appliance

1B Stage

< Oracle #n/iron(ent Setup

Sun &icros'ste(s) Inc*

Mount Options

Description allows t.e 7og 8riter to efficientl' trans(it t.e data to storage wit.out t.e latenc' associated wit. splitting I6O into page si1e transfers* Reco((ended for data files and redo logs* Note7 T!is option is not to +e used on 2racle +inary location or 3luster )egistry (iles*

bg vers

T.is option indicate t.at (ount is to retr' in t.e 0ackground if t.e ser/er:s (ount dae(on does not respond* T.e N/ersUO option ena0les t.e ad(inistrator to specif' t.e !>S /ersion to use* T.ere are two /ersions of !>S< /ersion 2 and /ersion B* Cersion B supports additional file operations t.at /ersion 2 did not .a/e* It is t.e t'pe of I" connection to 0e used* Sun and Oracle Suggest using TC"* TC" enforces i(portant reHuire(ents) suc. as packet ordering and packet acknowledg(ent for reacting to /arious network ano(alies* T.ese are to set t.e read and write packet si1es* It s.ould 0e a (ultiple of 12 0'tes* >or oracle data0ases) set it to B2?@A (B29)* T.is !>S (ount option was introduced to address perfor(ance issues 0' utili1ing local file locking rat.er t.an !>S: !etwork 7ock &anager (!7&)* T.is .as t.e potential to dra(aticall' i(pro/e perfor(ance* Ti(eout in 1610 of t.e second* Specif'ing @00) it is @0 seconds*

proto

rsize & wsize llock

timeo

Sun StorageTek 000 !AS appliance .a/e t.e 0usiness continuit' pre(iu( features suc. as snaps.ot and re(ote replication w.ic. can 0e i(ple(ented for RAC data0ase* "lease refer to t.e docu(ents under .ttp<66www*sun*co(6storagetek6oracle

Stage @ < Oracle 10gR2 Clusterware Installation

All t.e steps need to 0e perfor(ed fro( one of t.e node I unless and ot.erwise specified* 1* 7oad t.e Oracle 10gR2 Clusterware C$ and run t.e cluster preinstall c.eck* $ ./runcluvfy.sh stage -pre crsinst -n node-a,node-b -verbose Since we are le/eraging Sun Cluster) all t.e necessar' cluster packages are c.ecked 0' t.e utilit'* T.e c.ecking would fail on CI" nodes 0ut t.at can 0e ignored at t.is ti(e* 2* >or 4A@ 0ased Solaris) it is (andator' to run t.e rootpre.s! fro( t.e Clusterware install C$ (or) i(age* # cd <location>/clusterware/rootpre # ./rootpre.sh B* Set t.e $IS"7AF /aria0le and launc. t.e uni/ersal installer %UI* $ ./runIntaller ,* In t.e cluster configuration page) it.e %UI displa's Nclusternode*0privO and Nclusternode20privO as pri/ate node na(es w.ic. are deri/ed fro( Sun Cluster* *In t.e su0seHuent page) c.ange t.e following a) ;ost I"6na(e of t.e pu0lic network as Npu+licO 0) clprivenet0 as NprivateO c) T.e rest of t.e pri/ate interfaces as Ndo not useO Oracle 10gR2 RAC wit. Sun Cluster on StorageTek !AS Appliance

1, Stage @ < Oracle 10gR2 Clusterware Installation

Sun &icros'ste(s) Inc*

@* C.oose 6"ternal )edundancy and pro/ide one location for OCR file* /clusterware/ocr/ocr.dbf ?* C.oose 6"ternal )edundancy and pro/ide one location for Coting disks* /clusterware/voting/voting.dbf Note : 6"ternal )edundancy is used 0ecause) t.e !AS appliance uses RAI$2 to stripe data across dri/es* If additional protection is reHuired) t.en it is reco((ended to create RAI$ groups out of different set of dri/es and create file/olu(e to store additional copies of clusterware files* A* Run ora8nst)oot.s! and root.s! w.en pro(pted for on 0ot. t.e nodes one at a ti(e and not to 0e run in parallel* T.e script for(ats t.e /oting and OCR files* On t.e second node) w.ile t.e root*s. configures t.e clusterware file) it atte(ps to configure CI" 0' doing a silent launc. of /ipca* $epending on t.e I" address (if t.e address is of R>C1-1A co(pliant)) t.e registration of resources suc. as CI") %$S and O!S (a' not occur* -* If t.e pre/ious step was run successful wit.out errors) skip t.is step* Ot.erwise) fro( one of t.e node) as root user) set t.e $IS"7AF para(eter and launc. /ipca utilit'*
# $ORA_CRS_HOME/bin/vipca

"ro/ide t.e correct CI" addresses for 0ot. t.e nodes* T.e %UI t.en registers t.e CI")%S$ and O!S resources to t.e CRS* T.en it starts t.ose resources* T.e Oracle Clusterware installation can 0e /alidated using t.e crs4stat co((and (after running t.e root*s. fro( t.e last node)* 8it. t.at step) t.e Oracle Clusterware installation is co(plete*

Stage ? < Oracle RAC $ata0ase 0inaries Installation

"lease refer to Oracle docu(entation for step20'2step details on Oracle RAC $3 installation* It is possi0le to install t.e Oracle data0ase 0inaries on eit.er s.ared .o(e on !AS or local to t.e nodes*

If a s.ared Oracle .o(e is used) t.en ORAC7#Q;O&# (ust 0e t.e sa(e across nodes* A soft link can 0e created w.ic. points to t.e !AS director' (or) use t.e sa(e (ount point* /oracle/products/db/10.2.0 As oracle user) initiate t.e uni/ersal installer for Oracle RAC data0ase 0inar' installation $ export DISPLAY=<node>:0.0 $ cd <Database binary location>/.runInstaller "ro/ide t.e following infor(ation w.en asked for < 1* VORAC7#Q;O&# location to install Oracle R$3&S 0inaries ( oracle products d+ *0.2.0)* &ake sure t.at t.e director' is e(pt' and no ot.er RAC or Oracle 0inaries are installed in t.at director'* 2* "ro/ide s'ste( group I eit.er Nd0aO or NoinstallO B* C.oose to install N#nterprise #ditionO ,* C.oose to install NRAC data0ase 0inariesO and (ake sure t.at 0ot. t.e nodes are listed and c.ecked* * Fou (a' c.oose to install N$ata0ase Software Onl'O at t.is ti(e*

Oracle 10gR2 RAC wit. Sun Cluster on StorageTek !AS Appliance

1 Stage ? < Oracle RAC $ata0ase 0inaries Installation

Sun &icros'ste(s) Inc*

T.e data0ase 0inaries would 0e installed in t.e director' location oracle products d+ *0.2.0 w.ic. is s.ared 0etween nodes* ;ence no re(ote cop' operation occurs* 8.en pro(pted) as root user) run t.e oracle products d+ *0.2.0 root.s! on 0ot. t.e nodes one after t.e ot.er* 8it. t.at step) t.e installation of Oracle RAC 0inaries is co(plete*

$ata0ase Creation
T.e last step is to create t.e Oracle RAC data0ase I w.ic. can eit.er 0e created using Nd+caO %UI tool or using (anual scripts* T.e control files and data0ase files are s.ared a(ount t.e RAC instances* #ac. instance will .a/e its own Online redo logs* At t.e e/ent of a node failure) t.e ot.er sur/i/ing node perfor(s t.e instance reco/er' w.ic. (akes Oracle RAC infrastructure .ig.l' a/aila0le* If d+ca is used) it will also (ake appropriate entries in listener.ora9 tnsnames.ora and sqlnet.ora files* T.e' are) 0' default) would 0e in :2)A3;64.2<6 net,ork admin director'* Refer to Appendi5 for sa(ple files* All t.e data0ase files suc. as control files) online redo logs and datafiles are stored in a single file/olu(e* T.at wa') it 0eco(es easier to add storage to one /olu(e* It is reco((ended to use anot.er file/olu(e to store arc.i/ed redo logs and >RA I prefera0l' using a different 7U!* T.e following are t.e reHuire(ents for t.e Oracle RAC $ata0ase* 1* #ac. instance .as its own redo log files (t.reads) and s.ares t.e control files and data0ase files* >or easier and efficient (anage(ent) it is sufficient to .a/e one large file /olu(e to store all t.e data0ase files and online redo logs* If additional cluster node is added) t.en an additional set of redo logs wit. a new t.read nu(0er are to 0e created and ena0led* 2* #ac. instance na(e and instance nu(0er (ust 0e uniHue* T.e :2)A3;64.2<6 rd+ms log director' (0' default) is used as repositor' for files suc. as alert logs and trace files* T.e file na(es t'picall' .a/e t.e instance na(e* B* T.e startup and s.utdown of listener and data0ase instances are .andled 0' Oracle Clusterware and not 0' Sun Cluster* To 0ring t.e( up auto(aticall' after s'ste( 0ootups) (ake t.e appropriate entr' into var opt oracle orata+ file < $ORACLE_SID:$ORACLE_HOME:<N|Y>

Oracle 10gR2 RAC wit. Sun Cluster on StorageTek !AS Appliance

1@

Sun &icros'ste(s) Inc*

Conclusion
Sun StorageTek !AS offers a perfect and eas' platfor( for Oracle RAC deplo'(ent for easier i(ple(entation and (aintenance* StorageTek !AS also offers iSCSI luns t.at can 0e used for data as well as Huoru( disk for Cluster* Using Sun Cluster B*1 wit. Oracle Clusterware pro/ides superior a/aila0ilit') scala0ilit') sta0ilit' and perfor(ance to t.e Oracle 10g RAC infrastructure*

Reference
Sun Cluster $ata Ser/ice for Oracle Real Application Clusters guide for Solaris OS ("art S A1-20 AB210) Sun StorageTek B20 !AS $ocu(entation (.ttp<66www*sun*co(6storagetek6nas6 B206) Sun StorageTek B20 Oracle Usage $ocu(entation (.ttp<66www*sun*co(6storagetek6oracle) Oracle 10*2*0*1 Clusterware $ocu(entation Oracle 10*2*0*2 $ata0ase $ocu(entation Solaris 10 $ocu(entation

Oracle 10gR2 RAC wit. Sun Cluster on StorageTek !AS Appliance

1? Appendi5

Sun &icros'ste(s) Inc*

Appendi5
/etc/hosts # cat /etc/hosts # # Internet host table # 127.0.0.1 localhost 172.20.98.85 node-a 172.20.98.86 node-b ## VIP Nodes 172.20.200.200 node-a-vip 172.20.200.201 node-b-vip For ,-.1 /boot/solaris/filelist+ra#disk # cat filelist.ramdisk etc/rtc_config etc/system etc/name_to_major etc/driver_aliases etc/name_to_sysnum etc/dacf.conf etc/driver_classes etc/path_to_inst etc/mach etc/devices/devid_cache etc/devices/mdi_scsi_vhci_cache etc/devices/mdi_ib_cache kernel platform/i86pc/biosint platform/i86pc/kernel boot/solaris.xpm boot/solaris/bootenv.rc boot/solaris/devicedb/master boot/acpi/tables etc/cluster/nodeid /etc/"fstab
# Oracle Binaries b20-5310a:ora10gbin /oracle nfs yes rw,bg,hard,nointr,rsize=32768,wsize=32768,vers=3,noac,proto=tcp,suid # OCR and Voting Files b20-5310a:clusterware /clusterware nfs yes rw,bg,hard,nointr,rsize=32768,wsize=32768,vers=3,noac,proto=tcp,forcedirectio # Database Files b20-5310a:oradata /oradata nfs yes rw,bg,hard,nointr,rsize=32768,wsize=32768,vers=3,noac,proto=tcp,forcedirectio

loghost

Oracle 10gR2 RAC wit. Sun Cluster on StorageTek !AS Appliance

1A Appendi5

Sun &icros'ste(s) Inc*

ifconfig co##and
# ifconfig -a lo0: flags=2001000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL> mtu 8232 index 1 inet 127.0.0.1 netmask ff000000 bge0: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 2 inet 172.20.98.85 netmask ffffff00 broadcast 172.20.98.255 groupname sc_ipmp0 ether 0:14:4f:2a:93:3a bge0:1: flags=1040843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4> mtu 1500 index 2 inet 172.20.200.200 netmask ffffff00 broadcast 172.20.200.255 bge1: flags=1008843<UP,BROADCAST,RUNNING,MULTICAST,PRIVATE,IPv4> mtu 1500 index 4 inet 172.16.0.129 netmask ffffff80 broadcast 172.16.0.255 ether 0:14:4f:2a:93:3b bge2: flags=1008843<UP,BROADCAST,RUNNING,MULTICAST,PRIVATE,IPv4> mtu 1500 index 3 inet 172.16.1.1 netmask ffffff80 broadcast 172.16.1.127 ether 0:14:4f:2a:93:3c clprivnet0: flags=1009843<UP,BROADCAST,RUNNING,MULTICAST,MULTI_BCAST,PRIVATE,IPv4> mtu 1500 index 5 inet 172.16.193.1 netmask ffffff00 broadcast 172.16.193.255 ether 0:0:0:0:0:1 lo0: flags=2002000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv6,VIRTUAL> mtu 8252 index 1 inet6 ::1/128 bge1: flags=2008841<UP,RUNNING,MULTICAST,PRIVATE,IPv6> mtu 1500 index 4 inet6 fe80::214:4fff:fe2a:933b/10 ether 0:14:4f:2a:93:3b bge2: flags=2008841<UP,RUNNING,MULTICAST,PRIVATE,IPv6> mtu 1500 index 3 inet6 fe80::214:4fff:fe2a:933c/10 ether 0:14:4f:2a:93:3c

>or 4A@) scstat 2g ( notice no racQudl( resources )

# scstat -g -- Resource Groups and Resources -Group Name ---------Resources: rac-framework-rg Resources --------rac_framework rac_hwraid

-- Resource Groups -Group Name ---------Group: rac-framework-rg Group: rac-framework-rg Node Name --------node-b node-a State ----Online Online

-- Resources -Resource Name ------------Resource: rac_framework Resource: rac_framework Resource: rac_hwraid Resource: rac_hwraid Node Name --------node-b node-a node-b node-a State ----Online Online Online Online Status Message -------------Online Online Online Online

Oracle 10gR2 RAC wit. Sun Cluster on StorageTek !AS Appliance

1-

Sun &icros'ste(s) Inc*

>or Sparc s'ste()

# scstat -g -- Resource Groups and Resources -Group Name ---------Resources: rac-framework-rg Resources --------rac_framework rac_udlm rac_hwraid

-- Resource Groups -Group Name ---------Group: rac-framework-rg Group: rac-framework-rg Node Name --------node-a node-b State ----Online Online

-- Resources -Resource Name ------------Resource: rac_framework Resource: rac_framework Resource: rac_udlm Resource: rac_udlm Resource: rac_hwraid Resource: rac_hwraid Node Name --------node-a node-b node-a node-b node-a node-b State ----Online Online Online Online Online Online Status Message -------------Online Online Online Online Online Online

Sparc) scstat co((and

# scstat ------------------------------------------------------------------- Cluster Nodes -Node name --------node-a node-b Status -----Online Online

Cluster node: Cluster node:

------------------------------------------------------------------- Cluster Transport Paths -Endpoint -------node-a:bge2 node-a:bge1 Endpoint -------node-b:bge2 node-b:bge1 Status -----Path online Path online

Transport path: Transport path:

------------------------------------------------------------------- Quorum Summary --

Oracle 10gR2 RAC wit. Sun Cluster on StorageTek !AS Appliance

20

Sun &icros'ste(s) Inc*

Quorum votes possible: Quorum votes needed: Quorum votes present:

3 2 3

-- Quorum Votes by Node -Node Name --------node-a node-b Present Possible Status ------- -------- -----1 Online 1 Online

Node votes: Node votes:

1 1

-- Quorum Votes by Device -Device Name ----------/dev/did/rdsk/d7s2 Present Possible Status ------- -------- -----1 1 Online

Device votes:

------------------------------------------------------------------- Device Group Servers -Device Group -----------Primary ------Secondary ---------

-- Device Group Status -Device Group -----------Status ------

-- Multi-owner Device Groups -Device Group -----------Online Status -------------

------------------------------------------------------------------- Resource Groups and Resources -Group Name ---------Resources: rac-framework-rg Resources --------rac_framework rac_udlm rac_hwraid

-- Resource Groups -Group Name ---------Group: rac-framework-rg Group: rac-framework-rg Node Name --------node-a node-b State ----Online Online

-- Resources --

Oracle 10gR2 RAC wit. Sun Cluster on StorageTek !AS Appliance

21

Sun &icros'ste(s) Inc*

Resource Name ------------Resource: rac_framework Resource: rac_framework Resource: rac_udlm Resource: rac_udlm Resource: rac_hwraid Resource: rac_hwraid

Node Name --------node-a node-b node-a node-b node-a node-b

State ----Online Online Online Online Online Online

Status Message -------------Online Online Online Online Online Online

------------------------------------------------------------------- IPMP Groups -Node Name --------IPMP Group: node-a IPMP Group: node-b Group Status ---------sc_ipmp0 Online sc_ipmp0 Online Adapter Status -----------bge0 Online bge0 Online

i$C$I co##and e0a#(les 9in the nodes: : #iscsiadm list discovery Discovery: Static: disabled Send Targets: disabled iSNS: disabled
To #na0le static configuration<

# iscsiadm modify discovery --sendtargets disable # iscsiadm modify discovery --static-config enable #iscsiadm add static-config iqn.198603.com.sun:01:000e0c6754fa.456E271F.iscsiqdsparc,172.20.98.198 iqn.1986-03.com.sun:01:000e0c6754fa.456f1153.iscsiqdx86,172.20.98.198 # iscsiadm list static-config
To disa0le static configuration <

# iscsiadm modify discovery --static-config disable

To ena0le d'na(ic (send targets) disco/er'

# iscsiadm modify discovery --sendtargets enable # iscsiadm add discovery-address 172.20.98.198:3260 # iscsiadm list discovery-address -v 172.20.98.198:3260
To disa0le d'na(ic (send targets) disco/er'

# iscsiadm modify discovery --sendtargets disable

Oracle 10gR2 RAC wit. Sun Cluster on StorageTek !AS Appliance

22

Sun &icros'ste(s) Inc*

Oracle Clusterware Verification : -bash-3.00$ crs_stat -t Name Type Target State Host -----------------------------------------------------------ora....e-a.gsd application ONLINE ONLINE node-a ora....e-a.ons application ONLINE ONLINE node-a ora....e-a.vip application ONLINE ONLINE node-a ora....e-b.gsd application ONLINE ONLINE node-b ora....e-b.ons application ONLINE ONLINE node-b ora....e-b.vip application ONLINE ONLINE node-b Setting t.e (isscount (as root user)
$ORA_CRS_HOME/bin/crsctl set css misscount 600
iSCSI co m m a n ds

# # # # #

iscsiadm iscsiadm iscsiadm iscsiadm iscsiadm

modify discovery --sendtargets {enable| disable} add discovery-address <NAS ip address>:3260 list discovery list discovery-address list discovery-address -v <NAS ip address>:3260

# iscsiadm modify discovery --static {enable | disable} # iscsiadm {add | remove} static-config <iscsi name>,<NAS ip address> # iscsiadm list static-config ;O43C7!<=O6!/tnsna#es+ora 9Clients & 43C Nodes: racdb1 = (DESCRIPTION = (ADDRESS_LIST = (ADDRESS = (PROTOCOL = TCP)(HOST = node-a)(PORT = 1521)) ) (CONNECT_DATA = (SID = racdb1) ) ) racdb2 = (DESCRIPTION = (ADDRESS_LIST = (ADDRESS = (PROTOCOL = TCP)(HOST = node-b)(PORT = 1521)) ) (CONNECT_DATA = (SID = racdb2) ) )

Oracle 10gR2 RAC wit. Sun Cluster on StorageTek !AS Appliance

2B

Sun &icros'ste(s) Inc*

;O43C7!<=O6!/listener+ora 943C Nodes:

LISTENER_node-a = (DESCRIPTION_LIST = (DESCRIPTION = (ADDRESS_LIST = (ADDRESS = (PROTOCOL = IPC)(KEY = EXTPROC)) (ADDRESS = (PROTOCOL = TCP)(HOST = node-a)(PORT = 1521)) ) ) ) SID_LIST_LISTENER_node-a = (SID_LIST = (SID_DESC = (SID_NAME = PLSExtProc) (ORACLE_HOME = /oracle/products/db/10.2.0) (PROGRAM = extproc) ) (SID_DESC = (GLOBAL_DBNAME = racdb) (ORACLE_HOME =/oracle/products/db/10.2.0) (SID_NAME = racdb1) ) ) LISTENER_node-b = (DESCRIPTION_LIST = (DESCRIPTION = (ADDRESS_LIST = (ADDRESS = (PROTOCOL = IPC)(KEY = EXTPROC)) (ADDRESS = (PROTOCOL = TCP)(HOST = node-b)(PORT = 1521)) ) ) ) SID_LIST_LISTENER_node-b = (SID_LIST = (SID_DESC = (SID_NAME = PLSExtProc) (ORACLE_HOME = /oracle/products/db/10.2.0) (PROGRAM = extproc) ) (SID_DESC = (GLOBAL_DBNAME = racdb) (ORACLE_HOME =/oracle/products/db/10.2.0) (SID_NAME = racdb2) ) ) STARTUP_WAIT_TIME_LISTENER=0 CONNECT_TIMEOUT_LISTENER=10 TRACE_LEVEL_LISTENER=OFF TRACE_DIRECTORY_LISTENER=/oracle/products/db/10.2.0/network/trace TRACE File_LISTENER=listener.trc LOG_DIRECTORY_LISTENER=/oracle/products/db/10.2.0/network/log LOG_FILE_LISTENER=listener.log

Oracle 10gR2 RAC wit. Sun Cluster on StorageTek !AS Appliance