EXPERT TIPS AND

TOOLS FOR
MANAGING ORACLE
RAC 12C
L E I G H TON N E L S O N
L E AD D B A, W O R L D W I DE
T E C H N O L O G IE S

2 . 2 5 .20 15
ABOUT ME

Oracle DBA
Oracle ACE
Speaker
Co-Author Expert Oracle Enterprise Manager Cloud Control 12c
RAC Attack Ninja

http://blogs.griddba.com
linkedin.com/pub/leighton-nelson/4/6b8/a7a
@leight0nn
AGENDA

Oracle RAC 12c Features

Management Tools
SRVCTL
CRSCTL
Oracle Enterprise Manager
Support Tools
ORAchk
TFA
ADR
PDBs and Services in RAC
Whats New in 12.1.0.2
ORACLE RAC 12C OVERVIEW

Flex ASM
Flex Clusters
ORACLE RAC 12C OVERVIEW

Flex ASM
Pre-12c ASM instances required 1-to-1 mapping to database
instances
Flex ASM instance not needed on all nodes
ORACLE RAC 12C OVERVIEW

Flex Clusters
Hub and Leaf Nodes
Leaf nodes require shared storage
Requires Flex ASM
Only Hub node host databases
MANAGEMENT TOOLS

SRVCTL enhanced to accept full-word options

New SRVCTL commands have full-word options only
Existing SRVCTL commands support single letters for backwards
compatibility
More intuitive
Single letter options confusing at times
Inconsistent
Allows for more options
MANAGEMENT TOOLS

srvctl modify service s oltp d orcl r orcl1 a orcl2 P

BASIC e SELECT m BASIC w 60 j SHORT B SERVICE_TIME
MANAGEMENT TOOLS

srvctl modify service s oltp d orcl r orcl1 a orcl2 P

BASIC e SELECT m BASIC w 60 j SHORT B SERVICE_TIME

srvctl modify service service oltp db orcl preferred

orcl1 available orcl2 tafpolicy BASIC failovermethod
NONE timeout 60 clbgoal SHORT rlbgoal SERVICE_TIME

Show single letter options using the compatibile option after -help
MANAGEMENT TOOLS

Specify input file for passing lengthy options using file option
>cat services.in
-service radwwt_basic -db radwwt -serverpool wwt_sp -
cardinality uniform -clbgoal SHORT -rlbgoal SERVICE_TIME -
role PRIMARY -policy AUTOMATIC verbose

>srvctl add service -file services.in

MANAGEMENT TOOLS

What-If Command Evaluation

The eval option allows commands to be simulated without
modifying the system
Available for both SRVCTL and CRSCTL
Supported SRVCTL Commands
srvctl add database
srvctl add service
srvctl add srvpool
srvctl modify database
srvctl start database
srvctl start service
srvctl stop database
srvctl stop service
MANAGEMENT TOOLS

What-If Command Evaluation

MANAGEMENT TOOLS

What-If Command Evaluation

MANAGEMENT TOOLS

Using CRSCTL some attributes can be modified online without a

restart
Modification of ora.* resources enforced
crsctl modify ora.* resource fails with CRS-4995 in 12.1.0.2 and above
(Doc ID 1918102.1)
Only user-defined resources can be modified
There are exceptions
CRS_STAT commands deprecated as of 11.2
Replaced with CRSCTL status resource
MANAGEMENT TOOLS

What if?
MANAGEMENT TOOLS

Show what would happen if a resource failed and cannot be restarted

on the local node
SRVCTL predict

CRSCTL eval fail

MANAGEMENT TOOLS

Display status of all databases running running in a particular home

srvctl status databaseDB -thishome

Display status of all databases running with same version

srvctl status databaseDB -thisversion
MANAGEMENT TOOLS

Oracle Enterprise Manager

Cloud Control 12c
EM Express 12c
MANAGEMENT TOOLS
MANAGEMENT TOOLS

Oracle Enterprise Manager

SUPPORT TOOLS

ORAchk aka RACcheck

Trace File Analyzer (TFA) Collector
Automatic Diagnostics Repository (ADR)
SUPPORT TOOLS

ORAchk provides health checks for Oracle stack

Databases (Single Instance and RAC)
Enterprise Manager Repository
E-Business Suite
ORAchk now included in 12c binaries
$ORACLE_HOME/suptools
Recommended to download latest version from MOS
ORAchk - Health Checks for the Oracle Stack (Doc ID 1268927.2)
SUPPORT TOOLS

When to run ORAchk

After Initial Oracle RAC Deployment
Before maintenance
After maintenance
At least once every three months
SUPPORT TOOLS

Setup ORAChk to check start automatically

>./orachk auto_restart
Configure expect utility or password-less sudo
see http://expect.sourceforge.org
Configure scheduled collection
>./orachk -id DBA -set
"NOTIFICATION_EMAIL=leighton.nelson@wwt.com;AUTORUN_SCHEDU
LE=4,8,12,16,20 * * *;AUTORUN_FLAGS=-profile
dba;COLLECTION_RETENTION=30"
SUPPORT TOOLS

Use profiles to run specific checks

>./orachk profile <profilename>
SUPPORT TOOLS

Trace File Analyzer

Installed in $GI_HOME/suptools
Executed from $GI_HOME/tfa/bin
Collects diagnostics, traces, log files for DB, Clusterware, OS across
the cluster
Can be configured to trigger collection automatically
Starts automatically on reboot
SUPPORT TOOLS

Trace File Analyzer

Ability to analyze individual components
$ tfactl analyze node racdb01,racdb02 -comp crs -since 1h
Integrates with oratop, oswatcher etc
Some components require root or sudo access
Repository stored in GI_BASE/tfa/repository
Default size of 1GB
SUPPORT TOOLS

Trace File Analyzer

SUPPORT TOOLS

Automatic Diagnostics Repository for G.I.

Clusterware log files managed and treated like database related
logs, trace
$GI_BASE/diag/
adrci interface
SUPPORT TOOLS

ADRCI for CRS logs

SUPPORT TOOLS

ADRCI for CRS logs

SUPPORT TOOLS

Cluster Health Monitor (CHM/OS)

Introduced since 11.2.0.1
Collects OS level metrics using OS APIs
Stores data in management repository (GIMR)
Integrated with QoS and Oracle Enterprise Manager
SUPPORT TOOLS

Cluster Health Monitor (CHM/OS)

Processes
osysmond
ologgerd
oclumon
Clusterware Resource
$ crsctl status resource ora.crf -init
NAME=ora.crf
TYPE=ora.crf.type
TARGET=ONLINE
STATE=ONLINE on ngdcracdb01
PDBS AND SERVICES IN RAC

A default service is created with each PDB

Default service is the same global name as the PDB
Default service is NOT registered with Clusterware
Create additional services for application access
PDBs are mounted by default
Starting a service opens the PDB
SINGLETON service opens PDB on the node it runs on
Use UNIFORM service to access PDB on more than one node
Stopping a service doesnt close the PDB
PDB remains open but inaccessible
Remember PDBs exist on all RAC nodes running CDB!
Specify INSTANCES keyword when modifying PDB state
MANAGING PDBS

SQL Plus
SQL Developer
Oracle Enterprise Manager Cloud Control 12c
Oracle Enterprise Manager Express 12c
WHATS NEW IN 12.1.0.2

Ping Targets
Probe nodes in Virtual Cluster for network failure
Detects reachability for targets on public network
Rapid Home Cloning
Store gold images of database software in RHP server
Deploy images to dev/test environments
ASM Filter Driver
Consistent naming of ASM disks across cluster (replaces ASMLib)
Filters I/O to ASM disks to prevent accidental overwriting
WHATS NEW IN 12.1.0.2

Ping Targets
Configured during initial cluster install

Set after installation using srvctl

[root@racdb01 ~]# srvctl modify network -k 1 -pingtarget "10.255.224.198"
[root@racdb01 ~]# srvctl config network
Network 1 exists
Subnet IPv4: 10.255.224.0/255.255.255.0/eth0, static
Subnet IPv6:
Ping Targets: 10.255.224.198
Network is enabled
Network is individually enabled on nodes:
Network is individually disabled on nodes:
WHATS NEW IN 12.1.0.2

Rapid Home Cloning

Store gold images of database software in RHP server
rhpctl import image -image DB12102_base -path
/u01/db/oracle/12.1.0.2/db_1/ -imagetype ORACLEDBSOFTWARE
Deploy images to dev/test clusters
rhpctl add workingcopy -workingcopy DB12102Home -image
DB12102_base
Fast and standardized deployment of Oracle binaries
WHATS NEW IN 12.1.0.2

ASM Filter Driver

The verdict is still out
http://flashdba.com/2014/07/25/oracle-12-1-0-2-asm-filter-driver-first-
impressions/
REFERENCES

Oracle RAC 12c

http://www.oracle.com/technetwork/database/options/clustering/rac-
wp-12c-1896129.pdf
Oracle Clusterware Administration and Deployment Guide 12c
Release 1 (12.1)
Oracle RAC 12c Operational Best Practices
http://www.slideshare.net/MarkusMichalewicz
Oracle RAC, Oracle Data Guard, and Pluggable Databases: When
MAA Meets Oracle Multitenant
http://www.ludovicocaldara.net/dba/maa-multitenant-presentation/
FREE TRIAL--RESOLVE PERFORMANCE
ISSUES QUICKLY

Try Database Performance Analyzer FREE for 14 days

Insight into overhead caused by delays in RAC coordination and
cache operations
Improve root cause of slow performance
Quickly identify root cause of issues that impact end-user response time
See historical trends over days, months, and years
Understand impact of VMware performance
Agentless architecture, installs in minutes

www.solarwinds.com/dpa-download/

2015 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED. 41

QUESTIONS?

2015 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED. 42

 THANK YOU

The SOLARWINDS and SOLARWINDS & Design marks are the exclusive property of SolarWinds Worldwide, LLC, are registered with the U.S.
Patent and Trademark Office, and may be registered or pending registration in other countries. All other SolarWinds trademarks, service marks,
and logos may be common law marks, registered or pending registration in the United States or in other countries. All other trademarks
mentioned herein are used for identification purposes only and may be or are trademarks or registered trademarks of their respective companies.

2015 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED. 43