Oracle

Real Application Clusters

(RAC) – Roadmap for New Features

Markus Michalewicz, Senior Director of Product Management,

Database High Availability & Scalability Solutions, ST Development

December 7, 2018
Markus.Michalewicz@oracle.com
@OracleRACpm
http://www.linkedin.com/in/markusmichalewicz
http://www.slideshare.net/MarkusMichalewicz
Copyright © 2018, Oracle and/or its affiliates. All rights reserved. |
My Sangam18 Presentations
•  Friday, December 7: •  Saturday, December 8:

– 1:30pm / 02 - Ball Room 3 Grfloor – 9:00am / 02 - Ball Room 3 Grfloor

Oracle Real Application Clusters (RAC) Oracle MAA
Roadmap for new Features (Maximum Availability Architecture)
– 10:00am / 08 - CORAL First Floor
– 4:35pm / 02 - Ball Room 3 Grfloor Fireside Chat with Oracle RAC Experts
Oracle Sharding 18c for Data
Sovereignty and Massive Linear – 5:50pm / 01 - Ball Room 2 Grfloor
Scalability (Oracle) DBA Skills
to Have, to Obtain and to Nurture

Copyright © 2018, Oracle and/or its affiliates. All rights reserved. |

Safe Harbor Statement
The following is intended to outline our general product direction. It is intended for
information purposes only, and may not be incorporated into any contract. It is not a
commitment to deliver any material, code, or functionality, and should not be relied upon
in making purchasing decisions. The development, release, timing, and pricing of any
features or functionality described for Oracle’s products may change and remains at the
sole discretion of Oracle Corporation.

Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | 4

Program Agenda

1 Overview
2 Better Management
3 Better Availability
4 Better Scalability & Performance
5 The Road Ahead

Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | 5

Program Agenda

1 Overview
2 Better Management
3 Better Availability
4 Better Scalability & Performance
5 The Road Ahead

Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | 6

 Oracle RAC’s Journey into the Autonomous Database
20-years of continuous innovation*
Oracle 18c

Scalable Sequences
Continuous Application Availability
Oracle RAC Sharding
Cluster Domains
Cluster Health Advisor (CHA)
RAC Reader Nodes
Application Continuity (AC)
Oracle Flex ASM & Flex Clusters
Rapid Home Provisioning (RHP)
Cluster Health Monitor (CHM)
Oracle Quality of Service Management (QoS)
Policy-Based Cluster Management
Oracle RAC One Node & RACcheck
Oracle ASM Cluster File System (ACFS)
Oracle Grid Infrastructure (GI)
UCP and OCI Load Balancing Support for RAC
Cluster Verification Utility (CVU)
Cluster-Managed Services
Oracle Clusterware
Oracle Automatic Storage Management (ASM)
Oracle Real Application Clusters (RAC) Oracle 9i * Documented features list is selective; 20 years include development time

Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | 7
Oracle RAC Innovation Spin-Off 1
Oracle Autonomous Health and the Autonomous Health Framework (AHF)

http://oracle.com/goto/ahf

Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | 8

Oracle RAC Innovation Spin-Off 2
Oracle Fleet Patching & Provisioning (FPP) and Gold Image Management
formerly Oracle Rapid Home Provisioning (RHP)

http://oracle.com/goto/fpp
(http://oracle.com/goto/rhp)

Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | 9

Oracle RAC Innovation Spin-Off 3
Application Continuity & Continuous Application Availability – the new MAA* standard

http://oracle.com/goto/ac

* MAA = Maximum Availability Architecture

Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | 10

Oracle Autonomous Database Highlights

1 2 3 4 5

Self-Driving Self-Scaling Self-Repairing

Automates database and Scales online for Protects from all
infrastructure management, highest performance downtime including
monitoring, tuning and lowest cost planned maintenance

Enabled by Applied Machine Learning

Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | 11

Oracle RAC Development Focus Areas

Efficient management
Better scalability
for Better availability
& performance
large scale deployments

Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | 12

Program Agenda

1 Overview
2 Better Management
3 Better Availability
4 Better Scalability & Performance
5 The Road Ahead

Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | 13

What If…You have Hundreds of Environments
And what if…
– Software Installation
– Storage configuration
– Diagnostics setup

… would have to be performed only once
and can then be re-used multiple times?

… allowing you to save many hours
performing these tiring tasks?

Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | 14

Cluster Domain-Based Management
Simplification and efficiency through centralization
Facilitating …
–  Fleet Patching & Provisioning (FPP) /
formerly Rapid Home Provisioning (RHP)
–  Autonomous Health Framework (AHF)
–  Automatic Storage Management (ASM)

and hosted on a dedicated cluster
– the Domain Services Cluster (DSC) –
FPP AHF
all three management tasks can be
centralized and diagnostics can be optimized
Cluster
Domain Domain Services Cluster for Member Clusters in a Cluster Domain.

Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | 15

Cluster Domains @BT

Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | 16

Context – Existing RAC and Database Services in BT

•  Large scale, on-premises Enterprise Cloud

•  Oracle Database and RAC estate continuing to grow
•  Oracle RAC databases mostly on 12.1 or 12.2
•  12.2 clusters only since April 2018
•  Lots of automation already but …

•  We need smarter ways to

–  Deliver RAC clusters more quickly
–  Administrate more efficiently

•  Goal: Reduce overheads of many RAC clusters

© British Telecommunications plc
Cluster Domains – Why is this architecture attractive to us?

•  I/O Server
–  Replace hundreds of independent pools of storage with centralised pools
–  Increase storage on member clusters without infrastructure changes

•  Rapid Home Provisioning

–  Centralised management of Oracle software for installation, patching, upgrading

•  Autonomous Health Framework

–  Replace many independent management repositories with one

© British Telecommunications plc

What’s Next for Cluster Domains?
1.  Transitions (*18c+)
2.  Domain Services
Cluster Availability
3.  and Performance

Planned Transitions

Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | 19

Better Management Thanks to Your Feedback

Configure ASM on NFS

$ORACLE_HOME/gridSetup.sh

Separate Diskgroup for Grid

gridSetup and zip-based install ASM Management for Infrastructure Management
for Oracle Grid Infrastructure NFS-based Clusterware files Repository (GIMR)
NEW: RPM-based installs for the for easier management and allows for more flexibility during
Oracle Database and Oracle Client thereby better availability. Grid Infrastructure Installation

Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | 20

Adapting to Your Feedback
•  Desupport of Direct File System Placement for Oracle Clusterware Files
–  Introduced with Oracle Clusterware 12c Rel. 2 (12.2.0.1)
–  Effective with Oracle Clusterware 18c
–  Desupport revoked effective with Oracle Clusterware 19c

•  Oracle Grid Infrastructure Management Repository (GIMR)

–  Around since Oracle Grid Infrastructure 11g Release 2
–  Automatic Installation of the GIMR introduced with Grid Infrastructure 12.1.0.2
–  Separate diskgroup installation introduced with Grid Infrastructure 12c Release 2
–  Automatic install revised for Oracle Grid Infrastructure 19c
•  Plans foresee a GIMR installation outside of the Oracle Grid Infrastructure home for Standard Clusters
•  Centralized GIMR hosting on a Domain Services Cluster (for Member Clusters) remains unchanged

Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | 21

Oracle Flex Cluster – A Brief Review

Flex Cluster Massive Parallel Query Oracle RAC Oracle RAC Reader Nodes

Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | 22

Flex Cluster – Changes Down the Road

Flex Cluster Massive Parallel Query Oracle RAC Oracle RAC Reader Nodes
Leaf nodes deprecated deprecated to be implemented on Hub nodes

Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | 23

Deprecation = Room to Focus on Innovation
•  Oracle Flex Cluster
–  Introduced with Oracle Clusterware 12c Release 1 (12.1.0.1)
•  Target use case: Management of applications and database in the same cluster
•  Databases are meant to be hosted on HUB nodes, applications on Leaf nodes

–  Oracle Clusterware 12c Release 2 enabled two more uses cases:

1.  Massive Parallel Query Oracle Real Application Clusters (Oracle RAC)
2.  Oracle RAC Reader Nodes

–  Effective with Oracle Clusterware 18c

•  Leaf nodes as part of the Flex Cluster architecture have been deprecated
•  The “Massive Parallel Query Oracle RAC” use case has been deprecated
•  The “Oracle RAC Reader Nodes” use case remains and will be supported on HUB nodes instead

–  HA capabilities provided by Leaf nodes will be made available on Hub nodes as applicable
•  The desupport of Leaf nodes and their respective uses cases is currently planned for Oracle Clusterware 19c

Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | 24

Program Agenda

1 Overview
2 Better Management
3 Better Availability
4 Better Scalability & Performance
5 The Road Ahead

Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | 25

Three Availability Focus Areas

Application Continuity
Zero Brownout Zero Downtime
& Continuous Application
Reconfiguration Maintenance
Availability

Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | 26

Innovation Under the Hood
Multiple projects and features contribute to Zero Brownout Reconfiguration
1.5x
faster

4x
faster •  Oracle RAC reconfiguration times reduced by
–  4x between Oracle RAC 11.2.0.4 and 12 Rel. 2
–  1.5x between Oracle RAC 12c Rel. 2 & 18c
–  for a total of 6x improvements
between Oracle RAC 11.2.0.4 and 18c

Maximum Availability Architecture
•  Selection of contributing features:
–  Remastering Slaves (*12.1)
–  Support for 100 LMS’s (*12.2)
–  More Dynamic Remastering

Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | 27

Contributing Features
•  Remastering Slaves (1 slave per LMS)
–  Starting with Oracle RAC 12.1, the LMS offloads heavy remastering work to the slave
–  This improves LMS’s responsiveness for Cache Fusion requests during remastering

•  Support for 100 LMS’s – change in default value
–  Oracle RAC 12.2 supports up to 100 LMS’s (names: LMS0-LM99) as opposed to 35
–  On larger systems (lots of CPU, large SGA), more LMS’s will start by default
–  More LMS’s means better reconfiguration time without any impact during runtime

•  More Dynamic Remastering (DRM)
–  Starting with Oracle RAC 19c, DRM is planned to more adaptively consider the overall system state

Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | 28

Innovation on the Way to the Autonomous Database
Introducing Database Reliability Framework (DRF) (Working Name)
•  A proactive and automatic monitoring and correction framework
–  Some functionality first introduced in Oracle RAC 12c
–  Current framework used in Oracle RAC 18c; further enhancements planned for Oracle Database 19c

–  Monitors various (currently ~50) metrics across different layers continuously
•  Shares and considers information globally, but acts locally
–  Detects problems before any disruption of service occurs
•  v$ tables provide logs showing current system status and history of issues detected
–  Identifies root cause accurately, based on current system situation
•  Uses a combination of metrics to predict potential issues and identifies root cause(s)
–  Resolves problems with minimum disruption, ideally before it happens
•  Takes preventative action based on identified root cause
–  Serializes actions across the cluster to minimize resolution impact
•  Corrective actions are performed on per resource basis

Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | 29

Patching Improvements
•  OJVM is Oracle RAC rolling patch enabled with Oracle RAC 18c (18.4)
– Non-Java services are available at all times
– Java services are available all the time, except for a ~10 seconds brownout
•  No errors are reported during the brownout

•  Zero-Downtime Oracle Grid Infrastructure Patching (*18.3)

– Patch Oracle Grid Infrastructure without interrupting database operations
– Patches are applied out-of-place and in a rolling fashion with one node being patched
at a time while the database instance(s) on that node remain up and running
– Supported for Oracle RAC and RAC One Node clusters with two or more nodes

Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | 30

Program Agenda

1 Overview
2 Better Management
3 Better Availability
4 Better Scalability & Performance
5 The Road Ahead

Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | 31

Innovation Under the Hood
Multiple projects and features contribute to Better Scalability & Performance

•  Oracle RAC performance has improved up to

5x
faster –  5x between Oracle RAC 11.2.0.4 and 18.1
especially for high contention workloads
•  Selection of contributing features:
–  Leaf Block Split Optimizations (*12.2)
–  Scalable Sequences (*18c)
•  Exadata-based optimizations:
–  Undo RDMA-Read (*18c)
–  “Smart Fusion Block Transfer” (*12.2)
–  ExaFusion Direct-to-Wire OLTP Protocol (*12.2)

Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | 32

Oracle RAC Performance Features
Over two decades of innovation • Session Affinity
• PDB & Services Isolation
• Service-Oriented Buffer Cache
• Leaf Block Split Optimizations
• Self Tuning LMS
18c
• Multithreaded Cache Fusion
• ExaFusion Direct-to-Wire Protocol
• Smart Fusion Block Transfer • Scalable Sequences
• Undo RDMA-Read
• Commit Cache
• Connection Load Balancing • Database Reliability Framework
• Load Balancing Advisory
• Cluster Managed Services
12c
• Automatic Storage Management
11g
9i 10g • Universal Connection Pool (UCP) Support for Oracle RAC
• Support for Distributed Transactions (XA) in Oracle RAC
• Parallel Execution Optimizations for Oracle RAC
• Automatic Undo Management
• Affinity Locking and Read-Mostly Objects
• Cache Fusion
• Reader Bypass
• Oracle Real Application Clusters
• Flash Cache

Copyright © 2018, Oracle and/or its affiliates. All rights reserved. |

Contention – The Basics
https://www.slideshare.net/MarkusMichalewicz/oracle-rac-internals-the-cache-fusion-edition
write
write
write write write
write write write

Scaling out, contention can occur

Contention can occur in any multi- From a contention perspective,
between instances
user system (even in SI databases) the number of nodes is irrelevant.
(not only within an instance).

Note: for scalability, only write/write contention needs to be considered.

Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | 34

Contention – Considerations
https://www.slideshare.net/MarkusMichalewicz/oracle-rac-internals-the-cache-fusion-edition

Sequence

REDO

Contention can affect related data as

Frequent transactional changes to Pending redo must be written to much as it can affect the user data.
the same data blocks may result in log before the block can be Right growing indexes and
“write hot spots” transferred index contention are common.
In 99% of OLTP performance issues,
write hot spots occur on indexes.

Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | 35

Contention – The Solutions

ExaFusion Smart Fusion Block Transfer 3x faster Right-Growing Index

Direct-to-Wire OLTP Protocol Improves Cache Fusion latency by performance due to
allows the database to directly
Leaf Block Split Optimizations,
allowing LMS to serve dirty blocks Scalable Sequences,
call into the InfiniBand HW. as soon as a REDO flush is initiated Commit Cache

Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | 36

Undo RDMA-Read & Commit Cache
•  Undo RDMA-Read (*18c)
UNDO UNDO
–  In some workloads, more than half of the remote Inst Inst Inst
reads are for Undo Blocks to satisfy read consistency 1 RDMA 2 RDMA 3
–  Undo Block RDMA-read uses RDMA to directly and
rapidly access UNDO blocks in remote instances
•  Avoids having to activate a remote LMS to perform the operation

•  Commit Cache (*18c)

–  The Commit Cache maintains an in-memory, simple
hash table of recent transactions on each instance
–  Remote LMS directly reads the commit cache and
sends back commit times for requested transactions
•  Replaces having to send entire 8K transaction table block
•  Customer test results:
–  Up to 57% of the cache-fusion block traffic was eliminated
–  cluster wait time down by 22%; DB CPU time down by 7%

Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | 37

Scaling the World’s Most Complex Enterprise Workloads
•  Due to its market leading Cache Fusion algorithm, Oracle RAC scales

1.  any feature – e.g. Pluggable Databases, Oracle In-Memory and Oracle Data Guard

2.  most enterprise applications – e.g. Ebusiness Suite, SAP, Oracle Hospitality

3.  nearly all custom applications as used by many of Oracle’s 15000 RAC customers

•  Without the need for significant application changes
•  Especially on Oracle Exadata Database Machines
•  Designed to support Oracle Autonomous Database Cloud

Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | 38
Why Oracle Sharding?
•  Some applications hugely benefit from “data dependent routing”
–  Data dependent routing means that users are routed to where the data resides
–  Routing can be performed within an Oracle RAC database or across independent databases
–  Oracle Sharding is Oracle’s fully integrated data dependent routing solution

•  Oracle RAC Sharding
–  Affinitizes table partitions to instances
•  for better cache utilization and reduced block pings between instances
–  Takes advantage of direct routing API of Sharding:
•  Requests that a specify sharding key are routed to the instance that logically holds the corresponding partition
•  Requests that do not provide a partition key will be routed based on the default load balancing policy
–  Adding a sharding key improves OLTP performance requires no changes to the database schema
•  alter system enable affinity <TableName>;

Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | 39

Program Agenda

1 Overview
2 Better Management
3 Better Availability
4 Better Scalability & Performance
5 The Road Ahead

Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | 40

The Road Ahead Leads into the Autonomous Database Cloud
•  Future scalability & performance improvements
–  Tailor to scaling well within Exadata dimensions (“scale linear across 64 nodes, not 200”)
–  Are designed to meet ADB performance requirements and will grow as ADB enhances
–  Will leverage RDMA technology for server-less communication
–  Plan to use RoCE as the next-generation network for the cloud
•  Details in MOS note “Oracle RAC Interconnect Protocols – Support and Roadmap (ID 2434852.1)”
–  Will substitute storage access with network-based access to data on remote nodes
–  Are likely to utilize NVM for storage on independent servers (eliminate shared disk)

•  Future availability improvements

–  Will focus on reducing re-configuration times (brownouts) further to come closer to “zero”
–  Will provide even more ways to perform maintenance & admin tasks with no downtime

Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | 41

Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | 42