Exadata Evolution from V1 to X3 with

Deep Dive into the features

Date (2/7/13)
About Me

Satyendra Kumar Pasalapudi

Communities Knowledge Education

Practice Manager - Apps Associates
Saring
Co Founder & Vice President – All India Oracle Users Group
13+ Years of Exp in Oracle Technologies
Exadata Certified Professional
Speaker at Open World, IOUG Collaborate, AIOUG Sangam

satyendra.pasalapudi@appsassociates.com

Most of the slides source is oracle.com

Agenda

• Problem Definition
• Exadata Evolution from V1 to X3
• Exadata X3 What’s New
• Exadata Architecture
• Exadata Features ( V1 to X3)
• Exadata Smart Flash Cache Write - Back
Exadata Evolution
 Exadata Evolution – Features

Exadata V1 - Smart Scan / InfiniBand Scaleout

Exadata V2 - Smart Flash Cache, Storage Index, Columnar
Compression
Exadata X2 - Hardware DB Encryption , Automatic Service Request,
Data Mining Offload, Smart Flash Cache
Exadata X3 - Write back Flash Cache
How does the database process OLTP Caching

• Cache data in memory from disk to achieve fast query response

• For OLTP/ DSS workloads, memory should be big enough to hold as
much as possible

SGA
100% cache hit ratio is ideal
for OLTP workloads
Buffer
Cache
 How does the database process OLTP?

• CPU time is mainly consumed by I/O wait

Cache miss Cache Hit

OLTP performance issues

1. Data Size is 1. Huge amounts of data

increasing... 2. Not possible to cache

all data

SGA

Buffer
2. User Volume is Cache
increasing...
3. Many I/O operations
 Typical Problem of busy systems

800 I/O bottleneck 16.00

Low Throughput and Slow Response
TPS(Normal-100Thread=>100)

700 Low CPU usage 14.00

Response Time(Normal-
600 12.00

100Thread=>1.00)
500 TPS RES 10.00

400 8.00

300 6.00

200 4.00

100 2.00

0 0.00
Thread 100 200 300 400 500 600 700 800
Exadata Evolution
Exadata X3 Database In-Memory Machine
Exadata Integrates all into one
Oracle Exadata X3 – Database In-Memory Machine Systems

Oracle Exadata X3 Database In-Memory Machine systems leverage next-

generation technologies to deliver significant performance enhancements,
including:
• Four times the Flash memory capacity of the previous generation; with up to 40
percent faster response times and 100 GB/second data scan rates.
• 20 times more capacity for database writes through updated Exadata Smart
Flash Cache software.
• 33 percent more database CPU cores in the Oracle Exadata X3-2 Database In-
Memory Machine, using the latest 8-core Intel® Xeon E5-2600 series of
processors;
Oracle Exadata X3 – Database In-Memory Machine Systems
Generation Advances
Key Exadata Innovations
 Exadata Architecture

DB Server ( Single DB Server RAC DB Server RAC

Instance) Node1 Node 2

DB Instance DB Instance DB Instance

DBRM DBRM DBRM

ASM ASM ASM Single ASM Cluster

LIBCELL LIBCELL LIBCELL

iDB Protocal
InfiniBand Storage Switch / Network Over
Infiniband

Cell Server with Exadata Cell Server with Exadata Cell Server with Exadata
Storage server software Storage server software Storage server software

RS OEL RS OEL RS OEL

MS
Smart Flash
Cache MS
Smart Flash
Cache MS
Smart Flash
Cache
CellCLI/
IORM IORM IORM DCLI
Exadata Smart Scans

• Exadata cells implement smart scans to

greatly reduce the data that needs to be
processed by database
• Only return relevant rows and columns to
database
• Offload predicate evaluation
• Data reduction is usually very large
• Column and row reduction often decrease
data to be returned to the database by 10x
• Join Filtering
• Bloom filters used for join filtering in storage

• Telco wants to identify customers that

spends more than Rs 200 per phone call
• The information about these customer
occupies 2 MB in a Tera Byte Table
Traditional Scan Processing
Exadata Smart Scan Processing
 Exadata Smart Scan Query Example

Optimizer Exadata Storage

What were my Chooses
sales yesterday? Partitions and
Grid
Indexes to
Access
Oracle Database
Grid Scan compressed
blocks in
partitions/indexes
Select
Retrieve sales
sum(sales)
amounts for
where
Sept 24
Date=’24-Sept’

10 TB scanned
SUM 1 MB returned to servers
Exadata Smart Scan

Query Processing Query Processing in Exadata

Tradition Databases

CPU CPU
Select
customer_
Memory iDB
Memory name from protocal

calls where
Network Network
amount >
Amount >
100 100
I / O Controller I / O Controller

Disk Disk
 Exadata Architecture

DB Server ( Single DB Server RAC DB Server RAC

Instance) Node1 Node 2

DB Instance DB Instance DB Instance

DBRM DBRM DBRM

ASM ASM ASM Single ASM Cluster

LIBCELL LIBCELL LIBCELL

iDB Protocal
InfiniBand Storage Switch / Network Over
Infiniband

Cell Server with Exadata Cell Server with Exadata Cell Server with Exadata
Storage server software Storage server software Storage server software

RS OEL RS OEL RS OEL

MS
Smart Flash
Cache MS
Smart Flash
Cache MS
Smart Flash
Cache
CellCLI/
IORM IORM IORM DCLI
 Exadata Hybrid Columnar Compression ( From V2)

• Data is organized and compressed by column

• Dramatically better compression

• Speed Optimized Query Mode for Data

Warehousing
Query

• 10X compression typical

• Runs faster because of Exadata offload!

• Space Optimized Archival Mode for

infrequently accessed data
• 15X to 50X compression typical

Faster and Simpler

Backup, DR, Caching,
Reorg, Clone Benefits Multiply

25
Data Mining Model Scoring ( From X2)

Offload of Data Mining Model Scoring

• Data Mining model scoring is offloaded to Exadata. This makes the

deployment of data warehouses on Database Machine an even
better and more performant data analysis platform.
• All data mining scoring functions (e.g., prediction_probability) are
offloaded to Exadata for
processing. This will not only speed warehouse analysis but
reduce database server CPU consumption and the I/O load between
the database server and Exadata storage.
Automated Service Request (ASR) ( From X2)

FRU – Field Replaceable Unit

Smart Flash Log ( From X2)
Smart Flash Log ( From X2)
Exadata Smart Flash Cache

• Exadata has 22.4 TB of flash

• 56 Flash PCI cards avoid disk
controller bottlenecks

• Intelligently manages flash

• Smart Flash Cache holds hot data
• Gives speed of flash, cost of disk

• Exadata flash cache achieves:

• Over 1 million IO/sec from SQL (8K)
• Sub-millisecond response times
• 50 GB/sec query throughput
Exadata Smart Flash Cache – Write Operation ( before X3 )

DB

ACKNOWLEDGEMENT
1
Exadata Cell Server
3
Exadata Smart
Flash Cache

MS OEL
RS
IORM
2 4
Exadata Smart Flash Cache – Read

Cached Data Un Cached Data

Read Request

DB

Read Request
DB

1 3 1 2
Exadata Cell Server
Exadata Cell Server

Exadata Smart Exadata Smart

Flash Cache Flash Cache

MS OEL MS OEL
RS RS

IORM IORM 3
2
 Exadata Smart Flash Cache Write - Back

• Flash cache provides the enhanced read and write performance.

• From Exadata storage software version 11.2.3.2.0 writes will also go to Flash
cache.
• This means writes first happen on Flash cache and the database gives
acknowledgment.
• Cell software then writes them back onto hard disk when it finds it necessary.
Exadata Smart Flash Cache Write-Back

• Accelerates write intensive workloads

• Frequently updated tables and indexes
• 20X more write IOPS than disk on X3
• 10X more write IOPs than disk on V2 and X2
• Database writes go directly to flash cache
• Block is kept in cache until it LRUs out
• Could be months or years
• While it is cached, reads or writes will be serviced from cache
• If the block stops being accessed, block will eventually age out of
cache and will be written to disk
• Smart Caching
• For example RMAN Backup and Data Pump reads and writes are
not cached in flash
Smart Flash Cache Write-Back Availability

• Write-back cache is persistent across reboots

– Accelerates crash recovery
• No need to reload or recover flash cache
– Write-back speeds redo apply for Recovery and Data Guard
• Redo apply is write IO intensive
• Flash Card failure is transparently and automatically handled by
Exadata and ASM software
– Writes are mirrored across cells, so a database write I/O becomes a flash
write to 2 or 3 storage servers
– No interruption to application on flash card failure
– Contents of cache are recovered from mirrors on other cells
• No Administration is needed
– Flash cache is transparent to Database, ASM, RMAN
– As always, RMAN Backups are at the database level
– Automatically backs up latest data on disk and on flash
 Exadata Smart Flash Cache Write-Back

Write Caching primarily benefits very high-write workloads

• Applications that are not bottlenecked on writes will see little or no benefit
from the extra writes enabled by write-back

• Unlike reads, improved write latency doesn’t help the application since
writes are performed in background by DBWR

• Remember that the flash read cache offloads most the IOs from disks and
therefore frees the disks to perform more writes. So even though the read
cache does not directly cache writes, it improves writes a lot.

• Write IOPs are now so high that usually another bottleneck is reached
before writes are maxed – application, CPU usage, mid-tier, latches, etc
Exadata Smart Flash Cache Write-Back

• What is version of storage server software and Database server

software is required ?

• Write-back flash cache is available from Exadata Storage storage

server software version 11.2.3.2.0
• Oracle Database software 11.2.0.3 BP 9 or later is required for
write-back flash cache to be enabled.

• When should one use the write back flash cache?

• If the application writes intensively and if one finds significant waits
for "free buffer waits" or high IO times to check for write bottleneck
in AWR reports, then you should consider using the write back
flash cache
 Exadata Smart Flash Cache Write-Back

How can one enable the write back flash cache?

CellCLI> list cell attributes flashcachemode
WriteThrough

CellCLI> list cell detail

name: cel14
...
flashCacheMode: WriteThrough

CellCLI> ALTER CELL flashCacheMode=WriteBack

The cell server must be restarted to make the changes effective.

The cell services must be shut down before changing the flashCacheMode attribute.

If the attribute is modified from writeback to writethrough and if there is existing flash cache, then an
error is displayed.

The flash cache must be dropped before changing the attribute to writethrough.
Exadata Smart Flash Cache Write-Back

What is performance benefit we can get by enabling the write back

flash cache ?

Write-back flash cache significantly improves the write intensive

operations because writing to flash cache is faster than writing to Hard
disks.

Depending on your application, on X3-2 machines write performance

can be improved up to 20X IOPS than disk and 10X more write IOPs
than to disk on V2 and X2
ata X3 Database In-Memory Machine
Exadata Smart Flash Cache Write-Back

• How do I flush the data from flash cache to Disk?

The data which is not been synchronized (Dirty Data) with griddisk can be
synchronized using the FLUSH option
Synchronization of dirty data can be a lengthy process, depending on the
number of bytes to be synchronized.

CELLCLI>ALTER GRIDDISK grid_disk_name FLUSH

Use the following command to check the progress:

CELLCLI>LIST GRIDDISK ATTRIBUTES name, flushstatus, flusherror

The FLUSH option stops new data from being cached on the specified
grid disks until CELLSRV restarts, or the flush operation is canceled.
Satyendra.pasalapudi@appsassociates.com
Satyendra.kumar@aioug.org