Chapter 5: Foundations for a New Enterprise Application Development Era

Learning Map
The Future of Enterprise Compu'ng Founda'ons of Database Storage Techniques

In-Memory Database Operators

Advanced Database Storage Tech- niques

Founda'ons for a New Enterprise Applica'on Development Era

Implications on Application Development

Dr.-Ing. Jrgen Mller

Simplified Application Development

Tradi&onal Applica&on cache Database cache Column-oriented

No caches needed No redundant data/ objects No maintenance of indexes or aggregates Data movements are minimized

Prebuilt aggregates

Raw data

111

SAP ERP Financials on In-Memory Technology

In-memory column database for an enterprise system

Combined workload (parallel OLTP/OLAP queries) Leverage in-memory capabili&es to

Reduce amount of data Aggregate on-the-y Run analy&c-style queries (to replace materialized views) Execute stored procedures

112

Use Case: SAP ERP Financials solu&on

Post and change documents Display open items Run dunning job Analy&cal queries, such as balance sheet

Current Financials Solutions

113

The Target Financials Solution

Only base tables, algorithms, and some indexes

114

In-Memory Financials on SAP ERP

accounting documents dunning data sum tables

BKPF
BSEG

secondary indices

MHNK
MHND

change documents

BSAD
BSAK
BSAS
BSID
BSIK
BSIS

GLT0

CDHDR
CDPOS

LFC1

KNC1

115

In-Memory Financials on SAP ERP (prototype)

accounting documents

BKPF
BSEG

116

Feasibility of Financials on In-Memory Technology

Modica&ons on SAP Financials

Removed secondary indices, sum tables and pre-calculated and materialized tables Reduce code complexity and simplify locks Insert Only to enable history (change document replacement) Added stored procedures with business func&onality

European division of a retailer

ERP 2005 ECC 6.0 EhP3 5.5 TB system database size Financials:

23 million headers / 8 GB in main memory 252 million items / 50 GB in main memory

(including inverted indices for join a_ributes and insert only extension)

117

Reduction by a Factor 10
DBMS
BKPF
BSEG
8.7 GB
255 GB
263.7 GB
Secondary Indices
Sum Tables
Complete

118

IMDB
1.5 GB
50 GB
51.5 GB
-
-
51.5 GB

255 GB
0.55 GB
519.25 GB

Dunning Run

Dunning run determines all open and due invoices Customer dened queries on 250M records Current system: 20 min New logic: 1.5 sec

In-memory column store Parallelized stored procedures Simplied Financials

119

Why?

Being able to perform the dunning run in such a short &me lowers TCO Add more func&onality! Run other jobs in the mean&me! - in a mul&-tenancy cloud setup hardware must be used wisely

120

Bring Application Logic Closer to the Storage Layer

Select accounts to be dunned, for each:

Select open account items from BSID, for each: Calculate due date Select dunning procedure, level and area Create MHNK entries

Create and write dunning item tables

121

Bring Application Logic Closer to the Storage Layer

Select accounts to be dunned, for each:

1 SELECT 10,000 SELECTs 10,000 SELECTs 31,000 Entries

Select open account items from BSID, for each: Calculate due date Select dunning procedure, level and area Create MHNK entries

Create and write dunning item tables

122

Bring Application Logic Closer to the Storage Layer

Select accounts to be dunned, for each:

1 SELECT One single Select open account items from BSID, for each: stored procedure 10,000 SELECTs
Calculate due date Select dunning procedure, level and area Create MHNK entries

10,000 SELECTs 31,000 Entries

Create and write dunning item tables

123

Bring Application Logic Closer to the Storage Layer

Select accounts to be dunned, for each:

1 SELECT One single Select open account items from BSID, for each: stored procedure 10,000 SELECTs
Calculate due date Select dunning procedure, level and area Create MHNK entries

10,000 SELECTs

Create and write dunning item tables

124

Calculated on-the-y

31,000 Entries

Results
Original Version needed about 20 minutes Factor 800x accelera&on achieved
#
Opera'on

1
2
3
4
5
6
Select Open Items
Due date, dunning level
Filter 1 (Verify Dunning levels)
Filter 2 (Check Last Dunning)
Generate MHNK (Aggregate)
Generate MHND (Execute Filters)
Total

HANA2 Version

0.63s
27s
19s
15s
done in #1
done in #1
1 Minute

Variant 2

Variant 3

1.01s 0.6s (incl. T047 & KNB5 Join)

(incl. T047 & KNB5 Join)
deferred to aggrega&on
1.1s
0.8s
1.2s
140ms

(#3, #4 exec. in parallel)

0.5s
0.5s
0.4s
done in #1
done in #1

(#3, #4 exec. in parallel)

3.0s

1.5s

125

Hardware: 4 CPUs x 6 Cores (Intel Dunnington), 256GB RAM

Customer Data: 250mio line items, 380k open, 200k due

Dunning Application

126

Dunning Application

127

Available-to-Promise Check

Can I get enough quan&&es of a requested product on a desired delivery date? Goal: Analyze and validate the poten&al of in-memory and highly parallel data processing for Available-to-Promise (ATP) Challenges

Dynamic aggrega&on Instant rescheduling in minutes vs. nightly batch runs Real-&me and historical analy&cs Real-&me ATP checks without materialized views Ad-hoc rescheduling No materialized aggregates

Outcome

128

In-Memory Available-to-Promise

129

Demand Planning

Flexible analysis of demand planning data Zooming to choose granularity Filter by certain products or customers Browse through &me spans Combina&on of loca&on-based geo data with planning data in an in-memory database External factors such as the temperature, or the level of cloudiness can be overlaid to incorporate them in planning decisions

130

GORFID

HANA for Streaming Data Processing Use Case: In-Memory RFID Data Management Evalua&on of SAP OER Prototypical implementa&on of:

RFID Read Event Repository on HANA Discovery Service on HANA (10 billion data records with ca. 3 seconds response &me) Frontends for iPhone, iPad2

Key Findings:

HANA is suited for streaming data (using bulk inserts) Analy&cs on streaming data is now possible

131

Near Real-Time as a Concept

Read Event Read Event Read Event Repositories Repositories Repositories up 8,000 read up toto 8.000 read up to 8.000 read event notications event notications eventper notications second per second second per

Bulk load every 2-3 seconds: > 50,000 inserts/s

DiscoveryService Service Discovery Discovery Service

SAP HANA HANA SAP SAP HANA

132

Learning Map
The Future of Enterprise Compu'ng Founda'ons of Database Storage Techniques

In-Memory Database Operators

Advanced Database Storage Tech- niques

Verication Verication Verication Services Services Services upto to2.000 2,000 up up to 2.000 requests requests requests per second per second per second

P P A

A A

Founda'ons for a New Enterprise Applica'on Development Era

Views

Dr.-Ing. Jrgen Mller

Dynamic Views
Presentation Layer
Excel SAP Business Objects Explorer Any Software View View View

View Layer (Calculations, Filter, ...)

View

View

...

View

View

Other DB

Persistency Layer (Main Memory)

Node Tables

Node Tables

Node Tables

Node Tables

Store
DB Persistence

Restart

Object Hierarchy Recovery Write Complete Objects

135

Logical Log

Learning Map
The Future of Enterprise Compu'ng Founda'ons of Database Storage Techniques

In-Memory Database Operators

Advanced Database Storage Tech- niques

Founda'ons for a New Enterprise Applica'on Development Era

Bypass Solution

Dr.-Ing. Jrgen Mller

Bypass Solution Allows a Smooth Transition to In-Memory Technology

Millions of old un-op&mized lines of code at the customers site Transi&on required Row-store replacement Part-for-part replacement with bypass Transform row-store to column-store on the y Change of applica&on code

140

Bypass Solution for Transition

Todays System
ERP

Traditional BI
SAP

ETL

OLAP Engine

BIA

Business Objects

...

Traditional DB
w/ Cubes

Excel

Traditional DB

141

Bypass Solution for Transition

STEP 1: Install and run the in-memory database in parallel

Todays System with IMDB

ERP

Traditional BI
SAP

ETL

OLAP Engine

BIA

Business Objects

...

Traditional DB
w/ Cubes

Excel

Traditional DB

IMDB

SSD

Bypass Solution for Transition

STEP 1: Install and run the in-memory database in parallel STEP 2: Generate business value from the first day on

Todays System with IMDB

ERP

Traditional BI
SAP

ETL

OLAP Engine

BIA

Business Objects

...

Traditional DB
w/ Cubes

Excel

Traditional DB

IMDB

SSD

New Applications

Bypass Solution for Transition

STEP 1: Install and run the in-memory database in parallel STEP 2: Generate business value from the first day on STEP 3: Re-create traditional BI w/o cubes in IMDB

Todays System with IMDB

ERP

Traditional BI with IMDB

SAP

EL

OLAP Engine

BIA

Business Objects

...

w/o cubes

IMDB

Excel

Traditional DB

IMDB

SSD

New Applications

Bypass Solution for Transition

STEP 1: Install and run the in-memory database in parallel STEP 2: Generate business value from the first day on STEP 3: Re-create traditional BI w/o cubes in IMDB STEP 4: Introduce next-gen BI running in parallel without materialized views

Todays System with IMDB

ERP

Traditional BI with IMDB

SAP

EL

OLAP Engine

BIA

Business Objects

...

w/o Cubes

IMDB

Excel

Traditional DB

IMDB
OLAP Engine

BI 2.0
SAP Business Objects

SSD

New Applications

Excel

...

Bypass Solution for Transition

STEP 1: Install and run the in-memory database in parallel STEP 2: Generate business value from the first day on STEP 3: Re-create traditional BI w/o cubes in IMDB STEP 4: Introduce next-gen BI running in parallel without materialized views STEP 5: Eliminate all the traditional BI, virtualize all in-memory BI, using non-materialized views

Todays System with IMDB

ERP

Traditional DB

IMDB
OLAP Engine

BI 2.0
SAP Business Objects

SSD

New Applications

Excel

...

Bypass Solution for Transition

STEP 1: Install and run the in-memory database in parallel STEP 2: Generate business value from the first day on STEP 3: Re-create traditional BI w/o cubes in IMDB STEP 4: Introduce next-gen BI running in parallel without materialized views STEP 5: Eliminate all the traditional BI, virtualize all in-memory BI, using non-materialized views STEP 6: Run ERP and BI on IMDB

Future System with IMDB

ERP Future Releases OLTP & OLAP BI 2.0
SAP Business Objects

IMDB
OLAP Engine

SSD

New Applications

Excel

...