DMM106 Data Integration and Data Quality

Management on SAP HANA Cloud Platform

Public

Speakers
Las Vegas, Sept 19 - 23

Bangalore, October 5 - 7

Barcelona, Nov 8 - 10

Swapan Saha

Shibajee Dutta Gupta

Ben Hofmans

2016 SAP SE or an SAP affiliate company. All rights reserved.

Public

Disclaimer
The information in this presentation is confidential and proprietary to SAP and may not be disclosed without the permission of
SAP. Except for your obligation to protect confidential information, this presentation is not subject to your license agreement or
any other service or subscription agreement with SAP. SAP has no obligation to pursue any course of business outlined in this
presentation or any related document, or to develop or release any functionality mentioned therein.
This presentation, or any related document and SAP's strategy and possible future developments, products and or platforms
directions and functionality are all subject to change and may be changed by SAP at any time for any reason without notice.
The information in this presentation is not a commitment, promise or legal obligation to deliver any material, code or functionality.
This presentation is provided without a warranty of any kind, either express or implied, including but not limited to, the implied
warranties of merchantability, fitness for a particular purpose, or non-infringement. This presentation is for informational
purposes and may not be incorporated into a contract. SAP assumes no responsibility for errors or omissions in this
presentation, except if such damages were caused by SAPs intentional or gross negligence.
All forward-looking statements are subject to various risks and uncertainties that could cause actual results to differ materially
from expectations. Readers are cautioned not to place undue reliance on these forward-looking statements, which speak only
as of their dates, and they should not be relied upon in making purchasing decisions.

2016 SAP SE or an SAP affiliate company. All rights reserved.

Public

Agenda
Data integration to HCP with onPremise tools
Native data integration on HCP with HANA smart data integration
Data cleansing on HCP with SAP Data Quality Management, microservices for location data
Data Preparation on HCP with SAP Agile Data Preparation

Q&A

2016 SAP SE or an SAP affiliate company. All rights reserved.

Public

Data Integration to HCP with

onPremise Tools

Public

SAP HANA Cloud Connector

Connecting on-premise data integration tools to HCP
The SAP HANA Cloud Connector
establishes a secure and reliable tunnel so
that on-premise data integration tools (e.g.
SAP Data Services, SLT or 3rd party tools)
can connect to the HANA database on the
HANA Cloud Platform.
In SAP Cloud Connector create a database
tunnel through a service channel. All
ODBC/JDBC requests to this service
channel will be redirected to the HANA
instance on HCP

HANA Cloud Platform

HANA
HTTPS

cloud
on-premise

SAP HANA Cloud Connector

Service channel
<cloud connector host>:<port>

JCBD/ODBC
SLT
2016 SAP SE or an SAP affiliate company. All rights reserved.

SAP Data
Services

3rd party
tools
Public

SAP Data Services

An enterprise class data integration tool for any source & target
Any Source

Any Target

RDBMSs
(SAP & Non-SAP)

SAP Business Suite

SAP Data Services

Technical UI
(Data
Services)

SAP

Business UI
(Information
Steward)

Packaged Apps
(SAP & Non-SAP)
SaaS-based Applications

Profiling

Network Device Platforms

(Wired/Wireless)

batch /
real time

Data Quality

Streaming & Complex Event

Data

Text Analytics

One administration environment

(Scheduling, Security, User Management)

Cloud & Social Media

Big Data & Spatial Data

SAP Replication Server
REPLICATION

2016 SAP SE or an SAP affiliate company. All rights reserved.

SAP Landscape Transformation

TRANSFORMATION

SAP Databases (inl SAP

databases on HCP through
the SAP Cloud Connector))
3rd Party apps

LOAD

Non-SAP

batch /
real time

ETL

One runtime
architecture &
services

3rd Party Databases & DWs

Files
Message Buses & Services

Cloud & Big Data

Un-/Semi-Structured Data

Unified metadata

SAP BW

Web Services, OData

Public

Native Data Integration on HCP

with HANA smart data integration

Public

SAP HANA smart data integration

Overview
SAP HANA smart data integration delivers built-in data movement capabilities into the HANA
database to ingest data from a wide variety of sources,
as well as move data out (bi-directionally).
SAP HANA Platform

Native Integration Capabilities

Supports all styles of data delivery

Index Server
Virtual Tables

Virtualization / Federation
Real Time (some data sources)
Batch

Tables

Data Provisioning Server

Supports data distribution (bi-directional)

Architected for on premise, cloud or hybrid deployments

Data Provisioning Agent

Adapters

Web Based Design Tools

Open Framework

Out of the box adapters for common sources

Open SDK to build custom adapters
2016 SAP SE or an SAP affiliate company. All rights reserved.

RDBMS / Hadoop

Text Data

Structured Data

Social Network

Public

Data Provisioning Agent

The key to successful deployment in cloud with security top of mind
Lightweight installation inside the customers firewall
Small download from SAP Service Marketplace
A Deployment Unit (DU) for HANA
An onPremise agent
Simple installation and configuration
Runs on Windows or Linux

Securely transfers data from on-premise to cloud

Data sent over the cloud via HTTPS
Data are streamed from source to target and never persisted

Operates without firewall exceptions

Communication is always from the agent to the cloud
No need for VPN, reverse proxy, or other firewall exceptions
Agent uses long polling: places request to server and waits
for response when a task is ready to execute
2016 SAP SE or an SAP affiliate company. All rights reserved.

Public

10

SAP HANA smart data integration adapters

Built-in real time adapters for common sources
3rd party DBMS adapters
For Oracle, MSSQL, DB2, HANA, Sybase ASE and Teradata
Real time push through log based changed data capture (Oracle,
MSSQL, DB2, ASE) or trigger based changed data capture (HANA,
Teradata).

SAP ECC1 adapters

Implements real time push functionality on top of 3rd party DBMS sources
(Oracle, DB2, MSSQL)
In addition, it takes into account
SAP ABAP Transparent, Clustered and Pooled tables
SAP catalog hierarchy metadata

Twitter adapter
Real time access to Twitter content on hashtags, users, etc.
1. Minimum supported version is ECC 6; Details on NetWeaver version dependency in PAM
2016 SAP SE or an SAP affiliate company. All rights reserved.

Public

11

SAP HANA smart data integration adapters

Built-in batch adapters for common sources
OData adapter
Generally accessible from the cloud (e.g. Success Factors)
Runs on the DPserver in HANA, not on the agent

SOAP Adapter
Adapter for SOAP Webservices

Camel adapter for Facebook

Generic Camel adapter (100+ open source adapters available) https://camel.apache.org/components.html
In SP11 only tested for the Camel Facebook component

Hive adapter
Adapter for Hadoop

ABAP adapter
For business content extractors (aka BW datasources)
2016 SAP SE or an SAP affiliate company. All rights reserved.

Public

12

SAP HANA smart data integration

Use case: Data Federation (smart data access)
SAP HANA Platform

Data Federation enables users to see data from

remote sources as virtual tables in the HANA
database, without physically moving all data
into HANA.

Smart Data Access

Smart Data Integration

Virtual Tables

Data Provisioning
Server

SDA ODBC Adapters

Ideally suited for ad-hoc queries on data sets

that cannot be moved in HANA.

Initially delivered through Smart Data Access

Available since HANA SPS07

Was not available on HCP because the SDA architecture
was not cloud ready (local ODBC drivers). But

Now extended to HCP with Smart Data

Integration

RDBMS

Data Provisioning Agent

Database
Adapters
Non-database
Adapters

Custom
Adapters
(SDK)

Extends SDA functionality with more adapters that can run

outside of HANA
RDBMS

2016 SAP SE or an SAP affiliate company. All rights reserved.

Social Network

Text Data

Public

13

SAP HANA smart data integration

Use case: Data replication
With Data Replication data can be
replicated in real-time from the source
system into HANA, with minimal
transformations.

Simple web UI to create a replication

task:

1. Select remote source

2. Choose list of tables to replicate
3. Save and activate

Initial load followed by continuous realtime delta load

More advanced options available

Replicate table structure changes

Filters and mapping expressions

2016 SAP SE or an SAP affiliate company. All rights reserved.

Public

14

SAP HANA smart data integration

Use case: Data integration with flowgraphs
With flowgraphs more complex
data transformation rules can be
defined to extract, transform and
load data into HANA.

Rich set of transformations

available.

Also flowgraphs can be real-time

and replicate + transform data on
the fly.

2016 SAP SE or an SAP affiliate company. All rights reserved.

Public

15

SAP HANA smart data integration - Available on HANA Trial

During setup of a HANA MDC (Multitenant
Database Containers) instance, you can enable
the data provisioning server.
Data provisioning agent and deployment unit
available under a developer license on
tools.hana.ondemand.com (HCI tab).

http://scn.sap.com/community/developer-center/cloud-platform/blog/2016/07/14/step-by-step-setup-sdi-for-your-hcp-trial-account
2016 SAP SE or an SAP affiliate company. All rights reserved.

Public

16

SAP HANA smart data integration

Licensing and packaging for productive use
Check with your SAP account manager for latest details, but in summary:

HANA smart data integration is included in the premium and professional editions of the HCP
extension bundles.

HANA smart data integration is included in HCP, integration service bundles (Data and
Enterprise edition).

Note: HANA smart data integration (SDI) is replacing HANA Cloud Integration for data services
(HCI-DS) for use cases that involve loading data into HANA on HCP, so SDI has been included in
all bundles that have HCI-DS.

2016 SAP SE or an SAP affiliate company. All rights reserved.

Public

17

Data cleansing on HCP with

SAP Data Quality Management,
microservices for location data

Public

SAP Data Quality Management, microservices for location data

Benefits
A new microservice available on HCP for location data cleansing
can be called from HCP applications, other cloud applications or
onPremise applications.
Benefits:

Simple consumption model in the Cloud

Integration into many SAP Applications already

in development

Includes ABAP Add-on for ECC, CRM,

and Master Data Governance integration

SAP Data Quality Management, microservices

Address Cleansing
Geocoding
Reverse Geocoding

Self-Service purchase and consumption model

Leading solution w/ 30 years of experience in address cleansing and geocoding

capabilities (technology acquired from Firstlogic/BusinessObjects)

Strong partnerships with many worldwide postal authorities and data providers

Distributed or Centralized Configuration of complex address cleansing scenarios

with user interface in the Cloud

2016 SAP SE or an SAP affiliate company. All rights reserved.

SAP HANA
Cloud Platform
Customer/Partner
Applications

Public

19

SAP Data Quality Management, microservices for location data

Key capabilities
The initial release of this HTTP/JSON based
microservice will include 3 services :

Address Cleansing

Validate address per postal authority reference data as correct

Parse, Standardize, Correct, & Enhance

38 Countries supported to verify to street/house

number/secondary level

240+ Countries support to verify to city level

Geocoding

Append latitude and longitude given an address

28 Countries currently supported

Reverse Geocoding

Provide address(es) given a latitude and longitude

28 Countries currently supported

2016 SAP SE or an SAP affiliate company. All rights reserved.

Public

20

SAP Data Quality Management, microservices for location data

Example

1. Input

ADDRESS1

ADDRESS2 CITY

REGION

POSTAL

COUNTRY

121 Trellis Ln

<Blank>

escondid

California

<Blank>

USA

121 Trellis Lane

<Blank>

Escondido <Blank>

92026

USA

121 trelis Ln.

<Blank>

Escondido CA

92026

USA

SAP Data Quality Management, microservices

Address Cleansing

2. Send to microservice

Geocoding

SAP HANA
Cloud Platform

Reverse Geocoding

3. Output

DELIVERY_ADDRESS CITY

STATE

ZIP

ZIP4

LONGITUDE LATITUDE

121 Trellis Ln

Escondido

CA

92026

2078

-117.09002

33.14502399

121 Trellis Ln

Escondido

CA

92026

2078

-117.09002

33.14502399

121 Trellis Ln

Escondido

CA

92026

2078

-117.09002

33.14502399

2016 SAP SE or an SAP affiliate company. All rights reserved.

Public

21
21

SAP Data Quality Management, microservices for location data

Example (JSON)

2016 SAP SE or an SAP affiliate company. All rights reserved.

Public

22

SAP Data Quality Management, microservices for location data

Configuration UI (optional)
Mappings and
settings per
country or country
group

Specify input
mappings from your
data model to the
service

Specify address
formatting /
standardization
settings

Specify output
mappings to your
data model from the
service

2016 SAP SE or an SAP affiliate company. All rights reserved.

Public

23

SAP Data Quality Management, microservices for location data

Available as beta on trial (August 2016)

http://scn.sap.com/community/developer-center/cloud-platform/blog/2016/06/21/new-hcp-service-beta--sap-data-qualitymanagement-microservices-for-location-data#comment-683622
2016 SAP SE or an SAP affiliate company. All rights reserved.

Public

24

Data Preparation on HCP with

SAP Agile Data Preparation

Public

SAP Agile Data Preparation Key Capabilities

Discover, Profile, Combine and Share Data Sets
SAP Agile Data
Preparation
SAP BW

Analyst, Scientist,
Steward

Combine
Explore

SAP Business
Suite

Shape

BI Tools

Ingest data from

variety of sources

Profile data

Enrich

SAP HANA

Clean

ingest

distribute

SAP HANA
De-dup.

Databases

IT

Govern

4
Delimited Files

Delimited
Files

5
Sources

2016 SAP SE or an SAP affiliate company. All rights reserved.

Prepare

Targets

Combine, shape,
enrich, or cleanse data

Output data for

downstream uses

IT Governance team analyze and optimize

user processes

Public

26

SAP Agile Data Preparation

Available on the HANA Cloud Platform
Available as SaaS instantly at SAP Hana Cloud Platform from various global locations instantly.

Smart Data Integration with the data provisioning agent is used to get your data securely into HCP.

2016 SAP SE or an SAP affiliate company. All rights reserved.

Public

27

SAP TechEd Online

Continue your SAP TechEd
education after the event!
Access replays of

2016 SAP SE or an SAP affiliate company. All rights reserved.

Keynotes
Demo Jam
SAP TechEd live interviews
Select lecture sessions
Hands-on sessions

Public

28

Further information
Related SAP TechEd sessions:
DMM100 Lecture - EIM Overview in Data Integration, Quality, Preparation and Stewardship
DMM102 Lecture Define, Assess, and Improve Data Using SAP Agile Data Preparation
DMM160 Hands-on Experience SAP HANA Smart Data Integration and Smart Data Quality Firsthand
DMM811 Roadmap Q&A SAP HANA Smart Data Integration; SAP Agile Data Preparation
DMM163 Hands-on Prepare Your Data for Your Analytic Needs in Just a Few Clicks
ANP262 Hands-on SAP HANA Cloud Platform and SAP BusinessObjects Cloud In Action

SAP Public Web

on scn.sap.com :
Smart Data Integration on HCP : scn.sap.com/community/developer-center/cloud-platform/blog/2016/06/22/smartdata-integration-available-for-hcp
Data Quality microservices on HCP: scn.sap.com/community/developer-center/cloud-platform/blog/2016/06/21/newhcp-service-beta--sap-data-quality-management-microservices-for-location-data#comment-683622
More on Enterprise Information Management (EIM) in general : scn.sap.com/community/enterprise-informationmanagement

SAP Education and Certification Opportunities

www.sap.com/education

Watch SAP TechEd Online

www.sapteched.com/online
2016 SAP SE or an SAP affiliate company. All rights reserved.

Public

29

Feedback
Please complete your
session evaluation for
DMM106

2016 SAP SE or an SAP affiliate company. All rights reserved.

Contact information:
Ben Hofmans
Product Manager
Ben.Hofmans@sap.com

Public

30

 2016 SAP SE or an SAP affiliate company. All rights reserved.

No part of this publication may be reproduced or transmitted in any form or for any purpose without the express permission of SAP SE or an SAP affiliate company.
SAP and other SAP products and services mentioned herein as well as their respective logos are trademarks or registered trademarks of SAP SE (or an SAP affiliate
company) in Germany and other countries. Please see http://www.sap.com/corporate-en/about/legal/copyright/index.html for additional trademark information and notices.
Some software products marketed by SAP SE and its distributors contain proprietary software components of other software vendors.
National product specifications may vary.
These materials are provided by SAP SE or an SAP affiliate company for informational purposes only, without representation or warranty of any kind, and SAP SE or its
affiliated companies shall not be liable for errors or omissions with respect to the materials. The only warranties for SAP SE or SAP affiliate company products and
services are those that are set forth in the express warranty statements accompanying such products and services, if any. Nothing herein should be construed as
constituting an additional warranty.
In particular, SAP SE or its affiliated companies have no obligation to pursue any course of business outlined in this document or any related presentation, or to develop
or release any functionality mentioned therein. This document, or any related presentation, and SAP SEs or its affiliated companies strategy and possible future
developments, products, and/or platform directions and functionality are all subject to change and may be changed by SAP SE or its affiliated companies at any time
for any reason without notice. The information in this document is not a commitment, promise, or legal obligation to deliver any material, code, or functionality. All forwardlooking statements are subject to various risks and uncertainties that could cause actual results to differ materially from expectations. Readers are cautioned not to place
undue reliance on these forward-looking statements, which speak only as of their dates, and they should not be relied upon in making purchasing decisions.

2016 SAP SE or an SAP affiliate company. All rights reserved.

Public

31