1

2
3
4
When you hear the term Big Data, there are usually multiple problems
associated with it. Of course the size of the source information is one, but why
is the data so large? Because it is not ERP data but something else, such as
weblogs, social media data, and so on. This data is either completely
unstructured or semi-structured, and this data needs to be related back to the
ERP data.

Let us have a look at three examples of what one might want to call Big Data.
5
You might have an extremely large table in the database with billings you want
to sum up and show in a report for further analysis. This is an example of what
Hadoop can do very well, but so can databases, HANA in particular. So unless
the data is in Hadoop already, you would probably not use Hadoop for that
purpose.
6
A weblog is a nice example of what Hadoop can do very well. It is a large
dataset, it is not very well structured, you do not want to do simple aggregation
functions like How often was the URL xyz called, but more complex
algorithms like What is the time when a single user opened one page and then
another the potential reading time of the first page. Expressing these
calculations in SQL is almost impossible and even if it can be done, it would
execute for a long time.
7
For simple statistical means you might want to count the number of times a
word occurs. But how do you break apart a long text into its words in SQL? And
even if you can, the amount of data would grow so much that it cannot be
processed anymore.
8
Hadoop by itself is very simple, it is a file storage system and can execute
programs reading the files and writing new ones. The program needs to be
written in Java and can consist of two methods only, a map method and a
reduce (aggregation) method.
What is unique about Hadoop however is that if both requirements are met,
files as source and map-reduce based programs, the Hadoop environment can
execute these in a massively parallel way. So your input of the map method
might be the Amazon product review as suggesting in Example 3 and the Map
phase will output the list of words. The reduce method will then take the words
and count them across all reviews so that per product you can see how often
the word issue was found. Just as an example of a first simple approach, we
of course would use the Text Data Processing Transform for that.
Hadoop has two extensions we are using also, the Hive which gives you SQL
like options to join and summarize data. And PIG to write Map-Reduce logic not
in low level Java code but a higher level scripting language compiled into
Map_Reduce logic at runtime.
9
It does not make sense to fill up a database with terabytes of text data if this
data cannot be analyzed because the logic cannot be expressed in SQL or
other database languages. And if the source data is just rough data that has to
be processed in order to get a measure out of it. Then you use Hadoop for the
cheap storage of the data and for processing it, the results are then loaded into
HANA or any other system to enrich the analysis possible there.
10
And why would we use weblogs, text information entered by millions of users?
Because it can help us to find out the Why.
11
Combining the ERP numbers with the Hadoop-produced statistics for each
product will allow you to relate the two, for example, relate revenue numbers
with the hit rates and the things said about the product to get a big picture of
the sales situation.
12
13
It is quite possible that you may have many tools from different vendors
supporting your EIM and/or data management initiatives for ETL, data quality,
data profiling, metadata management, and text analytics.

What if you could simplify your IT environment with a single foundation to
deliver data services across your entire enterprise (for current or new projects).
Such a foundation needs to be open to support all data sources and targets,
scalable to handle small and extreme data volumes, and reusable so that you
can build once and reuse for other projects.

Such a Data Services foundation can be your standard for delivering all of the
critical information management capabilities across your enterprise. As a result,
you can significantly gain greater IT efficiency and deliver maximum business
effectiveness.
14
For us as Data Services users, Hadoop is just yet another source target. We
can read and write files into Hadoop, and we can push down logic into Hadoop
so we do not have to do that in the engine.
15
The internal flow logic when Data Services is using Hadoop is quite simple, as
Hadoop is quite simple. We prepare the file, we generate the processing logic,
and let this script then be executed in Hadoop.

One important limitation at the moment is that we cannot load a remote Hadoop
system, so a job server needs to be installed in the Hadoop system.
16
17
The CDC checkbox and type can only be set when creating the datastore,
it cannot be modified later on.
Once a CDC datastore is created, it will show tables only where CDC or
change tracking is activated.
18
Change Data Capture (CDC) and Change Tracking enable applications to
determine the Insert, Update, and Delete operations that were made to user
tables in a database.
19
After each successful execution (at the end of the job), the watermark is written
to a status table; for the next execution, only changes after this watermark are
extracted.

You can use the CDC subscription name (for change tracking only) when
multiple subscribers need to access the delta independently.

For the type CDC, low and high watermarks are set to ensure data
consistency. This applies to all dataflows in the job.

Additional CDC columns are available in the input schema.
20
21
22