Data Warehouse

Fundamentals

Chapter 9

Data Mining Basics

Instructor: Paul Chen
 Topics
1. How Data Mining Evolved?
2. Decision Processing Overview and Tasks
3. Data Mining, Whats it?
4. Data Mining vs. Data Warehousing
5. How Data Mining Works? And Its Applications
6. Data Mining Operations and Associated Techniques
7. The Data Mining Process
8. Data Mining Tools
9. Data Mining Techniques- A Summary
 Topic 1:How Data Mining Evolved?

Many businesses have invested heavily in information

technology to help them manage their businesses more
effectively and gain a competitive edge. Increasingly large
amounts of critical business data are being stored
electronically and this volume is expected to continue to
grow. The Data Mining technology is helping companies
leverage their existing data more effectively and obtain
insightful information giving them a competitive edge.
 How Data Mining Evolved?

1960s 1990s Late 1990s to

1970s-80s
Data OLAP and Now
RDBMS Data Mining
Collection DW

Time Line
 Topic 2: Decision Processing
Overview
Decision processing systems, and their underlying
analytical applications, provide business users with the
information they need to track and analyze business
trends, and to explore new business opportunities. As
businesses become increasingly competitive and
complex, effective decision processing systems are
essential for success.
 The Next Generation of Business
Intelligence
A decision processing system analyzes business
information captured from operational systems (Back-
and-front office, and e-business applications).
Distribution of business information to business users
is via corporate intranets and extranets.
The flow of data can be thought of as an information
supply chain whose objective is to convert operational
data into useful business information.
 The Decision Processing Information
Business
Supply Chain Metrics
Operational
Systems
External Analytic
E-Business Data Applications
Applications

Collaborative
DW &
Back-Office Office Systems
Transaction Business
Applications Intelligence
Information Tools
Staging
Area
Business
Front-Office Decisions
Applications
 Decision ProcessingFour Tasks***

Extracting and transforming information

This involves capturing data from operational systems,
transforming it into business information, and loading
Into a data warehouse information store.

Current extract templates on the market are primarily at

Capturing data from ERP (Enterprise Resource Planning)
Transaction processing systems for example: SAP Business
Information Warehouse and Peoplesoft BPM data warehouse)

*** Mentioned in chapter 2

 Decision ProcessingFour Tasks
(Contd)

Managing information

This task encompasses the maintenance of business

information in information stores, and how these
information stores are processed by business intelligence
tools and analytic applications.
The cornerstone of decision processing is data
warehousing, and warehouse information stores should
be organized and modeled into relational and
multidimensional database products.
 Decision ProcessingFour Tasks
(Contd)
Analyzing and modeling information
The traditional approach to decision
processing is to build a data warehouse
and supply business users with a set of
business intelligence tools (query,
reporting, OLAP and data mining, for
example) to process information in data
warehouse information stores.
A better approach is employ turn-key and
web-based analytic application packages
that are designed to provide
comprehensive analyses for the business
area being researched. Key business
metrics (ex. Revenue dollars per sales rep
per day) are useful.
 Decision ProcessingFour Tasks
(Contd)
Distributing information

Business intelligence tools and analytic applications distribute

information and the results of analysis operations to business
users via standard graphical and Web interfaces.
To help users uncover and organize this range of business
information, an enterprise information portal (EIP) is required.
An EIP provides a single point of entry to any piece of
business information, no matter where it resides.
The main components of an EIP are information assistant
(Web browser interface) , an information directory and a
subscription facility.
 Decision Making Under Risk

Decisions are made under three sets of conditions:

Certainty
The decision makers know everything in advance

of making the decision

Uncertainty
The decision makers know nothing about the

probabilities or the consequences of decisions

Risk
 Decision-Making Style
Decision-making styles of users are categorized as
either
Analytic or
Heuristic
 Analytic and Heuristic Decision
Making
Analytical Decision Maker
Learns by analyzing
Uses step-by-step procedure
Values quantitative
information and models
Builds mathematical models
and algorithms
Seeks optimal solution
Heuristic Decision Maker
Learns by acting
Uses trial and error
Values experiences
Relies on common sense
Seeks completely satisfying
solution
 Topic 3: Data Mining, Whats it?

Data Mining has been defined as a decision support

process in which a search is made for patterns of
information in data. To detect patterns in data, Data
Mining uses sophisticated statistical analysis and modeling
technologies to uncover useful relationships hidden in
databases. It predicts future trends and finds behavior
allowing businesses to make predictive, knowledge-driven
decisions.
 Data Mining, Whats it?
The process of extracting valid, previously unknown,
comprehensible, and actionable information from large
databases and using it to make crucial business
decisions, (Simoudis,1996).

Involves analysis of data and use of software techniques

for finding hidden and unexpected patterns and
relationships in sets of data.
 Data Mining, Whats it?
Reveals information that is hidden and unexpected, as
little value in finding patterns and relationships that
are already intuitive.

Patterns and relationships are identified by examining

the underlying rules and features in the data.

Tends to work from the data up and most accurate

results normally require large volumes of data to
deliver reliable conclusions.
 Data Mining, Whats it?
Starts by developing an optimal representation of
structure of sample data, during which time knowledge
is acquired and extended to larger sets of data.

Data mining can provide huge paybacks for companies

who have made a significant investment in data
warehousing.

Relatively new technology, however already used in a

number of industries.
 Topic 4: Data Mining vs. Data
Warehousing
Data Mining does not require that a Data Warehouse be
built. Often, data can be downloaded from the operational
files to flat files that contain the data ready for the data
mining analysis.

Data Mining can be implemented rapidly on existing

software and hardware platforms. Data Mining tools can
analyze massive databases to deliver answers to questions
such as, Which customers are most likely to respond to
my next promotional mailing, and why?
 Data Mining vs. Data
Warehousing
Major challenge to exploit data mining is identifying suitable data
to mine.

Data mining requires single, separate, clean, integrated, and self-

consistent source of data.

A data warehouse is well equipped for providing data for mining.

Data quality and consistency is a pre-requisite for mining to

ensure the accuracy of the predictive models. Data warehouses are
populated with clean, consistent data.
 Data Mining vs. Data
Warehousing
Advantageous to mine data from multiple sources to discover as
many interrelationships as possible. Data warehouses contain data
from a number of sources.

Selecting relevant subsets of records and fields for data mining

requires query capabilities of the data warehouse.

Results of a data mining study are useful if there is some way to

further investigate the uncovered patterns. Data warehouses
provide capability to go back to the data source.
 Topic 5: How Data Mining
Works?
How exactly is Data Mining able to tell you important
things that you didnt know or what is going to happen
next? The technique in Data Mining is called Predictive
Modeling which is knowledge discovery process via
relationships and patterns in broad sense.

Modeling is the act of building a model in one situation

where you know the answer and then applying it to another
situation that you dont.
 Examples of Applications of Data
Mining via relationships and patterns
Retail / Marketing
Identifying buying patterns of customers
Finding associations among customer demographic
characteristics
Predicting response to mailing campaigns
Market basket analysis
 Examples of Applications of Data
Mining via relationships and patterns
Banking
Detecting patterns of fraudulent credit card use
Identifying loyal customers
Predicting customers likely to change their credit
card affiliation
Determining credit card spending by customer
groups
 Examples of Applications of Data
Mining via relationships and patterns
Insurance
Claims analysis
Predicting which customers will buy new policies.

Medicine
Characterizing patient behaviour to predict surgery
visits
Identifying successful medical therapies for
different illnesses.
 Examples of Applications of Data
Mining via relationships and patterns
Customer profiling: characteristics of good customers are
identified with the goals of predicting who will become
one and helping marketers target new prospects.

Targeting specific marketing promotions to existing and

potential customers offers similar benefits.

Market-basket analysis: With Data Mining, companies can

determine which products to stock in which stores, and
even how to place them within a store.
 Examples of Applications of Data
Mining via relationships and patterns
Customer Relationships Management-Determines
characteristics of customers who are likely to leave for a
competitor, a company can take action to retain that
customer because doing so is usually for less expensive
than acquiring a new customer.

Fraud detection- With Data Mining, companies can

identify potentially fraudulent transactions before they
happen.
 Topic 6: Data Mining Operations
and Associated Techniques

In previous foils, predictive modeling in essence includes

other operations shown in the above table.
 Descriptive: The dealer sold 200 cars last month.

Operational (OLTP)

Explanatory: For every increase in 1 % in the

interest,
auto sales decrease by 5 %.
Traditional DW
OLAP

Predictive: predictions about future buyer behavior.

Data Mining
Level of Modeling vs. Level of Analytical Processing

Descriptive Explanatory Predictive

SIMPLE QUERIES WHAT IF

& REPORTS PROCESSING DETERMINE IF
ANY PATTERNS
ANALYZE WHAT EXIST BY REVIEWING
HAS PREVIOUSLY DATA RELATIONSHIPS
OCCURRED TO
BRING ABOUT THE
CURRENT STATE
OF THE DATA
Normalized Denormaliz + Statistical Analysis/
Tables ed Artificial Intelligence
Tables
Roll-up; Drill Down Classification & Value Predictio
 Predictive Modelling

Similar to the human learning experience

uses observations to form a model of the important
characteristics of some phenomenon.

Uses generalizations of real world and ability to fit

new data into a general framework.

Can analyze a database to determine essential

characteristics (model) about the data set.
 Predictive Modelling

Model is developed using a supervised learning

approach, which has two phases: training and testing.

Training builds a model using a large sample of

historical data called a training set.
Testing involves trying out the model on new,
previously unseen data to determine its accuracy
and physical performance characteristics.
 Predictive Modelling

Applications of predictive modelling include customer

retention management, credit approval, cross selling,
and direct marketing.

Two techniques associated with predictive modelling:

A. classification
B. value prediction, distinguished by nature of the
variable being predicted.
 Statistical Analysis of Actual Sales (dollars
and quantities) relative To these Signage
Variables-a predictive modeling example.
Content
Frequency
Depth
Focus
Depth
Scale
Length
Location

Statistical Analysis : Correlation, Regression, Experiment Design,

Optimization. Now it goes into real time analysis.
Signage
Signage
 PREDICTIVE MODELING

There are two techniques associated with predictive

modeling: classification and value prediction, which are
distinguished by the nature of the variable being
predicted.
 Predictive Modelling - Classification

Used to establish a specific predetermined class for

each record in a database from a finite set of possible,
class values.

Two specializations of classification: tree induction and

neural induction.
Example of Classification using
Tree Induction
Example of Classification using
Tree Induction
Customer renting
property
> No
2 years
Yes

Rent Customer age>45

property
No Yes

Rent Buy property

property
Example of Classification using
Neural Induction
 Example of Classification using
Neural Induction
Each processing unit (circle) in one layer is connected
to each processing unit in the next layer by a weighted
value, expressing the strength of the relationship. The
network attempts to mirror the way the human brain
works in recognizing patterns by arithmetically
combining all the variables with a given data point.

In this way, it is possible to develop nonlinear

predictive models that learn by studying
combinations of variables and how different
combinations of variables affect different data sets.
 Predictive Modelling - Value
Prediction
Used to estimate a continuous numeric value that is
associated with a database record.

Uses the traditional statistical techniques of linear

regression and non-linear regression.

Relatively easy-to-use and understand.

 Predictive Modelling - Value
Prediction
Linear regression attempts to fit a straight line through
a plot of the data, such that the line is the best
representation of the average of all observations at that
point in the plot.

Problem is that the technique only works well with

linear data and is sensitive to the presence of outliers
(i.e.., data values, which do not conform to the expected
norm).
 Predictive Modelling - Value
Prediction
Although non-linear regression avoids the main
problems of linear regression, still not flexible enough
to handle all possible shapes of the data plot.

Statistical measurements are fine for building linear

models that describe predictable data points, however,
most data is not linear in nature.
 Predictive Modelling - Value
Prediction
Data mining requires statistical methods that can
accommodate non-linearity, outliers, and non-numeric
data.

Applications of value prediction include credit card

fraud detection or target mailing list identification.
 Database Segmentation

Aim is to partition a database into an unknown number

of segments, or clusters, of similar records.

Uses unsupervised learning to discover homogeneous

sub-populations in a database to improve the accuracy
of the profiles.
 Database Segmentation

Less precise than other operations thus less sensitive to

redundant and irrelevant features.

Sensitivity can be reduced by ignoring a subset of the

attributes that describe each instance or by assigning a
weighting factor to each variable.

Applications of database segmentation include

customer profiling, direct marketing, and cross selling.
Example of Database Segmentation
using a Scatter plot
 Database Segmentation
Associated with demographic or neural clustering
techniques, distinguished by:
Allowable data inputs
Methods used to calculate the distance between
records
Presentation of the resulting segments for analysis.
Example of Database Segmentation
using a Visualization
 Link Analysis

Aims to establish links (associations) between records,

or sets of records, in a database.

There are three specializations

Associations discovery
Sequential pattern discovery
Similar time sequence discovery

Applications include product affinity analysis, direct

marketing, and stock price movement.
 Link Analysis - Associations
Discovery
Finds items that imply the presence of other items in
the same event.

Affinities between items are represented by association

rules.
e.g. When customer rents property for more than 2
years and is more than 25 years old, in 40% of cases,
customer will buy a property. Association happens
in 35% of all customers who rent properties.
 Link Analysis - Sequential Pattern
Discovery
Finds patterns between events such that the presence of
one set of items is followed by another set of items in a
database of events over a period of time.

e.g. Used to understand long term customer buying

behaviour.
 Link Analysis - Similar Time
Sequence Discovery
Finds links between two sets of data that are time-
dependent, and is based on the degree of similarity
between the patterns that both time series demonstrate.
e.g. Within three months of buying property, new
home owners will purchase goods such as cookers,
freezers, and washing machines.
 Deviation Detection

Relatively new operation in terms of commercially

available data mining tools.

Often a source of true discovery because it identifies

outliers, which express deviation from some previously
known expectation and norm.
 Deviation Detection

Can be performed using statistics and visualization

techniques or as a by-product of data mining.

Applications include fraud detection in the use of credit

cards and insurance claims, quality control, and defects
tracing.
 A Summary: Data-Driven
Techniques*
Data Visualization

Decision Trees

Clustering

Factor Analysis

Neural Network

Association Rules

Rule Induction

* Based on Sakhr Younesss book Professional Data Warehousing with SQL Server 7.0 and
OLAP Services
 Data Visualization
A pie chart showing the sales of a product by region is
Sometimes much more effective than presenting the s
Data in a text or tabular form.

9%
Northeast South 11 %
39% North

21 %
West
20 %
East
Decision Tree
 Cluster Analysis
First segment (high income>8,000)
Have
Children
Second Segment (8000>middle income >3000)
Married

Third Segment (low income < 3000) Last car is

A used one

Own car
 Factor Analysis
Unlike cluster analysis, factor analysis builds a model from data.
The technique finds underlying factors, also called latent
variables and provides models for these factors based on
variables in the data. For ex., a software company is considering a
survey to find out the nine most perceived attributes of one of
their products. They might categorize these products to categories
such as service for technical support, availability for training and
a help system.

Factor analysis is used for grouping together products based on a

similarity of buying patterns so that vendors may bundle several
products as one to sell them together at a lower price than their
added individual prices..
Neural Networks
 Association Rules

Association models are models that examine the extent to which

values of one field depend on, or are produced by, values of
another field. These models are often referred to as Market Basket
Analysis when they are applied to retail industries to study the
buying patterns of these customers, especially in grocery and
retail stores that issue their own credit cards. Charging against
these cards gives the store the chance to associate the purchases of
customers with their identities, which allows them to study
associations among other things.
 Rules Induction

This is a powerful technique that involves a large number of rules

using a set of if..then statements in the pursuit of all possible
patterns in the dataset. For ex., if the customer is a male then, if he
is between 30 and 40 years of ages, and his income is less than
$50,000 and more than $20,000, he is likely to be driving a car that
was bought as new.
 A Summary: Theory-Driven
Techniques
Correlations

T-Tests

Analysis of Variables

Linear Regression

Logistic Regression

Discriminate Analysis

Forecasting Methods
 Topic 7: The Data Mining Process

Define the problem.

Select the data.
Prepare the data.
Mine the data.
Deploy the model.
Take business action.
Are you ready for Data Mining?
 Define the problem
A successful data mining initiative always starts with
a well-defined project. To insure that the project produces
incremental value, include an assessment of the status quo
solution and a review of technology, organization, and
business processes.
 Select the data

This step involves defining your data source . (not every

data source and record is required.) The data is usually
extracted from the source system to a separate server.
 Prepare the data

This step represents up to 80 percent of the total project

effort. For data mining, the data must reside in one flat
table (each record has many columns). In addition to being
the most time consuming, the step is also the most critical.
The resulting models are only as good as the data used to
create them.
 Mine the data

Typically the easiest and shortest phase, this step involves

applying statistical and AI tools to create mathematical
models. Data mining typically occurs on a server separate
from the data warehousing and other corporate systems.
 Deploy the Model

Model deployment is the process of implementing the

mathematical models into operational systems to improve
business results.
 Take Business Action

Use the deployed model to achieve improved results to the

business problem identified at the beginning of the
process.
 Step to Implement Data Mining

Discovery (patterns, relations

Prior Knowledge
Associations, etc.)

Information Model

Validation

Deployment
 ARE YOU READY FOR DATA
MINING?

Just because you have a data warehouse doesnt mean

youre necessarily ready for data mining. Much of the
work our company does in the data mining arena has
more to do with data mining readiness assessment than
with actually performing data mining.
 Metrics you can use to gauge your data
mining readiness
Do you have a staff of experienced knowledge workers?
Do you have the data?
Do you have marketing processes in place that can use this
data?
Do you have a business champion who can embrace the
process and results?
Do you have the technology infrastructure to support
advanced analysis?
 Topic 8: Data Mining Tools

Data mining tools are typically classified by the type of

algorithm they use to identify hidden patterns. There are
many different algorithms in use, but the four most
popular are association, sequence, clustering (or
segmentation), and predictive modeling.
 Data Mining Tools

There are a growing number of commercial data

mining tools on the marketplace.

Important characteristics of data mining tools include:

Data preparation facilities
Selection of data mining operations
Product scalability and performance
Facilities for visualization of results.
 Data Mining vs. OLAP

They are two separate breeds of analysis with

entirely different objectives, not to mention
tools, skill sets, and implementation methods.
 Data Mining
With canned reports, ad hoc querying, and
OLAP, the end user defines a hypothesis and
determines which data to examine. With data
mining, the tool identifies the hypothesis, and it
actually tells the user where in the data to start
the exploration process.
 Data Mining
Rather than using SQL to filter out values and methodically
reduce the data into a concise answer set, data mining uses
algorithms that exhaustively review the relationships among
data elements to determine if any patterns exist. The whole
purpose of data mining is to yield new business information
that a business person can act on.
 OLAP vs. Data Mining Tools
OLAP Tools
Are ad hoc, shrink wrapped
tools that provide an interface
to data
Are used when you have
specific known questions
Looks and feels like a
spreadsheet that allow
rotation, slicing and graphic
Can be deployed to large
number of users
Data Mining Tools
Methods for analyzing
multiple data types
-- Regression Trees
-- Neural networks
-- Genetic algorithms
Are used when you dont
know what the questions are
Usually textual in nature
Usually deployed to a small
number of analysts
 Data Mining Tools

ASSOCIATION

Association, also frequently referred to as "affinity

analysis," reviews numerous sets of items and looks for
common groupings. An example of association is market
basket analysis, which involves reviewing the products
that consumers purchase in a single trip to the grocery
store.
 ASSOCIATION

Finds items that imply the presence of other items

in the same event.

Affinities between items are represented by

association rules.
e.g. When a customer rents property for more than 2
years and is more than 25 years old, in 40% of cases,
the customer will buy a property. This association
happens in 35% of all customers who rent properties.
 Data Mining Tools

SEQUENCE

Sequential analysis helps data miners identify a set of

order-specific items or events. Association identifies the
existence of patterns or groups of items; sequential
analysis identifies the order of those patterns or groups of
items.
 SEQUENCE

Finds patterns between events such that the presence of

one set of items is followed by another set of items in a
database of events over a period of time.
e.g. Used to understand long term customer buying
behavior.
 Link Analysis - Similar Time Sequence
Discovery
Finds links between two sets of data that are time-
dependent, and is based on the degree of similarity
between the patterns that both time series demonstrate.

e.g. Within three months of buying property, new home

owners will purchase goods such as cookers, freezers, and
washing machines.
 Data Mining Tools

CLUSTERING

Cluster analysis lets the data miner assemble data into

unforeseen groups containing similar characteristics. Also
known as "segmentation," this type of data
mining is probably the most widely used.
 CLUSTERING

Aim is to partition a database into an unknown number of

segments, or clusters, of similar records.

Uses unsupervised learning to discover homogeneous sub-

populations in a database to improve the accuracy of the
profiles.
 Data Mining Tools

PREDICTIVE MODELING

As the name implies, predictive modeling involves

developing a model from historical data for predicting a
future event. The power of predictive modeling engines is
that they can use a broad range of data attributes to identify
future behavior. Both cluster analysis and predictive
modeling tools identify distinct groups of items with
common attributes; the difference is that predictive modeling
focuses on the likelihood of a particular outcome for a
particular group.
 Topic 9: Data Mining Techniques- A
Summary
Artificial neural networks: Non-linear predictive models that
learn through training and resembles biological neural networks
in structure.
Decision Trees: Tree-shaped structures that represent sets of
decisions. These decisions generate rules for the classification of a
database.

Generic Algorithms: Optimization techniques that use processes

such as generic combination, mutation, and natural selection in a
design based on the concepts of revolution.

Rule induction: The extraction of useful if-then rules from data

based on statistical significance.
 Data Mining Techniques- A
Summary
Predictive modeling Classification
Value prediction
Database Segmentation Demographic clustering
Neural clustering
Link analysis
Association discovery
Sequential pattern discovery
Similar time sequence
discovery
Deviation detection Statistics
Visualization
 Two Types of Data Mining Modeling-
Verification and Discovery
The verification model utilizes a process that looks in a
database to detect trends and patterns in data that will help
answer some specific questions about the business.

In this mode, the user generates a hypothesis about the

data, issues a query against the data and examines the
results of the query looking for verification of the
hypothesis or the user decides that the hypothesis is not
valid.
 Verification Model

In this model, very little information is created in this

extraction process: either the hypothesis is verified or it is
not.

Common tools used in this mode are: queries,

multidimensional analysis and visualization. What all have
in common are that the user is essentially guiding the
exploration of the data being inspected.
 Discovery Model

A more popular model is the Discovery Model that utilizes

a process that looks in a database to discover and/or
predict future patterns. The discovery model is divided
into two modes: Descriptive and Predictive.
 Discovery Model- Descriptive Mode

The Descriptive mode finds hidden patterns without a

predetermined idea or hypothesis about what the patterns
may be. In other words, the Data Mining software or
program takes the initiative in finding what the interesting
patterns are, without the user thinking of the relevant
questions first. In this mode information is created about
the data with very little or guidance from the user. The
exploration of the data is done in such a way as to yield as
large a number of useful facts about the data in the shortest
amount of time.
 Discovery Model- Predictive Mode

In the Predictive mode patterns discovered from the database are used
to predict the future patterns or trends. Predictive modeling allows the
user to submit records with some unknown field values, and the
system will guess the unknown values based on previous patterns
discovered from the database.

In comparing the two models, one can state that Verification can be
very inefficient, timely and costly. Whereas, Discovery modeling
can be very efficient, cost effective, less dependent on user input and
increases modeling accuracy.