Lecture 1 - Course Introduction & Dimensional Modeling

DATA WAREHOUSING
Professor: John Shantz

Lecture 1
Carnegie Mellon University
Pittsburgh, PA
CLASS AGENDA
Introductions and background

Syllabus & Course expectations
Data Warehousing Basics
Dimensional Modeling Introduction
Retail Sales Case Study
Dimensional Modeling Exercise
Project / Assignments
Data Warehousing
MY BACKGROUND
President, Data Warehouse Consultants LLC a
Pittsburgh-focused database and data warehouse
consulting company
Started company in 2004 to focus solely on data
warehousing consulting opportunities
Former Deloitte Consulting Manager one of the
founding members of Deloittes DW public sector
practice.
15+ years of experience in design, development and
implementation of data warehouse projects
Successful implementation of many different data
warehouses in various businesses
Master of Business Administration from Tepper School
of Business and EE undergraduate degree from Penn
State University
Data Warehousing
CLASS AGENDA
Introductions and Background

Data Warehousing
SYLLABUS REVIEW
Textbook
Ralph Kimball and Margy Ross. The
Data Warehouse Toolkit: The
Definitive Guide to Dimensional
Modeling (Third Edition).
ISBN: 1-118-53080-2
Data Warehousing
BLACKBOARD
Blackboard will be used as for the
class website.
Lecture slides, handouts and
announcements will be posted via
the website.
Data Warehousing
COURSE GOALS
Understand the basic components of a
data warehouse
Design a data warehouse based on user
requirements
Create a prototype data warehouse using
established principles discussed in class
This will essentially be a project course
and most work will revolve around your
groups project.
Data Warehousing
COURSE GRADING
Grading Criteria
Quizzes (2)
30%
Project Requirements & Design 15%
Project Presentation 15%
Course Project
30%
Class Participation 10%
Due to the project nature of this course and the required

group work, Pass/Fail grading will not be permitted.
This course is also not available for official Audit credit
for the same reasons.
Data Warehousing
QUIZZES
Two scheduled in-class quizzes
Focused on key principles of data warehousing
discussed in class and in the handouts
Scheduled for Week 3 and Week 6 (subject to change)
If you miss a quiz you will receive a zero for that score
unless you make alternate arrangements with me IN
ADVANCE. No make-ups or alternate arrangements
will be made after the quiz is given.
Alternate arrangements are not guaranteed, however,
and are made solely at my discretion based on the
individual students circumstances.
Data Warehousing
COURSE PROJECT
Once the basics of data warehousing have been
covered, the course project will become the focal point
of the class
Groups will consist of 3-4 members (depending on
class size) and be assigned randomly by me
The project is due the final week of class. I strongly
encourage you to begin the project in week 3 once
your groups have been assigned.
Every group member should know the subject and
goals of his/her project. Failure to be knowledgeable
on your groups activities will negatively affect your
evaluation.
Data Warehousing
10
REQUIREMENTS & DESIGN

The first deliverable for your group project is
the requirements and design document.
This will describe your design and serves as
your plan for how you intend to build your
data warehouse.
It will be due during Week 5 of class.
Data Warehousing
11
GROUP PRESENTATION
Each group will be required to
present a short synopsis on their
project during the final class.
Presentations will be approximately
10 to 12 minutes in length.
All group members are strongly
encouraged to be present for your
groups presentation
The final presentations will be
during the last day of class.
Data Warehousing
12
GROUP PRESENTATION (CONT.)

During each groups presentation,
another group will be chosen to provide
a critique when the presentation is
complete.
Specifically, we will be looking for
opinions on items that were done well
and aspects that could be improved.
An effective critique will contribute to
your own teams score for that group.
Data Warehousing
13
COURSE PROJECT
The final project is due the last week of class
Detailed instructions of what is required for
the project will be posted later in the year
Every group member should know the subject
and goals of his/her project. Failure to be
knowledgeable on your groups activities will
negatively affect your evaluation.
Data Warehousing
14
COURSE PROJECT SOFTWARE

It is important to note that dimensional data
modeling and data warehousing is software
independent. It can be done correctly in almost
any type of database.
To maintain grading consistency, however, I
require the project to be done in Microsoft SQL
Server 2012. This software is available from the
Heinz computing center.
Further instructions on how to submit your
project using the above software will be posted
with
the course project
assignment.
15
Data Warehousing
CLASS EXPECTATIONS SOFTWARE
To complete labs that I have planned you must install

Microsoft SQL Server 2012 database. Be sure to install
MS Analytic Services (MSAS) and MS Integration
Services (MSIS) when you install the software. This is
also the software I recommend you use to complete
your project. If you download Express versions from
Microsoft, this may be an issue.
The software and installation instructions for both

software packages are available from Heinz Computing
Services.
WORK ON SOFTWARE INSTALLATIONS EARLY AS THEY DO

NOT GO SMOOTHLY SOMETIMES.
Data Warehousing
16
COURSE ENROLLMENT & WAITLIST
The enrollment size of this course is limited due to the

time required in-class to complete the presentations. It
is simply not possible to extend the enrollment beyond
the current size.
If you are currently on the waitlist and are very

interested in pursuing the course, I suggest coming for
the first two weeks to see who adds/drops the course.
Historically, 2-5 students get in because of late drops.
If you do not get into the course, please see your
department representatives for you options. I dont
have the ability to increase the enrollment because of
the structure of the course.
Right now the course is being offered again next mini,
and there is another section this mini.
Data Warehousing
17
CLASS AGENDA

Data Warehousing
18
DATA WAREHOUSE DEFINITIONS

A data warehouse is
subject-oriented,
integrated,
time-varying,
non-volatile
a collection of data that is used primarily in
organizational decision making [Inmon, 1992]
Typically a database that is maintained

separately from the organizations
operational databases.
Data Warehousing
19
DATA WAREHOUSE VS. DATA MARTS

Enterprise Warehouse: collects all information
about subjects (customers, products, sales, assets,
personnel) that span the entire organization.
Requires extensive business modeling
May take years to design and build
Data Mart: a logical and physical subset of a data

warehouse; in its most simplistic form, represents
data from a single business process (e.g., retail
sales, retail inventory, purchase orders) [Kimball,
2002]
Faster roll out, but complex integration in the long run
Data Warehousing
20
REAL-WORLD DEFINITIONS
Whatever the business says it is
The distinctions really break down:
Data mart
Decision Support System
Data warehouse Data Analysis
Environment
Cube
Access database
Reporting system
Analytic Workspace
Ask four experts and youll get five

definitions.
Everyone needs a data warehouse!
Data Warehousing
21
DATA WAREHOUSING MARKET

Includes hardware, database software, and tools
Multi-billion dollar market segment
A maturing market - warehouses deployed in
virtually every industry:
manufacturing (e.g., order shipment)
financial (e.g., claims analysis, fraud detection)
retail (e.g., user profiling, inventory management)
transportation (e.g., fleet management)
telecommunications (e.g., call analysis)
utilities (e.g., power usage analysis)
healthcare (e.g., cost and treatment outcomes)
Data Warehousing
22
WHY USE A SEPARATE SYSTEM?

Performance
Operational databases are designed and
tuned for known transactions and workloads.
Complex decision-support queries would
degrade performance for operational
transactions.
Special data organization, access and
implementation methods needed for
multidimensional views and queries.
Data Warehousing
23

Function
Historical Data: Decision support requires
historical data, which operational databases do
not typically maintain.
Data Consolidation: Decision support requires
consolidation (aggregation, summarization) of
data from many heterogeneous sources:
operational databases, external sources.
Data Quality: Different sources typically use
inconsistent data representations, codes, and
formats which have to be reconciled..
Data Warehousing
24

Goal
Goal of transactional system is to capture data
quickly. Users have to use the system to do
their jobs.
Goal of data warehouse is to help the
organization run better. Most data warehouses
are valuable, but not absolutely necessary
(until someone uses one to eat your lunch).
Target Audience
Helps to maintain one version of the truth.
Data Warehousing
25
DATA WAREHOUSE COMPONENTS

Data Warehouse Database Server
Most always a relational DBMS
All major database companies now have offerings
OLAP Servers
Relational OLAP (ROLAP): extended relational DBMS that
maps operations on multidimensional data to standard
relational operations.
Multidimensional OLAP (MOLAP): special purpose server
that directly implements multidimensional data and
operations.
Tools or Clients
Extraction, Transformation and Load tools

Query and reporting tools
Analysis tools
Data mining tools (e.g., trend analysis, prediction)
Data Warehousing
26
DATA WAREHOUSING TOOLS

Database Servers
Oracle
Microsoft SQL Server
Sybase
IBM DB2
Microsoft Access
ETL Tools
Informatica PowerPlay
Ascential DataStage (IBM)
Hyperion Application Link
Oracle PL/SQL
Microsoft Data Transformation Services
Many Others
Data Warehousing
27
Source System 1
Extracts
Source System 2
Source System 2
Extracts
Transformation Logic
Source System 1
Analytical Data Filter
SAMPLE DATA WAREHOUSE

ARCHITECTURE
OLAP Cubes
Operational Data
Store (ODS)
or
Staging Area
Data Warehouse
Standard Reports
Data marts
Extraction
Transformation & Load
User View
METADATA
Data Warehousing
28
COMPARISON OF LEADING TOOL

SUITES
Cognos series 7
C ognos PowerPlay
IBM Cognos 8 BI
Business Objects
Enterprise
PowerPlay (BI Mobile

Analysis)
Desktop Intelligence /
OLAP Intelligence
SQL Server Analysis

Services (SSAS)
SQL Server Management
Studio
Desktop query and

analysis tool
Web Intelligence
Web-based
multidimensional
analysis
Microsoft Excel
Spredsheet integration
C ognos PowerPlay Web Analysis Studio
MS SQL Server BI
Comments
PowerPlay Excel
IBM C ognos 8 BI
Analysis for Microsoft
Excel
Impromptu
C rystal Reports
ReportNet (Report
Studio + Query Studio) WebIntelligence
SQL Server Reporting

Services (SSRS)
Reporting
C ognos WebPortal
C ognos C onnection
InfoView
Performance Point Server
Web Portal
GO! Dashboard
Performance manager
Report Studio
XC elsius
Business Intelligence
Development Studio
Visual dashboards
Visualizer
Live office
Xcelsius
Dashboard builder
DecisionStream
Data Manager
Data integrator (BODI)
SQL Server Integration

Services (SSIS)
ETL and data integraion

tool
Framework Manager
Designer
Development Studio
Modeling application
Development Studio
Scorecarding
BusinessObjects
Enterprise XI
Business activity
monitoring
Business Planning and

C onsolidation
Planning application
PowerPlay Transformer
Impromptu
Administrator
Metrics Manager
Metrics Studio
Notice cast
Event Studio
Planning
Planning
C ontroller
C ontroller
Performance manager
Dashboard manager
http://www.bi-dw.info/cognos-bo-sqlserver.htm
Data Warehousing
29
CLASS AGENDA

Data Warehousing
30
DIMENSIONAL MODELING
Transactional systems
Designed to allow for quick transactional processing and
efficient storing of data.
To accomplish this, designers typically use some type of
normalization. Most strive for Third Normal Form.
Analytical systems
Designed to extract and query data quickly
Access speed is the main concern
Hence, normalization which is widely used for transactional
databases, is generally not appropriate for data warehouse
design
Design should reflect multidimensional view
This is called a dimensional model or star schema
Data Warehousing
31
Source System 1
Extracts
Source System 2
Source System 2
Extracts
Transformation Logic
Source System 1
Analytical Data Filter
SAMPLE DATA WAREHOUSE

ARCHITECTURE
OLAP Cubes
Operational Data
Store (ODS)
or
Staging Area
Data Warehouse
Standard Reports
Data marts
Extraction
Transformation & Load
User View
METADATA
Data Warehousing
32
THE PROBLEM
Transactional models, while efficient for
transaction processing, are not good for
analytics
How do we
determine
the average
grade in
biology for
CMU in a
given
semester?
Data Warehousing
33
THE SOLUTION
Organize the data so it can be pulled out more
efficiently.
The number
of students
can be
counted by a
simple
aggregate
query based
on the fact
table.
Data Warehousing
34
COMPONENTS
Fact Table
Primary table which stores the performance measurements
of the business
The term fact refers to a business measure
Each row in a fact table corresponds to a specific
measurement
Each measurement is taken at the intersection of all the
relevant dimensions (e.g., day, product, and store) this list
of dimensions defines the grain of the fact table
All measurements in a fact table must be at the same grain
Facts are either additive, semiadditive, or nonadditive
most are numeric
Contains two or more foreign keys to dimension tables
Expresses the many-to-many relationships between
dimensions in dimensional models
Data Warehousing
35
COMPONENTS
Dimension Tables
Contain the textual descriptors of the business
Usually low in cardinality, but very wide (50100 attributes not uncommon)
Dimension attributes used as query
constraints, groupings, and report labels
The more descriptive the dimension attributes,
the better
Often contain hierarchical relationships
(city=>state=>region)
Data Warehousing
36
COMPONENTS
Fact Table + Dimension Tables =
Dimensional Model (Star Schema)
Benefits of dimensional model
Simplicity
Easy for business users to understand
Improved query performance
Extensibility
Easily accommodates change (but not that
easily!)
Data Warehousing
37
DIMENSIONAL MODELING PROCESS

Consists of four main steps:
1. Select the business process to model
2. Declare the grain of the business process
3. Choose the dimensions that apply to each fact
table row
4. Identify the facts
Dimensional modeling is part science, and part art
Data Warehousing
38
CLASS AGENDA

Data Warehousing
39
RETAIL CASE STUDY

Large grocery chain:
100 grocery stores spread over 5-state area

Each store has the following departments: grocery,
frozen foods, dairy, meat, produce, bakery, floral, and
health/beauty aids
Each store has 60K individual products or stock
keeping units (SKUs)
Each individual product is assigned and labeled with an SKU,

regardless of whether it is produced externally or internally
When a purchase occurs, the bar code is scanned into the point
of sale (POS) system
Pricing and promotion decisions represent an especially

interesting aspect of the business
Data Warehousing
40
RETAIL CASE STUDY
Profit comes from charging as much as possible for each

product, lowering acquisition costs/overhead, and attracting as
many customers as possible
Promotions used to attract customers, and include temporary
price reductions (TPRs), newspaper ads, in-store displays, and
coupons
Large increases in volume can be created by dramatic price
reductions
e.g., a 50-cent reduction in price of paper towels, especially when

coupled with an ad and a display, can cause sale of paper towels
to jump by a factor of 10
However, such huge price reductions are not sustainable, since

goods are likely being sold at a loss
Thus, impact of promotions is an important part of the analysis
of operations in the grocery store
Data Warehousing
41
RETAIL CASE STUDY

1. Select the business process to model:
Requires an understanding of both

business requirements and available data
Management wants to better understand

customer purchases as captured by the
POS system
The chosen business process we will
model is POS retail sales
Data Warehousing
42
RETAIL CASE STUDY

2. Declare the grain of the business process:
Specify exactly what an individual fact table row

represents the grain conveys the level of detail
associated with fact table measurements
It is highly recommended to choose the most granular or

atomic information captured by the business process
Why?
The grain that we will use in our retail example, which is

the most granular data available, is an individual line
item on a POS transaction
Data Warehousing
43
RETAIL CASE STUDY

Choose the dimensions:
Primary dimensions determined from grain:
Date
Product
Store
We also want to be able to see the effects

of promotions on each sale:
Add a Promotion dimension
Data Warehousing
44
RETAIL CASE STUDY

Identify the facts:
Facts collected by POS:
Sales quantity
Revenue or Sales dollar amount
(sales quantity * unit price)
Cost dollar amount
Data Warehousing
45
RETAIL CASE STUDY

We are also interested in gross margin:
Gross margin = gross profit/sales dollar amount, where
Gross profit = sales dollar amount cost dollar amount
Should we choose to store gross profit or gross margin as

a fact?
Gross profit is additive across all dimensions, whereas gross
margin is nonadditive
A fact is additive if we can sum the fact across all dimensions and
obtain a valid and correct number
A fact is nonadditive if the summation of the fact across any dimension
results in a meaningless, nonsensical number
A fact is semiadditive if it is additive across some dimensions and
nonadditive across other dimensions
Data Warehousing
46
GROSS MARGIN EXAMPLE

Product A
Price is $10, Cost is $5
Gross Profit is $5
Gross Margin is 50% (5 / 10)
Product B
Price is $100, Cost is $90
Gross Profit is $10
Gross Margin is 10%
Assume we sell one of each product

Is the gross margin 60% (10% + 50%)?
What is the gross margin for both products?
Data Warehousing
47
GROSS MARGIN EXAMPLE

Assume we sell one of each product
Revenue is additive
$100 + $10 = $110
Gross Profit is additive

$10 + $5 = $15
Gross Margin is not additive

($10 + $5) / ($100 + $10) = 13.6%
This value could not be calculated only from
the gross margin on each individual product or
transaction
Data Warehousing
48
RETAIL CASE STUDY

Preliminary Star Schema
Data Warehousing
49
RETAIL CASE STUDY

Query Example What products have the highest gross
margin in January?
SELECT SKU,
SUM(GrossProfit),
SUM(GrossProfit) / SUM (SalesDollars) AS GrossMargin
FROM
RetailSalesTransactionFact F
INNER JOIN DateDim D

ON F.TransactionCalendarDate = D.CalendarDate
WHERE
MonthName = January
GROUP BY SKU
Data Warehousing
50
CLOSER LOOK AT THE DIMENSIONS

Date Dimension
Example attributes:
Date
Full Date Description
Month Number
Month Name
Month Short Name
Day Number in Month
Day of Week
Day Number in Year
Year
Fiscal Quarter
Fiscal Year
Holiday Indicator
First Day of Quarter Indicator
Selling Season
Etc.
Data Warehousing
All data warehouses have a

Date/Time dimension
It is possible to pre-populate
Date dimension
Relatively small dimension table,

e.g., 10 years of days is only
about 3650 rows
Multiple hierarchies exist within

Date dimension
51
DATE DIMENSION EXAMPLE
Data Warehousing
52

Product Dimension
Example attributes:
SKU Number (Natural Key)

UPC
Product Description
Brand Description
Category Description
Department Description
Package Type Description
Package Size
Fat Content
Diet Type
Weight
Weight Units of Measure
Storage Type
Data Warehousing
Recall there are about 60K SKUs
Product dimension will contain

about 150K rows when
accounting for different
merchandising schemes across
stores and historical products
Product hierarchy:
SKU=>Brand=>Category=>Department
53

Store Dimension
Example attributes:
Store Number
Store Name
Store Street Address
Store City
Store County
Store State
Store Zip Code
Store Manager
Store District
Store Region
Floor Plan Type
Selling Square Footage
First Open Date
Data Warehousing
Represents primary geographic

dimension
Store hierarchies include:

Store=>State
Store=>District=>Region
Any number of different

geography or sales hierarchies
can exist in this dimension
54

Promotion
Dimension
A causal dimension it
describes factors believed to
cause a change in product sales
Useful in determining whether a

promotion is effective, e.g.:
Example attributes:
Promotion Code
Promotion Name
Price Reduction Type
Promotion Media Type
Ad Type
Display Type
Coupon Type
Ad Media Name
Display Provider
Promotion Cost
Promotion Begin Date
Promotion End Date
Data Warehousing
Whether products under

promotion experienced a gain in
sales during promotional period
Whether cannibalization
occurred
The different promotion types are

highly correlated, e.g., TPR, ad,
coupon, and display often occur
together
55
RETAIL CASE STUDY

Assume the chain now switches POS systems and must
renumber their SKUs and store numbers.
Anyone see a problem with this model
Data Warehousing
56
SURROGATE KEYS
It is highly recommended to use surrogate keys for dimension table keys
Surrogate keys are simply integers assigned sequentially to a particular dimension

row
Operational codes (e.g., SKU number) can still be retained for analysis purposes
Benefits of surrogate keys:
Buffer the data warehouse from changes in operational codes

Can save space due to their small size compared to operational codes
Allow recording of conditions which do not have an operational code (e.g., No
Promotion)
Allow handling of changes to dimension table attributes (to be discussed later)
The main disadvantage of using surrogate keys is that it requires some effort to
implement
Always, always, always use surrogate

keys!
Data Warehousing
57
STAR SCHEMA WITH SURROGATE

KEYS
Data Warehousing
58
STAR SCHEMA SIZE ANALYSIS

Product Dimension
150,000 products x 1 KB per row = 150 MB
Date Dimension
3,650 dates (10 years) x 1 KB per row = 3.5 MB
Store Dimension
100 stores x 2 KB per row = 0.2 MB
Promotion Dimension
5,000 promotions x 1KB per row = 5 MB
Total Dimensions = 158.7 MB
Data Warehousing
59
STAR SCHEMA SIZE ANALYSIS

Fact Table
Assume 10,000 transactions per day per store
10,000 purchases x 3650 days x 10 products per purchase
x 1 promotion per purchase
365,000,000 records x 1KB per record = 365 GB
Total Size = Fact + Dimensions

Total Size = 365,000 MB + 158.7 MB = 365.2 GB
Sizing rule when calculating size, the size of the
dimension tables can usually be ignored.
Data Warehousing
60
CLASS AGENDA

Data Warehousing
61
COURSE PROJECT
Project groups will be assigned by
me after the late drop period next
week.
Groups will consist of 4-5 members
Data Warehousing
62
COURSE PROJECT
Its not too early to begin thinking
about your topic for the course
project.
What you will need
A business objective (real or
plausible)
A source of data
An interest in the topic (this could be
important!)
Data Warehousing
63
READING ASSIGNMENTS
Kimball Chapters 1, 2 and 3
Chaudhuri and Dayal, An
Overview of Data Warehousing and
OLAP Technology, Sections 1-7
(available on Blackboard)
Data Warehousing
64
CLASS AGENDA

Data Warehousing
65
DIMENSIONAL MODELING DESIGN EXERCISE

Pittsburgh, PA
66

Lecture 1 - Course Introduction & Dimensional Modeling

Caricato da

Informazioni sul documento

Descrizione originale:

Titolo originale

Copyright

Formati disponibili

Condividi questo documento

Condividi o incorpora il documento

Opzioni di condivisione

Hai trovato utile questo documento?

Questo contenuto è inappropriato?

Copyright:

Formati disponibili

Lecture 1 - Course Introduction & Dimensional Modeling

Caricato da

Copyright:

Formati disponibili

DATA WAREHOUSING

Professor: John Shantz

Introductions and background

Carnegie Mellon University

Carnegie Mellon University

Introductions and Background

Carnegie Mellon University

Carnegie Mellon University

Carnegie Mellon University

Carnegie Mellon University

Due to the project nature of this course and the required

Carnegie Mellon University

Carnegie Mellon University

Carnegie Mellon University

REQUIREMENTS & DESIGN

Carnegie Mellon University

Carnegie Mellon University

GROUP PRESENTATION (CONT.)

Carnegie Mellon University

Carnegie Mellon University

COURSE PROJECT SOFTWARE

CLASS EXPECTATIONS SOFTWARE

To complete labs that I have planned you must install

The software and installation instructions for both

WORK ON SOFTWARE INSTALLATIONS EARLY AS THEY DO

Carnegie Mellon University

COURSE ENROLLMENT & WAITLIST

The enrollment size of this course is limited due to the

If you are currently on the waitlist and are very

Carnegie Mellon University

Introductions and Background

Carnegie Mellon University

DATA WAREHOUSE DEFINITIONS

Typically a database that is maintained

Carnegie Mellon University

DATA WAREHOUSE VS. DATA MARTS

Data Mart: a logical and physical subset of a data

Carnegie Mellon University

Ask four experts and youll get five

Carnegie Mellon University

DATA WAREHOUSING MARKET

Carnegie Mellon University

WHY USE A SEPARATE SYSTEM?

Carnegie Mellon University

WHY USE A SEPARATE SYSTEM?

Carnegie Mellon University

WHY USE A SEPARATE SYSTEM?

Carnegie Mellon University

DATA WAREHOUSE COMPONENTS

Extraction, Transformation and Load tools

Carnegie Mellon University

DATA WAREHOUSING TOOLS

Carnegie Mellon University

Analytical Data Filter

SAMPLE DATA WAREHOUSE

Transformation & Load

COMPARISON OF LEADING TOOL

PowerPlay (BI Mobile

SQL Server Analysis

Desktop query and

C ognos PowerPlay Web Analysis Studio

SQL Server Reporting