Oracle 11g and OBIEE Data Warehousing and Analytics

Analytics and DW Using Oracle 11g and OBIEE 10g
Mark Rittman, Director, Rittman Mead Consulting
Who Am I?
Oracle BI&W Architecture and Development Specialist Co-Founder of Rittman Mead Consulting Oracle BI&W Project Delivery Specialists 10+ years with Discoverer, OWB etc Oracle ACE Director, ACE of the Year 2005 Writer for OTN and Oracle Magazine Longest-running Oracle blog http://www.rittmanmead.com/blog Chair of UKOUG BIRT SIG Co-Chair of ODTUG BI&DW SIG Second year of OU BI Masterclasses 18 countries visited in 2006-7
Rittman Mead Consulting

Oracle BI&DW Project Specialists Consulting, Training, Support Works with you to ensure OBIEE project success Small, focused team OWB, Oracle BI, DW technical specialists Clients in the UK, Europe, USA
The Oracle 11g Data Warehousing Platform

The latest release of the Oracle database Enterprise Edition contains a large number of data warehousing features Partitioning Bitmap indexes Star transformations Compression Options for in-database OLAP and Data Mining Oracle Warehouse Builder for database-centric ETL and Data Quality
The Oracle 10g Business Intelligence Platform

Based around Oracle Business Intelligence Enterprise Edition Dashboards, Answers, Delivers, Oracle BI Server Complete set of web-based query, reporting and analysis tools Based on Siebel Analytics Heterogeneous database access Logical Business Model Data integration capabilities Add-ons for data integration, real-time decisions
Oracle Fusion Middleware

Middleware architecture for Oracle products SOA-enabled J2EE environment Security, BI, user interaction, systems management, tools, BPM
Hyperion Financial Management Tools

Financial Management tools Hyperion Planning, Hyperion Performance Scorecard Hyperion Capital Expenses Planning, Hyperion Workforce Planning Essbase OLAP Server Hyperion System 9 infrastructure ex-Brio reporting and dashboard tools
The Future Oracle BI&DW Platform

Leverages the analytic capabilities of the Oracle database In-database OLAP, data mining, partitioning, analytic queries Scalable store of detail-level data for analysis Business logic and metadata layer provided by Oracle BI Server (OBIEE) Semantic model, calculations, time-series functions, hierarchies Oracle BI Presentation Server (OBIEE) plus Hyperion EPM tools provide Presentation Layer Three-tier BI&DW architecture leverages Oracle RDBMS and OBIEE/Hyperion toolset
The Data Services Layer

Based around a core Oracle Data Warehouse Or several federated data warehouses Data Warehouse has three tiers Staging Atomic/Normalized Performance/Dimensional ETL process for loading DW OWB (default ETL tool) ODI (heterogenous sources /targets) Data Services Layer also provides real-time and SOA feeds to OBIEE For applications not yet in DW For real-time reports For transitory requirements
The Business Logic Layer

Contains an integrated, logical model of the business Provides an abstraction layer for the business Can be auto-generated by the ETL tool OWB 11gR2 Maintained by the business Can connect to multiple sources in the data services layer The Oracle data warehouse Direct application access OLAP cubes Relational databases
The Presentation Layer

Data access layer, query tools, interfaces, MS Ofce integration Oracle BI Answers, Dashboard, BI Publisher BI Ofce integration SOAP (SOA) interface ODBC interface (BO, Cognos etc) Hyperion tools May access Essbase directly, by-passing Business Logic Layer

An in-depth look at the technologies behind Oracles future BI&DW platform Whats new in Oracle Database 11g for data warehousing How the Oracle BI Server provides the business logic for your BI&DW platform Creating the Enterprise Semantic Model, how the technology works and what are its strengths and weaknesses Where to place calculations, summaries and hierarchies in your architecture How Oracle BI Presentation Server provides dashboards and reports (and what its missing in the current release) Where the Hyperion tools t in Whats new in Oracle OLAP 11g, and an introduction to Oracle Essbase
Seminar Agenda
Introduction ETL and DW development using Oracle 11g and OWB 11g Coffee Creating the BI foundation later with OBIEE BI Server Lunch Reporting and Dashboards using OBIEE Presentation Server Coffee Analytics and OLAP Analysis using Oracle OLAP and Oracle Essbase Close
Seminar Material
Presentation slides are available in PDF format http://www.rittmanmead.com/presentations Leave a business card and I will forward on copies, answer questions Visit http://www.rittmanmead.com for more OBIEE material
Participant Introduction
The most important people in this room are you It helps if I know a bit about you Your Oracle BI experience Your course objectives Where you work Introduce yourself to me, and to the other seminar participants
Seminar Participation
Questions are welcome at any time During the session During breaks At the end of the day Hopefully Ill know the answer If not, Ill get back to you Your feedback is appreciated What worked well, what could be better Flow of slides and product coverage Fill in the evaluation forms (Oracle and mine)

Mark Rittman, Director, Rittman Mead Consulting
Analytics and DW Using Oracle 11g and Oracle BIEE

Lesson 1 : ETL and DW development using Oracle 11g and OWB 11g (and ODI)
Objectives
Understand the data warehousing features in the Oracle database See whats new in Oracle Database 11g for data warehousing Partitioning Cube Organized Materialized Views See whats new in OWB11g SCD2 Handling that works (sort of) DML Error Logging Get an introduction to ODI 10.1.3.4 Web Services functionality

Oracle Database 11g Oracle Warehouse Builder Oracle OLAP Option Oracle Data Mining Option Partitioning Option


Oracle 11g Enterprise Edition Code Features

Bitmap Indexes Materialized Views Star Transformations Cost-Based Optimizer Oracle Warehouse Builder Core Functionality
Oracle 11g Data Warehousing Options

Partitioning OLAP Data Mining OWB Data Quality Option OWB Enterprise ETL Option OWB ERP Connectors
Partitioning
Use table partitions to make large (fact) tables more manageable age out old data by dropping partitions add new partitions to receive new data set unchanging tablespaces to READ ONLY to optimize database backup speed data load by using partition exchange can reduce query time by partition elimination (pruning) parallel query
Annual Data
Jan Feb
Mar
Oracle OLAP Option

Adds multi-dimensional data types and a multi-dimensional engine Stores data in indexed arrays Adds an OLAP API and an SQL access layer (Can be) Faster than star schemas for loading and querying Supports time-series and dimensional queries Based on Express Server technology Provides consistently-fast access to summarized data
Oracle Data Mining Option

In-database data mining engine Accessed via Java and PL/SQL APIs Wide range of algorithms Anomaly detection Attribute importance Association rules Clustering Classication & regression Nonnegative matrix factorization Structured & unstructured data (text mining) BLAST (life sciences similarity search algorithm)
New Features in Oracle OLAP 11g

The latest release of the Oracle Database OLAP Option Continues the objective of embedding OLAP in the database engine SQL SELECT access Cubes stored in Analytic Workspaces Runs faster in 11g, uses SecureFiles instead of LOBs Metadata in the Oracle Data Dictionary Automatic creation of views over AW objects Optimizer now AW view aware CUBE_TABLE for easier direct SQL access CUBE access now detailed in execution plans
Cube Organized Materialized Views

Transparently enhance the query performance of BI applications Data is managed in an Oracle cube Fast refresh Fast query Manage a single cube instead of 100s of individual MVs Applications query the base relational tables Oracle automatically rewrites the queries to use the OLAP cube Access to the summary data in the OLAP cube is fully transparent
Positioned As the Oracle DW Enhancer

Transparent integration with SQL Swap-in replacement for Materialized Views Refreshed and loaded in the same way as Materialized Views Either accessed directly using SQL views, or indirectly through query rewrite
OLAP Cubes Now Feature in Explain Plans

Cube operations now feature in execution plans CUBE_SCAN shows data being retrieved from a Cube Organized MV
explain plan for select sum( u.units), c.market_segment_dsc, t.year_dsc, p.family_id from units_fact u, customer_dim c, time_dim t, product_dim p where u.ship_to_id = c.ship_to_id and u.month_id = t.month_id and u.item_id = p.item_id group by c.market_segment_dsc, t.year_dsc, p.family_id; PLAN_TABLE_OUTPUT ----------------------------------------Plan hash value: 3517365018 ------------------------------------------------------------------------------------------| Id | Operation | Name | Rows | Bytes | Cost (%CPU)| Time | ------------------------------------------------------------------------------------------| 0 | SELECT STATEMENT | | 19 | 2375 | 37 (6)| 00:00:01 | | 1 | HASH GROUP BY | | 19 | 2375 | 37 (6)| 00:00:01 | |* 2 | HASH JOIN | | 19 | 2375 | 36 (3)| 00:00:01 | |* 3 | HASH JOIN | | 19 | 2071 | 33 (4)| 00:00:01 | |* 4 | CUBE SCAN PARTIAL OUTER| CB$UNITS | 20 | 2000 | 29 (0)| 00:00:01 | | 5 | TABLE ACCESS FULL | TIME_DIM | 96 | 864 | 3 (0)| 00:00:01 | | 6 | TABLE ACCESS FULL | CUSTOMER_DIM | 61 | 976 | 3 (0)| 00:00:01 | -------------------------------------------------------------------------------------------
Demonstration
New In 11g - Partitioning Improvements

Interval Partitions Automatically creates partitions as they are needed Saves scripting the creation of new partitions each day/week/month Useful as long as youre not using Partition Exchange Loading
New in 11g - Virtual Columns

Dene a column based on other columns in the table Takes up no storage space Equivalent to calculated measures in Oracle OLAP Partitioning can be based on these virtual columns
New in 11g - REF Partitioning

Partition a table based on a column in another table Partition an detail table by the master table primary key Useful, but actually not relevant for star schemas The wrong way around, we want to partition the dimensions by the fact keys
New in 11g - More Partitioning Combinations

Two-dimensional partitioning schemes Partitions and sub-partitions Partition fact table primarily by date, then create list sub-partitions Some combinations were available in 9i, 10g Many more available in 11g
Oracle Warehouse Builder

Oracles Full DW Lifecycle tool Part of the Database Technology Stack Now at version 11g Release 1 Relational and OLAP Data Modeling Data Mapping Process Flows Data Proling, Auditing and Error Handling Metadata Management Change Management ERP Connectivity
New in OWB11g - DML Error Logging

Table operators in mappings can have Error Tables congured Erroneous rows (from 10.2 DB onwards) moved into error table Uses DML Error Logging Allows direct-path (set-based) loading without error roll-back Traps constraint errors, others still fail Free option, no dependency on Data Quality Option Actually faster than (row-based) Data Rules
Demonstration
DML Error Logging in OWB11g / 10.2.0.3
New in OWB11g - SCD2 Handling (That Works)

SCD2 handling now works in OWB11gR1 (and OWB 10.2.0.3) Changes to levels above base level now handled properly Hierarchy versioning All attributes in a row with trigger attributes must now also be triggers No ability to mix SCD1 and SCD2 in a single dimension Watch out for performance though Performance bug in 10.2.0.3 and 11gR1, look for upcoming patch
Recently Available - Experts on OTN

XML Input, AW Import, Dimension Derivation Experts etc Mapping from view/SQL SELECT
SELECT ! DECODE(GROUPING(channel_desc) , 1, 'Multi-channel sum' , channel_desc) AS Channel , DECODE (GROUPING (country_iso_code) , 1, 'Multi-country sum' , country_iso_code) AS Country , TO_CHAR(SUM(amount_sold) , '9,999,999,999') SALES$ FROM sales , customers , times , channels , countries WHERE sales.time_id=times.time_id AND sales.cust_id=customers.cust_id AND customers.country_id = countries.country_id AND sales.channel_id= channels.channel_id AND channels.channel_desc IN ('Direct Sales', 'Internet') AND times.calendar_month_desc= '2000-09' AND country_iso_code IN ('GB', 'US') GROUP BY CUBE(channel_desc, country_iso_code);
Alternative Data Warehouse Data Sources

Web Services Advanced Queues and Change Data Capture Streams Enterprise Service Bus Business Activity Monitoring
Web Service Data Sources

Many organizations are adopting SOA architectures Data can now arrive through services and events, as well as from databases ETL process need to be SOA-aware Web services and business events are now potentially DW data sources
SOA Architecture
BPEL Orchestration
Business Activity Monitoring Web Services Manager Declarative Rules Engine Enterprise Service Bus
ETL Process
Mappings Metadata
Bulk Data Processing
Advanced Queues and CDC Streams

Trickle feed of incremental data Arrives via queues, streams Often need to process in real-time Some platforms (i.e. Oracle RDBMS) have native support ETL tools need to be CDC-aware
Initial bulk load CDC for synchron ization Incremental load of new data CDC for loopback synchron ization
CDC
DW Sources
DW Target
-------------
CDC
Metadata
Oracle Data Integrator

Comprehensive data integration platform Extraction, transformation and load of data, to and from databases, events and Web services Batch and real-time loading Java-based, hot-pluggable, database independent
Oracle Data Integrator Architecture
Extensible Knowledge Modules

Provided out of the box, and are user extensible Written for specic source and target platforms User created example : Oracle 10g Data Pump
Event and Service-Based Integration

Integrates data from events (les arrive, data arrives on queue) and from services (SOA, Web services) Perform integration in real-time (Enterprise Information Integration) Oracle Data Integrator Reects todays reality that Event Conductor Service Conductor not all data lives in databases Event-oriented Service-oriented Integrates with Integration Integration Oracle SOA Suite
Metadata Declarative Design
Data-oriented Integration Data Conductor
ODI Web Service Functionality

Data Services Publish data stores (tables, views etc) as web services ODIInvokeWebService tool Request data from a web service as part of a package ODI Public Web Services Invoke an ODI scenario through a web service call
Oracle Data
Event-oriented
Service-
Metadata Declarative Design
ODI Data Services

Publish a data store as a web service Insert, Update, Select, Delete List (multiple requests in one) Inbuilt interface to Apache Axis2 1.2
ODIInvokeWebService Tool
Request data from a web service Called as a package step (in OWB terms, a process ow) Data is fetched into an XML document XML document then processed and loaded into a table Most suited to use in initial data preparation, then loaded into staging table for access by ODI interface
ODI Public Web Services

Invoke an scenario (a published ODI package) via a web service call Call ODI functionality from an application, a BPEL process Add as a partner link in the JDeveloper BPEL editor Add an Invoke activity to call the InvokeScenario operation Add an Assign activity to set operation parameters (repository, scenario etc)
Possible Web Services & ODI Scenarios

To provide bulk processing functionality to a business process BPEL orchestration of an ETL process ODI Public Web Services Service-based access to data mart or data warehouse data Data Services Gather data from a web service prior to an ETL operation Call a human workow process as part of an ETL control routine ODIInvokeWebService tool
Oracle Data Integrator and OWB

Complements OWB by adding advanced data integration functions Works in the staging/integration layer Pre-build packaged solutions for CDC, Web Services etc Supports event-based, service-based and real-time integration Complements the BI Server by handling large data volumes and data marts Provides heterogeneous functionality - OWB may be more appropriate for Oracle data warehouses Create ofine store (data mart) with history tables (SCD etc) Perform complex integration tasks Integrate SOA and event-based data Adds the ability to target heterogeneous platforms Supports real-time data integration
Next-Generation DW Objectives!
Incorporate data feeds from multiple sources Database tables Files Services Events Create a consolidated, integrated reporting database Incorporates batch-loaded and real-time data Uses database-centric and heterogeneous tools Tackle integration and standardization issues in a single place Provide an integrated store of data, indexed and optimized
Demonstration
ODI, Service and Event-Based Data Integration
Summary
Oracle Database 11g comes with a number of new data warehousing features Partitioning improvements Cube Organized Materialized Views OWB11g features bug xes, enhancements and support for 11g SCD2 handling that works (watch performance though) Support for DML Error Logging Data Warehousing in the Fusion era is no longer just about extracting from, and loading to, Oracle databases ODI provides access to heterogeneous data sources Non-Oracle databases Web services Events

Lesson 1 : ETL and DW development using Oracle 11g and OWB 11g (and ODI)

Lesson 2 : Creating the Business Logic Layer with OBIEE BI Server
Oracle BI Suite Enterprise Edition Plus

Oracle Next-Generation BI tools platform Based on Siebel Analytics Originally developed by nQuire Standards (J2EE) based Heterogeneous connectivity Server and Web-based Oracle BI Server Oracle Presentation Server Oracle BI Answers Oracle BI Delivers Oracle BI Administration Job Manager, Catalog Manager etc

The Business Logic Layer

Contains an integrated, logical model of the business Provides an abstraction layer for the business Can be auto-generated by the ETL tool OWB 11gR2 Maintained by the business Can connect to multiple sources in the data services layer The Oracle data warehouse Direct application access OLAP cubes Relational databases
OBIEE 10.1.3.2 Architecture

Web Server (IIS, Tomcat, Websphere, iPlanet)
SAW Bridge (J2EE/ISAPI) XML, HTML, XLS, PDF, TXT over HTTP/HTTPS
Web Browser
Javascript for Usability & Interactivity
Oracle BI Publisher
Delivery Server Layout Interfaces XSL Data Logic
Oracle BI Presentation Services

SOAP Web Services, XML and URL Interface Oracle Interactive Dashboards Oracle Answers
TCP/IP (SSL) Web Catalog Service XML Framework
User Profiling, Security and Session Mngmt Cache Services (Web) & Connection Mngmt
HTML, SOAP over HTTP/HTTPS
External Applications and Portals
ODBC over TCP/IP (SSL)
Oracle Delivers Server

Scheduling/Event Services Agent Execution Logic Device Adaptive Content
TCP/IP (SSL)
Oracle BI Server
Logical SQL ODBC/JDBC (Logical Business Model)
Session Management Intelligent Request Generation
Metadata Interchange
Load Balancer
Oracle BI Cluster Controller
Logical Request Generation Cache Services Navigator Multi-Pass / Sub-Request Logic Fragmentation Optimization Aggregate Navigator Optimized Query Rewrites Execution Engine
Oracle BI Administration
Metadata Management Services Multi-User Development Services Metadata Documentation Services Server Management Services
Externalized Authentication LDAP DB Authentication Custom Authenticator
Data Source Adapters

ODBC, CLI, OCI, XML, MDX Analytical and Operational Data Sources
System / Perf Monitoring
Security Services
Query Govern.
Oracle BI Server
Calculation and Data Integration Engine Virtual Data Warehouse Allows reporting across multiple sources Data accessed though metadata layer Connects to RDBMS and OLAP data Oracle Database (including Oracle OLAP) IBM DB2 Microsoft SQL Server Teradata Microsoft Analysis Services (OLAP) SAP BW 'info cubes' Security, Summary Management Translates incoming logical SQL in to optimized, physical SQL
Oracle BI Presentation Server

BI Application Server Single application that provides Oracle BI Answers, Dashboards Metadata stored in Web Catalog User Proling, Security and Session Management SOAP Web Services, XML and URL Interface
Web Browser

Oracle BI Delivers Server

Scheduling and distribution Uses iBots (Intelligent Bots) Integration (through Java) with BPEL iBot chaining, calling from BPEL, calling BPEL Permits alerts, through execution of Answers reports If report returns rows, alert triggered Alert appears on dashboard, via email, SMS, cell phone Can trigger narrative and Oracle Delivers Server conditionally-appearing reports
Scheduling/Event Services Agent Execution Logic Device Adaptive Content
Oracle BI Publisher Server

New in 10.1.3.2, based on Oracle XML Publisher Separate application, integrated with Oracle BI Presentation Server Initially usernames and passwords integrated Thereafter separately administered Provides high-delity layout, positioning of data, charts etc Multiple data sources OBIEE Metadata Layer Oracle BI Answers reports Oracle BI Publisher Any source accessible via SQL Delivery Server XML over HTTP Layout Interfaces
XSL Data Logic
Oracle BI Administration
MS Windows application for managing BI Server Creates and manages Common Enterprise Information Model Physical, logical and presentation layers Manage other aspects of BI Server Security Scheduler (Delivers server) Oracle BI Administration Jobs Joins Marketing Metadata Management Services Sessions Multi-User Development Services Cache Metadata Documentation Services Utilities and Tools Server Management Services Connect to BI Server via ODBC DSN
Core BI Server Functionality

Data Integration through Business Model View Calculation and analytical engine Support for RDBMS and OLAP data sources Intelligent Request Generation and Optimized Data Access Security and Data Scoping ODBC Interface out, native RDBMS and OLAP interfaces in Clustering, replication, support for Windows and Unix/Linux hosts, 32-bit and 64-bit
Data Access
Reporting data may be sourced from multiple databases, applications Oracle Database (including Oracle OLAP) IBM DB2 Microsoft SQL Server Teradata Microsoft Analysis Services (OLAP) SAP BW 'info cubes' SAP, Peoplesoft, Siebel, E-Business Suite Oracle BI Server joins this source data together and presents a logical business model Facts Dimensions Calculations
Customer
Common Enterprise Information Model

Metadata layer for OBIEE tools Three layers of metadata Physical layer, connections to sources Logical layer, denormalized facts and dimensions Presentation layer, report-centric folders Maintained using BI Administrator Model is the center of the OBIEE universe
Data Modeling using Oracle BI Server

Design goal for the BI Server metadata layer is to create a Star Schema Fact tables Dimension tables Drill paths Source data can either be pre-integrated (data mart, data warehouse) or integrated using the BI Server (federated) Federated can be faster to deliver (report in place) Assumes data sources share common key values Data marts are simpler and faster to query MDX (OLAP, Multi-dimensional) data is converted into RDBMS row-sets Logical layer is main integration point Presentation layer allows report centric folders
Model First Development

Business Model is centre of the OBIEE universe First element of design that is produced Integrated view of the organizations data Option 1: Star with model rst, then map on physical sources purest approach, uses theoretical model as basis Tricky to get joins and integration working Lots of manual logical table and column denition steps Option 2: Produce model on paper, then construct it using elements of physical model Quicker and faster, just drag and drop May end up with redundant columns Still needs a design phase at the start
Sample Business Model
Data Modeling Workow Steps

1. Dene logical model (on paper or using BI Administrator) 2. Import physical data sources 3. Create physical primary and foreign keys 4. Either drag physical tables to create logical tables, or map
Step 1: Create Physical Layer

Dene ODBC connection from client or server to database ODBC only needed to dene initial connection, import metadata Native connection (OCI etc) takes over from then Import tables, views, keys etc from data source Dene keys, foreign keys if needed Update row counts Dene table aliases
Physical Join Types

If source data has foreign key joins, they will be imported If not, they can be dened in the physical model Two types of join Foreign key joins (e.g. CUSTID = CUSTID) 1:N joins only Complex joins (TRAN_DATE between STARTDATEID and ENDDATEID) also supports 1:1, O:1, 1:O etc Important to create physical joins before importing in to logical layer Logical joins rely on physical joins to dene paths Makes life a lot simpler Create joins using Join Manager, or Diagrammer
Step 2: Create Logical Layer

Firstly, create new logical model Then create logical tables and columns Either create from scratch, then connect to physical Or dene by dragging and dropping physical tables Start off with single source for each table, then add connnections Dene dimensions to create drill paths Add calculations Use Rename Wizard to clean up table and column names
Calculations
OBIEE has its own SQL syntax Cross-platform Same SQL across all sources Calculations, analytics, time-series Calculation Wizard for time-series calculations AGO, TO_DATE Oracle BI Answers features SQL pass-thru Has to be enabled per user Allows native functions to be used Future: direct support for native SQL functions
Time-Series Calculations
OBIEE 10.1.3.2 supports limited time-series functions AGO value of a measure x months, quarters, years ago
AGO(Videostore."Sales fact view".Sales, Videostore.TimesDim."Month", 1)
TODATE value of a measure to the end of a month, quarter, year

AGO(Videostore."Sales fact view".Sales, Videostore.TimesDim."Quarter", 1)
Requires at least one time dimension in the metadata layer Requires at least one or more Chronological keys Monotonically increasing value (DAY etc)
Create Calculations in the Optimal Location

Whilst the BI Server can create calculations, it may be more appropriate to do this in the underlying database Oracle Database analytic functions (LAG, LEAD, RANK etc) Oracle Database OLAP functions (time-series, forecast, allocation) Use the available architecture efciently Perform calculations as close to the source data as possible Use dedicated OLAP technology if available Use BI Server to provide features not found in the sources Just because the BI Server can do something, doesnt mean it should do something
Logical Join Types

Logical layer supports both foreign key and complex joins Foreign key joins not recommended Restrict choices that BI server can make over join paths Only there for backward compatibility Complex joins are preferred Indicates relationship between tables Allows BI Server to determine best join path Useful when more than one Logical Table Source
Logical Table Sources

Logical tables can have more than one physical source Can create >1 Logical Table Source BI Server creates two physical queries Useful when mapping data from different levels and when using fragmentation Each Logical table source can have >1 table mapped BI server creates one query per LTS, joining as per physical foreign key joins Used when you need columns from other joined tables in your logical table (join only happens if additional columns requested)
Consider Where to Join Data

Again, consider where to integrate and join data Whilst the BI Server can join data, it may be faster to load into a data mart ETL tools can integrate data at time of load Denormalized data structures can be faster than joining at report time Indexing, Summary Management, Query Optimization Creating a data warehouse still is the optimal solution BIEE data source joining features are most appropriate when bringing new data in thats not yet in the data warehouse
Step 3: Create Presentation Layer

Business models themselves are not visible to users Users see presentation catalogs whose columns are dened in business models. Presentation catalogs appear as subject areas in the web UI Presentation catalogs are databases to client tools using ODBC To create a presentation catalog, drag business model to left panel. Then customize re-order, re-folder, re-name. Can create multiple Presentation Catalogs from single Logical Model
Initial Reporting Scenario

Global Company has a data warehouse, containing order and customer data Requirement is to take the existing warehouse (normalized) and turn it into a star schema (denormalized) For now, OBIEE will be used to map the data
Demonstration
Creating the initial Common Enterprise Information Model
Reporting Across Multiple Data Sources

OBIEE can combine and join data from multiple data sources Data is retrieved in separate SQL, MDX calls and joined using the BI Server Allows you to perform federated queries Watch out for performance though End goal should be to eventually move reporting data to a data warehouse
Federated Query Scenario

Additional data exists in an Excel spreadsheet Contains quotas, plus information on products and dates Quotas are at the product category and month level Existing data is at the product and day level Requirement is to join data, but only show quotas the relevant level
Joining A Second Data Source : Step 1

Import the new schema into the physical model, ensure FK and PK constraints are set

Use the Join Manager, or Physical Model Diagrammer, to create joins between tables in the two schemas

Add the new dimension tables as additional logical table sources to the exisiting logical dimension tables Set the dimension hierarchy level at which they are applicable

Add the additional measures to the fact table Set the dimension levels at which the measures become relevant
Demonstration
Adding a Second Data Source
Incorporating Real-Time Data

Initial data set consists of historical Customer Data Warehouse data Data also exists for real-time Order Bookings system Two logical models could be created, but ideal is to combine historical and real-time data in one model Known as Fragmentation
Fragmentation
Logical tables have >1 Logical Table Source Logical Table Source 1 = Historical Data Logical Table Source 2 = Real-Time Data
Fragmentation Step 1
Add new logical table source to Items logical table
Ensure mapped tables are added to new LTS
Add Fragmentation Content conditions
Repeat process for Dimensions Do not add fragmentation condition if same data in both dimension versions Add distinct clause instead
Demonstration
Adding real-time data using Fragmentation
Summary
The Business Logic layer is central to Oracles BI&DW Architecture Consists of physical, logical and presentation layers Foundation is properly-joined physical table sources Denormalized star schema built using complex logical joins Calculations (including time-series calculations) not available in the source database can be added to logical model Where possible though, use in-database analytics and OLAP Dimensions dened to determine drill paths Additional data sources can be combined in a single business model Real-time data can be incorporated using Fragmentation

Lesson 2 : Creating the Business Logic Layer with OBIEE BI Server

Lesson 3 : Reporting and Dashboards using OBIEE Presentation Server
Oracle BI Presentation Server

BI Application Server Single application that provides Oracle BI Answers, Dashboards Metadata stored in Web Catalog User Proling, Security and Session Management SOAP Web Services, XML and URL Interface
Web Browser



Oracle BI Answers
Query authoring tool for BI EE Report against business model Tables, Cross-tabs Graphs, Gauges View and column selectors Narratives Publish to Oracle BI Interactive Dashboards Database-neutral Connects through BI Server
Answers Views
Each report (request) contains a single query Possible to union, minus etc with similar queries Query has one or more views associated with it Table, pivot-table Chart, gauge, funnel chart Title, narrative View and column selectors Views are combined in the compound view Compound view generally displayed in dashboard
Answers Formatting Options

Columns can be formatted on column, or other column value Colors, fonts, Images and icons Can replace or supplement gures Images can be added to report titles Comes with a library of stock images
Oracle BI Interactive Dashboards

Main method of interacting with reports Fully interactive Drill on data, graphs Switch between views Drill out to detail Incorporate images, web content Role-based security Lightweight, DHTML
Dashboard Interactivity
Prompts can be added to dashboard Drop-down lists, multi-select Passes parameters to reports Switch between different views Change columns used in reports
Demonstration
Creating Reports and Dashboards using Oracle BI Presentation Server
Oracle BI Publisher
Publishes Answers reports as PDF, XLS, RTF etc Also direct access to metadata layer Pixel-perfect reports Used for distributionquality reporting Integrated into dashboard
Oracle BI Delivers
Scheduling and distribution Uses iBots (Intelligent Bots) Integration (through Java) with BPEL iBot chaining, calling from BPEL, calling BPEL Permits alerts, through execution of Answers reports If report returns rows, alert triggered Alert appears on dashboard, via email, SMS, cell phone Can trigger narrative and conditionally-appearing reports on dashboard
Dashboard Alerts & Guided Analytics

Enables content and layout of dashboard to change based on information analyzed Sections or links can be dened to only appear when there is interesting information in the data
Types of Guided Analytics

1.Adding a link to a dashboard, that only appears if a condition is met 2.Making reports appear on the dashboard, when a condition is met Requires two things A report to run, that determines whether the event has happened A report, or link, that you want to appear
Step 1 : Create a Trigger Report

Create a report that checks for your condition Has a branch fell below its sales threshold? Has a products year on year sales fallen? Use lters to check for the condition If the report returns rows, the trigger will be met Save to the Web Catalog
Step 2 : Add Guided Analytics Link to Dashboard

Drag and drop from Dashboard Objects palette Add to dashboard, just under the report to which it is related
Step 3 : Edit the Guided Analytics Link Properties

Select the report that is run to determine whether the link is displayed Select whether any rows, or no rows, is the trigger Enter link text Select report, dashboard or URL to link to
Step 4 : View Link in Dashboard (if Triggered)

Link only appears if condition is met Guides user to the next action they should take
Making Sections Display Conditionally

Individual sections can be displayed conditionally Select Guided Navigation from the section properties Pick the report to run
Demonstration
Guided Analytics and Conditional Display
Summary
Oracle BI Answers and Dashboards give you many ways to display and visualize data Many formatting and layout options are now possible Best practices exist for laying out dashboards and presenting data Reports and dashboards can be made interactive and navigable

Lesson 3 : Reporting and Dashboards using OBIEE Presentation Server

Lesson 4 : Analytics and OLAP Analysis using Oracle OLAP and Oracle Essbase




Oracle OLAP Option

Adds multi-dimensional data types and a multi-dimensional engine Stores data in indexed arrays Adds an OLAP API and an SQL access layer (Can be) Faster than star schemas for loading and querying Supports time-series and dimensional queries Based on Express Server technology Provides consistently-fast access to summarized data

The latest release of the Oracle Database OLAP Option Continues the objective of embedding OLAP in the database engine SQL SELECT access Cubes stored in Analytic Workspaces Runs faster in 11g, uses SecureFiles instead of LOBs Metadata in the Oracle Data Dictionary Automatic creation of views over AW objects Optimizer now AW view aware CUBE_TABLE for easier direct SQL access CUBE access now detailed in execution plans
Essbase Overview
Multi-dimensional database engine Calculations, aggregations, forecasting, allocation Foundation for many analytical applications Originally developed by Arbor, merged with Hyperion Software 1998 Part of Oracle Fusion Middleware End-User focused, very popular with business users
Essbase Architecture
ODBC connectivity to all major RDBMSs, plus SAP B/W Multi-dimensional OLAP engine Java and XML/A mid-tier Essbase query tools or OBIEE+
Essbase Analytic Capabilities

Drill, pivot, select, compare, rank, sort, lter, group, stack, calculate, annotate, write-back.... Reporting tools enabled by database outline Maps to the business
Essbase Calculation Ability

Wide range of functionality Simple aggregations to complex cross-dimensional allocations Results stored or calculated upon retrieval Relationship-aware Application-wide or subset-specic Conditional and procedural logic
Essbase Storage Options

Traditionally, Essbase uses the Block Storage Option Useful for dense, sparse/dense cubes Full range of calculations Slower to aggregate Recently introduced, the Aggregate Storage Option Useful for sparse data sets Generally read-only Fast to aggregate Similar storage model to MS AS, Oracle OLAP compressed composites
Query and Administration Languages

MaxL DDL for systems administration (SQL-like object denition language)
create or replace outline on database global.global
MDX for accessing data and metadata

With set [CUSTOMER_DIM3] as '[CUSTOMER_DIM].Levels(2).members' set [PRODUCT_DIM3] as '[PRODUCT_DIM].Levels(2).members' set [TIME_DIM3] as '[TIME_DIM].Levels(2).members' set [Q] as 'crossjoin ({[CUSTOMER_DIM3]},crossjoin ({[PRODUCT_DIM3]}, {[TIME_DIM3]}))' select { [Accounts].[SALES] } on columns, NON EMPTY {[Q]} properties ANCESTOR_NAMES on rows from [global.Global]
Essbase Development & Support Tools

Analytic Administration Services Analytic Integration Services Analytic Provider Services Hyperion Shared Services
Administration Services
Three-tier application for administering Essbase Console for editing, maintaining Essbase databases Java, launched through Web browser Allows you to maintain the OLAP database outline Denition of dimensions, hierarchies, measures
Integration Services
IDE for dening and loading an Essbase database Derive an Essbase cube from a relational star schema Dene dimensions, hierarchies, measures Populate cube Set up hybrid (HOLAP) storage Generate calculation scripts
OLAP Models and Metamodels

Metadata used by Integration Services OLAP Models dene the relational star schema used to source data for an Essbase database OLAP Metaoutline denes the dimensional model used in the Essbase database Administration Services can derive the metaoutline from the OLAP Model Outline can then be maintained in the Administration Services Console
Provider Services
Mid-tier that provides XML/A connectivity, Java API Fail-over, clustering Requirement for OBIEE and Smart View access
Smart View for Ofce

Microsoft Ofce Integration (Word, Excel, Powerpoint) General analytic capability Query designer, linked views Inclusion of Hyperion Visual Explorer Oracle Hyperion Planning Integration UI for Planning off-line BI+ repository Run-time prompts Multi-page support Editing POV of imported objects
Essbase Within Oracle BI Framework

Adds multi-dimensional analytic server to OBIEE BI Foundation Calculations, fast analysis, predictive modeling
Creating an Essbase Cube from Oracle Data

1. Create new OLAP Model using Integration Services 2. Connect to the source database 3. Create fact table and associated Account dimension 4. Create other dimensions, including a time dimension if required 5. Review dimensional model 6. Create hierarchies for the dimensions 7. Create the OLAP Metaoutline, dene the measures 8. Create the database using Administration Services, choose storage option 9. Use Integration Services to load members and data 10.Preview OLAP cube in Administration Services
Creating in Essbase Cube : Step 1

Start Integration Services console, select New OLAP Model Select ODBC Data Source that provides a connection to the source schema
Creating in Essbase Cube : Step 1

Start Integration Services console, select New OLAP Model Select ODBC Data Source that provides a connection to the source schema
Creating an Essbase Cube : Step 2

Start up the Integration Services Console, show Intelligent Help (optional) Select Create Fact Table from menu, select table that contains measures Check box to create Account dimension Uncheck box to create Time dimension

Start up the Integration Services Console, show Intelligent Help (optional) Select Create Fact Table from menu, select table that contains measures Check box to create Account dimension Uncheck box to create Time dimension

Select Create Dimensions from the application menu

Select Create Dimensions from the application menu

View initial model in Integration Services

Create Hierarchies for all of the dimensions Create TIME_DIM as a Time Dimension Validate and then Save model

Create Hierarchies for all of the dimensions Create TIME_DIM as a Time Dimension Validate and then Save model

Create new MetaOutline Base MetaOutline on Model that you just created

Select Autodetect and create the dimensions from existing hierarchies

Select Autodetect and create the dimensions from existing hierarchies

Drag and drop Accounts dimension over to dimension list Set dimensions to dense or sparse, as needed

Drag and drop Accounts dimension over to dimension list Set dimensions to dense or sparse, as needed

Use Administration Services to create a new application Select Block Storage or Aggregate Storage

Use Administration Services to create a new application Select Block Storage or Aggregate Storage

Use Integration Services to carry out a member and data load



Use Administration Services to preview the cube Note : if no values in cube, check that use default calc script was selected when performing the member and data load
Demonstration
Creating an Essbase Cube using Oracle Data
Querying Essbase Cubes using Oracle BI EE

Essbase is a native data source for Oracle BI Enterprise Edition Uses XMLA compatibility feature of OBIEE Requires OBIEE 10.1.3.3.2, or 10.1.3.3.1 with registry edit HKEY_CURRENT_USER\Software\Oracle Corporation\Oracle BI\Oracle BI Administration Tool\Hidden Features Create new DWORD value called Essbase, set to 1 Some restrictions in 10.1.3.3.1 release, treat as for evaluation only
OBIEE 10.1.3.3.1 Guidelines & Limitations

Measure hierarchies not supported, all measures shown on same level Import process does not know that dimension member names may be unique, you will need to inform OBIEE of this in the logical dimension member properties EVALUATE function can now work against MDX data sources EVALUATE_AGGREGATE function will need to be used for Essbase to natively handle time-series functions (not function shipped by default) Essbase UDAs suppressed by default, can be enabled using EVALUATE Requires Essbase 9.3.1 stack Essbase Server 9.3.1 Essbase Provider Services 9.3.1
Registering an Essbase Cube in OBIEE Step 1

Use Oracle BI Administrator to Import from XMLA Requires Essbase Provider Services to be installed and running

Enter the connection details for the Essbase server URL = <server_name>:13080/aps/XMLA Press Update to pre-populate the Data Source eld Username = Essbase login ID Password = Essbase password Not Shared Services login

Select the cube that you want to import Essbase cubes are hypercubes, contain measures of the same dimensionality Cube brings across measures and dimensions

Select the cube that you want to import Essbase cubes are hypercubes, contain measures of the same dimensionality Cube brings across measures and dimensions

Cube appears as new database data source in OBIEE Physical layer Note different icons for dimensions and measures All joins are created automatically

Drag physical cube into logical layer All dimensions, joins, tables are automatically created Note that level names do not get imported

Rename levels to correct names

Drag logical model into Presentation Layer Delete all the Level 0 levels Not required, if selected will cause request to error

Presentation layer should now only contain the tidied-up column names, as if it was a relational schema Now ready for reporting against

Create request as normal, results are returned super-fast
Demonstration
Importing and Reporting against Essbase using Oracle BI Enterprise Edition
Oracle Database 11g OLAP Option Overview

The latest release of the Oracle Database OLAP Option In-database OLAP, tightly integrated with the Oracle database Data stored in Analytic Workspaces Runs faster in 11g, uses SecureFiles instead of LOBs Queried using SQL, OLAP DML, OLAP API Metadata in the Oracle Data Dictionary Fully compatible with RAC / Grid Computing

All AWs run faster through optimized SQL access layer, use of SecureFiles New 11g Cube Mode CUBE_ORGANIZED materialized views CUBE_TABLE SQL access method AWs refreshed using DBMS_MVIEW.REFRESH AWs qualify for query rewrite New query expression syntax Enhancements to AWM Automatic creation of SQL views Data security based on Hierarchies

Transparently enhance the query performance of BI applications Data is managed in an Oracle cube Fast refresh Fast query Manage a single cube instead of 100s of individual MVs Applications query the base relational tables Oracle automatically rewrites the queries to use the OLAP cube Access to the summary data in the OLAP cube is fully transparent
Automatic Generation of SQL Views

SQL views are automatically created when AW objects are dened Views over dimensions Views over hierarchies Views over cube Denormalized view over cube + dims Fully solved Maintained automatically
Cost-Based Aggregation
Alternative to level-based pre-aggregation 0 = no pre-summarization 1 = build composite, no pre-summarization Typical gure is 20, 50 is quite high Higher gures tell AWM to precompute more of the aggregate space
Positioned As the Oracle Data Warehouse Enhancer

Transparent integration with SQL Swap-in replacement for Materialized Views Refreshed and loaded in the same way as Materialized Views Either accessed directly using SQL views, or indirectly through query rewrite
Creating an 11g Analytic Workspace Step 1

Start up Analytic Workspace Manager 11g Select OLAP 11g Cube Type

Create dimensions as normal Create dimension Create levels Create hierarchies Create mappings

Switch to the Materialized View tab on the dimension Enable MV refresh Select refresh mode Fast does not work on initial release Check compatibility checklist Check MV details

Build the dimension using the Maintain Dimension menu item Check progress of build, and new CommandOutput log le View dimension in Dimension Viewer

Build the dimension using the Maintain Dimension menu item Check progress of build, and new CommandOutput log le View dimension in Dimension Viewer

Analyze the dimension so that it is eligible for Query Rewrite Updates stats used by the Cost-Based Optimizer

Dene the cube as normal New option to aggregate using Cost-based setting % is of the total cube, starting from the bottom hierarchy levels Adds % on to total size of cube (if using compression)

Use the cube Materialized View tab to set MV parameters Fast refresh does not work (yet) Enable Query Rewrite Check that all conditions are OK Check that the MV Details panel shows the MV as being created

Use Partitioning Advisor to advise on cube partitioning New in 11g - Capstone Partition can be pre-summarized



Storage Advisor recommends storage options to use Compressed vs. regular, dense dimensions vs. sparse Analyzes source data to determine sparsity

Storage Advisor recommends storage options to use Compressed vs. regular, dense dimensions vs. sparse Analyzes source data to determine sparsity

Maintain Cube as normal Check new Build Log and CommandOutput log le

View cube results in AWM Cube Viewer Check that everything aggregates properly, gures look correct

Create calculations using new Create Calculated Measure dialog New calculation types New expression language SQL-style syntax Easier for non-OLAP developers to use (Not OLAP DML or MDX)

Create calculations using new Create Calculated Measure dialog New calculation types New expression language SQL-style syntax Easier for non-OLAP developers to use (Not OLAP DML or MDX)

(Optional) Create Cube Script to load cube Steps for clearing dims and measures, loading, analyzing, OLAP DML etc Note - not compatible with MV refresh

View SQL Views over AW objects in SQL Developer View Materialized Views created by AWM

Use the MV Advisor to prepare source data for rewrite Create MV logs Create source tab constraints Create relational dimensions

Write SQL against the base-level tables Rewritten against AW (initial release joins back to dims, in future this will go)
explain plan for select sum( u.units), c.market_segment_dsc, t.year_dsc, p.family_id from units_fact u, customer_dim c, time_dim t, product_dim p where u.ship_to_id = c.ship_to_id and u.month_id = t.month_id and u.item_id = p.item_id group by c.market_segment_dsc, t.year_dsc, p.family_id; PLAN_TABLE_OUTPUT ----------------------------------------Plan hash value: 3517365018 ------------------------------------------------------------------------------------------| Id | Operation | Name | Rows | Bytes | Cost (%CPU)| Time | ------------------------------------------------------------------------------------------| 0 | SELECT STATEMENT | | 19 | 2375 | 37 (6)| 00:00:01 | | 1 | HASH GROUP BY | | 19 | 2375 | 37 (6)| 00:00:01 | |* 2 | HASH JOIN | | 19 | 2375 | 36 (3)| 00:00:01 | |* 3 | HASH JOIN | | 19 | 2071 | 33 (4)| 00:00:01 | |* 4 | CUBE SCAN PARTIAL OUTER| CB$UNITS | 20 | 2000 | 29 (0)| 00:00:01 | | 5 | TABLE ACCESS FULL | TIME_DIM | 96 | 864 | 3 (0)| 00:00:01 | | 6 | TABLE ACCESS FULL | CUSTOMER_DIM | 61 | 976 | 3 (0)| 00:00:01 | -------------------------------------------------------------------------------------------
Demonstration
Creating an Oracle Database 11g OLAP Option Cube, Enabling for Query Rewrite and SQL Access
Summary
Essbase is the analytical OLAP engine provided by Hyperion Standalone analytical applications, engine behind Hyperion Planning etc Query using MDX, Smart View, XML/A applications Powerful query features, calculations, high uptake by business users Integrates with OBIEE via XML/A interface Oracle OLAP is the in-database analytical solution for the Oracle RDBMS Tight SQL integration Accessed through SQL views or through Cube Organized Materialized Views Both are highly capable OLAP servers Essbase aimed at standalone analytical apps, integration with OBIEE Oracle OLAP aimed at database-centric OLAP, also integrates well with OBIEE

Lesson 4 : Analytics and OLAP Analysis using Oracle OLAP and Oracle Essbase

Closing Slides

The latest release of the Oracle database Enterprise Edition contains a large number of data warehousing features Partitioning Bitmap indexes Star transformations Compression Options for in-database OLAP and Data Mining Oracle Warehouse Builder for database-centric ETL and Data Quality
The Oracle 10g Business Intelligence Platform

Based around Oracle Business Intelligence Enterprise Edition Dashboards, Answers, Delivers, Oracle BI Server Complete set of web-based query, reporting and analysis tools Based on Siebel Analytics Heterogeneous database access Logical Business Model Data integration capabilities Add-ons for data integration, real-time decisions



An in-depth look at the technologies behind Oracles future BI&DW platform Whats new in Oracle Database 11g for data warehousing How the Oracle BI Server provides the business logic for your BI&DW platform Creating the Enterprise Semantic Model, how the technology works and what are its strengths and weaknesses Where to place calculations, summaries and hierarchies in your architecture How Oracle BI Presentation Server provides dashboards and reports (and what its missing in the current release) Where the Hyperion tools t in Whats new in Oracle OLAP 11g, and an introduction to Oracle Essbase
And Finally..."
Thank you all for attending I hope today has been of interest to you Please remember to complete the seminar evaluations Leave your contact details with me to forward on the slides Have a safe journey home Goodbye!

Oracle 11g and OBIEE Data Warehousing and Analytics

Caricato da

Informazioni sul documento

Descrizione originale:

Copyright

Formati disponibili

Condividi questo documento

Condividi o incorpora il documento

Opzioni di condivisione

Hai trovato utile questo documento?

Questo contenuto è inappropriato?

Copyright:

Formati disponibili

Oracle 11g and OBIEE Data Warehousing and Analytics

Caricato da

Copyright:

Formati disponibili

Analytics and DW Using Oracle 11g and OBIEE 10g

Mark Rittman, Director, Rittman Mead Consulting

Rittman Mead Consulting

The Oracle 11g Data Warehousing Platform

The Oracle 10g Business Intelligence Platform

Oracle Fusion Middleware

Hyperion Financial Management Tools

The Future Oracle BI&DW Platform

The Data Services Layer

The Business Logic Layer

The Presentation Layer

Analytics and DW Using Oracle 11g and OBIEE 10g

Analytics and DW Using Oracle 11g and OBIEE 10g

Analytics and DW Using Oracle 11g and Oracle BIEE

The Oracle 11g Data Warehousing Platform

The Future Oracle BI&DW Platform

The Data Services Layer

Oracle 11g Enterprise Edition Code Features

Oracle 11g Data Warehousing Options

Oracle OLAP Option

Oracle Data Mining Option

New Features in Oracle OLAP 11g

Cube Organized Materialized Views

Positioned As the Oracle DW Enhancer

OLAP Cubes Now Feature in Explain Plans

New In 11g - Partitioning Improvements

New in 11g - Virtual Columns

New in 11g - REF Partitioning

New in 11g - More Partitioning Combinations

Oracle Warehouse Builder

New in OWB11g - DML Error Logging

New in OWB11g - SCD2 Handling (That Works)

Recently Available - Experts on OTN

Alternative Data Warehouse Data Sources

Web Service Data Sources

Bulk Data Processing

Advanced Queues and CDC Streams

Oracle Data Integrator

Oracle Data Integrator Architecture

Extensible Knowledge Modules

Event and Service-Based Integration

Data-oriented Integration Data Conductor

ODI Web Service Functionality

Metadata Declarative Design

ODI Data Services

ODI Public Web Services

Possible Web Services & ODI Scenarios

Oracle Data Integrator and OWB

Analytics and DW Using Oracle 11g and Oracle BIEE

Analytics and DW Using Oracle 11g and Oracle BIEE

Oracle BI Suite Enterprise Edition Plus

The Future Oracle BI&DW Platform

The Business Logic Layer

OBIEE 10.1.3.2 Architecture

Oracle BI Presentation Services

TCP/IP (SSL) Web Catalog Service XML Framework

HTML, SOAP over HTTP/HTTPS

External Applications and Portals

ODBC over TCP/IP (SSL)

Oracle Delivers Server

Session Management Intelligent Request Generation

Oracle BI Cluster Controller

Externalized Authentication LDAP DB Authentication Custom Authenticator