Turban Ch1 Ch6

Business Intelligence: A Managerial Approach nd Edition) (2
Chapter 1: Introduction to Business Intelligence
Business PressuresResponses Support Model
1-2
Copyright 2011 Pearson Education, Inc. Publishing as Prentice Hall
Business Environment Factors

FACTOR Markets DESCRIPTION Strong competition Expanding global markets Blooming electronic markets on the Internet Innovative marketing methods Opportunities for outsourcing with IT support Need for real-time, on-demand transactions Desire for customization Desire for quality, diversity of products, and speed of delivery Customers getting powerful and less loyal More innovations, new products, and new services Increasing obsolescence rate Increasing information overload Social networking, Web 2.0 and beyond Growing government regulations and deregulation Workforce more diversified, older, and composed of more women Prime concerns of homeland security and terrorist attacks Necessity of Sarbanes-Oxley Act and other reporting-related legislation Increasing social responsibility of companies Greater emphasis on sustainability
Consumer demand Technology
Societal
1-3
Organizational Responses
Be Reactive, Anticipative, Adaptive, and Proactive Managers may take actions, such as:

Employing strategic planning. Using new and innovative business models. Restructuring business processes. Participating in business alliances. Improving corporate information systems. Improving partnership relationships. Encouraging innovation and creativity. cont>
1-4
Organizational Responses, continued

1-5
Improving customer service and relationships. Moving to electronic commerce (e-commerce). Moving to make-to-order production and ondemand manufacturing and services. Using new IT to improve communication, data access (discovery of information), and collaboration. Responding quickly to competitors' actions (e.g., in pricing, promotions, new products and services). Automating many tasks of white-collar employees. Automating certain decision processes. Improving decision making by employing analytics.
Business Intelligence (BI)
BI is an evolution of decision support concepts over time.
Meaning of EIS/DSS

Then: Executive Information System Now: Everybodys Information System (BI)
BI systems are enhanced with additional visualizations, alerts, and performance measurement capabilities. The term BI emerged from industry apps.
1-6
Definition of BI
BI is an umbrella term that combines architectures, tools, databases, analytical tools, applications, and methodologies. BI a content-free expression, so it means different things to different people. BI's major objective is to enable easy access to data (and models) to provide business managers with the ability to conduct analysis. BI helps transform data, to information (and knowledge), to decisions and finally to action.
1-7
The Architecture of BI
A BI system has four major components:

a data warehouse, with its source data business analytics, a collection of tools for manipulating, mining, and analyzing the data in the data warehouse; business performance management (BPM) for monitoring and analyzing performance a user interface (e.g., dashboard)
1-8
A High-level Architecture of BI
1-9
Components in a BI Architecture
The data warehouse is the cornerstone of any medium-to-large BI system.
Originally, the data warehouse included only historical data that was organized and summarized, so end users could easily view or manipulate it. Today, some data warehouses include access to current data as well, so they can provide real-time decision support (for details see Chapter 2).
Business analytics are the tools that help users transform data into knowledge (e.g.,
queries, data/text mining tools, etc.).
1-10
Styles of BI
MicroStrategy, Corp. distinguishes five styles of BI and offers tools for each:
1. 2. 3. 4. 5.
report delivery and alerting enterprise reporting (using dashboards and scorecards) cube analysis (also known as slice-anddice analysis) ad-hoc queries statistics and data mining
1-11
The Benefits of BI
The ability to provide accurate information when needed, including a real-time view of the corporate performance and its parts A survey by Thompson (2004)

Faster, more accurate reporting (81%) Improved decision making (78%) Improved customer service (56%) Increased revenue (49%)
See Table 1.2 for a list of BI analytic applications, the business questions they answer and the business value they bring.
1-12
Intelligence Creation and Use

A Cyclical Process of Intelligence Creation And Use
BI practitioners often follow the national security model depicted in this figure.
1-13
Intelligence Creation and Use
Steps Involved

Data warehouse deployment Creation of intelligence
Identification and prioritization of BI projects

By using ROI and TCO (cost-benefit analysis) This process is also called BI governance
BI Governance
Who should do the prioritization?

Partnership between functional area heads Partnership between customers and providers
1-14
BI Governance Issues/Tasks
1.
2. 3. 4. 5.
1-15
Create categories of projects (investment, business opportunity, strategic, mandatory, etc.) Define criteria for project selection Determine and set a framework for managing project risk Manage and leverage project interdependencies Continuously monitor and adjust the composition of the portfolio
Intelligence and Espionage
Stealing corporate secrets, CIA,
Intelligence vs. Espionage The way that modern companies ethically and legally organize themselves to glean as much as they can from their customers, their business environment, their stakeholders, their business processes, their competitors, and other such sources of potentially valuable information
Intelligence
Problem too much data, very little value
Use of data/text/Web mining (see Chapter 4, 5)
1-16
Transaction Processing Versus Analytic Processing
Transaction processing systems are constantly involved in handling updates (add/edit/delete) to what we might call operational databases.
1-17
ATM withdrawal transaction, sales order entry via an ecommerce site updates DBs Online analytic processing (OLTP) handles routine on-going business ERP, SCM, CRM systems generate and store data in OLTP systems The main goal is to have high efficiency
Transaction Processing Versus Analytic Processing
Online analytic processing (OLAP) systems are involved in extracting information from data stored by OLTP systems
Routine sales reports by product, by region, by sales person, etc. Often built on top of a data warehouse where the data is not transactional Main goal is effectiveness (and then, efficiency) provide correct information in a timely manner More on OLAP will be covered in Chapter 2
1-18
BI and Business Strategy
To be successful, BI must be aligned with the companys business strategy.
BI cannot/should not be a technical exercise for the information systems department.
BI changes the way a company conducts business by

improving business processes, and transforming decision making to a more data/fact/information driven activity.
1-19
BI should help execute the business strategy and not be an impediment for it!
Real-time, On-demand BI

The demand for real-time BI is growing! Is real-time BI attainable? Technology is getting there

Automated, faster data collection (RFID, sensors, ) Database and other software technologies (agent, SOA, ) are advancing Telecommunication infrastructure is improving Computational power is increasing while the cost for these technologies is decreasing
1-20
Trent -> Business Activity Management
Issues for Successful BI
Developing vs. Acquiring BI systems

Developing everything from scratch Buying/leasing a complete system Using a shell BI system and customizing it Use of outside consultants? It is easier to quantify costs Harder to quantify benefits
Justifying via cost-benefit analysis

Most of them are intangibles
1-21
Issues for Successful BI
Security and Privacy

Still an important research topic in BI How much security/privacy? BI must integrate into the existing IS
Integration of Systems and Applications
Often sits on top of ERP, SCM, CRM systems
Integration to outside (partners of the extended enterprise) via internet
customers, vendors, government agencies, etc.
1-22
Major BI Tools and Techniques
Tool categories

Data management Reporting, status tracking Visualization Strategy and performance management Business analytics Social networking & Web 2.0 New/advanced tools/techniques to handle massive data sets for knowledge discovery
1-23

Chapter 2: Data Warehousing
What is a Data Warehouse?
A physical repository where relational data are specially organized to provide enterprisewide, cleansed data in a standardized format The data warehouse is a collection of integrated, subject-oriented databases designed to support DSS functions, where each unit of data is non-volatile and relevant to some moment in time
1-25
Characteristics of DW

1-26
Subject oriented Integrated Time-variant (time series) Nonvolatile Summarized Not normalized Metadata Web based, relational/multi-dimensional Client/server Real-time and/or right-time (active)
Data Mart
A departmental data warehouse that stores only relevant data
Dependent data mart A subset that is created directly from a data warehouse Independent data mart A small data warehouse designed for a strategic business unit or a department
1-27
Data Warehousing Definitions
Operational data stores (ODS) A type of database often used as an interim area for a data warehouse Oper marts An operational data mart Enterprise data warehouse (EDW) A data warehouse for the enterprise Metadata Data about data. In a data warehouse, metadata describe the contents of a data warehouse and the manner of its acquisition and use
1-28
DW Framework
No data marts option Data Sources Access ERP ETL Process Select Legacy Extract Transform Integrate Other OLTP/wEB Load Replication External data Enterprise Data warehouse Metadata Data mart (Engineering) Routine Business Reporting Applications (Visualization)
Data mart (Marketing)
Data/text mining
POS
Data mart (Finance)
OLAP, Dashboard, Web
Data mart (...)
Custom built applications
1-29
DW Architecture
Three-tier architecture
1. 2.
3.
Data acquisition software (back-end) The data warehouse that contains the data & software Client (front-end) software that allows users to access and analyze data from the warehouse
Two-tier architecture
First 2 tiers in three-tier architecture is combined into one
Sometimes there is only one tier

1-30 Copyright 2011 Pearson Education, Inc. Publishing as Prentice Hall
DW Architectures
Tier 1: Client workstation
Tier 2: Application & database server
1-31
A Web-based DW Architecture
1-32
Teradata Corp. DW Architecture
1-33
Data Warehousing Architectures

Ten factors that potentially affect the architecture selection decision:
1. Information interdependence between organizational units 2. Upper managements information needs 3. Urgency of need for a data warehouse 4. Nature of end-user tasks 5. Constraints on resources
1-34
6. Strategic view of the data warehouse prior to implementation 7. Compatibility with existing systems 8. Perceived ability of the in-house IT staff 9. Technical issues 10. Social/political factors
Data Integration and the Extraction, Transformation, and Load (ETL) Process
Extraction, transformation, and load (ETL)
1-35
Representation of Data in DW
Dimensional Modeling a retrieval-based system that supports high-volume query access Star schema the most commonly used and the simplest style of dimensional modeling
Contain a fact table surrounded by and connected to several dimension tables Fact table contains the descriptive attributes (numerical values) needed to perform decision analysis and query reporting Dimension tables contain classification and aggregation information about the values in the fact table
Snowflakes schema an extension of star schema where the diagram resembles a snowflake in shape
1-36
Multidimensionality
Multidimensionality
The ability to organize, present, and analyze data by several dimensions, such as sales by region, by product, by salesperson, and by time (four dimensions)
Multidimensional presentation
Dimensions: products, salespeople, market segments, business units, geographical locations, distribution channels, country, or industry Measures: money, sales volume, head count, inventory profit, actual versus forecast Time: daily, weekly, monthly, quarterly, or yearly
1-37
Star vs Snowflake Schema
1-38
Analysis of Data in DW
Online analytical processing (OLAP)

Data driven activities performed by end users to query the online system and to conduct analyses Data cubes, drill-down / rollup, slice & dice, Generating queries (query tools) Requesting ad hoc reports Conducting statistical and other analyses Developing multimedia-based applications
OLAP Activities

1-39
Analysis of Data Stored in DW OLTP vs. OLAP
OLTP (online transaction processing)
A system that is primarily responsible for capturing and storing data related to day-to-day business functions such as ERP, CRM, SCM, POS, The main focus is on efficiency of routine tasks A system is designed to address the need of information extraction by providing effectively and efficiently ad hoc analysis of organizational data The main focus is on effectiveness
OLAP (online analytic processing)
1-40
OLAP vs. OLTP
1-41
OLAP Operations

Slice a subset of a multidimensional array Dice a slice on more than two dimensions Drill Down/Up navigating among levels of data ranging from the most summarized (up) to the most detailed (down) Roll Up computing all of the data relationships for one or more dimensions Pivot used to change the dimensional orientation of a report or an ad hoc querypage display
1-42
OLAP
Slicing Operations on a Simple TreeDimensional Data Cube
A 3-dimensional OLAP cube with slicing operations
Sales volumes of a specific Product on variable Time and Region
Product
Cells are filled with numbers representing sales volumes
Geography
Ti m
Sales volumes of a specific Region on variable Time and Products
Sales volumes of a specific Time on variable Region and Products
1-43
DW Implementation Issues
11 tasks for successful DW implementation

Establishment of service-level agreements and data-refresh requirements Identification of data sources and their governance policies Data quality planning Data model design ETL tool selection Relational database software and platform selection Data transport Data conversion Reconciliation process Purge and archive planning End-user support
1-44
DW Implementation Guidelines

Project must fit with corporate strategy & business objectives There must be complete buy-in to the project by executives, managers, and users It is important to manage user expectations about the completed project The data warehouse must be built incrementally Build in adaptability, flexibility and scalability The project must be managed by both IT and business professionals Only load data that have been cleansed and are of a quality understood by the organization Do not overlook training requirements Be politically aware
1-45
Successful DW Implementation Things to Avoid

Starting with the wrong sponsorship chain Setting expectations that you cannot meet Engaging in politically naive behavior Loading the data warehouse with information just because it is available Believing that data warehousing database design is the same as transactional database design Choosing a data warehouse manager who is technology oriented rather than user oriented
1-46
Successful DW Implementation Things to Avoid - Cont.
Focusing on traditional internal recordoriented data and ignoring the value of external data and of text, images, etc. Delivering data with confusing definitions Believing promises of performance, capacity, and scalability Believing that your problems are over when the data warehouse is up and running Focusing on ad hoc data mining and periodic reporting instead of alerts
1-47
Failure Factors in DW Projects

Lack of executive sponsorship Unclear business objectives Cultural issues being ignored
Change management

1-48
Unrealistic expectations Inappropriate architecture Low data quality / missing information Loading data just because it is available
Real-time/Active DW/BI
Enabling real-time data updates for real-time analysis and real-time decision making is growing rapidly
Push vs. Pull (of data)

Not all data should be updated continuously Mismatch of reports generated minutes apart May be cost prohibitive May also be infeasible
Concerns about real-time BI

1-49
Real-time/Active DW at Teradata
1-50
Enterprise Decision Evolution and DW
1-51
The Future of DW
Sourcing

Open source software SaaS (software as a service) Cloud computing DW appliances Real-time DW Data management practices/technologies In-memory processing (super-computing) New DBMS Advanced analytics
Infrastructure

1-52
BI / OLAP Portal for Learning

MicroStrategy, and much more www.TeradataStudentNetwork.com Pw: <check with TDUN>
1-53

Chapter 3: Business Performance Management (BPM)
Business Performance Management (BPM) Overview
Business Performance Management (BPM) is A real-time system that alert managers to potential opportunities, impending problems, and threats, and then empowers them to react through models and collaboration. Also called, corporate performance management (CPM by Gartner Group), enterprise performance management (EPM by Oracle), strategic enterprise management (SEM by SAP)
1-55
Business Performance Management (BPM) Overview
BPM refers to the business processes, methodologies, metrics, and technologies used by enterprises to measure, monitor, and manage business performance. BPM encompasses three key components
A set of integrated, closed-loop management and analytic processes, supported by technology Tools for businesses to define strategic goals and then measure/manage performance against them Methods and tools for monitoring key performance indicators (KPIs), linked to organizational strategy
1-56
BPM versus BI
BPM is an outgrowth of BI and incorporates many of its technologies, applications, and techniques.

The same companies market and sell them. BI has evolved so that many of the original differences between the two no longer exist (e.g., BI used to be focused on departmental rather than enterprise-wide projects). BI is a crucial element of BPM.
1-57
BPM = BI + Planning (a unified solution)
A Closed-loop Process to Optimize Business Performance
Process Steps
1. 2. 3. 4.
Strategize Plan Monitor/analyze Act/adjust
Each with its own process steps

Strategize: Where Do We Want to Go?
Strategic objective A broad statement or general course of action prescribing targeted directions for an organization Strategic goal A quantified objective with a designated time period Strategic vision A picture or mental image of what the organization should look like in the future Critical success factors (CSF) Key factors that delineate the things that an organization must excel at to be successful
1-59
Strategize: Where Do We Want to Go?

90 percent of organizations fail to execute their strategies The strategy gap
Four sources for the gap between strategy and execution:

1. 2. 3. 4.
Communication (enterprise-wide) Alignment of rewards and incentives Focus (concentrating on the core elements) Resources
1-60
Plan: How Do We Get There?
Operational planning
Operational plan: plan that translates an organizations strategic objectives and goals into a set of well-defined tactics and initiatives, resources requirements, and expected results for some future time period (usually a year).

Operational planning can be

Tactic-centric (operationally focused) Budget-centric (financially focused)
1-61
Plan: How Do We Get There?
Financial planning and budgeting
An organizations strategic objectives and key metrics should serve as top-down drivers for the allocation of an organizations tangible and intangible assets Resource allocations should be carefully aligned with the organizations strategic objectives and tactics in order to achieve strategic success
1-62
Monitor: How Are We Doing?
A comprehensive framework for monitoring performance should address two key issues:
What to monitor

Critical success factors Strategic goals and targets
How to monitor
1-63
Diagnostic control system

A cybernetic system that has inputs, a process for transforming the inputs into outputs, a standard or benchmark against which to compare the outputs, and a feedback channel to allow information on variances between the outputs and the standard to be communicated and acted upon
1-64
Pitfalls of variance analysis
The vast majority of the exception analysis focuses on negative variances when functional groups or departments fail to meet their targets Rarely are positive variances reviewed for potential opportunities, and rarely does the analysis focus on assumptions underlying the variance patterns
1-65
What if strategic assumptions (not the operations) are wrong?
1-66
Act and Adjust: What Do We Need to Do Differently?

Harrahs Closed-Loop Marketing Model
1-67
Performance Measurement
Performance measurement system A system that assists managers in tracking the implementations of business strategy by comparing actual results against strategic goals and objectives
Comprises systematic comparative methods that indicate progress (or lack thereof) against goals
1-68
Performance Measurement KPIs and Operational Metrics
Key performance indicator (KPI) A KPI represents a strategic objective and metric that measures performance against a goal Distinguishing features of KPIs

Strategy Targets Ranges
Encodings Time frames Benchmarks
1-69
Key performance indicator (KPI)

Outcome KPIs vs. (lagging indicators e.g., revenues) Driver KPIs (leading indicators e.g., sales leads)
Operational areas covered by driver KPIs

Customer performance Service performance Sales operations Sales plan/forecast
1-70
The drawbacks of using financial data as the core of a performance measurement:
Financial measures are usually reported by organizational structures and not by the processes that produced them Financial measures are lagging indicators, telling us what happened, not why it happened or what is likely to happen in the future Financial measures are often the product of allocations that are not related to the underlying processes that generated them Financial measures are focused on the short term returns
1-71
Good performance measures should:

Be focused on key factors. Be a mix of past, present, and future. Balance the needs of all stakeholders (shareholders, employees, partners, suppliers, etc.). Start at the top and trickle down to the bottom. Have targets that are based on research and reality rather than be arbitrary.
1-72
BPM Methodologies
Balanced scorecard (BSC) A performance measurement and management methodology that helps translate an organizations financials, customer, internal process, and learning and growth objectives and targets into a set of actionable initiatives "The Balanced Scorecard: Measures (HBR, 1992) That Drive Performance
1-73
BPM Methodologies Balanced Scorecard
1-74
BPM Methodologies
In BSC, the term balance arises because the combined set of measures are supposed to encompass indicators that are:

Financial and nonfinancial Leading and lagging Internal and external Quantitative and qualitative Short term and long term
1-75
BPM Methodologies

Aligning strategies and actions A six-step process

1. 2. 3. 4. 5. 6.
Developing and formulating a strategy Planning the strategy Aligning the organization Planning the operations Monitoring and learning Testing and adapting the strategy
1-76
BPM Methodologies
Six Sigma A performance management methodology aimed at reducing the number of defects in a business process to as close to zero defects per million opportunities (DPMO) as possible
1-77
BPM Methodologies
How to Succeed in Six Sigma

Six Sigma is integrated with business strategy Six Sigma supports business objectives Key executives are engaged in the process Project selection is based on value potential There is a critical mass of projects and resources Projects-in-process are actively managed Team leadership skills are emphasized Results are rigorously tracked
1-78
BSC + Six Sigma = Success (see Tech. Ins. 9.3)
BPM Architecture and Applications
BPM applications
1. 2. 3. 4. 5.
Strategy management Budgeting, planning, and forecasting Financial consolidation Profitability modeling and optimization Financial, statutory, and management reporting
1-79
Performance Dashboards
Dashboards and scorecards both provide visual displays of important information that is consolidated and arranged on a single screen so that information can be digested at a single glance and easily explored
1-80
1-81
Dashboards versus scorecards
Performance dashboards Visual display used to monitor operational performance (free form) Performance scorecards Visual display used to chart progress against strategic and tactical goals and targets (predetermined measures)
1-82
Dashboards versus scorecards
Performance dashboard is a multilayered application built on a business intelligence and data integration infrastructure that enables organizations to measure, monitor, and manage business performance more effectively
- Eckerson
Three types of performance dashboards:

1. 2. 3.
Operational dashboards Tactical dashboards Strategic dashboards
1-83
Dashboard design
The fundamental challenge of dashboard design is to display all the required information on a single screen, clearly and without distraction, in a manner that can be assimilated quickly" (Few, 2005)
1-84

Chapter 4: Data Mining for Business Intelligence
Data Mining Concepts and Definitions Why Data Mining?

More intense competition at the global scale Recognition of the value in data sources Availability of quality data on customers, vendors, transactions, Web, etc. Consolidation and integration of data repositories into data warehouses The exponential increase in data processing and storage capabilities; and decrease in cost Movement toward conversion of information resources into nonphysical form
1-86
Definition of Data Mining
The nontrivial process of identifying valid, novel, potentially useful, and ultimately understandable patterns in data stored in structured databases - Fayyad et al., (1996) Keywords in this definition: Process, nontrivial, valid, novel, potentially useful, understandable Data mining: a misnomer? Other names: knowledge extraction, pattern analysis, knowledge discovery, information harvesting, pattern searching, data dredging
1-87
Data Mining Characteristics/Objectives
Source of data for DM is often a consolidated data warehouse (not always!). DM environment is usually a client-server or a Web-based information systems architecture. Data is the most critical ingredient for DM which may include soft/unstructured data. The miner is often an end user. Striking it rich requires creative thinking. Data mining tools capabilities and ease of use are essential (Web, Parallel processing, etc.).
1-88
Data in Data Mining
Data: a collection of facts usually obtained as the result of experiences, observations, or experiments Data may consist of numbers, words, and images Data: lowest level of abstraction (from which information and knowledge are derived)
- DM with different data types? - Other data types?
1-89
What Does DM Do? How Does it Work?
DM extracts patterns from data
Pattern? A mathematical (numeric and/or symbolic) relationship among data items Association Prediction Cluster (segmentation) Sequential (or time series) relationships
Types of patterns

1-90
A Taxonomy for Data Mining Tasks
1-91
Data Mining Applications
Customer Relationship Management

Maximize return on marketing campaigns Improve customer retention (churn analysis) Maximize customer value (cross- or up-selling) Identify and treat most valued customers
Banking & Other Financial

Automate the loan application process Detecting fraudulent transactions Maximize customer value (cross- and up-selling) Optimizing cash reserves with forecasting
1-92
Data Mining Applications (cont.)
Retailing and Logistics

Optimize inventory levels at different locations Improve the store layout and sales promotions Optimize logistics by predicting seasonal effects Minimize losses due to limited shelf life
Manufacturing and Maintenance

1-93
Predict/prevent machinery failures Identify anomalies in production systems to optimize manufacturing capacity Discover novel patterns to improve product quality
Brokerage and Securities Trading

Predict changes on certain bond prices Forecast the direction of stock fluctuations Assess the effect of events on market movements Identify and prevent fraudulent activities in trading
Insurance

Forecast claim costs for better business planning Determine optimal rate plans Optimize marketing to specific customers Identify and prevent fraudulent claim activities
1-94

1-95
Computer hardware and software Science and engineering

Government and defense Homeland security and law enforcement Travel industry Healthcare Highly popular application areas for data mining Medicine Entertainment industry Sports Etc.
Data Mining Process: CRISP-DM
1-96
Data Mining Process: CRISP-DM

Step Step Step Step Step Step
1: 2: 3: 4: 5: 6:
Business Understanding Data Understanding Data Preparation (!) Model Building Testing and Evaluation Deployment
Accounts for ~85% of total project time
The process is highly repetitive and experimental (DM: art versus science?)
1-97
Data Preparation A Critical DM Task
1-98
Data Mining Process: SEMMA
1-99
Data Mining Methods: Classification

1-100
Most frequently used DM method Part of the machine-learning family Employ supervised learning Learn from past data, classify new data The output variable is categorical (nominal or ordinal) in nature Classification versus regression? Classification versus clustering?
Classification Techniques

1-101
Decision tree analysis Statistical analysis Neural networks Support vector machines Case-based reasoning Bayesian classifiers Genetic algorithms Rough sets
Decision Trees

Employs the divide and conquer method Recursively divides a training set until each division consists of examples from one class
1. 2. 3.
A general algorithm for decision tree building
4.
Create a root node and assign all of the training data to it. Select the best splitting attribute. Add a branch to the root node for each value of the split. Split the data into mutually exclusive subsets along the lines of the specific split. Repeat the steps 2 and 3 for each and every leaf node until the stopping criteria is reached.
1-102
Cluster Analysis for Data Mining

1-103
Used for automatic identification of natural groupings of things Part of the machine-learning family Employ unsupervised learning Learns the clusters of things from past data, then assigns new instances There is no output variable Also known as segmentation
Clustering results may be used to

Identify natural groupings of customers Identify rules for assigning new cases to classes for targeting/diagnostic purposes Provide characterization, definition, labeling of populations Decrease the size and complexity of problems for other data mining methods Identify outliers in a specific domain (e.g., rare-event detection)
1-104
k-Means Clustering Algorithm

k : pre-determined number of clusters

Algorithm (Step 0: determine value of k)
Step 1: Randomly generate k random points as initial cluster centers. Step 2: Assign each point to the nearest cluster center. Step 3: Re-compute the new cluster centers. Repeat steps 3 and 4 until some convergence criterion is met (usually that the assignment of points to clusters becomes stable).
Cluster Analysis for Data Mining k-Means Clustering Algorithm
1-106
Association Rule Mining

A very popular DM method in business Finds interesting relationships (affinities) between variables (items or events) Part of machine learning family Employs unsupervised learning There is no output variable Also known as market basket analysis Often used as an example to describe DM to ordinary people, such as the famous relationship between diapers and beers!
1-107

Input: the simple point-of-sale transaction data Output: Most frequent affinities among items Example: according to the transaction data Customer who bought a laptop computer and a virus protection software, also bought extended service plan 70 percent of the time" How do you use such a pattern/knowledge?

Put the items next to each other for ease of finding Promote the items as a package (do not put one on sale if the other(s) are on sale) Place items far apart from each other so that the customer has to walk the aisles to search for it, and by doing so potentially see and buy other items
1-108
Representative applications of association rule mining include
In business: cross-marketing, cross-selling, store design, catalog design, e-commerce site design, optimization of online advertising, product pricing, and sales/promotion configuration In medicine: relationships between symptoms and illnesses; diagnosis and patient characteristics and treatments (to be used in medical DSS); and genes and their functions (to be used in genomics projects)
1-109
Artificial Neural Networks for Data Mining
clustering/segmentation
Artificial neural networks (ANN or NN) is a brain metaphor for information processing a.k.a. Neural Computing Very good at capturing highly complex non-linear functions! Many uses prediction (regression, classification), Many application areas finance, medicine,
marketing, manufacturing, service operations, information systems, etc.

Synapse
Dendrites
Biological NN Synapse
Axon Axon
Biological versus Artificial Neural Networks
Neuron
Dendrites
Neuron
x1 w1
Inputs
Artificial NN
Y1
Outputs
x2
w2
Weights
Processing Element (PE)
f (S )
i
i 1
X iW
Transfer Function
Summation
Y2 . . . Yn
xn
wn
Biological
Artificial Node (or PE) Input Output Weight Fast Few (102)
1-111
Neuron Dendrites Axon Synapse Slow Copyright 2011 Pearson Education, Inc. Publishing as Prentice Hall Many (109)
Data Mining Myths
Data mining

1-112
provides instant solutions/predictions. is not yet viable for business applications. requires a separate, dedicated database. can only be done by those with advanced degrees. is only for large firms that have lots of customer data. is another name for good-old statistics.
Common Data Mining Blunders

1. 2. 3. 4. 5.
Selecting the wrong problem for data mining Ignoring what your sponsor thinks data mining is and what it really can/cannot do Not leaving sufficient time for data acquisition, selection and preparation Looking only at aggregated results and not at individual records/predictions Being sloppy about keeping track of the data mining procedure and results
1-113
Common Data Mining Mistakes

6. 7. 8. 9. 10.
1-114
Ignoring suspicious (good or bad) findings and quickly moving on Running mining algorithms repeatedly and blindly, without thinking about the next stage Naively believing everything you are told about the data Naively believing everything you are told about your own data mining analysis Measuring your results differently from the way your sponsor measures them

Chapter 5: Text and Web Mining
Text Mining Concepts
85-90 percent of all corporate data is in some kind of unstructured form (e.g., text). Unstructured corporate data is doubling in size every 18 months. Tapping into these information sources is not an option, but a need to stay competitive. Answer: text mining
A semi-automated process of extracting knowledge from unstructured data sources a.k.a. text data mining or knowledge discovery in textual databases
1-116
Data Mining versus Text Mining

Both seek novel and useful patterns Both are semi-automated processes Difference is the nature of the data:

Structured versus unstructured data Structured data: databases Unstructured data: Word documents, PDF files, text excerpts, XML files, and so on
Text mining first, impose structure to the data, then mine the structured data
1-117
Text Mining Concepts
Benefits of text mining are obvious especially in text-rich data environments
e.g., law (court orders), academic research (research articles), finance (quarterly reports), medicine (discharge summaries), biology (molecular interactions), technology (patent files), marketing (customer comments), etc. Spam filtering Email prioritization and categorization Automatic response generation
Electronic communication records (e.g., Email)

1-118
Text Mining Application Area

Information extraction Topic tracking Summarization Categorization Clustering Concept linking Question answering
1-119
Text Mining Terminology

1-120
Unstructured or semistructured data Corpus (and corpora) Terms Concepts Stemming Stop words (and include words) Synonyms (and polysemes) Tokenizing
Text Mining Terminology

Term dictionary Word frequency Part-of-speech tagging Morphology Term-by-document matrix
Occurrence matrix Latent semantic indexing
Singular value decomposition
1-121
Natural Language Processing (NLP)
Structuring a collection of text

Old approach: bag-of-words New approach: natural language processing a very important concept in text mining. a subfield of artificial intelligence and computational linguistics. the study of "understanding" the natural human language.
NLP is

1-122
Syntax versus semantics based text mining
What is Understanding ?

Human understands, what about computers? Natural language is vague, context driven True understanding requires extensive knowledge of a topic Can/will computers ever understand natural language the same/accurate way we do?
1-123
Challenges in NLP

Part-of-speech tagging Text segmentation Word sense disambiguation Syntax ambiguity Imperfect or irregular input Speech acts
Dream of AI community
to have algorithms that are capable of automatically reading and obtaining knowledge from text
1-124
NLP Task Categories

1-125
Information retrieval Information extraction Named-entity recognition Question answering Automatic summarization Natural language generation & understanding Machine translation Foreign language reading & writing Speech recognition Text proofing Optical character recognition
Text Mining Applications
Marketing applications
Enables better CRM ECHELON, OASIS Deception detection
Security applications

example coming up
Medicine and biology
Literature-based gene identification
example coming up
Academic applications
Research stream analysis
- example coming up
1-126
Application Case 7.4: Mining for Lies
1-127
Application Case 7.4: Mining for Lies

Category Quantity Complexity Uncertainty Nonimmediacy Expressivity Diversity Informality Specificity Affect Example Cues Verb count, noun-phrase count, ... Avg. no of clauses, sentence length, Modifiers, modal verbs, ... Passive voice, objectification, ... Emotiveness Lexical diversity, redundancy, ... Typographical error ratio Spatiotemporal, perceptual information Positive affect, negative affect, etc.
1-128
Text Mining Process
The three-step text mining process

Text Mining Process
Step 1: Establish the corpus
Collect all relevant unstructured data (e.g., textual documents, XML files, emails, Web pages, short notes, voice recordings) Digitize, standardize the collection (e.g., all in ASCII text files) Place the collection in a common place (e.g., in a flat file, or in a directory as separate files)
1-130
Text Mining Process
Step 2: Create the TermbyDocument Matrix
1-131
Text Mining Process
Step 2: Create the TermbyDocument Matrix (TDM)
Should all terms be included?

Stop words, include words Synonyms, homonyms Stemming
What is the best representation of the indices (values in cells)?

Row counts; binary frequencies; log frequencies; Inverse document frequency
1-132
Web Mining Overview

Web is the largest repository of data Data is in HTML, XML, text format Challenges (of processing Web data)

The The The The The
Web Web Web Web Web
is too big for effective data mining is too complex is too dynamic is not specific to a domain has everything
1-133
Opportunities and challenges are great!
Web Mining
Web mining (or Web data mining) is the process of discovering intrinsic relationships from Web data (textual, linkage, or usage)
1-134
Web Content/Structure Mining

Mining of the textual content on the Web Data collection via Web crawlers Web pages include hyperlinks

Authoritative pages Hubs hyperlink-induced topic search (HITS) alg.
1-135
Web Usage Mining
Extraction of information from data generated through Web page visits and transactions
data stored in server access logs, referrer logs, agent logs, and client-side cookies user characteristics and usage profiles metadata, such as page attributes, content attributes, and usage data
Clickstream data Clickstream analysis
1-136
Web Usage Mining
Web usage mining applications

Determine the lifetime value of clients Design cross-marketing strategies across products. Evaluate promotional campaigns Target electronic ads and coupons at user groups based on user access patterns Predict user behavior based on previously learned rules and users' profiles Present dynamic information to users based on their interests and profiles
1-137
Web Usage Mining
(clickstream analysis)
1-138
Web Mining Success Stories

Amazon.com, Ask.com, Scholastic.com, etc. Website Optimization Ecosystem
1-139

Chapter 6: BI Implementation: Integration and Emerging Trends
Implementing BI An Overview
Critical Success Factors for BI Implementation

a. b. c. d. e. f. g.
Business driven methodology and project management Clear vision and planning Committed management support and sponsorship Data management and quality issues Mapping the solutions to the user requirements Performance considerations of the BI system Robust and extensible framework
1-141
Managerial Issues Related to BI Implementation

1. 2. 3. 4. 5. 6. 7. 8.
1-142
System development and the need for integration Costbenefit issues and justification Legal issues and privacy BI and BPM today and tomorrow Cost justification; intangible benefits Documenting and securing support systems Ethical issues BI Project failures
BI and Integration Implementation
Why integrate?

To better implement a complete BI system To increase the capabilities of the BI applications To enable real-time decision support To enable more powerful applications To facilitate faster system development To enhance support activities such as blogs, wikis, RSS feeds, etc.
1-143
BI and Integration Implementation
Levels of BI Integration
Functional integration can be within the same BI or across different BI systems
Integration across different BI systems can be accomplished in a loosely coupled fashion input output passing, messaging (SOA) Integration within a BI system is more cohesive with several sub-systems constituting the whole
Embedded Intelligent Systems
Serving as the intelligent agents within BI
1-144
Connecting BI Systems to Databases and Other Enterprise Systems
Virtually every BI application requires database or data warehouse access
Multi-tiered Application Architecture

On-Demand BI
The limitations of Traditional BI
Complex, time-consuming, expensive On-demand computing = Utility computing SaaS (Software as a service) Allows SMEs to utilize affordable BI On-demand function alternatives

The On-Demand Alternative

Internally sharing licenses within a firm Sharing licenses with many firms via an ASP
1-146
Benefits of On-Demand BI
Ability to handle fluctuating demand
Flexible use of the BI technology pool Hardware (servers and peripherals) Software (more features for less) Maintenance (centralized timely updates)
Reduced investment/cost

1-147
Embodiment of recognized best practices Better flexibility and connectivity with other systems via SaaS infrastructure Better RIO
The Limitations of On-Demand BI
Integration of vendors software with companys software may be difficult The vendor can go out of business, leaving the company without a service It is difficult or even impossible to modify hosted software for better fit with the users needs Upgrading may become a problem You may relinquish strategic data to strangers (lack of privacy/security of corporate data)
1-148
Issues of Legality, Privacy and Ethics
Ethics in Decision Making and Support

Electronic surveillance Software piracy Use of proprietary databases Use of intellectual property such as knowledge Computer accessibility for workers with disabilities Accuracy of data, information, and knowledge Protection of the rights of users
Use of corporate computers for non-workrelated purposes (personal use of Internet while working)
1-149
Emerging Topics in BI An Overview
Web 2.0 revolution as it relates to BI in (Section 6.7) Online social networks (Section 6.8) Virtual worlds as related to BI (Section 6.9) Integration social networking and BI (Section 6.10) RFID and BI (Section 6.11) Reality Mining (Section 6.12)
1-150
Emerging Topics in BI An Overview The Future of BI
Web 2.0 revolution as it related to BI (Section 6.7) Online social networks (Section 6.8) Virtual worlds as related to BI (Section 6.9) Integration social networking and BI (Section 6.10) RFID and BI (Section 6.11) Reality Mining (Section 6.12)
1-151
Emerging Topics in BI An Overview
In 2009, collaborative decision making emerged as a new product category that combines social software with business intelligence platform capabilities. In 2010, 20 percent of organizations will have an industryspecific analytic application delivered via software as a service as a standard component of their business intelligence portfolio. By 2012, business units will control at least 40 percent of the total budget for BI. By 2012, one-third of analytic applications applied to business processes will be delivered through coarse-grained application mashups. Because of lack of information, processes, and tools, through 2012, more than 35 percent of the top 5,000 global companies will regularly fail to make insightful decisions about significant changes in their business and markets.
1-152
The Web 2.0 Revolution
Web 2.0: a popular term for describing advanced Web technologies and applications, including blogs, wikis, RSS, mashups, usergenerated content, and social networks Objective: enhance creativity, information sharing, and collaboration Difference between Web 2.0 and Web 1.x Use of Web for collaboration among Internet users and other users, content providers, and enterprises
1-153
The Web 2.0 Revolution
Web 2.0: an umbrella term for new technologies for both content as well as how the Web works Web 2.0 has led to the evolution of Web-based virtual communities and their hosting services, such as social networking sites, video-sharing sites Companies that understand these new applications and technologiesand apply the capabilities early onstand to greatly improve internal business processes and marketing
1-154
Online Social Networking Basics and Examples
A social network is a place where people create their own space, or homepage, on which they write blogs; post pictures, videos, or music; share ideas; and link to other Web locations they find interesting.
The mass adoption of social networking Web sites points to an evolution in human social interaction
The size of social network sites are growing rapidly, with some having over 100 million members growth for successful ones 40 to 50 %
in the first few years and 15 to 25 % thereafter
1-155
Mobile Social Networking
Social networking where members converse and connect with one another using cell phones or other mobile devices MySpace and Facebook offer mobile services Mobile only services: Brightkite, and Fon11 Basic types of mobile social networks
1. 2.
Partnership with mobile carriers (use of MySpace over AT&T network) Without a partnership (off deck) (e.g., MocoSpace and Mobikade)

1-156
Mobile Enterprise Networks Mobile Community Activities (e.g., Sonopia)
Major Social Network Services
Facebook: The Network Effect
Launched in 2004 by Mark Zuckerberg (former Harvard student) It is the largest social network service in the world with over 500 million active users worldwide Initially intended for college and high school students to connected to other students at the same school In 2006 opened its doors to anyone over 13; enabling Facebook to compete directly with MySpace.
1-157
Implications of Business and Enterprise Social Networks
Business oriented social networks can go beyond advertising and sales Emerging enterprise social networking apps:
Finding and Recruiting Workers
See Application Case 14.2 for a representative example
Management Activities and Support Training Knowledge Management and Expert Location
e.g., innocentive.com; awareness.com; Caterpillar

1-158
Enhancing Collaboration Using Blogs and Wikis Within the Enterprise >
Implications of Business and Enterprise Social Networks
Survey shows that best-in-class companies use blogs and wikis for the following applications:

1-159
Project collaboration and communication (63%) Process and procedure document (63%) FAQs (61%) E-learning and training (46%) Forums for new ideas (41%) Corporate-specific dynamic glossary and terminology (38%) Collaboration with customers (24%)
Virtual Tradeshows
See iTradeFair.com
Social Networks and BI: Collaborative Decision Making
Collaborative decision making (CDM) combines social software and BI
CDM is a category of decision-support system for non-routine, complex decisions that require iterative human interactions. Ad hoc tagging regarding value, relevance, credibility, and decision context can substantially enrich both the decision process and the content that contributes to the decisions. Tying BI to decisions and outcomes that can be measured will enable organizations to better demonstrate the business value of BI.
1-161
How CDM Works
1-162
How does RFID work?
RFID system
a tag (an electronic chip attached to the product to be identified) an interrogator (i.e., reader) with one or more antennae attached a computer (to manage the reader and store the data captured by the reader) Active tag versus Passive tags
Tags
1-163
RFID for Supply Chain BI
RFID in Retail Systems
Functions in a distribution center
receiving, put-away, picking, and shipping
Sequence of operations at a receiving dock

unloading the contents of the trailer 2. verification of the receipt of goods against expected delivery (purchase order) 3. documentation of the discrepancy 4. application of labels to the pallets, cases, items 5. sorting of goods for put-away or cross-dock
1.
1-164
RFID for BI in Supply Chain
Better SC visibility with RFID systems
Timing/duration of movements between different locations especially important for products with limited shelf life Better management of out-of-stock items (optimal restocking of store shelves) Help streamline the backroom operations: eliminate unnecessary case cycles, reorders Better analysis of movement timings for more effective and efficient logistics
1-165
Reality Mining
Identifying aggregate patterns of human activity trends (see sensenetworks.com by MIT & Columbia University) Many devices send location information
Cars, buses, taxis, mobile phones, cameras, and personal navigation devices Using technologies such as GPS, WiFi, and cell tower triangulation
Enables tracking of assets, finding nearby services, locating friends/family members,
1-166
Reality Mining
Citisense: finding people with similar interests
A map of an area of San Francisco with density designation at place of interests See www.sensenetworks.com/city sense.php for real-time animation of the content.

Turban Ch1 Ch6

Caricato da

Informazioni sul documento

Descrizione originale:

Copyright

Formati disponibili

Condividi questo documento

Condividi o incorpora il documento

Opzioni di condivisione

Hai trovato utile questo documento?

Questo contenuto è inappropriato?

Copyright:

Formati disponibili

Turban Ch1 Ch6

Caricato da

Copyright:

Formati disponibili

Business Intelligence: A Managerial Approach nd Edition) (2

Chapter 1: Introduction to Business Intelligence

Business PressuresResponses Support Model

Copyright 2011 Pearson Education, Inc. Publishing as Prentice Hall

Business Environment Factors

Consumer demand Technology

Copyright 2011 Pearson Education, Inc. Publishing as Prentice Hall

Copyright 2011 Pearson Education, Inc. Publishing as Prentice Hall

Organizational Responses, continued

Copyright 2011 Pearson Education, Inc. Publishing as Prentice Hall

Business Intelligence (BI)

BI is an evolution of decision support concepts over time.

Then: Executive Information System Now: Everybodys Information System (BI)

Copyright 2011 Pearson Education, Inc. Publishing as Prentice Hall

Copyright 2011 Pearson Education, Inc. Publishing as Prentice Hall

A BI system has four major components:

Copyright 2011 Pearson Education, Inc. Publishing as Prentice Hall

Copyright 2011 Pearson Education, Inc. Publishing as Prentice Hall

The data warehouse is the cornerstone of any medium-to-large BI system.

Copyright 2011 Pearson Education, Inc. Publishing as Prentice Hall

Copyright 2011 Pearson Education, Inc. Publishing as Prentice Hall

Copyright 2011 Pearson Education, Inc. Publishing as Prentice Hall

Intelligence Creation and Use

Copyright 2011 Pearson Education, Inc. Publishing as Prentice Hall

Intelligence Creation and Use

Data warehouse deployment Creation of intelligence

Identification and prioritization of BI projects

Who should do the prioritization?

Copyright 2011 Pearson Education, Inc. Publishing as Prentice Hall

Copyright 2011 Pearson Education, Inc. Publishing as Prentice Hall

Intelligence and Espionage

Stealing corporate secrets, CIA,

Problem too much data, very little value

Use of data/text/Web mining (see Chapter 4, 5)

Copyright 2011 Pearson Education, Inc. Publishing as Prentice Hall

Transaction Processing Versus Analytic Processing

Copyright 2011 Pearson Education, Inc. Publishing as Prentice Hall

Transaction Processing Versus Analytic Processing

Copyright 2011 Pearson Education, Inc. Publishing as Prentice Hall

BI and Business Strategy

To be successful, BI must be aligned with the companys business strategy.

BI cannot/should not be a technical exercise for the information systems department.

BI changes the way a company conducts business by

Copyright 2011 Pearson Education, Inc. Publishing as Prentice Hall

Trent -> Business Activity Management

Copyright 2011 Pearson Education, Inc. Publishing as Prentice Hall

Issues for Successful BI

Developing vs. Acquiring BI systems

Justifying via cost-benefit analysis

Most of them are intangibles

Copyright 2011 Pearson Education, Inc. Publishing as Prentice Hall

Issues for Successful BI

Security and Privacy

Integration of Systems and Applications

Often sits on top of ERP, SCM, CRM systems

Integration to outside (partners of the extended enterprise) via internet

customers, vendors, government agencies, etc.

Copyright 2011 Pearson Education, Inc. Publishing as Prentice Hall

Major BI Tools and Techniques

Copyright 2011 Pearson Education, Inc. Publishing as Prentice Hall

Business Intelligence: A Managerial Approach nd Edition) (2

What is a Data Warehouse?

Copyright 2011 Pearson Education, Inc. Publishing as Prentice Hall