The architecture and components of a

BI system

After understanding what the BI system is, it's time to discover more about its components and

understand how these components work with each other. There are also some BI tools that help to

implement one or more components. The following diagram shows an illustration of the

architecture and main components of the Business Intelligence system:

The BI architecture and components differ based on the tools, environment, and so on. The

architecture shown in the preceding diagram contains components that are common in most of the

BI systems. In the following sections, you will learn more about each component.

The data warehouse

The data warehouse is the core of the BI system. A data warehouse is a database built for the

purpose of data analysis and reporting. This purpose changes the design of this database as well.

As you know, operational databases are built on normalization standards, which are efficient for

transactional systems, for example, to reduce redundancy. As you probably know, a 3NF-designed

database for a sales system contains many tables related to each other. So, for example, a report

on sales information may consume more than 10 joined conditions, which slows down the

response time of the query and report. A data warehouse comes with a new design that reduces

the response time and increases the performance of queries for reports and analytics. You will

learn more about the design of a data warehouse (which is called dimensional modeling) later in

this chapter.

Extract Transform Load

It is very likely that more than one system acts as the source of data required for the BI system. So

there is a requirement for data consolidation that extracts data from different sources and

transforms it into the shape that fits into the data warehouse, and finally, loads it into the data

warehouse; this process is called Extract Transform Load (ETL). There are many challenges in

the ETL process, out of which some will be revealed (conceptually) later in this chapter.

According to the definition of states, ETL is not just a data integration phase. Let's discover more

about it with an example; in an operational sales database, you may have dozen of tables that

provide sale transactional data. When you design that sales data into your data warehouse, you

can denormalize it and build one or two tables for it. So, the ETL process should extract data from

the sales database and transform it (combine, match, and so on) to fit it into the model of data

warehouse tables.

There are some ETL tools in the market that perform the extract, transform, and load operations.

The Microsoft solution for ETL is SQL Server Integration Service (SSIS), which is one of the best

ETL tools in the market. SSIS can connect to multiple data sources such as Oracle, DB2, Text

Files, XML, Web services, SQL Server, and so on. SSIS also has many built-in transformations to
transform the data as required. Chapter 4, ETL with Integration Services, is about SSIS and how

to do data transformations with this tool.

Data model – BISM

A data warehouse is designed to be the source of analysis and reports, so it works much faster

than operational systems for producing reports. However, a DW is not that fast to cover all

requirements because it is still a relational database, and databases have many constraints that

reduce the response time of a query. The requirement for faster processing and a lower response

time on one hand, and aggregated information on another hand causes the creation of another

layer in BI systems. This layer, which we call the data model, contains a file-based or memory-

based model of the data for producing very quick responses to reports.

Microsoft's solution for the data model is split into two technologies: the OLAP cube and the In-

memory tabular model. The OLAP cube is a file-based data storage that loads data from a data

warehouse into a cube model. The cube contains descriptive information as dimensions (for

example, customer and product) and cells (for example, facts and measures, such as sales and

discount). The following diagram shows a sample OLAP cube:

In the preceding diagram, the illustrated cube has three dimensions: Product, Customer,

and Time. Each cell in the cube shows a junction of these three dimensions. For example, if we

store the sales amount in each cell, then the green cell shows that Devin paid 23$ for a Hat on

June 5. Aggregated data can be fetched easily as well within the cube structure. For example, the

orange set of cells shows how much Mark paid on June 1 for all products. As you can see, the

cube structure makes it easier and faster to access the required information.

Microsoft SQL Server Analysis Services 2012 comes with two different types of modeling:

multidimensional and tabular. Multidimensional modeling is based on the OLAP cube and is fitted

with measures and dimensions, as you can see in the preceding diagram. The tabular model is

based on a new In-memory engine for tables. The In-memory engine loads all data rows from

tables into the memory and responds to queries directly from the memory. This is very fast in

terms of the response time. You will learn more about SSAS Multidimensional in Chapter 2, SQL

Server Analysis Services Multidimensional Cube Development, and about SSAS Tabular
in Chapter 3, Tabular Development of SQL Server Analysis Services, of this book. The BI

semantic model (BISM) provided by Microsoft is a combination of SSAS Tabular and

Multidimensional solutions.

Data visualization

The frontend of a BI system is data visualization. In other words, data visualization is a part of

the BI system that users can see. There are different methods for visualizing information, such as

strategic and tactical dashboards, Key Performance Indicators (KPIs), and detailed or

consolidated reports. As you probably know, there are many reporting and visualizing tools on the

market.

Microsoft has provided a set of visualization tools to cover dashboards, KPIs, scorecards, and

reports required in a BI application. PerformancePoint, as part of Microsoft SharePoint, is a

dashboard tool that performs best when connected to SSAS Multidimensional OLAP cube. You

will learn about PerformancePoint in Chapter 10, Dashboard Design. Microsoft's SQL Server

Reporting Services (SSRS) is a great reporting tool for creating detailed and consolidated

reports. SSRS is a mature technology in this area, which will be revealed in Chapter 9, Reporting

Services. Excel is also a great slicing and dicing tool especially for power users. There are also

components in Excel such as Power View, which are designed to build performance dashboards.

You will learn more about Power View in Chapter 9, Reporting Services, and about Power BI

features of Excel 2013 in Chapter 11, Power BI. Sometimes, you will need to embed reports and

dashboards in your custom written application. Chapter 12, Integrating Reports in Application,

of this book explains that in detail.

Master Data Management

Every organization has a part of its business that is common between different systems. That part

of the data in the business can be managed and maintained as master data. For example, an
organization may receive customer information from an online web application form or from a retail

store's spreadsheets, or based on a web service provided by other vendors.

Master Data Management (MDM) is the process of maintaining the single version of truth for

master data entities through multiple systems. Microsoft's solution for MDM is Master Data

Services (MDS). Master data can be stored in the MDS entities and it can be maintained and

changed through the MDS Web UI or Excel UI. Other systems such as CRM, AX, and even DW

can be subscribers of the master data entities. Even if one or more systems are able to change the

master data, they can write back their changes into MDS through the staging architecture. You will

learn more about MDS in Chapter 5, Master Data Management.

Data Quality Services

The quality of data is different in each operational system, especially when we deal with legacy

systems or systems that have a high dependence on user inputs. As the BI system is based on

data, the better the quality of data, the better the output of the BI solution. Because of this fact,

working on data quality is one of the components of the BI systems. As an example, Auckland

might be written as "Auck land" in some Excel files or be typed as "Aukland" by the user in the

input form.

As a solution to improve the quality of data, Microsoft provided users with DQS. DQS works based

on Knowledge Base domains, which means a Knowledge Base can be created for different

domains, and the Knowledge Base will be maintained and improved by a data steward as time

passes. There are also matching policies that can be used to apply standardization on the data.

You will learn more about DQS in Chapter 6, Data Quality and Data Cleansing.

You are currently viewing a free section

Access this and 7,000+ eBooks and Videos with a Packt subscription, with over 100 new titles
released each month.

Start your 10-day FREE trial

 Previous SectionNext Section

What Is BI Architecture?
Business intelligence architecture is a term used to describe standards and policies for
organizing data with the help of computer-based techniques and technologies that create
business intelligence systems used for online data visualization, reporting, and analysis.

One of the BI architecture components is data warehousing. Organizing, storing,

cleaning, and extraction of the data must be carried by a central repository system,
namely data warehouse, that is considered as the fundamental component of business
intelligence. But how exactly are they connected?

What Is Data Warehousing And Business

Intelligence?
Data warehousing and business intelligence are terms used to describe the process of
storing all the company’s data in internal or external databases from various sources with
the focus on analysis, and generating actionable insights through online BI tools.

One without the other wouldn’t function, and we will now explain premises that surround
their framework by using a BI architecture diagram to fully understand how data
warehouse enhances the BI processes.

BI Architecture Framework In Modern Business

There are various components and layers that business intelligence architecture consists
of. Each of that component has its own purpose that we will discuss in more detail while
concentrating on data warehousing. But first, let’s first see what exactly these
components are made of.

A solid BI architecture framework consists of:

1. Collection of data
2. Data integration
3. Storage of data
4. Data analysis
5. Distribution of data
6. Reaction based on insights

**click to enlarge**

We can see in our BI architecture diagram how the process flows through various layers,
and now we will focus on each.

1. Collection of data
The first step in creating a stable architecture starts in gathering data from various data
sources such as CRM, ERP, databases, files or APIs, depending on the requirements
and resources of a company. Modern BI tools offer a lot of different, fast and easy data
connectors to make this process smooth and easy by using smart ETL engines in the
background. They enable communication between scattered departments and systems
that would otherwise stay disparate. From a business point of view, this is a crucial
element in creating a successful data-driven decision culture that can eliminate errors,
increase productivity, and streamline operations. You have to collect data in order to be
able to manipulate with it.
2. Data integration
When data is collected through scattered systems, the next step continues in extracting
data and loading it to a data warehouse. This process is called ETL (Extract-Transform-
Load).

With an increasing amount of data generated today and the overload on IT departments
and professionals, ETL as a service comes as a natural answer to solve complex data
requests in various industries. The process is simple; data is pulled from external sources
(from our step 1) while ensuring that these sources aren’t negatively impacted with the
performance or other issues. Secondly, data is conformed to the demanded standard. In
other words, this (transform) step ensures data is clean and prepared to the final stage:
loading into a data warehouse.

3. Data storage
Now we approach the data warehousing and business intelligence concepts. While both
terms are often used interchangeably, there are certain differences that we will focus on
to get a more clear picture on this topic.
 **click to enlarge**

The main differences, as we can also see in the visual, between business intelligence
and data warehousing are indicated in these main questions:

a) What is the goal?

Business intelligence and data warehousing have different goals. While they are
connected and cannot function without each other, as mentioned earlier, BI is mainly
focused on generating business insights, whether operational or strategic efficiency such
as product positioning and pricing to goals, profitability, sales performance, forecasting,
strategic directions, and priorities on a broader level. The point is to access, explore, and
analyze measurable aspects of a business. On the other hand, a data warehouse (DWH)
has its significance in storing all the company’s data (from one or several sources) in a
single place. In a nutshell, BI systems and tools make use of data warehouse while data
warehouse acts as a foundation for business intelligence.

Exclusive Bonus Content: Wondering about BI & DWH? Learn right here!
Get the short and sweet version with our completely free summary!
b) What is the output?

The output data of both terms also vary. While BI outputs information through data
visualization, online dashboards, and reporting, data warehouse outlines data in
dimension and fact tables for upstream applications (or BI tools). Data cleansing,
metadata management, data distribution, storage management, recovery, and backup
planning are processes conducted in a data warehouse while BI makes use of tools that
focus on statistics, visualization, and data mining.

The output difference is closely interlaced with the people that can work with either BI or
data warehouse. But let’s see this through our next major aspect.

c) What is the audience?

To expand our previous point, the people involved in managing the data are quite
different. C-level executives or managers use modern BI tools in the form of a real-time
dashboard since they need to derive factual intelligence, create effective sales reports or
forecast strategic development of the department or company. CEOs or sales managers
cannot manage data warehouse since it’s not their area of expertise; they need a tool that
will translate the heavy IT data into insights that an average business user can fully
understand. That’s where business intelligence creates a solid bridge between DWH and
BI. On the other hand, a data warehouse is usually dealt with by data (warehouse)
engineers and back-end developers. They are the technical chain in a BI architecture
framework that design, develop, and maintain systems for future data analysis and
reporting a business might need.

d) What are the tools?

With the expansion of data processed and created in our digital age, the tools and
software needed to perform analysis expanded and developed in recent years in ways
we could not have imagined. In this context, the need for utilizing a proper tool, a
stable business intelligence dashboard and data warehouse increased exponentially. In
such environment, the data warehouse processes can be managed with a product such
as Amazon Redshift while the full support for BI insights needed to effectively generate
and develop sustainable business acumen with tools such as datapine. Visualization of
data is the core element that enables managers, professionals, and business users to
perform analysis on their own, without the need for heavy IT support or work.

Now that we have expounded what is data warehousing and business intelligence, we
continue with our next step: analyzing the BI architecture layers needed for establishing
a sustainable business development.

4. Analysis of data
In this step of our compact BI architecture, we will focus on the analysis of data after it’s
handled, processed, and cleaned in former steps with the help of data warehouse(s). The
ubiquitous need for successful analysis for empowering businesses of all sizes to grow
and profit is done through BI application tools. Especially when it comes to ad hoc
analysis that enables freedom, usability, and flexibility in performing analysis and helping
answer critical business questions swiftly and accurately.

This visual above represents the power of a modern, easy-to-use BI user interface.
Modern BI tools like datapine empower business users to create queries via drag and
drop, and build stunning data visualizations with a few clicks, even without profound
technological knowledge. This simplifies the process of creating business dashboards, or
an analytical report, and generate actionable insights needed for improving the
operational and strategic efficiency of a business. The data warehouse works behind this
process and makes the overall architecture possible.
5. Data distribution
Data distribution comes as one of the most important processes when it comes to sharing
information and providing stakeholders with indispensable insights to obtain sustainable
business development. Distribution is usually performed in 3 ways:

a) Reporting via automated e-mails: Created reports can be shared with selected
recipients on a defined schedule. The dashboards will be automatically updated on a
daily, weekly or monthly basis which eliminates manual work and enables up to date
information.

b) Dashboarding: Another reporting option is to directly share a dashboard in a secure

viewer environment. The users you share with cannot make edits or change the content
but can use assigned filters to manipulate data and interact with the dashboard. Another
option is to share via public URL that enables users to access the dashboards even if
they’re outside of your organization, as shown in the picture below:

c) Embedding: This form of data distribution is enabled through embedded BI. Your own
application can use dashboards as a mean of analytics and reporting without the need
for labeling the BI tool in external applications or intranets.

6. Reactions based on generated insights

The final stage where the BI architecture expounds its power is the fundamental part of
any business: creating data-driven decisions. Without the backbones of data
warehousing and business intelligence, the final stage wouldn’t be possible and
businesses won’t be able to progress. CEOs, managers, professionals, coworkers, and
all the interested stakeholders can have the power of data to generate valid, accurate,
data-based decisions that will help them move forward. Let’s see this through one of
our dashboard examples: the management KPI dashboard.

**click to enlarge**

This dashboard is the final product on how data warehouse and business intelligence
work together. The processes behind this visualization include the whole architecture
which we have described, but it would not be possible to achieve without a firm data
warehouse solution. Ultimately, this enables a high-level manager to get a
comprehension of the strategic development and potential decisions for creating and
maintaining a stable business.

On this particular dashboard, you can see the total revenue, as well as on a customer
level, adding also the costs. The targets are also set so that the dashboard immediately
calculates if they have been met or additional adjustments are needed from a
management point of view. As revenue is one of the most important factors when
evaluating if the business is growing, this management dashboard ensures all the
essential data is visualized and the user can easily interact with each section, on a
continual basis, making the decision processes more cohesive and, ultimately, more
profitable.

Exclusive Bonus Content: Wondering about BI & DWH? Learn right here!
Get the short and sweet version with our completely free summary!

Data Warehousing And Business Intelligence:

Solutions For A Forward-Looking Business
We have explained these terms and how they complement the BI architecture. These
processes are important to consider in today’s competitive business environment since
they bring the best data management practice that can only bring positive results.

Although the terms have been used as synonyms in recent years, today they function on
diverse levels, but the perspective is the same: analyze, clean, monitor, and evaluate the
data in the finest and most productive way possible.