What is BI Testing?

BI Testing is the process of validating the data,

format and performance of the reports, subject
areas and security aspects of the BI Projects.
Emphasis on a thorough BI Testing is key for
improving the quality of the BI Reports and user
adoption. However, testing of BI projects is
different from traditional web application testing
since the content of the reports is automatically
generated by the BI tool based on the BI tool
metadata. The focus of this article is to describe
the different aspects of BI testing.

BI Testing Categories
BI Functional Testing
Adhoc Report Testing
BI Security Testing
BI Regression Testing
BI Stress Testing
BI FUNCTIONAL TESTING

Unrestricted
When a new report or dashboard is developed for consumption by
other users, it is important to perform a few checks to validate the
data and design of the included reports.

Report or Dashboard Design Check

Verify that the new report or dashboard conforms to the report requirement /
design specifications. Some of the items to check are :
Verify that the report or dashboard page title corresponds to the content of
the reports.
For reports with charts, the axis should be labelled appropriately.
The aggregation level of the data in the reports should be as per the report
requirements.
Verify that the report or dashboard page design conforms to the design
standards and best practices.
Validate the presence and functionality of the report download and print
options.
Where applicable, verify that the report help text exists and is appropriate for
the report content.
Verify the existance of any required static display text in the report such as
FOIA text.

Example : A new dashboard page was created with too many reports and prompts in
one page which made it difficult for users to gain insights quickly. This affected used
adoption.

Prompts Check
Prompts are used to filter the data in the reports as needed. They can be of
different types but the most common type of prompt is a select list or
dropdown with a lis of values. Some of the key tests for prompts are :
Verify that all the prompts are available as per requirements. Also check if
the type of the prompt matches the design specification.
For each prompt verify the label and list of values displayed (where
applicable).
Apply each prompt and verify that the data in the report is getting filtered
appropriately.
Verify the default prompt selection satisfies the report or dashboard page
design specification.

Example: The default selection for the 'Quarter' prompt was supposed to be the
current quarter but it was hardcoded by the report developer to a specific quarter.

Unrestricted
Report data accuracy Check
Verify that the data shown in the report is accurate. As is evident, this check is
vital aspect of the report functional testing. Cross check the report with data
shown in a transactional system application that is trusted by the users as the
source of truth for the data shown in the report.
Come up with an equivalent database query on the target and source
databases for the report. Compare the results from the queries with the data
in the report.
Review the database query generated by the report for any issues.
Apply reports prompts and validate the database query generated by the
report as well as the query output.

Example : A set of canned reports were developed for a new BI project. When data
accuracy tests were done comparing the report data with the output of equivalent
queries in the source system, it was found that more than 50% of them failed the
testing. Upon further investigation, several ETL and BI tool modeling issues were
discovered by the development team.

Drilldown Report checks

It is common to have links to drilldown reports in a report so that the user can
navigate to those reports for further details. Links to these reports can be at
the column level or column headling level. For each link to the drilldown
report, verify the following items :
Verify that the counts are matching between the summary and detail report
where appropriate.
Verify that all the prompts from the summary reports are getting applied to
the detail report.
Check if the links to the detail report from the summary report are working
from charts, tables, table headings.
Verify the database SQL query for the drill down report is as expected.

Example: One of the prompt for the report was not applied to the drilldown report
when user navigated to it from the summary report. As a result, the amounts did not
match between the summary and drilldown report.

Report Performance checks

Verify that the reports and dashboard page rendering times are meeting SLA
requirements. Test the performance for difference prompt selections. Perform
the same checks for the drilldown reports.

Example: The report did not have any default prompt selections and the
performance of the report was extremely slow when no prompts (filters) were

Unrestricted
applied. It not only caused bad user experience but also unnecessary load on the
database because users often stopped execution of the report from UI without
getting any value from it.

Browser checks
Browser compatibility of the reports is often dictated by the support of these
browsers by the BI tool being used for the BI project. Any custom javascript
additions to the report or dashboard page can also result in browser specific
report issues.

Example: Although the BI tool supported both Firefox and IE, the reports were very
slow in IE because of the difference in the image caching features of the
corresponding browsers.

ADHOC REPORT TESTING

BI tools such as OBIEE and Business Objects empower the
business users by providing the capability to create their own
reports without the help of a developer. These tools generate the
database queries automatically for the reports based on the
measure and dimension definitions defined on top of the physical
data model. For OBIEE, this model is defined in the RPD while
Business Objects stores the model in the form of an universe.
Business users can select any combination of dimension and
measure attributes available in the subject area to come up with
their own adhoc report. From a testing perspective, this presents a
huge challenge since the number of different combination of
dimension and measures can be very large and impossible to test
manually.

Subject Area design check

Subject areas contain related dimensions and measures that can be used for
creating new reports on the fly. Verify that the subject areas follow the
requirement / design specifications. Some of the items to check are :
Verify that the subjec area, dimension folders, fact folders, dimension
attributes and measures following the standard naming convensions.
Verify that the attributes and measures are placed in the appropriate

Unrestricted
dimension and fact folders.
Valdate the help text for the subject area, folders, attributes and measures.
Check for any unrelated dimensions or measures in the subject area.
Alternatively, check for any missing dimensions or measures.

Example : Multiple subject areas were created by different developers with different
naming convensions which was confusing to the end users.

Mapping check
Verify that all the dimension attributes and measures are mapped properly to
available database tables and columns. This can be done by creating adhoc
reports by select all the attributes/measures from each folder. Run these
reports and validate that the data is as expected.
Example: One of the measure had a typo in the database column name which
caused errors when it was included in a report.

Joins check
When a combination of dimension attributes and measures are added to a
report, the database query generated includes joins between the tables
involved. If the joins have issues, it usually results in errors.
Create separate reports by picking all the attributes in each dimension folder
and one measure at time.
Select attributes from multiple dimension folders and one measure at a
time.
Verify the database query generated if the join condition makes sense.

Example 1 : When a new dimension table was added to the BI model, new joins
were created with the fact tables. However, the join condition for an existing report
got impacted because this new addition. As a result, the report did not return any
results.

Example 2 : When OBIEE is unable to determine the join between two entities, it
either throws an ODBC error or in some cases use 'CAST AS NULL' in the database
query. In the later case, the report won't throw any error but it won't return any data
either.

BI STRESS TESTING
Like any other web application, BI applications also have
authentication and authorization security requirements. BI
applications are often integrated with single signon or embeded with

Unrestricted
other transactional applications. It is important to test the security
aspects of the BI application just like other web applications.

Report access security

The objective of this test is to validate that the BI users have access to the BI
reports, subject areas and dashboards is limited according to their access
levels. Access to the reports is generally controlled by role based security in
the BI tool.
Understand how the access to reports is different for different roles.
Identify users with those roles for testing.
Login as these users and validate access to the reports and subject areas.

Example: In a HR analytics application, access to salary and compensation

reports are restricted to select users.

Data Security (or Position based security)

In this form of security, different users have access to a report but the data
shown in the report is different based on the person running the report.

Example: In a CRM analytics application, all sales managers have access to a

pipeline report but the data shown in the report is limited to the opportunity
pipeline of his or her direct reports.

Single signon security

Single signon is often used as the authentication mechanism for BI
applications in large enterprises. The objective of this testing is to ensure that
users are able to access BI applications using their single signon access (or
windows authentication).

Integrated security
BI Applications are sometimes embedded as part of other transaction system
applications using a common authentication mechanism. This integration
needs to be tested for different users.

BI REGRESSION TESTING

Unrestricted
BI tools make it easy to create new reports by automatically
generating the database query dynamically based on a predefined
BI Model. This presents a challenge from a regression testing
standpoint because any change change to the BI Model can
potentially impact existing reports. Hence it is important to do a
complete regression test of the existing reports and dashboards
whenever there is an upgrade or change in the BI Model.

Regression testing of report data

The primary focus of this test is to verify that the data shown by a report or
dashboard page is same before and after the change (or upgrade). In case of
the BI tool upgrade, it is possible that the look and feel of the report might
change. By validating the data of the report, we can be sure that the report
content is same.
Example : When a new dimension table was added to the BI model, new joins were
created with the fact tables. However, the join condition for an existing report got
impacted because of this new addition. As a result, the report did not return any
results.

Regression testing of report format

Verify that there is no change in the look and feel of the report.
Example: After a BI Tool upgrade, one of the chart in an existing report stopped
showing in the UI.

Regression testing of prompts

Verify that all the prompts are available as expected and the list of values
showng in the prompts match the expected values. Apply the prompts and
verify that the reports shows data as expected.
Example : One of the filter condition was accidentally deleted from an existing
report. As a result, the prompt values are not getting applied as filters to the report.

Regression testing of report database query

When data is changing in the underlying tables, it might be difficult to
understand if the differences in the report data before and after an upgrade is
because of a regression issue or data change. It is helpful to compare the
database SQL query in such cases.
Example : A recent database upgrade adversely affected the query execution plans
which resulted in performance issues for the reports.

Regression testing of report performance

Unrestricted
Any upgrade or change in the system can cause regression in performance of
existing reports. It is important to verify that the reports are still performing as
expected.
Example : A recent database upgrade adversely affected the query execution plans
which resulted in performance issues for the reports.

Regression testing of security

The BI security testing sections delves into various aspects of security testing.
From a regression testing standpoint, the objective is to validate that the
security is not adversely impacted.
Example : While migrating new report content to production, the BI administrator
accidentally modified the security setting for existing HR reports. As a result,
employee compensation reports were visible to all BI users.

BI STRESS TESTING
BI Stress testing is similar to testing of any web based application
testing. The objective is to simulate concurrent users accessing
reports with different prompts and understand the bottlenecks in the
system.

Simulating user behavior

A typical BI user will login to the system and navigate to reports (or
dashboards), apply prompts and drills down to other reports. After the report is
rendered, the BI User reviews the data for a certain time called think time. The
BI user eventually logs out of the system. The first step in conducting a stress
test is to identify a list of most commonly used reports or dashboards for the
testing.

Simulating concurrent user load

Conducting a stress test requires simulation of the above BI user behavior
concurrently for different user loads. So it is important to have a list of different
user logins for the stress test. When executing the reports, each user can pick
a different set of prompt values for the same report. The stress test should be
able to randomize the prompt selection for each user so as generate a more
realistic behavior.

BI Cache

Unrestricted
Most of the BI tools support a caching mechanism to improve the performance
of the reports. Database queries and the data blocks used to generate the
results are also cached in databases. The question that frequently comes up
whether cache should turned off for stress testing. While turning off caching is
a good way to understand system bottlenecks, our recommendation is to
perform the stress test with the caching turned on because it is more closer to
what a production system would look like. The following points should be kept
in mind to reduce the impact of caching :
1. Include a large set of reports and dashboard pages.
2. Randomize prompt values for a given report or dashboard page so that the
same report is requested with different filter conditions.
3. User a large pool of user login so that different user based security is
applied during the execution.

Stress testing is an iterative process. When the first round of stress test is
conducted, a particular component of the system (eg. database cache) might
max out and bring down the response times. Once this bottleneck has been
addressed another roud of stress test might find another bottleneck. This
process needs to be continued till the target concurrent user load and report
performance SLAs are met.

Measurements
There are several performance data points that need to captured while
running a stress test:
Initial and final response time for each of the report and dashboard pages.
Time spent for each request in the database and in the BI tool.
CPU and memory utilization in the server machines (BI Tool and database).
Load balancing across all the nodes in a cluster.
Number of active sessions and number of report requests.
Busy connection count in database Connection pool and current queued
requests.
Network load.
Database cache and disk reads.

Unrestricted
 In our previous blog, 5 keys to nailing a BI Implementation, we
focused on achieving strategic success in implementing Business
Intelligence applications. In this blog, we turn our attention to a
tactical, but important aspect Testing of Business Intelligence
Applications.

Effective integration of testing in the implementation process

builds trust and confidence among business users as they make
crucial strategic decisions, based on the BI data generated.

Testing of Data Warehouse/Business Intelligence (DW/BI)

applications is a little different than testing traditional
transactional applications as it requires data-centric testing
approach.

The typical challenges an enterprises faces while testing DW/BI

implementations include:

Data volume, variety and complexity

Data anomalies from disparate data sources

Unrestricted
 Data loss during data integration process and handshaking
between sources
Time consuming
No audit trails, reusability or methodology resulting into high
cost of quality
Specialized skills required to execute data validation and
verification process
To ensure data completeness, accuracy, consistency, security and
reliability throughout the life cycle, it is important to test all
these aspects at each data entry point in the BI architecture and
not just at the end through reports or dashboards.

BI Testing Strategy:
The goal of testing BI applications is to achieve credible data.
And data credibility can be attained by making the testing cycle
effective.

A comprehensive test strategy is the stepping stone of an

effective test cycle. The strategy should cover test planning for
each stage, every time the data moves and state the
responsibilities of each stakeholder e.g. business analysts,
infrastructure team, QA team, DBAs, Developers and Business
Users. To ensure testing readiness from all aspects the key areas
the strategy should focus on are:

Unrestricted
 Scope of testing: Describe testing techniques and types to be
used.
Test environment set up.
Test Data Availability: It is recommended to have production
like data covering all/critical business scenarios.
Data quality and performance acceptance criteria.
The below diagram depicts the data entry points and lists a few
sample checks at each stage. Data Collection, Data Integration,
Data Storage and Data Presentation.

Data Acquisition:
The primary aim of data completeness is to ensure that all of the
data is extracted that needs to be loaded in the target. During the
data acquisition phase it is important to understand the various
data sources, the time boundaries of the data selected and any

Unrestricted
 other special cases that need to be considered. The key areas this
phase should focus on are:

Validating the data required and the availability of the data

sources from which this data needs to be extracted.
Data profiling: Embedding data profiling activity helps in
understanding the data, especially identifying different data
values, boundary value conditions or any data issues at early
stages. Identifying data problems early on will considerably
reduce the cost of fixing it later in the development cycle.

Data Integration:
Testing within the data integration phase is the crux as data
transformation takes place at this stage. Business requirements
get translated into transformation logic. Once the data is
transformed, thorough testing needs to be executed to ensure
underlying data complies with the expected transformation
logic. Key areas this phase should focus on are:

Validating the Data Model: This involves validating the data

structure with business specifications. This can be done by
comparing columns and their data types with business
specifications and reporting column requirements ensuring data
coverage at source.

Unrestricted
 Reviewing the Data Dictionary: Verifying metadata which
includes constraints like Nulls, Default Values, Primary Keys,
Check Constraints, Referential Integrity, Surrogate keys,
Cardinality (1:1, m: n), etc.
Validating the Source to Target Mapping: Ensuring traceability
throughout will help build the quality aspects like consistency,
accuracy and reliability.
Data Storage: The data storage phase refers to loading of data
within the data warehouse/data mart or OLAP cubes. The data
loads can be one time, incrementally or in real-time. Key areas
this phase should focus on are:

Validating data loads based on time intervals.

Performance and Scalability: Testing of initial and subsequent
loads with performance and scalability aspect ensures that the
system is within acceptable performance limits and can sustain
further data growth.
Parallel Execution and Precedence: Verifying appropriate
parallel execution and precedence during ETL process is
important as it may impact directly on performance and
scalability of the system.

Unrestricted
 Validating the Archival and Purge Policy: Ensures data history
based on business requirements.
Verifying error logging, exception handling and recovery from
failure points.

Data Presentation:
This is the final step of the testing cycle and has the privilege of
having a graphical interface to test the data. Key areas this phase
should focus on are:

Validating the Report Model.

Report layout validation as per mockups and data validation as
per business requirements.
End to End Testing: Although individual components of the data
warehouse may be behaving as expected, there is no guarantee
that the entire system will behave the same. Thus execution and
validation of end-to-end runs are recommended. Along with data
reconciliation discrepancies, issues might surface such as
resource contention or deadlocks. The end-to-end runs will
further help in ensuring the data quality and performance
acceptance criteria are met.

Unrestricted
 While above considerations are given, one important aspect that
still remains to be addressed is the issue of Time. BitWise has
created a platform based on DW/BI Testing Best Practices that
automates and improves the overall effectiveness of DW/BI
Testing. If youre interested in learning more about this platform,
please contact us.

With the features and benefits of this platform, the intention is to

address most of the DW/BI testing challenges:

End-to-end traceability achieved right through source extraction

to reports.
100% requirement and test data coverage through test cases.
Test case automation of standard checks achieving considerable
time savings.
Up to 50% time savings through automated regression testing.

Unrestricted
 Defect or bug tracking involving all the stakeholders.
Improved testing cycle time through reusability.
Process improvements with analytical reporting showcasing test
data, test cases & defect trends.

Conclusion:
Testing BI applications is different than testing traditional
enterprise applications. To achieve truly credible data, each
stage of the lifecycle must be tested effectively Data
Collection, Data Integration, Data Storage and Data Presentation.
If youre not comfortable with your internal capabilities to test
your BI applications, turn to the BitWise DW/BI Testing
platform and lean on Bitwises expertise and experience gained
through testing business intelligence applications for clients over
the past decade.
A report is an output produced by the application under test. Reports come in
numerous varieties. Some reports mainly have numbers and/ or charts and some
reports mainly have text. Some reports are short and some run into pages. Whatever
type of report you test, you should find the following list of things to test in a report
handy. See the video, How to test software reports? or read on...

How to test reports

1. Does the report validate any input data (e.g. date range or
username) provided to it?
Tests for data 2. Is the input data entered by the user shown correctly in the
report?
3. Is the report created based on the input data given by the
user?

Unrestricted
 4. Does the report calculate values (e.g. subtotals by a unit
such as a reporting period, totals, averages, minimum,
maximum) correctly?
5. Is each data item on the report formatted correctly?
6. Does the report group data on the basis of a unit correctly?
7. Does the report show and use for calculation the correct
statutory details (e.g. tax rates)?
8. Is the chart according to the data in the report?
9. If required, does the report show the correct reference
number?
10. If required, does the report show the correct reference
number(s) of the previous/ related report(s)?
11. Does the report show the correct text (e.g. executive
summary, key drivers, approach used to create the report,
current issues and action plan)?
12. If required, does the report show the correct supporting
data?
13. Does the report include the correct date and time that it
was created on?
14. Does the report show the correct names of the author,
reviewer and approver?

15. Is the report layout as agreed?

16. Does the report show the correct heading (name, purpose
and the target audience of the report)?
17. If applicable, does the report show the correct table of
contents?
18. Does the report draw any tables correctly? Do the tables
clearly show the data? Is the data aligned within the tables?
Tests for presentation Are the tables aligned with each other?
19. Is the data correctly broken up into sections or pages?
20. Does similar data and text appear in the same font (faces,
sizes, colors etc.)? Are these fonts as agreed?
21. Is the header correct? Does the correct header appear on
every page?
22. Is the footer correct? Does the correct footer appear on
every page?
23. If it is a multi-page report, do the correct page numbers

Unrestricted
 appear on every page?
24. Are the graphics used (e.g. logo and background) correct?
25. Are the links on the report correct?
26. If required, does the report include notes for interpreting
the data, resources for further information or next steps?
27. Is the report similar in look and feel as the other reports
produced by the application?

28. Is the time taken to generate the report reasonable?

29. Is it possible to distribute the report (e.g. by email, shared
drive, feed etc.) as required? Does the distributed report open
Tests for other factors
on all supported devices?
30. Is it possible to export the report to all supported formats?
31. Does the report print? Completely and with the same data
and same presentation?

Unrestricted