Integrated Cost & Schedule Risk Analysis

Dynamic Integrated Cost Estimator (DICE) Model

Adelaide, Australia 28 June, 2011
This document contains Booz Allen Hamilton proprietary and confidential information and is intended solely for the use and information of the client to whom it is addressed. This data shall not be released to other contractors without written consent from Booz Allen Hamilton.

Booz Allen Proprietary/Not for Distribution

Outline
Introduction RealTime Analytics Overview of Joint Confidence Level Analysis NASAs Joint Confidence Level Policy Dynamic Integrated Cost Estimator (DICE) JCL Analysis Tool

Attempt at Humor

Introduction
Whether assessing/analyzing NASA, DoD or Intelligence Community owned projects, the story is the same each time: Programs are increasingly experiencing growth above and beyond their initial cost and schedule estimates This is not just a cosmetic problem: Cost and schedule growth delays capabilities and constraints the budgets of other programs causing a waterfall of instability Studies have examined the reasons behind this growth reaching similar conclusions 1. Early program optimism leading to optimistic estimates

2. Insufficient cost and schedule reserves available to cover risk

3. Weak independent validation of cost and schedule Recognizing this, many guides, including the GAOs Cost Estimating Handbook, have included Risk Analysis as a required step in best practice cost estimating processes

RealTime Analytics
One of the challenges in performing cost and schedule analysis is the time required to run simulations Risk analysis, required for all cost estimates by 2009 Weapons System Acquisition Reform Act of 2009 and GAO, requires the use of simulations Run-times of minutes or hours prohibit most risk analyses models from being decision making tools as they can not be re-run during meetings RealTime Analytics (RTA) is a collection of technologies, tools and methodologies allowing complex analytics to be performed far faster than using currently available methods This presentation will focus on the Dynamic Integrated Cost Estimator (DICE) and the methodologies it addresses: Joint Confidence Level Analysis (Integrated Cost & Schedule Risk Analysis) RTA tools allow simulations to run up to 99.99% faster than comparable industry tools Simulations formerly taking minutes or hours to run now finish in under 1 second Allows decision makers to run an unlimited number of excursion scenarios without ever leaving the meeting room

Introduction to Joint Confidence Level Analysis

Decision makers are starting to recognize that there is rarely a relation between cost risk analysis results and the programs schedule This can lead to risk adjusted cost estimates that, if come to pass, will almost always imply associated schedule growth From the other side, traditional schedule risk analysis typically leads to risk adjusted schedules that, if come to pass, will result in cost growth Even when both of these analyses are performed on a program, they are typically done by disjoint groups under different sets of assumptions Joint Cost & Schedule Risk Analysis is an attempt to integrate cost and schedule risk analysis in a way that produces meaningful, compatible results This presentation will cover:

NASAs JCL Policy

How JCLs are performed

What is Joint Confidence Level Analysis?

Historically, cost and schedule risk analyses are developed separately and their results are not compatible This can lead to risk adjusted cost estimates that, if come to pass, will almost always imply associated schedule growth and vice versa Joint Confidence Level Analysis combines cost and schedule risk analysis into a single, coherent output Joint Confidence Level (JCL) analysis1 results in a bivariate distribution of final projected cost and schedule pairs Probability Joint Confidence Levels represent the probability of finishing at or under both cost and schedule JCL Analysis allows a program to: Defend budgetary and scheduling decisions

70% Joint Confidence Level Curve

Prioritize risks and other threats based on their overall impact to the program and not simply their anticipated local impact
Develop more precise risk mitigation plans
1Also

Cost

known as Integrated Cost & Schedule Risk Analysis or Joint Cost & Schedule Risk Analysis
6

What is Joint Confidence Level Analysis?

The primary output from JCL analysis is the JCL scatter-plot Each point on the scatter plot represents 1 iteration of a Monte Carlo simulation performed on the JCL model JCL scatter-plot provides: Joint Confidence Levels (e.g. For NASA Budgeting) Relationship between cost and schedule (e.g. $ cost growth/days schedule growth)

Joint Confidence Level

Final Results
7/24/14
Cost FOC

Knee in the Curve 50% $787.7M 10/8/12

Values 70% $831.5M 1/29/13

Launch Date

1/5/14 6/19/13 12/1/12 5/15/12 10/28/11

JCL Distrubution Project Baseline = 61% 70% JCL 50% JCL

4/11/11 9/23/10
$603 $803

$1,003

$1,203

Total Cost ($M)

JCL Analysis: NASA Policy & Air Force Research

JCL Analysis becoming more common in industry In use by oil industry for years Already a requirement for NASA projects per NPD 1000.5

Air Force completing research on JCL Analysis1

Several methods available for integrating cost & schedule; preferred method depends on program phase and available data Parametric approaches: Typically used phase A and before or by oversight groups Divided into multiple regression and multivariate regression approaches Build-up approaches: Typically used following phase B by PMOs Cost risk analysis and schedule risk analysis performed separately Quality checks performed to make sure analyses are compatible

1Joint

Cost Schedule Model (JCSM): Recent AFCAA Efforts to Assess Integrated Cost and Schedule Analysis. Hogan, Greg (et al). SCEA 2011 8

JCL Analysis: Overview of Methods

Joint Risk Analysis:
Joint Confidence Level

Cost Loaded ScheduleBased Simulation (NASA Policy) Multivariate Regression

Final Results
7/24/14

Cost FOC

Knee in the Curve 50% $787.7M 10/8/12

Values 70% $831.5M 1/29/13

Launch Date

1/5/14 6/19/13 12/1/12 5/15/12 10/28/11

JCL Distrubution Project Baseline = 61% 70% JCL 50% JCL

Joint Risk Analysis methods estimate cost and schedule simultaneously; JCL scatter plot produced directly from analysis

4/11/11 9/23/10
$603 $803

$1,003

$1,203

Total Cost ($M)

Cost Risk Analysis: Inputs-Based Simulation Outputs-Based Simulation Scenario Based Parametric Analysis

When Joint Risk Analysis methods are not used, cost and schedule risk analyses must be combined using a Monte Carlo simulation

Joint Confidence Level

Final Results
Cost FOC

Knee in the Curve 50% $787.7M 10/8/12

Values 70% $831.5M 1/29/13

Schedule Risk Analysis: Parametric Analysis Schedule-Based Simulation

Monte Carlo Simulation

7/24/14

Launch Date

1/5/14 6/19/13 12/1/12 5/15/12 10/28/11

JCL Distrubution Project Baseline = 61% 70% JCL 50% JCL

4/11/11 9/23/10
$603 $803

$1,003

$1,203

Total Cost ($M)

When risk analyses are combined, care must be taken to ensure results are compatible
1Joint

Cost Schedule Model (JCSM): Recent AFCAA Efforts to Assess Integrated Cost and Schedule Analysis. Hogan, Greg (et al). SCEA 2011 9

Joint Confidence Level Analysis Process at NASA

JCL analysis can be performed using existing resources; likely risk management, cost estimating, and scheduling personnel

The artifacts needed to perform a JCL analysis are:

A program schedule (IMS or analysis schedule) with uncertainty bounds on task durations

Integrated Master Schedule

Cost Estimate

A quantified risk register (probabilities, cost and schedule impacts) where each risk is mapped to a task in the IMS
A cost estimate with uncertainty bounds that maps to the schedule Creation of these artifacts requires communication between programs cost estimating, scheduling and risk management staff

Program Risk Register

Integrated Risk Assessment

10

NASA Joint Confidence Level Analysis Policy

Joint Confidence Level Analysis has gained significant momentum recently NASA is leading the way in the development of this methodology

NASA Policy Directive 1000.5 mandates that programs will be baselined at the 70 percent confidence level using a joint cost and schedule probability distribution2
NPD 1000.5 also stipulates that projects are funded at no less than 50% of the JCL or as approved by the decision authority, maintaining JCLs through the program lifecycle The goal is to provide stronger assurance that NASA can meet cost and schedule targets3

A recent GAO report cites NASAs JCL policy as an effort to provide transparency on the effects of funding changes on the probability of meeting cost and schedule commitments 4
NASA Cost Analysis Division (CAD) has developed a handbook to provide more information and guidance on this topic Programs conducting JCL Analysis include James Webb Space Telescope and SOFIA While the methodology has made substantial strides, the cost and schedule communities must overcome political and technical obstacles before full adoption
NPD 1000.5 - http://www.hq.nasa.gov/office/codeq/doctree/10005.htm - January 15, 2009 JCL Status Report - http://www.nasa.gov/pdf/421542main_JCL%20Status%20Report-2010%20Feb.pdf February 2010 4 GAO Report NASA Assessments of Selected Large-Scale Projects - http://www.gao.gov/new.items/d11239sp.pdf - March 2011
2 3

11

NASA JCL Model Prototype: DICE

In Fall 2010, NASA CAD commissioned the development of a JCL model prototype The intention of this effort was to explore the value of producing a standard toolset for NASA programs conducting JCL analysis Booz Allen created the Dynamic Integrated Cost Estimator (DICE) with a focus on streamlining the JCL process and decreasing simulation runtimes Other key features of the DICE prototype development included: Rapid schedule import from MS Project Cost-Loading Discrete Risk Analysis JCL Scatter Plots and Iso-Curves Benchmarking effort with other JCL tools

It is important to note that there are many tools that projects can use to develop JCLs, but DICE is optimized for this analysis

12

DICE facilitates Joint Confidence Level Analysis

DICE is an Adobe Flex-based tool for cutting-edge cost and schedule risk analysis
Includes modeling capability for producing build-up Joint Confidence Levels (JCLs) Achieves industry-leading runtimes using Booz Allens RealTime Analytics

13

DICE Benchmarking against Primavera Risk Analysis

Runtimes (3200-line schedule)
Task Run Time

S-Curves (at 0.4 correlation)

DICE
Opening Tool Import Schedule Load Risk File Initialize Correlation Run Simulation Export Data 0:02 0:03 0:03 0:02 0:09 0:08

Primavera
0:05 2:05:27 0:01 0:05 2:27 0:00

Project End Date

Total Project Cost

Schedule

Cost

Key Points Identical input parameters to ensure consistency in benchmarking Importing risks from 3rd party template

Correlation

Outputs <1% variation from Primavera

Project End Date

14

DICE Demo

15

Conclusion
Booz Allens recent innovations in simulation technology enable analysts to support decision making in near real-time Decision makers can see the impact of changes to their program, real-time, without ever leaving the meeting room There is no longer a limit on the number of excursions that can be run on an analysis New Joint Confidence Level methods changes the way programs look at cost and schedule risk Analyses are no longer performed and viewed separately, but rather are integrated and optimized using a standard tool Decision makers have more insight into their program than ever before Opening of communication lines between cost, schedule and risk management staff results in better program management

16

Points of Contact
Eric Druker
Associate

Graham Gilmer
Associate

Booz | Allen | Hamilton

Booz Allen Hamilton Inc. St. Louis, MO Tel (314) 368-5850 Druker_Eric@bah.com
Booz Allen Hamilton Inc. 1530 Wilson Blvd., 10th Floor Arlington, VA 22209 Tel (703) 526-2413 Gilmer_Graham@bah.com

Booz | Allen | Hamilton

Colin Smith
Associate

Brandon Herzog
Consultant

Booz | Allen | Hamilton

Booz Allen Hamilton Inc. Suite 2100 230 Peachtree Street NW Atlanta, GA 30303 Tel (404) 658-8011 Smith_Colin@bah.com Booz Allen Hamilton Inc. 1530 Wilson Blvd., 10th Floor Arlington, VA 22209 Tel (703) 526-6040 Herzog_Paul@bah.com

Booz | Allen | Hamilton

17

DICE Functionality Gantt Chart and Cost/Schedule Uncertainty

Organizes project tasks, costs, constraints, schedule interrelationships, and adds uncertainty to individual cost and schedule items

18

DICE Functionality Cost-Loading

DICE can load schedules with time-dependent and time-independent costs, generating standard outputs such as JCL scatter plots and iso-curves

19

DICE Functionality Fiscal Year Segmenting

DICE accounts for costs (and uncertainty) by fiscal year aids budget planning

20

DICE Functionality Run-time Features

DICE incorporates blanket correlation across the model, or specifies individual correlation between schedule tasks, time-independent costs, or risks
The model is optimized for Joint Confidence Level Analysis Greatly increases understanding of how schedule growth impacts cost Contains robust schedule logic functionality and discrete risk integration DICE includes RealTime Analytics to enable industry-leading simulation runtimes Reduces the time required to evaluate decisions and provides decision makers with on-the-spot analysis Booz Allens RTA allows for quick initial runtimes and near-immediate re-runs Interactive features provide intuitive user experience and rapid evaluation of alternatives Enables comparison of multiple different scenarios or confidence levels of the same project Builds off of existing tools (like MS Project, Excel) for seamless data integration, cost visualization, and navigation of risk analysis

21

RealTime Analytics: Excel Tool

22

RealTime Analytics Excel Tool

Introduction
RealTime Analytics Excel Tool Excel add-in enabling fast running of simulations Similar in capabilities to Crystal Ball and @Risk

Benefits:
Runtime savings of >99% vs. COTS tools Analysis can be run without ever leaving the meeting room Decreased simulation runtimes allows running of unlimited number of excursion scenarios

Not a numerical approximation approach such as Method of Moments, it simply runs simulations faster than existing tools
Uses: Cost estimating/risk analysis/risk management Insurance pricing/actuarial models Portfolio optimization/trade-off analysis Cash flow/profit analysis

23

RealTime Analytics Excel Tool:

Benchmarking vs. Crystal Ball & @Risk
Simulation Run Times
Crystal Ball (Normal Mode) Crystal Ball (Extreme Mode) @Risk Real Time Analytics 13 Hrs 48 Min 6 Hrs 10 Min

File Size
265 MB

Baseline Scenario

12 Min 4 Min 10 Min 12 Min <1 Second

~ ~

Each "Excursion Scenario"

7 Hrs 48 Min
Crystal Ball

15 MB

RTA

Assumptions
Simulations include mix of triangular, normal, beta, lognormal and uniform distributions with same parameters in each model 444 correlated distributions with 9,620 forecasted values Baseline scenario is the time to prime the models; includes adding in correlation Each additional scenario is time to re-run model when a single distribution parameter is changed
100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0%
$25,000

Comparison of Simulation Results

Results
Cumulative Probability

$35,000

$45,000

$55,000 Cost

$65,000

$75,000

CB

RTA

'@ Risk

24