Using Parametric Software Estimates During Program Support Reviews

Chris Miller
Office of the Deputy Director, Software Engineering and System Assurance SYSTEMS & SOFTWARE ENGINEERING
Office of the Deputy Under Secretary of Defense for Acquisition and Technology US Department of Defense

October 2008
Version 2.0

OUSD (AT&L) Organization May 2006

USD, Acquisition Technology & Logistics DUSD, Acquisition & Technology

Dir, Joint Advanced Concepts

Dir, Systems and Software Engineering

Dir, Portfolio Systems Acquisition

Defense Acquisition University

Defense Procurement and Acquisition Policy

Industrial Programs

Small Business Programs

Defense Contract Management Agency

Systems & Software Engineering Chris Miller, October 2008

Slide 2

Systems and Software Engineering

Director, Systems & Software Engineering

Deputy Director Enterprise Development

Deputy Director Developmental Test & Evaluation

Deputy Director Software Engineering & System Assurance

Deputy Director Assessments & Support

Systems & Software Engineering Chris Miller, October 2008

Slide 3

Elements of a DoD Strategy for Software

Support Acquisition Success
Ensure effective and efficient software solutions across the acquisition spectrum of systems, SoS and capability portfolios

Improve the State-of-the-Practice of Software Engineering

Advocate and lead software initiatives to improve the state-of-thepractices through transition of tools, techniques, etc.

Leadership, Outreach and Advocacy

Implement at Department and National levels, a strategic plan for meeting Defense software requirements

Foster Software Resources to meet DoD needs

Enable the US and global capability to meet Department software needs, in an assured and responsive manner Promote World-Class Leadership for Defense Software Engineering
Systems & Software Engineering Chris Miller, October 2008

Slide 4

Notional View of Software Measurement

Generate SW appropriate WBS Use estimation tools, techniques, processes, & practices Earned Value Management (EVM) for SW

Determine methods of obtaining cost estimating data

Link SW quality indicators to EVM

Concepts - Requirements - Arch/Design Maintenance

Development -

Software Engineering and Systems Assurance (SSA) initiatives Program feasibility analysis using estimation models Software Resources and Data Report: Feasibility Study Revision of MIL-HDBK-881A to improve software guidance Integration of software metrics with EVM to assess consistency of estimates
Systems & Software Engineering Chris Miller, October 2008

Driving Technical Rigor Back Into Programs Program Support Reviews

Program Support Reviews provide insight into a programs technical execution focusing on:
SE as envisioned in programs technical planning T&E as captured in verification and validation strategy Risk management - integrated, effective and resourced Milestone exit criteria as captured in Acquisition Decision Memo Acquisition strategy as captured in Acquisition Strategy Report

Independent, cross-functional view aimed at providing risk-reduction recommendations

The PSR reduces risk in the technical and programmatic execution of a program
Systems & Software Engineering Chris Miller, October 2008

Program Support Review Activity

(as of August 2008)

Decision Support Reviews NunnPSRs/NARs completed: 57 AOTRs completed: 13 Nunn-McCurdy Certification: 13 Participation on Service-led IRTs: 3 Technical Assessments: 13
OTRR 9% Pre-MS C 19% McCurdy 13% DAE Review 5% Other 25% Pre-MS B 26% Pre-MS A 3%

Service-Managed Acquisitions
Air Force 33% Agencies 9%

Programs by Domain Area

Business 2% Space 4% Comms 3% Missiles 9% Fixed Wing 18%
Other 2%

Land 13%

Marine Corps 10% Navy 21%

Army 27%

Rotary Wing 21% Munitions 3% Ships 8%

C2-ISR 13% Unmanned 4%

Systems & Software Engineering Chris Miller, October 2008

DoD Software Performance: What Were Seeing*

Software issues are significant contributors to poor program execution
Schedule realism (compressed, overlapping) Software requirements not well defined, traceable, testable Immature architectures, COTS integration, interoperability, obsolescence (electronics/hardware refresh) Software development processes not institutionalized, planning documents missing or incomplete, reuse strategies inconsistent Software test/evaluation lacking rigor and breadth Lessons learned not incorporated into successive builds Software risks/metrics not well defined, managed

*Based on over 60 program reviews over the past 3 years

Systems & Software Engineering Chris Miller, October 2008

Software Engineering and System Assurance (SSA) Role

SSA produced software estimates to support several PSR teams in 2007

Program A: SSA was asked to assess software schedule feasibility prior to MS B Program B: Significant software issues

Opportunity for SSA to support program decision making by providing software estimates
Estimation activities aimed at gauging overall program feasibility and quantifying magnitude of top program risks Focus on support for engineering vs. budgeting decisions

Systems & Software Engineering Chris Miller, October 2008

Program A

PSR of aircraft program with ambitious schedule

Three years (36 months) from Milestone B to LRIP Modifications needed to meet U.S. requirements

Developed three software estimates

Software size (SLOC) estimates provided by program office Re-estimated both new and reused code
Based on reuse and code growth Estimates for optimistic, typical, and pessimistic were identified as Min, Mid, and Max

Used three parametric models

COCOMO II, SLIM, and SEER-SEM

Most input parameters set at nominal

Systems & Software Engineering Chris Miller, October 2008

10

Program A

Estimation & Feasibility Analysis

Given an adjusted code size (ASIZE) of 918K to 1590K SLOCs A range of 5375 to 9571 person months should be expected over a 62 to 74 calendar month schedule
COCOMO II
ASize (KSLOC) Effort (PM) Duration (CM)
Min Mid Max

918 5375 62

1,204 7147 68

1590 9571 74

All three models forecasted 65 to 68 months (assuming a 50% confidence-level)

Result:
Analysis revealed existing acquisition strategy was not feasible Service added more schedule to acquisition strategy DUSD(A&T) estimates change in acquisition strategy saved $5 billion
Systems & Software Engineering Chris Miller, October 2008

11

Program B

Major subcontractor with significant software content did not use any parametric estimating tool
Firm Fixed Price subcontract

At one point there was a substantial conflict on estimates between major subcontractor and prime, related to requirements
One portion of software grew from ~250KSLOC to 863KSLOC (3.5x), reflecting shall not degrade current capability requirement

Increase in software engineering staff by 57 percent (40 people) over 9 month period
Based on schedule, contractor should be drawing down software staff

Significant variation in code estimates during review

Systems & Software Engineering Chris Miller, October 2008

12

Program B
Changes in SLOC estimates
CSCI 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 D C E F G H H I G K L C M C 65,000 33,000 15,000 114,000 26,000 6,000 2,000 33,000 15,000 23,000 7,000 1,000 NA NA 752,000 283,255 373,587 7,879 218,609 25,622 16,580 34,806 42,355 126,238 26,404 12,500 29,121 100 2,600 2,078,969 25,467 100,129 7,879 218,609 34,544 16,580 34,806 42,355 119,626 26,404 12,500 29,121 1,000 2,600 1,413,521 18,367 68,129 2,000 16,409 8,922 15,580 17,206 31,655 6,433 2,604 1,000 25,621 1,000 400 7,100 32,000 5,879 202,200 25,622 1,000 17,600 10,700 113,193 23,800 11,500 3,500 Unavail 2,200 82% 99% 100% 99% 100% 100% 100% 99% 100% 91% 100% 100% 100% 81% 60% 99% 100% 95% 100% 100% 100% 97% 100% 91% 100% 100% 100% 43% C 325,000 415,249 401,732 131,921 269,811 98% 98% Supplier A B A Original 40,000 27,000 20,000 NM Review 374,739 50,363 38,962 2 wks. later 250,844 50,363 38,962 New 201,144 40,363 38,962 Reuse 49,700 10,000 0 Complete % 75% 90% 71% Test coverage to-date 68% 85% 60% DR's/Problem reports 47 0 5 1 5 0 0 0 1 0 2 2 0 1 0 1 1 7 0 0

Systems & Software Engineering Chris Miller, October 2008

13

Program B Schedule Estimates

75 KSLOC remaining based on last estimate
1200 1000
150 KSLOC (Optimal Schedule)

Staff Months

800
100 KSLOC (Optimal Schedule)

600 400
50 KSLOC (Optimal Schedule) 150 KSLOC (Nominal) 100 KSLOC (Nominal) 50 KSLOC (Nominal)

200 0 0 5 10 15 Months

20

25

30

Time from Detailed Requirements Complete Through End of Developmental Testing

Systems & Software Engineering Chris Miller, October 2008

14

Program B

Estimation and Feasibility Analysis

Contractor appears to be driven to meet schedules versus costs
Significant increase in effort for minimal schedule savings

Uncertainty in scope of remaining software development

Need to develop firmer size estimates

Recommendations
PMO reach a decision on unstable requirements to prevent further code growth in or out Program office bring in parametric estimating consultant to review contractors estimates for most volatile software components

Result
Acquisition Decision Memorandum requires Service to conduct software review

Systems & Software Engineering Chris Miller, October 2008

15

Integrating Management Indicators

Prioritized Risks

Estimation & Planning

Program Context

Risk Management
Insight Needed

Data & Insight

Risk Mitigation Status

SDP: WBS, IMP, IMS, Metric Definitions

Measurement & Analysis

Systems & Software Engineering Chris Miller, October 2008

16

SSA Software Measurement & Analysis Initiatives

Software EVM Study/Pilot
Focus: Development of cost controls for software component of program Execution: Partnering with DCMA, ARA, PA&E, Contractors

Cost Estimation / SRDR Feasibility Study

Focus: Analysis on software cost estimation practices, availability of historical data, and increasing confidence in estimates Execution: Partnering with PA&E/DCARC and Hill AFB STSC to evaluate existing data sources

Work Breakdown Structure Handbook Update

Focus: Recommendations for update of MIL HDBK 881 to improve handling of software Execution: Partnering with Services, DCMA, ARA, DAU, NDU, PA&E/DCARC and using NDIA expert panel
Systems & Software Engineering Chris Miller, October 2008

Software EVM Study/Pilot

Development of a methodology combining EVM and software metrics analysis to predict cost and schedule overruns Piloting on a 5-year ACAT 1D SW development program, as part of a cross-cutting study Pilot indicator shows ETC (estimate-to-complete) forecasts for:
Existing program management plans EVM measures Software metrics (i.e, growth profile of size, effort, and defects)

Equivalent ETC forecasts provides an increased confidence in project plans

Systems & Software Engineering Chris Miller, October 2008

Integrated EVM Concept

EV ETC SW Metric CPMO Spiral - Sample Summary ETC Performance Measurement Baseline
$1,800 $1,600

BAC = $1420
$1,400 $1,200
$1,051 $1,145 $1,251 $1,318 $1,370

$1,420 $1,471

$1,559

$1,579

Program Plan ETC

DOLLARS ($K)

$1,000
$792

$941 $1,032 $706 715 $478 $565

EV Metrics Calculated Schedule Range: 09/08/08 - 10/12/08 Calculated Estim ate at Com plete Range: $1,797K - $2,515K Cum ulative CPI = 0.79 SW Metrics Calculated Schedule Range: 05/14/08 - 06/18/08 Calculated Estim ate at Com plete Range: $1,570K - $1,710K

$800 $600 $400

$291

$338

$200

$228 $71 $47

Data Date

$0 Oct07

Nov07

Dec07

Jan08

Feb08

M ar08

Apr08

M ay08

Jun08

Jul08

Aug08

Sep08

Oct08

Nov08

Dec08

TIME (Months)
BAC BCWS (PV) BCWP (EV) ACWP (AC) ETC

Acquisition Oversight confidence increases as ETCs overlap i.e., as multiple measures indicate to same conclusion
Systems & Software Engineering Chris Miller, October 2008

SRDR Feasibility Study

SSA funded a STSC feasibility study on normalizing DCARC data
Purpose of the study was to determine if the DCARC data set could be used to develop a parametric software estimation model Dr. Randall Jensen at Hill AFB is the creator of Sage and SEERSEM

Preliminary study results

Data was submitted using a wide variety of definitions that made it difficult to use directly, but was successful in normalizing the data and validating the results The data collected in the SRDR forms allows a first-level parametric model to be constructed (lack of environment information prohibits the creation of a detailed model) Minimum development time projection model
Systems & Software Engineering Chris Miller, October 2008

Minimum Software Development Time Projection Model

Impossible Zone

Analysis of CSCI data: Apparent Maximum Size of 200 KSLOC (per CSCI) Apparent Minimum Schedule (Impossible Zone) Environment & Complexity are primary drivers of variation Apparent Maximum Schedule of 50 months
Source: Long, L. G. et al, Aerospace Corp Report,2004
Systems & Software Engineering Chris Miller, October 2008

Updating MIL-HDBK-881A for Software Overview

Revision objectives:
Revise to make handbook acceptable of software engineering practice Correct errors

Notable changes:
Replace 'material item' with ' acquisition program' Add words to make 'artifacts' equal to 'products' Include words that make 'product-oriented' and 'DoDAF architecture' views interchangeable WRT to creating a WBS

Results
Compiled a Comment matrix Drafted a new Appendix B (for software)

During final review and clean up, the working group was made aware of 90s effort (draft MIL-HDBK-171)
Systems & Software Engineering Chris Miller, October 2008

MIL-HDBK-881A Next Steps

The Software Cost Control working group goal is to provide an community-coordinated set of recommendations Reached out to industry members of NDIA to review the suggested changes
Ensure recommended changes are improvements from industry perspective Obtain additional examples to include in Appendix B NDIA feedback is due on September 19th

Submission of suggested changes to PA&E ARA will take place after collection of NDIA feedback and consolidate into a single baseline
Systems & Software Engineering Chris Miller, October 2008

Questions/Discussion

Contact Information: Chris Miller SSE/SSA Support Software Engineering & Systems Assurance ODUSD(A&T) Systems & Software Engineering Christopher.miller.ctr@osd.mil

Systems & Software Engineering Chris Miller, October 2008

24