Documenti di Didattica
Documenti di Professioni
Documenti di Cultura
Rasmus Vestberg
Supervisors:
Dag Swartling, Linköping Institute of Technology
Michael Trochez, Saab AB
Abstract
In traditional Project Management one determines the budget and measures the cost against
that budget as project goes on. In Earned Value Management, which has been adopted by
Saab’s Naval Systems Division less than two years ago, one also measures the actual
performance by assigning finished work a value.
As a consequence of the recent adoption of the Earned Value Management the Planning
Department asked for a study of following two questions; How far can one do detailed
planning and how should the value be measured?
The report is based on a studious literature study combined and analyzed together with
historical data from existing projects at Naval Systems Division.
The result indicates that the Naval Systems Division is not using Earned Value Management
at its full potential. A lot of money is spent to get the management system working, but many
things are not implemented in the best way. The figures provided to Naval Systems Division
is not showing the true Earned Value as could be if the system was used in a different way.
One can argue that the Naval Systems Division is still using traditional project management
only dressed in Earned Value figures. This is in large extent a consequence of not measuring
the value in the right way. By using other measurement methods the system will measure the
true performance instead of doing subjective judgment as per today.
As a consequence of this flawed use of Earned Value Management many of the people
working in the project does not accept the system. Also a lot of work is spent clearing up the
Earned Value figures to the customer, instead of using them as a helpful tool in the project
management.
By adapting the entire Earned Value concept several benefits will follow. Today the
forecasting of a projects final outcome is made in a time consuming traditional project
management way instead of using the mathematical approach provided by Earned Value
Management. The Earned Value Management forecasting method is very easy to use and will
save a lot of work and money for Naval Systems Division. This study shows that in this case,
the easier method even provides better accuracy.
2
Earned Value Management at
Vestberg, Rasmus Saab Naval Systems Division Sweden
The research also implies that detailed planning could be done very far in advance. No figures
provided by the investigation indicates that detailed planning of nearby work packages are
more accurate than work packages planned far away.
3
Vestberg, Rasmus Table of Contents
Table of Contents
1 INTRODUCTION ....................................................................................................................................... 8
2 METHOD................................................................................................................................................... 12
4
Earned Value Management at
Vestberg, Rasmus Saab Naval Systems Division Sweden
5 ANALYSIS................................................................................................................................................. 68
6 DISCUSSION............................................................................................................................................. 74
5
Vestberg, Rasmus Table of Contents
7 CONCLUSIONS........................................................................................................................................ 84
8 REFERENCES .......................................................................................................................................... 88
9 APPENDICES ........................................................................................................................................... 92
6
Earned Value Management at
Vestberg, Rasmus Saab Naval Systems Division Sweden
7
Vestberg, Rasmus Chapter 1 - Introduction
1 Introduction
This chapter will treat the background and subject of this master thesis.
Today SAAB has changed its name to Saab AB and is in popular speech called Saab
Technologies. The direction of the company has been further increased and Saab AB is today
split into three different segments; Defense and Security Solutions, Systems and Products and
Aeronautics. These different segments are further split into several business units. One of the
business units included in Defense and Security is Saab Systems, with over 1300 employees,
which is further split into five divisions. One of those divisions is the Naval Systems Division
Sweden (NSDS). 2
1.2 Background
The Earned Value Management (EVM) is quite recently adopted at Naval Systems Division.
The model is in use but without wide acceptance and cooperation from the staff. The level of
detail planning when setting up a project is subject for constant discussion. Guidance
regarding planning package conversion and overall Earned Value Management is needed so
projects can have a shorter start-up time without loosing quality.
1
http://saabnet.saabgroup.com
2
http://saabnet.saabgroup.com
8
Earned Value Management at
Vestberg, Rasmus Saab Naval Systems Division Sweden
1.3 Purpose
The purpose of this master thesis is to study the use of Performance Measurement Techniques
(PMT) and Planning Packages in Earned Value Management at Saab Systems Naval Systems
Division. Different PMT will be compared to each other to tell which ones are best suited for
Saab Systems Naval Systems Division. Studying the accuracy of the project planning and
measuring aims to provide guidance for conversion of Planning Packages. The thesis aims to
answer the following two main questions:
• In which time frame is the knowledge regarding future work high enough to
breakdown planning packages into work packages?
• How should NSDS measure the work to attain most valid Earned Value metrics?
1.4 Disposition
In chapter 2 the different methods used during this thesis is discussed in detail. The pros and
cons with used methods are discussed to provide the reader with information regarding
subject of errors in the work.
Chapter 3 treats the Frame of Reference subjective to the purpose of the thesis. Common
knowledge and facts found in literature is described in detail to give the reader a basic
background of Earned Value Management.
Chapter 4 will present the figures received from analyzing the historical data. The presented
figures will be described and analyzed in chapter 5. The connection between the analyzed
figures and the frame of reference is made first in chapter 6, where furthermore my own
reflections and ideas are presented. Also a description of how work is conducted today at
Naval Systems Division is displayed in chapter 4.
In chapter 7 the conclusions based on chapter 3 to chapter 6 are presented in relative compact
form aiming to answer the main questions in the purpose.
9
Vestberg, Rasmus Chapter 1 - Introduction
For people with wide knowledge about NSDS and EVM it is possible to skip chapter 3. Focus
in the reading should then be at chapter 2.2.2 and chapter 4 to chapter 7. However I
recommend reading the entire report for best understanding of conclusions made.
10
Earned Value Management at
Vestberg, Rasmus Saab Naval Systems Division Sweden
11
Vestberg, Rasmus Chapter 2 - Method
2 Method
This chapter will describe the different methods used for conducting this study. The chapter is
divided into three parts. The first part will cover the theory behind the used methods followed
up by an illustration of how they have been used. The last part will discuss the possible
disadvantages with the chosen methods.
2.1 Theory
This part will present a few theories behind some of the used methods.
3
Lekvall and Wahlbin (2001)
12
Earned Value Management at
Vestberg, Rasmus Saab Naval Systems Division Sweden
Reliability
Reliability is the probability of obtaining the same results if the research was conducted in the
same way once again. High reliability means that the result would be pretty much the same. In
qualitative surveys it might be difficult to obtain the same results again since for example the
interviewed persons might answer different from time to time. Quantitative research is more
likely to be of high reliability because of the mathematical methods used.4
Validity
The validity tells how well the research reflects the reality. A high level of validity indicates
that the research reflects the reality in a good way. Inversely from reliability the validity is
usually higher in qualitative research because of simplifications made in the quantitative
research.5
2.1.3 Interviews
Interviews can be made in several ways; for example personal interviews, telephone
interviews, written questionnaire etcetera.
In a personal interview there are some things to think about; the degree of standardization is
one of them. A completely standardized interview consists of identical questions posed in the
4
Lekvall and Wahlbin (2001)
5
Lekvall and Wahlbin (2001)
13
Vestberg, Rasmus Chapter 2 - Method
same order for all interviewed. The level of structuring is another thing to determine.
Structuring means to what extent the questions have predefined replying options.6
It is also important to gain the trust from the interviewed person if the answers should be
really truthful. Therefore the most sensitive questions should come in the end of the interview,
when as much trust as possible has been gained.7
Several articles have been used to extend the picture provided by PMI. Different alternative
theories about and complementing EVM has been studied in the search for the best suited
method for the Naval Systems Division.
The internal document “Projekthandbok” has been studied to get a basic knowledge about the
routines at the Naval Systems Division. This document describes when, why and how a
project should be managed at Naval Systems Division.
It could have been possible to attain knowledge regarding Earned Value Management from
people working at Naval Systems Division. However, this has been ignored to prevent any
influence of the understanding and conclusions.
6
Patel and Davidsson (2003)
7
Cassel and Symon (2004)
14
Earned Value Management at
Vestberg, Rasmus Saab Naval Systems Division Sweden
2.2.2.1 Interviews
To make sure the understanding, provided from the “Projekthandbok”, of the project work at
Naval Systems Division is the truly used and not just the hoped-for, personal interviews have
been conducted. This has provided the research with primary information. The above
mentioned “Projekthandbok” has not been updated since the introduction of Earned Value
Management, why this supplementation was absolutely necessary. The interviews have been
made with people at different positions to get the entire scope.
All the interviews have been made with a very low degree of standardization and structuring.
Basically the interviews consisted of a conversation where the interviewed was asked to
describe their work, their contact with the project organization and their contact with Earned
Value Management.
If the interviews would have been made with high standardization and structuring I do not
think I would have got the entire picture. The results would have been relying on my
capability to ask the right questions. Since many of the interviews were held early in the
process I did most likely not have had the understanding needed to ask the right questions, at
that time.
15
Vestberg, Rasmus Chapter 2 - Method
forecast is based on the knowledge about the work to be done, thus the same thing as when to
convert planning packages. See 3.2.2.5.
When the forecast is stabile within 10% of the Actual at Completion (the final outcome) the
Naval Systems Division would feel satisfied with the knowledge of the work to be done.8
This means that if we can, based on the EVM reports, show any trend indicating when the
forecasting stabilizes within 10% of its final outcome, this would give a scientific/statistical
point for planning package conversion.
8
Michael Trochez
16
Earned Value Management at
Vestberg, Rasmus Saab Naval Systems Division Sweden
Figure 1 is an example of the method. Each line represents the forecasting relative the Actual
at Completion for different work packages. The broken line highlights the 10% limit. As seen
in the figure, the forecasting stabilizes below 10% somewhere between 5-7 months before its
completion date (all work packages has their completion date at the end of the graph). This
would mean that we could plan at work package level 6 months ahead (the median in the
figure).
40%
30%
Forecast relative Actual at Completion
20%
10%
0%
6
06
7
6
06
7
06
07
apr-0
jun-0
jul-0
jan-0
apr-0
jun-0
sep-0
okt-0
feb-0
mar-0
aug-
nov-
d e c-
maj-
maj-
-10%
-20%
-30%
Date
The work packages to be studied had to be finished. This is the only way to know the precise
outcome of the work package. To determine which work packages that really where finished I
have studied the project Cost Performance Reports. Finished work packages have an Earned
Value equaling the Planned Value, regardless of the performance of that specific work
package.
When the work packages to be measured were selected the graphs were built through the
following formula:
17
Vestberg, Rasmus Chapter 2 - Method
It is obvious that as the work package are approaching completions there will be less work left
to forecast. Therefore the estimate will always approach the actual when the work package
reaches completion.
If we use Formula 1 with following figures; Actual Cost Cumulative = 900, Actual Cost at
Completion = 1000 and Estimated Cost at Completion = 1100. Then the Forecasting
Accuracy will be 10% from perfect. Since there is only 100 hours left of work but estimated
200 hours, it is obvious that the estimate is pretty bad although the figure looks okay. That
also makes it interesting to look at the ability to forecast the work packages relative the
amount of remaining work.
Using Formula 2 the simple example above will be 100% wrong instead of 10%.
This second method will be difficult to use when the baseline has been changed, why the
periods in the study with this problem have been left out. Also the case when the final cost
has been achieved is left out, since the formula will give division with zero and the work
package is finished anyway.
It could have been possible to determine the time frame of planning package conversion by
interviewing people responsible for the plans. This would however only represent the wished
for time frame without any scientific support.
18
Earned Value Management at
Vestberg, Rasmus Saab Naval Systems Division Sweden
2.3.1.1 Literature
There is always a good thing to establish the facts from several independent sources. This
could perhaps have been done more thoroughly by using even more different sources.
However, it is not likely that result from the literature study would have been different even if
more sources would have been used. The sources used are the fundamental source for the
subject in mind. And the publisher is in some way one of the owner of the method available in
Earned Value Management. The literature study about EVM could be said to be of both high
reliability and validity.
The internal “Projekthandbok” is not updated for a couple of years which could affect the
understanding of the situation at the Naval Systems Division. The main concept of this study
is anyway focused on EVM which is not included in the “Projekthandbok”. Therefore the
interviews have been the main source of this information and confusion based on non updated
documents would be no issue. Based on this the validity could be questioned but should
anyway not affect the study in general.
2.3.1.2 Interviews
It is always a risk of missing vital information while conducting an interview alone. One
might for example miss important body language while taking notes. Since the interviews
were made with very low level of standardization and structuring the reliability is low and
there is a great risk that one might get different answers doing a new interview. However the
validity is high since the interviews were made to build the entire picture of the situation at
Naval Systems Division. Several people have contributed to the understanding of NSDS’s
project management why the gathered picture would be very valid.
19
Vestberg, Rasmus Chapter 2 - Method
Old finished project have been studied to see if they could contribute to the gathering of data
but without success. Very little information is stored regarding old projects. When there is
information stored about old projects it is not the wanted, or the information is at a too high
level telling us nothing about the work packages. Although the findings can not be called
statistical they should work as a warning signal indicating a very obvious trend.
20
Earned Value Management at
Vestberg, Rasmus Saab Naval Systems Division Sweden
21
Vestberg, Rasmus Chapter 3 – Frame of Reference
3 Frame of Reference
This chapter will treat the Frame of Reference subjective to the purpose of the thesis
1600
Budget
1400
1200
Cost
1000
1000$
800
600
400
200
0
jan feb mar apr maj jun jul aug sep okt nov dec
Date
9
ANSI/EIA-748-A-1998
10
Fleming and Koppelman (2005)
22
Earned Value Management at
Vestberg, Rasmus Saab Naval Systems Division Sweden
In the end of the 1950’s the United States Navy introduced the Program Evaluation Review
Technique (PERT) as a network scheduling and risk management device. In 1962 PERT was
extended to PERT/costs aiming to add resources to the time scheduler. Since the computers
back then was not powerful enough PERT did not live on for many years. However PERT left
something to the afterworld. The implementation of PERT/costs had required some new
reporting formats from the contractors. One of those formats contained “value of work
performed” versus “actual cost”.
By the second half of the 1960’s the United States Air Force made a new effort to oversee the
industry performance. Instead of imposing a specific management control system they wanted
the industry to satisfy some criteria with their existing management control systems. The
result was an earned value concept called Cost/Schedule Control Systems Criteria (C/SCSC)
which comprise 35 criteria that one had to adapt while being a contractor for the Department
of Defense (DoD).
11
Fleming and Koppelman (2005)
23
Vestberg, Rasmus Chapter 3 – Frame of Reference
During the next three decades the C/SCSC evolved to be more and more difficult to use. The
basic criteria were added with implementation guidelines, surveillance manuals and
implementation checklist which were intended to be used as guidelines used with good
judgment. However the guidelines were not always used as guidelines but instead as
peremptory rules. Furthermore the private sector never adapted the C/SCSC. One of the main
reasons why the industry never adopted the C/SCSC, except the rigid guidelines, was the new
vocabulary. Instead of calling things what they are C/SCSC used their own expressions and
abbreviations. For example earned value was called “Budgeted Cost of Work Performed”,
BCWP, or simply “p”. Overrun was called OTB which stood for “Over Target Baseline”.
Some of those expressions are still used today by some people, although there are simpler
expressions meaning the same thing.
In 1995 the National Defense Industrial Association, NDIA, accepted the task of rewriting
DoD’s formal earned value criteria. The objective of the rewriting was to make the criteria
more practicable for the private industry. The 35 criteria became 32 and terms as Budgeted
Cost Work Performed (BCWP) and Budgeted Cost Work Scheduled (BCWS) where changed
to Earned Value and Planned Value. But the main change was in the attitude of all parties.
The private company was now adopting the technique since it had proven to be best-practice
instead of something that where forced from the government to their contractors. The
restriction of EVM to DoD was removed and in 1998 the new technique became accepted as
an ANSI/EIA document. 12
12
Fleming and Koppelman (2005)
13
Christensen David S (1998)
24
Earned Value Management at
Vestberg, Rasmus Saab Naval Systems Division Sweden
what has actually been performed. If the actual work is in the same magnitude as the actual
cost, the project is ahead of schedule and not really overspending. In the same way a project
that is spending less money than budget might not be as successful as it seems. If the actual
work performed also is less than planned the project is in fact behind schedule and not only
cost effective as traditional project measurement would say.
Let us look at an easy example to make things even clearer. We have a project extent over ten
months (see Figure 3) with a total budget of $1,000,000. Each month has a budget of
$100,000 and distinct milestones for each budgeted $100,000. Once a milestone is
accomplished the project will earn $100,000. The budget is labeled the “Planned Value”
which consists of two elements, the work scheduled and the budget for that work.
1200
Planned Value
1000
800
1000$
600
400
Actual Cost
Earned Value
200
0
jan feb mar apr maj jun jul aug sep okt nov
Date
After four months the project has spent the planned $400,000 and would by traditional project
management be performing well. The Actual Cost is therefore $400,000. When also looking
at the work performed, as in EVM, you notice that only three of the four milestones have been
achieved and the Earned Value is only $300,000. One could immediately see that the project
25
Vestberg, Rasmus Chapter 3 – Frame of Reference
is running behind and have a negative Schedule Variance at $100,000 (Schedule Variance =
Earned Value – Planned Value, see Figure 4).
Also shown in Figure 3 is the Actual Cost, $400,000. The Actual Cost is greater than the
Earned Value which tells us that the project is not cost effective. The Cost Variance is defined
as the Earned Value – Actual Cost (see Figure 4).
Earned Value is recommended to help managing company’s different projects and act as a
comparison between a company’s different projects. (Also other company’s projects using
Earned Value can be compared)14 Although some of the basics and theories behind Earned
Value can be used for continues business it is not recommended to use the entire Earned
Value Management method for that purpose.15
Voices have been raised against the Earned Value method, saying it is one of the larger cost
drivers in a project.16 So why should one use Earned Value? There are several benefits with
employing Earned Value and among others it can be said that; it provides reliable data. It
provides early warning signals. It makes reliable predictions possible etcetera.17 Also there is
a general phobia about Earned Value complexity which undermines the acceptance.18
The core basics of Earned Value is the measurement of actual work performed compared to a
baseline plan.19 Perhaps the most advantageous thing about Earned Value is the, through
measurement of performance, provided possibility of reliable prediction of the project. Some
people would say that this measurement requires too much effort compared to the
advantageous with the system.20 Those people are most likely just prejudice based on old
times complexity in Earned Value. Those doubters tell us that there are reasons to keep
14
Averstad, Dan (2003)
15
Fleming and Koppelman (2005)
16
Fleming and Koppelman (2005)
17
Christensen David S (1998)
18
Kondur, Mohan (2007)
19
Christensen David S (1999)
20
Fleming and Koppelman (2005)
26
Earned Value Management at
Vestberg, Rasmus Saab Naval Systems Division Sweden
Earned Value as simple as possible.21 To keep Earned Value simple should not be a problem.
The concept of Earned Value is often used as a daily routine. For example, when shopping
grocery many people look at the price per kilo. Concerns about what you get for the money
could in some extent be equal to Earned Value. Shopping based only at the total cost is more
similar to the traditional project management.
The concept of Earned Value requires a performance plan, the Planned Value, which
combined with the physical Earned Value and the Actual Cost provides the cost performance.
21
Kondur, Mohan (2007)
27
Vestberg, Rasmus Chapter 3 – Frame of Reference
1200
Planned Value
1000
800
1000$
600
Actual Cost
Cost Variance
400
Schedule Variance
Earned Value
200
0
jan feb mar apr maj jun jul aug sep okt nov
Date
The CPI in our example is 0.6 which in the same way as the SPI can tell us something about
the future of the project. For every dollar spent we have only got a value of $0.6 which gives
us the final cost of $1,666.666 for the project.
EV EV
CV = AC − EV , SV = PV − EV , CPI = , SPI =
AC PV
Formula 3 a-d
With these figures you can make the correct decisions about the project. It is not likely that
the estimated total cost of the project is possible to improve, unless you change the scope.
However the duration of the project can perhaps be improved by adding extra funds. Research
28
Earned Value Management at
Vestberg, Rasmus Saab Naval Systems Division Sweden
from hundreds of projects within the DOD, shows that the performance of a project at 20% of
completion is very accurate for the entire project. Very seldom a projects performance
improves after the 20% completion. If the performance changes it often gets worse and not
better.22 Therefore management no longer must wait until the total funds have been spent to
determine that additional budget is needed.
A useful tool for scooping the project is the Work Breakdown Structure (WBS). A WBS can
be developed in different kind of ways and take different shapes; the important thing is that it
helps grasping the scope.25 The type of WBS discussed further will be the one described in
PMBOK.26
A WBS looks like an organizational chart but it is important to realize that it is not. However
the WBS is for the project leader what the organizational chart is for the company executive.27
The WBS represents the project work to be done and the deliverables of the project. The
WBS is separated into different products or work in several levels. Either one chose to do a
22
Fleming and Koppelman (2005)
23
A Guide to the Project Management Body of Knowledge (2005)
24
ANSI/EIA-748-A-1998
25
Wenell, Torbjörn (2004)
26
A Guide to the Project Management Body of Knowledge (2005)
27
Fleming and Koppelman (2005)
29
Vestberg, Rasmus Chapter 3 – Frame of Reference
product or a work oriented WBS (se Figure 5 and Figure 6).28 A success factor is to let the
WBS reflect the way the manager actually plans to manage the project.29
At the top levels the product or work is described very briefly, called planning packages. It is
not uncommon that the owner of the project specifies the three top levels of the WBS. The
lower levels are extended by the project leader to contain work, costs and schedule, the work
packages. It is first at work package level that one can accurately estimate cost and schedule
and it is also at this level the Earned Value is computed.30 In very large projects it can be hard
28
Wenell, Torbjörn (2004)
29
Fleming and Koppelman (2005)
30
ANSI/EIA-748-A-1998
30
Earned Value Management at
Vestberg, Rasmus Saab Naval Systems Division Sweden
to plan everything on work package level from the beginning. The manager then has to do a
rolling planning bye and bye.31 It is also common that the WBS is reassessed from time to
time when the knowledge gets wider or when requirements changes.32 Each company must
determine its own standard for the conversion of planning package into work packages.33
Different deliverables need different levels of breakdown to clearly define the work. More
detailed breakdown gives better opportunities to control and manage the work. In the same
time too detailed breakdown will inhibit the productivity and lead to ineffective work. It is of
most importance for the project manager to keep the breakdown at a balanced level. 34
For the breakdown of the WBS-element it takes that different subprojects and deliverables is
split into its basic elements. Each element must be clearly defined and assigned to a specific
unit taking the responsibility for the task. When work has been defined it is possible to assign
the tasks to personnel through an Organization Breakdown Structure, OBS. The lowest level
of the WBS is expanded with more details into a WBS-dictionary. This dictionary is the
Statement of Work (SOW) for the project.
Of course there are other techniques used than the WBS to scope the work. The important
thing is naturally not which technique one chooses to use, but the result when grasping the
project. Since WBS is the superior most used technique and also the one used at Naval
System Division there is no reason to study other techniques further.
3.2.2.3 Scheduling
Independent of which management techniques are being used all projects has to schedule the
work, so also the ones applying Earned Value. Perhaps the ones using Earned Value
especially needs it.35 Larger projects may have multiple schedules correlated to each other
31
A Guide to the Project Management Body of Knowledge (2005)
32
ANSI/EIA-748-A-1998
33
ANSI/EIA-748-A-1998
34
A Guide to the Project Management Body of Knowledge (2005)
35
Kondur Mohan (2007)
31
Vestberg, Rasmus Chapter 3 – Frame of Reference
with a formal scheduling system. But also the smallest project needs at least a project master
schedule, PMS, containing the most important tasks to be performed.36
To implement Earned Value Management there are two basic needs. First, as earlier
discussed, one must define the project and all the work which has to be done. Second, the
work has to be put into a time frame so that performance can be measured against it.37 The
time based schedule together with the WBS is evidently the spinal column of the projects
baseline.
The Earned Value Project Management implies that there is no other way to establish a time
schedule except the Critical Path Method (CPM).38 The Critical Path Method is a technique
used for analyzing web plans. There are some different kinds of web planning methods
although they are all very similar. The concept of them all is to display the tasks in the project
in a web, showing the connections between the different tasks.39 The CPM and the
dependencies between the tasks are then used to show the shortest possible duration of the
project.
The difference between cost variance and schedule variance takes some extra attention except
what has been told earlier. A negative cost variance is always a serious matter telling us that
the project is not performing as planned. A negative schedule variance on the other hand does
not have to be a big issue. The schedule variance only tells us how much of the planned work
we have accomplished. That could of course be a big problem if we for example have a tight
schedule and an important deadline to keep. But it could also be the result of a late start, either
on some of the work packages or on the entire project. If the negative schedule variance is a
consequence of some single work packages starting late, and those not being on the critical
path, the variance in fact could be insignificant.40
36
Fleming and Koppelman (2005)
37
Fleming and Koppelman (2005)
38
Fleming and Koppelman (2005)
39
Project Management Body of Knowledge (2005)
40
Fleming and Koppelman (2005)
32
Earned Value Management at
Vestberg, Rasmus Saab Naval Systems Division Sweden
Most of the time schedule variance can also be compensated. In the best case it is enough to
reorganize the schedule to get back on track. Otherwise one can ad extra money to speed up
the work. A negative cost variance is seldom to compensate for. The basic reason for this
difference in cost and schedule variance is that the scope is almost never dependent on what
date the work is being done.
• Weighted Milestones
• Fixed Formula
• Percent Complete Estimates
• Percent Complete and Milestones
• Equivalent Completed Units
• Apportioned Relationship
• Level of Effort
Different techniques might suite different projects. The choosing of technique, to make it fit
the projects various work packages, is up to the project leader. One does not have to use the
same technique over the entire project. The important thing is to decide at planning stage
which technique to use and be consistent to that.41
41
Fleming and Koppelman (2005)
33
Vestberg, Rasmus Chapter 3 – Frame of Reference
packages are divided into several different milestones one must distribute the planned value
of that work package among the milestones. This could be done in a simple method where
you divide the planned value equal over all milestones. Since a milestone is unique and
measurable, for example a specific task being finished, the equal division might not fit. It is
therefore ok to divide the planned value uneven between the milestones. The important thing
is that the assigned value for each milestone reflects the actual planned work for achieving
that specific milestone.42
This method is preferred when looking at the performance measurement.43 However it is the
most difficult method when initially planning the project. It takes close cooperation between
scheduling people, resource-estimating functions and work packages managers to get the
milestones meaningful and correctly weighted.
This method will act well, as long as the work packages are short in duration. If the work
packages span over several report periods, this method will give a lag in the Earned Value. On
the positive side it is one of the simplest methods both to plan and measure. A suited
application for this method is purchased materials, earning 25% of the value when order has
been done and 75% when the material has been delivered.
42
ANSI/EIA-748-A-1998
43
Fleming and Koppelman (2005)
34
Earned Value Management at
Vestberg, Rasmus Saab Naval Systems Division Sweden
This method is the easiest of them all to administer which is probably why it have received
increasingly wide acceptance in the private industry.45 But it also has obvious disadvantages.
If one would like to play games with the Earned Value this method is the most vulnerable.
The person in charge of each work package is obviously able to put a gloss on the completed
work. Especially when the pressure is high from the senior managers, the estimates tend to be
a little bit on the large side. To minimize this phenomenon some companies have developed
procedures to incorporate some objectivity into the subjective estimate. Another disadvantage
is that it can sometimes be difficult to actually know how much work has been accomplished.
Of course the situation could be the other way around. The responsible manager
underestimates the performance since they know their reputation might be at stake. It is
always easier to defend modesty than exaggerations. Also, underestimating the progress will
give a buffer for the next report period.
44
ANSI/EIA-748-A-1998
45
Fleming and Koppelman (2005)
35
Vestberg, Rasmus Chapter 3 – Frame of Reference
This method is said to be the best out of two methods, the ease of Percent Complete and the
assured Milestones.46 This seems to work well in any industry and any type of project.
Especially it is a suited method for project with large work packages.
46
Fleming and Koppelman (2005)
47
ANSI/EIA-748-A-1998
36
Earned Value Management at
Vestberg, Rasmus Saab Naval Systems Division Sweden
will be similar in the apportioned task. However, the cost variance is individual for the related
tasks, since the cost is incurred for the apportioned task itself.
Using Level of Effort means that there could be no Schedule Variance, ever. But, since the
cost is still measured the same way as in all measurement techniques, the Cost Variance could
be very big. This is not good at all and it is doubtful that this even qualifies as Earned Value.49
In every project using Level of Effort on some work package, one should quantify the value
of Level of Effort. The quantification is made by dividing the value of Level of Effort with
the value of the total baseline. If the value of Level of Effort tasks exceeds 5-10% the project
is measuring the passage of time instead of true performance.50
A way of dealing with this is to “quarantine” the Level of Effort tasks. That simply means you
remove all task measured with Level of Effort from the baseline and place them in a separate
budget. Those tasks should not be bothered to measure, since they reflect nothing but the
passage of time. 51
One could also replace the Level of Effort with subjective Percent Complete measurements.
The Percent Complete method has its disadvantages as well, but it is still preferable to Level
of Effort52
48
ANSI/EIA-748-A-1998
49
Fleming and Koppelman (2005)
50
Fleming and Koppelman (2005)
51
Fleming and Koppelman (2005)
52
Fleming and Koppelman (2005)
37
Vestberg, Rasmus Chapter 3 – Frame of Reference
3.2.2.5 Forecasting
The Earned Value Management is a very helpful tool when predicting the outcome of the
project. To make accurate predictions, it is very important that the plan is of good quality. If
the plan is not credible the same thing will be evident for the predictions. Therefore the
managers really have to develop a plan that they do believe is possible to keep up with.
If the measurement of a project in progress shows a cost overrun or a schedule variance the
manager has to take aggressive actions. Unfortunately managers often do not like or accept
the final forecast results and choose not to take necessary action. If the managers did believe
in the plan they created, they should also believe in the predictions given to them concerning
their plan and statistic figures.
53
Fleming and Koppelman (2005)
38
Earned Value Management at
Vestberg, Rasmus Saab Naval Systems Division Sweden
This formula can be looked upon as a best case scenario. The project has incurred an overrun
which not likely will go away. Assuming that the rest of the project will follow the plan, this
gives the accurate estimate. Almost never the project will underrun in the future and
compensate the early overrun.54
To smoothen out the varieties of the project performance the used CPI is the cumulative to
date. This will not give the project team a perfect picture of their recent work, but it will
provide an accurate long-term forecasting. The Formula 5 can be simplified as follows.
Budget at Completion
Estimate at Completion =
Cost Performance Index
Formula 6
This method is often referred to as the “most likely” outcome of the project.55 Studies of
several projects show that the CPI at 20% completion will be stable for the rest of the
project.56
54
Fleming and Koppelman (2005)
55
Fleming and Koppelman (2005)
56
Christensen and Heise (1993)
39
Vestberg, Rasmus Chapter 3 – Frame of Reference
Since it is very common that project team strive to be on track with the schedule, this formula
is appropriate. If a project is behind schedule, it will take extra effort and probably also
money to get it on track again. By multiplying the CPI and SPI one gets a good performance
factor. If both the SPI and CPI is less than one (1), the outcome will be worse than in ordinary
cumulative CPI forecasting. This is perfectly natural, since it will take extra funds to
compensate the schedule slippage. On the other hand, if one of CPI or SPI is above one (1),
the performance factor indicates that the better of the two compensates the worse.
This calculation method is often referred to as the worst case scenario. Although, there are
some people claiming this to be the most likely outcome of the project.57
There is no statistic data provided showing this to be a valid forecasting method. Actually
some would say this is a to simplified way of predicting the completion date and that it takes
careful Critical Path work to do this properly. The SPI only indicates the status of the project
based on time and is not sufficient to forecast the completion date itself. Analyzing the
Critical Path combined with SPI calculation is the recommended method.58
57
Fleming and Koppelman (2005)
58
Fleming and Koppelman (2005)
40
Earned Value Management at
Vestberg, Rasmus Saab Naval Systems Division Sweden
measure and it correlates well with the cost performance which naturally also is measured in
money. However, this way of measuring the schedule performance has disadvantages.
Since the Schedule Variance and the Schedule Performance Index is calculated based on the
planned value, some false results will occur. When the project is supposed to be finished the
Planned Value will not change any more.59 This means that no matter how late the project
actually finishes, the Schedule Variance will approach zero and the Schedule Performance
Index approaches one. This is simply because the Earned Value always equals the Planned
Value at completion of a project; otherwise the project would not be completed.60
Because of this flaw, Earned Value Management is sometime labeled as a cost management
system.61
The remedy for this is to use Earned Scheduling. This is the same thing as Earned Value but
the Schedule Performance is measured in time instead of in money. The Earned Schedule, ES,
is reflecting the time at when the achieved work should have been done according to plan.
The Actual Time, AT, is the time of today. The planning of the project is exactly the same as
with Earned Value and one prepares a baseline to measure against. The only difference is that
one calculates the Schedule Variance as the horizontal difference against the baseline, instead
of a vertical difference against the baseline (see Figure 7). That is, the vertical axis represents
money while the horizontal axis represents schedule time. That will give us the following
formula for calculating SV(t) and SPI(t).62
ES
Schedule Variance(t ) = ES − AT Schedule Performance Index(t ) =
AT
Formula 8 a and b
The classic Schedule Variance, SV($), shows the difference between the amount of achieved
work and planned work. The SV(t) shows the difference in calendar time between today’s
59
Corovic Radenko (2007)
60
Lipke Walt (2003)
61
Lipke Walt (2006)
62
Lipke Walt (2003)
41
Vestberg, Rasmus Chapter 3 – Frame of Reference
date and the date when one should have accomplished the amount of work being finished of
today (se Figure 7).63
1600
Budget
1400
1200
SV($)
SV(t)
Cost
1000
1000$
800
600
400
200
0
jan feb mar apr maj jun jul aug sep okt nov dec
Date
Even if analyzing the Critical Path is probably the best way of forecasting the completion
date, this extension to Earned Schedule provides a tool suited also for time prediction. The
benefit with Earned Schedule against analyzing the Critical Path is the simplicity. In large
projects the network schedule is often of very high complexity, therefore a simple analyzing
tool as Earned Schedule suits well.64
Studies have been made, although not as extensive as DoD’s EVM-study, showing that this
new approach correlates well with the Earned Value basics. Not only do they work together
with EVM’s cost analyze, they also behave correctly over the entire project.65 This makes it
63
Abba Wayne (2002)
64
Henderson Kym (2005)
65
Henderson Kym (2003)
42
Earned Value Management at
Vestberg, Rasmus Saab Naval Systems Division Sweden
possible to forecast the outcome of the project, as well on schedule basis, without analyzing
the critical path.66
If one agrees that Earned Schedule provides the true picture of the Schedule Performance, we
can use it to see how incorrect the SV($) can be. All practitioners agree in that SV($) give
erroneous figures after the completion date. But, one can show that SV($) is unreliable over
the entire project when compared to SV(t). This is in the case when the project has a non
linear baseline (see Figure 8).
1600
1400 Budget
Earned Value
1200
1000
1000$
SV($)2
800
600
SVt)2
400
200
SV($)1
SV(t)1
0
1 2 3 4 5 6 7 8 9 10 11 12
Date
If looking at a project with perfect performance, only starting one month late, the problem
becomes obvious. The SV(t) is constant over the entire project (SV(t)1 = SV(t)2), while the
SV($) changes radically (SV($)1 ≠ SV($)2). A project with perfect performance would have
the same Schedule Variance through the entire project. If the baseline is perfectly linear, the
two different measurement techniques will provide the same data.67
66
Henderson Kym (2004)
67
Corovic Radenko (2007)
43
Vestberg, Rasmus Chapter 3 – Frame of Reference
Defining the work and developing a Work Breakdown Structure, WBS, is essential for the
successful use of EVM according to DACS. Each work package in the WBS must have a
short duration or be divided into measurable milestones. Further more it is important to keep
the WBS on the right level. To detail planning will create an overload of data which is hard to
track while lack of details may mask important information. The Gold Practice suggests 3-4
levels in the WBS for regular projects and up to 6 levels for very large and complex projects.
The Earned Value is suggested to be calculated as either 0% or 100% where 100% is given
when the task is completed and validated. The Gold Practice also suggests that one should
establish specific exit criteria for each task to make the measurement easier. Doing subjective
assessment to gain the earned value is strongly not recommended.
Later in the project when the work breakdown is done, the baseline is set according to that
specific planning package. Then you can not change the baseline and as a consequence not
either the budget of that working package. That means that the breakdown and detail planning
comprise splitting the budget on different work packages.
68
www.goldpractices.com
44
Earned Value Management at
Vestberg, Rasmus Saab Naval Systems Division Sweden
45
Vestberg, Rasmus Chapter 4 – Empirical Study
4 Empirical Study
This chapter will present the results from the analyzed work packages and the mapping of
how project work is conducted at NSDS. The figures used to compute the data is shown in
Appendix 1. All results are presented in graphs to increase the visibility.
69
Project Management Body of Knowledge (2005)
70
Henrik Olsson
46
Earned Value Management at
Vestberg, Rasmus Saab Naval Systems Division Sweden
more expensive than agreed there is not much the project leader can do. This type of project
model is called procurement model.71
The Naval Systems Division has been running projects in a more standardized way earlier but
a couple of benefits with today’s organization led to the changes. First of all, many of the
projects at Naval Systems Division develop or use the same technology or product. Roughly
you could say that the Naval Systems Division has a couple of systems that are developed or
tailor-made for special applications and customers. Therefore there are very much of the work
that is similar or perhaps exactly the same in different projects. The Naval Systems Division
uses something they have chosen to call “the Family”, which is a collection of existing
solutions for different problems. If someone improves something they got from “the Family”
this improvement will be saved to provide others with the latest solution. Hence the existing
organization makes double work less common.72
This type of organization also makes it easier to develop or use new technology since it is
more flagrant where and by whom the technology should be adapted. By the same reason it is
easier to merge organizations since the functions and responsibilities are more obvious.73
Much of the reasons can probably be referred to the size of the projects. The benefits with the
more traditional project organization were not very appreciable since the size made the project
leader loose control anyway. A fact is that the Naval Systems Division has increased its result
considerably with the new organization.74
Since the beginning of this study NSDS has made some changes in the project organizations.
These are not dealt with since they do not affect the purpose of the study and the analyzed
data derives from the described organization.75
71
Projekthandbok, SAAB Systems
72
Henrik Olsson
73
Person A
74
Person A
75
Trochez Michael
47
Vestberg, Rasmus Chapter 4 – Empirical Study
76
Ulf Karlsson
77
Stefan Larsson
78
Projekthandbok, SAAB Systems
79
Michael Trochez
48
Earned Value Management at
Vestberg, Rasmus Saab Naval Systems Division Sweden
The breakdown into work packages is made as a Bottom-Up planning. This means that the
sum of the work packages does not always equal the corresponding planning package. If the
difference is insignificant, no action is taken at all. If the difference is of major size, relative
the project, the baseline can be changed. If the baseline needs to be changed that decision has
to go through the managerial body.82
80
Projekthandbok, SAAB Systems
81
Projekthandbok, SAAB Systems
82
Henrik Olsson
49
Vestberg, Rasmus Chapter 4 – Empirical Study
Periodically, at least once a month, the person responsible for each work package updates the
plan and the planned cost for the work package. To make Earned Value Management work
changes in the original baseline is not recommended, SAAB uses therefore several baselines.
The original baseline is named “Baseline 10” and can only be changed if there is a change of
scope, the changing baseline will simply be denoted as “baseline”. If the work package has
used 50% of its time planned in the baseline the work package has earned 50% of the
baseline10. This means that as long as the plan is updated and the baseline is the latest
estimated cost and duration for each work package, the Earned Value will be automatically
generated based on the reported hours.
ActualWork
Earned Value = * Baseline10
Actual Work + Estimated Remaining Work
Formula 9
If the “Baseline 10” is budgeted as 500 hours, the actual work is 300 hours and the updated
baseline is set to 600 hours, then the Earned Value will be 250 hours. The actual work
represents 50% of the latest estimate of that specific work package, this means that the project
earns 50% of the original baseline; the “Baseline 10”.
83
Yngve Henriks
50
Earned Value Management at
Vestberg, Rasmus Saab Naval Systems Division Sweden
4.1.2.3 Scheduling
The planning group has the responsibility to schedule projects tasks. As the base of the
scheduling the planning group uses Microsoft Project Server, where all task are gathered and
connected with interdependencies. This Microsoft Project schedule contains for example
information about start and finish dates, budgeted costs and apportioned resources. This data,
as the core of Earned Value, is exported to another system handling the Earned Value data
and metrics.
Since the Naval Systems Division project organization is put together the way it is, one of the
major tasks for the planning group is to manage resources amongst the different projects. This
means keeping track on the covering of staff. Making sure each project gets its required staff
and keeping individuals from being over scheduled. The constant updating of the project
plans is necessary to schedule the resources associated with each work package. This
information is not provided when using performance indexes.85
84
Michael Trochez
85
Yngve Henriks
86
Björn Lindberg
51
Vestberg, Rasmus Chapter 4 – Empirical Study
The Managers at Naval Systems Division believe that using the CPI and SPI when predicting
the projects outcome is not justifying. A late start will have to large impact on the Estimate at
Completion when using the traditional Earned Value Management forecasting methods. The
experience and subjective judgment from the Project Leader is a much more reliable way of
predicting the projects future.87
87
Yngve Henriks
52
Earned Value Management at
Vestberg, Rasmus Saab Naval Systems Division Sweden
In Figure 9 each line represents one work package. The x-axis represents the timeline in
months based on the true dates for the studied work packages. The y-axis represents the
Forecasting Accuracy in percent of the entire work package. The dots on each curve
represents the time when each work package finished. The finish date is set when the Earned
Value has reached 100%. Some of the work packages have smaller changes after the finish
date, which is a consequence from the use of Level of Effort (see 6.1.3).
400,00%
300,00%
Forecast relative Actual at Completion
200,00%
100,00%
0,00%
6
jul-06
06
7
6
7
06
07
apr-0
jun-0
aug-0
jan-0
feb-0
apr-0
jun-0
sep-0
okt-0
dec-0
mar-0
nov-
maj-
maj-
-100,00%
-200,00%
Date
Figure 9 Difference in Forecast and Actual at Completion relative entire work package, true finish
dates
53
Vestberg, Rasmus Chapter 4 – Empirical Study
In Figure 10 the work packages has been horizontally displaced so that all work packages end
in June 2007. This displacement is made to make the analysing of forecasting accuracy trends
easier. As seen in Figure 10 the different curves start at different points. Those points are not
the true start date of the work packages; it is only the start date of the available historical data.
400%
300%
Forecast relative Actual at Completion
200%
100%
0%
6
jul-06
7
6
07
7
06
07
apr-0
jun-0
aug-0
nov-0
jan-0
feb-0
apr-0
jun-0
sep-0
okt-0
dec-0
maj-
mar-
maj-
-100%
-200%
Date
Figure 10 Difference in Forecast and Actual at Completion relative entire work package, finish
dates displaced to jun-07
54
Earned Value Management at
Vestberg, Rasmus Saab Naval Systems Division Sweden
Figure 11 is a more detailed through zoom example of Figure 10 to further increase the clarity
of the graphs. In Figure 11 two broken lines have also been inserted at ±10%. Before the
study took place the principal stated that when the forecasting is within 10% we can say that
the knowledge of the outcome is acceptable.
30%
20%
Forecast relative Actual at Completion
10%
0%
06
6
06
7
6
7
06
07
apr-0
jul-0
apr-0
jun-0
sep-0
jan-0
feb-0
jun-0
okt-0
mar-0
aug-
n o v-
dec-
maj-
maj-
-10%
-20%
-30%
Date
Figure 11 Difference in Forecast and Actual at Completion relative entire work package, finish
dates displaced to jun-07, zoomed in
Formula 12
55
Vestberg, Rasmus Chapter 4 – Empirical Study
In Figure 12 each line represents one work package. The x-axis represents the timeline in
months based on the true dates for the studied work packages. The y-axis represents the
Forecasting Accuracy in percent of the real remaining work. The work packages have been
horizontal displaced making them all end in June 2007.
1000%
800%
Forecast relative Actual at Completion
600%
400%
200%
0%
6
06
06
06
7
6
7
06
07
apr-0
jun-0
jul-0
jan-0
feb-0
apr-0
jun-0
okt-0
dec-0
mar-0
aug-
nov-
sep-
maj-
maj-
-200%
Date
Figure 12 Difference in Forecast and Actual at Completion relative remaining work, finish dates
displaced to jun-07
56
Earned Value Management at
Vestberg, Rasmus Saab Naval Systems Division Sweden
Figure 13 is a more detailed through zoom example of Figure 12 with the broken lines
inserted to highlight the ±10% area.
100%
80%
60%
Forecast relative Actual at Completion
40%
20%
0%
6
6
jul-06
7
6
07
06
07
apr-0
aug-0
apr-0
jun-0
sep-0
nov-0
dec-0
jan-0
feb-0
jun-0
okt-0
mar-
maj-
maj-
-20%
-40%
-60%
-80%
-100%
Date
Figure 13 Difference in Forecast and Actual at Completion relative remaining work, finish dates
displaced to jun-07, zoomed in
Provided those data we can compare the accuracy of the subjective method used at NSDS and
the mathematical CPI method.
57
Vestberg, Rasmus Chapter 4 – Empirical Study
10%
Forecast using CPI
Average: 3%
Forecast relative Actual at Completion
8%
6%
4%
Forecast using NSDS method
Average: 2%
2%
0%
10
11
12
13
14
15
1
-2%
Date
Figure 14 Work package 1, Forecast using CPI vs. Forecast using NSDS method, relative entire
work package
58
Earned Value Management at
Vestberg, Rasmus Saab Naval Systems Division Sweden
40%
30%
20%
Forecast relative Actual at Completion
10%
Forecast using NSDS method
Average: 9%
0%
1
10
11
12
13
14
15
-10%
-20%
-30%
-40%
Forecast using CPI
Average: 19%
-50%
-60%
Date
Figure 15 Work package 2, Forecast using CPI vs. Forecast using NSDS method, relative entire
work package
60%
40%
Forecast relative Actual at Completion
20%
10
11
12
13
14
15
-20%
-40%
-60%
-80%
Date
Figure 16 Work package 3, Forecast using CPI vs. Forecast using NSDS method, relative entire
work package
59
Vestberg, Rasmus Chapter 4 – Empirical Study
10%
0%
10
11
12
13
14
15
1
9
Forecast relative Actual at Completion
-10%
-20%
-40%
-50%
Forecast using NSDS method
Average: 16%
-60%
Date
Figure 17 Work package 4, Forecast using CPI vs. Forecast using NSDS method, relative entire
work package
90%
Forecast using NSDS method
Average: 75%
80%
70%
Forecast using CPI
Forecast relative Actual at Completion
Average: 61%
60%
50%
40%
30%
20%
10%
0%
1
10
11
12
13
14
15
Date
Figure 18 Work package 5, Forecast using CPI vs. Forecast using NSDS method, relative entire
work package
60
Earned Value Management at
Vestberg, Rasmus Saab Naval Systems Division Sweden
80%
Forecast using NSDS method
Average: 12%
70%
60%
Forecast relative Actual at Completion
50%
40%
30%
20%
Forecast using CPI
Average: 5%
10%
0%
1
10
11
12
13
14
15
-10%
-20%
Date
Figure 19 Work package 6, Forecast using CPI vs. Forecast using NSDS method, relative entire
work package
100%
80%
Forecast relative Actual at Completion
60%
40%
20%
0%
1
10
11
12
13
14
15
-20%
Figure 20 Work package 7, Forecast using CPI vs. Forecast using NSDS method, relative entire
work package
61
Vestberg, Rasmus Chapter 4 – Empirical Study
40%
30%
20%
Forecast relative Actual at Completion
10%
0%
1
10
11
12
13
14
15
-10%
-20%
-30%
Forecast using NSDS method
Average: 22%
-40%
-50%
Forecast using CPI
Average: 16%
-60%
Date
Figure 21 Work package 8, Forecast using CPI vs. Forecast using NSDS method, relative entire
work package
For the eight work packages represented in the graphs above, the total average inaccuracy is
22% respectively 21% for the subjective NSDS’s method and the CPI method.
62
Earned Value Management at
Vestberg, Rasmus Saab Naval Systems Division Sweden
1000%
800%
Forecast relative Actual at Completion
600%
400%
200%
Forecast using CPI
Average: 170%
0%
1
10
11
12
13
14
15
Forecast using NSDS method
Average: 247%
-200%
Date
Figure 22 Work package 1, Forecast using CPI vs. Forecast using NSDS method, relative remaining
work
80%
60%
40%
Forecast relative Actual at Completion
20%
Forecast using NSDS method
Average: 30%
0%
1
10
11
12
13
14
15
-20%
-40%
-80%
-100%
Date
Figure 23 Work package 2, Forecast using CPI vs. Forecast using NSDS method, relative remaining
work
63
Vestberg, Rasmus Chapter 4 – Empirical Study
300%
250%
200%
Forecast relative Actual at Completion
150%
100%
50%
Forecast using NSDS method
Average: 67%
0%
1
10
11
12
13
14
15
-50%
Forecast using CPI
Average: 62%
-100%
Date
Figure 24 Work package 3, Forecast using CPI vs. Forecast using NSDS method, relative remaining
work
40%
20%
Forecast relative Actual at Completion
0%
1
10
11
12
13
14
15
-20%
-60%
-100%
Date
Figure 25 Work package 4, Forecast using CPI vs. Forecast using NSDS method, relative remaining
work
64
Earned Value Management at
Vestberg, Rasmus Saab Naval Systems Division Sweden
600%
500%
Forecast relative Actual at Completion
400%
300%
200%
Forecast using NSDS method
Average: 236%
Forecast using CPI
Average: 156%
100%
0%
1
10
11
12
13
14
15
Date
Figure 26 Work package 5, Forecast using CPI vs. Forecast using NSDS method, relative remaining
work
400%
350%
Forecast using NSDS method
Average: 102%
300%
Forecast relative Actual at Completion
250%
200%
150%
100%
Forecast using CPI
Average: 59%
50%
0%
1
10
11
12
13
14
15
-50%
-100%
Date
Figure 27 Work package 6, Forecast using CPI vs. Forecast using NSDS method, relative remaining
work
65
Vestberg, Rasmus Chapter 4 – Empirical Study
250%
200%
Forecast relative Actual at Completion
150%
Forecast using NSDS method
Average: 73%
100%
50%
0%
1
10
11
12
13
14
15
-50%
Forecast using CPI
Average: 37%
-100%
Date
Figure 28 Work package 7, Forecast using CPI vs. Forecast using NSDS method, relative remaining
work
150%
100%
Forecast relative Actual at Completion
50%
0%
1
10
11
12
13
14
15
-50%
Forecast using NSDS method
Average: 42%
-100%
Date
Figure 29 Work package 8, Forecast using CPI vs. Forecast using NSDS method, relative remaining
work
66
Earned Value Management at
Vestberg, Rasmus Saab Naval Systems Division Sweden
For the eight work packages represented in the graphs above, the total average inaccuracy is
102% respectively 72% for the subjective NSDS’s method and the CPI method.
67
Vestberg, Rasmus Chapter 5 – Analysis
5 Analysis
This chapter contains the analyze of the result of the empirical study and the frame of
reference concerning Naval Systems Division. Deeper discussion about the findings will be
done in chapter 6 Discussion.
In the empirical study there have been two different approaches of how the forecasting
accuracy should be measured. Either one measures the accuracy relative the entire work
(Figure 9-Figure 11) or relative the actual remaining work (Figure 12 and Figure 13).
This way of measuring will always approach perfect prediction. The remaining work and
therefore also the prediction offset, approaches zero during the progress of the work package.
Since the entire work eventually will be much larger than the prediction offset the relative
offset will approach zero.
This approach could be looked upon as the measure of how well one could predict the entire
work package at different points.
68
Earned Value Management at
Vestberg, Rasmus Saab Naval Systems Division Sweden
Figure 9 displays the work packages in an authentic timeline. This is a little bit roundabout to
analyze. Therefore we look at Figure 10 and Figure 11 instead. Those display the same data
only displaced in time to ease analyzing. Figure 11 is a more detailed through zoom example
of Figure 10.
If starting to look at Figure 10 one can see that one year before completion, seven out of
eleven work packages are close to or more than 100% wrong. This means that the estimation
is less than half or more than twice the actual. At some point between seven and nine months
before completion the curves starts to close up on the zero line although they are not within
the 10% limit. Worth noticing is that it is more common with overestimates than
underestimates.
It is most likely that the close up depends on the relative entire work instead of the better
knowledge. Since the curves relative remaining work do not show the same trend at the same
point of time (see 5.1.2). It is more likely that the trend shows that seven to nine months
before completion there is not much work left on the work packages closing up on the final
outcome, instead of showing that the predictions are more valid within this time period.
If looking at the zoomed in Figure 11 we can see that more than a third of the work packages
do not enter within the 10% limit until the work is completed. For the rest of the work
packages they are within the 10% area between 3.5 and 9 months before completion with a
median of 5 months. The work packages entering within the limit at completion is probably
subject for trick based on individuals wish to always under run.
In the beginning of the work package this approach will give almost the same results as
measuring against the entire work package. This is because the remaining work is close to the
69
Vestberg, Rasmus Chapter 5 – Analysis
entire work. When work precedes the difference between the entire work and the remaining
work will increase, consequently the methods will give different results.
One should be aware of the possibility of extreme data in the very end of the project or when
there is very small amount of work left. If the remaining work is very small, insignificant
forecasting errors could give large turns on the relative value. Because of this one might not
pay to much attention to the figures provided by this method in the very last phase of the
projects.
This approach could be looked upon as the measure of how well one could predict the
remaining work at different times.
Figure 12 and Figure 13 displays the work packages relative the remaining actual work. The
figures are displaced in time to ease analyzing and Figure 13 is a detailed through zoom
example of Figure 12.
If looking at Figure 12 we can see that some work packages are very poor forecasted. At
worst one is more than eight times overestimated. One can also see the phenomena in the last
two months mentioned above.
If ignoring the last two months and looking at Figure 13 we would hopefully be able to
distinguish some trends suggesting the time frame for planning at work package level.
Unfortunately it is impossible to tell from the curves that the forecasting is getting better in a
certain time frame. In fact all work packages are only occasionally inside the 10% which
might as well be luck as good predictions.
5.2 PMT
One of the purposes of studying literature about Performance Measurement Techniques was
to provide suggestions of improvement within this area at Naval Systems Division.
Combining the information from the literature and the situation at NSDS provides the
fundament for later given suggestions. Some of those suggestions can be supported by some
comparing made in the empirical study.
70
Earned Value Management at
Vestberg, Rasmus Saab Naval Systems Division Sweden
The Gold Practices suggest that one should use Fixed Formula with the value 0% and 100%,
earning 100% when the work package is finished. This method is in practice the same thing as
Weighted Milestones. In Weighted milestones the work packages are a little bit bigger and
one chooses to split the work into measurable parts with the different milestones. In the Fixed
Formula method one splits the work at the milestones into separate small work packages. In
the Gold Practices there is strongly recommend not to use the subjective percent complete
method.
The use of Level of Effort is not recommended based on the non real value quality. If the
Level of Effort exceeds 5-10% of the work packages one measures the passage of time instead
of true performance. At NSDS the average of all work measured using LOE is 19.7%. In the
worst project the amount of LOE is 46.8%, that is almost half the project. Those figures truly
indicate that one is not using EVM at its best capacity at Naval Systems Division.
71
Vestberg, Rasmus Chapter 5 – Analysis
One of the great advantages with Earned Value is the easy forecasting provided by the
performance indexes. If NSDS changes the Performance Measurement Technique they will
no longer have the forecasting provided “for free”. They could of course continue with the
subjective judgments to predict to future work packages, this is in literature stated as probably
the best forecasting method. However, this method takes a lot of time (see 3.2.2.5). Instead of
sticking to the old forecasting one could start using the simple index based method
recommended by PMI. The method mentioned as the “most likely” is the Cumulative CPI
method (see 3.2.2.5.2). Therefore this method has been compared with NSDS forecasting
method in Figure 15 to Figure 29. Figure 15 to Figure 21 shows the forecasting error relative
entire work package while Figure 22 to Figure 29 shows the error relative remaining work.
The end points for all the curves are set to the time when all the work packages have reported
100% Earned Value. If first looking at the figures displaying relative entire work, we can see
the following things: The endpoint for the NSDS method does not always equal zero. This is
because the Earned Value has been reported as 100% although the estimates still indicating
that there is some work left. The reason for this is the problem behind Level of Effort.
Although the Level of Effort package has been displaced in time and the manager estimate the
work to be not finished, the value will be earned according to the original baseline. This is
why the NSDS method does not always equal zero in the end, as it is supposed to do with the
relative entire work approach. The CPI method always equals the actual at completion when
100% value has been earned. Besides this there is no larger difference in the precision of the
two methods. The average error for the CPI respectively the NSDS method is 21% and 22%,
which shows an insignificant difference.
If looking at the figures showing the errors relative the remaining work the differences
becomes more evident. The NSDS method is no longer expected to end at a zero error (see
5.1.2) but the CPI method still does. This makes the CPI method more stabile and the average
error for the CPI respectively the NSDS method is 72% and 102%. The CPI method is then
proven to be more valid in this case.
Important to notice is that this is only valid for this specific Division or perhaps only this
specific case. One also has to keep in mind that the CPI is based on the Earned Value. This
gives a circle relation in this case since the Earned Value is related to the forecasting. The
72
Earned Value Management at
Vestberg, Rasmus Saab Naval Systems Division Sweden
Earned Value has already been condemned why also the CPI method will be flawed.
However, if the Earned Value were more valid the CPI method would also provide better
accuracy.
73
Vestberg, Rasmus Chapter 6 – Discussion
6 Discussion
This chapter will discuss the findings made in the Frame of Reference and the Empirical
study. In this chapter I will also present my own ideas and conclusions based on the earlier
presented information and data.
6.1 PMT
This section will discuss findings concerning Performance Measurement Techniques.
However, on second thought, one might ask oneself, is this really earned value? Is this not in
fact only the traditional project follow-up dressed in earned value? If we do not measure the
value but instead calculate the value based on work performed we can not really tell anything
about the status of that particular work package. The only thing this will give us is the planned
value updated based on the projects performance so far. This is in fact the estimated budgeted
cost for work scheduled. The usage of cost performance index in this example does not help
as much as one might think. Since the cost performance index originate from the previous
earned value and the previous actual cost we will have a problem since the earned value was
not really earned value. Only the fully completed work packages will help with providing a
correct cost
74
Earned Value Management at
Vestberg, Rasmus Saab Naval Systems Division Sweden
The way of measuring the Earned Value at Naval Systems Division is in reality the same as
Percent Complete Estimates. Instead of estimating how much of the work already been done,
one estimates the amount of work still remaining. This gives the exact same result and is only
different in the approach of using the Performance Measurement Technique.
In both approaches it is possible to “lose” already earned value; however, I believe that it
would be greater probability for this to happen with the Naval Systems Divisions approach. It
would be contradictory for a manager to state one week that we have accomplished 40% of
the planned work and the next week decrease it to 35%. In the Naval Systems Divisions
approach, it is not very contradictory to state that one has realized that there is more work left
than what was expected the week earlier, although it has the exact same affect on the Earned
Value.
Is it good that the measurement technique makes it possible to lose Earned Value? On the one
hand it must be a good thing. Since the Earned Value obviously was not achieved one should
not be able to profit what one does not have. On the other hand the reports will be uncertain if
the Earned Value can fluctuate. Also, it must be annoying for the personnel to work in a
project where the progress is not certain. Further, if the value really has been achieved, then it
might be tragic that one can loose it because of complications in the remaining work. The
already earned value might in fact be valid and should not change at all.
75
Vestberg, Rasmus Chapter 6 – Discussion
The measurement technique used at NSDS is very vulnerable to old plans. If the plan is not
running updated the method will equal traditional project management without EVM. In
traditional Percent Complete estimates bad updates will result in no Earned Value, why the
incentive to update the plans are probably higher in this method than in NSDS’s method. The
way of measuring the Earned Value has its advantages concerning scheduling of resources,
but I still do not think it is true Earned Value. Also, if the forecast is as bad as the Empirical
Study shows, then it is pointless to use it for resource allocations.
These circumstances emphasize the importance of having a measurement technique that one
can truly trust. And I believe, based on the flaws with the different Performance Measurement
Techniques, that the only ones with true Earned Value is Weighted Milestones and Fixed
Formula. If the work is done and can be measured, then the value is earned.
The simple solution to this is to change the “Baseline 10” every time the LOE work package
is changed. Since LOE always should perform well this is the only solution if one wants to
use this measurement technique.
In the curves in Figure 10 some effects of NSDS’s wide use of Level of Effort is evident. Not
even when the work packages have earned 100% of its value the forecast is correct. This is
simply caused by the corruption made by Level of Effort measurement.
The average project at NSDS measures 19.7% of the work with LOE. The LOE work
packages are made up by overhead costs, such as support, quality, management, planning etc.
76
Earned Value Management at
Vestberg, Rasmus Saab Naval Systems Division Sweden
To reduce the amount of level of effort to be able to use EVM for real one could separate
those work packages and handle them as business overhead cost. This might not be the best
solution since different project might use those functions differently. Another way of dealing
with it is to use Apportioned Relationship or Percent Complete. The Percent Complete is not
recommended in ordinary cases, but it is still much more preferable than LOE. The
apportioned relationship is well suited as long as one could find related work packages to
connect the measurement to.
Note! Instead of only changing PMT for today’s LOE tasks the amount of LOE might work as
a warning signal. Is it really healthy to use 46.8% (the worst case at NSDS) of the time in a
project to management and support? Possibly the entire project resource allocation needs
revisiting.
6.2.1 Forecasting
The forecasting at Naval Systems Division is very simple thanks to the way of using Earned
Value calculation. In fact, forecasting using a revised plan is mentioned to be the most
reliable forecasting method of them all (3.2.2.5). However, this means that the Naval Systems
Division is using the forecasting method used in traditional project management instead of the
easy mathematical method provided by Earned Value Management. Since this feature is one
of the greatest advantages with Earned Value Management one loses some of the merits in
EVM by using this approach.
77
Vestberg, Rasmus Chapter 6 – Discussion
It might be that the estimate from a Project Leader or a Program Project Leader is more valid
than the estimate made using performance indexes. However, as earlier spoken, the way of
measuring Earned Value at Naval Systems Division is not real Earned Value. The
Performance Measurement Technique will need to be revised. If the constant updating of the
plan is no longer needed to generate the Earned Value this work would only be done based on
forecasting needs. That means we are back in the choice between the time-consuming and
accurate classic forecasting method and the almost as reliable but very easy EVM forecasting
method (more reliable in our case). Instead of as today, when one gets the classic forecasting
method on top of the not fully implemented Earned Value Management System.
In the empirical study the forecasting relative the entire work package is far better than the
one relative the remaining work. Which one should we then pay attention to? It would be the
same thing to predict the entire work package as predicting the remaining of the work
package. However I believe that the mind can be tricked to do different predictions based on
the way the question is put. No matter if the results would be different or not the fact remains;
at Naval Systems Division the question is always “How much work remains?” This means
that there is no excuse for not looking at the predictions based on the remaining work. Hence
we can conclude that the predictions of today are very poor.
If we use this approach to determine the time frame for conversion of planning packages, it is
important that the remaining of a work package can be equalled with a smaller work package.
One could argue that the remaining should not be equalled with a smaller work package, since
there could be things that one states as finished although their not. That would mean that there
is more work left to do than expected. There are two obvious counter-arguments for this
statement. First of all, if we are not certain on the Earned Value achieved we can seriously
question the entire method. Second, the same thing could be valid for smaller work packages,
the inputs from earlier work packages are not correct and there will be more work than
expected.
The reason for using NSDS forecasting method is based on the necessity to have updated
plans in order to make correct resource allocation. One might ask one self this question; is it
likely that the Project Leader or the Program Project Leader will perform better in updating
the far term plans than they did with the nearby work packages? If not, the index of how the
78
Earned Value Management at
Vestberg, Rasmus Saab Naval Systems Division Sweden
nearby work packages fall out might be at least as good as the subjective judgment. This line
of argument must also support the idea that it is possible to do long term resource planning
based on performance index. If the index indicates that one needs 1.25 hours for every
planned hour then that would be enough information for the resource planning. At least as
long as we agree in that the performance indexes are good enough to replace the updated
plans concerning estimates and long term planning.
The big difference in the Empirical Study between the forecast and the Actual at Completion
in some work packages can be explained in several ways. It could be that the person in charge
has very bad control over the entire work package. More likely is that some individuals prefer
to purposely overestimate the remaining work to be able to under run in the end. For them it is
more important to look good in the end than to have a realistic plan during the work. This is
confirmed likely based on the curves in Figure 11. All the work packages that first at
completion have a realistic picture are subject for under runs. Large overestimates are a larger
problem than one might think. This leads to higher prize to the customer and perhaps loss of
business.
It is interesting to see that the easy CPI method provides better accuracy than the strenuous
method used at NSDS today. It should also be mentioned at this point that the figures used to
generate the Cumulative CPI forecast is not true Earned Value as stated earlier. The fact that
the subjective judgment provides insecure figures strengthens the theory that there is not real
Earned Value measured when not using measurable defined work.
The baseline for the entire project (the core of EV) must be set in the beginning, although the
later part of the project is only planned briefly. As said in the Earned Value Project
79
Vestberg, Rasmus Chapter 6 – Discussion
Management88 the work package is what gives the baseline. How can one then determine the
later baseline when one has not done the breakdown into work packages yet? The answer is at
first sight pretty simple, one lets the planning packages generate the baseline! However, at
second thought one realizes this way of solving the problem has a flaw. The first part of the
baseline is composed by the work packages in detailed plan, but the second part originates in
les detailed plans and assumptions. Can one then really use the CPI and SPI that has been
calculated based on one type of figures and then apply it to a different kind of figures?
According to DoD and their massive research this seams to work.
To solve this problem I would suggest NSDS to do planning at work package level for the
entire project from the beginning. This is in somewhat controversial but can be supported by
some basic arguments.
First of all the empirical study does not show any signs indicating that people at NSDS do
better plans concerning the near future than far term work packages. Approximately one third
of the studied work packages are probably flawed by deliberate overestimates. However, the
rest of the studied work packages do not show any better trends.
Breakdown of a planning package tends to consist of adjusting the work packages to sum up
the original planning package. This means that the later planned work packages are of the
same granulate as the respective planning package.
Planning on a higher level is likely to provide overestimates because of the safety margin put
in the plan.
Most of the work in the project at NSDS is not research and development why one should
have pretty good knowledge about what to do. If you know how to do it, why not plan it. If
one chooses to do detailed plan one might not need to schedule exactly when it should be
done. But what and at what cost should not be a problem. The difference in the EVM-curves
will then be only according time and not money. That is, the CPI is still ok but the SPI might
show bad figures.
88
Fleming and Koppelman (2005)
80
Earned Value Management at
Vestberg, Rasmus Saab Naval Systems Division Sweden
If one anyway chooses to do rolling planning I believe it might be a good idea to be a bit more
flexible about the baseline. The beginning of the baseline is made up by work packages. The
beginning of the baseline is what makes the CPI and SPI which is why those also are based on
figures from work package level. If the later part of the baseline is made up by planning
packages one would combine data from different origin. This is why I suggest that one should
not be too afraid of changing baseline when planning packages is broken down.
6.3 Miscellaneous
This part will discuss findings outside the original problem.
SV(t) can if wanted be recalculated to $ again. It might sound a little odd to first replace the
measurement based on money with a measurement based on time, and then convert the time
into money again. Still it will not be the same thing. The SV($) is the vertical difference
between baseline and performance while the SV(t) is the horizontal difference. Converting the
SV(t) into money will not change this difference between the two approaches, only display
the variance in another unit. I would not recommend this conversion since it only complicates
81
Vestberg, Rasmus Chapter 6 – Discussion
a well working easy technique. The reason for doing this anyway would be if one must
display the figures in the same unit.
The more intuitive approach of the Schedule Variance combined with the less defective result
compared to the SV($) makes this method an obvious choice. Changing into Earned
Scheduling would not mean any revision of the plans or anything. The changes do only
comprise the way of calculating the variance when computing the Earned Value figures.
Unfortunately the system used at NSDS to manage Earned Value does not support Earned
Scheduling. However, all the data needed to calculate Earned Schedule is available in the
system and the only thing needed is the implementation of a new formula. Based on the
insignificant change in the system (probably only some few lines of code) and the size of
NSDS as a customer to the system provider, would make it possible to do this implementation
instantly.
Using several milestones in each work package makes it easier to really determine the status
of the project, in the same time they serve the purpose of engaging the staff. Having frequent
milestone will visualize the progress of the work.
82
Earned Value Management at
Vestberg, Rasmus Saab Naval Systems Division Sweden
83
Vestberg, Rasmus Chapter 7 – Conclusions
7 Conclusions
This chapter will in condensed form present the conclusions and suggestions for adaptations
at NSDS. The Empirical study is because of the quantity not statistical in any way. However,
the suggestions would be valid anyway based only on the Frame of Reference. The small
quantity of research does nevertheless support all conclusions.
7.2 PMT
The only true Earned Value is the objective measurable value. Therefore the Measurement
Technique to be used must be Weighted Milestones or Fixed Formula. This should be used
with very small work packages or larger work packages divided into several measurable
milestones.
All Level of Effort task must be separated from the project baseline and measured separately.
In cases where separation is not possible the work most preferably should be measured with
Apportioned Relationship and secondary Percent Complete. This will provide a measurement
on those tasks which is not present at all today.
84
Earned Value Management at
Vestberg, Rasmus Saab Naval Systems Division Sweden
7.3 Forecasting
The forecasting should be made as described in 3.2.2.5 with some of the mathematical
approaches preferably using all three together as described. The method is easy and provides
data good enough (and some times even better than today). Changing this routine is easy and
could be made instantly without any revision of the plans.
7.4 Miscellaneous
Today I would say that NSDS is not using Earned Value at all. This is mainly based on the
very high level of LOE measurement. Also the shocking reliability in the subjective Earned
Value measurement supports that point of view. One would probably provide more accurate
Earned Value by guessing. Even if other projects than the measured would be more accurate I
still would not say that NSDS is measuring true Earned Value. Using a subjective method will
in my mind never provide assured information. Thinking about it, measuring the performance
with subjective method is what has always been done in traditional project management. The
only difference is that in this case it is taken to a lower level and displayed in graphs.
By adapting this recommendations there will most likely be several benefits. The work with
Earned Value Management will be less. The data provided will be more reliable and therefore
useful. Valid data will most likely yield a much wider acceptance for the system. Detailed
measurable milestones will make it more difficult to strongly overestimate the work. If there
still are large overestimates they will be noticed early when a couple of milestones has been
achieved.
The cause of the extreme portion of LOE should be further investigated. Not necessarily, but
it could indicate bad use of resources. If half of the LOE tasks are misuse of resources (the
amount of LOE is twice the maximum recommended) there is 10% of the total project budget
to save.
Doing the changes in forecasting would mean that one could no longer use the same data as
above to determine the time frame for planning package breakdown. Instead, if adopting all
purposed changes, one could measure the accuracy against the original plan, since all work
85
Vestberg, Rasmus Chapter 7 – Conclusions
packages is recommended to be planned right away at project start. If there is a special time
frame telling how far in advance it is possible to plan, this will be obvious.
Measuring the accuracy in forecasting should be made running in the same way as in this
report. This provides a good indication of how well the used techniques work, both for
planning and measuring of Earned Value.
86
Earned Value Management at
Vestberg, Rasmus Saab Naval Systems Division Sweden
87
Vestberg, Rasmus Chapter 8 – References
8 References
Books
Cassell, C. and G. Symon (ed.) (2004) Essential Guide to Qualitative Methods in
Organizational Research. London. SAGE Publications Ltd
Fleming Q. and Koppelman J. (2005), Earned Value Project Management, third edition,
Pennsylvania, Project Management Institute, Inc.
(2005), A Guide to the Project Management Body of Knowledge: PMBOK guide. – 3rd ed.,
Pennsylvania, Project Management Institute, Inc.
Articles
Abba Wayne (2002), June 2002, Emerging Ideas, Relating EVM to “Real” Schedules,
unpublished presentation
National Contract Management Journal (1993), Spring 1993, Cost Performance Index
Stability, Christensen, D. S., Heise, S.
Acquisition Review Quarterly (1998), Fall 1998, The Cost and Benefits of the Earned Value
Management Process, Christensen David S. Ph.D.
Acquisition Review Quarterly (1999), Summer 1999, Using The Earned Value Cost
Management Report to Evaluate the Contractor’s Estimate at Completion, Christensen David
S, Ph.D.
88
Earned Value Management at
Vestberg, Rasmus Saab Naval Systems Division Sweden
The Measurable News (2007), Winter 2006-2007, Why EVM Is Not Good for Schedule
Performance Analyses, Corovic Radenko
The Measurable News (2003), Summer 2003, A Breakthrough Extension to Earned Value
Theory?, Henderson Kym
The Measurable News (2004), Spring 2004, Further Developments in Earned Schedule,
Henderson Kym
The Measurable News (2005), Spring 2005, Earned Schedule in Action, Henderson Kym
The Measurable News (2007), Summer 2007, Earned Value Business Solutions:
Implementation, Lessons Learned and Best Practices, Kondur Mohan
The Measurable News (2003), Mars 2003, Schedule Is Different, Lipke Walt
Interviews
Person A, Manager Naval Operations Project Office
Henriks Yngve, Manager Project Management Office, Naval Systems Division Sweden
89
Vestberg, Rasmus Chapter 8 – References
Trochez Michael, Head for the Planning and Control Function, Naval Systems Division
Sweden
Internet
http://saabnet.saabgroup.com/SaabSystems/About/index_sv.htm?NavigationType=SiteNaviga
tionDataSource, Accessed June 2007
Others
ANSI/EIA-748-A-1998, Arlington, Electronic Industries Alliance
90
Earned Value Management at
Vestberg, Rasmus Saab Naval Systems Division Sweden
91
Vestberg, Rasmus Chapter 9 – Appendices
9 Appendices
9.1 Appendix 1
92