Early recognition of factors that may result in project problems is especially important for identifying

projects that need special attention. It is also important to have a common structure for consistent
communication of project health across the IT development organization.

The focus of the following material is the:

 selection of PCSFs for IT projects

 identification and definition of KPIs and metrics to determine if the PCSFs are being
achieved across the project lifecycle
 collection and retention of the metrics
 compilation and presentation of the results of PCSF measurement and assessment in
dashboards

Project-critical success factors: PCSFs are conditions required for a project’s end results to be
measured and are an essential element in assessing a project’s success. PCSFs can be subjective
or objective. Examples of subjective PCSFs are those used by the United States Government
Accountability Office to assess federal IT programs:

 Program officials were actively engaged with stakeholders.

 Program staff had the necessary knowledge and skills.
 Senior department and agency executives supported the programs.
 End users and stakeholders were involved in the development of requirements.
 End users participated in testing of system functionality prior to formal end user acceptance
testing.
 Government and contractor staffs were stable and consistent.
 Program staff prioritized requirements.
 Program officials maintained regular communication with the prime contractor.
 Programs received sufficient funding.

For PCSFs evaluated objectively, KPIs with their metrics indicate if the PCSFs are being realized.
Objective PCSFs proposed as appropriate for IT projects are:

 Financial status – The project has adequate funding and has not overrun the project
budget.
 Human resources – Project staffing is stable, vacancies are filled expeditiously and the
project is fully staffed.
 Quality controls – The product or solution meets the requirements, specifications and
performance criteria and test results are satisfactory.
 Regulatory compliance – All required reviews are accomplished satisfactorily.
 Schedule status – The interim product and solution deliveries are within the agreed time
frame and the critical path has not slipped.
 Scope delivery – The product scope is stable and the deliverables, capabilities, features,
etc. satisfy the product scope statement (or statement of work/customer expectations).

A failure to continually achieve all of the PCSFs during the project lifecycle does not necessarily
predict the failure of the project. However, tracking the PCSFs by applicable KPIs enables the
project manager and team to recognize problem areas in project performance in time to develop and
carry out remedial actions. As these PCSFs develop, mature or improve, so increases the likelihood
that stakeholder expectations will be met (i.e., the project becomes increasingly healthy).
Key performance indicators and metrics: Determining the accomplishment of the PCSFs for a
project requires the observation and measurement of specific indicators related to each PCSF.
These indicators, or KPIs, are best determined for a given project by the stakeholders of that project,
preferably early in the SLM planning stage (see the appendix for a comprehensive definition of KPI,
metric and related terms). For each agreed KPI metric, the project stakeholders also need to
determine:

 The frequency of measurement

 Whether it is a point-in-time (snapshot) or cumulative metric
 The severity criteria (what determines a “red,” “yellow” or “green” severity level)
 Whether the data source exists or a new source must be identified or established

It should be noted that while numerous KPIs typically influence IT projects generally, the benefit of
tracking all of them may be outweighed by the cost to do so. Therefore, each project must carefully
select and manage those KPIs most relevant to its specific situation.

The accompanying Key Performance Indicator Template can be used to define KPIs. A template for
each different KPI and metric should be created and saved in a repository. This will eliminate
duplication of effort across projects when defining KPIs and metrics. For example, if Project A
defines a KPI for “critical path slippage,” Project B might use the same KPI or use its definition as a
basis to develop a variation.

As a starting point, the table included in the template gives some examples of possible KPIs for each
of the previously defined PCSFs and a metric (measure and dimension) for each KPI, with the
targets remaining to be defined.

Collection and retention of metrics: When stakeholders agree on the KPIs to use to assess the
health of a project, the sources of the necessary metrics must be identified. The likelihood is that
procedures will already be in place to collect and retain most of the metrics. If, however, there are
required metrics that are not being collected, then new collection and retention procedures will need
to be developed and implemented.

Retention of metrics is important as a basis for producing trend line charts for project-level
dashboards. Each time metric data is collected, the data should be retained to enable its future
retrieval for display in raw form and inclusion in trend line charts.

Project critical success factor dashboards: An ultimate objective of project health assessment is
to communicate the results to all project stakeholders and others concerned with the success of the
project. This provides a level of confidence that the project is on the right track, or if not, provides a
starting point for corrective or remedial actions.

An effective way to communicate this information is by an executive-level PCSF dashboard and

project-level PCSF dashboards that “display all of the required information on a single screen,
clearly and without distraction in a manner that can be assimilated quickly.” (Stephen Few)

1. Executive-level dashboard: A suggested executive-level dashboard would show selected

projects (possibly “high visibility” projects) in the left-hand column of a matrix presentation. The
PCSFs would be arrayed horizontally across the top of the matrix with a column for each. The
PCSFs would map to the projects in the matrix with a visual indicator such as a “traffic light” in each
intersecting cell. Clicking on a project row would open (drill down to) a project-level dashboard as
described below.
It may be desirable and helpful to display other relevant columns containing information such as:

 Project start date (with a planned/actual indicator)

 Project finish date (with a planned/actual indicator)
 Project start date variance from baseline
 Project finish date variance from baseline
 Percent work complete (calculated from the project schedule)
 Percent duration complete (calculated from the project schedule)

Information presented should be based on a discussion with IT management of the questions they
wish to answer by using this dashboard.

2. Project-level dashboard: A project-level dashboard would be established and maintained for

each project featured on the executive-level dashboard. The project-level dashboards would show
the executive-level indicator row for the project, an image or icon for each PCSF and a horizontal
trend line chart showing the trend line for each PCSF. Clicking on a particular PCSF image or icon
would open (drill down to) the detailed metrics for the PCSF, including historical data (if applicable).

Recommended Action Steps

Recommendations for determining the health of IT projects with action steps suggested for
implementing an objective, metrics-based project health assessment process are offered below.
Based on the research conducted and the resulting approach previously described, it is
recommended that CIOs:

1. Increase the priority of project health assessment with respect to competing activities and
communicate the priority to all concerned participants.
2. Designate key team members and set the time frame to accomplish steps 3 through 10.
3. Adopt a set of objective metrics-based PCSFs to uniformly and consistently apply to the
project health assessments of all projects in the enterprise or organization portfolio.
4. Include the definition of, and stakeholder agreement to, KPIs for projects in the SLM
planning stage.
5. Develop and implement procedures for collection and retention of KPIs and their metrics.
6. Design, develop and implement an executive-level PCSF dashboard to apply to the “high
visibility” projects in the enterprise or organization portfolio.
7. Design, develop and implement a standard project-level PCSF dashboard to apply to all
projects in the enterprise or organization portfolio.
8. Create a repository for KPIs and their associated metrics.
9. Integrate the executive-level PCSF dashboard, the project-level PCSF dashboards and
the KPI/metrics repository.
10. Develop and implement a project health assessment process with procedures for
conducting project health assessments and populating and maintaining the dashboards.

Sources

1. Bittner, Kurt. Measuring project health. IBM Software Group. Rational Software Division.

2006.
2. Department of the Air Force. Software Technology Support Center. Guidelines for
Successful Acquisition and Management of Software-Intensive Systems: Weapons
Systems, Command and Control Systems, Management Information
Systems. Condensed Version 4.0. February 2003. Chapter 11 Assessing Project Health.
3. Few, Stephen. Why Most Dashboards Fail. Perceptual Edge. 2007.
 4. Gonzalez, Tom. Dashboard Design: Key Performance Indicators and Metrics. 2005.
5. Government Accountability Office. Information Technology.  Critical Factors Underlying
Successful Major Acquisitions (GAO-12-7). October 2011.
6. Kerzner, Harold. Project Management Metrics, KPIs, and Dashboards: A Guide to
Measuring and Monitoring Project Performance. John Wiley & Sons: Hoboken. 2011.

Appendix

Key Performance Indicators and Metrics (From: Gonzalez, Tom. Dashboard Design: Key

Performance Indicators and Metrics. BrightPoint Consulting, Inc.)

Key Performance Indicator (KPI): a metric that is tied to a target.

Example: widget sales per week of $10,000

Metric: widget sales per week

Target: $10,000

Metric: a direct numerical measure that represents a piece of business data in the relationship of

one or more dimensions.

Example: gross sales by week

 Measure: dollars (gross sales)

 Dimension: time (week)

Hierarchy: levels within a dimension

Example: gross sales by day; gross sales by week; gross sales by month

 Hierarchy: day, week, month within the time dimension

Grain: a measure associated with a specific level of hierarchy within a dimension

Example: gross sales by day

 Measure: dollars
 Hierarchy: day level
 Dimension: time

Multi-dimensional analysis: looking at a measure across more than one dimension

Example: gross sales by week for Maryland

 Measure: dollars (gross sales)

 Dimension: time (week)
 Dimension: territory (Maryland)