Early Warning Signs in Complex Projects

possible.

1

Introduction

Project assessments are carried out worldwide. They are frequently referred to as reviews, audits, and health checks. The intent of these initiatives is to evaluate the extent to which the project documents are developed in accordance with expectations and formal criteria. The purpose is to support decision making and make sure that the projects are successful. One additional important aspect is, of course, to learn from past experience.

Still, we are often surprised at how projects turn out. Complex projects often do not behave in the way we expect, and in particular effects within complex projects are often time-delayed and take time to emerge. Project assessments need to look through the complexity of a given project in its context and identify the relevant early warning signs of project problems, project failure, under-performance, or cost overrun. Experience and current literature seems to indicate that we are not very good at picking up early warning signals.

Assessments are potentially important in securing the success of projects. To be effective, project assessment should be systematic and well adapted to the purpose and context of the assessment, as well as anchored in a governance framework established by the project owner/sponsoring organization. This study is designed to strengthen the basis for implementing effective project assessments.

Increasing complexity in projects and rapid changes makes the governance of projects increasingly challenging. Complexity in this context includes external complexities such as stakeholder relationships (Pryke & Smyth, 2006) and decision-making processes, and internal complexities such as technology and interfaces to existing systems (Williams, 1999). This increasing complexity suggests there may be a wide range of reasons why future projects may fail. In addition, the causal relations between early indications or incidents and later results are seldom obvious, but are often very complex. This makes it very difficult to know what to look for when assessing complex projects. This report investigates the practice of project assessments to identify how and to what degree the right early warning signs are identified.

1.1 Previous Work by the Research Teams

Previous work on governance frameworks has revealed how governance frameworks and project assessments are very closely linked. Existing governance frameworks include guidelines and instructions for performing project assessments and defining when and how they are used. Previous work by the research team has studied three existing frameworks in the UK and Norway (Klakegg, Williams, & Magnussen, 2009). By building on this work, extending it to Australia and the private sector, a wider platform of knowledge is established.

The previous work of Williams and associates covers a broad set of factors relating specifically to how to understand complexity (Williams, Eden, Ackermann, & Tait, 1995; Williams, 2005) and learn from projects (Williams, 2008). Understanding causal chains and the reasons for disruption and delay has been a major issue in their research (Eden et al. 2005). Further, the previous work of Andersen and associates on performance measurement adds to the methodological basis (Andersen & Fagerhaug, 2001). Earlier work by Walker and associates has built a strong platform for understanding the system sides of project assessments and diffusion of new knowledge (Walker & Rowlinson 2008). Also, work by Klakegg and associates from the Concept Research Programme in the governance of public projects and the performance of project assessments has added to the current knowledge in this field (Concept 2009).

1.2 Definitions and Basic Concepts

Most the Project Management Institute's general definitions of concepts are of relevance for this report (PMI, 2008). The authors have chosen the following specific concepts and definitions for this work. We use them throughout the report, although we also address other (more or less synonymous) words and concepts in the discussions and analysis.

Table 1-1 Definitions and Basic Concepts

A Office of Government Commerce (OGC) focus program management. The meaning is similar when the word program is replaced by project. In this report, the concept of project is in focus.

1.3 Scope and Limitations

This study looks at how project assessments may be used to identify early warning signals in complex projects. This implies several aspects of scope definition for the research project and important limitations to the use of the results. The most important scope definitions are:

Perspectives: This research represents a project owner's perspective (as seen from the organization or the firm). This is the strategic perspective—the view of a permanent parent organization. The success of a project has to be discussed in the owner's business environment and is a question of whether or to what degree the owner's intentions and purposes regarding the project are met. Success in terms of meeting targets within budget and time restraints (the supplier perspective) is still important but not enough to guarantee success (Atkinson, 1999; Jergeas, 2005).

Countries/Sectors: The study includes guidelines, interviews, and cases from three countries: Australia, Norway, and the United Kingdom. These are the countries covered by the research teams involved. These are all wealthy, developed countries with a relatively well-established project tradition and culture. The results can be transferred to countries with a similar starting point in terms of economic thinking, culture, and administrative traditions. The study covers both public and private sector organizations in the countries included in the study.

Project Types/Industries: The focus is on well-established industries with traditional investment projects including the development of some form of physical or digital infrastructure. Typically, this covers building and construction, energy, oil and gas, telecommunications, and information and communication technology (ICT). Research, product development, and defense projects are deliberately left out due to budget and time constraints, but references to interviews include those individual experts with experience from such projects.

Phases/Timeline/Life cycle: Project management (PM) has traditionally focused on the execution phase. We acknowledge that the success or failure of any project may be determined well before the execution phase starts; consequently, this study focuses more on the early phases—the front-end. However, the entire timeline of a project and the complete life cycle of a project's results have to be considered when identifying early warning signals.

This study does not cover projects in developing economies where the governance framework is not developed or where a PM tradition and knowledge is not established. The study does not cover the aid and development projects of nongovernment organizations (NGOs).

The study is limited to a few categories of typical contexts (characterized by degrees of complexity) and only principal early warning signs, not detailed considerations of causality and intricate multiple influences. Classifying important signs identified in documents, interviews and case studies into context categories may give an indication of what signals are important in which contexts (for medium and high complexity projects).

1.4 Research Questions and Methodology

As mentioned in the introduction above, projects do not always turn out as they were originally intended. The obvious way to improve such a situation is to identify what may go wrong and then initiate a means to ensure that it will not happen. A wide range of analytical methods, computer tools, and project assessments have been developed in order to help avoid problems. Still, project owners and project managers seem to have limited success in picking up early warning signs. This report establishes a firm academic footing based on a literature survey, which serves to confirm that we understand the full background behind the causes of this phenomenon.

Traditionally, performance measurement in projects has been preoccupied with simple measures of time, cost, and quality. However, these factors are consequences of activities, incidents, and other conscious or unconscious actions, or even lack of actions. For the purpose of early warning in projects, there must be additional measures that can provide more relevant information. In the search for indicators that can serve as early warning signs in projects, we have turned to sources describing factors of project success and failure. Project management literature contains descriptions of so-called project success factors, or sometimes the inverse, project pitfalls. This topic has been extensively researched and is reported in Chapter 2.

In terms of factors that can be made subject to measurement and monitoring for early warning indications, several different avenues have been pursued. In IT projects, research findings indicate that people-related and process-related risks scored higher than product-related risks as dominant warning signs of IT project failure (Kappelman, McKeeman, & Zhang, 2006). Post-mortem analyses of unsuccessful IT-projects have shown that there are evident signs of symptoms well in advance of the final failure. We interpret this as an indication that improvement is indeed possible.

Syamil, Doll, and Apigian (2002) argued that behavior-related performance measures evaluating a project process are mediating variables affecting the extent to which the chosen process contributes to the overall project result. Supporting this view, Hoegl, Weinkauf, and Gemuenden (2004) found that collaborative processes during a project have predictive properties with regard to later team performance and can serve as early warning indicators. Balachandra and Raelin (1980) presented a model indicating that project success factors could be used for developing a model for early warning—an approach supported by Sanchez and Perez (2004). Investigating construction and building projects, Hanna and Gunduz (2005) found that a selection of variables from the pre-construction phase represented a project success rate of more than 70%. The model they developed could be used to define early warning signs for use in the tender phase, but did not lead to an explanation of how to conduct measurement and follow-up of the areas requiring attention. Jaafari and Jabiri (2006) and Jaafari (2007) proposed an approach to project diagnostics based on so-called project health checks, building partly on excellence models known in the quality management field and partly on maturity model principles. We will pursue these varying approaches further in Chapter 2.

In summary, many different approaches have been proposed in terms of what to measure as early warning signs. What is lacking seems partly to be empirical or practical validation of whether these approaches work, that is, whether they are truly capable of predicting emergent problems at an early stage. In addition, little work exists on how to use these measures in project organization. How does one collect the required data? Who should have access to the measurements? In which forums should they be discussed? What action should be taken? This report tries to shed some light on these areas.

The first research question is:

(i) How successful are current project assessment methods in trying to uncover early warning signs of problems?

Continuing our study of current literature, and expanding into governance frameworks and guidelines for assessments of projects we ask:

(ii) How do current project assessment methods try to uncover early warning signs of problems?

(iii) Why do methods fail to detect early warning signals?

This study of documented guidelines and instructions considers how and to what extent project assessments in the chosen countries/sectors are executed under established governance frameworks. The survey will indicate how frequently project assessments are reported in literature and public bodies such as National Audit Offices. The literature search has an international profile, whereas the study of public bodies is limited to the countries represented in this study.

Thus far, we have taken a positivist view to uncover what sort of problems there are. To look into solutions, we took a more phenomenological stance (Easterby-Smith, Thorpe, & Lowe, 1991). We considered the following three questions:

(iv) How can project assessments handle the complexity in projects and their context?

(v) Which practices seem more appropriate in which contexts?

(vi) What are the most important early warning signals to look for in the different contexts?

In order to investigate these questions from a more practical angle and to collect empirical evidence, we conducted interviews with a limited number of experts, looking at what companies and public entities do to implement project assessments, focusing on methodological choices and the effects of established practices, that is, what they were looking for. Following this up, we investigated a limited number of case projects to find out what the project assessments had identified as early warning signs and whether these had actually been confirmed in the time after assessment, that is, whether the right things were looked for

Budgetary constraints have inhibited a full examination of these three questions, so the last portion of the report is preliminary only. The findings are considered valuable for identifying strengths and weaknesses in current approaches, for providing insight into which early indications and warning signs can be expected to predict success and failure in different settings, and finally for identifying what further research will be needed to develop effective project assessment practices in specific contexts.

1.5 Research Methodology and Structure of the Report

General

The analyses suggested here are basically qualitative. No quantitative analysis or statistical approach is used as part of this project. We chose an exploratory and explanatory approach based on deeper study of a few cases over a broader survey with a large number of interviews and cases. This excludes an open survey and reduces the risk of not having access to a sufficiently large number of respondents/cases. This is the preferred approach when working with a limited budget. The three main methods used in this study are document studies, interviews, and case studies, as described in the following chapters. Figure 1.1 shows the outline of the research method and connections to the chapters in the report.

Literature Review

The literature review documented in Chapter 2 is an important starting point. We build a basis for further studies by making a comprehensive literature survey. The point is not to present all relevant contributions on the matters covered in Chapter 2, but rather to ensure we have covered the main contributions to relevant issues. For this reason, the literature is presented under a series of headings in the form of questions derived from the research questions from section 1.4. We primarily aim to capture contributions representing the latest development and best practices in the different fields covered.

Studies of Governance Frameworks and Guidelines

In all three countries represented by the research team (as well as many other countries) there has been an increasing focus on governance of projects in the last decade, resulting in new governance frameworks, quality assurance schemes, and project management systems being implemented in the public and private sectors on all levels. These frameworks normally include project assessments and guidelines to be followed by the intended users. In order to find out how frequently such assessments are really made, and what the guidelines prescribe as best practice, we look into a selection of nine governance frameworks available to the research team. These guidelines cover both the public and private sector. The primary research methodology in this part of the study involves documented studies, but these are supplemented with interviews in those cases where some questions were not answered by available documentation. This part of the study is reported in Chapter 4.

Expert Interviews

Based on the literature (Chapter 2) and public documents (Chapter 4), the reasons why we fail to pick up or respond to early warning signs have been preliminarily documented. To go beyond the academic understanding of the causes and to validate the findings, the researchers performed a number of interviews with people in relevant positions and with experience from complex projects and governance as well as project assessments. This is reported in Chapter 5. A questionnaire was prepared based on preliminary findings and used as a guideline during interviews. The interviews were semi-structured. Respondents were carefully chosen among individuals within two groups:

Individuals with experience in performing project assessments in different settings.

Individuals on the owner/sponsor side or project managers with experience in being assessed and determining whether such assessments identify the right signals and/or are followed up.

Due to budget constraints, the number of respondents was not sufficient to provide validation for all settings. Therefore, the conclusions from this study are of a preliminary nature. Based on the initial study, three categories of settings (according to complexity: simple, moderate, and complex) were defined and the respondents selected had experience in two of these categories (moderate and complex). We performed 14 interviews, 5 in Norway, 5 in the UK, and 4 in Australia. The respondents covered both the public and private sector. The responses from the interviews were analyzed by structuring them according to the questionnaire and comparing them with findings from the available literature (the reference model, see Chapter 3).

Case Studies

The literature survey, documented studies, and interviews give a broad picture of the situation. While performing interviews the researchers also looked for relevant cases that illustrate the current practice. The researchers identified and studied both good and bad examples of project assessments in the public and private sector. The point here is not whether a project is a success or failure, but whether the project assessment represents best practice or an immature approach. In total, eight cases were documented and analyzed. Four of these cases are used as examples in the report to explain how project assessments work. Some cases are presented anonymously in the analysis.

The cases have been selected to represent the same categories of settings as used in the interviews. The number of cases is limited but sufficient to unveil patterns for the type of setting they represent. The analysis includes cross-sector, country, and setting comparisons, within the limitations set for this study (cf. section 1.3). Each case study in Chapter 6 included interviews (on-site or by telephone) with individuals in key positions in the case project and also studies of project documents illustrating the effect of project assessments.

Conclusions and Further Research

In the final chapter, the findings from document studies, interviews, and case studies are combined to look for additional patterns and findings. We conclude with the research questions and point out which issues need more research. We also identify areas for improvement in the current performance of project assessments which could lead to better identification of early warning signs and therefore the ability to act upon them.

The research in the report is explorative in nature and qualitative in methodology. If one theoretical basis has to be pointed out as a main approach, it might be sense making. From all of the observed guidelines, reported practices, and experiences, we obtained new knowledge about how to see through complexity and use this as a tool to improve the possibility for successful projects from the owners’ perspective. The concluding part is also somewhat normative in that it attempts to point out effective strategies for identifying and acting upon early warning signs.

2

Literature

2.1 Introduction

As discussed in Chapter 1, we are often surprised at how projects turn out. Experience and current literature seems to indicate we are not very good at picking up the early warning signals of project problems, project failure, under-performance, or cost overrun.

The classic paper on early warning