A Proposal for a Quality Improvement Framework

for Small Organizations

Johan Redestig, pt95jre@student.hk-r.se
Erik Starck, pt95est@student.hk-r.se
University of Karlskrona/Ronneby
Department of Software Engineering and Computer Science
Ronneby, Sweden
Master thesis in software engineering
2008-09-06
Supervisor: Conny Johansson
Abstract
The goal of this thesis is to present a quality framework for small
organizations that are developing software. This framework includes a
quality manual that could be used as first steps for small software
developing organizations that wish to work in a process oriented
manner and thereby improve the quality in the organization.

To be able to do this, an empirical study is performed, in which we tries

to find the areas in which small organizations normally fail. The quality
framework focuses on these areas, thereby making the quality
framework as compact as possible. This makes it suitable as an
introduction to a process oriented way of working for small software
developing organizations.

2
Acknowledgements
The authors would like to thank all that have participated in the development of this thesis in one
way or another. Especially our supervisor Conny Johansson for his excellent remarks and support.
We would also like to thank the people that have participated in the interviews for their time and
effort.

Johan Redestig

Erik Starck

3
 Table of contents
1INTRODUCTION........................................................................................................................................6
1.1ADDRESSED PROBLEMS..................................................................................................................................6
1.2GOALS........................................................................................................................................................7
2STRUCTURE AND ORGANIZATION OF THIS PAPER......................................................................8

3QUALITY......................................................................................................................................................9
3.1DEFINING QUALITY FOR SMALL ORGANIZATIONS..............................................................................................9
3.2EXISTING QUALITY MODELS AND SMALL ORGANIZATIONS...............................................................................14
3.3COMPANY QUALITY MANUAL, CQM...........................................................................................................17
3.4CONCLUSION.............................................................................................................................................17
4METHOD....................................................................................................................................................19
4.1QUALITATIVE STUDY...................................................................................................................................19
4.2RELIABILITY..............................................................................................................................................21
4.3ANALYSIS OF DATA.....................................................................................................................................22
4.4CONCLUSION.............................................................................................................................................22
5PROBLEMS AND HYPOTHESES..........................................................................................................24
5.1VIEW OF SOFTWARE DEVELOPMENT................................................................................................................24
5.2PLANNING.................................................................................................................................................25
5.3MANAGEMENT...........................................................................................................................................26
5.4SELECTING TECHNICAL SOLUTIONS.................................................................................................................27
5.5CONFIGURATION MANAGEMENT AND VERSION CONTROL....................................................................................27
5.6TESTING...................................................................................................................................................28
5.7REQUIREMENTS..........................................................................................................................................29
5.8CONCLUSION.............................................................................................................................................30
6INTERVIEW ANALYSES.........................................................................................................................31
6.1OVERVIEW................................................................................................................................................31
6.2COMPANY I...............................................................................................................................................32
6.3COMPANY II..............................................................................................................................................36
6.4COMPANY III............................................................................................................................................41
6.5COMPANY IV............................................................................................................................................46
6.6COMPANY V..............................................................................................................................................50
7ANALYSIS CONCLUSION......................................................................................................................54
7.1THE HYPOTHESIS........................................................................................................................................54
7.2CONCLUSION.............................................................................................................................................57
8QUALITY FRAMEWORK FOR SMALL ORGANIZATIONS...........................................................59
8.1QUALITY DEFINITION..................................................................................................................................59
8.2QUALITY MODEL.......................................................................................................................................61
8.3QUALITY MANUAL ....................................................................................................................................62
8.4CONCLUSION.............................................................................................................................................67
9FUTURE WORK........................................................................................................................................68
9.1THE CQM...............................................................................................................................................68
9.2COMPARISON TO A BIG COMPANY..................................................................................................................68
9.3THE NEXT STEP..........................................................................................................................................68
10REFERENCES.........................................................................................................................................69

11DEFINITIONS..........................................................................................................................................72

4
 11.1SMALL ORGANIZATION...............................................................................................................................72
11.2QUALITY.................................................................................................................................................72
11.3ISO 9000..............................................................................................................................................72
12APPENDIX A - QUESTIONS.................................................................................................................73
12.1INITIAL...................................................................................................................................................73
12.2VIEW OF SOFTWARE..................................................................................................................................73
12.3PLANNING...............................................................................................................................................74
12.4MANAGEMENT.........................................................................................................................................74
12.5SELECTING TECHNICAL SOLUTIONS...............................................................................................................74
12.6CONFIGURATION MANAGEMENT...................................................................................................................75
12.7TESTING ................................................................................................................................................75
12.8REQUIREMENTS........................................................................................................................................75
13APPENDIX B – OVERVIEW OVER THE ANSWERS TO THE QUESTIONS..............................76
13.1QUALITY.................................................................................................................................................76
13.2PLANNING ..............................................................................................................................................76
13.3SELECTING TECHNICAL SOLUTIONS ..............................................................................................................77
13.4CONFIGURATION MANAGEMENT...................................................................................................................77
13.5TESTING.................................................................................................................................................78
13.6REQUIREMENTS .......................................................................................................................................78

5
1 Introduction
Many software-developing organizations try to enhance the quality of the products that they
produce by focusing on a process-oriented approach of working [HAMMER+]. There exist
different models of process oriented approaches that companies can be certified in, one common
model is ISO 9000. According to Bo Manelius at ISOQA are small organizations underrepresented
among the certified organizations. The reasons are the cost and the extra work that a certification
involves.

We believe that small organizations are intimidated by the size of the existing quality models, such
as ISO 9000, CMM or TickIT. The reasons for this can be that the quality models may have a bad
reputation and are perceived to be bureaucratic and time consuming. The goal of this paper is to
construct a quality framework that can be used by small organizations. This framework includes a
quality definition, a quality model and a quality manual. The manual can be used as a first step
towards a more process oriented way of working. It should be far from as large as CMM or ISO
9000, to make sure that the size does not become an intimidating factor.

To be able to do this, we must focus on the aspects of software development that can be improved
with the least amount of work and bureaucracy. We assume that there are such aspects that can be
greatly improved with little effort, for example, only one third of the programmers are aware of
the concept of configuration management [BECK+]. Another example: One manager found
software development to be like “Jumping from a cliff with closed eyes, and hoping that one lands
on something soft”. The problem that he was referring to was that of choosing what technology to
base the products on. If the chosen technology did not measure up, then the entire product had to
be rewritten. That organization did not believe that there was any time to evaluate competing
technologies. If developers are not even aware of basic control methods such as configuration
management or know how to evaluate technical solutions, then we believe that it is possible to
increase the quality in many organizations by focusing on a few carefully selected areas. To find
these areas we make an empirical study on small software developing organizations in the south of
Sweden. In this study we will try to find the areas in which small organizations normally fail, but
also the areas in which they do not fail, as this does not have to be a part of the quality manual.

We will also examine what the concept of quality means for small organizations, so we know what
such organizations want to achieve. We studied a number of employment opportunities where
various organizations were looking for quality managers. All of these organizations were really
looking for test engineers. The conclusion is that quality still is interpreted to be testing. This
paper will show that quality management is about much more than pure testing even though
testing is important.

We believe that it is possible to develop high quality software within small organizations. In fact,
there is a great potential in smaller organizations. Organizations and projects can gain a lot from
being kept small [LIBERTY], [BROOKS], [MAGUIRE]. They can adopt fast when circumstances
change, they can use the potential of their team members or employees to a greater extent, and
they lack much of the overhead work that is typical for larger organizations. Introducing
formalized processes can help the organization bring out its full potential [ZAHRAN].

The intended reader of this paper is a person that is familiar with the basic concepts of quality
management in software developing organizations. It could be someone working on a small
company that wishes to introduce quality improving processes into their organization.

1.1 Addressed problems

We aim to address the following problems, in the context of software development in small
organizations.

6
 What are the most important quality improving measures? What are the most difficult
problems in developing software? We believe that it is important that the quality improving
measures can be easily incorporated into the organization, and that they make as much
improvement to the development cycle as possible. Otherwise they will not be accepted by the
organization and, as a consequence, not used. By focusing on the most difficult problems and then
use quality improving methods to cope with them, we believe that quality can be increased. The
problem is to find the most rewarding quality improving efforts for small organizations.

How do small companies define quality? We believe that a firm definition of the term ‘quality’
can help when striving for higher quality. Therefore the problem is to construct a quality definition
that suits small organizations.

Is it possible to construct a small framework for achieving higher quality in smaller

organizations? The problem is to construct a small quality framework that can be used in small
organizations with a desire to raise their quality. This is built on the conclusions that we drew from
the previous problems.

1.2 Goals
The goals of the paper is to:

Study awareness and usage of basic quality control methods in small software developing
organizations and locate the areas in which they can be improved with minimum effort.

Produce a set of guidelines and recommendations of the most important quality enhancing
methods. The guidelines must be easy to incorporate into a small organization, they must be
flexible and the organization must be able to extend them when their quality maturity is
increasing.

7
2 Structure and organization of this paper
This paper is organized in the following sections:

Quality – discusses the quality concept, and provides various definitions thereof. The quality
concept is the foundation for any further discussion about improving quality.

Method – Description of the method that we used during the empirical study of software
developing organizations. There is also a models used for analyzing the result of the study.

Problems – The problems that we believe small software developing organization faces.
These are the hypothesis for the empirical study. The problems discussed here are partially
based on the personal experience of the authors.

Interview– Contains a summary of each interview.

Analysis – Analysis of the empirical study, the result and various discussions.

Model – The conclusions drawn from the empirical study used to discuss what quality
improving activities are most important, and examples of such processes are given.

Appendix A – Contains the questions that we asked during the interviews.

Appendix B – Answers to the questions, these have been generalized.

8
3 Quality
The chapter gives a background and discussion on the concept of quality in general. The goal is to
provide a foundation for the discussion of quality as it applies for small organizations. The basic
software engineering principles [GILB] are practiced in order to construct a basis for the quality
concept. The principles states that in order to construct a system, the following must be known:

The goals of the system. “Projects without clear goals will not achieve their goals clearly”
[GILB].

The requirements on the system.

How to fulfill the requirements.

The actual implementation of the requirements.

We aim to construct the quality concept by the guidelines that [GILB] describes by the goals,
requirements and fulfillment. This chapter is divided into these sections:

Defining Quality for Small Organizations. The purpose of the quality definition is to
document the goals of quality improvement activities in general and a quality model in
particular. Quality is often defined in a single sentence or a short paragraph. The chapter
provides an overview of definitions available. ISO, for example, has one quality definition
and Total Quality Management is, in a way, a quality definition in itself. This is the goal of
the quality framework.

Existing Quality Models and Small Organizations. A quality model is the requirement
specification for quality improvement. Examples of quality models include ISO 9001 and
CMM; the chapter discusses these in further detail. They are auditable standards, which
means that a company can be reviewed to see whether they comply with the standard or not.
These are the requirements of the quality framework.

Company Quality Manual, CQM. The CQM is a strategy for fulfilling the requirements of a
quality model. The CQM is often company specific and contains well-defined processes that
are followed to achieve a certain result and can (or should) be used directly as working
instructions. This describes how to fulfill the requirements.

The documents that are produced or the results that become of following the CQM is the most
concrete level, and will not be covered by this thesis.

The purpose of this chapter is to focus on the first two parts of our quality model, and to a lesser
extent the CQM. It contains a general discussion over the concept of quality, and attempts to give
an overview that leads to a definition of quality that can be the ground on which the rest of this
thesis can stand on. It also contains an overview of existing quality models and their relation to
small companies.

3.1 Defining Quality for Small Organizations

The definition of quality should provide the goals of quality work. “If quality is to be managed, it
must first be understood.” (David Garvin). This chapter contains an elaboration on the definitions
of the concept.

9
 After the industrial revolution, quality became an important aspect of every company’s way to
success [ENCYC]. It was no longer enough to be productive and effective, it was necessary to
produce high quality products. Michael Hammer and James Champy express it like this:
“Adequate is no longer good enough. If a company can't stand shoulder to shoulder with the
world's best in a competitive category, it soon has no place to stand at all.” [HAMMER+]

We believe that it is fundamental for further work to agree upon a quality definition, since it is the
base for the next step in our model of quality models. When a quality definition has been
established we know what to strive for. There are, as we shall see, some major problems when
agreeing on a quality definition - it is in fact a very difficult concept to grasp. There is a plethora
of quality definitions available (see chapter Subjective experience below), we have chosen those
that we find the most interesting. We have also studied how the quality models that we give a
more in depth view on below (see Existing Quality Models and Small Organizations) define
quality.

To summarize, these are the sources we used for our quality definition:

Subjective experiences – The view of quality from the individual point of view of the authors
of this document, as well as collected opinions from different sources.

Total Quality Management - TQM has many insightful views on quality.

Existing quality models – Each model has its own definition, and they do not always overlap.

Conclusion – Bringing the pieces together.

3.1.1 Subjective experience

One common set of quality definitions focuses on the subjective experience of the individual.
There are many similarities with existentialism and this type of quality perception, accepting that
there are objects of various qualities, but it is the individual perception that matters. This means
that there are different quality definitions, based on the point of view of the person that makes the
definition. Quality for the manager is not necessarily the same thing as quality for the customer.

In [GARVIN:1] David Garvin identifies five different types of quality definitions: transcendent,
product-based, user-based, manufacturing-based and value-based. Below are examples of different
quality definitions that fit into the different types of definitions.

Transcendent Definition

“Quality is neither mind nor matter, but a third entity independent of the two... Quality is a
characteristic of thought and statement that is recognized by a non-thinking process. Because
definitions are a product of rigid, formal thinking, quality cannot be defined. But even though
Quality cannot be defined, you know what Quality is!” (Robert M. Pirsig, Zen and the Art of
Motorcycle Maintenance)

“The meaning of the word quality, like beauty, is in the eye of the beholder. ” (Binnie Perper, How
to sell quality... in other words)

“Everyone has an idea of what quality is, but no one can clearly and unambiguously define it.” (B.
Edvardsson, Kvalitetsutveckling : ett managementperspektiv)

Product-based Definition

"Differences in quality amount to differences in the quantity of some desired ingredient or

attribute." (Lawrence Abbott, “Quality and Competition”)

10
 User-based Definition

"Quality consists of the capacity to satisfy wants." (C.D. Edwards, Quality Progress, October,
1968)

"Quality is the degree to which a specific product satisfies the wants of a specific customer." (H.L.
Gilmore, Quality Progress, June, 1974)

"Quality is any aspect of a product...which influences the demand curve." (Dortman and Steiner,
American Economic Review)

"In the final analysis of the marketplace, the quality of a product depends on how well it fits
patterns of customer preference." (Kuehn and Day, Harvard Business Review, 1962)

Manufacturing-based Definition

"Quality means conformance to requirements." (Phil Crosby, Quality is Free)

Value-based Definition

"Quality is the degree of excellence at an acceptable price and the control of variability at an
acceptable cost." Broh, Managing Quality, 1982.

"Quality means best for certain customer conditions. These conditions are (a) the actual use and
(b) the selling price of the product." (A. Feigenbaum, Total Quality Control)

These quality definitions focus on the concept depending upon the situation. David Garvin has
built a model consisting of eight dimensions of quality, based on this fact [GARVIN:2]. The eight
dimensions are performance, features, reliability, conformance, durability, serviceability, aesthetics
and perceived quality. The problem, according to Garvin [GARVIN:2], is that the customer often
is talking about another dimension of quality than the manager is thinking about.

3.1.2 The Total Quality Management model quality definition

Total Quality Management (TQM) is a popular concept [TINGEY]. It is more of a philosophy than
a quality model, but the ideas behind it can very well be interesting for small organizations. It is
the implementation of it that might differ between a big and a small organization. CMM and ISO
9000 are all different ways of achieving Total Quality Management.

TQM was introduced by Dr. Feigenbaum in 1983 and became widely spread because of Japanese
companies who recognized the benefits of the model [ENCYC]. Today, the term TQM lacks a
universally agreed upon definition, but it basically consists of ten basic benchmarks, which
together form a view on the concept of quality. The benchmarks are:

1. Quality is an organization-wide process. This is means that having a quality department that
is separate from the rest of the company, or by focusing on for example just testing, optimal
quality will not be the result.

2. Quality is what the customer says it is. If there was no customer; no one who had the
requirements for the system, quality would not even be required. Since it is the customer who
the product is produced for, he or she is the most important part of any production chain.

3. Quality and cost are a sum, not a difference. Quality will, in the end, save money.

11
 4. Quality requires both individual and teamwork zealotry. Not only must quality be an
organization-wide process, but the different parts of the company must also be able to work
together in their efforts for higher quality. A well-built infrastructure is required.

5. Quality is a way of managing. Good management means bringing out the full capacity of
everyone in the organization.

6. Quality and innovation are mutually dependent. Some organizations seem to want the creative
process, before construction; to be as chaotic as possible, consequently bringing out the
maximum of creativity from the people involved. But the quality must be a partner of
development, from the beginning.

7. Quality is an ethic. No processes, numbers or metrics are enough to motivate people. What
they need to feel is that the work they are doing is done well. With a constant focus on quality,
they can be certain it will be.

8. Quality requires continuous improvement. You can never have enough quality. Quality is a
constantly upward moving target.

9. Quality is the most cost-effective, least capital-intensive route to productivity. This is why
process improvement is important. The race for quality never reaches the goal. There is
always a better way to do something.

10. Quality is implemented with a total system connected with customers and suppliers. Based on
the fact that the company can not operate without the customers or the suppliers. They are a
unit, a complete system. Implementing quality without including all parts of the system is
misleading.

TQM is not a quality model that you can use directly to follow processes for your work. One can
perceive TQM as the requirements that need to be set on a quality model for it to be complete. The
ten benchmarks described above makes quality a way of totally focusing the organization towards
serving the customer, as well as market success, employee satisfaction and cost leadership. This
has been summarized in shorter quality definitions of TQM, for example the following:

“A comprehensive set of tools, management philosophies, and improvement methods including:

customer orientation, the empowerment of employees, participative management, data-based
decisions, continual improvement, a ‘process’ orientation, and a set of quantitative tools for
process improvement.” [KLEIN]

But other definitions exist, for example:

”The infrastructure, tools, methods and rules which result in increased business success and
customer satisfaction by enabling continuous improvements to people, processes and products.”
[BROCKA]

3.1.3 The ISO 9000 model quality definition

Much has been written about ISO 9000. This chapter attempts to give an overview of the core
values and quality definitions that ISO 9000 is built upon, as well as an introduction to the
requirements that ISO 9000 puts on an organization.

ISO 9000 in itself does not define quality, but the ISO definition of quality is:

“The totality of features and characteristic of a product or service that bears on its ability to
satisfy stated or implied needs.” [ENCYC].

12
 Since this is the ISO definition, one would expect that it is the purpose of ISO 9000 to fulfill this.
This definition does not mention for example process improvement or employee satisfaction. ISO
9004 introduces Organizational Goals [TINGEY] that includes the formulation:

“In order to meet its objectives, the organization should ensure that the technical administrative,
and human factors affecting the quality of its products will be under control, whether hardware,
software, processed materials, or services. All such control should be oriented towards the
reduction, elimination, and, most importantly, prevention of quality nonconformities.”

This formulation implies that quality is about having control of the working process.

There is also an addition to the ISO 9000:1994 standard, which is called Quality Management
Principles [ISOTC176]. A quality management principle is defined as: “...a comprehensive and
fundamental rule or belief, for leading and operating an organization, aimed at continually
improving performance over the long term by focusing on customers while addressing the needs of
all other stakeholders.” There are eight different Quality Management Principles [ISOTC176]:

Principle 1: Customer-Focused Organization. “Organizations depend on their customers and

therefore should understand current and future customer needs, meet customer requirements and
strive to exceed customer expectations.” [ISOTC176]

Principle 2: Leadership. “Leaders establish unity of purpose and direction of the organization.
They should create and maintain the internal environment in which people can become fully
involved in achieving the organization's objectives.”[ISOTC176]

Principle 3: Involvement of People. “People at all levels are the essence of an organization and
their full involvement enables their abilities to be used for the organization's benefit.” [ISOTC176]

Principle 4: Process Approach. “A desired result is achieved more efficiently when related
resources and activities are managed as a process.” [ISOTC176]

Principle 5: System Approach to Management. “Identifying, understanding and managing a

system of interrelated processes for a given objective improves the organization’s effectiveness and
efficiency.” [ISOTC176]

Principle 6: Continual Improvement. “Continual improvement should be a permanent objective

of the organization.” [ISOTC176]

Principle 7: Factual approach to decision making. “Effective decisions are based on the
analysis of data and information.” [ISOTC176]

Principle 8: Mutually beneficial supplier relationships. “An organization and its suppliers are
interdependent, and a mutually beneficial relationship enhances the ability of both to create
value.” [ISOTC176]

These principles are somewhat similar to the benchmarks in TQM (see above).

3.1.4 The CMM model quality definition

There exist no written quality definition in CMM, but since it is a five level scale where an
immature organization is at level one, and a mature organization is at level five, in a sense the
CMM define quality to be maturity. The more advanced your processes are, the better you are,
according to CMM.

13
3.2 Existing Quality Models and Small Organizations
A quality model is a useful tool for a company to control their working processes and in the end
increase their quality. Without a process, improvement becomes harder [HUMPHREY] since the
same mistakes can be repeated. It also becomes easier to incorporate new people to the
organization. The principle 4, Process Approach [ISOTC176] in Quality Management Principles,
also states that processes make the organization more efficient.

There are many advantages with small organizations compared to large ones. Organizations, and
projects can gain a lot by being kept small [LIBERTY], [BROOKS], [MAGUIRE]. They can
adopt fast when circumstances change, they can use the potential of their team members to a
greater extent, and they lack much of the overhead work that is typical for larger organizations.
Also, all of the advantages of being small might just as well turn against the organization if special
considerations are not taken. Quick communication and high speed in taking decisions might lead
to less time for reflections and long term planning.

A study performed in 1995 [STANDISH] showed that only 19% of all software projects finished
on schedule and budget, and one of the main reasons for this was poor documentation and
requirements handling. That figure is quite low, and there are obviously big potential profits to be
made by having it increased. We believe that quality models can possibly aid in doing that.

Our personal experience is that the quality problem is as serious in small organizations as in large
ones, but the ways of controlling it are different. This becomes apparent when one studies the
requirements that CMM or ISO 9000 put on an organization. For example the need of independent
reviewers, or even a quality team, is practically hard to satisfy in an organization of very few
employees. This chapter attempts to enlighten the difficulties for small organizations to take
advantage of the existing quality models. It focuses on the following quality models:

CMM – Software developing quality model. Chosen because it focuses solely on software
development.

ISO 9000 – Large quality model in Sweden and most western countries. Chosen because it is
widely used and well known.

3.2.1 ISO 9000

ISO 9000 consists of two types of standards. The first type is a range of guidelines and is not
auditable. Amongst these are ISO 9000-1:1994, ISO 9004-1:1994 and others [TINGEY]. The
second type consists of auditable standards. This includes ISO 9001, ISO 9002 and ISO 9003. Of
these three, ISO 9001 is the broadest in scope.

ISO 9001 consists of 20 quality system requirements, and these are further broken down into 46
subsections [TINGEY]. The quality requirements include: “Management responsibility”, “Quality
System”, “Contract Review”, “Design Control”, “Purchasing”, “Control of Customer-Supplied
Product”, “Product Identification”, “Process Control”, “Inspection and Testing”, “Control of
Inspection, Measuring and Test Equipment”, “Inspection and Test Status”, “Control of
Nonconforming Product”, “Corrective and Preventative Action”, “Handling, Storage, Packaging,
Preservation and Delivery”, “Control of Quality Records”, “Internal Quality Records”, “Training”,
“Servicing” and “Statistical Techniques”. They are not divided into maturity levels as CMM.

The assessment is a 4-step process [TINGEY]:

1. Preaudit (optional). Preparation for the next step.

2. Audit (pass/fail). Detailed audit of the company’s quality management systems. Can last
several weeks.

14
 3. Registration. If passed in the previous step, the company is registered.

4. Surveillance audit (semi-annual/on-going). Continuous monitoring and verification.

ISO 9000-3 contains guidelines for adopting the ISO 9001 to a software developing organization.

The quality definition of ISO (see The ISO 9000 model quality definition above) implies a very
customer-oriented approach. But the ISO 9000 model is simpler than that. It can actually be
summarized in the following steps [LJUNGBERG]:

1. Say what you do (procedures).

2. Do what you say (responsibility).

3. Document what you did (document).

4. Show it (certifying/revision).

This means that as long as you document what you do, you can basically do anything. There are
some regulations in ISO 9000 that states that laws etc. must be followed, but very little beyond
that. In a sense ISO 9000 is thus a quality model that does not focus upon the quality of the
product being produced but more on the process of producing the product, which might seem
strange. A new version of ISO 9000, called ISO 9000:2000 is currently being developed
[LJUNGBERG], where they focus upon this and (and other) issues.

3.2.2 SEI CMM for Software

The Capability Maturity Model is aimed specifically towards software developing organizations. It
was developed by The Software Engineering Institute, which is operated by Carnegie Mellon
University. It is a model for software process assessments and it is broken into 5 maturity levels.
The maturity levels (except the first) are further divided into 18 Key Process Areas, which are
further broken down into 52 goals and 316 key practices. CMM have (intentionally?) avoided the
quality concept, they are referring to maturity.

The following levels are included in the CMM-model [PAULK+]:

1. Initial. None, or few, defined processes. Ad hoc development. Success depends on individuals
who perform “heroic” acts of programming.

2. Repeatable. Documented and stable estimating, planning and commitment processes are
available. People are more trained on software engineering concepts.

3. Defined. Integrated management and engineering processes are used across the organization.
Data are collected and used in all defined processes. People are constantly trained and
educated.

4. Managed. Processes are quantitatively understood and stabilized. Sources of individual

problems are focused on and eliminated. Data are used to understand the process and stabilize
it.

5. Optimizing. Processes are continuously and systematically improved. Common sources of

problems are understood and eliminated. Everyone is involved in process improvement. Data
are collected to evaluate and select process improvements.

15
Very few companies reach higher than level 3 and a minority reaches level 3, or even level 2.
During 1987 and 1994, 379 CMM assessments were conducted by SEI in USA. They gave the
following results from SEI/CMM:

Level 1: Initial 73,0%

Level 2: Repeatable 16,0%

Level 3: Defined 10,0%

Level 4: Managed 0,6%

Level 5: Optimizing 0,3%

0Table 1 CMM assessments 1987-1994

But the situation has improved. Recent assessments from 1993 to 1999, performed in 697
organizations, give the following results from SEI/CMM:

Level 1: Initial 55,2%

Level 2: Repeatable 25,8%

Level 3: Defined 15,6%

Level 4: Managed 2,7%

Level 5: Optimizing 0,6%

0Table 2 CMM assessments 1993-19999

This indicates a major improvement, and might be an indication of a change in the software
industry towards higher process maturity and quality.

Level 2 is the first level that puts any requirements on the organization. The Key Process Areas in
level 2 are:

Requirements management. Includes, amongst other, processes for documenting

requirements, keeping project plans and documents consistent with changing requirements
and reviewing new requirements before adopting them into the project [PAULK+].

Software project planning. Includes, amongst other, processes for documentation and
derivation of estimates, commitments to work, documenting project planning, identifying a
software lifecycle with predefined stages of manageable size, recording software planning
data and deriving the schedule for the project [PAULK+].

Software project tracking and oversight. Includes for example progress reporting, agreeing
to changes of commitments, communication of the status of the project, process for taking
corrective actions, process for project schedule tracking and basic risk management
[PAULK+].

Software subcontract management. Includes, amongst other, a process for selecting

software subcontractors, holding periodic technical reviews and interchanges with the
subcontractor, tracking the progress of the subcontractor, a process for inspecting the
subcontractors software engineering capabilities, conducting acceptance testing as part of the

16
 delivery of the products from the subcontractor and evaluating the performance of the
subcontractor on a periodic basis [PAULK+].

Software quality assurance. The purpose of this key process area is to make sure that
management is provided appropriate visibility into the process being used by the project and
the products built. It includes the forming of an SQA group, reviews of the projects
development plan, standards and procedures and measuring of the SQA groups activities to
determine cost and schedule status etc [PAULK+].

Software configuration management. Includes a commitment to control changes to

identified software work products and following a written organizational policy for
implementing software configuration management, establishing a configuration management
group, initiating, recording, reviewing, approving and tracking change requests and problem
reports for all configuration items according to a documented procedure and conducting
software baseline audits according to a documented procedure, amongst other [PAULK+].

We will not go into detail in the higher levels.

3.3 Company Quality Manual, CQM

The Company Quality Model describes how the company will fulfill the requirements of the
quality model. It is company-specific documentation of the processes being used in the company.
Absence of such processes can lead to ‘fire-fighting’ [ZAHRAN] which minimizes the team’s
capability of achieving the common goal.

A process can be defined in many ways. [ZAHRAN] presents a holistic view of the definition of a
process. In this view, a process has three aspects:

The Document specifying the process

The Knowledge in people’s brains to drive their behavior, and

The Results of the Process.

The CQM provides the first aspect of this view. The last two aspects are not covered by this thesis,
but would fall under the forth layer in our quality framework.

3.4 Conclusion
By following basic software engineering principles [GILB] we have constructed a model of a
quality framework, divided into the following parts:

A quality definition.

A quality model.

A documented process, or a CQM.

The documents that are produced or the results that become of following the CQM.

The task now becomes to establish a quality definition and a quality model and from this construct
a CQM for small organizations.

17
By looking at different existing quality definitions, as well as relying on our own experience, it has
been shown that quality is a subjective experience and very difficult to define in general. If you
want to define it, you must take a set of different aspects into consideration. David Garvin
introduces 5 different types of quality definitions [GARVIN:1]. They are Transcendent Definition,
Product-based Definition, User-based Definition, Manufacturing-based Definition and Value-
based Definition. The existing quality models’ quality definitions often lack important aspects,
such as process improvement, employee or customer satisfaction or business success.

The quality models ISO 9000 and CMM include a wide range of requirements, which all must be
fulfilled by an organization in order to be certified [TINGEY], [PAULK+].

The documented process, or CQM, describes how to fulfill the requirements of the quality model.
It is a documentation of the processes being used in the company. Absence of such processes can
lead to ‘fire-fighting’ [ZAHRAN].

18
4 Method
This chapter describes the method that is used during the scientific research and empirical studies,
in order to gather and analyze new information. The material in these chapters has largely been
based on [EJVEGARD] and [ELY+].
There are many types of methods available for scientific research, examples are case studies,
descriptions, classification, quantification, etc. The method that we choose to implement the paper
on is a qualitative study and hypothesis. Hypotheses are validated through interviews.

4.1 Qualitative study

We study the awareness of quality models and also the problems that small software developing
organization face, by performing a qualitative study based on interviews. The most accurate way
to study this is probably by conducting some type of active research [IVERSEN+] or an
ethnographic study. This is not a feasible way for us due to the short time available to produce the
paper. The best alternative that we have found is to perform interviews with a set of representative
organizations. The main problem with interviews is that we can not expect the organizations to
simply answer the questions “what problems are there?” There is a risk that such questions would
give the answer “there are no problems!” To remedy the situation we have developed a set of
problems that we suspect that these organizations face. The questions that make up the interview
have been designed in such a way that they can test the validity of the expected problems. It is
possible that this method does not reveal all problems that these organizations face, but we hope
that it reveals sufficiently many. The study is not going to measure the impact that the problem
may or may not have in the organization, it will only find presence or absence of the problem.

The empirical study is hence conducted as formal interviews. Formal interviews mean interviews
with a set of predefined questions. There is room for exploration within the interview, but the goal
is to stick to the questions. The questions are asked in the same order for each organization (where
applicable) to avoid that the order affects the answers of the questions. The purpose of the
interviews is to determine the validity of our hypothesis regarding small organizations.
During the interview, some steps are taken to ensure that the interview is not unnecessarily
interrupted. If it is possible we will conduct the interviews on some type of ‘neutral’ ground where
there are, for example, no telephones.
We plan to perform five interviews from five unique companies, with two persons from each
company.

4.1.1 Documentation
We are going to use tape recordings to document each interview, provided that the persons being
interviewed agree on being taped. If we may not use tape recordings then notes will be taken by
hand. We have chosen not to employ a video camera due to practical reasons, a video camera
would probably unnecessarily disturb the interview since there are so few persons present.
The tape recordings will later (preferably the same day as the interview) be transcribed into a log.
This log will be edited for size and later be sent to the persons that were interviewed for
acceptance.

4.1.2 Selection
How are the interviewed organizations selected? The purpose of the paper is to examine small
software developing organizations, so there are limits of the size of the organization. Small is
defined as less than or equal to 20 persons developing software. This criterion is approximate so
the organization may be larger if otherwise interesting. The organization must be developing
software products. It does not matter what type of software they are developing. It is also a

19
 purpose of this paper to examine organizations that currently are not certified according to some
quality model. Due to practical reasons only organizations in the south of Sweden are selected.
The organizations probably has an office in Malmö/Lund, or at least in Skåne.

It would be difficult to ensure objectivity if we have been working within the organization, so it is
desirable that we are not already familiar with the organization. By organization we do not
necessarily mean a company, an organization may be the software department within a company
that does other things, e.g. hardware development, than software products. The resulting
requirements for selecting organizations are:

Develops software products

Size, less than 20 persons developing software

Not certified according ISO 9000, CMM, or similar

Not already familiar with the organization

Located in the south of Sweden

A list of possible organizations that fulfill our requirements is compiled and then we randomly
select five from that list. Two persons from each company will be interviewed. The first person
that we are interested in interviewing has responsibility for the end product. He/she is developing
software, and is communicating with the customer. Ideally the person is some type of project
leader. The other person is a developer, a programmer, which probably has less contact with the
customer. To keep objectivity, we are not interested in interviewing people that we have had
previous personal contact with.

First person from the organization:

Responsibility for end product

Involved in software development

Customer contact

No previous personal contact

Second person from the organization:

Developing software, close to the code

Probably less customer contact

No previous personal contact

These selection criteria are the default, if we find an organization/person that does not fulfill
all the requirements and still is interesting this will be documented in the interview.

4.1.3 Time
Total time for each interview is estimated to be one hour. That time will be kept rather tightly, as it
is important that the interview stays focused and does not disappear into irrelevant discussions.
Time is valuable, and we do not wish to allocate the interviewee longer than necessary.

20
4.1.4 Confidentiality
The purpose of the paper is to analyze problems with current state of practice in small
organizations. It is thus necessary to discuss problems within certain organizations. The
organizations will probably not want to participate in an interview that results in a paper where the
organization is described as having problems. Therefore is it necessary to ensure that the
organization and person participating in the empirical study are guaranteed anonymity. The
interviews must be confidential. The questions have been designed in such a way that they do not
obviously reveal what organization that has answered them. In cases where products or product
descriptions are mentioned, that might lead to readers being able to figure out which company was
interviewed, these are changed to a more general form.

4.1.5 Access
We gain access to the organization by contacting them via e-mail. If there is interest within the
organization to participate in the interview then a time and a location is decided. The original e-
mail will describe who we are and the purpose of the interview; we will also describe the type of
person that we are interested in interviewing.

4.1.6 Questions
The questions that are not applicable to the organization at hand are ignored, but it is documented
what questions that have been skipped.
The questions have been designed in such a way that they will try to answer if the hypotheses are
correct, or wrong. It is easy to ask questions like “do you have problems with configuration
management?” this has been avoided since such questions are closed. The aim has been to design
as open questions as possible that can be used to determine the validity of our hypotheses.

4.1.7 Outline
The interview will begin with a presentation of the work that we are performing. After that the
person being interviewed is asked to describe the organization and what type of work that he/she is
performing in the organization. These are hopefully easy questions and will, hopefully, remove
some tension. The main questions are then worked through in the order they appear in the question
document. There will be room for small exploratory paths away from the questions, but the goal is
to stick to the questions.
When the interview is finally over the person that has been interviewed will be asked if he/she is
interested in receiving a copy of the final report.

4.2 Reliability
The paper is based mostly on the findings from the interviews. If we do not manage to produce
reliable results from the interviews the entire paper will become highly unreliable. We have
identified some aspects of reliability that we will try to resolve.

4.2.1 Objectivity
Observation can never be entirely objective [ELY+], but we will try to obtain as much objectivity
as possible. There is a problem with interviewing people working within software developing
organizations, since we are familiar with the environment and culture that often is present there.
That familiarity might make us see things the way we expect them to be, not how they are. We
believe that awareness of the problem of objectivity is the best way to ensure it.

21
4.2.2 Risks during the interview
Interviews are by no means an exact science. There are many possible ways that interviews can
result in erroneous conclusions. We have identified a set of risks, and strategies for minimizing or
avoiding them:

The person being interviewed says what we want to hear. This could possibly be avoided by
performing the interview on neutral ground like a cafeteria.

We are told how things should be, not the actual state of practice. There have probably been
discussions within the organization about how things should really be. There has however
never been any time to realize these plans. The questions must be designed in such a way that
they result in the current state of practice, what they are actually doing.

We do not interpret the person being interviewed correctly. Possible reasons for this might be
that we do not share the same definitions for the concepts that are discussed. It is not obvious
what requirement, test or problem actually means. The first step avoid this risk is to tape the
interview, that will minimize the risk of us not hearing what they are saying. The other risk is
more difficult, but we must remember to ask for definitions of the ‘fancier’ words.

The questions are designed in such a way that they lead the answer. There is a balance
between asking open enough questions, and asking specific questions. Too open questions
will not provide the answer that we are interested in and too specific questions risk leading the
answer. We have designed rather open questions, and we will ask counter questions if it is
obvious that it was misinterpreted.

4.3 Analysis of data

The following process is followed when analyzing the data:

1. The interviews are translated into English.

2. The interviews are sent to the interviewees for approval.

3. Each interview is analyzed and an analysis document is prepared. The analysis is based upon
the interview and the hypothesis regarding software development. The analyses are
summarized in a conclusion of the interview, which includes a grade to which we feel that the
hypothesis fit into the organization. This grade is from 1 to 5, where 1 means that the
hypothesis did not fit in at all, and 5 means we were right on the spot.

4. Each answer to each question is studied and generalized in order to find patterns among the
answers. These answers and patterns are available in Appendix A.

5. Finally, a conclusion of all the interviews are made and the grades for each hypothesis and
interview are added. A matrix of the hypothesis is produced and an average of the grades for
each hypothesis is calculated. An average value of 3.0 or more indicates that we were correct.

6. For each hypothesis, a conclusion is written.

4.4 Conclusion
A qualitative study based upon short formal interviews with a set of small Swedish software
developing organizations. The interviewed organizations are develops software products with
less than 20 persons, are not quality certified and that we are not already familiar with.

22
Two persons for each organization are interviewed where one person is some type of project
manager and the second person is more like a programmer. The purpose of the interviews is to be
able to validate a set of hypotheses.

The result of the interviews is analyzed with the objective of establishing compliance or non-
compliance with hypotheses. The strategy is to assign a score (1-5) to each hypothesis for each
company. If the score is more or equal to 3 then the corresponding hypothesis is validated for that
organization. The average of all scores is presented in the interview conclusion.

23
5 Problems and hypotheses
This chapter contains descriptions and discussions about the hypotheses that we have regarding
software development in small organizations. These hypotheses are used as the basis for the
qualitative study.

The hypotheses are based partially on our personal experience from developing software in the
industry and partially on literature studies; they may or may not be valid, that is what the empirical
study is going to show. Some of the organizations that we have encountered were not aware of
configuration management, no project plans were established products were not tested before
delivery, requirements were not documented and bugs were not corrected. Many organizations
realize that it is necessary to make nightly backups, but how many have actually verified that these
backups are correct? We wish to examine if our personal experiences are generally true. We have
identified the general problem domain, and some hints to be looking for in the analysis of the
interviews. We believe that small organizations face problems in these domains:

View of software development – Development is too short sighted.

Planning – No or poor project plans are produced and maintained.

Management – Is not working with the developers.

Selecting technical solutions – The solutions are based upon prejudices and commercials.

Configuration management and version control – There is no control over versions of

work products.

Testing – Basic ad-hoc testing is performed, but little is documented.

Requirements – It is not known what the customer really wants.

The domains are similar with the key process areas in CMM level 2 (repeatable) [PAULK+]. That
leads us to believe that these problems are those that are most necessary to correct before more
rigorous quality models can be introduced in the organization.

Each hypothesis description begins with a theoretical background. This background serves the
purpose of motivating the existence of the hypothesis and the indications therein. After the
background is a description of the hypothesis itself. The hypothesis describes a hypothetical state
in an organization. The hypothesis is concluded with a set of indications that the hypothesis is
correct; these indications will be used later on when analyzing results from interviews.

5.1 View of software development

Background. Developing new software products is an investment. It takes time and effort to
develop and the end product will in many cases survive for years to come. This is indicated by the
large cost of maintenance. Maintenance costs are as much as 40-70 percent of the cost during the
total lifecycle [BOEHM], where maintenance is defined to be post-delivery changes.

Hypothesis. The hypothesis is that the organization does not consider the long-term nature of
software when they are developing it. Planning is focused on the next deadline; or the next visit by
the customer. They are often aware that they should document, test etc, but they feel that there is
no time for such things. In the very short view of software lifetime, there is no time. When the
goal is to finish the current project and not long-term success then haste and stress follows. Small

24
 organizations can probably not develop strategic plans for the next ten years, but it should be
possible to predict that there will be another project. There is a problem with maintenance of old
products.

If the organization realized that they should have a more long-term view of their software
investment than we believe that the next deadline would not be as important as continuos
improvement. With a longer-term view of software development, quality systems and continuos
improvement is easier to motivate.

One of the most important aspects of software development is often time-to-market due to the
competitive advantage of getting to market sooner [CROW]. So there is no time for too much
control, or too formal processes. When the focus on time-to-market is high and a project risk
becoming late, management hires more people and adds them to the project. Adding more peoples
to a late project risk making it later [BROOKS].

Indications.

Quality System. There is currently no existing quality system concerning the entire
organization and the processes by which they produce software.

View of ISO 9000. Their view of for example ISO 9000 is based on prejudices. They think
that it will cost too much to formalize the way they work into a process. It may or may not be
true that the cost is too high, the focus here is that the organization does not know if this is the
case. The reason that we have chosen to focus on ISO 9000 and no other quality models such
as CMM in the hypothesis is that we believe that ISO 9000 is the only major well known
model in small software organizations. If the view of ISO 9000 is based on prejudices we
conclude that the same would be the case of other models as well.

Experience preservation. They do not in a formalized way preserve experiences from

previous projects. When experiences are not taken care of the organization risk facing the
same problems in the next project.

Brooks Law. When late for a deadline they add people to the project or work overtime in
stead of discussing a more long-sighted solution with the customer. Brooks law: “Adding
manpower to a late software project makes it later” [BROOKS].

5.2 Planning
Background. Poor planning is the source of problems more than any other [METZGER]. Large
projects must be broken down into smaller manageable tasks (divide and conquer), called
activities or miniature milestones. Activities are only a few days worth of time and defined to be
‘done’ or ‘not done’ [MCCONNEL]. The reason to break down projects into smaller activities is to
be able to track progress, “Software progress monitoring was so poor that several well-known
software disasters were not anticipated until the very day of expected deployment” [JONES:2]. We
conclude that formal progress reporting is essential for any software project. Without control on
daily basis the project risk being delayed, “How does a project get to be a year late? One day at a
time.” [BROOKS].

There is a risk in putting too much faith in a detailed project plan as well; the anti pattern
[BROWN+] discusses this risk and calls it Death by planning. “The deliverables plans should be
updated weekly to ensure appropriate planning and controls that reduce project risks”
[BROWN+].

Hypothesis. The hypothesis is that there are no or vague project plans. The delivery dates of the
products are decided by the management, which seldom asks the employees first. A plan is often
produced at the beginning of a project, but it is not maintained during the project. There is no way

25
 to tell if the project is running late, or if the deadlines will be held. The plan is not based on
activities that have been produced by dividing the project into smaller, manageable pieces. It is
difficult to know if the project is on schedule or not, since no progress reporting is performed.

Indications.

Divide and conquer. Projects are not divided into activities (or miniature milestones). The
project is planned as a whole, work tasks are identified but these are very large and involve
many peoples. It is difficult to measure progress in such big activities. Project plans are not
revised. Establishing a plan early in the project is not worth much if it is not revised to mirror
reality as time goes by.

Progress reporting. Progress reporting is non-existent. Project progress is based upon gut
feelings, thus is it difficult to know the current state of the project. There is a risk to fall for
the 80-20 problem. That is, when one believe to be 80% finished, only 20% is actually.

Deadlines and milestones. There exists no deadlines or milestones. Given enough time will
make any problem go away. A project with no deadlines can never be late. The deadlines are
decided by management, with little or no input from the people responsible for implementing
the project.

Use of estimates. Estimations that are the basis for project planning are based solely upon
experience and gut feelings. It is difficult for new employees who lack experience to produce
reasonably reliable plans.

5.3 Management
Background. We have not found any theoretical background for this hypothesis; it is based solely
upon our personal experiences developing software in small organizations. This makes the
hypothesis clearly weaker than the others presented in this chapter. The reason that we have
chosen to include it in the paper is that it is a personal experience and we wish to examine if this is
generally the case.

Hypothesis. The hypothesis regarding management is that the organizations has been founded by
one ‘older’ male with excellent knowledge of some domain and that this person is controlling the
organization like a despot. This person hired a few ‘programmers’, and they all set out to develop
new software. The management person had a very hard time to realize the software development
specific problems, and the programmers are not trained in seeing the greater picture. The
programmers are excellent ‘coders’, but they do not see anything beyond technical solutions. It is
possible that the organization lacks a project manager, or some type of authority between the
management and the ‘coders’. The manager is not taking active part in the development of the
project, but believes that the developers should be left alone, which is odd since it is he that sets
the deadlines. As the projects proceeds and the manager gets new ideas for the end product new
requirements are added to the end product.

Indications.

Project managers. There are not any project managers. Absence of a person who is
responsible to managing communication between higher management and developers indicate
this problem.

Management part of development. Management does not take active part of development
and has therefore little knowledge of software development problems. Despite this he is
setting the deadlines, defining requirements and signing the contracts. This leaves the
developers in an impossible situation since someone else decides when they are going to be
finished.

26
 Adding requirements. Requirements are added without the influence of the developers. The
manger is in contact with the customer, and promises new features.

Developers domain knowledge. The developers have poor knowledge of the end user
environment that they are developing software for. The manager who has this knowledge
assumes that the developers are as familiar with the domain as he is.

5.4 Selecting technical solutions

Background. Solutions are the means to provide implementations of the requirements on a
software product. The technological solutions that are selected define the end product. In order to
select such solution must the requirements be know; the qualities of a suggested solution must be
known, along with the drawbacks of the solution and the resource consumption required [GILB+].
When a set of possible solutions have been found an evaluation must be performed in order to find
the solution that best suits the problem [GILB+]. “A solution worth considering is one whose
positive contribution to your requirements outweighs its negative ones” [GILB+].

Hypothesis. The hypothesis is that the technical solutions that are selected are the result of brief
discussions, or selects products from company X alone, because they have always done so. There
is no technical reason why they select the solutions they do. The organization is limited by the
technologies they have experience in, and they are applying the known technologies for any
problem. Programmers refuse to consider alternative languages, or database solutions. When new
technologies are introduced into a project is it unknown if the product will meet the necessary
requirements. The decision to select a specific solution is not taken by the project as a whole but
rater by management, with little input from developers.

Indications.

Vendor lock-in. A company is using product X from vendor Y because they have always
done so. Technologies are selected mainly from previous experience; this results in problems
to adopt new technologies. The concept has been borrowed from the anti-pattern with the
same name [BROWN+].

Evaluating technologies. New technologies are not evaluated. New untested technologies are
accepted as solutions without evaluating them or comparing them with similar technologies.
When using new technologies is not validated that it will perform according to the
requirements. New technologies are not subjected to proof of concept tests in advance of the
selection. It is thus unknown if the technology is sufficient.

Golden hammer. A single solution is applied to all problems. The company uses the same
solution for any problem. The concept has been borrowed from the anti-pattern with the same
name [BROWN+]

New technologies evaluated by management only. Management takes decisions of

technologies on strategic meetings, without developers’ influence.

5.5 Configuration management and version control

Background. Software configuration management is the process by which change and evolution
of software products is controlled. Configuration management improves productivity and product
quality [FEILER]. Configuration management reduces overall software development costs
[TOMAYKO]. “Lack of automated source-code control is a common and irksome inefficiency”
[MCCONNEL].

27
 One of the most important configuration management problems is management of simultaneous
updates [TOMAYKO]. If this is not managed then there is a risk that different changes distort the
end product and thus causes software failure. [TOMAYKO]. There are configuration management
strategies that allow simultaneous access to source code or other documents. This strategy requires
manual effort to merge changes made by different persons. This is the strategy used by cvs
(http://www.cyclic.com/) for example. During the interviews we will ask how they ensure atomic
file access, if the answer is that this is not necessary since they accept merging changes then the
indication is wrong.

Hypothesis. Our hypothesis is that problems occur when the company has many customers and
many versions of the same software at different customer sightings. What if customer A finds a
bug in a specific version, and some other customers are affected as well. Does the company know
which version of the product is installed where? The organization has problems with ensuring that
only one person is working with a file at a given time. It is not possible to restore the last version
of a file, except by restoring backup tapes.

Indications.

CM tool. Without tools the management of versions and files are ignored or managed by
hand. Managing these issues by hand is time consuming, difficult and boring.

Working version control. Rely on backup copies for previous versions of files. There are
many problems with backups for example (1) there is no history information (2) information
is backed up only once or twice a day (3) it is time consuming to restore backups.

Delivery control. The company does not know which version of the software is installed
where. It is not even sure that the customer knows what version he/she has installed. Lack of
this information makes it difficult to restore the customer configuration for finding defects. It
is not possible to reproduce versions being delivered to the customer. It is not possible to
restore the customer configuration in the office. Since the company is unable to do this, they
have very hard time to find defects.

File control. It is possible that two persons work in the same file simultaneously. They do
thus risk ruining each other’s work, or produce inconsistencies in the files.

5.6 Testing
Background. It is necessary to test software products. There are developers that try to avoid
testing by developing defect-free software using various methods but “attempts to construct
perfect software are doomed to fail” [HAMLET]. What testing method is then the best to use? “No
one method, no one set of methods, can guarantee finding all bugs” [BEIZER]. Even though it is
impossible to find all bugs in a software product, it is not possible to avoid testing. In all test
strategies is it not only necessary to document how to test but also to predict the outcome of the
test. If the outcome can not reliably be predicted then there are misconceptions as to what it is the
routine should and is doing [BEIZER]. The purpose of testing is to find the important defects, in
order to remove them, before they make their way to the customer. Testing the actual end product
is not the only work product to test. If defects can be found and eliminated before they make their
way to the end product large savings can be done; “the cost of removing defects from software
increases dramatically with the time the defects remain in the product” [BOEHM]. Software
inspections have reported test execution cost reduction of 5 to 10 times, since there are fewer
defects to find [GILB+].

Hypothesis. Some type of testing is performed, that is the source code is compiled and executed.
Possibly some type of ad hoc testing as well. There is no formal test plan. When the customer
inspects partial deliveries, or even the end product, everyone is very nervous that major defects
will be detected. No regression testing is performed; it is common that defects that have been

28
 corrected reappear again and again. The organization is not familiar with simple testing concepts
such as coverage tests, or statistical testing.

Indications.

Testing strategies. None or just one testing strategy is followed. The testing strategy is
probably input validation. No single testing strategy can find all defects, relying only on input
validation is thus not sufficient.

Test cases documentation. Test cases are not documented. With undocumented test cases the
process of finding defects is ad-hoc. Each test run is different from the previous.

Binary testing focus. Only binaries are tested. Other types of tests such as peer reviews or
inspections are not performed. Inspections can be performed earlier in the development cycle
than tests on binaries and can therefore potentially save time and money.

Validation. The result of a test is not known before the test. If a test is performed, and the
result of the test is not known in advance, than any result is acceptable. Some defects such as
program crashes can be found but more subtle defects are difficult to find.

Root cause analysis. Root cause analysis, the process by which one attempts to find the real
reason to a problem. Found defects can reappear or produce follow-up errors, since no root
cause analysis is performed.

5.7 Requirements
Background. Requirements for software products are the description of the desired properties of a
system. The requirements define what to be implemented. Requirements are also one of the most
difficult aspects of software development. Incomplete and changing requirements are two of the
main reasons to why project are delivered late, over budget and with less functionality than
desired [STANDISH]. A European study also showed that the management of customer
requirements was one of the principal problem areas in software development [KOTONYA+]. The
risk of adding features that are “good to have” is known as feature creep. Projects that fail to
control creeping requirements are highly accessible to excessive schedule pressure, it can even
destabilize the product to such a degree that is can not be finished at all [JONES].

Hypothesis. Our hypothesis is that the products developed are based upon the customer
specifications, but little time is spent on analyzing the requirements to ensure that they are the
correct. The customer is maybe not sure what he/she really needs. The set of existing requirements
is in constant flux. Also, it is very difficult to know that you have interpreted the customers’
demands correctly. The customer often has very deep domain knowledge, but is uncertain about
the terms used in software development or the industry in general, which leads to
misunderstandings and communication problems. Requirement changes are discussed over the
phone or mentioned in meetings. These changes are not managed, and thus easily forgotten or
misinterpreted.

Indications.

Requirements specification. Initial requirements are poorly documented. The initial

requirements are unknown and correspondence with the customer is based on phone calls and
e-mail, the information from this communication is not documented. It is thus not known
what to build.

Feature creep. New requirements are poorly documented which makes feature creep a
problem. The requirements are in constant flux. The customer constantly changes his/her
mind; the reason to this is that they do not really know what they want.

29
 Requirements verification. There exist no formal way of communicating requirements with
the customer. The requirements analysis is based upon ad-hoc processes, it is not known if the
agreed upon requirements are the right requirements.

5.8 Conclusion
The chapter presents a set of hypotheses that are used as the basis for the empirical study, these are
partially based upon personal experiences of the authors and partially on literature studies.

View of software development. The hypothesis is that the organization does not consider the
long-term nature of software when they are developing it. Planning is focused on the next
deadline, or the next visit by the customer. Longer-term quality achievements are not prioritized
or encouraged.

Planning. There are no or vague project plans. A plan is often produced at the beginning of a
project, but it is not maintained during the project. There is no way to tell if the project is running
late, or if the deadlines will be held.

Management. One single male person that controls all aspects of the development manages the
organization. This person is controlling the organization like a despot.

Selecting technical solutions. The organization is limited by the technologies they have
experience in. Programmers refuse to consider alternative languages, or database solutions. When
new technologies are introduced into a project is it unknown if the product will meet the necessary
requirements.

Configuration management and version control. There exist no plan for controlling
configuration of source code, documentation, releases or any other work product. It is not possible
to retrieve older versions of work products.

Testing. There is no formal test plan. When the customer inspects partial deliveries, or even the
end product, everyone is very nervous that major defects will be detected. No regression testing is
performed; it is common that defects that have been corrected reappear again and again.

Requirements. Products developed are based upon the customer specifications, but little time is
spent on analyzing the requirements to ensure that they are the correct. The customer is maybe not
sure what he/she really needs. The set of existing requirements is in constant flux.

30
6 Interview analyses
This section contains the analysis of each interview performed. The actual transcriptions from the
interviews are not available in this document. The analyses are based on our hypotheses and on the
answers from the interviews. The analysis has been as objective as possible, but we have probably
been affected by our impressions from the meeting as well.

Two companies declined the opportunity to participate in the interview series. The stated reason
was that they could not spare the time. The total time for the interview was two hours for each
company. The companies that could not plan for two extra hours in the next three weeks probably
had important views upon the software quality concept. There is hence a risk that we have not
interviewed the companies that are most important.

6.1 Overview
Each interviewed company is analyzed in a separate chapter. The chapter begins with a short
description of the company and its domain, and the persons interviewed. The analysis is based on
the hypothesis presented earlier; the goal here is to find if those hypotheses are consistent with the
interviews or not. To do this, we make a judgment of how well the hypothesis stood up against
reality for each company. To quantify this judgment, a score from 1 to 5 measures the consistency,
where a score of 5 indicates full consistency and 1 means no consistency. Since there is a range
between 1 and 5 we have chosen to set the value 3 as the limit for when our hypothesis was right
or wrong. This means that when all the interviews are summarized the hypothesis’s that received
an average value of 3.0 or higher are verified.

These scores are the results of evaluating the indications of the hypothesis with evidence found
during the interview. Since the interviews only lasted for an hour each, these scores are affected by
our subjective impression of the company, as well as the answers they gave to our questions.

The general structure of an analysis is as follows.

6.1.1 Name of hypothesis

Indication 1…n. Describes how the indication stands up against reality.

Conclusion. A conclusion of the hypothesis and a final motivation for the grade.

Name of hypothesis #grade

31
6.2 Company I
Only one person is interviewed in this company. The company consists of two persons so we do
not believe that a second person would have contributed with much extra information. The
interview was conducted in the home of the interviewee.

This company is more a hobby than a full time work. It might not be fair to compare them to
other software developing companies, but we have chosen to include them in the survey since this
is the way that many larger companies have been founded. When the company grows it probably
becomes more necessary to structure the work, but the more people involved the more difficult it
probably is.

6.2.1 Presentation of the company

The company is developing and maintaining software products and web pages. Two persons
founded the company in 1996. The company is focusing on database front-ends in Visual Basic. It
is a small organization, where the employees are the owners and do not depend on the company’s
income for their own living. It is closer to a hobby than a full time job.

6.2.2 Presentation of the interviewed

Person I

He is 40 years old and is working for Company I at a part time basis only. Does not have any
formal education in software development. The other person working within the organization has a
formal education as a programmer.

6.2.3 The hypothesis

View of software development

Company I’s view of quality is based on the assumption that it is the software in itself that carries
the quality; the customer is involved only as an end user of the product. Usability and stability are
the main concerns. These two make quality in software development difficult. By usability they
mean a “...clean user interface”. This is based on personal experience and the domain knowledge
of the company. The second most important factor is the structure of the database, and ensuring
the relations. The problems are hence more or less technical by nature.

Quality System: The company had not been discussing how or if they needed to structure their
work. The company currently does not use any quality system, although they think that doing so
could improve the way they work.

Experience preservation: No project conclusions are made. Experience gained is focused on

technologies and programming.

View of ISO 9000: They have no or little relations to ISO 9000, but believe that it might help
software-developing organizations to structure their work.

Brooks Law: The are only two persons working within the company, and they do not hire more
people even if they would be late, so they do not risk falling for Brooks law.

Conclusion: There is no quality system implemented in the organization, nor are there any
concrete plans to do so. Problems addressed in earlier projects are not preserved. There exist no
plans for improving the way that they work. This is because Company I is producing software
because they think it is fun, and as a hobby. They may not have many of the problems that larger

32
organizations do, but there is no long-term plan or view of software development so the hypothesis
scores 5.

View of software development 5

Planning

Divide and conquer: The project is not broken down into activities; the short term planning
is managed with TODO lists, which is a common way of keeping track of activities. The fact
that they do not have any deadlines makes further planning unnecessary since they can never
become late. This also leads to the fact that estimates are unnecessary and measurement of
progress seems to be non existent. The schedule is based upon what windows and tables that
have been designed.

Progress reports: Progress is not documented, or measured.

Milestones and Deadlines: Since the projects the company has been involved in were fully
developed before being marketed, they did not have a deadline in the normal sense of the word so
the problems involved with having a deadline to work against was non existent. They develop a
basic plan for the project, but there are no real deadlines. They did not promise any external
customer any release dates, so they do not find a more thorough plan necessary.

Use of estimates: Size in length or time is not estimated.

Conclusion: Projects are not divided into activities, nor is any project plan established. Progress is
not measured. This is absence of planning, which the hypothesis suggests, and the score is
therefore 5.

Planning 5

Management

This hypothesis does not apply for this type of organizations, since the two employees are also
developing their products.

Management N/A

Selecting technical solutions

Vendor lock-in: Although they did not explicitly state it, we believe that a certain major
company is providing all their development tools. For example, when asked how they
preserve experiences they said: “Microsoft is improving the standard both on development
tools and web browsers.” They also used Microsoft Access as their database “...because we
believe that it is fastest”. When talking about a project that they have made, it was a rework of
an older program: “We decided to develop an alternative to this application with modern tools
such as Access and Visual Basic.”

Evaluating technologies: They do not perform any evaluation of the tool that they select to
use, the only requirement is that of experience. No validation of the technologies used was
performed. They used Visual Basic as the programming language “...because we are most
experienced with that”.

Golden hammer: The company has been involved in too few projects for us to make a
conclusion about this.

New technologies evaluated by management only: Not applicable.

33
Conclusion: Even though the number of projects that they company has been involved in are too
few to draw any real conclusions. We still believe that our hypothesis is true and give it a 5, since
the situation complies with that described by the hypothesis.

Selecting technical solutions 5

Configuration management and version control

CM Tool: The organization does not use any CM tool.

Working version control: There is no CM tool, which means that it is very difficult to revert
to older versions of existing files.

Delivery control: The fact that their products have only been delivered to single customers
reduces the problem of distributing updates so less version control might be necessary.

File control: The company consists of two persons, and they had only one computer to work with.
So it is impossible to have to people working on the same file simultaneously.

Conclusion: The organization is not using any CM tool, which the hypothesis suggests. There is
no control for managing delivered products, few products has been delivered so this might be a
non-issue. The situation is like that discussed in the hypothesis, even though the implications are
not so difficult.

Configuration management and version 5

control

Testing

Testing strategies: No formal tests are performed; the only type of test that seems to be
performed is rudimentary input validation.

Test cases documentation: There are no test plans, and the result of a test case is not known
in advance of the test. The tests are performed by hand and no regression testing is performed.
Bugs, defects or deviations are not classified or documented.

Binary testing focus: Only binaries are tested.

Root cause analysis: No root cause analysis is done.

Conclusion: The organization is not preparing any formal test plans. The result of the tests
executed is not known in advance, and is based solely on the binaries. Defects discovered are not
subject to root cause analysis, which could prevent them from reoccurring. This is the situation
that the hypothesis suggests so the score is 5.

Testing 5

Requirements

Requirement specification: One of the projects they have done is a conversion of an older
program into a new Windows-based modern application. This made the requirements handling
rather easy, since they had an original product to base their new work on. They used this older
program as a base, and added their own experience of the domain with modern development tools.

Feature creep: Since they had no deadlines and no specified requirements, there was no risk
for feature creep.

34
Requirement verification: A customer, with newer requirements, was never involved. Since the
applications were developed with no customer involved and no real deadline, they had much
freedom in doing their work. It was totally up to them selves to decide. They are developing
prototypes as they way of communicating requirements with the customer.

Conclusion: They developed the product for company where one of them had previously worked
and had thus deep knowledge of the domain. Since they know their domain very well, and can use
this knowledge to build user-friendly products. Our hypothesis about requirements handling does
not apply and get a 1.

Requirements 1

35
6.3 Company II
This interview was conducted on site, in a meeting room. Each person was interviewed
individually, beginning with Person I.

Company II is a software developing organization that is actively working with introducing a

quality system. The system is not currently a part of each software developers everyday work,
which means that the hypothesis regarding configuration management for example seem to hold.

6.3.1 Presentation of the company

Company II is a company with very high domain knowledge in a certain technical domain. When
they started, 15 years ago, they did not do any software development at all, but over the years this
has increased and now a larger portion of the company is involved in software development. They
have customers nation-wide and in more than 30 countries around the world. The general
education level is master of engineer in their domain. Some of the employees are systems
engineer, but they are a minority.

The company has been working together with a large, international consultant company, and has
learnt a lot about project management from them.

6.3.2 Presentation of the interviewed

Person I

Person I have a Technical Ph.D. and has worked in the company for 3 years. She is working as a
technical consultant and project manager. She does not write any code, but do large-scale system
design on some products. Customer contact is a large part of her work.

Person II

Person II is a master of engineer in a domain closely related to the companies. She is currently
working mostly as a software developer.

6.3.3 The hypothesis

View of software development

Quality System. There is a company quality manual that describes project management over
larger software projects. This manual seems to focus on project management and customer
relation’s aspects, but there seems to be less focus on the developers. The developers are not even
using the manual.

View of ISO 9000. “ISO 9000 contains a lot that our organization would benefit from. But the
risk is that it takes longer time, and it can not take extra days to write a document that the
customer has to pay for.” There seems to be some doubts about the usage of ISO 9000 and that it
might lead to too much paperwork. Paperwork in itself is not a bad thing though, in the context of
avoiding misunderstandings is the strategy to document. “We try to get it on paper. A lot of paper
work, but you have to do it. We do status reports that the customer has to sign. That is our goal.”
The perception about ISO 9000 is not based on studies or actual experience.

Experience preservation. Formal conclusion reports are produced after longer projects, which is
another sign of their project management maturity. The developers do not seem to be involved in
the process, so the programming work are still not as developed as the project management
processes.

36
Brooks Law. When deadlines can not be held as agreed upon the solution is to work more. “Then
you work overtime and you get really stressed up.”

Conclusion. There was a company quality manual, it was not based on ISO 9000, but that is not a
bad thing. This manual was mainly used by management, the development teams had little contact
with it. Our hypothesis is regarding the entire software developing organization so there was some
compliance with it. The fact that conclusion reports were produced is contrary to the hypothesis.
Finally was the hypothesis more or less wrong, due to some compliance did it make the score 2.

View of software development 2

Planning

Divide and conquer. TODO-lists for short term planning of what to do during the day, and in
shorter projects. Projects are not broken down into small activities, at least not smaller projects.
There were large differences between large and small projects, where larger projects were more
formal.

Progress reporting. There is no formal way of measuring progress in a project, yet it is important
to be able to predict when the current plan is going wrong. The reason for the overtime is: “Lack
of resources. Even though we try to plan, you do not always have the resources you need. Perhaps
we do not have anybody to put in to work.”

Deadlines and milestones. There are deadlines and additional milestones, these are planned and
the project plan is revised when it is necessary.

Use of estimates. There are planning strategies but they are often difficult to implement, the main
reason is that estimation is difficult. “You think that everything will go much smoother than it
does.”

Conclusion. Project plans are established and these are updated when necessary. The project is not
divided into activities in the sense; TODO lists perform short term planning. This is consistent
with the hypothesis. The fact that it is necessary to work a lot of overtime during some periods
imply weaknesses in project planning. Estimations are not based on metrics or other formal
method. It is definitely not full compliance with the hypothesis, but some indications are correct,
so an average score is reasonable.

Planning 3

Management

Project managers. There are project managers that are responsible for the larger projects. The
requirements on the product are negotiated with the customer. The developers are taking active
part in the project.

Management part of development. The CEO or other management person performs no extra
control over the project. This weakens our hypothesis, that there is one person on the company
that decides everything.

Adding requirements. There have been problems to reach project conclusions, the customer was
never satisfied, more and more issues had to be corrected. This was not the result of higher
management intervention though.

Developers domain knowledge. Company II started as a company within a specific domain, and
has gradually moved over into becoming a software developing organization. This means that the

37
developers have a very high degree of knowledge in the domain, though there is less software
engineering expertise.

Conclusion. The organization is clearly not governed by a despotic manager, which the hypothesis
states. No compliance with any of the indications was found, so a very low score is natural.

Management 1

Selecting technical solutions

Vendor lock-in. The solutions used in projects vary, languages etc. depend on the project at hand.
“We change for example language and tools between projects.” The solutions are not limited to
previous experiences. There is also no risk that a single solution is applied to any problem.

Evaluating technologies. Technologies are not evaluated and verified that they will work in
practice and solve the problems stated by the requirements. “You do not. It can be too slow. You do
not know that.”

Each employee have a personal development plan, when there is nothing scheduled then the time
is spent on learning for instance new languages. This ensures that new knowledge is obtained by
the organization.

Golden hammer. Some solutions are limited due to lack of alternatives, there is only one main
vendor in the domain and it is necessary to use their products.

New technologies evaluated by management only. Each project is responsible for selecting
suitable solutions. This means that management is not involved the way our hypothesis suggests.

Conclusion. There are little indications that our hypothesis is correct here. Solutions vary from
project to project, depending on the requirements at hand. However, many projects require
products from a single large vendor. New technical solutions that have not previously been
employed are not evaluated in advance, which the hypothesis predicts. There is thus little correct
indications so the score is as low as 2.

Selecting technical solutions 2

Configuration management and version control

CM tool. Person I’s view: “Most often, it is one-person projects, and then there are no problems.
If it is bigger ones, then it is divided so that you do not share components between persons. Every
person is responsible for his or her piece. You do not share files.” There is no CM-tool in use, but
since files are never shared this was not a problem. Person II answer differs: “In the beginning
that was not a problem, but then problems occurred. [...] But it’s important, it’s so very important.
It’s dangerous to change things at the same time.”

Working version control. Reverting to older versions of documentation and source code is
possible by retrieving backups. The developer usually saves temporary versions locally “You may
have an older copy left, but it is not always that you do.”
Delivery control. The state of the product before delivery is saved, with the purpose of keeping
track of what version is stored where. It does happen though that changes are made to this state.
“But if changes are coming in after the delivery, we are a bit sloppy with documenting it.”

File control. There is no official way of ensuring atomic usage of source code, or other
documents.

38
Conclusion. Based upon the words by the developer interviewed we conclude that many of the
indications are correct. A CM plan and tool is desired by the organization. Older versions of
source code are retrieved from backups and there is no control on the file level. It is not always
possible to reproduce customer’s versions since late changes are not always documented. This
indicates full compliance with the hypothesis.

Configuration management and version 5

control

Testing

Testing strategies. There are three types of tests involved (1) simple tests, (2) unit test and (3)
integration testing. Simple tests are performed by the programmer during the development of the
product.

Test cases documentation. These tests do not follow any formal plan, nor are the results of the
test known in advance. A person other than the programmer performs unit tests; these are also
undocumented with unknown results. The final type of testing, integration testing is more formal
with prepared test procedures but still with unknown results.

Binary testing focus. The testing strategies employed test the produced binaries alone.

Validation. Expected results from tests are not documented.

Root cause analysis. Little regression testing is performed to ensure that old defects do not
reoccur, or that patches do not produce more defects. Defects do not result in a root cause analysis.

Conclusion. There was a set of different testing strategies; this is directly inconsistent with the
hypothesis. Most of the tests are undocumented with unknown results, though. No inspections or
test other than binaries are performed and subsequent versions are not subject to regression tests.
We still believe that the score is average since so many different testing strategies are performed.

Testing 3

Requirements

Requirements specification. The product requirements are documented. “A lot of paper work, but
you have to do it.” There have been situations where changed requirements have not been
documented. The result of such changing requirements has been feature-creep.

Feature creep. There has been situations like “If the customer has changed his mind, and we say
“yes, yes”, and then we have not documented it and it becomes expensive.”

Requirements verification. The requirements are communicated with the customer with the help
of prototypes, milestones etc. The use of evolutionary delivery decreases problems related to
requirements handling.

Conclusion. Requirement specifications are rigorously documented, which is inconsistent with the
hypothesis. There have however been indications of feature creep. The strategy to avoid
misunderstandings is to develop increments or prototypes so the customer can observe
development progress. There are risks associated with evolutionary prototyping [CONNELL+]
development or evolutionary delivery [GILB]. One of the largest is that the next deadline becomes
too important, and you forget about the long-term lifetime of the software that you are developing
[MCCONNELL]. To have a strong design, for example, becomes secondary in comparison to have
an executable binary on the next delivery. The score is low due to requirement specification and
incremental delivery.

39
Requirements 2

40
6.4 Company III
The interview was performed at the office in a meeting room. The two persons were interviewed
simultaneously.

6.4.1 Presentation of the company

The organization is developing products and is consulting on EDI (electronic data interchange)
solutions. The primary platform is the IBM AS/400.

6.4.2 Presentation of the interviewed

Person I

Is working, as the manager for the consult department, is also responsible for two internal
products. He has a shorter academic education, but there was not much computer education
available at the time. There have naturally been many courses within the organization as well.

Person II

Works as an on site consultant is also responsible for one product. He is usually adapting existing
solutions, but may spend time developing products on site. He has one year of academic
education in designing computers and software, also internal education within the organization.
The main knowledge has been obtained according to learn-by-doing.

6.4.3 The hypothesis

View of software development

Quality System: The interviewed has a customer oriented quality perception that is also partially
existential; “It is an experience”. They mean that quality is personal and what some people
perceive as quality, may not be quality for some one else. Project planning is perceived to be one
of the major obstacles when it comes to quality in software development and testing is the second
major problem factor. Later in the interview it becomes apparent that they have no type of formal
testing.

The company is clearly interested in improvement. “We are in a process where we are getting used
to the idea of improving quality if that process leads to ISO 9000 or something else we do not
know.” By improvement they mean to document more of the way they work, “It might be
necessary to become more formal and spend more time on documentation.” They also state that:
“The larger the organization becomes the more important it is to document what one is doing.”

The company wants to introduce some quality framework but do not know which are available.
They feel that they should improve themselves but they have not taken any serious steps in that
direction as of yet. The current projects and work-tasks leave little time for focusing on
documenting processes etc.

View of ISO 9000: The general perception of ISO 9000 is that it is too bureaucratic, “...many of
our customers are certified. The fact that they are certified leads to an enormous paperwork each
time there is a deviation report. For small companies this leads to very much administrative work
the times that I have seen it.” They also seem a little skeptical about ISO 9000 since it does not
help them develop better products.

Experience preservation: There is currently little possibility to preserve experiences from one
project to the next project, and this has caused some problems for the company. Not preserving

41
these experiences indicates that they are more focused upon the current project, than long-term
success, which verifies our hypothesis. Only the problems of today are important. There seem to
be some interest in improving this situation though. One solution is to keep a log of events
occurring within each project.

Brooks Law: Since they have no deadlines, they have no projects that are late for deadlines
and Brooks’ law is not applicable.

Conclusion: As all the others, the company is interested in improvement, but does not have a plan
for implementing it. They believe that ISO 9000 is too bureaucratic and does not believe that it
will help them produce better products. They do not preserve experience, which has caused some
problems in the past.

All in all, this strengthens our hypothesis of short-term view of software development, the current
project is more important than long term improvement.

View of software development 4

Planning

Divide and conquer: Short term planning is based upon TODO lists that are managed
individually by each person. There is no central planning but group meetings ensure that workload
is equally divided. Longer scope planning is based upon contracts or projects; there are no shorter
activities.

Progress reports: They do not have any deadlines, but lack of deadlines does implicate that it is
not necessary to report progress, “one feels where one is.”

Milestones and Deadlines: The most interesting aspect of project planning is that they did
not establish any deadlines. They rather agreed upon “what time in the year that we will be
finished.” The reason that they worked this way was that they are primarily selling consulting
and resources, not developing their own software. Since they prioritize paying customers they
believe that it is impossible to set any deadlines for the internal projects. Without deadlines
the company risks feature-creep and becoming late. “The time plan for our internal projects is
not very strict, we use it as a general aim. There is thus no ‘deadlines’ when we have to be
finished.”

Use of estimates: No estimates are done.

Conclusion: The company does consulting services and on-sight development, which means that
the internal project’s suffered in planning. Project planning was perceived by the company to be
one of the major obstacles when it comes to quality in software development. If they do not know
where they want to go, then it does not matter what path they chooses.

This strengthens our hypothesis and we give it a 4.

Planning 4

Management

Project managers: The company had product responsible that in a way acted as project
managers.

Management part of development: It is a collective responsibility to select projects. There

do not seem to be any despot on the top that decides what to do. This is probably because of
the high technical competence in the organization.

42
 Adding requirements: New requirements were added with the influence of the developers.

Developer’s domain knowledge: The developer’s domain knowledge was very high.

Conclusion: The developers, as well as the managers, have very good knowledge of the end users
environments. It is a collective responsibility to select projects. The developers take part when
adding new requirements. Adding it all together makes our hypothesis wrong.

Management 1

Selects technical solutions

Vendor lock-in: All software is developed for the AS/400 server from IBM, which is vendor
lock-in. This is the situation described in the hypothesis but it is on the other hand one of the
business ideas of the organization.

Evaluating technologies: “There are strategic meetings within the organization that select
major technical solutions.” They discussed an example of such strategic decision and the
arguments for that technology was that “It is very well documented” and “It is a big known
provider” - it is good because other people use it. Some type of internal, thorough evaluation
of the product before the organization decides to implement all of their software with it, is not
performed.

Golden hammer: “When we have selected to tool to be used for development then all must
projects adhere to that.” To select one solution that all projects must use is clearly similar to the
golden hammer [BROWN+], where a single solution is applied to any problem, and strengthens
our hypothesis.

New technologies evaluated by management only: This was not true in this organization.

Conclusion: All software is developed for specific server software, which is vendor lock-in. They
have strategic meetings, deciding on new technologies, but does not seem to make any thorough
evaluation. Management does not decide on new technologies, but the facts that they are trusting
brand or lack of alternatives strengthens our hypothesis.

Selects technical solutions 4

Configuration management and version control

CM Tool: No CM tool is used. The reason that they have not evaluated any version control system
is “there is not any versioning or generations management tool automatically available for
AS/400.” This does not change our hypothesis. The reason for not using a configuration
management tool does not matter to our hypothesis.

Working version control: Their strategy for reproducing older versions of their products is based
upon a backup system. The backup system goes three weeks back, but anything older than that is
difficult to reproduce. The fact that they rely on backup copies for older versions of files
strengthens our hypothesis.

Delivery control: They do not keep track of what version is installed where, and they do not
preserve older versions of their products, only the latest version is available at the office. Recently
a feature has been added that allows the customer to see which version they have installed. This
leads us to believe that they have had problems with this earlier. Corrections to detected defects
are distributed in so called program temporary fixes (PTF). These are distributed to the customers
that have reported the defect.

43
 File control: They have not had any problems with overwriting each other’s files, since they
are working in separate files. There is no way of ensuring that people actually are working in
separate files other than oral communication.

Conclusion: The company does not use any CM tool. They do not have any overall strategy
for CM handling and does not keep control of where different versions of their software were
installed. This has made them adding a new feature to their programs that lets the customer
see which version they are running, probably because they have had that kind of problems
before. There is no formal way of ensuring that files are not corrupted as a consequence of
overwriting each others work.

All of our indications were right, giving the hypothesis a 5.

Configuration management and version 5

control

Testing

Testing strategies: The main concern about testing is that a non-technical person should
perform it. This person does not follow any formal test plan; he tests based on previous
experience. There are hence little formal test plans, or test protocols.

Test cases documentation: The result of a test is not known in advance of the test. Some type of
regression testing is performed on their PTF before they are delivered to their customers. Little or
no tests are automated.

Binary testing focus: Only binaries are tested.

Root cause analysis: They do not perform any root cause analysis of the defects that they find, so
they risk repeating the problems over and over. The defects are not documented and used in future
tests, so there is a risk that they will return. There have been problems with defect corrections that
lead to new defects.

Conclusion: There are no test specifications, no peer reviews etc. The simplest form of monkey
testing is performed, they only condition is that the person who is testing should be a non-
technical person. Adding it all together, our hypothesis is verified and strengthened, which gives a
high score.

Testing 5

Requirements

Requirement specification: The company is mainly maintaining their product, and the
requirement analysis made is based upon customer criticisms on previous versions. The product
has matured over the years, and there are industry standards that must be followed, so they do not
find the requirement analysis to be any problem.

Feature creep: They are aware of the risk with feature creep but do not associate it with
requirement analysis.

The facts that they have existing products which they continuously add PTF:s to might
indicate that feature creep has occurred. They do not however, make new products or are
involved in new projects. This means that our hypothesis indication of Feature creep does not
really apply.

44
 Requirement verification: The product responsible handles requirement verification. “As
responsible for a product you have to be able to view things from the customer perspective.”

Conclusion: Their deep domain knowledge makes it easier for them to understand customer
requirements and suggestion. There are no formal processes for managing requirements, but we do
not find our hypothesis strengthened due to their domain knowledge.

Requirements 2

45
6.5 Company IV

6.5.1 Presentation of the company

Develops interactive media in the form of web services and CD-ROM. The customers are average
to large sized companies. Examples of products are shockwave games, presentation support, sales
support and education. The company consists of two persons, but they hire resources during larger
projects. Large projects employ up to four people.

Company IV strong side seems to be their way of communicating with the customer. They want
the customer to feel at home with what they do. This makes the customer feel safe about their
investment.

They have not been in contact with software engineering, and the skills they have are from
experience and ideas from other companies.

6.5.2 Presentation of the interviewed

Person I

Is working with project management, but is also responsible for the graphics and is a 3D
specialist. Has unfinished studies at LTH and some miscellaneous jobs behind him. Has largely
been learnt by doing.

Person II

Is responsible for programming. Do sometime work as the project manager. In his academic
background there is a degree in system engineering and a short course in multimedia.

6.5.3 The hypothesis

View of software development

Quality System: Since the company is developing web services and multimedia applications, they
are somewhat separated from the other companies, since much of the work is done by graphic
tools, and WYSIWYG (What You See Is What You Get) editors. This decreases the complexity
generally found in code-based products, and the importance of writing maintainable code.

They do not have a complete quality system, but they do follow simple processes concerning for
example customer contacts. They do not write conclusions of projects, although they realize that
as they grow, this will become increasingly important.

Experience preservation: When asked if they document their experiences they say: “We should
really document more.” They continue: “If we grow to a certain point of size such methods will
become important, but there is currently no such method. “

View of ISO 9000: They think that ISO 9000“...feels too big and nasty”. This indicates that their
view of ISO 9000 is based on prejudices.

Brooks Law: This is not applicable since their projects are so small. They can not afford to add
more people. If the project is too big, they turn it down.

Conclusion: Most of the companies we have interviewed states that they will get better at
documenting what they do, and Company IV is no exception. It is impossible for us to know if this
is a sincere desire to improve their processes or something they know that we want to hear.

46
They follow some simple processes, but do not have a complete quality model and they do not
document their experiences. The general impression we got from the company, added together
with their answer’s give our hypothesis a 3 when applied to Company IV.

View of software development 3

Planning

Divide and conquer: When asked how they make their project planning they answer: “When we
are planning a project, we divide it into phases. Each phase is then estimated for length. The sum
of all phases indicates how long the entire project is. A phase can be a few days.” This means that
they divide projects into smaller parts and do estimates on the parts.

Since they do not have any progress reporting, one can assume that the project plan is not updated.

Progress reporting: They do not have any kind of progress reporting – “It is like cooking - you
see what happens”.

Deadlines and milestones: Deadlines and milestones are used.

Use of estimates: They use estimates, so the indication was wrong.

Conclusion: In a small company such as Company IV, long-sighted planning is very difficult. Still,
Company IV seems to have an organized way of working. They follow simple processes from
PROMISE (Producers Of Multimedia In SwEden) when contacting customers, and since they
teach classes and hold lectures, they must follow some sort of scheme.

The company has some sort of a planning strategy. Three of our indications in our hypothesis were
wrong, but since they are only two people some of the indications might not apply. Adding it
together gives our hypothesis a 2.

Planning 2

Management

This hypothesis does not apply for this type of organization. The company is too small.

Management N/A

Selecting technical solutions

Vendor lock-in: When working with multimedia and web based development, new tools and
techniques are constantly emerging. It is also important that you use the latest tools in order to
be able to do the most advanced effects and implement the latest features. This means that you
are in some way stuck to the tool vendor that you are most familiar with.

Evaluating technologies: When asked how to choose technology, they answer: “We choose the
technology that we know best. It takes too long to learn a new programming language for each
project so we stick to the technology that we are competent in using.” This would strengthen our
hypothesis, but in the next sentence they also add: “If we do not know the required technology we
are forced to turn down the customer.” They can not afford to evaluate alternatives, but they
evaluate the ones they are familiar with. Still, this is a verification of our hypothesis.

When it comes to whether or not the product will work in practice, they regulate that in the
requirements. This means they are at least aware of the problem.

47
Golden hammer: Company IV seems to know their limitations and does not take jobs that are not
inside their domain. They also know what their tools are capable of, and turn down jobs that are
not possible to implement with the technologies used. This implies that our hypothesis indication
of Golden Hammer is not correct. They use a hammer, but only to hammer nails.

New technologies evaluated by management only: Not applicable due to the size of the
organization.

Conclusion: Being a multimedia creator, Company IV is very closely dependent on the tools on
the market. It is crucial to know the right tool and to use it optimally. Company IV seems to be
stuck to the technologies they are using, but still they know their limitations. This means that our
hypothesis is almost correct and is given a 4.

Selecting technical solutions 4

Configuration management and version control

CM Tool: The Company does not use a CM tool. They have some sort of version control but
it is manually managed. They do not work in a networked environment, which could possibly
speak against investing in a tool, but a configuration management tool brings with it a lot of
useful side effects. Also, the network can be seen as a part of the CM tool itself. Today the
two persons in the company have different version control strategies. They both rely on
backup copies for older versions of source files.

Working version control: Person II says: “There has never been any version management in that
way, and there has never been any problem with version management. There is no need of such
control. There have however been problems with your own versions, so some formalization might
be valuable.” This indicates that they might improve their configuration management process.
They do not currently have any version handling process.

Delivery control: Each product is only delivered to one customer, so this is not applicable.

File control: Since they do not work in a networked environment, two persons can not access the
same file simultaneously but that does not mean that the same document can not end up on both of
their computers.

Conclusion: Even though the company seems to have a good control of the files and versions of
products, they lack a common CM strategy and do not use a tool. It is up to the employee to keep
track of changes or different versions of files. They only deliver to one customer, so they have no
delivery plans. This gives our hypothesis a 4.

Configuration management and version 4

control

Testing

Testing strategies: When making a CD-ROM product, thorough testing is of the essence. Once
printed, the CD-ROM is irreversible. This has lead to a testing process that seems to capture most
of the errors in the end product. It is not, however, documented or formalized.

Test cases documentation: More than one testing strategy was followed, but only binaries were
tested. The result of a test is known before the test, during the user testing. Test protocols were
used, and reapplied when errors are found.

Binary testing focus: Only binaries are tested. No testing or reviewing of source code or
documents is done.

48
 Root cause analysis: Found defects are not subjected to root cause analysis, which means
that there is no way of preventing them from reoccurring.

Conclusion: The company follows a testing strategy that appears to work very well, but it is
not documented. Only binaries are tested. No root cause analysis is done but more than one
testing strategy is followed, and they appears to be working well. This results in a 2 to our
hypothesis.

Testing 2

Requirements

Requirements specification: We asked what the difficulties involved in software

development related to quality was, and Person II answered: “It is to understand what the
customer wants, or rephrasing the customer needs into a requirement specification. This is
one of our strong sides; we are good at explaining to the customer what the customer really
wants.” Requirements are thoroughly documented and no work is done until consensus has
been achieved.

Feature creep: New requirements are incorporated in the requirement specification, which
prevents feature creep.

Requirement verification: They had a complete view on the software lifecycle, when it comes to
testing and verifying requirements. They were for example always sure that new requirements are
verified before beginning to work on them. But still the process was not formalized in any way. It
was totally based on the experience of Person I and II.

Conclusion: Company IV knows a lot about how to do things wrong, because they have been
working as teachers, teaching multimedia. They have seen their students do all the classic
mistakes and they have also learnt how to speak to people that does not know as much about
technology as they do.

Our hypothesis indications were wrong at most parts. Even though they do not have a formalized
requirement’s handling, they follow informal processes that work well. This gives our hypothesis
about poor requirement’s management a 2.

Requirements 2

49
6.6 Company V
The interview took place in a meeting room at the companies office. The two representatives of
the company were interviewed at the same time.

Company V was the last company we interviewed and the only that had come into any contact
with ISO 9000. They were very pleased with it, and showed us their quality manual, which was
easy to read, and very small (a couple of pages). They saw no problem with “too much
documentation” or “too much bureaucracy” that the other companies were concerned about.

6.6.1 Presentation of the company

Company V developed their own products and offered their customer extra resources. There were
ten employees in Company V at the time of the interview. The typical customers were smaller
producing companies in Scandinavia. The main product that Company V developed was a
production-planning tool that can be connected to the customers business systems. They also had
customers in trotting business.

6.6.2 Presentation of the interviewed

Person I

Person I was the CEO of the company. He had been working there from the start and have been
studying economy, but has no formal degree.

Person II

Person II is quality responsible, and a programmer. He has been working at Company V for two
years and has no academic degree, although he did study computer science for some time.

6.6.3 The hypothesis

View of software development

Quality System. The organization is currently in the process of developing a quality system, with
the aim of becoming ISO 9000 certified.

View of ISO 9000. The view of ISO 9000 is based upon facts and hand on experience. The
process of becoming certified is actively supported by external expertise.

Experience preservation. The view is that documenting a process is not enough; it must be
improved upon with time according to the experiences learnt. “We must continuously improve our
processes.”

Brooks Law. When late for a deadline is nights and weekends utilized, rather than removing
requirements. The main strategy is to call the customer and work something out, though.

Conclusion. The hypothesis is wrong on most points here. There is a quality system actually being
developed, and that system is based on ISO 9000. The organization is taking courses led by
authorized ISO 9000 personnel and the view of ISO 9000 is thus not based on prejudices. The
organization finds it necessary to preserve experiences in order to improve upon the processes.
This makes the hypothesis score very low.

View of software development 1

50
Planning

Divide and conquer. The project-plans are prepared and these are based upon earlier experience,
and in-house expertise. Project-planning and cost budgeting are some of the most important
aspects of software development. These plans are partially broken down into smaller sub-
processes, or activities. Each sub-process is a rather large part of the entire project.

Progress reporting. There is no formal way of measuring progress on the work tasks. “One
learns from experience how long time that each work task should take.”

Deadlines and milestones. Deadlines and milestones schedule the project. These are defined in
the project-plans and are necessary to structure the work.

Revision of project plan. The plan is revised when necessary, no plan can last forever and is a
living document.

Use of estimates. The plans and deadlines are not based upon previous measured metrics or
formal method. The method used is based solely on earlier experiences.

Conclusion. The hypothesis is mostly wrong in these aspects. Formal project-plans are produced,
and are considered as important aspects of a project as a whole. These plans are broken down into
sub-processes, deadlines and milestones. The sub-processes are large parts of a project are not
activities, as we perceive them. There is no way of reporting progress except verbally in meetings.
The project-plan is updated throughout the project, based on informal progress reports. The initial
plan is based on estimates from earlier projects, no metrics etc. are collected. Project-plans are
important and used, the lack of progress reporting and formal estimations makes the score 2
though.

Planning 2

Management

Project managers. There are project managers who are responsible for ensuring the project
schedule, and allocates resources for work tasks.

Management part of development. The management is not directly in control over the
requirement of the software products. There are formal methods for controlling projects.

Adding requirements. Small changes are accepted without any formal discussion, but if a change
would imply many hours of work than this must be formally documented and planned for.

Developers domain knowledge. The developers have a varying background not necessarily from
the domain that the organization is developing products within.

Conclusion. The hypothesis is clearly wrong in all indications. There are project managers and no
senior management person dictates the requirements in the project. The developers have a varying
background. The formal quality structures are inconsistent with our hypothesis. All of this implies
that the score should become very low.

Management 1

Selecting technical solutions

Vendor lock-in. “We usually select Microsoft products, since they can be trusted to survive the
next few years. Microsoft is also the only company that develops interesting technologies.”

51
Technologies are selected based upon what vendor have developed them. Previous experience is
also very important when selecting solutions in projects.

Evaluating technologies. There is no way of knowing if the selected technology will work in
practice, since they are not evaluated and no proofs of concept are developed. Selecting a new
technology is taking a ‘chance’.

Golden hammer. It is important to follow the technological evolution to keep up with current
events. It is however each developers’ responsibility to study new technologies in their spare time.

New technologies evaluated by management only. Each project is in control over the selection
of solution and no higher management decides what technologies are used.

Conclusion. Stating that Microsoft is the only company that develops new and interesting
technologies seems as a high conformance with vendor lock-in. New technologies are accepted
without formal evaluations, they take chances in stead. Expecting that the personnel study new
technologies seems to risk that they do not, and thus sticking to old technologies. It is however the
projects responsibility to select techniques, not management. This makes the score high, but not
full.

Selecting technical solutions 4

Configuration management and version control

CM tool. A CM tool is used, the tool is Visual Source Safe delivered by Microsoft.

Working version control. The tool makes it unnecessary to rely on backups for version control.

Delivery control. The products are delivered to unique customers, this makes it relative easy to
keep track over what version that is installed where. The possibility to make labels in the tool also
makes it easy to retrieve older versions of entire products.

File control. The tool makes it easy to ensure unique file access by file-locks.

Conclusion. This company was the only one interviewed that used a CM tool. This tool is greatly
appreciated. They deliver their software to unique customers, so the version control problem is
non existent. All of the hypothesis indications are wrong and the score can only be 1.

Configuration management and version 1

control

Testing

Testing strategies. There is a set of different testing types: functionality testing, flow testing, beta
testing and provocative testing.

Test cases documentation. Only beta testing is formally documented, with expected result for
each test case. The other testing strategies are performed by the person responsible for the code, or
his/hers peers.

Binary testing focus. Only the binaries are tested. Most test strategies are variations of input
testing.

Root cause analysis. Found defects do not result in root cause analysis, and can hence return in a
form or another.

52
Conclusion. There were different types of strategies for testing and this is contrary to our
hypothesis. They appeared to have very good testing strategies, which weakens out hypothesis.
However, the testing was based on the binaries only, which conforms to the hypothesis. Finally,
beta tests are documented with expected results and thus the score is rather low.

Testing 2

Requirements

Requirements specification. Requirements are formally documented and the customer signs all
specifications. Prototypes are used as the basis for communication with the customer, and to
ensure that misunderstandings are avoided.

Feature creep. Changing requirements are documented as far as possible; the customer must also
sign all changes. Smaller changes may be implemented without extra paperwork. The
requirements changes but in an orderly manner.

Requirements verification. The requirements are documented in a requirements specification,

these are discussed with the customer using prototypes and other means of communication.

Conclusion. Requirements are documented and verified with the customer. Prototyping is used to
ensure that the requirements are correct. Changing requirements are documented. None of the
indications for this hypothesis were right, the final score was thus 1.

Requirements 1

53
7 Analysis conclusion
This chapter contains the conclusions that we draw from the analysis of the interviews. Table 3
contains the resulting averages from the interviews. The management hypothesis could not be
performed on two organizations; the reason to this was that those company where too small to
have any apparent management.

The grades in Table 3 are from 1 to 5 and the corresponding hypothesis is considered to be true if
the average value is more than or equal to 3.0. The grades are based on how many of the
indications in the hypotheses that the company showed signs of, so 3.0 would mean 60%
compliance with the hypothesis. For the purpose of this paper, that is a sufficient indication of the
hypothesis. Two hypotheses are thus not ‘true’ and those are Management and Requirements. The
hypotheses that are the most ‘true’ are the ones regarding Configuration Management and
Selecting Technical Solution.
Management

Requirements

software dev.
View of

Planning

Testing

solution
technical
Selecting

control
version
CM and
Cmp. I N/A 1 5 5 5 5 4

Cmp. II 1 2 2 3 3 2 5

Cmp. III 1 2 4 4 5 4 5

Cmp. IV N/A 2 3 2 3 4 4

Cmp. V 1 1 1 2 2 4 1

Average

Table 3, the averages of the companies in the survey

7.1 The hypothesis

Each hypothesis is discussed in further detail in this chapter. The grade is motivated and examples
and quotes from the interviews are presented.

7.1.1 View of software development

We have shown that these companies have a too shortsighted view of software development. The
current project or activity is more important than long term improvement. Introducing quality
systems into an organization takes time and effort and these organizations do not believe that they
have the time or the resources to do so. It is common that the companies have an internal
discussion regarding quality improvement, and they feel that it is necessary to document more, but
little actual activity is performed in order to achieve this.

There are examples of companies that have successfully introduced, or are in the process of
introducing, quality systems into their organizations. None of these organizations believed that
such system consumed too much time or effort. We have not found that ISO 9000 is not suited as a
quality system for small organizations. Some believed that ISO 9000 was too bureaucratic and
time consuming but their views was built on believes and no actual experience. The company that

54
 was in the process of becoming certified according to ISO 9000 meant that there is nothing wrong
with ISO 9000, since you are yourself responsible for developing processes.

We asked the companies what quality, when related to software, was for them. The most common
answers during the interviews were some form of “customer satisfaction” or “well-tested
software”. Three of the companies said that software quality is that the program does not crash.
We did not find any company with a stated quality definition. These quality definitions are from
the organization point of view. More accurately, from the person that received the question point
of view. The definitions are seldom concerned with quality of what.

There are dangers in focusing only on these two. It is, for example, not always clear whom the
customer is, the one paying for the product, or the one using it. For how long should the customer
be satisfied, one hour or ten years? Such definitions easily neglect the long-term perspective of
software development. The products that were developed 20 years ago, and today suffer from the
Y2K problem, are they of low quality? Were they not enough tested when they were released? To
achieve customer satisfaction it is necessary to know what the customer expects. The budget or
time schedule may not be as important as fulfillment of requirements for some customers. Such
customer may very well be satisfied with the result, even if it was delivered late, or cost more than
estimated.

One of the companies we interviewed gave us the following answer: “It is to meet the customer’s
demand. [...] Quality is the right thing, on time and on budget. There is no good or bad quality,
you have to be at the same level as the customer.” The person who gave us this answer was a
project manager at the company. The company had been discussing quality and how to increase
their quality for a longer period of time. It is interesting to note that the developer at the same
company gave us the following reply on the same question: “[...] it is well tested. It never becomes
bug-free, but that it is tested is very important.” This clearly shows that different persons, even at
the same company, have different views on quality.

Another company answered “Quality when it comes to software development is that everyone is
working according to the same principles and that everything is documented.” This company was
on the way of getting ISO 9000 certified, and this answer was the only one where testing or
customer satisfaction was not even mentioned. If this indicates process maturity or too much
influence from the ongoing ISO 9000-certification process is impossible to say, but it does show
yet another perspective of quality.

7.1.2 Planning
Many software projects are rather roughly planned. There are examples of companies that do not
issue deadlines or milestones in their projects. Most companies do however perform high level
planning, but we have not found any company that breaks the larger scale plan down into smaller
activities (such as reviewing a document, meetings, finishing a certain module of software, testing
a certain module of software etc.). The short-term schedule is instead managed ad-hoc using
TODO lists. There are no examples of companies who are measuring progress in their projects.
Some uses their large-scale project plan to identify progress in their projects.

On the single developer level progress is based upon experience, which makes it difficult for new
employees with little experience to estimate progress. Large project happens one day at a time,
one hour at a time [BROOKS], it is thus necessary to keep track of the smaller activities. The
estimations for larger activities are also based upon experiences, which might be less a problem
since there are probably experienced persons available for that, but a more formal estimation
process could potentially increase estimations correctness.

7.1.3 Management
Our personal experience with small software developing organizations was that the management
did pay little attention to the developers. Management controlled requirement changes and project

55
 deadlines without discussing this with the people responsible for developing the products. Our
study has shown that this is not generally the case. In the larger companies there were project
managers who are responsible for the project and communication with higher management. In
smaller organization all of the employees were part of the development process.

7.1.4 Selecting technical solutions

The solutions that are used to implement the project requirement are selected with an ad-hoc
process, and most often came from the same vendor (3 of the companies). None of the companies
was able to know that the selected solution would actually work in practice. The selection of
technology does thus seem like chancing. The selection is often based upon previous experience,
or by trusting some vendor.

All companies agree that technology changes very fast and that it is important to be aware of new
emerging technologies. In that light it does seem dangerous to trust personal experience and a
specific vendor. There is one example of a company that made a strategic decision regarding what
technology their entire project was to be developed with.

Selecting technologies to implement requirements is an important part of any project, if a

technology is chosen that is not capable of solving the problems and hand are must work wasted.
It does seem sensible that selecting technologies based upon the requirements in the project, and
an evaluation of the possible solutions. The evaluation would then contain a proof of concept to
ensure that the technology does actually deliver the desired requirements. It may very well be the
case that a single vendor develops suitable products, but neglecting to evaluate alternatives may
lead to the usage of inferior solutions.

7.1.5 Configuration management and version control

We have only found one company that used a configuration management tool. All companies were
aware of the existence of such type of tool, but were not using them. The reason for not using such
a tool was usually that they could not find any reason to do so. Every company felt that it was
necessary to keep track of older versions of the product, but that was conveniently managed using
some backup strategy.

The developers kept personal older copies of the source code locally on their own workstations.
The personal backup strategy is risky, since it depends on human remembering to update files back
and forth. Two companies did not work in a network environment and did not believe that there
was any risk of overwriting each other’s work. All companies were thus subject to the problem
that a tool solves, and they were trying to solve these problems manually. Tools are cheap and
easily installed, so there are no reason that a software developing organization should not use
them.

Most companies developed products that was unique to each customer, and it was thus no
problems of keeping track of what versions was installed where.

7.1.6 Testing
We have shown that the testing processes in small software developing organizations have
insufficiencies. Most of the companies have some sort of testing strategy, but it has been shown
that one testing strategy is not enough [BEIZER] to find a maximum amount of defects.

Only binaries, i.e. the functionality of the program as seen by the user, are tested. No reviews or
inspections of code are performed. No coverage testing is performed. The most common way of
testing seems to be to have someone follow a test protocol, report and fix the deviations, and
iterate until no more defects are found. When errors are found outside the testing protocol, they

56
 are seldom incorporated in the testing process. Root cause analysis is seldom made, error fixing is
almost ad hoc.

We believe that most companies are unaware of other testing strategies than for example input
validation (the program should not crash on erroneous input) and a validation against the system
or requirement specification (to make sure the program does what it is supposed to do).

7.1.7 Requirements
Most of the companies we interviewed were highly competent in their specific domain. This made
requirements handling less of a problem than we had expected. Most of the companies were also
very good at documenting the requirements. Also, the companies told us stories about how they
had failed with their requirement handling previously and learnt a lot from this. It seems as if this
is one of the first fields that companies get better in. If you run a company you learn early that
documenting the requirements are important for your own survival.

Prototypes were used extensively, which might lead to other problems [MCCONNELL], but they
help a lot in discussing requirements with the customer.

But it was not just positive. Most of the companies lacked a formalized process for requirements
handling, and new requirements were often added without being documented which leads to
feature creep.

None the less, our hypothesis did not stand up against the actual reality, which is clearly shown by
the low grade of the requirement hypothesis.

7.2 Conclusion
The study has shown that companies have shortsighted view of software development, the
hypothesis “View of software development” received the grade 3.0. The organizations are not
considering the long-term nature of software while developing it. The organizations do not believe
that they have the time or the resources to introduce a quality plan. Planning is focused on the next
deadline only. Even though the time to market is becoming increasingly important [CROW], one
can not forget about the long-term quality of the software being produced. This is very important,
as late changes are much more expensive than earlier fixes [BOEHM]. In fact, fixing an error
while the software is in operation can be as much as 1000 times more expensive than if the error is
corrected in the first stages of development [BOEHM]. We believe that by introducing formal
processes in the form of a CQM, this can be remedied, a suggestion for such CQM is presented in
the next chapter.

Project planning is inadequate and the hypothesis received the grade 3.2. The hypothesis is that
there are no or vague project plans. According to [METZGER] poor planning is the most common
source of problems. During the interviews we found that many software projects are rather
roughly planned. Some companies do not issue deadlines or milestones in their projects. None of
the companies are measuring progress in their projects. We believe that this can be improved by
introducing project-planning processes.

Companies select technical solutions without evaluating them. The organizations are limited by
the technologies they have experience in. This hypothesis “Selecting technical solutions” received
the grade 3.8. It was based on the anti-patterns [BROWN] Vendor Lock-In and Golden Hammer
and our own experience of the industry. We believe that by introducing processes for evaluating
technical solutions this can be improved.

Configuration management is inadequate. The hypothesis is that problems occur when the
company has many versions of the same software at different customer sightings. There are
problems with ensuring that only one person is working with a file at a given time. It is not

57
possible to restore the last version of a file, except by backup. The hypothesis received the grade
3.8, which means that it is verified by our study. A CM tool and a CM process can improve the
situation at most companies [TOMAYKO], [FEILER].

Testing is inadequate. The hypothesis regarding software testing is that some type of testing is
performed, but there exist no formal test plan. No regression testing is performed; it is common
that defects that have been corrected reappear again and again. The organization is not familiar
with simple testing concepts such as coverage tests, or statistical testing. This hypothesis received
the grade 3.6. We believe that by introducing a testing process, this can be improved.

All of the above hypotheses were verified by our study. The following hypotheses, however,
proved to be wrong.

The hypothesis “Requirements” received the grade 1.6. The hypothesis is that the products
developed are based upon the customer specifications, but little time is spent on analyzing the
requirements to ensure that they are the correct and that changing requirements are poorly
handled. This is serious because it can destabilize the product to such a degree that it can not be
finished at all [JONES]. But since most of the companies we interviewed were highly competent
in their specific domain, it made requirements handling less of a problem than we had expected.
This seems to be one of the first areas that companies get better in.

The hypothesis “Management” received the grade 1.0, which means that is was completely wrong.
This hypothesis is that one person on the company decides everything as a despot. He (it is often a
he) sets the deadlines and adds new requirements, but can not understand the software
development specific problems. The programmers on the other hand are not trained in seeing the
project from the managers’ point of view. It is possible that the organization lacks a project
manager, or some type of authority between the management and the developers. This does not
seem to be the case in the industry today, according to our study.

We will now use this knowledge to construct a basic Quality Manual for small organizations. It
will focus on the 5 issues that proved to be a problem in existing organizations.

58
8 Quality framework for small organizations
The purpose of this chapter is to use the results from the analysis in chapter: 7, and construct a
quality framework from this, that can be used to construct a CQM structure especially focused on
small organizations.

In chapter 3 we discussed the concept of quality. We constructed a 4-level model of quality

abstractions, which included:

Quality Definition. A quality definition describing goals.

Quality Model. A quality model documenting the requirements.

Quality Manual. A CQM that is implementing the requirements.

Produced documents.

In order to construct a CQM for small organizations, we must take the previous two levels into
consideration, thus the first task becomes to find a quality definition. From this quality definition,
we will construct a quality model and from the quality model build a quality manual. The idea is
that the quality manual should be constantly updated and improved, while the quality model is a
more stable document, which provides the ground on which the manual stands on. The quality
definition on the other hand, should very seldom be modified.

Why not use ISO 9000 or CMM? ISO 9000 suffers from a bad reputation. During our interviews,
nearly all the companies replied that ISO 9000 was “too much bureaucracy” or “too much
documentation”. But this does not have to be the case. One of the companies, which was on their
way to be ISO 9000-certified, showed us their quality manual, and it was as thin as a video
manual. But even though ISO 9000 might be better than its reputation, the reputation itself lowers
the use of it. It is also quite large, and it might be a big first step for a company with no formalized
quality organization.

The way that CMM divides companies into maturity levels makes it tempting to think that CMM
is excellent for small organizations, since companies of all sizes can “fit in” to the 5-level scale.
But the statistics presented in chapter 3 clearly shows that this is not the case. More than 50
percent of the companies does not even reach level 2! CMM might be good in a later stage, but the
first step is obviously too steep.

The existing quality models are thus unsuitable to be used as a first step. They might be useful in a
later stage, but in the initial phases of quality improvement, they seem to be too large and
incomprehensible. What we are looking for is something that can be used in the next (or even the
current) project, not something that takes years to develop.

8.1 Quality Definition

One of our hypotheses about problems in small organizations is that they are not considering the
long-term perspective of software development. This was strengthened by our empirical study. To
avoid that such problems is introduced by a weak quality definition we have chosen to slightly
alter the definition that companies seems to agree on. The definition is quite similar to one of the
Total Quality Management definitions, but we do not see this as a drawback. Our definition of
quality is divided into two parts. The first part is:

“Quality is long-term customer satisfaction, business success and continuous improvements.”

59
This definition includes 3 aspects of quality: business success, customer satisfaction and
continuous improvement. The company should keep their customers happy, make money doing so,
and keep getting better at it. Simply stating that quality is customer satisfaction might feed the
misconception that the customer should be happy only on the delivery day. The customer should
of course be happy on the delivery day, but the customer should also be happy a year after the
delivery day. The delivered product should withstand changing requirements, maintenance etc.
The long-term perspective must be reflected in the quality definition. This perspective is caught
with the “continuous improvements”.

To summarize, the three aspects of our first part of our quality definition are:

Customer satisfaction. Customers are satisfied when they receive the expected product and
service. The customer may expect the product or service to be delivered on time, within budget
and according to the requirements, but it does not have to be so. The customer probably expects
the written agreements to be fulfilled, but the contents of them depend on the customer and
project. It difficult to generalize on customer’s expectations, it is something that has to be explored
for every new customer.

Business success. Business success is the ultimate goal for most companies. It is crucial for its
survival to make money, but business success is not simply making quick money. Business success
is a long term survival and expansion of an organization.

Continuous improvements. It is important to make continuous improvements to the

organization’s quality plan. This is crucial for many reasons:

The quality manual must be flexible. If one part of it is not good, or not used, then remove it
or rewrite it!

Learning from previous mistakes is a sure way of becoming really good at something.

Processes that lead to mistakes must be removed or changed.

You gain a competitive advantage by always aiming for higher quality.

The world is changing.

The idea of continuous improvement does also prevent the idea that quality is something that can
be achieved – which of course it is not. If the organization believes that it has “achieved quality”
then what is the point in becoming better?

The quality models available does not simply concentrate on customer satisfaction; they are
concerned with, more or less, the entire organization. This is where the second part of our quality
definition comes in. It is not possible to achieve long-term success if half the organization is
counteracting the purpose. For example: if not enough effort is put into the well being of the
employees, they will not produce as well as they could potentially do. Our definition of quality for
the entire organization is a recursive definition:

“The quality of a system, is the quality of its sub systems.”

Quality can increase when each subsystem is increasing their quality. We believe that many
organizations, especially small ones, focus too much on one or two subparts, forgetting the other.
This is reflected in our hypothesis.

60
8.1.1 Conclusion
Our empirical study showed that companies have a shortsighted view of software development.
When me make our quality definition, it is important to take a more longsighted view of software
developing into account. We now have a quality definition, which is the first part of the 4-level
model. The quality definition is:

“Quality is long-term customer satisfaction, business success and continuous improvements.”

And:

“The quality of a system, is the quality of its sub systems.”

The task now becomes to define the system, i.e., the organization, and its subparts. This will
become our quality model in the 4-level abstraction from chapter 3.

8.2 Quality Model

The purpose of this chapter is to describe a Quality Model that we can use to build a Quality
Manual from. The model should build on the Quality Definition that we defined in chapter 59. It
must also be as simple as possible so that it can easily be incorporated in a small company. To
fulfill this, we make a model of a small company, including all the parts of the company. Then we
set up quality values for each part of the company, so that we can fulfill the definition that: “The
quality of a system, is the quality of its sub systems.”

This philosophy is quite different from for example ISO 9000 or CMM, which merely lists the
requirements of the quality manual [PAULK+], [TINGEY]. We have chosen not to do this,
because we want a more flexible model, which suits the dynamic structure of a small organization.

8.2.1 The Organization

An organization, in our model, consists of producers, owners, customers and products.

Producers:

Employees (developers)

Management (short sighted economical interests)

Project leaders

Owners:

Owners (long sighted economical interests)

Customers:

Customers, in a very wide sense of the concept. Customers are those that are buying the
product and the people that are going to use the product.

Product:

One or more software product(s)

In small organizations, one person may have many roles, for example being an owner, an
economical manager and a developer. Each role has its own perspective of quality.

61
8.2.2 Perspectives of Quality
The concept of quality depends on ones point of view. The various stakeholders have different
quality expectations:

Employees: The employees, especially the developers, are an essential part of a software
producing organization. The employees of the organization value personal growth, a comfortable
working environment and a reasonable salary.

Management: Economical managers have economical responsibility for one specific software
project. Their values include a low price, a controlled development process that allows for
planning and effective development.

Project leaders: Project leaders value well-defined specifications, sufficient resources, time and
clear goals. They want control of the project.

Owners: The owners make long term investments in the organizations, and therefore their values
include making a profit, having a clear strategy and satisfied customers.

Customers: The customer is the strongest driver for quality. Without a customer, quality would
not even be required. The customer wants the expected software at the expected time and for an
expected price.

Product: The product does not, of course, have any values of its own. There are however
attributes of the product that are not of main interest of the customer. Such attributes are testability,
maintainability, reusability etc. These internal requirements are easily forgotten if too much focus
is spent on ‘customer satisfaction’.

According to our quality definition (see chapter Quality Definition above), a system has quality
when its sub systems have quality. In our model, this means that when all quality perspectives are
taken into account, the result is an organization that can fulfill the other quality definition:

“Quality is long-term customer satisfaction, business success and continuous improvements.”

8.2.3 Conclusion
Since we have shown in chapter 3 that the definition of quality is dependent on the subjective view
of the person making the definition, we built our quality model around a model of an organization
that suits most small companies. This will ensure that, as many views of quality as possible will be
taken into account. We then set up quality values for each subpart of the organization, thus
ensuring that each part of the organization can have its view of quality fulfilled. This leads to
conflicts – it is not always possible to satisfy everybody.

8.3 Quality Manual

It has been shown that small companies have weaknesses in the following activities: project
planning, selecting technical solutions, configuration management and testing. This chapter
provides a suggestion for an initial quality manual that attempts to reduce those problems. A
complete CQM would have to cover many more areas that those that we are suggesting here, but
we focus on the issues that was found during the interviews. The processes suggested here is a
suggestion of what processes could look like, they are by no means absolute.

It is important that the manual is simple to use and can easily be extended. It should be used as a
first step towards a bigger manual for a small company, and can not bring with it a lot of extra
“unnecessary” work. Otherwise the manual risks ending up on the shelves collecting dust. “A
small organization can lose the spirit of the CMM (or any other process model) while attempting
to apply the model to the letter, introducing excessive documentation and formality that can

62
 impede project work” [WIEGERS]. It is also important that the manual can be used directly in the
organization, preferably in on-going projects. Many of the companies that we interviewed said that
they were about to develop a quality manual for their organization, but they did not take the first
step towards doing so. By keeping the manual small, the first step becomes an easier one.

One of the interviewed companies that was in the process of developing a CQM of their own
claimed that processes should be documented as flowcharts. The reason to this was to reduce the
amount of text and thus making the CQM smaller. This is probably an excellent way to document
the CQM; we have however documented our process suggestions as text. The processes are in the
following format:

Name: The name of the process.

Forces: Why does the process exist? It is important that the people using the process understand
why it must be followed.

Preconditions: What needs to be done in order to start the process.

Process: A list of activities that may or must be followed.

Result: The expected result of the process.

Rationale: Explains why the process looks like it does and how it relates to the quality model.
Which subparts of the organization in our quality model are involved? How are their quality
values affected? It also contains references to the interviews.

8.3.1 Divide and estimate

Name: Divide and estimate

Preconditions: None.

Assisting material: Data from previous similar projects.

Process: The Project Managers and the Employees have a meeting in which they together divide
the project into as small activities as possible. To make the estimates more accurate, let all the
people estimate on all the activities. Then note invariance and let the people who made the
estimates argue for it. Modify the numbers and repeat this until consensus has been reached. The
person who is responsible for the activity always has the final say. This should be done once, at the
beginning of the project, and at major milestones. The estimation meeting should not take longer
than approximately 1-2 hours.

Activities are only a few days worth of time and defines to be ‘done’ or ‘not done’
[MCCONNEL]). When all activities are short enough, and the participant’s feel assured that no
activities have been missed, the process is finished.

Result: A list of activities, and a time for each activity.

Rationale: By dividing the project into smaller activities, estimating becomes simpler. It is also
easy to forget large parts of work, if the work is not divided into smaller tasks. Estimating the time
each activity will take is crucial for a realistic project plan.

The Employees must know what to do and what is expected from them. By being a part of the
project planning they feel that they have a saying in what they are supposed to do. The interviews
showed that this was not always the case.

63
 Project Managers must be able to make correct plans. By asking the Employees the plans will
most probably be more correct. This means that the Managers can give the Customers accurate
price estimations.

It also becomes simpler for new Employees to make correct estimations if they make the
estimations in the company of more experienced people.

8.3.2 Progress reporting

The interviews showed that some companies totally neglected to measure progress in the projects.
By not doing this, they risk that the project is running out of control. A project gets late one hour
at the time.

Name: Progress reporting

Preconditions: A list of activities.

Assisting material: None.

Process: A project plan must be kept, and updated, during the entire project. If an activity takes
too long time, it must be re-estimated and the plan must be updated accordingly.

An activity has two states: Working and Finished.

Result: A project plan that is consistent with the actual progress of work in the project.

Rationale: During the project, it is important that activities does not take too long time. This will
make the entire project later. The plan must be updated during the entire project.

8.3.3 Milestones and deadlines

Two of the interviewed companies did not have any deadlines. Overtime was used a lot during
some periods of time in two companies. Planning was proven to be a problem in the companies we
interviewed. One of the companies that we asked for interviews could not even participate because
they were working too much.

Name: Milestone and deadlines

Preconditions: A project plan.

Assisting material: Data from previous similar projects.

Process: The project is divided into a number of deadlines. Each deadline is divided into a range
of miniature milestones. The difference between deadlines and miniature milestones is that the
deadlines are project-wide while the miniature milestones are personal for each Employee or for a
small team of Employees. Each deadline and miniature milestone should have a clear definition of
what needs to be done in order to pass the milestone or deadline.

Result: A list of deadlines and miniature milestones, and the requirements for each.

Rationale: By focusing on miniature milestones, the project is kept on track and keeps the correct
focus. Deadlines help increase the pressure on Management and Developers. It is however very
important that the Developers have a say when deciding the deadlines.

64
 By keeping miniature milestones, the Developers know what is expected of them. Since they have
themselves agreed on the milestone, a commitment culture is grown. Managers can more easily
track the progress of the project by seeing how many milestones are kept.

If a deadline is not kept, precautions must be taken. It is dangerous to believe that the time lost
will be regained in a later stage [MCONNELL]. More likely, the entire project will be delayed by
the same percentage of time that the deadline was delayed. The best way of ensuring that
deadlines are kept is to remove functionality requirements [MCONNELL]. The interviews showed
that this is seldom done. In stead, overtime and stress is used to try to get the project on its feet.
This will lead to lower quality of the Product and burned out Employees.

8.3.4 Evaluation technique

Many of the companies in the interviews used the same tool for many different tasks. They also
used tools from a specific vendor consistently. As a small company, this might be the simplest
solution. But in the long run, it is important to keep an open mind towards new technologies.

Name: Quantify the Needs

Preconditions: The system requirements of the system.

Steps performed: Make a matrix with different tools on the Y-axis and non-functional
requirements on the X-axis. Examples of non-functional requirements include performance, ease-
of-use, portability, maintainability etc. Put grades on each tool and non-functional requirement.
Base the grade on experience, reviews from technical magazines and where it is possible,
experiments and facts. All the people involved in using the tool should be able to grade it. The tool
with the highest score is the most suitable technical solution. The grades may be weighted if some
requirements are more important than other are.

Result: A matrix with grades of the different tools.

Rationale: Selecting a good technical solution can greatly enhance the quality and lifetime of the
Product.

8.3.5 Configuration management

Only one of the companies used a CM tool, the other companies used local backups or relied on
system backups for older versions of files. It is our conclusion that a tool can greatly improve or at
least simplify CM. A tool alone does not provide perfects CM, but it is a step in the right direction.

Name: Configuration Management

Preconditions: A CM tool simplifies matters a lot.

Steps performed: Every project has a list of configuration items, which must be under CM tool
control. All of these items must have a version number, and changes to them must be documented.
After a delivery (see Milestones and deadlines above), all items included in the are labeled with an
unique identifier. This enables the organization to retrieve old deliveries.

Result: N/A

Rationale: The Employees will have better control of the configuration items. Managers and
Employees will be able to trace changes on the items.

65
8.3.6 Line coverage
Name: Line coverage

Preconditions: Software source code has been produced.

Steps performed: A set of scenarios is executed through a tool that monitors the level of line
coverage. It is likely that all lines of code can not be executed at once. It is not a goal to reach 100
percent coverage since this is often not realistic. Coverage as close to 100 percent as possible
should be a goal, though.

Result: Software code that is ready for statistical testing.

Rationale: Line coverage is the method by which one ensure that all lines of software code has
been executed at least once. This method does naturally not find all defects, but there is very good
tool support, which makes the activity easy to perform. It thus makes improvements for the
Developers, making it easier for them to find errors, and increases the quality in the Product.

8.3.7 Statistical testing

The second most problematic aspect of software development is ensuring that “The program does
not crash”. The most common strategy to prevent program crashes was to perform software tests.
Most people interviewed said that it is difficult to test the system the way that the user is going to
use it. “The user is incredibly creative in finding ways to use the system that we could not
anticipate”.

Name: Statistical testing

Preconditions: Software deliverables has been produced.

Steps performed: A model over usage behavior is established, this model is a Markov diagram
with probabilities on the arcs. The model is then used to ‘generate’ test specifications that are
similar to the way the user would use the products.

Result: A document describing the defects found.

Rationale: The idea behind statistical testing is to test software the way that the user is using the
system. This results in that the most important defects are found first, thus improving quality in
the Product.

8.3.8 Peer reviews

Being able to reduce the number of defects found in the deliverables could potentially reduce the
testing effort. Full-blown software Inspections as [GILB+] describes are probably too much for
smaller software developing organizations, but the peer reviews of CMM [PAULK+] are probably
realistic. The reason for reviews is that no organization interviewed tested anything but the
binaries. What work items that are subject to Reviews is defined by the organization.

Name: Peer reviews

Preconditions: Software work item has been prepared, along with a checklist containing
information on what defects to look for.

Steps performed: The person(s) performing the review checks the work item for the possible
defects described in a checklist. The defects that he/she finds are documented in a defect log.

Result: Document describing defects found.

66
 Rationale: It has been shown [GILB+] that reviews or inspections greatly increase the quality of
the work produced. It makes the Product better, and improves the work performed by the
Developers.

8.3.9 Conclusion
This chapter presented a Company Quality Manual to be used as a first step towards a process-
oriented approach of working. It contained processes for all the areas under which the empirical
study showed that small companies fall behind on.

8.4 Conclusion
The goal of this thesis was to present a quality manual made for small organizations that are
developing software. To be able to do this, an empirical study was performed, and it shows that
small companies have weaknesses in the following activities: project planning, selecting technical
solutions, configuration management and testing. From the conclusions drawn from this, we
constructed a quality framework that was presented in this chapter.

The quality framework consisted of a quality definition, a quality model and a quality manual. The
manual should be used as a first step, and will probable have to be reworked to fit into the
organization that uses it. This is not a drawback, though. A quality manual must be constantly
updated and improved. The quality framework presented in this chapter should be used as a very
first step only. It is more important to begin than to get it right the first time.

67
9 Future Work
This chapter gives suggestions as to what future work could be interesting in the light of this
thesis.

9.1 The CQM

Our proposal for a CQM has not been tested in the industry. It would be interesting to implement
this into some small organizations and observing its usefulness. The findings would be use to
iterate over the design of the CQM in order to produce one that is more efficient for its purpose.

9.2 Comparison to a big company

The intent is that the CQM should be used in a small organization. But how does a larger
organization differ from a small one in this context? Are the same problems applicable there?

9.3 The next step

This thesis provides the first step in moving towards higher quality for a small organization. But
there is still a long way to go before all the requirements for an ISO 9000 certification can be
achieved. A proposal for future work is to make The Next Step, which would include a more
detailed CQM, possibly including activities such as Risk Management, Software Metrics and
Inspections.

68
10 References
[BECK+] Back, Leland L., and Thomas E. Perkins; A Survey of Software Engineering
Practice: Tools, Methods, and Results; 1983; IEEE Transactions on Software
Engineering SE0

[BEIZER] Boris Beizer; Software System Testing and Quality Assurance; 1984; Van
Nostrand Reinhold

[BOEHM] B. W. Boehm; Software Engineering Economics; 1981; Prentice-Hall, Inc.

[BROCKA] Bruce Brocka and M. Suzanne Brocka; Quality Management: Implementing

the Best Ideas of the Masters; 1992; Irwin Professional Pub

[BROOKS] Brooks, Fredrik P; The Mythical Man Month: Essays on Software Engineering;
1995; Addison-Wesley Pub Co.

[BROWN+] Brown, William H. Brown, William J Malveau, Raphael C; Antipatterns; 1998;

John Wiley & Sons (C)

[CONNELL+] John Connell, Linda Shafer; Object oriented rapid prototyping; 1995; Yourdon
Press

[CROW] Kenneth A. Crow; Improving Time-to-Market Through Planning and Resource

Management; 1996; DRM Associates (http://www.npd-
solutions.com/resources.html)

[EDVARDSSON] B. Edvardsson, B. Thomasson; Kvalitetsutveckling : ett managementperspektiv;

1992 ; Studenlitteratur AB

[EJVEGARD] Rolf Ejvegard; Vetenskaplig metod; 1996; Studentliteratur

[ELY+] Margot Ely et.al; Kvalitativ forskningsmetodik i praktiken. – circklar innom

circlar; 1993; Studentliteratur

[ENCYC] John Marciniak (editor); Encyclopedia of Software Engineering; 1994; John

Wiley & Sons, New York

[FEIGENBAUM] A. Feigenbaum; Total Quality Control; 1991; McGraw Hill

[FEILER] Peter H. Feiler; Software Configuration Management: Advances in Software

Development Environments; 1990; Carnegie-Mellon University, Software
Engineering Institute

[GARVIN:1] David Garvin; What does product quality really mean?; 1984; Sloan
Management Review, 26,1, 1984

[GARVIN:2] David Garvin; Five Definitions of Quality; 1988; Free Press

[GILB] Tom Gilb; Principles of Software Enginnering Management; 1988; Addison-

Wesley

[GILB+] Tom Gilb, Dorothy Graham; Software Inspections; 1996; Addison-Wesley

69
[HAMMER+] Michael Hammer, James Champy; Reengineering the Corporation : A
Manifesto for Business Revolution, 1994; Harperbusiness

[HAMLET] Dick Hamlet; Are we testing for true realiability?; 1992; IEEE Software

[HUMPRHEY] Watts S. Humphrey; Introduction to the Personal Software Process; 1996;

Addison-Wesley

[ISOTC176] ISO/TC 176/SC 2/WG 15/N131;Draft Brochure on Quality Management

Principles and Guidelines On Their Application; 1997; ISO

[IVERSEN+] Jakob H, Iversen, Peter Axel Nielsen, Jacob Norbjerg; Problem Diagnosis in
Software Process Improvement; 1998; CTI, Technical University of Denmark

[JONES:2] Capers Jones; Patterns of Large Software Systems: Failure and Success; 1987;
IEEE Software

[JONES] Capers Jones; Assessment and Control of Software Risks; 1994; Yourdon Press

[KLEIN] Robert A. Klein; Achieve Total Quality Management; 1991; Chemical

Engineering Progress

[KOTONYA+] Gerald Kotonya, Ian Sommerville; “Requirements Engineering”; 1998; John

Wiley & Sons Ltd.

[LIBERTY] Liberty, Jesse; Clouds to code; 1997; Wrox Press Inc.

[LJUNGBERG] Lars Ljungberg; Buzzwords in Software; 1999; Q-Labs

[LÖÖV+] Peter Lööv, Gunnar Arnberg; Approaching CMM Level 2; 1996; Department of
Computer Science and Business Administration, University College of
Karlskrona/Ronneby

[MAGUIRE] Steve Maguire; Debugging the development process: Practical Strategies for
Staying Focused, Hitting Ship Dates, and Building Solid Teams; 1994;
Microsoft Press

[MCCONNELL] Steve McConnell; Rapid Development; 1996; Microsoft Press

[METZGER] Philip Metzger; Managing a Programming Project, 2d ed; 1981; Prentice Hall

[PAULK+] Mark C. Paulk, Bill Curtis, Mary Beth Chrissis, and Charles V.
Weber.;Capability Maturity Model for Software;Version 1.1. CMU/SEI-93-TR-
24. Pittsburgh: Software Engineering Institute;1993.

[PERPER] Binnie Perper; How to sell quality... in other words. 1996; Writing By Design

[STANDISH] Sample research paper (http://www.standishgroup.com/chaos.html); The

CHAOS Report; 1995; Standish Group

[TINGEY] Michael O. Tingey; Comparing ISO 9000, Malcolm Balbridge and the SEI
CMM for software; 1997; Prentice Hall

[TOMAYKO] James E. Tomayko; Software Configuration Management; 1990; Carnegie-

Mellon University, Software Engineering Institute

70
[WIEGERS] Karl E. Wiegers; Software Process Improvement: Eight Traps to Avoid; May
1996; Software Development Magazine

[ZAHRAN] Sami Zahlan; Software Process Improvement; 1998; Addison-Wesley

[WIEGERS] Karl E. Wiegers; Software Process Improvement: Eight Traps to Avoid; May
1996; Software Development Magazine

71
11 Definitions

11.1 Small organization

This thesis studies small organizations. By this we mean small software developing organizations.
Small in this context are less than 20 persons actively taking part in development of software
products. An organization is not necessarily a company, there can be several organizations within
a larger company. The studies companies consisted of only one organization each, though.

11.2 Quality
There are many definitions of the quality concepts, some of them are discussed in the quality
chapter. Other quality related concepts that are defined in that chapter are system, framework and
model.

11.3 ISO 9000

When we refer to ISO 9000 in this paper, we refer to the family of such quality certifications, with
ISO 9000-3 in particular.

72
12 Appendix A - Questions
This section contains the actual questions that are asked during each interview. The headings serve
the purpose of organizing the questions, but these will not be communicated to the interviewee.
Each question is commented for the purpose of that question, we will use these comments during
the interview to clarify if necessary. These comments will also be used when the interview is later
analyzed.

The questions generally ask for consciousness about software development problems. There is
only time for one hour of questions so it is not possible to go to any depth in any area. The
strategy then is to ask for issues that either we have personal experience from or are discussed in
frameworks such as CMM.

12.1 Initial
These questions are discussed initially. They do not serve any scientific purpose but they are here
to get the discussion going.

Describe your role in the organization, time of employment, education?

What is the average level of education in the organization (average number of study points)
(what type of people are working within the organization. Are there mostly self-learned, or do
they have some formal education?)

12.2 View of software

What is quality? Many organizations say proudly that they are developing high quality
products. If they have a real interest in high quality, then they should be aware of what quality
is, or at least given the concept some thought.

What are the problems that affect product quality? We are interested in the spontaneous
answer to this question. What aspect of software development is perceived to be most
difficult? If the spontaneous answer is “bugs”, then the organization might be fire fighting. If
requirements and understanding the customer is central then the organization probably have
reached a higher level of maturity.

What is your relation to ISO 9000? Why? (It is probably safe to use ISO 9000 as a concept,
everyone has heard of it?) We need to know the perceived problems with existing models, and
then we know what is expected of our model. It is also interesting to find if the feeling about
ISO 9000 is based upon prejudices and hand on experiences.

Have you studied other certification frameworks? Which? Having studied other quality
frameworks clearly indicated awareness of other frameworks. Such awareness can probably
be interpreted as more than brief interest in improving quality.

How do you ensure that the experiences from one project are taken care of in the next? If an
organization does not explicitly document their experiences from each project, then how do
they share their experiences? Experiences are one of the bases for improvement, and if that is
not taken care of, how do they improve?

73
12.3 Planning
How do you know what to do when you come to work each morning? Some organizations say
proudly that when you come to work each morning you never know what to do! This
indicates that there is little control over what and when activities are performed.

Do you know what to do next week, next month? How long in advance do the organization
plan its projects. It is probably better to have a small or weak plan than not having any at all.
However, the plan needs to be consistent with reality so it needs to be updated regularly. Do
the organization do this? Try to find out if they have longer plans than simply this project.
Short plans do basically have two implications; either do they not have any future customers,
or do they not believe to have any time except for the current project.

How do you know if you are going to meet a deadline? Are they measuring progress? Are
people reporting what and how much they have done. Have they been collecting some type of
metrics, or even experiences that help them to know how the project is going? The opposite is
a very naïve approach where all deadlines will be held.

What happens if you are late for a deadline? There are many possible strategies for managing
late projects, for example: Adding more people to the project and risking breaking Brooks law
[BROOKS], or simply ignoring the problem.

How much overtime does each employee use each month? Approximately none, 1, 2 or many
hours per month. Much overtime indicates that the organization have difficulties in planning.
If the answer is that it depends, then they are sometimes using very much overtime, and that
indicates poor planning.

Who is going to use the documentation? (Is the documentation actually used, or is it produced
because they believe that it is necessary?)

What happens if a given person gets sick or quits? (Are some people more valuable than
other? How do they manage the transition from one developer to another?)

12.4 Management
How come you are developing the type of products you are? (The thesis is that a person with
much domain knowledge once founded the organization. That person did then hire a set of
programmers to develop products. This indicates that the organization do not find software
engineering problems as main problems.)

What control does management have over the development? What role does the management
person have? Is he/she involved with the actual development, or does he/she only provide
customers and requirements specifications?

12.5 Selecting technical solutions

How do you select technical solutions? Is it a conscious choice, or are they using XYZ
because they have always done so.

How do you know if the product will work in practice? Have they performed some evaluation
among possible choices? Is it a rational choice or are they working with ‘Microsoft’.

74
12.6 Configuration management
How do you ensure that two persons are not working in the same file? This is CM in its purest
form. Lack of CM means that they spend time merging files every now and then. If they are
using some CM tool then they must be considered to be aware of this problem.

How do you do if you want to revert to last weeks state or that before lunch?

Can you reproduce an old version of a product? Being able to reproduce old versions means
that they have some formalized method for managing the versions that they deliver.

Do you know what version is installed where? Lack of management of installed versions may
lead to problems with updating client’s software.

12.7 Testing
What is a bug? (Defect? Error?) (Is there a difference between them? Is a bug only a system
crash?)

Describe the testing cycle. (Is there for example a preset stability ambition, or is the products
just tested until they do not seem to crash anymore.)

How do you ensure that old defects do not return? (Someone said, “all bugs have to be fixed
three times”, while this may or may not be true, lack of regression testing can harm the
project.)

Describe the defect cycle. (Is the new version sent to all the customers?)

12.8 Requirements
How do you know that you are developing the right application? What type of analysis has
been performed in order to reach the requirements specification? Is the organization simply
accepting customer requirements, are they developing prototypes, or what?

How do you avoid misunderstandings? The provider and the customer might very well reach
consensus, but how are they sure that they both agree on the same thing? From personal
experience we have found that if specifications are too technical the customer might not really
understand what they mean, but accept them anyway.

How are changing requirements managed? The requirements for any software product will
inevitably change [KOTONYA+]. These changes risk leading to feature creep. Projects that
fail to control creeping requirements are highly susceptible to excessive schedule pressure
[JONES].

75
13 Appendix B – Overview over the answers to the
questions
The answers of the questions have been generalized and are presented here. A person may have
answered many things, and a person may have not answered a question. It is noted for each answer
what organization and person that gave the answer, these are presented within brackets.

The answers to the questions concerning management varied from company to company and we
failed to find any interesting patters, and are therefore not listed in this chapter.

What is the average level of education in the organization?

Does not have any formal education, perhaps some courses [1, 3.1, 3.2, 4.1, 4.2 and 5]. The most
common reason is that there were no or little opportunities for formal education when they started.
Has an exam from a university [2.1, 2.2, 5.1]

13.1 Quality
What is quality?
The program does not crash [1.1, 2.2, 4.2]
Meeting customer requirements [2.1, 2.2, 3.2, 4.1]
Is difficult to grasp, or define [3.1]
Working according to principles and everything is documented [5.1, 5.2]
What are the problems that affect product quality?
Bugs and testing [2.1, 2.2, 3.1]
Requirements, understanding the customer and avoiding feature creep [1.1, 2.1, 4.1, 4.2]
Planning and documentation [2.2, 3.1, 3.2, 5.1, 5.2]
What is you relation to ISO? Why?
It is too bureaucratic [2.1, 2.2, 3.1, 3.2, 4.1, 4.2]
It would probably help us [1.1]
It is helping us [5.1, 5.2]
Something, but not ISO, would help us. [2.1, 3.1]
It does not help us to develop, just to document [3.1]
(The most common perception is thus that ISO is too bureaucratic. However as seen in the next
answer is that the people that find ISO that have not had any real experience with it either.)

Has actual experience of ISO [1.1, 5.1, 5.2]

Have not worked with ISO [2.1, 2.2, 3.1, 3.2, 4.1, 4.2]
Have you studied other certification frameworks? Which?
No [1.1, 2.2, 3.1, 3.2]
Yes (Not CMM, TQM or Malcom Balbridge though) [2.1, 4.1, 4.2, 5.1, 5.2]
How are you ensuring that the experiences from one project are taken care of in the next?
One learns new things during each project [1.1, 2.2, 5.1, 5.2]
Conclusion reports [2.1]
We do not [3.1, 3.2, 4.1, 4.2]

13.2 Planning
How do you know what to do when you come to work each morning? (Short term planning)
TODO lists [1.1, 2.1, 2.2, 3.1, 4.1]

76
 One person is responsible for delegating work tasks [5.1, 5.2]
Do you know what to do next week, next month?
There is no problem to know what to do [2.1, 3.1, 3.2, 5.1]
Project plans [4.2]
As it happens [4.1, 2.2]
How do you know if you are going to meet a deadline?
We have never been late for a deadline [1.1, 3.1]
Experience [3.1, 3.2]
Project plans and systems specifications [2.1, 5.1, 5.2]
You do not [4.1]
(No one answered that they measured progress)

(Company 1 and 3 did not plan for any deadlines)

What happens if you are late for a deadline?

Overtime [2.1, 2.2]
Discussion with the customer, cutting down requirements for example [2.1, 4.1, 4.2]
Take resources from other projects [5.1, 5.2]
How much overtime does each employee use each month?
Very much during some periods [2.1, 2.2]
We preferably work weekends [3.1, 3.2]
Does not work any (project-related) overtime [1.1, 4.1, 4.2, 5.1, 5.2]
What happens if a given person gets sick or quits?
Our time-plan is not so tight that we can not handle that [3.1, 3.2]
The organization is two or less [1.1, 4.1, 4.2]
Our quality system enables us the share work [5.1, 5.2]
Someone else has to take over [2.1, 2.2]

13.3 Selecting technical solutions

How do you select technical solutions?
We use applications from a major software vendor [1.1, 3.1, 3.2, 5.1]
It depends on the project and the customer [2.1, 2.2]
The technologies that we are experienced in [4.1, 4.2, 1.1, 5.2]
How do you know if the product will work in practice?
We do not know that [2.1, 2.2, 5.1, 5.2]
We use applications from a major software vendor [3.1, 3.2, ]
It is specified in the requirement specification [4.1, 4.2]

13.4 Configuration management

How do you ensure that two persons are not working in the same file?
We do not [1.1, 2.1, 2.2, 3.1, 3.2, 4.1, 4.2]
We are using a configuration management tool that ensures this [5.1, 5.2]
How do you do if you want to revert to last weeks state or that before lunch?
It is not possible [1.1, 2.1, 2.2, 4.1]
By restoring yesterdays backup [3.1, 3.2, 4.2]
From our configuration management tool [5.1, 5.2]
Can you reproduce an old version of a product?
Yes [1.1, 2.1, 2.2, 4.1, 4.2, 5.1, 5.2]

77
 No [3.1, 3.2]
Do you know what version is installed where?
Yes, we develop customer unique applications so it is easy to keep track of versions [1.1, 2.2, 4.2,
5.1, 5.2]
Yes, but we have had problems with that. [2.2]
No [3.1, 3.2]

13.5 Testing
Describe the testing cycle
There are no (or little) formal test-plans or specifications. Each developer is responsible for his/her
work [1.1, 3.1, 3.2, 2.1]
Alpha/Beta strategies [5.1, 5.2]
The person that has written the code does not test it. [3.1, 2.2]
There are test specification, with documented expected results [5.1, 5.2]
It is not possible to perform a complete system test. The user is incredibly creative in finding ways
to use the system that we could not anticipate [2.1, 2.2, 3.1, 3.2, 4.2]
How do you ensure that old defects do not return?
Once we have fixed a problem, it does not return [1.1, 3.1, 3.2, 4.1, 4.2]
We do not know if old defects return [2.2, 2.1]
We have a bug-tracking system, and write deviation reports [5.1, 5.2]
Describe the defect cycle
The customer reports the defect, and we fixe it as soon as possible [1.1]
The above but, if the defect is major then the test specifications have to be re-executed [3.1, 3.2,
5.1, 5.2]
(Most people did not answer this question)

13.6 Requirements
How do you know that you are developing the right application? (and how do you avoid
misunderstandings?)
Have long experience in the domain, so we know [1.1,3.1]
Prototypes [4.1, 2.1, 5.2, 1.1]
Meetings, email and phone [2.1, 2.2, 4.1, 4.2]
Documentation and specifications [2.1, 3.1, 5.1,5.2]
External standards [3.2]
How are changing requirements managed?
Have never experienced changing requirements [1.1]
That is regulated in contracts and specifications [4.1, 4.2, 5.1, 5.2]
There is no specific way of managing changes [2.1, 2.2, 3.1, 3.2]

78