Nine Steps to Delivering Defect-Free Software

Copyright © 1997, 1998 Terence M. Colligan

Hello. I am Terry Colligan, president of Tenberry Software, Inc. I have been a software developer for over 30
years, and have been managing software development for over 20 years. Tenberry Software (formerly Rational
Systems) has a reputation for producing high-quality software and for having extremely good engineers.

Although I thought I understood the importance of quality, and took pride in the quality of the software we
produced, I never believed that delivering defect-free software was possible. After all, everyone knows that all
software has lots of bugs, right?

Well, no, not necessarily! Certainly, most experiences with today's software quality are not encouraging.
Although few people can name even one piece of software which they use that has no bugs, defect-free software
is possible to create. We know it is possible, because we're doing it.

It started with a single engineer. This engineer was consistently producing work with a defect rate more than
one hundred times smaller than our other engineers. She has done so for us for over three years now. During the
same time, she has produced three to five times as much code as any other engineer.

I found this so exciting that I determined to find out how she did it, and to see if we could teach our other
engineers to achieve the same quality results.

I later discovered that one of my best friends, an independent consultant in the mainframe/Cobol world, has
been similarly producing defect-free results on his projects.

We have developed a process to produce guaranteed defect-free software. (We are continuing to refine our
process, but it works now.) To help improve general software quality, we are sharing the nine steps of our
process:

1. Believe Defect-Free Software is Possible

Surprisingly, the first reaction that I get when I describe Defect-Free Software is to be told that it's just not
possible. Defect-Free Software seems to be self-contradictory. Some folks even act as if "Defect-Free Software"
is an attempt at computer humor.

In fact, this attitude is the biggest obstacle preventing the delivery of defect-free software! The most striking
difference between the two defect-free engineers and our other engineers (including me!) is their attitude
towards software defects.

The average engineer acts as though defects are inevitable. Sure, they try to write good code, but when a defect
is found, it's not a surprise. No big deal, just add it to the list of bugs to fix. Bugs in other people's code are no
surprise either. Because typical engineers view bugs as normal, they aren't focused on preventing them.

The defect-free engineers, on the other hand, expect their code to have no defects. When a (rare) bug is found,
they are very embarrassed and horrified. When they encounter bugs in other people's code, they are disgusted.
Because the defect-free engineers view a bug as a public disgrace, they are very motived to do whatever it takes
to prevent all bugs.

In short, the defect-free engineers, who believe defect-free software is possible, have vastly lower defect rates
than the typical engineer, who believes bugs are a natural part of programming. The defect-free engineers have
a markedly higher productivity.
In software quality, you get what you believe in!
2. Think Defect-Free Software is Important
Why is defect-free software important?

Delivering defect-free software reduces support costs.

Delivering defect-free software reduces programming costs.
Delivering defect-free software reduces development time.
Delivering defect-free software can provide a competitive advantage.

If you (or your manager) don't think delivering defect-free software is important, you won't spend the effort
necessary to deliver it.

3. Commit to Delivering Defect-Free Software

In the past, I was the single biggest obstacle to producing defect-free code at Tenberry. Because I didn't really
believe that defect-free code was possible, I made decisions primarily focused on short schedule times.

In retrospect, virtually every decision against trying for defect-free and in favor of short schedule time was
wrong and resulted in longer schedules, more bugs, more support, higher costs and smaller profits!

Making a firm commitment to defect-free code and holding to that commitment, in spite of schedule and other
pressures, is absolutely necessary to producing defect-free code.

As a nice side benefit, you will see improved schedules and reduced costs!

4. Design Your Code for Simplicity and Reliability

After attitude and commitment, program design and structure have the biggest impact on defect-free code. A
clean, well structured design simplifies producing reliable code. A poor design cripples the engineer, and will
make it impossible to achieve defect-free code.

Each function should be precise -- it should have only one purpose. Each action or activity should be
implemented in exactly one place. When programs are structured this way, the engineer can easily find the right
place to make a change. In the unlikely event that a bug is discovered in testing, the engineer can go directly to
the code with the defect and promptly correct it. This saves time and is the major cause of the faster schedules
experienced with Defect-Free Software.

In addition to designing for clarity, it's important to keep the defect-free goal in mind. You want to choose
designs that will be least likely to have bugs. In other words, avoid tricky code. Don't start to optimize code
unless you are sure there is a performance problem.

5. Trace Every Line of Code When Written

To me, one of the most surprising techniques used by our defect-free engineer was the deliberate tracing in a
debugger of each line of new code when it is written.

As each line of code is about to be executed, you should try to predict what the effect will be -- what data will
be changed, which path a conditional will follow, etc. If you can't predict what the effect will be, then you don't
understand the program you are working on -- a very dangerous situation. If you don't predict correctly, you
have probably discovered a problem that should be addressed.

Tracing all new code shows:

Code that hasn't been tested. By stepping through each line of code, you ensure that the new code is fully tested.
Inefficient code. For example, calling the same function multiple times. The first time engineers watch a trace
of their code, they are often surprised to see exactly how much redundant and inefficient code is executed.

Code that is working for the wrong reason.

Confirmation that the design is functioning as intended.

Tracing all new code will ensure that your code will be tested and is functioning as designed -- both important
characteristics of defect-free code.

6. Review Code by Programmer Peers

Peer code reviews have consistently been shown to be the single most cost-effective way of removing bugs
from code. The process of explaining a new section of code to another engineer and persuading that second
engineer the code is defect-free has several positive impacts:

Exposes the design and implementation, with benefits similar to tracing the code.

Forces the engineer to articulate assumptions. About ten percent of our code reviews are stopped in progress as
the authoring engineer suddenly says, "Oops! Never mind!" because he suddenly realized that he had made an
invalid assumption. (The review later resumes with the revised code.)

Allows more than one engineer to look at the code while it can still be easily changed. Code will be made
simpler and easier to understand.

Encourages cross-training and sharing of techniques. By discussing design strategies and implementation
techniques, each engineer learns from the experience of their peers.

Peer code reviews seem to work best. Code reviews done by managers or senior technical staff can have some
of the same benefits, but sometimes are less effective due to the interpersonal dynamics.

7. Build Automated QA into Your Code

Obviously, to build defect-free code, you have to be able to test your code. In addition to including a testing
plan/strategy into the implementation, you should design specific code to provide for full, automated testability.

The most effective testing we use is fully automated or regression testing. This is a series of fully automated
tests that are run after each build of a program. The tests are designed to exercise every part of the program, and
produce a success/failure report. The idea is to use the power of the computer to make sure that the program
hasn't been adversely affected by a change.

If the design is well structured, most changes should not have side effects. The purpose of these automated tests
is to provide insurance that the coding assumptions are valid, and that everything else still works. By making
the tests completely automated, they can be run frequently and provide prompt feedback to the engineer.

If tests are run by manually testing the program, we have the chance of human error missing a problem. Manual
testing is also very expensive, usually too expensive to run after every change to a program.

There are a number of commercial testing tools available which are designed to help you automate your testing,
particularly in GUI environments such as Windows. Although they are no doubt better than manual testing, we
have not found them to be effective, for a number of reasons. (For more details, check out our automated testing
strategy.)
By building support for automated testing into your program, you can approach 100% automated testing.
Without this customized, built-in testability, you will be lucky to achieve 35% automated testing, even with the
best commercial QA testing tool. We recommend that you budget five percent of total engineering time to
creating support for automated QA testing.

Of course, each new piece of code should have a corresponding set of tests, added at the same time as the code
is added, for the automated QA suite.

In order for fully automated execution of testing to be of value, the tests that are automatically executed and
checked must cover the software fully. To the extent that they don't, running the tests doesn't tell you anything
about the part of your software that wasn't exercised by the testing. (This is true for all testing, whether
automated or manual.)

8. Build and Test Daily

Once you have a fully automated test suite, you should run it after every build. This gives developers feedback
about the changes they are making, and it gives management clear, objective feedback about the project status.

Clear, objective feedback about project status help managers make better estimates and plans. This feedback
can help you identify and address problems while you still have time to do something about them. In addition,
this clear, objective feedback puts managers in a better position to provide correct feedback to their managers
(or shareholders). Finally, this objective feedback helps managers decide when a project can be shipped or
deployed.

The more prompt the feedback to the programmers, the more useful it is. The shorter the time between the
creation of a defect and its discovery, the easier it is for the programmer to understand just what they have done
wrong. Prompt feedback of failing tests can work as a kind of positive reinforcement for development
techniques that work and negative reinforcement for techniques that don't.

By automating the build process as well, you can schedule builds of your system daily. By building daily, you
will maximize the feedback to both your programmers and your management.

9. Use Automated Checking Wherever Possible

There are a lot of existing tools that can be used to find errors in your code in an automatic or semiautomatic
manner. Your programmers should be using these tools wherever possible.

These tools should be used in addition to the clean design, rather than instead of. No matter how much you use
automated checking tools, using these tools alone will never turn poorly designed, buggy code into defect-free
code. You can however, find a lot of bugs that would otherwise take much more time and effort to find and fix.

Useful automated checking tools include:

Setting your compiler's warning level to maximum.

Using the memory trace subsystem, such as Microsoft's MFC/DBGWIN combination, that may be provided
with your compiler.

Lint. This is a program which does extra fussy checking of C code. Some people find it very helpful. There is a
PC version and a C++ version available from Gimpel Software.
These kinds of tools are important, particularly for catching the kinds of errors that don't have obvious
symptoms, such as memory leaks.

Summary. That's the overview of how we create defect-free software. Obviously, there is a lot of work
involved. There are also lots of details that will need to be adapted to your specific situation.

Applying these defect-free methods to an existing program will be worthwhile as well. Although it's harder to
achieve a totally defect-free result with existing code (usually due to the design), applying these steps will result
in a significant reduction in an existing program's defect rates.

You can deliver defect-free software -- all you have to do is demand it. By following these steps and working
constantly towards the defect-free goal, you will see more and more of your software become defect-free.

Tenberry helps companies deliver defect-free software in two ways: We produce defect-free software under
contract, and we consult with companies to help them produce their own defect-free software processes.