What Does It Cost to Fix a Defect?

By Johanna Rothman Summary: We all have different attitudes and policies toward finding and fixing defects. The choice about whether and when to fix defects depends upon many factors, one of the least understood being the actual cost of fixing a defect. In this wee !s column, testing expert Johanna Rothman shares a formula for calculating the system test cost to fix defects and how to factor that into the bigger picture of your pro"ect.

#an is a developer wor ing on a pro"ect, along with four other people. They spent the first eight months of the pro"ect developing the product without fixing their defects, unless the defect prevented them from moving forward with development. #an and his team thought it would be more cost$ effective to fix their defects all at once. %o at month nine, about a month away from the desired release date, they decided it was time to fix the defects. &very is a pro"ect manager at a company with a virtual loc on the mar et. Because the customers are captive, each customer wants a beta copy immediately, so they can start using the software. 'oncerned that a beta with many defects would ma e their customers angry, &very decided it would be more cost$effective and less ris y for the developers to find and fix the ma"ority of the defects before they started system test. Two pro"ects, two completely different approaches toward finding and fixing defects. We all have different attitudes toward fixing defects, specifically which ones to fix and when. The choice to fix or not to fix depends upon many factors( the ind of product you!re developing) the ris s associated with shipping nown or un nown defects) your development processes) and what it will cost you to fix the defect when you do fix it. *ne of the least understood factors is the actual cost of fixing a defect. This cost feeds bac into your choice of development lifecycle and development process and can help you decide how much ris you!re willing to ta e to ship or not ship the product. +owever, many people don!t actually now what it costs their organi,ations to fix a defect. If you!re not sure either, here is an estimation techni-ue to measure that cost( In system test, when people are .// percent dedicated to finding and fixing defects, count the number of fixes. 0ou now how many people 1developers, testers, and anyone else2 wor ed on the pro"ect, and you now the duration of system test. This allows you to calculate the cost to fix a defect during this phase of the pro"ect. +ere!s how you can find the average cost to find and fix a defect(

&verage cost to fix a defect 3

14umber of people 5 number of days2 5 cost per person$day 14umber of fixed defects2

4ote that the number of defects you find isn!t enough information6it!s the number of defects you fix. #etecting defects is only the first step. 7ocating the failure, deciding how to fix it, developer testing 1a. .a. unit testing2 the fix, system testing the fix, and loo ing for other defects this fix caused is why the fix value is what!s important. 7et!s loo at some examples. In these examples, I assume a person$ day is 89//. #an!s pro"ect is uncovering large numbers of defects now that they!re in system test, and even though most of the defects are -uic to fix, some ta e very long. &very!s pro"ect is uncovering very few defects in system test, but because there!s such a long time between the time each defect is uncovered, it appears to ta e a long time to fix a defect. :sing the calculation above, here are #an and &very!s data for system test(

'ompany 1for a specific release2 #an &very

4umber 'ost per of people

4umber 4umber test fixes

person$days &verage days to fix during

%ystem test cost to fix a

person$day of days of system

system test defect 9 ./ 89// 89// ;/ </ .<9 ?/ <// <// ..= =.@ 8>// 8?,???

&s long as you!re loo ing at the whole picture of the pro"ect, this estimation metric is good for system test fix cost. But, loo at how high &very!s cost to fix is. In reality, &very!s pro"ect met its beta release date 1</ wor days in system test2, at a very low ris of disappointing the customers. #an!s test time too two months 1;/ wor days2, and even though the team has fixed .<9 defects, they have over ?// defects not fixed. &very!s cost$to$fix is high, because his team wor ed hard to prevent defects before system test. In fact, using the estimation techni-ue above, &very!s cost to fix a defect is highly inflated in system test, because they found and fixed most of the defects beforehand. Because &very!s pro"ect detected and fixed most of the defects before system test, the estimation techni-ue is not valid. Instead, &very!s pro"ect can calculate the actual cost of finding and fixing their defects. &very uses his average of > hours system$test time to find and fix a defect. +ere!s the table with &very!s more realistic estimate of system test cost(

'ompany 1for a specific release2 #an

4umber 'ost per of people

4umber 4umber test fixes

person$days &verage days to fix during

%ystem test cost to fix a

person$day of days of system

system test defect 9 89// ;/ .<9 <// ..= 8>//

&very

./

89//

</

?/

<//

../

8<//

:sing the updated data, here!s a clearer picture of the cost to #an and &very!s pro"ects to fix a defect(

Implementation Costs Aro"ect 4umber 'ost per of people on the pro"ect #an &very 9 ./ 89// 89// not trac ed < hours not trac ed 8.<9 not trac ed <9/ not trac ed 8?.,<9/ &verage time to Implementation 4umber of cost to fix implementation fixes Implementation %ystem test

person$day fix for

Implementation total cost before

System Costs &verage time to fix during %ystem to fix a 4umber %ystem test total cost %ystem test fixes .<9 8.//,/// ?/ 8=,///

test cost of

system test defect ..= ../ 8>// 8<//

Post-Release Costs Are$release &verage time Aost total cost to fix post$ release Aost$release Aost release Total pre and post release

release number of cost to fixes fix

total cost cost

8.//,/// .9 person$

8@,9//

<? 8.@<,9//

8<@<,9//

days 8./?,>9/ 9 person$ days &very has high system test costs, because his pro"ect spends more time loo ing for defects than fixing defects +owever, &very!s total defect fix cost for &very!s larger pro"ect is lower than #an!s smaller pro"ect. &nd &very!s post$release fix cost is substantially lower. Bach pro"ect will have its own cost to fix a defect, because the cost depends on the activities underta en in the pro"ect and when you start trac ing defects, as well as cost to fix. :se your fix cost to decide how you want to proceed with this pro"ect or the next one. If your cost is too high, and you!re not yet in system test, you could try some defect detection and prevention techni-ues. Just ma e sure that if everyone is associated with finding and fixing defects that you don!t only count the fix time, that you count the detection time also. If your find$and$fix cost is high in system test, what!s the ris of releasing earlyC &very might have used his find$and$fix cost of 8???? to chose to end system test early and release early, nowing that his post$release cost would rise. *nly &very and his management could estimate the ris of releasing early. :se the pre$release fix costs to see if you and your staff are being cost$effective in your pre$release activities. I!ve found that each organi,ation has a typical post$release cost, not necessarily tied to the pro"ect. %o I use the post$release cost to help define release criteria. This helps you manage the ris of too many defects to fix after the release. Dnowing how much it costs you to find and fix a defect allows you to as -uestions about how you!re finding, fixing, and verifying defects. 0et another way to build a system with the appropriate -uality. 8<,9// < 89,/// 8./>,>9/

About the Author Johanna Rothman observes and consults on managing high$technology product development, loo ing for the leverage points that will ma e her clients more successful. Johanna was the 'onference 'hair for %EB!s %oftware Fanagement 1%F2 'onference 1..$.9 Geb.2. %he conducted a management$improv tutorial and participated in a panel discussion of mentoring and manager$ ma ing at the conference. %he recently co$moderated a RoundTable HFa ing the Transition to FanagementI with Bsther #erby. 0ou can reach Johanna at "rJ"rothman.com or by visiting www."rothman.com. Bac to Top

Rea er Comments

&dd 0our 'omments

Bac to Top

I would li e to than the author for introducing a more up to date metric for calculating the cost of a defect. IKve always compared defects costs to the JA7 study from the early L/s, that for every earlier stage in the development cycle one can catch and remediate a sw defect, a savings of ./x is achieved 1for e.g., ./,/// to fix a sw defect once in testing, down to 8./ if caught while writing the code itself2. IKm going to ta e another loo at the white paper, MGrom %oftware Euality 'ontrol to Euality &ssuranceM with renewed eyes. Implementing proactive -uality assessment and assurance tools, plus the use of impact analysis tools may save an organi,ation even more money than I once imagined. Than s again.....1/<N</N/<2 Alex Forbes &uthorKs Response( &lex, when IKve measure the ratio of cost, it wasnKt a nice even .(./(.//(./// 1re-uirements(design(implementation and test(post$release2, it was more li e .(</(;9(<9/. But, these are my measurements only, "ust on the pro"ects IKve wor ed on, and IKm not an academic. *n the other hand, fixing a problem post$release, and having it cost <9/ times what it cost to fix in re-uirements was a large$enough hammer for me to use on my management, to change what we did for the next pro"ect.....1/<N<.N/<2Johanna Rothman While I applaud any effort to produce management$friendly 1read as money2 metrics, we have to be careful not to measure something "ust because we can. I have < concerns( .2 We all now that the cost to fix a defect s yroc ets after release, when all manner of damage control and recovery may be re-uired. Fa ing decisions based on how much it will cost to fix a defect does not ta e into account the cost of 4*T fixing it. <2 I agree with the earlier comment that only a thorough time$trac ing system can give you defensible metrics. *therwise, the sample si,e is usually so small 1statistically spea ing2 that minor variances 1someone ta es vacation or is out sic 2 can cause wide swings that render the results meaningless or at least sub"ect to debate.....1/<N</N/<2 Linda Hayes &uthorKs Response( 7inda, I first started using cost$to$fix post$release. Fy management thought it "ust too us a Mcouple of daysM to ma e fixes, and we could choose to fix numerous defects post$release. When I measured how long it too , it was roughtly ; person$wee s to fix each defect. I had a sample si,e of 9/ post$release defects for that release. Bven after that release, our post$release fix cost didnKt decrease much, but by changing what we did during the pro"ect, we were able to significantly reduce the number of defects going out to the field, and therefore the number of defects we needed to fix.....1/<N</N/<2Johanna Rothman It was fascinating to see the testing process -uantified li e this $$ much better than declaring this or that testing method KobviouslyK the best. *n the other hand, IKm left with the impression that weKre still trying to measure the si,e of a cloud. Reading the article, I ept thin ing of non$trivial factors that affect the total cost of testing and that had no place in the formulae( .. +ow much

time did &veryKs team waste fixing bugs in code that was eventually eliminated or radically rewritten before code completeC <. #id #anKs </ days give them enough time to rewrite the parts of the code that were fundamentally flawed $ or did they settle for superficial patches that made the code inefficient and unreadableC ?. Was &veryKs team concentrating on nit$pic ing details at a time when they should have been wearing their designersK hatsC ;. What was the cost of the loss of customer goodwill, and therefore future purchases, from #anKs high error rateC 9. #id the effort &veryKs team ma e early on ma e them unwilling to scrap and rewrite parts of the product that were poorly designedC =. +ow much time did #anKs team waste finding out which were the ?// bugs not worth fixingC @. +ow disruptive to the developers were the testing groupKs findings early on in the pro"ectC &nd on and on. IKm glad Johanna has made such an excellent start on this and has brought a little light into a very dar area. ....1/<N.LN/<2 Tek Wallah &uthorKs Response( Te , IKm glad you could see that this is "ust the beginning of the story. 7et me try to answer some of the -uestions( .. &veryKs team didnKt 5feel5 as if they wasted time fixing defects, because even if they redesigned and rewrote code, they learned from the fixing. <. #anKs </ days were insufficient. The customers eventually ic ed out the 'B* and reorgani,ed the company, based on too many defects for too many releases. 1%mall customer base.2 ?. 4ot everyone on &veryKs team was able to stop nit$pic ing during high level design and architecture, but they had ways to deal with those nits. ;. The customers too over #anKs companyKs board ($2 9. &veryKs team was not loyal to the code, but to the product. That meant that they were able to scrap when they needed to scrap. =. ItKs not clear to me that any of #anKs defects didnKt need to be fixed. +mm, let me say that more clearly. 4one of the defects #anKs pro"ect found were trivial enough to postpone. @. Testers didnKt disrupt the development on &veryKs pro"ect $ the testers and developers wor ed together.....1/<N<.N/<2 Johanna Rothman I worry about this metric, Johanna. &nyone who calculates a Mcost to fixM metric is going to want to use that information to extrapolate future costs, rightC I mean, if the metric tells me that fixing a bug costs 89//, then I should be allowed to reason that ma ing a decision 5not5 to fix ./ bugs would save me 89///, but that wouldnKt be a valid inference, would itC Gixing them might cost pennies, depending on the type of fix. *r may cost vastly more 4*T to fix. #eciding not to fix a set of bugs may not save me a single day on the pro"ect, if, for instance, the schedule is driven by some factor other than fixing bugs, or bug fixes happen in parallel, by developers who would be sitting idle if they werenKt fixing bugs that fell in their area of expertise.....1/<N.>N/<2 James Bach &uthorKs Response( James, youKre right to worry. :sing this metric by itself can be meaningless, unless you plan to change where you put the effort on a pro"ect. :nfortunately, many software pro"ects are driven by how many of which defects they can fix in the last phase of the pro"ect. If managers understood how much it cost to find and fix defects at the end of the pro"ect, maybe, "ust maybe, they would change how they planned the next pro"ect.....1/<N</N/<2 Johanna Rothman *ne thing I found interesting in these examples is that the Mcost to fixM defects is calculated in

combination with the effort of testing re-uired to find the defect to be fixed. Testing and debugging, whoever does the actual wor , are two different activities with their own metrics and implications. Gor example, if one team finds and fixes .// defects and the other team finds ;// defects and fixes .//, all else being e-ual, the second team is at a much greater advantage. It means they have had the opportunity to fix the highest priority defects first, and have a better basis to calculate the ris of releasing with the nown unfixed defects. They also have information about nown defects and wor arounds to put in the readme or to eep in the customer support database. The number that stic s out in my mind about the relative cost of fixing a defect in production vs. discovering it at the re-uirements phase, when the defect was traced bac to faulty re-uirements, is <@/ times more. &nd in general, the later the defect is found, the more costly it is to fix. There was a study that supported these trends, but it is somewhat intuitive. %uppose there is turnover in #anKs team between month . and month >. & new person is debugging someone elseKs code. & -uestion I have too is, when #anKs team decides not to fix bugs until month L, does this mean also that they are not unit and integration testing either until month LC If not, they are wasting the opportunity to do the testing when their mind is fresh on the specs and how they implemented them. IKm not sure if my comments miss the point of this article, but I feel it is important to .2encourage developers to do unit and integration testing before system testing in any case 1even internal system testers can get cran y when a system has too many bugs2 and <2 recogni,e the difference between the merits of the testing process vs. the debugging 1fixing of bugs2 process. Than s for the article.....1/<N.>N/<2 Mary Steinborn &uthorKs Response( Fary, at the end of the pro"ect, assuming everyone is focused on finding and fixing defects, I donKt now how to separate the testing from the fixing. &re you testing if youKre trying to replicate a problemC &re you testing if youKre trying to verify a fix is goodC &re you fixing if youKre testing your fixC &re you fixing if you read the other code around what youKve fixedC If I new how to appropriately categori,e the activities, I would, but there are too many overlapping activities when MeveryoneM is in frantic find$and$fix mode. BTW, IKm not sure that the people who find ;// defects and fix .// defects are better off than the people who only find .//. It depends on how many more defects there are to find, 5and5 if the developers actually fix the highest exposure defects first. ($2 1I suspect we agree more than we disagree, but "ust numbers arenKt a good enough indication of whatKs going on.2 I absolutely do agree with you that the more product testing everyone does earlier, the better off the pro"ect is..... 1/<N<.N/<2Johanna Rothman Oreat article, JohannaP 0ouKve captured the cost basis of errors. Fix this with the relative efficiency of finding and fixing via reviews compared to Mtrying to recreate the problemM debug Q fix, and youKve got the big picture. 'ontinuous unit testing and .//R review coverage not only catch faults early, but also point s-uarely at the location needing repair $ saving =/$>/R of Mbug fixM time. The progression from M nown problemM to M nown causeM is not cheap in system test..... 1/<N.>N/<2 Robert E Lee & good article and the author made it very easy to calulate the cost by simple tables. +ere are my

views) .. It doesnKt matter in my opinion how many days it ta es to fix a problem. #evelopers wor on many activities in a day and duration canKt be calculated by elapsed days. What matters a lot is the effort it has ta en 1in terms of actual hours the developer spent wor ing for the fix. Fany defect trac ing tools now have a seperate field where the effort in hours can be entered for each defect. We can run a -uery to find out the total hours, get the actual number of days and use it in the calculation. <. We can use the cost calculated and the parameters used for calculation into meaningful metrics such as number of defects fixed per ./// hours of effort, per 8./// of money ...etc sothat we will now whether efficiency is going up in the team. The cost of fixing the defects go up if the team members are new or in$ade-uate information is presented in defect database. 'ost of reproducing the defect involves significant cost and effort. ?. I would be happy to hear how exactly we can calculate the cost saved by preventing the defects at each and every phase. It is a bold step for any organi,ation to calculate the cost of fixing the defects. This will definitely create an awareness that defects have to be prevented at each an every phase to reduce the cost. It is a good method to start with and than s to the author for sharing this idea..... 1/<N.>N/<2 Srini!asan "esikan