S

T
E
S
T
S
m
a
r
T
E
S
T
S
A SMART APPROACH TO
SOFTWARE TESTING
EGBERT BOUMAN
SmarTEST
SSSS
A SMART APPROACH TO
SOFTWARE TESTING
S
m
a
r
T
E
S
T
S
SmarTEST
A smart approach to
software testing
Egbert Bouman
For more information about this edition and other Sdu editions,
please contact:
Sdu Customer Service
P.O. Box 20014
2500 EA The Hague
phone: +31 (0) 70 3789880
www.sdu.nl/klantenservice
Sdu Uitgevers bv, Den Haag
Academic Service is an imprint of Sdu Uitgevers bv
Translation: Landl, Paul de Wit
TyeseingIrilschyomaakredaclieLeiden
CoverRosannaZiloZIDIineAgrahicsAmersfoorl
Print: Drukkerij Wilco, Amersfoort
ISBN: 9789012582902
NUR: 980
All rights reserved. No part of this publication may be repro-
ducedsloredinarelrievaIsyslemorlransmiedinanyformor
by any means, electronic, mechanical, photocopying, recording
or otherwise, without the publishers prior consent.
WhiIeeveryeorlhasbeenmadeloensurelhereIiabiIilyoflhe
information presented in this publication, Sdu Uitgevers neither
guarantees the accuracy of the data contained herein nor accepts
responsibility for errors or omissions or their consequences.
Contents 5
Contents
Foreword 11
Preface 13
How to use this book 21
PART 1 THE WHAT & WHY OF TESTING 23
1 Good commissioning and IT Governance 25
RequiredGriandinsighl25
usinessob|eclivesasslarlingoinl26
ITmaers28
Archilecluredrivendesign28
usinesscaseandrequiremenls29
LegisIalionandreguIalioncomIiance30
TeslinginconneclionvilholherdisciIines32
2 There is more to success than the IT system alone 35
RiskconlroIbasedonrequiremenls35
Iroduclrisksversusro|eclrisks35
Nol|uslasyslembulrocessandinformalionasveII36
2.4 Integral management of requirements and acceptance
crileria37
SuccessQuaIilyxAccelance39
Irevenlionandcure40
Issixouloflenenough41
3 System, information and process quality 43
TheSmarTISTscoe43
WhyquaIilymodeIs43
TheIISreferencemodeIforroduclquaIily43
SyslemquaIilyandlheISOmodeI44
InformalionquaIilyandlheIDQmodeI45
IrocessquaIilyandlheIOQmodeI46
3.7 Models made to measure, for testers and developers
aIike47
ToconcIudelhischalerandarl48
PART 2 A GRIP ON TESTING 51
4 Test approaches: an inventory by comparison 53
WhaldoesagoodleslaroachIookIike53
Thebeslleslaroach54
TeslingandmodernsyslemdeveIomenl56
SmarTIST58
6 Contents
TMa60
TeslIrame61
ISTQ62
5 The SmarTEST process 65
IrocessmodeI65
Teslaclivilies66
Ihasing69
Teslroducls73
OrganisalionlesloIicyandinfraslruclure75
6 Test levels and the W-model, special tests 79
TeslIeveIs79
IromVmodeIloWmodeI80
MainlenanceTesling85
Conlinuouslesling86
TheoeralionaIleslshadovrun88
OeralionaImoniloring90
Teslingackagesoflvare91
Indloendlesling91
7 Test planning and estimation 95
Tesleslimalion95
TeslslralegyandchoosingleslIeveIs98
Theleslro|eclIan103
SmarlleslinginIRINCIro|ecls105
8 Test competencies and test roles 109
Isleslingarofessioninilsovnrighl109
Thecomelencysquare110
Userinulvialheleslrocess112
TeslroIesleslosilionsandcareeralhs114
IeoIeissuesandcuIlure115
9 Test organisation and outsourcing 117
Consideralionsvhenorganisinglheleslrocess117
TeslorganisalioninlheIine118
Teslorganisalioninro|ecls120
IrofessionaITeslServicesandManagedTeslServices122
Oulsourcingandsmarlsourcing125
ThefaclorymodeI127
10 Test process assessment and improvement 129
Aboulambilionandragmalism129
TIITMMiandTIS130
ToconcIudelhischalerandarl132
Contents 7
PART 3 EXPERT TESTING 133
11 Intake, review and inspection 135
Documenlvericalionandslaliclesling135
InseclionsrevievsandvaIklhroughs136
Iaganinseclion139
IndividuaIrevievlechniquesSmarlRevievSQR141
12 Risk assessment 145
Norisknolesl145
ImorlanceandfaiIurerobabiIilydelerminelherisk146
Whendoveneedariskassessmenl146
Decomosilionoflheleslob|ecl147
OnedimensionaIriskassessmenllheTRAmelhod148
TvodimensionaIriskassessmenllheIRIMAmelhod149
TheIRIMAsleedIan151
12.8 How to proceed from here? The conscience function of the
roduclriskmalrix151
13 Requirements and acceptance criteria 153
SlakehoIdersandaccelingarlies153
Iromrequiremenlsloaccelancecrileria156
13.3 SMART wording of requirements and acceptance
crileria156
13.4 Acceptance criteria play a pivotal role in the test
rocess157
14 Test specications 161
TeslsecicalionslhinkrslaclIaler161
Tesldesignlechniquesandcoverage162
TeslcasesIogicaIandhysicaI166
ThereIalivilyofleslcases167
ComaclandrevievabIeleslvarevilhleslscenarios169
Theconlenlsofaleslscenario171
14.7 The SmarTEST terminology compared to that of TMap,
ISTQandTeslIrame174
Teslsecicalionlisfromraclice176
15 Test execution 179
SlebysleIanforleslexeculion179
IxIoralorylesling180
WhenlosloleslinglheCorneodiagram183
16 Defect management 185
Theimorlanceofdefeclmanagemenl185
Slalusov187
TheroIes190
ThedisalchermodeIandlhemonilormodeI191
Thedefeclmanagemenlmeeling193
8 Contents
Aribulesinlhedefecldalabasenolloomany193
DefeclmanagemenllooIs196
RoolcauseanaIysisandslalislicaIanaIysis197
Defeclmanagemenllisfromraclice198
17 Reports and advice 201
ThefouriIIarsoflransarenlreorling201
Reorlseraccelancecrilerion202
TheReIeaseCharl204
MelricsandlheDefeclTrendOverviev205
Theleslreorl207
ReIeasecrileriaandreIeaseadvice207
18 Automated testing 211
TeslingvhiIeyouareasIee211
TheloofcommerciaIandfreeoensourcelooIs212
Yelanolhersoflvarero|ecl212
Teslingvilhaclionvords213
18.5 When does automated functional testing become cost-
eeclive214
AulomalederformanceIoadandslresslesling214
OlherareasvherelooIscanheI216
18.8 Prevent shelfware: selection and implementation of test
looIs217
Managemenlcommilmenl218
Inshorl219
19 Testing data quality 221
TheimorlanceofdalaquaIily221
Dalameladalaandinformalion222
TeslingdalaquaIily223
TolaIDalaQuaIilyManagemenl224
TheoverofDalaIroIing226
TooIsforDalaIroIing229
20 Testing process and organisation quality 231
IrocessvaIidalion231
IDIandITaudiling232
20.3 Why auditors and testers come across each other
reealedIy234
Ioinlersforleslers236
Contents 9
APPENDICES 239
A The ISO 9126 software qualitymodel: denitions 241
B Examples of quality characteristics with
acceptance criteria 247
C Requirements for requirements, with a few
examples 251
D Requirements engineering, REQUIRESmart

255
E Work instruction risk analysis with the PRIMA
matrix 259
F Sample Test scenario WITH test cases 265
G Sample Test scenario WITHOUT test cases 269
Trademark acknowledgments 271
Literature 273
Index 279
Foreword 11
Foreword
Dear reader, you may know more about technology than I do but
like you, I am fascinated by IT and the possibilities that Informa-
lionTechnoIogyoerslousersonaIIyandlosocielyalIargeAs
Mayor of Knowledge City Almere, the most digital city in the
Netherlands, I know a thing or two about that.
For example, we created an Online Products and Services appli-
cation for Almere, enabling citizens to report faulty street lights
or loose footway slabs round-the-clock. Using the citys digital
database every resident can access information on road works,
for example by means of a digital map similar to Google Earth.
That is all wonderful, and not even that unusual anymore. How-
ever, I do meet quite a few administrators and entrepreneurs who
at best display a love-hate relationship with respect to IT and IT
projects. The national government too, knows what I am talking
about, and it was like that when I was Deputy Prime Minister and
Cabinet Minister. Your average IT project is no bed of roses. You
will experience that yourself as commissioning party, entrepre-
neur or administrator.
I believe that to a large degree this is because in large projects
lhereisloomuchvaelhalordinaryeoIecanlseelhrough
IndeaIingvilhITeoIeyousomelimes|uslhaveloeeIoa
few layers of techno-speak and one-sidedness before you get to
talk about something substantial. For example, take the Digital
Counter. That in itself is by no means the most complicated part,
unIikelheorganisalionanddalabasesoeralingbehindilThese
had to be turned upside down, actually. We were forced to struc-
lureourrocessesdierenlIyandullhecuslomermoreallhe
centre of what we do. That is a tall order in an organisation of
civiIservanlsbulIen|oybeingabIelouIIlhalsorloflhingo
After all, this is about something substantial, of course: innova-
tion is a necessity for the social and economic development of our
country. For that reason, Knowledge City Almere is an impor-
tant partner as a driving force behind innovation through ICT.
We enter into all sorts of partnerships, but one needs to make sure
that one stays in charge. As the commissioning party, you want
to stay in control, as the book puts it.
I embrace everyone who helps me with that. They are like guard-
ian angels: IT experts who not only speak my language, but also
rovide secic reIiabIe informalion as veII Indeendenl ad-
vice of people without tunnel vision, such as testers, is always
12 Foreword
valuable. That is why I am happy with this book. As far as I am
concerned, it is an example of the approach that IT people should
adopt when going about their business: smart, knowledgeable
and versaliIe And aIso IeveIheaded exibIe and unbIinkered
And they should always look at their work through the eyes of
their customer and commissioning party.
Now that I have read this book, I have seen how important test-
ing really is. That does not apply to IT projects only, but to the
people involved as well. Egbert Bouman explains the discipline
of testing to a relatively wide audience. He does so without wear-
ing blinkers. The connections he makes are to the point, and you
dont have to be an IT specialist to see that. It is not over-simpli-
edandaIvaysagoodandnicereadIamconvincedlhisbook
will contribute to the increasing number of successful IT projects.
Annemarie Jorritsma
Almere, March 2008
Annemarie Jorritsma is Mayor of Almere, the most digital city in the
Netherlands. Previously, she was Deputy Prime Minister, Minister for
Transport, Public Works and Water Management and Minister for Eco-
ncnicAairsinicra|ia
Preface 13
Preface
The relevance of testing
I grew up in a family-run bakery shop. My father had pride in his
craflsmanshieingabakerisanerofessionlheimorlance
of bread is evident. I am a software tester and maybe you are one
loooryouemIoyleslersThalisanolhernerofessionforlhe
importance of testing does not require much explanation either.
Nowadays, testing of information systems is almost as relevant to
human existence as bread. In 2003, the National Institute of Stand-
ards and TechnologyinlheUniledSlalesubIishedavehundred
page report on the damage to the American economy that could
bearibuledlosoflvarebugsThaldamageamounlslocIoseon
sixty billion dollars a year. That would be more than twenty-two
biIIion doIIars Iess if enough money vere invesled in lesling
Testing can therefore save a lot of money. That is it for the good
nevsvhichhasnolchangedsincelherslDulchedilionoflhis
book in 2004.
IT quality in the Netherlands: 20 years of stagnation?
The middle of 1987: Handboek Testen (A Guide to Testing) was
published after questions in the Dutch Parliament about shoddy
IT
1
systems and poor quality control with the Dutch Inland Rev-
enue Service. This was TMap (Test Management Approach) ver-
sion 0.1, which by now has become a popular test approach. The
Maintain company was founded, which developed into Valori.
The end of 2007: Questions were asked in the Dutch Parliament
and there was an emergency debate on persistent IT problems in
the Inland Revenue Service. The responsible State Secretary, Mr
De Jager from the Dutch Ministry of Finance, expressed his con-
cerns about the stability of the automation systems in the Dutch
Tax and Customs Administration. February 2008: 730,000 tax
1 We used to speak of IT, then the ICT abbreviation was all the fashion,
and now we often say IT again. That is what we do in this book, too.
making money with
testing
failed IT projects
14 Preface
returns were rendered unusable on account of a late end-to-end
test. Around the same time, research showed that the Dutch gov-
ernmenl aIone sends four lo ve biIIion Iuros on faiIed aulo
mation projects each year.
This story looks like bad news. Is this the result of twenty years
of professionalisation? Including that of the testing discipline?
From virtually zero to thousands of full-time software testers,
and dozens of specialised testing companies? From zero to more
lhanlvenlyheflybooksonbeerlesling
Are we doing something wrong?
Testing does not make quality
Testers do make mistakes but they dont create software bugs or
unstable systems. Testers just pass judgment on quality: is this
the right system (validation) and has it been built adequately
(vericalion Novadays ve are abIe lo make assessmenls in-
creasingly early in the software life cycle, with greater degrees of
objectivity and reliability. Yet, the old truth still applies: One can-
not test quality into a system.
Of course, that is too easy. A carefully designed test process can
and shouId make a reaI dierence ossibIy lhe dierence be-
tween success and failure for an IT project. Managers who are not
provided with hard data and factual quality information by test-
ers can only rely on gut feeling and on the advice of the most
vociferous person. They lack the support and the insight that are
indispensable for adequate supplier commissioning and project
management, and are unable to anticipate and avert impending
disaster. In short, they are not in control.
Are you a good commissioning party?
Valori frequently organises Meetings of Minds where interest-
ing themes draw interesting speakers and customers. I remember
oneoflhesevonderfuIsessionshighuinalaIIoceloverin
Roerdam The roosilion vas IT ro|ecls faiI rimariIy be-
cause of poor commissioning. We felt that that was a very pro-
vocative proposition, but much to our surprise, the majority of
lheaendingmanagersacluaIIyagreedMaybelhehenomenaI
view of the city reinforced the awareness of what they lacked in
their own projects: an unobstructed view, overview and insight.
Insight into project management and into the quality of what has
been delivered. Because if you lack grip, you cannot be a good
what are we doing
wrong?
validation &
verication
project management
good commissioning
Preface 15
commissioning party and the supplier will be unable to deliver as
expected.
Testing does make a difference!
Testing is the essential element in the continuous feedback loop
of system development processes. It is the ongoing game between
software builders and critical, independent testers, and these two
parties keep each other sharp and focused. After all, the human
mind is such that we soon become blind to our own mistakes.
Moreover lesling rovides lhe secic and facluaI informalion
that the commissioning party, the client, needs to have.
Judging from the numerous approaches to testing, the compre-
hensive nature of the handbooks, the maturity of the training
courses and the quality of the current generation of test profes-
sionals, one would think that that is no problem at all in 2008.
Structured approaches to testing, such as TMap

and TestFrame

,
ensurelhalaenlionisgivenloleslingalanearIyslageIgrev
up with it, so to speak, and I could no longer do without it. If a
structured method is used well, we are in pole position when test-
ing starts and we address the risks as early as possible, which is
cheaper and safer than doing so at the very last minute.
However, we touch upon a crucial issue here: knowing the meth-
ods, techniques and tools is one thing. Applying that knowledge
eecliveIy lhough is quile anolher lhingAs a maer of facl il
means that we should continuously be able to provide our clients
with the information needed at the right time, at the appropriate
level of detail. That requires a sense for strategic targets, for hu-
man relations, for adaptability, transparency and quantity.
Furthermore, these structured approaches also have their down-
sides, the pitfall of any comprehensive framework: a tendency for
too much formalism and rigidity and overkill of procedures and
documentation.
Why SmarTEST?
SmarTEST is the exception. SmarTEST originated on the shop
oor and il oers a conceluaI and slralegic framevork vhere
the usefulness and the practicality do survive confrontation with
reaIro|eclsInlheendveonIymakeadierenceifvecanoer
added value in the real world, I would almost say: in a streetwise
way. That requires smart testing: Strategic, valuing Man over ma-
chine, AdaliveRiskbasedandTransarenlSMARTforshorl
blind to own
mistakes
adaptability,
transparency and
quantity
surviving the
confrontation with
reality
16 Preface
SmarTEST combines the thoroughness and reliability of tradi-
tional test methods with a smart, pragmatic approach, geared to-
wards the modern practice of system development. SmarTEST is
focused on results: a complete and reliable quality assessment for
an acceptable fee. We do not have to exclude every single risk, for
the speed and market-focus of an organisation must not be ham-
ered by a rigid and riskavoiding ailude Thal vas once ex-
pressed aptly in the following saying:
Does SmarTEST make other test approaches redundant? No, it
does nolAdmiedIy SmarTIST can be used comIeleIy on ils
own but it also combines very well with (elements from) other
aroaches and books Hovever lhis book does oer an inde-
pendent introduction for everyone who does not want to lose
sight of the essence. At this moment in time, it is also the only
book lo oer a fuII and ob|eclive overviev of lhe various lesl
methods and the corresponding literature for the Dutch market.
ThalisnosmaIImaerforilissafelocIaimlhallheNelherIands
play a major role internationally in the professionalisation of soft-
ware testing. There are those who even maintain that testing is
the most important IT export product of the Netherlands [Loen-
houd, 2007].
Why SmarTEST in English?
ecauseoflhedemandSmarTISTisincreasingIyusedininler
nalionaI ro|ecl seings vhere IngIish nol Dulch is used as
lingua franca to bridge the communication gaps. SmarTEST in
IngIishislhusonIyseenlobeingIlviIIbroadenlheaudience
IIeasenolelhalSmarTISTisofDulchoriginrmIygroundedin
the Dutch (IT) culture and business reality. We did not change
that in the process of translating. Dont worry: there is enough to
en|oyevenifyourinlernalionaIseingmaybeaIiIebildierenl
You will probably recognise your situation close enough in the
examples given. So let us continue with an observation that may
not be typically Dutch at all.
pragmatic approach
The man who makes the decision and may sometimes lack preci-
sion, boosts the coffers more than the man who seeks perfection,
and misses the connection
(J. Smit at a congress called Decision-making in the organisation)
The Netherlands
leading in testing
Preface 17
SmarTEST in (academic) teaching
SmarTEST has been well-received in education circles. The Dutch
EXIN has chosen SmarTEST as the basis for its Testing Founda-
lioncoursesandcerlicalionrogrammeandSmarTISTlogelh-
er vilh TMa for Tesling IrofessionaI The Roerdam High
SchooIRoerdamUniversity) was inspired by SmarTEST and it
hasbeenleachingrslyearandsecondyearsludenlsSmarTIST
as an approach to iterative development methods. We are very
happy with that, because testing deserves to be more widely
known among students. After all, it is among one of the most re-
quested IT disciplines
2
.
Therefore, the integration of testing in education is not a problem.
Hovever lhe inlroduclion of lesling in scienlic research is
somevhal more comIicaled as lesling is rsl and foremosl a
discipline based on practice. A hard-core scientist is likely to dis-
missilaslooragmalicConverseIylyicaIIyscienlicsub|ecls
as correctness proofs and formal notations such as TTCN3 strug-
gle to demonstrate their added value in the everyday practice of
ITro|eclsIorlhalreasonvehavedevoledreIaliveIyIiIesace
to this for now. However, the academic world and the corporate
world are growing closer to each other in the area of testing. May-
belhereviIIbemoredeveIomenlsloreorlinlhenexledilion
Who are the target readers of this book?
This book is aimed at a wide audience: commissioning parties, IT
managers, test managers, testers and accepting parties. After all,
the gap between IT and the business must be closed and testing
is a domain in which these two come together. Testers should
therefore learn to speak the language of commissioning parties
and users, while commissioning parties need to understand what
testing is really about. I wrote this book as a contribution in bridg-
ing that gap.
After you have read this book, you will know whats going on in
testing, what to look out for in testing in the real world and you
will be able to use testing as a strategic risk instrument. Undoubt-
edly, you will remember several practical tips that you will be
able to apply in your daily practice. You can combine them with
more details, tools and templates, which are available at www.
smartest.nl.
2 Source: Computable, March 9, 2007.
testing most
requested
IT discipline
testing scientic?
wide audience
single language
testing in the real
world
www
18 Preface
Incidentally, I have used the personal pronoun he throughout
the book, but it goes without saying that I mean he or she, for
while ICT is the work of man, it obviously is not an exclusively
male domain.
Acknowledgments
This book would never have seen the light of day without the big
support I received. First I would like to thank the many Valori
colleagues who reviewed the book. I am particularly grateful to
those who larded their critical comments with compliments and
notes of encouragement. I have enjoyed processing their feed-
back. It proves that a tester or reviewer also has to bring good
news.
This English edition posed an extra challenge. Again, I was not
left alone. I look back at a pleasant co-operation with professional
translator Paul de Wit. The Valori review team did an indispensa-
ble job by helping to update parts of the content and to work the
standard English into IT and testing English, which was by no
means easy. Id like to say thank you very much to: Gerard
Dri|oulKeIvinGeerIingsIrankHelemThi|svanKoeveringe
Martin Kooij, Jaap van der Leer, Chris Maliepaard, Eric van der
Mark, Andy Meijer, Pieter Smelt, Patrick Smulders, Ralph Tjong-
Akiet, Binne van der Veen and Danielle Verduin-Slotboom. Many
lhankscoIIeaguesilvasaIeasure
Many thanks too, to all those Valori customers with whom I was
privileged to work, where the confrontation with the real world
took place. Ideally, I would like to thank all the people with
whom I worked with so much pleasure personally. That, howev-
er, would be a long list, and would probably lead to a few unin-
lenlionaIyelainfuIomissionsSoIelilsuceinsleadlomenlion
the organisalions lhal have conlribuled in some secic vay lo
the SmarTEST approach and this book: ABN-AMRO, Achmea,
ANWB, Arboned, ASML, ASR, Cordares, Delta Lloyd, ING Bank,
Interpay,Interpolis, Kadaster, KPN mobile, The Dutch Ministry
of Security and Justice, Menzis, PGGM, Politie Haag landen (The
Haaglanden Police Force), Rabobank, Randstad I-Bridge, RDW,
Stater, VGZ and Univ. A separate acknowledgment is due to
HosseinChamanioflheRoerdamUniversilyandhiscoIIeague
Jurgen van Raak for their enthusiastic and valuable feedback
fromlhemomenllhersledilionvasubIished
Tim Koomen, co-author of TMap NEXT, gave valuable tips. Erik
van Veenendaal, the driving force behind TMMi in the Nether-
lands, checked the section on TMMi. The ISTQB section is up-to-
colleagues
organisations
test gurus
Preface 19
date and reliable thanks to the critical view of Rik Marselis, a
member of the ISTQB board.
I am grateful to the publisher, Academic Service for their profes-
sional support, in particular Marcel Roozeboom and Judith Ruijg-
rokandIrilsIrilschyforlhelyeseingWilhoullheheIand
vision of Ric GieIen and MicheI Koning lhere denileIy vouId
not have been a book; they did everything to avoid disturbing me
on my book days, even if my professional presence was urgently
needed. As far as I know it did not lose Valori any customers, but
I was not always so sure about that.
Ingrid Oevanger an oId lesling acquainlance of mine and co
founder of the TestNet association, also needs to be mentioned.
As an Independent linguist Ingrid has made an indispensable
conlribulion lo lhe naI reviev of lhe Dulch manuscril These
days, Ingrid has her own copywriting and translation agency,
andsheisdoingveII
I have saved the real heroes and background characters of this
book till last: my wife Forien and my children Jaap, Muril and
Aleid, who were always willing to give up a few square metres of
our living space and pamper me to boot. Without all these other
people I mentioned before this book would not have existed, but
vilhoulyouIvouIdbenovhere
Egbert Bouman, March 2008 and September 2011
heroes
How to use this book 21
How to use this book
On the three parts
SmarTEST, a smart approach to software testing comprises three
parts, the sizes of which are in a ratio of 1:2:4.
Part 1, The What and Why of Testing, is for everyone who wants
to know why testing is important and what one can and cannot
expect from testing. It deals with the question of how testing is a
strategic tool for a commissioning party that wants to be in con-
trol and takes IT Governance seriously. The What, the scope of
lhe SmarTIST rocess is dened by lhe IIS modeI an inlegraI
approach to system, data and process quality.
Part 2, A Grip on Testing, presents the most important testing
methods and discusses the essentials of the How of testing. Test
policy, planning, organising and improving the test process will
all be dealt with in this part. This part is aimed at (project) manag-
ers and everyone else who want to gain insight into and get a grip
on the test process.
Part 3, Expert Testing, is meant for everyone who really would
like to develop a feel for the best practices and the professional
side of the testing discipline. This part contains a more in-depth
analysis and is particularly suited to readers who are closely in-
volved in the test process, such as test managers, test leads and
testers. This part can serve as a handbook, especially in combina-
tion with the website. Chapters 19 and 20 on testing data quality,
process quality and auditing can be appreciated on their own and
may interesting to readers of part 1 and 2.
TheaendicesconlainmodeIsIislsofdenilionsexamIesand
some work instructions.
what & why
grip on testing
expert testing
22 How to use this book
The website
The book is linked to a vebsilevvvsmarleslnIYouviIInd
more detailed information, templates, examples, tools and topical
news there. The book contains frequent references to the site, so
look around on the site to see what else is there.
The website is in Dutch, but tools and key texts are available in
English, and will be increasingly so.
Other literature
In addition to the website, there are other handbooks you can use
alongside SmarTEST. SmarTEST is a way of looking at and deal-
ing with testing that works well on its own and also in combina-
tion with other approaches.
Differences compared with the rst edition
This second edition is completely revised, updated, expanded
and improved both on content and on way of presenting. The
moslimorlanldierencescomaredvilhlherslDulchedi-
tion are:
Improved accessibility through a more consistent structure,
aimed at:
readers with a general interest (part 1);
managers and those who commission tests (part 2);
professional testers (part 3).
MoreaenlionlolhecIienlserseclive
An objective overview of all standard approaches to testing, with
an emphasis on the Dutch market.
eer comalibiIily vilh videIy used lerminoIogy arlicuIarIy
in ISTQB and TMap.
UdaleoflooIlisandlooIreferencesvilhagooddeaIofaen-
tion to open-source tools.
More aenlion lo lesl eslimalion and imrovemenl of lhe lesl
process.
Separate chapters on testing data quality and process quality.
Etc., etc.
The ve rinciIes of SMART lesling are sliII fuIIy aIicabIe
Strategic, valuing Man over machine, Adaptive, Risk-based and
Transparent. Nothing has changed in that respect.
www
principles unchanged
PART 1
THE WHAT & WHY OF
TESTING
This part is intended for everyone who wants to know why testing
is important and what one can and cannot expect from testing. It
deals with the question of how testing is a strategic tool for a com-
missioning party that wants to be in control and takes IT Govern-
ance seriously.
Subsequently, we determine the scope of testing. This goes beyond
merely evaluating software: there is more to success than the IT
system alone. This is illustrated by the introduction of the IPS model
for the quality of Information, Processes and Systems. This is the
scope for the next two parts of the book.
Chapter 1 Good commissioning and IT governance 25
1 Good commissioning and IT
Governance
1.1 Required: Grip and insight
In the preface, we already noted that good commissioning is cru-
cial for successful IT projects. Commissioning is a process that is
initiated and managed by the commissioning party. This does not
include the construction. What it does include is requirements
engineering, architecture, management, checking and (accept-
ance) testing of the deliverables. The extent to which the commis-
sioning party is able to get this process up and running and to
manage it, largely determines the success of projects.
Who is actually the commissioning party? By commissioning
party, or client for short, we mean everyone that is responsible
for the management of IT suppliers and IT project managers.
Mosl of lhe lime lhis lask resls vilh a sleering commiee or a
management board. For various reasons, this is right and desira-
ble, but still, IT remains the work of humans. Therefore, an IT
suIierdeserveslodeaIvilhsomeonemadeofeshandbIood
vhoersonieslhecIienlandexeculeshisduliesvilhcommil-
ment, passion and personal responsibility. That is the second
principle of SMART testing: the M of Man over machine. This is
aIsoasociaIandeconomiclrendaflerseveraInanciaIscandaIs
new compliance codes (such as the Tabaksblat code in the Neth-
erlands, SOX in the US) focus on personal responsibility and lia-
bility. Later in this chapter, we will return to legal compliance in
detail.
It is not easy being a good commissioning party for IT suppliers.
In current architecture, we are talking not only about isolated ap-
plications, but about complex chains. A client will seek transpar-
ency in his systems and projects and wishes to be well-informed
at every level, with a full grasp of the facts. Some of the terms we
hear with increasing frequency in this context are Corporate Gov-
ernance lo run an enlerrise veII ecienlIy and resonsibIy
and IT Governance (the IT side of Corporate Governance). And
lhesedayslhalexlendsbeyondlhenanciaIerseclivevhich
is traditionally monitored by accountants and controllers. This is
good commissioning
human factor
IT Governance
26 Part 1
about business-IT alignment from the ground up, as an organic
part of system development and maintenance [Hoogervorst,
2008].
Even today, IT projects still fail in dramatic ways. And one thing
has become clear by now: the blame cannot be laid solely at the
suppliers feet. In government projects, it has been painfully dem-
onstrated that ill-informed and wavering commissioning parties
are major contributors to failure
3
. Shareholders and stakeholders
demand transparency in governance and supporting processes.
That includes a certain level of knowledge and a certain degree of
insight on the clients side. Many IT providers assume that their
clients know exactly what is needed. The reality is often rather
dierenllhereisanenormousamounlofmiscommunicalionbe-
tween suppliers and commissioning parties
4
. It is true that some
IT providers will exploit this situation, but instead of taking the
moraI high ground cIienls had beer lry and gel more gri on
their projects, and be more in control. That requires a good re-
view and test process.
Good commissioning parties use testing as a strategic tool for risk
conlroINolaslhenaIsafelynelbulasinlegraIarloflhelolaI
life cycle, all the way from the business case to the utilisation and
maintenance phase. That calls for a good groundwork at the start.
The next paragraph will discuss this in more detail.
1.2 Business objectives as starting point
Why do organisations actually spend money on information sys-
tems? As a means to achieve a higher strategic objective, as illus-
trated in Figure 1.1.
Testing informalion syslems oers insighl inlo lhe quality and
progress, two closely connected concepts. Progress without quali-
ty is no progress, witness the sighs over what should have been
lhehomeslraighlandlhecommonvisdomlhallhenaIer
cent of the work takes 80 per cent of the time. Distrust anyone
who claims that something is ready, by and large. That usually
really means that only 20 per cent of the work has been done. The
reslisneededloaainanaccelabIequaIilyIeveI
3 Source: report from the Dutch Court of Audit, late November 2007
4 Source: research by the Marqit agency, 10-10-2006
business-IT
alignment
transparency
in control
Testing is a strategic tool for risk control
strategic tool
for risk control
progress without
quality is no progress
Chapter 1 Good commissioning and IT governance 27
An information system in the broad sense of the word is the complex
of people, process and tools in which data are collected, recorded
and processed systematically. The information system in a strict
sense is a combination of hardware and software components for
secicaIicalionsTheaIicalionissuorledbylheIT infra-
structure: systems software, network and storage. The term (in-
formation) system is usually meant in the strict sense in this book.
To know whether an information system suits the business objec-
tives, you need to know with some precision what these objec-
tives are. That is not all that simple. The answer to that question
will depend on the person you are asking. The Financial Director
thinks in terms of revenue and value to shareholders; IT just hap-
pens to cost money [Berdowski, 2003]. The Managing Director
will be talking about the contribution to the companys mission,
core competencies and strategic objectives. The Marketing Direc-
tor hammers on image, markets and customers and the corre-
sponding products and services. The department manager
believes it is all about his teams and the business processes. With
aIiIebilofIucklhey|uslmighlagreeonlhecritical success fac-
tors for the organisation.
denition
information system
Information system
Goal
Your organisation

Business processes
Figure 1.1 The information system serves business processes and objectives
mission
core competences
critical success
factors
28 Part 1
Figure 1.2 shows the two levels in formulating business objec-
tives: strategic, such as mission, core competencies and goals and
tactical, such as the choice for a promising product/market combi-
nation. Strategic and tactical business objectives subsequently
give rise to the concrete products and services which, in turn, ne-
cessilale cerlain secic business processes and organisation
types.
1.3 IT matters!
In 2008, all this is supported and facilitated by information sys-
tems to an ever-increasing extent. And this trend does not stop
here. IT is interwoven with operational management and the
company value chain in all sorts of ways. Disruptions are no
Ionger |usl an inlernaI robIem bul have a knockon eecl in
various vays aecl lhe cuslomer direclIy and cause negalive
ubIicily Hence lhe eorls lo achieve reliability, stability and
availability. At the same time, there is the need to create a com-
petitive advantage through ever smarter information systems.
Although Nicolas Carr temporarily enjoyed a great deal of sup-
orlinvilhhisarlicIeITdoesnlmaerCarrIThas
certainly not become a mass product that does not allow you to
slandoulfromlhereslAdmiedIyyoudonolseeaIolofcus-
tom-made work anymore; standard packages and components
areusednearIyaIIoflhelimeHoveveryouviIIndilindis-
tinctive, cleverly designed and well-balanced combinations, em-
bedded in a carefully thought-out architecture.
1.4 Architecture driven design
Every self-respecting organisation has an IT strategy with an IT
blueprint or IT architecture. More and more often, we also see an
Enterprise Architecture based on a generic architectural frame-
work. Within that framework, people are working with architec-
Strategic:
Mission, core competences, strategic goals
Tactical:
Products, services, operational processes, organisation
With critical succes factors en Key Performance Indicators
Figure 1.2 From business strategy to concrete results
product/market
combination
IT interwoven with
the total value chain
competitive
advantage
Enterprise
Architecture
Chapter 1 Good commissioning and IT governance 29
tural principles that ensure that IT projects and systems are
internally coherent and contribute to the business objectives.
Some common frameworks are:
TOGAF: The Open Group Architecture Framework, an open-
source framework
The Zachman framework
IAF: Integrated Architecture Framework
NORA: Nederlandse Overheids Referentie Architectuur (= refer-
ence architecture for the Dutch government)
Some examples of principles and guidelines as part of an Enter-
prise Architecture are:
We choose SOA (Service-Oriented Architecture): IT building
blocks must have service architecture and match our Enterprise
Service Bus.
We choose oensourcesoflvareAloicaIgovernmenllheme
We apply a re-use before buy, before build strategy.
In other words: developing in-house is only an option if there are
no avaiIabIe comonenls for reuse and ve cannol nd vhal is
needed on the market.
We work with .NET. (and, by implication, not with J2EE)
Teslers can benel from lhis vhen slrucluring lheir ovn vork
and aligning it with the stakeholders on a strategic level. An ex-
ample would be the TOGAF
5
and SmarTEST-based approach for
testing infrastructure components, which will be discussed in
paragraph 6.4 Continuous testing.
1.5 Business case and requirements
So now we are sure, IT is indeed a critical success factor for near-
ly every organisation. The complexity and the connection with
operational management puts heavy demands on the capability
of the business to manage and control. If only to keep costs under
control. The construction of an information system is therefore
usually supported by the so-called business caselhe|uslicalion
for the project, not in the least in terms of money: return on invest-
ment. The business case also covers the reasons and objectives for
the project, the estimated costs and risks and the expected bene-
lsandsavingsAveIImanagedro|eclischeckedreguIarIyfor
its compatibility with the business case and conversely, the busi-
ness case is checked for topicality and reality. Indeed, popular
project management methods, such as PRINCE2, pay a great deal
ofaenlionlolhis
5 See also the article TOGAF gaat het maken (TOGAF will be a hit) in
Computable, February 8, 2008.
TOGAF
SOA
business leading
PRINCE2
30 Part 1
A business case alone is not enough to unleash the computer pro-
grammers and start coding. It has to be translated into design
documents and models. To that end, the so-called requirements
are collected and documented.
The requirements should relate both to functionality, and to the
way that functionality will be presented: speed, reliability, user-
friendliness, manageability and security, among other things.
These characteristics are also colloquially known as non-function-
al requirements. The requirements are to formulate the answer to
the WHAT question, the question WHAT the system should be
and do, formulated in terms the accepting party, the intended us-
ers and the IT support team will understand. However, they do
not yet dwell too much on HOW this is to be achieved.
All sorts of environmental factors also decide what is possible
and desirable, for example, available technology (technology
push), developments in society, as well as legislation and regula-
lionSeegure
1.6 Legislation and regulation, compliance
Legislation and regulation have far-reaching consequences for IT
systems. For years now, we have had the equivalent of the Data
Protection Act in the Netherlands. It has a big impact on IT pro-
jects. For sure, any IT specialist who has been working in the
health care sector lately will vouch for this. It is partly because of
requirements
technology push
IT preconditions
IT blueprint: information policy,
architecture, infrastructure,
and COTS-components
Business goals
Mission, Core competences, strategic goals
Products and services, and business processes
Critical succes factors with KPIs
Competition,
Technology push,
Compliance,
Society, etc.
Project en product requirements
High level in the business case (the nancial justication for the project)
and elaborated upon during the project
Figure 1.3 Business requirements and IT preconditions decisive for an
information system
DPA
Chapter 1 Good commissioning and IT governance 31
those preconditions that the Electronic Patient Record (Electronic
Health Record) and the integration of hospital information sys-
lems are sliII slruggIing lo gel o lhe ground The more recenl
Dutch WALVIS Act is intended to force a reduction of the admin-
istrative burden for customers and providers of employee insur-
ance Hovever lhe modicalions lo lhe syslems of insurance
companies and temporary employment agencies that are re-
quired for WALVIS have already made many an IT managers
hair go grey.
In lhe vorId of inlernalionaI aymenl lrac ve currenlIy see
many SEPA projects (Single Euro Payments Area); a programme
ofmodicalionslolheIuroeanbankingsyslemsloharmonise
and slandardise lhe aymenl lrac inside and oulside Iuroe
which will run for quite a few more years. Unfortunately, the
Netherlands was already leading the way with customer-friendly
aymenllracsolheaverageconsumermayveIIenduay-
ingmoreforIessserviceullhalsanolhermaer
Financial malpractices in big corporations have resulted in laws,
particularly in the United States, such as the Sarbanes-Oxley Act,
sometimes shortened to SOX (Some take the view that these laws
havegoneloofarSOXismeanlloreslorelheshakencondence
in the capital market and accountants, states the responsibility of
directors of listed companies and prescribes detailed rules for su-
pervision. All that made the so-called SOX compliance an impor-
tant theme for all companies that are listed on the American stock
market directly or indirectly. In Europe we already had the so-
called Basel II Accords and in the Netherlands there was the
Tabaksblat code.
What all these laws have in common is their far-reaching impact
on IT systems
6
and the requirement to demonstrate clearly that
these laws have been complied with, hence the term compli-
ance. The traditional role of highly paid accountants and control-
Iers is suIemenled by or even laken over by IT sla vho
number many testers among them, and who have developed into
genuine compliance specialists. More detailed information can be
found in chapter 20. Since directors, board members, commis-
sioners and chairmen are severally liable for irregularities, the
businesscaseisnolloodicuIlhereandhasbecomeralherer-
sonal: I just want to get this sorted, I dont fancy being locked up
for a couple of years.
6 See also the monthly magazine Informatie, special issue Regulering
en SOX compliance, maart 2008: De gevolgen van Sarbanes-Oxley
zijn veel groter dan iemand ooit had kunnen bedenken (Regulation
andSOXcomIianceMarchTheeeclsofSarbanesOxIeyare
much bigger than anyone could ever have conceived).
EPD
WALVIS
SEPA
SOX
Basel II
Tabaksblat code
compliance
32 Part 1
1.7 Testing in connection with other disciplines
We have seen that testing is one of the key activities to allow a
commissioning party to keep on top of his IT projects and sys-
tems. However, in order to be in control and stay in control more
than just testing is needed. For it is impossible to test properly
without input from architecture, requirements and design.
Moreover, testing is pointless if the products tested are not admin-
istered properly. Therefore, SmarTEST presupposes close co-
operation among seven disciplines:
1 Initiation Plan, objective and frameworks
2 OcniiicnRequiremenlsdenilionandbusinessarchileclure
3 Design IT architecture and high-level design
4 Realisation Development of the IT system (i.e. design, coding,
unit/module/module integration tests)
5 Checks System and acceptance tests, audits, reviews
6 Operation Maintenance, IT support and performance manage-
ment
7 Co-ordination Project management
The emphasis of testing lies with Checks here. Figure 1.4 shows
lhesedisciIinesandlheircohesioneIovveviIIbrieyexIain
each of these and their relation with SmarTEST.
co-operation
Commissioning party
good
commissioning
Accepting party
Initiation
Realisation
Co-ordination
Denition
Design
Operation
Checks
Figure 1.4 Testing (Checks) in relation to adjacent disciplines
Chapter 1 Good commissioning and IT governance 33
Initiation: From idea to plan
During the initiation phase, ideas are elaborated on and turned
into a basic business case. The idea is discussed with stakeholders
and the objectives and frameworks are established. In this phase,
we go from a preliminary idea to a concrete plan. The most im-
portant result of this phase, the business case, is the departure
point for the project and the basis for monitoring and control by
the commissioning party. Testing and the corresponding risk as-
sessment (PRIMA, Product Risk Matrix) play an important part
in validating and if necessary recalibrating the business case.
Denition: Requirements denition and business architecture
This comprises exploring and listing the high-level requirements
and acceptance criteria: the main requirements of the client. That
constitutes the basis for detailed and unambiguous commission-
ing as well as the starting documents for the development and
testing phases. Detailed and unambiguous commissioning on the
bases of well understood requirements prevents costly misunder-
standings on the suppliers part and serves as a reference for sys-
tem acceptance.
Design: IT architecture and high-level design
In the design phase it is shown how the IT system and the project
will manifest itself in the real world. An architecture is developed
for the system that matches the company-wide Enterprise Archi-
tecture, as discussed in paragraph 1.4. The project is taking shape
bychoosingfromarangeofossibIereaIisalionscenariosreecl-
ing the commissioning partys needs. The system test will be
based on these agreed manifestations.
Realisation: System development
The detailed design and the actual construction (coding, unit and
integration testing) of the IT system are carried out by the IT pro-
vider usuaIIy an exlernaI arly Dala denilion and migralion
and (re)design of processes and organisation happen in this stage.
The commissioning partys challenge is to keep exactly the right
amount of grip on these processes. Not focusing too much on de-
tail, as that is what the internal and external suppliers get paid
for, but not leaning back too far either, as any project derailments
might then surface too late. The detailed design and the interface
secicalions are lhe basis for lhe module tests and integration
tests.
Checks: System and acceptance tests
In the checking phase, it is all about accepting the system that has
been delivered. In this phase, the tests that are directly relevant
for the client, that is to say the system and acceptance tests, are
carried out. In SmarTEST-terms, these are the test activities at the
top of the W-model, as discussed in chapter 6. Actually, testing
business case
requirements
design
construction
checks, testing
34 Part 1
will in practice be interwoven with the entire process: the prepa-
ration of testing starts much earlier than the actual test execution
hase eg denilion of lesl scenarios and buiIding lesls can aI-
ready be performed in the Realisation phase. The same applies to
reviews and inspections of various key documents in the project
(see chapter 11).
Operation: Maintenance, IT support and performance
management
After testing and acceptance, the developed product is deployed
in an IT-environment that is well managed from a functional and
technical point of view. This allows for business continuity and
continuous adaptation to the ever changing business and techni-
cal needs. The maintenance of systems, data and documentation
in an inter-related way is thus an ongoing activity across projects.
Application Service Management contacts in particular, serve as
a link between business and IT, crucial for keeping up with the
needs of the business and translating these needs into the realm
of the possibilities and restrictions within the IT support and
maintenance organisation. In this phase performance is moni-
tored and optimised as well, building on performance tests that
were designed and executed in the project (see also the paragraph
Operational Monitoring in chapter 6). All testware developed in
due course of the project is transferred to the support organisation.
Co-ordination: Project management
The scale and complexity of todays IT projects makes it neces-
sary for the client to play a strong coordinating role. Coordina-
tion entails the continuous monitoring of the process as a whole
and the transitions between the phases. This must be done in a
goal-oriented way, ensuring the achievement of the business case
within the agreed frameworks and in accordance with the de-
ned ob|eclives The SmarTIST rocess rovides lhe ro|ecl
manager with factual quality and progress data.
Summarising: success comes from excellent execution of the en-
tire chain from Initiation to Operation. The strength of this chain
from Initiation to Operation is determined by its weakest link.
The SmarTEST process is both a strong link within the chain
as well as a measuring instrument guarding the strength of the
chain as a whole.
maintenance
project management
Chapter 2 There is more to succes than the IT system alone 35
2 There is more to success than
the IT system alone
2.1 Risk control based on requirements
Ieclive IT soIulions come aboul vhere commissioning arlies
know what they want and are able to make it explicit. They need
to make and keep their requirements precise. To list, manage and
verify them, is all related to risk control. If there are no require-
ments, there will be no risk that the result will not meet them.
Because there will obviously always be requirements, we do our
best to make these explicit and formulate them in a measurable
way. That in itself limits the risk that the expectations of commis-
sioning parties, accepting parties and developers diverge. How-
ever, active risk management is also needed. Risk management can
be divided into two focal areas: project management and the pro-
ject result, and linked to these are project risks and product risks
respectively. These focal areas are connected, but they all require
their own approach.
2.2 Product risks versus project risks
Risk control through testing is aimed at the end product: that
which is left when the project is no longer there. Connected with
this are the product risks. By product we mean the project result
in a broad sense. It can be an application or a system, but it might
also include data sets, plus the organisational structure and em-
bedded rocessesAnd lhe secicalions lerms and condilions
and brochures that go with a new type of mortgage, perhaps. In
other words, the information system in the broadest sense.
In addition to this, we have project risks, that is to say, risks con-
nected with project management, with the processes in the project.
IorexamIeIanningnanceandslangrisksIndireclIyro-
ject risks are obviously all to do with the chance of a good end
resuIlSeegure
active risk
management
product risks
project risks
36 Part 1
Project risks are the project managers responsibility, not the test-
ers primarily. Product risks are also the project managers re-
sponsibility, but they are the testers core business.
The purpose of testing is the assessment of the end product as re-
gards quality and risks, two sides of the same coin. The so-called
acceptance criteria, a central element in the smarTEST approach,
are criteria for the end product, formulated in a measurable way.
2.3 Not just a system, but process and information as well
Risks are aached lo aII roducls resuIling from a ro|ecl Ini-
tially, we will think of the information system to be delivered,
which usually consists largely of software.
Subsequently, it is also important that the system functions prop-
erly in the business processes and that people can work with it.
Therefore, its also about the rocessandorganisalioncongura-
tion. For example, the process of dealing with customer queries
arriving via e-mail. Or the entire process of the daily, weekly or
monthly supply of source data to a data warehouse. Most IT pro-
jects or programmes include the assignment to set up these pro-
cesses, and to determine the required organisation.
Another end result of a project could be a clean, complete and
up-to-date data set, for example the customer base, or stock ad-
ministration.
Therefore, the end product is not merely the information system,
but also the processes around it and the corresponding data sets.
These have properties too, to which criteria apply which we want
to verify through testing. A stock-control system, for example,
musl be IIed vilh lhe correcl slarling dala from day one No
maer hov veII lhe syslem erforms if lhe slarling dala are
incorrect, customer orders will be accepted when there is no
testing is
product focused
Risk management
Based on requirements and acceptance criteria
Product Risks
Related to the end result,
when the project has been
decommissioned
Project Risks
Related to planning,
progress, budget, politics,
people issues, etc.
Figure 2.1 Product and project risks
software
process and
organisation as well
and data
Chapter 2 There is more to succes than the IT system alone 37
stock. Or conversely, customers will be referred to competitors
while the warehouse is full of unsold product.
This leads us to the following division into three parts of the end
product to be tested:
1 Information
The information and data that are to be processed by the informa-
tion system must be correct, complete, available, on time, up-to-
date and relevant.
2 Processes and Organisation
The methods of work and procedures must be well adjusted. And
lheorganisalionincIudingslangmuslberearedforlhis
3 System
The information system delivered must satisfy requirements and
secicalionsandvorkveIIvilhad|acenlsyslems
This tripartition is brought together in the SmarTEST IPS model,
asshovningure
The three submodels are described in more detail in the next
chapter.
Iachoflheselhreeareasmayharbourasignicanllhreallolhe
ultimate success of the new information system.
2.4 Integral management of requirements and acceptance
criteria
Half of the investments by governments and companies goes to
IT (source: US Dept. of Commerce, 2002). Many IT projects fail.
According to the Gartner research bureau, 50 per cent of IT pro-
|eclsdonolmeellheirob|eclivesTheseareguresfromlheUnil-
ed States, where a quarter of all investments by the government
and companies, several billions of dollars, just go down the drain.
tripartition
IPS model
Product requirements
The result the business owners will
get once the project has been
decommissioned
Information
Data to and from
the system
Systems
The delivered
information systems
Processes
Organisation and
processes
P S I
Figure 2.2 The IPS model and the three quality dimensions
50% IT projects fail
38 Part 1
Thesearedramalicguresandscoresofresearchro|eclsdem-
onslralelhallhingsarehardIydierenlinIuroe
For a large part, the failed projects are the consequence of a poor
handling of requirements, acceptance criteria and risks, and a
lack of joined-up management. Entire projects go wrong because
people see requirements as a document instead of a process. Ad-
miedIyrequiremenlsaredocumenledallheoulselofaro|ecl
but with every design and elaboration stage, the solution be-
comes further and further removed from the original require-
ments. Moreover, understanding increases over time, and this
resuIls in modicalions lo lhe requiremenls vhiIe ilems have
aIreadybeenbuiIlOnecancomarelhisloereclinglhevaIIsof
a house at the same time as laying its foundation. Since progres-
sive insight is a fact of life and the world keeps changing, trying
to freeze the requirements is a rear-guard action.
Partly because of this, cyclical development methods (design a
IiIebuiIdaIiIeleslaIiIehavebecomeouIarHoveveril
issliIIdicuIllomonilorvhelhervearesliIIaimingallherighl
target and whether we are still doing what we set out to achieve
to begin with. What we need for that at the very least is an up-to-
date and actively managed set of requirements with acceptance
criteria, couched in terms that accepting parties can understand,
plus a process of checks that is actually used to perform checks
based on these requirements and criteria. Moreover, that process
of checks must be able to respond to new or previously unrecog-
nised requirements and risks.
This means that an integral approach of requirements, acceptance
criteria and risks is called for. SuchanaroachIeadslobenels
in terms of Time, Money and Quality:
Time building and testing based on up-to-date and complete ac-
ceptance criteria will, on average, be quicker.
Money co-operation prevents duplication of work and misun-
derstandings and therefore saves money.
Quality geing il righl lhe rsl lime yieIds beer quaIily lhan
making corrections later on.
increasing
understanding
Trying to freeze requirements is a rear-guard action.
The important question is whether you can manage them.
actively managed
requirements are
needed
Time, Money
and Quality
Managing requirements and acceptance criteria costs money.
Working with incomplete and obsolete requirements costs more
money.
Chapter 2 There is more to succes than the IT system alone 39
So, in summary, we need:
an actively managed set of up-to-date requirements, couched in
terms that the client understands;
a set of acceptance criteria that is linked to the former item, to
vhichlheaccelingarlyisaIsocommied
a testing and checking process that is designed to handle this set
of criteria, including changes.
2.5 Success = Quality x Acceptance
To meet the outlined broad range of expectations, a project must
do the right things and do them right. The result should then be
goodquaIilyroduclsAccordingloaIIociaIdenilionsquaI-
ity is meeting objectives (lnessforuroseandlheexIiciland
implicit expectations of the stakeholders. An important part of
the quaIilyfocusedeorlsisaimedalacknovIedginglheimIic-
il aern of execlalion lhal is lhe lacil or even unconscious
execlalionsandmakingilsecicandmeasurabIeIorquaIily
alone is no guarantee of success. A product or a system should
also catch on and appeal widely. Success, therefore, is the result
of quality and acceptance.
Most quality forms are not intrinsic. That is to say, they are not a
property of the product itself, but of the combination of the prod-
uct and its environment. Here is a concrete example: manageabil-
ity can be achieved independently up to a point, but it exists
primarily in the interaction between the system and the mainte-
nance organisation. An application may have excellent adminis-
trative support possibilities, but if IT support does not know how
to use them, the whole thing will remain unsupported. Quality is
not an isolated concept.
We will go one step further: People determine the quality. They
do so in two ways: providing quality is the work of humans as is
perceiving quality. The interaction between system, information
and processes is not optimal unless someone creates quality and
someone else experiences it as such.
AhaacircuIarargumenllhemoreaenlivereadersareIikeIylo
observe now. Thats right, quality and accelanceinuenceeach
other. The success of a new information system with everything
that comes with it is determined by the quality across a broad
spectrum, but even more so by the extent to which the accepting
arlies recognise and areciale lhis quaIilyAnd lhe Iaer as-
pect is probably the result of the degree to which they, perhaps
acceptance
criteria
denition of quality
quality is not isolated
quality is determined
by people
Quality is in the eye of the beholder
acceptance
40 Part 1
onlheinsligalionofaleslmanagerarlicialedindeningand
evaluating the quality requirements.
Dening quaIily requiremenls logelher vilh lhe acceling ar-
ties poses a hidden danger: the bar may easily be raised too high.
It becomes clear that this can work out wrong once we realise that
quaIily is dened by some as Iercelion minus Ixeclalions
[Muntinga et al., 1998]. In other words: if something conforms to
expectations, the accepting parties will not experience this as
high quality and may even be slightly disappointed. The verdict
onlhesameroduclviIIhoveverbeosiliveifilisbeerlhan
expected. That makes expectation management, the art of creat-
ing and stimulating realistic expectations, an essential part of de-
ciding on the acceptance criteria. Deciding together what cannot
be expected under any circumstances is as important as deciding
what has to be provided.
2.6 Prevention and cure
As Benjamin Franklin already said: An ounce of prevention is
worth a pound of cure. IarIyreclicalionismanylimescheaer
lhanIalecorreclionaslhesocaIIedoehmscurveshovsing-
ureTheIalermislakesaresoedlhehigherlhecoslofrec-
licalionviIIbeoehmTheemhasisinquaIilycarehas
traditionally been on prevention. Testing emphasises the safety-
net function: evaluation and correction of defects. However, we
should not take the split between prevention and safety net too
far. Testing actually covers both, for example, with intake reviews
as a prophylactic measure.
For some requirements, revenlivequaIilymeasuresviIIsuce
others will have to be tested in depth after delivery. Sometimes
expectation
management
early rectication
cheaper
Rectication costs
increase exponentially
over time
C
o
s
t

o
f

r
e
c
t
i

c
a
t
i
o
n
System life cycle
Design Requirements Realisation Implementation Deployment
Figure 2.3 Boehms curve
prevention and
correction
Chapter 2 There is more to succes than the IT system alone 41
both approaches are needed. At any rate, an integral approach
oerslhebeslchancesforaro|ecllhaldoeslherighllhingsand
does it well.
2.7 Is six out of ten enough?
Which quality rating do we want? Is six out of ten enough? Would
veralherhaveeighlouloflenDenileIynollenforlhalisvery
exensiveQuaIilydoesnolhavelobeolimisedalaIIcoslsThal
means that we may have to deal with failure costs every now and
then: costs of repair and costs resulting from production errors.
That could be internal damage, but it could also mean external
damage (customers), possibly involving claims and a tarnished
image as a consequence. From an operational point of view, it is
advisable to put up with a certain amount of this kind of failure
costs. After all, if we wanted to eliminate them completely, we
would be asking for perfection, and that would require enormous
investments in prevention and testing.
IigureheIsyoundlherighlbaIanceThereisanoptimum
in the division of the two cost components:
1 test and prevention costs;
2 costs of repair and (commercial) damage.
This sort of illustration always proves to be very useful in discus-
sions on the purpose and necessity of testing. Testing is not a goal
ersenorismoreleslingnecessariIyaIvaysbeerIrofessionaI
testers are always focused on reaching the optimum cost point.
Thal is lhe movemenl fromA lo in lhe gure bul no furlher
This is the point where you are earning money for the commis-
sioning party.
accept failure costs
Optimum
A
A
B
B
Effort
C
o
s
t
s
Test and
prevention costs
Total
Costs of repair and
(commercial) damage
Figure 2.4 Optimum division of costs of prevention, tests and repair
aim for the optimum
Egbert Bouman is product-
manager testen bij Valori.
Hij heeft meer dan twintig jaar
ervaring met IT-kwaliteit en
testen van informatiesystemen.
Als testmanager, adviseur,
docent en publicist heeft hij
zijn sporen verdiend.
Egbert Bouman is product
manager testing at Valori,
The Netherlands. In his
career he has tested almost
everything, from submarines
to datawarehouses. He is the
godfather of the SmarTEST
approach and an experienced
trainer, presenter and thought
leader.
978 90 125 8290 2
980 NUR
ISBN
A SMART APPROACH TO SOFTWARE TESTING
Since the rst edition in 2004 the SmarTEST approach to software
testing has gained increasing popularity in the Netherlands. SmarTEST
combines the thoroughness and reliability of classic test methods with
a smart, pragmatic approach, geared towards the modern practice of
system development.
SmarTEST is a real world approach for a complete and reliable quality
assessment at acceptable costs. Risk and requirements based, whilst
not hampered by a rigid and risk-avoiding attitude.
SmarTEST can be used completely on its own but it also combines
very well with (elements from) other approaches and books. The book
offers the best possible introduction into the state of the art and the
state of the practice of modern software testing.
This book is a handbook for everyone with a keen eye for software
quality in the real world. Its aimed at a wide audience of IT managers,
project managers, test managers and testers.

SmarTEST completely ts the no-nonsense business environment of
our Randstad company. It suits the team and I can recommend it to
everyone. Reiant Mulder, I-bridge management team. I-bridge supports
all of Randstad holdings IT
Easy to read, inspiring to anyone that needs to know more about
testing. The author created a very SMART book for testers and
managers alike. Tim Koomen, independent test consultant, co-author
of TMap Next and TPI
SmarTEST is an easy read. Page by page the book shows the deep
practical experience of the author. Evert van Donk and Wim van
Dommelen in the EDP auditors monthly
SmarTEST is the ideal companion to the iterative and agile develop-
ment methods that we teach. Hossein Chamani, Rotterdam University
SmarTEST is a great best practices framework. Best practices such
as risk based testing are put in perspective. A balanced and accessible
book, deserving the largest audience. Bob van der Burgt, chairman of
Testnet, Program Chair EuroSTAR 2008 and founder of Professional
Testing B.V.
I am condent this book will give rise to an increasing number of
successful IT projects. Annemarie Jorritsma, mayor of Almere, former
vice prime minister
SmarTEST
www.academicservice.nl