ISSN 1866-5705

www.testingexperience.com free digital version print version 8,00 € printed in Germany 11

The Magazine for Professional Testers

Metrics
September 2010

© sandra zuerlein - Fotolia.com

 © Otmar Smit - Fotolia.com

Indicators of Useful or Useless Indicators

by Issi (Issachar) Hazan-Fuchs

During the past few years, I have been involved in producing and
reporting testing indicators. I have observed good and less good
indicators, learned from my own and others successes and mis-
takes, while reaching agreement with stakeholders on format.
I would like to share my own experience, both good and bad, and
give indicators to assist you in evaluating the usefulness of your
own testing indicators.
it3.
Note: Following the spirit of this article, these are only indicators
and not absolute facts. Like any indicators, you should use them
only as aids for the evaluation of your indicators.
Useful Indicators:
• Indicators produce a discussion that initiates decisions

I am going to specifically discuss my experiences with quality in-
dicators (bug counts) and with test execution indicators.
Before we start, we must define what “useful” means in our case.
My understanding of “useful” indicators has been shaped by trial
and error and by reading the thoughts of others1.
The most simple definition that I can define as a baseline for this
article is:
A good indication that our indicators are useful is when the-
se three parts - Indicators, Discussion and Decision - exist and
connect in a cause and effect connection. Otherwise indicators
are just wasting time for the people who produce or view them.
Discussions without decisions might indicate that the indicators
were interesting. but did not provide the data for decision ma-
king (unless we have a decision-making problem). If we have indi-
cators which initiate decisions by themselves, we probably have a
problem of over- valuing the indicators.

Useful indicators aid the decision-making process

Note that I used the term indicators rather than metrics, to em-
phasize the fact that metrics are not employed to measure for the
sake of measurement, but to aid decision making and the resul-
ting actions. I also highlighted the verb aid to mark that useful
indicators are not a direct baseline for decisions, but triggers for
inquiries that will be the baseline for making decisions. The in-
quiry itself cannot be done by any formula, but by interviews of
people, observations and analysis2.

I will try to list a few indications that I found to be signs of useful

indicators, and indications, that I found to be signs of non-useful
indicators.

In addition to the play on words in the headline of this articles,

describing the “Indicators of useful or useless indicators” from
my own experience is an opportunity to share my own learning
process of observing an indication and deriving a heuristic from

1 I highly recommend reading Michael Bolton’s post “Meaningful

Metrics” which describes an approach for meaningful metrics. See
http://www.developsense.com/blog/2009/01/meaningful-metrics/
3 Thanks to my friend and colleague Shmuel Gershon for review-
2 Ibid. ing the draft of this article.

74 The Magazine for Professional Testers www.testingexperience.com

Although I have never heard of a machine which automatically
makes the decision to release a build according to the number of
reported bugs on the code base, the problem of viewing the num-
bers as targets instead of aids to the decision making process, is
common.
As opposed to the continuously changing indicator set described
above, the opposite could also occur. Where the indicator set is an
old one that you inherited without questioning, there is a good
chance that it is out of date for your up-to-date needs.
“Use with caution” indications

Discussions and decisions will not happen each time we extract • When a second mathematical function is involved
the indicators. When defining the indicator sets, imagining what-
if scenarios and assessing whether they are capable of inducing Some indicators introduce a weight function that unifies a few
discussion that will lead to decisions, can be a good way to assess items that are measured together. For example, if there are some
them. components of our software which are more risky than others,
we will use a function that will assign a weight to components
Useless (or less useful) Indicators’ Indications:
according to our risk level analysis.
• Indicators are happy while the project is sad
This approach can be useful since the indicators looks more
While corrections, exceptions and complimentary informati- unified and complete, but the mix between the subjective inter-
on are essential to complete the picture drawn from indicators, pretations and the numbers can be misleading. In the correct
when time after time you have to complete the picture by dra- context it is definitely OK, but in some contexts, especially when
wing an opposite picture to the one that is apparent from the indicators are a sensitive subject, it’s better to have the bare num-
quantities indicators, your indicators might be wrong, or the bers followed by a verbal interpretation for discussion, so that the
whole activity is going in the wrong direction. For example, week numbers themselves will not be subject to debate.
after week getting green “pass” bars for the test execution while
you have to express your concerns that the product itself is not I hope that I have given you some good indicators and food for
useable or too buggy can indicate that you are publishing the less thought so that in future you will ask your own indicators, “Do
important indicators, or that a major activity of your test team is you work for me (and the project’s success)? Or do I work for you?”
focused on a less critical path of your product quality. – Such questioning can be a good start to better and more useful
indicators.
I could say the same about “Sad Indicators in a happy project”.
While working on this article a software developer that I know
told me about a project she was involved in. The project mana-
ger asked to delay the release because the high-severity defect
numbers were too high, but inquiry proved that the defects were
not classified correctly and they were only minor cosmetic ones.
The project manager understood his mistake and suggested to
release the product, but the developer objected. Not due to the
number of open bugs, but because the type of bugs showed that
the test team spent the time on finding minor, redundant bugs
instead of testing the relevant functionality.

• Extracting the indicators consumes more than a few minu-

tes

Although there is no direct relationship between the time it takes

to extract the indicators and their quality. In cases where extrac-
ting the indicators becomes a long task that consumes too much
precious time, this can indicate that the indicator set is too com-
plicated or too large. Automating the indicators is a good idea
in such cases, but before you set off, you should first revisit your Biography
approach, in order that you will help your process and not just
Issi (Issachar) Hazan-Fuchs has been testing drivers,
solve the time problem whilst preserving the over-complicated firmware and software for more than 10 years in the
approach. Functional Testing Lab of Intel® Design center in Jerusa-
lem, Israel.
• Indicator set continuously change during the project During the last years, Issi was testing technical lead of
several projects, and was involved in defining and pro-
Although being responsive and tuned to the stakeholders’ needs ducing indicators for the product development team
is a good thing, so is being flexible about your own process. In and other stakeholders.
cases where there are repetitive change requests to the indicator Issi publishes his thoughts about testing in his blog at:
http://testermindset.blogspot.com
set during the project, it can indicate that you have a fundamen-
tal problem with the indicator set. Either the stakeholders do not
understand or agree on the purpose of the indicators, or your in-
dicators are not defined in a useful manner.

• Indicator set remains the same for ages

www.testingexperience.com The Magazine for Professional Testers 75