Economics

Mark Esposito Terence Tse

Professor of Business Associate Professor of

and Economics at Finance, ESCP Europe
Grenoble Ecole de Business School and
Management, Harvard Head of Competitiveness
University Extension and Studies, i7 Institute
IE Business School. for Innovation and
Competitiveness

The Fallacy of
Measurement
BY MARK ESPOSITO AND TERENCE TSE

In the past two decades, the number of perfor-
mance indices has proliferated substantially.
There are good reasons why performance indices
have blossomed: they catch attention and they
make information easily understandable. In this
article, Terence Tse and Mark Esposito discuss
the abilities of indices and glean insights into
how they work by highlighting their strengths
and weaknesses.
Through such manipulation, not surprisingly, statis-
tics has become perhaps one of the most unloved sub-
jects. Popularised by Mark Twain, the phrase “There are
three kinds of lies: lies, damned lies, and statistics” de-
“There are three scribes the persuasive power of numbers, particular-
kinds of lies: lies, ly when used deliberately to bolster weak arguments.2
Others in history often show no more leniency. For in-
damned lies, stance, in his book “How to lie with statistics”, a guide that
and statistics”
describes the to, Daniel Huff lamented “Crooks already know these

I
t is no secret that statistics have are often have
been manipulated to alter perceptions. Just con-
sider the following example: In the late 1970s, the
Mexico City government increased the capacity of its
four-lane motorway by re-painting lines to make it six
lanes wide, which, in effect, gave it a 50% rise in ca-
pacity. Unfortunately, this also resulted in an increase
in fatal accidents. So, after a year it was changed
back to a four-lane road, representing a 33% capac-
ity reduction. Yet, the government, in an attempt to
support its claims of social progress, subtracted that
33% drop from the 50% increase to claim a net in-
crease in capacity of 17%.1
persuasive power tricks. Honest men must learn them in self defence”.3
Numerical indicators can often speak louder than
of numbers, qualitative evidence. In our view, the worst culprit
particularly when of them all is the calculation of Gross Domestic
used deliberately Products (GDP). Even though different govern-
to bolster weak ments, research institutes as well as economists fre-
quently estimate GDP differently, its fundamental
arguments. compositions always include the following:
GDP = Consumption + Investment + Government
spending + (Export - Import)
history, possible to establish how much a nation
produces. And the equation is only matched by its

34 The World Financial Review September - October 2015

 Indices – of which there is now seemingly
one for everything – can therefore come
in as a handy tool to exert social pressure,
especially on governments and politicians
to take (corrective) actions.

pollution, from which the environmental damage can

now be hardly reversed. Russia’s stellar GDP growth in
the past decade did not reveal the problem of solely de-
pending on oil and gas export,5 the dire consequence of
which has led to the country’s current economic plight.

Indices: Old Tricks in New Packaging

Befuddling with numbers has lately – and unfortu-
nately – gone on a new height: in the form of indices
and rankings, which weaves data on lots of measures
into a single easy-to-understand ranking.6
In the past two decades, the number of performance
indices has proliferated substantially (see Figure 1 on the
next page). There are good reasons why performance
indices have blossomed: they catch attention and they
make information easily understandable. But perhaps
the most potent of them all: backed by the appearance
mathematical elegance. Yet, when Simon Kuznets
- index serves as a named, rank-ordered representation
try’s income, he expressively pointed out that many of a past or projected performance that uses numerical
data to simplify otherwise very hard-to-describe social
if not impossible, to quantify. He stated that, phenomenon. This, in turn, allows for direct compari-
“The valuable capacity of the human mind to sim- son between countries or persons and to evaluate their
plify a complex situation in a compact characterization performance by reference to one or more standards.7
becomes dangerous when not controlled in terms of Indices – of which there is now seemingly one for
- everything – can therefore come in as a handy tool to
- exert social pressure, especially on governments and
gests, often misleadingly, a precision and simplicity in politicians to take (corrective) actions.
the outlines of the object measured. Measurements -
of national income are subject to this type of illusion tics and abilities to apply pressure as well as shortcom-
and resulting abuse, especially since they deal with ings. Subsequently, we turn to look at four examples of
indices to glean insights into how indices work by high-
social groups where the effectiveness of an argument lighting their strengths and weaknesses. This article ends
4
with some concluding remarks as well as suggestions.
Despite such warning, over time, we have focused
- Indices I Love You
ed those that are not. It has become the sine qua non for Indices – also called rankings and composites –
economic progress. The grand result of such bamboo- compile numerically-rendered data and repackage
zle with numbers is now felt: in the pursuit of GDP statistical information. Given the numerical nature of
growth, the Chinese government has for years ignored indices, they have the characteristics that follow:

35
 Economics
Simplification of complex reality.
of rankings is that they are often numerical representa-
tions of complex phenomena. The result is that they are
capable of presenting a picture as well as telling a story in
and social activities that range from state qualities (e.g.
“transparency”) to state policies (e.g. “press freedom”)
to prevalent social practices (e.g. “corruption”).8
Admittedly, boiling down a great deal of contex-
tual information to some single numbers or “scores”
as well as stripping down meaning from the observed
phenomenon may sound like a self-defeating exer-
cise. But indices do offer something that are easy-
to-understand, convenient and easy-to-use. Hence,
they provide an instant idea of the success of coun-
tries and policies in the areas of concern. Moreover,
they repackage statistical information but do not nec-
* As of 2011
Source: http://www.
essarily engage in statistics, their users need nothing
economist.com/news/ more than basic arithmetic to make sense of results.
leaders/21631025-learn- Viewing from this perspective, rankings are an excel-
ruses-international-country- lent tool that everybody can understand and that ev-
rankings-how-lie-indices erybody can use for a wide range of purposes.
Figure 1. Number of indices in the world
very complex reality is often reinforced by the name of
the indices themselves: the name is often designed in a
way to stress what the index claims to measure or rank.
Calling an indicator a measure of “transparency” or
“human development” purport that such phenomena
are directly observable and the numerical representation
36 The World Financial Review September - October 2015
that measures them are in existent. An indicator may
even create the phenomenon it claims to measure, as IQ
9
Indeed, names
are sometimes deliberately designed to create controver-
sy, thereby used for grabbing more attention, faster.
Conciseness and comparability. The beauty of
rending vague issues into precise scores, in addi-
tion to making indices easy reading, is that the eco-
nomic, social and political aspects of our lives can
now be rated: indices have become “psychological
rules of thumb” that are clear and simple symbols,
which in turn help the examiners, the examinees and
third parties deal with the complex reality.10 In ad-
dition to simplicity, concise indicators can add pro-
cessing speed as they can be scanned in moments,
while reading the underlying reports on which they
are based would involve substantially more time.11
that they can be placed in orders, making it possible
for countries to be ranked and explicit comparison
among units to be fostered. Furthermore, scores can
be averaged, thereby establishing “norms” and “stan-
dards”, against which units can be compared.12
the scores calculated and complied periodically, time-
series analysis that shows how countries perform on a
certain issue over time can also be conducted.
Subject matter expertise (to the index builder).
methodologies and robust data, others are founded
on a little more than “guesstimation”. Even so, these
latter indices can deploy various techniques to generate
got The Economist to publish an article called “How
to lie with statistics”.13
the rankings seem so persuasive that they cannot be
ignored, which in turn reinforces the expertise.14
This may explain why performance index build-
ers often rely on respect and credibility that are af-
-
ence. For instance, an index developed in Switzerland
is likely to carry a great deal of weight than a similar
index that is created in Russia. This is not necessarily
because the Swiss have more coercive power; instead,
they could have a much higher credibility for the pur-
poses of such a rating.15
You’ve Got the Power
Given the ease of processing and understanding, as
well as the perception of expertise, most performance
indices, especially the prominent ones, can assert in-
- also have been over-estimated. One of the major
itoring: continuously observing and checking the
progress or quality of a policy, practice or condi-
tion. The result is that they can mobilise and inform
transnational pressure. Explicit comparisons, once
public discourse.16 Rankings can therefore be a par-
ticularly potent lever for creating social pressure.
grew dramatically in countries covered by the index.
Moreover, there is a fear of international opprobri-
um. A 2010 press release from Pakistan’s Interior
“improve the stature of Pakistan before the world”.
Similarly, Germany’s underperformance in OECD’s
Programme for International Student Assessment
(PISA), which compares the academic performance
of pupils of 15 years of age in dozens of countries,
pushed the country to reform school as well as the cre-
ation of a €4 billion education support programme.17
What We Should be Aware of with Indices
In 2008, we heard the following statements18:
• The director-general of the Malaysian Industrial
Development Authority insists that “Malaysia
aims to move from the 24th to a top 10 position in
Doing Business index”;
• Akylbek Zhaparov, Minister of the Economic
Development in Kyrgyzstan, expressed the wish
that the country “shall rank among the top twenty
countries in the Doing Business rating in three
years”, from its current position of 94th;
• Macedonia placed a one-page advert in The
Economist's
in 2008” to announce that it had improved their
position on the Doing Business Index.
- liberately chosen message. Others are often made up
able, raising awareness does not always lead to actions.
In the 2015 ranking, Malaysia and Kyrgyzstan holds
the 18th and 102th position, respectively.19 Granted,
there are plenty of reasons that prohibit countries
from improving their ranking. Yet, even for those who
are able to go up on the list, increasing in ranking does
not necessarily mean improving the real performance.
Countries can choose to focus their efforts on those
criteria with the goals of boosting their ranks.20
The “attention-grabbing” power of indices may
weaknesses, in particular with those rankings that
include a large number of countries, is that media is
likely to only be interested in those that are at the top
and bottom – how those countries in-between have
performed are often overlooked. The result is that
indices may not have as much social power as many
people believe.
Another problem is that the ordinal structure of
ranking could lead to over-reaction. Each year when
a ranking is published, much public attention focuses
One of on the relative rank of a country to its rank in prior
the major years and to the rank position of competing coun-
tries. For instance, when Pakistan (ranked 138th on the
weaknesses, Human Development Index) bypassed India (ranked
in particular 139th) in 1998, The Tribune wrote: “Pak beat India, both
lose!” In 2001, when Canada lost the number one po-
with those sition, it prompted The National Post
rankings that not No. 1! Canada drops in UN rankings... Prime
Minister Jean Chretien often refers to the report in
include a public statements and speeches”. A past study shows
large number that they do not necessarily mean that countries have
performed less well in absolute terms. Yet, the order
of countries, of things has created the wrong perception.21 This
is that media shows that statements based on ordinal comparisons
have to be interpreted with great care.
is likely to
only be Conclusions and Implications
Indices, especially the most prominent ones, are often
interested in tools of social power that can affect the behaviour
those that of policy makers and governments. This is not the
least because they can turn complex matters into
are at the top concise indicators that can be scanned in a matter
and bottom.
issues allow for immediate and direct comparisons
and benchmarking.
But as much as indices could tell a story, these
stories are often told in a distorted way. Some indices
are created just to put forward a subjectively and de-
with a limited set of perspectives. To Michael Porter
and his team, for instance, there are other aspects in
the society that matter at least as much as national
competitiveness. They include access to opportuni-
ties, suicide, property rights, school attendance, at-
titudes toward immigrants and minorities, oppor-
tunity for women, religious freedom, nutrition and
launched Social Progress Index.22
37
 Economics
direction, as existing indices are often overwhelming-
ly one-dimensional. In a world where most indices
delve into more narrow spectra of our lives, it makes
more sense for us to widen our scope. Surely, access
to data is often a constraint. Yet, we see that supra-na-
tional organisations such as the United Nations hold
a great number of very rich datasets. As the different
agencies have opened up their databases, we can see
that there is much to be gained for everyone.
In a world Yet here we are not suggesting any “super index”;
we simply subscribe to the view that if we can
where most combine more of the different and diverse facets of
indices delve our lives into single composites, governments and
into more power that many current indices have potently pro-
jected, it is possible to create a better future in this
narrow way for all of us. After all, like in any relationship, we
spectra of should not simply be asking “what’s wrong with you”.
our lives, it “how we can make everything around us better”. Our
passion for indices should not be different.
makes more
sense for About the Authors
Dr. Mark Esposito is a Professor of Business
us to widen and Economics, teaching at Grenoble Ecole de
Management, Harvard University Extension and IE
our scope. Business School. He serves as Institutes Council Co-
Leader, at the Microeconomics of Competitiveness
program (MOC) at the Institute of Strategy and
Competitiveness, at Harvard Business School. Mark
consults in the area of corporate sustainability, com-
plexity and competitiveness worldwide, including ad-
vising to the United Nations Global Compact, na-
tional banks and NATO through various Executive
Development Programs. From 2013-14, Mark advised
the President of the European Parliament, Martin
Schulz, in the analysis of the EU systemic crisis and
Economic Forum reports on Innovation Driven
Entrepreneurship. Mark holds a PhD in Business
and Economics from the International School of
Management, in a joint program with St. John’s
University in New York. He tweets as @Exp_Mark
Dr. Terence Tse is an Associate Professor of Finance
at ESCP Europe Business School. He is also Head of
Competitiveness Studies at i7 Institute for Innovation
and Competitiveness, an academic think-tank based in
Paris and London. In addition to working in govern-
mental advisory capacity, Terence writes extensively
38 The World Financial Review September - October 2015
and appears on television programmes in China,
France, Greece and Japan, discussing the subjects of
competitiveness and economic affairs. Before joining
academia, he worked in mergers and acquisitions at
Schroders, Citibank and Lazard Brothers in Montréal
and New York. Terence also worked as a business
consultant both independently and at Ernst & Young
PhD from Cambridge in Management Studies. He
tweets as @Terencecmtse.
References
1. http://www.economist.com/node/160593
2. The origin of the quote remains to this day uncertain.
See http://www.york.ac.uk/depts/maths/histstat/lies.htm
3. Huff, Daniel (1991) How to Lie with Statistics, Penguin Business
4. Kuznets, Simon (1934) National Income 1929–1932. A report
6.http://www.economist.com/news/leaders/21631025-
learn-ruses-international-country-rankings-how-lie-indices#mm
7. Davis, Kevin E., Kingsbury, Benedict and Merry, Sally Engle
(2012) “Indicators as a Technology of Global Governance”,
Law & Society Review, 46 (1), 71-104
8. Kelly, Judith G. and Simmons, Beth A. (2014) “Politics
by Number: Indicators as Social Pressure in International
Relations”, American Journal of Political Science (forthcoming)
9. Davis, Kevin E., Kingsbury, Benedict and Merry, Sally Engle
(2012) “Indicators as a Technology of Global Governance”,
Law & Society Review, 46 (1), 71-104
10. Sinclair, Timothy (2005) The New Masters of Capital: American
Bond Rating Agencies and the Politics of Creditworthiness, Cornell
University Press
11. Löwenheim, Oded (2008) “Examining the State: A Foucauldian
Perspective on International ‘Governance Indicators’, Third World
Quarterly, 29(2) 255-274
Benchmarks, Shame, and Learning in the ILO Labor Standards
Monitoring Regime”, International Studies Quarterly, 44, 643-666
13. http://www.economist.com/news/leaders/21631025-learn-
ruses-international-country-rankings-how-lie-indices#mm
(2012) “The tyranny of international index rankings”, Journal of
Development Economics, 97(1), 1-14
15. Kelly, Judith G. and Simmons, Beth A. (2014) “Politics
by Number: Indicators as Social Pressure in International
Relations”, American Journal of Political Science (forthcoming)
16. Andreas, Peter and Greenhill, Kelly M. (2010) Sex, Drugs,
,
Cornell University Press
17.http://www.economist.com/news/inter national/
21631039-international-comparisons-are-popular-influential-
18. These examples are drawn from Høylanda, Bjørn, Moeneb,
index rankings”, Journal of Development Economics, 97(1), 1-14
Doing Business
(2012) “The tyranny of international index rankings”, Journal of
Development Economics, 97(1), 1-14
Maximilian (2010)
Error: Evidence from the Human Development Index
Paper 16572
22. http://www.socialprogressimperative.org/