ASSESSING CANDIDATES

So Who Have We Really Been Hiring?

We make assumptions about the tools we use to hire people.
But sometimes, we’re assuming a bit too much.
By Gary C. Townsend, head of HR Business Integration and Workforce Analytics, Coca-Cola Africa

easurement as the “scientific” basis of the hiring zation has to get it right. Everything else is a consequence
M process is generally unquestioned.The assumption
is that the tools in use have been validated and are generally
of this choice: coaching, communication, creativity and
innovation, diversity, empowerment, initiative and risk-
accepted in practice as being statistically sound. As a taking, mentoring, personal integrity, planning and
consequence, most of the ongoing efforts of recruiters have organizing, problem solving and decision making, quality
been focused on figuring out how the assessments are of results, teamwork, technical competency, vision, and the
helping delivery of all these strategies such as “recruiting list goes on. All said and done, the selection process, more
the best,” “building world-class talent,” “driving diversity often than not, is really a guesstimate based on interviewer
through creative recruiting strategies,” “finding more experience, some “timeless” tools (and accompanying
innovative ways to hire the best,” and “building pipelines assumptions) we inherit with it, the odd gut-feel (often
for leadership talent.” lumped into the “fit-factor”), and the internal politics of the
Similarly, there’s debate about the transactional aspects day.
of assessments — should we administer Given the aforementioned, maybe the
These very
them with paper or online, for example. In recruiting community should pause to
other words, the use of assessments has
impressive-looking consider the tremendous implications of
become a given. In many ways, the manuals, usually how we construct our assessments and how
perception of the recruiting community is filled with equally they can be affected by things that may seem
that it has moved along to bigger and better impressive statistics, trivial or academic. It is often left in the
things, as though the assessment part of the are often not worth hands of the apparent experts from various
process has been neatly packaged and dealt
the paper they are outsourced companies to select and
with by virtue of its historical contribution. administer these products because of the
written on.
The question, however, remains: Are the perceived overly technical nature of this part
assessments we’re using in fact providing the solid of the recruiting exercise. This is understandable, given
foundation we assume they are? the effort it takes to go through any assessment’s detailed
While all the inspirational strategies provide most technical report to validate its proposed accuracy and
recruiters with great material for planning sessions and applicability. These very impressive-looking manuals,
personal performance-development plans, one cannot but usually filled with equally impressive statistics, are often not
reflect on where it all points to. What is fundamentally worth the paper they are written on.
driving all the fanciful models, and more important, what The truth of the matter is that any interpretation can only
is everyone ultimately attempting to achieve? Once distilled, be as good as the quality of its measure. Herein lies the
it appears relatively straightforward — measuring for the fundamental problem we’re facing — that recruiters are
most suitable person for the job — nothing more and often unaware of how these measures are constructed.
nothing less. When one considers the diligence exhibited throughout the
Hiring is the one objective opportunity that every organi- recruiting processes, the selection of tools and instruments,

©2007 ERE Media, Inc. Journal of Corporate Recruiting Leadership | crljournal.com | JUNE 2007 9
 ASSESSING CANDIDATES

FUZZY MATH

and sometimes even the choice of statistical analysis, it
would make sense to focus equally on the primary question
of the basis of the measurement construction. It’s
impossible to make inferences about who is the better
person for the job without rigorous measurement. This
has to be the pivotal issue once we decide to use any form
of psychological assessment as part of the selection battery.
So, to the point. If we’re looking for the best, how do we
measure for “the best”? And even more important, how
do we know we have a tool that is calibrated to measure the
best in a way that is invariant and can be reproduced
whenever necessary?
For a bit more clarity, let’s deconstruct this using a typical
example of how these rating scales are put together when
developing an occupational or personality instrument. First,
the researcher would establish a group of items
intended to measure a specific construct such
as leadership. After these items were
We have
administered to a selected sample of that we are
individuals, the responses would in turn be measuring and not discussion. If our items reflect a distinct
aggregated and presented as a total scale value. simply counting difference in the levels of endorsement they
In our example, let’s assume that the observations.
researcher is developing a scale to measure
leadership, with high total scores representing
more of the qualities being measured and low scores
indicating less. The items are scored on a five-point Likert
scale: 1 (never), 2 (seldom), 3 (neither most of the time nor
seldom), 4 (most of the time), 5 (always). Given this
context, let’s assume that the following represent a few of
the items measuring this leadership construct.
Figure 1
Let us now assume that the rating-scale response for an
individual on the three items represented above is 1, 3,
and 5, respectively. Traditionally, this person would be
assigned a score of 9 on the leadership scale, and this 9
would then be used as the measure on all further statistical
analyses. Now let’s consider another individual who
responded 4, 4, and 1. It is patent that this person receives
exactly the same score of 9.
Considering this situation, it is clear that we are making
the following assumptions when we add up the rating in this
fashion: 1) that each item contributes equally to the
measurement of the construct, and 2) that each item is
measured on the same scale.
When considering the first assumption, we’re concluding
that each item has exactly the same qualitative value when
measuring this imaginary idea about what constitutes
leadership. However, it is glaringly obvious that each of the
items represented above brings distinctively different
qualitative values to the overall leadership measure we’re
attempting. Consider items 1 and 3 in Figure 1. It is
obvious that a high score on item 1 should carry more
weight than an equally high score on item 3. One could
argue as well that item 2 also holds a relatively
low standing on this leadership hierarchy
to ensure relative to item 1 and possibly a higher ranking
than item 3. This brings us to the heart of the
bring to the leadership ability we’re seeking to
measure, then we cannot avoid analyzing our
data in a way that distinguishes the value that
each item brings to the measurement of the whole thing.
There can be very little doubt that a score of 4 on item 1
contributes significantly more to leadership than a score of
4 on item 3.
The second assumption in turn raises the issue of the
interval scale. By including the five-point Likert scale, we
assume that it represents a uniform
distribution between each point on
the item scale as well as across all
the items. Let’s consider the first
item in our example. Given the
assumption of each point on the
scale being equidistant, we would
imagine that (N) never is equally far from (S) seldom as
(M) most of the time is from (A) always or (S) seldom is
from (NSM) neither seldom nor most of the time, etc.
However, considering the item statement Exploits
information from various constituents to formulate plans
for competing in the marketplace, it could very well be
possible that (N) never and (S) seldom are psychologically

10 Journal of Corporate Recruiting Leadership | crljournal.com | JUNE 2007 ©2007 ERE Media, Inc.
 ASSESSING CANDIDATES
FUZZY MATH
much closer to each other (in the minds of the
respondents), as are (M) most of the time and (A) always.
Let us explore this graphically using item 1 (see Figure 2).
Responses based on the assumption of linearity and
equidistance would graphically look something like this (the
assumption being that the difference between N and S is the
same as that between S and NSM, and so on).
Figure 2
In reality, and as explained above, there’s a very distinct
difference in respondents’ psychological interpretation of
the distances between these various options. Respondents
to this and other items would find the choice between (M)
more often than not and (A) always much easier than
between (S) seldom and (NSM) neither seldom nor most
of the time. (N) never and (S) seldom are also much more
difficult to choose between in terms of endorsement.
If we were to graphically lay this out, we would note that
the relative distances in terms of strength of endorsement
look something like this:
Figure 3
This spatial representation of the psychological impact of
making these choices demonstrates that there’s a large
psychological difference between endorsing A on this
fictitious leadership item and rejecting it N — (d). The
decision is relatively definitive — you either exploit
information from various constituents or you don’t.
However, when one considers the psychological shift that
has to be made in deciding to endorse either (M) more
often than not or (A) always, the boundaries start blurring
considerably — (b).
The choice now becomes more reflexive. This could be
based on a host of psychological precursors. The
©2007 ERE Media, Inc.
respondent, for example, could have just completed an
exercise gathering strategic market-related intelligence.
On the other hand, the respondent could be accessing
information based on similar but more historical personal
actions. These two scenarios could be all that separate the
choice of always from the choice of most of the time!
However, it’s very unlikely that, given the suggested context
above, the second scenario
would result in a tossup between
(NSM) neither seldom nor most
of the time and (M) most of the
time — (c).
In much the same way as explained above, this lack of
linearity is usually evident across all the items as well. Here’s
what I mean by that. The psychological distance (value)
between NSM and M — (c) on item 1 may be very different
from that of item 3.
Two prospective applicants with exactly the same psycho-
logical style and associated skills being identified by item 1
could very possibly view the psychological distinction
between NSM, M, and A very differently.
One individual, for example, could feel comfortable
selecting M as a response, while another would have no
problem selecting NSM. This
would have minimal impact when
hiring some frontline supervisor,
but when we start dealing with
high-stakes personnel, it can so
easily translate into either an
unqualified success or devastating
failure for the organization. So one
cannot make the assumption that
the “value” of a move from NSM to M is the same as that
of a move from M to A.
Further, simply tallying raw scores and using them as an
indicator of the strongest candidate on this particular item
will most definitely result in a bias against the individual who
selected (M) most of the time as opposed to (A) always.
In essence, raw item ratings are unable to factor in this
lack of linearity both within and across the various items
measuring a specific quality such as leadership. The
majority of instruments currently in circulation perpetuate
this fundamental weakness in their designs. They confuse
counts with measures.
Journal of Corporate Recruiting Leadership | crljournal.com | JUNE 2007 11
 ASSESSING CANDIDATES

FUZZY MATH

The quantitative observations they use to arrive at a final

score are grounded in counting observed events or, as in
this example, leadership properties, whereas for any
measurement to be meaningful, it must be based on the
arithmetical properties of the interval scales used. So, before
we even begin to consider whether one candidate is better
suited for a particular job than another, we have to be
assured of this fundamental prerequisite — that
measurement can only take place once a calibrated
measure, with a well-defined origin and unit, has been
constructed in a way that can be consistently and reliably
used. We have to ensure that we are measuring and not
simply counting observations.
To date we have readily accepted the various reports and
analyses stemming from a host of assessments without any
reservations. I propose that the next time we pick up the
phone to call our favorite assessment consultancy or reach
for one off the shelf, we are able to answer the question,
Does this instrument allow me to independently estimate
the measure of the person’s ability or endorsement level
on the latent trait and the level of difficulty of the various
items on the same latent trait and yet compare them
explicitly to one another? After all, how accurate could any
selection process be if the determining psychological
construct has never been measured comprehensively in the
first place?

Gary C.Townsend is head of HR Business Integration

and Workforce Analytics at Coca-Cola Africa.
gtownsend@afr.ko.com
Gary Townsend joined Coca-Cola Southern & East
Africa Division as part of a team that developed and
managed a unique marketing management training
program for graduates. He was tasked with the curriculum design,
assessments, recruitment, training, management, and placement.After
three years, he moved into the role of human resources information
strategy manager for the Coca-Cola Southern and East Africa
Division. In 2005, he assumed the role of head of HR Business
Integration and Workforce Analytics as a member of the Center of
Excellence for Coca-Cola Africa, with an accountability extending
across 56 countries. In this role he provides strategic direction and
thought leadership in supporting the Africa Group and local division
strategies.

12 Journal of Corporate Recruiting Leadership | crljournal.com | JUNE 2007 ©2007 ERE Media, Inc.