See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/331765650

Does the Number of Response Options Matter? Psychometric Perspectives

Using Personality Questionnaire Data

Article  in  Psychological Assessment · March 2019

DOI: 10.1037/pas0000648

CITATIONS READS

19 2,554

4 authors, including:

Leonard J Simms Kerry A. Zelazny

University at Buffalo, The State University of New York University at Buffalo, The State University of New York
80 PUBLICATIONS   4,565 CITATIONS    4 PUBLICATIONS   146 CITATIONS   

SEE PROFILE SEE PROFILE

Trevor Williams
University at Buffalo, The State University of New York
7 PUBLICATIONS   156 CITATIONS   

SEE PROFILE

Some of the authors of this publication are also working on these related projects:

Latent Structure of Narcissistic Personality Disorder View project

MMPI-2-RF work View project

All content following this page was uploaded by Leonard J Simms on 17 March 2019.

The user has requested enhancement of the downloaded file.

Psychological Assessment
Does the Number of Response Options Matter?
Psychometric Perspectives Using Personality
Questionnaire Data
Leonard J. Simms, Kerry Zelazny, Trevor F. Williams, and Lee Bernstein
Online First Publication, March 14, 2019. http://dx.doi.org/10.1037/pas0000648

CITATION
Simms, L. J., Zelazny, K., Williams, T. F., & Bernstein, L. (2019, March 14). Does the Number of
Response Options Matter? Psychometric Perspectives Using Personality Questionnaire Data.
Psychological Assessment. Advance online publication. http://dx.doi.org/10.1037/pas0000648
 Psychological Assessment
© 2019 American Psychological Association 2019, Vol. 1, No. 999, 000
1040-3590/19/$12.00 http://dx.doi.org/10.1037/pas0000648

Does the Number of Response Options Matter? Psychometric Perspectives

Using Personality Questionnaire Data
Leonard J. Simms, Kerry Zelazny, Trevor F. Williams, and Lee Bernstein
University at Buffalo, The State University of New York

Psychological tests typically include a response scale whose purpose it is to organize and constrain the
options available to respondents and facilitate scoring. One such response scale is the Likert scale, which
initially was introduced to have a specific 5-point form. In practice, such scales have varied considerably
This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.

in the nature and number of response options. However, relatively little consensus exists regarding
This document is copyrighted by the American Psychological Association or one of its allied publishers.

several questions that have emerged regarding the use of Likert-type items. First, is there a “psycho-
metrically optimal” number of response options? Second, is it better to include an even or odd number
of response options? Finally, do visual analog items offer any advantages over Likert-type items? We
studied these questions in a sample of 1,358 undergraduates who were randomly assigned to groups to
complete a common personality measure using response scales ranging from 2 to 11 options, and a visual
analog condition. Results revealed attenuated psychometric precision for response scales with 2 to 5
response options; interestingly, however, the criterion validity results did not follow this pattern. Also,
no psychometric advantages were revealed for any response scales beyond 6 options, including visual
analogs. These results have important implications for psychological scale development.

Public Significance Statement

We studied several aspects of questionnaire response scales and their impact on the reliability and
validity of a personality measure. We found that smaller numbers of response options have a negative
impact on the measurement precision of a scale, which has important implications how psychological
measures should be developed and used in practice.

Keywords: Likert scales, scale development, psychometrics, reliability, validity

Supplemental materials: http://dx.doi.org/10.1037/pas0000648.supp

Measurement tools in psychology and the social sciences more
generally—including a wide variety of self-report, interview-based,
and observational methods—typically include a response scale
whose purpose it is to organize and constrain the options available
to respondents and facilitate scoring. Today, response scales vary
considerably in terms of the nature and number of response op-
tions. Arguably the most common response scale used is that
introduced by Rensis Likert in 1932, as part of his doctoral
Editor’s Note. Yossef S. Ben-Porath, Editor, served as the sole action
editor for this submission.
Leonard J. Simms, Kerry Zelazny, Trevor F. Williams, and Lee Bernstein,
Department of Psychology, University at Buffalo, The State University of
New York.
This study was supported by a research grant to Leonard J. Simms from
the National Institute of Mental Health (R01MH080086).
Correspondence concerning this article should be addressed to Leonard J.
Simms, Department of Psychology, University at Buffalo, The State Univer-
sity of New York, Park Hall 218, Buffalo, NY 14221. E-mail: ljsimms@
buffalo.edu
dissertation, to measure a range of psychological attitudes. The
original “Likert” scale (pronounced “lick urt,”/‘lIk.ɘrt/) included
five symmetrical and balanced options reflecting degree of agree-
ment: strongly agree, agree, undecided/neither, disagree, or strongly
disagree. However, despite the ubiquitous place that response
scales occupy in psychological measurement, the Likert scale has
been extended and elaborated in numerous ways in the years that
have passed since its original introduction, often with little empir-
ical justification. Although a small (but inconsistent) literature
exists regarding the nature and number of response options to
include in psychological measures, measurement lore rather than
data typically guides the choices scale developers and researchers
make regarding response scales.
Given this backdrop, a number of questions frequently are
posed regarding the use of Likert-type items on psychological
questionnaires. First, is there a “psychometrically optimal”
number of response options? Second, is it better to include an
even or odd number of response options? And finally, do visual
analog scales—in which respondents simply make a mark (or
move a slider) along a line ranging from agree to disagree—
offer any advantages over traditional Likert-type items (e.g.,
permitting the possibility finer distinctions along the response
scale)? In this article, we briefly review the literatures related to

1
 2 SIMMS, ZELAZNY, WILLIAMS, AND BERNSTEIN

each of these questions and then offer a summary of fresh data
that were collected to address each.
Is There a “Psychometrically Optimal” Number of
Response Options?
Despite the central importance of response scale to most ques-
tionnaire and rating scale measures of psychological constructs,
little consensus has emerged in the literature regarding the number
of points to include in a Likert-type rating scale. Likert’s (1932)
original scale included five options, as described above, but even
a casual look at the measures used in practice and research reveals
that measures range widely in the number of response options they
This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.
“I would like the work of a librarian,” on a 10-point Likert scale
ranging from 1 ⫽ very strongly disagree to 10 ⫽ very strongly
agree (see Table 1 for an example of all point labels across such
a scale). How finely can a respondent discriminate along this
scale? Is a response of 9 (strongly agree) reliably different than a
response of 10 (very strongly agree)? Although an empirical
question, anecdotal evidence from years of developing and using
scales of various lengths in our laboratory suggests that there is a
point of diminishing returns with respect to the number of response
options.
Some literature is consistent with this perspective. Bendig (1953)
reported equal reliability for three, five, six, or nine response
options but a reliability decrease for 11 options. More recently,

offer respondents. Several lines of thought may influence one’s

numerous studies have concluded that measurement precision (and
This document is copyrighted by the American Psychological Association or one of its allied publishers.

view on the optimal number of points to include on a Likert scale.

First, given the importance of score variance in classical test
theory, response scales that result in increases in valid score
variance should increase measurement precision and, thus, maxi-
mize validity coefficients. By this way of thinking, some scale
development lore suggests that longer response scales are prefer-
able since they will increase variability in total scores and thus
maximize precision and validity. But the key is that the variance in
scores must reflect reliable and valid distinctions among individ-
uals on the psychological characteristic being evaluated. Some
evidence would seem to support this perspective. Several studies
suggest that estimates of score reliability increase with the number
of response options (e.g., Finn, Ben-Porath, & Tellegen, 2015;
Flamer, 1983; Hilbert, Küchenhoff, Sarubin, Nakagawa, & Büh-
ner, 2016; Weng, 2004).
Second, we might expect measurement precision and validity to
asymptote (or perhaps even decrease) at some number of response
options if the added score variance offered by more differentiated
response scales reflects psychometric noise rather than signal. That
is, it is possible (perhaps even likely) that humans’ ability to make
fine-grained distinctions about relatively fuzzy and complex psy-
chological constructs is not without limits (Cox, 1980; Symonds,
1924; Tourangeau, Rips, & Rasinski, 2000). For example, imagine
a hypothetical respondent trying to provide a response to the item,
sometimes validity) does indeed asymptote. Lee and Paek (2014)
concluded, based on simulated data, that four to six response
options is the ideal number, although they examined only up to six
response options. Lozano, García-Cueto, and Muñiz (2008) con-
cluded that 4 to four to seven increases after seven options. Preston
and Colman (2000) investigated all possible response scales be-
tween two and 11 options on a measure of service elements
associated with a recently visited store or restaurant, finding that
(a) two to four response options performed poorly in terms of
reliability, validity, discriminating power, and (b) performance on
these metrics improved up to seven response options. However,
although Preston & Colman’s study had strong resolution for
determining a potential point of asymptote, it was limited in
several important ways. First, the scale they used was not a
traditional Likert scale, which limits its generalizability. Second,
this study included only 149 participants, all of whom completed
all 10 of the different response options under consideration, which
may have resulted in respondent fatigue and/or memory effects
that could have affected the results. Finally, this study focused on
measurement of attitudes regarding an experience in a restaurant
rather than a deeper psychological construct, such as personality. It
is an open question what a study with similar resolution would
show if applied to traditional personality scales.

Table 1
Summary of Likert Response Labels Used in This Study
Option 2-point 3-point 4-point 5-point 6-point 7-point 8-point 9-point 10-point 11-point Visual Analog

1 Disagree Disagree Strongly Strongly Disagree Strongly Strongly Disagree Very Strongly Very Strongly Very Strongly Very Strongly Disagree
Disagree Disagree Disagree Disagree Disagree Disagree
2 Agree Neither Agree nor Disagree Disagree Disagree Disagree Strongly Strongly Disagree Strongly Strongly Disagree Agree
Disagree Disagree Disagree
3 Agree Agree Neither Agree nor Slightly Slightly Disagree Disagree Disagree Disagree Disagree
Disagree Disagree
4 Strongly Agree Slightly Neither Agree nor Slightly Slightly Disagree Mostly Mostly Disagree
Agree Agree Disagree Disagree Disagree
5 Strongly Agree Agree Slightly Agree Slightly Neither Agree nor Slightly Slightly Disagree
Agree Disagree Disagree
6 Strongly Agree Agree Slightly Agree Slightly Neither Agree nor
Agree Agree Disagree
7 Strongly Agree Strongly Agree Mostly Agree Slightly Agree
Agree
8 Very Strongly Strongly Agree Agree Mostly Agree
Agree
9 Very Strongly Strongly Agree
Agree Agree
10 Very Strongly Strongly Agree
Agree
11 Very Strongly
Agree
 NUMBER OF RESPONSE OPTIONS
In contrast to the preceding paragraphs, a number of studies also
have suggested no clear trends in scale psychometrics as a function
of the number of response options (e.g., Bendig, 1954; Capik &
Gozum, 2015; Matell & Jacoby, 1972). Moreover, some recent
studies have painted a contradictory picture of response option
effects. Finn et al. (2015), following up on similar studies by Cox
and colleagues (2012) and Cox, Courrégé, Feder, & Weed (2017)),
compared 2- and 4-point response scales as applied to the items of
the restructured form of the Minnesota Multiphasic Personality
Inventory (MMPI)-2 (MMPI-2-RF; Tellegen & Ben-Porath, 2008/
2011). In this line of research, increasing the number of response
options resulted in improved internal consistency but, interest-
ingly, no advantage in convergent validity. However, it is an open
This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.
question whether the validity results will replicate with response
This document is copyrighted by the American Psychological Association or one of its allied publishers.
scales with more than four options.
Despite these diverging conclusions regarding the optimal num-
ber of response options to include for psychological test items,
several tentative conclusions are possible. First, most studies sug-
gest that two to three response options result in attenuated psy-
chometrics. Second, although several studies show improved psy-
chometrics as the number of response options increases, the bulk
of recent studies point to the existence of an asymptote above
which scale psychometrics level off or decrease. Third, there is
inconsistency in the specific point of diminishing returns, with four
to seven response options variously reported as the number needed
for maximum measurement precision. Finally, recent work sug-
gests that improvements in measurement precision may not result
in concomitant improvements in convergent validity. In the present
study, we aim to replicate and extend these previous findings using
a well-known measure of a prominent personality model. Our
design includes all response options ranging from two to 11
response options, large sample sizes, and thus adequate resolution
to derive stronger conclusions that the previous studies conducted
in this domain.
Is It Better to Have an Even or Odd Number of
Response options?
Another question that faces psychological scale developers—
and one about which there is relatively more opinion than data—is
whether Likert-type items should include an even or odd number
of response options (Kulas & Stachowski, 2013; Nadler, Weston,
& Voyles, 2015). With an odd number of response options, the
middle option (i.e., neither agree nor disagree) has ambiguous
meaning, which should increase measurement error if respondents
use that option in different ways that do not reflect their perceived
standing on the characteristic being measured. Kulas and Sta-
chowski (2013) recently studied the reasons respondents may have
for selecting the middle option on an oddly numbered Likert scale,
which may include (a) the response reflects moderate standing on
the item/trait (which arguably would be the ideal), (b) the respon-
dent has difficulty deciding his or her standing one the item, (c) the
respondent is confused about the item’s meaning, and/or (d) the
respondent feels that his or her response is context-dependent (i.e.,
what the authors called the “it depends” reason for using the
middle option). The latter three options are less than ideal, since
they represent construct-irrelevant reasons for using a particular
response option and thus should increase measurement error and
thus attenuate validity.
3
Aside from these considerations, Garland (1991) suggested that
midpoint responses may be related to socially desirable respond-
ing. Velez and Ashworth (2007) argued that midpoint endorsement
decreases with clearer items. Hernández, Drasgow, & González-
Roma (2004) identified different classes of individuals in terms of
how they use middle response options, and suggested small per-
sonality differences across these groups. Thus, the reasons for and
impact of allowing a middle-point option have been considered
elsewhere in the literature, albeit infrequently and without clear
conclusions about the question under consideration here: Should
scale developers use even or odd numbers of response options for
Likert-type items? Thus, our second aim is to directly address this
question.
Do Visual Analog Response Scales Offer Advantages
Relative to Likert-Type Scales?
Visual analog scales (Aitken, 1969; Flynn, van Schaik, & van
Wersch, 2004; Hayes & Patterson, 1921) ostensibly are continuous
measurement devices that have been used in many literatures to
represent both full constructs and/or single items on longer scales
of psychological constructs. Historically, visual analogs appeared
as a single horizontal line on a page, with two opposing anchors on
either end of this line (e.g., agree vs. disagree, true vs. untrue, good
vs. bad, etc.). Respondents are asked to simply mark their standing
on the given construct with a short vertical line, and scoring is
completed by physically measuring the location of the mark using
a measure (e.g., a ruler) with reasonable precision. This can be a
labor-intensive process, given that each mark must be hand-
measured. However, with the advent of computer- and Internet-
based survey programs, visual analogs now can be neatly repre-
sented (and more easily scored) using a slider bar that respondents
move using a mouse, trackpad, or other input device.
Given the greater resolution and flexibility associated with
visual analogs, some have suggested that they might maximize
measurement precision, even for items on traditional psychological
scales (e.g., Reips & Funke, 2008; Russell & Bobko, 1992). That
said, relatively little work has directly compared visual analog
scales to traditional Likert-type items as the basis for scale devel-
opment. The work that has been conducted has led to competing
conclusions, with some studies showing no differences in mea-
surement properties of visual analog scales as compared with
traditional Likert scale (e.g., Bergman, 2009) and others showing
modest evidence of psychometric superiority for visual analogs
(e.g., Hilbert et al., 2016; Russell & Bobko, 1992). Thus, although
computers and online data collection make the use of such scales
much easier than previously, it is an open question whether hu-
mans can make fine-grained distinctions along these scales in a
way that actually improves precision.
Present Study
Given the importance of measurement precision across all areas
of psychology (and the social sciences more generally), the limited
research and lack of scientific consensus regarding the nature and
number of response options for questionnaire items is surprising.
With this article, we sought to add meaningfully to this literature
by studying three issues related to response scales: (a) the optimal
number of response options for a Likert scale, (b) even versus odd
 4 SIMMS, ZELAZNY, WILLIAMS, AND BERNSTEIN
numbers of response options, and (c) visual analog scales versus
traditional Likert scales. Our study extends previous work by
simultaneously assessing each of these three questions using a
well-powered study with adequate resolution to yield more con-
clusive results. Given the previous (albeit inconclusive) work in
this area, as well as considerations drawn from basic psychometric
theory, we predicted that measurement precision (and subsequent
markers of validity) would (a) asymptote after six to seven re-
sponse options, (b) show no advantage for odd numbers of re-
sponse options relative to matched evens, and (c) be no stronger
for visual analog scales relative to traditional Likert scales.
Method
This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.
This document is copyrighted by the American Psychological Association or one of its allied publishers.
Participants and Procedures
Participants from this study were 1,358 undergraduate students
who participated in exchange for course research credit at a large
university in the Northeastern United States. All study procedures
were reviewed and approved by the university’s cognizant insti-
tutional review board. Participants were 51% female; Mean age
was 19.1 years old (SD ⫽ 1.9). The sample was 60.8% White,
22.8% Asian American, 8.0% African American, and 5.1% His-
panic/Latino. Participants participated in groups of as many as
eight participants. Questionnaires were completed using computers
in private carrels in a laboratory setting. All participants completed
the Big Five Inventory (BFI; John & Srivastava, 1999) twice,
before and after completing the Personality Inventory for DSM–5
(PID-5; Krueger, Derringer, Markon, Watson, & Skodol, 2012).
Response scales for the BFI were manipulated across groups, as
described below.
The study was structured with both cross-sectional and repeated-
measures components. Participants were randomly assigned to one
of six groups. Within the first five groups, response scales were
paired to facilitate comparisons between odd and even numbers of
response options. Each group was designated to complete one
odd-even comparison (e.g., four vs. five response options). Re-
sponse options, which are presented in Table 1 for all groups, were
developed such that matched odd- and even-numbered response
scales were identical except that paired odd-numbered scales in-
cluded a middle “neither agree nor disagree” option. This yielded
five groups with varying numbers of response options: (a) two-
and three-response options (n ⫽ 225), (b) four- and five-response
options (n ⫽ 232), (c) six- and seven-response options (n ⫽ 221),
(d) eight- and nine-response options (n ⫽ 219), and (e) 10- and
11-response options (n ⫽ 212). Order or response scale presenta-
tion was counterbalanced within groups. In addition, a sixth group
completed the BFI twice using a visual analog scale for each item
(n ⫽ 249), which was presented as a slider bar with a marker that
could be moved by participants using a mouse. The visual analog
scale was scored based on marker position on a 0 to 1000 scale.
Measures
Big Five Inventory (BFI; John & Srivastava, 1999). The
BFI is a 44-item scale that is designed to use a 5-point Likert-type
rating scale, ranging from 1 (strongly disagree) to 5 (strongly
agree), and provides scores on the domains of the Big Five Model
of personality (Neuroticism, Extraversion, Conscientiousness,
Agreeableness, and Openness). Benet-Martinez and John (1998)
reported good internal consistency for BFI scales and good con-
vergence between BFI scales and other established measures of the
Big Five Model. A noted above, we manipulated the response
scale across groups for the BFI, and the resultant reliabilities and
validities are presented below, representing the primary analyses
for this article.
Personality Inventory for DSM–5 (PID-5; Krueger et al.,
2012). The PID-5 includes 220 self-report items assessing the 25
maladaptive traits and five higher order domains of the Alternative
Model of Personality Disorder (American Psychiatric Association,
2013). Responses are provided on a four-point Likert scale from 0
(very false or often false) to 3 (very true or often true), and scale
scores are means of associated items, with higher scores represent-
ing greater pathology. Recent studies have supported the construct
validity of the PID-5 five-factor structure (e.g., Wright & Simms,
2014). In the current study the average alpha coefficients were .94
(range ⫽ .92–.96) and .87 (range ⫽ .68 –.96) across domains and
traits, respectively. We included the PID-5 to compare the impact
of differing response options on BFI criterion validity.
Analyses and Results
To facilitate our primary aims, we conducted analyses to compare
descriptive statistics, internal consistency reliabilities, short-term re-
test correlations, and criterion validity correlations. First, descriptive
statistics across scales and numbers of response options— calculated
as Cohen’s d effect size differences on item means, compared to two
response options—are presented in Figure 1. What is clear from these
results is that item means decreased a full standard deviation between
two and four response options. Between four and seven response
options, the item means tended to remain relatively consistent, but
then additional, albeit small, decreases in item means were observed
beyond seven response options. Significance testing results for these
descriptive statistics are presented in Supplemental Table 1: Within-
and between-subjects t tests generally corroborated the pattern de-
scribed here. We also examined scale skewness and kurtosis, finding
that skewness and kurtosis generally decreased from two to four
options, but did not change dramatically after four response options.
We next examined internal consistency reliabilities across scales
and numbers of response options. Results related to our first
aim— determining the optimal number of response options to
include on a Likert scale—appear in Figures 2. Two findings can
be gleaned from this figure. First, alpha coefficients varied across
scales, with BFI Neuroticism and BFI Extraversion demonstrating
consistently stronger alphas than the remaining three BFI scales.
Second, the mean alpha line neatly summarizes the results: Internal
consistency reliabilities appeared to increase gradually until six
response options and then leveled off. These results reveal no
psychometric advantage for any scale beyond six response options.
In Figure 3, we rearranged the internal consistency values to
facilitate an examination of our second and third study aims—
examining odd versus even numbers of response options and
visual analog scales. In the first paired comparison—2- versus
3-point scales—the mean alpha coefficients increased from two to
three response options, M alphas ⫽ .73 and .78, respectively. Mean
alphas were the same for the 4- and 5-point scales, both alphas ⫽
.80. Mean alphas decreased slightly from six to seven response
options, M alphas ⫽ .83 and .82, respectively. Similarly, mean
 NUMBER OF RESPONSE OPTIONS
This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.
This document is copyrighted by the American Psychological Association or one of its allied publishers.
Figure 1. Cohen’s d effect size differences (based on item means) as a function of scale and number of
response options. VA ⫽ visual analog scale.
alphas decreased slightly from eight to nine response options, M
alphas ⫽ .82 and .81, respectively. The same pattern was observed
from 10 to 11 response options, M alphas ⫽ .83 and .82, respec-
tively. Finally, the visual analog scale yielded a slightly lower
alpha compare d to the most other alphas between six and 11re-
sponse options, M ␣ ⫽ .81. Taken together, these results suggest
a slight decrement in alpha for odd numbers of response options,
beyond six, and an additional slight decrement for visual analog
scales. That said, the differences between matched even and odd
numbers of response options were small enough that the practical
impact likely is minimal.
We also tested for significant differences among the alpha coeffi-
cients described above, using methods described by Feldt, Woodruff,
and Salih (1987) and implemented in “cocron,” an open-source pro-
gram on the R platform (Diedenhofen & Musch, 2016). These
.90
.85
.80
Coefficient Alpha
.75
.70
.65
.60
2 3 4 5
Figure 2. Internal consistency reliabilities as a function of scale and number of response options. VA ⫽ visual
analog scale.
5
results—presented in Supplemental Table 2—revealed several pat-
terns. First, there were relatively few significant differences among
the alphas presented in Figures 2 and 3. Second, the bulk of
significant alpha differences involved the 2-point scale, which was
significantly lower than the 3-point scale for four of five traits (i.e.,
all but conscientiousness), ␹2s (1) ranged from 10.1 to 16.6, p ⬍
.01. Similarly, the 2-point scale yielded alphas that were signifi-
cantly lower than those for all more differentiated scales for
extraversion, agreeableness, and openness, ␹2s (1) ranged from 3.6
to 16.6, p ⬍ .05, whereas the 2-point scale was less consistently
different than other response scales for conscientiousness and
neuroticism. Finally, to detect alpha differences between even and
odd numbers of response options, 7 of 25 matched comparisons
yielded alpha differences in the predicted direction, ␹2s (1) ranged
from 5.2 to 16.6, p ⬍ .05, whereas the remaining tests were not
BFI Extraversion
BFI Agreeableness
BFI Conscienousness
BFI Neurocism
BFI Openness
Mean
6 7 8 9 10 11 VA
Response Opons
 6 SIMMS, ZELAZNY, WILLIAMS, AND BERNSTEIN

.90

.85

.80
Coefficient Alpha

.75

BFI Extraversion
.70 BFI Agreeableness
This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.

BFI Conscienousness
This document is copyrighted by the American Psychological Association or one of its allied publishers.

BFI Neurocism
.65 BFI Openness
Mean

.60
2 3 4 5 6 7 8 9 10 11 VA
Response Opons

Figure 3. Internal consistency reliabilities as a function of scale and number of response options, highlighting
matched even-odd pairs. VA ⫽ visual analog scale.

significant. Thus, these significance tests painted a more conser-
vative picture of the differences than was evident through visual
inspection of the figures; through this lens, the most robust finding
is that 2-point scales are impoverished relative to more differen-
tiated scales with respect to internal consistency reliability.
We next studied short-term retest correlations within each
group. Given the short interval—approximately 30 min— between
test and retest, these coefficients are akin to dependability corre-
lations (see Watson, 2004). These results—which appear in Figure
4—showed that retest correlations (a) all were strong and greater
than .84, and (b) asymptoted for the six versus seven or eight
versus nine, and 10 versus 11 comparisons, M retest correlations ⫽
.895, .892, and .896, respectively. Full retest results are presented
in Supplemental Table 3, including Z tests for significant differ-
ences among the correlations. Although the figures suggest no
advantage for any scale beyond six response options, the signifi-
cance testing revealed a more conservative pattern: Short-term
retest coefficients were significantly lower for the two-to-three
group relative to all other groups for Neuroticism, Extraversion,
and Openness, Zs ranged from 2.01 to 2.68, p ⬍ .05. No other
differences among these correlations were significant.
Finally, we studied convergent validity as a function of scale and
number of response options. Results for the BFI Extraversion, Agree-
ableness, Conscientiousness, and Neuroticism scales are presented in

Figure 4. Short-term retest correlations as a function of scale and number of response options.
 NUMBER OF RESPONSE OPTIONS
This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.
This document is copyrighted by the American Psychological Association or one of its allied publishers.
Figure 5. Mean convergent validity correlations, measured against the Personality Inventory for DSM–5, as a
function of scale and number of response options. For each BFI scale, convergent validities were included in the
means only if at least one correlation was greater than or equal to .40. All correlations are absolute values. VA ⫽
visual analog scale.
Figure 5 and Supplemental Tables 4 to 8. For each BFI scale, we
calculated mean convergent validity correlations against the scales of
the PID-5, Convergent validities were included in the means only if at
least one absolute correlation was greater than or equal to .40. No
correlates met this threshold for BFI Openness. Given the premise that
unreliability should attenuate validity, we predicted increases in con-
vergent validity as the number of response options increased. Results
varied across scales, with no single pattern of validities emerging
across scales as a function of response scale. The mean validity
data—represented by a bold, dashed line in Figure 5—revealed a
relatively flat line across response formats, with two exceptions. First,
there was a slight dip in validity for those who completed the 4- and
5-point BFI scales, mean rs ⫽ .42 and .41, respectively, compared to
the mean validity across all scales and response scales, mean r ⫽ .45.
Second, there was a notable drop in validity for those completing he
BFI using the visual analog format, mean r ⫽ .41. Notably, however,
none of the mean validity correlations presented in Figure 5 were
significantly different from one another, all Zs ⬍ 1.96, ps ⬎ .05.
Discussion
In this article, we have attempted to bring clarity to three
questions involved in determining a response scale for psycholog-
ical test items: (a) is there a psychometrically optimal number of
response options to include for Likert-type items, (b) is there any
psychometric advantage or disadvantage associated with using odd
or even numbers of response options, and (c) how does the visual
analog format compare to traditional Likert-type items with re-
spect to their resultant psychometric characteristics. To address
these questions, we examined descriptive statistics, internal con-
sistency reliability, short-term retest correlations, and convergent
validity across different numbers of response options and with five
broad personality scales. In this discussion, we will summarize and
consider the implications of our results related to each of these
three questions.
7
How Many Response Options?
As noted previously, traditional Likert scales (Likert, 1932)
included 5 symmetrical response options ranging from strongly
disagree to strongly agree, but many variations on that original
structure have emerged since the scales original publication, often
with little or no evidence offered to support the deviation. Our
results revealed several themes regarding the ideal or optimal
number of response options to include on such a scale. First,
although we did not make any specific prediction regarding de-
scriptive statistics, our results show that changing the number of
response options has a non-negligible impact on basic scale norms.
Large differences in scale means were revealed for smaller num-
bers of response options. And although these differences stabilized
a bit for scales with four or more response options, item means
continued to decline at a smaller rate. Thus, for example, if
response scales are modified for a given research study or clinical
application, simple proration of published norms—to account for
changes in the number of response options—appears to be a
problematic exercise.
Second, response scales with two and three response options
(and to a lesser extent four and five response options) generally
attenuated the psychometric precision associated with these five
BFI scales, which suggests that scale developers wishing to adopt
such response scales do so at some risk of reducing measurement
precision. Practically speaking, to have scales with minimally
acceptable measurement precision (e.g., ␣ ⫽ .80; see Clark &
Watson, 1995), using fewer response options will require scale
developers to include more items. Thus, these results reveal a
trade-off between response scale simplicity and measurement im-
precision that must be adequately reconciled at the scale develop-
ment stage. Some developers may desire simpler response scales
for nonpsychometric reasons (e.g., readability, simplicity), but
more items will be needed for such scales if minimally acceptable
measurement precision is desired.
 8 SIMMS, ZELAZNY, WILLIAMS, AND BERNSTEIN
Third, no improvements in psychometric precision were identi-
fied past 6 response options. Thus, there appears to be little
psychometric basis to arguments that additional numbers of re-
sponse options will translate to increased scale reliability while
holding the number of items constant. Going beyond 6 options
may confuse participants who perhaps have difficulty perceiving
differences between similarly worded response options (e.g., strongly
agree vs. very strongly agree). Alternatively, more differentiated
response scales may pose important challenges to the ability of
humans to make fine-grained distinctions regarding responses to
relatively coarse psychological test items. That is, there likely are
important cognitive information processing variables (e.g., mem-
ory, perception, discrimination, intelligence) at play that have not
This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.
been adequately studied with respect to how they interact with
This document is copyrighted by the American Psychological Association or one of its allied publishers.
participants’ approaches to psychological test items. More work is
needed in this area.
Interestingly, the criterion validity results defied the pattern of
attenuated psychometrics for smaller numbers of response options,
such that the lower psychometric precision associated with smaller
numbers of response options did not translate into attenuated
validity as would be expected from classical test theory (e.g.,
Crocker & Algina, 1986; Gulliksen, 1950; McDonald, 1999). This
extends recent work suggesting that moving from two to four
response options results in no improvement in convergent validity
correlations (Cox et al., 2012, 2017; Finn et al., 2015). In present
study, we replicated this finding and extended it to more differen-
tiated response formats up to 11 options (and the visual analog
scale). Several possibilities could explain this pattern of results.
First, the BFI scales we used for the validity analyses are psycho-
metrically strong scales with a long literature showing evidence of
their reliability and validity (e.g., see John & Srivastava, 1999). In
particular, it is notable that even in our own results, the short-term
retest correlations, although slightly lower for less differentiated
scales, did not reveal dramatic differences in scale dependability.
Arguably, measurement imprecision related to temporal differ-
ences (i.e., retest reliability or dependability; Watson, 2004) might
be more detrimental to criterion validity than imprecision related
to content heterogeneity (i.e., internal consistency reliability). If
so, then our failure to see a consistent impact of response options
on criterion validity is not as surprising.
Alternatively, unknown sampling differences across our groups
could have affected these findings, despite large samples and
random assignment. Also, our criterion variables were limited to
the scales of another personality questionnaire with known limi-
tations (e.g., Al-Dajani, Gralnick, & Bagby, 2015). Given that both
our primary measure and convergent validity measure were self-
report questionnaires with similar formats, the impact of shared
method variance could not be disentangled in the current study
(e.g., Courrégé & Weed, 2018). Taken together, it would be
important to study the impact on validity of different numbers of
response options using a broader range of test and nontest criteria,
such as behavioral observations or real-word indicators of person-
ality. Regardless, although the precision results point to an asymp-
totic point of diminishing returns following 6 response options, the
criterion validity results limit the strength of this conclusion. This
result deserves further study since it calls into question long-
standing tenets of classical test theory regarding the relation be-
tween reliability and validity. All things being equal, measurement
impression should attenuate validity correlations (e.g., Gulliksen,
1950). That did not occur in this study.
Another point that deserves discussion is that statistical signif-
icance testing showed a more conservative pattern than the visual
pattern of results presented in the figures. The most conservative
conclusion based on significance testing is that 2- and 3-point
response formats are impoverished with respect to measurement
precision (but not validity). Differentiation beyond four response
options generally was not supported by significance testing. How-
ever, the visual patterns appear to show replicable additional
advantages, albeit small and nonsignificant, between four and five
and six and seven options. Thus, we maintain that scale developers
should strongly consider using six or seven response options
unless pilot testing or other considerations make such a differen-
tiated format undesirable.
Notably, some readers may find this recommendation curious
given that the validity results failed to support the need for such a
differentiated response scale. We acknowledge this discrepancy.
However, the decision regarding how many options to offer on a
personality questionnaire like the BFI involves more than a con-
sideration of validity. Deficits in measurement precision alone are
directly related to increases in the standard error of measurement
of a scale (and thus the confidence intervals around scores), which
has implications for the precision of point estimates of scores used
in clinical and applied settings. Thus, even in the absence of
validity implications (about which the conclusions are still un-
clear), it is our view that response scales should be adequately
differentiated to maximize measurement precision.
Is It Better to Have an Even or Odd Number
of options?
Given the ambiguity associated with the use of the middle option
on odd-numbered response scales (Kulas & Stachowski, 2013), we
predicted that odd-numbered Likert scales would show no advantage,
psychometrically speaking, over matched even-numbered scales. This
result was generally supported, as alphas and criterion validity corre-
lations generally revealed no advantage for odd-numbered scales
relative to matched even-numbered scales. Taken together with the
conclusions presented above, these results suggest little psychometric
justification for Likert scales with more than six or seven response
options. Moreover, although the psychometric differences between
six and seven response options were small to nonexistent, we would
argue that six response options are preferable to seven options on the
grounds of parsimony. That said, the most honest appraisal of our
results is that it probably doesn’t matter much. Those unconcerned
with the ambiguity of the middle option may continue to use odd-
numbered Likert scales given that there is no clear and unequivocal
psychometric penalty for doing so.
Do Visual Analog Scales Offer Any Advantages Over
Traditional Likert-Type Items?
Visual analog scales offer the promise of added measurement
precision given that they do not rely on a limited number of anchor
points and rather allow participants to simply mark their response
anywhere they wish along a continuous line. That said, our results
failed to show any psychometric advantage for visual analog items
relative to traditional Likert-type items. In fact, most of our anal-
 NUMBER OF RESPONSE OPTIONS
yses appeared to reveal small nonsignificant decrements in psy-
chometric performance, including criterion validity, for scales
comprised of visual analog items. Similar to that which was
discussed above, it appears that the promise of added psychometric
precision is not realized in practice with scales based on visual
analog items, perhaps because humans are unable to reliably make
meaningful and valid fine-grained distinctions for coarse items
reflecting complex psychological characteristics. Although the dif-
ferences in results for visual analogs generally were not large or
statistically significant, nothing in our results would lead us to
recommend their use as the basis for psychological test items.
Moreover, although not directly assessed in the present study, the
present results also call into question the common practice of using
This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.
visual analog scales as single-item scales to reflect underlying
This document is copyrighted by the American Psychological Association or one of its allied publishers.
latent traits and characteristics.
Limitations and Future Directions
Strengths of our study include a large sample size, resolution
permitting us to examine all common numbers of response options
between two and 11 (including a visual analog option), and the use
of a widely used and psychometrically strong measure of a prom-
inent measure of personality as the basis for the study. In addition,
we examined the impact of response scale on both reliability and
validity, something not often done in this literature. That said, our
results must be interpreted in the context of several limitations.
First, our results, by design, apply only to traditional Likert-type
agree-disagree scales. Other metrics—such as scales related to
frequency or intensity or similarity—might yield different results
and thus must be studied in a manner similar to what we presented
in this article. That said, the basic finding—that there is a point of
diminishing returns in terms of the number of response options—is
likely to generalize to all possible response scales for psycholog-
ical scales, although the exact point might be a function of the
nature of the response scale under consideration. For example,
Likert scales are inherently bipolar in nature; unipolar scales (e.g.,
not at all to extremely) may not yield as many meaningfully
different anchor points. All such variations in response scales
ought to be carefully studied during the scale development process.
Second, our analyses and results were confined to a single
multiscale measure of personality. Although the BFI is a strong
and widely studied measure of a prominent model of personality,
it does not fully represent all possible constructs psychological
researchers and clinicians may wish to measure. Taken together
with the finding that our patterns of results appeared to vary
somewhat even within the BFI scales, generalizing the present
findings to other measures and different constructs must be done
with caution until the present results are replicated with a broader
range of measures and constructs, including different domains,
such as psychopathology, mood/affect, attitudes, values, and con-
structs used in other areas of the social sciences (e.g., political or
moral attitudes, market research evaluations, etc.).
Finally, our sample, although large, was comprised solely of
college undergraduates who completed the study to fulfill a course
research requirement. Replicating the present work in a broader,
more ecologically valid community and psychiatric samples would
be important aims for future work in this area. Although this is a
common limitation in empirical studies involving student samples,
there are specific reasons to highlight this limitation in the present
9
article. College samples typically are comprised of relatively well-
educated participants with average to above-average intellectual
ability and reading levels. Given the cognitive information pro-
cessing demands that test items likely place on those providing
responses, college samples likely represent an upper bound on the
amount of complexity that it is reasonable to include in an item
response format. Thus, samples comprised of participants with
lower cognitive ability might yield different results from those
identified here.
Final Thoughts
Despite the above caveats, the present study offers important
guidance to aid scale developers who wish to base their work on
evidence-based practices. We conclude that a 6-point response
scale is the most reasonable format based on the present results,
especially for measures of personality constructs like those we
assessed in this study. Going much beyond six response options
seems to challenge humans’ basic ability to make fine-grained
distinctions about the complex psychological constructs we tend to
study. We also made suggestions regarding the use of odd-
numbers of response options and visual analog items. However, an
important take-home message from this article is that the choice of
response format should be given the same amount of thought and
examination as typically is put into the development of the test
items themselves.
If you’re building a new scale and wish to deviate from the
suggestions listed here, we strongly recommend pilot testing.
References
Aitken, R. C. (1969). Measurement of feelings using visual analogue
scales. Proceedings of the Royal Society of Medicine, 62, 989 –993.
Al-Dajani, N., Gralnick, T. M., & Bagby, R. M. (2015). A Psychometric
Review of the Personality Inventory for DSM–5 (PID-5): Current Status
and Future Directions. Journal of Personality Assessment, 98, 1–20.
American Psychiatric Association. (2013). Diagnostic and Statistical Man-
ual of Mental Disorders (5th ed.). Washington, DC: Author.
Bendig, A. W. (1953). The reliability of self-ratings as a function of the
amount of verbal anchoring and of the number of categories on the scale.
Journal of Applied Psychology, 37, 38 – 41. http://dx.doi.org/10.1037/h0
057911
Bendig, A. W. (1954). Reliability and the number of rating-scale categories.
Journal of Applied Psychology, 38, 38 – 40. http://dx.doi.org/10.1037/h0
055647
Benet-Martínez, V., & John, O. P. (1998). Los Cinco Grandes across
cultures and ethnic groups: Multitrait multimethod analyses of the Big
Five in Spanish and English. Journal of Personality and Social Psychol-
ogy, 75, 729 –750. http://dx.doi.org/10.1037/0022-3514.75.3.729
Bergman, R. D. (2009). Testing the measurement invariance of the Likert
and graphic rating scales under two conditions of scale numeric pre-
sentation (Doctoral dissertation). Retrieved from Proquest Dissertations
& Theses Global. (Order No. 3360158)
Capik, C., & Gozum, S. (2015). Psychometric features of an assessment
instrument with Likert and dichotomous response formats. Public
Health Nursing, 32, 81– 86. http://dx.doi.org/10.1111/phn.12156
Clark, L. A., & Watson, D. (1995). Constructing validity: Basic issues in
objective scale development. Psychological Assessment, 7, 309 –319.
http://dx.doi.org/10.1037/1040-3590.7.3.309
Courrégé, S. C., & Weed, N. C. (2018). The role of common method
variance in MMPI-2-RF response option augmentation. Unpublished
manuscript.
 10 SIMMS, ZELAZNY, WILLIAMS, AND BERNSTEIN
Cox, A. C., Courrégé, S. C., Feder, A. H., & Weed, N. C. (2017). Effects
of augmenting response options of the MMPI-2-RF: An extension of
previous findings. Cogent Psychology, 4, 1323988. http://dx.doi.org/10
.1080/23311908.2017.1323988
Cox, A., Pant, H., Gilson, A. N., Rodriguez, J. L., Young, K. R., Kwon, S.,
& Weed, N. C. (2012). Effects of augmenting response options on
MMPI-2 RC scale psychometrics. Journal of Personality Assessment,
94, 613– 619. http://dx.doi.org/10.1080/00223891.2012.700464
Cox, E. P. (1980). The optimal number of response alternatives for a scale:
A review. Journal of Marketing Research, 17, 407– 422. http://dx.doi
.org/10.2307/3150495
Crocker, L., & Algina, J. (1986). Introduction to classical & modern test
theory. Orlando, FL: Holt, Rinehart and Winston.
Diedenhofen, B., & Musch, J. (2016). cocron: A web interface and R
This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.
package for the statistical comparison of Cronbach’s alpha coefficients.
This document is copyrighted by the American Psychological Association or one of its allied publishers.
International Journal of Internet Science, 11, 51– 60.
Feldt, L. S., Woodruff, D. J., & Salih, F. A. (1987). Statistical inference for
coefficient alpha. Applied Psychological Measurement, 11, 93–103.
http://dx.doi.org/10.1177/014662168701100107
Finn, J. A., Ben-Porath, Y. S., & Tellegen, A. (2015). Dichotomous versus
polytomous response options in psychopathology assessment: Method or
meaningful variance? Psychological Assessment, 27, 184 –193. http://dx
.doi.org/10.1037/pas0000044
Flamer, S. (1983). Assessment of the multitrait-multimethod matrix valid-
ity of Likert scales via confirmatory factor analysis. Multivariate Be-
havioral Research, 18, 275–306. http://dx.doi.org/10.1207/s15327906
mbr1803_3
Flynn, D., van Schaik, P., & van Wersch, A. (2004). A comparison of
multi-item Likert and visual analogue scales for the assessment of
transactionally defined coping function. European Journal of Psycho-
logical Assessment, 20, 49 –58. http://dx.doi.org/10.1027/1015-5759.20
.1.49
Garland, R. (1991). The mid-point on a rating scale: Is it desirable.
Marketing Bulletin, 2, 66 –70.
Gulliksen, H. (1950). Theory of mental tests. New York, NY: Wiley.
http://dx.doi.org/10.1037/13240-000
Hayes, M. H. S., & Patterson, D. G. (1921). Experimental development of
the graphic rating method. Psychological Bulletin, 18, 98 –99.
Hernández, A., Drasgow, F., & González-Romá, V. (2004). Investigating
the functioning of a middle category by means of a mixed-measurement
model. Journal of Applied Psychology, 89, 687– 699. http://dx.doi.org/
10.1037/0021-9010.89.4.687
Hilbert, S., Küchenhoff, H., Sarubin, N., Nakagawa, T. T., & Bühner, M.
(2016). The influence of the response format in a personality question-
naire: An analysis of a dichotomous, a Likert-type, and a visual analogue
scale. TPM-Testing, Psychometrics, Methodology in Applied Psychol-
ogy, 23, 3–24.
John, O. P., & Srivastava, S. (1999). The Big Five Trait taxonomy: History,
measurement, and theoretical perspectives. In L. A. Pervin & O. P. John
(Eds.), Handbook of personality: Theory and research (2nd ed., pp.
102–138). New York, NY: Guilford Press.
Krueger, R. F., Derringer, J., Markon, K. E., Watson, D., & Skodol, A. E.
(2012). Initial construction of a maladaptive personality trait model and
inventory for. DSM–5. Psychological Medicine, 42, 1879 –1890. http://
dx.doi.org/10.1017/S0033291711002674
Kulas, J. T., & Stachowski, A. A. (2013). Respondent rationale for neither
agreeing nor disagreeing: Person and item contributors to middle cate-
gory endorsement intent on Likert personality indicators. Journal of
View publication stats
Research in Personality, 47, 254 –262. http://dx.doi.org/10.1016/j.jrp
.2013.01.014
Lee, J., & Paek, I. (2014). In search of the optimal number of response
categories in a rating scale. Journal of Psychoeducational Assessment,
32, 663– 673. http://dx.doi.org/10.1177/0734282914522200
Likert, R. (1932). A technique for the measurement of attitudes. Archives
de Psychologie, 140, 5–53.
Lozano, L. M., García-Cueto, E., & Muñiz, J. (2008). Effect of the number of
response categories on the reliability and validity of rating scales. Method-
ology: European Journal of Research Methods for the Behavioral and
Social Sciences, 4, 73–79. http://dx.doi.org/10.1027/1614-2241.4.2.73
Matell, M. S., & Jacoby, J. (1972). Is there an optimal number of alterna-
tives for Likert-scale items? Effects of testing time and scale properties.
Journal of Applied Psychology, 56, 506 –509. http://dx.doi.org/10.1037/
h0033601
McDonald, R. P. (1999). Test theory: A unified treatment. Mahwah, NJ:
Erlbaum.
Nadler, J. T., Weston, R., & Voyles, E. C. (2015). Stuck in the middle: The
use and interpretation of mid-points in items on questionnaires. Journal
of General Psychology, 142, 71– 89. http://dx.doi.org/10.1080/00221309
.2014.994590
Preston, C. C., & Colman, A. M. (2000). Optimal number of response
categories in rating scales: Reliability, validity, discriminating power,
and respondent preferences. Acta Psychologica, 104, 1–15. http://dx.doi
.org/10.1016/S0001-6918(99)00050-5
Reips, U. D., & Funke, F. (2008). Interval-level measurement with visual
analogue scales in Internet-based research: VAS Generator. Behavior
Research Methods, 40, 699 –704. http://dx.doi.org/10.3758/BRM.40.3
.699
Russell, C. J., & Bobko, P. (1992). Moderated regression analysis and
Likert scales: Too coarse for comfort. Journal of Applied Psychology,
77, 336 –342. http://dx.doi.org/10.1037/0021-9010.77.3.336
Symonds, P. M. (1924). On the loss of reliability in ratings due to
coarseness of the scale. Journal of Experimental Psychology, 7, 456 –
461. http://dx.doi.org/10.1037/h0074469
Tellegen, A., & Ben-Porath, Y. S. (2008/2011). Minnesota Multiphasic
Personality Inventory–2 Restructured Form: Technical manual. Minne-
apolis, MN: University of Minnesota Press.
Tourangeau, R., Rips, L. J., & Rasinski, K. (2000). The psychology of
survey response. Cambridge, England: Cambridge University Press.
http://dx.doi.org/10.1017/CBO9780511819322
Velez, P., & Ashworth, S. D. (2007). The impact of item readability on the
endorsement of the midpoint response surveys. Survey Research Meth-
ods, 1, 69 –74.
Watson, D. (2004). Stability versus change, dependability versus error:
Issues in the assessment of personality over time. Journal of Research in
Personality, 38, 319 –350. http://dx.doi.org/10.1016/j.jrp.2004.03.001
Weng, L. J. (2004). Impact of the number of response categories and
anchor labels on coefficient alpha and test-retest reliability. Educational
and Psychological Measurement, 64, 956 –972. http://dx.doi.org/10
.1177/0013164404268674
Wright, A. G. C., & Simms, L. J. (2014). On the structure of personality
disorder traits: Conjoint analyses of the CAT-PD, PID-5, and NEO-PI-3
trait models. Personality Disorders, 5, 43–54. http://dx.doi.org/10.1037/
per0000037
Received January 26, 2018
Revision received June 28, 2018
Accepted June 30, 2018 䡲