Expert Performance in SCRABBLE: Implications for the Study of the

Structure and Acquisition of Complex Skills

Michael Tuffiash, Roy W. Roring, and K. Anders Ericsson
Florida State University
Applied psychologists have long been interested in examining expert performance in complex cognitive
domains. In the present article, we report the results from a study of expert cognitive skill in which
elements from two historically distinct research paradigms are incorporated the individual differences
tradition and the expert-performance approach. Forty tournament-rated SCRABBLE players (20 elite, 20
average) and 40 unrated novice players completed a battery of domain-representative laboratory tasks
and standardized verbal ability tests. The analyses revealed that elite- and average-level rated players
only significantly differed from each other on tasks representative of SCRABBLE performance. Fur-
thermore, domain-relevant practice mediated the effects of SCRABBLE tournament ratings on repre-
sentative task performance, suggesting that SCRABBLE players can acquire some of the knowledge
necessary for success at the highest levels of competition by engaging in activities deliberately designed
to maximize adaptation to SCRABBLE-specific task constraints. We discuss the potential importance of
our results in the context of continuing efforts to capture and explain superior performance across
intellectual domains.
Keywords: expert performance, skill acquisition, individual differences, protocol analysis, SCRABBLE
Scientists from a number of disciplines have long been intrigued
by expertise and expert performance. In the case of psychology,
expertise provides a challenge to researchers interested in the
structure and development of skills and abilities because it repre-
sents an extreme point on the spectrum of human performance.
Moreover, the study of expert performance provides researchers
with opportunities to evaluate the generalizability of broader the-
ories about basic cognitive processes and capacities. For these
reasons, expert performance and related research topics constitute
a set of touchstone phenomena (Charness, 1998) around which a
number of both theoretically important and practically relevant
inquiries can be made with respect to the range and limits of
human achievement.
In this article, we describe the expert-performance approach as
a technique for studying superior performance outside the labora-
tory, and we relate it to other approaches traditionally used to study
human skills and abilities. We then present the findings of a study
in which we apply the expert-performance approach to the domain
of SCRABBLE to determine how the acquisition and cognitive
structure of this skill are related to traditional measures of ability.
Capturing and Explaining Superior Cognitive
Achievement: The Individual Differences Approach
Some of the earliest empirical studies of superior cognitive
ability involved research on basic cognitive processes and general
capacities as reflected by performance on elementary laboratory
tasks, such as simple reaction time and perceptual discrimination.
This early work (e.g., see Wissler, 1901) indicated that simple
perceptual and psychomotor tasks do not generally well-predict
individual differences in performance on more complex, real-
world tasks, such as school grades. The difficulties encountered in
this early research, along with the subsequent development of
more sophisticated statistical methods (e.g., factor analysis), led to
a shift in research toward measuring individual differences in more
complex cognitive ability constructs, such as psychometric g (cf.
Carroll, 1993; Jensen, 1998). These test batteries are frequently
cited as one of the best predictors of real-world cognitive achieve-
ment, namely because of their strong correlations with several
measures of occupational success (Gottfredson, 1997; Hunter &
Hunter, 1984).
To better understand why job-related skills might be predicted
by standardized cognitive ability measures, it may be useful to
examine laboratory research on the structure of skill acquisition.
For example, Fitts and Posner (1967) argued that cognitive skills
are typically acquired in phases. During the first phase, individuals
Michael Tuffiash, Roy W. Roring, and K. Anders Ericsson, Department
of Psychology, Florida State University.
Financial support for this study was provided by a Florida State Conradi
Eminent Scholar grant to the third author (K. Anders Ericsson).
HASBRO is the owner of the registered SCRABBLE trademark in the
United States and Canada. 2007 HASBRO. All rights reserved.
We thank John Chew, Joe Edley, Jane Ratsey Williams, Sherrie St.
John, and John Williams Jr., from the National SCRABBLE Association
for their generous logistical support and thoughtful feedback on method-
ological issues pertaining to our study. We would also acknowledge the
dedicated and skillful contributions of Shelby Cohen, Patrick Collopy,
Nicole Levins, Justin Provido, Jessica Ramos, Jessica Riblet, and Justin
Rufty toward the completion of the scoring and data entry procedures
required for this study. Special thanks also to Maria Borzycki and Diane
Halpern.
Correspondence concerning this article should be addressed to Michael
Tuffiash, Florida State University, Department of Psychology, Tallahassee,
FL 32306-4301. E-mail: tuffiash@psy.fsu.edu
Journal of Experimental Psychology: Applied Copyright 2007 by the American Psychological Association
2007, Vol. 13, No. 3, 124134 1076-898X/07/$12.00 DOI: 10.1037/1076-898X.13.3.124
124
need to expend significant effort trying to understand the task, and
they must concentrate heavily on what they are going to do to
reduce gross mistakes. With more experience (the second phase),
their obvious mistakes become increasingly rare, and individuals
are able to integrate sequences of steps resulting in smoother and
more efficient performance. After some additional period of train-
ing and experiencefrequently less than 50 hr for most recre-
ational and everyday activities, such as skiing, tennis, and driving
a carthe third phase is attained in which an acceptable standard
of performance is produced without much need for effortful atten-
tion.
Ackerman (1987, 2000) showed that performance on cognitive
skill measures is correlated with different types of abilities during
these phases. For many tasks, measures of general abilities corre-
lated the highest with skill during the initial phase, whereas mea-
sures of more specific abilities became stronger correlates of skill
during later periods. However, most laboratory studies on phases
of skill acquisition have focused on skills that can be attained
within 50 hours of practice (for review, see Ericsson & Williams,
2007)and frequently much shorter practice periods across only a
handful of laboratory sessionswhich raises the issue of whether
these findings would generalize to expert performance in naturally
occurring task environments after hundreds or thousands of hours
of domain experience. To date, evidence from the latter type of
domains indicates significant relationships between basic cogni-
tive processing capacities or general cognitive abilities and per-
formance during the earliest stages of skill acquisition, but weaker
or nonsignificant relationships among highly skilled individuals.
For instance, several independent research teams have failed to
observe significant relationships between IQ or general cognitive
ability test scores and skill in chess as measured by official
tournament performance outcomes (Djakow, Petrowski, & Rudik,
1927; Doll & Mayr, 1987; Horgan & Morgan, 1990), even in
samples ranging widely in skill and ability ranges (Grabner,
Neubauer, & Stern, 2006; Unterrainer, Kaller, Halsband, & Rahm,
2006), and other researchers have little evidence for the influence
of more specific measures, such as visual-spatial aptitude and
visual-memory capacity in adult samples (Doll & Mayr, 1987;
Waters, Gobet, & Leyden, 2002).
General physiological capacity and basic reaction time measures
have also been unsuccessful at predicting maximal performance in
athletic sports and other domains of cognitive and perceptual
expertise in which objective measures of skill are readily available
(Ericsson & Lehmann, 1996; Starkes & Deakin, 1984; Ward &
Williams, 2005). Furthermore, some recent research has suggested
that the general ability correlation in domains like chess dimin-
ished as individuals continued practicing in a domain (Bilalic,
McLeod, & Gobet, 2007), similar to earlier theoretical proposals
(e.g., Ackerman, 1987). The failure of individual differences in
performance on traditional, standardized ability tests to account for
most of the variance in high-level competitive performance sug-
gests the need for an alternative approach to studying the structure
and acquisition of superior ability in these domains.
Capturing and Explaining Superior Cognitive
Achievement: The Expert-Performance Approach
To address the critical issue of capturing and analyzing the
structure and acquisition of superior achievement, one must be
able to measure and reproduce the associated superior performance
under controlled conditions. With this goal in mind, Ericsson and
Smith (1991) developed a three-stage methodological framework
now commonly referred to as the expert-performance approach. In
this approach, the investigator first attempts to identify a repre-
sentative task or tasks that capture the essence of expertise in a
target performance domain. These representative tasks should re-
flect as closely as possible the demands facing experts during
challenging activities in their domain, and the careful design and
selection of representative tasks permits the investigator to repro-
duce experts superior performance under rigorously controlled,
yet ecologically valid task conditions. Moreover, the success of
this first step can be assessed by correlating performance on the
laboratory-based representative tasks with an external criterion of
performance in the target domain. When the first step is success-
fully completed, the investigator next attempts to identify the
mechanisms that mediate the observed superior performance. This
second step frequently involves the observation of performance by
highly skilled (and sometimes also unskilled) individuals on intact
representative tasks in conjunction with the collection of process-
tracing data, such as verbal reports or think-aloud protocols
(Ericsson, 2006b; Ericsson & Simon, 1993). However, various
component demands within a given representative task may also
be isolated and experimentally manipulated to test hypotheses
regarding the mediating mechanisms. During the third and final
stage, the investigator attempts to account for the development of
the previously identified mechanisms. Research participants, and
sometimes also their relatives or teachers, may be asked to provide
critical information about significant milestones in domain partic-
ipation or performance (e.g., age at introduction to the domain, age
at first significant domain achievement, age at commencement
and/or termination of domain instruction), as well as estimates of
the quantity and type of past and present domain-relevant activity,
and other potentially relevant background characteristics that may
be critical factors in the acquisition of domain skill.
Expert Performance in SCRABBLE
In this paper, we present the results of a study in which the
expert-performance approach is applied to the domain of
SCRABBLE. Our initial interest in this particular domain was
partly stimulated by the strong measurement properties of com-
petitive SCRABBLE play. As in chess, every SCRABBLE game
results in an objective, quantifiable outcome (i.e., win, lose, or
draw based on each players game score), and every person com-
peting in a SCRABBLE tournament receives a numerical skill
rating that directly reflects his or her performance against other
rated opponents. Thus, the performance metric associated with
competitive SCRABBLE play provides researchers with mathe-
matically rigorous, yet ecologically valid criterion measure against
which one may validate performance on laboratory-based repre-
sentative tasks. In addition, the study of expert-performance in
SCRABBLE presents an opportunity to test the generalizeability
of findings from studies of individual differences and superior
performance in characteristically nonverbal skill domains to those
that are characteristically verbal. Hence, just as earlier researchers
explored possible linkages between performance on standardized
measures of spatial ability and skilled performance in spatially
oriented game domains like Go and chess (Masunaga & Horn,
125
SPECIAL ISSUE: EXPERT PERFORMANCE IN SCRABBLE
2000; Waters et al., 2002), one might posit analogous relationships
between scores on standardized tests that are frequently associated
with verbal aptitude or verbal intelligence constructs (e.g., vocab-
ulary or word fluency; cf. Carroll, 1993) and skilled performance
in the game of SCRABBLE. From a theoretical perspective based
on general aptitudes or basic cognitive capacities, the level of
proficiency at which SCRABBLE players can demonstrate supe-
rior domain-specific skills may also be somewhat dependent on
their relative levels of general verbal knowledge or efficiency of
word retrieval. This framework would thus predict a relationship
between standardized tests designed to capture these abilities (e.g.,
word fluency or vocabulary tests), and SCRABBLE-specific task
performance. However, there are reasons to be skeptical about the
relationship between standardized measures of verbal ability or
aptitude and success in SCRABBLE based on the alternative
framework of the expert-performance approach.
According to several studies of expert performance, attaining
the highest levels of achievement in a domain requires thousands
of hours of deliberate practice (Ericsson, 2006a; Ericsson,
Krampe, & Tesch-Romer, 1993), during which the aspiring indi-
vidual works to improve specific aspects of performance. By
repeatedly modifying ones strategies, processes, and representa-
tions over this extensive period of training, an individual may
eventually achieve maximal adaptation to domain-specific task
constraints (Ericsson & Lehmann, 1996). This extended adaptation
to the constraints of a particular performance domain often result
in the acquisition of cognitive processing mechanisms that are
highly specific to a domain and, thus, are unlikely to transfer to
other domains (cf. Ericsson & Kintsch, 1995). Hence, it follows
that in the case of SCRABBLE, extended deliberate practice would
result in superior performance only on SCRABBLE-related tasks.
Suggestions about the specific content of such practice may be
gleaned from first-hand reports of aspiring players (Fatsis, 2002)
as well as instructional resources written by established experts
(Edley & Williams, 2001). These sources emphasized activities
such as the targeted study and memorization of word lists, activ-
ities to strengthen facility in anagramming, and exercises to im-
prove tactical strategies specific to the game of SCRABBLE.
However, SCRABBLE players are not required to know the mean-
ing of a word nor how to pronounce it to use it during game play.
It is therefore reasonable to argue that deliberate practice in
SCRABBLE may result in the development of specialized skills
related to the retrieval of tactically important words and their
visual recognition among a set of scrambled letters without nec-
essarily requiring superior verbal aptitude or intelligence in the
broader sense, and we refer to the hypothesis of a distinct ability
to access words based on both visual letter cues (anagramming)
and their value in SCRABBLE play as the anagrammatic word-
identification skill hypothesis. Hence, better players have acquired
highly specific skills relevant to anagramming words and identi-
fying them as legal plays. This hypothesis makes several concrete
predictions that can be examined using the expert-performance
approach.
In this paper, we first describe how we created a representative
SCRABBLE problem-solving task involving simulated SCRAB-
BLE game positions while following some of the same design
principles successfully applied to the study of expert performance
in chess (De Groot, 1946/1978; van der Maas & Wagenmakers,
2005). Given that this task is designed to capture both hypothe-
sized aspects of SCRABBLE skill (namely anagramming and
knowledge of game-relevant words), we predict that performance
on the SCRABBLE problem-solving test will strongly differentiate
between players who are known to perform at superior levels in
SCRABBLE tournaments and players who are known to perform
at only average levels based on tournament ratings. Second, we
attempt to test our anagrammatic word-identification skill hypoth-
esis about the cognitive underpinnings of SCRABBLE expertise
by incorporating methodological elements from both the individ-
ual differences and expert-performance research approaches. Fol-
lowing the expert-performance approach, if we were able to suc-
cessfully capture high-level SCRABBLE skill with our
SCRABBLE move-selection task, then a systematic analysis of the
verbal protocols collected during this task should be helpful in
detecting the presence or absence of the specialized word gener-
ation and detection mechanisms that are proposed to mediate
superior SCRABBLE performance as specified by the anagram-
matic word-identification skill hypothesis. However, we should
also be able to relate the underlying structure of these mechanisms
to a set of verbal ability tasks that are either highly representative
of the particular demands imposed by the SCRABBLE game
environment as well as to similar tasks that are not representative.
Our hypothesis predicts that SCRABBLE skill will predict perfor-
mance among skilled players only on tasks that are related to
SCRABBLE task demands. Finally, we collect retrospective esti-
mates of current and accumulated SCRABBLE practice activities
to obtain potential insights into the development of the cognitive
mechanisms mediating superior SCRABBLE performance. Based
on the anagrammatic word-identification skill hypothesis, as well
as the broader theory of deliberate practice (Ericsson et al., 1993),
we hypothesize that expert SCRABBLE players have acquired the
knowledge necessary for success in high-level tournament play by
engaging in outside activities designed to improve specific aspects
of SCRABBLE performance. Thus, rather than merely playing
SCRABBLE against other people, aspiring players will engage in
focused study activities designed to increase their skill at anagram
solution and improve their knowledge of strategically important
words. An addendum to this hypothesis is that individuals who
engage in SCRABBLE-relevant activities may be differentiated
from non-SCRABBLE players with respect to their broader verbal
abilities or aptitudes, either due to self-selection into the domain or
due to the requirement for high initial levels of these abilities for
competitive performance levels. We thus assess the degree to
which rated SCRABBLE players are representative of the greater
population of individuals presumed to have at least average levels
of verbal ability, but who have only minimal experience with this
particular skill domain.
Method
Participants
A total of 40 tournament-rated SCRABBLE players (29 male,
11 female), ranging from 21 to 83 years of age (M 47.7, SD
14.9) participated in the study after responding to public announce-
ments about the project on SCRABBLE-related Websites. These
players were selected solely on the basis of their self-reported skill
rating in the NSA (National SCRABBLE Association), which
were verified against their official ratings as reported in publicly
126
TUFFIASH, RORING, AND ERICSSON
available lists (see http://www.scrabble-assoc.com/ratings/). Dur-
ing recruitment, we specifically targeted players from two rating
ranges: 800 to 1200, or 1600 and above. The lower rating range
was intended to capture the level of skill representative of the
average player in the NSA. The higher rating range was intended
to capture the level of skill representative of players who typically
compete in the top division of the National SCRABBLE Cham-
pionship (i.e., those who would be eligible for the title of National
SCRABBLE Champion), and who established players would be
most likely to regard as experts (cf. Fatsis, 2002).
The final elite rated player subsample consisted of 20 partici-
pants (19 male, 1 female) with a mean NSA rating of 1772 (SD
89). Given that the variances of the two samples were unequal, we
used the appropriate t test for this case. The mean NSA skill rating
for this group was not significantly different from the mean skill
rating of all Division 1 contestants (n 153) at the 2004 NSA
Championship (M 1788, SD 107, range 1498 to 2051),
t(26.6) 0.76, ns. The final average rated player subsample
consisted of 20 participants (10 male, 10 female) with a mean NSA
rating of 1005 (SD 93). The mean NSA rating for this group was
not significantly different from the mean rating of all other active
NSA members (n 6,993) at the start of the study (August, 2004
rating list: M 979, SD 368, range 283 to 2052), t(20.7)
1.19, ns. All rated participants completed testing at various
locations near SCRABBLE tournament sites or local SCRABBLE
clubs between August 2004 and August 2005, and were paid $40
for completing the 2-hr test session plus an additional $10 for
completing a mail-back survey about their past and present
SCRABBLE-related activities.
The mean age of the elite rated player subsample (M 42.7,
SD 11.3 years) was significantly lower than that of the average
rated player subsample (M52.7, SD 16.6 years), t(38) 2.23,
p .05, d 0.71. However, these two subsamples did not differ
significantly from each other with respect to their self-reported
years of education (elite player subsample M 16.3, SD 3.0
years; average player subsample M 17.4, SD 3.6 years),
t(38) 1.02, ns; nor did they differ significantly from each other
with respect to their self-reported years of SCRABBLE experience
(elite player subsample M27.9, SD 11.6 years; average player
subsample M 35.2, SD 11.7 years), t(38) 1.97, ns.
An additional sample of 40 unrated, novice SCRABBLE players
(5 male, 35 female), ranging from 18 to 22 years of age (M18.6,
SD 0.6 years) was obtained from a pool of undergraduate
students enrolled in an introductory university psychology course
during the 2005/2006 academic year. All novice participants re-
ported playing SCRABBLE at least once in their lifetime, but none
of them reported any previous SCRABBLE tournament experi-
ence. Participants in the novice sample completed the same battery
of tests taken by the tournament-rated SCRABBLE players, but
received credit toward the fulfillment of experimental participation
requirements for their introductory psychology course instead of
cash payments.
Representative SCRABBLE Problem-Solving Task
To accurately reproduce the essential cognitive and behavioral
demands imposed by competitive SCRABBLE, while minimizing
the potential effects of previous exposure to specific game situa-
tions (such as those found in published strategy books or annotated
championship tournament games), we generated a collection of
novel game positions using a widely available computerized ver-
sion of SCRABBLE (Atari, 2001). Positions were reached after
allowing the program to play against itself, with the softwares
skill level set to mimic the performance of the best SCRABBLE
tournament players in the world (rating of around 2100; see
Sheppard, 2002). The selection of specific positions was partly
guided by the desire to have solutions that expert SCRABBLE
players should know based on their purported study of certain
word lists (three- and four-letter words, words containing the
letters J, Q, X, or Z) or common bingo words (i.e.,
words that are typically seven or eight letters in lengthplays that
use all seven letters on a players rack result in a 50 point bonus
and thus typically constitute the most desirable plays; cf. Edley &
Williams, 2001; Fatsis, 2002). A preliminary set of 15 game
positions were presented to a small sample of local tournament-
rated players (novice through subexpert) to determine the effi-
ciency of the task at differentiating between varying skill levels, to
estimate the general time frame required for skilled players to
choose a reasonable move, and to check the frequency with which
experienced players might recognize the best solutions to the
positions without any thought or deliberation (due to either their
exceptionally high frequency of occurrence or to the similarity
between the test position and a position from a well-known pre-
vious game, such as those found in published strategy books or
annotated championship matches). Twelve positions (1 opening,
10 midgame, 1 endgame) were extracted from the original 15-item
set to create a diverse sample of positions that might be encoun-
tered during highly competitive SCRABBLE play, but that could
be completed in approximately 1 hr at a time limit of 5 minutes per
position. The game boards for the final 12 positions were con-
verted to high-resolution computer files, enlarged to match the size
of a standard SCRABBLE grid, printed on white nonglossy poster
paper, and mounted on black mat board to allow for rapid presen-
tation and repeated use (see Halpern & Wai, 2007, p. 81, for a
photo of an actual SCRABBLE game board).
During test administration, participants were shown one game
position at a time, along with a plastic SCRABBLE tile rack
containing the wooden letter tiles that were assigned to the human
player for the corresponding position during the simulated source
game. To further mimic the natural conditions of competitive play,
each participant received an official tournament scoresheet with
each game position showing the total point scores obtained by both
of the simulated players (self and opponent) up to the current point
in the simulated game, and participants were permitted to write
down any information that they wished on these sheets. Once all of
the materials for a given test position were placed in front of the
participants, they were instructed to find the best move for that
position in 5 minutes or less. They were permitted to move and
rearrange the letter tiles on the tile rack in any manner that they
wished, and when they were ready to select their moves, partici-
pants were instructed to place their letter tiles onto the game board
exactly as if they were playing a real SCRABBLE game. Partici-
pants were also instructed to provide both concurrent and retro-
spective think out loud reports (Ericsson & Simon, 1993) during
each problem in the SCRABBLE problem-solving test. Think-
aloud instructions followed a standard script, with all participants
completing a series of non-SCRABBLE practice problems (i.e.,
letter arithmetic) prior to the start of the SCRABBLE problem-
127
SPECIAL ISSUE: EXPERT PERFORMANCE IN SCRABBLE
solving test to orient them to the verbal reporting procedure
(contact authors for verbatim think-aloud instructions and
warm-up problems).
Participants move choices were scored according to the raw
number of points that their chosen plays would have produced for
each position, providing a potential range from 0 for any illegal move
to 98 for the highest-scoring best move (Item 12). The total number of
points across the 12 different positions was the primary dependent
variable for this task. Participants from both rated-player groups were
largely successful in selecting legal move choices: There were only
15 nonlegal plays (words that were not listed in the SCRABBLE
dictionary; Merriam-Webster, 1996) across all 12 test positions (11
by average-rated players, 4 by elite players), comprising approx-
imately 3% of all move choices. We computed an internal consis-
tency coefficient (Cronbachs alpha) of .90 for the final 12-item
SCRABBLE problem-solving test, with an average interitem cor-
relation of .55both figures roughly conforming to the guidelines
for reliability in applied measures of narrowly defined constructs
(Clark & Watson, 1995; Nunnally, 1967).
Standardized Verbal Ability Tests
Digit-symbol. The WAIS III Digit-symbol coding test
(Wechsler, 1997) was administered as a proxy measure of
perceptual speed. Participants were given 2 minutes to complete
as much of the coding sheet as possible, working from left to
right without skipping any symbol response boxes or rows.
Scores were obtained by counting the total number of symbols
correctly coded within the 2-minute time limit. This test also
served as a filler task to reduce potential contamination between
the previous SCRABBLE move-selection test and subsequent ver-
bal ability tests in which participants might be asked to generate or
respond to words that they previously encountered or generated
during the SCRABBLE problem-solving test positions. Wechsler
reported an average testretest reliability coefficient of .86 for this
test in a standardization sample ranging from 16 to 89 years of age.
Letter fluency. Participants were asked to verbally generate as
many words as possible containing certain letters. Each participant
was prompted to generate words beginning with F, A, S, and
UN-, as well as four-letter nouns, with a time limit of 1 minute
per letter category. The selection of the F-, A-, and S-
prompts was based on their use as a standard neuropsychological
measure of verbal fluency (Borkowski, Benton, & Spreen, 1967).
UN-words (words with the prefix UN) were used because
SCRABBLE players claimed to study lists of words with common
prefixes as a training technique (Edley & Williams, 2001;
Schachner, 1998). Four-letter nouns were added as a control cat-
egory representing a word-related constraint (i.e., part of speech)
hypothesized to have minimal correspondence with the demands
placed on word generation by the rules of SCRABBLE game play.
Responses were audiotaped for postsession scoring and analysis.
Internal consistency (Cronbachs .87) for the three traditional
verbal fluency items (FAS) was in line with estimates reported in
the clinical neuropsychology literature (Spreen & Strauss, 1991).
Vocabulary. Two standardized tests were used to measure
general English vocabulary. First, participants completed the vo-
cabulary test from the StanfordBinet Intelligence Test (SB III;
Terman & Merrill, 1960), in which a set of 45 increasingly
difficult words were read aloud by the test administrator, and the
respondent was prompted to provide an oral definition for each
word. Responses were tape recorded for later scoring and analysis.
Second, participants completed the North American Adult Reading
Test (NAART; Blair & Spreen, 1989), in which they were given a
printed list of 61 words with instructions to read each and every
word aloud at their own pace. Scoring on this test is based on
correct pronunciation of the words according to a standard key.
The test took approximately 2 to 3 minutes to complete, during
which time the participants oral responses were audiotaped for
postsession scoring and analysis. Both vocabulary tests were
scored independently by two judges, and interrater reliability co-
efficients for the SB III vocabulary test, r(38) .86, and the
NAART, r(38) .91, were in line with values reported for these
tests in the published literature (Terman & Merrill, 1960; Uttl,
2002).
Anagrams. Fifty-one English anagrams created by Borzycki
(2001) were presented via computer using E-Prime (E-prime,
version 1.1.; Schneider, Eschman, & Zuccolotto, 2002). These
anagrams ranged from five to eight letters in length, and were
composed from source words at one of three levels of written word
frequency (low 5, for all words; medium, M 116.7, SD
8.96; high, M 470.5, SD 149.6; Kucera & Francis, 1967). In
the present study, the original 51-item set was divided into two
testing blocks. During the first block (42 items), participants had
5 s to view and solve each anagram while thinking aloud. In the
second block (9 items), participants had 20 s to view and solve
each anagram while thinking aloud, and were then instructed to
provide a retrospective think-aloud report for up to 10 s. The
presentation order of the individual anagrams within each test
block was randomized across participants to control for potential
interitem facilitation or interference effects. Estimates of internal
consistency (Cronbachs alpha) for the anagram task were .95 and
.90 for the 5-s and 20-s blocks, respectively, and total scores for
the two anagram blocks correlated highly with each other, r(78)
.91, p .001. Due to the strong correlation between the first and
second parts of this task, all results reported for anagramming
performance are based on the combined total number of anagrams
solved across the two blocks.
The measures described above were administered in the follow-
ing fixed order to all participants SCRABBLE problem-solving
task, digit-symbol, verbal fluency, NAART, SB III vocabulary,
and anagrams.
Questionnaire Assessment of SCRABBLE Experience
Participants received a seven-page background survey to be com-
pleted at home and returned by mail. The survey contained items
regarding the participants general background, SCRABBLE-related
performance and experience milestones, current SCRABBLE-related
activities, cumulative SCRABBLE activities, and personality traits.
Current SCRABBLE-related activities included the number of hours
of study in a typical week spent analyzing other peoples past
SCRABBLE games, analyzing ones own past SCRABBLE games,
making word lists, studying word lists, playing against a SCRABBLE
computer program, playing SCRABBLE against other people (out-
side of tournamentsas in club meetings or casual games played
against friends), solving anagrams, solving theoretical SCRABBLE
problems or tactical exercises, and playing other word games. Partic-
ipants also reported their cumulative SCRABBLE activity within two
128
TUFFIASH, RORING, AND ERICSSON
broad categoriessolitary study and competitive playby providing
retrospective estimates of time spent on each activity during a typical
week of the year at each age since they first began playing SCRAB-
BLE. Although reliability estimates are not available for this partic-
ular survey, previous research eliciting similar self-report practice
estimates repeatedly from experts in sports and music have reported
testretest reliabilities at or above .80 (Bengtsson et al., 2005; Cote,
Ericsson, & Law, 2005). The entire SCRABBLE experience survey
took approximately 30 minutes to complete, and participants received
$10 for returning their completed surveys.
Results
Representative Task Performance
Average scores on the move-selection task for the three tested
skill groups are shown in Table 1. In support of our first predic-
tionthat our SCRABBLE move-selection task would capture the
skills that are supposedly reflected by SCRABBLE tournament
ratingswe found that the SCRABBLE move-selection task
scores and SCRABBLE tournament performance ratings were
highly correlated, r(38) .89, and virtually unaffected by poten-
tial confounds related to age or years of education, partial r(36)
.88. In addition, results from a univariate analysis of covariance
test (ANCOVA) with age and years of education as covariates
indicate that our representative move-selection task strongly dif-
ferentiated between our rated players, F(1, 36) 573.49, MSE
8,055, p .001,
p
2
.94. Note that a test comparing the
variances of the mean move-selection scores in these two groups
was not significant, Levines F(1, 38) 0.39, ns, but the mean
score difference was statistically significant both with and without
the presence of the age and education covariates. The large effects
reported here are consistent with the hypothesis that a laboratory
measure requiring move decisions in response to representative
SCRABBLE game situations could successfully capture individual
differences in skill as measured by an ecologically valid perfor-
mance criterion.
SCRABBLE Problem-Solving Verbal Reports
We conducted a preliminary analysis of our think-aloud ver-
bal protocols to validate the anagrammatic word-identification
skill hypothesis of skill group differences in representative
SCRABBLE task performance. Because the verbal reports of our
SCRABBLE players mainly consisted of strings of candidate
solution words (e.g., CONNIES, CONES, CON, COSINE,
CONINES), we simply coded each word from the verbalized
word sequence as a bingo word, a nonbingo word, or a word that
was illegal for the current position according to the Official
SCRABBLE Players Dictionary (1996). For this specific five-
word example, CONNIES would be coded as an illegal word,
CONES, CON, and COSINE would be coded as nonbingo words,
and CONINES would be coded as a bingo word (given that playing
this in this position requires all seven letter tiles). We then calcu-
lated the total number of words verbalized in each of these three
categories by each participant during each SCRABBLE move
selection.
In the present analysis, we investigated protocols from 2 of the
12 positions, 1 containing a high-scoring bingo solution, and 1 in
which the highest-scoring solution was a nonbingo word. Ob-
served move scores on both items were significantly correlated
with overall SCRABBLE test performance for the bingo, r(38)
.87, p .001, and the nonbingo position, r(38) .34, p .05.
Consistent with our hypothesis that SCRABBLE experts have
studied specific lists of tactically important words, ANCOVA
analyses controlling for age and education reveal that the number
of verbalized bingo words was predicted by skill group in both the
bingo, F(1, 35) 43.87, MSE .19, p .001,
p
2
.56; and
nonbingo problem, F(1, 35) 10.42, MSE .22, p .05,
p
2

.23. In addition, we found that the overall number of words

generated during the 5-minute time limit was negatively predictive
of SCRABBLE expertise group for the bingo problem, F(1, 35)
6.027, MSE 54.30, p .05,
p
2
.15; but not for the nonbingo
problem, F(1, 35) 0.30, MSE 103.85, ns. Finally, given our
prediction that anagramming skill would differentiate stronger
from weaker players, we examined the number of letters in the first
candidate solution verbalized during both move-selection prob-
lems. We found that skill group membership was significantly
related to the number of letters contained in the first candidate
solution generated during both the bingo, F(1, 35) 4.78, MSE
1.32, p .05,
p
2
.12; and nonbingo problems, F(1, 35)
14.93, MSE 3.34, p .001,
p
2
.29; consistent with better
players having enhanced skill at mentally manipulating letters to
form words.
Verbal Ability Tests
The mean scores of the two rated skill groups, as well as the
novice group on the verbal ability tests are reported in Table 1. To
evaluate the predictions of the anagrammatic word-identification
skill hypothesis dealing with individual differences in performance
on our verbal ability task battery, separate univariate ANCOVA
tests were performed on each task to make comparisons between
different skill groups while controlling for age and years of edu-
cation. As predicted, significant differences between the two rated-
skill groups were observed on the anagramming task, F(1, 36)
38.6, MSE 39.60, p .001,
p
2
.51; and the UN-words letter
fluency test item, F(1, 36) 12.63, MSE 20.11, p .05,
p
2

.27. There were no significant differences, however, between elite

and average rated players on Digit-symbol coding, F(1, 36)
1.64, MSE 286.95, ns,
p
2
.04; FAS letter fluency, F(1, 36)
Table 1
Comparison of SCRABBLE Skill-Group Mean Scores (Standard
Deviations in Parentheses) on Representative and Standardized
Verbal Ability Tasks
Task Elite Average Novice
SCRABBLE move choice 452.5 (51.1) 261.8 (45.8) 115.8 (42.8)
Digit-symbol coding 86.8 (22.2) 86.6 (16.5) 84.9 (12.5)
Letter fluency
FAS words
a
65.2 (13.7) 61.9 (10.3) 41.5 (9.2)
UN words 17.0 (5.4) 12.2 (3.8) 7.2 (2.6)
Four-letter nouns 17.3 (5.7) 14.8 (5.6) 7.0 (4.0)
Stanford-Binet vocabulary 33.4 (5.3) 33.0 (4.2) 23.4 (3.1)
NAART 46.6 (4.9) 46.7 (5.5) 33.3 (5.9)
Anagrams 37.4 (6.9) 21.9 (7.5) 7.4 (3.8)
Note. NAART North American Adult Reading Test.
a
Combined total from F-, A-, and S- word prompts.
129
SPECIAL ISSUE: EXPERT PERFORMANCE IN SCRABBLE
0.22, MSE 152.11, ns; four-letter noun fluency, F(1, 36) 0.83,
MSE 34.69, ns; SB III vocabulary, F(1, 36) 0.81, MSE
22.80, ns; or the NAART, F(1, 36) 0.01, MSE 29.15, ns.
We also ran a series of t tests to compare the combined elite- and
average-rated SCRABBLE subsamples with the 40 unrated novice
players on our verbal ability test battery (see Table 1). These
comparisons showed that the rated players demonstrated signifi-
cantly higher mean performance than the college students on the
SB III vocabulary test, t(78) 10.89, p .001, d 2.46; the
NAART, t(78) 10.62, p .001, d 2.37; FAS letter fluency,
t(78) 9.33, p .001, d 0.68; UN-word fluency, t(78) 7.95,
p .001, d 0.41; and four-letter noun fluency, t(78) 8.03, p
.001, d 1.79. However, we did not find a significant difference
in performance between rated players and novice college students
on the Digit-symbol coding test, t(78) 0.49, ns.
SCRABBLE-Related Activities and Cumulative Practice
Estimates
Skill group means for the elite- and average-rated players on the
complete set of current SCRABBLE-related activities are listed at
the top of Table 2. Consistent with our third hypothesis, elite
SCRABBLE players spent significantly more time engaged in
current activities involving focused study of SCRABBLE-related
task materials; specifically, analysis of their own previous games,
F(1, 35) 6.03, MSE 3.50, p .05,
p
2
.15; and solving
anagrams, F(1, 35) 5.47, MSE 80.57, p .05,
p
2
.14; but
significantly less time playing word games other than SCRAB-
BLE, F 4.92 (1, 35), MSE 188.15, p .05,
p
2
.12.
However, elite- and average-rated players did not differ in their
estimates of time spent during a typical week analyzing other
players previous games, F(1, 35) 0.27, MSE 0.15, ns;
playing in SCRABBLE tournaments, F(1, 35) 1.04, MSE
8.92, ns,
p
2
.02; playing SCRABBLE against other people for
fun, F(1, 35) 1.42, MSE 24.16, ns
p
2
.03; playing
SCRABBLE against a computer program, F(1, 35) 0.15,
MSE 11.96, ns; studying word lists, F(1, 35) .12, MSE
7.50, ns,
p
2
.05; making word lists, F(1, 35) 0.01, MSE
1.47, ns; or solving theoretical SCRABBLE problems, F(1, 35)
0.55, MSE 0.88, ns.
Skill group differences in retrospective estimates of SCRAB-
BLE activity accumulated over longer time intervals are shown in
the lower portion of Table 2. Using the participants self-report
survey estimates of their hours spent on each activity during a
typical week of each year, we calculated cumulative estimates of
serious SCRABBLE study time and serious SCRABBLE playing
time over three long-term intervals lifetime, past 10 years, and
all years since joining a SCRABBLE club. Although the table
contains group means and standard deviations for the raw time
estimates, these estimates were log transformed prior to group
comparisons in accordance with common practice for these types
of variables (see Charness, Krampe, Mayr, 1996). Statistical com-
parisons (ANCOVA) of the differences between our elite- and
average-level player groups yielded statistically significant and
large skill effects for all six of the (log transformed) cumulative
activity variables after accounting for age and years of education.
Specifically, skill group predicted lifetime accumulated study, F(1,
36) 21.28, MSE 6.28, p .001,
p
2
.37; accumulated study
in clubs, F(1, 35) 20.22, MSE 4.52, p .001,
p
2
.37;
accumulated study over the past 10 years, F(1, 34) 17.44,
MSE 3.20, p .001,
p
2
.34; lifetime competitive play, F(1,
36) 28.72, MSE 3.00, p .001,
p
2
.44; lifetime club play,
F(1, 36) 28.25, MSE 3.30, p .001,
p
2
.44; and
competitive play over the past 10 years, F(1, 36) 14.99, MSE
1.94, p .001,
p
2
.29. Furthermore, self-report estimates of
anagramming practice during a typical week were significantly
correlated with anagramming task performance, r(37) .39, p
.05.
Discussion
Overall, our study was successful in demonstrating how the
expert-performance approach can be applied to improve under-
standing of superior performance in a specific skill domain. We
successfully completed the first step of the expert-performance
approach by demonstrating a strong statistical relationship be-
tween move-selection during representative SCRABBLE game
situations and skill ratings extracted from real SCRABBLE tour-
naments. Subsequent analyses of the think-aloud verbalizations
generated during the move-selection task and the scores from a
variety of standardized and laboratory-based verbal ability tasks
then allowed us to successfully complete the second step of the
expert-performance approach by identifying critical cognitive sub-
components of SCRABBLE play that were consistent with our
anagrammatic word-identification skill hypothesis. Specifically,
our preliminary analyses of the think-aloud reports showed that
experts generated more bingo words than do novices, and that their
initial response to a problem tended to have more letters, both
Table 2
Rated-Player Skill Group Mean Time Estimates (in Hours;
Standard Deviations in Parentheses) for Current and
Cumulative SCRABBLE-Related Activities
Current activities Elite Average
Analysis of own previous
SCRABBLE games
0.9 (0.8) 0.2 (0.6)
Analysis of others previous
SCRABBLE games
0.2 (0.2) 0.1 (0.4)
Anagramming 4.2 (5.0) 0.8 (1.4)
Playing other word
games/puzzles
0.1 (0.3) 4.0 (8.8)
Playing SCRABBLE against
others (tournament)
2.1 (2.1) 1.7 (3.7)
Playing SCRABBLE against
others (nontournament)
7.3 (4.7) 5.6 (5.5)
Playing SCRABBLE against a
computer
2.1 (3.1) 2.0 (3.6)
Studying word lists 2.6 (3.6) 0.8 (1.0)
Making word lists 0.9 (1.1) 0.6 (1.1)
Solving (theoretical)
SCRABBLE problems
0.2 (0.4) 0.5 (1.2)
Cumulative activities
Serious study: Lifetime 5,084.1 (4,818.4) 1,318.2 (1,464.6)
Serious study: Club years 4,830.6 (4,785.7) 1,224.6 (1,465.2)
Serious study: Past 10 years 3,541.0 (3,467.2) 886.6 (801.9)
Competitive play: Lifetime 6,043.4 (4,019.8) 2,378.7 (2,364.6)
Competitive play: Club years 5,522.1 (4,205.4) 1,823.6 (1,931.3)
Competitive play: Past 10 years 3,901.0 (2,609.2) 1,626.9 (1,575.9)
130
TUFFIASH, RORING, AND ERICSSON
findings was consistent with the notion that better players have
accumulated more orthographic word knowledge and are thus
better able to locate high-scoring solutions from anagrammatic
letter arrays. Complementary findings from the broader task bat-
tery then illustrated how laboratory and standardized measures of
verbal ability corresponding to facility in orthographically based
word retrieval and recognition (i.e., letter fluency and anagram-
ming) could differentiate between elite and average-rated players,
even though the two groups were not significantly different from
each other on other measures of verbal knowledge strongly em-
phasizing the semantic or phonological properties of language (i.e.,
word definition or pronunciation vocabulary). Finally, we uncov-
ered information about the types of training experiences support-
ing the acquisition of superior SCRABBLE performance by using
self-report estimates of current and accumulated time spent on
different types of SCRABBLE-related activities. SCRABBLE ex-
pertiseas represented by performance on both the external cri-
terion measure (i.e., NSA ratings) as well as performance on
laboratory measures (i.e., move selection and anagramming)
was significantly associated with the quantity of time spent on
SCRABBLE-related activities that best met the theoretical descrip-
tion of deliberate practice, but was not significantly related to the
quantity of time spent on other activities that would be considered
suboptimal forms of training according to the principles of delib-
erate practice theory.
Implications for the Study of Representative Task
Performance
Analyses concerning the reliability and validity of our central
representative taskSCRABBLE game move-selectionare en-
couraging on multiple levels. First, the external validity of our
representative task was at or above the same level found in other
recent studies employing similar techniques. For instance, van der
Maas and Wagenmakers (2005) reported correlations of .78 and
.81 between chess tournament ratings and two versions of a 40-
item chess move-choice task. As in our task, their measure also
employed static game positions and required participants to select
the best move under a specified time constraint. However, our task
did just as well, even though we employed a much smaller pool of
items and administered them to a relatively small sample. It is also
worth noting that we were able to obtain a high degree of external
validity from samples of widely divergent skill levels while simul-
taneously collecting think-aloud reports. This finding is particu-
larly important in light of long-standing concerns about the poten-
tial interference of verbal report methods with the cognitive
processes required to complete difficult problem-solving tasks
(Ericsson, 2006b; Ericsson & Simon, 1993).
Implications for the Study of Cognitive Mechanisms
Mediating Superior Verbal Abilities
Our analysis of expert SCRABBLE performance is broadly
consistent with the notion that experts acquire domain-specific
representations to support effective encoding and access from
long-term memory as a form of working memory (LTWM;
Ericsson & Kintsch, 1995). For instance, it is possible that such
structures mediate the solution of difficult anagrams, particularly
when large numbers of letters are mentally manipulated under time
constraints (cf. Borzycki, 2001). However, further research is
needed to formulate and test more detailed accounts of how
LTWM theory might account for exceptional skill in word gener-
ation during SCRABBLE games or related word-retrieval activi-
ties like anagramming.
With respect to the role of individual difference in verbal
abilities, our findings are consistent with research from other skill
domains. Although our task battery was not an IQ test, our results
are generally consistent with the common finding that IQ and other
measures of general cognitive ability become weaker correlates
of task-specific skills with increasing domain activity (e.g.,
Ackerman, 1987; Bilalic et al., 2007; Hulin, Henry, & Noon, 1990;
Keil & Cortina, 2001). However, a more surprising finding was the
discovery that our rated SCRABBLE players as a whole exhibited
superior performance relative to college students for all but one
test (Digit-symbol) in our ability battery. This difference is con-
sistent with the findings of Halpern and Wai (2007), who also
reported a significant difference between the vocabulary test
scores of rated SCRABBLE players and undergraduate college
students, as well as with the results from Andreasen (1987), who
reported significantly higher vocabulary scores of eminent writers
when compared to an age- and education-matched control group.
Consequently, competitive SCRABBLE players level of perfor-
mance on standardized verbal ability measures are probably not
representative of verbal ability test performance within the general
population of college students.
To investigate whether our data were consistent with the no-
tion of a verbal ability threshold for acquiring expertise in
SCRABBLE, we searched for clear counterexamples of individu-
als within our elite-rated sample performing at or below the mean
score for our college student novice sample on our tests of verbal
ability. This search revealed several such cases for one of our
standardized vocabulary measures (SB III) as well as one of our
contrived verbal fluency measures (four-letter nouns). The exis-
tence of individuals exhibiting characteristically normal levels
of verbal ability test performance who have achieved superior
levels of SCRABBLE performance suggests that high scores on
verbal ability tests should not be considered a prerequisite for
becoming a SCRABBLE expert. More interesting, Halpern and
Wai (2007) reported a significant correlation (r .45) between
vocabulary and SCRABBLE ratings in the text of their document,
but in their Table 4, the same correlation is reported as zero, due
to missing values for some rated players (D. Halpern, personal
communication, February 28, 2007). This would imply that the
effect reported in the text was driven by a couple of cases, and
thus, not necessarily generalizeable to the rest of the sample. Thus,
their data is consistent with ours given that for most of their
participants the correlation between the vocabulary test and the
SCRABBLE ratings was not significant.
The lack of substantial correlations between the vocabulary and
NAART tests on the one hand, and SCRABBLE expertise may not
be a decisive indicator of the lack of ability correlations with
individual differences in SCRABBLE expertise, because these
tests only capture part of the wide range of abilities that underlie
verbal IQ. It may be that these tests are not at either the optimal
level of specificity/generality, or that they are not assessing critical
verbal abilities for SCRABBLE performance, such as might be
obtained with ability measures like memory for meaningful mate-
rial, phonetic decoding, spelling, and so on (e.g., see Horn, 1989).
131
SPECIAL ISSUE: EXPERT PERFORMANCE IN SCRABBLE
See Wittmann & Su (1999) for a discussion of Brunswik sym-
metry and the selection of predictors for criterion task performance
issues.
One plausible explanation for the superior verbal ability test
scores of the rated SCRABBLE players in our study is that the
types of individuals who play SCRABBLE at a competitive level
already had a strong interest in verbal activities prior to their first
encounter with this game. Indeed, it seems highly likely that
SCRABBLE novices could benefit from possessing above-average
amounts of English word knowledge acquired through casual
experience. Prior to significant engagement in specific activities
designed specifically to improve SCRABBLE performance, indi-
viduals have minimal SCRABBLE-specific knowledge to call on
when making game-related decisions. Thus, novices are likely to
rely more heavily on the kinds of verbal knowledge gained through
exposure to the standard school curriculum or everyday leisure
activities. This explanation could potentially account for the sig-
nificant correlations between vocabulary and skill ratings in sam-
ples composed primarily of nonrated amateurs and intermediate-
level rated SCRABBLE players (Borzycki, 2001; Halpern & Wai,
2007; for a related discussion of crossword puzzle solvers, see
Hambrick, Salthouse, & Meinz, 1999). However, it is also possible
that the difference between rated and unrated SCRABBLE players
in performance on the standardized verbal ability tests is itself
partly a consequence of extensive SCRABBLE competition and
intensive study of verbal materials.
If the most effective deliberate practice activities in SCRAB-
BLE primarily shape only SCRABBLE-specific performance
mechanisms, then our failure to detect a significant difference in
vocabulary between elite- and average-rated SCRABBLE players
with roughly equal amounts of education on two different stan-
dardized vocabulary tests may simply be evidence supporting the
popular belief that success at the highest levels of competition has
little, if anything, to do with ones knowledge of word meanings
(Edley & Williams, 2001; Fatsis, 2002). Unfortunately, our small
sample size, quasi-experimental design, and cross-sectional obser-
vations prohibit us from drawing stronger conclusions about this
and other potential explanations. We suggest that in the future,
researchers interested in these issues consider incorporating lon-
gitudinal measurement techniques, large random samples, and a
broader selection of standardized aptitude measures to examine
whether individual differences in general abilities or basic cogni-
tive processing capacities have any causal influence on the devel-
opment of skilled SCRABBLE performance.
Implications for the Theory of Deliberate Practice
Finally, the pattern of significant skill-group differences that we
observed for estimates of cumulative and recent time spent on
SCRABBLE-related activities is largely consistent with the results
from other studies of practice in SCRABBLE (Borzycki, 2001;
Halpern & Wai, 2007), music (Ericsson et al., 1993; Sloboda,
Davidson, Howe, & Moore, 1996), chess (Charness et al., 1996;
Charness, Tuffiash, Krampe, Reingold, & Vasyukova, 2005), and
sports (Ward, Hodges, Williams, & Starkes, 2004; Ward &
Williams, 2005). We found significant differences between elite
and average-rated players on estimates of both cumulative
SCRABBLE study and playing time, with many of the top players
reporting cumulative estimates of between 5,000 and 10,000 hr
since their first exposure to the game. However, we did not find
particularly strong skill-group differences in estimates of current
SCRABBLE activities that were broken down into more specific
task categories. To some degree, this may simply reflect week-to-
week variability in the amount and type of domain-relevant activ-
ities undertaken, with many serious players possibly concentrating
the majority of their study time during the weeks or days imme-
diately preceding an important tournament (see Charness et al.,
1996). However, we also acknowledge that our SCRABBLE ex-
perience questionnaire was only intended to serve as a crude
instrument for assessing an individuals SCRABBLE-related prac-
tice and playing habits, and thus may not have captured all of the
activities that are relevant to maintaining or improving ones
SCRABBLE skill. Conceivably, other more intensive measures
(e.g., daily time diaries; Deakin, Cote, & Harvey, 2006) could
provide a more precise account of a SCRABBLE players daily or
weekly activities.
General Conclusions
In most previous applications of the expert-performance ap-
proach to skilled and expert performance, the identification and
selection of representative tasks, specification of the mediating
mechanisms, and delineation of domain-relevant deliberate prac-
tice activities have been completed over a series of consecutive
studies. In this paper, however, we conducted all three of these
steps simultaneously. Although potentially a very efficient form of
empirical inquiry, this particular implementation of the expert-per-
formance approach is not without risk. Successful simultaneous
application of the three steps requires that the hypothesized critical
mechanisms (in this study, anagrammatic word-identification
skill), be correctly inferred prior to the collection of any actual
data. If our initial selection of laboratory tasks for capturing the
representative demands of the true SCRABBLE task environment
had been unsuccessful, then we would not have been able to
identify the mechanisms that mediated superior performance, and
we would have been unable to make inferences about the devel-
opmental factors that led to such performance. As research with
the expert-performance approach matures, and researchers develop
more sophisticated means of eliciting and engineering knowledge
about cognitive processing mechanisms, we believe that more
investigators working in both laboratory and field settings will be
able to save valuable time and resources by simultaneously con-
ducting the three steps. At the same time, it is clear that the first
studies in a domain will never account for all of the potentially
relevant information about the structure and acquisition of expert
performance in that domain: There will always be the need for
subsequent studies to replicate the initial analyses, refine the
measurement instruments, explore theoretically incongruent find-
ings, and develop practical applications.
More generally, our study of the structure and acquisition of
expert SCRABBLE performance is consistent with an increasing
body of knowledge that shows that the development of expert
performance cannot be simply extrapolated from the study of skill
acquisition after a few hours in the laboratory. Accounting for the
superior performance of experts in everyday life appears to require
a different approach, where performance is captured by carefully
designed representative tasks and then systematically investigated
with process-tracing techniques and experimental manipulations.
132
TUFFIASH, RORING, AND ERICSSON
Previous applications of this approach have tended to focus on
domains like chess, music, and sports, because they readily permit
the direct observation and reproduction of superior performance by
experienced individuals under controlled conditions. However, the
challenge facing contemporary expert-performance researchers is
to demonstrate how the principles and mechanisms proposed to
explain high achievement in these kinds of skill domains can be
used to explain human performance more generally. Although
some scholars have already taken important steps in this direction
(Ackerman, 2005; Ericsson, 2004), there is clearly a need for more
research synthesizing the findings from studies of elite perfor-
mance in competitive domains like SCRABBLE and chess with
the findings from field studies of skilled performance in occupa-
tional or everyday realms as well as from controlled laboratory
studies of unskilled performance by novice learners. Nevertheless,
we believe that by directly investigating the essential properties of
superior performance on domain-representative tasks, researchers
can derive generalizeable predictions about the development and
nature of human abilities that both meet the traditional standards of
scientific measurement while providing useful information about
learning in real-world contexts. Hence, studies designed in the
spirit of the expert-performance approach can satisfy both basic
and applied cognitive research goalsallowing academic scien-
tists to generate and test theoretically driven hypotheses about
fundamental cognitive processes in the laboratory, while simulta-
neously permitting applied practitioners interested in improving
real-world achievement to benefit from cognitive psychologys
powerful methodological toolbox.
References
Ackerman, P. L. (1987). Individual differences in skill learning: An inte-
gration of psychometric and information processing perspectives. Psy-
chological Bulletin, 102, 327.
Ackerman, P. L. (2000). A reappraisal of the ability determinants of
individual differences in skilled performance. Psychologische Beitrage,
42, 417.
Ackerman, P. L. (2005). Ability determinants of individual differences in
skilled performance. In R. J. Sternberg & J. E. Pretz (Eds.), Cognition
and intelligence: Identifying the mechanisms of the mind (pp. 142159).
New York: Cambridge University Press.
Andreasen, N. (1987). Creativity and mental illness: Prevalence rates in
writers and their first-degree relatives. American Journal of Psychiatry,
144, 12881292.
Atari Interactive Inc. (2001). SCRABBLE CD-ROM crossword game
(version 2.0) [Computer software]. New York: Author.
Bengtsson, S. L., Nagy, Z., Skare, S., Forsman, L., Forssberg, H., & Ullen,
F. (2005). Extensive piano practicing has regionally specific effects on
white matter development. Nature Neuroscience, 8, 11481150.
Bilalic, M., McLeod, P., & Gobet, F. (2007). Does chess need intelligence:
A study with young chess players. Intelligence, 35, 457470.
Blair, J. R., & Spreen, O. (1989). Predicting premorbid IQ: A revision of
the National Adult Reading Test. The Clinical Neuropsychologist, 3,
129136.
Borkowski, J., Benton, A., & Spreen, O. (1967). Word fluency and brain
damage. Neuropsychologia, 5, 135150.
Borzycki, M. (2001). Ageing, cognition and word play: An examination of
the potential benefits of verbal expertise in later adulthood (Doctoral
dissertation, University of New South Wales, 2001). Dissertation Ab-
stracts International, 64, 440.
Carroll, J. B. (1993). Human cognitive abilities: A survey of the factor-
analytic studies. New York: Cambridge University Press.
Charness, N. (1998). Explaining exceptional performance: Constituent
abilities and touchstone phenomena. Behavioral and Brain Sciences, 21,
410411.
Charness, N., Krampe, R. T., & Mayr, U. (1996). The role of practice and
coaching in entrepreneurial skill domains: An international comparison
of life-span chess skill acquisition. In K. A. Ericsson (Ed.), The road to
excellence: The acquisition of expert performance in the arts and sci-
ences, sports and games (pp. 5180). Mahwah, NJ: Erlbaum.
Charness, N., Tuffiash, M., Krampe, R., Reingold, E., & Vasyukova, E.
(2005). The role of deliberate practice in chess expertise. Applied Cog-
nitive Psychology, 19, 151165.
Clark, L. A., & Watson, D. (1995). Constructing validity: Basic issues in
objective scale development. Psychological Assessment, 7, 309319.
Cote, J., Ericsson, K. A., & Law, M. (2005). Tracing the development of
athletes using retrospective interview methods: A proposed interview
and validation procedure for reported information. Journal of Applied
Sport Psychology, 17, 119.
Deakin, J. M., Cote, J., & Harvey, A. S. (2006). Time budgets, diaries,
and analyses of concurrent practice activities. In K. A. Ericsson, N.
Charness, P. Feltovich, & R. R. Hoffman (Eds.), Cambridge handbook
of expertise and expert performance (pp. 303318). Cambridge, En-
gland: Cambridge University Press.
De Groot, A. D. (1978). Thought and choice in chess (2nd ed.). The Hague,
The Netherlands: Mouton. (Original work published 1946)
Djakow, I. N., Petrowski, N. W., & Rudik, P. A. (1927). Psychologie des
schachspiels [Psychology of chessplayers]. Berlin: De Gruyter.
Doll, J., & Mayr, U. (1987). Intelligenz und schachleistung: Eine unters-
chung an schachexperten [Intelligence and chess performance: An in-
vestigation of chess experts]. Psychologische Beitrage, 29, 270289.
Edley, J., & Williams, J. D., Jr. (2001). Everything SCRABBLE (2nd ed.).
New York: Pocket Books.
Ericsson, K. A. (2004). Deliberate practice and the acquisition and main-
tenance of expert performance in medicine and related domains. Aca-
demic Medicine, 79, S70S81.
Ericsson, K. A. (2006a). The influence of experience and deliberate prac-
tice on the development of superior expert performance. In K. A.
Ericsson, N. Charness, P. Feltovich, & R. R. Hoffman (Eds.), Cambridge
handbook of expertise and expert performance (pp. 685706). Cam-
bridge, England: Cambridge University Press.
Ericsson, K. A. (2006b). Protocol analysis and expert thought: Concurrent
verbalizations of thinking during experts performance on representative
task. In K. A. Ericsson, N. Charness, P. Feltovich, & R. R. Hoffman
(Eds.), Cambridge handbook of expertise and expert performance (pp.
223242). Cambridge, England: Cambridge University Press.
Ericsson, K. A., & Kintsch, W. (1995). Long-term working memory.
Psychological Review, 102, 211245.
Ericsson, K. A., Krampe, R. T., Tesch-Romer, C. (1993). The role of
deliberate practice in the acquisition of expert performance. Psycholog-
ical Review, 100, 363406.
Ericsson, K. A., & Lehmann, A. C. (1996). Expert and exceptional per-
formance: Evidence of maximal adaptation to task constraints. Annual
Review of Psychology, 47, 273305.
Ericsson, K. A., & Simon, H. A. (1993). Protocol analysis: Verbal reports
as data (Rev. ed.). Cambridge, MA: MIT Press.
Ericsson, K. A., & Smith, J. (1991). Prospects and limits of the empirical
study of expertise: An introduction. In K. A. Ericsson & J. Smith (Eds.),
Towards a general theory of expertise (pp. 138). Cambridge, England:
Cambridge University Press.
Ericsson, K. A., & Williams, A. M. (2007). Capturing naturally-occurring
superior performance in the laboratory: Translational research on expert
performance. Journal of Experimental Psychology: Applied, 13(3), 115
123.
Fatsis, S. (2002). Word freaks: Heartbreak, triumph, genius, and obsession
133
SPECIAL ISSUE: EXPERT PERFORMANCE IN SCRABBLE
in the world of competitive SCRABBLE players. New York: Penguin
Books.
Fitts, P. M., & Posner, M. I. (1967). Human performance. Belmont, CA:
Brooks/Cole.
Gottfredson, L. (1997). Why g matters: The complexity of everyday life.
Intelligence, 24, 79132.
Grabner, R. H., Neubauer, A. C., & Stern, E. (2006). Superior performance
and neural efficiency: The impact of intelligence and expertise. Brain
Research Bulletin, 69, 422439.
Halpern, D., & Wai, J. (2007). The world of competitive SCRABBLE:
Novice and expert differences in visuospatial and verbal abilities. Jour-
nal of Experimental Psychology: Applied, 13, 7995.
Hambrick, D. Z., Salthouse, T. A., & Meinz, E. J. (1999). Predictors of
crossword puzzle proficiency and moderators of age-cognition relations.
Journal of Experimental Psychology: General, 128, 131164.
Horgan, D. D., & Morgan, D. (1990). Chess expertise in children. Applied
Cognitive Psychology, 4, 109128.
Horn, J. L. (1989). Cognitive diversity: A framework of learning. In P. L.
Ackerman, R. J. Sternberg, & R. Glaser (Eds.), Learning and individual
differences: Advances in theory and research (pp. 61116). New York:
Freeman.
Hulin, C. L., Henry, R. A., & Noon, S. L. (1990). Adding a dimension:
Time as a factor in the generalizability of predictive relationships.
Psychological Bulletin, 107, 328340.
Hunter, J. E., & Hunter, R. F. (1984). Validity and utility of alternative
predictors of job performance. Psychological Bulletin, 96, 7298.
Jensen, A. R. (1998). The g factor. London: Praeger.
Keil, C. T., & Cortina, J. M. (2001). Degradation of validity over time: A
test and extension of Ackermans model. Psychological Bulletin, 127,
673697.
Kucera, H., & Francis, W. N. (1967). Computational analysis of present-
day American English. Providence, RI: Brown University Press.
Masunaga, H., & Horn, J. (2000). Characterizing mature human intelli-
gence: Expertise development. Learning and Individual Differences, 12,
533.
Nunnally, J. C. (1967). Psychometric theory. New York: McGraw-Hill.
Official SCRABBLE players dictionary (3rd ed.). (1996). Springfield, MA:
Merriam-Webster.
Schachner, R. W. (1998). The official SCRABBLE brand word-finder (Rev.
ed.). New York: Black Dog & Leventhal.
Schneider, W., Eschman, A., & Zuccolotto, A. (2002). E-prime reference
guide. Pittsburgh, PA: Psychology Software Tools.
Sheppard, B. (2002). World-championship-caliber-SCRABBLE. Artificial
Intelligence, 134, 241275.
Sloboda, J. A., Davidson, J. W., Howe, M. J. A., & Moore, D. G. (1996).
The role of practice in the development of performing musicians. British
Journal of Psychology, 87, 287309.
Spreen, O., & Strauss, E. (1991). A compendium of neuropsychological
tests: Administration, norms, and commentary. New York: Oxford Uni-
versity Press.
Starkes, J. L., & Deakin, J. (1984). Perception in sport: A cognitive
approach to skilled performance. In W. F. Straub & J. M. Williams
(Eds.), Cognitive sport psychology (pp. 115128). Lansing, NY: Sport
Science.
Terman, L. M., & Merrill, M. A. (1960). Stanford-Binet Intelligence Scale:
Manual for the third revision. Boston: Houghton Mifflin.
Unterrainer, J. M., Kaller, C. M., Halsband, U., Rahm, B. (2006). Planning
abilities and chess: A comparison of chess and non-chess players on the
Tower of London task. British Journal of Psychology, 97, 299311.
Uttl, B. (2002). North American Adult Reading Test: Age norms, reliabil-
ity, and validity. Journal of Clinical and Experimental Neuropsychol-
ogy, 24, 11231137.
van der Maas, H. L. J., & Wagenmakers, E. J. (2005). A psychometric
analysis of chess expertise. American Journal of Psychology, 118, 29
60.
Ward, P., Hodges, N. J., Williams, A. M., & Starkes, J. L. (2004).
Deliberate practice and expert performance. In A. M. Williams & N. J.
Hodges (Eds.), Skill acquisition in sport (pp. 231258). London: Rout-
ledge.
Ward, P., & Williams, A. M. (2005). Perceptual and cognitive skill
development in soccer: The multidimensional nature of expert perfor-
mance. Journal of Sport & Exercise Psychology, 25, 93111.
Waters, A. J., Gobet, F., & Leyden, G. (2002). Visuospatial abilities of
chess players. British Journal of Psychology, 93, 557565.
Wechsler, D. (1997). WAISIII: Administration and scoring manual. San
Antonio, TX: Psychological Corporation.
Wissler, C. (1901). The correlation of mental and physical tests. Psycho-
logical Review, Monograph No. 3.
Wittmann, W. W., & Su, H. M. (1999). Investigating the paths between
working memory, intelligence, knowledge, and complex problem-
solving performances via Brunswick symmetry. In P. L. Ackerman,
P. C. Kyllonen, & R. D. Roberts (Eds.), Learning and individual
differences: Process, trait, and content determinants (pp. 77108).
Washington, DC: American Psychological Association.
Received November 30, 2006
Revision received June 25, 2007
Accepted June 29, 2007
134
TUFFIASH, RORING, AND ERICSSON