Journal of Personality Assessment

ISSN: 0022-3891 (Print) 1532-7752 (Online) Journal homepage: https://www.tandfonline.com/loi/hjpa20

The Centrality of the Response Process in

Construct Validity: An Illustration via the
Rorschach Space Response

Joni L. Mihura, Nicolae Dumitrascu, Manali Roy & Gregory J. Meyer

To cite this article: Joni L. Mihura, Nicolae Dumitrascu, Manali Roy & Gregory J. Meyer
(2018) The Centrality of the Response Process in Construct Validity: An Illustration via the
Rorschach Space Response, Journal of Personality Assessment, 100:3, 233-249, DOI:
10.1080/00223891.2017.1306781

To link to this article: https://doi.org/10.1080/00223891.2017.1306781

Published online: 27 Apr 2017.

Submit your article to this journal

Article views: 391

View related articles

View Crossmark data

Citing articles: 5 View citing articles

Full Terms & Conditions of access and use can be found at

https://www.tandfonline.com/action/journalInformation?journalCode=hjpa20
JOURNAL OF PERSONALITY ASSESSMENT
2018, VOL. 100, NO. 3, 233–249
https://doi.org/10.1080/00223891.2017.1306781

ARTICLES

The Centrality of the Response Process in Construct Validity: An Illustration via the
Rorschach Space Response
Joni L. Mihura,1 Nicolae Dumitrascu,2 Manali Roy,1 and Gregory J. Meyer1
1
Department of Psychology, University of Toledo; 2The Danielsen Institute, Boston University

ABSTRACT ARTICLE HISTORY

Recently, psychologists have emphasized the response process—that is, the psychological operations and Received 5 December 2016
behaviors that lead to test scores—when designing psychological tests, interpreting their results, and Revised 18 January 2017
refining their validity. To illustrate the centrality of the response process in construct validity and test
interpretation, we provide a historical, conceptual, and empirical review of the main uses of the
background white space of the Rorschach cards, called space reversal (SR) and space integration (SI) in the
Rorschach Performance Assessment System. We show how SR and SI’s unique response processes result in
different interpretations, and that reviewing their literatures with these distinct interpretations in mind
produces the expected patterns of convergent and discriminant validity. That is, SR was uniquely related
to measures of oppositionality; SI was uniquely related to measures of cognitive complexity; and both SR
and SI were related to measures of creativity. Our review further suggests that the Comprehensive System
use of a single space code for all uses of white space likely led to its lack of meta-analytic support as a
measure of oppositionality (Mihura, Meyer, Dumitrascu, & Bombel, 2013). We close by discussing the use
of the response process to improve test interpretation, develop better measures, and advance the design
of research.

Historically, many professionals and laypersons alike have
regarded the Rorschach inkblot task (Rorschach, 1921/1942)
with skepticism (Crews, 2004; Goode, 2001; Jensen, 1965).
Why? A simple explanation could be that how the Rorschach
works is rarely discussed. The link between the psychological
operations in the response process (Schachtel, 1966) and the
interpretation of the related score is rarely explained in the con-
temporary psychology literature (Exner, 2003; Exner & Erd-
berg, 2005). Consequentially, some criticisms in the popular
press have referred to Rorschach interpretation as being like
astrology, palm reading, and tea leaf reading (Wood, Nezwor-
ski, Lilienfeld, & Garb, 2003).
Although mystery about how the Rorschach works can be an
intriguing feature of the test, and historically some clinicians
have capitalized on this potential mystery by using what Wei-
ner (2003) called the “Ouija board” approach to interpreta-
tion—the related opaqueness can lead to mistrust and doubt by
professionals and the public. Instead, a framework for concep-
tualizing how the test works should be grounded in reasonable
and documented mechanisms regarding how the mind works.
A disjuncture between a Rorschach response and the resulting
interpretation of personality features makes the test difficult to
teach and also challenging to discuss its results with clients in a
meaningful and empowering way. Although it is conceivable
that some psychologists might be motivated to keep the Ror-
schach shrouded in mystery, this opaqueness can also be prop-
agated inadvertently—simply because the direct links between
the Rorschach variable’s coding and its interpretation were
never taught to the psychologist. This omission in training
could occur for various reasons, perhaps because the link was
lost over time or it was never fully explicated in the first place.
An illustration of how the link between a variable’s coding
and its interpretation can be lost over time can be shown with
the Rorschach Comprehensive System’s (CS; Exner, 2003) Pair
score. A Pair is coded when the respondent reports seeing two
of something (e.g., “people” or “animals”), one symmetrically
on each side of the card. Pairs are included in the CS Egocen-
tricity Index along with the Reflection score, which is coded
when a person sees a reflection in the symmetrical “sides” of
the inkblot (e.g., “a person looking at their reflection in the mir-
ror”). The CS interprets Reflections as an indicator of narcis-
sism, a term derived from Narcissus in Greek mythology who
falls in love with his reflection. Psychoanalytic theory subse-
quently used the term narcissism to denote autoeroticism
(Freud, 1914). But how do Pair responses fit this interpretation
when Pairs are not a reflection, but two separate objects? For
Exner, that interpretive link did initially exist. In a 1969 article,
in which Exner explored the relationship between Rorschach
responses and narcissism, he stated, “[Pair] responses were
included in the analysis in that they represent a type of mirror
image although clearly in a much more subtle form”
(Exner, 1969a, p. 327). However, over time, Exner’s conceptual
link between the Pair response and egocentricity was lost
because it was not discussed in his CS test manuals.

CONTACT Joni L. Mihura joni.mihura@utoledo.edu Department of Psychology, University of Toledo, Mail Stop 948, Toledo, OH 43606.
Color versions of one or more of the figures in the article can be found online at www.tandfonline.com/hjpa.

© 2018 Taylor & Francis

234 MIHURA, DUMITRASCU, ROY, MEYER
In contrast, Urist’s (1977) Mutuality of Autonomy scale
(MA)—an empirically supported psychoanalytic scale
(Graceffo, Mihura, & Meyer, 2014) depicting levels of develop-
ment ranging from engulfing and overwhelming (Level 7) to
mutual and autonomous (Level 1)—makes this link more
explicit. MA Level 4, representing the reflection-mirroring
stage, is coded when “one figure is seen as the reflection, or
imprint, of another” (Urist, 1977, p. 5). Holaday and Sparks
(2001) further elaborated Urist’s coding of pairs in Level 4 by
emphasizing that the pairs must be described as indistinguish-
able, by using descriptors like “identical,” “twins,” or “exactly
the same,” which fits Kohut’s (1971) conceptualization of nar-
cissism that includes mirroring and twinship. Without this
elaboration of sameness, pairs of objects do not fit the theoreti-
cal construct of narcissism. In fact, it appears that Exner
(1969a) might have followed this more restrictive rule for cod-
ing Pair responses than in his final CS coding guidelines,
because the Pair means he reports in that article are substan-
tially lower than his nonpatient CS means.1
Another way that the link between a Rorschach variable’s
coding and its interpretation is unclear is when the original
link was “discovered” by chance or through a strict empirical
method of scale construction. The Reflection score also serves
as a good example of a CS variable for which a finding was ini-
tially discovered by chance. As Exner (1969a) stated, “Some-
what by accident, it was noted during data analysis for another
Rorschach study, that a relatively high frequency of “reflection”
responses, occurred in Rorschach protocols of overt homosex-
uals”2 (p. 325). Exner (2003) used exploratory statistical meth-
ods to develop his six CS constellation indexes, using a method
of test construction similar to that used to create the original
Minnesota Multiphasic Personality Inventory (MMPI) scales
(the “traditional scales”) that relies strictly on empirical meth-
ods to choose the items (Hathaway & McKinley, 1940). That is,
scale items were not chosen based on their content or their con-
ceptual link to a personality characteristic, but based solely on
their statistical ability to differentiate target and control sam-
ples. The empirical method of item selection starts with a large
number of items, which are administered to the two groups
(e.g., depressed vs. nonpatients) and statistically winnowed
down to the best discriminatory items.
The emergence of a new Rorschach system (Exner, 1974)
that emphasized the empirical method of variable selection and
deemphasized theory was similar to and synchronous with
major transformations of the Diagnostic and Statistical Manual
of Mental Disorders (DSM; American Psychiatric Association,
1968, 1980). The CS was introduced as a new empirically based
atheoretical Rorschach system during a time when serious
empirical challenges were made to the reliability and accuracy
of the DSM–II (American Psychiatric Association, 1968). The
first edition of the CS test manual was published in 1974
(Exner, 1974), the same year that Spitzer and Fleiss (1974)
1
In Exner (1969a), the Pair means were 1.95 and 2.61 (SDs D 1.32 and 1.56), which
are notably lower than nonpatient means using the standard CS coding guide-
lines: Exner (1974) D 7.3 (no SD given), Exner (2003) D 8.52 (SD D 2.18), and
Exner (2007) D 8.82 (SD D 3.08).
2
At the time Exner wrote this article, homosexuality was theoretically due to auto-
eroticism or narcissism (see Exner, 1969a).
published their seminal paper challenging the diagnostic reli-
ability of the DSM–II. Essentially, Exner was to the Rorschach
what Spitzer was to the DSM, in that they each helped usher in
a new atheoretical, empirically focused version of the Ror-
schach and the DSM, respectively: the Rorschach CS (Exner,
1974) and the DSM–III (American Psychiatric Association,
1980). By doing so, Exner helped bring respect for the Ror-
schach as a scientific instrument and dissociate the test from
the disparaged psychoanalytic theory.
However, as many scholars have noted, strict reliance on
empiricism in test development—referred to as “dustbowl
empiricism” (Butcher, 2000)—can come at a cost. Statistically,
a strict empirical method of item selection capitalizes on
chance. Therefore, the individual items, not simply the test
scales themselves, must be cross-validated. Failure to cross-vali-
date item selection was a criticism of the strict empirical
method of test construction used to develop the MMPI
(Helmes & Reddon, 1993) and the Rorschach CS constellation
indexes (Wood, Nezworski, & Stejskal, 1996). A strict empirical
method of item selection can result in items with low face valid-
ity—that is, it might not be readily apparent how an item is
assessing its scale’s construct (and, indeed, it might not be a
valid measure). Using the MMPI–2 (Butcher, Graham, Telle-
gen, & Kaemmer, 1989) as an example, items with low face
validity were referred to as “subtle items” and items with a clear
link to pathology were called “obvious items” (Hollrah,3
Schlottmann, Scott, & Brunetti, 1995). Due to the lack of cross-
validation of MMPI items selected for the traditional scales,
many subtle items might be on their scales simply by chance
(e.g., Weed, Ben-Porath, & Butcher, 1990).4
In sharp contrast to dustbowl empiricism, a more contem-
porary view of test development and construct validity
(Borsboom, Mellenbergh, & van Heerden, 2003, 2004) argues
that theory and an understanding of the response process that
leads to the score should be primary in developing scales. Bors-
boom et al. (2004) compellingly argued that:
The concept of validity may never have been necessary because the
instruments were generally set up on the basis of an idea of how
they would work. In that case, the question of what it is, precisely,
that is measured can simply be resolved by pointing to the processes
that lead to the measurement outcomes. (p. 1067)
Regarding our definition of the response process, as
described in Meyer, Viglione, Mihura, Erard, and Erdberg
(2011):
[The response process refers to] the factors that lead to or produce
the task behaviors captured by a particular code. They are the psy-
chological elements that are present in the process of generating a
response with a particular set of coded attributes. The relevant fac-
tors or psychological elements are embedded in the respondent’s
coded behavior and imagery, which include his or her abilities,
organizational efforts, styles of processing, feelings, ideas, motives,
and conflicts. These factors include longstanding personality
3
Joni L. Hollrah is the same person as the first author, Joni L. Mihura.
4
On the other hand, subtle items exist for which the link to the scale’s construct
becomes clear after reflection, and are therefore less likely to be error. For exam-
ple, although it might not be obvious to a layperson how answering false to “I
like to flirt” belongs on MMPI Scale 2 (Depression), with just a little reflection a
psychologist could connect this experience with the loss of libido seen in
depressed patients.

characteristics, current state-like circumstances, and reactions pres-
ent in the context of being assessed by a particular examiner at a
particular time. (p. 330)
Therefore, our use of the term response process specifically
targets the link between the psychological operations and
behaviors that occur while giving a response and the associated
interpretation as applied to the case.5 We discuss the response
process further in subsequent sections, using the experiential
knowledge gained by understanding the role of the response
process in the interpretation of how respondents might inte-
grate the background white space of the Rorschach card into
their responses.
The purpose of this article is to illustrate how validity can be
improved by conceptually realigning constructs or interpretive
inferences with the response processes that lead to measurement
outcomes. To illustrate the importance of aligning the response
process with a measure’s construct, we focus on different ways a
Rorschach respondent can use the inkblot’s background white
space to produce a response. First, we review the history of dif-
ferent codes that have been used to designate the use of white
space to show how their unique response processes differentially
align with distinct interpretations. Next, we review the empirical
literature for these codes to evaluate the convergent and discrim-
inant validity findings that would be expected based on these dis-
tinct response processes. We close with a general discussion of
how to link response processes with test interpretation to
improve interpretive accuracy, develop better measures, refine
existing ones, and advance the design of psychological research.
How the lack of empirical support for the CS Space
variable led to a sharper focus on its response process
The CS test manual (Exner, 2003) cites five studies that support
the validity of the CS Space variable. In contrast, the construct
validity meta-analyses by Mihura, Meyer, Dumitrascu, and
Bombel (2013) reviewed a more comprehensive set of studies
and did not find support for the CS Space variable as a measure
of oppositionality, either the overt behavior or the associated
emotion (anger). In fact, the effect size was essentially zero for
validity criteria assessed by self-report (r D .03, k D 11),6 and
for validity criteria whose method required an external assess-
ment of the person—for example, psychiatric diagnosis,
observer ratings, offender status (r D .01, k D 13).
Given the wide discrepancy between Exner’s (2003) and
Mihura et al.’s (2013) reviews of the CS Space validity literature,
a reasonable conclusion could be that one of the reviews is
wrong. However, the actual situation is more complicated—but
also enlightening at the same time. When Mihura et al. were
summarizing the validity research cited in the CS manual
(Exner, 2003) for their Discussion section, they realized the
validity studies Exner cited for the Space variable were pub-
lished two to three decades before the CS came into existence
(Exner, 1974). These pre-CS studies included four articles
(Bandura, 1954a; Counts & Mensh, 1950; Fonda, 1951; Rosen,
5
Exner (2003) also used the term response process, but targeting the more gen-
eral processes that occur when responding to the Rorschach task of “What might
this be?” such as scanning, identifying, discarding, and selecting responses.
6
k D number of findings.
CENTRALITY OF THE RESPONSE PROCESS IN CONSTRUCT VALIDITY 235
1952) and one book (Rapaport, Gill, & Schafer, 1946). This is
important because, in contrast to Rorschach’s original system
and other subsequent systems, the CS does not assign alterna-
tive codes to differentiate the ways in which the background
white space is used; rather, a single Space code is given any
time the respondent incorporates the background white into a
response. As will be discussed, though, Space coding differen-
tials are interpretively meaningful.
History of white space coding and the differences
across Rorschach systems
Since its inception in Rorschach’s (1921/1942) Psychodiagnos-
tik, the white space variable (S) has gone through several itera-
tions. In Psychodiagnostik, Rorschach defined his S response as
“Intermediate forms (S) [Zwischenformantworten] … in which
the white spaces are interpreted rather than the black or col-
ored parts of the figure [inkblot] which surround them”
(Rorschach, 1921/1942, p. 39, italics added). Stated differently,
“S is the symbol for interpretations in which the black parts of
the figures are neglected in favor of the white spaces they out-
line” (p. 186). In fact, Rorschach deliberately provided promi-
nent white space areas that would be available for responses on
three cards (Cards II, VII, and IX; p. 52) and indicated any par-
allel series of inkblots should be similarly designed (p. 53).
Rorschach noted that Space responses were “most common in
stubborn, eccentric normals and in negativistic, scattered schiz-
ophrenics;” he believed that “S answers always indicate some
sort of tendency to opposition” (p. 39).
Subsequently, all five early U.S. Rorschach systems included
a version of this variable (Exner, 1969b). However, no two sys-
tems coded it exactly the same. Some differentiated types of
white space use and some did not. Hertz (1970), Klopfer and
Kelley (1942), and Rapaport et al. (1946) differentiated between
white space (a) used as the location for the main part of a per-
son’s response (e.g., on Card VII, “a white lamp” to the DS7
area), and (b) used to embellish the main response (e.g., on
Card I, “a fox’s head” with the white spaces at DdS26 as “its
eyes and teeth”). Figure 1 provides a non-Rorschach inkblot
illustration. The bottom, central white space can be seen as the
form of a teddy bear with the top two pinkish-red spots as the
Figure 1. Inkblot example to illustrate two different types of Rorschach Space
responses: Figure–ground reversal Space responses and responses using the space
to embellish responses provided to the inkblot proper. (Artwork by Elena Yeager.)
236 MIHURA, DUMITRASCU, ROY, MEYER
inside of its ears and the two pinkish-red spots below as its eyes.
The figure–ground reversal that focuses on the white space to
create the teddy bear would historically be coded “Primary” or
“Main” Space. On this same inkblot, the lighter central pink-
ish-red area can be seen as a mask with the lower white spot as
its mouth. The mouth on the response would be a “Secondary”
or “Additional” Space response in the earlier Rorschach sys-
tems because the inkblot proper forms the mask and the white
space, as the mouth, is an embellishment. As discussed later,
the newer Rorschach Performance Assessment System (R–
PAS; Meyer et al., 2011) calls these codes, respectively, Space
Reversal and Space Integration, a convention we use in all sub-
sequent sections.
Beck, Beck, Levitt, and Molish (1961) and Piotrowski (1957)
did not differentiate between these two types of white space
responses, although Piotrowski did say, “However, one might
assign a value of a whole point to each of the rare pure S [i.e.,
Space Reversal], and a half point to each of the combined S [
i.e., Space Integration]” (p. 95), but this was not his convention.
To complicate things, Rorschach systems often used different
codes for essentially the same variable. For example, when a
small or rare white space area was used for the main response,
Klopfer, Ainsworth, Klopfer, and Holt (1954) used the code
“S,” Hertz (1946) used the code “s,” and Beck (1944) used the
code “Dds.” Furthermore, over time, some Rorschach systemat-
izers changed their coding for white space. For example, ini-
tially Klopfer’s system used the code s for responses provided
to small or rare white space areas (Klopfer & Sender, 1936);
but, subsequently, it coded all Space responses as S, although
continuing to distinguish between Space Reversal and Space
Integration responses in the interpretation (Klopfer et al., 1954;
Klopfer & Davidson, 1962; Klopfer & Kelley, 1942). Fonda
(1951) and Exner (1969b) provided more information about
the differences in coding white space across the early popular
U.S. Rorschach systems.
When Exner compiled the key aspects of existing Rorschach
systems into the CS (Exner, 1974), he also included a white
space variable. However, like Beck and Piotrowski, he did not
differentiate between the different types. Instead, Exner (1974)
decided that “The symbol S is included in the scoring for loca-
tion whenever a white space area of the blot is included in the
response” (p. 57). Instead of using formal codes to differentiate
uses of white space, Exner (1974) recommended that the “type
of S (common, uncommon, separate, or combined) can be
noted … in the qualitative evaluation of the protocol” (p. 57).
Yet subsequent CS manuals (e.g., Exner, 2003) omitted this rec-
ommendation. Therefore, the CS distinction between the Space
responses formed by figure–ground reversals versus other uses
of white space faded over time.
Space interpretation based on response process
The consequences of Exner’s (1974) decision to use just one
Space code for all uses of white space becomes clear when one
reviews the literature on previous Rorschach systems. Histori-
cally, many authors emphasized the link between the psycho-
logical maneuvers—or “response processes”—involved in
giving a Space Reversal response and Rorschach’s claim that “S
answers always indicate some sort of tendency to opposition”
(Rorschach, 1921/1942, p. 39). The response process link of
Space Reversal to oppositionality was highlighted by Rapaport
et al. (1946) when they said, “But how is it that space responses
seem to refer to an oppositional tendency? The first answer
appears to lie in the common-sense consideration that a subject
who interprets the white spaces is actually doing the reverse of
what the instructions imply” (p. 177, italics added). Fonda
(1977) similarly stated:
Although [Rorschach] made no reference to the concept of the
reversal of figure and ground, it is clear that such was the perceptual
process he had in mind as mainly responsible for these responses.
… A person who produces S does indeed seem to be performing in
a way contrary to instructions. Instead of interpreting the black or
colored parts of the ink blot, as Rorschach quite naturally expected,
the subject interprets a white space. (p. 114)
Fonda (1977) astutely noted that this Space Reversal
response-process interpretation is consistent with the definition
of “negativism”7 provided by Bleuler, with whom Rorschach
trained and completed his dissertation. Bleuler (1911/1950)
defined negativism and illustrated the construct of negativism
with numerous examples, which are summarized here to pro-
vide a better understanding of the intended construct:
We subsume under the term, negativism, a number of symptoms
which have the common characteristic that a reaction which would
be expected in a positive sense occurs in the negative sense instead.
The patients cannot or will not do what is expected of them (passive
negativism); or they do just the very opposite or, at least, something
else than what is expected (active or contrary negativism).
When patients should be getting up, they want to stay in bed. When
they are supposed to be in bed, they want to get up. … To “Good
day,” they say, “good-bye.” They do their work all wrong; sew but-
tons on the wrong side of the clothes. They eat their soup with a
fork and their desert with a soup spoon. They continually sit down
in someone else’s place, enter every bed but their own. They call
our children by their surnames (which they have picked up some-
how) instead of by their nicknames. A hebephrenic is asked to saw
some wood; he hauls some small boards instead, then he has to
bring small boards, which he puts on the wrong rack. He is sup-
posed to go down the staircase but resists; then suddenly takes the
whole flight in one great leap. (pp. 191–192)8
As Bleuler (1911/1950) explained regarding the etiology of
negativism in schizophrenia:
In schizophrenia, the patient’s relationship to the outside world has
been altered; it has on the whole become a hostile one. The patients
live in their autistic worlds. Thus it can be shown in many cases
that the patients consider all stimuli emanating from the outside
world, which they cannot block off, as unpleasant disturbances. As
a result of this, negativistic attitudes develop. (p. 442)
Fonda (1977) noted that Rorschach’s interpretation of the
oppositional Space Reversal response process was “unequivo-
cally pejorative.” However, Fonda described several alternative
positive interpretations by considering the potential healthy
functions of the figure–ground reversal process. These include
Freud’s (1913/1959) concept of the “general instinct of
7
Schizophrenic negativism is not the same as negative symptoms in
schizophrenia.
8
Bleuler’s (1911/1950) concept of negativism was not linked with the physical
aggression that many Rorschach authors have associated with oppositionality,
and which resulted in studies exploring the relationship between Space and
aggression, violence, and psychopathy (see Mihura et al., 2013, p. 593).

mastery,” whose aim Hendrick (1943) described as “to control
or alter a piece of the environment, an ego-alien situation, by
the skillful use of perceptual, intellectual, and motor techni-
ques” (p. 314). Fonda also invoked Angyal’s (1965) concept of
autonomy in the white space figure–ground reversal and
White’s (1959) role of competence as an aspect of autonomy.
The figure–ground reversal inherent in giving a Space
Reversal response—or “doing the reverse of what the instruc-
tions imply”—is not part of the Space Integration response pro-
cess, the main distinctiveness of which, by nature of its coding,
is that the white space is integrated with the inkblot proper.
Historically, the Space Integration variable received scant atten-
tion in the theoretical and empirical literature. The most rele-
vant information appears to be Beck’s (1933) measure of
cognitive organizational effort—the Z score—which includes
instances in which the white space is integrated with the inkblot
proper. Exner (1974, 2003) followed Beck’s convention of
including the integrated Space response into his Z score. All of
this suggests that producing a Space Integration response
requires the ability to cognitively integrate and organize one’s
perceptions.
There are distinct response processes in each type of
Space response
Existing research is consistent with the view that the response pro-
cesses involved in Space Reversal and Space Integration responses
are unique and distinct. Bandura (1954b) and Nelson (1954) both
found that the perceptual figure–ground reversal process—based
on the rate of perceptual reversals of a Necker Cube and other
reversible figures—was more characteristic of Space Reversal than
Space Integration—that is, rs D .32 versus .19 (Bandura, 1954b)
and .48 versus .06 (Nelson, 1954). These findings support the idea
that the figure–ground reversal process is more specific to the Space
Reversal than the Space Integration variable.
Similarly, the very low associations between these two differ-
ent types of Space variables suggest they are measuring unique
constructs. Holding the number of Responses constant, Ban-
dura (1954b) found that the partial correlation between Space
Reversal and Space Integration was only .08. Using R–PAS cod-
ing guidelines for these variables (Meyer et al., 2011), and
excluding responses in which both scores were present,
Dumitrascu, Mihura, Meyer, and Onofrei (2011) found that the
correlation between Space Reversal and Space Integration was
only .08. One would not expect to see such low correlations if
these two types of Space responses were targeting the same con-
struct—especially because they are both derived from a perfor-
mance-based monomethod measurement (Campbell & Fiske,
1959; Meyer, 1996, 1997).
Previous validity reviews of the Rorschach Space
variable
We identified four Rorschach white space validity reviews—a
chapter by Fonda (1977), an article by Frank (1993), the CS test
manual (Exner, 2003), and Mihura et al.’s (2013) CS validity
meta-analyses. None of these reviews differentiated between
the different types of Space responses when formulating their
conclusions. Fonda’s review initially emphasized the
CENTRALITY OF THE RESPONSE PROCESS IN CONSTRUCT VALIDITY 237
phenomenological difference between Space Reversal and Space
Integration; however, his conclusions did not. He also included
studies in which the type of Space variable was undefined.
Frank’s literature review did not differentiate between Space
Reversal and Space Integration, and he concluded that “there
does not seem to be any consistency in the findings” (p. 1113).
However, he suggested, it is “possible that S may have different
meanings … whether space is used as a main response … or is
an addendum to a main response” (p. 1113), for which he
stated “a program of research is needed” (p. 1114).
Exner’s (2003) review of the validity literature supported the
CS Space variable as a measure of oppositionality, which was in
diametric contrast with Mihura et al.’s (2013) meta-analyses
that found an association of essentially zero with relevant valid-
ity criteria (r D .01). However, as previously noted, Exner’s
review did not include any studies that used the CS coding cri-
teria for Space, which was required for Mihura et al.’s meta-
analyses. This is crucial because 87.5% of CS Space is due to
Space Integration. Only one third9 of the CS Space responses
consist of the figure–ground reversals with a response process
that fits with an interpretation of oppositionality or negativism.
Thus, it is understandable why Mihura et al.’s meta-analyses
did not support CS Space as a measure of oppositionality.
The R–PAS Space variables go back in time to capture
the response process intended by Rorschach
A revitalized understanding of the unique response processes
for Space Reversal and Space Integration provides a new lens
from which to view the validity literature. Applying a response
process-focused approach, Dumitrascu, Mihura, and Meyer
(2010) conducted a preliminary review of the early Space litera-
ture and found support for the oppositionality interpretation of
the figure–ground reversal variable. This was intriguing
because, from the beginning, Rorschach’s proposed interpreta-
tion of Space was based on this response process, which was
lost when the two types of Space responses were combined.
Dumitrascu et al. (2011) investigated the convergent and
discriminant validity of Space Integration and Space Reversal
using level of education as a proxy for cognitive complexity in
the Romanian CS normative sample CS (Dumitrascu, 2007),
which provided a substantial range of education (i.e., 4–17
years). Dumitrascu et al. recoded protocols using a prepublica-
tion version of the R–PAS Space Reversal and Space Integration
coding guidelines. The relationship with education was strong
for Space Integration (r D .48) and moderate for Space Reversal
(r D .25); however, when Space Reversal codes were limited to
responses without Space Integration, its correlation with educa-
tion dropped to .08. Therefore, the use of white space for inte-
gration or elaboration was related to education as a proxy for
cognitive complexity, whereas the figure–ground reversal use of
white space was not. Further, these two Space variables showed
very small associations with each other (.08). Thus, these initial
findings suggested that Space Reversal and Space Integration
were assessing two different constructs.
9
Space Integration responses outnumber Space Reversal responses almost 3:1
(M D 2.8 vs. 1.1; Meyer et al., 2011, p. 310).
238 MIHURA, DUMITRASCU, ROY, MEYER
The preliminary review and data in Dumitrascu et al. (2010;
Dumitrascu et al., 2011), combined with the revitalized inter-
pretation of Space responses via their conceptually based
response processes, guided decisions in the most recent
Rorschach system: R–PAS (Meyer et al., 2011). The construct
validity meta-analyses by Mihura and colleagues (2013) played
a central role in choosing which variables to include in R–PAS
(Meyer, Viglione, & Mihura, 2017; Meyer et al., 2011). That
review placed the CS Space variable in the “no validity support”
category (see Mihura et al, 2013, Table 3, p. 570), so it was not
included in R–PAS. Instead, R–PAS returned to the older litera-
ture distinguishing between the Space Reversal and Space Inte-
gration types of responses.
R–PAS named these scores Space Reversal (SR) and Space
Integration (SI) to emphasize their inherent response processes
(Meyer & Mihura, 2010; Meyer et al., 2011). SR was placed on
Page 1 of the Summary Scores and Profiles results, which is for
the variables with the best empirical support; SI was placed on
Page 2 for more tentative interpretation because it had a clear
response process interpretation but relatively little research.
The R–PAS developers created guidelines so SR and SI could
be reliably scored. The reliability reported for the R–PAS norms
was conducted using an early version of the coding guidelines,
and the disagreements that arose from that coding were used to
revise the final coding guidelines (see Meyer et al., 2011, pp.
435–439). The reliability study relied on six blinded coders,
consisting of two R–PAS developers—Gregory J. Meyer and
Joni L. Mihura—and four of their graduate students: Nicolae
Dumitrascu, Sandra Horn, Wei-Cheng Hsiao, and Joshua
Eblin. A total of 60 nonpatient adult protocols were coded
independently by five of the examiners, with the remaining
examiner coding 50 of those records. The mean pairwise intra-
class correlations (ICCs) across all six coders were .85 for SR
and .88 for SI. For the two R–PAS developers (Mihura and
Meyer), the ICCs were .89 for SR and .96 for SI.
Using a mixed nonpatient and patient sample of 50 child and
adult records and draft versions of the R–PAS coding guidelines,
two graduate students (Amy C. Blume-Marcovici and Heidi L.
Miller) trained by another R–PAS developer—Donald J.
Viglione—coded the records for all R–PAS variables (Viglione,
Blume-Marcovici, Miller, Giromini, & Meyer, 2012). Their pro-
tocol-level ICCs for SR and SI were .91 and .86, respectively.
Kivisalu, Lewey, Shaffer, and Canfield (2016) recently completed
an R–PAS interrater reliability study using student coders and a
sample of 50 adult nonpatients. Response-level ICCs for two
independent coders were .82 for SR and .85 for SI. Using
Cicchetti’s (1994) interrater reliability guidelines—where poor
agreement is < .40, fair agreement is .41 to .59, good agreement
is .60 to .74, and excellent agreement is .75 to 1.00—these scor-
ing reliability values for SR and SI are excellent.
A conceptual review of the Rorschach Space response
literature
In subsequent sections, we review the research literature using
methodology similar to that used in Dumitrascu, Mihura, and
Meyer (2013) to provide a conceptual understanding of Space
Reversal and Space Integration. We were interested in both
convergent and discriminant validity (Campbell & Fiske, 1959).
Method
We reviewed the Space literature using a systematic methodol-
ogy, which we believe is important when describing any body
of literature, but particularly the Rorschach’s, which has come
under especially harsh scrutiny over the past couple of decades
(e.g., Wood et al., 2003). Therefore, we describe our review
methodology in detail.
Defining the Rorschach Space variables and their construct
labels
Space variables
We reviewed the empirical literature for studies that differenti-
ated between white space coding based on figure–ground rever-
sal and white space used as an elaboration or embellishment of
a response to the inkblot proper. The variables that fit these cat-
egories were (a) Rorschach’s (1921/1942) Space, (b) Primary
and Secondary Space (Hertz, 1970), (c) Main and Additional
Space (Klopfer et al., 1954; Klopfer & Kelley, 1942), (d) SR and
SI (R–PAS; Meyer et al., 2011), and (e) any study that modified
its Space coding to fit either of these categories (typically based
on Fonda’s [1951] article that distinguished between these two
types of Space responses). Because the CS Space variable is
mostly comprised of SI (87.5%; see Meyer et al., 2011, p. 310),
we also were interested to see if it assessed cognitive complexity
and so included it as a proxy for SI.
Space Reversal and Space Integration constructs
To choose relevant findings, we used the formal R–PAS SR and
SI interpretations to target their core constructs (Meyer et al.,
2011). R–PAS interprets SR within the Self and Other Repre-
sentation domain, and states that it “may be prompted by crea-
tivity, individuality, oppositionality, or healthy self-assertive
strivings” (Meyer et al., 2011, p. 362). We viewed the opposi-
tionalism inherent in the SR response process as consonant
with Bleuler’s (1911/1950) view of negativism, not as consonant
with the overt aggression that characterizes oppositional defiant
disorder in the DSM–5 (American Psychiatric Association,
2013). Therefore, in contrast to Exner (2003), we concluded
that physical aggression and psychopathy do not target the
bullseye of SR’s core construct.
R–PAS locates SI within the Engagement and Cognitive Proc-
essing domain. Its interpretive postulates focus on the constructs of
“integration” and “synthesis,” also stating that SI is “indicative of
complex and flexible thinking” (Meyer et al., 2011, p. 367), to which
the Response Process section adds “and possibly creativity” (p.
333). Therefore, for SI, we focused on findings with criterion varia-
bles that targeted these cognitive constructs.
We required the criterion variables for SR and SI to be behavior-
ally based. To define “behavior,” we used the definition provided by
Baumeister, Vohs, and Funder (2007), which states:
A study qualified as having behavior if any element involved behav-
ior—that is, if the study used any manipulation, any dependent
measure, or even used behavior as the conduit for manipulating the
independent variable (e.g., taking a test and getting feedback on it).
… Self-reports of past behaviors or of hypothetical behaviors did
not count. The use of archival behavioral data (e.g., crime statistics;
donating blood) qualified. Reading about someone else’s behavior
was not counted as behavior. (p. 399)

Literature search
We conducted PsycINFO and MEDLINE database searches
using SR and SI construct keywords derived from the previ-
ously described R–PAS manual’s interpretations of these varia-
bles (Meyer et al., 2011, pp. 333, 362, 367). The final search to
update our database was conducted on November 4, 2016. We
initially used the search terms “Rorschach AND Space,” but
many relevant articles did not use the term “Space” in their title
or abstract. Therefore, we searched titles and abstracts of these
databases using the search terms “Rorschach AND (Space OR
opposition OR assertive OR individuality OR creativ)” for
SR and “Rorschach AND (cognitive OR complex OR creativ
OR integrati OR flexib OR synthe)” for SI.
We limited the search to articles published in the English lan-
guage with methodology classified as an empirical study. After
deleting duplicates, there were 102 articles to review for SR and 356
articles to review for SI, for a total of 398 articles (an overlap of 60
articles). We also reviewed articles cited in the previously described
Space reviews (Exner, 2003; Fonda, 1977; Frank, 1993) for poten-
tially relevant findings, which added 12 articles to review for SR.
For discriminant validity purposes, we included Mihura et al.’s
(2013) meta-analytic findings for the CS Space variable in relation-
ship to externally assessed criteria targeting oppositionality, which
added nine articles with 13 findings.10
We focused on the peer-reviewed journal literature with two
exceptions. First, Charek, Meyer, and Mihura (2016) provided
validity data for SI by documenting that it was lower under
conditions of mental fatigue or ego depletion. In an as yet
unpublished dissertation, Charek (2016) did not replicate these
findings. Because two of the authors (Joni L. Mihura and
Gregory J. Meyer) were on Charek’s dissertation committee
and aware of these findings, we believed it was important to
include this unsuccessful replication. Charek (2016) also
reported the relationship between SR and SI with cognitive
ability measures, so those findings are included as well. Second,
we included the unpublished results from Dumitrascu et al.
(2011) because that research was undertaken for this article.
Therefore, in total, there were 412 studies to review for conver-
gent validity findings—116 for SR and 356 for SI (with an over-
lap of 60 studies).
Selection procedure
The first author conducted the literature review, consulting with
the second author for difficult decisions. We reviewed all relevant
studies reported in three Space reviews (Exner, 2003; Fonda, 1977;
Frank, 1993). For the PsycINFO and MEDLINE search, we did not
obtain the full-text article if the title or abstract indicated that (a)
the relevant Space variable was not being used (e.g., for SR, the CS
Space response was being studied), or (b) the relevant constructs
were not being assessed (e.g., for SI, the criterion variable only tar-
geted oppositionality). The validity criterion had to clearly target
the SR and SI characteristics described in the R–PAS manual and
used in the search terms. If we could not reliably determine from
10
To be consistent with Exner’s (2003) interpretation of the Space response, Mihura
et al.’s (2013) meta-analytic criteria for CS Space included conduct disorder, psy-
chopathy, and physical aggression; however, as previously noted, this is not con-
sistent with R–PAS’s view of SR’s response process interpretation.
CENTRALITY OF THE RESPONSE PROCESS IN CONSTRUCT VALIDITY 239
the full-text source which type of Space variable was used, we
excluded the study.
Study quality
To ensure better study quality, we excluded results that selectively
reported significant findings or that contained errors for which (a)
the correct value was not computable from the article, or (b) the
study authors were not able to provide the correct data. Coding reli-
ability was a study quality criterion. We did not expect Rorschach
studies published before Weiner’s (1991) recommended minimum
level (i.e., 80% agreement) to report coding reliability. Therefore,
we only excluded findings that provided evidence of low reliability,
not studies that did not report reliability. We used the same level of
reliability for both Rorschach and criterion variables as used in
Graceffo et al. (2014) of ICC < .60, Pearson r < .70, or percent
agreement < 80% (Weiner, 1991).
Finally, we considered number of Responses (R) as a rele-
vant study quality characteristic. The fact that people can pro-
vide a variable number of responses to the 10 inkblots has been
discussed as a potential problem for decades (Cronbach, 1949;
Meyer, 1992). Although we did not generally require studies to
control for R, the Beck method of Rorschach administration is
an exception because it instructs respondents to “be sure to tell
the examiner everything [italics added] that you see on the card
as you look at it” (Beck et al., 1961, p. 2). The Beck approach
results in a large and variable number of Responses (e.g., see
Katko, Meyer, Mihura, & Bombel, 2010, p. 595). Therefore,
study findings were excluded if they used the Beck administra-
tion method and did not control for R.
Discriminant validity
Using the multitrait–multimethod approach to construct validity
(Campbell & Fiske, 1959), we evaluated both convergent validity
and discriminant validity. That is, if a study reported the associa-
tion between one type of Space response and a measure assessing
its targeted construct (e.g., SR and oppositionality), as well as the
association between the other type of Space response and the same
criterion measure (e.g., SI and oppositionality), we report each of
these findings. By comparing only within-study results, we hold
study design constant so that it is not a confounding variable.
Results
Description of sample
Our search strategy identified 23 independent studies with 25
findings that fit the convergent validity criteria and 17 findings
that fit the discriminant validity criteria.11 Of these, 10 articles
with 19 convergent validity criteria were identified in the Psy-
cINFO and MEDLINE search strategy. Two additional articles
were included from the published Space reviews (Exner, 2003;
Fonda, 1977; Frank, 1993); Stein (1973) and De Koninck and
Crabb
e-Decleve (1971) each contributed one convergent valid-
ity finding. Charek’s (2016) dissertation had three findings that
fit the convergent validity criteria and three that fit the
11
We included two articles (Bandura, 1954a; Stein, 1973) that used a Space variable
that combined SR and SI responses because the methodology purposefully
weighted the SR response heavily.
240 MIHURA, DUMITRASCU, ROY, MEYER
discriminant validity criteria. Dumitrascu et al. (2011) had one
finding that fit the convergent validity criteria and one that fit
the discriminant validity criteria. Nine studies with 13 discrimi-
nant validity findings were included from the Mihura et al.
(2013) CS meta-analyses. In total, there were 14 SR studies
with seven convergent validity findings and 13 discriminant
validity findings. There were nine SI studies with 18 convergent
validity findings and four discriminant validity findings. The
reasons for study exclusions are described in subsequent
sections.
Reasons for excluded studies
Space reversal
Ninety-nine studies were excluded from the SR PsycINFO and
MEDLINE search because they were either (a) not investigating an
SR variable (n D 70); (b) using criterion variables that we judged as
not targeting the bullseye of the SR construct, or were based on
self-report or another inkblot variable (n D 18); (c) not investigat-
ing Rorschach validity (n D 8; e.g., reliability studies); or (d) con-
tained errors (n D 1). Regarding the latter, Weltman and Wolfson
(1964) was excluded because its summary table of information was
incomplete and its mathematically nested cells could not be recon-
structed from contradictory information contained in the text. All
but one of the 18 studies whose criterion variables we judged as not
targeting the bullseye were not designed to validate the Rorschach;
instead, they used the Rorschach as a valid psychological test to
understand how certain populations differed from control groups
on various psychological characteristics (e.g., comparing controls
to women who have had abortions, people who have chronic
schizophrenia, or people who have nightmares). For the remaining
study, we disagreed with the author about the suitability of the cri-
terion as an indicator of oppositionality. Fonda (1951) considered
it oppositional to answer “?” to a questionnaire with the response
options of Yes, ?, and No. Although oppositionality could be one
reason for responding ?, there are other possible reasons—such as
pure uncertainty as to how to respond.
Regarding the 12 potential SR articles from the published
Space validity reviews (Exner, 2003; Fonda, 1977; Frank, 1993)
that were not identified in our PsycINFO and MEDLINE
search,12 six were excluded because they did not use a figure–
ground reversal coding for Space (Borgatta & Eschenbach,
1955; Counts & Mensh, 1950; Eschenbach & Borgatta, 1955;
Lord, 1950; Murray, 1957; Ray, 1963). Four were excluded
because we judged their criterion variables as not targeting the
bullseye of SR’s construct. Rosen’s (1952) criterion was psy-
chopathy and Schachtel’s (1951) was delinquency. As previ-
ously noted, we decided a priori that psychopathy and physical
aggression do not target the bullseye of SR’s response process.
Finn and Neuringer (1968) used left-handedness as a proxy for
oppositionality, and there is no solid research to support this
association. Fox and Blatt (1969) tested Rapaport, Gill, and
Schafer’s (1945) hypothesis that scoring higher on Digits Back-
wards (DB) than Digits Forward (DF) might indicate negativ-
ism and a tendency to be oppositional. We had initially
included their positive findings, although we did so cautiously
12
We report these exclusions in detail because they have been included in previ-
ous Space reviews.
due to the number of alternative reasons for why a person
might score higher on DB than DF. The ability to mentally
visualize digits in order and then reverse them might share psy-
chological operations with reversing figure and ground in ink-
blot imagery. However, this is not encompassed by our target
constructs for SR, and we eventually decided to exclude this
finding when it was questioned by a reviewer. Finally, although
we included Ingram (1954), we excluded several of his findings
because they violated our interrater reliability criterion.
Space integration
Regarding the 349 excluded articles from the SI PsycINFO and
MEDLINE search, based on the titles and abstracts, 67 were
not investigating Rorschach scale validity (e.g., reliability, the-
ory, Rorschach mentioned but not studied) and 91 did not
include a relevant criterion variable—the latter included studies
investigating sex differences, cultural differences, the relation-
ship between the Rorschach and self-report measures, various
medical conditions that are not expected to significantly affect
cognitive abilities (e.g., hypospadias in childhood, myasthenia,
Tourette’s, asthma), and other conditions not expected to differ
on SI (e.g., dissociative identity disorder vs. borderline person-
ality disorder, sex offenders vs. other offenders). The abstracts
did not indicate that any of these studies were designed to study
the Space variable. The remaining 191 articles were obtained
for review, and 187 articles did not report SI results. Of the
remaining four articles, two were not written in English. A
third article did not hypothesize differences between groups on
SI, but selectively reported results of several Rorschach varia-
bles that differentiated between their groups. The last article
was excluded due to a data-reporting error that was unable to
be corrected by the author (reporting an average proportion of
60% Space responses in a schizophrenia sample, which is
implausibly high).
Orientation to the SR and SI construct validity results
The SR and SI convergent and discriminant validity findings
are reported in Tables 1 and 2. Each table reports the study,
sample size, study population(s), criterion variable, Rorschach
system, coding reliability (if present), whether the analyses con-
trolled for R, and columns for the convergent and discriminant
validity effect sizes. As noted in the Method section, CS Space
is reported as a proxy for SI.
Effect size calculation and reporting
All findings were converted to the same metric (Pearson’s r) for
equivalent comparison across studies. When reporting SR and
SI findings in Tables 1 and 2, all effects are reported as positive
when in line with theoretical expectation and negative when in
the opposite direction. On average, we expect cross-method
associations (r) for Rorschach variables to be around .27 (see
Mihura et al., 2013). Readers can use this benchmark when
reviewing the results in Tables 1 and 2.
Main SR and SI construct validity results
Convergent validity results are in Table 1 for SR and Table 2 for
SI. Discriminant validity results are reversed and in Table 2 for
 CENTRALITY OF THE RESPONSE PROCESS IN CONSTRUCT VALIDITY 241

Table 1. Space Reversal versus Space Integration relative to criteria of oppositionality, creativity, individuality, independence, or self-assertive strivings.

SR convergent SI discriminant
Rorschach Coding Control for validity ESr validity ESr
Study n Criterion variable system reliability R? SR CS S as SI proxya

Pathological interpretation (oppositionality –the behavior or the affect)

Mihura et al. (2013) 790 Results from CS meta-analysis CS Varies Varies .01 (k D 13)
Bandura (1954a) 59 Negativism, teacher ratings of high Hertz, weighted None Yes .34
school students
Stein (1973) 40 Oppositionality, supervisor ratings Klopfer, None Yes .53
of Naval enlisted men weighted
Ingram (1954) 16 Hostility expressed: Insulting Beck None No .50b
interview
Hostility observed: Challenging (coded Primary ¡.50b
puzzle S)
(male undergraduates)
Healthy interpretation (creativity, individuality, independence, or self-assertive strivings)
De Koninck & Crabb
e- 14 Field independence vs. Klopfer None Yes .74
Decleve (1971) dependence, rod-and-frame test
(psychology undergraduates)
Bandura (1954a) 59 Assertiveness, teacher ratings of Hertz, weighted None Yes .14
high school students
Ramachandra (1994) 60 Accomplished artists (creativity) vs. Klopfer None Yes .31
normal controls

Note. Column shown in italics represents discriminant validity. SR D Space Reversal; SI D Space Integration; ESr D effect size, Pearson r; CS D Comprehensive System; k D
number of effect sizes.
a
Because the CS S score is largely composed of SI, it is used as its proxy. No studies investigated the actual SI score in relationship to oppositionality or the other criterion
variables reported in this table.
b
Effect sizes were estimated based on being significant at the .05 level.

SR and Table 1 for SI. Overall, the SR and SI construct validity
findings showed the expected convergent and discriminant
validity effect sizes. For SR, six of the seven convergent validity
findings are in the expected direction (rs D .14–.74; Table 1).
The one negative finding was from a study by Ingram (1954),
who also observed one positive finding. College students with
at least two SR responses, compared to those with no SR
responses, expressed more hostility when their career potential
was insulted during an interview; however, they expressed less
hostility when completing a Tower of Hanoi task. Discriminant
validity for SR, across four effect sizes, showed the expected
lack of association with measures of cognitive complexity
(Table 2). For SI (Table 2), all five convergent validity coeffi-
cients for the R–PAS SI variable are in the expected direction
(rs D .14–.48), and the 13 CS Space coefficients showed similar
convergent validity (rs D .11–.72). With respect to discriminant
validity for SI, as would be expected, the CS Space variable
showed no meaningful association with oppositionality criteria
across 13 effect sizes (i.e., from Mihura et al.’s [2013] meta-
analyses, shown in the first data row in Table 1).
Discussion
Rorschach scores that make use of the background white
space were used to illustrate the role of the response process
in construct validity. A historical and conceptual review
illustrates how the coding of Space became disconnected
from the original rationale for its interpretation. After
realigning the interpretation of white space use with the
response processes that led to its coding, a review of the
research literature revealed the expected patterns of conver-
gent and discriminant validity that support differential inter-
pretations of SR and SI. That is, the SR response process
aligns with the original rationale as a measure of opposition-
ality or negativism (Rorschach, 1921/1942) in that the
examinee is “doing the reverse of what the instructions
imply” (Rapaport et al., 1946, p. 177, italics added). In con-
trast, the SI response process does not align with the con-
struct of oppositionality or negativity; instead, integrating
the white background with the inkblot proper is consistent
with complex, integrative, flexible, and possibly creative
thinking.
Systematic review of the Space Reversal and Space
Integration literature
Three previous narrative reviews of the Space literature exist
(Exner, 2003; Fonda, 1977; Frank, 1993), in addition to the CS
meta-analyses by Mihura et al. (2013). Ours is unique in both
(a) differentiating between SR and SI types of white space cod-
ing, and (b) using a systematic review. The latter is important
because 18 of the 20 articles mentioned in the previous narra-
tive reviews were available for all three reviews (i.e., published
in 1971 or before), yet there is minimal article overlap. No arti-
cle is cited in all three reviews, and 12 of the 20 articles are cited
in only one review. Although all studies in these three reviews
were designed to target SR’s construct, almost half (9/20) did
not use an SR type of criterion variable. Five of the 11 remain-
ing studies, conducted half-century or more ago, used criterion
variables that we judged as not targeting the bullseye of SR’s
construct (psychopathy or delinquency) or as questionable
measures of its construct (left-handedness, responding “?” [vs.
Yes or No] to a questionnaire, DB > DF). We excluded two
other studies reported in these narrative Space reviews because
one contained significant errors and the other used self-report
criterion variables.
Space Reversal
Seven study findings fit our SR inclusion criteria. Two stud-
ies targeted the classic negativism interpretation of
242 MIHURA, DUMITRASCU, ROY, MEYER

Table 2. Space Integration versus Space Reversal relative to criteria of complex, flexible, synthetic, integrative, or creative thinking.

SI convergent validity ESr

SR discriminant
Rorschach Coding Control for CS S as validity ESr
Study n Criterion variable system reliability R? SI proxya SI SR w/o SIb

Ferracuti, Cannoni, 40 Creativity (Torrance)–Verbal CS None No

Burla, & Lazzari Fluidity .37
(1999) Flexibility .42
Originality .35
Creativity (Torrance)–Figural
Fluidity .11
Flexibility .23
Originality .15
Elaboration .33
(graduate and postgraduate students) M D .28
Schwartz & Canetti 40 Creativity (Remote Associations Test) CS ICC D .98 Yes .25
(2014) (community volunteers)
Charek, Meyer, & Mihura 97 Controls vs. ego depletion R-PAS ICC D .81 Yes .23
(2016) manipulation (undergraduates)
Charek (2016) 135 Controls vs. ego depletion R-PAS ICC D .76 No .14
manipulation ICC D .84 .00
70 Judgment of line orientation accuracy R-PAS ICC D .76 No .41
ICC D .84 ¡.11
70 Delis–Kaplan Executive Function R-PAS ICC D .76 No .28
System (D–KEFS) Design Fluency ICC D .84 ¡.05
total (undergraduates)
Lorio et al. (2010) 110 Healthy controls vs. progressive CS None No .32c
multiple sclerosis
Tegtmeyer & Gordon 38 Full-scale IQ (nonpatient children) CS None Yes .24
(1983)
Di Nuovo, Buono, 46 Full-scale IQ (psychiatric & CS None No .35
Colucci, & Pellicciotta rehabilitation inpatients)
(2004)
Dumitrascu, Mihura, 111 Range of education (Grade 4–17, R-PAS ICC D .98 No .48
Meyer, & Onofrei community volunteers ICC D .94 .08
(2011)
Smith, Bistis, Zahka, & 27 Rey–Osterrieth Complex Figure CS ICC D .92 No
Blais (2007) Delay .31d
Organization (child & adolescent .72d
outpatients)

Note. Column shown in italics represents discriminant validity. CS S D CS Space coding; SI D Space Integration; SR D Space Reversal; ESr D effect size, Pearson r; CS D
Comprehensive System; ICC D intraclass correlation; R-PAS D Rorschach Performance Assessment System.
a
Because the CS S score is largely composed of SI, it is used as its proxy.
b
Limited to SR responses on which SI was not also coded.
c
Cohen’s d for the CS space score reported in Lorio et al. (2010, Table 6) should be –.69 not –.38. We converted the correct value to r.
d
Controlling for age and IQ.

oppositionality (i.e., doing the opposite as instructed, not
the aggression inherent in oppositional defiant disorder).
Bandura (1954a) found that high school students rated high
on negativism by their teachers had significantly more SR
responses (r D .34).13 Stein (1973) found that Navy enlisted
men rated high on oppositionality by their immediate
supervisors gave more SR responses (r D .53).14 Therefore,
there are replicated findings for the relationship between SR
and the oppositionality seen in negativism (Bleuler, 1911/
1950). One study investigating the emotion related to oppo-
sitionality (hostility) found mixed results. Ingram (1954)
found that SR was positively related to observer-rated
13
As described in Bandura’s (1952) dissertation, in Bandura (1954a) “negativism”
was rated on a 5-point scale ranging from “Stubborn and resistive, refuses to do
things unless forced” at the high end to “Accepts discipline and orders without
14
hesitation, docile and compliant” (p. 81) at the low end.
For Stein (1973), oppositionality was rated on three 7-point semantic differential
scales with endpoints of cooperative–stubborn, agreeable–oppositional, and
acquiescent–contrary.
hostility expressed during an insulting interview but nega-
tively related (the opposite of the predicted direction) to
observer-rated hostility while completing a challenging
Tower of Hanoi type puzzle task. Ingram had defined these
situations as interpersonally versus intellectually frustrating,
respectively. Therefore, although this finding needs to be
replicated (Ingram had a very small n of 16), it is possible
that the interpersonal and insulting dynamic of the inter-
view pulled for a more negativistic response than the intel-
lectually challenging puzzle.
Regarding the more uniformly healthy SR interpretations
(creativity, individuality, independence, and self-assertive striv-
ings), there were three relevant findings, each targeting a differ-
ent construct. De Koninck and Crabb
e-Decleve (1971) found
that Canadian college students who scored higher on a field
independence measure (the rod-and-frame test) were more
likely to produce SR responses (r D .74). However, this effect
size is artificially inflated because the study used an extreme
groups design with a very small sample size (i.e., out of 52

tested, the seven highest and lowest scores on the rod-and-
frame test), and should be replicated. Bandura (1954a) found a
minimal relationship between SR and high school students
rated as assertive by their teachers (r D .14).15 Finally, Rama-
chandra (1994) found that accomplished Indian artists (those
receiving prestigious awards) in music, literature, painting,
sculpture, and dance produced more SR responses compared to
normal controls.
Space Integration
In almost every instance, SI showed the expected relationships
across a wide range of criterion variables that targeted complex,
flexible, synthetic, integrative, or creative thinking, whereas SR
was not related to these criteria. SI was positively related to
different measures of creativity (Ferracuti, Cannoni, Burla, &
Lazzari, 1999; Schwartz & Canetti, 2014), various neuropsycho-
logical measures (Charek, 2016; Smith, Bistis, Zahka, & Blais,
2007), IQ (Di Nuovo, Buono, Colucci, & Pellicciotta, 2004;
Tegtmeyer & Gordon, 1983), and education (Dumitrascu et al.,
2011), and it was less common for persons with progressive
multiple sclerosis compared to controls (Lorio et al., 2010).
Charek et al. (2016) found that participants exposed to an
experimental ego depletion manipulation had less complex
protocols (e.g., fewer SIs) than controls, but the methodologi-
cally more complex follow-up study (Charek, 2016) did not.
This negative finding in the SI review might be less a nonsup-
portive finding for SI and more the absence of an ephemeral
ego depletion effect (e.g., Carter, Kofler, Forster, & McCul-
lough, 2015), given that Charek (2016) documented clear valid-
ity for SI with cognitive performance criteria.
Creativity and the Space response
Interestingly, even though SR and SI were unrelated to each
other statistically, they were each related to creativity. This
might initially seem counterintuitive, but the SR and SI
response processes likely capture unique aspects of creativity.
For example, as shown in Table 1, Ferracuti et al. (1999) found
that CS Space, which is largely comprised of SI, was related to
all three types of verbal creativity (fluidity, flexibility, and origi-
nality), but to only two of the four types of figural creativity: fig-
ural flexibility and elaboration, not figural fluidity or originality.
One would expect figural originality (i.e., the ability to create
unusual, new, and different types of ideas with figures) to be
more related to SR than to SI due to SR’s perceptual figure–
ground reversal response process. Researchers should further
investigate SR and SI’s unique associations with different types
of creativity.
Using the response process to improve the accuracy of test
interpretation and empathic understanding of clients
As previously noted, the R–PAS developers emphasized the
response processes when deciding which Rorschach variables
to include in the system (Meyer et al., 2011), and formal
15
As described in Bandura’s dissertation, in Bandura (1954a) “assertiveness” was
rated on a 5-point scale ranging from “Insistent in making his opinions known,
tries to do all of the talking, tends to force his views” to “Quiet and submissive,
rarely or never asks questions or volunteers comments” (p. 81).
CENTRALITY OF THE RESPONSE PROCESS IN CONSTRUCT VALIDITY 243
interpretation strongly emphasizes the response process. The
R–PAS test manual provides two levels of interpretation: the
first at the response level (the response process for individual
coded variables, like Vista [V]), and the second at the protocol
or summary level (integrating the response process of all coded
variables in a score, like YTVC0 ).
In addition to focusing on the response process inherent in
any particular test score, the interpreter must also consider the
context, such as the inkblot, the instructions, the interpersonal
situation, and the reason for the referral. For example, seeing
numerous aggressive images on the Rorschach clearly indicates
that aggressive images are on a person’s mind. However, it is
fairly easy for people to notice and suppress these images
(Benjestorf, Viglione, Lamb, & Giromini, 2013), which might
occur when a person accused of a violent crime attempts to
appear nonaggressive or when a person with obsessive–com-
pulsive disorder has ego-dystonic intrusive violent images.
The interpreter should consider the interpersonal situation
during the Rorschach administration and how similar processes
might play out in analogous situations in everyday life. For
example, the task of having to explain to the examiner why one
perceives things the way one does on the Rorschach is like the
everyday task of recognizing how one is feeling and reacting to
experiences and being able to communicate that to another
person. A respondent who has difficulty recognizing and artic-
ulating his or her experiences to the examiner in the testing sit-
uation is displaying an alexithymic process (an inability to
describe the nuances and subtleties of one’s experiences or
emotions). Accordingly, scores on Rorschach variables that
reflect this process (e.g., FD, Blends, Form%) correspond to
measures of alexithymia (Porcelli & Mihura, 2010).
When discussing a client’s Rorschach results with them, the
clinician should stay as close as possible to the response pro-
cess, and provide the client analogous examples as to how this
process occurs in everyday life. Prioritizing the response pro-
cess in an assessment enables a clinician to experientially empa-
thize with clients through their test results compared to simply
applying a score’s label. This approach to interpretation should
resonate better with the client. Of course, the response process
occurring during a person’s test performance is not usually
entirely isomorphic to analogous behaviors in everyday life;
therefore, the interpreter should be cognizant of the similarities
and differences when applying test results to the client’s life.
This caveat applies to all methods of assessment. As noted by
Mihura and Meyer (in press), “On the WAIS–IV Block Design
subtest, clinicians do not interpret the person’s ability to put
together blocks in everyday life; they generalize the psychologi-
cal operations that occur in the process of generating a
response—e.g., visual analysis and synthesis in the case of Block
Design—to similar operations in everyday life.”
To illustrate the differential interpretations of SR—including
pathological versus nonpathological oppositionality—we pro-
vide brief excerpts from three cases. We use a form of configu-
rational (Bram & Peebles, 2014) and structural (Weiner, 2003)
analysis, which considers the nature of the Space response and
accompanying protocol features. We focus on the following
major themes to help with the differential interpretations of SR
responses. One key consideration with “doing the opposite of
the instructions” in the SR figure–ground reversal is whether
244 MIHURA, DUMITRASCU, ROY, MEYER
the intended target of the act is interpersonal. Using a response
process perspective, interpersonal oppositionality can be ascer-
tained from a Rorschach administration in two main ways: (a)
when SR responses are accompanied with human content or
human movement, and (b) the respondents’ interactions with
the examiner are oppositional. Pathological oppositionality
would be more likely if the responses contained a high degree
of hostility and aggression and the respondent was hostile,
aggressive, or uncooperative with the examiner. Pathological
oppositionality would be less likely if the respondent was very
cooperative, and expressed pleasure and interest in the Ror-
schach task. Finally, the degree to which movement scores that
accompany the SR responses are passive or active could provide
information as to how the oppositionality will be expressed.
Subsequently, we provide three cases to illustrate how the
response process varied in these key ways in a manner consis-
tent with analogous situations observed in other parts of the
assessment process or occurring outside the assessment.
SR example Case #1: Max
Kaakinen, Muzio, and S€a€av€al€a (in press) presented a violence
risk assessment case conducted in Finland with Max, a 22-year-
old visual communications student, due to vague threats he
had made at school. Max has a history of two documented
assaults (against a previous girlfriend who was breaking up
with him, and a man who insulted him). School officials
expressed fear of Max both as a potential aggressor toward
others and himself. Throughout the assessment, Max was con-
descending and verbally aggressive. He easily lost his temper
when asked questions that implied criticism (e.g., about his pre-
vious assaults). Max’s only assessment question was “I would
really like to know why every time I try to express my artistic
needs someone is calling the police!”
Max produced an extremely high number of SR responses
(7, SS D 141) as well as an extremely high number of responses
(R D 38, SS D 143). SR was still elevated after adjusting for pro-
tocol Complexity (SS D 141). Max’s behaviors during the Ror-
schach administration were hostile, oppositional, and
uncooperative. For example, his high number of Rorschach
responses can only be accomplished with R–PAS by “violating”
the instructions to give “two … maybe three responses” per
card and by not complying with the repeated reminders of the
instructions. Max expressed irritation to the examiner each
time she reminded him. Eventually she stopped giving
reminders due to his irritation with her. Max’s SR responses
were “a roguish face” (formed largely by only the white spaces
on Card I), “a screaming face with a fearful, panic-like look,” “a
torso of a body wearing a suit,” “a semi-surprised smiling face,”
“an Aztec priest,” “a shock wave in an explosion,” and “a wom-
an’s hipbone.” Hence, six of Max’s seven SR responses contain
human contents, none of which are whole human representa-
tions, and five that were coded for Poor Human Representa-
tions. Max reported an extremely high number of aggressive
images (AGC D 7, SS D 127). He had an equal number of pas-
sive and active movement scores with his SR responses. How-
ever, his two active movement responses were forceful and
aggressive: the “screaming face” and the “explosion,” which
Max elaborated as “a really powerful explosion that can be seen
at a distance of hundreds of kilometers.” Therefore, the
response processes accompanying Max’s protocol suggest an
overt pathological oppositionality interpretation of his elevated
SR, which is consistent with his behavior observed outside the
Rorschach task.
SR example Case #2: Tala
Tala is a highly intelligent 24-year-old woman who presented
for therapy with the first author after several failed therapies.
Tala had been diagnosed with avoidant personality disorder,
and she dropped out of a visual arts school due to depression.
In the intake, Tala had several piercings—including a nose
ring, lip ring, and numerous ear piercings—and her dress and
attitudes were notably avant-garde. Tala had moved back in
with her parents, rarely left her bedroom, creatively painted
over her bedroom windows, and had not answered long-term
friends’ phone calls or emails for over 2 years (about which she
felt guilty). Tala summed up her motivation for therapy as “I
want to have a life.”
Tala gave a high number of SR responses (6, SS D 141), as
well as a very high number of Rorschach responses (R D 37,
SS D 138). Like with Max, her SR was still elevated after adjust-
ing for Complexity (SS D 141). However, unlike Max, Tala was
cooperative during the Rorschach administration, after initially
expressing concerns to her therapist about revealing her
responses with someone she did not know. Also different than
Max’s Rorschach administration, the examiner was using an
early version of R–PAS R-Optimized administration, which did
not consistently build in the reminder for “two … maybe three
responses.” The examiner did, however, ask for the card back
each time Tala gave four responses. Therefore, Tala might have
begun to think that four responses per card was one of the
rules.
Tala’s SR responses were “a wedding dress,” “a praying per-
son wearing a robe,” “people with their backs together looking
at papers,” “a cup in a museum from Ancient Greece,” “a tiny
person on the horizon with snow on the ground,” and “the
white face of a goat with a Fu Manchu mustache.” Hence, three
of her six SR responses were whole humans, two of which had
Good Human and one Poor Human Representation codes (the
latter for low perceptual accuracy, FQ-). Tala had an average
number of aggressive images (AGC D 3, SS D 100). Three of
her SR responses were coded for movement and all were pas-
sive. Therefore, the response processes accompanying the SRs
on Tala’s Rorschach protocol are notably different from Max’s.
They do suggest that her oppositionality is interpersonally
directed, but her protocol and her test behaviors are not hostile
or aggressive, and she has evidence of healthy whole human
representations.
These findings helped the therapist formulate and test the
hypothesis that an underlying passive resistance might be driv-
ing Tala’s current state of living at home yet rarely coming out
of her room as a compromise formation to individuate from
her family. Her avant-garde presentation, attitudes, and as her
therapist later learned, her artistic style, were likely also an
expression of oppositionality. Although not as pathological as
Max’s case, this oppositional style served as a compromise to
simultaneously be different and get attention, yet keep most
people somewhat distant.

SR example Case #3: Jill
Kamphuis, de Saeger, and Mihura (in press) presented a case
conducted in the Netherlands of Jill, a 15-year-old female with
posttraumatic stress due to chronic bullying at school and asso-
ciated persistent depression. During the time she was being bul-
lied, she was sexually assaulted by a same-age uncle. Jill had
stopped attending school and was taking online classes. She
was experiencing nightmares related to the bullying, engaging
in self-harm, and, during one period of extreme stress, heard
voices and believed she could communicate with plants. Jill’s
parents described her as quiet and serious. She was not making
progress in a psychotherapy using eye movement desensitiza-
tion and reprocessing, and was referred for an assessment.
Jill produced three SR responses, which is elevated for her
age (SS D 142). Jill’s SR responses were “a UFO in the night
sky,” “the night sky … because the white spots look like stars,”
and “a lamp with a shade.” Hence, none of her SR responses
contained human representations or movement scores. She was
very cooperative with the Rorschach administration, during
which she stated that she liked to draw and sketch. She told the
examiner that she enjoyed taking the Rorschach because it gave
her the opportunity to “express [her] creativity.” Therefore, the
expression of her figure–ground response process in everyday
life might be less likely to be overtly interpersonal. Her SR
responses, instead, hinted at a rather lonely and distant experi-
ence. Jill also produced a very elevated number of aggressive
images (AGC D 8, SS D 134). However, as noted in the R–PAS
manual (see Meyer et al., 2011, p. 345), respondents’ history
and their attitude toward the aggressive images is important to
interpretation. In Jill’s case, we know she has a history of being
aggressed against. Most of her AGC contents were relatively
benign (e.g., “African rhinoceros,” “pointy things on the head
of a dinosaur,” “a dragon—I rather like dragons, it looks like
how I might draw a dragon,” “a witch with super large eyes,”
“an arrow,” “two wolves”). Otherwise, Jill saw a “demon cat,”
to which she spontaneously expressed fear (i.e., “pretty scary
actually”). Her most viscerally poignant response was “A bro-
ken zombie … it looks like the skin is completely pulled down,
the skin is torn,” which seems to be a representation of how Jill
probably feels in the world of her adolescent peers. In fact, in
the discussion portion of Jill’s assessment, she resonated with
this image as a reflection of her emotional state.
Therefore, the response processes in Jill’s Rorschach proto-
col suggested that her SR responses did not indicate interper-
sonal oppositionality. Instead, it is as if Jill is using this
psychological figure–ground reversal process to help herself
disconnect from a traumatizing human world, similar to Bleu-
ler’s (1911/1950) explanation of schizophrenic negativism as an
attitude towards an altered outside world that has become hos-
tile. In response, Jill is dissociated and talking with plants. Her
SR responses suggest that she is imaginatively seeking light in
the night sky of her lived experience. One recommendation
that evolved from the R–PAS interpretation was that her treat-
ment more directly address her current social situation and her
interpersonal skills in the context of a supportive interpersonal
alliance. Jill and her family were offered a mentalization-based
treatment or dialectical behavior treatment (DBT), and they
opted for DBT. On last accounts, Jill was actively participating
in the DBT program and her emotional distress had improved.
CENTRALITY OF THE RESPONSE PROCESS IN CONSTRUCT VALIDITY 245
Align SI interpretation with its response process
Our main SI interpretive caveat is that the nature of the
respondent’s SI responses fit the interpretation. Through a
response process lens, the typical face response to Card I might
not be the best SI example of “complex, flexible, synthetic, inte-
grative, or creative thinking,” because the white spaces that
comprise the facial features (DdS26) are critical bits that help
form the percept rather than an elaboration to the main
response, such as using the Card I white spaces as designs on a
butterfly or moth. In contrast, less obvious uses of white space
as facial features can indicate creative and flexible thinking,
such as using DdS32 on Card VIII (inverted Whole) as the eyes
on a mask. Other SIs that fit a more complex psychological
operation occur when the inkblot proper is used to embellish
or elaborate on a response in which SR constitutes the central
object(s)—for example, the white spaces in Card I seen as “four
ghosts dancing” and the inkblot incorporated as “… in the
darkness,” or the middle white space on Card II (DS5) seen as a
white spaceship flying through the night sky (D6).
Conclusion
Clinicians should closely attend to the response process with all
assessment methods when making interpretations. For other
Rorschach examples of using the response process in interpre-
tation, see the cases discussed in Using the Rorschach Perfor-
mance Assessment System (R–PAS; Mihura & Meyer, in press).
For examples beyond the Rorschach (e.g., self-report instru-
ments, clinical interviews, informant-ratings, narrative
approaches like the Thematic Apperception Test, intelligence
tests, implicit association tests), see Mihura and Graceffo
(2014).
Using the response process to develop better measures
and refine existing ones
There are important ways that using the response process to
conceptualize psychological test variables can result in better
test validity research designs. As a general logical outcome of
our major thesis—when choosing validity criteria for test valid-
ity studies, researchers should map the response process of the
test variable onto the criterion variable, considering the ways in
which they are similar and different. Returning to the Block
Design response process example, note how its response pro-
cess components are similar and dissimilar to the R–PAS Syn-
thesis response. For example, their response process similarities
include the ability to analyze and synthesize abstract visual
information. Yet, Block Design is dissimilar from Synthesis in
that it requires physical manipulation of the stimuli. The Syn-
thesis response also seems to be the result of language-medi-
ated, top-down mental processes, compared to the more
bottom-up, visual-spatial mental processes activated in Block
Design.
As another dissimilarity between Synthesis and Block
Design, the Rorschach is a “typical performance test” (what a
person will typically do when left to their own predilections)
compared to IQ tests, which are “maximal performance tests”
(what a person can do when asked to perform optimally to
clear guidelines; Cronbach, 1960). R–PAS instructions do not
ask the respondent to give responses that draw relationships
246 MIHURA, DUMITRASCU, ROY, MEYER
between the blot components (i.e., Synthesis). Therefore, a
research design could use an experimental manipulation to
bring the demand situations of the two tests closer together to
test the components of their respective response processes. For
example, if respondents were provided maximal performance
instructions on the Rorschach specific to the Synthesis response
process—for example, “Rather than reporting just one thing,
your goal is to report two or more things in some relationship
to each other”—respondents’ Synthesis scores might be more
strongly related to their Block Design scores than with the tra-
ditional Rorschach instruction of “What might this be?”
Bornstein has written several articles and conducted experi-
ments that capitalize on the response process by using experi-
mental manipulations with a framework he refers to as the
“process dissociation approach” (e.g., Bornstein, 2002, 2011,
2012). The experimental manipulation in the previous example
with WAIS Block Design and the R–PAS Synthesis score uses a
process somewhat like his process dissociation approach. Born-
stein described a three-step process to conducting process dis-
sociation research: (a) determine the score’s underlying
processes and the context variables that alter these processes,
(b) turn these process-altering variables into experimental
manipulations, and (c) interpret the outcome in the context of
these processes as well as any limiting conditions.
As described in our introduction, Borsboom and colleagues
(2003, 2004) espoused a view of construct validity similar to
Bornstein’s (2002, 2011), although they did not articulate the
validation steps as clearly. Conceptually, Borsboom et al.
argued that theory and an understanding of the response pro-
cess that leads to the score should be primary in developing test
scales, and that this requires experimental manipulation to
determine cause and effect. An illustration of applying the
approach Borsboom espoused can be seen in the studies by
Charek (Charek, 2016; Charek et al., 2016)—cited in Table 1
(this article)—that used experimental manipulations of ego
depletion prior to administering the Rorschach and cognitive
tests compared to the results of a control group. Test variables
assessing cognitive complexity, such as SI, were hypothesized
to be less common in the experimental group (in which
induced depletion should reduce cognitive complexity) com-
pared to controls.
In the past decade, there has been a rise in studies targeting
the neurological processes underlying Rorschach scores. For
example, findings from an eye-tracking study by Dauphin and
Greene (2012) are particularly relevant to the Space response.
That is, although Rorschach designed Card II with a “promi-
nent intermediate figure” (DS5) and Card IX with a “definite
intermediate figure” (DS8), he intentionally designed Card VII
to primarily target the figure–ground reversal process. About
Card VII, Rorschach (1921/1942) stated, “The essential part is
the white intermediate figure, a rather obvious oil lamp, rather
than the black figures. This Plate presents the converse of Plate
V, in that normals rarely see the lamp while schizophrenics fre-
quently do” (p. 52, italics added). Rorschach’s expectation that
schizophrenics are more likely to report SRs to Card VII is con-
sistent with, and possibly driven, by Blueler’s (1911/1950) view
of schizophrenic negativism. Using a nonpatient sample, Dau-
phin and Greene (2012) found that Card VII was the card with
the shortest initial saccade latency. Respondents’ initial Card
VII focus was on the middle white space; yet, their focus very
quickly shifted to the inkblot proper, suggesting that the white
space is not perceived as the goal of the “What might this be?”
instruction even though it is the most prominent component.
In the past few years, several studies have shown an associa-
tion between the Human Movement response (M) and mirror
neuron activity in the brain (Ando et al., 2015; Giromini,
Porcelli, Viglione, Parolin, & Pineda, 2010). Mirror neurons
fire when primates act, and when they observe the same action
by others. The functions of the mirror neuron system have
been implicated in the ability to mentalize and empathize, con-
sistent with M’s interpretation. Several other recent brain-
focused studies have supported Rorschach variables’ interpreta-
tions and are consistent with the presumed underlying
response processes. For example, using fMRI, Jimura, Konishi,
Asari, and Miyashita (2009) found an association between Ach-
romatic Color (C0 ) and medial frontal lobe activity when
receiving negative feedback, consistent with the interpretation
of C0 as internally focused negative emotion. Asari and col-
leagues found evidence that unusual perceptions (as in FQu)
are associated with brain activity that suggests personal emo-
tional processing is involved—that is, with the amygdala modu-
lating the connections between the temporopolar regions and
the anterior prefrontal region and the bilateral occipitotempo-
ral regions (Asari et al., 2008, 2010a, 2010b). This is only a sam-
pling of the existing Rorschach-focused brain-based research,
which can and should be incorporated into our understanding
of the response processes that underlie Rorschach variables.
We strongly encourage psychological test researchers to
familiarize themselves with test validation frameworks that tar-
get the response process. A framework from which to start is
provided by the expert consensus classification process used in
the intelligence and achievement test literatures (Flanagan,
Ortiz, Alfonso, & Mascolo, 2002, 2006; McGrew & Flanagan,
1998). Applying this model to the response process compo-
nents of other psychological tests would require assessment
experts with the relevant specialties (e.g., psychosis, stress reac-
tions, personality disorders) and assessment methods (e.g., self-
report, inkblot, storytelling tests) to provide judgments as to
what response process components lead to the test score, how
these components map onto the targeted psychological con-
struct, and identify the limiting conditions.
Conclusions
Historically, the psychological test validity literature has almost
exclusively focused on the associations between different test
variables rather than an understanding of the psychological
operations inherent in the respondent’s response process. How-
ever, in the past decade or so, there has been an emerging inter-
est in creating, validating, and interpreting psychological tests
through an understanding of their response processes (Born-
stein, 2002, 2011; Borsboom et al., 2003, 2004; Meyer et al.,
2016; Meyer et al., 2011). To illustrate the importance of the
response process in test interpretation, we provided a historical,
conceptual, and empirical review of the different types of Space
responses to show how focusing on their unique response pro-
cesses leads to a clearer and more accurate understanding of
their psychological constructs. We encourage clinicians and

researchers to apply the response process conceptual frame-
work to interpreting psychological test results, designing new
measures, and refining existing ones. It is vital that psycholo-
gists more clearly and systematically understand the response
processes inherent in different assessment methods.
Disclosure
Joni L. Mihura and Gregory J. Meyer receive royalties from a Rorschach
test manual (Meyer, Viglione, Mihura, Erard, & Erdberg, 2011) and associ-
ated products.
References
References marked with an asterisk indicate studies included in the sys-
tematic review results.
American Psychiatric Association. (1968). Diagnostic and statistical man-
ual of mental disorders (2nd ed.). Washington, DC: Author.
American Psychiatric Association. (1980). Diagnostic and statistical man-
ual of mental disorders (3rd ed.). Washington, DC: Author.
American Psychiatric Association (2013). Diagnostic and statistical man-
ual of mental disorders (5th ed.). Washington, DC: Author.
Ando, A., Salatino, A., Giromini, L., Ricci, R., Pignolo, C., Cristofanelli, S.,
… Zennaro, A. (2015). Embodied simulation and ambiguous stimuli:
The role of the mirror neuron system. Brain Research, 1629, 135–142.
https://doi.org/10.1016/j.brainres.2015.10.025
Angyal, A. (1965). Neurosis and treatment: A holistic theory. New York,
NY: Wiley.
Asari, T., Konishi, S., Jimura, K., Chikazoe, J., Nakamura, N., & Miyashita,
Y. (2008). Right temporopolar activation associated with unique per-
ception. Neuroimage, 41, 145. doi:10.1016/j.neuroimage.2008.01.059
Asari, T., Konishi, S., Jimura, K., Chikazoe, J., Nakamura, N., & Miyashita,
Y. (2010a). Amygdalar enlargement associated with unique perception.
Cortex, 46, 94–99. doi:10.1016/j.cortex.2008.08.001
Asari, T., Konishi, S., Jimura, K., Chikazoe, J., Nakamura, N., & Miyashita,
Y. (2010b). Amygdalar modulation of frontotemporal connectivity dur-
ing the inkblot test. Psychiatry Research: Neuroimaging, 182, 103–110.
doi:10.1016/j.pscychresns.2010.01.002
Bandura, A. (1952). A study of some of the psychological processes associ-
ated with the Rorschach white space response (Unpublished doctoral
dissertation). University of Iowa, Iowa City.

Bandura, A. (1954a). The Rorschach white space response and “opposi-
tional” behavior. Journal of Consulting Psychology, 18, 17–21.
doi:10.1037/h0056911
Bandura, A. (1954b). The Rorschach white space response and perceptual
reversal. Journal of Experimental Psychology, 48, 113–118. doi:10.1037/
h0063608
Baumeister, R. F., Vohs, K. D., & Funder, D. C. (2007). Psychology as the
science of self-reports and finger movements: Whatever happened to
actual behavior? Perspectives on Psychological Science, 2, 396–403.
doi:10.1111/j.1745-6916.2007.00051.x
Beck, S. J. (1933). Configurational tendencies in Rorschach responses. The
American Journal of Psychology, 45, 433–443. doi:10.2307/1415041
Beck, S. J. (1944). Rorschach’s test: Vol. I. Basic processes. New York, NY:
Grune & Stratton.
Beck, S. J., Beck, A. G., Levitt, E. E., & Molish, H. B. (1961). Rorschach’s
test: Vol. I. Basic processes (3rd ed.). New York, NY: Grune & Stratton.
Benjestorf, S. T., Viglione, D. J., Lamb, J. D., & Giromini, L. (2013). Sup-
pression of aggressive Rorschach responses among violent offenders
and nonoffenders. Journal of Interpersonal Violence, 28, 2981–3003.
doi:10.1177/0886260513488688
Bleuler, E. (1950). Dementia praecox or the group of schizophrenias. New York,
NY: International Universities Press. (Original work published 1911)
Borgatta, E. F., & Eschenbach, A. E. (1955). Factor analysis of Rorschach
variables and behavioral observation. Psychological Reports, 1, 129–136.
doi:10.2466/PR0.1.129-136
CENTRALITY OF THE RESPONSE PROCESS IN CONSTRUCT VALIDITY 247
Bornstein, R. F. (2002). A process dissociation approach to objective-pro-
jective test score interrelationships. Journal of Personality Assessment,
78, 47–68. doi:10.1207/S15327752JPA7801_04
Bornstein, R. F. (2011). Toward a process-focused model of test score
validity: Improving psychological assessment in science and practice.
Psychological Assessment, 23, 532–544. doi:10.1037/a0022402
Bornstein, R. F. (2012). Rorschach score validation as a model for 21st-
century personality assessment. Journal of Personality Assessment, 94,
26–38. doi:10.1080/00223891.2011.627961
Borsboom, D., Mellenbergh, G. J., & van Heerden, J. (2003). The theoreti-
cal status of latent variables. Psychological Review, 110, 203–219.
doi:10.1037/0033-295X.110.2.203
Borsboom, D., Mellenbergh, G. J., & van Heerden, J. (2004). The concept
of validity. Psychological Review, 111, 1061–1071. doi:10.1037/0033-
295X.111.4.1061
Bram, A. D., & Peebles, M. J. (2014). Psychological testing that matters: Cre-
ating a road map for effective treatment. Washington, DC: American
Psychological Association.
Butcher, J. N. (2000). Starke Rosecrans Hathaway: Biography of an empiri-
cist. In G. A. Kimble & M. Wertheimer (Eds.). Portraits of pioneers in
psychology (Vol. IV, pp. 245–261). Washington, DC: American Psycho-
logical Association.
Butcher, J. N., Graham, J. R., Tellegen, A., & Kaemmer, B. (1989). Manual
for the restandardized Minnesota Multiphasic Personality Inventory:
MMPI–2. Minneapolis: University of Minnesota Press.
Campbell, D. T., & Fiske, D. W. (1959). Convergent and discriminant vali-
dation by the multitrait–multimethod matrix. Psychological Bulletin,
56, 81–105. https://doi.org/10.1037/h0046016
Carter, E. C., Kofler, L. M., Forster, D. E., & McCullough, M. E. (2015). A
series of meta-analytic tests of the depletion effect: Self-control does
not seem to rely on a limited resource. Journal of Experimental Psychol-
ogy: General, 144, 796–815. doi:10.1037/xge0000083

Charek, D. B. (2016). Differentiating maximal and typical performance meas-
ures: The impact of ego depletion on measures of maximal and typical cogni-
tion (Unpublished doctoral dissertation). University of Toledo, Toledo, OH.

Charek, D. B., Meyer, G. M., & Mihura, J. L. (2016). The impact of an ego
depletion manipulation on performance-based and self-report assess-
ment measures. Assessment, 23, 637–649. doi:10.1177/
1073191115586580
Cicchetti, D. V. (1994). Guidelines, criteria, and rules of thumb for
evaluating normed and standardized assessment instruments in
psychology. Psychological Assessment, 6, 284–290. doi:10.1037/
1040-3590.6.4.284
Counts, R. M., & Mensh, I. N. (1950). Personality characteristics in hyp-
notically-induced hostility. Journal of Clinical Psychology, 6, 325–330.
doi:10.1002/1097-4679(195010)6:4<325::AID-JCLP2270060404>3.0.
CO;2-G
Crews, F. C. (2004). Out, damned blot! The New York Review of Books, 51
(12), 22.
Cronbach, L. J. (1949). Statistical methods applied to Rorschach scores: A
review. Psychological Bulletin, 46, 393–429. doi:10.1037/h0059467
Cronbach, L. J. (1960). Essentials of psychological testing (2nd ed.). New
York, NY: Harper & Row.
Dauphin, B., & Greene, H. H. (2012). Here’s looking at you: Eye movement
exploration of Rorschach images. Rorschachiana, 33, 3–22. doi:10.1027/
1192-5604/a000025

De Koninck, J. M., & Crabb
e-Decleve, G. (1971). Field dependence and
Rorschach white-space figure–ground reversal responses. Perceptual
and Motor Skills, 33, 1191–1194.

Di Nuovo, S. F., Buono, S., Colucci, G., & Pellicciotta, A. (2004). Psychopathol-
ogy and mental retardation: A study using the Rorschach inkblot test. Psy-
chological Reports, 94, 1313–1321. doi:10.2466/PR0.94.3.1313-1321
Dumitrascu, N. (2007). Rorschach Comprehensive System data for a sam-
ple of 111 adult nonpatients from Romania. Journal of Personality
Assessment, 89(Suppl. 1), S142–S148. doi:10.1080/00223890701583648
Dumitrascu, N., Mihura, J. L., & Meyer, G. J. (2010, March). Scoring guide-
lines and norms for Space Reversal (SR) and Space Integration (SI) Ror-
schach responses. Paper presented at the annual meeting of the Society
for Personality Assessment, San Jose, CA.
248 MIHURA, DUMITRASCU, ROY, MEYER
Dumitrascu, N., Mihura, J. L., & Meyer, G. J. (2013, March). Space Reversal
and Space Integration: A validity review. Paper presented at the annual
meeting of the Society for Personality Assessment, San Diego, CA.

Dumitrascu, N., Mihura, J. L., Meyer, G. J., & Onofrei, C. (2011, March).
The relationship between the education level (as a proxy for complexity)
and selected R–PAS variables. Paper presented at the annual meeting of
the Society for Personality Assessment, Boston, MA.
Eschenbach, A. E., & Borgatta, E. F. (1955). Testing behavior hypotheses
with the Rorschach: An exploration in validation. Journal of Consulting
Psychology, 19, 267–273. doi:10.1037/h0047818
Exner, J. E. (1969a). Rorschach responses as an index of narcissism. Jour-
nal of Projective Techniques & Personality Assessment, 33, 324–330.
doi:10.1080/0091651X.1969.10380156
Exner, J. E. (1969b). The Rorschach systems. New York, NY: Grune &
Stratton.
Exner, J. E. (1974). The Rorschach: A comprehensive system. New York, NY:
Wiley.
Exner, J. E. (2003). The Rorschach: A comprehensive system (4th ed.). New
York, NY: Wiley.
Exner, J. E., Jr. (2007). A new U.S. adult nonpatient sample. Journal of Per-
sonality Assessment, 89, S154–S158. https://doi.org/10.1080/
00223890701583523
Exner, J. E., & Erdberg, P. (2005). The Rorschach: A comprehensive system.
Vol. 2: Advanced interpretation (3rd ed.). Hoboken, NJ: Wiley.

Ferracuti, S., Cannoni, E., Burla, F., & Lazzari, R. (1999). Correlations
for the Rorschach with the Torrance tests of creative thinking. Per-
ceptual and Motor Skills, 89, 863–870. doi:10.2466/PMS.89.7.863-
870
Finn, J. A., & Neuringer, C. (1968). Left-handedness: A study of its relation
to opposition. Journal of Projective Techniques & Personality Assess-
ment, 32, 49–52. doi:10.1080/0091651X.1968.10120446
Flanagan, D. P., Ortiz, S. O., Alfonso, V. C., & Mascolo, J. T. (2002). The
achievement test desk reference: Comprehensive assessment of learning
disabilities. Boston, MA: Allyn & Bacon.
Flanagan, D. P., Ortiz, S. O., Alfonso, V. C., & Mascolo, J. T. (2006). The
achievement test desk reference: A guide to learning disability identifica-
tion (2nd ed.). Hoboken, NJ: Wiley.
Fonda, C. P. (1951). The nature and meaning of the Rorschach white space
response. The Journal of Abnormal and Social Psychology, 46, 367–377.
doi:10.1037/h0058752
Fonda, C. P. (1977). The white space response. In M. A. Rickers-Ovsian-
kina,(Ed.), Rorschach psychology (pp. 113–156). Huntington, NY:
Krieger.

Fox, E., & Blatt, S. J. (1969). An attempt to test assumptions about some
indications of negativism on psychological tests. Journal of Consulting
and Clinical Psychology, 33, 365–366.
Frank, G. (1993). On the validity of Rorschach’s hypotheses: The relation-
ship of space responses (S) to oppositionalism. Psychological Reports,
72, 1111–1114. doi:10.2466/pr0.1993.72.3c.1111
Freud, S. (1914). On narcissism: An introduction. Standard Edition, 14,
73–102.
Freud, S. (1959). The predisposition to obsessional neurosis. In Collected
papers (Vol. 2, pp. 122–132). New York, NY: Basic Books. (Original
work published 1913)
Giromini, L., Porcelli, P., Viglione, D. J., Parolin, L., & Pineda, J. A. (2010).
The feeling of movement: EEG evidence for mirroring activity during
the observations of static, ambiguous stimuli in the Rorschach cards.
Biological Psychology, 85, 233–241. doi:10.1016/j.biopsycho.2010.07.008
Goode, E. (2001, February 20). What’s in an inkblot? Some say, not much.
The New York Times, p. F1.
Graceffo, R. A., Mihura, J. L., & Meyer, G. J. (2014). A meta-analysis of an
implicit measure of personality functioning: The Mutuality of Auton-
omy Scale. Journal of Personality Assessment, 96, 581–595. doi:10.1080/
00223891.2014.919299
Hathaway, S. R., & McKinley, J. C. (1940). A multiphasic personality
schedule (Minnesota): I. Construction of the schedule. The Journal of
Psychology: Interdisciplinary and Applied, 10, 249–254. doi:10.1080/
00223980.1940.9917000
Helmes, E., & Reddon, J. R. (1993). A perspective on developments in
assessing psychopathology: A critical review of the MMPI and MMPI–
2. Psychological Bulletin, 113, 453–471. doi:10.1037/0033-
2909.113.3.453
Hendrick, I. (1943). Work and the pleasure principle. Psychoanalytic Quar-
terly, 12, 311–329.
Hertz, M. R. (1946). Frequency tables to be used in scoring the Rorschach
inkblot test (3rd ed.). Cleveland, OH: Department of Psychology, Case
Western Reserve University.
Hertz, M. R. (1970). Frequency tables for scoring Rorschach responses: Code
charts, normal and rare Details, FC and F- responses, Popular responses,
Original responses. Cleveland, OH: Press of Case Western Reserve
University.
Holaday, M., & Sparks, C. L. (2001). Revised guidelines for Urist’s Mutual-
ity of Autonomy Scale (MOA). Assessment, 8, 145–154. doi:10.1177/
107319110100800203
Hollrah, J. L., Schlottmann, R. S., Scott, A. B., & Brunetti, D. G. (1995).
Validity of the MMPI subtle items. Journal of Personality Assessment,
65, 278–299. doi:10.1207/s15327752jpa6502_5

Ingram, W. (1954). Prediction of aggression from the Rorschach. Journal
of Consulting Psychology, 18, 23–28. doi:10.1037/h0055820
Jensen, A. R. (1965). Review of the Rorschach Inkblot Test. In O. K. Buros
(Ed.), The sixth mental measurements yearbook (pp. 501–509). High-
land Park, NJ: Gryphon.
Jimura, K., Konishi, S., Asari, T., & Miyashita, Y. (2009). Involvement of
medial prefrontal cortex in emotion during feedback presentation.
Neuroreport: For Rapid Communication of Neuroscience Research, 20,
886–890. doi:10.1097/WNR.0b013e32832c5f4d
Kaakinen, S., Muzio, E., & S€a€av€al€a, H. (in press). Using R–PAS in violence
risk assessment: A case study. In J. L. Mihura & G. J. Meyer (Eds.),
Applications of the Rorschach Performance Assessment System (R–PAS).
New York, NY: Guilford Press.
Kamphuis, J., de Saeger, H., & Mihura, J. L. (in press). The broken zombie:
Using R–PAS in the assessment of a bullied adolescent with borderline
personality features. In J. L. Mihura & G. J. Meyer (Eds.). Applications
of the Rorschach Performance Assessment System (R–PAS). New York,
NY: Guilford Press.
Katko, N. J., Meyer, G. J., Mihura, J. L., & Bombel, G. (2010). A principal
components analysis of Rorschach aggression and hostility variables.
Journal of Personality Assessment, 92, 594–598. doi:10.1080/
00223891.2010.513309
Kivisalu, T. M., Lewey, J. H., Shaffer, T. W., & Canfield, M. L. (2016). An investi-
gation of interrater reliability for the Rorschach Performance Assessment
System (R–PAS) in a nonpatient U.S. sample. Journal of Personality Assess-
ment, 98, 382–390. doi:10.1080/00223891.2015.1118380
Klopfer, B., Ainsworth, M. D., Klopfer, W. G., & Holt, R. R. (1954). Devel-
opments in the Rorschach technique (Vol. 1). New York, NY: Harcourt,
Brace & World.
Klopfer, B., & Davidson, H. H. (1962). The Rorschach technique. New
York, NY: Harcourt, Brace & World.
Klopfer, B., & Kelley, D. M. (1942). The Rorschach technique. Oxford, UK:
World Book.
Klopfer, B., & Sender, S. (1936). A system of refined scoring symbols.
Rorschach Research Exchange, 1, 19–22. doi:10.1080/
08934037.1936.10381485
Kohut, H. (1971). The analysis of the self: A systematic approach to the psy-
choanalytic treatment of narcissistic personality disorders. New York,
NY: International Universities Press.
Lord, E. (1950). Experimentally induced variations in Rorschach perfor-
mance. Psychological Monographs: General and Applied, 64, 1–34.
doi:10.1037/h0093628

Lorio, R., Moressa, G., Meneghello, F., Salcuni, S., Stabile, M. R., Zennaro,
A., … Tonin, P. (2010). Personality functioning in patients with a pro-
gressive course of multiple sclerosis. Psychological Reports, 107, 629–
646. doi:10.2466/02.09.15.19.PR0.107.5.629-646
McGrew, K. S., & Flanagan, D. P. (1998). The intelligence test desk reference
(ITDR): Gf-Gc cross-battery assessment. Needham Heights, MA: Allyn
& Bacon.
Meyer, G. J. (1992). Response frequency problems in the Rorschach:
Clinical and research implications with suggestions for the future.
Journal of Personality Assessment, 58, 231–244. doi:10.1207/
s15327752jpa5802_2

Meyer, G. J. (1996). The Rorschach and MMPI: Toward a more scientifi-
cally differentiated understanding of cross-method assessment. Journal
of Personality Assessment, 67, 558–578. doi:10.1207/
s15327752jpa6703_11
Meyer, G. J. (1997). On the integration of personality assessment methods:
The Rorschach and MMPI. Journal of Personality Assessment, 68, 297–
330. doi:10.1207/s15327752jpa6802_5
Meyer, G. J., Huprich, S. K., Bornstein, R. F., Mihura, J. L., Smith, J. D.,
Weiner, I. B., & Blais, M. A. (2016). From screening to integrative multi-
method assessment: Implications for training and practice. Manuscript
submitted for publication.
Meyer, G. J., & Mihura, J. L. (2010, June). Rorschach Performance Assess-
ment System (R–PAS): Evidentiary foundations and clinical applica-
tions. Workshop conducted at Charles University, Prague, Czech
Republic.
Meyer, G. J., Viglione, D. J., & Mihura, J. L. (2017). Psychometric founda-
tions of the Rorschach Performance Assessment System (R–PAS). In
R. Erard & B. Evans (Eds.), The Rorschach in multimethod forensic
practice (pp. 23–91). New York, NY: Routledge.
Meyer, G. J., Viglione, D. J., Mihura, J. L., Erard, R. E., & Erdberg, P.
(2011). Rorschach Performance Assessment System: Administration,
coding, interpretation, and technical manual. Toledo, OH: Authors.
Mihura, J. L., & Graceffo, R. A. (2014). Multimethod assessment and
treatment planning. In C. J. Hopwood & R. F. Bornstein (Eds.),
Multimethod clinical assessment (pp. 285–318). New York, NY:
Guilford Press.
Mihura, J. L., & Meyer, G. J. (Eds.). (in press). Using the Rorschach Performance
Assessment System (R–PAS). New York, NY: Guilford Press.

Mihura, J. L., Meyer, G. J., Dumitrascu, N., & Bombel, G. (2013). The
validity of individual Rorschach variables: Systematic reviews and
meta-analyses of the Comprehensive System. Psychological Bulletin,
139, 548–605. doi:10.1037/a0029406
Murray, D. C. (1957). An investigation of the Rorschach white space
response in an extratensive experience balance as a measure of out-
wardly directed opposition. Journal of Projective Techniques, 21, 40–46.
doi:10.1080/08853126.1957.10380745
Nelson, W. D. (1954). An evaluation of the white space response on the Ror-
schach as figure–ground reversal and intellectual opposition (Unpub-
lished doctoral dissertation). Michigan State College of Agriculture and
Applied Science, East Lansing.
Piotrowski, Z. A. (1957). Perceptanalysis: A fundamentally reworked,
expanded, and systematized Rorschach method. New York, NY:
Macmillan.
Porcelli, P., & Mihura, J. L. (2010). Assessment of alexithymia with the
Rorschach Comprehensive System: The Rorschach Alexithymia Scale
(RAS). Journal of Personality Assessment, 92, 128–136. doi:10.1080/
00223890903508146

Ramachandra, S. (1994). Rorschach and the creative artists. Journal of
Projective Psychology & Mental Health, 1, 39–50.
Rapaport, D., Gill, M., & Schafer, R. (1945). Diagnostic psychological test-
ing: The theory, statistical evaluation, and diagnostic application of a
battery of tests (Vol. 1). Chicago, IL: Year Book.
Rapaport, D., Gill, M., & Schafer, R. (1946). Diagnostic psychological test-
ing: The theory, statistical evaluation, and diagnostic application of a
battery of tests (Vol. 2). Chicago, IL: Year Book.
CENTRALITY OF THE RESPONSE PROCESS IN CONSTRUCT VALIDITY 249
Ray, J. B. (1963). The meaning of Rorschach white space responses. Journal
of Projective Techniques & Personality Assessment, 27, 315–323.
doi:10.1080/0091651X.1963.10120054
Rorschach, H. (1942). Psychodiagnostik [Psychodiagnostics]. Bern, Swit-
zerland: Bircher. (Original work published 1921)
Rosen, E. (1952). MMPI and Rorschach correlates of the Rorschach white
space response. Journal of Clinical Psychology, 8, 283–288. doi:10.1002/
1097-4679(195207)8:3<283::AID-JCLP2270080311>3.0.CO;2-W
Schachtel, E. G. (1951). Notes on Rorschach tests of 500 juvenile
delinquents and a control group of 500 non-delinquent adolescents.
Journal of Projective Techniques, 15, 144–172. doi:10.1080/
08853126.1951.10380370
Schachtel, E. G. (1966). Experiential foundations of Rorschach test. New
York, NY: Basic Books.

Schwartz, H., & Canetti, L. (2014). Creativity and emotional distress on
the Rorschach test. Rorschachiana, 35, 23–41. doi:10.1027/1192-5604/
a000043

Smith, S. R., Bistis, K., Zahka, N. E., & Blais, M. A. (2007). Perceptual-
organizational characteristics of the Rorschach task. The Clinical Neu-
ropsychologist, 21, 789–799. doi:10.1080/13854040600800995
Spitzer, R. L., & Fleiss, J. L. (1974). A re-analysis of the reliability of psychi-
atric diagnosis. British Journal of Psychiatry, 125, 341–347. doi:10.1192/
bjp.125.4.341

Stein, M. L. (1973). An empirical validation of the relation between Ror-
schach white-space and oppositionality. Perceptual and Motor Skills,
37, 375–381. doi:10.2466/pms.1973.37.2.375

Tegtmeyer, P. F., & Gordon, M. (1983). Interpretation of white space
responses in children’s Rorschach protocols. Perceptual and Motor
Skills, 57, 611–616. doi:10.2466/pms.1983.57.2.611
Urist, J. (1977). The Rorschach Test and the assessment of object relations.
Journal of Personality Assessment, 41, 3–9. doi:10.1207/
s15327752jpa4101_1
Viglione, D. J., Blume-Marcovici, A. C., Miller, H. L., Giromini, L., &
Meyer, G. (2012). An inter-rater reliability study for the Rorschach Per-
formance Assessment System. Journal of Personality Assessment, 94,
607–612. doi:10.1080/00223891.2012.684118
Weed, N. C., Ben-Porath, Y. S., & Butcher, J. N. (1990). Failure of Wiener
and Harmon Minnesota Multiphasic Personality Inventory (MMPI)
subtle scales as personality descriptors and as validity indicators. Psy-
chological Assessment: A Journal of Consulting and Clinical Psychology,
2, 281–285. doi:10.1037/1040-3590.2.3.281
Weiner, I. B. (1991). Editor’s note: Interscorer agreement in Rorschach
research. Journal of Personality Assessment, 56, 1.
Weiner, I. B. (2003). Principles of Rorschach interpretation (2nd ed.). Mah-
wah, NJ: Erlbaum.
Weltman, R., & Wolfson, W. (1964). Rorschach S: Oppositional tendencies
or mastery strivings. Perceptual and Motor Skills, 18, 821–824.
White, R. W. (1959). Motivation reconsidered: The concept of compe-
tence. Psychological Review, 61, 297–333.
Wood, J. M., Nezworski, M. T., Lilienfeld, S. O., & Garb, H. N. (2003).
What’s wrong with the Rorschach? Science confronts the controversial
inkblot test. New York, NY: Jossey-Bass.
Wood, J. M., Nezworski, M. T., & Stejskal, W. J. (1996). The comprehen-
sive system for the Rorschach: A critical examination. Psychological Sci-
ence, 7, 3–10. doi:10.1111/j.1467-9280.1996.tb00658.x