Complex Information Processing

Paper #106
June 26. 1967

CHAPTER VII
THE ANALYSIS OP PROBLEM-SOLVING PROTOCOLS

As we turn from the examination of the structure of task

environments to the psychological aspects of human behavior in
those environments, we are confronted with novel problems of data
\

analysis. In this chapter we will discuss a method of analysing

thinking-aloud protocols of human subjects. Since such protocols
prove to be among the most informative data that can be brought
to bear in testing theories of human thinking, it is important
that we have objective and systematic ways of handling them. To
make the discussion concrete, we will use as an example of such
data the protocol of a subject doing the cryptarithmetic task that
was introduced in Chapter IV.
Testing In format ion-Process ing Theories
Building and testing psychological theories in information--
processing terms calls for somewhat different approaches than are
typical today in experimental psychology. First, a great deal of
effort must go, and has gone., into tool building developing a
technology for constructing IPS*s having desired properties. As
we have already seen, much of this tool building has been directed
toward psychological theorizing. The tools themselves often take
the form of computer programming languages.
A second kind of research activity may be called "sufficiency
analysis. 11 It is aimed at constructing programs that can perform
difficult cognitive tasks of the kinds people can perform, with-
CXP #106 2

out detailed concern for performing them in the same way, but
with concern that no gross abilities are assumed that lie beyond
known human abilities. This style of analysis represents a first
stage of transition to building theories that explain or predict
specific human performances. Those among the problem-solving
programs already described that do not call for large amounts of
search would fit this category of sufficiency analysis$ The Logic
Theorist, perhaps; and the NS8 program and MATER among the chess-
playing programs. The Greenblatt chess program, on the other hand,
though a strong player, sometimes examines thousands of positions,
hence would have to be put on the artificial intelligence side of
the line.
ft third kind of information-processing research is directed
at constructing and testing generalized models, which do not pre-
dict the behavior of individual human beings, but, at a more gen-
eral level, the behavior of populations. This line of inquiry
parallels traditional research in experimental pyschology more
closely than the other lines, for this is the kind of generalized
prediction to which psychology has usually aspired, and to which
most experimental procedures in the field have been adapted. In
some later chapters of this book, we shall have occasion to dis-
cuss briefly some generalized models £PAM, a theory of human
verbal learning is an example -that have been tested by comparing
their behavior with the average behavior of samples of human sub-
jects.
GIF #106 3

The fourth, and final, approach has been to construct IPS 9 3

to explain in detail a segment of a single individual's behavior.
Theories of this kind can be subjected to severe testing by com-
paring the predicted behavior with the moment-to-moment record of
the subject's behavior.
The Paradigm for protocol ftnalys.is
The use of thinking-aloud protocols as data in psychology is
not at all new. Their connection with the introspective method,
especially of the Wurzburgers ( ), goes back to the first decades
of this century. They served Duncker well in his classic contri-
but ion to the psychology of problem so3,ving in 1935 ( ). They
formed the primary material in the forties for an intensive study
of thinking in chess by the Dutch psychologist De Groot (recently
revised and translated into English ( ) )» However, free verbal
report fell into relative disuse within the mainstream of bebavior-
ist psychology, especially in the United States. Andnot until
the advent of the computer« with the corresponding conceptual dev-
elopment in programming, has it been possible to couple protocols
with precise models of process.,
Spelling out a little more the over-all strategy of this
approach, we cen divide the analysis into four stages;
1. The subject is given a problem to solva. He is instructed
to say aloud whatever occurs to him while he is ^forking on
the problem.
2. The tape recording of his verbal behavior (along with a
record of non-verbal behaviors, such as expressions that
CIP #106 4

were written down) becomes the raw record of the ex-

periment. This is the protocol* The protocol includes
the experimenter's behavior also, although his partici-
pation is usually minimized intentionally.
3. After intensive analysis of the protocol (hours for
every minute of subject behavior), a proposal emerges
for a scheme of information processing that will simu-
late the subject's behavior. This proposal may* for
example, take the form of a flow diagram.
4. A computer program is coded and debugged that outputs a
record, called the trace, which purports to correspond
to the behavior indicated in the protocol.
The process may sometimes be abridged in certain ways. In
particular, if the flow diagram is not too complicated, it may be
used directly, without actually coding a computer program and
producing the trace, to predict the trace or important features
of it by hind simulation. Most of the detailed comparisons of
theory with behavior in this book will take this abridged form.
In the case of the General Problem Solver, several versions, as
we have seen, have been run on computers, but in examining the
similarities and differences between these and human protocols,
we will also frequently have recourse to hand-simulated varia-
tions.
Th@ paradigm just presented has three salient features.
First, it deals with the dynamics of an individual episode of
behavior that re precise and highly specific. Third, it deals
with the content of the tasko* Involvement with content is also
CIP #106

More precisely, a simple and completely specified interpreter

is sufficient to translate the statement of the theory into ade-
quate behavior.

reflected in the use of freely-produced linguistic utterances 33

the primary source of data. In this respect the protocol is a
natural data form for this type of theory. It is appropriate,
also, in providing a large amount of information about the subject
per unit of time. The necessity for this becomes apparent on con-
sidering how to identify a system as complex as a problem-solving
human.
Th& major problems in protocol analysis arise from these same
dominant features. We will mention two prominent problems before
turning to a third that is the main concern of this chapter.
The Assessment of Fit
In assessing the adequacy of the program to describe or ex-
plain the subject's behavior, several things are missing to which
psychologists have become accustomed. First, there is no accept-
able way to quantify the degree of correspondence between the trace
of the program and the protocol. This is not a problem of making
the inference definite or public Trace and protocol can be laid
side by side, as is done in Figure 1, where a fragment of a sub-
ject's behavior is compared with a portion of a trace produced
one version of GPS. However* an elaborate output statement and
free linguistic utterance must still be compared.
CIP #106

PI0URE X
CIP #106 7

Numerical measures are not easily adequate to such compari-

sons. In chess* for example* both trace and protocol identify
not only moves made or considered, but also features of the board
that are noticed and evaluated* decisions points in the analysis,
and so on. Although a human can assess each instance qualitati-
vely* there are no techniques available for quantifying the com-
parison, or summarizing the results of a large set of compari-
sons.* This difficulty stems directly from the richness of the
*
A close analogy is the problem of interpreting microphotographs
of biological specimens. Historically* biologists have always
been satisfied with more rough-and-ready methods of confronting
theory with data than have bean psychologists. We will refrain
from speculation as to whether this has had anything to do with
the slower rate of advance of psychology.

theory the kinds of things it can assert about human behavior.

Still* it makes the apparatus of statistical testing largely in-
applicable »
In addition f the program has been created partly with the
subject's protocol in view. Writing a program that will repro-
duce known data is not a trivial task; it amounts to finding a
simple pattern hidden in a set of inter-connected time series.
In one sense* it is analogous to fitting curves to data* where a
ce t.ain number of free parameters are available to be estimated
from the data themselves. But programs are not parameterized in
any simple way, and no analytic framework exists for counting de~
CIP #106 8

grees of freedom, and thus assessing quantitatively how much flexi-

bility is available.
What is the Theory?
Programs are symbolic structures that specify the behavior
through time of a system in the same manner that a set of diff-
erence equations does ( Thus, one has the same justification
).
for calling a program a theory as for giving that name to a set
of difference or differential equations (e.g.* Newton's laws of
motion)o In both cases program and equations--the theory ex-
plains the behavior of the system by giving a rule that specifies
how it is changing at any given moment as a function of its curr-
ent state at that moment.
Still a certain discomfort is felt in the idea of programs
as theories (Reitman, chapter 2). The discomfort stems partly
from the specificity of the theory, which seems to be limited
to explaining the behavior of a single person during a single
episode. While in other areas, such as the earth's geological
development, we are content to construct a theory of the history
of an individual system, clearly little scientific interest atta-
ches per se to the history of a particular college student in our
laboratory on a particular day. Thus, we must view these indi-
vidualized theories microtheories would be an appropriate term*

Even though this term is currently used in a somewhat broader

sense to denote a theory applying to a miniature domain of be-
havior e.g. , a theory of rat behavior in the T-maze.

simply as a technical means for bringing a more general theory

CIP #106 9

into contact with its data in an objective and systematic way.

The more general theory, of course* is neither so formalized
nor precise as the microtheories to which it gives rise. In part
it includes the basic possibility of viewing a human in a precise
way as an information processing system. But it also includes a
theory of how problem solving is accomplished what mechanisms are
common to all humans; what methods are possible and under what
conditions they are evoked? and so on* A program like GPS des-
cribed in the last chapter is such a general theory* insofar as
it incorporates such common mechanisms, contains a set of methods,
and evokes them under particular conditions. But we would expect
that GPS would have to undergo some modification in order to fit
the behavioral data from any particular subject, for the program
certainly makes no explicit provision for individual differences.
As our experience grows with this kind of theorizing* we will
gradually learn how to "parameterize" programs to admit individual
differences* and how to distinguish the "essential" structure of
the program--*i.e. * the theory-- from details (including boundary
conditions) that reflect such individual differences. At present,
we are able to do this only in an informal, commonsensical way.
Of a similar nature is the problem of differentiating those
parts of the program that have psychological import hence, that
are to be regarded as parts of the theory from those that are
only included to enable the program to run on a digital computer.
It is not only that not everything in the trace can be expected
to appear in the verbal protocol for a subject may fail to ver-
balize things of which he is conscious* and must fail to verbalise
r GIF #106 10

those that occur in his processing but of which he is unconscious.

There is the further consideration that as one moves from gross
behavior to the details of machine code, both the organization of
the program and its instructions reflect increasingly the struc-
ture of existing programming languages (and ultimately, of com-
puters) , instead of the structure of the central nervous system
and its processes. As theory, the program must be taken seriously
only down to the level of pyachologically meaningful elementary
information processes; and at the present time it is not known
with precision just where this level is, and where the boundary
should be drawn.
Such problems have confronted sciences before, notably in
fields like chemistry and genetics where entities (e.g., planetary
electrons, genes) were postulated and assigned properties on in-
direct evidence before there was any way of detecting and examin-
ing them by relatively direct methods. Under such conditions,
scientists have sometimes been guilty of the fallacy of misplaced
concreteness for example, of inventing an ether so that electro-
magnetic waves would have something to ruffIs but there does not
appear to be any simple way, whose costs are not greater than the
gain, for avoiding such errors. We must simply maintain an atti-
tude of appropriate tentativeness toward the entities the thec/y
postulates, our scepticism increasing as we move downward irco
successive layers of detail, but decreasing over time to the ex-
tent that the detail leads to successful predictions of behavior.
CIP #106 11

The Procedure to be Followed

The point of view toward data analysis taken in this book
reflects these considerations. The fit of an individual program
to a single segment of human behavior will be treated, so to speak,
as a single data point. That is, the program that accomplishes
the fit is viewed as a completely particularizec"! version (with
specific parameter values and particular boundary conditions) of
a more general theory that has not yet been formulated precisely.
The common structure of this more general theory and the nature
of the individual differences are to be inducted by cumulating
the number of pieces of data treated in this way.
The present chapter carries out this ki.Md of analysis for a
single protocol of a subject doing the cryptarithmetic task. Other
protocols, for this and other tasks, will be analysed in later
chapters. Our initial emphasis will be upon the extraction of
information from the protocol. The final step, writing an indi-
vidualized computer program (see step 4 in the previous section) c
will not be taken, but most of the main features that would have
to be incorporated in the program will become evident from the
analysis.
We start with an analysis of the task, introducing the
technical apparatus needed to describe the subject's behavior.
Then we give a gross description of the protocol, followed by
the detailed analysis.
CIP #106 129

is by no means a trivial task to use Whatever degrees of freedom are available

to construct a program — any program — that will come even close to fitting
the data in detail.