Critical Thinking Assessment

Author(s): Robert H. Ennis

Source: Theory into Practice, Vol. 32, No. 3, Teaching for Higher Order Thinking (Summer,
1993), pp. 179-186
Published by: Taylor & Francis, Ltd.
Stable URL: http://www.jstor.org/stable/1476699
Accessed: 18-05-2015 22:26 UTC

REFERENCES
Linked references are available on JSTOR for this article:
http://www.jstor.org/stable/1476699?seq=1&cid=pdf-reference#references_tab_contents

You may need to log in to JSTOR to access the linked references.

Your use of the JSTOR archive indicates your acceptance of the Terms & Conditions of Use, available at http://www.jstor.org/page/
info/about/policies/terms.jsp

JSTOR is a not-for-profit service that helps scholars, researchers, and students discover, use, and build upon a wide range of content
in a trusted digital archive. We use information technology and tools to increase productivity and facilitate new forms of scholarship.
For more information about JSTOR, please contact support@jstor.org.

Taylor & Francis, Ltd. is collaborating with JSTOR to digitize, preserve and extend access to Theory into Practice.

http://www.jstor.org

This content downloaded from 50.155.233.239 on Mon, 18 May 2015 22:26:22 UTC
All use subject to JSTOR Terms and Conditions
Robert H. Ennis
Critical Thinking Assessment
LTHOUGHcritical thinking has often been urged
as a goal of educationthroughoutmost of this
century (for example, John Dewey's How We Think,
1910; and the EducationalPolicies Commission's The
Central Purpose of American Education, 1961), not a
great deal has been done about it. Since the early
1980s, however, attentionto critical thinkinginstruc-
tion has increased significantly-with some spillover
to critical thinking assessment, an area that has been
neglectedeven morethancriticalthinkinginstruction.
Partly as a result of this neglect, the picture I
paint of critical thinking assessment is not all rosy,
though there are some bright spots. More explicitly,
my major theme is that, given our current state of
knowledge, critical thinking assessment, albeit diffi-
cult to do well, is possible. Two subthemes are that
(a) the difficulties and possibilities vary with the pur-
pose of the criticalthinkingassessmentand the format
used, and (b) thereare numeroustrapsfor the unwary.
In pursuitof these themes, I consider some pos-
sible purposes in attemptingto assess critical think-
ing, note some traps, list and comment on available
critical thinking tests (none of which suit all of the
purposes), and finish with suggestions for how to de-
velop your own critical thinking assessment, includ-
ing a discussion of some major formats. But first,
some attentionmust be paid to the definition of criti-
cal thinking,becausecriticalthinkingassessmentrequires
thatwe be clearaboutwhat we are tryingto assess.
Robert H. Ennis is professor of education at the Univer-
sity of Illinois at Urbana-Champaign.
THEORYINTO PRACTICE,Volume 32, Number 3, Summer 1993
Copyright 1993 College of Education, The Ohio State University
This content downloaded from 50.155.233.239 on Mon, 18 May 2015 22:26:22 UTC
All use subject to JSTOR Terms and Conditions
Defining Critical Thinking
The upperthree levels of Blooms' taxonomy of
educationalobjectives (analysis, synthesis, and eval-
uation) are often offered as a definition of critical
thinking.Sometimes the next two levels (comprehen-
sion and application)are added. This conception is a
good beginning, but it has problems. One is that the
levels are not really hierarchical,as suggested by the
theory, but rather are interdependent.For example,
although synthesis and evaluation generally do re-
quire analysis, analysis generally requires synthesis
and evaluation (Ennis, 1981).
More significantly, given our concern here, the
three (or five) concepts are too vague to guide us in
developing and judging critical thinking assessment.
Consider analysis, for example. What do you assess
when you test for ability to analyze? The difficulty
becomes apparentwhen we consider the following
varietyof thingsthatcan be labeled"analysis":analysis
of the political situationin the Middle East, analysis
of a chemical substance,analysis of a word, analysis
of an argument,and analysis of the opponent'sweak-
nesses in a basketball game. What testable thing do
all these activities have in common?None, except for
the vague principle that it is often desirable to break
things into parts.
A definition of critical thinking that I at one
time endorsed is that critical thinking is the correct
assessing of statements (Ennis, 1962). If I had not
elaborated this definition, it would be as vague as
Bloom's taxonomy.But even whenelaborated,it suffers
THEORYINTO PRACTICE/ Summer 1993
Teachingfor Higher Order Thinking
from excluding creative aspects of critical thinking,
such as conceiving of alternatives, formulatinghy-
potheses and definitions, and developing plans for
experiments.I now think the contemporaryconcep-
tion of critical thinking includes these things, so the
"correct assessing" definition is more narrow than
standardusage, and thus could interfere with com-
municationamong proponentsof critical thinking.
The following definition seems to be more in
accord with contemporaryusage and thus, I hope,
will minimize confusion in communication:"Critical
thinking is reasonablereflective thinking focused on
deciding what to believe or do." As it stands,howev-
er, this definition is also as vague as Bloom's taxon-
omy. It too needs elaboration.Here is an abridgment
of the elaborationsI have providedand defendedelse-
where (Ennis, 1987, 1991, in press):
In reasonably and reflectively going about de-
ciding what to believe or do, a person characteristi-
cally needs to do most of these things (and do them
interdependently):
1. Judgethe credibilityof sources.
2. Identifyconclusions,reasons,and assumptions.
3. Judgethe qualityof an argument,includingthe ac-
ceptabilityof its reasons,assumptions,andevidence.
4. Develop and defenda positionon an issue.
5. Ask appropriate clarifyingquestions.
6. Planexperimentsandjudge experimentaldesigns.
7. Define termsin a way appropriate for the context.
8. Be open-minded.
9. Try to be well informed.
10. Drawconclusionswhenwarranted, butwithcaution.
This interdependentlist of abilities and disposi-
tions can provide some specificity for guiding criti-
cal thinking testing. The elaborations,of which the
list is an abridgment,are more thorough,but the sim-
plicity of this list can make it useful. It can serve as a
set of goals for an entire critical thinkingcurriculum
or as a partialset of goals for some subject-matteror
other instructionalsequence. It can be the basis for a
table of specificationsfor constructinga criticalthink-
ing test. (A table of specifications provides the areas
that a test is supposed to assess and indicates the
weighting assigned to each.)
The elaborationalso can be used as a guide in
judging the extent to which an existing critical think-
ing test is comprehensive, and whether it assesses
critical thinking at all. One of my chief criticisms of
most existing critical thinking tests is their lack of
comprehensiveness.For example, they typically fail
180
This content downloaded from 50.155.233.239 on Mon, 18 May 2015 22:26:22 UTC
All use subject to JSTOR Terms and Conditions
to test for such importantthings as being open mind-
ed, and many even fail to test for judging the credi-
bility of sources. Without some defensible concep-
tion of critical thinking, judgments about tests are
likely to be erratic-or worse.
Two other well-known definitions of critical
thinkingare McPeck's "reflectiveskepticism"(1981,
p. 7) and Paul's "strong sense" definition (1987).
Paul's definition is similar in broad outline to the
definitionproposedhere, but emphasizesmoreheavily
being aware of one's own assumptions and seeing
things from others' points of view. However, neither
of these definitionsprovidessufficient elaborationfor
developing critical thinking tests. Furthermore,
McPeck's definitionis negative.Criticalthinkingmust
get beyond skepticism.
Purposes of Critical Thinking Assessment
Not only must we have a defensible elaborated
definition of critical thinking when selecting, criti-
cizing, or developing a test, we must also have a
clear idea of the purpose for which the test is to be
used. A varietyof possible purposesexist, but no one
test or assessmentprocedurefits themall. Herearesome
majorpossiblepurposes,accompaniedby comments:
1. Diagnosing the levels of students' critical
thinking. If we are to know where to focus our in-
struction,we must "startwith where they are"in spe-
cific aspects of critical thinking.Tests can be helpful
in this respect by showing specific areas of strength
and weakness (for example, ability to identify as-
sumptions).
2. Giving studentsfeedback about their critical
thinking prowess. If students know their specific
strengthsand weaknesses, their attemptsto improve
can be better focused.
3. Motivating students to be better at critical
thinking.Thoughfrequentlymisusedas a motivational
device, tests can and do motivate students to learn
the materialthey expect to be covered on the test. If
critical thinking is omitted from tests, test batteries,
or other assessment procedures,studentswill tend to
neglect it (Smith, 1991; Shepard, 1991).
4. Informingteachers about the success of their
efforts to teach students to think critically. Teachers
can use tests to obtain feedback about their instruc-
tion in critical thinking.
5. Doing researchabout critical thinkinginstruc-
tionalquestionsand issues.Withoutcarefulcomparison

of a variety of approaches,the difficult issues in crit-
ical thinking instructionand curriculumorganization
cannot be answered. But this research requires as-
sessment, so that comparisonscan be made.
6. Providing help in deciding whethera student
should enter an educationalprogram. People in some
fields already use assessed critical thinking prowess
to help make admissions decisions. Examples are
medicine, nursing,law, and graduateschool in gener-
al. The idea seems good, but the efficacy of existing
efforts in selecting bettercriticalthinkershas not been
established. Researchneeds to be done in this area.
7. Providing informationfor holding schools
accountablefor the critical thinkingprowess of their
students. A currentlypopularpurpose for testing, in-
cluding critical thinkingtesting, is to pressureschools
and teachers to "measure up" by holding them ac-
countable for the test results of their students.
Purposes 6 and 7 typically constitute "high-
stakes"testing, so called because much often depends
on the results. The science reasoning section of the
American College Test (ACT),much of the new Med-
ical Colleges AdmissionsTest (MCAT),College Board
Advanced Placement (AP) tests, Iowa Test of Educa-
tional Development,and the analytic and logical rea-
soning sections of the GraduateRecord Examination
(GRE) and the Law School Aptitude Test (LSAT)are
examples of high-stakes critical thinking tests.
Traps
In pursuingthe above purposes, educatorsneed
to be aware of several traps, including the following:
1. Test results may be compared with norms,
and the claim made that the difference, or similarity,
is the result of instruction. There are usually other
possible explanationsof the result, such as neighbor-
hood influences.Currently-popular accountabilitytest-
ing invites us into this trap.
2. A pretestand a posttestmay be given without
comparingthe classto a controlgroup.The lackof a con-
trolgrouprendersthe pretest-to-posttestresultsdubious,
since many things otherthan the instructionhave hap-
pened to the students,and could accountfor the results.
3. The use of the same test for the pretest and
posttest has the problem of alerting the students to
the test questions. On the other hand, the use of dif-
ferent forms of (allegedly) the same test for pretest-
posttest comparisons,given that the testing is for crit-
ical thinking,is probablyworse, since differentforms
This content downloaded from 50.155.233.239 on Mon, 18 May 2015 22:26:22 UTC
All use subject to JSTOR Terms and Conditions
Ennis
CriticalThinkingAssessment
are actually different tests. Comparabilityis always
suspect, since so much depends on the specific con-
tent of the test.
4. Most critical thinking tests are not compre-
hensive, especially those that are easiest to use, the
multiple-choicetests. These tests typically miss much
that is importantin critical thinking.
5. Another problem in the use of (especially)
multiple-choicetests lies in differencesin background
beliefs and assumptionsbetween test maker and test
taker. Since a critical thinkeremploys a grasp of the
situation, different beliefs about the situation can
sometimes result in justifiably different answers to
test questions (see Norris & Ennis, 1989).
6. Significant results may be expected in too
short a time period. Learningto think critically takes
a long time. Much reflective practice with many ex-
amples in a variety of situationsis required.
7. High-stakespurposes often interferewith the
validity of a test. This is partlybecause they motivate
cram-schools, which teach students how to do well
on thetestswithoutthe students'havingthecriticalthink-
ing prowessfor whichthe test is supposedlytesting.The
studentsoften learntricksfor takingthe tests.
This interference with validity occurs also in
part because the high-stakes situation pressures the
test makers to avoid taking risks with items, the an-
swers to which might be subject to challenge. So the
pressureis for them to limit their testing to multiple-
choice deductive-logic items of various sorts, that is,
items in which the conclusionnecessarilyfollows from
the premises (thus limiting the test's comprehensive-
ness and content validity). Deductive-logic items are
the most immuneto complaintaboutthe keyed answer.
8. Scarce resources (indicated by low assess-
ment budgets and overworkedteachers)often lead to
compromisesthat affect test validity. Because of the
expense of, and/orteacher gradingtime requiredfor,
the tests necessary to assess critical thinking, many
testingprogramshave resortedto multiple-choicetests
that are arguablyless valid than short answer, essay,
and performancetests of critical thinking.
Published Critical Thinking Tests
Although a numberof tests incorporatecritical
thinking (including the high-stakes tests just men-
tioned), only a few have critical thinking (or some
aspect of critical thinking) as their primaryconcern.
None exist for studentsbelow fourth grade.
181
THEORYINTO PRACTICE/ Summer 1993
Teachingfor Higher Order Thinking
This dearthof critical thinking tests is unfortu-
nate; many more are needed to fit the various situa-
tions and purposesof critical thinkingtesting. In Ta-
ble 1, I have attemptedto identify all currentlyavail-
able publishedtests thathave criticalthinkingas their
primaryconcern. The tests are groupedaccordingto
whether they aim at a single aspect of critical think-
ing or more than one aspect. The essay test is more
comprehensivethan the others.
It would also make sense to group the tests ac-
cording to whether they are subject specific or gen-
eral-contentbased. Subject-specific critical thinking
tests assess critical thinkingwithin one standardsub-
ject matterarea, whereas general-content-basedcriti-
cal thinkingtests use content from a variety of areas
with which test takers are presumed to be already
familiar. A committee of the National Academy of
Education has recommendedthat there be a strong
effort to develop subject-specifichigher orderthink-
ing tests (The Nation's Report Card, 1987, p. 54). A
full understandingof any subjectmatterarearequires
that the person be able to think well in that area.
Regrettably,I can find no subject-specificcriti-
cal thinking tests (that is, tests whose primarypur-
pose is to assess critical thinking in a subject matter
area), althoughparts of some tests (such as the ACT
section on science reasoning) fit this criterion. So
there is no subject-specificgroupingin this listing of
tests primarilycommittedto critical thinking. All of
the tests listed here are general-content-basedtests.
Unfortunately,the NationalAcademy committee
also recommendedthe neglectof general-content-based
higherorderthinkingtests (p. 54). This is a mistake.We
need general-content-based tests to checkfor transferof
criticalthinkinginstructionto everydaylife, regardless
of whetherthinkinginstructionis embeddedin subject
matterinstructionor whetherit is offeredin a separate
courseor unit, or some combinationof the two.
Since I am a coauthor of some of the listed
tests, my conflict of interest in presenting and dis-
cussing this listing is obvious. I have tried not to let
it interfere with my objectivity, but do recommend
Arter & Salmon's Assessing Higher Order Thinking
Skills: A Consumer's Guide (1987), which provides
more extensive coverage. A generaldiscussion of the
problems,prospects,and methodsof critical thinking
testing can be found in Evaluating Critical Thinking
(Norris & Ennis, 1989).
Since statistical informationabout tests can be
misleading, it is importantto make one's own infor-
182
This content downloaded from 50.155.233.239 on Mon, 18 May 2015 22:26:22 UTC
All use subject to JSTOR Terms and Conditions
mal judgment about the validity of the content. Per-
sons who are seriously considering using any test
should take the test and score it themselves. There is
no better way to get a feel for the test's content va-
lidity. One should not depend solely on the name
given to the test by the author and publisher. The
following questions should be considered:
1. Is the test basedon a defensibleconceptionof criti-
cal thinking?
2. How comprehensiveis its coverageof this concep-
tion?
3. Does it seem to do a good job at the level of your
students?
Though these questions might seem obvious, they
are often neglected.
In varying degrees, all of the listed tests can be
used for the first five purposes specified earlier (all
but the high-stakes purposes). Their use for high
stakes is problematicfor two reasons:(a) There is no
security on the tests, so prospective examinees can
secure copies, and (b) most of the tests are not suffi-
ciently comprehensive to provide valid results in a
high-stakes situation. Let me elaborate on this sec-
ond problem.
As indicated earlier, existing multiple-choice
tests do not directly and effectively test for many
significant aspects of critical thinking, such as being
open mindedand drawingwarrantedconclusions cau-
tiously. In response to this problem, some people
will hold that the various aspects of critical thinking
are correlatedwith each other, so the lack of direct
testing of specific aspects does not matter.For exam-
ple, being open minded correlateshighly with judg-
ing the credibilityof sources and identifyingassump-
tions, making all of these good indicatorsof the oth-
ers, so the argumentgoes.
However, when the stakes are high, people pre-
pare the contentareasthatareexpectedto be on the
for
tests. Even thoughthese contentareasmight correlate
highly with other critical thinking aspects when the
stakes are low, specific preparationfor the expected
aspects in order to deal with them on the tests will
lower the correlations, destroying their validity as
indirect measures of the missing aspects of critical
thinking. The danger is to accept correlations ob-
tained in low-stakes situations as representativeof
the correlationsobtainablein a high-stakessituation.
A possible exception to this warning about the
use of the listed tests for high-stakes situationsis the

Table 1
An Annotated List of Critical Thinking Tests
Tests Covering More Than One Aspect of Critical
Thinking
The California Critical Thinking Skills Test: Col-
lege Level (1990) by P. Facione. The California Aca-
demic Press, 217 LaCruz Ave, Millbrae, CA 94030.
Aimed at college students, but probably usable with ad-
vanced and gifted high school students. Incorporates in-
terpretation,argument analysis and appraisal, deduction,
mind bender puzzles, and induction (including rudimen-
tary statistical inference).
Cornell Critical Thinking Test, Level X (1985) by
R.H. Ennis and J. Millman. Midwest Publications, PO
Box 448, Pacific Grove, CA 93950. Aimed at Grades 4-
14. Sections on induction, credibility, observation, de-
duction, and assumption identification.
Cornell Critical Thinking Test, Level Z (1985) by
R.H. Ennis and J. Millman. Midwest Publications, PO
Box 448, Pacific Grove, CA 93950. Aimed at advanced
or gifted high school students, college students, and oth-
er adults. Sections on induction, credibility, prediction
and experimental planning, fallacies (especially equivo-
cation), deduction, definition, and assumption identifica-
tion.
The Ennis-Weir Critical ThinkingEssay Test (1985)
by R.H. Ennis and E. Weir. Midwest Publications, PO
Box 448, Pacific Grove CA 93950. Aimed at grades 7
through college. Also intended to be used as a teaching
material. Incorporates getting the point, seeing the rea-
sons and assumptions, stating one's point, offering good
reasons, seeing other possibilities (including other possi-
ble explanations), and responding to and avoiding equiv-
ocation, irrelevance, circularity, reversal of an if-then
(or other conditional) relationship, overgeneralization,
credibility problems, and the use of emotive language to
persuade.
Judgment: Deductive Logic and Assumption Rec-
ognition (1971) by E. Shaffer and J. Steiger. Instruction-
al Objectives Exchange, PO Box 24095, Los Angeles,
CA 90024. Aimed at grades 7-12. Developed as a crite-
rion-referenced test, but without specific standards. In-
cludes sections on deduction, assumption identification,
and credibility, and distinguishes between emotionally
loaded content and other content.
New Jersey Test of Reasoning Skills (1983) by V.
Shipman. Institute for the Advancement of Philosophy
for Children, Test Division, Montclair State College,
Upper Montclair, NJ 08043. Aimed at grades 4 though
college. Incorporates the syllogism (heavily represent-
ed), assumption identification, induction, good reasons,
and kind and degree.
Ross Test of Higher Cognitive Processes (1976) by
J.D. Ross and C.M. Ross. Academic Therapy Publica-
tions, 20 Commercial Blvd., Novato, CA 94947. Aimed
at grades 4-6. Sections on verbal analogies, deduction,
This content downloaded from 50.155.233.239 on Mon, 18 May 2015 22:26:22 UTC
All use subject to JSTOR Terms and Conditions
Ennis
Critical ThinkingAssessment
assumptionidentification,word relationships,sentence se-
quencing, interpretinganswers to questions, information
sufficiency and relevance in mathematics problems, and
analysis of attributesof complex stick figures.
Test of Enquiry Skills (1979) by B.J. Fraser. Aus-
tralian Council for Educational Research Limited, Fred-
erick Street, Hawthorn, Victoria 3122, Australia. Aimed
at Australian grades 7-10. Sections on using reference
materials (library usage, index, and table of contents);
interpreting and processing information (scales, averag-
es, percentages, proportions, charts and tables, and
graphs); and (subject-specific) thinking in science (com-
prehension of science reading, design of experiments,
conclusions, and generalizations).
Test of Inference Ability in Reading Comprehen-
sion (1987) by L.M Phillips and C. Patterson. Institute
for Educational Research and Development, Memorial
University of Newfoundland, St. John's, Newfoundland,
Canada A1B 3X8. Aimed at grades 6-8. Tests for ability
to infer information and interpretations from short pas-
sages. Multiple choice version (by both authors) and con-
structed response version (by Phillips only).
Watson-Glaser Critical ThinkingAppraisal (1980)
by G. Watson and E.M. Glaser. The Psychological Cor-
poration, 555 Academic Court, San Antonio TX 78204.
Aimed at grade 9 through adulthood. Sections on induc-
tion, assumption identification, deduction,judging wheth-
er a conclusion follows beyond a reasonable doubt, and
argument evaluation.
Tests Covering Only One Aspect of Critical Thinking
Cornell Class Reasoning Test (1964) by R.H. Ennis,
W.L. Gardiner,R. Morrow, D. Paulus, and L. Ringel. Illi-
nois CriticalThinking Project, University of Illinois, 1310
S. 6th St., Champaign,IL 61820. Aimed at grades 4-14.
Tests for a variety of forms of (deductive)class reasoning.
Cornell Conditional Reasoning Test (1964) by R.H.
Ennis, W. Gardiner,J. Guzzetta, R. Morrow, D. Paulus,
and L. Ringel. Illinois Critical Thinking Project, Univer-
sity of Illinois, 1310 S. 6th St., Champaign, IL 61820.
Aimed at grades 4-14. Tests for a variety of forms of
(deductive) conditional reasoning.
Logical Reasoning (1955) by A. Hertzka and J.P.
Guilford. SheridanPsychological Services, PO Box 6101,
Orange, CA 92667. Aimed at high school and college
students and other adults. Tests for facility with class
reasoning.
Test on Appraising Observations (1983) by S.P.
Norris and R. King. Institute for Educational Research
and Development, Memorial University of Newfound-
land, St. John's, Newfoundland, Canada, A1B 3X8.
Aimed at grades 7-14. Tests for ability to judge the cred-
ibility of statements of observation. Multiple choice and
constructed response versions.
183
THEORYINTO PRACTICE/ Summer 1993
Teachingfor Higher Order Thinking
criticalthinkingessay test, which does test more com-
prehensively than others. But it is not secure. Fur-
thermore,it is more expensive in time and/or money
than multiple-choice tests to administer and score.
The problem is serious in high-stakestesting. We do
not yet have inexpensive critical thinking testing us-
able for high stakes. Research and development are
needed here.
The listed multiple-choice tests can, to varying
degrees, be used for the first five listed lower stakes
purposes:diagnosis, feedback, motivation, impact of
teaching, and research. But discriminatingjudgment
is necessary. For example, if a test is to be used for
diagnostic purposes, it can legitimately only reveal
strengthsand weaknesses in aspects for which it tests.
The less comprehensivethe test, the less comprehen-
sive the diagnosis.
For comprehensiveassessment,unless appropri-
ate multiple-choice tests are developed, open-ended
assessment techniquesare probablyneeded. Until the
published repertoireof open-ended critical thinking
tests increases considerably,and unless one uses the
publishedessay test, or partsof otheropen-endedtests,
such as College Board's advanced placement (AP)
tests, it is necessary to make your own test.
Making Your Own Test
In making your own test, it is probably better
that it be at least somewhatopen ended anyway, since
makinggood multiple-choicetests is difficult and time
consuming, and requiresa series of revisions, tryouts,
and more revisions. Suggestions for makingmultiple-
choice critical thinkingitems may be found in Norris
and Ennis (1989), but I do not present them here,
because open-ended assessment is better adapted to
do-it-yourself test makers and can be more compre-
hensive. Norris and Ennis also make suggestions for
open-endedassessment,and the discussionhere grows
out of that presentation.
Multiple-choiceassessment is labor intensive in
the constructionand revision of the tests. Open-end-
ed assessment is labor intensive in the grading, once
one has developed a knack for framing questions.
One promising approachis to give a multiple-choice
item, thus assuring attentionto a particularaspect of
critical thinking, and to ask for a brief written de-
fense of the selected answer to the item.
As in the previous example, open-endedassess-
ment can be fairly well structured.Or it can be much
184
This content downloaded from 50.155.233.239 on Mon, 18 May 2015 22:26:22 UTC
All use subject to JSTOR Terms and Conditions
less structured-in the form of naturalisticobserva-
tion of a student. Greater structureusually means
greatereffort beforehand,but also greaterassurance
that there will be opportunitiesto assess specific as-
pects of critical thinking.Less structuregenerally re-
quires greatereffort duringand after the observation,
and gives the opportunityfor more life-like situa-
tions, but provides less assurancethat a broad range
of specific aspects of critical thinking will be as-
sessed. The sections thatfollow illustrateseveraltypes
of open-endedcritical thinkingtests that teacherscan
make themselves.
Multiple Choice With Written Justification
In the Illinois CriticalThinkingProject-in con-
junction with the Alliance for Essential Schools in
Illinois-we are currently exploring the use of the
multiple-choice-plus-written-justification format.We
have taken 20 items from the Cornell Critical Think-
ing Test, Level X, and requesteda brief writtenjusti-
fication of the student's answer to each. In the fol-
lowing example of an item focusing on judging the
credibilityof a source, the situationis the exploration
of a newly-discoveredplanet:
WHICHIS MOREBELIEVABLE? Circleone:
A. The healthofficer investigatesfurtherand says,
"Thiswatersupplyis safe to drink."
B. Several others are soldiers. One of them says,
"Thiswatersupplyis not safe."
C. A and B are equallybelievable.
YOURREASON:
One advantageof this promising format is that spe-
cific aspects of critical thinking can be covered (in-
cluding an aspect not effectively tested in existing
multiple-choicetests: being appropriatelycautious in
the drawingof conclusions).Anotheradvantageis that
answersthatdifferfromthosein the key, if well defend-
ed, can receive full credit.Answersthatdifferfrom the
key, as I notedearlier,are sometimesdefensible,given
thatthe test takerhas differentbeliefs aboutthe world
than the test maker. We have found that high inter-
rater consistency (.98) can be obtained if guides to
scoringare carefullyconstructedand if the scorersare
proficientin the same conceptionof criticalthinking.
I recommendthis approachto makingyour own
test. It is fairly quick, can be comprehensive, pro-
vides forgivenessfor unrefinedmultiple-choiceitems,
and allows for differencesin studentbackgroundsand
interpretationof items.

Essay Testing of Critical Thinking
Several approachesto making one's own essay
test of critical thinking are viable, depending on the
purpose.
High structure. The use of the argumentative
essay to assess critical thinking can vary consider-
ably in degree of structure.The Ennis-WeirCritical
ThinkingEssay Test is an example of a highly struc-
turedessay test. It provides an argumentativepassage
(a letter to an editor) with numberedparagraphs,most
of which have specific built-in errors. Students are
asked to appraisethe thinking in each paragraphand
the passageas a whole, and to defendtheirappraisals.
A scoring guide assigns a certainnumberof pos-
sible points to the appraisal of each paragraphand
the passage as a whole, and provides guidance for the
grader. But the grader must be proficient in critical
thinkingin orderto handle responses that differ from
standardresponses in varyingdegrees. Responses that
are radically different, if well defended, receive full
credit. Scoring by proficient graders takes about 6
minutes per essay.
Medium structure. Structurecan be reduced by
providing an argumentativepassage and requesting
an argumentativeresponse to the thesis of the pas-
sage and its defense-without specifying the organi-
zation of the response. College Board AP tests use
this approach.
Scoring can be either holistic (one overall grade
for the essay), or analytic (a grade for each of several
criteria).Holistic scoring is quicker and thus less ex-
pensive. Proficient graderstake roughly 1 or 2 min-
utes for a two-pageessay. Analytic scoringgives more
information and is more useful for most purposes.
Proficientgraderstakeroughly3 to 6 minutesfor a two-
page essay, dependingon how elaboratethe criteriaare.
Minimal structure. Structurecan be furtherre-
duced by providingonly a question to be answeredor
an issue to be addressed.The Illinois Critical Think-
ing Essay Contest uses this approach(Powers, 1989).
In one year, studentswere asked to take and defend a
position about the possible regulation of music tele-
vision, a topic of great interest to students. Reduced
structuregives students more freedom, but provides
teachers with less assurance of diagnostic informa-
tion, not a problem for the essay contest. Again, ei-
ther holistic or analytic scoring is possible.
At Illinois we are also using this formatfor the
developmentof the IllinoisCriticalThinkingEssay Test.
This content downloaded from 50.155.233.239 on Mon, 18 May 2015 22:26:22 UTC
All use subject to JSTOR Terms and Conditions
Ennis
Critical ThinkingAssessment
We developeda six-factoranalyticscoring system, an
adaptationof scoring guides developedby the Illinois
StateBoardof Education,and have securedhigh inter-
raterconsistency(.94). This approachalso looks prom-
ising. Gradingtakes us about 5 minutesper essay for
essays writtenin 40 minutesof class time.
Performance Assessment
Performanceassessment is the most expensive
of all, since it requires considerableexpert time de-
voted to each student.It has the greatestface validity
for whateveris revealed,since the situationsare more
realistic-possibly real life situations. However, the
greaterthe realism, the less assuranceof comprehen-
siveness. In real life situations, people generally re-
veal only what the situation requires, and most ob-
servable situationsdo not requireall aspects of criti-
cal thinking.So real-life performanceassessment en-
countersa difficulty similarto one found in multiple-
choice assessment: reduced comprehensiveness.An-
other possible danger in performanceassessment is
excessive subjectivity.
The least structuredperformanceassessment is
naturalistic observation, as in a case study (Stake,
1978). Here, a trainedobservertakes extensive notes
describing a series of events and focuses on the ac-
tivities of one person or group.Interpretationis inev-
itable, but "rich"descriptionis the goal.
An example of slightly more structuredperfor-
mance assessment is the use of a student's portfolio
of work to determine graduationfrom high school
(recommended, for example, by Sizer in Horace's
Compromise,1984). Validity of this type of assess-
ment is not yet established. It is an attractiveidea,
but many problems exist, including probablelack of
comprehensivenessof critical thinking assessment.
A more structuredperformance assessment is
exemplified by an exploratoryassessment effort by
the NationalAssessmentof EducationalProgress(Blum-
berg, Epstein,MacDonald,& Mullis, 1986). A student
is given a varietyof materialsand asked to investigate
what factors affect the rate at which sugar cubes dis-
solve. The observerasks questionsand watchesto see
whetherthe studentgoes aboutthe taskscientifically.In
this kind of performanceassessment, structureis pro-
vided by the assignmentof a task, which is designed
to check things of interest.
Performanceassessmentseems valid on the face
of it. Expense, possible lack of comprehensiveness,
185
THEORYINTO PRACTICE/ Summer 1993
Teachingfor Higher Order Thinking
possible excessive subjectivity, and lengthy reports
are dangers.
Summary
Critical thinkingtesting is possible for a variety
of purposes.The higher the stakes and the greaterthe
budgetaryrestraints,the fewer the purposes that can
be served. In particular,comprehensivenessof cover-
age of aspects of critical thinking is threatened in
high-stakes testing.
A number of published tests focus on critical
thinking. Almost all are multiple-choicetests, an ad-
vantage for efficiency and cost, but currentlynot for
comprehensiveness.More research and development
are needed.
Viable alternativesinclude the additionof justi-
fication requests to multiple-choice items, essay test-
ing with varyingdegreesof structure,andperformance
assessment.All are considerablymore expensive than
multiple-choice testing when used on a large scale,
but on a small scale, they offer a feasible alternative
in terms of validity and expense. However, grading
them does take more time than grading a prepack-
aged multiple-choice test.
Note: The author deeply appreciates the helpful com-
ments of Michelle Commeyras, Marguerite Finken, Ste-
phen Norris, and Amanda Shepherd.
References
Arter, J.A., & Salmon, J.R. (1987). Assessing higher or-
der thinking skills: A consumer's guide. Portland,
OR: Northwest Regional Educational Laboratory.
Blumberg, F., Epstein, M., MacDonald, W., & Mullis, I.
(1986). A pilot study of higher-order thinking skills
assessment techniques in science and mathematics.
Princeton, NJ: National Assessment of Educational
Progress.
Tip
186
This content downloaded from 50.155.233.239 on Mon, 18 May 2015 22:26:22 UTC
All use subject to JSTOR Terms and Conditions
Dewey, J. (1910). How we think. Boston: D.C. Heath.
Educational Policies Commission. (1961). The central
purpose of American education. Washington, DC:
National Education Association.
Ennis, R.H. (1962). A concept of critical thinking. Har-
vard Educational Review, 29, 128-136.
Ennis, R.H. (1981). Eight fallacies in Bloom's taxono-
my. In C.J.B. Macmillan (Ed.), Philosophy of edu-
cation 1980 (pp. 269-273). Bloomington, IL: Phi-
losophy of Education Society.
Ennis, R.H. (1987). A taxonomy of critical thinking dis-
positions and abilities. In J. Baron & R. Sternberg
(Eds.), Teaching thinking skills: Theory and prac-
tice (pp. 9-26). New York: W.H. Freeman.
Ennis, R.H. (1991). Critical thinking: A streamlined con-
ception. Teaching Philosophy, 14(1), 5-25.
Ennis, R.H. (in press). Critical thinking. Englewood
Cliffs, NJ: Prentice-Hall.
McPeck, J.E. (1981). Critical thinking and education.
New York: St. Martin's Press.
The nation's report card. (1987). Cambridge, MA: Na-
tional Academy of Education, Harvard Graduate
School of Education.
Norris, S.P., & Ennis, R.H. (1989). Evaluating critical
thinking. Pacific Grove, CA: Midwest Publications
Paul, R.W. (1987). Dialogical thinking: Critical thought
essential to the acquisition of rational knowledge
and passions. In J. Baron & R. Sternberg (Eds.),
Teaching thinking skills: Theory and practice (pp.
127-148). New York: W.H. Freeman.
Powers, B. (Ed.). (1989). Illinois critical thinking annu-
al. Champaign, IL: University of Illinois College of
Education.
Shepard, L.A. (1991). Will national tests improve stu-
dent learning? Phi Delta Kappan, 73, 232-238.
Sizer, T. (1984). Horace's compromise. Boston: Hough-
ton-Mifflin.
Smith, M.L. (1991). Put to the test: The effects of exter-
nal testing on teachers. Educational Researcher,
20(5), 8-11.
Stake, R.E. (1978). The case study method in social in-
quiry. Educational Researcher, 7(2), 5-8.