Designing Social Inquiry

SCIENTIFIC INFERENCE IN
QUALITATIVE RESEARCH

Gary King
Robert 0. Keohane
Sidney Verba

PRINCETON UNIVERSITY PRESS

PRINCETON, NEW JERSEY

 Contents

Copyright@ 1994 by Princcton Universily Press Preface ix

r:u.blishcd by Princcton Univcrsity Press, 41 William Strcet,
1 nnceton, New Jersey 08540
l~l '.hc Unitcd Kingdom: Princeton Univcrsity Press, 1 The Science in Soda! Science 3
Cluchcstci~ West Sussex 1. l lntroduction 3
All Rights Reservcd 1.1.1 Two Styles of Research, One Logic of Inference 3
Li/m1ry ofC011gress C11ta/ogi11g ..j11-P11/lficntio11 Data 1.1.2 Defining Scientific Research in Social Sciences 7
King, Cary. Ll.3 Science and Complexity 9
Designing social inquiry: scicnlific infercncc in qw1litiiltivc 1.2 Major Components of Research Design 12
rcscarch / Gary King, Robert 0. Keoh,mc, Sidney Verba.
p. cm. 1.2.1 Improving Research Qnestions 14
Tncludcs bibliogrnphical rcfcrcnccs and index :t.. 2.2 Improving Theory 19
I~BN 0-69]-03470-2 (cloth : alk. papcr) 1.2.3 hnproving Data Quality 23
1SBN (J..691-03471-0 (pbk.: alk. p<1pcr) 1.2.4 Tmproving the Use of Existing Data 27
1. Soda! sdcnces-"-Mcthodology. 2. SociGl scicnces-
Rcsearch. 3. Infcrcnce 1.3 Themes of This Volume 28
1. I<cohane, Robert Owen. 1.3.1 Using Observable lmplications to Connect Theory
II. Vcrb<1, Sidncy. III. Tit!e.
f-l6U(5437 1994 93-39283
and Data 28
300'.72·-dc20 C[p 1.3.2 Maximizing Leverage 29
1.3.3 Reporting Uncertainty 31
This book has becn composcd in Adobe Palatino
1.3.4 Thinking like a Social Seien/ist: Skepticisn1
Pr!nc:ton Univcrsity Press books nrc printcd 011 and Rival Hypot:heses 32
ac1d-trec papcr mld nwet thc guidc!ines
for pcrmancnce <1nd durnbility of thc Com.rnittce 2 Descriptive Inference 34
on Production Cuidelincs for Book Longevity 2.1 General Knowledge and Particular Facts 35
of the Council on Librnry Rcsources
2.1.1 "Interpretation" and Inference 36
Printcd in thc United States of America
2.1.2 "Uniqueness," Complexity, and Si,nplification 42
10 9 8 7 6 5 4 J 2 2.1.3 Co1nparative Case Studies 43
10 9 8 7 6 5 4 3 2 2.2 Inference: the Scientific Purpose of Data Collection 46
(l'bk.)
2.3 Formal Models of Qualitative Research 49
2.4 A Formal Model of Data Collection 51
2.5 Sun1marizing Historical Detail 53
2.6 Descriptive Inference 55
2.7 Criteria for Judging Descriptivc Infcrcnces 63
2.7.1 Unbiased Inferences 63
2.7.2 Efficiency 66
Preface

IN THIS 1300K we develop a unified approach to valid descriptive and

causal inferencc in qualitative research, where nurnerical 1neasure-
ment is either impossible or undesirable. We argue that the logic of
good quantitative and good qualitative research designs do not funda-
mentally differ. Our approach applies equally to these apparcntly dif-
ferent forms of scholarship.
Our goal in writing this book is to encourage qualitative researchers
to take scientific inference seriously and to incorporate it into their
work. We hope that our unified logic of inferencc, and our attempt to
demonstnüe that this unified logic can bc helpful to qualitative re-
searchcrs, will hclp improvc the work in our discipline and perhaps
aid research in othcr social scienccs ,1..s well. Thus, wc hope that this
book is read and critically considered. by political scientists and other
sodal scientists of all persuasions and career stages-frmn qualitative
field researchers to statistical analysts, fron1 advanced undergraduates
and first-year graduate students to senior scholars. We use some math-
ematical notation becausc it is especially helpful in clarifying concepts
in qualitative 1nethods; however, we assun1e no prior knowlcdge of
mathen1atics or statistics, and 1nost of the notation can be skipped
without loss of continuity.
University adn1inistrators often speak of the con1ple1nentarity of
teaching and research. Indccd, teaching and research are very nearly
coincident, in that they both entail acquiring new knowlcdge and com-
municating it to others, albeit in slightly different forms. This book at-
tests to the synchronous nature of these activities. Since 1989, we have
been working cm this book and jointly teaching the graduate se1ninar
"Qualitative Methods in Social Science" in Harvard University 1 s De-
parh11ent of Government. The se1ninar has been very livcly, and it
oftcn has spilled into thc halls and onto the pages of lengthy memos
passed among ourselves and our students. Our intellectual battles
have c1lways been friendly, but our rules of engagement meant that
11
agreeing to disagree" and c01npron1ising wcre high crin1es. If onc of
us wc1s not truly convinced of a point, wc took it as our obligation to
continuc the debatc. Tn thc end, we each learned. a great dcal about
qualitative and qucmtitalive resec1rch from one another and fr01n our
students and changed 111any of our initial positions. In addition to its
primary purposes, this book is a state1nent of our hard-won unani-
mous position on scientific inferencc in qualitative research.
 Prcfoce xi
x · Prcface
Political Science Association mcetings (W<1shington, D.C:; ~9 ~\ugu 5t-
Wc con1plctcd the first vcrsion of this book in 199·1 and have revised l September l99l), the Seminar in the Methodology and [ hilosophy of
it extensively in thc years since. Gary King first suggested that we
the Social Sciences (Harvard Univcrsity,. Center _for Internatlm~al .~~-
write this book, drafted the first vcrsions of 1nost chapters, and took -. ·" 25 se,pternber ·1992) the Colloqmum Senes of the Intcrd1sc1-
the lcad through the long proccss of revision. However, the book has tc111s, - ' 1· U . ~ ,·t
plinary Consortium for Statistical Applications (Inc _iana n1vers1 y,
been rewritten so extensively by Robert Keohane and Sidney Verba, as , ·t991) the Institute for Global Cmiperation and Change
weil as Gary King, that it would be impossible for us to idcntify the 4 [) ecem b er . , ~ , r:: '1991)
scininar series (University of California, Berk~ley, L, JantW~\ c · ; ,
authorship of many passages and sections reliably.
and the University of Jllinois, Urbana-Champaign (18 March 1993).
During this long process, we circulated drafts to colleagues around
the United States and are indebted to them for the extraordinary gen-
erosity of their comments. We are also grateful to the graduate stu- Gary King
dents who have been exposed to this manuscript both at .Harvard and Robert O. Keohanne
at othcr universities and whose reactions have been important to us Sidney Verba
in making revisions. Trying to list all thc individuals who were helpful
in a project such as this is notoriously hazardous (we estimate the Cn111bridgc, Massach11setts
probability of inadvertently 01nitting son1eonc whose comments wcre
important to us to be 0.92). We wish to acknowledge the following
individuals: Christopher H. Achen, John Aldrich, Hayward Alker,
Robert I-[. Bates, Ja111es Battista, Nathaniel Heck, Nancy Burns, Michael
Cobb, David Collier, Gary Cox, Michael C. Desch, David Dessler, Jorge
Dorninguez, George Downs, 'Mitchell Duneier, Matthew Evangelista,
John Ferejohn, Andrew Gelman, Alexander George, Joshua Goldstein,
Andrcw Green, David Green, Robin f-Ianna, Michael Hiscox, Jmnes E.
Jones, Sr., Miles Kahler, Elizabeth King, Alexander Kozhemiakin, Ste-
phen D. l<rasner, Herbert Kritzer, James Kuklinski, Nathan Lane, Peter
Lange, Tony Lavelle, Judy Layzer, Jack S. Levy, Daniel Little, Sean
Lynn-Jones, Lisa L. Martin, Helen Milner, Gerardo L. Munck, Timothy
P Nokken, Joseph S. Nye, Charles Ragin, Swarna Rajagopalan, Sha-
mara Shantu Riley, David Rocke, David Rohde, Frances Rosenbluth,
David Schwieder, Collins G. Shackelford, Jr., Kenneth Shcpsle, Daniel
Walsh, Carolyn Warner, Steve Aviv Yetiv, Mary Zerbinos, and Mi-
chael Zürn. Our appreciation goes to Steve Voss for preparing the
index, and to the crew at Princeton University Press, Walter Lippin-
cott, ·Makoln1 DeBevoise, Peter Dougherty, and Alessandra Bocco. Our
thanks also go to the National Science Foundation for research grant
SllR-9223637 to Gary King. Robert 0. Keohane is grateful to the John
Simon Guggenheün Men1orial Foundation for a fellowship during the
term of which work on this book was completed.
We (in various pertnutations and combinations) were also extren1ely
fortunate to have had the opportunity to present earlier versions of
this book in seminars and panels at thc Midwest Political Science As-
sociation meetings (Chicago, 2-6 April 1990), the Political Methodol-
ogy Group meetings (Duke University, "18-20 July 1990), the American
Designing Social Inquiry
CHAPTER 1

The Science in Social Science

1.1 !NTRODUCT!ON

TI-IIS BOOK is about rescarch in the social sciences. Our goal is practical:
designing research that will produce valid inferences about social and
political life We focus on political science, but our argun1ent applies to
other disciplines such as sociology, anthropology, history, econ01nics,
and psychology and to nondisciplinary areas of study such as legal
evidence, education research, and clinical reasoning.
This is neither a work in the philosophy of the social sciences nor a
guide to specific research tasks such as the design of surveys, conduct
of field work, or analysis of statistical data. Rather, this is a book about
rcsearch design: how to posc questions and fashion scholarly research
to make valid descriptive and causal inferences. As such, it occupies a
1niddle ground between abstract philosophical debates and the hands-
on techniques of the researcher and focuses on the essential logic un-
derlying all social scientific research.

1.1.1 'Tivo Styles of Research, One Logic of lnference

Our 111ain goal is to connect the traditions of what are conventionally
11
denoted "quantitative" and qualitative" research by applying a uni-
fied logic of infercnce to both. Thc two traditions appcar quite differ-
ent; indeed they s01netin1es seen1 to be at war. Our view is that these
differences are 1nainly ones of style and specific technique. The same
underlying logic provides the frainework for each research approach.
This logic tends to be explicated and formalized clearly in discussions
of quantitative research 1nethods. But the smne logic of infercnce un-
derlies the best qualitative research, and all qualitative and quantita-
tive researchers would bcncfit by morc cxplicit attention to this logic
in the course of designing rcsearch.
The styles of quantitative and qualitative research are very different.
Quantitative research uses numbers and statistical methods. lt tends to
be based on numerical 1neasure1nents of specific aspects of phenon1-
ena; it abstracts fron1 particular instances to seek general description
or to test causal hypotheses; it seeks measure1nents and analyses that
are easily replicable by other rcsearchers.
 4 The Scicnce in Sochl Sciencc
QualHat~v~ ~·csearch, in contrast, covers a wide rang:e of approaches,
but by defmthon, none of thesc approaches relics on nurnerical mea-
surements. Such work has tcnded to focus on one or a s1nall nurnbcr
of ~·ases, to us? intensive intervicws or depth analysis of historical ma-
tenals, tobe drn.cursive in method, and tobe concerned with a rounded
or c_°mprehcns1ve account of somc event or unit. Even though they
have ~ small number of ca~e~, qw.1~itative researchers generally un-
ea~th .enormous arnounts of mtormat10n from their studies. Somctimes
~his kmd of work in the social sciences is Iinkcd with area or case stud-
1~s w!1ere the fc~cus is cm a particular event, decision, institution, Ioca-
hon, 1ssue, or piece of legislation. As is also the case with quantitative
rescarcl~, ~he 1_~1stance is (~ften im~ortant_ in its own right: a major
c~1ang~ I~ ,~ n~llon, an el~ct10n, a maJor dcc1sion, or a world crisis. Why
d1d the East Cerrnan reg1me collapse so suddenly in 1989? More gener-
~lly: ~h~ d1~ almost ~11 the con11ntmist regin1es of Eastern Europc col-
lapse m ~ 989?, S~)metunes, but c_ertainly not always, the event may be
chosen _as an exernpl~r _of a parhcular type of event, such as a political
r~volut10n_ or the de~1s1on of a particular cornn1unity to rejcct a waste
d1sposal site. So~netunes this kind of work is linked to area studies
where thc focus ~s on the history and culture of a parlicular part of the
worl_d. The parhcular place or cvent is analyzed closely and in full
detail.
For se~cral dccades, political scientists have debated the 1nerits of
case .s,t:idies ;,er~us ~~a~fstical_ studies, arca studies versus con1parative
studies, m:d ~c1entific stud1es of politics using quantitative inethods
versus "h1stoncal" investigations relying on rich textual and contex-
tual t~n~~rs:m_1ding. So~e _quantitative researchers believe that sys-
tcn1at1c statisbcal analys1s 1s the only road to truth in the social sci-
ci:ices. Advocat~s. of qualitative research vehe1nently disagree. This
difference of opm1on le~ds to lively debate; but unfortunately, it also
~1_fu:cat:s the social sc1~nc~s into a q:1a_ntit~tive-systematic-general-
1zmg btanch and a quahtative-humamshc-d1scursive branch. As the
f~)rmer becomes m?rc and more sophisticated in the analysis of statis-
tlcal data (and_ the1r work becmnes less con1prehensible to thosc who
have_ not stud1ed the techniques), the latter becomes more and more
~~n~mced, of the irrelevm:cc of such analyses to the seemingly non-
~cphcablc c1nd nongenerahzable events in which its practitioners are
1ntercsted.
A major purpose of this book is to show that lhe differences betwcen
the quantita_ttve and qualitative traditions are only stylistic and arc
methodolog1cally and substantively unirnportant. All good research
can be understood-indeed, is best understood-to derivc fron1 the
samc underlying logic of inference. Both quantitative and qualitative
[ntroduction 5
rcscarch can bc systematic and scicntific. f···Iü,torical rescarch can be an-
alytical, seeking to evaltrnte alternative cxplanations through a proccss
of valid causal inference. 1-Iistory, or historical sociology, is not incmn-
patiblc with social sciencc (Skocpol l 984: 374-86).
Breaking down thesc barriers requircs tha t we begin by questioning
the vcry concept of "qualitative" research. We have used the term in
our title to signal our subjcct matter, not to irnply that "qualitative"
rescarch is fundarncntally different from "quantitative" rcsearch, ex-
cept in style.
Most research does not fit clearly into onc category or the other. The
best often cmnbincs features of each. In the same rescarch project,
smne data may be collected that is amenablc to statistical analysis,
while other equally significant infonnation is not. Patterns and trends
in social, political, or econmnic behavior are more readily subjected to
quantitative analysis than is the flow of ideas an1ong people or the
difference n1ade by exceptional individual leadership. If we arc to un-
d erstand the rapidly changing social world, we will nced to include
inforn1ation that cannot be easily quantified as well as that which can.
Furthermore, all social science requires con1parison, which entails
11
judgmcnts of which phenon1cna are "1nore or "lcss" alike in degrec
(i.e., quantitative diffcrences) or in kind (i.c., qualitative differcnccs).
Two excellent recent studies exemplify this point. In Coercive Cooper-
ation (1992), Lisa L. Martin sought to explain thc degree of interna-
tional cooperation on economic sanctions by quantitatively analyzing
ninety-nine cases of atte1npted cconon1ic sanctions frmn the post-
World War II era. Although this quantitative analysis yielded much
valuable information, ccrtain causal infercnces suggested by thc data
wcre ambiguous; hence, Martin carried out six detailed case studies of
sanctions episodcs in an attcn1pt to gather morc cvidence relevant to
her causal inference. For Making Democmcy Work (1993), Robert D. Put-
nain and his colleagues intcrviewed l12 Italian regional councillors in
1970, 194 in 1976, and 234 in 1981-1982, and ll5 community leaders in
1976 and 118 in 1981-1982. They also sent a mail questionnaire to over
500 con1111unity leaders throughout the country in 1983. Four nation-
wide 1nass surveys werc undcrtaken espccially for this study. Never-
theless, betwecn 1976 and 1989 Putnan1 and his colleagucs conducted
detailed case studies of the politics of six regions. Seeking to satisfy the
"interocular traun1atic tcst," thc investigators "gained an intimatc
knowledge of thc internal political 1naneuvering and personalitics that
have animated regional politics ovcr thc last two decadcs" (Putnam
1993:190).
Thc lessons of these efforts should be clear: ncither quantitative nor
qualitative research is superior to the othcr, regardless of thc rcsearch
 6 The Sciencc in Social Science
Jntroduction 7
problem being addressed. Since many subjects of intcrest to social sci-
entists cannot be meaningfully formulated in ways that permit statisti- . . ' _, uestionnaire on psychopathy inay not bc th~ t~;st
tenng a hve-page q , ·t d AJb,,rt Finstein who said as
J l1 s l irnpl'ler once c1 e
strategy. osep ~ ~L , · , rtain they do not say anything about
cal testing of hypotheses with qumüitative data, we do not wish to ' ·' ' '
encourage the exclusive use of quantitative techniqucs. We arc not try- far as our proposit10nls arde- c~ ' a;1ythin{T about reality, they are not
ing to get all social scientists out of the library and into the computer l ·t nd as far as t wy o say c t:) • t ·
center, or to replace idiosyncratic conversations with structured intcr- real y, a ' '
· « (S )1umpeter [1930
"] 1991·298-99)
· · Yet even though certam· y is
d
certam. ,bl' c .
e can unprovc t11c rc 1rn
_ . bilitv validitv ccrtamty, an
;, ' ;, . 'f
views. Rather, we argue that nonstatistical research will producc n1ore unattama e, w - ' . b . ttention to the rules of sc1enh 1c
rcliable rcsults if researchers pay attention to the rules of scientific in- honesty of our conclus10ns y pay1ng a 1 , t nrnke descriptivc and
Tl .al science we espouse see <s o <
ference-rules that are sometimes rnore clearly stated in the style of
quantitative rcsearch. Precisely defined statistical methods that under-
inference. 1e soc1, , -_ i i Th )Se who do not sharc the as-
causal. inferc_nces .about t!1e ::;.:;t· kno~ability and the aspiration for
gird quantitative research represent abstract formal models applicable sumphons ot partial and imp d' ·11 have to look elsewhere for
to all kinds of research, even that for which variables cannot be 1nea- dcscriptive and causal understan mg ':"1 '. _,
sured quantitatively. The very abstract, and even unrealistic, nature of • . . f d' inatic battles in wh1ch to engage.
insp1ration
1 . or. or para ig 'd ' . C'S for scientific empirical research.
statistical rnodels is what makes the rules of inference shine through so we do not prov1 e rec1p , , d' ·
clearly. n smn, 0 . -s and rules but these are ineant to 1sc1-
The rules of inference that we discuss are not relevant to all issues
W<: offer a nu~ber _f_,~r-~~~pt[~1' both u;ntitative and qualitative rc-
plme thought, not stlfk it. f t q1· ation of theoretical standards
that are of significance to social scientists. Many of the n1ost ü11portant l ein the unper ec app ic, . . 1
searc 1, we enga~ - . n crfect rescarch designs and en1p1nca
11
questions concerning political life-about such concepts as agcncy, ob- of inference to mherently P . _ _ tions but we can ask that
ligation, legitirnacy, citizenship, sovcreignty, and the proper relation- 111
data. Any n1eanil:~ful rule~ /d ~t of e:trefm li~~tions for the reliabil-
ship betwcen national societies and international politics-are philo- cxceptions be jusbfied ,expdhc1ti,
sophical rather than empirical. But the rules are relevant to all rcsearch .t of research be assesse ' an t\~~~t ' t~~= unce~tainty of conclusions be
where thc goal is to learn focts about the real world. Indeed, the dis- ~fportecl. We seek not dogma, but disciplincd thought.
tinctivc characteristic that sets social science apart fr01n casual obser-
vation is that social science seeks to arrive at valid inferences by the
1.'l .2 Defining Scientific Research in the Social Sciences
syste1natic use of wcll-establishcd proccdures of inquiry. Our focus
here on empirical research rneans that we sidestep rnany issues in the Our definition of "s~ie1:tific research" is a~~ ~d:~s\0 c:~~~:;, ::1~ ~~L~~~
philosophy of social science as well as controversies about the ro!e of t·t t·ve or quahtat1ve researc11, even 1 1 1l. I
postmodernis111, the nature and cxistence of truth, relativism, and rc- quan 1 .a 1 t' 1 Yet we neecl a cl e1·1111·t·1011 (_,f good research, for w 11c . I1
approxnna ·101 , . . tor, , 1·1 · word con1es wlt 1
, · ,-f f, " asour d escnp
lated subjects. We assume that it is possib]e to have some knowledge d 11, 11 5
we use the wor seien 1 IC c t d or inappropriate or down-
of the external world but that such knowledge is always uncertain. t t. 18 that are unwarran e < •
n1any conno a 101 , ' ' . . , Hence wc prov1dc
Furthcrmore, nothing in our set of rules implies that we 111ust run . d' f H smne quahtat1ve researc11ers. '
the perfect experiment (if such a thing existed) or collect all relevant right m.c~n rn~y ..c . - As should be clear, we do not regard quan-
an exphclt defmit10n here. , ' , . " tific than qualitative research.
data beforc we can 1nake valid social scientific inferences. An impor- titative research to be any l:°~ne soenh an be c1uantitative or quali-
Good rescarc 1, a L , ~
tant topic is worth studying even if very little information is availablc. 1 tl1 t ·s scienhfic researc , c, ' .
. t 'f earch has the follow1ng
The result of applying any research design in this situation will be tative in style. In design, however, seien I IC res ' c

relatively uncertain conclusions, but so long as we honestly report our four characteristics:
uncertainty, this kind of study can be very uscful. Limited information
is often a necessary feature of social inquiry. Because the social world l. The goal is inference. Scicntific res~~arch is d~s.i~~;~:~f:~:,:~: :i~~~~;~t!~:
changes rap.idly, analyses that help us und.erstand those changes re-
quire that we describc them and seek to und erstand them contempora-
:~)::x ,~~:~t;~:;i
1 1
:;~:::;;:(;:~: :~:e
s~:s(~;fi~f ;1~:~~;l~~:na arc
1
often indispens-
neous1y, even when uncertainty about our conclusions is high. The ur- . - ,. , - th _, word, "c1uasi-cxperirnent." Either i1 research
gency of a problem may be so great that data gathercd by the most 1Wc rcjcct thc conccpt, 01 al leaSl e ,· , nd vrtlucs of the key rnrn;al
. . . t01. )1ltrol ovcr the observrt 1ons " ' ,
useful scientific n1ethods might be obsolete before it can be accumu- dcsign involvcs tnvest1ga Ce , ) .. , , not (in which case it is noncxper-
. 1 . 1 , . , ·t i, an expernnent 01 1 c1ot 3 . _ .
variables (m w HC) 1 ca.sc l s' d .· ental n'scarch havc thctr at1V<lll·-
lated. lf a distraught person is running at us swinging an ax, adminis- 1 B lh 'x11erinwnta! <111 noncxpc1 nn - ' --
imcntal
trtgcs an drcsearc c .· situ<1tions than the othcr.
1 · O
d rnw b·,cks·
, , one 1s not b. e•tt,. .
u 111 a 11 research
· '
8 The Science in Social Sciencc
nble to scientific research, but thc accunmlation of focts a!one is not suffi-
cicnt. Fi:!cts can bc collected (by qualitative or quantitative researchers)
rnore or less systcmatically, and the former is obviously better thm1 the
hütet~ but our particular definition of science requires the additiorn1! step
of attempting to infer beyond thc immediate data to something broader
that is not dircctly observed. That something may involve dcscriplive in-
fercncc--·-using obscrvations from thc world to learn about other unob-
served facts. Or that something may involve causal infcrence-learning
about causal effects from thc data observcd. Thc dornain of inference can
be restrictcd in space and time-voting behavior in American elections
since 1960, social movements in Eastern Europe since 1989~-or it can be
cxtcnsive--hum,:m behavior since thc invention of agriculture. In either
case, the key distinguishing mark of scientific research is the goal of m,1k-
ing infercnces that go beyond the parhcular observations collected.
2. The procedures are public. Scientific research uses explicit, codified, and
public methods to generatc and analyze data whose reliability can there-
fore be assesscd. Much socia\ research in the qualitative style follows
fewer precise rulcs of research procedure or of inference. As Robert K.
Merton ([1949.I 1968:71-72) put it, "The sociological analysis of qualitative
data often residcs in a private world of penetrating but unfathomable
insights and ineffoble understandings .... [I-Iowever,] sciencc .. , is pub-
lic, not private." Merton's statement is not true of all qualitative research-
ers (and it is unfortunately still true of some quantitative analysts), but
many procecd as if they had no method~sometimes as if thc use of ex-
plicit methods would diminish their creativity. Nevertheless they cannot
hclp but use some method. Somehow they observe phenomcna, ask ques-
tions, infer information about the world frcnn these observations, and
make inferences about causc and effect. If the method and logic of a re-
searcher's observations and inferences are left implicit, the scholarly com-
munity has no way 'of judging the validity of what was done. We cannot
evziluate the principles of selection that were used to record observations,
the ways in which observations were processed, and the !ogic by which
conclusions were drawn. We cannot learn from thcir methods or replicate
their results. Such rcsearch is not a public act. Whether or not it makes
good reading, it is not a contribution to social science.
All methods-whether explicit or not--have limitations. The advan-
tage of explicitness is that those limitations can be understood and, if pos-
sible, addressed. In addition, the methods can be taught and sharcd. This
process allows rcscarch results lo be comparcd ,:1eross separate rescarch-
ers and rescarch projecls sludies lo be replicated, and scholars to learn.
3. The conclusions are uncertain. ßy definition, inference is an imperfect
process. Ils goal is to usc quantitative or qualitative data to learn about
the world that produced them. Reaching perfectly certain conclusions
lntroduction 9
from uncertain data is obviously impossible. Indced, uncertainty ~s a cen~
trnl aspect of all research and all knowledge about thc world. Without a
reasonable estitnatc of uncertainty, <1 description of thc real world or an
inforence about a causal effect in the real world is uninterprctable. A re-
searcher who fails to face the issue of uncertainty directly is either assert-
ing that hc or she knows everything perfectI_r or that h_e ~)-~·._:h~\l~as. n~
idea how certain or uncertain the results are. btther way, mfc1ences with
out uncertainty estimates are not science as we define it.
4• The content is the method. Finally, scientific research adheres to a se~ of
rules of inference on which its validity dcpends. Explicating the :nost ,~n:-
portant rules is a major task of this book. 2 .The content_ of "sc1ence ~s
primarily the methods and rules, not the.subJect_matt:r, smc~ ~e c:n.,u~e
these methods to study virtually anythmg. Tlus p01~1t was 1e,~ogm~ed
over a ccntury ago when Karl Pearson (1892: 16) explamed that thc held
of scicnce is unlimited; its material is endless; every group of natural phe-
nomena, every phasc of social lifc, evcry stage of past or p1_·esent dev~Io_p~
ment is material for science. The unity of all science cons1sts alone m ils
method, not in its material."
These four features of scicncc havc a further implication: science at
its best is a social enterprise. Every researcher or temn of ~csearchers
labors under limitations of knowledgc and i~sight, and mtstakes are
unavmL · i able, yet· such crrors will likely be pomted out . by ·others.
· Un- ·t
derstanding thc social character of science cm~ ?~
hbcrahng s1nc? 1
1 111 eans that our work need not to bc bcyond cnt1c1s1n to n1ake an 11:n-
portant contribution-whcthcr to the description of a proble,m or, lt~
conceptualization, to theory or to the evaluabon of thcory. As long as
our wor k exp11c1 . ·tl y a ddresses
, , (or 'attcmpts to redirect) the .concerns
. f of
the con1munity of scholars and uses publ~c 1nethods to ~rnve at .111 er~
ences that arc consistent with rulcs of sc1ence and thc 1nforn1~h01: at
our disposal, it is likcly to make a contribution. An;~ tl~e _cont~1~,ut10n
of evcn a 111inor article is greater than that of thc great wm k that
stays forever in a desk drawer or within the confines of a computer.
LL, Scicnce and Complexity
Social science constitutes an attcmpt to 111akc sense of social ~ituations
lhat we perceive as 1nore or less cornple~. "W_e nee_d ,to__ r~co~rnz~, _:10~-
cver, that what wc pcrceive as complex1~y. 1s n~t ent~:cly 1,nhero~t 11~
phenomena: the wor1d is not naturally d1v1dcd 1nto simple and com
2 A!though WC do covcr thc vast majority of the importun[ rul~s of s~i~n:ifi~ in_f~rc: :c~
1
they me not rnmplcte. Indecd, rn~)St p_hil~>sophers agree th,it ,1 compktc, cxh,1ustlVl m
ductivc logk is impossible, cvcn m pnnnplc.
 rn The Scic11ce in Social Science
[ntroduction ll
plcx
t· scts_ of_ ewnts
·, ' · On- t!1c, con. t· raz•Y, t.11e pcrce1ved . complexity of a situ- course of events which is altered through 111odifications in one or more
~ ion _c1epen~is u~ _part on how we11 we GUl simplify reality, an'd' our 'condilions"' (Weber [19051 1949:173). Thc application of this idea in
~ap~czty_ to s1mp!1fy de~1ends on whether we can specify outcomes and a systematic, scientific way is illustrated in a particularly extre1ne ex-
exp analory vanables tn a cohcrent way. Having more observ"'t' ample of a rare event fron1 geology and evolutionary biology, both
may assist us in this b . ,, 1011s
·t ,, . ' ' _ L • proccss ut 1s usually insufficient. Thus "com Jfcx-
.'
historically oriented natural sciences. Stephen J. Gould has suggested
1 !! .", l"'.rtly condlfwnal on the state of our theory. / .
that one way to distinguish systematic features of evolution from sto-
Sc1ent1hc methods can be as valuable for intrinsicall
events as for simpler ones. Complexity is likel t k, ' Y. complex chastic, chance events may be to ü11agine what the world would be
~~, o ~1a _e_ our mferences ]ike if all conditions up to a specific point were fixed and then the rest
less certain but should not make the
and limited data should not cause
On the contrary· the b · t ff
l: ,
~~1:~:~~;)~~e~1t_1-~1c~.JJ~1ce~"tainty
. seien 1 1c research.
of history wcre renm. He contends that if it were possible to "replay
the tape of life," to let evolution occur again from the beginning, the
' . . igges payo for usmg the rules of scientif . world's organisms today would be a completely different (Gould
ference occurs prec1sely whe 1 d t . r . IC m-
flawed, measure1nents are ~11ncl:a; ~1~d u;:~te/, o~.ervation tools arc 1989a).
With clear relationships 'nd lt b'. e a wns ips are uncertain. A unique event on which students of evolution have recently fo-
na111 1guous d'lh. 111 t1 d b
. u
unportant, since even partially flawed rul,, f ~ 'f ~ 1,~ may e less cused is the sudden extinction of the dinosaurs 65 million years ago.
Gould (1989a:318) says, "we n1ust assume that consciousness would
answer~ that are roughly correct. es o m erence 1nay produce
not have evolved on our planet if a costnic catastrophe had not
Cons1der some complex, and in smne s , " . " .
enormous nunificatio11s Tl1 II , f ense umque, events with claimed the dinosaurs as victims." If this statement is true, the extinc-
" e eo apse o · the R E , tion of thc dinosaurs was as important as any historical evcnt for
French Revolution, the American Ci~il W W !~man •mp1re, the
caust, and the reunification of 'cerrnan .ar, c cor . "War l, the Holo-. hun1m1 beings; however, dinosaur extinction does not fall neatly into
such events. These events seem tob' t! ,Y 1'.1 11990 ate all ~xamples of a dass of events that could be studied in a syste1natic, comparative
of many forces whose , . . e 1e resu t (~f complex mteractions fashion through the application of general laws in a straightforward
taken place That . . c~n1unc~u1e appears crucial to the event having way.
forces co:1v~r red a:: ~: epen_ ently c~used s~q:iences of events and Nevertheless, dinosaur cxtinction can be studied scientifically: alter-
to bring abot~ the C'Vegn,vtsenbep1·I1,,1ce abnd tnnde, their interaction appearing native hypotheses can be developed and tested with respect to their
· · g o serve (H' J 1970) observable irnplications. One hypothcsis to account for dinosaur ex-
more, it is often difficult to b l'' , I .
products of large-scale historic:l'efv~ t
. . . ,
:~t usc unan
these events were inevitable
" mces. sorne seern to hav- :l"
. Further-
d d
tindion, developed by Luis Alvarez and collaborators at Berkeley in
the late 1970s (W. Alvarez 1990), posits a cosmic collision: a meteor-
m part, on 1d10syncracies of ersonalitie, . , . . ' e c .epen e '
ments. Indeed fr,,1n tl p . s, mstitutions, or social tnove- ite crashed into the earth at about 72,000 kilmneters an hour, creating
, 1e perspecttve of O tl · a blast greater than that from a full-scale nuclcar war. lf this hypothe-
seems to have played 1.f . ur 1eones, chance often
' a ro e. actors outside th, , f h sis is correct, it would havc the observable implication that iridium
provided crucial Jinks in the sequences of e t e scope o t e theory
One wa tO d . ven s. (an element com1non in meteorites but rare on earth) should be found
'Y un erstand such events is b , ",k' . .
conceptualizing each case , 8 b 'f Y see mg generahzat10ns: in the particular layer of the earth's crust that corresponds to sedi-
' ,, a n1em er o · a class of ~ , . f, b .
meaningful generalizations can be inade. This m:~~flcf.'-i aftut wh1ch ment laid down sixty-five million years ago; indeed, thc discovery of
iridium at predicted layers in the earth has been taken as partial con-
w~ll for ordinary wars or revolutions but s ' , o o en ~orks
b~m? n1uch rnore extre1ne than othe1~s ar~?,me_~,a~~ ~nd rev,olu.h~ns, firming evidence for the theory. Although this is an unambiguously
chstnbution. Furthermore' n,,t bl 1' outhers m the statisttcal unique event, thcre are many other observable implications. For one
, a e ear y wars or r, 1 t'
:1~~~'.~ :;~:~"ir et~:~.~~~:~ ~7: ex::
example, it should be possible to find the 111etorite's crater somewhere
i:;~r~!c~~h s;~~:::t:~:t ev/n;s of on Earth (and several candidates have already been found). 3
comparing them with th ~·. , . n h~t caubon is necessary in The issue of thc causc(s) of dinosaur extinction re1nains unresolved,
eu successors wh1ch 1nay be t , although the controversy has generated 1nuch valuable research. For
1
he product of imitation. Expanding tl~e 1 , , f ', o some extent
but it is not always appropriate. c ass o events can be useful, 3 Howcver, an alternative hypothesis, that cxtinction was caused by vokanic erup~

Another way of dealing scientifically with r _, 1 ' . , lions, is also consistcnt with thc prcscncc nf iridium, ilnd sccms more consistcnt th,m
to engage in countcrfactual analysis: "the n~1er1e,t,a]arge-scale events IS the rncteorite hypothesis with the finding that all thc spccies extinctions did not occur
construction of a simultaneously.
 12 The Science in Social Science
<~ur purposes, the point of this example is that scientific generaliza-
hons arc useful in studying even highly unusual events that do not fall
into a large dass of events. The Alvarez hypothesis cannot be tested
with referencc to a sd of common events, but it does have observable
ilnplications for other phenmnena that can be evaluated. We should
note, ho':ever, t.h~t a hypothesis is not considered a reasonably certain
explanahon unhl lt has been C:~valuated ernpirically and passcd a nuin-
b_er of de~nanding tests. At a minünu1n, its implications rnust be con-
s1stent wJth our knowledge of the external world; at best, it should
predict what In1re Lakatos Cl 970) refers to as "new facts," that is, those
formerly unobserved.
The point is that even apparently unique events such as dinosaur
~xtinction can be studied scientifically if we pay attention to improv-
mg theory data, and our use of the data. hnproving our theory
through conccptual clarification and specification of variables can
gcnerate morc observable implications and evcn test causal theorics
of unique events such as dinosaur cxtinction. lmproving our data al-
!ows us to observe 1nore of these observable ünplications, and ünprov-
mg our use of data permits more of thcse implications to be cxtracted_
from existing data. That a sct of events tobe studied is highly complex
does not render careful research design irrelevant. Whether we study
rnany phcnomena or few-or even one-·-the study will be improved if
we c.ollect data on as 1nany observable irnplications of our theory as
poss1ble.
1.2 MAJOR CoMPONENTS oF RESEARCH DESIGN
S?cial science research at its best is a creative process of insight and
d1scovery taking place within a well-established structure of scientific
inquiry. Th~ first-rate social ~cientist does not regard a research design
a.s a b~uepnnt for a 1nechamcal process of data-gathering and evalua-
t10n. lo the contrary, the scholar must have the flexibility of mind to
ov~rturn old ways. of looking at the world, to ask new questions, to
rev1se research des1gns appropriately, and then to collect rnore data of
a different type than originally intended. However, if the researcher's
findings are to bc valid and accepted by scholars in this field all these
revisions and reconsiderations 1nust take place according t~ explicit
prc~cedures consistcnt with the rulcs of inferencc. A dynainic proccss
of 1nquiry occurs within a stable structure of rules.
Social scientists often begin research with a considcred design, col-
lect son1e data, a~1d draw conclusions. But this proccss is rarely a
smooth one and 1s not always best donc in this order: conclusions
rarely follow easily fron1 a research design and data collected in accor-
Major Componenls of Research Design 13
dance with it. Once an investigator has collected data as provided by
a research design, he or she will often find an imperfect fit arr1.ong the
main research questions, the theory and the dala at hand. At th1s stage,
researchers often become discouraged. They rnistakenly believe tl1at
other social scientists find close, immediate fits bctween data and re-
search. 'T'his perception is due to the fact that investigators often take
down the scaffolding alter putting up their intellectual buildings, leav-
ing little trace of the agony and uncertainty of constructim:. Thus th.e
process of inquiry seerns morc mechanical and cut-and-dned than lt
~~~~. .
Some of our advice is directed toward researchers who are trymg to
make connections between theory and data. At tünes, they can design
more appropriate data-collection procedures in order to evaluate i:l
theory better; at other times, they can use the data. ~hey have an? recast
a thcoretical question (or even pose an entirely düferent quesbon that
was not originally foreseen) to produce a more important rescarch
project. The research, if it adhercs to rules of inference, will still be
scientific and producc rcliable inferences about the world.
Wherever possible, researchers should also improve their research
designs before conducting any field research. However,. data has a
way of disciplining thought. lt is extremely common to fmd Hlat the
best rcscarch design falls apart when the very first obscrvations are
collected-it is not that the theory is wrong but that the data are not
suited to answcring the questions originally posed. Understanding
fron1 thc outset what can and what cannot be done at this later stage
can help the researcher anticipate at least some of the problerns when
first designing thc research. . .
For analytical purposes, wc divide all research des1gns 1nto f~)ur
components: the research question, the theory, thc data, and the use oj" thc
data. These components are not usually developed separately and
scholars do not attend to then1 in any preordained order. In fact, for
qualitative researchers who begin their field work before choosing a
precise research question, data comes first, followed ~y ~he otl:ers.
Howevcr, this particular breakdown, which we explam m sect~ons
1.2.1-'J.2.4, is particularly useful for understanding the nature ~t- re-
search designs. In order to clarify precisely what could be done If re-
sources were redirected, our c1dvicc in the renrninder of this section
assumes that rescarchcrs have unlimited tin1c and resources. Of
course, in any actual research situation, onc must always m_ake com-
prmniscs. Wc believe that undcrstanding the advice in the tou: cat~-
gorics that follow will hclp rcsearchers n1ake thesc com.prom1ses ~n
such a way as to ünprove their research designs most, even when m
fact their rescarch is subject to external constraints.
 14 Thc Sciencc in Social Science
M<1jor Cornponcnt8 of Research De8ign 15
1.2J lmproving Research Questions . , ,· _, t'st and latet~ to choose a particular research
to be~orn~c~ ~octl 1 tf1~yc1:1~ay co1{<;titute the "real" rcc1sons for engaging
Throughout this book, we consider what to do once we identify the ,1uest1011. s Suc , . d ·. t ly so ßut no matter
object of research. Givcn a rcsearch qucstio11, what are the ways to . articular research proJect-an appropna e , : ' . .
in a p, ersonal or idiosyncratic lhe reasons for choosmg a top1c, thl~
conduct that research so that wc ca11 obtai11 valid explanatio11s of social
how p . d rules of inference discussed m th1s book w1
a11d political phe11ome11a? Our discussio11 begi11s with a research qucs- inetho~i' ~!r:c:~:::1~ore ~owerful research designs. Frorn the per-
tio11 and then proceeds to the stages of designing and conducti11g the help sc 10 '. 'ntial contribution to social science, personal rea-
research. But where do research questions originate? How does a spect1ve ot .a ~ote_ -~, ar nor sufficient justificatio11s for the chmce
scholar choosc the topic for analysis? There is no simple answer to this sons are. neither n~cess\ ,y tl - -hould not appcar in our scholarly
question. l..ike others, Karl Popper Cl 968:32) has argued that "there is 0 f a top1c. In most cases, 1ey s . I s
. mos t d'lfCC t!y but quite indehcate y, 110 I 011e care,n
:;~,lt
'
'tings "fb put lt t
no such thing as a logical method of having new ideas .... Discovery
contains 'an irrational eleme11t,' or a 'creative intuition."' The rules of ~~ think-the scholarly community only cares w 1a. we ca
choice at the earliest stages of the research process are less formalized den1onstrate. ,· t pic do not exist there are
than are the rules for other research activities. There are texts on de- Though precise rules for c11oosmg a o . . ' l : , likel
ways-beyon . d 111. d'IV!'d lkal preferences-of detenmnmg ·tt 1eIdeallyy
signing laboratory experiinents on social choice, statistical criteria on
\ , of a research enterprise to the scholar1y commum y. . ' . '
drawing a sample for a survey of attitudes on public policy, and man- va
8 ue ' ~ . . l ocial sciences should satisfy two cntena.
uals on co11ducting participant observation of a bureaucratic office. But
there is no rule for choosing which research project to conduct, nor if
'.·e:e~;~):r~l',';;;;
,ll a
1~:/~1i:,:,~/pos: question that is ''.itnport~nf' in the :""'.
Ftrst, . -I ld be consequential for pohtical, social, or eco
we should decide to conduct field work, are there rules governing
where we should conduct it. world. ~l~e t,;~~c ~~~~rstanding something that significantly affelcts
1101111c 1 e, f d . t ding and predtctmg events t 1at
We can propose ways to select a san1ple of comn1unities in order to rnany people's lives, or or uni e1s Sa111·vel '199()··15) Second, o research
f I b , ,ficia (sec 11 · Y · ·
study the ünpact of alternative educational policies, or ways to concep-
rnig'.~~ ~1~}~~,~':,:;,e ,~rsp:~ific c;,t!;i/,ution to an identifi~/Jle scholarfl~ l:te;":
tualize ethnic conflict in a n1cmncr conducive to the forrnulatio11 and
pro~c~ ,1. zcreasing our collcctive ability to construct verif1ed sctentt tc .cxp a
testing of hypotheses as to its incidence. But there are no rules that teil
us whether to study educational policy or ethnic conflict. In terms of ::~~:0,(s ;/ some aspect of the world. This latter critedon ~e·s, n;: f :;:i~
social science methods, there are better and worse ways to study the that all research that contributes tok~u: ::~1:l~i°i~;;:~c;;1e~~)lnetin1es
collapse of the East Gerrna11 government in 1989 just as there are better t' . fact aitns directly at n1a ing ' ' ,. d
na wns :n /1 'wledge in a Held is such that much fact-findi11g an
and worse ways to study the relationship between a candidate's posi- the stat~ o .(110 ·' e can t·1.ke o11 the challenge of expla-
tion on taxcs and the likelihood of electoral success. But thcre is no deseript1on 1s needed before w ' . ' · ·t ·11 be descriptive
'b · f smgle pro1ec w1
way to determine whether it is better to study the collapse of the East nation. Often the contn utionl o a t e~en be descriptive i11ference
Gennan regime or the role of taxes in U.S. electoral politics. · f s etünes the goa may no · '
m erence. l ~ om ·11 be the close ob serva 11011 . of particular
< '
events or the. sun1-
•
The specific topic that a social scientist studies may have a personal but rat lferl :,"t~ . al detail These however, meet our second cntenon
and idiosyncratic origin. It is no accident that research on particular mary o us onc, , · , , .
groups is likely to be pioneered by people of that group: women have l rec1uisites to explanat10n.
because t 1ey are .pre . ' . . tl ·' t' to the real world of polities
often led the way in the history of women, blacks in the history of Our first cntenon d1rects ow a en 1011 . . I d f the
. ~ d t the current and h1stonca recor o
blacks, immigrants in the history of im1nigration. Topics 1nay also be and social phenomena an. ,~ eo le's lives. Whether a research
influenced by personal inclination and values. The student of third- events and pr,oblern~. tha\:i~1~s:J1tiJI a societal judgment. The sec-
world politics is likely to havc a greatcr desire for travel and a greater questJon meets th1s c11teno . I Y, . 1 I rly literature of socrnl
tolerance for difficult living conditions than the student of congres- ond criterion directs our attentlon to t 1e sc 10 a :! t uzzles that re-
. II t
science, to thc mte ec ua P '-: ~ , . I uzzles not yet pose,, 0 P ·,
, , d ,t} d, available
sional policy making; the analyst of international coopcration 111ay
have a particular distastc for violent conflict. main to be solved, a11d to the sc1entific theones an me 10 s'
These personal experiences and valucs often provide the n1otivation to solve them. . . f d' sub'ect matter that
Political scientists have no d1fhculty m mg , J
 Major Cornponents of Rcsemch Design 17
·16 Thc Scifnce in Social Science
Choose an (lCCeptcd hypothcsis in thc literaturc t.hat we susp.cct _!s ·fo.lse
meets our first critcrion. Ten major wars during the last four hundrcd Z. (or one wc bdieve has not been adequately conhrrncd) ~md 11we_stigalc
yenrs lrnvc killed almost thirty million people (Lcvy J 985:372); some l ,lher it is indeed falsc or whcther some othcr theory 1s corrcct.
"limited wc1rs," such as those betwccn the United Statcs and North :t::mpt to rcsolve or provide further evidence of onc side of a .~ontro-
Vietnam and between [ran and Iraq, havc each clailned over a million · t·l1e t·tei·atttre-perhaps
J. versy ll1 demonstratc that the controvc1sy was
1 ,
lives; and nudear war, were it to occur, could kill billions of human
unfounded from the start. . .· , .
beings. Political n1ismanagement, both do1nestic and international, has ·
4. Des1gn resecat·cl"', to illuminate or evaluate unquest1oned (1Ssun1pt1ons m
led to economic privation on a global basis-as in the l 930s--as weII
the literature. . d
as to regional and local depression, as evidenced by the tragic expe- S. Argue that an important topic has be~n overlooked in .t~1e ltterature an
riences of much of Africa and Lfltin Americc1 during the 1980s. In then proceed to contribute a systematic study to the mea. . . .
general, cross-national variation in political institutions is associated Show that theories or evidence designed for some purp~)S~ m on~.hte~a-
with great variation in the conditions of ordinary human life, which 6. ture could be applied in another literature to solve an ex1stmg but dppar-
are reflected in differences in life expectancy and infant mortality be-
ently unrelated problem.
tween countries with similar levels of econornic development (Russett
1978:913-28). Within the United States, programs designcd to alleviatc Focusing too inuch 011 mal<ing a contribution to a scho~arly litera-
poverty or social disorganization seern to have varied greatly in their ture withottt somc attention to topics that. ha:'e :~al-world i1:1p~rtanc~
efficacy. lt cannot be doubted that research which contributes even runs the risk of descending to politically ms1gmfic~nt questions. Co.n
1narginally to an understanding of these issues is important. , l attention to the current political agenda withont reg~rd. to 1s-
While social scientists have an abundance of significant questions verse y, ' b']'t f b1'ect to systematic study withm the
sues of thc ainena 11 y o a su d ..,[ , ..
that can be investigated, the tools for understanding them arc scarc_e ramewor k o f a
( body of social science knowledge
. lea s to cmc ess
f
and rather crude. Much has been written about war or social misery work that adds Jittlc to our deeper undcrstandmg. , .
that adds little to the understanding of these issues because it fails Our two criteria for choosing research queshons are not ne~essanlj
either to describe these phenomena syste1natically or to makc valid · ·t· t one a11 othcr In the long run, understand1ng rea -
111 oppos1 1011 o c • • d l f of
causal or dcscriptive inferences. Brilliant insights can contribute to un- 1i henmnena is enhanced by the generat1011 an eva ua 1011
dcrstanding by yiclding intcresting new hypothcses, but brilliance is worl c Pt hypotheses through the use of the scientific method. But
exp ana ory ' · · b t'cal use
not a method of empirical research. All hypotheses need to be evalu- in the short term, there rn~y ~e. a contradict1c:n etw~en prt~ 1 ((199Ü)
ated empirically beforc they can makc a contribution to knowledge. fulness and long-tenn scient1hc value. For 1nsta:1ce, Man 1w .
This book offers no ad vice on becoming brilliant. What it can do, how- points out that macroeconornic theory and apphed rnacroeco~o~11cs
evcr, is to emphasize the ü11portancc of conducting research so that it . _ d sharply during the 1970s and 1980s: models that ha
d 1verge een
constitutes a contribution to knowledge.
, '
shown to be thcoretically incoherent were st1 use . o
·n . d t forccast - '~· , "
the
Our second criterion for choosing a research question, "making a . t·
d 1rec f th _, u s econmny while the new theoretical models dc-
1011 o e · · ' . · , d "P not suffi-
contribution," means explicitly locating a research design within the signed to correct these flaws re1nmne~ spec~11at1ve an we c
framework of thc existing social scientific literature. This ensures that ciently refined to 1nake accurate pred1ct10ns. .
thc investigator understand the "statc of the art" and n1inirnizes the The criteria of practical applicability to the real world and contribu-
chance of duplicating what has already been done. It also guarantecs tion to scientific progress n1ay seem opposed to. o11e a~1oth~~- when {1
that the work done will be important to others, thus improving the researcher chooses a topic, Some resear~l:ers will beg1n wlt~ a re~ -
success of the comn1unity of scholars taken as a whole. Making an ex- world problen1 that is of great social sigmficance: th~ t.hre~~ ?f 11ucl~:~
plicit contribution to the literature can be done in 111any different ways. the incorne gap between n1en and women, the transit10n to
We !ist a few of the possibilities hcrc: war, . 1 Eastern Europe. Others may start with nn intelledual prob-
mocracy 11 .,, t i · t· betwccn
len1 encratcd by the social science literature: a con Ta( 1c 1011 -.
'1. Choose a hypothesis scen as important by scholars in thc liternture but s~ve~al experimental studies of dccision-makit:g under .u1:ccrt~1n~y.~ot;
for which no one has completcd ,:1 system.atic study. If wc find evidencc an inconsistency between theories of congress10na~ vo.tu:g an r~ce,n
in favor of or opposed to the favored hypothcsis, wc will be making a ·
e Iechon ou t cmn es-, . 1'1,e. distinction
. between the cntena 1s, of course,
contribulion.
 18 The Scie11ce in Sociol Science
not hard and fast. Some research questions satisfy both criteria from
the beginning, but in dcsigning rcsearch, researchcrs often bcgin
nearer one than the othcr. 4
Wherevcr it begins, the proccss of designing research to answer a
specific question should move toward the satisfaction of our two cri-
teria. And obviously our direction of 1noven1ent will depend on where
we start. If we are 1notivated by a social scientific puzzle, we must ask
h.ow_ t.o make that_ research topic morc relevant to rcal-world topics of
s1g111ficance-for mstance, how rnight laboratory experimcnts bctter
illurninate rcal-world strategic choiccs by political decision-n1akers or,
what behavioral consequcnces might the theory have. If we begin with
a real-world problem, we should ask how that problern can be studied
with modern scientilic methods so that it contributes to the stock of
social science explanations. lt may be that we will decide that moving
too far frmn one criterion or the other is not the most fruitful approach.
Laboratory experimenters may argue that the search for external refer-
ents is prematur~ and that n1ore progress will be 111ade by refining the-
ory m:d method m the more controlled environment of the laboratory.
And m terms of a long-term research program, they may be right.
Conversely, the scholar motivated by a real-world problem 1nay argue
that accurate description is nceded beforc 1noving to explanation. And
such a rc~earcher may also be right. Accurate description is an in1por-
tant step m explanatory research programs.
In either case, a research program, and if possible a specific research
projcct, should aim to satisfy our two criteria: it should dcal with a
significant real-world topic and be designed to contribute, directly or
indirectly, to a specific scholarly litcrature. Since our rnain concern in
this .book is making qualitative research more scientific, we will pri-
manir address the researcher who starts with the "real-world" per-
spective. ßut our analysis is relevant to both types of investigator.
If we begin with a significant real-world problern rather than with
an established literature, it is essential to devise a workable plan for
studying it. A proposed topic that cannot be refined into a specific research
project permitting valid descriptive or causal inference should be modified
afong the way or abandoned. A proposed topic that will make no contri-
4 The dilcmina is not unlikc that foced by natural scicntists in deciding whether to
conduct applied or basic rcseurch. For exarnplc, c1pplicd rcsemch in relation to a particu-
!ar drug or disease may, in the short run, improve mcdical care withou! contributing as
rnuch to thc general knowkdgc of the under!ying biologic<1\ mechanisms. B<1sic rcscarch
rnay have thc opposite consequence. Most rcscmchcrs would arguc, as wc do for thc
social scie1!ccs, th~t the dichotomy is false <rnd that basic rescarch will ultimately lcad to
the powertul apphcd results. [-Iowever, <1ll agrec that the best research design is onc that
somehow maiwges both to be dircct!y relevant to solving rcal-world problems 811d to
furthcring thc go<1ls of a spccific scientific litcrature.
M'1jor Componcnts of Resecuch Design ·19
bution to somc scholarly literature should ~ünilarly l~e cl:a~1g.c(,:·, f~Ia~,-
ing tentatively chosen a topic, we enter n dialogue wlth the_ litercllure.
What questions of interest to us have already been answcrcd? f-I~w
' fJose and refine our c1uestion so that it seen1s capable of bemg
can We , .l b . . .
answered with the tools available? We 1nay start wit.1 a urmng ISSt~e,
~ut wc will have to come to grips both with thc hterature of socrnl
science and the problems of infcrence.
1.2.2 lmproving Theory
A social science theory is a reasoned and precise speculation about the
answer to a research question, including a statement about why the
proposed answer is correct. Thcories usually imply se~.era: mor~ ,s\e-
cific dcscriptive or causal hypotheses. A theory rnust be c01:s1stcnt
with prior evidence about a research question. "A the~)ry that 1.g:1~res
existing cvidence is an oxymoron. If we had the eqmvalcnt of tlut~1
in advcrtising' legislation, such an oxy1noron should not be called a
thcory" (Licbcrson 1992:4; see also Woods and Walton 1982).
The developmcnt of a theory is often presented. as thc fust stcp of
rescarch. Tt smnetimes comes first in pract1cc, but lt need not. In fad,
we cannot develop a theory without knowlege of prior work on thc
subject and thc collection of some data, since .even thc r~scarch ques-
tion would be unknown. Neverthelcss, dcspitc whatever amount of
data has already been collected, there are some ge1:eral _ways to evalu-
ate and improve the usefulness of a theory. We bnefly introducc each
of these here but save a more detailed discussion for later chapters ..
.First, choose theorics that could be wrong. Indee~i, vastly more 1s
learned fn_)n1 theories that are wrong than from theones that are stated
5
so broadly that they could not be wrong even in principk. Wc need to
be able to givc a direct answer to the questlon: What evtd~nce wo.uld
convince us that we are wrong? 6 If there is no answer to th1s quest10n,
then we do not have a theory. .
Second to make sure a theory is falsifiable, choose one that 1s cap~-
blc of g:nerating as 111any observabfe implications as possible. Tlus
choice will allow 1nore tcsts of thc theory with more d~-~a and a gr.eatcr
variety of data, will put the theory at risk of being fals111ed more hmes,
and will makc it possible to collect data so as to btnld strong ev1dence
for the theory.
'.i This is thc principle of falsifiability {Popper 1968). It is an issu~ on whi.ch there c~re
v,uicd positions in the philosophy of sdence. However, vcry fcw otthem d1sc1grce wtth
thc principle tlwt theories should be statcd dcarly ct~ough ~o t~1al ti1~~
~~)~;\d bt: w,r~-n~~
6This is probably thc most comrnonly askcd qucstion ,1t JOb mtc1vw1,,, s 111 out dq<1i l
tnent and many others.
 20 Thc Sciencc in Socia[ Scicncc
Third, in. dcsigning thcories, bc as concrctc ;Js possible. Vaguely
stated theones and hypothescs servc no purpose but to obfuscate. The-
ories that are stated precisely and rnake spccific predictions can be
shown rnon) casily to be wrong and are therefore better.
Smne rcsearchers rccornmend following the principle of "parsi-
rnony." Unfortunately, the word has been used in so rnany ways in
casual conversation and scholarly writings that the principlc has be-
come obscured (sec Sober [19881 for a complete discussion). The clear-
est definition of_ parsimony was given by Jeffrcys (1961:47): "Simple
~hconcs have lugher pnor probabilities. 07 Parsirnony is therefore a
Judgment, or even assurnption, about the nature of the world: it is as-
s~1mcd to be ~imple. The principle of choosing theorics that ü11ply a
snnple world 1s a rule that clcarly applics in situations wherc there is
a high dcgree of certainty that the world is indecd simple. Scholars in
physirnseem to find parsimony appropriate, but those in biology oftcn
tlunk ot lt as absurd. In the social scienccs, smne forcefully defend par-
simony in their subfields (c.g., Zcllner 1984), but we believe it is only
occas1onall~ appropriatc. Given the prccise definition of parsilnony as
an assumphon ~1b?ut thc wor~d, we should never insist on parsünony
as a gencral pnncrple of des1gning theories, but it is useful in those
situations where wc have s0111e knowledgc of the sü11plicity of the
world we are studying.
Our po.int is that ~e do not advise researchers to seck parsimony as
an essential good, s1ncc thcrc seems little reason to adopt it unless we
alre~dy know a lot about a subject. We do not even nced parsin1ony to
avo1d ex_ccssively complicated theorics, since it is dircctly implied by
the maxun that the thcory should be just as cmnplicated as all our
cvidence suggest. Situations with insufficient evidence relative to the
complexity of the theory being investigated can lead to what we call
"indetcrminc1te rese~rch designs" (see section 4.1), but these are prob-
len1s of rescarch des1gn and not assumptions about the world.
All our ad vice thus far applies if wc have not yet col!ected our data
and begun any analysis. Howcver, if we have already gathcred the
data, wc can ccrtainly use these rules to modify our theory and gather
ncw data, and thus gcnerate new obscrvable implications of the new
theory. Of coursc, this process is expensive, time consuming, and
probably wasteful of the data already collccted. What then about thc
situation where our thcory is in obvious need of improven1ent but
we cannot afford to collect additional data? This situation~in which
researchers often find themselves~demands great caution and self-
7
This phrase hns cornc to bc known as the "Jeffrcys-Wrinch Simplicity Postulate." Thc
conccpt is simllar to Occmn's wzor.
Mnjor Components of Research Design 2·1
. · t Any intelligent scholar can come up with a "plausible" the-
rcstram · , - --~, I
ory for any set of data aftcr thc fact: yct to do so .de1:10n~trat~s _1:c~: 1-
. b t tl,c' vcncity of the theory. ,he theory will fit thc data rncely
=·
111g i:l Ol1
l
- '
.
,[i]] may be wildly wrong---.jndced, demonstrably wrong w1t 1
. ,t other data. Human bcings are very goou at recogmzmg pa - , •.• .
~
' 'n ,see
· ]
ntos
b t not very good at rccogmz1ng . . nonpattcrns. (M os.- t o- f us, eve
utterns in random ink blots!) Ad hoc adjustments 111 a thcory that
~~es not fit existing data must be used rarcly and with considerable
. 1· . 8
discip me. .. . ·l , :l - .
There is still the problem of what to do when w.e have .ruus 1eL orn
data collection and analysis and wish to work on nnpr?vm~ a thcory.
In this situation, we recomrnend following two rules: hr.st: 1f our pre-
diction is conditional on several variables and we ~~e wilhng to ~~op
one of the conditions, we may do so. For examplc, lt w~ hypothesized
originally that democratic countries with advanc:d .socrnl wdfarc ~ys-_
tems do not fight each other, it would be pernussiblc to extend tha t
h othesis to all n1odern dcmocracics and thus evalu~t: our theory
ar!'inst rnore cascs and in~rease its chances of being _fals1fH~d. :hc ~en-
eral point is that after see111g the data, we n1ay 1nod1fy orn. theory 111 a
way that makes it apply to a larger range of pheno1~~na .. S111ce su~l'. an
alteration in our thesis exposes it more fully to fals1hcat10n, 1:1od1fica-
. 1 in this direction should not lcad to ad hoc explanatlons
t101 , , . that
mcrely appear to "savc" an inadequate theory by res.tnctmg Its ~an~c
to pheno1ncna that have already been observc~ tobe 111 ~ccord ~ith lt.
The opposite practice, however, .is gcncrally ma~p:opnate. -~tter ob-
scrving the data, wc should not Jl1St add a .r~str~chve cond1tion and
then procecd as if our thcory, with that quahficat1on, has bcen s.hown
to bc correct. If our original theory was that modern democrac1es d~J
not fight wars with one anothcr due to their constitutional systen1s, lt
would bc lcss pen11issible, having found exccptions to our. "rule," to
restrict the proposition to democracie~ with .adv,~nced socia_l welfare
systems oncc it has been ascertaincd by mspcctwn of thc data that such a
qualification 1.oould appear to 1-nake our proposiUon correct. Or suppos~ ,that
our original theory was that rcvolutions only occL~r ~rndcr con~ttmns
of sevcre economic dcprcssion, but we find that th1S 1s not truc 111 one
of our case studies. [n this situation it would not bc reasonabl~ merely
to add genen11 conditions such as, rcvolutions never occur d~1~1ng pen-
ods of prosperity except when the military is weak, the polll1cal lcad-
ership is repressive, the cconomy is based on a sn1all nun1ber of prod-
B [f wc have chosen a topic of rcal-workl importancc '.rnd/or c:nc \-.vhich mnh'-: sc'.nw
conlribulion to ,1 scholnrly literaturc, thc sudal 11<1lurc ot academia will corrcd tlus situ-
atic>n: soinconc will rcplicate our study with anothcr ;,et of dMa c1nd dcmonstrntc thcü
\.VC wcrc wrong.

22 · The Sciwce in Soda! Sciencc
ucts, and the climate is warm. Such a formulation is rnercly a foncy
(and rnisleading) way of saying "my thcory is correct, except in coun-
try x." Since we havc already discovered that our theory is incorrcct
for country x, it does not hclp to turn this falsification into a spurious
generalization. Without efforts to collect new data, we will have no
admissible evidence to support the ncw version of the thcory.
So our basic rule with respect to altering our theory a.Her observing
the data is: ·we can ,nake fhe fhcory lcss rcstricfive (so that it covers a broader
range of phenotnena and is exposed to nwre opportunitics for fals1fication),
lmt ·we should not nwkc it rnore restrictive withouf collecting neru data to test
the ncw version of the theory. If we cannot collect additional data, then
we are stuck; and we do not propose any mngical way of getting un-
stuck. At smne point, deciding that we are wrong is best; indeed, nega-
tive findings can be quite valuable for a scholarly literature. Who
would not prefer one solid negative finding over any nwnber of flin1sy
positive findings based on ad hoc theories?
Moreover, if we are wrong, we need not stop writing after ad1nitting
defeat. We may add a section to our article or a chapter to our book
about future cmpirical rcsearch and current theorctical speculation. In
this context, we have considerably morc freedon1. We may suggest ad-
ditional conditions that might be plausibly attached to our thcory, if
wc believe they might solve the problcm, propose a modification of
another existing theory or propose a range of entirely different theo-
ries. In this situation, we cannot conclude anything with a great deal of
certainty (except perhaps that the theory we stated at the outsct is
wrong), but we do have the ]uxury of inventing new research designs
or data-collection projects that cottld be used to dccide whether our
speculations are correct. These can be very valuable, especially in sug-
gesting areas where future rcsearchers can look.
Adrnittedly, as we discussed above, social science does not operate
strictly according to rules: the need for creativity sometünes mandates
that the textbook be discarded! And data can discipline thought.
Hence researchers will sornetimes, after confronting data, have inspi-
rations about how they should have constructed the theory in the first
place. Such a rnodification, even if restrictive, may be worthwhile if we
can convince ourselves and others that modifying the theory in the
way that we propose is smnething we could have done before we col-
lected the data if we had thought of it. But until tcsted with new data,
the status of such a thcory will rcn1aü1 very uncertain, and it should be
labeled as such.
One in1portant consequence of thesc rules is that pilot projccts are
often very useful, cspccially in research where data must be gathered
by interviewing or othcr particularly costly 1neans. Prcliminary data-
gathering may lcad us to alter the research qucstions or modify the
Major Cornponents of Research Design 23
thcory. Then new data can be gathcrcd to lest the ncw theory, and the
problcn1 of using the samc data to generate and test a theory can be
avoided.
J.2.3 f,nproviug Data Quality
"Data" are systematically collected elemcnts of information about the
world. They can be qualitative or quantitative in style. Sometimes data
are collected to evaluate a very spccific thcory, but not so infrequently,
;cholars collect data before knowing precisely what they are interested
in finding out. Morcovcr, even if data circ collectcd to evaluate a spc-
cific hypothesis, researchers may ultimately be interested in questions
that had not occurred to thcm previously.
[n either case-when data are gathered for a specific purpose or
when data are used for some purpose not clearly in mind when they
were gathered-ccrtain rules will in1prove the quality of thosc data. In
principle, we can think about these ~ules fo~ ilnproving data se~arately
frorn the rules in section 1.2.2 for nnprovmg theory. In pract1ce any
data-collection effort requires srnnc degree of theory, just as fonnulat-
ing any theory requires s01ne data ~see.Coom~s 1964)'. .
Our first and n1ost in1portant gu1dehne for nnprovmg data quahty
is: record and report the process by which the data are generated. Without
this inforn1ation we cannot detennine whether using standard proce-
durcs in analyzing the data will produce biased infcrcnces. Only by
knowing thc process by which the data were generated will webe able
to produce valid descriptive or causal inferences. In a quantitative
opinion poll, recording the data-generation process requires that we
know the exact method by which the sa1nple was drawn and the spc-
cific questions thcit were asked. In a qualitative comparative case
study, reporting the precise rules by which we cho?se the s~nal~ nu11:1-
ber of cases for analysis is criticaL We give addit10nal gu1delmes m
chapter 6 for case selection in qualitative research, but even more im-
portant than choosing a good n1ethod is being. ccireful ~o record and
report whatever method was used and all the 1nformat10n necessary
for son1eone else to apply it. 9
In section 1.2.2 we argued for thcories that are ccipable of genernting
9 We find that 111,;my graduate students arc unnecessarily afraid of sharing dat.:1 and
thc information neccssary to replicate their results. Thcy are afrnid thnt sonwonc will
steul thcir hard work or cven provc that thcy wcre wrong. These are <lll common fcms,
but they <lrt' <1lmost <1lwc1y:, unwarranted. Publinüion (or <1t least sending copi<.':l ?f re-
search pcipers to othcr scholms) m1d shc1ring data is the best w<1y to gmmmtcc crc\ht for
one's contributions. Morcovcf, shming datil will only help othcrs follow nlong m thc
rcscarch you startcd. Whcn their rcscnrch is publishcd, thcy will citc yoor cffort m1d
a<lv,1ncc your visibility and rcputation.
24 Thc Scie/lcc in Soda! Scicncc
many observable implications. Our scccmd guidcline for improving
d~ltJ qua!ity is in order bef/:er to evaluale a theory, collecl dnl'a on as nmny
of its ohscrvable implications as possil,ic. This means collccting as much
data in as many diverse contexts as possiblc. Each additional irnpli-
cation of our thcory which we observc providcs another context in
which to cvaluatc Hs veracity. The more observable implications
which i:lre found to be consistent with thc thcory, the morc powcrful
the cxplanation and thc more certain the results.
When adding data on new observable implications of a theory, we
can (a) collect rnorc observations on the same depcndent variable, or
(b) record additional dependent variables. We can, for instance, dis-
aggregate to shortcr time periods or smaller geographic areas. We can
also collect information on dependent variables of less direct interest;
if the results are as the thcory predicts, we will have n1orc confidence
in the thcory.
For cxample, consider the rational deterrence thcory: potential initi-
ators of warfare calculatc the costs and bencfits of attacking other
statcs, and these calculations can be influenced by crcdible threats of
retaliation. The most direct tcst of this theory would be to asse_ss
whether, given threats of war, dccisions to attack are associated with
such factors as thc balnnce of military forces between the potential at-
tacker and thc defender or the interests at stake for the defender (Huth
"1988). However, even though using only cascs in which threats are
issued constitutes a set of observable implications of the theory, they
are only part of thc observations that could be gathered (and used
alone may lead to sclection bia::;), since situations in which thrcats
then1selves are deterred would be excludcd fr0111 the data set. Hencc it
might be worthwhile also to collect data on an additional dependent
variable (i.e.f a different set of observable implications) bcised on a
measurcn1ent of whethcr threats are made by states that have some
inccntives to do so.
Insofar as sufficient good data on deterrence in international politics
is lacking, it could also be helpful to lest a different theory, onc with
similar motivational assumptions, for a different dependcnt variable
under different conditions but which is still an observable implication
of thc samc theory. For instance, we could construct a laboratory ex-
perimcnt to sec whcther, under simulated conditions, "thrcats" arc de-
terred ra.ther than accentuated by nlilitary power and firm bargaining
behcivior. Or we could examine whcther other adors in analogous sit-
uations, such as oligopolistic firms competing for market sharc or or-
ganized-crime fomilies cornpeting for turf, use detcrrence strategies
and how successful thcy are under varying conditions. Indecd, econo-
mists working in the field of industrial organization have used non-
Major Components of Research Design 25
coo P''f"tive
.:. ,, nv·ame theory, on which detcrrcncc theory. .also rdics, . to
·tudy such problems as entry into markets and pncmg stratcg1es
~Fudenberg and Tirole 1989). Given tl.1e close sitnilarit~ bctwc~n. th~
thcories, cn1pirical evidence supportmg gm:1~ . theo_ry s pred1chons
bout firm behavior would increase the plaus1b1hty ot related hypoth-
:scs about statc bchavior in international politics. Uncertaint:y would
remain about thc applicability of conclusions frorn one domam to a~1-
other, but thc issuc is important cnough to warrant attempts to gam
insight and evidence wherever thcy can be found. .
Obviously, to collect data forever without doing any analys1s would
preclude rather than facilitate completion of useful ~escarch. In pr.ac-
·ce Jimited th11e and rcsources will always constram data-collechon
t1 ; . .
efforts. Although more information, additional cascs, extra ~ntcrv:ewsf
another variable, and other relevant forms of data collechon will ~1-
ways ünprovc the certainty of our inferences to S~)n1e deg~ee, pr0111~s-
ing, potential scholars can bc ruined by too n111ch mformatlon ~s caSI~Y
as by too Jittle. lnsisting on rcading yet another book or gdt1~g still
one more data set without ever writing a word 1s a prcscnpt10n for
bcing unprodudive. . .
Our third guideline is: tnaximize thc valuftty of our rne~sure1nents. Va-
lidity refers to measuring what we think we are 1neasunng. The une1n-
ployment rate may be a good indicator o~ t~1c st~te of the e~o1:mny, ~ut
the two are not synonyme.ms. [n generat tt 1s eas1cst to max1m1ze vahd-
ity by adhering to the data and not allowing unobserved or unmeas:lf-
able concepts get in the way. If an informant. responds to ~ur qucst10n
by indicating ignorance, thcn we know hc satd that he was ignorant. ~f
that, we havc a valid 1ncasurement. However, what he really 1nea.nt 1s
an altogether different conccpt-·onc that cannot be n1easured w1tl: a
high degree of confidence. For example, in countrics wit~1 reprc~s.1ve
governme11tsf exprcssing ignorance n1ay be a w.a~ of makmg a c~It~c~l
political statement for some pcople; for othersf it 1s a way of saymg I
don't know."
Our fourth guideline is: ensure that data-collcction methods are reliable.
Reliability 1neans that applying thc same procedur~ in the sa111e "':'?
will always produce the same measure. When a reha?le proccdur~ 1s
applied at different tit11es and nothing has happened m ~he mcanhmc
to change the "true" statc of thc objcct we are n1casunng; the same
rcsult will bc obscrved. 10 Reliable measures also produce the same re-
w We mn check rcli«bility oursclvcs by mc,:1suring thc samc quantity twicc and scc'.ng
whcther thc me«sures arc the senne. Sornctirncs this sccrns easy, such a~ litcrally ask:ng
the s«mc question at different times during an interview. Howcvc1~ uskmg the ques.tion
oncc mciy influcncc thc rcspondcnt to rcspond in a consistcnt ü~shion the sccond tmw,
so we nced tobe cmcful that thc two mcasurcmcnts arc indccd mdependent.
26 The Sci('l!cc in Socia! Science
sults whcn applicd by different rcsearchers, and this oulcomc dc-
pends, of coursc\ upon thcre bcing cxplicit procedures that can be fol-
lowed.11
Our final guidcline ü,: all data and analyses should, insofar as possible,
be repficah/e. Replicability applics not only to data, so that we can see
whether our measures are reliable, but to thc entire rcasoning process
used in producing conclusions. On the basis of our rcsearch report, a
new researcher should be able to duplicate our data and trace the logic
by which we reached our conclusions. Replicability is irnportant even
i/ no one actually replicates our study. Only by reporting the study in
sufficient detail so that it can be rcplicated is it possible to evaluate the
proccdures followed and rnethods used.
Replicability of data rnay be difficult or impossible in sorne kinds of
research: intervicwces may die or disappear, and direct observations
of real-world cvents by witncsses or participants cannot be repeated.
Replicability has also cmne to 111ean different things in different re-
scarch traditions. In quantitative research, scholars focus on repli-
cating the analysis after starting with the samc data. As anyonc who
has ever tried to replicate the quantitative results of even prmninc_nt
published works knows wcll, it is usually a lot harder than it should
be and always rnore valuable than it secn1s at the outset (sec Dewald
et al. 1986 on replication in quantitative research).
The analogy in tradHional qualitative research is provided by foot-
notes and bibliographic essays. Using these tools, succecding scholars
should be able to locate thc sources used in published work and makc
their own evaluations of the inferences claimed frorn this inforrnation.
For research based cm direct observation, replication is 1nore difficult.
One scholar could borrow another's field notes or tape recorded inter-
views to see whether they support the conclusions rnade by the origi-
nal investigator. Since so rnuch of the data in field research involve
conversations, in1pressions, and other unrecorded participatory infor-
rnation, this reanalysis of results using the same data is not often done.
However, some important advances 1night be achieved if rnore schol-
ars tried this type of replication, and it would probably also encouragc
others to keep more complete field notes. Occasionally, an entire re-
scarch project, including data collection, has been replicated. Since we
cannot go back in tüne, the replication cannot be perfect but can be
quite valuable nonetheless. Perhaps thc rnost extensive rcplication of
11
J\n cxnmplc is lhc usc of morc than onc coder to extrnct systemc1lic information
from trnnscripts of in-dcpth intcrvi1.cws. [f two peoplc use the sc1mc coding rnlcs, we rnn
sec how oftcn they produce the samc judgmcnt. {f thcy do not produce rclinble mec1-
surcs, thcn wc can makc thc coding rules morc preci$c ,md try ngnin. Evcntunlly, a sct
of rulcs rnn oftcn be gencrntcd so that the npplication of thc sc1me prncedure by different
codcrs will yield thc satllc rcsult.
Major Components of Research Design · 27
l ualitative study is the sociological study of Middletown, [ndiana,
~, ~un by Robert and Helen Lynd. Their first "Middletown" study was
~1,!ished in 1929 and was replicated in a book published in 1937.
ßver fifty years after the original study, a long se.n~s of boc~ks and
. "] , . ai·e~ being· published that replicate these ong1nal studies (see
ar t cc e.5 . All 1· .
Cctplow et al., 'J983a, 1983.b and th.e citations. ther~m)., -, , \1ua It:t.1-~~
replication need not be this extensive, .but th~s 111aJor 1esemch p10Jed
·l cKild servc ,:1s an exemplar for what is poss1ble.
s 1 All research should atte1npt to achieve as muc 11 rep 1·1ca bT1 tty as pos-
sible: scholars should always record the exact m~thods, rules, and pro-
cedures used to gather information and draw 111ferences so that an-
c~ther rcsearcher can do thc same thing and draw (one hopes) the same
conclusion. Replicability also rneans that scholars who usc unpub-
lishcd or private records should endeavor to ensure tha~ future schol-
ars will have access to the material on sirnilar tenns; taking advantag.e
of privileged access with(~Ut seekin.g a~c~ss for. others prccludes ~eph-
Gltion and calls into quest1on the sc1entific qualtty of the w~1r~: Ustwlly
our work will not be rcplicated, but we have the respons1b1hty. to act
as if somcone 111ay wish to do so. Even if the work 1s not rephcated,
providing the materials for such replication will enable readers to un-
derstand and evaluate what we havc done.
1.2.4 /,nproving the Use of Existing Data
Fixing data problems by collecting new and better d,1ta is al~11ost al-
ways an improvement on trying to use existing, flawed da~a 1n bd~er
ways; however, thc former app~oach is not ~lways poss~bl~. ,5ocial
scientists often find thernselves with problematic data and httle chancc
to acquire anything better; thus, they havc to make the best of what
they have. .. . .
hnproving the use of previously collect~d .data 1s the n~am to~i.c
taught in classes on statistical 1nethods and is, .mdeed,r the chief contn-
bution of inferential statistics to the social sc1ences. fhe precepts on
this topic that are so clear in the study of infere~1tial statistics a~so
apply to qualitative rcsearch. The rcmait:der of th1s boo.k deals. with.
these precepts rnore fully. Here we prov1de .merely a bnef outhne of
the guidelincs for in1proving the use of prcv1ously collected. data.
First, whenever possible, we should usc data to generatc mferenc~s
that are "unbiascd," that is, corrcct on average. To understand th1s
very specific idea from statistical resea~ch,. irnagine appl~ing thc s~~nc
methodology (in quantitative or qualitative research) for analyzmg
and drawing conclusions from data across many data sets. Becau~e of
small errors in the data or in thc application of the procedure, a s1ngle
application of this rnethodology would probably never be cxactly cor-
 Thcmcs of This Volumc 29
28 Thc Scicncc in Soci,il Sciencc
usl guide our data collection, and help distinguish rclev,111t from ir-
rect . A n " un l,rnscc
· l" proccd ure will bc corred wl101 _, _, ~ ,. 111
relevant facts. [n chapter 2.6 wc discuss how theory affccts data collec-
age across many '1: anpl't ca t'1ons-even
, . . tc1kcn as. an avcr-
~, 1f no smg"l, 1. . tion, as well as how data disciplines theoretical imagination. Here, we
l he procedure will not syst"' 11· t· ·11 '] e app icatwn is,.correct.
' , ...1 a 1cc1 y t1 t the out· , 1 ·
L

or another. c m e 11l onc tnrcction want to strt.$S that theory and cn1pirical research must be tightly con-
11ectcd. Any theory that does real work for us has implications for cm-
Achieving unbiased inferences de e :i , f pirical investigation; no empirical investigation can be successful with-
nal collection of thc d·1ta at d ·t. l tp. l1l s, o coursc, both on the origi-
c c c 1 I s a Cl usc· 'U :l .
fore, it is always best to antici at-' bl ' ' l l ' as wc pomted out be- 0ut thcory to guide its choice of questions. Theory and data collection
gins. However, we n1cntion fi1c:ept:
-~~n: ~eforc data collection be-
using the data we need to b - , t' s, ues nefly here because when
are both essential aspects of thc process by which we seek to decide
whethcr a thcory should be provisionally vicwcd true or false, subjcct
sources of bic;s were -,,v„rI,,e kpa: idculc:~rly careful to Hnalyze whether as it is in both cascs to the uncertainty that characterizes all inference.
c O eu unng dat· ]! · We should ask of any theory: What are its observable implications?
source, which can lea,J t·c, b1'a.' sed m . f·erences ' isd t1eo
1 t ectrnn.
f 1 One · such
choosing observations' - a mc1nner , ·' ' a o se ect10n bias: We should ask about any e1npirical investigations·. Are thc observa-
111 th"lt syst ~ t· 11 · tions relevant to the in1plications of our theory, and, if so, what do thcy
populatlon fron1 which they were draw1~ , . erna ·,ca y ~1storts the
ple is deliberately choosi11g o11!y l._Ahlthough an obv1ous exam- enable us to infer about the correctness of the theory? In any social
. . ,. cascs w uc supp t .1 scientific study, the ünplications of the theory and the observation of
t1on bias can occur in n1uch 1110 c , bt or our t 1eory, selec-
result from omitted v·1r1'abl 'b'· ~ slu Jle ways. Another difficulty can focts need to mcsh with one another: social science concl usions cannot
' ' e ldS w uc 1 reters t ) tl , J · be considered reliable if thcy are not based on thcory and data in
control variable that m1'gl1t 1· I1fl uence
' a seernin ( le ,exc l us1on of. some
tw_ een our explanatory va1·1··1bl .' l g causa connedrnn be- strong connection with one another and forged by fonnulating and
. ' ' es ano t 1at whicl .
Wc d1scuss these and cxamining thc observable implications of a theory.
. numero · l , o 1er potentnll p'tf
,s tl . we want
11 · · to explam..
m; b iased inferences in chapters 2-6. ' ' a s m producmg
fhe second
ciency": an effi guideline
. _, t· ,
is ba s·e ~J on tlle statist1cal
. . concept of "effi- 1..3.2 Maxirnizing Lcvcragc
c1en use of data mvolve , . . .
used for descriptive ,,r C" . l . f -.s max1_m1z1ng the inforrnation The scholar who searchcs for additional implications of a hypothesis
. ,,usa m erencc Ma · · · ff' . is pursuing one of the most irnportant achievements of all soci~J sci-
quires not only using all our data bt t l . :' xm11z1ng e. 1c1ency re-
mation in thc data to ü11p . t' I a so us1ng all the relevant infor- ence: cxplaining as rnuch r1s possible with as little as possib[e. Good social
. ' rove m erences. For e 1 ·r . science secks to increasc the significance of what is explained relative
d1saggregated into 'sn1all ' g eograp . 1uca . l muts . w,xampl e, Ild the data . are
way, not just as a national a . ,.re _, , . ' e s 1ou use It that tu the infonnation uscd in the explanation. If we can accurately ex-
!arger degrees of uncerta1·ngt·g , gate._ Thde s~aller aggregates will havc plain what al /irst appears to be a complicated effect with a single
. ( y dSSOClate with tl " b .. causal variable or a few variables, the Ie.ve.rage we have over a problem
. ' ~ ' e 1mp11cahons
at least 111 part observabl . . . letn, ut 1f they are
of th-' t1 , ·] . ' is very high. Conversely, if we can explain many effeds on the basis of
tarn son1e informati,,11c
w11· l b b
Je l can e rought t0 b
e wory, l 1ey
l will
. con-
problen1. - ear on t 1c mference one or a few variables we also have high leverage. Leverage is low in
the social sciences in gcneral and even n1ore so in particular subjcct
areas. This n1ay be because scholars do not yet know how to increase
l.3 TtTEMES Ol' THIS VOLUME it or because nature h8.ppens not tobe orgc1nizcd in a convenient fash-
ion or for both of thcse rcasons. Areas convcntionally studicd qualita-
1
We condude this ovcrview cha ter b 1 . l . .
themes in developing resear p l . y 11g 1hghtmg the fom important
·1 ( c 1 c., e:,ignti llw.t we lnvc :r, d 1
tively are often those in which lcverage is low. Explanation of any-
thing seen1s to require a host of explanatory variables: we use a lot to
an(l w1 l claborate throughout this book. ' - c iscusse 1ere
cxplain a Jittle. In such cases, our goal should be to design research
with more levcragc.
1.3.1 Using Observable ltnpiications to Connecl Theory and Data Therc arc various ways in which we can incrcase our leverage ovcr
a resenrch proble1n. Thc prirn,;1.ry way is to increase the numbcr of
In ~his chapter we have e111phasized that e - , . _, observable implicc1tions of our hypothcsis and seek confinnation of
wh1le, musl ha ve implicatio11' b ve1 y lheory, to be worth-
find if the theory is correct Tl~ .~'
.
"t
'theb ob_serv_at1ons we expcct to
esc o J~r, ua Je unplzcatwns of the theory
those implications. As we have dcscribed abovc, this task can involve
 30 The Scirnce in Soda! Science
CU improving thc thcory so thal it has n1on) obscrvable implications,
(2) improving the data so rnore of these implications are indeed ob-
served and used to cvaluate the thcory, and (3) in1proving the use of
the data so that 1nore of thcse implications are extracted from existing
data. None of these, nor the general concept of n1aximizing leverage,
are the same as the concept of parsimony, which, as wc explained in
section 1.2.2, is an assumption about thc nature of the world rather
than a rulc for designing research.
Maxin1izing levernge is so important and so general that we strongly
re_com1nend that resenrchers routincly !ist all possible observable implications
of thcir hypothesis /hat might bc obscrvcd in t/,eir data or in othcr data. It
may be possible to test son1e of lhese new implications in the original
data set-as long as the ünplication does not "come out of" the data
but is a hypothesis independently suggested by the theory or a differ-
ent data set. But it is better still to turn to other data. ThtJS we should
also consider implications that rnight appear in other data--such as
data about other units, data about other aspects of the units under
study, data from different levcls of aggregation, and data fron1 other
time periods such as predictions about the near future-and evaluate
the hypothesis in those scttings. The rnorc evidence wc can find in
varicd contexts, thc more powcrful our explanation becon1es, and the
more confidence we and olhers should have in our conclusions.
At first thought, sorne researchers rnay object to the idea of collect-
i~1g ob~ervable implications fron1 any source or at any level of aggrega-
t1on different from that for which the theory was dcsigned. For exarn-
ple, Lieberson (1985) applies to qualitative rcsearch thc statistical idca
of "ecological fallacy"~incorrectly using aggregate data to make in-
ferences about individuals-to warn against cross-level inference. !2
We certainly agree that we can use aggregate data to make incorrect
inferences about individuals: if we are interested in individuals, then
studying individuals is generally a better strategy if we can obtain
these data. However, if the inference we seck to make is more than a
very narrowly cast hypothcsis, our theory may have implications at
111any levels of analysis, and wc will often bc able to use data from all
these levels to providc some infonnation about our theory. Thus, even
if we arc prin1arily interested in an aggreg.ate lcvcl of analysis, wc can
12
The phra~e "?c.olngic,1! follacy" is confusing bcrnuse thc proccss of reasoning from
~1ggrcgMc- to mchv1dual-lcvcl proccsscs is ncithcr ccologica! nor a follacy. "Ecological"
15 m~ unfor_tunatc choice of word to dcscribe lhe aggrcgate levcl of ,rnalysis. Although
Robm~on (!990) condudcd in his original arlidc about this topic that using aggrcgatc
<1nalys1s to reason about individm1ls i:; ,1 fatlacy, qunntitiitivc social sckmti:;ts c1.nd sl'iltisti-
dan:; now \.Videly rc~ognizc thal some information nbout individuals doc:; cxist at aggrc-
gc1.tc levcls of nnalysis, and many rnetilods of unbiascd "ccological" infcrtcnc:e have been
dcvcloped.
Themes of This Volume 31
oftcn gain leverage about our theory's veracity by looking at the data
from these othcr lcvcls.
For example, if we dcvelop a theory to explain rcvolutions, wc
should look for observablc ünplications of that theory not only in ovcr-
all outcon1es but also such phenomena as the responses to in-depth
intervicws of revolutionaries, thc reactions of people in sn1all commu-
nities in 111inor parts of the country, and official staten1ents by party
lcaders. We should be willing to take whatever information we can
acquire so long as it helps us lcarn about the veracity of our theory. If
we can test our theory by cxamining outcomes of revolutions, fine, ßut
in most cases vcry little information cxists at that lcvel, perhaps just
one or a few observations, and their values are rarely unan1biguous or
measured without error. Many different theories are consistent with
the existence of a revolution. Only by delving dceper in the prescnt
case, or bringing in relevant information existing in other cases, is it
possiblc to distinguish among previously indisting;uishable theories.
The only issue in using inforn1ation at other levels and fron1 other
sources to study a theory designed at an aggregate lcvel is whether
thcsc new observations contain senne information that is relevant to
evaluating in1plications of our theory. If these new observations hclp
to test our thcory, they should bc used even if thcy arc not the irnplica-
tions of greatest interest. For exmnple, we n1ay not care at all about the
views of revolutionaries, but if their answers to our questions arc con-
sistcnt with our thcory of revolutions, then the theory itself will be
more likely tobe correct, and thc collection of additional inforrnation
will have becn uscful. In fact, an observation at thc rnost aggregate
lcvel of data analysis-the occurrence of a predicted revolution, for
example--is merely one obscrved implication of thc theory, and be-
cause of the small an1ount of information in it, it should not be privi-
lcgcd over other observable implications. We need to collect informa-
tion on as 1nany observable implications of our theory as possiblc.
L3.3 Reporling U11ccrtainty
All knowledge and all inference-in quantitative and in qualitative
rcsearch-is uncertain. Qualitative measurement is error-prone, as
is quantitative, but the sources of error may diffcr. Thc qualitative in-
terviewer conducting a long, in-depth interview with a respondcnt
whose background hc has studicd is less likely to mismeasurc the sub-
ject's real political ideology than is a survey researcher conducting a
structured interview with a randomly sclccted rcspondent about
whorn he knows nothing. (Although the oppositc is also possible if,
for instance, hc rclies too lwavily on an inform.ant who is not trust-

32 Thc Scil'llce in Social Scicnce
worthy.) l lowcvcr, the survcy researchcr is less likely to generalizc
inappropriatcly from thc particular cases interviewed to the broader
population than is the in-depth researcher. Neither is immune fro111
the uncertaintics of measurement or thc undcrlying probabilistic na-
ture of thc social world. - Jate in life. If this wcrc thc case, those who d1d not have a hcart attack
All good social scientists-~whethcr in the quantitative or qualitative
traditions---report estimates of the uncertainty of their inferences. Pcr-
haps the single most serious problem with qualitative rcsearch in po-
litical science is the pervasive failure to provide rcasonable estünates
of the uncertainty of the investigator's infcrences (sec King '!990). We
can make a valid inference in alrnost any situation, no matter how litn-
ited the evidence, by fo!lowing the rules in this book, but wc should
avoid forging sweeping conclusions from wcak data. Thc point is not
that rcliable infcrences arc in1possiblc in qualitative research, but
r~ther that we should always rcport a reasonable estimate of the dcgree
of certainty we have in cach of our inferences. Neustadt and May
(1986:274), dealing with arcas in which precise quantitative estünates
are difficult, proposc a useful 111ethod of cncouraging policymakers
(who are often faccd with the necessity of reaching conclusions about
what policy to follow out of inadequatc data) to judgc the unccrtainty
of their conclusions. They ask "How rnuch of your own n1oney would
you wager on it?" This makes sense as long as we also ask, "At what
odds?"
Themes of '['his Volume 33
style charactcristics) that might e~riain th~ rcsult? ~ight .wc have in~
advcrtently reversed cause and ettect? [t 1s hard to nnagme how .nc~t
having a heart attack might cause one to c~t less red meat but it 1s
possiblc. Perhaps people lose their appetite t.or hamburgers and steak
(for whatcver rcason) would live longer and eat less nwat. This fact
would produce the sa111e relationship that led the researchers to con-
clude that n1eat was the culprit in heart attacks.
H is not our purpose to call such medical studics into question.
Rather we wish merely to illustrate how social scientists approach the
issue of causal inference: with skepticism and a concern for alternative
explanations that may have been overlooked. Causal inference thus
becomes a process whcreby each conclusion becomes the occasion for
further rescarch to refine and test it. Through successivc approxima-
tions we try to cmne closer and closer to accurate causal inference.

1.3.4 Thinking like a Social Scientist: Skepticisn,

and Rival Hvpotheses
The uncertainty of causal inferences 111eans that good social scientists
do not casily acccpt thcn1. When told A causes B, someone who
"thinks like a social scientist" asks whether that connection is a true
causal onc. It is easy to ask such questions about the research of others,
but it is more important to ask then1 about our own research. There arc
rnany reasons why we might be skcptical of a causal account, plausible
though it 1nay sound at firnt glance. We rcad in the newspapcr that thc
Japanese eat less red n1eat and have fewcr hcart attacks than A1neri-
cans. This observation alcme is interesting. In addition, thc explana-
tion-too much stcak leads to the high rate of l1eart discase in the
United States-is plausible. 'Thc skcptical social scientist asks about the
accuracy of thc data (how do wc know about eating habits? what sa1n-
plc was used? are lwart attacks classified similarly in Japan and the
United States so that we arc con1paring sitnilar phcnomcna?). Assurn-
ing thclt the data arc accurate, what else n1ight explain the effects: Are
thcrc other variables (othcr dietary differenccs, genetic fcatures, lifc-
230 [ncreasing the Number of Observations
valuable research. Fortunately, the appropriate n1ethodological issues
for qualitative researchers to understand arc precisely the ones that all
other scientific researchers need to follow. Valid inference is possible
only so long as the inherent logic underlying all social scientific re-
search is understood and followed.
References
Achen, Christopher H. 1986 StaUslicnl Analysis of Quasi-Experiments. Berkeley:
University of California Press.
Achen, Christopher H., and Duncan Snidal. 1989. "Rational Deterrence Thc-
ory and Comparative Case Studies." World Po/itics 4l, no. 2 (January): l43-
69.
Alvarez, Waltet~ and Frank Asaro. 1990. "An Extraterrestrial Impact." Scientific
A1ncrican (October): 78--84.
Anselin, Luc. 1988. Spatial Econmnetrics: Methods and Models. Boston: Kluwer
Academic Publishcrs.
Barnett, Vic. 1982. Cotnparative Statistical lnference. 2d ed. New York: Wiley.
Baumol, William J. 1990. "St. John versus the Hicksians, or a Theorist Malgre
Lui?" Thc Journal of Economic Literature 28, no. 4: l708-15.
ßeck, Nathaniel. l991. "Alternative Dynamic Structurcs." Pofiticaf Analysis 3:
51-87.
Becker, Howard S. 1966. "Whose Side Are We On?" Sociaf Problems l4: 239-47.
ßecke1~ Howard S., and Charles C. Ragin. l992. Wfwt Is a Casc? Exploring thc
Foundations of Social lnquiry. New York: Cambridge University Press.
Blainey, Geoffrey. 1973. The Causes of War. New York: Free Press.
Bollen, Kenneth A., Barbara Entwisle, and Arthur S. Alderson. l 993. "Macro-
comparative Research Methods." ln Judith Blake, ed. Macrocomparative Re-
search Methods Palo Alto, Calif. Annual Reviews, Inc.
Cain, ßruce, John Ferejohn, and Morris Fiorina. l 987. The Personal Vate: Constit-
ucncy Service and Electoral Indcpendence. Cambridge: Harvard University
Press.
Caplow, Theodore, Howard M. ßahr, ßruce A. Chadwick, and Dwight W.
Hoover. 1983a. All Fail11ful Pcoplc: Change and Continuity in Middletown's Reli-
gion. Minneapolis: University of Minnesota Press.
____ ,,. .. l983b. Middletown Families: Pifty Years of Change and Continuity. New
York: Bantam Books.
Caro, Robert. 1983. Thc Years of Lyndon Johnson. New York: Vintage ßooks.
Collict~ David. l991. "The Comparntive Method: Two Decades of Change." In
Dankwart A. Rustow and Kenneth Paul, eds. Comparative Political Dynamics:
Global Research Perspectives. New York: Harper Collins.
___ --. l993. "The Comparative Method." In Ada W. Finifter, ed. Politirnl Sci-
c11cc: Tlic State of t/11: Discipfi11e. Washington, D.C.: American Political Science
Association.
Cook, Karen Schweers, and Margaret Levi, eds. l 990. The Limits of Rationalify.
Chicago: Univcrsity of Chicago Press.
Coombs, Clydc H. 1964. A Theory of Dolo. New York: Wilcy.
Courtillot, Vincent E. l 990. "A Volcanic Eruption." Scfcntific Amerirnn (Octo~
ber): 78--84.
232 Refercnccs
Dahl, Robert. l 961. Who Covcrns? Dcmocmcy and Power in an Anwrican City.
New (--Iavcn: Yale University Press.
Dessler, David. 199"1. "ßeyond Correlations: Toward a Causa! Theory of War."
international Studies Quarterly 3, no. 35 (September): 337---55.
Dcwald, William C., Jcrry C. Thursby, and Richard G. Andcrson. l986. "The
Journal of Money, Credit and ßanking Project." Americnn Economic Review
76, no. 4 (September): 587-603.
Dü11T1ond, Larry and Marc F. Plattner, cds. 1993, The Global Resurgencc of De-
11wcrac11- Baltimore: Johns 1-Iopkins University Press.
Duneier,,Mitchell. 1993. Slim's ?r1/Jlc. Chicago: University of Chicago Press.
Easton, David. 1965. A Systems Analysis of Politicaf Life. New York: Wiley.
Eckstein, Harry. 1969. "Anthority Relations and Covernmental Performance."
Comparative Political Studies 2: 269-325.
1975. "Case Stndy and Theory in Political Science." In Fred L Green-
stein and Nelson W. Polsby, eds. Handbook of Politirnl Science, vol. l, Political
Scicnce: Scope and Theory. Rcading, Mass.: Addison-Wesley.
Elster, Jon. l983. Explaining Technical Change: A Case Study in the Philosaphy of
Science New York: Cambridge University Press.
Fearon, James D. l991. "Counterfactuals and Hypothesis Testing in Political
Sciencc." World Pofitics (January) 43, no. 2: l69-95.
Fenno, Richard F. 1978. I-lome Style. Boston: Little, Brown.
Fcrejohn, John. l993. "Structure and ldeology: Change in Parliament in Early
Stuart England." In Judith Goldstein and Robert 0. Keohane, eds. ldeas nnd
Foreign Policy: Be/iefs, lnstitutions and Political Change. lthaca: Cornell Univer-
sity Press.
Ferguson, Yalc I-I., and Richard W. Mansbach. 1988. The Elusive Quest: Theory
and International Folitics. Colnmbia: University of South Carolina.
Feynman, Richard P. 1965. The Character of JJhysical Law. Cambridge, Mass.:
MIT Press.
Fiorina, Morris, and Charles R. Flott. 1978. "Committee Decisions under Ma-
jority Rnle." American Political Science Review 72, no. 2 (June): 575-98.
Fisher, Ronald A. 1935. The Design of f."xperiments. New York: Hafner Pub-
lishing.
Fogcl, Robert William. 1989. Without Consent or Contract: The Rise and Fall of
American Slavery. New York: W.W. Norton.
Friedrich, Carl J. 1958. "Political Philosophy and the Science of Politics." In
Roland Young, ed. Approaches to thc Study of JJofitics. Chicago: University of
Chicago Press.
Fudenberg, Drcw, and Jean T'irole. l 989. "Noncooperative garne theory for in-
dustrial organization: an introdnction and overview." In Richard Schrna-
lenscc and Robert D. Willig, eds. l-landbook of Industrial Organization, vol. 1
Amsterdarn: North Holland.
Garfinkcl, H. l964. "Studies of the Routine Grounds of Evcryday Activitics."
Social Problems l1: 225-50.
Ceddes, Barbara. '1990. "How thc Cases You Choosc Affect the Answers You
Cet: Sclection Bias in Cornparativc Politics." Politica/ Analysis, 2:131-52.
Gccrlz, Clifford. 1973. An lntcrprdation of Cull-ures. New York: Basic ßooks.
Referei,ces 233
1983. "Local I<nowledge: Fact and Law in Comparative Pcrspective,"
In Clifford Geertz, ed. Locaf Knowledge: Purther Ess11ys in lnterpretive Anthro„
pology. New York: Basic ßooks.
Gelman, Andrcw, and Gary King. 1990. "Estimating Incumbency Advantage
without Bias." American Journal of Political Science 34, no. 4 (November):
ll42-64.
1993. "Why Are U.S. Presidential Election Poils So Variable When the
Vote Is So Predictable." ßritish Journal of Political Science (forthcoming).
George, Alexander L. ·r 982. "Casc Studies and Theory Devclopment." Paper
presentcd at thc Second Annual Symposium on Information Processing in
Organizations, Carnegie-Mellon University, October 15-16.
George, Alexander L., and Tirnothy J. McKeown. l 985. "Casc Studies and Thc-
ories of Organizational Decision Making. Advances in [11ftmnation Proccssing
in Organizations 2: 21-58.
George, Alexander L., and Richard Smoke. 1974. Deterrence in American Foreign
Poficy. New York: Columbia University Press.
Gigerenze1~ Gerd, Zeno Swijtink, Theodore Porter, Lorraine Daston, John
Beatty, and Lorenz Kruger. 1989. The En1pire of Chance: How Probability
Changed Science and L:veryday Life. New York: Cambridge University Press.
Gilpin, Robert. 1981. War and Change in World l'olitics. New York: Cambridge
Univcrsity Press.
Goldbergei~ Arthur. 199l. A Course in f.'conometrics. Cambridge, Mass: Harvard
University Press.
Goldstein, Judith, and Robert 0. Keohane, eds. 1993. Ideas and Foreign Policy:
Beliefs, lnstitutions and Political Change. Ithaca: Cornell Univcrsity Press.
Could, Stephen J. 1989a. Wonderful Life: The Burgess Shale and the Nature of.l·-Iis-
tory. New York: Norton.
1989b. "The Horn of Triton." Natural Hislory (Decernber): 18-27.
Cranger, G.W.J., and P. Newbold. 1977. Forecasting Economic Time Series. New
York: Academic Press.
Gulick, Edward V. 1967. Europe's Classicaf Balance of Power. New York: Norton.
Hall, Peter A., ed. 1989. The Political Power of Economic ldeas: Keynesianis1n
Across Nations. Princeton: Princeton University Press.
Halpern, Nina, 1993. "Stalinist Political Econorny," ln Jndith Goldstein and
Robert 0. Kcohane, eds. ·1993, Ideas nnd Foreign Poficy: Beliefs, lnstitutions and
Politica/ Change. [thaca: Cornell University Press.
Hermens, F. A. 1941. Democracy or llnarchy: A Study of Proportio11al Rcpresenta-
tion. South Bend, lnd.: University of Notrc Dame Press.
i"lirschman, Albert 0. 1970. "The Search for Paradigrns as a I-Iindrance to Un-
derstanding." World Politics 22, no. 3 (April): 329-43.
Hoffmann, Stanley. l 960. Conte111porary Tltcory i11 International R.elatio11s. Engle-
wood Cliffs, N.J.: Prentice-I--Iall.
Holland, Paul. 'l 986. "Statistics and Cansal ·foference." Journal of Ute American
Statistical Association s·1: 945-60.
Horowitz, Donald. l993. "Comparing Democratic Systems." In Larry Dia-
rnond and Marc F. Plattner, eds. The Global Resurgcnce of Democracy. Balti-
more: Johns Hopkins University Press.
234 References
Hulh, Paul. l988. "Extcndcd Detcrrcnce and the Outbreak of War." All!crican
PoliUcal Scieuce Review 82: no. 2 (June): 423-43.
f-Iuth, Paul and Brucc Russett. 1990. "Testing Deterrcncc Theory: Rigor Makes
a Difference." World PoliUcs 42, no. 4 (July): 466-.SOl.
Hsiao, C. 1983. "Identification." In Zvi Criliches and Michael Intriligator, eds.
vol. 1, l-landbook of Econometrics. Amsterdam: North~Holland.
(nkelcs, Alex, and Peter Rossi. ·r 956. "National Comparisons of Occupc1.tional
Prestige." American Journal of Sociology 61: 329-39.
Iyengm~ Satis, and Jod B. Creenhouse. 1988. "Selcction Models and the File
Drawer Proble1n." Slatistical Sciencc 3, no. 1 (Fcbruary): HJ9-35.
Iyengm~ Shanto, and Donald Kinder. 1987. News Tlwt Matters. Chicago: Uni-
versity of Chicago Press.
Jeffreys, Harold. l96l. Theory of Probability. Oxford: Clarendon Press.
Jervis, Robert. 1976. Perception and Mispcrception in lntcmational Pofitics. Prince-
ton: Princeton University Press.
Jcrvis, Robert, Richard Ned Lcbow, and Janice Cross Stein. 1985. Psycholop;y
and Detcrcncc. Baltimore: Johns Hopkins University Press.
Jones, E. L. l981. Thc European Miracle: Environments, Economics, and Geopolitics
in the Histon; of Europe and J\sia. Cambridge: Cambridge University Press.
Johnston, J. l984. Econometric Metlwds. 3d ed. New York: McGraw I--Iill.
Kahnen1cm, Daniel, Paul Slovik, zmd Arnos Tversky, cds. 1982. Judgement wider
Uncertainty: l1c11ristics a11d Hiascs. New York: Cambridge Univcrsity Press.
Katzenstein, Peter J. 1985. Smalf States in World Markcts: Jndustrial Policy in Eu-
rope. Ithaca: Cornell University Press.
Keane, Mark T. l988. Analogien! Prob/ein Solving. Chichester, West Sussex: Ellis
Horwood.
Kennedy, Paul. 1987. The Rise ond Fall of the Great Powers. New York: Random
Housc.
I<eohane, Robert 0. l980. "Thc Thcory of Hegemonie Stability and Changes in
International Economic Regimes, 1967-1977." In Ole R. Holsti, Randolph M.
Siverson, Alexander L. George, eds. Change in the International Systern. ßoul-
dcr: Westview Press.
1984. After J-Iegenwny: Cooperation and Discord in the World Political Econ-
Ollllj. Princeton: Princeton University Press .
.................·...... 1988. "International Institutions: Two Approaches." !11tcrnational Stud-
ies Quarter/,; 32: 379.
-----·-·-·· 1989.' lntcmatümal l11st-itutions and State Power: Essoys in International
Relations Theory. I3oulder: Westview.
Kcohane, Robert 0., and Joseph S. Nye, Jr. 1977. Power ond lnterdependence:
Worfd Politics in 'fransition. Boston; Littlc, Brown.
Khong, Yuen Foong. 1992. A11afogics 11/- War: r<ort'a, Municlt, Dien Bie11 Pliu, and
the Vietnam Decisions of :l965. Princeton: Princeton University Press.
King, Gary. 1989. Llnifying Political Methodology: The Likelilwod Theory of Statis/-i-
cal lnference. New York: Cmnbridge University Press.
l 993. "The Methodology of Presidency Research," In George Ed-
wards m, John H. Kessel and Bert A. R.ockrnan, eds. Researching Uze Presi-
Rcferenccs
235
dency: Vital Quesf-io11s, New Approachc:,. Piltsburgh: Univcrsity of Pittsbu,vh
Press. 0
... 19~1a .. ".Const!tuency Service and fncurnbency Advantage." British
Journal oJ Politu:al Sc1ence 21, no. "J (January): U9·-28.
--·-·-······ 199"Jb. "Stochastic Variation: A Comment on Lewis-Beck and Sblla-
ban's 'The R-Square'." Political Analysis 2: 185-200.
. 1991c. "On Pohtical Mcthodology." Political Anolysis 2: ·t-30.
Kohli, Atul. 1987. T1-ze State and Poverty in lndia: The Politics of Reform. New
York: Cambridge University Press.
Kreps, David M. 1990. "Corporate Culture and Economic Theory," In
James E. Alt and I<enneth Shepsle, eds. Pcrspectives on Positive Political Econ-
omy. New York: Cambridge University Press.
Laitin, David D. 1986. Uegemony and Culturc: Politics and Religious Clu111ge
among the Yorulm. Chicago: University of Chicago Press.
Lakatos, Irnre. 1970. "Falsification and the Methodology of Scientific Research
Progrnms." In I. Lakatos and A. Musgrave, eds. Criticism and the Crowth of
Knowlcdge. Cambridge: Cambridge University Press.
Lakeman, Enid, and James D. Lmnbert. 1955. Voting in Denwcmcies. London:
Faber and Faber.
Leame1~ Edward E. 1978. Spec1fication Searcl1es: Ad Hoc lnference With No11exper-
imental Data. New York: Wiley.
1983. "Let's T1ke thc Con Out of Econometrics." Al/lerirnn f.'conomic
Review 73, no. 1 (March): 31-43.
Levy, Jack S. 1985. "Theories of General War." World Politics 37, no. 3 (April):
344-74.
--· "Quantitative Studies of Deterrence Success and Failure." In Paul C.
Stern, Robert Axelrod, Robert Jervis, and Roy Radner, eds. Perspectives on
Deterrencc. New York: Oxford University Press.
Lieberson, Stanley. 1985. Making lt Count: The Improvement of Social Research and
Theory. Berkeley: University of California Press.
-..... 1992. "Einstein, Renoir, and Grceley: Some Thoughts about Evidence
in Sociology." Anwrica11 Sociofogical Review 56 (February); 1~15.
Lijphart, Arend. 1971. "Comp<1rative Politics and Comparative Method,"
A///erican Political Science Review 65, no. 3 (Septgmber): 682-98.
Lindberg, Leon N., and Stuart A. Scheingold. 1971. Regional lntegmtion: Theory
and Research. Cambridge: Harv,1rd Univcrsity Press.
Lindberg, Leon N., and Stuart A. Scheingold. 1970. Europe's Would-Be Polity:
Patterns of Change in l·he European Community. Englewood Cliffs, N.J.: Pren-
tice-Hall.
Linz, Juan J. 1993. "The Pcrils of Prcsidentialisrn." In Diamond, Larry and
Marc F. Plattner, eds. The Global Res1.11srnce of Dcmocmcy. Baltirnorc: Johns
Hopkins University Press. 108~26.
Lipset, Seymour Martin. 1963. Thc First New Nation: The U11ited Stotes in Co111-
porativc and Historical Perspective. New York: Basic Books.
Little, Daniel. 1991. Varietics of Socia! fxpl1111r1tion: A11 Introduction to llie Plziloso-
phy of Socia{ Science. ßoulder Colo.: Wcstview.
236 References
Longino, l-Ielen E. 1990. Scic11ce as Sociol f<nowledge: Vof11fs (lnd 0/;jectivity in
Scien/-ific lnquivy. Princcton: Princeton Univcrsity Press.
Lowcnthal, Abrnham F. 1972. The Do1ninican lnl"ervention. Cambridge: Harvard
University Press.
Mankiw, N. Gregory. 1990. "A Quick Refrcsher Course in Macroeconomics,"
Journal of Economic Litemture (December) 28, no. 4: 1645-60.
Martin, Lisci L. 1992. Coercive Cooperution. Princeton: Princeton Univcrsity
Press.
Merck & Co., lnc., ]989. Annual Report. Rayway, New Jersey: Merck & Co., Inc.,
1989.
Merton, Robert K. I1949] 1968. Social Theory and Socinl Strudure. Reprint. New
York: Free Press.
Mill, John Stuart. 1843. A Systent of Logic. Publisher unknown.
Miller, David. 1988. "Conjectural Knowlcdge: Popper's Solution of the Prob-
lem of lnduction." In Paul Levinson, ed. ln Pursuit of Truth. Atlantic High-
lands, N .J .: I-Iumanities Press.
Milner, Helen V. 1988. Resisting Protectionism: Global lndustries and the Politics of
lnternntfonal Tmdc. Princeton: Princeton University Press.
Moc, Terry M. 1990. "The Politics of Structural Choke: Toward a Theory of
Public Burwucracy." In Oliver Williamt.on, ed. Organization Theory: Frmn
Chester ßnrnard fo the Present and Beyond. New York Oxford University
Press.
Moon, Donald J. 1975. "The Logic of Political Inquiry: A Synthesis of Opposed
Perspectives." In Fred 1. Creenstein and Nelson W. Polsby, eds. Handbook of
Political Science, vol. 1, PoliUcal Science: Scope and Theory. Reading, Mass.: Ad-
dison-Wesley.
Neustadt, Richard E., and Earnest R. May. 1986. Thinking in Tirne: T1w Uses of
Hisfory for Decision-Mnkers. New York: Free Press.
Nye, Joseph S. 197l. Peace in Parts. Boston: Little, Brown.
O'Hear, Anthony. 1989. lntroduction to the Fhilosophy of Science. Oxford: Claren-
don Press.
Ordeshook, Peter C. 1986. Garne Theory and Po/itical Theory: An lntroduction.
New York: Cambridge University Press.
Palfrey, Thmnas R., ed. 1991. L.a!Joratory Research in Political Economy. Ann
Arbor: University of Michigan Press.
Pearson, Karl. l892. The Gramnuzr of Science. London: J.M. Dent & Sons,
Ltd.
Plott, Charles R., and Michael E. Levine. 1978. "A Model of Agenda Influence
on Co1n111ittee Dcdsions." A111ei-ic11n Econo1nic Review 68, no. 1 (March): 146-
60.
Poppc1~ Karl R. 1968. 'l'he Logic of Scientific Oiscovery. New York: Harper and
Row.
·-· FJ82. "The Open Universe: An Argument for Indeterminism." In W.W.
Ihrtley III, cd. 'fhe Postscript to l'he Logic of Scientific Discovery. Totowa, N.J.:
Rowman and Littlefield.
[\)rter, Michael E. 1990. The Competitive Advantage of Nations. New York: Free
Press.
Rcfcrences 237
Prze~orski, Adan:: ai:d I--Ien:y Teune. 1982. The Logic of Compamtive Social Jn-
qwry. Malabar, Honda: I<neger Publishing Company.
Psathas, Ccorge. 1968. "Ethnomcthods and [lhcnomenology." Social Research
35: 500-20.
Putnam, Robert D., with Robert Leonardi and Raffaella Y. Nanetti, 1993. Mak-
ing Democracy Work: Civic Traditions in Modem Italy. Frinceton: Princeton
University Press.
Ragin, Charles C. 1987. The Cornparative Method: Movinr.; bevond Qualitative and
Quantitative Strategies. Berkeley: Univcrsity of Califo~nü~ Press.
Ri\ers, Douglas, and Morris P. Fiorin,1. 1989. "Constituency Service, Reputa-
tion, and the Incumbency Advantage," In Morris P. Fiorina and David
Rohde, eds. ffome Style and Washington Work. Ann Arbor: University of
Michigan Press.
Robinson, William S. 1990. "Ecological Correlations and the Behavior of Jndi-
viduals." Americnn Sociofogicnl Review 15: 351-.S7.
Rogowski, Ronald. 1987. "Trade and the Variety of Democratic Institutions."
lnternntion11l Organization 4l, no. 2 (Spring): 203-24.
Ro~enau, Pauline. l990. "Once Again into the Fray: International Relations
Confronts the I-Iumanities." Mi!leniwn: Journal of International Stw.iies 19,
no. 1 (Spring): 83-llO.
Rosenstone, Steven R. 1983. Forecasting Presidential Clcctions. New Haven: Yale
University Press.
Roth, Alvin E. 1988. "Laboratory Experimentation in Econornics: A Methodo-
logical Overview." The Econornics Journal 98 (Dcccrnbcr): 974-··103·1.
Rubin, Donald B. 1974. "Estimating Causal Effects of 1featments in Random-
ized and Nonrandomized Studies." Journal of Educntional Psychofooy 66: 688-
701. "
1978. "Bayesian Infcrence for Causa! Effccts: Thc Role of Randomiza-
tion." The Annals of Statislics 6: 34-58. \
Russett, Bruce. 1978. "The Marginal Utility of Income Transfers to thc Third
World." International Organization 32, no. 4: 913-28.
Sanday, Peggy Reves. 1983. "The Ethnographie Paradigm(s)." In John Van
Maanen, ed. Qualitative Methodology. Sage: Beverly Hills.
Schumpeter, Joesph A. [1936] l99l. "Can Capitalism Survive?" ln Richard
Swedberg, ed. The Econo1nics of Sociofogy nnd Capitalis1n, Princeton: Princeton
Univl:~rsity Press.
Shepsle, I<enncth A. 1986. "lnt.titutional Equilibrium and Equilibrium [nstitu-
tions." In Herbert F. Weisberg, ed. Pofiticnl Science: The Science of Politics.
New York: Agathon Press.
Shively, W. Phi!lips. 1990. Thc Cmft of Political Research. 3d ed. Englewood
Cliffs, N.J.: Prentice-lfoll.
Simon, I~Icrbert.f:· 198~. "Human Nature in Politics: The Dialogue of Psychol-
ogy w1th Polttical Sc1encc." Amcrican /Joliticnl Science R,cvicw. 79, no. 2 (June):
293-305.
Skocpol, Theda. 1979. Stntes and Social Revolutions. Cambridge University
Press.
--- ·· l984. "Emerging Agendat. and Recurrent Strategies in HistoriGJl Soci-
238 References
ology." ln Thcda Skocpol, cd. Vision and Method in J-Iistorirnl Sociology. New Index
York: Cambridge University Press. ··-----·-----··-----
Snyder, Glenn I-l., and Paul Dicsing. 1977. Couflict amons Nations: Bar~ainini
Decision Making and Syste111 Structurc in International Crises. Frinccto~1'.
Princeton University Press.
Snyder, Jack. 1991. Myths of Empire: Do111estic Politics anä [nfcrnational Ambitio11.
Ithaca: Cornell Univcrsity Press. acu te criscs, '134-35 causa! mechanü;ms, 85-87, 225
Sober, Elliot. l988. Rcconstructing the Past: Parsi1nony, Evolution and lnfereuce. A!varez, Luis, H-··"12 causa\ thcorics, 99-100
Cambridge: MIT Press. analogy, reasoning by, 212-13 rulcs for constructing, 99-.J"!4
Suppes, Patrick C. 1970. A Probabilistic Tlteory of Causality. Amsterdmn: North- applied rcsearch, 18n causnlity, 76-91
1-Iolland. assignment, 94, 115, 196-97, 229 Chile, 219
Tawney, R.H. l 935. Religion and the Rise of Capitalism. New York: Harcourt, c1symmetdc causality, 89-91 Christian trnditions in Nigeria, '147, "186·-
Australia, 205 87, 205-6
Brace & Co.
Tilly, Charles, cd. 1975. The Formation of National Statcs in Western Europc. autocorrclation, 223n coding, '157
availability heuristic, 2I3n collecting data, 51
Princeton: Princeton University Press. average, 53, 65, 80-8l
Verba, Sidncy. 1967. "Somc Dilemmas of Political Research." World Politics 20 to compensate for fundamental vari-
ability, 214
(October): lll-28. basic research, l8n and cfficiency, 214
Waltz, Kenneth N. 1979. Theory of lnternational Politics. Reading, Mass: Ad- bias, 27-28, 63-65, 97-99, l50-Sl ensuring reliability, 25-26, l51
dison-Wesley. compared to efficiency, 70-74 using forma! modcls for, 51-53, lOS-
Webb, Eugcne J., D. T. Campbcll, R. D. Schwartz, and L. Sechrest. 1966. Unob- and endogcneity, 195-96 106
trusivc Measurcs. Chirngo: Rand McNally. formal example of, 65, 97-99 guidelines for improving, 23-26, 46-47
Webb, Eugene J., and Karl E. Weid<. 1983. "Unobtrusivc Measures in Organi- and measurement error, 161-65 level of aggregation, 48·-A9, 'f 17
zational Theory: A Rcminder." In John Van Maanen, ed. Qualitative Method·· and substantive bias, 64 to maximize lcverage, 204-205
bivariate <mülysis, 172 pilot projects before, 22-23
ology. Sage: ßeverly Bills.
Boolcan Algebra c1pproc1ch, 90n quantification in, 43-44
Weber, Max. {1905] 1949. "Critical Studies in the Logic of the Cultural Sci-
cnces." In Max Weber, ed. The Me/:hodology of thc Social Sciences. Translated rccording process of, 23
Canada, 205 reliability as a goa\, 25-26
and edited by Ed.ward A. Shils and Henry A. Fluch. New York: Free Press. case-contro! study, 141 replicability as a goal, 26-27
Weiner, Myron. '1991. T'hc Child and Uze State in [ndia. Princeton: Princcton Uni- case studies, usc of, 5, 43-46, 52-"53, 60, and small~11 problcrn, 213-17
versity Press. 67-68 collinearity, 2·13-15. See also multi-
Wcndt, Alexander. 1992. "Anarchy is What States Make of lt: The Sodal Con- and causal mechanisms, 85-87, 225 collincarity
struction of Power Politics." International Organization 64, no. 2 (Spring): and constant causal cffects, 93 comparative approach, 212-13
391-426. crucia!, 208-12 comparntive politics field
Wolfinger, Ray1nond, and Steven Rosensterne. l980. Who Votes. New Haven: multiple measures over time, 221 importance of description in, 44-45
and sclection bias, H7 complexity of events, 9-Hl, 42-43, 93-94
Yale University Press.
single, 208-12, 227n
Woods, John, and Douglas Walton. l982. Argu1ncnt: Tlw Logic of the Falfacies.
and unit homogeneity, 93
conditional inder,endcnct'
/ . "', ·1·1c,.
., 91 , 94-()"'
New York: McGraw-Hill Ryerson Ltd. congruence proccdure, 45-46
of the Weimar Repub!ic, l89-91 consistency, l67n
Zelditch, Morris Jr. "l9Tl. "Intelligible Comparisons." .In Ivan Valliet~ cd. Com- cases, 52-53, H6-17, 208·-U constmlt effect assurnption, 92-93, l16
pamtivc Methods in Sociology. Berkeley and Los Angeles: University of Cali- cm1sc1l effect, s·1-s2, 84, 88 constants, selcction of, 146·"49
fornia Press. fonnal rnodcl for, 95-97 construct validity, 25
Zellner, Arnold. l971. An lntroduction to Bnyesi1111 lnference in Econo111etrics. New overestim,1ting, l39 contribution to scientific liternture, '16-17
York: Wiley. c.111s;1l hypothcsis, 40
contro!, 125, 168, l85, l96~·2ff!
1984. Basic [ssucs in F,conomefrics. Chicago: University of Chicago Press. causc1l infcrencc, 8, "18, 32·-33, 75-ll4, rl6 of the research situation, ·t 99--206
assumptions foi~ 91-97 and small-11 problcms, 196·-200
and case studies, 45 control variables, 77
correcting biased, ·t87-88 counterfactual mrnlysis, 10-U, 77-78, 88--
critcria for judging, 97-99 89, 192
with measurcrnent error, 156 crucial c,1se studies, 209-"!2
unccrtainty oJ, 2'13-15 culture, 109-10, l9l··-92, 226