Documenti di Didattica
Documenti di Professioni
Documenti di Cultura
Brian D. Haig
Abstract
This chapter provides a philosophical examination of a number of different quantitative research
methods that are prominent in the behavioral sciences. It begins by outlining a scientic realist
methodology that can help illuminate the conceptual foundations of behavioral research methods. The
methods selected for critical examination are exploratory data analysis, statistical signicance testing,
Bayesian conrmation theory, meta-analysis, exploratory factor analysis, and causal modeling. Typically,
these methods contribute to either the detection of empirical phenomena or the construction of
explanatory theory. The chapter concludes with a brief consideration of directions that might be taken
in future philosophical work on quantitative methods.
Key Words: scientic realism, methodology, exploratory data analysis, statistical signicance testing,
Bayesianism, meta-analysis, exploratory factor analysis, causal modeling, latent variables, phenomena
detection, hypothetico-deductive method, inference to the best explanation
Introduction
Historically, philosophers of science have given
research methods in science limited attention, concentrating mostly on the nature and purpose of theory in the physical sciences. More recently, however,
philosophers of science have shown an increased
willingness to deal with methodological issues in sciences other than physicsparticularly biology, but
also psychology to a limited extent. There is, then,
a developing literature in contemporary philosophy
of science that can aid both our understanding and
use of a variety of research methods and strategies in
psychology (e.g., Trout, 1998).
At the same time, a miscellany of theoretically
oriented psychologists, and behavioral and social
scientists more generally, have produced work on
the conceptual foundations of research methods that
helps illuminate those methods. The work of both
professional philosophers of science and theoretical
scientists deserves to be included in a philosophical
examination of behavioral research methods.
Scientic Realism
The philosophies of positivism, social constructionism, and scientic realism just mentioned are
really family positions. This is especially true of scientic realism, which comes in many forms. Most
versions of scientic realism display a commitment
to at least two doctrines: (1) that there is a real world
8
of which we are part and (2) that both the observable and unobservable features of that world can be
known by the proper use of scientic methods. Some
versions of scientic realism incorporate additional
theses (e.g., the claims that truth is the primary aim
of science and that successive theories more closely
approximate the truth), and some will also nominate
optional doctrines that may, but need not, be used
by scientic realists (e.g., the claim that causal relations are relations of natural necessity; see Hooker,
1987). Others who opt for an industrial strength
version of scientic realism for the physical sciences
are more cautious about its successful reach in the
behavioral sciences. Trout (1998), for example, subscribes to a modest realism in psychology, based on
his skepticism about the disciplines ability to produce deeply informative theories like those of the
physical sciences.
Given that this chapter is concerned with the
philosophical foundations of quantitative methods,
the remaining characterization of scientic realism
will limit its attention to research methodology.
t h e p h i l o s o p h y o f q u a n t i tat i v e m e t h o d s
methods are concerned with the effective organization of data, the construction of graphical and
semi-graphical displays, and the examination of
distributional assumptions and functional dependencies. Two additional attractive features of Tukeys
methods are their robustness to changes in underlying distributions and their resistance to outliers
in data sets. Exploratory methods with these two
features are particularly suited to data analysis in psychology, where researchers are frequently confronted
with ad hoc sets of data on amenable variables, which
have been acquired in convenient circumstances.
Consider the hypothetico-deductive and inductive conceptions of scientic methods, which are
mentioned here as candidate models for data analysis. Most psychological researchers continue to
undertake their research within the connes of the
hypothetico-deductive method. Witness their heavy
preoccupation with theory testing, where conrmatory data analyses are conducted on limited sets of
data gathered in accord with the dictates of the test
predictions of theories. In this regard, psychologists
frequently employ tests of statistical signicance
to obtain binary decisions about the credibility of
the null hypothesis and its substantive alternatives.
However, the use of statistical signicance tests in
this way strongly blunts our ability to look for
more interesting patterns in the data. Indeed, the
continued neglect of exploratory data analysis in
psychological research occurs in good part because
10
t h e p h i l o s o p h y o f q u a n t i tat i v e m e t h o d s
checking for the accuracy of data entries, identifying and dealing with missing and outlying data, and
examining the data for their t to the assumptions
of the data analytic methods to be used. This important, and time-consuming, preparatory phase of
data analysis has not received the amount of explicit
attention that it deserves in behavioral science education and research practice. Fidell and Tabachnick
(2003) provide a useful overview of the task and
techniques of initial data analysis.
Conrmation of the initial data patterns suggested by exploratory data analysis is a just checking strategy and as such should be regarded as a
process of close replication. However, it is essential
to go further and undertake constructive replications to ascertain the extent to which results hold
across different methods, treatments, subjects, and
occasions. Seeking results that are reproducible
through constructive replications requires data analytic strategies that are designed to achieve signicant
sameness rather than signicant difference (Ehrenberg & Bound, 1993). Exploratory data analysis,
then, can usefully be regarded as the second in
a four-stage sequence of activities that, in turn,
attend to data quality, pattern suggestion, pattern
conrmation, and generalization.
11
has undertaken a useful extensive review of the controversy since its beginning. Despite the plethora
of critiques of statistical signicance testing, most
psychologists understand them poorly, frequently
use them inappropriately, and pay little attention
to the controversy they have generated (Gigerenzer,
Krauss, & Vitouch, 2004).
The signicance testing controversy is multifaceted. This section will limit its attention to a
consideration of the two major schools of signicance testing, their hybridization and its defects, and
the appropriateness of testing scientic hypotheses
and theories using tests of statistical signicance.
Psychologists tend to assume that there is a single unied theory of tests of statistical signicance.
However, there are two major schools of thought
regarding signicance tests: Fisherian and NeymanPearson. Initially, Neyman and Egon Pearson sought
to build on and improve Fishers theory, but they
subsequently developed their own theory as an alternative to Fishers theory. There are many points of
difference between the two schools, which adopt
fundamentally different outlooks on the nature of
scientic method. The uncritical combination of
the two schools in psychology has led to a confused
understanding of tests of statistical signicance and
to their misuse in research.
t h e p h i l o s o p h y o f q u a n t i tat i v e m e t h o d s
neo-Fisherian alternative. Key elements of this alternative are that the probability of Type I error is not
specied, pvalues are not misleadingly described
as signicant or nonsignicant, judgment is suspended about accepting the null hypothesis on the
basis of high p-values, the three-valued logic that
gives information about the direction of the effect
being tested is adopted, accompanying effect size
information is provided, and use is made of adjunct
information such as condence intervals, where
appropriate. Two things to note here about this neoFisherian perspective are that it is concerned with
signicance assessments but not null hypothesis signicance tests and that it is concerned with statistical
tests as distinct from the tests of scientic hypotheses. There are empirical studies in psychology that
approximate this modied Fisherian perspective on
signicance tests.
13
and a statistical concept of power, Neyman and Pearsons approach marks an improvement over Fishers
signicance tests. However, as noted earlier, Neyman and Pearson were concerned with measuring
the reliability of errors in decision making in the
long run rather than with evidence for believing
hypotheses in a particular experiment; Type I and
Type II error are objective probabilities, understood as long-run relative frequencies. As such, they
belong to reference classes of endless series of trials
that might have happened but never did. They are
not about single events such as individual experiments. Further, being concerned with decision
making understood as behavioral courses of action,
Neyman-Pearson statistics do not measure strength
of evidence for different hypotheses and thus do not
tell us how condent we should be in our beliefs
about those hypotheses.
It would seem, then, that the Neyman-Pearson
approach is suitable for use only when the focus is on
controlling errors in the long run (e.g., quality control experiments). However, this does not happen
often in psychology.
The Hybrid
Although the Neyman-Pearson theory of testing
is the ofcial theory of statistical testing in the eld
of professional statistics, textbooks in the behavioral sciences ensure that researchers are instructed to
indiscriminately adopt a hybrid account of tests of
statistical signicance, one that is essentially Fisherian in its logic but that is often couched in the
decision-theoretic language of Neyman and Pearson.
The hybrid logic is a confused and inconsistent
amalgam of the two different schools of thought
(Acree, 1979; Gigerenzer, 1993; Spielman, 1974;
but see Lehmann, 1993, for the suggestion that the
best elements of both can be combined in a unied
position). To the bare bones of Fisherian logic, the
hybrid adds the notion of Type II error (opposed by
Fisher) and the associated notion of statistical power
(which Fisher thought could not be quantied)
but only at the level of rhetoric (thereby ignoring
Neyman and Pearson), while giving a behavioral
interpretation of both Type I and Type II errors (vigorously opposed by Fisher)! Because the authors of
statistical textbooks in the behavioral sciences tend
to present hybrid accounts of signicance testing,
aspiring researchers in these sciences almost always
acquire a confused understanding of such tests. It is
most unfortunate that many writers of statistics textbooks in the behavioral sciences have unwittingly
perpetuated these basic misunderstandings.
14
To make matters worse, this confusion is compounded by a tendency of psychologists to misrepresent the cognitive accomplishments of signicance
tests in a number of ways. For example, levels
of statistical signicance are taken as measures of
condence in research hypotheses, likelihood information is taken as a gage of the credibility of the
hypotheses being tested, and reported levels of signicance are taken as measures of the replicability
of ndings (Gigerenzer, Krauss, & Vitouch, 2004).
t h e p h i l o s o p h y o f q u a n t i tat i v e m e t h o d s
long and hard to answer this important and challenging question by developing theories of scientic
conrmation. Despite the considerable fruits of
their labors, there is widespread disagreement about
which theory of conrmation we should accept. In
recent times, a large number of philosophers of
science have contributed to Bayesian conrmation
theory (e.g., Earman, 1992; Howson & Urbach,
2006). Many philosophical methodologists now
believe that Bayesianism, including Bayesian philosophy of science, holds the best hope for building
a comprehensive and unied theory of scientic
inference.
Bayesianism is a comprehensive position. It comprises a theory of statistical inference, an account
of scientic method, and a perspective on a variety of challenging methodological issues. Today, it
also boasts a fully edged philosophy of science. In
this section, attention is limited to a consideration
of the strengths and weaknesses of Bayesian statistical inference, the ability of Bayesian conrmation
theory to improve upon the hypothetic-deductive
method, and the question of whether Bayesianism
provides an illuminating account of the approach
to theory evaluation known as inference to the best
explanation.
Pr(H) Pr(D/H)
Pr(D)
15
informational output of a traditional test of signicance is the probability of the data, given the truth
of our hypothesis, but it is just one input in the
Bayesian scheme of things. A second stated desirable feature of the Bayesian view is its willingness to
make use of relevant information about the hypothesis before the empirical investigation is conducted
and new data are obtained, explicitly in the form of
a prior probability estimate of our hypothesis. Traditional tests of statistical signicance are premised
on the assumption that inferences should be based
solely on present data, without any regard for what
we might bring to a study in the way of belief or
knowledge about the hypothesis to be testeda
position that Bayesians contend is hardly designed to
maximize our chances of learning from experience.
To achieve their goal of the systematic revision of
opinion on the basis of new information, Bayesians
are able to employ Bayes theorem iteratively. Having obtained a posterior probability assignment for
their hypothesis via Bayes theorem, they can then
go on and use that posterior probability as the new
prior probability in a further use of Bayes theorem
designed to yield a revised posterior probability, and
so on. In this way, the Bayesian researcher learns
from experience.
t h e p h i l o s o p h y o f q u a n t i tat i v e m e t h o d s
Pr(H1 ) Pr(D/H1 )
Pr(H2 )Pr(D/H2 )+Pr(H1 )Pr(D/H1 )
17
Meta-Analysis
In the space of three decades meta-analysis has
become a prominent methodology in behavioral science research, with the major developments coming
from the elds of education and psychology (Glass,
McGaw, & Smith, 1981: Hedges & Olkin, 1985;
Hunter & Schmidt, 2004). Meta-analysis is an
approach to data analysis that involves the quantitative analysis of the data analyses of primary empirical
studies. Hence, the term meta-analysis coined by
Glass (1976). Meta-analysis, which comes in a variety of forms (Bangert-Drowns, 1986), is concerned
with the statistical analyses of the results from many
individual studies in a given domain for the purpose of integrating or synthesizing those research
ndings.
The following selective treatment of metaanalysis considers its possible roles in scientic
explanation and evaluation research before critically
examining one extended argument for the conclusion that meta-analysis is premised on a faulty
conception of science.
t h e p h i l o s o p h y o f q u a n t i tat i v e m e t h o d s
haig
19
Glasss position seems to be that although program treatments can be causally responsible for their
measured outcomes, it matters little that knowledge of this gleaned from evaluation studies does
not tell us how programs produce their effects,
because such knowledge is not needed for policy
action.
Glass is surely correct in asserting that scientists
are centrally concerned with the construction of
causal theories to explain phenomena, for this is the
normal way in which they achieve understanding
of the empirical regularities they discover. However,
he is wrong to insist that proper evaluations should
deliberately ignore knowledge of underlying causal
mechanisms. The reason for this is that the effective
implementation and alteration of social programs
often benets from knowledge of the relevant causal
mechanisms involved (Gottfredson, 1984), and
strategic intervention in respect of these is often
the most effective way to bring about social change.
Although standard versions of scientic realism are
wrong to insist that the relevant causal mechanisms
are always unobserved mechanisms, it is the case that
appeal to knowledge of covert causal mechanisms
will frequently be required for understanding and
change.
To conclude this highly selective evaluation of
Glasss rationale for meta-analysis, science itself is
best understood as a value-laden, problem-oriented
human endeavor that tries to construct causal
explanatory theories of the phenomena it discovers.
There is no sound way of drawing a principled contrast between scientic and evaluative inquiry. These
critical remarks are not directed against the worth of
evaluation as such or against the use of meta-analysis
in evaluating program or product effectiveness. They
are leveled specically at the conception of evaluation research that appears to undergird Glasss
approach to meta-analysis.
t h e p h i l o s o p h y o f q u a n t i tat i v e m e t h o d s
21
t h e p h i l o s o p h y o f q u a n t i tat i v e m e t h o d s
23
using exploratory factor analysis to facilitate judgments about the initial plausibility of hypotheses will
still leave the domains being investigated in a state
of considerable theoretical underdetermination. It
should also be stressed that the resulting plurality of competing theories is entirely to be expected
and should not be thought of as an undesirable
consequence of employing exploratory factor analysis. To the contrary, it is essential for the growth
of scientic knowledge that we vigorously promote
theoretical pluralism (Hooker, 1987), develop theoretical alternatives, and submit them to critical
scrutiny.
24
Causal Modeling
During the last 50 years, social and behavioral
science methodologists have developed a variety of
increasingly sophisticated statistical methods to help
researchers draw causal conclusions from correlational data. These causal modeling methods, as they
have sometimes been called, include path analysis, conrmatory factor analysis, and full structural
equation modeling.
Despite the fact that psychological researchers are
increasingly employing more sophisticated causal
t h e p h i l o s o p h y o f q u a n t i tat i v e m e t h o d s
25
The guess-and-test strategy of the hypotheticodeductive method takes predictive accuracy as the
sole criterion of theory goodness. However, it
seems to be the case that in research practice, the
hypothetico-deductive method is sometimes combined with the use of supplementary evaluative criteria such as simplicity, scope, and fruitfulness. When
this happens, and one or more of the supplementary
criteria have to do with explanation, the combined
approach can appropriately be regarded as a version
of inference to the best explanation, rather than just
an augmented account of the hypothetico-deductive
method (Haig, 2009). This is because the central
characteristic of the hypothetico-deductive method
is a relationship of logical entailment between theory and evidence, whereas with inference to the
best explanation the relationship is also one of
explanation. The hybrid version of inference to
26
Many causal modeling methods are latent variable methods, whose conceptual foundations are to
be found in the methodology of latent variable theory (Borsboom, 2005; 2008). Central to this theory
is the concept of a latent variable itself. However,
the notion of a latent variable is a contested concept,
and there are fundamental philosophical differences
in how it should be understood.
A clear example of the contested nature of the
concept of a latent variable is to be found in the
two quite different interpretations of the nature of
the factors produced by exploratory factor analysis. One view, known as ctionalism, maintains
that the common factors, the output of exploratory
factor analysis, are not theoretical entities invoked
to explain why the observed variables correlate the
way that they do. Rather, these factors are taken
to be summary expressions of the way manifest
variables co-vary. Relatedly, theories that marshal
t h e p h i l o s o p h y o f q u a n t i tat i v e m e t h o d s
Conclusion
The philosophy of research methods is an aspect
of research methodology that receives limited attention in behavioral science education. The majority
of students and research practitioners in the behavioral sciences obtain the bulk of their knowledge of
research methods from textbooks. However, a casual
examination of these texts shows that they tend to
pay little, if any, serious regard to the philosophy
of science and its bearing on the research process.
As Kuhn pointed out nearly 50 years ago (Kuhn,
1962; 1996), textbooks play a major role in dogmatically initiating students into the routine practices
of normal science. Serious attention to the philosophy of research methods would go a considerable
way toward overcoming this uncritical practice. As
contemporary philosophy of science increasingly
focuses on the contextual use of research methods in
the various sciences, it is to be hoped that research
methodologists and other behavioral scientists will
avail themselves of the genuine methodological
insights that it contains.
Future Directions
In this nal section of the chapter, I suggest a
number of directions that future work in the philosophy of quantitative methods might take. The rst
three suggestions are briey discussed; the remaining
suggestions are simply listed.
27
t h e p h i l o s o p h y o f q u a n t i tat i v e m e t h o d s
Additional Directions
Space considerations prevent discussion of additional future directions in the philosophy of quantitative methods. However, the following points
deserve to be on an agenda for future study.
Develop a modern interdisciplinary conception of research methodology.
Give more attention to investigative strategies
in psychological research.
Take major philosophical theories of scientic
method seriously.
Apply insights from the new experimentalism in the philosophy of science to the understanding of quantitative research methods.
Develop the philosophical foundations of theory construction methods in the behavioral sciences.
Assess the implications of different theories of
causality for research methods.
Examine the philosophical foundations of
new research methods such as data mining, structural equation modeling, and functional neuroimaging.
Author note
Brian D. Haig, Department of Psychology, University of Canterbury Christchurch, New Zealand,
Email: brian.haig@canterbury.ac.nz
References
Acree, M. C. (1979). Theories of statistical inference in psychological research: A historico-critical study. Dissertation
Abstracts International, 39, 5073B. (UMI No. 7907000).
Anastasi, A., & Urbina, S. (1997). Psychological testing (7th ed.)
Upper Saddle River, NJ: Prentice-Hall.
Bangert-Drowns, R. L. (1986). Review of developments in metaanalytic method. Psychological Bulletin, 99, 388399.
Behrens, J. T., & Yu, C-H. (2003). Exploratory data analysis. In
J. A. Schinka & W. F. Velicer (Eds.), Handbook of psychology,
Vol. 2 (pp. 3364). New York: Wiley.
Berger, J. O. & Selke, T. (1987). Testing a point null hypothesis:
The irreconcilability of p values and evidence (with comments). Journal of the American Statistical Association, 82,
112139.
Bhaskar, R. (1975). A realist philosophy of science. Brighton,
England: Harvester.
Bhaskar, R. (1979). The possibility of naturalism. Brighton,
England: Harvester.
Block, N. J. (1976). Fictionalism, functionalism, and factor
analysis. In R. S. Cohen, C. A. Hooker, & A. C. Michalos (Eds.), Boston Studies in the Philosophy of Science, Vol. 32,
(pp. 127141). Dordrecht, the Netherlands: Reidel.
Bolles, R. C. (1962). The difference between statistical and
scientic hypotheses. Psychological Reports, 11, 629645.
Borsboom, D. (2005). Measuring the mind: Conceptual issues in
contemporary psychometrics. Cambridge, England: Cambridge
University Press.
Borsboom, D. (2008). Latent variable theory. Measurement:
Interdisciplinary Research and Perspectives, 6, 2553.
Bunge, M. (2008). Bayesianism: Science or pseudoscience?
International Review of Victimology, 15, 165178.
Carruthers, P. (2002). The roots of scientic reasoning: Infancy,
modularity, and the art of tracking. In P. Carruthers, S. Stich,
& M. Siegal (Eds.), The cognitive basis of science (pp. 7395).
Cambridge, England: Cambridge University Press.
Chow, S. L. (1996). Statistical signicance: Rationale, validity, and
utility. London, England: Sage.
Cliff, N. (1983). Some cautions concerning the application of
causal modeling methods. Multivariate Behavioral Research,
18, 115126.
Cohen, J. (1994). The earth is round (p <.05). American
Psychologist, 49, 9971003.
haig
29
30
t h e p h i l o s o p h y o f q u a n t i tat i v e m e t h o d s