The Oxford Handbook of

Quantitative Methods
OX F O R D L I B R A RY O F P S Y C H O L O G Y

Editor-in-Chief

Peter E. Nathan

A r e a E d i t o r s:

Clinical Psychology
David H. Barlow

Cognitive Neuroscience
Kevin N. Ochsner and Stephen M. Kosslyn

Cognitive Psychology
Daniel Reisberg

Counseling Psychology
Elizabeth M. Altmaier and Jo-Ida C. Hansen

Developmental Psychology
Philip David Zelazo

Health Psychology
Howard S. Friedman

History of Psychology
David B. Baker

Methods and Measurement

Todd D. Little

Neuropsychology
Kenneth M. Adams

Organizational Phychology
Steve W. J. Kozlowski

Personality and Social Psychology

Kay Deaux and Mark Snyder
 OX F O R D L I B R A RY O F P S Y C H O L O G Y

Editor-in-Chief peter e. nathan

The Oxford Handbook

of Quantitative
Methods
Edited by
Todd D. Little

Vo lu m e ı : Fo u n d at i o n s

1
3
Oxford University Press, Inc., publishes works that further
Oxford University’s objective of excellence in
research, scholarship, and education.

Oxford New York

Auckland Cape Town Dar es Salaam Hong Kong Karachi
Kuala Lumpur Madrid Melbourne Mexico City Nairobi
New Delhi Shanghai Taipei Toronto

With offices in
Argentina Austria Brazil Chile Czech Republic France Greece
Guatemala Hungary Italy Japan Poland Portugal Singapore
South Korea Switzerland Thailand Turkey Ukraine Vietnam

© 2013 by Oxford University Press, Inc.

Published by Oxford University Press, Inc.

198 Madison Avenue, New York, New York 10016
www.oup.com

Oxford is a registered trademark of Oxford University Press

All rights reserved. No part of this publication may be reproduced,

stored in a retrieval system, or transmitted, in any form or by any means,
electronic, mechanical, photocopying, recording, or
otherwise, without the prior permission of Oxford University Press

Library of Congress Cataloging-in-Publication Data

The Oxford handbook of quantitative methods / edited by Todd D. Little.
v. cm. – (Oxford library of psychology)
ISBN 978–0–19–993487–4
ISBN 978–0–19–993489–8
1. Psychology–Statistical methods. 2. Psychology–Mathematical models. I. Little, Todd D.
BF39.O927 2012
150.72 1—dc23
2012015005

9 8 7 6 5 4 3 2 1
Printed in the United States of America
on acid-free paper
SHORT CONTENTS

Oxford Library of Psychology vii

About the Editor ix

Contributors xi

Table of Contents xvii

Chapters 1–479

Index 480

v
This page intentionally left blank
OX F O R D L I B R A RY O F P SYC H O LO GY

The Oxford Library of Psychology, a landmark series of handbooks, is published by

Oxford University Press, one of the world’s oldest and most highly respected publish-
ers, with a tradition of publishing significant books in psychology. The ambitious goal
of the Oxford Library of Psychology is nothing less than to span a vibrant, wide-ranging
field and, in so doing, to fill a clear market need.
Encompassing a comprehensive set of handbooks, organized hierarchically, the
Library incorporates volumes at different levels, each designed to meet a distinct
need. At one level are a set of handbooks designed broadly to survey the major
subfields of psychology; at another are numerous handbooks that cover important
current focal research and scholarly areas of psychology in depth and detail. Planned
as a reflection of the dynamism of psychology, the Library will grow and expand
as psychology itself develops, thereby highlighting significant new research that will
impact on the field. Adding to its accessibility and ease of use, the Library will be
published in print and, later, electronically.
The Library surveys psychology’s principal subfields with a set of handbooks that
capture the current status and future prospects of those major subdisciplines. This ini-
tial set includes handbooks of social and personality psychology, clinical psychology,
counseling psychology, school psychology, educational psychology, industrial and
organizational psychology, cognitive psychology, cognitive neuroscience, methods
and measurements, history, neuropsychology, personality assessment, developmen-
tal psychology, and more. Each handbook undertakes to review one of psychology’s
major subdisciplines with breadth, comprehensiveness, and exemplary scholarship.
In addition to these broadly conceived volumes, the Library also includes a large
number of handbooks designed to explore in depth more specialized areas of schol-
arship and research, such as stress, health and coping, anxiety and related disorders,
cognitive development, or child and adolescent assessment. In contrast to the broad
coverage of the subfield handbooks, each of these latter volumes focuses on an espe-
cially productive, more highly focused line of scholarship and research. Whether
at the broadest or most specific level, however, all of the Library handbooks offer
synthetic coverage that reviews and evaluates the relevant past and present research
and anticipates research in the future. Each handbook in the Library includes intro-
ductory and concluding chapters written by its editor to provide a roadmap to the
handbook’s table of contents and to offer informed anticipations of significant future
developments in that field.
An undertaking of this scope calls for handbook editors and chapter authors
who are established scholars in the areas about which they write. Many of the

vii
 nation’s and world’s most productive and best-respected psychologists have agreed
to edit Library handbooks or write authoritative chapters in their areas of expertise.
For whom has the Oxford Library of Psychology been written? Because of its breadth,
depth, and accessibility, the Library serves a diverse audience, including graduate
students in psychology and their faculty mentors, scholars, researchers, and practi-
tioners in psychology and related fields. Each will find in the Library the information
they seek on the subfield or focal area of psychology in which they work or are
interested.
Befitting its commitment to accessibility, each handbook includes a comprehen-
sive index, as well as extensive references to help guide research. And because the
Library was designed from its inception as an online as well as a print resource,
its structure and contents will be readily and rationally searchable online. Further,
once the Library is released online, the handbooks will be regularly and thoroughly
updated.
In summary, the Oxford Library of Psychology will grow organically to provide
a thoroughly informed perspective on the field of psychology, one that reflects
both psychology’s dynamism and its increasing interdisciplinarity. Once published
electronically, the Library is also destined to become a uniquely valuable interactive
tool, with extended search and browsing capabilities. As you begin to consult this
handbook, we sincerely hope you will share our enthusiasm for the more than 500-
year tradition of Oxford University Press for excellence, innovation, and quality, as
exemplified by the Oxford Library of Psychology.

Peter E. Nathan
Editor-in-Chief
Oxford Library of Psychology

viii oxford library of psychology

ABOUT THE EDITOR

Todd D. Little
Todd D. Little, Ph.D., is a Professor of Psychology, Director of the Quantitative
training program, Director of the undergraduate Social and Behavioral Sciences
Methodology minor, and a member of the Developmental training program. Since
2010, Todd has been Director of the Center for Research Methods and Data Analysis
(CRMDA) at Kansas University. Little is internationally recognized for his quanti-
tative work on various aspects of applied SEM (e.g., indicator selection, parceling,
modeling developmental processes) as well as his substantive developmental research
(e.g., action-control processes and motivation, coping, and self-regulation). In
2001, Little was elected to membership in the Society for Multivariate Experimental
Psychology. In 2009, he was elected President of APA’s Division 5 (Evaluation, Mea-
surement, and Statistics) and in 2010 was elected Fellow of the division. In 2012,
he was elected Fellow in the Association for Psychological Science. He founded,
organizes, and teaches in the internationally renowned KU “Stats Camps” each June
(see crmda.KU.edu for details of the summer training programs). Little has edited
five books related to methodology including The Oxford Handbook of Quantitative
Methods and the Guilford Handbook of Developmental Research Methods (with Brett
Laursen and Noel Card). Little has been principal investigator or co-principal inves-
tigator on more than 15 grants and contracts, statistical consultant on more than 60
grants and he has guided the development of more than 10 different measurement
tools.

ix
This page intentionally left blank
 CONTRIBUTORS

Leona S. Aiken Timothy A. Brown

Department of Psychology Department of Psychology
Arizona State University Boston University
Tempe, AZ Boston, MA
Rawni A. Anderson Trent D. Buskirk
Center for Research Methods Department of Community Health-
and Data Analysis Biostatistics Division
University of Kansas Saint Louis University
Lawrence, KS Saint Louis, MO
Luc Anselin Noel A. Card
GeoDa Center for Geospatial Analysis Family Studies and Human Development
and Computation University of Arizona
School of Geographical Sciences and Tucson, AZ
Urban Planning Deborah M. Casper
Arizona State University Family Studies and Human Development
Tempe, AZ University of Arizona
Amanda N. Baraldi Tucson, AZ
Department of Psychology Daniel R. Cavagnaro
Arizona State University Department of Psychology
Tempe, AZ The Ohio State University
David E. Bard Columbus, OH
Department of Pediatrics Rand D. Conger
University of Oklahoma Health Sciences Department of Human and Community
Center Development
Oklahoma City, OK University of California at Davis
Theodore P. Beauchaine Davis, CA
Department of Psychology David Cook
Washington State University Abt Associates Inc.
Pullman, WA Thomas D. Cook
Gabriëlla A.M. Blokland Institute for Policy Research
Genetic Epidemiology Laboratory Northwestern University
Queensland Institute of Evanston, IL
Medical Research Stefany Coxe
School of Psychology and Centre Department of Psychology
for Advanced Imaging Arizona State University
University of Queensland Tempe, AZ
Brisbane, Australia R.J. de Ayala
Annette Brose Department of Educational
Max Plank Institute for Human Psychology
Development University of Nebraska Lincoln
Berlin, Germany Lincoln, NE
Max Plank Institute for Human Cognitive
and Brain Sciences

xi
Pascal R. Deboeck Brian D. Haig
Department of Psychology Department of Psychology
University of Kansas University of Canterbury
Lawrence, KS Canterbury, New Zealand
Sarah Depaoli Kelly Hallberg
Department of Educational Psychology Institute for Policy Research
University of Wisconsin Madison Northwestern University
Madison, WI Evanston, IL
Cody S. Ding Lisa L. Harlow
College of Education Department of Psychology
University of Missouri-Saint Louis University of Rhode Island
Saint Louis, MO Kingston, RI
M. Brent Donnellan Emily J. Hart
Department of Psychology Department of Psychology
Michigan State University University at Buffalo
East Lansing, MI The State University of New York
Dawnté R. Early Buffalo, NY
Department of Human and Community Kit-Tai Hau
Development The Chinese University of Hong Kong
University of California at Davis Hong Kong
Davis, CA Joop J. Hox
Craig K. Enders Department of Methodology and Statistics
Department of Psychology Utrecht University
Arizona State University Utrecht, The Netherlands
Tempe, AZ James Jaccard
David M. Erceg-Hurn Department of Psychology
School of Psychology Florida International University
University of Western Australia Boca Raton, FL
Crawley, WA, Australia Paul E. Johnson
Aurelio José Figueredo Department of Political Science
Department of Psychology Kansas University
School of Mind, Brain, & Behavior Lawrence, KS
Division of Family Studies and Human Kelly M. Kadlec
Development Department of Psychology
College of Agriculture and Life Sciences University of Southern California
University of Arizona Los Angeles, CA
Tucson, AZ David Kaplan
Rafael Antonio Garcia Department of Educational
Department of Psychology Psychology
School of Mind, Brain, & Behavior University of Wisconsin-Madison
University of Arizona Madison, WI
Tucson, AZ Ken Kelley
Amanda C. Gottschall Department of Management
Department of Psychology University of Notre Dame
Arizona State University Notre Dame, IN
Tempe, AZ Harvey J. Keselman
Michael J. Greenacre Department of Psychology
Department of Economics and Business University of Manitoba
Universitat Pompeu Fabra, Barcelona Winnipeg, Canada
Barcelona, Spain

xii contributors
Neal M. Kingston Professor Emeritus
School of Education Macquarie University
University of Kansas † April 16, 1928 – October, 29, 2011
Lawrence, KS Sarah E. Medland
Yasemin Kisbu-Sakarya Genetic Epidemiology Laboratory
Department of Psychology Queensland Institute of Medical Research
Arizona State University School of Psychology
Tempe, AZ University of Queensland
Laura B. Kramer Brisbane, Australia
School of Education Peter C. M. Molenaar
University of Kansas Department of Human Development
Lawrence, KS and Family Studies
Todd D. Little Pennsylvania State University
Center for Research Methods and Data University Park, PA
Analysis Alexandre J.S. Morin
Department of Psychology Department of Psychology
University of Kansas University of Sherbrooke
Lawrence, KS Sherbrooke, Quebec, Canada
Richard E. Lucas Miriam A. Mosing
Department of Psychology Genetic Epidemiology Laboratory
Michigan State University Queensland Institute of Medical Research
East Lansing, MI School of Psychology
David P. MacKinnon University of Queensland
Department of Psychology Brisbane, Australia
Arizona State University Keith E. Muller
Tempe, AZ Department of Health Outcomes
Patrick Mair and Policy
Institute for Statistics and Mathematics University of Florida
Vienna University of Economics Gainesville, FL
and Business Eun-Young Mun
Vienna, Austria Center of Alcohol Studies
Herbert W. Marsh Rutgers University
Department of Education Piscataway, NJ
University of Oxford Alan T. Murray
Oxford, UK GeoDa Center for Geospatial Analysis
Katherine E. Masyn and Computation
Graduate School of Education School of Geographical Sciences
Harvard University and Urban Planning
Cambridge, MA Arizona State University
John J. McArdle Tempe, AZ
Department of Psychology Jay I. Myung
University of Southern California Department of Psychology
Los Angeles, CA The Ohio State University
Roderick P. McDonald† Columbus, OH
Sydney University Benjamin Nagengast
Sydney, Australia Department of Education
Professor Emeritus Oxford University
University of Illinois Oxford, UK
at Urbana-Champaign

contributors xiii
Sally Gayle Olderbak André A. Rupp
Department of Psychology Department of Measurement, Statistics,
School of Mind, Brain, & and Evaluation (EDMS)
Behavior University of Maryland
University of Arizona College Park, MD
Tucson, AZ Gabriel Lee Schlomer
Jamie M. Ostrov Division of Family Studies and
Department of Psychology Human Development
University at Buffalo College of Agriculture and Life Sciences
The State University of New York University of Arizona
Buffalo, NY Tucson, AZ
Trond Peterson Christof Schuster
Department of Sociology Department of Psychology
University of California-Berkeley Justus-Liebig-Unversitat Giessen
Berkeley, CA Giessen, Germany
Mark A. Pitt James P. Selig
Department of Psychology Department of Psychology
The Ohio State University University of New Mexico
Columbus, OH Albuquerque, NM
Larry R. Price Paul E. Spector
College of Education and College of Science Department of Psychology
Texas State University-San Marcos University of South Florida
San Marcos, TX Tampa, FL
Nilam Ram Peter M. Steiner
Department of Human Development and Department of Educational Psychology
Family Studies University of Wisconsin-Madison
Pennsylvania State University Madison, WI
University Park, PA Carolin Strobl
Max Plank Institute for Human Ludwig-Maximilians-Universität
Development München Faculty of Mathematics,
Berlin, Germany Informatics and Statistics Institute
Sergio J. Rey of Statistics
GeoDa Center for Geospatial Analysis Munich, Germany
and Computation Bruce Thompson
School of Geographical Sciences and Baylor College of Medicine
Urban Planning Austin, TX
Arizona State University Terry T. Tomazic
Tempe, AZ Department of Sociology and
Joseph L. Rodgers Criminal Justice
Department of Psychology Saint Louis University
University of Oklahoma Saint Louis, MO
Norman, OK James T. Townsend
Robert Rosenthal Department of Psychological and
Department of Psychology Brain Sciences
University of California, Riverside Indiana University
Riverside, CA Bloomington, IN
Ralph L. Rosnow Trisha Van Zandt
Department of Psychology Department of Psychology
Temple University The Ohio State University
Philadelphia, PA Columbus, OH

xiv contributors
Alexander von Eye Keith F. Widaman
Departments of Psychology Department of Psychology
Michigan State University University of California at Davis
East Lansing, MI Davis, CA
University of Vienna Rand R. Wilcox
Vienna, Austria Department of Psychology
Stefan von Weber University of Southern California
Department of Mechanical Engineering Los Angeles, CA
Hochschule Furtwangen University Lisa M. Willoughby
Furtwangen im Schwarzwald, Germany Department of Psychology
Karin J.H. Verweij Saint Louis University
Genetic Epidemiology Laboratory Saint Louis, MO
Queensland Institute of Medical Research Coady Wing
School of Psychology Institute for Policy Research
University of Queensland Northwestern University
Brisbane, Australia Evanston, IL
Theodore A. Walls Pedro Sofio Abril Wolf
Department of Behavioral Science Department of Psychology
University of Rhode Island University of Cape Town
Kingston, RI Cape Town, South Africa
Lihshing Leigh Wang Vivian Wong
Educational Studies Program School of Education and Social Policy
University of Cincinnati Northwestern University
Cincinnati, OH Evanston, IL
Amber S. Watts Carol M. Woods
Center for Research Methods and Data Center for Research Methods and
Analysis and Data Analysis and
Lifespan Institute, Gerontology Center Department of Psychology
University of Kansas University of Kansas
Lawrence, KS Lawrence, KS
William W.S. Wei Wei Wu
Department of Statistics Center for Research Methods and
Temple University Data Analysis and
Philadelphia, PA Department of Psychology
Zhonglin Wen University of Kansas
South China Normal University Lawrence, KS
Stephen G. West Ke-Hai Yuan
Department of Psychology Department of Psychology
Arizona State University University of Notre Dame
Tempe, AZ Notre Dame, IN

contributors xv
This page intentionally left blank
CONTENTS

1. Introduction 1
Todd D. Little
2. The Philosophy of Quantitative Methods 7
Brian D. Haig
3. Quantitative Methods and Ethics 32
Ralph L. Rosnow and Robert Rosenthal
4. Special Populations 55
Keith F. Widaman, Dawnté R. Early, and Rand D. Conger
5. Theory Construction, Model Building, and Model Selection 82
James Jaccard
6. Teaching Quantitative Psychology 105
Lisa L. Harlow
7. Modern Test Theory 118
Roderick P. McDonald
8. The IRT Tradition and its Applications 144
R.J. de Ayala
9. Survey Design and Measure Development 170
Paul E. Spector
10. High-Stakes Test Construction and Test Use 189
Neal M. Kingston and Laura B. Kramer
11. Effect Size and Sample Size Planning 206
Ken Kelley
12. Experimental Design for Causal Inference: Clinical Trials and
Regression Discontinuity Designs 223
Kelly Hallberg, Coady Wing, Vivian Wong, and Thomas D. Cook
13. Matching and Propensity Scores 237
Peter M. Steiner and David Cook
14. Designs for and Analyses of Response Time Experiments 260
Trisha Van Zandt and James T. Townsend
15. Observational Methods 286
Jamie M. Ostrov and Emily J. Hart

xvii
 16. A Primer of Epidemiologic Methods, Concepts, and Analysis with
Examples and More Advanced Applications within Psychology 305
David E. Bard, Joseph L. Rodgers, and Keith E. Muller
17. Program Evaluation: Principles, Procedures, and Practices 332
Aurelio José Figueredo, Sally Gayle Olderbak, Gabriel Lee Schlomer,
Rafael Antonio Garcia, and Pedro Sofio Abril Wolf
18. Overview of Statistical Estimation Methods 361
Ke-Hai Yuan and Christof Schuster
19. Robust Statistical Estimation 388
David M. Erceg-Hurn, Rand R. Wilcox, and Harvey J. Keselman
20. Bayesian Statistical Methods 407
David Kaplan and Sarah Depaoli
21. Mathematical Modeling 438
Daniel R. Cavagnaro, Jay I. Myung, and Mark A. Pitt
22. Monte Carlo Analysis in Academic Research 454
Paul E. Johnson

Index 480

xviii contents
 CHAPTER

Introduction
1
Todd D. Little

Abstract

In this introductory chapter to The Oxford Handbook of Quantitative Methods, I provide an overview of
the two volumes. More specifically, I describe the rationale and motivation for the selected topics that
are presented in volumes. I also list out my instructions to the chapter authors and then describe how
the chapters fit together into thematic groupings. I also extend my sincerest gratitude to the persons
who assisted me along the way, as no work this comprehensive can be done without the considerable
help and assistance of many persons. I conclude with how pleased I am with the quality and
comprehensiveness of the chapters that are included.
Key Words: Overview; Quantitative Methods; Methodology; Statistics

Oxford Introduction synthetic Library of Handbooks covering the field

Handbooks provide a crucial venue to communi-
cate the current state of the field. They also provide
a one-stop source for learning and reviewing cur-
rent best practices in a field. The Oxford Handbook of
Quantitative Methods serves both of these functions.
The field of quantitative methods is quite broad, as
you can probably imagine. I have tried to be thor-
ough in my selection of topics to be covered. As with
any handbook of this magnitude, some topics were
all set to have a contribution submitted, only to have
some unforeseen hindrance preclude its inclusion
at the last minute (e.g., graphical representations
of data, ecological inference, history of quantita-
tive methods). Some topics overlap with others and
may not have found their way to become a separate
chapter, but their fundamental elements are found
in parts of other chapters.
This handbook is one of many that Oxford
University Press (OUP) is assembling but will be
the capstone methodology handbook. As many of
you know, OUP is building a comprehensive and
of psychology (the Editor-in-Chief of the library
is Peter Nathan, University of Iowa Foundation
Distinguished Professor of Psychology and Public
Health). The library comprises handbooks in the
truest sense of the word: books that summarize and
synthesize a topic, define the current scholarship,
and set the agenda for future research. Each hand-
book is published as a bound book, and it will also
be developed for electronic delivery. In this format,
the content will be integrated across topics and avail-
able as a fully integrated electronic library. I think
the idea of a comprehensive electronic library is very
forward-thinking. This format is a very attractive
opportunity to have a fully comprehensive and up-
to-date handbook of methods in our field. Hence,
I agreed to take on the role of editor of The Oxford
Handbook of Quantitative Methods.
I am very pleased with the quality of the work
that each author provided. As per my request to the
contributing authors, each chapter is meant to be
both accessible and comprehensive; nearly all the

1
authors were very responsive to my requests. The
guidelines I asked authors to consider were:
• Handbook chapters should be comprehensive
and authoritative; readers will rely heavily on these
chapters, particularly when they move to the online
format.
• Handbook chapters should present not only
the strengths of the topic covered but also any
limitations.
• Handbook chapters should make all assump-
tions underlying the topic explicit.
• Regarding citations, handbook chapters
should cover the historical origins as well as the
recent renditions of a given key topic.
• Handbook chapters should not present one-
sided views on any debate; rather, they should report
the issues and present the arguments—both pro and
con. Authors can direct readers to other platforms
where a position piece is presented.
• To facilitate the online linkages, handbook
chapters should point to other online resources
related to the topic presented.
• Every element of every formula presented must
be explicitly explained; assume no knowledge of how
to read formulae.
• Examples, examples, examples, and, when
in doubt, provide an example! Concrete examples
are absolutely critical to communicate quantitative
content.
• Avoid jargon and acronyms. Please spell out
acronyms, and if you use jargon, please remind the
reader of the meaning or definition of the jargon
every three to four times it is used; similarly, if you
use an acronym, then remind the reader of what it
means every three to four times it is used.
• Use active voice, and do not shy away from the
use of I/me or we/us. Channel how you lecture on
the topic. It will create a crisp and enjoyable read.
• Do not start a sentence with “This” followed
by a verb. The referent to “this” must be restated
because of the ambiguity this creates. This general
guideline should be followed as a rule!
Authors, like editors, have preferences and habits,
so you will find places, chapters, and so on where
some of my admonitions were not followed. But the
quality of the product that each chapter provides is
nonetheless uncompromised. We have established a
Wiki-based resource page for the handbook, which
can be found at crmda.KU.edu/oxford. Each author
has been asked to maintain and upload materials to
2 introduction
support his or her chapter contribution. At the top of
that page is a link that encourages you to offer com-
ments and suggestions on the topics and coverage
of the handbook. These comments will be reviewed
and integrated into future editions of this handbook.
I encourage you, therefore, to take advantage of this
opportunity to help shape the directions and content
coverage of this handbook.
Statistical software has blossomed with the advent
of hardware that provides the necessary speed and
memory and programming languages coupled with
numerical algorithms that are more efficient and
optimized than yesteryear. These software advances
have allowed many of the advances in modern statis-
tics to become accessible to the typical end-user.
Modern missing data algorithms and Bayesian esti-
mation procedures, for example, have been the
beneficiaries of these advances. Of course, some of
the software developments have included simplified
interfaces with slick graphic user interfaces. The crit-
ical options are usually prefilled with default settings.
These latter two aspects of advancing software are
unfortunate because they lead to mindless applica-
tions of the statistical techniques. I would prefer
that options not be set as default but, rather, have
the software prompt the user to make a choice (and
give good help for what each choice means). I would
prefer that a complete script of the GUI choices and
the order in which steps were taken be automatically
saved and displayed.
I have organized the handbook by starting with
some basics. It begins with the philosophical under-
pinnings associated with science and quantitative
methods (Haig, Chapter 2, Volume 1) followed
by a discussion of how to construct theories and
models so that they can be tested empirically
and the best model selected (Jaccard, Chapter 5,
Volume 1). I then turn to an enlightened discus-
sion of ethics in the conduct of quantitative research
(Rosnow & Rosenbloom, Chapter 3, Volume 1)
and related issues when quantitative methods are
applied in special populations (Widaman, Early, &
Conger, Chapter 4, Volume 1). Harlow (Chapter 6,
Volume 1) follows with an encompassing and
impassioned discussion of teaching quantitative
methods.
The theme in the next grouping of chapters
centers on measurement issues. First, the late
McDonald (Chapter 17, Volume 1) provides a thor-
ough overview of Modern Test Theory.1 De Ayala
(Chapter 8, Volume 1) adds a detailed discussion
of Item Response Theory as an essential mea-
surement and analysis tool. After these principles
of measurement are discussed, the principles and
practices surrounding survey design and measure
development are presented (Spector, Chapter 9,
Volume 1). Kingston and Kramer (Chapter 10,
Volume 1) further this discussion in the context of
high-stakes testing.
A next grouping of chapters covers various design
issues. Kelley (Chapter 11, Volume 1) begins
this section by covering issues of power, effect
size, and sample size planning. Hallberg, Wing,
Wong, and Cook (Chapter 12, Volume 1) then
address key experimental designs for causal infer-
ence: the gold standard randomized clinical trials
(RCT) design and the underutilized regression dis-
continuity design. Some key quasi-experimental
procedures for comparing groups are discussed
in Steiner and Cooks’ (Chapter 13, Volume 1)
chapter on using matching and propensity scores.
Finally, Van Zandt and Townsend (Chapter 14,
Volume 1) provide a detailed discussion of the
designs for and analyses of response time experi-
ments. I put observational methods (Ostrov & Hart,
Chapter 15, Volume 1), epidemiological methods
(Bard, Rodgers, & Mueller, Chapter 16, Volume 1),
and program evaluation (Figueredo, Olderbak, &
Schlomer, Chapter 17, Volume 1) in with these
chapters because they address more collection and
design issues, although the discussion of program
evaluation also addresses the unique analysis and
presentation issues.
I have a stellar group of chapters related to esti-
mation issues. Yuan and Schuster (Chapter 18,
Volume 1) provide an overview of statistical estima-
tion method; Erceg-Hurn, Wilcox, and Keselman
(Chapter 19, Volume 1) provide a nice comple-
ment with a focus on robust estimation tech-
niques. Bayesian statistical estimation methods are
thoroughly reviewed in the Kaplan and Depaoli
(Chapter 20, Volume 1) contribution. The details
of mathematical modeling are synthesized in this
section by Cavagnaro, Myung, and Pitt (Chapter 21,
Volume 1). This section is completed by John-
son (Chapter 22, Volume 1), who discusses the
many issues and nuances involved in conducting
Monte Carlo simulations to address the what-
would-happen-if questions that we often need to
answer.
The foundational techniques for the statistical
analysis of quantitative data start with a detailed
overview of the traditional methods that have
marked social and behavioral sciences (i.e., the
General Linear Model; Thompson, Chapter 2,
Volume 2). Coxe, West, and Aiken (Chapter 3,
Volume 2) then extend the General Linear Model
to discuss the Generalized Linear Model. This dis-
cussion is easily followed by Woods (Chapter 4,
Volume 2), who synthesizes the various techniques
of analyzing categorical data. After the chapter on
configural frequency analysis by Von Eye, Mun,
Mair and von Weber (Chapter 5, Volume 5), I then
segway into nonparametric techniques (Buskirk,
Tomazic, & Willoughby, Chapter 6, Volume 2) and
the more specialized techniques of correspondence
analysis (Greenacre, Chapter 7, Volume 2) and spa-
tial analysis (Anselin, Murry, & Rey, Chapter 8,
Volume 2). This section is capped with chapters
dedicated to special areas of research—namely,
techniques and issues related to the analysis of
imaging data (e.g., fMRI; Price, Chapter 9, Vol-
ume 2). The closely aligned worlds of behavior
genetics (i.e., twin studies; Blokland, Mosing,
Verweij, & Medland, Chapter 10, Volume 2)
and genes (Medland, Chapter 11, Volume 2)
follows.
The foundations of multivariate techniques are
grouped beginning with Ding’s (Chapter 12, Vol-
ume 2) presentation of multidimensional scaling
and Brown’s (Chapter 13, Volume 2) summary
of the foundations of latent variable measurement
models. Hox layers in the multilevel issues as han-
dled in both the manifest regression framework
and the latent variable work of structural equation
modeling. McArdle and Kadlec (Chapter 15, Vol-
ume 2) detail, in broad terms, different structural
equation models and their utility. MacKinnon,
Kisbu-Sakarya, and Gottschall (Chapter 16, Vol-
ume 2) address the many new developments in
mediation analysis, while Marsh, Hau, Wen, and
Nagengast (Chapter 17, Volume 2) do the same for
analyses of moderation.
The next group of chapters focuses on repeated
measures and longitudinal designs. It begins with
a chapter I co-wrote with Wu and Selig and pro-
vides a general overview of longitudinal models
(Wu, Selig, & Little, Chapter 18, Volume 2).
Deboeck (Chapter 19, Volume 2) takes things fur-
ther into the burgeoning world of dynamical systems
and continuous-time models for longitudinal data.
Relatedly, Walls (Chapter 20, Volume 2) provides
an overview of designs for doing intensive longitu-
dinal collection and analysis designs. The wonderful
world of dynamic-factor models (a multivariate
model for single-subject data) is presented by Ram,
Brose, and Molenaar (Chapter 21, Volume 2). Wei
(Chapter 22, Volume 2) covers all the issues of tradi-
tional time-series models and Peterson (Chapter 23,
little 3
Volume 2) rounds out this section with a thorough
coverage of event history models.
The volume finishes with two small sections.
The first focuses on techniques dedicated to find-
ing heterogeneous subgroups in one’s data. Rupp
(Chapter 24, Volume 2) covers tradition clustering
and classification procedures. Masyn and Nylund-
Gibson (Chapter 25, Volume 2) cover the model-
based approaches encompassed under the umbrella
of mixture modeling. Beauchaine (Chapter 26, Vol-
ume 2) completes this first group with his coverage
of the nuances of taxometrics. The second of the
final group of chapters covers issues related to sec-
ondary analyses of extant data. I put the chapter on
missing data in here because it generally is applied
after data collection occurs, but it is also a little out
of order here because of the terrific and powerful fea-
tures of planned missing data designs. In this regard,
Baraldi and Enders (Chapter 27, Volume 2) could
have gone into the design section. Donnellan and
Lucas (Chapter 28, Volume 2) cover the issues asso-
ciated with analyzing the large-scale archival data
sets that are available via federal funding agencies
such as NCES, NIH, NSF, and the like. Data mining
can also be classified as a set of secondary modeling
procedures, and Strobl’s (Chapter 29, Volume 2)
chapter covers the techniques and issues in this
emerging field of methodology. Card and Casper
(Chapter 30, Volume 2) covers the still advancing
world of meta-analysis and current best practices
in quantitative synthesis of published studies. The
final chapter of The Oxford Handbook of Quantita-
tive Methods is one I co-authored with Wang, Watts,
and Anderson (Wang, Watts, Anderson, & Little,
Chapter 31, Volume 2). In this capstone chapter,
we address the many pervasive fallacies that still
permeate the world of quantitative methodology.
A venture such as this does involve the gener-
ous and essential contributions of expert reviewers.
Many of the chapter authors also served as review-
ers for other chapters, and I won’t mention them by
name here. I do want to express gratitude to a num-
ber of ad hoc reviewers who assisted me along the
way (in arbitrary order): Steve Lee, Kris Preacher,
Mijke Rhemtulla, Chantelle Dowsett, Jason Lee,
Michael Edwards, David Johnson (I apologize now
if I have forgotten that you reviewed a chapter for
me!). I also owe a debt of gratitude to Chad Zim-
merman at OUP, who was relentless in guiding us
through the incremental steps needed to herd us all
to a final and pride-worthy end product and to Anne
Dellinger who was instrumental in bringing closure
to this mammoth project.
4 introduction
Author note
Partial support for this project was provided by
grant NSF 1053160 (Todd D. Little & Wei Wu,
co-PIs) and by the Center for Research Methods
and Data Analysis at the University of Kansas (Todd
D. Little, director). Correspondence concerning this
work should be addressed to Todd D. Little, Center
for Research Methods and Data Analysis, Univer-
sity of Kansas, 1425 Jayhawk Blvd. Watson Library,
470. Lawrence, KS 66045. E-mail: yhat@ku.edu.
Web: crmda.ku.edu.
Note
1. This chapter was completed shortly before Rod’s unex-
pected passing. His legacy and commitment to quantitative
methods was uncompromising and we will miss his voice of wis-
dom and his piercing intellect; R.I .P., Rod McDonald and, as
you once said, pervixi… .
References
Anselin, L., Murry, A. T., & Rey, S. J. (2012). Spatial analysis.
In T. D. Little (Ed.), The Oxford Handbook of Quantitative
Methods (Vol. 2, pp. 154–174). New York: Oxford University
Press.
Baraldi, A. N. & Enders, C. K. (2012). Missing data methods.
In T. D. Little (Ed.), The Oxford Handbook of Quantitative
Methods (Vol. 2, pp. 635–664). New York: Oxford University
Press.
Bard, D. E., Rodgers, J. L., & Muller, K. E. (2012). A primer of
epidemiology methods with applications in psychology. In T.
D. Little (Ed.), The Oxford Handbook of Quantitative Methods
(Vol. 1, pp. 305–349). New York: Oxford University Press.
Beauchaine, T. P. (2012). Taxometrics. In T. D. Little (Ed.),
The Oxford Handbook of Quantitative Methods (Vol. 2,
pp. 612–634). New York: Oxford University Press.
Brown, T. A. (2012). Latent variable measurement models. In T.
D. Little (Ed.), The Oxford Handbook of Quantitative Methods
(Vol. 2, pp. 257–280). New York: Oxford University Press.
Buskirk, T. D., Tomazic, T. T., & Willoughbby, L. (2012).
Nonparametric statistical techniques. In T. D. Little (Ed.),
The Oxford Handbook of Quantitative Methods (Vol. 2,
pp. 106–141). New York: Oxford University Press.
Card, N. A. & Casper, D. M. (2012). Meta-analysis and quan-
titative research synthesis. In T. D. Little (Ed.), The Oxford
Handbook of Quantitative Methods (Vol. 2, pp. 701–717).
New York: Oxford University Press.
Cavagnaro, D. R., Myung, J. I., & Pitt, M. A. (2012). Mathe-
matical modeling. In T. D. Little (Ed.), The Oxford Handbook
of Quantitative Methods (Vol. 1, pp. 438–453). New York:
Oxford University Press.
Coxe, S., West, S. G., & Aiken, L. S. (2012). Generalized lin-
ear models. In T. D. Little (Ed.), The Oxford Handbook of
Quantitative Methods (Vol. 2, pp. 26–51). New York: Oxford
University Press.
De Ayala, R. J. (2012). The IRT tradition and its applications.
In T. D. Little (Ed.), The Oxford Handbook of Quantitative
Methods (Vol. 1, pp. 144–169). New York: Oxford University
Press.
Deboeck, P. R. (2012). Dynamical systems and models of con-
tinuous time. In T. D. Little (Ed.), The Oxford Handbook
 of Quantitative Methods (Vol. 2, pp. 411–431). New York:
Oxford University Press.
Ding, C. S. (2012). Multidimensional scaling. In T. D. Little
(Ed.), The Oxford Handbook of Quantitative Methods (Vol. 2,
pp. 7–25). New York: Oxford University Press.
Donnellan, M. B. & Lucas, R. E. (2012). Secondary data analysis.
In T. D. Little (Ed.), The Oxford Handbook of Quantitative
Methods (Vol. 2, pp. 665–677). New York: Oxford University
Press.
Erceg-Hurn, D. M., Wilcox, R. R., & Keselman, H. H. (2012).
Robust statistical estimation. In T. D. Little (Ed.), The Oxford
Handbook of Quantitative Methods (Vol. 1, pp. 388–406).
New York: Oxford University Press.
Figueredo, A. J., Olderbak, S. G., & Schlomer, G. L. (2012)
Program evaluation: Principles, procedures, and practices.
In T. D. Little (Ed.), The Oxford Handbook of Quantitative
Methods (Vol. 1, pp. 332–360). New York: Oxford University
Press.
Greenacre, M. J. (2012). Correspondence analysis. In T. D. Little
(Ed.), The Oxford Handbook of Quantitative Methods (Vol. 2,
pp. 142–153). New York: Oxford University Press.
Haig, B. D. (2012). The philosophy of quantitative methods.
In T. D. Little (Ed.), The Oxford Handbook of Quantitative
Methods (Vol. 1, pp. 7–31). New York: Oxford University
Press.
Hallberg, K., Wing, C., Wong, V., & Cook, T. D. (2012). Exper-
imental design for causal inference: Clinical trials and regres-
sion discontinuity designs. In T. D. Little (Ed.), The Oxford
Handbook of Quantitative Methods (Vol. 1, pp. 223–236).
New York: Oxford University Press.
Harlow, L. (2012). Teaching quantitative psychology. In
T. D. Little (Ed.), The Oxford Handbook of Quantitative Meth-
ods (Vol. 1, pp. 105–117). New York: Oxford University
Press.
Hox, J. J., (2012). Multilevel regression and multilevel struc-
tural equation modeling In T. D. Little (Ed.), The Oxford
Handbook of Quantitative Methods (Vol. 2, pp. 281–294).
New York: Oxford University Press.
Jaccard, J. (2012). Theory construction, model building, and
model selection. In T. D. Little (Ed.), The Oxford Handbook
of Quantitative Methods (Vol. 1, pp. 82–104). New York:
Oxford University Press.
Johnson, P. E. (2012). Monte Carlo analysis in academic research.
In T. D. Little (Ed.), The Oxford Handbook of Quantitative
Methods (Vol. 1, pp. 454–479). New York: Oxford University
Press.
Kaplan, D. & Depaoli, S. (2012). Bayesian statistical methods.
In T. D. Little (Ed.), The Oxford Handbook of Quantitative
Methods (Vol. 1, pp. 407–437). New York: Oxford University
Press.
Kelley, K. (2012). Effect size and sample size planning. In
T. D. Little (Ed.), The Oxford Handbook of Quantitative Meth-
ods (Vol. 1, pp. 206–222). New York: Oxford University
Press.
Kingston, N. M. & Kramer, L. B. (2012). High stakes test con-
struction and test use. In T. D. Little (Ed.), The Oxford
Handbook of Quantitative Methods (Vol. 1, pp. 189–205).
New York: Oxford University Press.
MacKinnon, D. P., Kisbu-Sakarya, Y., & Gottschall, A. C.
(2012). Developments in mediation analysis. In T. D.
Little (Ed.), The Oxford Handbook of Quantitative Meth-
ods (Vol. 2, pp. 338–360). New York: Oxford University
Press.
Marsh, H. W., Hau, K-T., Wen, Z., & Nagengast, B. (2012).
Moderation. In T. D. Little (Ed.), The Oxford Handbook
of Quantitative Methods (Vol. 2, pp. 361–386). New York:
Oxford University Press.
Masyn, K. E. & Nylund-Gibson, K. (2012). Mixture modeling.
In T. D. Little (Ed.), The Oxford Handbook of Quantitative
Methods (Vol. 2, pp. 551–611). New York: Oxford University
Press.
McArdle, J. J. & Kadlec, K. M. (2012). Structural equation mod-
els. In T. D. Little (Ed.), The Oxford Handbook of Quantitative
Methods (Vol. 2, pp. 295–337). New York: Oxford University
Press.
McDonald, R. P. (2012). Modern test theory. In T. D. Little
(Ed.), The Oxford Handbook of Quantitative Methods (Vol. 1,
pp. 118–143). New York: Oxford University Press.
Medland, S. E. (2012). Quantitative analysis of genes. In T.
D. Little (Ed.), The Oxford Handbook of Quantitative Meth-
ods (Vol. 2, pp. 219–234). New York: Oxford University
Press.
Ostrov, J. M. & Hart, E. J. (2012). Observational methods. In T.
D. Little (Ed.), The Oxford Handbook of Quantitative Methods
(Vol. 1, pp. 286–304). New York: Oxford University Press.
Peterson, T. (2012) Analyzing event history data. In T. D. Little
(Ed.), The Oxford Handbook of Quantitative Methods (Vol. 2,
pp. 486–516). New York: Oxford University Press.
Price, L. R. (2012). Analysis of imaging data. In T. D. Lit-
tle (Ed.), The Oxford Handbook of Quantitative Methods
(Vol. 2, pp. 175–197). New York: Oxford University
Press.
Ram, N., Brose, A., & Molenaar, P. C. M. (2012). Dynamic fac-
tor analysis: Modeling person-specific process. In T. D. Little
(Ed.), The Oxford Handbook of Quantitative Methods (Vol. 2,
pp. 441–457). New York: Oxford University Press.
Rosnow, R. L. & Rosenthal, R. (2012). Quantitative methods
and ethics. In T. D. Little (Ed.), The Oxford Handbook of
Quantitative Methods (Vol. 1, pp. 32–54). New York: Oxford
University Press.
Rupp, A. A. (2012). Clustering and classification. In T. D. Little
(Ed.), The Oxford Handbook of Quantitative Methods (Vol. 2,
pp. 517–611). New York: Oxford University Press.
Spector, P. E. (2012). Survey design and measure development.
In T. D. Little (Ed.), The Oxford Handbook of Quantitative
Methods (Vol. 1, pp. 170–188). New York: Oxford University
Press.
Steiner, P. M. & Cook, D. (2012). Matching and Propensity
Scores. In T. D. Little (Ed.), The Oxford Handbook of Quan-
titative Methods (Vol. 1, pp. 237–259). New York: Oxford
University Press.
Strobl, C. (2012). Data mining. In T. D. Little (Ed.), The Oxford
Handbook of Quantitative Methods (Vol. 2, pp. 678–700).
New York: Oxford University Press.
Thompson, B. (2012). Overview of traditional/classical statisti-
cal approaches. In T. D. Little (Ed.), The Oxford Handbook of
Quantitative Methods (Vol. 2, pp. 7–25). New York: Oxford
University Press.
Van Zandt, T., & Townsend, J. T. (2012). Designs for and
analyses of response time experiments. In T. D. Little
(Ed.), The Oxford Handbook of Quantitative Methods (Vol. 1,
pp. 260–285). New York: Oxford University Press.
von Eye, A., Mun, E. U., Mair, P., & von Weber, S. Configural
frequency analysis. In T. D. Little (Ed.), The Oxford Hand-
book of Quantitative Methods (Vol. 2, pp. 73–105). New York:
Oxford University Press.
little 5
Walls, T. A. (2012). Intensive longitudinal data. In T. D. Little
(Ed.), The Oxford Handbook of Quantitative Methods (Vol. 2,
pp. 432–440). New York: Oxford University Press.
Wang, L. L., Watts, A. S., Anderson, R. A., & Little, T. D. (2012).
Common fallacies in quantitative research methodology. In
T. D. Little (Ed.), The Oxford Handbook of Quantitative Meth-
ods (Vol. 2, pp. 718–758). New York: Oxford University
Press.
Wei. W. W. S. (2012). Time series analysis. In T. D. Little (Ed.),
The Oxford Handbook of Quantitative Methods (Vol. 2, pp.
458–485). New York: Oxford University Press.
Widaman, K. F., Early, D. R., & Conger, R. D. (2012). Special
populations. In T. D. Little (Ed.), The Oxford Handbook of
6 introduction
Quantitative Methods (Vol. 1, pp. 55–81). New York: Oxford
University Press.
Woods, C. M. (2012). Categorical methods. In T. D. Little
(Ed.), The Oxford Handbook of Quantitative Methods (Vol. 2,
pp. 52–73). New York: Oxford University Press.
Wu, W., Selig, J. P., & Little, T. D. (2012). Longitudinal data
analysis. In T. D. Little (Ed.), The Oxford Handbook of Quan-
titative Methods (Vol. 2, pp. 387–410). New York: Oxford
University Press.
Yuan, K-H., & Schuster, C. (2012). Overview of statistical esti-
mation methods. In T. D. Little (Ed.), The Oxford Handbook
of Quantitative Methods. (Vol. 1, pp. 361–387). New York:
Oxford University Press.
 CHAPTER

The Philosophy of Quantitative Methods

2
Brian D. Haig

Abstract

This chapter provides a philosophical examination of a number of different quantitative research

methods that are prominent in the behavioral sciences. It begins by outlining a scientific realist
methodology that can help illuminate the conceptual foundations of behavioral research methods. The
methods selected for critical examination are exploratory data analysis, statistical significance testing,
Bayesian confirmation theory, meta-analysis, exploratory factor analysis, and causal modeling. Typically,
these methods contribute to either the detection of empirical phenomena or the construction of
explanatory theory. The chapter concludes with a brief consideration of directions that might be taken
in future philosophical work on quantitative methods.
Key Words: scientific realism, methodology, exploratory data analysis, statistical significance testing,
Bayesianism, meta-analysis, exploratory factor analysis, causal modeling, latent variables, phenomena
detection, hypothetico-deductive method, inference to the best explanation

Introduction This chapter undertakes a philosophical examina-

Historically, philosophers of science have given
research methods in science limited attention, con-
centrating mostly on the nature and purpose of the-
ory in the physical sciences. More recently, however,
philosophers of science have shown an increased
willingness to deal with methodological issues in sci-
ences other than physics—particularly biology, but
also psychology to a limited extent. There is, then,
a developing literature in contemporary philosophy
of science that can aid both our understanding and
use of a variety of research methods and strategies in
psychology (e.g., Trout, 1998).
At the same time, a miscellany of theoretically
oriented psychologists, and behavioral and social
scientists more generally, have produced work on
the conceptual foundations of research methods that
helps illuminate those methods. The work of both
professional philosophers of science and theoretical
scientists deserves to be included in a philosophical
examination of behavioral research methods.
tion of a number of different quantitative research
methods that are prominent in the behavioral sci-
ences. It begins by outlining a scientific realist
methodology that can help illuminate the concep-
tual foundations of behavioral research methods.
The methods submitted to critical examination are
exploratory data analysis, statistical significance test-
ing, Bayesian confirmation theory, meta-analysis,
exploratory factor analysis, and causal modeling
methods. The chapter concludes with a brief and
selective consideration of directions that might be
taken in future philosophical work on quantitative
methods.
Quantitative Methods and Scientific
Realism
The three major philosophies of science
that bear on psychology are empiricism, social
constructionism, and scientific realism (Greenwood,

7
1992; Manicas & Secord, 1983). Nineteenth
century British empiricism had a major influence
on the development of British statistics in the first
half of the twentieth century (e.g., Mulaik, 1985).
The statistical methods developed in that intellectual
milieu remain an important part of psychology’s sta-
tistical research practice. For example, Karl Pearson’s
product moment correlation coefficient was taken
by its founder to be the quantitative expression of
a causal relation viewed in empiricist terms. Simi-
larly, Fisher’s endorsement of inductive methods as
the proper view of scientific method stemmed from
a commitment to the empiricism of his day. Even
in the current postpositivist philosophical climate,
authors of research methods textbooks sometimes
portray quantitative research as essentially positivist
in its empiricist commitments (Yu, 2006). Among
other things, positivism restricts its attention to
what can be observed and regards theories as instru-
ments that organize claims about observables but
that do not explain them by appeal to hidden
causes.
Qualitative methodologists also often bolster
their preferred conception of qualitative research by
comparing it with an unflattering positivist picture
of quantitative research. They tend to adopt the phi-
losophy of social constructionism, which is opposed
to the traditional notions of truth, objectivity, and
reason, maintaining that our understanding of the
world is determined by social negotiation. In one or
another of its various forms, it is the philosophy of
choice for many qualitative researchers, and it tends
to be employed by those who are opposed, or indif-
ferent, to quantitative methods. I shall not consider
it further in this chapter.
In what follows, I will adopt a scientific realist
perspective on research methods. Although the sub-
ject of considerable debate, and opposed by many
antirealist positions, scientific realism is the domi-
nant philosophy of science today. It is also the tacit
philosophy of most working scientists. This fact,
combined with its current heavy emphasis on the
nature of scientific practice, makes scientific realism
a philosophy for science—not just a philosophy of
science.
Scientific Realism
The philosophies of positivism, social construc-
tionism, and scientific realism just mentioned are
really family positions. This is especially true of sci-
entific realism, which comes in many forms. Most
versions of scientific realism display a commitment
to at least two doctrines: (1) that there is a real world
8 t h e p h i l o s o p h y o f q u a n t i tat i v e m e t h o d s
of which we are part and (2) that both the observ-
able and unobservable features of that world can be
known by the proper use of scientific methods. Some
versions of scientific realism incorporate additional
theses (e.g., the claims that truth is the primary aim
of science and that successive theories more closely
approximate the truth), and some will also nominate
optional doctrines that may, but need not, be used
by scientific realists (e.g., the claim that causal rela-
tions are relations of natural necessity; see Hooker,
1987). Others who opt for an “industrial strength”
version of scientific realism for the physical sciences
are more cautious about its successful reach in the
behavioral sciences. Trout (1998), for example, sub-
scribes to a modest realism in psychology, based on
his skepticism about the discipline’s ability to pro-
duce deeply informative theories like those of the
physical sciences.
Given that this chapter is concerned with the
philosophical foundations of quantitative methods,
the remaining characterization of scientific realism
will limit its attention to research methodology.
Scientific Realist Methodology
Scientific realism boasts a rich conception of
methodology, which is of considerable help in
understanding and guiding research. The resource-
fulness of realist methodology is suggested in the
following description of its major characteristics (see
Hooker, 1987; Nickles, 1987). First, realist method-
ology has three major tasks: to describe how methods
function; to evaluate methods critically against their
rivals; and to recommend how to use particular
methods to pursue chosen research goals.
Second, realist methodology is critically aim-
oriented. At a broad level, it recommends the pursuit
of valuable truth, explanatory understanding, and
effective control as primary research goals; and it is
concerned with the mutual adjustment of methods
and research goals.
Third, realist methodology is naturalistic—that
is, it is a substantive domain that uses the meth-
ods of the various sciences to study method itself.
Proctor and Capaldi (2001) advocate a naturalistic
approach to methodology in psychology in which
the empirical justification of methodological ideas
is emphasized.
A fourth feature of realist methodology is that it
is both generative and consequentialist. Generative
methodology involves reasoning to, and accept-
ing, knowledge claims in question from warranted
premises. Exploratory factor analysis is a prominent
example of a method in psychology that involves
a generative justification of the factorial hypotheses
to which it gives rise. By contrast, consequentialist
methodology focuses on reasoning from knowledge
claims in question to their testable consequences.
The widely used hypothetico-deductive method,
with its emphasis on predictive accuracy, clearly
exhibits a consequentialist approach to justifying
knowledge claims.
Fifth, realist methodology acknowledges the need
for two quite different approaches to justifying
knowledge claims. In philosophy these are com-
monly known as reliabilism and coherentism. With
reliabilism, a belief is justified to the extent that it is
acquired by reliable processes. In general, the innu-
merable methods that contribute to the detection of
empirical phenomena are concerned with reliabilist
justification. With coherentism, a belief is justified
in virtue of its coherence with other beliefs. Tha-
gard’s (1992) theory of explanatory coherence, used
for the comparative evaluation of scientific theories,
embodies an illuminating coherentist perspective on
knowledge justification. These two forms of justifi-
cation are different, complementary, and of equal
importance.
As a sixth feature, realist methodology regards
science as a problem-oriented endeavor in which
problems are conceptualized as constraints on their
effective solution (Haig, 1987; Nickles, 1981). On
this formulation, the constraints are actually con-
stitutive of the problem itself; they characterize the
problem and give it structure. Further, by includ-
ing all the constraints in the problem’s articulation,
the problem enables the researcher to direct inquiry
effectively by pointing the way to its own solution. In
a real sense, stating the problem is half the solution!
Finally, realist methodology takes the researcher’s
make up as a “knowing subject” seriously. Among
other things, the researcher is regarded as a satisficer
who makes heavy use of heuristics to guide her
inquiries. For example, McGuire (1997) discusses
many useful heuristics that can be employed to facil-
itate the generation of hypotheses in psychological
research.
Scientific realist methodology undergirds a wide
variety of methods, strategies, and heuristics that
have been successfully used to produce worthwhile
knowledge about both empirical phenomena and
explanatory theories. If quantitative researchers in
psychology engage this literature seriously, then they
will find resources for enhancing their understand-
ing of research methods.
I turn now to a philosophical consideration of
the selected research methods.
Exploratory Data Analysis
In psychological research, the major emphasis in
data analysis is placed on statistical inference, where
the task is to find out whether a data set exhibits a
designated feature of interest characterized with ref-
erence to a probabilistic model. Unfortunately, the
dominance of this goal has had the effect of discour-
aging a concerted examination of data sets in terms
of their quality and structure. Detailed explorations
of data are important in science, and it often makes
good sense to conduct them instead of a probabilis-
tic model or before the model is formulated and
adopted.
Consistent with this emphasis on the close exam-
ination of data, the last 30 years have witnessed the
strong development of an empirical, data-oriented
approach to statistics. One important part of this
movement is exploratory data analysis, which con-
trasts with the more familiar traditional statistical
methods with their characteristic emphasis on the
confirmation of knowledge claims.
Exploratory Data Analysis and John Tukey
Spelling out a philosophy of exploratory data
analysis is difficult, and few methodologists have
attempted to do so (for an initial attempt to do
this from a Bayesian perspective, see Good, 1983).
However, the intellectual progenitor of modern
exploratory data analysis, John Tukey, has devel-
oped a systematic perspective on the subject that
has helped to highlight its importance to research.
It deserves to be considered as a philosophy of data
analysis in its own right. Therefore, this brief exami-
nation of the philosophy of exploratory data analysis
pays particular attention to Tukey’s thinking on the
topic.
According to Tukey (1980), data analysis should
be treated as a two-stage compound process in
which the patterns in the data are first sug-
gested by exploratory data analysis and then crit-
ically checked through the use of confirmatory
data analysis procedures. Exploratory data analysis
involves descriptive—and frequently quantitative—
detective work designed to reveal structure or pattern
in the data sets under scrutiny. The data analyst
is encouraged to undertake an open-eyed investi-
gation of the data and perform multiple analyses
using a variety of intuitively appealing and easily
used techniques.
The compendium of methods for the explo-
ration of data, many of which were developed by
Tukey (1977), is designed to facilitate both dis-
covery and communication of information. These
haig 9
methods are concerned with the effective organi-
zation of data, the construction of graphical and
semi-graphical displays, and the examination of
distributional assumptions and functional depen-
dencies. Two additional attractive features of Tukey’s
methods are their robustness to changes in under-
lying distributions and their resistance to outliers
in data sets. Exploratory methods with these two
features are particularly suited to data analysis in psy-
chology, where researchers are frequently confronted
with ad hoc sets of data on amenable variables, which
have been acquired in convenient circumstances.
Exploratory Data Analysis and Scientific
Method
In his writings on data analysis, Tukey (1969)
has emphasized the related ideas that psychology is
without an agreed-upon model of data analysis and
that we need to think more broadly about scientific
inquiry. In an invited address to the American Psy-
chological Association in 1968, Tukey presented the
following excerpt from a prominent psychologist for
his audience to ponder. I quote in part:
I have the feeling that psychology is currently
without a dominant viewpoint concerning a model
for data analysis. In the forties and early fifties, a
hypothetico-deductive framework was popular, and
our mentors were keen on urging the design of
“crucial” experiments for the refutation of specific
predictions made from one or another theory.
Inductive empiricism was said to be disorderly and
inefficient. You and I knew then, as we know now,
that no one approach is uniformly most powerful.
(Tukey, 1969, p. 90)
Consider the hypothetico-deductive and induc-
tive conceptions of scientific methods, which are
mentioned here as candidate models for data anal-
ysis. Most psychological researchers continue to
undertake their research within the confines of the
hypothetico-deductive method. Witness their heavy
preoccupation with theory testing, where confirma-
tory data analyses are conducted on limited sets of
data gathered in accord with the dictates of the test
predictions of theories. In this regard, psychologists
frequently employ tests of statistical significance
to obtain binary decisions about the credibility of
the null hypothesis and its substantive alternatives.
However, the use of statistical significance tests in
this way strongly blunts our ability to look for
more interesting patterns in the data. Indeed, the
continued neglect of exploratory data analysis in
psychological research occurs in good part because
10 t h e p h i l o s o p h y o f q u a n t i tat i v e m e t h o d s
there is no acknowledged place for such work in
the hypothetico-deductive conception of inquiry
(Wilkinson & The Task Force, 1999).
I think the worth of the inductive method as a
model for data analysis is dismissed too quickly in
the above quotation. The major failing of the induc-
tive account of scientific method lies not so much
with its perspective on data analysis, but with its
prohibition of the formulation of explanatory the-
ories. A modern conception of inductive method is
embedded in the important scientific process of phe-
nomena detection. Phenomena are relatively stable
recurrent general features of the world that we seek
to explain (Woodward, 1989), and their detection
frequently involves an inductive process of empiri-
cal generalization. With its emphasis on phenomena
detection, inductive method reserves an important
place for the exploratory analysis of data. In detect-
ing phenomena, one is concerned to extract a signal
from the noise of data, and for this the inten-
sive search of large amounts of data is frequently
essential. It is precisely because securing a heavy
information yield for our data is likely to throw
up potentially interesting data patterns that might
turn out to be genuine phenomena. In this con-
text, data mining is encouraged, and the capabilities
of exploratory techniques in this regard often make
them the appropriate methods of choice.
By contrast, Behrens and Yu (2003) suggest that
the inferential foundations of exploratory data anal-
ysis are to be found in the idea of abduction, or
explanation (and by implication, not in the notions
of hypothetico-deductive testing and inductive gen-
eralization). However, exploratory data analysis is a
descriptive pattern-detection process that is a pre-
cursor to the inductive generalizations involved in
phenomena detection. As will be seen later in the
consideration of exploratory factor analysis, abduc-
tive inference is reserved for the construction of
causal explanatory theories that are introduced to
explain empirical phenomena. Beherens and Yu’s
suggestion conflates the quite different ideas of
descriptive and explanatory inference.
Exploratory Data Analysis and a Model of
Data Analysis
In the spirit of Tukey’s (1962; 1980) push for
breadth of vision in data analysis, one might use-
fully take a perspective on data analysis that extends
Tukey’s two-stage model (Haig, 2005b). Before
exploring data for patterns of potential interest,
researchers should assiduously screen their data for
their quality. This initial data analysis involves
checking for the accuracy of data entries, identify-
ing and dealing with missing and outlying data, and
examining the data for their fit to the assumptions
of the data analytic methods to be used. This impor-
tant, and time-consuming, preparatory phase of
data analysis has not received the amount of explicit
attention that it deserves in behavioral science edu-
cation and research practice. Fidell and Tabachnick
(2003) provide a useful overview of the task and
techniques of initial data analysis.
Confirmation of the initial data patterns sug-
gested by exploratory data analysis is a “just check-
ing” strategy and as such should be regarded as a
process of close replication. However, it is essential
to go further and undertake constructive replica-
tions to ascertain the extent to which results hold
across different methods, treatments, subjects, and
occasions. Seeking results that are reproducible
through constructive replications requires data ana-
lytic strategies that are designed to achieve significant
sameness rather than significant difference (Ehren-
berg & Bound, 1993). Exploratory data analysis,
then, can usefully be regarded as the second in
a four-stage sequence of activities that, in turn,
attend to data quality, pattern suggestion, pattern
confirmation, and generalization.
Resampling Methods and Reliabilist
Justification
Since the 1980s, statisticians have been able to
exploit the massive computational power of the
modern computer and develop a number of com-
puter intensive resampling methods, such as the
jackknife, the bootstrap, and cross-validation (Efron
& Tibshirani, 1993). These methods constitute one
important set of confirmatory procedures that are
well suited to the task of checking on the data
patterns thrown up by exploratory data analysis.
By exploiting the computer’s computational power,
these resampling methods free us from the restric-
tive assumptions of modern statistical theory, such as
the belief that the data are normally distributed, and
permit us to gage the reliability of chosen statistics
by making thousands, even millions, of calculations
on many data points.
It is important to appreciate that the resampling
methods just mentioned make use of a reliabilist
approach to justification. Here, the reliability checks
on emergent data patterns are provided by consis-
tency of test outcomes, which are time-honored
validating strategies. Our willingness to accept the
results of such checks is in accord with what Tha-
gard (1992) calls the principle of data priority. This
principle asserts that statements about observational
data, including empirical generalizations, have a
degree of acceptability on their own. Such claims
are not indubitable, but they do stand by them-
selves better than claims justified solely in terms of
what they explain. What justifies the provisional
acceptance of data statements is that they have
been achieved by reliable methods; what strength-
ens our provisional belief in the patterns thrown
up by exploratory data analysis is their confirma-
tion through use of computer-based resampling
methods.
Further, it is important to appreciate that the
acceptability of claims provided by the reliabilist jus-
tification of computer-intensive resampling meth-
ods can be enhanced by making appropriate use of
a coherentist approach to justification. One impor-
tant form of coherence is explanatory coherence, and
one method that delivers judgments of explanatory
coherence is the theory of explanatory coherence
(Thagard, 1992). According to this theory, data
claims, including empirical generalizations, receive
an additional justification if and when they enter
into, and cohere with, the explanatory relations of
the theory that explains them.
A Philosophy for Teaching Data Analysis
An underappreciated, but important, feature of
Tukey’s writings on exploratory data analysis is the
illuminating remarks on the teaching of data analysis
that they contain. These remarks can be assem-
bled into a constructive philosophy for teaching data
analysis, which can properly be regarded as an aspect
of an overall philosophy of exploratory data analy-
sis. This philosophy of teaching advises us to think
about and teach data analysis in a way that is quite
different from the prevailing custom.
Provocatively, Tukey (1980) maintained that the
proper role of statistics teachers is to teach that
which is most difficult and leave that which is more
manageable to good textbooks and computers. He
recommended teaching data analysis the way he
understood biochemistry was taught, concentrat-
ing on what the discipline of statistics has learned,
perhaps with a discussion of how such things were
learned. The detail of methods should be assigned
to laboratory work, and the practice of learning
data analytic techniques should be assigned to a
different course in which problems arose. He fore-
saw that such a redirection in teaching data analysis
would have to be introduced in phases. In Tukey’s
(1962) words, “The proposal is really to go in the
opposite direction from cookbookery; to teach not
haig 11
‘what to do,’ nor ‘how we learned what to do,’ but
rather, ‘what we have learned”’ (p. 63). This advice
is broadly consistent with the idea that we should
teach research methods in terms of their accompa-
nying methodology, a recommendation considered
at the end of this chapter.
Another prominent feature of Tukey’s philosophy
of teaching data analysis is his recommendation that
we should teach both exploratory and confirmatory
data analysis and that we have an obligation to do so.
Tukey’s strong promotion of the value of exploratory
data analysis was intended as a counter to the dom-
inance of confirmatory data analysis in statistical
practice. However, for Tukey, exploratory data anal-
ysis was not to be understood as more important
than confirmatory data analysis because both are
essential to good data analysis.
Tukey also suggested that exploratory data anal-
ysis should probably be taught before confirma-
tory data analysis. There are several reasons why
this recommendation has merit. Properly taught,
exploratory data analysis is probably easier to learn,
and it promotes a healthy attitude to data anal-
ysis (encouraging one to be a dataphile without
becoming a data junkie). It requires the investiga-
tor to get close to the data, analyze them in various
ways, and seek to extract as much as possible poten-
tially important information from the data. This is
done to detect indicative patterns in the data before
establishing through confirmatory data analysis that
they are genuine patterns.
Tukey emphasized that learning exploratory data
analysis centrally involves acquiring an appropri-
ate attitude toward the data, which includes the
following elements: exploratory data analysis is suf-
ficiently important to be given a great deal of time;
exploratory data analysis should be carried out flex-
ibly with multiple analyses being performed; and
exploratory data analysis should employ a multiplic-
ity of methods that enhance visual display.
Statistical Significance Testing
It is well known that tests of statistical significance
are the most widely used methods for evaluating
hypotheses in psychology (e.g., Hubbard & Ryan,
2000). These tests have been popular in psychology
for nearly 50 years and in statistics for about 75 years.
Since the 1960s, there has developed a massive crit-
ical literature in psychology regarding their worth.
Important early contributions to this debate are col-
lected in Morrison and Henkel (1970; see also Giere,
1972). Cohen (1994) provides a short perceptive
review of the controversy, whereas Nickerson (2000)
12 t h e p h i l o s o p h y o f q u a n t i tat i v e m e t h o d s
has undertaken a useful extensive review of the con-
troversy since its beginning. Despite the plethora
of critiques of statistical significance testing, most
psychologists understand them poorly, frequently
use them inappropriately, and pay little attention
to the controversy they have generated (Gigerenzer,
Krauss, & Vitouch, 2004).
The significance testing controversy is multi-
faceted. This section will limit its attention to a
consideration of the two major schools of signifi-
cance testing, their hybridization and its defects, and
the appropriateness of testing scientific hypotheses
and theories using tests of statistical significance.
Psychologists tend to assume that there is a sin-
gle unified theory of tests of statistical significance.
However, there are two major schools of thought
regarding significance tests: Fisherian and Neyman-
Pearson. Initially, Neyman and Egon Pearson sought
to build on and improve Fisher’s theory, but they
subsequently developed their own theory as an alter-
native to Fisher’s theory. There are many points of
difference between the two schools, which adopt
fundamentally different outlooks on the nature of
scientific method. The uncritical combination of
the two schools in psychology has led to a confused
understanding of tests of statistical significance and
to their misuse in research.
The Fisherian Significance Testing School
The Fisherian school of significance testing (e.g.,
Fisher, 1925) tests a hypothesis or theory of sub-
stantive interest against the null hypothesis that the
experimental effect to be demonstrated is in fact
absent. Fisher argued that an experiment is per-
formed solely to give the data an opportunity to
disprove the null hypothesis. No alternative hypoth-
esis is specified, and the null hypothesis is the
hypothesis to be nullified; it need not be the hypoth-
esis of zero difference. Because one cannot accept
the null hypothesis, no provision is made for Type
II error, and relatedly, there is no place for a sta-
tistical concept of power. Most importantly, and as
noted earlier, Fisher subscribed to an inductive con-
ception of scientific method and maintained that
significance tests are vehicles of inductive reasoning.
As such they are concerned with evidence for beliefs.
Should We Use Fisher’s Significance Tests?
The question of whether behavioral scientists
should use Fisherian significance tests as a defensible
form of hypothesis testing largely centers on whether
p−values are good measures of scientific evidence.
Although many psychological researchers think, and
some methodologists argue, that p−values can be
used to measure strength of evidence, others hold
them to be deeply problematic in this respect (e.g.,
Royall, 1997; Hubbard & Lindsay, 2008).
Some have argued that the concept of evidence
adopted by Fisher is defective. For one thing, it is
widely agreed by philosophers of science that the-
ory or hypothesis evaluation is a comparative affair
in which evidence against one hypothesis is evidence
for another hypothesis (e.g., Sober, 2008). However,
as just noted, Fisher only countenanced the null,
and without an alternative hypothesis with which
to compare the null, the logic of his significance
testing is importantly incomplete; one cannot have
evidence against the null without it being evidence
for another hypothesis. The idea that one might
allow for alternative hypotheses by joining Fisher’s
perspective with that of Neyman and Pearson will
be seen by many to make matters worse. As will be
noted shortly, Neyman and Pearson were concerned
with the reliability of errors in decision making in
the long run rather than with evidence for believing
hypotheses in a particular experiment.
Others contend that a further major problem
with Fisher’s p-value is that it doesn’t measure evi-
dence properly (e.g., Goodman, 1993). In this
regard, four claims about p-values are thought to
disqualify it as a proper measure of evidence. First,
the p-value is not a direct measure of the probability
that the null is false; it is a conditional probability
of obtaining the data, calculated on the assumption
that the null hypothesis is true. Second, to take a
number that quantifies rare data under the null is to
confuse the strength of evidence with the probability
of its occurrence (Royall, 1986). These two things
are different because the probability is an indication
of the long-run Type I error rate, which is separate
from strength of evidence. Third, the calculation of
a p-value combines the rarity of an obtained result
with the probability of results that didn’t happen. As
Jeffreys (1939) stated long ago, “What the use of P
implies . . . is that a hypothesis that may be true may
be rejected because it has not predicted observable
results that have not occurred” (p. 136). Finally, it is
claimed that the p-value can exaggerate the strength
of evidence against the point null and small inter-
val hypotheses (Berger & Selke, 1987), which are
frequently tested in psychology.
These criticisms are not convincing to all. For
example, Hurlbert and Lombardi (2009) recently
considered these arguments and recommend a shift
in focus from the classical Fisherian framework to a
neo-Fisherian alternative. Key elements of this alter-
native are that the probability of Type I error is not
specified, p−values are not misleadingly described
as “significant” or “nonsignificant,” judgment is sus-
pended about accepting the null hypothesis on the
basis of high p-values, the “three-valued logic” that
gives information about the direction of the effect
being tested is adopted, accompanying effect size
information is provided, and use is made of adjunct
information such as confidence intervals, where
appropriate. Two things to note here about this neo-
Fisherian perspective are that it is concerned with
significance assessments but not null hypothesis sig-
nificance tests and that it is concerned with statistical
tests as distinct from the tests of scientific hypothe-
ses. There are empirical studies in psychology that
approximate this modified Fisherian perspective on
significance tests.
The Neyman-Pearson Hypothesis Testing
School
Neyman and Pearson rejected Fisher’s notion of a
significance test and its use of a threshold p-value as a
basis for rejecting the null hypothesis. In this regard,
they added the requirement of the specification of an
alternative hypothesis as well as the null hypothesis,
and they replaced Fisher’s evidential p-value with the
Type I error rate, α (e.g., Neyman & Pearson, 1933).
In addition, Neyman and Pearson permitted a more
liberal formulation of the null hypothesis than did
Fisher and regarded it as legitimate to speak of its
acceptance. Thus, Type II error was admitted, and
explicit provision was made for a statistical concept
of power. To capture these differences, Neyman and
Pearson spoke of their approach as hypothesis testing
rather than significance testing.
However, the Neyman-Pearson school differs
from the Fisherian school most fundamentally in
maintaining that significance tests are rules of induc-
tive behavior rather than vehicles for inductive rea-
soning. On this view, significance testing is regarded
as a theory of prudential decision-making; accept-
ing or rejecting a hypothesis amounts to adopting an
appropriate course of action, rather than believing
it to be probably true or false. At root, Neyman and
Pearson held different views from Fisher about the
nature of science.
Should We Use Neyman and Pearson’s
Hypothesis Tests?
It might seem that by focusing on hypothesis test-
ing and making provision for alternative hypotheses
haig 13
and a statistical concept of power, Neyman and Pear-
son’s approach marks an improvement over Fisher’s
significance tests. However, as noted earlier, Ney-
man and Pearson were concerned with measuring
the reliability of errors in decision making in the
long run rather than with evidence for believing
hypotheses in a particular experiment; Type I and
Type II error are objective probabilities, under-
stood as long-run relative frequencies. As such, they
belong to reference classes of endless series of trials
that might have happened but never did. They are
not about single events such as individual exper-
iments. Further, being concerned with decision
making understood as behavioral courses of action,
Neyman-Pearson statistics do not measure strength
of evidence for different hypotheses and thus do not
tell us how confident we should be in our beliefs
about those hypotheses.
It would seem, then, that the Neyman-Pearson
approach is suitable for use only when the focus is on
controlling errors in the long run (e.g., quality con-
trol experiments). However, this does not happen
often in psychology.
The Hybrid
Although the Neyman-Pearson theory of testing
is the official theory of statistical testing in the field
of professional statistics, textbooks in the behav-
ioral sciences ensure that researchers are instructed to
indiscriminately adopt a hybrid account of tests of
statistical significance, one that is essentially Fish-
erian in its logic but that is often couched in the
decision-theoretic language of Neyman and Pearson.
The hybrid logic is a confused and inconsistent
amalgam of the two different schools of thought
(Acree, 1979; Gigerenzer, 1993; Spielman, 1974;
but see Lehmann, 1993, for the suggestion that the
best elements of both can be combined in a unified
position). To the bare bones of Fisherian logic, the
hybrid adds the notion of Type II error (opposed by
Fisher) and the associated notion of statistical power
(which Fisher thought could not be quantified)
but only at the level of rhetoric (thereby ignoring
Neyman and Pearson), while giving a behavioral
interpretation of both Type I and Type II errors (vig-
orously opposed by Fisher)! Because the authors of
statistical textbooks in the behavioral sciences tend
to present hybrid accounts of significance testing,
aspiring researchers in these sciences almost always
acquire a confused understanding of such tests. It is
most unfortunate that many writers of statistics text-
books in the behavioral sciences have unwittingly
perpetuated these basic misunderstandings.
14 t h e p h i l o s o p h y o f q u a n t i tat i v e m e t h o d s
To make matters worse, this confusion is com-
pounded by a tendency of psychologists to misrep-
resent the cognitive accomplishments of significance
tests in a number of ways. For example, levels
of statistical significance are taken as measures of
confidence in research hypotheses, likelihood infor-
mation is taken as a gage of the credibility of the
hypotheses being tested, and reported levels of sig-
nificance are taken as measures of the replicability
of findings (Gigerenzer, Krauss, & Vitouch, 2004).
Significance Tests and Theory Testing
Meehl (1967, 1978, 1997) has made one of the
strongest criticisms of the use of tests of statisti-
cal significance in psychology. He argued that the
widespread use of tests of statistical significance to
test substantive hypotheses and theories is deeply
flawed because the support for a hypothesis or the-
ory obtained by rejecting the null hypothesis is very
weak.
Sometimes psychological researchers test a
hypothesis of substantive interest against the point
null hypothesis that the difference between the rel-
evant population parameters is exactly zero. But a
fact, long known to professional statisticians and
appreciated by Meehl, is that the point null hypoth-
esis is virtually always false in the behavioral and
social sciences. The reason for this is that in these
sciences, most things are related to one another at
least to some small extent. In many parts of psy-
chology, “everything in the brain is connected with
everything else,” resulting in a large positive man-
ifold in which many variables correlate positively
with one another to a significant degree. Thus, in
the “softer” precincts of psychology, where “true”
experiments are often not possible, obtaining a rea-
sonable sample size makes the achievement of a
statistically significant result the likely outcome of an
empirical study. Meehl (1967) reasoned that if the
null hypothesis of zero group differences is almost
always false, then with sufficient power, directional
hypotheses in these parts of psychology have a 50:50
chance of achieving statistical significance! Meehl
and Lykken provided some empirical evidence for
this claim more than 40 years ago (Meehl, 1967; see
also Meehl, 1997). A recent simulation study on real
data carried out by Waller (2004) confirmed Meehl’s
claim.
One can better appreciate what is wrong with
using tests of statistical significance to appraise psy-
chological theories by considering the logic involved
in such testing. It is helpful to begin by observ-
ing the important distinction between scientific
hypotheses and the statistical hypotheses that may
be derived from them (Bolles, 1962). Often in
psychology, scientific theories about psychological
processes or structures are formulated, and then
statistical hypotheses are derived to facilitate their
empirical testing. The former will characteristically
invoke causal mechanisms for explanatory purposes,
whereas the latter will appeal to statistical tests
of null hypotheses about the population parame-
ters of observed variables. Meehl has argued that
psychological researchers tend to conflate the sub-
stantive theory and the statistical hypothesis and
unwarrantedly take the successful refutation of the
null hypothesis as grounds for concluding that the
substantive theory has been strongly confirmed.
However, if we have good grounds for believing that
the point null hypothesis is probably false at the out-
set, and we use the null as the observation hurdle for
our theories to surmount, then support for a theory
by rejecting this implausible null alternative is quite
feeble.
For good reason, then, Meehl (1990) has urged
psychologists to abandon tests of statistical signifi-
cance for purposes of substantive theory testing in
psychology. He suggests that psychologists should
replace them with a strategy that is adapted from the
philosopher of science, Imre Lakatos (1970), who
argued against Popper’s strict falsificationist posi-
tion on theory testing for the comparative theory
appraisal of research programs over time. Meehl has
maintained that one should defend and amend a
theory only if it has a good track record of successful
or near-miss predictions of low prior probability.
In conclusion, it is clear that there are fundamen-
tal philosophical differences between the Fisherian
and Neyman-Pearson schools of statistical thought.
Fisher’s statistical contributions can been seen as
a deliberate attempt to develop an objective alter-
native to Bayesian statistical thinking popular in
Europe at the time, whereas those of Neyman and
Pearson can be seen as an attempt to develop a
position that is even more objective than Fisher’s.
Nevertheless, Bayesian thinking today is an
attractive option for many statisticians who hold
misgivings about one or both of these schools of
thought. It is also the focus of much attention in
the philosophy of science. It is to the elements of
the Bayesian position that we now turn.
Bayesian Confirmation Theory
What is it for empirical evidence to provide con-
firmation or disconfirmation of a scientific hypothe-
sis or theory? Methodologists of science have worked
long and hard to answer this important and chal-
lenging question by developing theories of scientific
confirmation. Despite the considerable fruits of
their labors, there is widespread disagreement about
which theory of confirmation we should accept. In
recent times, a large number of philosophers of
science have contributed to Bayesian confirmation
theory (e.g., Earman, 1992; Howson & Urbach,
2006). Many philosophical methodologists now
believe that Bayesianism, including Bayesian phi-
losophy of science, holds the best hope for building
a comprehensive and unified theory of scientific
inference.
Bayesianism is a comprehensive position. It com-
prises a theory of statistical inference, an account
of scientific method, and a perspective on a vari-
ety of challenging methodological issues. Today, it
also boasts a fully fledged philosophy of science. In
this section, attention is limited to a consideration
of the strengths and weaknesses of Bayesian statis-
tical inference, the ability of Bayesian confirmation
theory to improve upon the hypothetic-deductive
method, and the question of whether Bayesianism
provides an illuminating account of the approach
to theory evaluation known as inference to the best
explanation.
Bayesian Statistical Inference
The Bayesian approach to statistical inference is
so called because it makes central use of a theorem
of the mathematical calculus of probability known
as Bayes’ theorem. This theorem can be written in a
simple form as:
Pr(H) × Pr(D/H)
Pr(H/D) =
Pr(D)
With the proviso that Pr (D) and Pr (H) cannot
be zero, the theorem says that the posterior probabil-
ity of the hypothesis is obtained by multiplying the
prior probability of the hypothesis by the probabil-
ity of the data, given the hypothesis (the likelihood),
and dividing the product by the prior probability of
the data. It is through use of this and other ver-
sions of Bayes’ Theorem that Bayesians are able to
implement their view of statistical inference, which
is the orderly revision of opinion in the light of new
information.
For Bayesians, a couple of features of this gloss
on Bayesian statistical inference recommend them-
selves. Most importantly, the Bayesian approach
squares with the stated purpose of scientific inquiry
noted above—namely, securing the probability of a
hypothesis in the light of the relevant evidence. The
haig 15
informational output of a traditional test of signifi-
cance is the probability of the data, given the truth
of our hypothesis, but it is just one input in the
Bayesian scheme of things. A second stated desir-
able feature of the Bayesian view is its willingness to
make use of relevant information about the hypoth-
esis before the empirical investigation is conducted
and new data are obtained, explicitly in the form of
a prior probability estimate of our hypothesis. Tra-
ditional tests of statistical significance are premised
on the assumption that inferences should be based
solely on present data, without any regard for what
we might bring to a study in the way of belief or
knowledge about the hypothesis to be tested—a
position that Bayesians contend is hardly designed to
maximize our chances of learning from experience.
To achieve their goal of the systematic revision of
opinion on the basis of new information, Bayesians
are able to employ Bayes’ theorem iteratively. Hav-
ing obtained a posterior probability assignment for
their hypothesis via Bayes’ theorem, they can then
go on and use that posterior probability as the new
prior probability in a further use of Bayes’ theorem
designed to yield a revised posterior probability, and
so on. In this way, the Bayesian researcher learns
from experience.
Criticisms of Bayesian Hypothesis Testing
Although my consideration of the merits of
conventional significance tests and their Bayesian
alternative is both sketchy and selective, some read-
ers will sense that the Bayesian view provides an
attractive alternative to the traditional approach,
particularly when the latter assumes its hybrid form.
However, as with all theories of confirmation, the
Bayesian approach has come in for its share of crit-
icism. These criticisms have tended to focus on the
alleged problematic nature of prior probabilities. In
this regard, it is objected that because Bayesians
adopt a subjectivist conception of probability and
resort to personal estimates of the prior probabilities
of their hypotheses, they introduce an ineliminable,
but highly undesirable, subjective element into their
calculations. To this objection, the Bayesians have
two plausible replies: they can concede that personal
estimates of prior probabilities are subjective, that
they may differ markedly from person to person, and
that they are often very rough estimates and then go
on to point out that when prior estimates err, they
are brought into line by freshly obtained sets of data;
or, they may appeal to the failure of strictly empiri-
cist theories of confirmation, which hold that one
may obtain an adequate test of a hypothesis solely
16 t h e p h i l o s o p h y o f q u a n t i tat i v e m e t h o d s
on the basis of evidence and logic, and assert that in
real-life situations, there is no alternative to relying
on a subjective component in our testing efforts.
In deciding whether to adopt a Bayesian posi-
tion on statistical inference, it should be kept in
mind that one does not have to embrace a gen-
eral Bayesian theory of scientific confirmation rather
than, say, the hypothetico-deductive alternative.
One might be a Bayesian when dealing with prob-
lems of statistical inference but remain wedded to a
general hypothetico-deductive conception of scien-
tific method. Or, more plausibly, one might employ
Bayesian statistical methods when concerned with
inferential problems about hypotheses for which
we have the relevant probabilistic information, but
adopt a non-probabilistic count of theory evaluation
such as Thagard’s theory of explanatory coherence,
which will be referred to later in the chapter. The
general point to be made here is that Bayes’ theorem
can help us deal with some problems of statistical
inference, but clearly, a great deal of scientific work
will be done with the use of other methods—some
of them statistical and some of them not.
Bayesianism and the Hypothetico-Deductive
Method
One of the clear achievements of Bayesianism is
its ability to improve on the unsatisfactory approach
to hypothesis and theory appraisal taken by the
hypothetico-deductive method. The hypothetico-
deductive method has long been the method of
choice for the evaluation of scientific theories (Lau-
dan, 1981), and it continues to have a dominant
place in psychology. Despite its popularity, it is
usually characterized in an austere manner: The
researcher takes a hypothesis or theory of interest
and tests it indirectly by deriving from it one or
more observational predictions that are themselves
directly tested. Predictions borne out by the data
are taken to confirm the theory to some degree;
those predictions that do not square with the data
count as disconfirming instances of the theory.
Normally, the theory is not compared with rival
theories in respect of the data, only with the data
themselves.
The hypothetico-deductive method, in some-
thing like this form, has been strongly criticized
by methodologists on a number of counts (e.g.,
Glymour, 1980; Rozeboom, 1997). One major crit-
icism of the method is that it is confirmationally
lax. This laxity arises from the fact that any posi-
tive confirming instance of a hypothesis submitted
to empirical test can confirm any hypothesis that is
conjoined with the test hypothesis, regardless of how
plausible it might be. This state of affairs is known as
the fallacy of irrelevant conjunction, or the tacking
problem, because confirmation of a test hypothesis
also confirms any conjunct that is attached to the
test hypothesis. The fallacy of irrelevant conjunc-
tion arises with the hypothetico-deductive method
because predictions are deduced from hypotheses
only by making use of auxiliary hypotheses drawn
from background knowledge.
Clearly, this is an unacceptable state of affairs.
Bayesians have challenged the assumption that the
occurrence of the consequences of a theory con-
firm the theory and its conjuncts holistically. They
argue that the Bayesian approach enables the dif-
ferential support of the elements of a theory, spec-
ifying conditions showing that E never increases
the probability of H conjoined with any additional
hypothesis by more than it increases the probability
of H.
Another major criticism of the hypothetico-
deductive method is that it tests a single hypothesis
or theory of interest against the empirical evidence;
it does not test a hypothesis or theory in relation to
rivals in respect of the evidence. This is held to be
a major flaw because it is widely agreed that theory
evaluation is a comparative affair involving simul-
taneous evaluation of two or more hypotheses or
theories.
The comparative nature of theory evaluation is
straightforwardly handled by the Bayesian position
by rewriting the simple form of Bayes’ theorem
given earlier to deal with two or more hypothe-
ses. Here, Bayes’ theorem is presented for the
case of two hypotheses, where the theorem can be
written for each hypothesis in turn. For the first
hypothesis,
Pr (H1 /D)
Pr(H1 ) × Pr(D/H1 )
=
Pr(H2 )×Pr(D/H2 )+Pr(H1 )×Pr(D/H1 )
This says that the posterior probability of the
first hypothesis is obtained by multiplying its prior
probability by the probability of the data, given that
hypothesis (the likelihood), and dividing the prod-
uct by the value that results from adding the prior
probability of the second hypothesis, multiplied
by the likelihood for that hypothesis, to the prior
probability of the first hypothesis, multiplied by its
likelihood. Bayes’ theorem for the second hypothesis
is written in a similar way.
Bayesianism and Inference to the Best
Explanation
Recently, some Bayesians have claimed that
their perspective on scientific method can also pro-
vide an enhanced characterization of the important
approach to theory evaluation known as inference
to the best explanation. Inference to the best expla-
nation is based on the belief that much of what
we know about the world is based on consider-
ations of explanatory worth. In contrast to the
Bayesian approach, accounts of inference to the best
explanation take theory evaluation to be a quali-
tative exercise that focuses on explanatory criteria
rather than a quantitative undertaking in which
one assigns probabilities to theories (Haig, 2009;
Thagard, 1992).
Although inference to the best explanation has
typically been regarded as a competitor for Bayesian
theory evaluation, Lipton (2004) has recently
argued that the two approaches are broadly compat-
ible and that, in fact, their proponents “should be
friends.” In broad terms, he suggests that judgments
of the loveliest explanation, which are provided by
the evaluative criteria of inference to the best expla-
nation, such as unificatory power, precision, and
elaboration of explanatory mechanisms, contribute
to assessments of the likeliest explanation, which
are provided by the probabilities of the Bayesian
approach. Specifically, Lipton maintains that the
explanatory considerations invoked in inference to
the best explanation guide determination of the
prior probabilities (and the likelihoods) that are
inserted in Bayes’ Theorem.
However, although appeal to explanatory matters
might be one way in which Bayesians can determine
their prior probabilities, Lipton does not suggest
how this might be done. Further, those who hold
inference to the best explanation to be a norma-
tive approach to scientific theory evaluation, with
its own distinctive character, will worry that Lipton
relegates it to a descriptive role within a Bayesian
normative framework (e.g., Psillos, 2004).
Another way of showing the compatibility of
inference to the best explanation and Bayesianism is
to translate the evaluative criteria employed within
inference to the best explanation into probabilis-
tic terms. McGrew (2003) has done this by taking
the important theoretical virtue of consilience, or
explanatory breadth, and showing that its Bayesian
form leads to higher posterior probabilities of the
hypotheses being evaluated. Nevertheless, McGrew
has acknowledged that by translating consilience
into its “flattened” probabilistic form, it no longer
haig 17
remains a genuine explanatory virtue. Not only is
there no guarantee that consilience will be concerned
with an explanation of the evidence, there is no
way that probabilistic translations of the explana-
tory virtues can refer to the causal connections that
are often appealed to in scientific explanations. Fur-
ther, Weisberg (2009) has recently argued that the
explanatory loss incurred in such translations will
occur for any distinctively explanatory virtue that
is given such probabilistic treatment. In short, it
would seem that Bayesianism cannot capture the
intuitively important notion of explanatory power
without significant loss.
What Should We Think About
Bayesianism?
Philosophical assessment of the worth of
Bayesianism range from claims that it is without peer
as a theory of scientific reasoning to the view that it is
fundamentally wrong-headed. Howson and Urbach
(2006) exemplify the former view, claiming that sci-
entific reasoning is both inductive and probabilistic
and that the axioms of probability suffice to articu-
late such reasoning. The latter view is exemplified by
Bunge (2008), who has argued that Bayesianism is
fundamentally wrong for three reasons: (1) it assigns
probabilities to statements rather than taking them
as objective features of the world; (2) it conceives
of probabilities as subjective; and (3) it appeals to
probabilities in the absence of randomness.
To add to this mix of views, many statisticians
take Bayesian statistical inference to be a superior
alternative to classical statistical inference, for the
reasons stated earlier. Finally, some advocates of
Bayesianism see it as a comprehensive theory of
confirmation, whereas others see it as having only
context-specific application.
The difficulties of deciding just what to think
about Bayesianism are captured well by the ambiva-
lence of John Earman (1992), a Bayesian philoso-
pher of science. He confesses to being an enthusiastic
Bayesian on Mondays, Wednesdays, and Fridays.
But on Tuesdays, Thursdays, and Saturdays, he
holds doubts about the totalizing ambitions of
Bayesianism and indeed whether it can serve as
a proper basis for scientific inference. Faced with
such difficulty, it is probably prudent to settle for
a contextual application of Bayesian thinking, as
indicated earlier in this section. For example, in par-
ticular domains such as medical diagnosis, where the
relevant probabilistic information is often available,
scientists sometimes appeal to the Bayesian corpus to
justify the selective use of its methods. By contrast,
18 t h e p h i l o s o p h y o f q u a n t i tat i v e m e t h o d s
in domains where the evaluation of explanatory
hypotheses and theories are of primary concern, sci-
entists have, for good reason, often employed some-
thing like inference to the best explanation. Like it
or not, the intending Bayesian scientist will have to
consult the relevant philosophical literature, among
other methodological literatures, to furnish an
informed justification for their Bayesian practices.
Meta-Analysis
In the space of three decades meta-analysis has
become a prominent methodology in behavioral sci-
ence research, with the major developments coming
from the fields of education and psychology (Glass,
McGaw, & Smith, 1981: Hedges & Olkin, 1985;
Hunter & Schmidt, 2004). Meta-analysis is an
approach to data analysis that involves the quantita-
tive analysis of the data analyses of primary empirical
studies. Hence, the term meta-analysis coined by
Glass (1976). Meta-analysis, which comes in a vari-
ety of forms (Bangert-Drowns, 1986), is concerned
with the statistical analyses of the results from many
individual studies in a given domain for the pur-
pose of integrating or synthesizing those research
findings.
The following selective treatment of meta-
analysis considers its possible roles in scientific
explanation and evaluation research before critically
examining one extended argument for the con-
clusion that meta-analysis is premised on a faulty
conception of science.
Meta-Analysis and Explanation
Meta-analysis is a prominent example of a dis-
tinctive use of statistical methods by behavioral
scientists to aid in the detection of empirical phe-
nomena. By calculating effect sizes across primary
studies in a common domain, meta-analysis helps us
detect robust empirical generalizations (cf. Schmidt,
1992). By using statistical methods to ascertain the
existence of such regularities, meta-analysis can be
usefully viewed as the statistical analog of direct
experimental replication. It is in this role that meta-
analysis currently performs its most important work
in science.
However, given that the detection of empiri-
cal phenomena and the construction of explana-
tory theories are quite different research tasks, the
recent suggestion that meta-analysis can directly
contribute to the construction of explanatory the-
ory (Cook et al., 1992; Schmidt, 1993) is an
arresting methodological claim. In approving this
extension of meta-analysis beyond a concern with thereby confer an explanatory role on meta-analysis
phenomena detection, Schmidt has acknowledged itself. One does not assign status simply on the basis
that scientific explanation normally involves the of association.
causal explanation of observed phenomena. Nev-
ertheless, he maintains that it is appropriate to
Meta-Analysis and Evaluative Inquiry
take scientific explanation to include “all research
It is surprising that methodological discussions of
processes that contribute ultimately to theory build-
meta-analysis and its applications have shown little
ing, including the first step of determining what
regard for the rationale that Glass originally pro-
the relationships are among important variable or
vided for its use. Glass claims that many researchers
constructs and how stable these relationships are”
misunderstand meta-analyses of outcome research
(Schmidt, 1993, p. 1164). Thus, the demonstration
because they fail to take cognizance of his rationale.
of a general effect, such as the pervasive influence
Specifically, this failure is offered by him as the rea-
of psycho-educational treatments on adult surgi-
son for the widespread misunderstanding of Smith,
cal patients, is deemed to be a meta-analysis at
Glass, and Miller’s (1980) original meta-analysis of
the “lowest level of explanation.” On the other
psychotherapy outcome studies.
hand, the use of meta-analysis to test compet-
In a number of different publications, Glass
ing theories of how patients cope with the stress
insists that meta-analysis should be understood as
of surgery is viewed as higher level explanatory
an exercise in evaluative research rather than in
meta-analysis.
scientific research (Glass, 1972; Smith, Glass, &
However, this attempt to extend the role of meta-
Miller, 1980; Glass & Kleigl, 1983). The core
analytic methods beyond phenomena detection to
of Glass’s underlying rationale for meta-analysis
explanation obscures the basic methodological dis-
involves drawing a strong distinction between scien-
tinction between phenomena detection and scien-
tific and evaluative inquiry. Glass’s position is that
tific explanation. As noted earlier in the chapter, the
researchers as scientists are concerned to satisfy their
stable general effects gleaned from meta-analysis are
curiosity by seeking truthful conclusions in the form
empirical phenomena, and statements about phe-
of theories comprising explanatory laws. By con-
nomena are the objects of scientific explanations;
trast, evaluators undertake research on behalf of a
they are not the explanations themselves. The ques-
client that is aimed at producing useful decisions
tion, “What do statements of empirical phenomena
based on descriptive determinations of the worth
explain?” occasions no natural reply. This is not sur-
of particular products or programs. For Glass, the
prising, for the successful detection of phenomena
meta-analysis of outcome studies properly involves
is essentially a descriptive achievement that involves
the integration of the products of evaluative research
investigative practices that are, for the most part,
only.
quite different from explanatory endeavors. In psy-
The methodology for this conception of meta-
chology, these methods are often statistical in kind.
analysis fashions the distinction between scientific
By contrast, scientific explanation is often causal-
and evaluative inquiry in terms of the relevance
mechanistic in nature (Salmon, 1984). On this
for each of the concepts of truth, explanation,
view, explanation requires the identification of the
values, problems, and generalizations. Because
mechanisms that underlie and give rise to empirical
of space limitations, I will consider just one of
phenomena, along with a detailing of the ways in
these contrasts—that of explanation. Glass contends
which those mechanisms produce the phenomena
that scientific inquiry involves the continual search
we seek to understand.
for subsurface explanations of surface phenomena.
When meta-analysis enters into the process of
Evaluative inquiry, on the other hand, does not seek
testing explanatory theories, it contributes to an
explanations:
evaluation of those theories in terms of predictive
success. However, this common strategy for evaluat- “A fully proper and useful explanation can be
ing scientific theories is not directly concerned with conducted without producing an explanation of why
their explanatory adequacy. To repeat, it is not being the product or program being evaluated is good or
denied that meta-analytic methods can be employed bad of how it operates to produce its effects . . . [It] is
in the course of testing theories, but meta-analysis usually enough for the evaluator to know that
itself is not an approach to theory testing (Chow, something attendant upon the [product or program]
1996). To employ meta-analysis to assist in the pre- is responsible for the valued outcomes.” (Glass, 1972,
dictive testing of an explanatory theory does not pp. 5–6)

haig 19
 Glass’s position seems to be that although pro-
gram treatments can be causally responsible for their
measured outcomes, it matters little that knowl-
edge of this gleaned from evaluation studies does
not tell us how programs produce their effects,
because such knowledge is not needed for policy
action.
Glass is surely correct in asserting that scientists
are centrally concerned with the construction of
causal theories to explain phenomena, for this is the
normal way in which they achieve understanding
of the empirical regularities they discover. However,
he is wrong to insist that proper evaluations should
deliberately ignore knowledge of underlying causal
mechanisms. The reason for this is that the effective
implementation and alteration of social programs
often benefits from knowledge of the relevant causal
mechanisms involved (Gottfredson, 1984), and
strategic intervention in respect of these is often
the most effective way to bring about social change.
Although standard versions of scientific realism are
wrong to insist that the relevant causal mechanisms
are always unobserved mechanisms, it is the case that
appeal to knowledge of covert causal mechanisms
will frequently be required for understanding and
change.
To conclude this highly selective evaluation of
Glass’s rationale for meta-analysis, science itself is
best understood as a value-laden, problem-oriented
human endeavor that tries to construct causal
explanatory theories of the phenomena it discovers.
There is no sound way of drawing a principled con-
trast between scientific and evaluative inquiry. These
critical remarks are not directed against the worth of
evaluation as such or against the use of meta-analysis
in evaluating program or product effectiveness. They
are leveled specifically at the conception of eval-
uation research that appears to undergird Glass’s
approach to meta-analysis.
Meta-Analysis and the Nature of Science
Proponents of meta-analysis often justify the
use of these methods by pointing out the need to
glean valuable knowledge from the information in a
domain that lies dormant in the cornucopia of scat-
tered primary studies. However, Sohn (1996) has
expressed concern about the quality of empirical psy-
chological studies that are used in meta-analysis. He
has urged resistance to the generally accepted view
that meta-analysis is a form of research rather than
a review of research, and he has balked at Schmidt’s
(1992) revisionist model of possible future science
20 t h e p h i l o s o p h y o f q u a n t i tat i v e m e t h o d s
as “. . . a two-tiered research enterprise [where] one
group of researchers will specialize in conducting
individual studies [and] another group will apply
complex and sophisticated meta-analysis methods to
those cumulative studies and will make the scientific
discoveries” (p. 1180). Sohn’s primary concerns are to
challenge the claim that meta-analysis is an impor-
tant vehicle of scientific discovery and to identify the
major problems besetting mainstream psychological
research.
Sohn (1996) has questioned the basic idea of
meta-analysis as a standalone literature review capa-
ble of discovering truths, whereas traditionally sci-
entific discoveries were contained in the empirical
findings of the primary studies themselves. For
Sohn, the idea that meta-analytic literature reviews
can make discoveries about nature rests on the
assumption that the primary research literature is
a proxy for nature. It is an assumption that he has
roundly rejected.
Noting the tendency of meta-analysts to paint a
bleak picture of progress in twentieth century psy-
chology, Sohn (1996) has suggested that although
meta-analysis has been introduced to improve mat-
ters in this regard, it is in fact symptomatic of its
poor progress. In his judgment, this lack of good
progress is a consequence of psychology adopting a
hypothesis-testing view of science. For Sohn, this
view of science seeks knowledge by testing research
hypotheses about the relationship of descriptive
variables without regard for causal mediating vari-
ables. Essentially, the approach amounts to the
hypothetico-deductive testing of outcome studies
through use of significance tests and effect size
measures. Sohn maintains that there are, in fact,
two deleterious consequences of such an approach
to research: one is the lack of agreement about
outcomes, and the other is the absence of knowl-
edge of the causal mechanisms that are responsi-
ble for those alleged outcomes. Meta-analysis is
indicted by Sohn for failing to remedy both types of
defect.
However, Sohn has supported his claim that
meta-analysis does not produce demonstrable evi-
dence for treatment effects in a curious way. He
has acknowledged that Smith, Glass, and Miller’s
(1980) well-known meta-analytic treatment of the
benefits of psychotherapy has been corroborated
by subsequent meta-analyses yet has maintained
that this does not constitute evidence for replica-
ble effects. He has expressed a distrust of research
that relies on statistical methods for making claims
about replicable effects. This distrust appears to
be founded in part on an extension of the view
attributed to Lord Rutherford that if an experimen-
tal study requires statistics, then the experiment is
in need of improvement. For Sohn, “If one’s science
needs [meta-analysis], one should have done better
science.”
However, this view flies in the face of widely
accepted scientific practice. Woodward’s (1989)
detailed examination of the practice of phenomena
detection in science strongly supports the view that
different parts of the various sciences, from physics
to anthropology, appropriately make extensive use of
statistical methods in the detection of empirical phe-
nomena. It is hard to imagine that statistics would
exist as we currently know it unless it provided a
necessary armament for science. In this regard, it
is worth noting that Sohn has acknowledged the
claim made by Hedges and Olkin (1985) that meta-
analysis in some form or other has a long history
of use in the hard sciences. Sohn has stated his dis-
agreement with this position, but he has not argued
against it. Space limitations preclude further analy-
sis of Sohn’s (1996) argument, but perhaps enough
has been said to suggest that it should be regarded
with some skepticism.
More work on the philosophical foundations
of meta-analysis is clearly needed. However, from
this highly selective examination of its conceptual
foundations, it can be concluded that meta-analysis
receives its primary justification in scientific research
by articulating one, but only one, way in which
researchers can fashion empirical generalization
from the findings of primary studies. It derives its
importance in this role directly from the impor-
tance accorded the goal of phenomena detection in
science.
Exploratory Factor Analysis
Despite the advanced statistical state and fre-
quent use of exploratory factor analysis in the
behavioral sciences, debate about its basic nature
and worth abounds. Thurstone (1947) has appro-
priately emphasized the exploratory nature of the
method, and many methodologists take it to be
a method for postulating latent variables that are
thought to underlie patterns of correlations. Some,
however, understand exploratory factor analysis as a
method of data reduction that provides an econom-
ical description of correlational data. The present
section considers this and other important foun-
dational issues that have figured prominently in
discussions of the method.
Factor Analytic Inference
Alongside the debate between the fictionalist and
realist interpretations of factors, there is a difference
of view about whether the basic inferential nature
of factor analytic inference is inductive or abductive
in nature. Expositions of exploratory factor analysis
seldom consider its inferential nature, but when they
do, the method is usually said to be inductive in char-
acter. This is not surprising, given that exploratory
factor analysis can be plausibly located histori-
cally within seventeenth- and eighteenth-century
empiricist philosophy of science and its inductive
conception of inquiry (Mulaik, 1987). However,
even if one relaxes the Baconian ideal that inductive
method is an algorithm that produces incorrigi-
ble knowledge, an inductive characterization of
exploratory factor analysis seems inappropriate. This
is because inductive inference, being descriptive
inference, cannot take the researcher from mani-
fest effects to theoretical entities that are different in
kind from those effects. However, abductive infer-
ence, which is concerned with the generation and
evaluation of explanatory hypotheses, can do so.
For this reason, exploratory factor analysis is bet-
ter understood as an abductive method of theory
generation (Haig, 2005a), a characterization that
coheres well with its general acceptance as a latent
variable method. With exploratory factor analysis,
abductive inference is explanatory inference that
leads back from presumed effects to underlying
causes.
There are different forms of abductive reason-
ing. Exploratory factor analysis is a method that
can facilitate the drawing of explanatory inferences
that are known as existential abductions. Existen-
tial abductions enable researchers to hypothesize
the existence, but not the nature, of entities previ-
ously unknown to them. The innumerable examples
of existential abduction in science include the ini-
tial postulation of hidden entities such as atoms,
genes, tectonic plates, and personality traits. In
cases like these, the primary thrust of the ini-
tial abductive inferences is to claims about the
existence of theoretical entities to explain empiri-
cal facts or phenomena. Similarly, the hypotheses
given to us through the use of exploratory factor
analysis postulate the existence of latent variables
such as Spearman’s g and extraversion. It remains
for further research to elaborate on the first rudi-
mentary conception of these variables and their
interrelation.
The factor analytic use of existential abduction
to infer the existence of, say, the theoretical entity
haig 21
g , can be coarsely reconstructed in accordance with
the following argument schema:
The surprising empirical phenomenon of the positive
correlations among different tests of ability is
identified.
If g exists, and it is validly and reliably measured by a
Weschler intelligence scale (and/or some other
objective test), then the positive manifold would
follow as a matter of course.
Hence, there are grounds for judging the hypothesis
of g to be initially plausible and worthy of further
pursuit.
Note that the schema for abductive inference and
its application to the generation of the hypothesis
of g is concerned with the form of the arguments
involved, rather than with the actual generation
of the explanatory hypotheses. The explanatory
hypothesis is given in the second premise of the argu-
ment. An account of the genesis of the explanatory
hypothesis must, therefore, be furnished by some
other means. It is plausible to suggest that reasoning
to explanatory hypotheses trades on our evolved cog-
nitive ability to abductively generate such hypothe-
ses (Carruthers, 2002). Whatever its origin, an
informative methodological characterization of the
abductive nature of factor analytic inference must
appeal to the scientist’s own psychological resources
as well as those of logic. This injunction is motivated
by the realist methodological thesis of naturalism
stated near the beginning of the chapter.
Although exploratory factor analysis exemplifies
well the character of existential abduction, it is
clearly not an all-purpose method for abductively
generating explanatory hypotheses and theories.
With its focus on common factors, it can prop-
erly serve as a generator of elementary theories only
in those multivariate domains that have common
causal structures.
The Principle of the Common Cause
It is well known that exploratory factor analysis is
a common factor analytic model in which the latent
factors it postulates are referred to as common factors.
Less well known is the fact that there is an important
principle of scientific inference, known as the prin-
ciple of the common cause. (e.g., Sober, 1988), that
can be used to drive the nature and shape of the exis-
tential abductive inferences involved in exploratory
factor analysis. The principle of the common cause
can be formulated concisely as follows: “Whenever
two or more events are improbably, or significantly,
22 t h e p h i l o s o p h y o f q u a n t i tat i v e m e t h o d s
correlated, infer one or more common causes unless
there is good reason not to.” Clearly, the princi-
ple should not be taken as a hard-and-fast rule,
for in many cases, proper inferences about corre-
lated events will not be in terms of common causes.
The qualifier, “unless there is good reason not to,”
should be understood as an injunction to consider
causal interpretations of the correlated events other
than the common causal kind. For example, in a
given research situation, the correlated events might
be related as direct causes, or their relationship
might be mediated by a third variable in a causal
sequence.
Although exploratory factor analysis is used to
infer common causes, expositions of common factor
analysis that explicitly acknowledge the importance
of the principle of the common cause are rare. Kim
and Mueller’s (1978) textbook exposition of factor
analysis is a noteworthy exception. In discussing
the conceptual foundations of factor analysis, these
authors evince the need to rely on what they call
the postulate of factorial causation. The postulate
of factorial causation is characterized by them as
“the assumption that the observed variables are lin-
ear combinations of underlying factors and that
the covariation between observed variables solely
results from their common sharing of one or more
of the common factors” (p. 78). The authors make
clear that the common factors mentioned in the
assumption are to be regarded as underlying causal
variables. Understood as a methodological injunc-
tion, this postulate functions as a variant of the
principle of the common cause. Without appeal
to this principle, factor analysts could not iden-
tify the underlying factor pattern from the observed
covariance structure.
There are two features of the principle of the com-
mon cause that make it particularly suitable for use
in exploratory factor analysis. First, it can be applied
in situations where we do not know how likely it
is that the correlated effects result from a common
cause. The abductive inference to common causes
is a basic explanatory move that is non-probabilistic
and qualitative in nature. It is judgments about the
soundness of the abductive inferences, rather than
the assignment of probabilities, that confer initial
plausibility on the factorial hypotheses spawned by
exploratory factor analysis. Second, the principle
can also be used in situations where we are essentially
ignorant of the nature of the common cause. With
this second feature, the principle of the common
cause accommodates the fact the exploratory factor
analysis trades in existential abductions.
 Further, it is important to appreciate that the
principle of the common cause does not function
in isolation from other methodological constraints.
Embedded in exploratory factor analysis, the prin-
ciple helps one limit existential abductive inference
to those situations where we reason back from
correlated effects to one or more common causes.
Although covariation is an important basic datum in
science, not all effects are expressed as correlations,
and of course, not all causes are of the common
causal variety. It follows from this that one should
not always look for common causal interpreta-
tions of multivariate data, for there are numerous
alternative latent variable models.
The Underdetermination of Factors
The methodological literature on exploratory fac-
tor analysis has given considerable attention to the
indeterminacy of factors in the common factor
model. Factor indeterminacy arises from the fact
that the common factors are not uniquely deter-
mined by their related manifest variables. As a
consequence, a number of different common fac-
tors can be produced to fit the same pattern of
correlations in the manifest variables.
Although typically ignored by factor analytic
researchers, factor indeterminacy is an epistemic
fact of life that continues to challenge factor ana-
lytic methodologists. Some methodologists regard
factor indeterminacy as a serious problem for com-
mon factor analysis and recommend the use of
alternative methods such as component analysis
methods because they are considered to be deter-
minate methods.
One constructive perspective on the issue of fac-
tor indeterminacy has been suggested by Mulaik and
McDonald (Mulaik & McDonald, 1978; McDon-
ald & Mulaik, 1979; Mulaik, 1987). Their position
is that the indeterminacy involved in interpreting
the common factors in exploratory factor analysis
is just a special case of the general indeterminacy
of theory by empirical evidence widely encountered
in science, and it should not, therefore, be seen as
a debilitating feature that forces us to give up on
common factor analysis.
Indeterminacy is pervasive in science. It occurs in
semantic, metaphysical, and epistemological forms
(McMullin, 1995). Factor indeterminacy is essen-
tially epistemological in nature. The basic idea of
epistemological or, more precisely, methodological
indeterminacy is that the truth or falsity (better,
acceptance or rejection) of a hypothesis or theory is
not determined by the relevant evidence (Duhem,
1954). In effect, methodological indeterminacy
arises from our inability to justify accepting one
theory among alternatives on the basis of empirical
evidence alone.
Mulaik (1987) sees underdetermination in
exploratory factor analysis as involving inductive
generalizations that go beyond the data. How-
ever, inductive underdetermination should be seen
as applying specifically to the task of establishing
factorial invariance where one seeks constructive
or external replication of factor patterns. However,
for exploratory factor analysis there is also need to
acknowledge and deal with abductive underdeter-
mination involved in the generation of explanatory
factorial theories. The sound abductive generation
of hypotheses is essentially educated guess work.
Thus, drawing from background knowledge, and
constrained by correlational empirical evidence, the
use of exploratory factor analysis can reasonably be
expected to yield a plurality of factorial hypotheses
or theories that are thought to be in competition.
This contrasts strongly with the unrealistic expec-
tation held by many earlier users of exploratory
factor analysis that the method would deliver them
strongly justified claims about the one best factorial
hypothesis or theory.
How then, can exploratory factor analysis deal
with the specter of underdetermination in the con-
text of theory generation? One plausible answer is
that exploratory factor analysis narrows down the
space of a potential infinity of candidate theories
to a manageable subset by facilitating judgments
of initial plausibility (Haig, 2005a). It seems clear
enough that scientists often make judgments about
the initial plausibility of the explanatory hypothe-
ses and theories that they generate. However, it
is less clear just to what this evaluative criterion
amounts (cf. Whitt, 1992). With an abductive
conception of exploratory factor analysis, judg-
ments of the initial plausibility of theories are
judgments about the soundness of the abductive
arguments employed in generating those theories.
It seems reasonable to suppose that those who
employ exploratory factor analysis as an abductive
method of theory generation often make com-
pressed judgments of initial plausibility. By confer-
ring judgments of initial plausibility on the theories
it spawns, exploratory factor analysis deems them
worthy of further pursuit, whereupon it remains
for the factorial theories to be further developed
and evaluated, perhaps through the use of confir-
matory factor analysis. It should be emphasized that
haig 23
using exploratory factor analysis to facilitate judg-
ments about the initial plausibility of hypotheses will
still leave the domains being investigated in a state
of considerable theoretical underdetermination. It
should also be stressed that the resulting plural-
ity of competing theories is entirely to be expected
and should not be thought of as an undesirable
consequence of employing exploratory factor anal-
ysis. To the contrary, it is essential for the growth
of scientific knowledge that we vigorously promote
theoretical pluralism (Hooker, 1987), develop the-
oretical alternatives, and submit them to critical
scrutiny.
Exploratory Factor Analysis and
Confirmatory Factor Analysis
The aforementioned consideration of exploratory
factor analysis supports the conclusion that there
is an important role for its use in factor analytic
research. However, this conclusion raises the ques-
tion of how exploratory factor analysis relates to its
confirmatory namesake. In contrast to popular ver-
sions of the classical inductivist view of science that
inductive method can generate secure knowledge
claims, the use of exploratory factor analysis as an
abductive method of theory generation can only fur-
nish researchers with a weak logic of discovery—one
that gives them educated guesses about underlying
causal factors. It is for this reason that those who use
exploratory factor analysis to generate theories need
to supplement their generative assessments of the
initial plausibility of those theories with additional
consequentialist justification in the form of confir-
matory factor analytic testing or some alternative
approach to theory appraisal.
However, in the factor analytic literature, there is
a division of opinion about whether exploratory fac-
tor analysis and confirmatory factor analysis should
be viewed as complementary or competing methods
of common factor analysis. Quite a number of factor
analytic methodologists have expressed views that
discourage their complementary use in factor ana-
lytic research. For example, Gorsuch (1983), in his
well-known book on factor analysis, has expressed
a view about the relative importance of exploratory
and confirmatory factor analysis that seems to be
quite widely held today:
Although the next three chapters [of Factor analysis]
are primarily concerned with exploratory factor
analysis, the space and time given to that technique is
a function of the complexity of resolving its
problems, not of its theoretical importance. On the
24 t h e p h i l o s o p h y o f q u a n t i tat i v e m e t h o d s
contrary, confirmatory factor analysis is the more
theoretically important—and should be the much
more widely used—of the two major factor analytic
approaches. (p. 134)
Although Gorsuch makes his claim in emphatic
terms, he provides no justification for it. He seems
to assume that theory testing is more important
than theory generation. However, this belief is diffi-
cult to defend, given the fact that there are many
other important phases of scientific inquiry that
together demand most of the researcher’s method-
ological time. Recall, for example, the importance
to science of the detection of empirical phenomena
and the generation, development, and comparative
appraisal of theories. Viewed in this light, theory
testing is just one, albeit important, part of scien-
tific method (cf. Simon, 1968). Given the fact that
science is as much concerned with theory genera-
tion as it is with theory testing, and acknowledging
that exploratory factor analysis is a useful abduc-
tive method of theory generation, exploratory factor
analysis deserves to be regarded as important as con-
firmatory factor analysis in the theory constructor’s
toolkit.
To conclude, despite the fact that exploratory
factor analysis has been frequently employed in
psychological research, the extant methodological
literature on the method seldom acknowledges the
explanatory and ontological import of the method’s
inferential nature. Abduction is a major form of cre-
ative reasoning in science, and the principle of the
common cause is a maxim of scientific inference with
important application in research. By incorporating
these two related elements into its fold, exploratory
factor analysis is ensured an important, albeit cir-
cumscribed, role in the construction of explanatory
theories in psychology and other sciences. By gen-
erating structural models about common causes,
exploratory factor analysis can serve as a valuable
precursor to confirmatory factor analysis.
Causal Modeling
During the last 50 years, social and behavioral
science methodologists have developed a variety of
increasingly sophisticated statistical methods to help
researchers draw causal conclusions from correla-
tional data. These causal modeling methods, as they
have sometimes been called, include path analy-
sis, confirmatory factor analysis, and full structural
equation modeling.
Despite the fact that psychological researchers are
increasingly employing more sophisticated causal
modeling methods in place of simple regression
and partial correlation procedures, worries about
both their accompanying methodology and their
misuse have been expressed (e.g., Cliff, 1983). In
this section, I consider some philosophical aspects
of three foundational issues that have been dis-
cussed in the literature on causal modeling: the
different ideas of causation presupposed by causal
modeling; the suggestion that causal modeling
can be viewed as a form of inference to the best
explanation; and the contested nature of latent
variables.
Causal Modeling and Theories of Causation
One central methodological issue in the debates
about causal modeling has to do with the appropri-
ateness of the nature of causation involved in various
causal modeling procedures. A popular view of the
matter is clearly expressed by Kenny (1979), who
points out that three conditions must be satisfied
for a researcher to claim that one variable is the
cause of another. The first condition is that the rela-
tionship be asymmetric. The second condition is
that a functional relationship be present between
cause and effect. The third condition is that the
causal relationship be direct or non-spurious. These
three conditions are exactly those of the regular-
ity theory of causation, which depicts the causal
relationship between events in terms of their reg-
ular succession, covariation, and contiguity. The
regularity theory, which is more or less Humean
in character, provides an important part of the
epistemic backdrop against which traditional causal
modeling methods like path analysis have been
understood.
However, like other parts of the standard empiri-
cist enterprise, this theory has received strong criti-
cism. Its claimed limitations can best be appreciated
by contrasting it with a scientific realist alternative
known as the generative theory of causation (Harré
& Madden, 1975). Briefly stated, the generative the-
ory depicts causation as a relationship where, under
appropriate conditions, a causal mechanism produces
its effect. For this to happen, the causal mechanism
must connect to its effect and have the power to
generate that effect, usually when stimulated by the
appropriate causal condition. It is the productivity
of a generative mechanism that makes it a causal
mechanism, and for this to occur, there must be a
naturally necessary connection that allows for the
transmission of power from cause to effect. This
causal power exists irrespective of whether it is cur-
rently being exercised. As such, it is properly viewed
as a tendency—that is, an existing state of an object,
which, if unimpeded, will produce its effect. We
are, therefore, able to infer abductively the presence
of the causal mechanism on the basis of knowl-
edge of the triggering condition and/or its presumed
effect.
Advocates of the generative theory of causa-
tion claim it has a number of important advan-
tages over the regularity theory. One advantage of
the generative theory is that it is able to accom-
modate deep structural, explanatory theories that
postulate unobserved generative mechanisms. It is
argued that we need a theory of causation that
affords us the conceptual space to do this, because
many of the world’s causal mechanisms are not
open to direct inspection. The latent variables of
many of our causal modeling methods are thought
by many to be precisely of this kind. A related
advantage of the generative theory is that it is
needed for enlightened social policy because, as
noted in the discussion of evaluation research ear-
lier, the possibility of ameliorative action depends
on effecting change based on an understanding
of how things work, and for this, knowledge of
the relevant underlying causal mechanisms is often
essential.
A third, and significant, advantage of the the-
ory of generative causation is that it enables us to
draw the important distinction between empirical
regularities and genuine causal laws. An adequate
methodology of causal modeling must be able to
draw the distinction between empirical regularities
and causal laws, because the ability to do so is a
conceptual requirement of being able to differen-
tiate properly direct causal relations from spurious
correlations. By collapsing this distinction, empiri-
cists, with their regularity theory of causation, are
unable to articulate a satisfactory notion of spuri-
ousness. For example, Simon’s (1985) influential
analysis of spurious correlation explicitly rejects the
generative theory of causation and endeavors to
ground the distinction between true and spurious
correlations on a commitment to an empiricist view
of causation. The common or intervening causes
that bring about spurious correlations will typically
be unobserved. However, for a statistical treat-
ment of these variables to be consistent with the
regularity theory, Simon’s view of causation forces
researchers to focus on altogether different variables
at the manifest level. But this cavalier ontologi-
cal slide wrecks our efforts to obtain worthwhile
causal knowledge, because the manifest replacement
variables cannot act as effective surrogates for their
haig 25
presumed common and intervening causes. They
are ontologically distinct from such causes and,
although as causal conditions they may trigger their
latent counterparts, they do not function as major
causal mechanisms that can bring about spurious
correlations.
Although it can plausibly be argued that a gener-
ative view of causation is required to make sense of
research that embraces hidden causal mechanisms, it
does not follow, as is often supposed (e.g., Manicas,
1989; Sayer, 1992), that the regularity theory has
no place in a realist conception of science. With
its emphasis on the ideas of regularity, it would
seem to be a suitable account of causation for claims
about phenomena that take the form of empirical
generalizations. Nor should it be thought that the
regularity theory and the generative theory together
give one a full understanding of causation in science.
For example, structural equation modeling provides
knowledge of causal networks. As such, it does not so
much encourage the development of detailed knowl-
edge of the nature of latent variables as it specifies
the range and order of causal relations into which
latent and manifest variables enter. For this type of
research, a network theory of causation is needed
(Thagard, 1999).
The suggestion that different conceptions of cau-
sation are relevant to causal modeling fits with a
philosophy of causal pluralism, which is increasingly
being recommended in contemporary methodolog-
ical studies of the nature of causation (Godfrey-
Smith, 2009).
Structural Equation Modeling and
Inference to the Best Explanation
The guess-and-test strategy of the hypothetico-
deductive method takes predictive accuracy as the
sole criterion of theory goodness. However, it
seems to be the case that in research practice, the
hypothetico-deductive method is sometimes com-
bined with the use of supplementary evaluative crite-
ria such as simplicity, scope, and fruitfulness. When
this happens, and one or more of the supplementary
criteria have to do with explanation, the combined
approach can appropriately be regarded as a version
of inference to the best explanation, rather than just
an augmented account of the hypothetico-deductive
method (Haig, 2009). This is because the central
characteristic of the hypothetico-deductive method
is a relationship of logical entailment between the-
ory and evidence, whereas with inference to the
best explanation the relationship is also one of
explanation. The hybrid version of inference to
26 t h e p h i l o s o p h y o f q u a n t i tat i v e m e t h o d s
the best explanation being considered here will
allow the researcher to say that a good explana-
tory theory will rate well on the explanatory criteria
and, at the same, boast a measure of predictive
success. Most methodologists and scientists will
agree that an explanatory theory that also makes
accurate predictions will be a better theory for
doing so.
Although the use of structural equation mod-
eling in psychology often involves testing models
in hypothetico-deductive fashion, it also contains
a minority practice that amounts to inference to
the best explanation in the sense just noted. This
latter practice involves the explicit comparison of
models or theories in which an assessment of
their goodness-of-fit to the empirical evidence is
combined with the weighting of the fit statistics
in terms of parsimony indices (Kaplan, 2000).
Here goodness-of-fit provides information about the
empirical adequacy of the model, whereas parsi-
mony functions as a criterion having to do with
the explanatory value of the model. Both are used
in judgments of model goodness. Markus, Hawes,
and Thasites (2008) recently have suggested that
in structural equation modeling, model fit can
be combined with model parsimony, understood
as explanatory power, to provide an operational-
ized account of inference to the best explanation.
They discussed the prospects of using structural
equation modeling in this way to evaluate the com-
parative merits of two- and three-factor models of
psychopathy.
Do Latent Variables Exist?
Many causal modeling methods are latent vari-
able methods, whose conceptual foundations are to
be found in the methodology of latent variable the-
ory (Borsboom, 2005; 2008). Central to this theory
is the concept of a latent variable itself. However,
the notion of a latent variable is a contested concept,
and there are fundamental philosophical differences
in how it should be understood.
A clear example of the contested nature of the
concept of a latent variable is to be found in the
two quite different interpretations of the nature of
the factors produced by exploratory factor analy-
sis. One view, known as fictionalism, maintains
that the common factors, the output of exploratory
factor analysis, are not theoretical entities invoked
to explain why the observed variables correlate the
way that they do. Rather, these factors are taken
to be summary expressions of the way manifest
variables co-vary. Relatedly, theories that marshal
descriptions of such factors are properly considered
to serve the instrumentalist function of econom-
ically redescribing the original correlational data.
This interpretation of exploratory factor analysis
has been quite influential in psychometrics (Block,
1976) and has been taught to generations of psychol-
ogy students through textbooks on psychological
testing (e.g., Anastasi & Urbina, 1997). Fictional-
ism seems to be the preferred option of many factor
analytic researchers in the domains of personality
and intelligence.
However, fictionalism is a difficult position to
defend, and it seems to fail in factor analysis for
the reason it fails in science generally: it inappro-
priately grants ontological significance to a sharp
distinction between observation and theory to but-
tress the claim that only observable, or manifest,
entities exist, when observability is really a matter
of degree. Fictionalists argue that because we do
not have perceptual experience of theoretical enti-
ties, we do not have grounds for saying they exist;
we only have grounds for claiming that observable
entities exist. But realist philosophers of science (e.g.,
Maxwell, 1962) assert in reply that fictionalists can-
not maintain a sharp distinction between what is
observable and what is unobservable. What can-
not be seen directly by the unaided eye might be
observable through a magnifying glass and what
cannot be observed through a magnifying glass
might be observed through a microscope. Impor-
tantly, how we draw the observable/unobservable
distinction at a particular time is a function of
prior knowledge, our physiological make-up, and
available instrumentation. Thus, the distinction
provides no basis for deciding what entities do, and
do not, exist. To assert that factors are theoretical
entities is not to regard them as having a special
existence; rather, it is to acknowledge that we come
to know them indirectly in terms of their correlated
effects. On this realist interpretation, the factors are
regarded as latent variables that underlie, and give
rise to, the correlated manifest variables. Borsboom
(2005) has made a strong case for adopting a realist
attitude to latent variables more generally by com-
bining an argument similar to Maxwell’s, along with
other foundational considerations in philosophy of
science and psychometrics.
This general argument against fictionalism simul-
taneously supports the doctrine of realism in sci-
ence, but it does not by itself establish that
the factors of exploratory factor analysis should
be given a realist interpretation. Whether this
should happen depends also on whether exploratory
factor analysis can facilitate the drawing of sound
abductive inference about the existence of latent
factors.
This highly selective consideration of the phi-
losophy of causal modeling points to three con-
clusions: (1) that causation in causal modeling
manifests itself in a number of different ways;
(2) that causal modeling can transcend the limi-
tations of the hypothetico-deductive method and
adopt the practice of inference to the best expla-
nation; and (3) that latent variables deserve to be
given a realist interpretation as genuine theoretical
entities.
Conclusion
The philosophy of research methods is an aspect
of research methodology that receives limited atten-
tion in behavioral science education. The majority
of students and research practitioners in the behav-
ioral sciences obtain the bulk of their knowledge of
research methods from textbooks. However, a casual
examination of these texts shows that they tend to
pay little, if any, serious regard to the philosophy
of science and its bearing on the research process.
As Kuhn pointed out nearly 50 years ago (Kuhn,
1962; 1996), textbooks play a major role in dogmat-
ically initiating students into the routine practices
of normal science. Serious attention to the philos-
ophy of research methods would go a considerable
way toward overcoming this uncritical practice. As
contemporary philosophy of science increasingly
focuses on the contextual use of research methods in
the various sciences, it is to be hoped that research
methodologists and other behavioral scientists will
avail themselves of the genuine methodological
insights that it contains.
Future Directions
In this final section of the chapter, I suggest a
number of directions that future work in the philos-
ophy of quantitative methods might take. The first
three suggestions are briefly discussed; the remaining
suggestions are simply listed.
Understand Quantitative Methods Through
Methodology
A proper understanding of research methods can-
not be had without an appreciation of their accom-
panying methodology (see Proctor & Capaldi,
2001). Methodology is the interdisciplinary field
that studies methods. It draws from the disciplines
haig 27
of statistics, philosophy of science, and cognitive
science, among others. And yet, the professional
literature of these disciplines does not figure in
the content of research methods courses. Further,
it is important to appreciate that methodology
has descriptive, critical, and advisory dimensions:
Again, the typical methods curriculum does not sys-
tematically deal with research methods with these
considerations in mind. It is not surprising, there-
fore, that psychologists’ understanding of research
methods often leaves a lot to be desired.
A realist-oriented methods curriculum would
profitably consider methods in the light of the pri-
mary characteristics of realist methodology outlined
early in the chapter. To mention just three of these:
Greater prominence would be given to generative
methodology in which reasoning well to hypothe-
ses and theories would figure in the assessment
of those knowledge claims. The sound abductive
reasoning to factorial hypotheses using exploratory
factor analysis is perhaps psychology’s best example
of generative justification. Similarly, the coherentist
justification of explanatory theories using methods
of inference to the best explanation would feature
much more prominently than it does at present.
Finally, in adopting methods that are apt for us
as knowing subjects, heuristic procedures would
receive much more explicit attention in the methods
curriculum as realistic guides to our thinking.
The British Psychological Society now takes con-
ceptual and historical issues as one of psychol-
ogy’s seven core areas. Teaching methods through
methodology is the appropriate way to employ this
core area in the teaching of research methods. The
American Psychological Association and the Asso-
ciation of Psychological Science would do well to
follow suit, for it is only by making considered use
of methodology that a genuine education in research
methods can be achieved.
Rethink the Quantitative/Qualitative
Distinction
A major feature of the methodological landscape
has been the discussion of the distinction between
quantitative and qualitative methods. Although per-
haps necessary in establishing a legitimate role for the
use of qualitative methods in research, the distinc-
tion is now the subject of critical scrutiny. The way
the original distinction was drawn has been ques-
tioned (e.g., Michell, 2004), and the combination
of qualitative and quantitative methods in mixed
methods strategies has been strongly promoted in
recent times.
28 t h e p h i l o s o p h y o f q u a n t i tat i v e m e t h o d s
However, the quantitative/qualitative debate has
not considered the possibility that most methods
have both quantitative and qualitative dimensions.
In many cases, we are likely to gain a better under-
standing of the research methods we use not by view-
ing them as either qualitative or quantitative but by
regarding them as having both qualitative and quan-
titative dimensions. Three examples are mentioned
here. First, grounded theory (e.g., Strauss, 1987),
the most prominent extant qualitative methodol-
ogy, is in good part the product of a translation
from some sociological quantitative methods of the
1950s. Moreover, there is nothing in principle to
stop researchers using quantitative methods within
the fold of grounded theory. Exploratory factor
analysis, for example, could sometimes be used for
generating grounded theory.
Second, although exploratory factor analysis
itself is standardly characterized as a multivariate sta-
tistical method, the inferential heart of the method
is the important scientific heuristic known as the
principle of the common cause. Importantly, this
principle, which guides the factor analytic inference
from correlations to underlying common factors,
can be effectively formulated in qualitative terms.
Finally, the theory of explanatory coherence
(Thagard, 1992), which evaluates theories in terms
of their explanatory power, is a qualitative method
of theory appraisal, but it is implemented by a com-
puter program that is part of the method proper,
and that has a connectionist architecture that is
mathematically constrained.
It is recommended, then, that methodologists
and researchers seriously entertain the prospect that
individual methods are likely to have a mix of
qualitative and quantitative features—that is, that
individual methods are themselves mixed methods.
Evaluate the Philosophical Critiques of
Quantitative Research Methods
Most of the occasional references to scientific real-
ism in psychology are to Bhaskar’s (1975; 1979)
critical realism (e. g., Manicas & Secord, 1983),
a philosophy that has had considerable impact on
the social sciences (e.g., Sayer, 1992). Interestingly,
critical realists have expressed strong reservations
about the use of statistical methods in quantitative
research. Bhaskar himself goes so far as to say that
causal models should be “totally discarded.” There
are various reasons for this attitude (see Pratschke,
2003), but perhaps the most fundamental one is
the claim that statistical models themselves do not
provide researchers with the substantive models that
are sought in causal modeling research.
However, this claim rests on a mistaken concep-
tion of the relation between statistical models and
substantive theoretical models. It is hard to deny
that consideration of much more than the statistical
machinery of causal modeling is needed to ground
substantive conclusions. Indeed, it is difficult to
see how any statistical method could be properly
understood and used in research without appeal to
suprastatistical matters. Consider, for example, the
oft-made claim that factors of exploratory factor anal-
ysis are statistical entities and that the method cannot,
therefore, be used to favor one substantive facto-
rial theory of intelligence over another (e.g., Gould,
1996). This claim is false because factor analysts typ-
ically transcend the statistical level of the method
and makes use of the relevant part of Latent Vari-
able Theory to generate plausible hypotheses about
the existence of latent variables. Of central relevance
here is the fact that exploratory factor analysis exploits
theso-called“principleofthecommoncause”tosanc-
tion inferences to the initial plausibility of interpreted
latent variables. We saw earlier that inferences from
manifest to latent variables made in accordance with
this principle are abductive, or explanatory, in nature
and are made by factor analysts themselves. Although
the statistical machinery of multiple regression and
partial correlation theory is obviously an impor-
tant part of exploratory factor analysis, its primary
function is to facilitate researchers’ suprastatistical
inferences to latent factors.
It is important to appreciate that the interpretive
dimension on causal modeling methods is a proper
part of its methodology. There is nothing in critical
realism, or other variants of scientific realism, that
prevents one from taking such an outlook on causal
modeling. Indeed, scientific realism comports well
with causal modeling methods that countenance
latent variables.
Additional Directions
Space considerations prevent discussion of addi-
tional future directions in the philosophy of quan-
titative methods. However, the following points
deserve to be on an agenda for future study.
• Develop a modern interdisciplinary concep-
tion of research methodology.
• Give more attention to investigative strategies
in psychological research.
• Take major philosophical theories of scientific
method seriously.
• Apply insights from the “new experimental-
ism” in the philosophy of science to the understand-
ing of quantitative research methods.
• Develop the philosophical foundations of the-
ory construction methods in the behavioral sciences.
• Assess the implications of different theories of
causality for research methods.
• Examine the philosophical foundations of
“new” research methods such as data mining, struc-
tural equation modeling, and functional neuroimag-
ing.
Author note
Brian D. Haig, Department of Psychology, Uni-
versity of Canterbury Christchurch, New Zealand,
Email: brian.haig@canterbury.ac.nz
References
Acree, M. C. (1979). Theories of statistical inference in psy-
chological research: A historico-critical study. Dissertation
Abstracts International, 39, 5073B. (UMI No. 7907000).
Anastasi, A., & Urbina, S. (1997). Psychological testing (7th ed.)
Upper Saddle River, NJ: Prentice-Hall.
Bangert-Drowns, R. L. (1986). Review of developments in meta-
analytic method. Psychological Bulletin, 99, 388–399.
Behrens, J. T., & Yu, C-H. (2003). Exploratory data analysis. In
J. A. Schinka & W. F. Velicer (Eds.), Handbook of psychology,
Vol. 2 (pp. 33–64). New York: Wiley.
Berger, J. O. & Selke, T. (1987). Testing a point null hypothesis:
The irreconcilability of p values and evidence (with com-
ments). Journal of the American Statistical Association, 82,
112–139.
Bhaskar, R. (1975). A realist philosophy of science. Brighton,
England: Harvester.
Bhaskar, R. (1979). The possibility of naturalism. Brighton,
England: Harvester.
Block, N. J. (1976). Fictionalism, functionalism, and factor
analysis. In R. S. Cohen, C. A. Hooker, & A. C. Micha-
los (Eds.), Boston Studies in the Philosophy of Science, Vol. 32,
(pp. 127–141). Dordrecht, the Netherlands: Reidel.
Bolles, R. C. (1962). The difference between statistical and
scientific hypotheses. Psychological Reports, 11, 629–645.
Borsboom, D. (2005). Measuring the mind: Conceptual issues in
contemporary psychometrics. Cambridge, England: Cambridge
University Press.
Borsboom, D. (2008). Latent variable theory. Measurement:
Interdisciplinary Research and Perspectives, 6, 25–53.
Bunge, M. (2008). Bayesianism: Science or pseudoscience?
International Review of Victimology, 15, 165–178.
Carruthers, P. (2002). The roots of scientific reasoning: Infancy,
modularity, and the art of tracking. In P. Carruthers, S. Stich,
& M. Siegal (Eds.), The cognitive basis of science (pp. 73–95).
Cambridge, England: Cambridge University Press.
Chow, S. L. (1996). Statistical significance: Rationale, validity, and
utility. London, England: Sage.
Cliff, N. (1983). Some cautions concerning the application of
causal modeling methods. Multivariate Behavioral Research,
18, 115–126.
Cohen, J. (1994). The earth is round (p <.05). American
Psychologist, 49, 997–1003.
haig 29
Cook, T. D., Cooper, H., Cordray, D. S., Hartmann, H.,
Hedges, L. V., Light, R. J., et al. (1992). Meta-analysis for
explanation: A casebook. New York: Russell Sage Foundation.
Duhem, P. (1954). The aim and structure of physical theory (2nd
ed., P. P. Weiner, Trans.). Princeton, NJ: Princeton University
Press.
Earman, J. (1992). Bayes or bust? A critical examination of Bayesian
confirmation theory. Cambridge, MA: MIT Press.
Efron, B., & Tibshirani, R. (1993). An introduction to the
bootstrap. New York: Chapman & Hall.
Ehrenberg, A. S. C., & Bound, J. A. (1993). Predictability and
prediction. Journal of the Royal Statistical Society, Part 2, 156,
167–206.
Fidell, L. S., & Tabachnick, B. G. (2003). Preparatory data anal-
ysis. In J. A. Schinka & W. F. Velicer (Eds.), Handbook of
psychology, Vol. 2 (pp. 115–121). New York: Wiley.
Fisher, R. A. (1925). Statistical methods for research workers.
Edinburgh, Scotland: Oliver & Boyd.
Giere, R. N. (1972). The significance test controversy. British
Journal for the Philosophy of Science, 23, 170–180.
Gigerenzer, G. (1993). The superego, the ego, and the id in
statistical reasoning. In G. Keren & C. Lewis (Eds.), A hand-
book for data analysis in the behavioral sciences (pp. 311–339).
Hillsdale, NJ: Lawrence Erlbaum.
Gigerenzer, G., Krauss, S., & Vitouch, O. (2004). The null
ritual: What you wanted to know about significance testing
but were afraid to ask. In D. Kaplan (Ed.), The Sage handbook
of quantitative methodology for the social sciences (pp. 391–
408). Thousand Oaks, CA: Sage.
Glass, G. V. (1972). The wisdom of scientific inquiry on
education. Journal of Research in Science Teaching, 9, 3–18.
Glass, G. V. (1976). Primary, secondary, and meta-analysis of
research. Educational Researcher, 5, 3–8.
Glass, G. V., & Kleigl, R. M. (1983). An apology for research inte-
gration in the study of psychotherapy. Journal of Consulting
and Clinical Psychology, 51, 28–41.
Glass, G. V., McGaw, B., & Smith, M. L. (1981). Meta-analysis
in social research. Beverly Hills, CA: Sage.
Glymour, C. N. (1980). Theory and evidence. Princeton, NJ:
Princeton University Press.
Godfrey-Smith, P. (2009). Causal pluralism. In H. Beebee, C.
Hitchcock, & P. Menzies (Eds.), The Oxford handbook of
causation (pp. 326–337). Oxford, England: Oxford Univer-
sity Press.
Good, I. J. (1983). The philosophy of exploratory data analysis.
Philosophy of Science, 50, 283–295.
Goodman, S. N. (1993). P values, hypothesis tests, and likeli-
hood: Implications for epidemiology of a neglected historical
debate. American Journal of Epidemiology, 137, 485–495.
Gorsuch, R. L. (1983). Factor analysis (2nd ed.). Hillsdale, NJ:
Lawrence Erlbaum.
Gottfredson, G. D. (1984). A theory-ridden approach to
program evaluation. American Psychologist, 39, 1101–1112.
Gould, S. J. (1996). The mismeasure of man (2nd ed.). New York:
Norton.
Greenwood, J. D. (1992). Realism, empiricism, and social
constructionism. Theory and Psychology, 2, 131–151.
Haig, B. D. (1987). Scientific problems and the conduct of
research. Educational Philosophy and Theory, 19, 22–32.
Haig, B. D. (2005a). Exploratory factor analysis, theory genera-
tion, and scientific method. Multivariate Behavioral Research,
40, 303–329.
Haig, B. D. (2005b). An abductive theory of scientific method.
Psychological Methods, 10, 371–388.
30 t h e p h i l o s o p h y o f q u a n t i tat i v e m e t h o d s
Haig, B. D. (2009). Inference to the best explanation: A neglected
approach to theory appraisal in psychology. American Journal
of Psychology, 122, 219–234.
Harré, R., & Madden, E. H. (1975). Causal powers. Oxford,
England: Blackwell.
Hedges, L. V., & Olkin, I. (1985). Statistical methods for meta-
analysis. New York: Academic Press.
Hooker, C. A. (1987). A realistic theory of science. New York: State
University of New York Press.
Howson, C., & Urbach, P. (2006). Scientific reasoning: The
Bayesian approach (3rd ed.). La Salle, IL: Open Court.
Hubbard, R., & Lindsay, R. M. (2008). Why p values are not a
useful measure of evidence in statistical significance testing.
Theory and Psychology, 18, 69–88.
Hubbard, R., & Ryan, P. A. (2000). The historical growth of
statistical significance testing in psychology—and its future
prospects. Educational and Psychological Measurement, 60,
661–681.
Hunter, J. E., & Schmidt, F. L. (2004). Methods of meta-
analysis: Correcting error and bias in research findings (2nd
ed.). Newbury Park, CA: Sage.
Hurlbert, S. H., & Lombardi, C. M. (2009). Final col-
lapse of the Neyman-Pearson decision theoretic framework
and rise of the neoFisherian. Annales Zoologici Fennici, 46,
311–349.
Jeffreys, H. (1939). Theory of probability. Oxford, England:
Clarendon.
Kaplan, D. (2000). Structural equation modeling: Foundations and
extensions. Thousand Oaks, CA: Sage.
Kenny, D. (1979). Correlation and causation. New York: Wiley.
Kim, J-O., & Mueller, C. W. (1978). Introduction to factor
analysis. Beverly Hills, CA: Sage.
Kuhn, T. S. (1996). The structure of scientific revolutions (3rd
ed.). Chicago, IL: University of Chicago Press (originally
published, 1962).
Lakatos, I. (1970). Falsification and the methodology of scien-
tific research programmes. In I. Lakatos & A. Musgrave, A.
(Eds.). Criticism and the growth of knowledge (pp. 91–195).
Cambridge, England: Cambridge University Press.
Laudan, L. (1981). Science and hypothesis. Dordrecht, The
Netherlands: Reidel.
Lehmann, E. L. (1993). The Fisher, Neyman-Pearson theories
of testing hypotheses: One theory or two? Journal of the
American Statistical Association, 88, 1242–1249.
Lipton, P. (2004). Inference to the best explanation (2nd ed.).
London, England: Routledge.
Manicas, P. T. (1989). Explanation and quantification. In B.
Glassner & J. D. Moreno (Eds.), The qualitative-quantitative
distinction in the social sciences (pp. 179–205). Dordrecht,
The Netherlands: Kluwer.
Manicas, P. T., & Secord, P. F. (1983). Implications for psychol-
ogy of the new philosophy of science. American Psychologist,
38, 399–413.
Markus, K., Hawes, S. S., & Thasites, R. (2008). Abductive infer-
ence to psychological variables: Steiger’s question and best
explanations in psychopathy. Journal of Clinical Psychology,
64, 1069–1088.
Maxwell, G. (1962). The ontological status of theoretical entities.
In H. Feigl & G. Maxwell (Eds.), Minnesota Studies in the
Philosophy of Science, Vol. 3 (pp. 3–28). Minneapolis, MN:
University of Minnesota Press.
McDonald, R. P., & Mulaik, S. A. (1979). Determinacy of com-
mon factors: A nontechnical review. Psychological Bulletin,
86, 297–306.
McGrew, T. (2003). Confirmation, heuristics, and explanatory
reasoning. British Journal for the Philosophy of Science, 54,
553–567.
McGuire, W. J. (1997). Creative hypothesis generating in psy-
chology: Some useful heuristics. Annual Review of Psychology,
48, 1–30.
McMullin, E. (1995). Underdetermination. The Journal of
Medicine and Philosophy, 20, 233–252.
Meehl, P. E. (1967). Theory testing in psychology and physics: A
methodological paradox. Philosophy of Science, 34, 103–115.
Meehl, P. E. (1978). Theoretical risks and tabular asterisks: Sir
Karl, Sir Ronald, and the slow progress of soft psychology.
Journal of Consulting and Clinical Psychology, 46, 806–834.
Meehl, P. E. (1990). Appraising and amending theories: The
strategy of Lakatosian defense and two principles that warrant
using it. Psychological Inquiry, 1, 108–141, 173–180.
Meehl, P. E. (1997). The problem is epistemology, not statis-
tics: Replace significance tests by confidence intervals and
quantify accuracy of risky numerical predictions. In L. L. Har-
low, S. A. Mulaik, & J. H. Steiger (Eds.), What if there were
no significance tests? (pp. 393–425). Mahwah, NJ: Lawrence
Erlbaum.
Michell, J. (2004). The place of qualitative research in psychology.
Qualitative Research in Psychology, 1, 307–319.
Morrison, D. E., & Henkel, R. E. (1970). The significance test
controversy. Chicago: Aldine.
Mulaik, S. A. (1985). Exploratory statistics and empiricism.
Philosophy of Science, 52, 410–430.
Mulaik, S. A. (1987). A brief history of the philosophical foun-
dations of exploratory factor analysis. Multivariate Behavioral
Research, 22, 267–305.
Mulaik, S. A., & McDonald, R. P. (1978). The effect of additional
variables on factor indeterminacy in models with a single
common factor. Psychometrika, 43, 177–192.
Neyman, J., & Pearson, E. S. (1933). On the problem of the
most efficient tests of statistical hypotheses. Philosophical
Transactions of the Royal Society of London A, 231, 289–337.
Nickerson, R. S. (2000). Null hypothesis significance testing:
A review of an old and continuing controversy. Psychological
Methods, 5, 241–301
Nickles, T. (1981). What is a problem that we might solve it?
Synthese, 47, 85–118.
Nickles, T. (1987). Twixt method and madness. In N. J. Nerses-
sian (Ed.), The process of science (pp. 41–67). Dordrecht, The
Netherlands: Martinus Nijhoff.
Pratschke, J. (2003). Realistic models? Critical realism and
statistical models in the social sciences. Philosophica, 71,
13–38.
Proctor, R. W., & Capaldi, E. J. (2001). Empirical evaluation
and justification of methodologies in psychological science.
Psychological Bulletin, 127, 759–772.
Psillos, S. (2004). Inference to the best explanation and Bayesian-
ism. In F. Stadler (Ed.), Induction and deduction in the sciences
(pp. 83–91). Dordrecht, The Netherlands: Kluwer.
Royall, R. M. (1986). The effect of sample size on the meaning
of significance tests. American Statistician, 40, 313–315.
Royall, R. M. (1997). Statistical evidence: A likelihood paradigm.
New York: Chapman & Hall.
Rozeboom, W. W. (1997). Good science is abductive, not
hypothetico-deductive. In L. L. Harlow, S. A. Mulaik, &
J. H. Steiger (Eds.), What if there were no significance tests?
(pp. 335–391.). Hillsdale, NJ: Lawrence Erlbaum.
Salmon, W. C. (1984). Scientific explanation and the causal
structure of the world. Princeton, NJ: Princeton University
Press.
Sayer, A. (1992). Methods in social science: A realist approach (2nd
ed.). London, England: Routledge.
Schmidt, F. L. (1992). What do data really mean? Research find-
ings, meta-analysis, and cumulative knowledge in psychology.
American Psychologist, 47, 1173–1181.
Schmidt, F. L. (1993). Meta-analysis and cumulative knowledge.
Contemporary Psychology, 38, 1163–1165.
Simon, H. (1985). Spurious correlation: A causal interpretation.
In H. M. Blalock (Ed.), Causal models in the social sciences
(2nd ed.) (pp. 7–21). New York: Aldine.
Simon, H. A. (1968). On judging the plausibility of theories. In
B. van Rootselaar & J. F. Staal (Eds.), Logic, methodology, and
philosophy of science, Vol. 3 (pp. 439–459). Amsterdam, The
Netherlands: North-Holland.
Smith, M. L., Glass, G. V., & Miller, T. I. (1980). The benefits
of psychotherapy. Baltimore, MD: Johns Hopkins University
Press.
Sober, E. (1988). The principle of the common cause. In J.
H. Fetzer (Ed.), Probability and causality (pp. 211–229).
Dordrecht, The Netherlands: Reidel.
Sober, E. (2008). Evidence and evolution: The logic behind the
science. Cambridge, England: Cambridge University Press.
Sohn, D. (1996). Meta-analysis and science. Theory and Psychol-
ogy, 6, 229–246.
Spielman, S. (1974). The logic of tests of significance. Philosophy
of Science, 14, 211–225.
Strauss, A. L. (1987). Qualitative analysis for social scientists. New
York: Cambridge University Press.
Thagard, P. (1992). Conceptual revolutions. Princeton, NJ:
Princeton University Press.
Thagard, P. (1999). How scientists explain disease. Princeton, NJ:
Princeton University Press.
Thurstone, L. L. (1947). Multiple factor analysis. Chicago, IL:
University of Chicago Press.
Trout, J. D. (1998). Measuring the intentional world: Realism,
naturalism, and quantitative methods in the behavioral sciences.
New York, NY: Oxford University Press.
Tukey, J. W. (1962). The future of data analysis. The Annals of
Mathematical Statistics, 33, 1–67.
Tukey, J. W. (1969). Analyzing data: Sanctification or detective
work? American Psychologist, 24, 83–91.
Tukey, J. W. (1977). Exploratory data analysis. Reading, MA:
Addison Wesley.
Tukey, J. W. (1980). We need both exploratory and confirmatory.
American Statistician, 34, 23–25.
Waller, N. G. (2004). The fallacy of the null hypothesis in soft
psychology. Applied and Preventive Psychology, 11, 83–86.
Weisberg, J. (2009). Locating IBE in the Bayesian framework.
Synthese, 167, 125–143.
Whitt, L. A. (1992). Indices of theory promise. Philosophy of
Science, 59, 612–634.
Wilkinson, L., & the Task Force on Statistical Inference. (1999).
Statistical methods in psychology journals: Guidelines and
explanations. American Psychologist, 54, 594–604.
Woodward, J. (1989). Data and phenomena. Synthese, 79, 393–
472.
Yu, C-H. (2006). Philosophical foundations of quantitative
research methodology. Lanham, MD: University Press of
America.
haig 31
 CHAPTER

Quantitative Methods and Ethics

3
Ralph L. Rosnow and Robert Rosenthal

Abstract

The purpose of this chapter is to provide a context for thinking about the role of ethics in quantitative
methodology. We begin by reviewing the sweep of events that led to the creation and expansion of
legal and professional rules for the protection of research subjects and society against unethical
research. The risk–benefit approach has served as an instrument of prior control by institutional
review boards. After discussing the nature of that approach, we sketch a model of the costs and
utilities of the “doing” and “not doing” of research. We illustrate some implications of the expanded
model for particular data analytic and reporting practices. We then outline a 5 × 5 matrix of general
ethical standards crossed with general data analytic and reporting standards to encourage thinking
about opportunities to address quantitative methodological problems in ways that may have mutual
ethical and substantive rewards. Finally, we discuss such an opportunity in the context of problems
associated with risk statistics that tend to exaggerate the absolute effects of therapeutic interventions
in randomized trials.
Key Words: Accountability,American Psychological Association (APA) Ethics Code, Belmont Report,
ethical principles, health statistics, institutional review board (IRB), moral dilemmas, Nuernberg
(Nuremberg) Code, quantitative methodology, risk–benefit assessment, statistical illiteracy,
transparency, volunteer bias

Introduction not be seen as mutually exclusive, as it can often be

In this chapter we sketch an historic and heuristic
framework for assessing certain ethical implications
of the term quantitative methods. We use this term
in the broadest sense to include not only statisti-
cal procedures but also what is frequently described
as quantitative research (in contrast to qualitative
research) in psychology and some other disciplines.
As defined in the APA Dictionary of Psychology, the
traditional distinction between these two general
types of research rests on whether “the approach
to science” does (quantitative research) or does
not (qualitative research) “employ the quantification
(expression in numerical form) of the observa-
tions made” (VandenBos, 2007, pp. 762–763). Of
course, quantitative and qualitative methods should
illuminating to use both types in the same research.
For example, in the typical psychological experi-
ment in which the observations take a numerical
form, it may be edifying to ask some of the par-
ticipants in postexperimental interviews to reflect
on the context in which the experiment was con-
ducted and to speculate on the ways in which it
may have influenced their own and other partici-
pants’ behaviors (Orne, 1962, 1969). By the same
token, it is usually possible to quantify nonquantita-
tive observations by, for example, decomposing the
qualitative subject matter element by element and
then numerically and visually analyzing and summa-
rizing the results. Blogs and online discussion groups
are currently a popular source of qualitative subject

32
matter, which researchers have trolled for patterns
or relationships that can be quantified by the use of
simple summary statistics (e.g., Bordia & Rosnow,
1995) or coded and visually mapped out using social
network analysis to highlight links and nodes in
the observed relationships (e.g., Kossinets & Watts,
2006; see also Wasserman & Faust, 1994). Whether
blogs and online discussion groups’ data are treated
quantitatively or qualitatively, their use may raise
ethical questions regarding the invasion of privacy.
The fact that bloggers and participants in online
discussion groups are typically fully aware that their
communications are quite public minimizes the risk
of invasion of privacy.
The term ethics was derived from the Greek ethos,
meaning “character” or “disposition.” We use the
term here to refer to the dos and don’ts of codified
and/or culturally ingrained rules by which morally
“right” and “wrong” conduct can be differentiated.
Conformity to such rules is usually taken to mean
morality, and our human ability to make ethical
judgments is sometimes described as a moral sense
(a tradition that apparently goes back to David
Hume’s A Treatise of Human Nature in the eigh-
teenth century). Philosophers and theologians have
frequently disagreed over the origin of the moral
sense, but on intuitive grounds it would seem that
morality is subject to societal sensitivities, group
values, and social pressures. It is not surprising
that researchers have documented systematic biases
in ethical judgments. For example, in a study by
Kimmel (1991), psychologists were asked to make
ethical judgments about hypothetical research cases.
Kimmel reported that those psychologists who were
more (as compared to less) approving in their eth-
ical judgments were more often men; had held an
advanced degree for a longer period of time; had
received the advanced degree in an area such as
experimental, developmental, or social psychology
rather than counseling, school, or community psy-
chology; and were employed in a research-oriented
context as opposed to a service-oriented context.
Citing this work of Kimmel’s (1991), an Amer-
ican Psychological Association (APA) committee
raised the possibility that inconsistent implemen-
tation of ethical standards by review boards might
result not only from the expanded role of review
boards but also from the composition of particu-
lar boards (Rosnow, Rotheram-Borus, Ceci, Blanck,
& Koocher, 1993). Assuming that morality is also
predicated on people’s abilities to figure out the
meaning of other people’s actions and underlying
intentions, it might be noted that there is also empir-
ical evidence of (1) individual differences in this
ability (described as interpersonal acumen) and (2) a
hierarchy of intention–action combinations ranging
from the least to most cognitively taxing (Rosnow,
Skleder, Jaeger, & Rind, 1994).
Societal sensitivities, group values, and situa-
tional pressures are subject to change in the face of
significant events. On the other hand, some moral
values seem to be relatively enduring and univer-
sal, such as the golden rule, which is frequently
expressed as “Do unto others as you would have
them do unto you.” In the framework of quanti-
tative methods and ethics, a categorical imperative
might be phrased as “Thou shalt not lie with statis-
tics.” Still, Huff, in his book, How to Lie with
Statistics, first cautioned the public in 1954 that the
reporting of statistical data was rife with “bungling
and chicanery” (Huff, 1982, p. 6). The progress
of science depends on the good faith that scien-
tists have in the integrity of one another’s work
and the unbiased communication of findings and
conclusions. Lying with statistics erodes the cred-
ibility of the scientific enterprise, and it can also
present an imminent danger to the general pub-
lic. “Lying with statistics” can refer to a number of
more specific practices: for example, reporting only
the data that agree with the researcher’s bias, omit-
ting any data not supporting the researcher’s bias,
and, most serious of all, fabricating the results of
the research. For example, there was a case reported
in 2009 in which an anesthesiologist fabricated the
statistical data that he had published in 21 journal
articles purporting to give the results of clinical trials
of a pain medicine marketed by the company that
funded much of the doctor’s research (Harris, 2009).
Another case, around the same time, involved a
medical researcher whose accounts of a blood test
for diagnosing prostate cancer had generated con-
siderable excitement in the medical community, but
who was now being sued for scientific fraud by his
industry sponsor (Kaiser, 2009). As the detection of
lying with statistics is often difficult in the normal
course of events, there have been calls for the public
sharing of raw data so that, as one scientist put it,
“Anyone with the skills can conduct their own anal-
yses, draw their own conclusions, and share those
conclusions with others” (Allison, 2009, p. 522).
That would probably help to reduce some of the
problems of biased data analysis, but it would not
help much if the shared data had been fabricated to
begin with.
rosnow, rosenthal 33
 In the following section, we review the sweep of
events that led to the development and growth of
restraints for the protection of human subjects and
society against unethical research.1 A thread running
throughout the discussion is the progression of the
APA’s code of conduct for psychological researchers
who work with human subjects. We assume that
many readers of this Handbook will have had a
primary or consulting background in some area of
psychology or a related research area. The develop-
ment of the APA principles gives us a glimpse of the
specific impact of legal regulations and societal sensi-
tivities in an area in which human research has been
constantly expanding into new contexts, includ-
ing “field settings and biomedical contexts where
research priorities are being integrated with the
priorities and interests of nonresearch institutions,
community leaders, and diverse populations” (Sales
& Folkman, 2000, p. ix). We then depict an ideal-
ized risk–benefit approach that review boards have
used as an instrument of prior control of research,
and we also describe an expanded model focused
on the costs and utilities of “doing” and “not doing”
research. The model can also be understood in terms
of the cost–utility of adopting versus not adopting
particular data analytic and reporting practices. We
then outline a matrix of general ethical standards
crossed with general data analytic and reporting
standards as (1) a reminder of the basic distinction
between ethical and technical mandates and (2) a
framework for thinking about promising opportu-
nities for ethical and substantive rewards in quanti-
tative methodology (cf. Blanck, Bellack, Rosnow,
Rotheram-Borus, & Schooler, 1992; Rosenthal,
1994; Rosnow, 1997). We discuss such an opportu-
nity in the context of the way in which a fixation on
relative risk (RR) in large sample randomized trials
of therapeutic interventions can lead to misconcep-
tions about the practical meaning to patients and
health-care providers of the particular intervention
tested.
The Shaping of Principles to Satisfy Ethical
and Legal Standards
If it can be said that a single historical event in
modern times is perhaps most responsible for ini-
tially galvanizing changes in the moral landscape of
science, then it would be World War II. On Decem-
ber 9, 1946 (the year after the surrender of Germany
on May 8, 1945 and the surrender of Japan on
August 14, 1945), criminal proceedings against Nazi
physicians and administrators who had participated
34 q u a n t i tat i v e m e t h o d s a n d e t h i c s
in war crimes and crimes against humanity were
presented before a military tribunal in Nuernberg,
Germany. For allied atomic scientists, Hiroshima
had been an epiphany that vaporized the old iconic
image of a morally neutral science. For researchers
who work with human participants, the backdrop
to the formation of ethical and legal principles to
protect the rights and welfare of all research partici-
pants were the shocking revelations of the war crimes
documented in meticulous detail at the Nuernberg
Military Tribunal. Beginning with the German inva-
sion of Poland at the outbreak of World War II,
Jews and other ethnic minority inmates of concen-
tration camps had been subjected to sadistic tortures
and other barbarities in “medical experiments” by
Nazi physicians in the name of science. As method-
ically described in the multivolume report of the
trials, “in every one of the experiments the subjects
experienced extreme pain or torture, and in most of
them they suffered permanent injury, mutilation, or
death” (Trials of War Criminals before the Nuernberg
Military Tribunals under Control Council Law No.
10, p. 181). Table 3.1 reprints the principles of the
Nuernberg Code, which have resonated to varying
degrees in all ensuing codes for biomedical research
with human participants as well as having had a gen-
erative influence on the development of principles
for the conduct of behavioral and social research.
We pick up the story again in the 1960s in the
United States, a period punctuated by the shock-
ing assassinations of President John F. Kennedy in
1963 and then of Dr. Martin Luther King, Jr., and
Senator Robert F. Kennedy in 1968. The 1960s
were also the beginning of the end of what Pat-
tullo (1982) called “the hitherto sacrosanct status”
of the human sciences, which moved “into an era
of uncommonly active concern for the rights and
welfare of segments of the population that had tra-
ditionally been neglected or exploited” (p. 375).
One highly publicized case in 1963 involved a noted
cancer researcher who had injected live cancer cells
into elderly, noncancerous patients, “many of whom
were not competent to give free, informed consent”
(Pattullo, p. 375). In 1966, the U.S. Surgeon Gen-
eral issued a set of regulations governing the use
of subjects by researchers whose work was funded
by the National Institutes of Health (NIH). Most
NIH grants funded biomedical research, but there
was also NIH support for research in the behav-
ioral and social sciences. In 1969, following the
exposure of further instances in which the welfare
of subjects had been ignored or endangered in
biomedical research (cf. Beecher, 1966, 1970; Katz,
Table 3.1. The Nuernberg Principles of 1946–1949 for Permissible Medical Experiments∗
1. The voluntary consent of the human subject is absolutely essential.
This means that the person involved should have legal capacity to give consent; should be so situated as to be able to
exercise free power of choice, without the intervention of any element of force, fraud, deceit, duress, over-reaching,
or other ulterior form of constraint or coercion; and should have sufficient knowledge and comprehension of the
elements of the subject matter involved as to enable him to make an understanding and enlightened decision. This
latter element requires that before the acceptance of an affirmative decision by the experimental subject there should
be made known to him the nature, duration, and purpose of the experiment; the method and means by which it is
to be conducted; all inconveniences and hazards reasonably to be expected; and the effects upon his health or person
which may possibly come from his participation in the experiment.
The duty and responsibility for ascertaining the quality of the consent rests upon each individual who initiates,
directs or engages in the experiment. It is a personal duty and responsibility which may not be delegated to another
with impunity.
2. The experiment should be such as to yield fruitful results for the good of society, unprocurable by other methods or
means of study, and not random and unnecessary in nature.
3. The experiment should be so designed and based on the results of animal experimentation and a knowledge of the
natural history of the disease or other problem under study that the anticipated results will justify the performance
of the experiment.
4. The experiment should be so conducted as to avoid all unnecessary physical and mental suffering and injury.
5. No experiment should be conducted where there is an a priori reason to believe that death or disabling injury will
occur; except, perhaps, in those experiments where the experimental physicians also serve as subjects.
6. The degree of risk to be taken should never exceed that determined by the humanitarian importance of the problem
to be solved by the experiment.
7. Proper preparations should be made and adequate facilities provided to protect the experimental subject against even
remote possibilities of injury, disability, or death.
8. The experiment should be conducted only by scientifically qualified persons. The highest degree of skill and care
should be required through all stages of the experiment of those who conduct or engage in the experiment.
9. During the course of the experiment the human subject should be at liberty to bring the experiment to an end if he
has reached the physical or mental state where continuation of the experiment seems to him to be impossible.
10. During the course of the experiment the scientist in charge must be prepared to terminate the experiment at any
stage, if he has probable cause to believe, in the exercise of the good faith, superior skill and careful judgment required
of him that a continuation of the experiment is likely to result in injury, disability, or death to the experimental
subject.
∗ Reprinted from pp. 181–182 in Trials of War Criminals before the Nuernberg Military Tribunals under Control Council Law No. 10, October
1946–April 1949, Vol. II. Washington, DC: U.S. Government Printing Office.

1972), the Surgeon General extended the earlier
safeguards to all human research. In a notorious
case (not made public until 1972), a study con-
ducted by the U.S. Public Health Service (USPHS)
simply followed the course of syphilis in more than
400 low-income African-American men residing in
Tuskegee, Alabama, from 1932 to 1972 (Jones,
1993). Recruited from churches and clinics with the
promise of free medical examinations and free health
care, the men who were subjects in this study were
never informed they had syphilis but only told they
had “bad blood.” They also were not offered peni-
cillin when it was discovered in 1943 and became
widely available in the 1950s, and they were warned
not to seek treatment elsewhere or they would be
dropped from the study. The investigators went so
far as to have local doctors promise not to treat the
men in the study with antibiotics (Stryker, 1997). As
the disease progressed in its predictable course with-
out any treatment, the men experienced damage to

rosnow, rosenthal 35
their skeletal, cardiovascular, and central nervous
systems and, in some cases, death. In 1972, the
appalling details were finally made public by a lawyer
who had been an epidemiologist for the USPHS,
and the study was halted (Fairchild & Bayer, 1999).
The following year, the Senate Health Subcommit-
tee (chaired by Senator Edward Kennedy) aired the
issue of scientific misconduct in public hearings.
The early 1960s was also a period when emotions
about invasions of privacy were running high in the
United States after a rash of reports of domestic wire-
tapping and other clandestine activities by federal
agencies. In the field of psychology, the morality of
the use of deception was being debated. As early
as the 1950s, there had been concerned statements
issued about the use of deception in social psycho-
logical experiments (Vinacke, 1954). The spark that
lit a fuse in the 1960s in the field of psychology
was the publication of Stanley Milgram’s studies on
obedience to authority, in which he had used an
elaborate deception and found that a majority of
ordinary research subjects were willing to adminis-
ter an allegedly dangerous level of shock to another
person when “ordered” to do so by a person in
authority, although no shock was actually admin-
istered (cf. Blass, 2004; Milgram, 1963, 1975).
Toward the end of the 1960s, there were impas-
sioned pleas by leading psychologists for the ethical
codification of practices commonly used in psycho-
logical research (Kelman, 1968; Smith, 1969). As
there were new methodological considerations and
federal regulations since the APA had formulated
a professional code of ethics in 1953, a task force
was appointed to draft a set of ethical principles for
research with human subjects. Table 3.2 shows the
final 10 principles adopted by the APA’s Council of
Representatives in 1972, which were elucidated in
a booklet that was issued the following year, Ethi-
cal Principles in the Conduct of Research with Human
Participants (APA, 1973). An international survey
conducted 1 year later found there were by then two
dozen codes of ethics that had been either adopted
or were under review by professional organizations
of social scientists (Reynold, 1975). Although viola-
tions of such professional codes were supported by
penalties such as loss of membership in the organiza-
tion, the problem was that many researchers engaged
in productive, rewarding careers did not belong to
these professional organizations.
By the end of the 1970s, the pendulum had
swung again, as accountability had become the
watchword of the decade (National Commission on
Research, 1980). In 1974, the guidelines provided
36 q u a n t i tat i v e m e t h o d s a n d e t h i c s
by the Department of Health, Education, and Wel-
fare (DHEW) 3 years earlier were codified as gov-
ernment regulations by the National Research Act of
July 12, 1974 (Pub. L. 93–348). Among the require-
ments instituted by the government regulations was
that institutions that received federal funding estab-
lish an institutional review board (IRB) for the
purpose of making prior assessments of the possi-
ble risks and benefits of proposed research.2 This
federal act also created the National Commission
for the Protection of Human Subjects of Biomed-
ical and Behavioral Research. Following hearings
that were held over a 3-year period, the document
called “The Belmont Report” was issued in April,
1979 (available online and also reprinted in Sales &
Folkman, 2000). Unlike other reports of the Com-
mission, the Belmont Report did not provide a list of
specific recommendations for administrative action
by the DHEW, but the Belmont Report recom-
mended that the report be adopted in its entirety
as a statement of DHEW policy. In the pream-
ble, the report mentioned the standards set by the
Nuernberg (“Nuremberg”) Code as the prototype
of many later codes consisting of rules, some gen-
eral and others specific, to guide researchers and
assure that research involving human participants
would be carried out in an ethical manner. Not-
ing that the rules were often inadequate to cover
complex situations, that they were often difficult to
apply or interpret, and that they often came into
conflict with one another, the National Commis-
sion had decided to issue broad ethical principles
to provide a basis on which specific rules could
then be formulated, criticized, and interpreted. As
we track the development of the APA principles
in this discussion, we will see that there has been
a similar progression, and later we will emphasize
some broad ethical principles when we discuss the
interface of ethical and technical standards in quan-
titative methodology. For now, however, it can be
noted that the Belmont Report proposed that (1)
respect for persons, (2) beneficence, and (3) jus-
tice provide the foundation for research ethics. The
report also proposed norms for scientific conduct
in six major areas: (1) the use of valid research
designs, (2) the competence of researchers, (3) the
identification of risk–benefit consequences, (4) the
selection of research participants, (5) the importance
of obtaining informed voluntary consent, and (6)
compensation for injury.3
In 1982, the earlier APA code was updated, and
a new version of Ethical Principles in the Conduct
of Research with Human Participants was published
Table 3.2. The Ethical Principles Adopted in December, 1972, by the Council of Representatives of
the American Psychological Association∗
The decision to undertake research rests upon a considered judgment by the individual psychologist
about how best to contribute to psychological science and to human welfare. The responsible psychologist
weighs alternative directions in which personal energies and resources might be invested. Having made
the decision to conduct research, psychologists must carry out their investigation with respect for the
people who participate and with concern for their dignity and welfare. The Principles that follow make
explicit the investigator’s ethical responsibilities toward participants over the course of research, from the
initial decision to pursue a study to the steps necessary to protect the confidentiality of research data.
These Principles should be interpreted in terms of the context provided in the complete document offered
as a supplement to these Principles.
1. In planning a study the investigator has the personal responsibility to make a careful evaluation of
its ethical acceptability, taking into account these Principles for research with human beings. To
the extent that this appraisal, weighing of scientific and humane values, suggests a deviation from
any Principle, the investigator incurs an increasingly serious obligation to seek ethical advice and to
observe more stringent safeguards to protect the rights of the human research participants.
2. Responsibility for the establishment and maintenance of acceptable ethical practice in research always
remains with the individual investigator. The investigator is also responsible for the ethical treatment
of research participants by collaborators, assistants, students, and employees, all of whom, however,
incur parallel obligations.
3. Ethical practice requires the investigator to inform the participant of all features of the research that
reasonably might be expected to influence willingness to participate and to explain all other aspects of
the research about which the participant inquires. Failure to make full disclosure gives added emphasis
to the investigator’s responsibility to protect the welfare and dignity of the research participant.
4. Openness and honesty are essential characteristics of the relationship between investigator and research
participant. When the methodological requirements of a study necessitate concealment or deception,
the investigator is required to ensure the participant’s understanding of the reasons for this action
and to restore the quality of the relationship with the investigator.
5. Ethical research practice requires the investigator to respect the individual’s freedom to decline to
participate in research or to discontinue participation at any time. The obligation to protect this
freedom requires special vigilance when the investigator is in a position of power over the participant.
The decision to limit this freedom increases the investigator’s responsibility to protect the participant’s
dignity and welfare.
6. Ethically acceptable research begins with the establishment of a clear and fair agreement between the
investigator and the research participant that clarifies the responsibilities of each. The investigator
has the obligation to honor all promises and commitments included in that agreement.
7. The ethical investigator protects participants from physical and mental discomfort, harm, and danger.
If the risk of such consequences exists, the investigator is required to inform the participant of that
fact, secure consent before proceeding, and take all possible measures to minimize distress. A research
procedure may not be used if it is likely to cause serious and lasting harm to participants.
8. After the data are collected, ethical practice requires the investigator to provide the participant
with a full clarification of the nature of the study and to remove any misconceptions that may have
arisen. Where scientific or human values justify delaying or withholding information, the investigator
acquires a special responsibility to assure that there are no damaging consequences for the participant.

rosnow, rosenthal 37
 Table 3.2. (Continued)
9. Where research procedures may result in undesirable consequences for the participant, the investigator
has the responsibility to detect and remove or correct these consequences, including, where relevant,
long-term aftereffects.
10. Information obtained about the research participants during the course of an investigation is con-
fidential. When the possibility exists that others may obtain access to such information, ethical
research practice requires that this possibility, together with the plans for protecting confidentiality,
be explained to the participants as a part of the procedure for obtaining informed consent.
∗ Quoted from pp. 1–2 in Ethical Principles in the Conduct of Research with Human Participants. Washington, DC: American
Psychological Association. Copyright © 1973 by the American Psychological Association.

by the APA. In the earlier version and in the 1982
version, the principles were based on actual ethi-
cal problems that researchers had experienced, and
extensive discussion throughout the profession was
incorporated in each edition of Ethical Principles.
The principles in the 1982 code are reprinted in
Table 3.3. Notice that there were several new terms
(subject at risk and subject at minimal risk) and also an
addendum sentence to informed consent (referring
to “research with children or with participants who
have impairments that would limit understanding
and/or communication”). The concept of minimal
risk (which came out of the Belmont Report) means
that the likelihood and extent of harm to the partic-
ipants are presumed to be no greater than what may
be typically experienced in everyday life or in routine
physical or psychological examinations (Scott-Jones
& Rosnow, 1998, p. 149). In actuality, the extent of
harm may not be completely anticipated, and esti-
mating the likelihood of harm is frequently difficult
or impossible. Regarding the expanded statement
on deception, the use of deception in research had
been frowned upon for some years although there
had long been instances in which active and passive
deceptions had been used routinely. An example was
the withholding of information (passive deception).
Randomized clinical trials would be considered of
dubious value in medical research had the experi-
menters and the participants not been deprived of
information regarding which condition was assigned
to each participant. On the other hand, in some
areas of behavioral experimentation, the use of
deception has been criticized as having “reached
a ‘taken-for-granted’ status” (Smith, Kimmel, &
Klein, 2009, p. 486).4
Given the precedence of federal (and state) regu-
lations since the guidelines developed by the DHEW
were codified by the National Research Act in 1974
(and revised as of November 6, 1975), researchers
were perhaps likely to take their ethical cues from
the legislated morality and its oversight by IRBs as
opposed to the aspirational principles embodied in
professional codes, such as the APA code. Another
complication in this case is that there was a frac-
tious splintering of the APA in the late-1980s, which
resulted in many members resigning from the APA
and the creation of the rival American Psycholog-
ical Society, subsequently renamed the Association
for Psychological Science (APS). For a time in the
1990s, a joint task force of the APA and the APS
attempted to draft a revised ethics code, but the
APS then withdrew its participation following an
apparently irresolvable disagreement. In 2002, after
a 5-year revision process, APA adopted a reworked
ethics code that emphasized the five general princi-
ples defined (by APA) in Table 3.4 and also “specific
standards” that fleshed out these principles.5 The
tenor of the final document was apparently intended
to reflect the remaining majority constituency of
the APA (practitioners) but also the residual con-
stituency of psychological scientists who perform
either quantitative or qualitative research in fun-
damental and applied contexts. Of the specific
principles with some relevance to data analysis or
quantitative methods, there were broadly stated rec-
ommendations such as sharing the research data
for verification by others (Section 8.14), not mak-
ing deceptive or false statements (Section 8.10),
using valid and reliable instruments (Section 9.02),
drawing on current knowledge for design, standard-
ization, validation, and the reduction or elimina-
tion of bias when constructing any psychometric
instruments (Section 9.05). We turn next to the
risk–benefit process, but we should also note that
ethical values with relevance to statistical practices
are embodied in the codes developed by statistical
organizations (e.g., American Statistical Association,
1999; see also Panter & Sterba, 2011).

38 q u a n t i tat i v e m e t h o d s a n d e t h i c s
Table 3.3. The Revised Ethical Principles Adopted in August, 1982, by the Council of Representatives of the
American Psychological Association for Research with Human Participants∗
The decision to undertake research rests upon a considered judgment by the individual psychologist about how best to
contribute to psychological science and human welfare. Having made the decision to conduct research, the psychologist
considers alternative directions in which research energies and resources might be invested. On the basis of this consid-
eration, the psychologist carries out the investigation with respect and concern for the dignity and welfare of the people
who participate and with cognizance of federal and state regulations and professional standards governing the conduct
of research with human participants.
A. In planning a study, the investigator has the responsibility to make a careful evaluation of its ethical acceptability. To
the extent that the weighing of scientific and human values suggests a compromise of any principle, the investigator
incurs a correspondingly serious obligation to seek ethical advice and to observe stringent safeguards to protect the
rights of human participants.
B. Considering whether a participant in a planned study will be a “subject at risk” or a “subject at minimal risk,” according
to recognized standards, is of primary ethical concern to the investigator.
C. The investigator always retains the responsibility for ensuring ethical practice in research. The researcher is also
responsible for the ethical treatment of research participants by collaborators, assistants, students, and employees, all
of whom, however, incur similar obligations.
D. Except in minimal-risk research, the investigator establishes a clear and fair agreement with research participants, prior
to their participation, that clarifies the obligations and responsibilities of each. The investigator has the obligation
to honor all promises and commitments included in that agreement. The investigator informs the participants of all
aspects of the research that might reasonably be expected to influence willingness to participate and explains all other
aspects of the research about which the participants inquire. Failure to make full disclosure prior to obtaining informed
consent requires additional safeguards to protect the welfare and dignity of the research participants. Research with
children or with participants who have impairments that would limit understanding and/or communication requires
special safeguarding procedures.
E. Methodological requirements of a study may make the use of concealment or deception necessary. Before conducting
such a study, the investigator has a special responsibility to (1) determine whether the use of such techniques is justified
by the study’s prospective scientific, educational, or applied value; (2) determine whether alternative procedures are
available that do not use concealment or deception; and (3) ensure that the participants are provided with sufficient
explanation as soon as possible.
F. The investigator respects the individual’s freedom to decline to participate in or to withdraw from the research at any
time. The obligation to protect this freedom requires careful thought and consideration when the investigator is in
a position of authority or influence over the participant. Such positions of authority include, but are not limited to,
situations in which research participation is required as part of employment or in which the participant is a student,
client, or employee of the investigator.
G. The investigator protects the participant from physical and mental discomfort, harm, and danger that may arise
from research procedures. If risks of such consequences exist, the investigator informs the participant of that fact.
Research procedures likely to cause serious or lasting harm to a participant are not used unless the failure to use
these procedures might expose the participant to risk of greater harm or unless the research has great potential benefit
and fully informed and voluntary consent is obtained from each participant. The participant should be informed
of procedures for contacting the investigator within a reasonable time period following participation should stress,
potential harm, or related questions or concerns arise.
H. After the data are collected, the investigator provides the participant with information about the nature of the study
and attempts to remove any misconceptions that may have arisen. Where scientific or human values justify delaying
or withholding this information, the investigator incurs a special responsibility to monitor the research and to ensure
that there are no damaging consequences for the participant.

rosnow, rosenthal 39
Table 3.3. (Continued)
I. Where research procedures result in undesirable consequences for the individual participant, the investigator has the
responsibility to detect and remove or correct these consequences, including long-term effects.
J. Information obtained about a research participant during the course of an investigation is confidential unless otherwise
agreed upon in advance. When the possibility exists that others may obtain access to such information, this possibility,
together with the plans for protecting confidentiality, is explained to the participant as part of the procedure for
obtaining informed consent.
∗ Quoted from pp. 5–7 in Ethical Principles in the Conduct of Research with Human Participants. Washington, DC: American Psychological Association.

Copyright ©1982 by the American Psychological Association.

Expanding the Calculation of Risks and
Benefits
After the Belmont Report, it seemed that every-
thing changed permanently for scientists engaged
in human subject research, and it made little dif-
ference whether they were engaged in biomedical,
behavioral, or social research. As the philosopher
John E. Atwell (1981) put it, the moral dilemma
was to defend the justification of using human sub-
jects as the means to an end that was beneficial
in some profoundly significant way (e.g., the pro-
gression of science, public health, or public policy)
while protecting the moral “ideals of human dignity,
respect for persons, freedom and self-determination,
and a sense of personal worth” (p. 89). Review
boards were now delegated the responsibility of mak-
ing prior assessments of the future consequences of
proposed research on the basis of the probability
that a certain magnitude of psychological, physi-
cal, legal, social, or economic harm might result,
weighed against the likelihood that “something of
positive value to health or welfare” might result.
Quoting the Belmont Report, “risk is properly con-
trasted to probability of benefits, and benefits are
properly contrasted with harms rather than risks of
harms,” where the “risks and benefits of research
may affect the individual subjects, the families of
the individual subjects, and society at large (or
special groups of subjects in society).” The moral
calculus of benefits to risks was said to be “in a
favorable ratio” when the anticipated risks were out-
weighed by the anticipated benefits to the subjects
(assuming this was applicable) and the anticipated
benefit to society in the form of the advance-
ment of knowledge. Put into practice, however,
researchers and members of review boards found it
difficult to “exorcize the devil from the details” when
challenged by ethical guidelines that frequently con-
flicted with traditional technical criteria (Mark,
Eyssell, & Campbell, 1999, p. 48). As human
beings are not omniscient, there was also the prob-
lem that “neither the risks nor the benefits . . . can
be perfectly known in advance” (Mark et al., 1999,
p. 49).
These complications notwithstanding, another
catch-22 of the risk–benefit assessment is that it
focuses only on the doing of research. Some years ago,
we proposed a way of visualizing this predicament—
first, in terms of an idealized representation of the
risk–benefit assessment and, second, in terms of an
alternative model focused on the costs and benefits
of both the doing and not doing of research (Rosen-
thal & Rosnow, 1984). The latter model also has
implications for the risk–benefit (we prefer the term
cost–utility) of using or not using particular quanti-
tative methods (we return to this idea in a moment).
First, however, Figure 3.1 shows an idealized repre-
sentation of the traditional risk–benefit assessment.
Risk (importance or probability of harm) is plotted
from low (C) to high (A) on the vertical axis, and
the benefit is plotted from low (C) to high (D) on
the horizontal axis. In other words, studies in which
the risk–benefit assessment is close to A would pre-
sumably be less likely to be approved; studies close
to D would be more likely to be approved; and stud-
ies falling along the B–C “diagonal of indecision”
exist in a limbo of uncertainty until relevant infor-
mation nudges the assessment to either side of the
diagonal. The idea of “zero risk” is a methodological
conceit, however, because all human subject research
can be understood as carrying some degree of risk.
The potential risk in the most benign behavioral
and social research, for example, is the “danger of
violating someone’s basic rights, if only the right of
privacy” (Atwell, 1981, p. 89). However, the funda-
mental problem of the traditional model represented
in Figure 3.1 is that it runs the risk of ignoring the
“not doing of research.” Put another way, there are
also moral costs when potentially useful research is
forestalled, or if the design or implementation is

40 q u a n t i tat i v e m e t h o d s a n d e t h i c s
Table 3.4. General Principles Adopted in 2003 by the American Psychological Association∗
General Principles
General Principles, as opposed to Ethical Standards, are aspirational in nature. Their intent is to guide and inspire
psychologists toward the very highest ethical ideals of the profession. General Principles, in contrast to Ethical Standards,
do not represent obligations and should not form the basis for imposing sanctions. Relying upon General Principles for
either of these reasons distorts both their meaning and purpose.
Principle A: Beneficence and Nonmaleficence
Psychologists strive to benefit those with whom they work and take care to do no harm. In their professional actions,
psychologists seek to safeguard the welfare and rights of those with whom they interact professionally and other affected
persons, and the welfare of animal subjects of research. When conflicts occur among psychologists’ obligations or
concerns, they attempt to resolve these conflicts in a responsible fashion that avoids or minimizes harm. Because
psychologists’ scientific and professional judgments and actions may affect the lives of others, they are alert to and
guard against personal, financial, social, organizational, or political factors that might lead to misuse of their influence.
Psychologists strive to be aware of the possible effect of their own physical and mental health on their ability to help
those with whom they work.
Principle B: Fidelity and Responsibility
Psychologists establish relationships of trust with those with whom they work. They are aware of their professional
and scientific responsibilities to society and to the specific communities in which they work. Psychologists uphold
professional standards of conduct, clarify their professional roles and obligations, accept appropriate responsibility for
their behavior, and seek to manage conflicts of interest that could lead to exploitation or harm. Psychologists consult
with, refer to, or cooperate with other professionals and institutions to the extent needed to serve the best interests
of those with whom they work. They are concerned about the ethical compliance of their colleagues’ scientific and
professional conduct. Psychologists strive to contribute a portion of their professional time for little or no compensation
or personal advantage.
Principle C: Integrity
Psychologists seek to promote accuracy, honesty, and truthfulness in the science, teaching, and practice of psychology.
In these activities psychologists do not steal, cheat, or engage in fraud, subterfuge, or intentional misrepresentation of
fact. Psychologists strive to keep their promises and to avoid unwise or unclear commitments. In situations in which
deception may be ethically justifiable to maximize benefits and minimize harm, psychologists have a serious obligation
to consider the need for, the possible consequences of, and their responsibility to correct any resulting mistrust or other
harmful effects that arise from the use of such techniques.
Principle D: Justice
Psychologists recognize that fairness and justice entitle all persons to access to and benefit from the contributions of
psychology and to equal quality in the processes, procedures, and services being conducted by psychologists. Psychol-
ogists exercise reasonable judgment and take precautions to ensure that their potential biases, the boundaries of their
competence, and the limitations of their expertise do not lead to or condone unjust practices.
Principle E: Respect for People’s Rights and Dignity
Psychologists respect the dignity and worth of all people, and the rights of individuals to privacy, confidentiality, and
self-determination. Psychologists are aware that special safeguards may be necessary to protect the rights and welfare
of persons or communities whose vulnerabilities impair autonomous decision making. Psychologists are aware of and
respect cultural, individual, and role differences, including those based on age, gender, gender identity, race, ethnicity,
culture, national origin, religion, sexual orientation, disability, language, and socioeconomic status and consider these
factors when working with members of such groups. Psychologists try to eliminate the effect on their work of biases based
on those factors, and they do not knowingly participate in or condone activities or others based upon such prejudices.
∗ Quoted from the American Psychological Association’s Ethical Principles of Psychologists and Code of Conduct (http://www.apa.org/
ethics/code2002.html). Effective date June 1, 2003, copyrighted in 2002 by the American Psychological Association.

compromised in a way that jeopardizes the integrity
of the research (cf. Haywood, 1976).
Figure 3.2 shows an alternative representing a
cost–utility assessment of both the doing and not
doing of research. In Part A, the decision plane
model on the left corresponds to a cost–utility
appraisal of the “doing of research,” and the model
on the right corresponds to an appraisal of the “not
doing of research.” We use the terms cost and utility
each in a collective sense. That is, the cost of doing
and the cost of not doing a particular research study
include more than only the risk of psychological or

rosnow, rosenthal 41
 A B physical harm; they also include the cost to society,
High funding agencies, and to scientific knowledge when
imagination and new scientifically based solutions
are stifled. As one scientist observed, “Scientists
know that questions are not settled; rather, they are

on
given provisional answers for which it is contingent
Risk of doing

isi
ec
upon the imagination of followers to find more illu-

nd
fi minating solutions” (Baltimore, 1997, p. 8). We
lo
a
on

also use utility in a collective sense, not just in the

iag
D

way that a “tool” can immediately be instrumentally

useful, but in a way that may have no immedi-
ate application and instead “speaks to our sense
Low
of wonder and paves the way for future advances”
Low High (Committee on Science, Engineering, and Public
Benefit of doing
C D Policy, 2009, p. 3). These figurative definitions of
cost and utility aside, Part B of Figure 3.2 suggests
Figure 3.1 Idealized decision-plane model representing the rel-
ative risks and benefits of research submitted to a review board
a way of transforming the three dimensions of Part
for prior approval (after Rosenthal & Rosnow, 1984; Rosnow & A to a two-dimensional model. Suppose an A–D
Rosenthal, 1997). “decision diagonal” for each of the decision planes

DOING RESEARCH NOT DOING RESEARCH

A B A' B'
High DON’T DO High DO
n

n
io

io
cis

cis
de

de
Cost

Cost
in

in
of

of
al

al
on

on
iag

iag
D

Low Low
C DO D C' DON’T DO D'
Low High Low High
Utility Utility
(Decision diagonal A'–D' of right plane in Part A)

D'
DON’T DO
Decision not to do

n
io
ecis
nd
fi
o
al
on
iag
D

Low
DO
A'
A D
Decision to do
(Decision diagonal A–D of left plane in Part A)

Figure 3.2 Decision-planes representing the ethical assessment of the costs and utilities of doing and not doing research (after Rosenthal
& Rosnow, 1984, 2008). (A) Costs and utilities of doing (left plane) and not doing (right plane) research. (B) Composite plane
representing both cases in Part A (above).
42 q u a n t i tat i v e m e t h o d s a n d e t h i c s
in Part A (in contrast to B–C and B’–C’, the diag-
onals of indecision). For any point in the plane of
doing, there would be a location on the cost axis
and on the utility axis, where any point could be
translated to an equivalent position on the decision
diagonal. Thus, if a point were twice as far from
A as from D, the transformed point would then be
located two-thirds of the way on the decision diago-
nal A–D (closer to D than to A). Similar reasoning is
applicable to not doing, with the exception that close-
ness to A would mean “do” rather than “not do.”
Points near D tell us the research should be done,
and points near D’ tell us the research should not be
done.6
Figure 3.2 can also be a way of thinking about
cost–utility dilemmas regarding quantitative meth-
ods and statistical reporting practices. In the 2009
edition of the U.S. National Academy of Sciences
(NAS) guide to responsible conduct in scientific
research, there are several hypothetical scenarios,
including one in which a pair of researchers (a post-
doctoral and a graduate student) discuss how they
should deal with two anomalous data points in a
graph they are preparing to present in a talk (Com-
mittee on Science, Engineering, and Public Policy,
2009). They want to put the best face on their
research, but they fear that discussing the two out-
liers will draw people’s attention away from the bulk
of the data. One option would be to drop the out-
liers, but, as one researcher cautions, this could be
viewed as “manipulating” the data, which is unethi-
cal. The other person comments that if they include
the anomalous points, and if a senior person then
advises them to include the anomalous data in a
paper they are drafting for publication, this could
make it harder to have the paper accepted by a
top journal. That is, the reported results will not
be unequivocal (a potential reason for rejection),
and the paper will also then be too wordy (another
reason to reject it?). In terms of Figure 3.2, not
including the two anomalous data points is anal-
ogous to the “not doing of research.” There are, of
course, additional statistical options, which can also
be framed in cost–utility terms, such as using a suit-
able transformation to pull in the outlying stragglers
and make them part of the group (cf. Rosenthal
& Rosnow, 2008, pp. 310–311). On the other
hand, outliers that are not merely recording errors
or instrument errors can sometimes provide a clue
as to a plausible moderator variable. Suppressing
this information could potentially impede scientific
progress (cf. Committee on Science, Engineering,
and Public Policy, 2009, p. 8).
Unfortunately, there are also cases involving the
suppression of data where the cost is not only that
it impedes progress in the field, but it also under-
mines the authority and trustworthiness of scientific
research and, in some instances, can cause harm
to the broader society, such as when public pol-
icy is based on only partial information or when
there is selective outcome reporting of the efficacy
of clinical interventions in published reports of ran-
domized trials (Turner, Matthews, Linardatos, Tell,
& Rosenthal, 2008; Vedula, Bero, Scherer, & Dick-
ersin, 2009). In an editorial in Science, Cicerone
(2010), then president of the NAS, stated that his
impression—based on information from scattered
public opinion polls and various assessments of lead-
ers in science, business, and government—was that
“public opinion has moved toward the view that
scientists often try to suppress alternative hypotheses
and ideas and that scientists will withhold data and
try to manipulate some aspects of peer review to pre-
vent dissent” (p. 624). Spielmans and Parry (2010)
described a number of instances of “marketing-based
medicine” by pharmaceutical firms. Cases included
the “cherry-picking” of data for publication, the sup-
pression or understatement of negative results, and
the publication (and distribution to doctors) of jour-
nal articles that were not written by the academic
authors who lent their names, titles, and purported
independence to the papers but instead had been
written by ghost writers hired by pharmaceutical and
medical-device firms to promote company products.
Spielmans and Parry displayed a number of screen
shots of company e-mails, which we do not usually
get to see because they go on behind the curtain. In
an editorial in PLoS Medicine (2009) lamenting the
problem of ghost writers and morally dubious prac-
tices in the medical marketing of pharmaceutics, the
editors wrote:
How did we get to the point that falsifying the
medical literature is acceptable? How did an industry
whose products have contributed to astounding
advances in global health over the past several
decades come to accept such practices as the norm?
Whatever the reasons, as the pipeline for new drugs
dries up and companies increasingly scramble for an
ever-diminishing proportion of the market in
“me-too” drugs, the medical publishing and
pharmaceutical industries and the medical
community have become locked into a cycle of
mutual dependency, in which truth and a lack of bias
have come to be seen as optional extras. Medical
journal editors need to decide whether they want to
rosnow, rosenthal 43
 roll over and just join the marketing departments of
pharmaceutical companies. Authors who put their
names to such papers need to consider whether doing
so is more important than having a medical literature
that can be believed in. Politicians need to consider
the harm done by an environment that incites
companies into insane races for profit rather than for
medical need. And companies need to consider
whether the arms race they have started will in the
end benefit anyone. After all, even drug company
employees get sick; do they trust ghost authors?
Ethical Standards and Quantitative
Methodological Standards
We turn now to Table 3.5, which shows a matrix
of general ethical standards crossed with quantitative
methodological standards (after Rosnow & Rosen-
thal, 2011). We do not claim that the row and
column standards are either exhaustive or mutually
exclusive but only that they are broadly represen-
tative of (1) aspirational ideals in the society as a
whole and (2) methodological, data analytic, and
reporting standards in science and technology. The
matrix is a convenient way of reminding ourselves
of the distinction between (1) and (2), and it is also
a way of visualizing a potential clash between (1)
and (2) and, frequently, the opportunity to exploit
this situation in a way that could have rewarding
ethical and scientific implications. Before we turn
specifically to the definitions of the row and col-
umn headings in Table 3.5, we will give a quick
example of what we mean by “rewarding ethical
and scientific implications” in the context of the
recruitment of volunteers. For this example, we
draw on some of our earlier work on specific threats
to validity (collectively described as artifacts) deriv-
ing from the volunteer status of the participants for
research participation. Among our concerns when
we began to study the volunteer was that ethical
sensitivities seemed to be propelling psychological
science into a science of informed volunteers (e.g.,
Rosenthal & Rosnow, 1969; Rosnow & Rosenthal,
1970). It was long suspected that people who vol-
unteered for behavioral and social research might
not be fully adequate models for the study of behav-
ior in general. To the extent that volunteers differ
from nonvolunteers on dimensions of importance,
the use of volunteers could have serious effects on
such estimated parameters as means, medians, pro-
portions, variances, skewness, and kurtosis. The
estimation of parameters such as these is the princi-
pal goal in survey research, whereas in experimental
44 q u a n t i tat i v e m e t h o d s a n d e t h i c s
research the focus is usually on the magnitude of
the difference between the experimental and con-
trol group means. Such differences, we and other
investigators observed, were sometimes affected by
the use of volunteers (Rosenthal & Rosnow, 1975,
2009).
With problems such as these serving as beginning
points for empirical and meta-analytic investiga-
tions, we explored the characteristics that differen-
tiated volunteers and nonvolunteers, the situational
determinants of volunteering, some possible inter-
actions of volunteer status with particular treatment
effects, the implications for predicting the direction
and, sometimes, the magnitude of the biasing effects
in research situations, and we also thought about
the broader ethical implications of these findings
(Rosenthal & Rosnow, 1975; Rosnow & Rosenthal,
1997). For example, in one aspect of our meta-
analytic inquiry, we put the following question to
the research literature: What are the variables that
tend to increase or decrease the rates of volunteering
obtained? Our preliminary answers to this question
may have implications for both the theory and prac-
tice of behavioral science. That is, if we continue
to learn more about the situational determinants of
volunteering, we can learn more about the social
psychology of social influence processes. Method-
ologically, once we learn more about the situational
determinants of volunteering, we should be in a bet-
ter position to reduce the bias in our samples that
derives from the volunteer subjects being systemat-
ically different from nonvolunteers in a variety of
characteristics. For example, one situational corre-
late was that the more important the research was
perceived, the more likely people were to volun-
teer for it. Thus, mentioning the importance of the
research during the recruitment phase might coax
more of the “nonvolunteers” into the sampling pool.
It would be unethical to exaggerate or misrepresent
the importance of the research. By being honest,
transparent, and informative, we are treating people
with respect and also giving them a well-founded
justification for asking them to volunteer their valu-
able time, attention, and cooperation. In sum, the
five column headings of Table 3.5 frequently come
precorrelated in the real world of research, often with
implications for the principles in the row headings
of the table.
Turning more specifically to the row headings in
Table 3.5, rows A, B, C, and E reiterate the three
“basic ethical principles” in the Belmont Report,
which were described there as respect for persons,
beneficence, and justice. Beneficence (the ethical
 Table 3.5. General Ethical Standards Crossed with Quantitative Methodology Standards (after Rosnow &
Rosenthal, 2011)
Quantitative methodological standards
Ethical standards 1. Transparency 2. Informativeness
A. Beneficence
B. Nonmaleficence
C. Justice
D. Integrity
E. Respect
ideal of “doing good”) was conflated with the princi-
ple (b) of nonmaleficence (“not doing harm”), and the
two were also portrayed as obligations assimilating
two complementary responsibilities: (1) do not
harm and (2) maximize possible benefits and mini-
mize possible harms. Next in Table 3.5 is justice, by
which we mean a sense of “fairness in distribution”
or “what is observed” (quoting from the Belmont
Report). As the Belmont Report went on to explain:
“Injustice occurs when some benefit to which a per-
son is entitled is denied without good reason or
when some burden is imposed unduly.” Conced-
ing that “what is equal?” and “what is unequal?”
are often complex, highly nuanced questions in a
specific research situation (just as they are when
questions of justice are associated with social prac-
tices, such as punishment, taxation, and political
representation), justice was nonetheless considered a
basic moral precept relevant to the ethics of research
involving human subjects. Next in Table 3.5 is
integrity, an ethical standard that was not distinctly
differentiated in the Belmont Report but that was
discussed in detail in the NAS guide (Committee
on Science, Engineering, and Public Policy, 2009).
Integrity implies honesty and truthfulness; it also
implies a prudent use of research funding and other
resources and, of course, the disclosure of any con-
flicts of interest, financial or otherwise, so as not
to betray public trust. Finally, respect was described
in the Belmont Report as assimilating two obliga-
tions: “first, that individuals should be treated as
autonomous agents, and second, that persons with
diminished autonomy are entitled to protection.”
In the current APA code, respect is equated with
civil liberties—that is, privacy, confidentiality, and
self-determination.
Inspecting the column headings in Table 3.5, first
by transparency, we mean here that the quantitative
3. Precision 4. Accuracy 5. Groundedness
results are presented in an open, frank, and can-
did way, that any technical language used is clear
and appropriate, and that visual displays do not
obfuscate the data but instead are as crystal clear as
possible. Elements of graphic design are explained
and illustrated in a number of very useful books
and articles, particularly the work of Tufte (1983,
1990, 2006) and Wainer (1984, 1996, 2000, 2009;
Wainer & Thissen, 1981), and there is a burgeon-
ing literature in every area of science on the visual
display of quantitative data. Second, by informa-
tiveness, we mean that there is enough information
reported to enable readers to make up their own
minds on the basis of the primary results and enough
to enable others to re-analyze the summary results
for themselves. The development of meta-analysis,
with emphasis on effect sizes and moderator vari-
ables, has stimulated ways of recreating summary
data sets and vital effect size information, often from
minimal raw ingredients. Third, the term precision
is used not in a statistical sense (the likely spread of
estimates of a parameter) but rather in a more gen-
eral sense to mean that quantitative results should
be reported to the degree of exactitude required by
the given situation. For example, reporting the aver-
age scores on an attitude questionnaire to a high
degree of decimal places is psychologically mean-
ingless (false precision), and reporting the weight
of mouse subjects to six decimal places is pointless
(needless precision). Fourth, accuracy means that a
conscientious effort is made to identify and cor-
rect mistakes in measurements, calculations, and
the reporting of numbers. Accuracy also means not
exaggerating results by, for example, making false
claims that applications of the results are unlikely to
achieve. Fifth, groundedness implies that the method
of choice is appropriate to the question of interest, as
opposed to using whatever is fashionable or having
rosnow, rosenthal 45
a computer program repackage the data in a one-
size-fits-all conceptual framework. The methods we
choose must be justifiable on more than just the
grounds that they are what we were taught in grad-
uate school, or that “this is what everyone else does”
(cf. Cohen, 1990, 1994; Rosnow & Rosenthal,
1995, 1996; Zuckerman, Hodgins, Zuckerman, &
Rosenthal, 1993).
Clinical Significance and the Consequences
of Statistical Illiteracy
To bring this discussion of quantitative meth-
ods and ethics full circle, we turn finally to a
problem that has been variously described as innu-
meracy (Paulos, 1990) and statistical illiteracy. The
terms are used to connote a lack of knowledge or
understanding of the meaning of numbers, statisti-
cal concepts, or the numeric expression of summary
statistics. As the authors of a popular book, The
Numbers Game, put it: “Numbers now saturate the
news, politics, life. . . . For good or for evil, they
are today’s preeminent public language—and those
who speak it rule” (Blastland & Dilnot, 2009, p. x).
To be sure, even people who are most literate in the
language of numbers are prone to wishful thinking
and fearful thinking and, therefore, sometimes sus-
ceptible to those who use numbers and gimmicks
to sway, influence, or even trick people. The math-
ematician who coined the term innumeracy told of
how his vulnerability to whim “entrained a series of
ill-fated investment decisions,” which he still found
“excruciating to recall” (Paulos, 2003, p. 1). The
launching point for the remainder of our discussion
was an editorial in a medical journal several years
ago, in which the writers of the editorial lamented
“the premature dissemination of research and the
exaggeration of medical research findings” (Schwartz
& Woloshin, 2003, p. 153). A large part of the prob-
lem is an emphasis on RR statistics that hook general
readers into making unwarranted assumptions, a
problem that may often begin with researchers,
funders, and journals that “court media atten-
tion through press releases” (Woloshin, Schwartz,
Casella, Kennedy, & Larson, 2009, p. 613). Con-
fusion about risk and risk statistics is not limited to
the general public (cf. Prasad, Jaeschke, Wyer, Keitz,
& Guyatt, 2008), but it is the susceptible public
(Carling, Kristoffersen, Herrin, Treweek, Oxman,
Schünemann, Akl, & Montori, 2008) that must
ultimately pay the price of the accelerating costs
of that confusion. Stirring the concept of statistical
significance into this mix can frequently produce a
46 q u a n t i tat i v e m e t h o d s a n d e t h i c s
truly astonishing amount of confusion. For example,
writing in the Journal of the National Cancer Insti-
tute, Miller (2007) mentioned that many doctors
equate the level of statistical significance of cancer
data with the “degree of improvement a new treat-
ment must make for it to be clinically meaningful”
(p. 1832).7
In the space remaining, we concentrate on mis-
conceptions and illusions regarding the concepts of
RR and statistical significance when the clinical sig-
nificance of interventions is appraised through the
lens of these concepts in randomized clinical trials
(RCTs). As a case in point, a highly cited report
on the management of depression, a report that was
issued by the National Institute for Health and Clin-
ical Excellence (NICE), used RR of 0.80 or less as a
threshold indicator of clinical significance in RCTs
with dichotomous outcomes and statistically signif-
icant results.8 We use the term clinical significance
here in the way that it was defined in an authorita-
tive medical glossary, although we recognize that it is
a hypothetical construct laden with surplus meaning
as well (cf. Jacobson & Truax, 1991). In the glos-
sary, clinical significance was taken to mean that “an
intervention has an effect that is of practical mean-
ing to patients and health care providers” (NICHSR,
2010; cf. Jeans, 1992; Kazdin, 1977, 2008). By
intervention, we mean a treatment or involvement
such as a vaccine used in a public health immu-
nization program to try to eradicate a preventable
disease (e.g., the Salk poliomyelitis vaccine), or a
drug that can be prescribed for a patient in the doc-
tor’s office, or an over-the-counter medicine (e.g.,
aspirin) used to reduce pain or lessen the risk of an
adverse event (e.g., heart attack), or a medication
and/or psychotherapy to treat depression. By tradi-
tion, RCTs are the gold standard in evidence-based
medicine when the goal is to appraise the clinical
significance of interventions in a carefully controlled
scientific manner. Claims contradicted by RCTs are
not always immediately rejected in evidence-based
medicine, as it has been noted that some “claims
from highly cited observational studies persist and
continue to be supported in the medical literature
despite strong contradictory evidence from random-
ized trials” (Tatsioni, Bonitsis, & Ioannidis, 2007).
Of course, just as gold can fluctuate in value, so
can conclusions based on the belief that statistical
significance is a proxy for clinical significance, or
when it is believed that given statistical significance,
clinical significance is achieved only if the reduc-
tion in RR reaches some arbitrary fixed magnitude
(recall, for example, NICE, 2004). The challenge
 Table 3.6. Final results for myocardial infarction (MI) and hemorrhagic stroke (HS) for the
aspirin (325 mg every other day) component of the Physicians’ Health Study (Steering
Committee of the Physicians’ Health Study Research Group, 1989). The increase in relative
risk (RRI) for HS was more than twice the reduction in relative risk (RRR) for MI. Having one
more case of HS in the aspirin group would have yielded a chi-square significant at p < 0.05,
RR = 2.0, and RRI = 100%. In the combined samples, the event rate of MI (378/22,071 =
0.0171, or 1.71% ) exceeded the event rate of HS (35/22,071 = 0.0016, or 0.16% ) by a ratio
of about 10:1, and a difference of 1.71% – 0.16% = 1.55%. In the subtable on the right, RRI is
the relative risk increase, computed as RRR (see Table 3.7), but indicated as RRI when the
treatment increases the risk of the adverse outcome.
Myocardial infarction (Heart attack) Hemorrhagic stroke

MI No MI Total HS No HS Total

Aspirin 139 10,898 11,037 Aspirin 23 11,014 11,037

Placebo 239 10,795 11,034 Placebo 12 11,022 11,034

Total 378 21,693 22,071 Total 35 22,036 22,071

Chi-square 26.9, p = 2.1 × 10−7 Chi-square 3.46, p = 0.063

RR 0.58 RR 1.92

RRR 42% RRI 92%

r(phi) 0.035 r(phi) –0.013

is to reverse the accelerating cost curve of statistical
illiteracy in an area that affects us all (see, for example,
Gigerenzer, Gaissmaier, Kurz-Milcke, Schwartz, &
Woloshin, 2008).
Table 3.6 helps us illustrate the folly of a delicate
balancing act that is sometimes required between
statistical significance and RR. The table shows a
portion of the results from the aspirin component
of a highly cited double-blind, placebo-controlled,
randomized trial to test whether 325 milligrams of
aspirin every other day reduces the mortality from
cardiovascular disease and whether beta-carotene
decreases the incidence of cancer (Steering Com-
mittee of the Physicians’ Health Study Research
Group, 1989). The aspirin component of the study
was terminated earlier than planned on finding “a
statistically significant, 44 [sic] percent reduction in
the risk of myocardial infarction for both fatal and
nonfatal events . . . [although] there continued to
be an apparent but not significantly increased risk
of stroke” (p. 132). RR (for relative risk) refers to
the ratio of the incidence rate of the adverse event
(the illness) in the treated sample to the control sam-
ple; RRR is the relative risk reduction; and RRI, is
the relative risk increase (the computation of these
indices is described in Table 3.7). When tables of
independent counts are set up as shown in Tables
3.6 and 3.7, an RR less than 1.0 indicates that the
treated sample fared better than the control sample
(thereby implying RRR), and an RR greater than 1.0
indicates the treated sample did more poorly than
the control (thereby implying RRI). Observe that
the “slightly increased risk of stroke” (RRI = 92% )
was actually more than twice the reduction in risk of
heart attack (RRR = 42% )! Suppose the study had
continued, and one more case of stroke had turned
up in the aspirin group. The p-value would have
reached the 0.05 level, and the researchers might
have arrived at a different conclusion, possibly that
the benefit with respect to heart attack was more
than offset by the increased risk in stroke. Appar-
ently, a p-value only a hair’s-breadth greater than
0.05 can trump a RR increase of 92% . On the
other hand, the event rate of stroke in the study as
a whole was only 0.16% , less than one-tenth the
magnitude of the event rate of 1.7% of heart attack
in the study as a whole.9 However, we would never
know this from the RR alone.
The fact is that RR statements are oblivious to
event rates in the total N . To give a quick exam-
ple, suppose in a study with 100 people each in the
treated and control samples that 1 treated person
and 5 untreated people (controls) became ill. RR
and RRR would be 0.20 and 80% , respectively.
Stating there was an 80% reduction in risk of the
adverse event conveys hope. However, suppose we
increase each sample size to 1,000 but still assume
1 case of illness in the treated sample and 5 cases

rosnow, rosenthal 47
 Table 3.7. Six studies each with total sample size (N ) of 2,000 and 1% event rates in Studies 1 and 4 (20
cases out of 2,000), 25% event rates in Studies 2 and 5 (500 cases out of 2,000), and 50% event rates in
Studies 3 and 6 (1,000 cases out of 2,000). RR, the relative risk or risk ratio, indicates the ratio of the
incidence rate of level of risk in the treated group to the level of risk in the control group. With cells labeled
A, B, C, D from upper left (A) to upper right (B), to lower left (C), to lower right (D), RR =
[A/(A+B)]/[C/(C+D)], where RR < 1.0 favors the treatment effect (risk reduction) and RR > 1.0 favors the
control effect (risk increase). OR, the odds ratio, also called relative odds or the cross-product ratio, is the
ratio of A/B to C/D, or the cross-product AD/BC. RRR, the relative risk reduction, is the reduction in risk
of the adverse outcome (e.g., illness) in the treated sample relative to the control, which is indicated as a
percentage by dividing RD (defined next) by [C/(C+D)] and then multiplying by 100. RD, the risk
difference, also called the absolute risk reduction (ARR), is the reduction in risk of the particular adverse
outcome (e.g., cancer, heart attack, stroke) in the treated group compared with the level of baseline risk in
the control—that is, [A/(A+B)]–[C/(C+D)]. Multiplying RD (or ARR) times 10,000 estimates the number
of people in a group of 10,000 that are predicted to benefit from the treatment. NNT = 1/RD = 1/ARR, is
the number needed to treat to prevent a single case of the particular adverse outcome.
Study 1 (N = 2,000) Study 2 (N = 2,000) Study 3 (N = 2,000)
Adverse outcome (1% ) Adverse outcome (25% ) Adverse outcome (50% )

Condition Yes No Yes No Yes No

Treatment 1 999 25 975 50 950

Control 19 981 475 525 950 50

Chi-square 16.4 (p < 0.0001) 540.0 (p < 0.0001) 1,620.0 (p < 0.0001)

RR 0.05 0.05 0.05

OR 0.05 0.03 0.003

RRR 94.7% 94.7% 94.7%

r(phi) 0.090 0.52 0.90

RD=ARR 0.018 0.45 0.90

NNT=1/ARR=1/RD 55.6 2.2 1.1

ARR(10,000) 180 4,500 9,000

Study 4 (N = 2,000) Study 5 (N = 2,000) Study 6 (N = 2,000)

Adverse outcome (1% ) Adverse outcome (25% ) Adverse outcome (50% )

Condition Yes No Yes No Yes No

Treatment 9 991 225 775 450 550

Control 11 989 275 725 550 450

Chi-square 0.2 (p = 0.89) 6.7 (p = 0.01) 20.0 (p < 0.0001

RR 0.82 0.82 0.82

OR 0.82 0.77 0.67

RRR 18.2% 18.2% 18.2%

r(phi) 0.01 0.06 0.10

RD=ARR 0.002 0.05 0.10

NNT=1/ARR=1/RD 500 20 10

ARR(10,000) 20 500 1,000

48 q u a n t i tat i v e m e t h o d s a n d e t h i c s
 Study 1 Study 2 Study 3 RR RRR
1000

800

Number of 600
Patients
400
0.05 95%
200

0
Study 4 Study 5 Study 6
1000

800

Number of 600
Patients
400
0.82 18%
200

0
Treatment

Treatment

Treatment
Control

Control

Control
Group
Assignment

Event
1% 25% 50%
Rate

Figure 3.3 Histograms based on the six studies in Table 3.7, in which the total sample size (N ) was 2,000 in each study. Darkened
areas of the bars indicate the number of adverse outcomes (event rates), which increased from 1% (20 cases out of 2,000) in Studies 1
and 4, to 25% (500 cases out of 2,000) in Studies 2 and 5, to 50% (1,000 cases out of 2,000) in Studies 3 and 6. However, the relative
risk (RR) and relative risk reduction (RRR) were insensitive to these vastly different event rates. In Studies 1, 2, and 3, the RR and
RRR remained constant at 0.05 and 94.7%, respectively, whereas in Studies 4, 5, and 6, the RR and RRR remained constant at 0.82
and 18.2%, respectively.

of illness in the control sample. We would still find
RR = 0.20 and RRR = 80% . It makes no differ-
ence how large we make the sample sizes, as RR and
RRR will not budge from 0.20 and 80% so long as
we assume 1 case of illness in the treated sample and
5 cases of illness in the control sample. Suppose we
now hold the N constant and see what happens to
the RR and RRR when the event rate in the over-
all N changes from one study to another. In Figure
3.3, we see the results of six hypothetical studies in
which the event rates increased from 1% in Stud-
ies 1 and 4, to 25% in Studies 2 and 5, to 50% in
Studies 3 and 6. Nonetheless, in Studies 1, 2, and 3,
RR remained constant at 0.05 and RRR remained
constant at an attention-getting 95% . In Studies 4,
5, and 6, RR and RRR stayed constant at 0.82 and
18% , respectively.
Further details of the studies in Figure 3.3 are
given in Table 3.7. The odds ratio (OR), for the
ratio of two odds, was for a time widely promoted
as a measure of association in 2 × 2 tables of
counts (Edwards, 1963; Mosteller, 1968) and is
still frequently reported in epidemiological studies
(Morris & Gardner, 2000). As Table 3.7 shows, OR
and RR are usually highly correlated. The absolute
risk reduction (ARR), also called the risk differ-
ence (RD), refers to the absolute reduction in risk
of the adverse event (illness) in the treated patients
compared with the level of baseline risk in the con-
trol group. Gigerenzer et al. (2008) recommended
using the absolute risk reduction (RD) rather than
the RR. As Table 3.7 shows, RD (or ARR) is sen-
sitive to the differences in the event rates. There
are other advantages as well to RD, which are dis-
cussed elsewhere (Rosenthal & Rosnow, 2008, pp.
631–632). Phi is the product-moment correlation
(r) when the two correlated variables are dichoto-
mous, and Table 3.7 shows it is sensitive to the
event rates and natural frequencies. Another useful
index is NNT, for the number of patients that
need to be treated to prevent a single case of the
adverse event. Relative risk may be an easy-to-handle
description, but it is only an alerting indicator that
tells us that something happened and we need to
explore the data further. As Tukey (1977), the con-
summate exploratory data analyst, stated: “Anything

rosnow, rosenthal 49
that makes a simpler description possible makes the
description more easily handleable; anything that
looks below the previously described surface makes
the description more effective” (p. v). And, we can
add, that any index of the magnitude of effect that
is clear enough, transparent enough, and accurate
enough to inform the nonspecialist of exactly what
we have learned from the quantitative data increases
the ethical value of those data (Rosnow & Rosenthal,
2011).
Conclusion
In a cultural sphere in which so many things com-
pete for our attention, it is not surprising that people
seem to gravitate to quick, parsimonious forms of
communication and, in the case of health statis-
tics, to numbers that appear to speak directly to
us. For doctors with little spare time to do more
than browse abstracts of clinical trials or the sum-
maries of summaries, the emphasis on parsimonious
summary statistics such as RR communications in
large sample RCTs may seem heavily freighted with
clinical meaning. For the general public, reading
about a 94.7% reduction in the risk of some ill-
ness, either in a pharmaceutical advertisement or in a
news story about a “miracle drug that does wonders,”
is attention-riveting. It is the kind of information
that is especially likely to arouse an inner urgency
in patients but also in anyone who is anxious and
uncertain about their health. Insofar as such infor-
mation exaggerates the absolute effects, it is not only
the patient or the public that will suffer the conse-
quences; the practice of medicine and the progress
of science will as well. As Gigerenzer et al. (2008)
wrote, “Statistical literacy is a necessary precondi-
tion for an educated citizenship in a technological
democracy” (p. 53). There are promising opportuni-
ties for moral (and societal) rewards for quantitative
methodologists who can help us educate our way
out of statistical illiteracy. And that education will be
beneficial, not only to the public but to many behav-
ioral, social, and medical researchers as well. As that
education takes place, there will be increased clar-
ity, transparency, and accuracy of the quantitative
methods employed, thereby increasing their ethical
value.
Future Directions
An important theoretical and practical question
remains to be addressed: To what extent is there
agreement among quantitative methodologists in
their evaluation of quantitative procedures as to the
50 q u a n t i tat i v e m e t h o d s a n d e t h i c s
degree to which each procedure in a particular study
meets the methodological standards of transparency,
informativeness, precision, accuracy, and ground-
edness? The research program called for to address
these psychometric questions of reliability will surely
find that specific research contexts, specific disci-
plinary affiliations, and other specific individual dif-
ferences (e.g., years of experience) will be moderators
of the magnitudes of agreement (i.e., reliabilities)
achieved. We believe that the results of such research
will demonstrate that there will be some disagree-
ment (that is, some unreliability) in quantitative
methodologists’ evaluations of various standards of
practice. And, as we noted above, that is likely to
be associated with some disagreement (that is, some
unreliability) in their evaluations of the ethical value
of various quantitative procedures.
Another important question would be addressed
by research asking the degree to which the spe-
cific goals and specific sponsors of the research
may serve as causal factors in researchers’ choices of
quantitative procedures. Teams of researchers (e.g.,
graduate students in academic departments rou-
tinely employing quantitative procedures in their
research) could be assigned at random to analyze
the data of different types of sponsors with differ-
ent types of goals. It would be instructive to learn
that choice of quantitative procedure was predictable
from knowing who was paying for the research and
what results the sponsors were hoping for. Recogni-
tion of the possibility that the choice of quantitative
procedures used might be affected by the finan-
cial interests of the investigator is reflected in the
increased frequency with which scientific journals
(e.g., medical journals) require a statement from
all co-authors of their financial interest in the com-
pany sponsoring the research (e.g., pharmaceutical
companies).
Finally, it would be valuable to quantify the costs
and utilities of doing and not doing a wide variety of
specific studies, including classic and not-so-classic
studies already conducted, and a variety of studies
not yet conducted. Over time, there may develop
a disciplinary consensus over the costs and util-
ities of a wide array of experimental procedures.
And, although such a consensus is building over
time, it will be of considerable interest to psychol-
ogists and sociologists of science to study disci-
plinary differences in such consensus-building. Part
of such a program of self-study of disciplines doing
quantitative research would focus on the quantita-
tive procedures used, but the primary goal would
be to apply survey research methods to establish
the degree of consensus on research ethics of the
behavioral, social, educational, and biomedical sci-
ences. The final product of such a program of
research would include the costs and utilities of
doing, and of not doing, a wide variety of research
studies.
Notes
1. Where we quote from a document but do not give the page
numbers of the quoted material, it is because either there was no
pagination or there was no consistent pagination in the online and
hard copy versions that we consulted. Tables 3.1–3.4 reprint only
the original material, as there were slight discrepancies between
original material and online versions.
2. Pattullo (1982) described the logical basis on which “rule-
makers” (like DHEW) had proceeded in terms of a syllogism
emphasizing not the potential benefits of research but only the
avoidance of risks of harm: “(a) Research can harm subjects; (2)
Only impartial outsiders can judge the risk of harm; (3) There-
fore, all research must be approved by an impartial outside group”
(p. 376).
3. Hearings on the recommendations in the Belmont Report
were conducted by the President’s Commission for the Study
of Ethical Problems in Medicine, Biomedical, and Behavioral
Research. Proceeding on the basis of the information pro-
vided at these hearings and on other sources of advice, the
Department of Health and Human Services (DHHS) then
issued a set of regulations in the January 26, 1981, issue of
the Federal Register. A compendium of regulations and guide-
lines that now govern the implementation of the National
Research Act and subsequent amendments can be found in
the DHHS manual known as the “Gray Booklet,” specifically
titled Guidelines for the Conduct of Research Involving Human
Subjects at the National Institutes of Health (available online at
http://ohsr.od.nih.gov/guidelines/index.html).
4. Smith, Kimmel, and Klein (2009) reported that 43.4%
of the articles on consumer research in leading journals in the
field in 1975 through 1976 described some form of deception in
the research. By 1989 through 1990, the number of such articles
increased to 57.7% , where it remained steady at 56% in 1996
through 1997, increased to 65.7% in 2001 through 2002, and
jumped to 80.4% in 2006 through 2007. The issue of deception
is further complicated by the fact that active and passive decep-
tions are far from rare in our society. Trial lawyers manipulate
the truth in court on behalf of their clients; prosecutors surrep-
titiously record private conversations; journalists get away with
using hidden cameras and undercover practices to obtain stories;
and the police use sting operations and entrapment procedures to
gather incriminating evidence (cf. Bok, 1978, 1984; Saxe, 1991;
Starobin, 1997).
5. The document, titled “Ethical Principles of Psycholo-
gists and Code of Conduct,” is available online at http://www.
apa.org/ETHICS/code2002.html.
6. Adaptations of the models in Figures 3.1 and 3.2 have been
used to cue students about possible ethical dilemmas in research
and data analysis (cf. Bragger & Freeman, 1999; Rosnow, 1990;
Strohmetz & Skleder, 1992).
7. The confusion of statistical significance with practical
importance may be a more far-reaching problem in science. In
a letter in Science, the writers noted that “almost all reviews
and much of the original research [about organic foods] report
only the statistical significance of the differences in nutrient
levels—not whether they are nutritionally important” (Clancy,
Hamm, Levine, & Wilkins, 2009, p. 676).
8. NICE (2004) also recommended that researchers use a
standardized mean difference (SMD) of half a standard devia-
tion or more (i.e., d or g ≥ 0.5) with continuous outcomes as
the threshold of clinical significance for initial assessments of sta-
tistically significant summary statistics (NICE, 2004). However,
effects far below the 0.5 threshold for SMDs have been associated
with important interventions. For example, in the classic Salk
vaccine trial (Brownlee, 1955; Francis, Korns, Voight, Boisen,
Hemphill, Napier, & Tolchinsky, 1955), phi = 0.011, which
has a d -equivalent of 0.022 (Rosnow & Rosenthal, 2008). It is
probably the case across the many domains in which clinical sig-
nificance is studied that larger values of d or g are in fact generally
associated with greater intervention benefit, efficacy, or clinical
importance. But it is also possible for large SMDs to have little
or no clinical significance. Suppose a medication was tested on
100 pairs of identical twins with fever, and in each and every
pair, the treated twin loses exactly one-tenth of 1 degree more
than the control twin. The SMD will be infinite, inasmuch as
the variability (the denominator of d or g ) will be 0, but few
doctors would consider this ES clinically significant. As Cohen
(1988) wisely cautioned, “the meaning of any given ES is, in the
final analysis, a function of the context in which it is embedded”
(p. 535).
9. The high RR of HS in this study, in which participants
(male physicians) took 325 milligrams every other day, might
explain in part why the current dose for MI prophylaxis is
tempered at only 81 milligrams per day.
Author Note
Ralph L. Rosnow is Thaddeus Bolton Professor
Emeritus at Temple University, Philadelphia, PA
(rosnow@temple.edu). Robert Rosenthal is a Dis-
tinguished Professor at the University of California
at Riverside (robert.rosenthal@ucr.edu) and Edgar
Pierce Professor of Psychology, Emeritus, Harvard
University.
References
Allison, D. B. (2009). The antidote to bias in research. Science,
326, 522.
American Psychological Association. (1973). Ethical principles in
the conduct of research with human participants. Washington,
DC: Author.
American Psychological Association. (1982). Ethical principles in
the conduct of research with human participants. Washington,
DC: Author.
American Statistical Association (1999). Ethical guidelines for
statistical practice. http://www.amstat.org/profession/index.
cfm?fusaction=ethical
Atwell, J. E. (1981). Human rights in human subjects research.
In A. J. Kimmel (Ed.), New directions for methodology of social
and behavioral science: Ethics of human subject research (No. 10,
pp. 81–90). San Francisco, CA: Jossey-Bass.
Baltimore, D. (January 27, 1997). Philosophical differences. The
New Yorker, p. 8.
rosnow, rosenthal 51
Beecher, H. K. (July 2, 1966). Documenting the abuses. Saturday
Review, 45–46.
Beecher, H. K. (1970). Research and the individual. Boston: Little
Brown.
Blanck, P. D., Bellack, A. S., Rosnow, R. L., Rotheram-Borus,
M. J., & Schooler, N. R. (1992). Scientific rewards and con-
flicts of ethical choices in human subjects research. American
Psychologist, 47, 959–965.
Blass, T. (2004). The man who shocked the world: The life and
legacy of Stanley Milgram. New York: Basic Books.
Blastland, M., & Dilnot, A. (2009). The numbers game. New
York: Gotham Books.
Bok, S. (1978). Lying: Moral choice in public and private life. New
York: Pantheon.
Bok, S. (1984). Secrets: On the ethics of concealment and revelation.
New York: Vintage Books.
Bordia, P., & Rosnow, R. L. (1995). Rumor rest stops on
the information highway: A naturalistic study of transmis-
sion patterns in a computer-mediated rumor chain. Human
Communication Research, 25, 163–179.
Bragger, J. D., & Freeman, M. A. (1999). Using a cost–benefit
analysis to teach ethics and statistics. Teaching of Psychology,
26, 34–36.
Brownlee, K. A. (1955). Statistics of the 1954 polio vaccine trials.
Journal of the American Statistical Association, 50, 1005–1013.
Carling, C., Kristoffersen, D. T., Herrin, J., Treweek, S., Oxman,
A. D., Schünemann, H., et al. (2008). How should the
impact of different presentations of treatment effects on
patient choice be evaluated? PLoS One, 3(11), e3693.
Cicerone, R. J. (2010). Ensuring integrity in science. Science,
327, 624.
Clancy, K., Hamm, M., Levine, A. S., & Wilkins, J. (2009),
Organics: Evidence of health benefits lacking. Science, 325,
676.
Cohen, J. (1988). Statistical power analysis for the behavioral
sciences (2nd ed.). Hillsdale, NJ: Erlbaum.
Cohen, J. (1990). Things I have learned (so far). American
Psychologist, 45, 1304–1312.
Cohen, J. (1994). The earth is round (p < .05). American
Psychologist, 49, 997–1003.
Committee on Science, Engineering, and Public Policy (2009).
On being a scientist: A guide to responsible conduct in research
(3rd ed.). Washington, DC: National Academies Press.
Edwards, A. W. F. (1963). The measure of association in
a 2×2 table. Journal of the Royal Statistical Society, 126,
109–114.
Fairchild, A. L., & Bayer, R. (1999). Uses and abuses of Tuskegee.
Science, 284, 918–921.
Francis, T., Jr., Korns, R. F., Voight, R. B., Boisen, M., Hemphill,
F., Napier, J., & Tolchinsky, E. (1955). An evaluation of
the 1954 poliomyelitis vaccine trials: A summary report.
American Journal of Public Health, 45(5), 1–63.
Gigerenzer, G., Gaissmaier, W., Kurz-Milcke, E., Schwartz, L.
M., & Woloshin, S. (2008). Helping doctors and patients
make sense of health statistics. Psychological Science in the
Public Interest, 8, 53–96.
Harris, G (March 10, 2009). Doctor admits pain studies were
frauds, hospital says. The New York Times. Retrieved March
20, 2009, www.nytimes.com/2009/03/11/health/research/
11pain.html?ref=us.
Haywood, H. C. (1976). The ethics of doing research . . . and
of not doing it. American Journal of Mental Deficiency, 81,
311–317.
52 q u a n t i tat i v e m e t h o d s a n d e t h i c s
Huff, D. (1982). How to lie with statistics. New York: Norton.
Jacobson, N. S., & Truax, P. (1991). Clinical significance:
A statistical approach to defining meaningful change in
psychotherapy research. Journal of Consulting and Clinical
Psychology, 59, 12–19.
Jeans, M. E. (1992). Clinical significance of research: A growing
concern. Canadian Journal of Nursing, 24, 1–2.
Jones, H. H. (1993). Bad blood: The Tuskegee syphilis experiment
(Revised edition). New York: Free Press.
Kaiser, J. (September 18, 2009). Researcher, two universities sued
over validity of prostate cancer test. Science, 235, 1484.
Kazdin, A. E. (1977). Assessing the clinical or applied impor-
tance of behavior change through social validation. Behavior
Modification, 1, 427–451.
Kazdin, A. E. (2008). Evidence-based treatment and practice:
New opportunities to bridge clinical research and practice,
enhance the knowledge base, and improve patient care.
American Psychologist, 63, 146–159.
Katz, J. (1972). Experimentation with human beings. New York:
Russell Sage.
Kelman, H. C. (1968). A time to speak: On human values and
social research. San Francisco, CA: Jossey-Bass.
Kimmel, A. J. (1991). Predictable biases in the ethical decision
making of psychologists. American Psychologist, 46, 786–788.
Kossinets, G., & Watts, D. J. (2006). Empirical analysis of an
evolving social network. Science, 311, 88–90.
Mark, M. M., Eyssell, K. M., & Campbell, B. (1999). The
ethics of data collection and analysis. In J. L. Fitzpatrick &
M. Morris (Eds.), Ethical issues in program evaluation (pp.
47–56). San Francisco, CA: Jossey-Bass.
Milgram, S. (1963). Behavioral study of obedience. Journal of
Abnormal and Social Psychology, 67, 371–378.
Milgram, S. (1975). Obedience to authority: An experimental view.
New York: Harper Colophon Books.
Miller, J. D. (2007). Finding clinical meaning in cancer data.
Journal of the National Cancer Institute, 99(24), 1832–1835.
Morris, J. A., & Gardner, M. J. (2000). Epidemiological studies.
In D. A. Altman, D. Machin, T. N. Bryant, & M. J. Gard-
ner (Eds.). Statistics with confidence (2nd. ed., pp. 57–72).
London: British Medical Journal Books.
Mosteller, F. (1968). Association and estimation in contingency
tables. Journal of the American Statistical Association, 63, 1–28.
National Commission for the Protection of Human Subjects of
Biomedical and Behavioral Research. (April 18, 1979). The
Belmont Report: Ethical principles and guidelines for the pro-
tection of human subjects of research. Retrieved April 8, 2009,
http://ohsr.od.nih.gov/guidelines/belmont.html.
National Commission on Research. (1980). Accountability:
Restoring the quality of the partnership. Science, 207,
1177–1182.
NICE. (2004). Depression: Management of depression in pri-
mary and secondary care (Clinical practice guideline No.
23). London: National Institute for Health and Clinical
Excellence
NICHSR. (2010). HTA 101: Glossary. Retrieved March 3, 2010
from http://www.nlm.nih.gov/nichsr/hta101/ta101014.html
Orne, M. T. (1962). On the social psychology of the psy-
chological experiment: With particular reference to demand
characteristics and their implications. American Psychologist,
17, 776–783.
Orne, M. T. (1969). Demand characteristics and the concept
of quasi-control. In R. Rosenthal & R. L. Rosnow (Eds.).
Artifact in behavioral research (pp. 143–179). New York:
 Academic Press. (Reissued in Rosenthal & Rosnow, 2009,
pp. 110–137).
Panter, A. T., & Sterba, S. K. (2011). Handbook of ethics in
quantitative methodology. Taylor & Francis.
Pattullo, E. L. (1982). Modesty is the best policy: The federal
role in social research. In T. L. Beauchamp, R. R. Faden, R. J.
Wallace, Jr., & L. Walters (Eds.), Ethical issues in social research
(pp. 373–390). Baltimore, MD: Johns Hopkins University
Press.
Paulos, J. A. (1990). Innumeracy: Mathematical illiteracy and its
consequences. New York: Vintage Books.
Paulos, J. A. (2003). A mathematician plays the stock market. New
York: Basic Books.
PLoS Medicine Editors (September, 2009). Ghostwriting: The
dirty little secret of medical publishing that just got bigger.
PLoS Medicine, 6, Issue 9, e1000156. Accessed September 19,
2009 from http://www.plosmedicine.org/static/ghostwriting.
action.
Prasad, K., Jaeschke, R., Wyer, P., Keitz, S., & Guyatt, G. (2008).
Tips for teachers of evidence-based medicine: Understanding
odds ratios and their relationship to risk ratios. Journal of
General Internal Medicine, 23(5), 635–640.
Reynold, P. D. (1975). Value dilemmas in the professional con-
duct of social science. International Social Science Journal, 27,
563–611.
Rosenthal, R. (1994). Science and ethics in conducting, ana-
lyzing, and reporting psychological research. Psychological
Science, 5, 127–134.
Rosenthal, R., & Rosnow, R. L. (1969). The volunteer subject.
In R. Rosenthal & R. L. Rosnow (Eds.), Artifact in behavioral
research (pp. 59–118). New York: Academic Press. (Reissued
in Rosenthal & Rosnow, 2009, pp. 48–92).
Rosenthal, R., & Rosnow, R. L. (1975). The volunteer subject.
New York: Wiley-Interscience. (Reissued in Rosenthal &
Rosnow, 2009, pp. 667–862).
Rosenthal, R., & Rosnow, R. L. (1984). Applying Hamlet’s ques-
tion to the ethical conduct of research. American Psychologist,
45, 775–777.
Rosenthal, R., & Rosnow, R. L. (2008). Essentials of behav-
ioral research: Methods and data analysis (3rd ed.). New York:
McGraw-Hill.
Rosenthal, R., & Rosnow, R. L. (2009). Artifacts in behavioral
research. New York: Oxford University Press.
Rosnow, R. L. (1990). Teaching research ethics through role-play
and discussion. Teaching of Psychology, 17, 179–181.
Rosnow, R. L. (1997). Hedgehogs, foxes, and the evolving
social contract in psychological science: Ethical challenges
and methodological opportunities. Psychological Methods, 2,
345–356.
Rosnow, R. L., & Rosenthal, R. (1970). Volunteer effects in
behavioral research. In K. H. Craik, B. Kleinmuntz, R. L.
Rosnow, R. Rosenthal, J. A. Cheyne, & R. H. Walters,
New directions in psychology (pp. 211–277). New York: Holt,
Rinehart & Winston.
Rosnow, R. L., & Rosenthal, R. (1995). “Some things you
learn aren’t so”: Cohen’s paradox, Asch’s paradigm, and the
interpretation of interaction. Psychological Science, 6, 3–9.
Rosnow, R. L., & Rosenthal, R. (1996). Contrasts and inter-
actions redux: Five easy pieces. Psychological Science, 7,
253–257.
Rosnow, R. L., & Rosenthal, R. (1997). People studying people:
Artifacts and ethics in behavioral research. New York: W. H.
Freeman.
Rosnow, R. L., & Rosenthal, R. (2008). Assessing the effect size
of outcome research. In A. M. Nezu & C. M. Nezu (Eds.),
Evidence-based outcome research (pp. 379–401). New York:
Oxford University Press.
Rosnow, R. L., & Rosenthal, R. (2011). Ethical principles in
data analysis: An overview. In A. T. Panter & S. K. Sterba
(Eds.), Handbook of ethics in quantitative methodology (pp.
37–58). New York: Routledge.
Rosnow, R. L., Rotheram-Borus, M. J., Ceci, S. J., Blanck,
P. D., & Koocher, G. P. (1993). The institutional review
board as a mirror of scientific and ethical standards. American
Psychologist, 48, 821–826.
Rosnow, R. L., Skleder, A. A., Jaeger, M. E., & Rind, B. (1994).
Intelligence and the epistemics of interpersonal acumen: Test-
ing some implications of Gardner’s theory. Intelligence, 19,
93–116.
Sales, B. D., & Folkman, S. (Eds.). (2000). Ethics in research with
human participants. Washington, DC: American Psychologi-
cal Association.
Saxe, L. (1991). Lying: Thoughts of an applied social psycholo-
gist. American Psychologist, 46, 409–415.
Schwartz, L. M., & Woloshin, S. (2003). On the prevention
and treatment of exaggeration. Journal of General Internal
Medicine, 18(2), 153–154.
Scott-Jones, D., & Rosnow, R. L. (1998). Ethics and mental
health research. In H. Friedman (Ed.), Encyclopedia of mental
health (Vol. 2, pp. 149–160). San Diego, CA: Academic
Press.
Smith, M. B. (1969). Social psychology and human values.
Chicago, IL: Aldine.
Smith, N. C., Kimmel, A. J., & Klein, J. G. (2009). Social con-
tract theory and the ethics of deception in consumer research.
Journal of Consumer Psychology, 19, 486–496.
Spielmans, G. I., & Parry, P. I. (2010). From evidence-
based medicine to marketing-based medicine: Evidence from
internal industry documents. Journal of Bioethical Inquiry:
doi:10.1007/s11673-010-9208-8.
Starobin, P. (January 28, 1997). Why those hidden cameras hurt
journalism. The New York Times, p. A21.
Steering Committee of the Physicians’ Health Study Research
Group. (1989). Final report on the aspirin component of the
ongoing Physicians’ Health Study. New England Journal of
Medicine, 321, 129–135.
Strohmetz, D. B., & Skleder, A. A. (1992). The use of role-play
in teaching research ethics: A validation study. Teaching of
Psychology, 19, 106–108.
Stryker, J. (April 13, 1997). Tuskegee’s long arm still touches a
nerve. The New York Times, p. 4.
Tatsioni, A., Bonitsis, N. G., & Ioannidis, J. P. A. (2007). Per-
sistence of contradicted claims in the literature. Journal of the
American Medical Association, 298, 2517–2526.
Trials of War Criminals before the Nuernberg Military Tribunals
under Control Council Law No. 10, October 1946-April
1949, Vol. II. Washington, DC: U.S. Government Printing
Office.
Tufte, E. R. (1983). The visual display of quantitative information.
Cheshire, CT: Graphics Press.
Tufte, E. T. (1990). Envisioning information. Cheshire, CT:
Graphics Press.
Tufte, E. T. (2006). Beautiful evidence. Cheshire, CT: Graphics
Press.
Tukey, J. W. (1977). Exploratory data analysis. Reading, MA:
Addison-Wesley.
rosnow, rosenthal 53
Turner, E. H., Matthews, A. M., Linardatos, E., Tell, R. A., &
Rosenthal, R. (2008). Selective publication of antidepressant
trials and its influence on apparent efficacy. New England
Journal of Medicine, 358(3), 32–40.
VandenBos, G. R. (Ed.). (2007). APA dictionary of psychology.
Washington, DC: American Psychological Association.
Vedula, S. S., Bero, L., Scherer, R. W., & Dickersin, K.
(2009). Outcome reporting in industry-sponsored trials
of Gabapentin for off-label use. New England Journal of
Medicine, 361(20), 1963–1971.
Vinacke, W. E. (1954). Deceiving experimental subjects. Ameri-
can Psychologist, 9, 155.
Wainer, H. (1984). How to display data badly. American
Statistician, 38, 137–147.
Wainer, H. (1996). Depicting error. American Statistician, 50(2),
101–111.
Wainer, H. (2000). Visual revelations: Graphical tales of fate and
deception from Napoleon Bonaparte to Ross Perot. Mahwah, NJ:
Erlbaum.
54 q u a n t i tat i v e m e t h o d s a n d e t h i c s
Wainer, H. (2009). Picturing the uncertain world. Princeton, NJ:
Princeton University Press.
Wainer, H., & Thissen, D. (1981). Graphical data analysis.
Annual Review of Psychology, 32, 191–241.
Wasserman, S., & Faust, K. (1994). Social network analysis:
Methods and applications. Cambridge, England: Cambridge
University Press.
Woloshin, S., Schwartz, L. M., Casella, S. L., Kennedy, A. T., &
Larson, R. J. (2009). Press releases by academic medical cen-
ters: Not so academic? Annals of Internal Medicine, 150(9),
613–618.
Zuckerman, M., Hodgins, H. S., Zuckerman, A., & Rosenthal,
R. (1993). Contemporary issues in the analysis of data: A
survey of 551 psychologists. Psychological Science, 4, 49–53.
 CHAPTER

Special Populations
4
Keith F. Widaman, Dawnté R. Early, and Rand D. Conger

Abstract

Special populations offer unique opportunities and challenges for mathematical/statistical modeling of
data. First, we discuss several ways of construing the notion of special populations, including the basis
on which we argue that the general notion of special populations is a rather recent one. Then, we
discuss four key methodological implications when considering special populations: (1) properly
defining and accessing participants from the special population; (2) ensuring that the same dimensions
are present across multiple populations; (3) assessing whether empirical implications of psychological
theories hold across populations; and (4) exploiting unusual variation in special populations that may
allow tests of unique hypotheses. Next, we provide examples that illustrate how to deal with each of
the methodological issues. We close with a discussion of issues occasioned by our discussion of
special populations, issues that represent topics for future developments.
Key Words: Special populations, group differences, factorial invariance, measurement invariance,
structural invariance, individual differences

Introduction methods to capture better behavioral phenomena in

The topic of the current chapter is the place or
importance of special populations, particularly with
regard to how quantitative methods or techniques
can be used to understand or characterize special
populations or inform about the nature of special
populations. Research on special populations has
burgeoned during the past quarter-century, and the
pace of development of quantitative methods has
also expanded rapidly during this period. In this
chapter, we deal with the intersection of these two
streams of research—special populations and quan-
titative methods—to illuminate both. That is, we
discuss ways in which the use of state-of-the-art
quantitative techniques can help explain the nature
of special populations in unique and informative
ways. In turn, we hope that consideration of spe-
cial populations may provide feedback that will
lead to interesting developments in quantitative
these groups.
We develop several goals for the chapter based on
our considerations of the application of quantitative
methods to special populations. An initial goal is to
identify the nature or conception of special popu-
lations. Here, we discuss our observations on how
special populations are identified. A second goal is
to explore how research on special populations offers
challenges to or ready application of methodological
or quantitative approaches. To meet this goal, we dis-
cuss four major implications we draw when thinking
about conducting research with special populations,
and we discuss techniques that would be partic-
ularly appropriate in pursuing these implications.
Our third goal is to describe a series of applications
of quantitative techniques in the study of special
populations to provide a substantive instantiation of
how quantitative techniques can be used to clarify

55
the nature of special populations and the dynam-
ics of psychological and biological processes in these
groups. We close with conclusions and a series of
questions that represent issues for future research.
In pursuing the nature and implications of special
populations for quantitative methods, we performed
several literature searches to help bound or circum-
scribe our thinking about special populations. A
PsycINFO search in August 2010 using “special
population” and “special populations” as possible
title words yielded 397 citations. The oldest cita-
tion was to a publication by Uehling in 1952, the
next oldest was to a publication in 1975, and 390
(or more than 98%) of the 397 citations were to
publications in 1980 or later. Thus, the clear major-
ity of publications listing “special population” as a
title word have occurred relatively recently, within
the last 30 years.
More inclusively, we next conducted a PsycINFO
search using “special population” and “special popu-
lations” as possible keywords. This search turned up
1546 citations, a far broader net of citations than
provided by the title-word search. Here, the oldest
citation was to a chapter in a volume resulting from
a White House Conference on Child Health and
Protection, published in 1931 during the Hoover
administration (see Folks, 1931). The second oldest
citation was the Uehling (1952) paper that was the
oldest publication in the title-word search, and 1503
(or more than 97%) of the 1546 citations were to
publications in 1980 or later.
Finally, a PsycINFO search using “special pop-
ulation” or “special populations” as phrases to be
found anywhere in the database led to 4266 cita-
tions, our search that led to the most inclusive list of
reference citations. The two oldest citations were the
Folks (1931) and Uehling (1952) papers uncovered
in the earlier searches, and a full 4221 (or more than
98.9%) of the citations were to works published in
1980 or later. Consistent with the preceding two
searches, the vast bulk of citations under the explicit
title or heading of “special populations” occurred
within the past three decades.
Each of these three searches outlined above sup-
ports the conclusion that work on or thinking about
special populations under that specific rubric is a
fairly recent phenomenon. However, work on spe-
cial populations has been a hallmark of research in
psychology for more than 100 years, if not more,
even if this research has not been published under
the heading of special populations. For example,
research and theory on persons with mental illness
can be traced back more than 200 years, as can
56 s p e c i a l p o p u l at i o n s
research on persons with mental retardation or intel-
lectual disability. Clinical methods in use more than
200 years ago for dealing with persons with mental
illness or intellectual disability appear barbaric to the
twenty-first-century practicing scientist or informed
citizen; indeed, many clinical methods in use only
50 or 75 years ago seem rather unfortunate and
misguided. Thus, research on special populations
has long been pursued in psychology, allied behav-
ioral and social sciences, and medical sciences. But,
the time is ripe for renewed exploration of behav-
ioral phenomena in special populations to provide a
fuller understanding of persons in these populations,
which should lead to improved ways of treating and
improving the lives of persons in special populations.
Conceptions of Special Populations
To consider methodological implications and
applications related to special populations, one must
consider first what a special population is. The term
special population has no obvious and ready refer-
ent. That is, if one asked 10 different psychologists
what was meant by “special populations,” one might
get 10 different answers with little in the way of
overlap aside from the indication that “special pop-
ulation” implies deviation from general population
norms. To confront this definitional problem, we
read a random sample of publications identified in
our literature searches to identify the ways different
researchers used the term “special populations.” In
the rest of this section, we discuss several ways in
which investigators have used the term, admitting
that our categorization is not exhaustive.
Disability Groups
One common use of the term special popula-
tion appears, implicitly, to represent a less pejorative
way of referring to one or another disability group.
Research on persons with a slowed rate of mental
development was long published under the head-
ing of research on mentally retarded persons, with a
clear implication that this was a special population.
More recently, the term mentally retarded persons was
replaced by persons with mental retardation, which
was a less pejorative term; still more recently, the
accepted term is now persons with intellectual disabil-
ity. Historical changes in these terms are reflected in
the name of the leading professional organization
for research and practice in this domain, which has
changed its name from the American Association
on Mental Deficiency to the American Association
on Mental Retardation and then to the American
Association on Intellectual and Developmental Dis-
abilities. Regardless of the precise term used to
refer to this population, the key defining feature is
that a person is considered to be a member of the
special population if he or she has exhibited a signif-
icantly slower rate of mental development relative
to persons in the general population.
As noted above, persons with mental illness are
another group that is treated as a special popula-
tion, a special population defined on the basis of
a perceived disability. Many additional disability
groups can be identified, including persons with
visual deficits or blindness, persons with hearing loss
or deafness, persons with physical disabilities, and
individuals with learning disabilities, to name only
a few. The one family resemblance shared by all of
these uses of the term “special population” is that
the generic label is applied to a group to denote the
presence of a behavioral disability that is common
to all members of the group.
One problem with the use of the term “special
population” in connection with disability groups
arises in the context of comorbidity, or the presence
of more than one identifier of disability. For exam-
ple, consider the case of a person with intellectual
disability who is also deaf. Is this person a member
of two different special populations—that is, is such
a person a member of the special population of per-
sons with intellectual disability and also a member
of the special population of persons who are deaf?
Or, is the person a member of a new and still more
special population, the population of persons with
dual diagnoses of intellectual disability and deafness?
No resolution of this issue is proposed here. Rather,
we merely raise this issue to highlight a vexing issue
in evolving notions of disability groups.
Research on disability groups takes a number
of forms, and we survey a few of these here. As
one example, Martz and Daniel (2010) collected
data in the United States and Kenya to determine
whether disability prototypes for four disability
groups (persons with AIDS, hearing impairment,
mental illness, and spinal cord injury) differed across
the four groups and whether this was moderated
by country of residence. As a second example, Or,
Cohen, and Tirosh (2010) used discriminant anal-
ysis to determine whether measures from a parent-
rated questionnaire could discriminate among three
groups of students, including those with attention-
deficit hyperactivity disorder (ADHD), those with
a learning disability, and those with a combination
of ADHD and learning disability. Finally, Cobb,
Lehman, Newman-Gonchar, and Alwell (2009)
synthesized results from prior meta-analyses of self-
determination interventions for various categories
of persons with disability. In some meta-analyses,
research on students from any disability group were
included, whereas other meta-analyses focused on
more restricted groups, such as students with intel-
lectual disability or developmental disabilities or
students with ADHD. The synthesis by Cobb et
al. ended on an unfortunate note, concluding that
interventions to enhance levels of self-determination
appear to have relatively weak effects, perhaps
because of heterogeneity of the special populations
combined in the meta-analyses.
Echoing issues identified above, Barton (2009)
has decried the lack of a common definition of
the basic term disability by persons with disabili-
ties or their advocates. In 1990, the U.S. Congress
passed the landmark Americans with Disabilities
Act, which outlawed discrimination on the basis of
disability and outlined many ways in which accom-
modations for persons with disabilities must be
provided. Given this legislation and the relatively
large numbers of persons in the United States who
fall in one or another disability category, the pop-
ulation of persons with disability could and should
be a potent political force. But, without a common
and inclusive definition of the term disability, the
political clout of this population typically is watered
down, as advocates for particular disability groups
(e.g., persons with intellectual disability, students
with deafness) follow their own special agendas. If
persons with disability and their advocates could
unite subgroups under a commonly accepted and
inclusive definition of disability, this would almost
certainly lead to a more united stance on issues, lead-
ing to greater power. Until that time, the political
power of persons with disabilities is likely to remain
fragmented and therefore weaker.
“Superability” Groups
In contrast to the notion of disability group-
ing, “special population” can also be used to define
individuals with superior levels of ability or per-
formance, which can be termed superability groups
to contrast with the term disability group. One
of the most common “superability” groups is the
intellectually gifted, often defined as persons with
intelligence quotients (IQs) greater than or equal to
130. But, giftedness has many dimensions; some
individuals are considered gifted in general, whereas
others may be deemed gifted in narrower domains
such as the arts or specific academic domains.
widaman, early, conger 57
 One issue in the study of superability groups
is the early identification of such individuals. Two
notable attempts at early identification were under-
taken during the twentieth century. In the first of
these, Terman established what became known as
the Genetic Studies of Genius. Terman had recently
developed the Stanford Binet Scale of Intelligence,
which was published in 1916. Then, from 1921 to
1923, he asked fifth grade teachers in California to
nominate the three brightest children in their classes
and the youngest child. After testing these children
with the new Stanford Binet scale, children who
scored 130 or above (i.e., two or more SDs above the
population mean) were invited into the study. Ini-
tially, more than 1450 children were enrolled in the
study, and later additions resulted in a final sample
size of 1528 children.
The second large-scale study of early identifica-
tion was the work begun in 1971 by Stanley to
identify junior high school students with very high
levels of math skill, an undertaking subsequently
dubbed the Study of Mathematically Precocious
Youth (SMPY). These youth were identified at age
12 or 13 years and had to score within the top 3% on
a test of school achievement. Then, the young peo-
ple were given the SAT, a standard test for college
admission, and had to score within the top 1% on
the test. Once selected into a cohort, the young stu-
dents were given a variety of accelerated academic
experiences to facilitate their learning in domains
of science and math. As described by Lubinski
and Benbow (2006), who are now the co-directors
of SMPY, a total of five cohorts of SMPY youth
were recruited. Recruitment for the various cohorts
took place between 1972 and 1992, and a total
of more than 5300 youth have participated in the
program.
A more recent example of early identification of
gifted individuals is a study by Kuo, Maker, Su, and
Hu (2010), who described an early identification
protocol used in Taiwan to identify gifted preschool-
ers. Children so identified were then enrolled in
an enrichment program to offer an optimal envi-
ronment for them to increase their problem-solving
abilities in multiple modalities. The protocol used
various kinds of information—from interviews,
checklists, portfolios, intelligence tests, and observer
ratings—to identify giftedness in several domains.
Although most children were deemed gifted in one
or another of the domains, almost 20% of children
were identified as gifted in more than one domain.
Whether these very-early identified children will
remain characterized as gifted at later points in their
58 s p e c i a l p o p u l at i o n s
lifespan will be an interesting result to track in the
future.
In an interesting twist, a superior level of ability
or performance in one area can be exhibited in the
presence of rather low performance in other areas.
For example, Olson, Berryhill, Drowos, Brown,
and Chatterjee (2010) reported on a patient who
had rather severe impairments in episodic memory
that presumably arose as a result of anoxia during
birth. The impairments in episodic memory were
quite general, resulting in rather poor performance
on many memory measures. However, the patient
had extremely accurate ability to recall calendar data
with regard to day, month, and year, and this unusu-
ally high skill enables the patient to recall the precise
date of many of his personal experiences. Another
example is that of hyperlexia, or precocious devel-
opment of single-word reading, which has typically
been identified only in persons who have a devel-
opmental disability (Grigorenko, Klin, & Volkmar,
2003). As a result, the identification of a partic-
ular form of performance as a “superability” does
not insure that the person displaying such an abil-
ity is thereby a member of a generally advantaged
group. Rather, the superability may merely represent
an unusually superior level of performance in the
context of a generally depressed level of functioning.
Demographic Groups
Another way of using the “special population”
term is to refer to groups of individuals who are
identified on the basis of demographic character-
istics. The most common characteristics used to
classify individuals in the United States are sex, age,
ethnic (or racial) status, and socioeconomic status
(SES) grouping, although other demographic vari-
ables are also used. For example, Sussman (2006)
has discussed the prevention of adolescent alcohol
problems in various special populations, citing vari-
ation across gender, ethnicity, region of the country,
and SES groups.
The issue of special populations often arises when
investigating the differential validity of psychologi-
cal tests with persons from different ethnic groups.
Cleary (1968) offered a series of statistical tests using
multiple regression analysis to determine whether
intercept bias and/or slope bias existed in the use
of test scores when evaluating students from differ-
ent ethnic groups for college admission. If neither
intercept nor slope bias were found to occur, then a
test would be deemed unbiased for use as a selection
device. If fewer applicants from a special population
(e.g., African-American) were selected using such
a test, then one could justify the result by claim-
ing that the lower level of selection resulted from
lower levels of scores obtained by members of the
special population on unbiased tests. Although so-
called “Cleary tests” have been used routinely for
more than four decades in college admissions and in
personnel selection, concerns about the utility of the
approach remain (e.g., Meade & Tonidandel, 2010;
Colarelli, Han, & Yang, 2010).
Retrieving literature that references “special pop-
ulations,” we were struck by the frequency with
which authors who referred to special populations
cited groups such as women, children, the elderly,
and minority ethnic groups. This designation of
special populations implies that non-women, non-
children, non-elderly, non-minority individuals—
that is, adult, White males and, perhaps, college
sophomores—are “the” populations that serve as
common reference groups, and anyone who devi-
ates from these norms is a member of a special
population.
Why White adult males and college sophomores
became standard reference groups is anyone’s guess.
Most likely, the emergence of these reference groups
resulted from not just one but a combination of
factors. Since the founding of the United States of
America, with few notable exceptions, adult White
males have tended to occupy the highest positions
of power in business, academia, and government.
Regardless of the optimality of this distribution
of power, the mere presence of this demographic
in positions of power may have led, explicitly or
implicitly, to acceptance of adult White males as the
reference group against which outcomes for other
groups would be compared. Additionally, the use of
a single demographic, such as adult White males,
might have been considered a way to reduce hetero-
geneity that might otherwise cloud research results.
Assuming that results for different demographic
groups should not differ dramatically, reduction of
heterogeneity might allow trends in data to be seen
more clearly.
Other reasons are, almost surely, responsible for
the use of college sophomores as a standard or
reference group. The top reason for selecting col-
lege sophomores as “typical” research participants
must be convenience. When taking introductory
psychology courses in college, students often must
participate in experiments so that they learn about
how studies are conducted, and college sophomores
are a common demographic in introductory psy-
chology classes. A quarter-century ago, Sears (1986,
updated by Henry, 2008; Sears, 2008) decried the
use of college sophomores as typical research partici-
pants, arguing that college sophomores may provide
systematically different responses than members
of the general population on many, if not most,
experimental questionnaires and paradigms. Col-
lege sophomores are in a stage of life when they are
attempting to “find” or define their identities and
thus may be much more susceptible to various influ-
ences, such as experimental inductions, than would
others. Despite the potentially limited utility of col-
lege sophomores as research participants, given their
unrepresentativeness relative to the population, con-
venience in obtaining their participation in research
is a leading cause of their continued predominance
in studies in social and personality psychology.
Political pressure often appears to be another fac-
tor related to the choice of research participants,
either restricting or promoting research on a given
topic. If a researcher chooses to study a topic (e.g.,
romantic love) or a group (e.g., homosexual males)
that a member of Congress finds objectionable, the
research project might be highlighted as a waste of
taxpayer money. Senator William Proxmire (D—
Wisconsin) made headlines in the 1970s and 1980s
when announcing his Golden Fleece Awards, which
derided programs of research he personally opposed.
On the other hand, the decades-long support for
research on persons with intellectual disability by the
members of the Kennedy family in Congress led to
far more research funding in this area than otherwise
would have occurred. Indeed, continued research
support for any special population usually requires
the presence of a special champion in Congress for
that population, given conflicting funding priorities
at the national level.
Yet another reason why certain groups are not
often represented as research participants is the
sheer difficulty in finding and recruiting partici-
pants in these groups. Masten and colleagues (e.g.,
Obradoviæ, Long, Cutuli, Chan, Hinz, Heistad,
& Masten, 2009) have been studying homeless and
highly mobile children, and identifying members
of this group and then tracking them longitudi-
nally has been difficult. Other challenges often face
researchers who study court-related samples, such
as victims of physical or sexual abuse or children or
adolescents in foster care, where concurrence of legal
entities are yet another impediment to research.
Regardless of the basis for the designation of
reference populations, signs of the breakup of the
hegemony of White adult males and college sopho-
mores as reference groups are clearly in evidence.
widaman, early, conger 59
For example, when applying for grants through the
National Institutes of Health, researchers must care-
fully describe the projected sample with regard to
sex, age, and ethnicity, and any exclusion of persons
from a particular demographic category must be
justified in convincing fashion. Further, concerted
efforts have been made to correct the demographic
imbalance in prior research on health. The Women’s
Health Initiative (http://www.nhlbi.nih.gov/whi/)
was a major program of research funded by the
government to investigate various outcomes in
postmenopausal women, focusing on cardiovas-
cular disease, cancer, and osteoporosis as com-
mon causes of disability, morbidity, and death
of women. Furthermore, the Office of Minor-
ity Health of the U.S. Department of Health and
Human Services has pursued Minority Health Ini-
tiatives (http://minorityhealth.hhs.gov/) to investi-
gate causes of disease, morbidity, and death in
minority populations that deviate from patterns
common in the majority (i.e., White) population.
We look forward to the day when women, children,
the elderly, and ethnic minorities are not consid-
ered special populations but are considered major
portions of the general population that deserve just
as much attention as research subjects as any other
demographic group.
Functional Groups
The term special population may also be applied
to identify persons who, in our terminology, are
members of identifiable functional groups. By func-
tional groups, we refer to individuals who have
particular combinations of behavioral profiles or
life situations that may have unique importance for
understanding their behavior. That is, individuals
can be characterized by the full repertoire of pos-
itive and negative behaviors they exhibit and the
life situations they have selected or that have been
imposed on them. Any of these factors, particularly
in combination, may define groups that function
in unique fashion to determine their behavior and
their susceptibility to environmental presses, such as
treatments.
In our literature searches on special populations,
we found a great many of the articles retrieved con-
cerned treatment outcomes in special populations,
where these special populations met our description
of functional groups. Over a decade ago, Polinsky,
Hser, and Grella (1998) described the extremely
varied types of clients who received services from
drug treatment programs in Los Angeles County,
60 s p e c i a l p o p u l at i o n s
highlighting characteristics such as the health sta-
tus, ethnicity, language needs, and gender-related
issues of clients. A key issue Polinsky et al. dis-
cussed was the application of types of treatments
to types of clients, under the assumption that client
characteristics may alter the effectiveness of partic-
ular treatments. Taking the issue further, Polinsky
et al. proposed that treatments might be spe-
cially adapted to the characteristics of the client to
obtain maximal success. More recently, Diaz, Hor-
ton, McIlveen, Weiner, and Nelson (2009), using
data from substance abuse programs, found that
almost half (48%) of the clients in their sample
had dysthymia. Building on this finding, Diaz et
al. recommended that treatment programs consider
whether clients have psychological disorders such as
dysthymia when treatments are formulated, because
treatment success may depend on the presence of
significant comorbid characteristics.
Expanding the reach of functional groups beyond
personal traits or characteristics, aspects of personal
life situations may also be used to define functional
groups. Individuals are born into families, these
families live in communities, and communities are
nested within larger geographical entities. In many
publications, Bronfenbrenner (e.g., 1977, 1986a,
1986b, 1999) laid out the ever-expanding circles of
embedded environments from micro- to macrosys-
tems. A recent book edited by Little, Bovaird,
and Card (2007) was dedicated to presenting sta-
tistical and other methodological solutions to the
modeling of contextual effects, such as embedded
social systems, that are rife in studies conducted
in representative, everyday contexts. In one con-
tribution to the Little et al. volume, Widaman
(2007) discussed the integration of embedded ver-
sions of both the social environment and the physical
environment as these combine with personal char-
acteristics to influence behavior. Although any of
these embedded levels may play a role in moderat-
ing behavioral change, variables associated with the
more proximal environments of family and com-
munity probably play a larger role than do more
distal variables. In a recent study, Chassin, Knight,
Vargas-Chanes, Losoya, and Naranjo (2009) found
that treatments to reduce certain forms of negative
behavior were effective only if families were involved
in the treatment. The upshot of this finding is that
personal characteristics and aspects of both the social
and physical life situations within which an individ-
ual functions should be considered when attempting
to understand the behavior and adaptability of the
individual.
Biological or Genetic Markers of Group
Membership
A fifth and final type of special population is
perhaps the most current and state-of-the-art way
of defining special populations—by the presence of
specific biological or genetic markers that have ties to
behavior. Since its rebirth in the early 1970s, behav-
ior genetic research has accumulated at a seemingly
ever-increasing rate. The general thrust of findings in
this field is that most, if not all, behavioral traits have
some heritability, and certain important traits (e.g.,
intelligence) have high levels of heritability. Consis-
tent with this focus, research continues to focus on
biological markers related to behavior, where biolog-
ical markers are broadly defined, from the molecular
level of particular genes or single-nucleotide poly-
morphisms (SNPs) to the molar level of performance
on experimental tasks.
Searching for the molecular genetic bases of
behavioral traits is a major focus at the present time
(Petrill, 2010). The successful identification of par-
ticular genes related to a certain condition is often
touted in the press. For example, Chakrabarti et al.
(2010) recently reported that certain genes associ-
ated with sex steroids and neural growth are related
to autistic traits and Asperger syndrome. But, the
mere presence of certain SNPs may not be the key.
Rather, gene expression at the molecular levels may
be the important feature, as Simunovic et al. (2009)
reported in their study of the pathology underlying
Parkinson’s disease. Although successes in finding
certain SNPs or particular forms of gene expression
related to a disease or behavior have been reported,
most of these involve very small portions of vari-
ance explained and suffer from lack of replication in
follow-up investigations.
Perhaps more promising is research at a more
molar level. For example, Pennington et al. (2008)
reported synaptic and metabolic abnormalities in
the prefrontal cortex in persons with schizophre-
nia or bipolar disorder. Focusing on the anterior
cingulate, Eastwood and Harrison (2010) found
increased synaptic transmission and plasticity in
persons with bipolar disorder. At a still more
molar level, Koychev, El-Deredy, Haenschel, and
Deakin (2010) reported visual information process-
ing deficits in persons with schizotypy, which is
a marker of vulnerability to schizophrenia. And,
Reichenberg, Caspi, Harrington, Houts, Keefe,
Murray, Poulton, and Moffitt (2010) argued that
they had identified patterns in performance on
standard psychological tests that reflected cogni-
tive deficits related to childhood schizophrenia. The
varied levels at which this research is undertaken—
from the level of synaptic processes to molar patterns
in behavior—is remarkable, yet the patterns uncov-
ered all point to the biological nature of the processes
involved.
Two additional ways of understanding biologi-
cal markers and their effects deserve mention here.
First, the search for a solitary SNP or a small number
of SNPs responsible for a particular psychological
or behavioral trait is almost certainly a rather unre-
alistic goal. A more likely outcome is reflected in
research on phenylketonuria (PKU). As reported on
the Phenylalanine Hydroxylase Locus Knowledge-
base website (http://www.pahdb.mcgill.ca/), more
than 500 mutations on the phenylalanine hydroxy-
lase (PAH) gene have been identified, and all of these
lead to reduced metabolism of phenylalanine into
tyrosine, the underlying problem in PKU. Although
a small number of mutations (e.g., 5 or 6) may
account for the majority of mutations in particular
populations (e.g., European), any of the mutations
can cause PKU. Further, many of the mutations
have been categorized with regard to the severity
of the mutation, indexed by the degree of disrup-
tion of phenylalanine metabolism. If a large number
of mutations are found to underlie a single, rather
restricted phenotype such as PKU, we should expect
that very large numbers of mutations or SNPs are
related to broader phenotypes such as intelligence,
intellectual disability, or personality disorders.
Second, we think that researchers must pay at
least as much attention to the environment as to gene
SNPs when searching for genes that affect behav-
ior, an approach that has been characterized as the
search for Gene X Environment, or G X E, interac-
tions. Specific genes or SNPs are important in the
G X E approach, but the SNPs alone do not directly
herald the emergence of a behavioral trait. Rather,
behavioral differences among groups identified with
different genetic alleles may arise only in particu-
lar environmental circumstances or may be clearly
exacerbated in such environments, so main effects
of genes or environments supply less-than-complete
information. Rather than main effects, the G X E
interaction indicates that the effects of genetic alle-
les is moderated by environmental circumstances,
so environments modulate how genetic factors are
expressed in behavior. Two early papers on the G
X E approach, by Caspi et al. (2002) and by Caspi
et al. (2003), were both published in Science, the
leading general journal in all of science. In the sec-
ond of these papers, Caspi et al. (2003) investigated
the 5-HTTLPR region, which is associated with
widaman, early, conger 61
serotonin function. The 5-HTTLPR region is char-
acterized by either short (s) or long (l) alleles; because
individuals obtain one copy from mother and one
from father, individuals can be characterized as s/s,
s/l, or l/l based on whether they have (1) two short,
(2) one short and one long, or (3) two long alleles,
respectively. Caspi et al. presented data suggesting
that persons with the l/l allele were relatively imper-
vious to stressful environments and therefore tended
to have lower levels of negative outcomes (e.g.,
depressive symptoms, suicide ideation/attempts) in
the most stressful environments. In contrast, indi-
viduals with the s/s allele tended to have the most
negative outcomes in the most stressful environ-
ments, and persons with the s/l allele had outcomes
that fell midway between the l/l and s/s groups.
However, these differences did not hold in all envi-
ronmental situations. Indeed, in the least stressful
environments, essentially no differences across allele
groups were found. Thus, in low-stress environ-
ments or in environments with no maltreatment,
the allele groups did not differ in depressive out-
comes, but differences across groups appeared only
as the stressfulness of the environment increased.
More recently, researchers have been investigat-
ing the notion of the genetic basis for differential
susceptibility to the environment. The studies by
Caspi et al. (2002, 2003) support the contention
of G X E interactions, but the allele group that
fared worst in the most stressful environments rarely
exhibited any benefit versus the other groups in
the least stressful environments. However, as Bel-
sky, Bakermans-Kranenburg, and van IJzendoorn
(2007) argued, the Caspi et al. studies may not
have investigated the widest possible range of envi-
ronmental circumstances, instead generally looking
only at stressful versus average environments. How-
ever, if one studied the entire range of environmen-
tal circumstances—from worst through average to
superior environments—then a true cross-over G X
E interaction may be found. That is, persons with
certain genetic alleles (e.g., the l/l allele from the
5-HTTPLR) may do relatively well in very poor
environments but also may not do much better in
superior environments, representing a group of per-
sons who are relatively impervious to environmental
circumstances and therefore have low susceptibility
to effects of the environment. In contrast, per-
sons with other alleles (e.g., the s/s allele from the
5-HTTPLR) may indeed perform rather poorly in
the worst environments but might show the best
outcomes of all groups in superior environments. If
this were to occur, these individuals would be the
62 s p e c i a l p o p u l at i o n s
most susceptible to environmental influence, with
their behavioral outcomes tracking the negativity or
positivity of the environments within which they
have developed. Although firm conclusions about
the presence of differential susceptibility and the
resulting cross-over G X E interactions has not yet
been provided, many experimental results published
in the last few years seem to support this idea. If G X
E interactions—particularly cross-over G X E inter-
actions consistent with the differential susceptibility
notion—are present in many behavioral domains, a
more nuanced picture must be drawn, with genes
and environments having co-equal status as the basis
for behavioral phenotypes. Groups may still be iden-
tified by their genetic alleles, but the implications of
their genes for these special populations can only
be understood by considering the environments in
which the persons have developed.
Summary
As a reading of this section demonstrates, the
notion of special populations is often invoked but
can refer to very different kinds of groupings of
individuals. We have identified five ways of char-
acterizing special populations, based on disability
status, superability status, demographic characteris-
tics, functional characteristics, and genetic markers.
Others might be able to identify additional classes of
variables that might be used to designate special pop-
ulations. Although the distinctions among different
types of special populations underscore the hetero-
geneous nature of the alternate bases for groupings of
persons, all of the distinctions among groups have an
important familial resemblance: Researchers must
investigate whether patterns of empirical results vary
in important ways across special populations or
when moving from the general population to a spe-
cial population. If results vary importantly across
groups, then special population status is a modera-
tor of results, and conclusions about “the ways that
things work” do not generalize across groups. Thus,
special populations constitute a crucible for research
in the social sciences, and we must guard against
unwarranted generalization of findings across
groups unless research supports such conclusions.
Methodological Implications of Special
Populations
Having established some guidelines for distin-
guishing among special populations, we turn next
to the methodological or quantitative implications
that arise when considering special populations. In
most graduate education in psychology, quantitative
experts teach classes in which students are prepared
to obtain data from a sample (typically described in
nebulous terms); take standard, off-the-shelf statis-
tical methods; estimate population parameters in
a particular analytic model, and use their results
to make conjectures about the population, as if
the population were a single, monolithic entity.
But, once one acknowledges the presence of spe-
cial populations (which represent specially identified
subsets of the larger population) our thinking about
methodological and statistical procedures must be
amended. Rather than estimating “the” population
parameter in monolithic population, we should
begin trying to understand the nature of special
populations, how parameter estimates might vary
across populations, when it is possible to compare
parameter estimates across groups, and similar dif-
ficult questions. In this section, we slice up this
task under four headings—identifying and accessing
special populations, measuring the same constructs
across groups, exploring the bounds of psychological
theories, and exploiting unusual variation in spe-
cial populations—and discuss the methodological
implications of the substantive issues that arise.
Identifying and Accessing Participants in
Special Populations
The first task of any investigation into partici-
pants from special populations is gaining access to
the participants. But, this characterization of “gain-
ing access to the participants” masks several neces-
sary steps in the design and conduct of an empirical
investigation. To provide some cognitive structur-
ing of this first concern, we have broken down the
research process associated with drawing a sample
from a special population into four steps or issues.
The first task is the clear identification of the spe-
cial population to be studied. As earlier sections
of this chapter have shown, a single, overarching
notion of what constitutes a special population is
not present in the research literature. Rather, vari-
ous ways of defining special populations can be used,
and some of these will provide partially overlapping
subsets of possible participants. Because of the seem-
ingly murky nature of some special populations, we
offer only the most general recommendations of how
to deal with the problem of identification of the pop-
ulation. Researchers should be careful to develop a
clear statement of the population to be studied. If
the target population is all persons receiving mental
health services in a catchment area, then no explicit
or implicit exclusion criteria should be used or one
might inadvertently draw an unrepresentative sam-
ple. Of course, a more restricted population could be
the topic of a study, such as persons from minority
groups who seek mental health services with regard
to alcohol problems but without attendant drug use
problems. We hope the reader is clear on the issue
of identification of a special population: Both inclu-
sion and exclusion criteria for participants must be
clearly elucidated and justified by the nature of the
study, and the research team should remain vigilant
to ensure that no unexpected factors are biasing the
nature of the population defined.
A second issue is the development of a plan
to access participants from the special population.
If one is interested in studying clinical popula-
tions, access to potential participants will often be
sought through professional agencies, such as men-
tal health centers, regional centers that offer services
for members of particular populations (e.g., intel-
lectual disability), state agencies, or the like. If the
goal is to study students in elementary school, devel-
oping ties to schools is the most obvious approach.
But, members of some populations are much harder
to access. For example, researchers who study child
physical or sexual abuse or neglect must often access
their participants through arrangements with court
systems, child welfare organizations, and so forth.
In these research endeavors, investigators frequently
find that a great deal of time must be spent and
a large amount of red tape must be surmounted to
gain access to research subjects. Still, one should not
let the difficulty of the access alter the research goals
of a program of research. To provide the most valid
data from the most representative settings, proper
and optimal access must be developed.
A third step, once the special population has
been identified and modes of accessing participants
have been developed, is to develop a plan for sam-
pling participants from the population. In some
situations, a sampling plan can be relatively straight-
forward. For example, a researcher might select each
birth (or every second birth) from a set of hospi-
tals during a given time period as the sample to
be drawn, and this will often lead to the draw-
ing of a representative sample from the population
that is served by the hospitals in question. Study-
ing birth outcomes in low SES families might be
approached in this fashion, if the hospitals selected
tended to serve the low SES families in a community.
But, if a researcher intends to investigate differ-
ences across certain strata (e.g., African-American,
Hispanic [or Latino]) and participants from these
widaman, early, conger 63
strata are unequally represented in the population,
then oversampling the low-frequency participants
and then re-weighting in the analysis stage can
ensure statistical analyses that have greater external
validity or generalizability. Full discussion of sam-
pling plans is beyond the scope of this chapter; here,
we merely emphasize the need to consider the sam-
pling plan with as much attention and concern as
other aspects of study design.
The final step in an empirical study is the recruit-
ment of participants from the special population.
This step involves the initial contact with potential
participants and their recruitment into the study.
The recruitment rate into a study is a crucial statis-
tic and should be reported in every published report.
Recruitment rates vary across types of studies, so a
low recruitment rate for a given study may not be
a fatal flaw if this is representative of the studies in
the domain. However, researchers should collect as
much basic data as possible on potential participants
so that trends in participation versus nonparticipa-
tion might be discerned. If nonparticipants differ
systematically from participants on any variables,
this may limit the generalizations to be drawn from
the study. Researchers should report the recruitment
rate in any published paper, so readers will have a
basis for placing the research results in context.
Implication 1: Informed identification of special
populations and members of such groups is often a
difficult task. But, the failure to identify, sample, and
recruit members of special populations in appropriate
ways may render any empirical results of dubious
value.
The history of research in psychology tends to
reflect the conduct of studies on samples of con-
venience. Researchers often are careful to describe
the basic demographic characteristics of their partic-
ipants but infrequently discuss whether the sample
is representative of any larger population. As psy-
chology matures as a science, greater attention will
be paid to the issues of drawing appropriate sam-
ples from well-defined populations. Our hope is
that as researchers pay greater attention to this set
of issues, research results will begin demonstrating
greater replication across studies than has often been
the case to date.
Measuring the Same Constructs Across
Groups
Perhaps the initial analytic task to undertake
when studying special populations is to determine
64 s p e c i a l p o p u l at i o n s
the meaning and the metrics of dimensions of indi-
vidual differences within and among populations
or subpopulations. Researchers typically assess indi-
vidual differences on key dimensions in a domain
in their research projects. For example, a study
on mental abilities may include measures of fluid
and crystallized intelligence, and a study of per-
sonality is likely to assess the Big Five dimensions
of Extraversion, Agreeableness, Conscientiousness,
Neuroticism, and Openness. Further, the listing
of certain dimensions of intelligence and person-
ality represents only a meager sampling of the many
different types of characteristics we assess in our
work as researchers in psychology. However, using
a standard measure developed and normed on the
general population to assess individuals from a spe-
cial population is fraught with problems, and a
researcher cannot assume that the individual differ-
ences reflected in scores on the measure are directly
comparable across populations.
Consider the use of a widely employed measur-
ing device in a special population. For example, a
researcher might want to assess dimensions of per-
sonality in a large sample of persons with intellectual
disability. The researcher may intend to determine
whether persons with intellectual disability have dif-
ferent mean levels on the personality dimensions
(e.g., higher levels of Agreeableness) and whether
individual differences on the personality traits relate
in meaningful ways to success in community place-
ments. To do so, the research might select the Big
Five Inventory (BFI) (John, Donahue, & Kentle,
1991), a 44-item measure of the major dimensions
of personality that contains 8 to 10 items for each of
the five dimensions. On each item, the respondent
is asked to indicate, using a 1-to-5 rating scale, his
or her degree of agreement with certain adjectives
describing personal behaviors or descriptions.
In a situation like this, a researcher may blithely
assume that responses by persons with intellectual
disability can easily be compared at the scale level
with responses by persons who do not have intel-
lectual disability. Thus, one might sum up the
item scores on the Extroversion scale and com-
pare mean and variance differences on scale scores
between relevant samples of persons with and with-
out intellectual disability. However, psychometric
investigations over the past 50 years and more have
demonstrated that incorrect conclusions may be
drawn in such situations unless one is confident
that one is assessing “the same constructs” across
the different populations. Methods of verifying that
one is assessing “the same constructs” across groups
have been published under several rubrics, including
methods to assess measurement invariance, factorial
invariance, or lack of differential item function-
ing (DIF). The upshot of this concern leads to
an important implication when researching special
populations:
Implication 2: Establishing the measurement
invariance of instruments across groups is a key result
in the investigation of any special population.
Measurement invariance must hold for meaningful
comparisons to be made across samples from
different populations.
Measurement invariance is a broad topic, one
that subsumes research and statistical models that
cross many boundaries. For example, some work
on measurement invariance has looked at predic-
tion models, which may be used to predict some
behavioral outcome (e.g., college grade point aver-
age) from relevant predictors (e.g., high school grade
point average, admission test scores). Multiple mod-
els can be employed in such research, and regression
analysis is often used. In regression analysis, ques-
tions arise regarding any potential intercept bias
or slope bias across groups when predicting the
outcome.
When measurement invariance is considered
within the context of factor analysis models, the
term applied is usually factorial invariance, which is
a restricted form of measurement invariance (Mered-
ith, 1993). In factor models, we typically begin with
a data model of the form:
Yji = τj + λj1 η1i + · · · + λjr ηri + εji , (1)
where Yji is the score of person i(i = 1, . . . , N ) on
manifest variable j(j = 1, . . . , p), τj is the intercept
for manifest variable j, λjk is the factor loading (or
regression weight) for predicting manifest variable j
from latent variable k(k = 1, . . . , r), ηki is the factor
score for person i on latent variable k, and εji is the
score of person i on the unique factor for manifest
variable j. The model in Equation 1 is termed the
linear factor analysis data model to signify the fact
that the linear model was developed as a model for
understanding persons’ scores on manifest variables
as linear functions of their scores on latent variables.
Writing Equation 1 in matrix notation results in:
Y = τ + Λη + ε, (2)
 
where Y is a p × 1 vector of scores  for person i
on the p manifest variables, τ is a p × 1 vector of
intercepts for the p manifest variables, Λ is a p × r
matrix of loadings of the p manifest variables on the
r latent variables, η is an (r × 1) vector of scores
 for
person i on the r latent variables, and ε is a p × 1
vector of scores of person i on the p unique factors.
One can use the model in Equation 2 to develop
moment expectations for the manifest variables,
moment expectations that yield expressions for the
covariance structure and the mean structure of the
manifest variables. In a single-group case, these
expectations are:
Σ = ΛΨ + Θ (3a)
μ = τ + Λα, (3b)
 
where Σ is the p × p matrix of population covari-
ances among manifest variables, Ψ is the (r × r)
matrixof covariances
 among the latent variables, Θ
is the p × p matrix (usually diagonal)
 of covari-
ances among unique factors, μ is a p × 1 vector
of population means on manifest variables, α is
an (r × 1) vector of means on the latent variables,
and other symbols were defined above. Equation 3a
is the population covariance structure model, and
Equation 3b is the population mean structure
model.
In any sample from a population, we observe
sample covariances among manifest variables, which
we signify as S, and sample means on the manifest
variables, which we can signify as Ȳ. Given these
sample estimators of population values, we can write
the sample covariance and mean models as:
S∼
= Λ̂Ψ̂Λ̂ + Θ̂=Σ̂ (4a)
Ȳ ∼
= μ̂ = τ̂ + Λ̂α̂, (4b)
where carets (ˆ) are added to matrices to indicate that
sample estimates of population parameters are con-
tained in the matrices, and all symbols are defined
above.
The covariance structure model in Equation
4a signifies that the matrix of sample covariances
among manifest variables S is approximated by the
covariance structure model, Λ̂Ψ̂Λ̂ +Θ̂; with esti-
mates in the three parameter matrices Λ̂, Ψ̂, andΘ̂,
the matrix expression yields an estimate of the pop-
ulation covariances among manifest variables, Σ̂,
under the assumption that the model is correct
in the population. The mean structure model in
Equation 4b shows that the sample means are esti-
mators of population means, and these are approx-
imated as a function of intercepts, factor loadings,
and means of the latent variables.
widaman, early, conger 65
 If we generalize the model in Equations 4a and
4b to the multiple-group context, we arrive at:
∼ Λ̂g Ψ̂g Λ̂ g + Θ̂g = Σ̂g
Sg = (5a)
Ȳg ∼
= μ̂g =τ̂g + Λ̂g α̂g , (5b)
where the subscript g (g = 1, . . . , G) has been
added to each matrix or vector to indicate that the
elements of equations are derived from group g , and
all other symbols are defined above.
Levels of factorial invariance. Given the multiple-
group model shown in Equations 5a and 5b, con-
sideration of factorial invariance can commence. A
simple rendition of factorial invariance is this: The
same factors should be present in multiple groups.
But, this simple statement masks key issues. How
can we tell if the same factors are present in differ-
ent groups? What empirical results would give us
confidence that we have identified the same factors
in different groups?
One way to get a bit more definite about how
to verify that invariant factors have been identified
in multiple groups is to consider a mathematical
statement regarding the expectations of the
 mani-
fest variables, which can be written as E Y|η, g .
This equation states that the expected values of the
manifest variables in Y are a function of the com-
mon latent variables in η and
 the group
 g to which
a person belongs. Now, if E Y|η, g = E (Y|η), or
if the expectations of the manifest variables given η
and g equal the expectations of the manifest variables
given just η, then the expectations are not depen-
dent on the group of which a person is a member.
If this identity holds, then the same latent variables
are present in the different groups.
To translate the expectation equations into impli-
cations regarding data, consider the factor model
shown in Equations
 5aand 5b above. The expecta-
tion equality E Y|η, g = E (Y|η) will hold only
if the common factors are “translated” into manifest
variable scores in the same fashion in each group.
This requirement has led researchers to discuss sev-
eral levels of factorial invariance. Many researchers
(e.g., Byrne, Shavelson, & Muthén, 1989; Chen,
Sousa, & West, 2005; Cheung & Rensvold, 1999;
Ferrer, Balluerka, & Widaman, 2008; Hancock,
Kuo, & Lawrence, 2001; Little, 1997; McArdle,
1988; Meredith & Horn, 2001; Millsap & Mered-
ith, 2007; Nesselroade, 1983; Rensvold & Cheung,
1998) have written on the topic of factorial invari-
ance, and the preceding listing of contributions
merely scratches the surface of work in this area and
therefore necessarily misses many contributions that
should also be cited.
66 s p e c i a l p o p u l at i o n s
In our presentation here, we will follow the sum-
mary of approaches to factorial invariance research
provided by Widaman and Reise (1997), who syn-
thesized prior work by Jöreskog (1971), Horn,
McArdle, and Mason (1983), and Meredith (1993)
to arrive at four levels of factorial invariance.
These four levels constitute levels of increasing
restriction on parameters of the factor analysis
model.
Horn et al. (1983) termed the first level of invari-
ance configural invariance. By configural invariance,
we mean that the same pattern of fixed and free
loadings in Λ is observed in each group. This form
of invariance states merely that within each group,
each manifest variable is predicted by the same latent
variable(s) as occurs in other groups. Under config-
ural invariance, the g subscript is still employed for
each Λ matrix, because different estimates can be
found for particular factor loadings across groups
and the stipulation of configural invariance will still
be satisfied.
The second level of factorial invariance was
termed weak factorial invariance by Widaman and
Reise (1997) and refers to a model in which factor
loadings are constrained to be invariant, or iden-
tical, on a one-by-one basis across groups. Thus,
not only must the factor loadings display the same
pattern of fixed and free loadings across groups, but
the free loadings across groups are constrained to
invariance. If the invariance constraint on factor
loadings is supported, then regression weights for
predicting manifest variables from latent variables
are invariant across groups. This equality is one key
element in showing that latent variables are trans-
lated into manifest variables in the same fashion.
If this holds and the factor loading matrices can
be constrained to invariance across groups, the g
subscript can be deleted from the loading matrices,
leading to:
Sg ∼
= Λ̂Ψ̂g Λ̂ + Θ̂g = Σ̂g (6a)
Ȳg ∼
= μ̂g = τ̂g + Λ̂α̂g , (6b)
where all symbols in Equations 6a and 6b are
defined above.
The third level of invariance, strong factorial
invariance, adds invariance across groups of the
measurement intercepts to the weak factorial invari-
ance model. Thus, under strong factorial invariance,
the regression equation for predicting each manifest
variable from the set of latent variables is invariant
across groups, satisfying a key criterion for measure-
ment invariance—that the latent variables related to
manifest variables in the same fashion across groups.
The resulting equations are:
Sg ∼
= Λ̂Ψ̂g Λ̂ + Θ̂g = Σ̂g (7a)
Ȳg ∼
= μ̂g =τ̂ + Λ̂α̂g , (7b)
where all symbols are defined above. Note that both
the factor loading matrix and the vector of intercepts
have the g subscript deleted, because these matrices
have invariant estimates across groups. Two key out-
comes accompany strong factorial invariance: (1) all
differences across groups in the means on manifest vari-
ables are due to mean differences on the latent variables;
and (2) group differences in means and variances on
the latent variables are identified in a comparable met-
ric across groups, enabling comparisons at the latent
variable level across groups. Thus, the latent variable
model in Equations 7a and 7b identifies the latent
variables as the sources of differences across groups in
mean levels on the manifest variables, and variances
on latent variables can also be compared.
The fourth and most restricted level of invari-
ance is strict factorial invariance. Under this level
of invariance, the unique factor variances are addi-
tionally constrained to invariance across groups. The
resulting equations for the covariance structure and
mean structure, respectively, are:
Sg ∼
= Λ̂Ψ̂g Λ̂ + Θ̂ = Σ̂g (8a)
Ȳg ∼
= μ̂g =τ̂ + Λ̂α̂g , (8b)
where all symbols are defined above. As shown in
Equation 8a, the g subscript is deleted from the
unique factor covariance matrix Θ̂ because esti-
mates in this matrix are invariant across groups.
Under strict factorial invariance, all group differ-
ences in mean levels and in variances on the manifest
variables are due to mean and variance differences,
respectively, in the latent variables. Thus, under strict
factorial invariance, we have a concise representa-
tion of all between-group differences on manifest
variables. Although strict factorial invariance is the
most concise of the levels of factorial invariance,
researchers often find that equality constraints on
unique factor variances across groups are too restric-
tive. This is not a problem, because comparisons
across groups on the latent variables are justified if
strong factorial invariance holds.
Representing mean and variance/covariance dif-
ferences across groups. If at least strong factorial
invariance is satisfied for a set of data, then latent
variables are identified in a form that allows inves-
tigation of group differences on the latent variables.
These differences across groups are contained in
particular matrices in Equations 8a and 8b. Specif-
ically, group differences in mean levels are obtained
from the α̂g matrices in Equation 8b. Models are
often identified by fixing factor means to zero in one
group (e.g., group 1, which serves as the reference
group), so mean values for the other groups (e.g.,
groups 2, . . . , G) are estimated as mean differences
from the reference group.
Group differences in variance on the latent vari-
ables or covariances among latent variables are
obtained from the Ψ̂g matrices in Equation 8a. If
latent variables are identified by fixing latent vari-
able variances to unity in the reference group (e.g.,
group 1), then variances on the latent variables are
estimated relative to this reference group.
In summary, if strong factorial invariance holds
for a set of data, latent variables are identified
in an invariant fashion across groups. Given this,
between-group comparisons on mean and/or vari-
ance on the latent variables are justified, and these
comparisons can be made at the error-free, latent
variable level.
Exploring the Bounds of Psychological
Theories
As discussed above, obtaining strong factorial
invariance allows the researcher to assume that
mean and/or variance differences on dimensions of
individual difference are interpretable across popu-
lations. However, psychological theories frequently
lead to predicted relations among constructs. For
example, many researchers have sought to outline
dimensions of parenting styles, and the isolation and
replication of such dimensions—particularly across
groups (e.g., across ethnic groups)—is an impor-
tant matter. But, once these initial steps have been
taken, researchers are typically interested in whether
parenting styles have impacts on, or at least con-
sistent patterns of asymmetric relations with, other
variables, such as child behavior. To pursue such
research, we must focus on the relations among
latent variables, determining whether the patterns
and strength of relations among latent variables is
similar or different across groups.
Implication 3: Results of studies of the general
population should not be generalized to special
populations without research that supports such
generalization. Investigations of the structural
relations among latent variables across populations
hold the key to determining whether conclusions
regarding relations among variables generalize to
special populations.
widaman, early, conger 67
In the structural modeling literature, researchers Equation 9 for η leads to:
often distinguish between the measurement model
η = (I − B)−1 ζ, (10)
and the structural model. The measurement model
consists of the relations of the latent variables to where I is an (r × r) identify matrix, the superscript
the manifest variables, and the structural model −1 indicates the inverse of the associated matrix,
contains relations among the latent variables. In and all other symbols were defined above.
the foregoing section, we described several levels of The covariance expectations for Equation 10 are:
factorial invariance—such as weak and strong facto-    −1
rial invariance—and these concerned invariance of E ηη = (I − B)−1 Ψ I − B , (11)
parameter estimates in the measurement model. where all symbols are defined above. To ease inter-
Once at least strong factorial invariance is estab- pretation of two parameter matrices, we will dis-
lished, we may pursue other forms of invariance tinguish between independent latent variables and
that are of great importance for generalizing the- dependent latent variables, using superscript (i) and
oretical conclusions across groups. These additional (d ), respectively. Thus, we will write the matrix of
forms of invariance fall under the rubric we are covariances among latent variables as:
calling structural invariance, because they involve  (i) 
invariance of the parameter estimates in the struc- Ψ 0
Ψ= (12)
tural model. A great deal of work has been done 0 Ψ(d )
on measurement or factorial invariance, but rela-
and the matrix of factor means as
tively little research has been published under the  (i) 
heading of structural invariance. To document this, α
α= , (13)
PsycINFO searches were made with “measurement α(d )
invariance” and “factorial invariance” as keywords;
these searches returned 412 and 402 citations, where Ψ(i) contains free covariances among the
respectively. A search with “structural invariance” as independent latent variables, Ψ(d ) contains covari-
keyword returned only 101 citations, a much lower ances among residuals of the dependent latent vari-
number. ables, the independent latent variables are assumed
In addition, the term structural invariance to be uncorrelated with the residuals of the depen-
seemed to be used in a much less consistent, more dent latent variables, α(i) is a vector of means of
confused fashion in prior research. Many authors the independent latent variables, α(d ) is a vector
used the term “structural invariance” synonymously of intercepts of the dependent latent variables, and
with what we have called factorial invariance, other symbols are as defined above.
arguing that satisfying tests of factorial invariance Placing Equations 11, 12, and 13 into Equations
implied that the factors were structurally invari- 7a and 7b for a multiple-group version of the strong
ant across groups. When using the term “structural factorial invariance model yields:

invariance” to refer to elements in the structural  −1 Ψ̂(i) 0
model, “structural invariance” has typically been Sg =∼ Λ̂ I − B̂g g
0 Ψ̂(d
g
)
interpreted as denoting the invariance of the pattern
of significant and nonsignificant directed relations  −1
among latent variables. I − B̂ g Λ̂ + Θ̂g = Σ̂g (14a)
To discuss levels of structural invariance, we need 
α̂g(i)
to revise slightly the structural model shown in Ȳg ∼
= μ̂g = τ̂ + Λ̂ , (14b)
Equations 8a and 8b. If we return to Equation 2 α̂g(d )
and allow directed relations among latent vari- where g subscripts on vectors or matrices indicate
ables, we can write an equation for the latent the presence of differing parameter estimates across
variables as: groups, and all symbols are defined above.
Levels of structural invariance. Here, we propose
η = Bη + ζ, (9)
that different levels of invariance can be distin-
guished for the structural model, levels that are
where B is an (r × r) matrix of regression weights analogous to the levels of factorial invariance. In
for predicting latent variables from other latent offering this proposal, we hope we can lead to
variables, ζ is an (r × 1) vector of latent variable research on structural invariance that is as illumi-
residuals, and other terms are defined above. Solving nating as work on factorial invariance and, in the

68 s p e c i a l p o p u l at i o n s
process, help establish a common nomenclature for
discussing structural invariance.
The first and most basic form of structural invari-
ance is configural structural invariance of the pattern
of fixed and free regression weights in the B̂g matrices
in Equation 14a. If a restricted pattern of directed
paths is estimated from independent latent variables
to dependent latent variables and among depen-
dent latent variables, configural structural invariance
implies that the same latent variables have directed
effects on the same outcome latent variables in each
group.
The second level of structural invariance is invari-
ance of the regression weight estimates for predicting
certain latent η variables from other latent variables.
Paralleling distinctions made above for factorial
invariance, placing invariance on the regression
weights leads to what may be called weak structural
invariance. These regression weights are contained
in the B̂g matrices in Equation 14a. If parame-
ter estimates in the B̂g matrices are constrained to
invariance across groups, the g subscript would be
dropped from these matrices, leading to


−1 Ψ̂(i)
0
Sg ∼
= Λ̂ I − B̂
g constraints on these latent intercepts leads to strong
0 Ψ̂(d
g
)
 −1
I − B̂ Λ̂ + Θ̂g = Σ̂g , (15)
where all terms are defined above, and the lack of
subscripts on the B̂ matrices indicates that across-
groups invariance constraints have been imposed on
parameter estimates in this matrix.
At least two issues must be mentioned with regard
to constraints on regression weights in the B̂ matri-
ces. First, across-group constraints on the B̂ matrices
are interpretable only if weak or strong factorial
invariance has been established. Thus, if only con-
figural factorial invariance holds for a given set of
data, no clear and convincing substantive interpre-
tation can be placed on constraints on elements
in the B̂ matrices. Data must support at least the
hypothesis of weak factorial invariance to yield inter-
pretable constraints on the B̂ matrices, and still
stronger interpretations of such constraints accom-
pany the successful specification of strong factorial
invariance. For this reason, we placed the restricted
B̂ matrices in the strong factorial invariance model
in Equation 15, because strong factorial invariance
allows a more adequate basis for discussing these
constraints.
Second, if across-group constraints are imposed
on the B̂ matrices, the constraints should generally
be placed on raw score regression weights (or equiva-
lents of these), rather than standard score regression
weights. As in typical multiple regression analy-
ses, raw score regression weights are presumed to
be invariant across samples from a common pop-
ulation, whereas standardized regression weights
are expected to vary as a result of range restric-
tion resulting from sampling. Now, with latent
variable models, the scale of each latent variable
may be fixed in any of several ways, and the “raw
score regression weights” among the latent η vari-
ables will vary as a result. However, if the scale
of each latent variable is fixed in one group and
strong factorial invariance constraints are placed on
the Λ̂ and τ̂ matrices, then the latent η variables
are on the same scale across groups. As a result,
the regression weights in the B̂ matrices are anal-
ogous to raw score regression weights and are on
a comparable metric, so invariance constraints on
these weights are reasonable a priori hypotheses
to test.
A third form of structural invariance involves
the intercepts for the dependent latent variables,
contained in the αg(d ) matrices. Placing invariance
structural invariance. If across-group constraints on
latent intercepts are imposed, then the resulting
equation for mean expectations has the form:
 
α̂g(i)
Ȳg ∼
= μ̂g = τ̂ + Λ̂ , (16)
α̂(d )
where all terms are defined above, and the lack of
subscripts on the α(d ) matrix indicates the presence
of cross-group invariance constraints on parameter
estimates in this matrix.
More importantly, if these latent intercept terms
are constrained to invariance across groups with
no significant loss in model fit, then any group
differences in mean level on the dependent latent
variables result from group differences in mean level
on the independent latent variables. This condition
is analogous to the distinction at the measured vari-
able level that distinguishes weak factorial invariance
from strong factorial invariance; hence, our dis-
tinction here between weak and strong structural
invariance.
The fourth and final type of structural invariance
to be represented and tested involves the residual
covariances among the dependent latent variables,
contained in the Ψ̂(d g
) matrices. If these residual
covariances are constrained to invariance, the result-
ing model is characterized as conforming to strict
widaman, early, conger 69
structural invariance. Under this model, Equation 15
becomes:
 ulations is the frequent finding of unique forms of
 −1 Ψ̂(i) 0
Sg ∼
= Λ̂ I − B̂ g variability in the special populations. That is, rela-
0 Ψ̂(d )
 −1
I − B̂ Λ̂ + Θ̂g = Σ̂g , (17)
where all terms are defined above, and the lack of
subscripts on certain parameter matrices indicates
cross-group invariance constraints on parameter
estimates in these matrices.
As a summary of issues in structural invariance,
distinctions among levels of constraints similar to
those made for factorial invariance may be drawn.
Specifically, invariance constraints on the α̂(d ) and B̂
matrices are the most important for tests of sub-
stantive theory. Once cross-group constraints are
imposed on the α̂(d ) and B̂ matrices, identical raw
score regression models—both the intercepts and
regression weights—hold in each group at the latent
variable level. Invariance of the α̂(d ) and B̂ matrices
is a reasonable a priori hypothesis; if sustained, then
the lack of group differences at this level is an impor-
tant finding. By comparison, additional constraints
on the Ψ̂(d )
g matrices are nice but not necessary. In
fact, there are reasonable bases for expecting that the
Ψ̂(d )
g matrices will vary significantly across groups
under sampling from a population, although the
α̂(d ) and B̂ matrices may display invariance across
groups.
Finally, we reiterate our earlier statement that
invoking strong or strict structural invariance con-
straints makes no sense substantively unless at least
strong factorial invariance constraints have been
placed on the model. Only if invariant latent vari-
ables are identified in an identical metric across
samples does it make sense to test whether invariant
regression parameters hold among these latent vari-
ables. Moreover, if the α̂(d ) and B̂ are invariant across
the general population and specific special popula-
tions, then processes bringing about the dependent
latent variables are the same across populations,
allowing one to generalize theoretical conclusions
across groups. On the other hand, if the α̂(d ) and B̂
are not invariant across the general population and
special populations, then results from the general
population cannot be extended to the special popu-
lations, and theories regarding the nature of relations
among latent variables would require modifications
in connection with special populations.
70 s p e c i a l p o p u l at i o n s
Exploiting Unusual Variation in Special
Populations
A final issue that arises in the study of special pop-
tive to the general population, individuals in a given
special population may exhibit substantial variation
on key variables that does not exist in marked form
in the general population. For example, adaptive
behaviors are forms of behavior that enable one to
live independently in the community. Because of
the way that adaptive behaviors are measured, per-
sons with IQ scores that are at or above the mean
of the population may exhibit little variability on a
measure of adaptive behavior, as they score at the
highest level on every item. This is a quite reason-
able outcome, as persons with IQ scores at or above
the population mean do not have problems living
independently in the community. One tends to find
substantially more variability in ratings of adaptive
behavior in samples of persons with intellectual dis-
ability; again, this is a reasonable outcome, as many
persons with intellectual disability have difficulties
in one or more domains of adaptive functioning,
limiting the quality of their independent living in
the community. Of course, if a special population is
a subset of the general population, then the marked
variation on key variables does exist in the general
population. But, if the special population is a rather
small subset of the population, then extreme vari-
ability on certain variables may be submerged in data
on the full population and therefore go unnoticed.
This issue of unusual variation within special
populations leads to our fourth methodological
implication of the presence of special populations:
Implication 4: Special populations may exhibit
unique forms of variability that can be exploited to
test theoretical conjectures in ways unavailable in the
general population.
We have not uncovered any particular method-
ological techniques that are unique to the issue of
unusual variation within special populations. In
the preceding sections, we outlined how quantita-
tive techniques could provide informative ways of
answering questions regarding measurement equiv-
alence and structural invariance. If unusual variation
were present on one or more variables in a special
population, then this unusual variation could well
be uncovered in particular matrices in the general
structural equation model we described.
Here, we merely highlight the issue of poten-
tial unusual variation in a special population. In
emphasizing this issue, we encourage researchers to
be vigilant for the presence of unusual variation.
Unusual variation may take any of several forms. Per-
sons in a special population may exhibit variation on
dimensions on which members of the general pop-
ulation may show no variability at all. An example
of this is special artistic abilities, such as individual
differences in composing symphonies or directing
movies. No one doubts the presence of individual
differences among composers and movie directors,
and preference for certain music or movies is often
a lively topic of discussion. But, we rarely reflect
on the fact that relatively few individuals compose
symphonies or direct movies, so no individual dif-
ferences on these dimensions are evident outside the
small group of persons who pursue these distinctive
endeavors.
Alternately, members of special populations
may exhibit unusually large or small variation on
behavioral traits relative to the general population.
Persons in a special population may show a particu-
lar symptomatic behavior and so exhibit essentially
no variation in this behavior, which may fall at one
end of a behavioral continuum on which the general
population exhibits wide individual variability. For
example, persons with Down syndrome often have
facial features, including small chin, round face,
almond eye shape, and so forth, that are an unusual
combination in the general population. Thus, the
common facial appearance of persons with Down
syndrome, which is distinct from other members of
the population, is an early signal that the individual
is a member of this special population. Or, per-
sons in a special population may exhibit enhanced
or enlarged variability on a dimension relative to
the general population. Regardless of the form of
unusual variability, such variability may offer unique
ways to understand differences across groups.
Examples of Quantitative Explorations of
Special Populations
Having outlined various methodological issues
and approaches that can be used to characterize spe-
cial populations, we now describe empirical studies
that embody each of the methodological issues that
we have discussed. The approaches we have taken
and the results obtained can be extended readily by
other researchers to the study of special populations.
Identifying and Accessing Participants in
Special Populations
One example of the problems that arise when
recruiting members of a special population was
reported by Nary, Froehlich-Grobe, and Aaronson
(2011), who sought to recruit wheelchair users into
a randomized controlled exercise trial. Nary et al.
set out to recruit 180 participants for their study, or
60 persons in each of three cohorts. The research
team used a wide variety of recruiting strategies,
beginning with the traditional approach of con-
tacting hospitals, health-care organizations, health-
care providers, and disability agencies. When these
sources of recruitment led to discouraging results,
Nary et al. began using more innovative approaches,
including putting flyers in movie theaters and public
transportation (e.g., busses), advertising in news-
papers, newsletters, and so forth, and employing
direct mail coupon packets (identified as ValPak).
These recruitment efforts were extremely varied, as
Nary et al. listed approximately 30 different loca-
tions/activities associated with their recruitment.
When participants were recruited, they were asked
where they heard about the project. Somewhat sur-
prisingly, the single most effective method was the
ValPak direct mail coupon packet and other contacts
through the media (e.g., radio and TV advertise-
ments, newspaper advertisements); a full third of
the sample of participants was recruited through the
media contacts. Although the research team did not
fully meet their recruitment goal of 180 participants,
they did come close to the goal—and only did so by
employing a wide array of approaches that cost much
more in time and effort than the research team had
anticipated. The Nary et al. paper is an entertaining,
if sobering, accounting of the myriad approaches the
researchers took to recruit participants and the rel-
ative value of the different approaches. Nary et al.
have summarized their experiences in a series of five
“lessons learned,” such as needing to be cognizant
that recruiting of persons with disabilities will be
more difficult than recruiting persons without dis-
abilities and that more time, effort, and money will
be required in recruiting members of special popu-
lations than initially thought. The most unfortunate
lesson learned was the fourth lesson, which related to
the research team finding that health-care agencies
were not especially helpful in recruiting participants,
although the special population of wheelchair users
has, in general, greater reliance on such agencies.
The Nary et al. paper is a wonderful place to start
when planning to recruit members of populations
with disabilities, and learning from the efforts of
these researchers might ease recruitment in future
studies.
A second example of ways researchers have dealt
with identifying and accessing participants in a
widaman, early, conger 71
special population comes from our California Fam-
ilies Project (CFP), a study of 674 Mexican-origin
families in California (e.g., Conger, Song, Stock-
dale, Ferrer, Widaman, & Cauce, 2012; Conger,
Stockdale, Song, Robins, & Widaman, in press).
We worked through the school districts in two cities
in Northern California to identify potential mem-
bers of our Mexican-origin population, which we
defined as the population of fifth grade public or
Catholic school children whose four grandparents
were born in Mexico or whose forebears were born
in Mexico. Children were drawn at random from the
student rosters for the school districts of these two
cities. Families of these children were then recruited
by telephone or, for cases in which there was no listed
phone number, by a recruiter who went to their
home. Of the 982 families contacted, 68.6% of these
eligible families (N = 674) agreed to participate
in the study. All family members were of Mexican
origin as determined by their ancestry and their self-
identification as being of Mexican heritage. First-,
second-, and third-generation children of Mexican
origin were eligible for the study. Also, the focal child
had to be living with his or her biological mother.
Either two-parent (82% of the sample) or single-
parent (18% of the sample) families were eligible to
participate. In two-parent families, the father had to
be the child’s biological father.
In addition to the initial recruitment, we face
challenges in tracking our families over time, espe-
cially given high mobility of this population during
times of economic downturn and resulting eco-
nomic pressure on families. We are currently in
the sixth year of assessing families in our longi-
tudinal study and are happy to report that we
have a retention rate over 90% for the families
in our study. This has taken a number of special
approaches to retaining families, including going
door-to-door if attempts to contact a family by
phone are unsuccessful, contacting persons who
know a family (especially if the family has moved),
and following families for assessments as they move
to other states (e.g., Texas, Arizona) or back to Mex-
ico. In the future, we should document all of the
different strategies we have used to keep contact
with our sample. An article detailing the different
approaches and their relative success would provide
a useful comparison to the recruitment problems
confronted by Nary et al. (2011) in their study
of wheelchair users. Only by keeping recruitment
and retention at the highest possible levels will the
data generated by our study and by others (e.g.,
Nary et al., 2011) be optimal for drawing the
72 s p e c i a l p o p u l at i o n s
conclusions we wish to draw. Optimal recruitment
and retention is an unexpectedly time-consuming
and expensive proposition, and we have our own
list of lessons learned in this necessary, thankless
task.
Measuring the Same Constructs Across
Groups
Economic pressure and depression. As one exam-
ple of measuring the same constructs across groups,
we present an example of measurement invari-
ance across groups using data from three ethnically
diverse samples living in the United States. Data
for these samples were obtained from the follow-
ing studies: (1) The Family and Community Health
Study (FACHS), a study of 889 African-American
children and their families; (2) the CFP, a study
of 674 Mexican-origin families and children; and
(3) the Family Transitions Project (FTP), a study
of 550 European American children and families.
For these analyses, we focused our attention on
the impact of economic pressure on husbands’ and
wives’ depressive symptoms. Because we were inter-
ested in impacts on two-parent families, all three
samples were reduced in number, yielding 300 fam-
ilies from the FACHS study, 482 families from
the CFP study, and 281 families from the IYFP
study, for a total of 1063 families across the three
groups.
The confirmatory factor model fit to the data
for the three samples contained seven manifest vari-
ables and three latent variables and is illustrated in
Figure 4.1. In the figure, standard figural notation
is used: Triangles denote the unit constant used
to estimate means (αk ) or intercepts (τj ), squares
or rectangles represent manifest variables, circles
or ellipses stand for latent variables, single-headed
arrows reflect directed effects (e.g., factor loadings
λj , regression weights), and double-headed arrows
represent latent variable (ψkk ) and unique factor (θj )
variances or covariances (ψkk  ).
In Figure 4.1, three latent variables are shown:
Economic Pressure, Husband Depression, and Wife
Depression. All three studies included the same
three indicators of Economic Pressure: (1) Unmet
Material Needs (U), the average of 4 items assessing
unmet material needs in different domains, includ-
ing “having enough money to afford the kind
of home, clothing and food your family needs”
(response scale 1 = “strongly agree” to 4 = “strongly
disagree”); (2) Can’t Make Ends Meet (C), the
average of 2 items measuring difficulties in having
money to cover expenses, including “difficulty pay-
ing bills” and “the amount of money left at the end of
the month” (response scale: 1 = “no difficulty at all”
to 4 = “a great deal of difficulty”); and (3) Financial
Cutbacks (F), the number of 11 areas in which the
family had to make cutbacks, including “the family
postponing major household purchases, changing
residence, and or eliminating medical insurance”
(each area scored dichotomously, 0 = “no cut-
backs” and 1 = “cutbacks”). Husbands and wives
provided answers on the preceding scales, and aver-
age responses across husbands and wives were used
as analyzed scores. For families in which husbands
did not participate, scores on the economic pres-
sure variables were treated as missing data. Scores
on all three indicators were reverse-scored, when
necessary, so that higher scores indicated greater
economic pressure on the family.
Across the three ethnic groups, the depression
latent construct was obtained from the Mini-Mood
and Anxiety Questionnaire (Mini-MASQ; Clark
& Watson,1997). For this analysis, we used the
five items that measure general distress—depression.
These items ask both husband and wife to self-report
on how much they had felt depressed, discour-
aged, and or worthless in the past week. Each
item was answered on a scale ranging from 1 =
“not at all” to 4 = “very much.” Two parcels (one
two-item and one three-item parcel) were formed,
and these are denoted M1H, M2H, M1W, and
M2W for parcels 1 and 2 from husband and wife,
respectively.
α1
1
ψ21
ψ11 Economic
ψ22
Pressure
λ1 λ2 λ3 λ4
U C F M1H
θ1 θ2 θ3 θ4
τ1 τ2 τ3 τ4
Figure 4.1 A figural presentation of the relations between latent variables for Economic Pressure, Husband Depression and Wife
Depression and their respective indicators (Note: triangle stands for the unit constant, squares for manifest variables, circles or ellipses
for latent variables, single-headed arrows for direct effects, and double-headed arrows for symmetric effects such as variances or
covariances)
Assessing factorial invariance across groups. We
then fit a series of three-group confirmatory fac-
tor models to test factorial invariance of the model
across the three samples. The first model, Model
1, was a three-factor model that had the same pat-
tern of fixed and free factor loadings in each group.
This configural invariance model fit the data well,
χ 2 (33, N = 1063) = 41.99, with RMSEA of
.028, and CFI and TLI values of 0.996 and 0.992,
respectively.
Next, we imposed invariance constraints on fac-
tor loadings in the Λ matrix across gender and
ethnicity. For example, the loadings for the two indi-
cators of husband depression were constrained to be
equal to the loadings for the two indicators of wife
depression within each sample; these loading were
then constrained to equality across all three samples.
The fit of the resulting model, Model 2, was slightly
worse; however, indicators of model fit remained
in an acceptable range, χ 2 (42, N = 1063) =
61.25, p = 0.02, with RMSEA of 0.036, and CFI
and TLI scores of 0.991 and 0.987, respectively.
The latter indices of practical fit suggest that the
invariance constraints imposed on Model 2, which
represent weak factorial invariance, are appropri-
ate. In prior research, several authors have discussed
changes in TLI or CFI values that might be consid-
ered practically significant when comparing nested
models. Thus, Widaman (1985) considered a differ-
ence in the TLI of less than 0.01 when comparing
two models to be a practically unimportant differ-
ence in fit, and Cheung and Rensvold (1999) and
α2 α3
ψ 31
ψ 32
Husband Wife
ψ 33
Depression Depression
λ5 λ6 λ7
M2H M1W M2W
θ5 θ6 θ7
τ5 τ6 τ7
widaman, early, conger 73
Little, Card, Slegers, and Ledford (2007) argued
that the CFI must meet this criterion (i.e., a dif-
ference in CFI of 0.01 or more between models)
to be deemed a difference in fit worthy of note.
Using these standards, the changes in fit when
moving from the configural invariance model to
the weak factorial invariance model were small and
unimportant.
Because fit indices for Model 2 were acceptable,
we next invoked invariance constraints on the man-
ifest variable intercepts in τ, leading to Model 3.
Invariance constraints on all but one τ value in each
sample were acceptable; the τ value allowed to vary
freely across the three samples was the intercept for
the third economic pressure indicator. This partial
strong invariance model resulted in poorer statisti-
cal fit than prior models, χ 2 (49, N = 1063) =
95.29, p < 0.001, reflecting a significant worsen-
ing of fit relative to the weak invariance model,
χ 2 (7, N = 1063) = 34.04, p < 0.001. How-
ever, practical fit indices were only modestly worse,
with an RMSEA of 0.052, and CFI and TLI values
of 0.979 and 0.973, respectively. Despite having
worse fit that slightly exceeded the “change of 0.01”
criterion for practically significant difference in fit,
the practical fit indexes for Model 3 were still accept-
able (i.e., the CI for the RMSEA included 0.05, and
both TLI and CFI values were above 0.95), indi-
cating that the partial strong invariance constraints
applied were appropriate.
The final confirmatory model, Model 4, imposed
invariance constraints on the unique factor variances
in Θ and therefore corresponded to strict factorial
invariance. Model 4 resulted in relatively poor fit,
χ 2 (65, N = 1063) = 317.84, p < 0.001, with
RMSEA of 0.105, and CFI and TLI scores of 0.886
and 0.890, respectively. This represented a signifi-
cant worsening of both the statistical index of fit,
χ 2 (16, N = 1063) = 222.55, p < 0.001, and
practical indices of fit when compared to Model 3.
Based on the poor practical fit indexes of the strict
χ 2 (3, N = 1063) = 12.96, p = 0.01,
factorial invariance model, we accepted the partial
strong factorial invariance model (Model 3) as the
optimal model for the data.
Group differences in means and variances on latent
variables. Given the fit of the partial strong facto-
rial invariance model, mean and variance differences
across groups could be evaluated in an unbiased fash-
ion. For the latent variables of Economic Pressure
and Husband Depression, means were fixed at 0 and
χ 2 (4, N =
variances at unity for the FTP sample, and means
and variances were estimated for all other latent vari-
ables. In the FTP sample, wives had significantly
74 s p e c i a l p o p u l at i o n s
higher levels of depression (M = 0.40 [SE =
0.10], SD = 1.46) relative to their husbands. The
FACHS sample had elevated means of 0.12 [SE =
0.10](SD = 1.03), 0.28 [SE = 0.13](SD =
1.37), and 0.45 [SE = 0.12](SD = 1.27),
on the Economic Pressure, Husband Depression,
and Wife Depression latent variables, respectively,
and the CFP sample also showed higher means
on the three factors, respectively, of 0.60 [SE =
0.09](SD = 0.97), 0.36 [SE = 0.11](SD =
1.47), and 0.69 [SE = 0.11](SD = 1.52). Thus,
relative to European American families, African-
American and Mexican-origin families showed both
higher mean levels and greater variability on the
three factors assessed in these confirmatory factor
models.
Exploring Bounds of Psychological Theories
Ethnic differences in effects of economic pressure on
depression. We turn next to ways of exploring the
bounds of psychological theories. A substantial body
of prior research supports the conclusion that family
economic pressure has negative effects on husband
and wife depression. Much of this research has been
based on data from European American families,
and we wanted to see whether the effect of economic
pressure on husband and wife depression was similar
across ethnic groups.
To answer this question, we returned to the
three-group data used to test factorial invariance
but examined models that substituted direct effects
of economic pressure on husband and wife depres-
sion in place of the correlations estimated in prior
models. In the first of these models, we imposed
invariance constraints on the path coefficients lead-
ing from economic pressure to depression across
husbands and wives within each sample. This cross-
gender constraint resulted in a significant worsening
in fit compared to the partial strong invariance
model,
with moderately worse practical fit indexes, RMSEA
of 0.055, and CFI and TLI values of 0.975 and
0.969, respectively. Next, we freed the cross-gender
constraints, but imposed equality constraints on the
path coefficients of economic pressure on husband
depression across groups and on wife depression
across groups. This model also resulted in a sig-
nificant worsening of fit when compared to partial
strong factorial invariance model,
1063) = 19.65, p < 0.001, with worse practical
fit indexes, RMSEA of 0.057, and CFI and TLI
values of 0.972 and 0.967, respectively.
 In our final model, we imposed invariance con-
straints on the path coefficients for economic pres-
sure on husband and wife depression across the two
minority samples, allowing the corresponding coef-
ficients to be free for the European American sample.
This resulted in a nonsignificant change in model
fit when compared to the partial strong invariance
model,
χ 2 (2, N = 1063) = 1.52, p = 0.47. The However, contrary to research on the general popu-
practical fit indexes for this model were essentially
identical to those for the strong factorial invari-
ance model, so we chose this model as the optimal
representation of our data.
Parameter estimates in this model suggest that the
impact of economic pressure on the level of depres-
sion for minority couples differs from that for Euro-
pean American couples. Specifically, economic pres-
sure appears to have the strongest effect (β = 0.69)
on depression in minority husbands (both African-
American and Mexican-American husbands), and a
significantly weaker effect (β = 0.47) on depression
in minority wives. In contrast, economic pressure
had a stronger effect (β = 0.54) on European Amer-
ican wives than on European American husbands
(β = 0.23), who were least affected by economic
pressure.
Parenting styles and child behavior. A second
example of exploring the bounds of psychologi-
cal theories developed on the general population
is worthy of note. In this study, Widaman and
Borthwick-Duffy (April, 1990) reported results
from 109 families with a child with intellectual dis-
ability. Key findings on parenting styles, or styles of
parenting behaviors, by researchers such as Baum-
rind (1968, 1971, 1991; Baumrind, Larzelere, &
Owens, 2010) and Hoffman (1975, 1979, 1994)
supported the presence of several dimensions of par-
enting behavior. The general findings of this research
was that authoritative parenting (high control, plus
high nurturance/warmth) was associated with opti-
mal forms of child behavior and that authoritarian
parenting (high control and power assertion, plus
low nurturance and warmth) and permissive par-
enting (low control, plus high nurturance and
warmth) both led to less optimal levels of child
behavior.
Widaman and Borthwick-Duffy (April, 1990)
isolated a total of seven dimensions of par-
enting behavior in their study, including (1)
nurturance/warmth, (2) induction, (3) maturity
demands, (4) promoting autonomy, (5) firm con-
trol, (6) love withdrawal, and (7) power assertion.
Consistent with research on the general population,
Widaman and Borthwick-Duffy found that the
positive parenting behaviors of induction, maturity
demands, and promoting autonomy had the
strongest, positive effects on longitudinal changes
in different forms of adaptive functioning, such as
practical skills (e.g., dressing oneself ), conceptual
competence (e.g., functional academics), and social
competence (e.g., making and keeping friends).
lation, Widaman and Borthwick-Duffy found that
the authoritarian dimension of power assertion
consistently had the strongest effects on longitu-
dinal changes in negative behaviors such as social
maladaption (e.g., aggression, property destruc-
tion) and personal maladaption (e.g., self-injurious
behavior). Although the standardized regression
weights were not large (βs ranged from 0.15 to
0.24), power assertion was the only parenting style to
impact changes in maladaptive behaviors. Thus, the
proscription of power assertion as a less useful form
of parenting behavior was based on research on fam-
ilies in the general population. However, to reduce
the levels of maladaptive behaviors in children with
intellectual disability, power assertion appears to be
the only viable parenting option.
Exploiting Unusual Variability in Special
Populations
Adaptive behavior in persons with intellectual dis-
ability. The final analytic issue to illustrate is the
exploiting of unusual variability in special popula-
tions. One example of unusual variability in special
populations involves the domain of adaptive behav-
ior, which attains significance in the population
of persons with intellectual disability. The three
key dimensions of adaptive behavior are practical
competence (or independent living skills), concep-
tual competence (or cognitive skills), and social
competence (or skills); all three dimensions repre-
sent everyday behaviors that enable a person to live
independently in the community.
Widaman, Gibbs, and Geary (1987) utilized a
database maintained by the Department of Devel-
opmental Services (DDS) of the State of Cali-
fornia. DDS required all persons receiving state
services for developmental disability to be assessed
on a 66-item instrument of adaptive behavior. The
instrument assessed the three key dimensions of
practical, conceptual, and social competence and
also assessed three additional dimensions of motor
competence, social maladaption, and personal mal-
adaption. Widaman et al. extracted 14 groups of
individuals based on a crossing of age (children,
widaman, early, conger 75
adolescents, and adults), levels of intellectual dis-
ability (mild, moderate, and severe), and placement
(home, community). Thus, these 14 groups are
special subgroups defined by demographic and func-
tional characteristics from the special population
of persons with intellectual disability. Widaman et
al. confirmed essentially identical six-factor solu-
tions across all 14 groups; interested readers are
referred to the published article for details. Of
importance here is the fact that, given the way the
66 items are phrased, persons who do not have
an intellectual disability would likely score at the
highest scale point on each item, therefore failing
to exhibit any variability in responses on the items.
Only persons with intellectual disability exhibit sub-
stantial variability on the items, so the dimensions
of adaptive behavior only have ready application in
this special population.
However, this research on adaptive behavior
offers a chance to make recommendations for future
research. The instrument developed by the Cal-
ifornia DDS contained items that exhibit much
variability only in samples of persons with develop-
mental disabilities, because persons without devel-
opmental disabilities would have ceiling effects on
all items. This does not mean, however, that persons
without developmental disabilities do not display
individual differences in adaptive forms of behav-
ior. More likely, the lack of variance in a sample
of persons without developmental disabilities is a
measurement issue or problem. This measurement
problem could be confronted by developing a set
of harder items for each dimension of adaptive
behavior, so that items would have a higher ceil-
ing and persons without developmental disabilities
might not “top out” on every item. If this were
done, then a computerized adaptive testing (CAT)
approach could be developed to administer a unique
set of items to each examinee, presenting items
that would enable precise measurement of the indi-
vidual’s standing on each dimension, although a
minimal number of items were needed to do so.
Waller and Reise (1989) described how to apply
the CAT approach to personality measurement, and
Reise and Waller (2003) discussed application of
sophisticated techniques to assessing psychopathol-
ogy. Similar approaches could undoubtedly be used
to assess adaptive behaviors across the entire popu-
lation in a more adequate way.
Effects of prenatal exposure to phenylalanine. A
second example of unusual variability in special
populations that provides a unique opportunity to
explore relations among variables is derived from
76 s p e c i a l p o p u l at i o n s
the Maternal PKU Collaborative (MPKUC) study.
Phenylketonuria is a well-known genetically based
disorder, which results in disrupted metabolism of
phenylalanine (PHE) into tyrosine. If left untreated,
infants with PKU who are normal at birth suffer
permanent brain damage that leads to severe mental
retardation (mean IQ of 50) by age 2 years. However,
with early identification and placement of infants on
a diet low in PHE, the negative effects of PKU can be
circumvented, and children with PKU can grow to
adulthood with no evidence of intellectual disability
(Koch & de la Cruz, 1999a, 1999b).
However, the story of maternal PKU is more
complex. If a woman with PKU does not maintain
a low-PHE diet when pregnant, then the increased
PHE in her blood crosses the placental barrier and
exposes the developing fetus to high levels of PHE.
For persons with PKU, levels of endogenous PHE
over 6 mg|dL have been found to be teratogenic,
which means that such levels lead to negative effects
on behavior. However, for the developing fetus, lev-
els of exogenous PHE (i.e., PHE from the mother)
that were sufficient to cause teratogenic effects had
never been identified. The MPKUC study (Koch, de
la Cruz, & Azen, 2003) began in 1984 as an inter-
vention study to help pregnant women maintain
low-PHE diets and therefore maintain low blood
PHE levels during pregnancy. A secondary goal of
the MPKUC study was to monitor maternal PHE
levels throughout pregnancy to study the relation
between PHE levels and child outcomes.
Table 4.1 shows descriptive statistics for partic-
ipants from the MPKUC study for three mother
variables (Full Scale IQ, PHE level when on an
unrestricted diet, and average PHE level during
pregnancy) and three child variables (Verbal, Perfor-
mance, and Full Scale IQ). Various forms of unusual
variability are contained in this table. First, all four of
the IQ variables exhibit substantial deviation from
the population mean: The mean mother IQ was
approximately a full standard deviation below the
population mean of 100, and the three mean child
IQ scores are more than one-half a standard devia-
tion below the population mean. Furthermore, the
three child IQ scores show markedly greater variabil-
ity than in the population, with SDs greater than
21 relative to a population SD of 15. The exces-
sive variability in child IQ scores is the result of a
larger-than-expected number of observations with
rather low IQ, presumably resulting from higher
levels of prenatal exposure to PHE. The indices of
mother PHE levels also exhibit unusual variability.
Persons who do not have the PKU genetic defect
 Table 4.1. Descriptive Statistics on Variables from the Maternal PKU
Study
Variable N Mean SD Min Max

Mother full-scale IQ 379 85.90 13.65 40 130

Mother PHE level on unrestricted diet 413 22.03 9.18 3.30 51.10

Mother PHE level during pregnancy 412 8.23 4.49 1.30 28.30

Child verbal IQ 7 years 284 92.06 22.39 40 142

Child performance IQ 7 years 285 92.00 21.86 40 133

Child full-scale IQ 7 years 284 91.35 23.21 35 139

Note: Mother full-scale IQ based on Wechsler Adult Intelligence Scale—Revised; Mother PHE
levels on regular diet and during pregnancy are in mg | dL; Child verbal, performance, and
full-scale IQ based on Wechsler Intelligence Scale for Children—Revised.

would exhibit extremely low PHE values, below 1.0
and near 0, on these two measures. In contrast to
this, mother PHE levels on an unrestricted diet var-
ied between 3.3 and 51.1, and mother PHE levels
during pregnancy varied between 1.3 and 28.3.
To model the effect of prenatal PHE exposure on
child Full Scale IQ at 7 years, one could use a linear
regression model. The result of fitting this model is
the following equation:
predicted IQ = 119.5 − 3.53 (PHE)
The standard errors for the intercept and regres-
sion slope were 2.13 and 0.24, respectively, so
interval estimates (95% CIs) for the two parameter
estimates were approximately 117.2 to 123.7 and
−3.05 to −4.00 for the intercept and slope, respec-
tively. The above equation represents the unrealistic
expectation that a child’s IQ at age 7 years would
be approximately 120 if his/her mother had main-
tained a PHE level of 0 during pregnancy, and that
IQ would drop about 3.5 points for every 1 mg|dL
increase above this value of 0.
An alternative regression model is a two-piece
linear spline, with three parameter estimates: an
intercept, a knot point that estimates the PHE expo-
sure at which a teratogenic effect begins to occur, and
the linear slope representing the teratogenic effect
after the knot point. The results for this model were:
predicted IQ = 103.9−4.14 (PHE), knot = 5.50.
In this equation, children who experienced pre-
natal PHE levels between 0 and 5.50 all had an
expected 7-year IQ of about 104, near the popu-
lation mean. Moreover, after the knot point of 5.50,
the predicted IQ drops over 4.1 IQ points for every
1 mg|dL increased. Because the standard errors for
the three coefficients were 1.70, 0.32, and 0.62, the
point estimates and interval estimates for the coef-
ficients are: intercept = 103.9, 95% CI [100.5,
107.3], regression slope for PHE = −4.14, 95%
CI [ −3.50, −4.78], and knot point = 5.50, 95%
CI [4.25, 6.75].
Exploiting the unusual variability of mother PHE
levels during pregnancy allowed us to reach several
important goals. First, we could verify the nature of
the teratogenic effect of prenatal exposure to PHE
on child cognitive outcomes, which is nonlinear
in nature with a threshold for teratogenic effects.
Second, we could estimate the level of exogenous
PHE at which the teratogenic effect begins to occur,
which is close to the level of endogenous PHE often
assumed to have teratogenic effects for persons with
PKU. Third, these results could be used as the basis
for recommendations for monitoring PHE levels
in mothers with PKU, attempting to ensure that
mothers with PKU keep their PHE levels below
the level at which teratogenic effects are likely to
occur. Additionally, all of these goals were accom-
plished only because of the unusual variability in
PHE levels exhibited by mothers with PKU in the
MPKUC study. Readers are referred to Widaman
(2009) for an informative summary of findings of
the MPKUC study.
Conclusions
The presence of special populations—however
delineated—provides opportunities beyond those
afforded by drawing repeated samples that are rep-
resentative of the general population. If we disre-
garded the presence of special populations, then we
might develop a corpus of scientific findings that

widaman, early, conger 77

applies to “the” population—whatever that is—but
in actuality applies to very few individuals or, at
least, fails to apply to members of important special
populations. The mere existence of special popula-
tions, however these are defined, challenges us to
verify that the most important of our findings are
not moderated substantially as a function of special
population status.
We outlined four primary methodological issues
that arise when investigating phenomena in the
presence of special populations. These four issues
involve identifying and assessing participants in spe-
cial populations, verifying that we are measuring the
same things across populations, discovering whether
sociobehavioral processes unfold in the same ways
across populations, and searching for unusual vari-
ability that might provide unique ways of viewing
behavioral phenomena and testing theoretical con-
jectures. Given the rather recent upsurge of research
pursued under the explicit rubric of special popula-
tions, the implications of special populations on the
way we do science will only increase in the future.
Furthermore, we expect that the recognition of the
presence of special populations will only enrich the
way we view social and behavioral science as we move
further into the twenty-first century.
Future Directions
Several future directions for research and the-
ory with regard to quantitative methodology as
applied to special populations can be drawn from
the material presented earlier in this chapter. These
are provided as a series of questions to guide future
research.
Question 1: What is the best way to conceptualize special
populations? Or, are alternative bases for identifying
special populations the best way to proceed?
Question 2: Is a “one-size-fits-all” conception of special
populations possible, or will the definition of special
populations vary as a function of the research
question asked?
Question 3: How should a researcher proceed if full
strong factorial invariance does not hold but only
partial strong invariance is exhibited by data? How
seriously does partial strong factorial invariance
impede scientific conclusions relative to full
invariance of all factor loadings and intercepts in the
strong factorial invariance model?
Question 4: How large must differences be across special
populations before differences are considered
important? Large sample sizes lead to increased power
to detect statistically significant differences across
78 s p e c i a l p o p u l at i o n s
groups, but how should we characterize the
magnitudes of effects, what magnitude of effects
should be considered practically important, and
would the magnitude of effects considered important
vary across domains of research?
Question 5: When does unusual variability represent a
valid measurement outcome, and when does unusual
(or different) variability across groups represent a
failure to assess individual differences adequately or in
comparable fashion across groups? Measurement is
the basis on which the whole enterprise of science is
erected, and concerted attention to accurate
measurement of individual differences across the
entire span of a dimension is crucial to answering
questions such as these. Use of the most up-to-date
measurement approaches, such as CAT, would go a
long way to resolving issues of differences in variance
across groups.
The future is bright for methodological and
quantitative innovations in the study of special pop-
ulations. As we documented at the start of this
chapter, labeling groups as special populations is
a relatively recent phenomenon, largely a product
of the past three decades. Moreover, the majority
of statistical methods and techniques for studying
differences across groups are also of recent origin.
Many of the most advanced techniques have been
available for, at most, 30 years, and new ways in
which these methods can be used to illuminate sim-
ilarities and differences across populations are being
developed on an almost daily basis. Improved, more
sophisticated understanding of the nature of special
populations is occurring at the genetic, biologi-
cal, and psychological/behavioral levels, and optimal
use of methodological approaches and quantitative
techniques is a crucial element that will push this
endeavor forward. Indeed, new quantitative tech-
niques or innovative use of existing techniques may
well be the key that unlocks the door to advanced
understanding of why special populations deserve
their distinctive status.
Author Note
Support for this work was provided by grants
from the National Institute of Child Health and
Human Development, the National Institute on
Drug Abuse, and the National Institute of Men-
tal Health (DA017902, HD047573, HD051746,
and MH051361).
Correspondence regarding this manuscript should
be addressed to Keith Widaman, Department
of Psychology, University of California at Davis,
One Shields Avenue, Davis, CA 95616. E-mail:
kfwidaman@ucdavis.edu
References
Barton, B. (2009). Dreams deferred: Disability definitions, data,
models, and perspectives. Journal of Sociology and Social
Welfare, 36(4), 13–24.
Baumrind, D. (1968). Authoritarian vs. authoritative parent
control. Adolescence, 3, 255–272.
Baumrind, D. (1971). Current patterns of parental authority.
Developmental Psychology, 4 (1, Pt. 2), 1–103.
Baumrind, D. (1991). The influence of parenting style on ado-
lescent competence and substance abuse. The Journal of Early
Adolescence, 11, 56–95.
Baumrind, D., Larzelere, R. E., & Owens, E. B. (2010). Effects
of preschool parents’ power assertive patterns and practices
on adolescent development. Parenting: Science and Practice,
10, 157–201.
Belsky, J., Bakermans-Kranenburg, M. J., van IJzendoorn, M. H.
(2007). For better and for worse: Differential susceptibility to
environmental influences. Current Directions in Psychological
Science, 16, 300–304.
Bronfenbrenner, U. (1977). Toward an experimental ecology of
human development. American Psychologist, 32, 513–531.
Bronfenbrenner, U. (1986a). Ecology of the family as a context
for human development: Research perspectives. Developmen-
tal Psychology, 22, 723–742.
Bronfenbrenner, U. (1986b). Recent advances in research on
the ecology of human development. In R. K. Silbereisen,
K. Eyferth, & G. Rudinger (Eds.), Development as action in
context: Problem behavior and normal youth development (pp.
287–309). Heidelberg: Springer-Verlag.
Bronfenbrenner, U. (1999). Environments in developmental
perspective: Theoretical and operational models. In S. L.
Friedman & T. D. Wachs (Eds.), Measuring environment
across the life span: Emerging methods and concepts (pp. 3–28).
Washington, DC: American Psychological Association.
Byrne, B. M., Shavelson, R. J., & Muthén, B. (1989). Testing for
the equivalence of factor covariance and mean structures: The
issue of partial measurement invariance. Psychological Bulletin,
105, 456–466.
Caspi, A., McClay, J., Moffitt, T. E., Mill, J., Martin, J., Craig,
I. W., et al. (2002). Role of genotype in the cycle of violence
in maltreated children. Science, 297, 851–854.
Caspi, A., Sugden, K., Moffitt, T. E., Taylor, A., Craig, I. W.,
Harrington, H., et al. (2003). Influence of life stress on
depression: Moderation by a polymorphism in the 5-HTT
gene. Science, 301, 386–389.
Chakrabarti, B., Dudbridge, E., Kent, L., Wheelwright, S., Hill-
Cawthorne, G., Allison, C., et al. (2010). Genes related
to sex steroids, neural growth, and social-emotional behav-
ior are associated with autistic traits, empathy, and Asperger
syndrome. Autism Research, 2, 157–177.
Chassin, L., Knight, G., Vargas-Chanes, D., Losoya, S. H., &
Naranjo, D. (2009). Substance use treatment outcomes in a
sample of male serious juvenile offenders. Journal of Substance
Abuse Treatment, 36, 183–194.
Chen, F. F., Sousa, K. H., & West, S. G. (2005). Testing mea-
surement invariance of second-order factor models. Structural
Equation Modeling, 12, 471–492.
Cheung, G. W., & Rensvold, R. B. (1999). Testing factorial
invariance across groups: A reconceptualization and proposed
new method. Journal of Management, 25, 1–27.
Clark, L. A., & Watson, D. (1997). The Mini Mood and
Anxiety SymptomQuestionnaire (Mini-MASQ). Unpublished
manuscript, University of Iowa.
Cleary, T. A. (1968). Test bias: Prediction of grades of Negro and
White students in integrated colleges. Journal of Educational
Measurement, 5, 115–124.
Cobb, B., Lehmann, J., Newman-Gonchar, R., & Alwell, M.
(2009). Self-determination for students with disabilities: A
narrative metasynthesis. Career Development for Exceptional
Individuals, 32, 108–114.
Colarelli, S. M., Han, K., & Yang, C. (2010). Biased against
whom? The problems of “group” definition and membership
in test bias analyses. Industrial and Organizational Psychology:
Perspectives on Science and Practice, 3, 228–231.
Conger, R. D., Song, H., Stockdale, G. D., Ferrer, E., Widaman,
K. F., & Cauce, A. M. (2012). Resilience and vulnerability of
Mexican origin youth and their families: A test of a culturally-
informed model of family economic stress. In P. K. Kerig, M.
S. Schulz, & S. T. Hauser (Eds.), Adolescence and beyond:
Family processes and development (pp. 268–286). New York:
Oxford University Press.
Conger, R. D., Stockdale, G. D., Song, H., Robins, R. W., &
Widaman, K. F. (in press). Predicting change in substance
use and substance use cognitions of Mexican origin youth
during the transition from childhood to early adolescence. In
Y. F. Thomas, L. N. Price, & A. V. Lybrand (Eds.), Drug use
trajectories among African American and Hispanic Youth. New
York: Springer.
Diaz, N., Horton, E. G., McIlveen, J., Weiner, M., & Nelson, J.
(2009). Dysthymia among substance abusers: An exploratory
study of individual and mental health factors. International
Journal of Mental Health and Addiction, 7, 357–367.
Eastwood, S. L., & Harrison, P. J. (2010). Markers of glutamate
synaptic transmission and plasticity are increased in the ante-
rior cingulate cortex in bipolar disorder. Biological Psychiatry,
67, 1010–1016.
Ferrer, E., Balluerka, N., & Widaman, K. F. (2008). Factorial
invariance and the specification of second-order latent growth
models. Methodology, 4, 22–36.
Folks, H. (1931). Committee C-1—Socially handicapped—
Dependency and neglect. In White House Conference on
Child Health and Protection, White House conference 1930:
Addresses and abstracts of committee reports, White House con-
ference on child health and protection called by President Hoover
(pp. 319–340). New York: The Century Co.
Grigorenko, E. L., Klin, A., & Volkmar, F. (2003). Annota-
tion: Hyperlexia: disability or superability? Journal of Child
Psychology and Psychiatry, 44, 1079–1091.
Hancock, G. R., Kuo, W.-L., & Lawrence, F. R. (2001). An
illustration of second-order latent growth models. Structural
Equation Modeling, 8, 470–489.
Henry, P. J. (2008). College sophomores in the laboratory
redux: Influences of a narrow data base on social psychol-
ogy’s view of the nature of prejudice. Psychological Inquiry, 19,
49–71.
Hoffman, M. L. (1975). Altruistic behavior and the parent-child
relationship. Journal of Personality and Social Psychology, 31,
937–943.
Hoffman, M. L. (1979). Development of moral thought, feeling,
and behavior. American Psychologist, 34, 958–966.
Hoffman, M. L. (1994). Discipline and internalization. Devel-
opmental Psychology, 30, 26–8.
Horn, J. L., McArdle, J. J., & Mason, R. (1983). When is
invariance not invariant: A practical scientist’s look at the
widaman, early, conger 79
 ethereal concept of factorial invariance. Southern Psychologist,
1, 179–188.
John, O. P., Donahue, E. M., & Kentle, R. L. (1991). The Big
Five Inventory-Versions 4a and 54. Berkeley, CA: University
of California, Berkeley, Institute of Personality and Social
Research.
Jöreskog, K. J. (1971). Simultaneous factor analysis in several
populations. Psychometrika, 36, 409–426.
Koch, R., & de la Cruz, F. (1999a). Historical aspects and
overview of research on phenylketonuria. Mental Retardation
and Developmental Disabilities Research Reviews, 5, 101–103.
Koch, R., & de la Cruz, F. (Eds.). (1999b). Phenylketonuria.
Mental Retardation and Developmental Disabilities Research
Reviews, 5, 101–161.
Koch, R., de la Cruz, F., & Azen, C. G. (Eds.). (2003). The
Maternal Phenylketonuria Collaborative Study: New devel-
opments and the need for new strategies. Pediatrics, 112,
1513–1587.
Koychev, I., El-Deredy, W., Haenschel, C., & Deakin, J. F. W.
(2010). Visual information processing deficits as biomarkers
of vulnerability to schizophrenia: An event-related potential
study in schizotypy. Neuropsychologia, 48, 2205–2214.
Kuo, C.-C., Maker, J., Su, F.-L., & Hu, C. (2010). Identify-
ing young gifted children and cultivating problem solving
abilities and multiple intelligences. Learning and Individual
Differences, 20, 365–379.
Little, T. D. (1997). Mean and covariance structures (MACS)
analyses of cross-cultural data: Practical and theoretical issues.
Multivariate Behavioral Research, 32, 53–76.
Little, T. D., Bovaird, J. A., & Card, N. A. (Eds.) (2007). Mod-
eling contextual effects in longitudinal studies. Mahwah, NJ:
Erlbaum.
Little, T. D., Card, N. A., Slegers, D. W., & Ledford, E. C.
(2007). Representing contextual effects in multiple-group
MACS models. In T. D. Little, J. A. Bovaird, & N. A. Card
(Eds.), Modeling contextual effects in longitudinal studies (pp.
121–147). Mahwah, NJ: Erlbaum.
Lubinski, D., & Benbow, C. P. (2006). Study of Mathematically
Precocious Youth after 35 years: Uncovering antecedents for
the development of math-science expertise. Perspectives on
Psychological Science, 1, 316–345.
Martz, E., & Daniel, S. (2010). Comparing disability proto-
types in the United States and Kenya. Journal of Applied
Rehabilitation Counseling, 41, 19–25.
McArdle, J. J. (1988). Dynamic but structural equation modeling
of repeated measures data. In J. R. Nesselroade & R. B. Cattell
(Eds.), Handbook of multivariate experimental psychology (2nd
ed., pp. 561–614). New York: Plenum.
Meade, A. W., & Tonidandel, S. (2010). Not seeing clearly with
Cleary: What test bias analyses do and do not tell us. Indus-
trial and Organizational Psychology: Perspectives on Science and
Practice, 3, 192–205.
Meredith, W., & Horn, J. (2001). The role of factorial invariance
in modeling growth and change. In L. M. Collins & A. G.
Sayer (Eds.), New methods for the analysis of change (pp. 203–
240). Washington, DC: American Psychological Association.
Meredith, W. M. (1993). Measurement invariance, factor
analysis and factorial invariance. Psychometrika, 58, 525–543.
Millsap, R. E., & Meredith, W. (2007). Factorial invariance:
Historical perspectives and new problems. In R. Cudeck &
R. C. MacCallum (Eds.), Factor analysis at 100: Historical
developments and new directions (pp. 131–152). Mahwah, NJ:
Erlbaum.
80 s p e c i a l p o p u l at i o n s
Nary, D. E., Froehlich-Grobe, K., & Aaronson, L. (2011).
Recruitment issues in a randomized controlled exercise trial
targeting wheelchair users. Contemporary Clinical Trials, 32,
188–195.
Nesselroade, J. R. (1983). Temporal selection and factor invari-
ance in the study of development and change. In P. B. Baltes
& O. G. Brim, Jr. (Eds.), Life-span development and behavior
(Vol. 5; pp. 59–87). New York: Academic Press.
Obradoviæ, J., Long, J. D., Cutuli, J. J., Chan, C.-K.,
Hinz, E., Heistad, D., & Masten, A. S. (2009). Aca-
demic achievement of homeless and highly mobile children
in an urban school district: Longitudinal evidence on risk,
growth, and resilience. Development and Psychopathology, 21,
493–518.
Olson, I. R., Berryhill, M. E., Drowos, D. B., Brown, L., & Chat-
terjee, A. (2010). A calendar savant with episodic memory
impairments. Neurocase, 16, 208–218.
Or, D., Cohen, A., & Tirosh, E. (2010). The reliability and valid-
ity of the Aggregate Neurobehaviroal Student Health and
Educational Review Parent’s Questionnaire (ANSER-PQ).
Journal of Child Neurology, 25, 157–164.
Pennington, K., Beasley, C. L., Dicker, P., Fagan, A.,
English, J., Pariante, C. M., et al. (2008). Prominent
synaptic and metabolic abnormalities revealed by pro-
teomic analysis of the dorsolateral prefrontal cortex in
schizophrenia and bipolar disorder. Molecular Psychiatry, 13,
1102–1117.
Petrill, S. A. (2010). Editorial: Identifying the genetic and envi-
ronmental influences on psychological outcomes. Journal of
Child Psychology and Psychiatry, 51, 745–746.
Polinsky, M. L., Hser, Y.-I., & Grella, C. E. (1998). Considera-
tion of special populations in the drug treatment system of a
large metropolitan area. Journal of Behavioral Health Services
& Research, 25, 7–21.
Reichenberg, A., Caspi, A., Harrington, H., Houts, R.,
Keefe, R. S. E., Murray, R. M., et al. (2010). Static
and dynamic cognitive deficits in childhood schizophrenia:
A 30-year study. The American Journal of Psychiatry, 167,
160–169.
Reise, S. P., & Waller, N. G. (2003). How many IRT parameters
does it take to model psychopathology items? Psychological
Methods, 8, 164–184.
Rensvold, R. B., & Cheung, G. W. (1998). Testing measure-
ment model for factorial invariance: A systematic approach.
Educational and Psychological Measurement, 58, 1017–1034.
Sears, D. O. (1986). College sophomores in the laboratory: Influ-
ences of a narrow database on social psychology’s view of
human nature. Journal of Personality and Social Psychology,
57, 515–530.
Sears, D. O. (2008). College student-it is redux. Psychological
Inquiry, 19, 72–77.
Simunovic, F., Yi, M., Wang, Y., Macey, L., Brown, L. T.,
Krichevsky, A. M., et al. (2009). Gene expression profiling
of substantia nigra dopamine neurons: Further insights into
Parkinson’s disease pathology. Brain, 132, 1795–1809.
Sussman, S. (2006). Prevention of adolescent alcohol problems in
special populations. In M. Galanter (Ed.), Alcohol problems
in adolescents and young adults: Epidemiology, neurobiology,
prevention, and treatment (pp. 225–253). New York: Springer
Science & Business Media.
Uehling, H. F. (1952). Rorschach “shock” for two spe-
cial populations. Journal of Consulting Psychology, 16,
224–225.
Waller, N. G., & Reise, S. P. (1989). Computerized adaptive
personality assessment: An illustration with the Absorption
scale. Journal of Personality and Social Psychology, 57, 1051–
1058.
Widaman, K. F. (1985). Hierarchically nested covariance struc-
ture models for multitrait-multimethod data. Applied Psycho-
logical Measurement, 9, 1–26.
Widaman, K. F. (2007). Intrauterine environment affects infant
and child intellectual outcomes: Environment as direct effect.
In T. D. Little, J. A. Bovaird, & N. A. Card (Eds.), Model-
ing contextual effects in longitudinal studies (pp. 387–436).
Mahwah, NJ: Erlbaum.
Widaman, K. F. (2009). Phenylketonuria in children and moth-
ers: Genes, environments, behavior. Current Directions in
Psychological Science, 18, 48–52.
Widaman, K. F., & BorthwickDuffy, S. A. (April, 1990). Parental
influences on the development of adaptive behaviors. Paper
presented at the Gatlinburg Conference on Research and The-
ory in Mental Retardation and Developmental Disabilities,
Brainerd, MN.
Widaman, K. F., Gibbs, K. W., & Geary, D. C. (1987). The
structure of adaptive behavior: I. Replication across fourteen
samples of nonprofoundly retarded persons. American Journal
of Mental Deficiency, 91, 348–360.
Widaman, K. F., & Reise, S. P. (1997). Exploring the measure-
ment invariance of psychological instruments: Applications
in the substance use domain. In K. J. Bryant, M. Windle,
& S. G. West (Eds.), The science of prevention: Method-
ological advances from alcohol and substance abuse research
(pp. 281–324). Washington, DC: American Psychological
Association.
widaman, early, conger 81
 CHAPTER

Theory Construction, Model Building,

5 and Model Selection

James Jaccard

Abstract

General issues involved in (1) building causal theories, (2) translating those theories into a set of
mathematical representations, (3) choosing an analytic strategy to estimate parameters in the
equations implied by the theory, and (4) choosing the “best” model from a set of competing models
are discussed. Theory construction fundamentally relies on six relationship types, including direct
effects, indirect effects, moderated effects, reciprocal effects, spurious effects and unanalyzed
relationships. Once specified, each type of effect can be represented mathematically, thereby
translating a path diagram into a set of (linear) equations. Parameters in the equations are then
estimated using either limited information estimation approaches or full information estimation
approaches, taking into account measurement properties, population distributions, and matters of
robustness. Choices between competing models are based on indices of relative fit with the data and
relative fit with competing models, as well as more general theoretical criteria (e.g., parsimony,
consistency with related theories).
Key Words: Theory; theory construction; modeling; parameter estimation

Few people dispute that theory is at the heart of
the scientific enterprise. We use theories to explain
phenomena and to help solve important applied
problems. With theoretical propositions in hand,
we design research to gain perspectives on the viabil-
ity of the theory and imbue those propositions with
a certain degree of confidence based on the results
of research and the quality of the research design.
Theory is fundamental to the social sciences.
Advances in statistical analysis have been con-
siderable during the past 30 years. Our statistical
toolbox was at one time somewhat limited, and it
was not uncommon for analysts to adopt practices
that forced data to conform to the assumptions of
our statistical tools so that we could apply inferen-
tial methods of analysis to those data. For example,
although we knew that a construct like depression
was non-normally distributed in the population and
that our measures of depression and data reflected
this, we would transform data on depression so
that it would approximate a normal distribution
and be amenable to analysis using statistical meth-
ods that assumed normality. Such days, fortunately,
are over. The statistical tools available now allow
us to give precedence to theory and model test-
ing without being slave to many of the traditional
assumptions made by methods of analysis. The
present chapter discusses issues that analysts need
to take into account as they move from theory to
analysis. I focus first on the nature of theory in
the social sciences, with an emphasis on describ-
ing theories that invoke the language of causality.
Although there are many other approaches to the-
ory and modeling, one cannot deny the prominence
and pervasiveness of causal thinking in social sci-
ence research. Hence, causal frameworks capture
the bulk of my attention. Next, I discuss issues
involved in moving from a well-specified causal

82
theory to a mathematical representation of that the-
ory for purposes of statistical analysis. I then discuss
a range of issues that must be considered as one
moves from a set of equations representing a the-
ory to formal data analysis, including measurement
considerations, full versus limited information esti-
mation strategies, distributional assumptions, and
the possible presence of outliers. Finally, I consider
issues in model selection, which refers to the process
of choosing one model as being the “best” from a set
of candidate models.
In the present chapter, I use the terms theory
and model interchangeably, although some scien-
tists do not do so. As examples, various authorities
contend that models are a special type of theory
(e.g., Coombs, Dawes, & Tversky, 1970, p. 4;
Kaplan, 1964, p. 263), models are portions of the-
ories (Sheth, 1967, p. 444; Torgerson, 1958, p. 4),
models are derived from theories (e.g., Pap, 1962,
p. 355), models are simplified versions of theories
(e.g., Carnap, 1971, p. 54), models represent corre-
spondence between two or more theories (Brodbeck,
1968), or theories represent specific interpretations
of (i.e., are derived from) models (e.g., Green &
Tull, 1975, p. 42). Others consider the terms to be
synonymous (cf, Dubin, 1976; Simon & Newell,
1956). Although there may indeed be meaningful
distinctions between theories and models, it also is
the case that models, like theories, involve variables
and relationships between variables, usually invok-
ing the concept of causality. Accordingly, I will use
the terms theory and model interchangeably.
Specifying a Causal Theory
The Nature of Causality
Theories take many forms in the social sciences.
One common form involves specifying presumed
relationships between variables while invoking the
concept of causality. The nature of causality has
been debated extensively by philosophers of sci-
ence (e.g., Bunge, 1961; Cartwright, 2007; Frank,
1961; Morgan & Winship, 2007; Pearl, 2000; Rus-
sell, 1931; Rubin, 1974, 1978; Shadish, Cook, &
Campbell, 2002) and most agree that causality is
an elusive concept that is fraught with ambiguities.
It is beyond the scope of this chapter to delve into
the underlying philosophical issues (see Jaccard &
Jacoby, 2010). Rather, I emphasize here a “working
model” of causality that is adopted by most social
scientists.
The concept of causality is usually thought of in
terms of change—that is, X is a cause of Y if changes
in X produce changes in Y (but see Sowa, 2000, and
Lewis, 2000, for alternative conceptualizations).
Four properties of causality are typically emphasized.
First, a cause always must precede an effect in time.
Second, the time that it takes for a change in X to
produce a change in Y can vary, ranging from almost
instantaneous change to years, decades, centuries, or
millennia. Third, the nature and/or strength of the
effect of X on Y can vary depending on context. X
may influence Y in one context but not another con-
text. Finally, cause and effect must be in some form
of spatial contact or must be connected by a chain
of intermediate events. We return to these ideas in
later sections of this chapter.
The Nature of Theories that Use
Causality
The focus of most causal theories is on explaining
why variation in one or more variables exists. Some
people make a great deal of money and others are
poor. Why? What can account for this variation?
Some people are able to remember complex mate-
rial easily whereas for other people, it is difficult
to do so. Why? What explains this variability? We
answer these questions by specifying the presumed
causes of the variability, and then we seek to test our
theoretical proposition(s).
In any given causal theory, there are six funda-
mental types of relationships that can be referenced;
these are illustrated in Figure 5.1. These six relation-
ship types are the cornerstone of causal theories and
define the universe of causal relationships that theo-
rists draw upon. In Figure 5.1, a variable is indicated
by a box, and a causal influence is represented by a
straight arrow emanating from the cause and point-
ing to the effect. I discuss the bidirectional curved
arrow in Figure 5.1 shortly.
Referring to Figure 5.1, a direct causal relation-
ship is one in which a given cause is assumed to
have a direct causal impact on some outcome vari-
able. For example, exposure to violence in the media
is assumed to influence aggressive behavior. Or,
the quality of the relationship between a mother
and her adolescent child is assumed to influence
whether the child uses drugs. By contrast, an indi-
rect causal relationship is when a variable influences
another variable indirectly through its impact on an
intermediary variable (see Fig. 5.1). For example,
failing to accomplish a goal may lead to frustra-
tion that, in turn, causes someone to aggress against
another. In this case, the failure to obtain a goal
is an indirect cause of aggression. It influences
aggression through its impact on frustration. Frus-
tration is formally called a mediating variable or,
jaccard 83
 Direct Causal Relationship
X Y
Spurious Relationship
Z X
X Y
Unanalyzed Relationship
X Y
Figure 5.1 Relationships in Causal Models
more informally, a mediator, because other vari-
ables “work through” it to influence the outcome.
Indirect relationships are sometimes called mediated
relationships.
Whenever a theorist posits an indirect relation-
ship, the issue of whether to specify partial or
complete mediation presents itself. In partial medi-
ation, the distal variable has a direct effect on
the outcome variable over and above its effect on
the mediator. In complete mediation, all of the
impact of the distal variable on the outcome vari-
able is accounted for by the mediator. With partial
mediation, in addition to the two causal paths char-
acterizing the indirect effect illustrated in Figure 5.1,
one adds an additional causal path linking the distal
variable (X) and the outcome variable (Y) directly. As
an example, we might posit that the quality of the
relationship between a mother and her adolescent
child (X) impacts the child’s motivation to perform
well in school (Z) and that one’s motivation to do
well in school, in turn, impacts (negatively) the ten-
dency for an adolescent to use drugs (Y). As we think
about the matter of specifying partial versus com-
plete mediation, we might decide that there are other
mechanisms by which the quality of the relationship
between parent and adolescent can impact drug use,
such as by lessening the attention that adolescents
84 theory construction, model building
Indirect Causal Relationship
X Z Y
Bi-directional Causal Relationship
Y
Moderated Causal Relationship
X Y
give to peers who use drugs. We therefore decide to
posit partial mediation to reflect this fact and add a
direct causal path from X to Y.
If we are able to specify another mechanism by
which Z influences Y over and beyond Z, then why
not just incorporate that additional mediator into
the theory? Of course, we could very well do this,
but then the issue becomes whether the two media-
tors in the theory, considered together, are complete
or partial mediators of the causal effect of X on Y.
This might lead us to speculate about yet a third
mechanism, and once we have specified it, the issue
of partial or complete mediation will present itself
yet again. At some point, we must decide to close
out the system and just let a direct path between X
and Y stand so as to reflect partial mediation with-
out formally bringing additional mediators into the
model. If pressed, we could articulate one, but we
simply do not want to complicate the theory further.
A spurious relationship is one in which two vari-
ables are related because they share a common cause
but not because either causes the other (see Fig.
5.1). As an example, if we select a random sam-
ple of people in the United States and calculate
the association between height and length of hair,
then we would find a moderate relationship between
the two variables: People with shorter hair grow
taller. Does this mean that a causal relationship exists
between these variables—that is, that cutting one’s
hair will make one grow taller? Of course not. The
reason the variables are correlated is because they
share a common cause: gender. Males tend to have
shorter hair than females and men tend to grow
taller than females. The common cause of gender
produces a correlation between length of hair and
height, but it is spurious.
A moderated causal relationship, like spurious and
indirect relationships, involves at least three variables
(see Fig. 5.1). In this case, the causal relationship
between two variables, X and Y, differs depending
on the value of a third variable, Z. For example,
it might be found that a given type of psychother-
apy (X) is effective for reducing depression (Y) for
females but not for males. In this case, the causal
relationship between being exposed to psychother-
apy and depression is moderated by gender. When
gender has the value “female,” X impacts Y. How-
ever, when gender has the value “male,” X does
not impact Y. Gender is called a moderator vari-
able because the relationship between the presence
or absence of psychotherapy (X) and depression (Y)
changes as a function of (or is “moderated by”) the
levels of gender.
A bidirectional causal relationship exists when
two variables are conceptualized as influencing each
other (see Fig. 5.1). For example, in the area of
reproductive health, a theorist might posit a bidirec-
tional influence between a woman’s belief that the
rhythm method is effective at preventing pregnancy
(X) and her attitude toward the rhythm method
(Y). A woman may have a positive attitude toward
the rhythm method because she believes it is effec-
tive. Simultaneously, she may believe it is effective,
in part, because she has a positive attitude toward
it, via a mechanism that involves rationalization of
behavior.
Technically, there can never be simultaneous
reciprocal causation because there always must be
a time interval, no matter how infinitesimally small,
between the cause and the effect that follows from
that cause. If we observed the causal dynamics
within the appropriate time frames, then the true
dynamic underlying a reciprocal causal relationship
would appear as follows:
Xt1→ Yt2→ Xt3→ Yt4,
where Xt1 is variable X at time 1, Yt2 is variable
Y at time 2, Xt3 is variable X at time 3, and Yt4
is variable Y at time 4. As an example, suppose we
conduct a cross-sectional study and at a given point
in time we measure adolescent drug use and grade
point averages in school. It is likely that the mea-
sured drug use reflects the influence of prior poor
performance in school because adolescents who do
poorly in school might turn to drugs as a coping
mechanism or as a way of spending free time that
normally would have been directed to school work.
Similarly, the measured school performance likely
reflects the effects of any prior drug use, which can
cause students to lose interest in school and to not
concentrate on tests and studying. It would be wrong
to assume there is unidirectional causality from one
construct to the other in this study. More realisti-
cally, the two measures reflect a more fine-grained
process that has played itself out—that is, poor per-
formance in school at time t influenced drug use at
time t + 1, which in turn influenced school perfor-
mance at time t + 2, which in turn influenced drug
use at time t + 3, and so on. It is only when we are
unable to capture the more fine-grained time inter-
vals and we must instead work with coarser time
intervals that the dynamic of the reciprocal causal
relationship as illustrated in Figure 5.1 applies. By
working with coarser time units, the more fine-
grained temporal causal dynamics are assumed to
have already played themselves out (often referred to
as the equilibrium assumption). In this sense, there
exists reciprocal causality per Figure 5.1.
The final type of relationship that can occur in
a causal model is an unanalyzed relationship. In
Figure 5.1, the two variables for this type of relation-
ship are connected by a double-headed curved arrow.
This arrow signifies that the two variables are pos-
sibly correlated but that the theorist is not going to
specify why they are correlated. The correlation may
be spurious or it may result from a causal connection
of some kind. The theorist wants to recognize the
possible correlation between the variables, but trying
to explain it is beyond the scope of the theoretical
effort. The relationship will remain unanalyzed.
Most causal models have more than one of these
six types of relationships in them. An example of
a multivariate causal model appears in Figure 5.2,
which was based on an endeavor that developed
an intervention to reduce unprotected sexual inter-
course to reduce the spread of sexually transmitted
diseases (STDs). The intervention is represented as
a two-level, qualitative variable in which individuals
are randomly assigned to either an intervention
group or a control group. The intervention is
designed to influence (1) the belief that having
unprotected sex increases the risk of contracting an
STD (see path a in Fig. 5.2) and (2) the belief
jaccard 85
 Belief
Unprotected Sex
leads to STDs
a
Intervention
versus
Control
b
Belief STDs
are Serious
Figure 5.2 A Multivariate Causal Model
that contracting an STD is bad for one’s health
(see path b in Fig. 5.2). These two beliefs, in turn,
are assumed to influence the tendency for a person
to engage in unprotected sex (paths c and d ). The
tendency to engage in unprotected sex, in turn, is
thought to impact the extent to which the person
contracts STDs (path e). Paths a through e each
represent direct causal relationships. There also is
aspurious relationship in this model, as seen by the
fact that the two beliefs share a common cause,
the intervention (paths a and b). Because of this
common cause, we expect the two beliefs to be
correlated to some extent, although there is no pre-
sumed causal connection between them. There also
are several indirect causal relationships in the model.
For example, the intervention indirectly affects the
occurrence of STDs through the two belief media-
tors and, in turn, their influence on the tendency to
engage in unprotected sex.
In causal theories, distinctions are made between
exogenous and endogenous variables. Any variable
that has a straight arrow going to it in a path dia-
gram is called an endogenous variable. Endogenous
variables, essentially, are outcome variables—that is,
they are presumed to be influenced by another vari-
able in the theoretical system. Variables that do not
have a straight arrow going to them are called exoge-
nous variables. They do not have presumed causes
that are elaborated upon in the theory.
In sum, the fundamental orientation to con-
structing a causal theory is to explain variation in
one or more outcomes. This is accomplished with
reference to six types of relationships, direct effects,
indirect effects, reciprocal effects, moderated effects,
spurious effects, and unanalyzed relationships. In
the theory construction process, it is not uncommon
for the theorist to first identify the outcome variable
he or she wishes to explain and then to specify a
few direct causes of that variable. One or more of
86 theory construction, model building
c
Engages in
e
Unprotected Has an STD
Sex
d
the direct causes can then be turned into an indirect
effect by elaborating the mechanisms by which the
cause produces the effect. The “mechanisms” are,
essentially, mediators. The theory might be further
elaborated by specifying the boundary conditions
of effects, thereby introducing moderated relation-
ships into the framework (e.g., the effect holds when
condition A is operative but not when conditions
B is operative; or the effect holds for one type
of individual but not another type of individual).
For a discussion of the many heuristics scientists
use to identify mechanisms and boundary condi-
tions when developing causal theories see Jaccard
and Jacoby (2010).
Theories can be represented in the form of path
diagrams using the graphical schemes illustrated in
Figures 5.1 and 5.2. Such representations are pos-
sible no matter what research design is used to test
a theory—that is, we can represent the underlying
theory with path diagrams for experiments just as
readily as for purely observational designs. Figure 5.2
is an example of a causal theory that incorporates an
experimental design.
Causal Theories with Explicit Temporal
Dynamics
An important structure of many (but not all)
causal theories is a focus on longitudinal dynam-
ics. Causal theories that focus on longitudinal
dynamics contain one or more of the six types
of relationships described above, but there also is
an explicit temporal dynamic that is of theoretical
interest. A nomenclature has emerged around such
theories, which I briefly describe here.
One popular type of causal model that includes
longitudinal dynamics is called a panel model in
which multiple variables are modeled at multi-
ple points in times, also called waves. Figure 5.3
 Alcohol Use a Alcohol Use
Time t Time t+1
d d
c c
Delinquency Delinquency
Time t b Time t+1
Figure 5.3 A Panel Model
presents an example of a two-variable, four-wave
panel model. This model examines the relationship
between adolescent alcohol use and delinquency
across the 4 years of high school. Heavy alcohol use
at time t is thought to have a direct effect on alcohol
use at time t + 1, and the same type of dynamic is
thought to hold for delinquency. A path that spec-
ifies a variable at time t + 1 as being a function
of that same variable at time t is said to reflect an
autoregressive effect (see paths a and b in Fig. 5.3).
In the model in Figure 5.3, the autoregressive effects
are first-order effects, because the variable at a given
time, t, is assumed to be a function of that same
variable at the time just prior to it. A second-order
autoregressive effect is one where the variable at time
t is impacted (also) by the same variable at time t −2.
A third-order autoregressive effect implies a direct
effect between a variable at time t and that same
variable at time t − 3. And so on.
The theory in Figure 5.3 also specifies that drink-
ing alcohol at time t predisposes one to delinquency
at that same time period under the rationale that
people often commit acts of delinquency when they
are drunk. When a causal relationship exists between
two variables at the same period of time, it is said to
represent a contemporaneous causal effect (see path c
in Fig. 5.3). Finally, when a variable at time t has
an effect on a different variable at time t + 1, it is
referred to as a lagged effect (see path d in Fig. 5.3).
In this case, alcohol use at time t is thought to have
a delayed effect on delinquency at time t + 1 inde-
pendent of the other indirect causal chains that link
alcohol use at time t to delinquency at time t + 1.
In sum, it is common in longitudinal models to
theorize about autoregressive effects, contempora-
neous effects, and lagged effects. These effects are
common features of panel models (Collins, 2006;
Finkel, 2008).
A second type of theory that formally incorpo-
rates temporal dynamics is theory based on growth
processes. This approach views variation in out-
comes across time as arising from an unobserved
a Alcohol Use a Alcohol Use
Time t+2 Time t+3
d
c
Delinquency Delinquency
b Time t+2 b Time t+3
“growth” process that causes changing values of
an outcome over time. These models are typically
associated with latent growth curve models in the
social science literature or, more simply, latent curve
models (LCMs). The classic form of these models is
presented in Figure 5.4, as applied to the alcohol-
delinquency example considered earlier. Observed
variables or measures are represented by boxes, and
latent (unobserved) variables are represented by cir-
cles. This model parameterizes a latent “growth”
or “maturation” process for alcohol use as students
progress through high school (represented by the
variable called “Alcohol Slope” in Fig. 5.4) as well
as a latent growth or maturation process for delin-
quency as students progress through high school
(see the variable “Delinquency Slope” in Fig. 5.4).
The “growth process” for alcohol use is assumed to
impact the “growth process” for delinquency across
the four time periods. One often will encounter
longitudinal causal theories expressed in this form
instead of in the more traditional panel model form
(Collins, 2006).
Of course, it is possible that both types of dynam-
ics in the two types of models operate. When the two
types of processes are integrated into a single model,
we obtain what is called an autoregressive latent tra-
jectory model (Bollen & Curran, 2006). Figure 5.5
presents this model.
In sum, when constructing theories that incorpo-
rate longitudinal dynamics, one will explicitly take
into account the possible causal dynamics described
by panel models, by LCMs, or by autoregressive
latent trajectory models.
Multilevel Causal Theories
Another common type of causal model that has
received considerable attention in the social sciences
is one that incorporates multiple levels of analysis, or
what is called a multilevel model. These models deal
with scenarios where there is nesting—for example,
where individuals are nested within different, higher
jaccard 87
 Baseline
Alcohol
Alcohol
Slope
Use

Alcohol Use Alcohol Use Alcohol Use Alcohol Use

Time t Time t+1 Time t+2 Time t+3
a

Delinquency Delinquency Delinquency Delinquency

Time t Time t+1 Time t+2 Time t+3

Baseline Delinquency
Delinquency Slope

Figure 5.4 A Latent Curve Model

Baseline Alcohol
Alcohol Slope

Alcohol Use a Alcohol Use a Alcohol Use a Alcohol Use

Time t Time t+1 Time t+2 Time t+3
e
d d d
c c c

Delinquency Delinquency Delinquency Delinquency

Time t b Time t+1 b Time t+2 b Time t+3

Baseline Delinquency
Delinquency Slope

Figure 5.5 An Autoregressive Latent Trajectory Model

88 theory construction, model building

level organizational units. For example, students are (a)
nested within schools and both characteristics of the X Y δ
students and characteristics of the schools can influ-
ence an outcome variable, such as performance by
students on a standardized math test. Employees are
nested within organizations, and both characteristics
of the employees and characteristics of the organiza- Z
tion can influence employee behavior. Patients are
nested within hospitals and both characteristics of (b)
the patients and characteristics of the hospitals can
influence patient recovery. δ δ2

Summary of Causal Model Forms

In sum, causal theories seek to explain variation in
Smoking Drug Use
one or more outcome variables by identifying causes
of those variables. Social scientists posit theories or
models to represent the presumed causal relation-
ships among variables, and these theories typically
have one or more of the six fundamental relation- Gender
ships types in them—namely, direct effects, indirect
effects, spurious effects, moderated effects, recip-
rocal causality, and unanalyzed relationships. The (c)
theories can focus on a single time period or explic-
itly deal with temporal dynamics, If they deal with
temporal dynamics, then this usually takes the form
of a panel model, a LCM, or an autoregressive latent δ1 δ2
trajectory model. Theorizing also can occur at a sin-
gle level of analysis or at multiple levels of analysis
in which lower order units are nested within higher
order units, with characteristics of the units at both Smoking Drug Use
levels influencing outcomes.

Theories and Disturbance Terms

There is a more subtle facet of theory construc-
tion beyond those elucidated thus far, and this
concerns the concept of disturbance terms. Con-
sider the simple theory in Figure 5.6a. This theory
has two direct causes where variables X and Z are
assumed to influence variable Y. A fourth “variable”
is represented in the system by a circle. This “vari-
able” reflects all unspecified variables that influence
Y other than X and Z. It formally recognizes that the
theory is incomplete and that we have not specified
every cause of the outcome variable. This “vari-
able” is called a disturbance term, and it represents
the totality of all unspecified causal effects on the
endogenous variable it is associated with. The pres-
ence of a disturbance term explicitly recognizes that
not all causal influences on a variable have been spec-
ified in the model. Traditionally, each endogenous
variable in a theory has a disturbance term associated
with it.
Gender
Figure 5.6 Theories with Disturbance Terms. (a) Theory with
Disturbance Term (b) Smoking and Drug Example with Uncor-
related Disturbance Terms (c) Smoking and Drug Example with
Correlated Disturbance Terms
Consider another example in Figure 5.6b. There
are two endogenous variables, and they share a com-
mon cause. One of the endogenous variables is
adolescent tobacco use, and the other is adolescent
drug use. The common cause is gender. The theory
posits that boys are more likely than girls to smoke
cigarettes and that boys also are more likely than girls
to use drugs. There is a disturbance term for each
of the endogenous variables. These terms recognize
that factors other than gender impact tobacco use
and drug use.
But there is a problem with this theory. According
to the theory, the only reason that smoking cigarettes

jaccard 89
and drug use in adolescence are correlated is because
they share the common cause of gender. In real-
ity, there are many other common causes of these
two constructs. For example, social class impacts
both tobacco use and drug use during adolescence,
with more economically disadvantaged youth hav-
ing an increased tendency to smoke cigarettes and
to use drugs. Essentially, social class resides within
the disturbance term for smoking cigarettes and it
also resides within the disturbance term for drug
use. If the same unspecified cause is in each distur-
bance term, then we would expect the two distur-
bance terms to be correlated. Figure 5.6c presents
a more plausible theory that includes this correla-
tion between disturbances. According to this theory,
there are two reasons why adolescent cigarette smok-
ing and adolescent drug use are correlated. One
reason is because they share the common cause of
gender. Another reason is that they share other com-
mon causes that are unspecified by the theory and
that reside in both disturbance terms.
A well-developed theory provides explicit state-
ments about which disturbance terms in the frame-
work are correlated and which disturbance terms
are not. The lazy way out for a theorist is to simply
assume all disturbance terms are correlated. But this
is not satisfactory, and it can create considerable dif-
ficulties for testing the theory empirically. A better
approach is to carefully consider every pair of dis-
turbance terms and to articulate a possible common
cause that resides in each of the separate disturbance
terms. If such a common cause can be articulated,
then it makes sense to posit correlated disturbances.
If one cannot articulate any such variable, or if its
effects are thought to be trivial, then one does not
posit correlated disturbances.
For models with a longitudinal component,
many theorists have a “knee-jerk” reaction that
disturbances directed at the same variable at two
different points in time must be correlated. Again,
if one can articulate a compelling rationale for cor-
related disturbances, then by all means, correlated
disturbances should be incorporated into the the-
ory. Otherwise, correlated disturbances should be
viewed with theoretical skepticism.
If a theorist is able to articulate a variable that
resides in two disturbance terms to create correlated
disturbances, then why not explicitly incorporate
the variable into the theoretical system? For exam-
ple, for the smoking cigarette and drug use example
in Figure 5.6, why not explicitly bring social class
into the theoretical system? This, of course, is desir-
able. But at some point, we want to close out
90 theory construction, model building
the theoretical system and work just with the vari-
ables we have specified. By including disturbance
terms and correlated disturbances, we are explic-
itly recognizing the operation of other variables, but
we choose not to give them central focus in our
theory.
Latent Variables, Structural Models, and
Measurement Models
Some researchers take matters a step further and
also incorporate a measurement theory into their
conceptual frameworks when they are performing
an empirical test of the theory. This goes beyond
the typical province of theory construction per se,
but I mention the ideas here as they ultimately
impact data analysis and the choice of statisti-
cal methods. The integration of conceptual and
measurement theories is something that should be
standard practice for social science research.
An empirical test of a theory necessarily requires
developing and using measures of the theoretical
constructs in the theory. Just as one can build a the-
ory linking one concept to another concept, so too
can one build a theory linking a construct to a mea-
sure of that construct. Some theorists combine both
types of theories into a single overarching frame-
work. Traditional measurement models make a
distinction between a latent variable and an observed
measure of that variable. The latent variable is the
true construct about which one is interested in
making statements, such as depression. Although
we can directly observe many of the symptoms of
depression, we can’t directly observe the “seat” of
depression in a person’s mind. Rather, we rely on
some observable response(s) to assess the latent vari-
able, such as a multi-item inventory of depression
that a person completes. Figure 5.7a presents one
representation of a measurement model. The latent
variable of depression is contained in a circle, and
the observed measure thought to reflect depression
is contained in a square (the label “AR” stands for
adolescent report of depression). A causal path is
drawn from the latent variable to the observed mea-
sure, under the assumption that how depressed a
person is influences how he or she responds to the
questions on the inventory. There also is an error
term that reflects measurement error—that is, there
are factors other than depression that influence a
person’s responses on the inventory. Ideally, mea-
surement error is minimal, but it is a fact of life for
many research endeavors. The relationship between
the latent construct and the observed indicator is
(a) (b)
ε ε1 ε2
AR AR MR
Depression Depression
Figure 5.7 Measurement Models. (a) Single Indicator (b) Mul-
tiple Indicators
usually assumed to be linear, but it could also be
nonlinear.
Sometimes we obtain multiple indicators of a
construct. For example, a researcher might obtain a
self-report of depression from an adolescent as well
as a report from the adolescent’s mother about how
depressed the child is (MR). Figure 5.7b presents
a measurement model for this scenario. The latent
variable of depression is assumed to influence both of
the observed measures, and each measure is assumed
to have some measurement error as reflected by the
presence of error terms. The errors are assumed to
be uncorrelated because we cannot articulate any
viable reason why we would expect them to be
correlated. However, one can introduce correlated
measurement error, if appropriate.
Figure 5.8 presents an example of a more elabo-
rate theoretical framework that incorporates a theory
about the relationship between constructs as well as a
measurement theory. Although it appears somewhat
intimidating, it is a straightforward model. There are
five latent constructs, and the main substantive the-
ory is focused on them. The portion of the diagram
focused on the causal relations among the latent
variables is called the structural model. The primary
outcome variable in this model is the birth weight
of a newborn. Birth weight is thought to be influ-
enced by two factors: how much alcohol the mother
consumes during her pregnancy and how much she
smokes during her pregnancy. Both of these variables
are thought to be influenced by two other variables.
The first determinant is the extent of support the
mother has from friends and relatives who can help
her quit smoking and drinking. The second is the
mother’s locus of control. Locus of control refers to
the extent to which the mother believes that what
happens to her is beyond her control. The theory
is that the more a mother thinks that what happens
is not under her control, the more likely she will
be to keep smoking and drinking during pregnancy.
These two latent exogenous variables are assumed to
be correlated. The three latent endogenous variables
each have a disturbance term, indicated by a circle
with a δ inside of it. The disturbances are assumed
to be uncorrelated.
The portion of the diagram with arrows from
the latent constructs to the observed measures con-
stitutes the measurement model. Each of the latent
variables has multiple indicators. In other words, the
researcher obtains three measures of each construct,
with the exception of birth weight, which is mea-
sured using two different indicators. In the interest
of space, we do not describe these measures, but note
that each is assumed to be fallible (i.e., subject to
some measurement error; see the circles ranging from
ε1 to ε14). The measurement errors are assumed
to be uncorrelated. Figure 5.8 provides an explicit
roadmap for a researcher to test the combined
structural theory and measurement theory.
In experiments that involve a formal manipula-
tion, the manipulation typically is considered to be
an observed variable in its own right, with no latent
construct underlying it (see Fig. 5.2). In some cases,
the manipulation is designed to reflect or produce
differences in an underlying psychological state (e.g.,
one’s mood when studying the effects of mood on
memory). In this case, a measurement model may
be introduced into the causal system, treating the
manipulation as a formative rather than a reflec-
tive indicator of the construct (see Schumacker &
Lomax, 2004, for elaboration).
Many social scientists view measurement mod-
els as interesting causal theories in their own right
rather than just a methodological feature of the-
ory testing. The most common case is when the
conceptual focus is on specifying the facets or dimen-
sions of a construct vis-à-vis factor analysis. The
causal theory underlying factor analysis is that one or
more (possibly correlated) unmeasured latent vari-
ables are each a common cause of a set of observed
measures of constructs that are of interest in their
own right. For example, theorists have suggested
there are four facets of social support: informational
support, emotional support, tangible support, and
companionship support. Each of these facets is con-
ceptualized as a latent variable that impacts distinct
manifestations of social support. For elaboration of
theory-based expressions of measurement models,
see Brown (2006).
jaccard 91
 ε1 ε2 ε3 ε4 ε5 ε6

S1 S2 S3 A1 A2 A3
ε13 ε14

W1 W2
Support Alcohol

δ1
Birth-
weight

δ2
δ3

Control Smoking

C1 C2 C3 K1 K2 K3

ε7 ε8 ε9 ε10 ε11 ε12

Figure 5.8 Example of Integrated Structural and Measurement Model

In sum, a well-developed theory will not only
include causal relationships between constructs but
also will include disturbance terms and a theory
of the correlational structure of those disturbance
terms. As one moves to an empirical test of a theory,
one also specifies a measurement model that links
theoretical constructs to measures obtained in the
empirical test.
From Theories to Mathematical
Representations
Specifying Core Model Equations
With a well-articulated theory in hand, the next
step for choosing a form of statistical analysis is to
translate the theory into a set of equations that then
guide the statistical analysis. The basic strategy for
doing so can be illustrated by making some simpli-
fying assumptions, which I relax later. First, unless
otherwise specified, I will assume that all variables in
the theoretical system are continuous. Second, I will
assume that the measures of all variables have at least
interval level properties. Finally, I will assume that
all relationships between variables are linear. I adopt
a strategy whereby the theory under consideration
is expressed in the form of a set of linear equations.
This does not mean, however, that researchers always
parameterize data analysis in terms of the linear
model. Many well-known methods of analysis, such
as t tests, analysis of variance, and analysis of covari-
ance, evolved outside of the context of the linear
model. However, these methods can be re-expressed
as linear equations, and in this sense, they are com-
patible with the current approach. Complex models
of dichotomous, nominal, and count variables also
can be approached from the perspective of linear
equations using what is known as the generalized
linear model (see Yuan & Schuster, Chapter 18,
Volume 1). Finally, newer methods of analysis
that focus on robust indices of central tendency

92 theory construction, model building

 ε3

Y3
β4 β2

β1
X Y2 Y1

β5 β3
Y4 ε2 ε1

ε4

Figure 5.9 Path Model for Defining Equations

and variability can be viewed in the context of lin-
ear equations but where the parameterization shifts
from means and variances to trimmed means, M
estimators, and the like (see Erceg-Hurn, Wilcox, &
Keselman, Chapter 19, Volume 1).
Consider the case where the initial theory is
represented as a path or influence diagram, such
as the theory in Figure 5.9. A path diagram can
be viewed as a pictorial representation of a set of
equations. There is a separate equation for each
endogenous variable in the theory. More specifi-
cally, given the aforementioned assumptions, each
endogenous variable can be expressed as being a
linear function of all variables with arrows going
directly to and explicitly touching the box repre-
senting the endogenous variable. For the model in
Figure 5.9, there are four linear equations that are
of primary theoretical interest because there are four
endogenous variables.
Using the above rule, the relevant equations are:
Y1 = α1 + β1 Y2 + ε1,
Y2 = α2 + β2 Y3 + β3 Y4 + ε2,
Y3 = α3 + β4 X + ε3, and
Y4 = α4 + β5 X + ε4,
where the various α are intercepts, the various β are
linear coefficients, and the various ε are disturbance
terms. Primary interest of the research is estimat-
ing and interpreting the parameters α1 through α4 ,
β1 through β5 , and the variances of ε1 through
ε4 relative to the variances of Y1 through Y5 . We
select statistical methods of analysis that provide
the best and most well-behaved estimates of these
parameters.
The rule for expressing a path diagram in terms
of a set of core equations also applies to models with
latent and observed variables, such as the model in
Figure 5.10. For this model, the structural model
has the following core equations:
LY = α1 + β1 LM + δ1
and
LM = α2 + β2 LX + β3 LZ + δ2 ,
with each equation focusing on a latent endoge-
nous variable as an outcome. We are interested in
estimating the parameters in these equations, but
the task is more challenging statistically because we
do not have direct access to a person’s scores on
the latent variables. Nevertheless, statisticians have
derived methods for obtaining such estimates (see
McDonald, Chapter 7, Volume 1).
The measurement model for the theory in
Figure 5.10 implies the following equations (again,
using the rule of specifying an equation for each
endogenous variable, in this case, the observed
endogenous variables):
X1 = α3 + β4 LX + ε1,
X2 = α4 + β5 LX + ε2,
Z1 = α5 + β6 LZ + ε3,
Z2 = α6 + β7 LZ + ε4,
M1 = α7 + β8 LM + ε5,
M2 = α8 + β9 LM + ε6,
Y1 = α9 + β10 LY + ε7,
and
Y2 = α10 + β11 LY + ε8.

jaccard 93
 ε1 ε2

ε5 ε6 ε7 ε8
X1 X2

β4 β5
M1 M2 Y1 Y2

LX
β2 β8 β9 β10 β11

β1
LM LY

β3

LZ δ2 δ1

β6 β7

Z1 Z2

ε3 ε4

Figure 5.10 Path Model with Latent Variables to Define Equations

It is not uncommon to express the set of core respect to two latent η, and p = 4 observed endoge-
equations in a theory in more succinct form than nous variables with respect to the two latent ξ . We
the above using matrix algebra. Readers unfamiliar further specify an mXm matrix (B) representing the
with matrix algebra can skip to the summary para- linear (regression) coefficients regressing the η onto
graph at the end of this section. As an example, the the η and a mXq matrix () representing the linear
above structural model is expressed by making dis- (regression) coefficients regressing the η onto the ξ :
tinctions between the latent endogenous variables


β11 β12 γ11 γ12
and the latent exogenous variables, defining a vector B= = .
of latent endogenous variables (η) and a vector of β21 β22 γ21 γ22
latent exogenous variables (ξ ) as In the example in Figure 5.10, let LY = η1 ,


 LM = η2 , LX = ξ1 , and LZ = ξ2 . Then
η1 ξ1
η= ξ= ,


η2 ξ2 0 β12 0 0
B= =
0 0 γ21 γ22
and a vector of intercepts (α) and a vector of
disturbance terms (δ) for the latent η as and the structural model is defined as



δ1 α1 η = α + Bη + ξ + δ.
δ= α= .
δ2 α2
The measurement model for the latent endoge-
In the present example, there are m = 2 latent nous and exogenous variables defines separate matri-
endogenous variables, q = 2 latent exogenous vari- ces of factor loadings for the linear (regression)
ables, r = 4 observed endogenous measures with coefficients from the latent endogenous variables to

94 theory construction, model building

the observed endogenous variables/indicators with variables, and X is a column vector of the observed
respect to them (λY which is an rXm matrix) and endogenous variables with respect to the latent
for the linear (regression) coefficients from the latent exogenous variables.
exogenous variables to the observed endogenous In addition to these matrices and vectors, one also
variables/indicators of them (λX , which is a pXq typically specifies a matrix of covariances between
matrix): the latent exogenous variables, a matrix of covari-
⎛ ⎞ ances for the disturbance terms, a matrix of covari-
λY11 λY12
⎜ λY ances for the error terms, and a vector of means
λY22 ⎟
λY = ⎜ 21
⎝ λY31 λY32 ⎠
⎟ for the latent exogenous variables, although these
are not typically part of the “core” equations in
λY41 λY42
⎛ ⎞ the model. Nevertheless, statisticians make use of
λX11 λX12 these matrices in the analysis of data and model
⎜ λX λX22 ⎟ evaluation.
λX = ⎜ 21
⎝ λX31 λX32 ⎠ .
⎟
In sum, given a well-developed causal theory that
λX4 λX42 takes the form of a path or influence diagram, one
can translate that diagram into a set of core equations
For the present example, let the four observed
that guide statistical analysis. This process is central
endogenous variables with respect to the latent
to building correspondence between our theoret-
endogenous variables be Y1 , Y2 , M1 , and M2 ,
ical models and our statistical models. The core
respectively, and the four observed endogenous vari-
equations can be represented in traditional algebraic
ables with respect to the latent exogenous variables
terms or in matrix form, but either way, they guide
be X1 , X2 , Z1 , and Z2 , respectively. Then the above
the choice of the statistical model for purposes of
matrices are
⎛ ⎞ data analysis.
λY11 0
⎜ λY 0 ⎟
λY = ⎜ ⎝ 0
21 ⎟
λY32 ⎠ Some Qualifications
0 λY42 For models with moderated causal effects, the
⎛ ⎞ rule for translating a path diagram into a set of core
λX11 0 equations must be slightly modified. Moderated
⎜ λX 0 ⎟
λX = ⎜ ⎝ 0
21 ⎟. effects are traditionally (but not always) represented
λX32 ⎠ by including product terms in the equation. Con-
0 λX42 sider a simple causal model of the following form:
A vector of intercepts (τY ) and a vector of error
terms (εY ) are defined for the observed indicators of Z
the latent endogenous variables:
⎛ ⎞ ⎛ ⎞
τY1 εY1
⎜ τY ⎟ ⎜ εY ⎟
τY = ⎜ 2 ⎟ ⎜ 2 ⎟
⎝ τY3 ⎠ εY = ⎝ εY3 ⎠,
X Y
τY4 εY4
and the observed indicators of the latent exogenous The core equation in this case describes the out-
variables: come variable Y, as a linear function of the two
⎛ ⎞ ⎛ ⎞ variables involved in the moderated relationship and
τX1 εX1
⎜ τX ⎟ ⎜ εX ⎟ the product of the two variables:
τX = ⎜ 2 ⎟ ⎜ 2 ⎟
⎝ τX3 ⎠ εX = ⎝ εX3 ⎠,
Y = α + β1 X + β2 Z + β3 XZ + δ.
τX4 εX4
The coefficient associated with the product term
and the measurement models are defined as:
reflects the degree of moderation that is operating.
Y = τY + λY η + εY Traditionally, the component parts of the product
term are included in the core equation as separate
X = τX + λ X ξ + ε X ,
predictors in addition to the product term because
where Y is a column vector of the observed endoge- doing so protects against arbitrary scaling affect-
nous variables with respect to the latent endogenous ing conclusions. For elaboration, see Blanton and

jaccard 95
Jaccard (2006) and Yuan and Schuster (Chapter
18, Volume 1). The above equation suggests that
another way of diagramming a moderated effect is
to include three separate direct effects, one from
each of the component parts of the product and a
third from the product term. Interaction analysis
does not have to be pursued through product terms
in a linear equation (Wilcox, 2005), but this is the
most common approach.
Another qualification to specifying core equations
is whether the relationship between variables is non-
linear rather than linear. For example, the relation-
ship between an outcome variable and a causal vari-
able might be characterized by a logarithmic func-
tion, an exponential function, a power function,
a polynomial function, or a trigonomic function,
among others. In such cases, the form of the core
equation differs from those specified earlier. Non-
linear modeling can be complex and consideration
of such models is beyond the scope of this chapter.
Readers are referred to Jaccard and Jacoby (2010)
and Yuan and Schuster (Chapter 18, Volume 1).
Although there are other nuances that can fur-
ther shape the nature of the equations one derives
from a causal theory, the general point I stress here is
that (1) it is possible to represent most causal mod-
els in the form of one or more equations and (2)
that translating a model into equation form is a fun-
damental step for choosing an appropriate analytic
method for testing the viability of a causal model.
Additional Considerations for Choosing
Analytic Strategy
With a carefully specified set of equations that are
theoretically guided, the next task of the researcher
is to choose a data analytic method to estimate
the parameters in those equations. The present
section considers the role of facets that must be
taken into account when making such decisions,
including measurement/metric considerations, the
use of full versus limited information estimation,
distributional assumptions of estimation strategies,
and the anticipated presence of outliers. Of course,
there are a host of more nuanced considerations that
researchers must account for, and these are devel-
oped in the different chapters in this volume. My
focus is on categories of more general considerations.
Measurement and Metric Considerations
Strategies for estimating parameters in core
equations, or the act of defining the parameters
themselves, are influenced by the psychometric
96 theory construction, model building
properties of the measures of the variables that
comprise the theory. One important metric consid-
eration is the level of measurement of the measures.
The classic distinction between nominal, ordinal,
interval, and ratio level measurement is particularly
important, because different methods of analysis
often are called for depending on the operative levels
of measurement. Nominal measurement involves
the assignment of numbers to levels of variable that
are categorical in nature and that have no inher-
ent ordering on an underlying dimension. For the
variable gender, a researcher might assign the val-
ues of 0 to females and 1 to males. The fact that
one number is larger than the other has no sub-
stantive interest. The numbers function much like
non-numeric labels.
Ordinal, interval, and ratio level measures can
be defined relative to the measurement equations
between quantitative latent variables and observed
measures described earlier. In describing the prop-
erties of these levels of measurement, I will assume
no measurement error is operating, to simplify the
presentation. Assuming no measurement error, an
interval level measure is one that is a linear function
of the underlying latent variable—that is,
X = α + βLX,
where X is the observed measure and LX is the latent
variable. The intercept and the linear coefficient
can take on any value. For ratio level measures, the
underlying latent variable, LX, should have a mean-
ingful conceptual zero-point (such as zero weight
when measuring how heavy an object is) and the
intercept in the measurement equation must equal
zero (as would be the case for a weight measure
in pounds or a measure in grams). For ordinal
measures, the relationship between X and LX is non-
linear but monotonic. For example, X might be a
power function of LX, in which case the measure
has ordinal properties.
As discussed in the other chapters in this volume,
analytic strategies for parameter estimation vary as a
function of these measurement levels. For example,
when a measure of an exogenous variable in a causal
model is nominal, it is common to represent that
variable in a core equation using dummy variables
(Chapter 18). If one wants to compare two groups
on their average heights, then a measure of height
that has ordinal properties is potentially problem-
atic and not well-suited to comparing means on the
underlying latent construct of height.
An important but underappreciated point when
taking measurement levels into account is the fact
that the distinction between the different measure-
ment levels is best viewed as approximate rather than
all-or-none. Metric properties are not inherent in
scales but, rather, are inherent in data and, hence,
are influenced by all of the facets of data collection.
The extent to which a measure has interval prop-
erties not only depends on the scale used to make
observations but also possibly on the particular set of
individuals on which the observations are made, the
time at which the data are collected, the setting in
which the data are collected, and so on. Consider the
following simplistic yet pedagogically useful exam-
ple. The height of five individuals is measured on
two different metrics, inches and a rank order of
height:
Individual Height in Inches Rank Order Height
A 72" 5
B 71" 4
C 70" 3
D 69" 2
E 67" 1
As is well known, the measures taken in inches
have interval level properties. For example, a differ-
ence of 1 between any two scores corresponds to the
same physical difference on the underlying dimen-
sion of height. The actual height difference between
individuals A and B corresponds to the same true
underlying height difference between individuals C
and D, and the metric reflects this (i.e., 72−71 = 1
and 70 − 69 = 1). Similarly, the difference between
D and E is 69 − 67 = 2, and the difference
between A and C is 2. These differences also reflect
the same amount on the underlying dimension of
height. Note, however, that these properties do not
hold for the rank order measure. The difference
in scores between individuals A and B is 1 (i.e.,
5 − 4), and the difference in scores for individu-
als D and E is also 1 (i.e., 2 − 1). These identical
differences correspond to differing degrees of height
disparities on the underlying dimension of height
(i.e., the true difference between individuals D and
E is larger than the true difference between indi-
viduals A and B, as is evident for the measure
using inches). For these individuals, the rank order
measures have ordinal properties but not interval
properties.
Now consider five different individuals with the
following scores:
Individual Height in Inches Rank Order Height
A 72" 5
B 71" 4
C 70" 3
D 69" 2
E 68" 1
Note that for these five individuals, the rank order
measure has interval level properties. The difference
in scores between individuals A and B is 1, as is
the difference between individuals D and E. These
differences correspond to the exact same amount
on the underlying physical dimension. In this case,
what we think of as traditionally being an ordinal
“scale” actually yields measures with interval level
properties. Suppose that individual E was not 68”
tall but instead was 67.9” tall. In this case, the rank
order measure is not strictly interval. But it is close
and probably can be treated as if it is interval level
without adverse effects.
This example illustrates that the crucial issue is
not whether a set of measures is interval or ordinal.
Rather, the critical issue is the extent to which a set of
measures approximates interval level characteristics.
If the approximation is close, then the data often
can be effectively analyzed using statistical methods
that assume interval level properties. If the approx-
imation is poor, an alternative analytic strategy is
called for. In this sense, we often can apply statis-
tical strategies that assume interval level measures
to ordinal data without consequence as long as the
ordinal data reasonably approximate interval level
properties.
Some researchers confuse the concept of measure-
ment level with measurement precision. Precision
refers to the number of scale points of a measure,
such as a 5-point scale, a 10-point scale, or a 21-
point scale. Measures can have interval level proper-
ties, for example, but be imprecise, or they can have
ordinal properties yet be relative precise. Precision
of a measure may shape the way that one chooses to
model data to evaluate a theory, but to the extent
it does, it is not because of levels of measurement
of the measures. With coarse and imprecise mea-
sures of continuous constructs, analytic methods
that assume high levels of precision can be problem-
atic and alternative analytic strategies are required
(see Yuan & Schuster, Chapter 18, Volume 1).
Another measurement matter that shapes the
method of data analysis is whether the outcome mea-
sure is discrete and zero-bounded (such as measures
of counts, like the number of times an adolescent has
jaccard 97
smoked marijuana) and whether measures are cen-
sored. Censoring occurs when a value occurs outside
the range of a measuring instrument. For example,
a bathroom scale might only measure up to 300
pounds. If a 350-pound individual is weighed using
the scale, then we would only know that the indi-
vidual’s weight is at least 300 pounds. For details of
how these metric qualities affect analytic strategy, see
Long (1997) and Yuan and Schuster (Chapter 18,
Volume 1).
Full Information versus Limited
Information Estimation
Given a set of equations derived from a causal
theory, one approach to estimating parameters in
those equations is to use a full information estima-
tion approach. In full information estimation, the
coefficients in all equations in the model are esti-
mated simultaneously. This can be contrasted with
a limited information estimation approach where
the coefficients in the equations are estimated one
equation at a time or, alternatively, focusing on only
a portion of the larger model rather than the entire
model. Full information estimation is common in
structural equation modeling (SEM), but there also
are limited information variants of SEM (see Bollen,
2001). Full information estimation has the advan-
tage that parameter estimates typically (but not
always) are more efficient (in a strict statistical sense
of the term) than limited information estimators as
long as the model being tested is correctly specified.
Limited information estimation has the advantage
that it often is less susceptible to adverse effects of
specification error, because the effects of specifica-
tion error are limited to the particular portion of
the model where the error occurs. It also allows one
to use the strongest methods of analysis available
dependent on the properties of measures and vari-
ables in different portions of the model rather than
applying the same homogenous algorithm through-
out the model. A decision point for analysts when
choosing a statistical method to estimate parameters
defined by a theoretical model is whether to adopt a
full information or limited information estimation
strategy.
Statistical Assumptions
Another issue that analysts must consider when
moving from equations to estimation is the dis-
tributional assumptions of the estimation method
and whether they are appropriate for the task at
98 theory construction, model building
hand. Three assumptions are typical: (1) normal-
ity of scores in the population, (2) homogeneity
of variances in the population, and (3) indepen-
dence of replicates. The ways these assumptions
are instantiated vary by analytic method, and other
assumptions also can come into play (Maxwell &
Delaney, 2004). I focus on the above assumptions
primarily to make general points about approaches
to distributional assumptions. I will make reference
to the robustness of a test to violations of underlying
assumptions. A statistical test is said to be robust to
violations of assumptions if (1) the nominal Type I
error rate (alpha level) set by the investigator a priori
(usually 0.05) is maintained in the face of assump-
tion violations and (2) the statistical power of the
test is relatively unaffected by assumption violations.
For a more technical and nuanced discussion of
robustness, see Wilcox (2005).
A common perception of many researchers is
that traditional F and t tests in analysis of vari-
ance (ANOVA) and regression are quite robust to
violations of normality and homogeneity of vari-
ance assumptions. Because of this, these methods
are often applied to data where the population
assumptions are tenuous. Studies have shown that
ANOVA and regression are not necessarily robust
to assumption violations of normality and variance
heterogeneity (see Keselman et al., 1998; Maxwell
& Delaney, 2004; Wilcox, 2005). One strategy for
dealing with assumption violations is to perform
a preliminary test of the viability of the assump-
tion in question and, if the test suggests a problem,
perform a metric transformation or use a robust
analytic alternative. This two-step strategy is contro-
versial for several reasons. First, many preliminary
tests lack power without large sample sizes and
yield nonsignificant results for testing an assumption
violation, even when the violation is problematic
(Wilcox, Charlin, & Thompson, 1986; Wilcox,
2003). Second, the crucial issue is not whether the
null hypothesis of normality or variance homogene-
ity can be rejected but, rather, estimating the degree
to which the assumption is violated and making
a decision as to whether the degree of violation is
consequential. This requires documenting the mag-
nitude of the assumption violation in the sample
data and then taking sampling error into account
when making decisions. For example, we might find
that a variance ratio comparing the variances of two
groups is 4.0, with a margin of error of plus or minus
3.0. The margin of error suggests that the variance
could be as large as 7.0, which could be problem-
atic. Unfortunately, it is rare for researchers to take
margins of error into account when evaluating pre-
liminary tests. Third, many tests of non-normality
are based on asymptotic theory and only perform
adequately with large sample sizes (Shapiro & Wilk,
1965). However, with large N , such tests tend to
detect minor departures from normality that may
be of little consequence. In addition, normality tests
can be differentially sensitive to different types of
non-normality. Some tests are sensitive mostly to
skew, whereas others are sensitive mostly to kurtosis.
Fourth, the preliminary tests often make assump-
tions in their own right and may perform poorly
when their assumptions are violated. For example,
many tests of variance homogeneity assume the pop-
ulation data are normally distributed (Carroll &
Schneider, 1985; Keyes & Levy, 1997; Parra-Frutos,
2009). If the population data are non-normal, then
the preliminary test of variance homogeneity may
be invalid. Fifth, using preliminary tests as a screen
can change the sampling distribution of F tests and
t tests in unpredictable ways. Although it seems rea-
sonable, the strategy of conducting preliminary tests
of model assumptions faces numerous difficulties.
Transformation strategies for dealing with
assumption violations have also been criticized. For
example, Budescu and Appelbaum (1981) found
that transformations to address variance heterogene-
ity can create more problems than they solve in
inferential tests because they can adversely impact
normality (see also Blaylock et al., 1980; Milligan,
1987; Doksum & Wong, 1983; Wilcox, 1996,
1998). Transformed variables often are difficult to
interpret (e.g., the mean log of annual income is
not easily interpreted). In models with multiple
predictors, transformations of the dependent vari-
able can create specification error that undermines
covariate control because it alters the relationships
between the outcome variable and all predictors in
the equation. Years ago, before high-speed comput-
ers were widespread, analysts had little choice but
to use transformations to make measures conform
to the assumptions of the limited number of para-
metric statistical strategies available. Such practices
are rarely needed today given the array of modern
methods of analysis that are available.
A growing number of statisticians recommend
that analysts simply abandon the more traditional
tests that make strong population assumptions un-
less they are confident in assumption viability based
on theory or extant research. Rather, analysts should
routinely use modern-day robust methods of analysis
or, at the very least, routinely supplement traditional
methods with modern robust methods (Keselman
et al., 2008; Wilcox, 2005). These scientists recog-
nize that cases may occur where defaulting to robust
analytic strategies will result in some loss of statistical
power and inaccurate probability coverage of confi-
dence intervals (CIs). However, the argument is that
in the long run, the use of robust methods will result
in better Type I error protection, increased power to
detect effects, and CIs that more accurately reflect
the desired probability coverage (Wilcox, 1998). Of
course, it is always important to explore the shapes of
distributions and dispersions of data. However, the
recommendation is to view traditional tests of model
assumptions and remedial strategies based on trans-
formations with caution, deferring instead to the use
of more modern robust analytic methods.
Earlier, I discussed causal theories that are multi-
level in nature, such as theories of how characteristics
of organizations as well as characteristics of indi-
viduals affect the behavior of individuals within an
organization. Research that tests multilevel mod-
els often strategically samples organizations (called
Level 2 units) and individuals nested within those
organizations (called Level 1 units). In such cases,
the statistical assumption of independent residu-
als/errors often is untenable because of the impact
that individuals within an organization have on
one another, either directly or indirectly. In such
cases, specialized statistical methods must be used
to deal with the dependencies (see Yuan & Schuster,
Chapter 18, Volume 1).
In sum, the choice of an analytic method to
use with data is impacted by the equations used to
represent a theory, the psychometric properties of
the measures, whether one seeks full information
estimation or limited information estimation, and
the population distributional assumptions that the
statistical tools available to the researcher make rela-
tive to the actual population distributions. A grow-
ing number of scientists suggest adopting robust
methods in favor of traditional methods because of
the complexities of two step strategies that rely on
preliminary tests of assumptions.
Outliers
Outliers are unusually small or large scores that
distort basic trends in the data. For example, for
the scores 2, 3, 4, 5, 6, 7, 8, 9, 10, and 50, the
last score is an outlier that distorts the mean and
makes the use of the mean suspect as a way to char-
acterize the central tendency of the data. Simple
methods for outlier detection compare the results
of an analysis when the case is included versus the
jaccard 99
results of an analysis when the case is deleted. Such
approaches, however, can be nondiagnostic when
multiple outliers are present. For example, if there
are two individuals in an analysis who distort a mean
upward, deleting only one of them may not reveal
an “outlier” effect as long as the second outlier is still
in the data. Only when both outliers are removed
is their distorting character revealed. Outlier iden-
tification is a complex enterprise, with some of the
most sophisticated work being pursued in the liter-
ature on robust statistics (see Wilcox, 2003, 2005,
for elaboration).
Wilcox (1998, 2006) objects to applying tradi-
tional inferential methods to data that have elim-
inated outliers based on simple outlier detection
methods. He argues that doing so invalidates the sta-
tistical theory on which the inferential tests are based
because of dependencies that outlier elimination cre-
ates. Others recommend conducting analyses with
and without outliers to determine if conclusions
change. If conclusions do change, then one moves
forward with any conclusions on a tentative basis.
Probably the most effective strategy for dealing with
outliers is to focus on parameter estimation methods
that are resistant to outliers. For an introduction to
these methods, see Wilcox (2005).
Model Selection
Model selection refers to the process of choosing
what one believes is the “best” model for describ-
ing a phenomenon from a set of candidate models.
The set of plausible models might consist of many
models or it might consist of just one or two
models. Criteria for model selection can be consid-
ered in general terms using a mindset of specifying
criteria to evaluate the overall quality of a theory
in general; or they can be discussed in specific,
quantitative terms when choosing between compet-
ing models within an experiment. I consider both
perspectives.
General Criteria for Evaluating Theories
Consensual validation is one basis by which the-
ories are evaluated. This refers to the degree of con-
sensus among the scientific community about the
validity of the theory. If a theory enjoys widespread
acceptance, then it is seen as being a “good” theory.
Shaw and Costonzo (1982) have argued that three
criteria are crucial for the acceptance of a theory.
First, the theory must be logically consistent—that
is, the theoretical statements within the concep-
tual system must not be contradictory. Second, the
100 theory construction, model building
theory must be in agreement with known data and
facts. Third, the theory must be testable—that is,
a theory must ultimately be subject to empirical
evaluation.
In addition to these criteria, Shaw and Costonzo
(1982) have specified six additional features of a the-
ory that are desirable but not necessarily critical. First,
a theory should be stated in terms that can be under-
stood and communicated to most other scientists.
Second, the theory should be parsimonious in that it
adequately explains a phenomenon but with a min-
imum of concepts and principles. All other things
being equal, preference is given to theories that make
fewerassumptions.Third, althoughwerecognizethat
theories are occasionally so novel that they upset the
theoretical apple cart, a theory should be consistent
with other accepted theories that have achieved con-
sensus among the scientific community—that is, it
should be able to be integrated into existing bod-
ies of theory. A fourth desideratum is scope. Other
things being equal, the greater the range of the the-
ory, the better it is thought to be. That said, there are
times when narrow range theories tend to hold up
better over time than broad range theories. Creativ-
ity or novelty is a fifth criterion sometimes suggested
for evaluating a theory. A theory that explains the
obvious is generally not as highly valued by the scien-
tific community as one that provides a novel insight
into an interesting phenomenon. Finally, many sci-
entists suggest that a good theory is one that generates
research activity.
Brinberg and McGrath (1985) have noted that
the various theory desiderata sometimes conflict
with each other. For example, parsimonious theories
tend to be more limited in scope. As such, the-
orists often make trade-offs as they construct the-
ories to maximize what is valued by the scientific
community.
Choosing Between Models in a Given Study
As one pursues the theory construction process,
one may develop competing theories that either
make opposite predictions or that account for the
same phenomena but using different explanations
and assumptions. Faced with such scenarios, we
design research to help us choose between the com-
peting theories. Many scientists consider research
that chooses between two or more logical and plau-
sible theories to be inherently more interesting than
studies that yield results regarding a single theory
(Platt, 1964).
In some cases, competing models making quali-
tatively opposite predictions about how data should
pattern themselves in a given study. In these cases,
deciding which theory better accounts for the data
is reasonably straightforward. As an example, sup-
pose we describe the personal qualities of a political
candidate to a person who he or she has not heard of
by providing the person with three pieces of infor-
mation about the candidate. Suppose that the three
pieces of information all are quite positive (e.g., the
candidate is said to be honest, smart, and empathic).
For purposes of developing this example, we char-
acterize how positive each piece of information is
considered to be using a metric that ranges from
0 to 10, with higher numbers reflecting higher
degrees of positivity. Suppose we want to predict
how favorable a person will feel toward the can-
didate based on the three pieces of information.
One plausible model states that the overall degree
of favorability is a function of the sum of the posi-
tivity of each individual piece of information—that
is, that people “tally up” the affective implications of
each piece of information when forming their over-
all impression. Another model, by contrast, specifies
a different function—namely, an averaging func-
tion. In this case, the overall feeling of favorability
is thought to be a function of the average positivity
of the information presented.
What are the implications of specifying the func-
tion as being summative versus averaging in form?
It turns out, they are considerable. Let’s explore
the summation model first. Suppose the positivity
values of the three pieces of information are 8, 8,
and 8 respectively. The overall feeling of favorability
toward the candidate will be a scaled function of
8 + 8 + 8 = 24. Now suppose we describe a sec-
ond candidate to this person using the same three
pieces of information but we add a fourth descrip-
tor to them (cunning) that has a positivity value of
4. According to the summation model, the over-
all feeling of favorability toward this new candidate
will be a scaled function of 8 + 8 + 8 + 4 = 28, and
the person will prefer the second candidate to the
first candidate. Psychologically, it is as if the sec-
ond candidate has all the same qualities as the first
candidate, and then “as a bonus,” you get a fourth
positive attribute as well. Hence, the person prefers
the second candidate to the first candidate.
Now consider instead the averaging function.
The overall feeling toward the first candidate is pre-
dicted to be (8 + 8 + 8)/3 = 8.0, and the overall
feeling toward the second candidate is said to be (8
+ 8 + 8 + 4)/4 = 7.0. In the averaging model, exactly
the reverse prediction is made in terms of candi-
date preference—namely, the person now will prefer
Table 5.1. Correlations for Intelligence
Example
Y1 Y2 Y3 Y4 Y5
Y1 1.00 0.72 0.63 0.54 0.45
Y2 0.72 1.00 0.56 0.48 0.40
Y3 0.63 0.56 1.00 0.42 0.35
Y4 0.54 0.48 0.42 1.00 0.30
Y5 0.45 0.40 0.35 0.30 1.00
the first candidate to the second candidate. Psycho-
logically, the first candidate has nothing but very
positive qualities, whereas the second candidate has
very positive qualities but also some qualities that are
only somewhat positive. The person prefers the first
candidate, who has nothing but very positive quali-
ties, to the second candidate, who has very positive
qualities but also moderately positive qualities.
In the above example, the summation and averag-
ing models make opposite predictions about candi-
date preference, and it is straightforward to choose
between the theories based on an empirical study
that asks people which of the two candidates they
prefer. Of course, if the study found that neither
candidate tended to be preferred, then this would
question both models.
Although studies like the above are compelling, it
is common to conduct studies where competing the-
ories do not make qualitatively opposite predictions,
but, rather, they make predictions about how data
should pattern themselves that allows researchers to
choose between them. As an example, in the area
of intelligence testing, theorists agree that there are
different kinds of intelligence and cognitive abilities,
such as math skills, vocabulary breadth, spatial skills,
motor skills, and memory. Studies suggest that mea-
sures of these constructs are moderately correlated
with one another. Some theorists believe that the
correlations among them result from the common
influence of general intelligence, sometimes called g .
We refer to this as a “single factor model” because it
posits that the correlations among the measures can
be accounted for by a single underlying factor. Other
theorists believe that g does not exist and that the
correlations among the different abilities are a result
of a more complex constellation of determinants
consisting of multiple factors.
Suppose in a population of individuals the corre-
lations among the five variables are as presented in
Table 5.1. It can be shown using somewhat involved,
jaccard 101
but tedious, algebra that the one factor model pre-
dicts that the correlations should pattern themselves
in a certain way—that is, they should follow cer-
tain mathematical regularities. For example, if we
choose any two columns and ignore entries where
a 1.00 occurs, then the ratios of the respective row
entries for the two columns should be equal in value
to the comparable ratio for any row in the matrix.
For example,
For columns 1 and 2:
.63/.56 = .54/.48 = .45/.40 = 1.125
and
For columns 2 and 5 :
.72/.45 = .56/.35 = .48/.30 = 1.600.
By contrast, models that posit more complex pat-
terns of underlying determinants (e.g., a two-factor
model, a three-factor model) predict a different set
of regularities, which we do not elaborate here in
the interest of space (see McDonald, 1985). The
model selection task is to choose which of the com-
peting models (e.g., a one-factor model, a two-factor
model, etc.) is most consistent with the data.
Statisticians have developed quantitative indices
of fit that represent the overall degree of correspon-
dence between the predicted pattern of data by a
model and the observed data pattern. When there
are competing models, a fit index is derived for
each model and then the fit indices are compared to
identify the model that best accounts for the data.
The technical details of such comparisons can be
quite complex, and these are elaborated through-
out the many chapters in this volume. The point
I am emphasizing here is that even if two models
do not make qualitatively opposite predictions, as
was the case for the example on summation versus
averaging, models often can be compared in terms
of their relative consistency with the data, with one
model ultimately being declared as being more con-
sistent with the data than another model based on
the comparison of quantitatively defined fit indices.
There is, however, an additional complication in
model selection. In practice, it would be surprising
if sample data yielded a set of correlations that per-
fectly followed the predicted regularities of, say, a
one-factor model, even if the one-factor model was
operative in the population. This is because sam-
ple correlations are subject to sampling error and
will randomly deviate to a greater or lesser extent
from the true population correlations. Even if this
is the case, one expects that if the one factor model
102 theory construction, model building
is true in the population, then the sample correla-
tion matrix should at least reasonably approximate
the regularities predicted by the one-factor model.
A challenge for scientists when comparing models
vis-à-vis indices of fit (or, for that matter, evaluating
a single model using an index of fit) is to take into
account such sampling error.
A final strategy that one encounters when scien-
tists evaluate competing models is the case where
scientists prefer the model that explains the most
variation in an outcome variable. In these cases,
one typically cannot differentiate models in terms
of degree of fit to the data, as in the previous
examples we considered. Nevertheless, one can dif-
ferentiate between them in terms of whether one
model accounts for significantly more variation in
an outcome than the other model. For example,
when explaining adolescent drug use, one might
compare a model that assumes drug use is a sim-
ple additive function of gender and grade in school
versus a model that assumes drug use is an additive
function of gender, grade, plus the interaction effect
between gender and grade. If the models account
for about the same amount of variation in drug
use, then preference for the first model will take
precedence on grounds of parsimony and because
taking into account the interaction effect does not
seem to matter. By contrast, if the three-parameter
model explains substantially more variation in drug
use than the two-parameter model, then the three-
parameter model is preferred.
As scientists consider the complex issues sur-
rounding model selection, a paramount concern
is not to mistakenly embrace a model that is mis-
specified. A misspecified model is a model that is
wrong because it (1) left out an important variable
from the model whose omission biases parame-
ter estimates and leads to faulty conclusions (also
called left-out-variable-error or LOVE problems);
(2) assumed one type of function between variables
(e.g., linear) when, in fact, a different type of func-
tion was operative (e.g., a threshold function), with
such misspecification leading to faulty conclusions;
and/or (3) incorrectly modeled the temporal dynam-
ics within a model in such a way that parameter
estimates and conclusions are non-trivially biased.
Throughout this volume, chapters elaborate on
issues of model misspecification and ways to gain
perspectives on it. Cudeck (1989) has argued that
the crucial issue is not whether a model is misspeci-
fied because misspecification is inevitable in so much
of the research we conduct. Rather, the more cen-
tral issue is whether the degree of misspecification is
sufficiently large that it leads to conclusions that are
incorrect or misplaced.
In sum, model selection can involve two pro-
cesses: (1) evaluating a model by and of itself to
determine whether it meets criteria the scientific
community judges to be important (such as being
logical, testable, consistent with known facts/data,
etc.), and (2) quantitatively comparing a model with
other competing models within a given study to
identify which model best accounts for the observed
data or explains the most variation in an out-
come. Three strategies are typically used to choose
between models. First, one designs a study where
the competing models make qualitatively opposite
predictions. Second, one deduces from the model
how data should pattern themselves and then derives
a quantitative (fit) index representing the corre-
spondence between the predicted patterns and the
observed patterns. The model that “fits” the data
best is the preferred model, everything else being
equal. Third, one selects a model that can explain
the most meaningful variation in a targeted outcome
variable. There are many nuances surrounding the
above strategies, and these are elaborated in other
chapters in this volume.
Concluding Comments
This chapter has described general issues involved
in (1) building causal theories, (2) translating those
theories into a set of mathematical representations,
(3) choosing an analytic strategy to estimate param-
eters and sampling error vis-à-vis those equations,
and (4) choosing the “best” model from a set of
competing models. There is a vast array of technical
issues to consider as one approaches these four tasks,
and these technicalities are elaborated throughout
the chapters in this volume. My purpose here was to
provide a “big picture” view of the broader enterprise
so that we can “see the forest through the trees” as
we approach the task of building cumulative bodies
of knowledge and solving social problems.
References
Blanton, H. & Jaccard, J. (2006). Tests of multiplicative mod-
els in psychology: A case study using the unified theory of
implicit attitudes, stereotypes, self esteem and self concept.
Psychological Review, 122, 155–169.
Blaylock, J., Salathe, L. & Green, R. (1980). A note on the Box-
Cox transformation under heteroskedasticity. Western Journal
of Agricultural Economics, 45, 129–135.
Bollen, K.A. (2001). Two-stage least squares and latent variable
models: Simultaneous estimation and robustness to misspec-
ifications. In Robert Cudeck, Stephen Du Toit, & Dag
Sörbom (Eds.), Structural equation modeling: Present and
future, A festschrift in honor of Karl Jöreskog. (pp. 119–138).
Lincoln, IL: Scientific Software.
Bollen, K.A., & Curran, P.J. (2006). Latent curve models: A
structural equation approach. Hoboken, NJ: Wiley.
Brinberg, D. & McGrath, J. (1985). Validity and the research
process. Newbury Park: Sage.
Brodbeck, M. (1968). Readings in the philosophy of the social
sciences. New York: Macmillan.
Brown, T. (2006). Confirmatory factor analysis for applied research.
New York: Guilford.
Budescu, D. & Appelbaum, M. (1981). Variance stabilizing
transformations and the power of the F test. Journal of
Educational and Behavioral Statistics, 6, 55–74.
Bunge, M. (1961). Causality, chance, and law. American Scientist,
69, 432–488.
Carnap, R. (1971). A basic system of inductive logic, Part 1. In
R. Carnap & R. C. Jeffrey (Eds.), Studies in inductive logic
and probability. Vol. I, (pp. 18–39). Berkeley, CA: University
of California Press.
Carroll, RJ, & Schneider, H. (1985). A note on Levene’s test
for equality of variances. Statistics & Probability Letters, 3,
191–194.
Cartwright, N. (2007). Hunting causes and using them. New York:
Cambridge University Press.
Collins, L. (2006). Analysis of longitudinal data: The integration
of theoretical model, temporal design, and statistical model.
Annual Review of Psychology, 57, 505–528.
Coombs, C., Dawes, R. & Tversky, A. (1970). Mathematical
psychology: An elementary introduction. Englewood Cliffs, NJ:
Prentice Hall.
Cudeck, R. (1989). Analysis of correlation matrices using covari-
ance structure models. Psychological Bulletin, 105, 317–327.
Doksum, K.A., & Wong, C.-W. (1983). Statistical tests based
on transformed data. Journal of the American Statistical
Association, 78, 411–417.
Dubin, R., (1976). Theory building in applied areas. In Marvin
D. Dunnette (Ed.), Handbook of industrial and organizational
psychology. (pp. 17–39). Chicago: Rand McNally.
Erceg-Hurn, D. M., Wilcox, R. R., & Keselman, H. H. (2012).
Robust statistical estimation. In T. D. Little (Ed.), The Oxford
Handbook of Quantitative Methods (Vol. 1, pp. 388–406).
New York: Oxford University Press.
Finkel, S. (2008). Linear panel analysis. In S. Menard (Ed.),
Handbook of longitudinal research (pp. 475–504). New York:
Elsevier Press
Frank, P. (1961). Modern science and its philosophy. New York:
Collier Books.
Green, P. & Tull, D. S. (1975). Research for marketing decisions.
Englewood Cliffs, NJ: Prentice-Hall.
Jaccard, J. & Jacoby, J. (2010). Theory construction and model
building skills: A practical guide for social scientists. New York:
Guilford.
Kaplan, A. (1964). The conduct of inquiry. San Francisco, CA:
Chandler.
Keselman, H., Algina, J., Lix, L., Wilcox, R. & Deering, K.
(2008). A generally robust approach for testing hypotheses
and setting confidence intervals for effect sizes. Psychological
Methods, 13, 110–129.
Keselman, H. J., Huberty, C. J., Lix, L. M., Olejnik, S., Cribbie,
R., Donahue, B., et al. (1998). Statistical practices of educa-
tional researchers: An analysis of their ANOVA, MANOVA,
and ANCOVA analyses. Review of Educational Research, 68,
350–386.
jaccard 103
Keyes, T. & Levy, M.S. (1997). Analysis of Levene’s test under
design imbalance. Journal of Educational and Behavioral
Statistics, 22, 227–236.
Lewis, D. (2000) Causation as influence. Journal of Philosophy,
97, 182–197.
Long, S. (1997).Regression models for categorical and limited
dependent variables. Thousand Oaks, CA: Sage.
Maxwell, S. E. & Delaney, H.D. (2004). Designing experiments
and analyzing data: A model comparison perspective. Mahwah,
NJ: Erlbaum.
McDonald, R. (1985). Factor analysis and related methods.
Hillsdale, NJ: Erlbaum.
McDonald, R. P. (2012). Modern test theory. In T. D. Little
(Ed.), The Oxford Handbook of Quantitative Methods (Vol. 1,
pp. 118–143). New York: Oxford University Press.
Milligan, G. (1987). The use of the arc-sine transformation
in the analysis of variance. Educational and Psychological
Measurement, 47, 563–573.
Morgan, S. & Winship, C. (2007). Counterfactuals and causal
inference. New York: Cambridge University Press.
Pap, A. (1962). An introduction to the philosophy of science.
Glencoe, IL: The Free Press of Glencoe (MacMillan).
Parra-Frutos, I. (2009). The behaviour of the modified Lev-
ene’s test when data are not normally distributed. Journal
Computational Statistics, 24, 671–693.
Pearl, J. (1999). Causality: Models, reasoning, and inference. New
York: Cambridge University Press.
Platt, J. (1964). Strong inference. Science, 146, 347–353.
Rubin, D. (1974). Estimating causal effects of treatments in ran-
domized and nonrandomized studies. Journal of Educational
Psychology, 66, 688–701.
Rubin, D. (1978). Bayesian inference for causal effects: The role
of randomization. Annals of Statistics, 6, 34–58.
Russell, B. (1931). The scientific outlook. London: George Allen
& Unwin.
Schumacker, R. E. & Lomax, R. G. (2004). A beginner’s guide to
structural equation modeling. Mahwah, NJ: Erlbaum.
104 theory construction, model building
Shadish, W. R., Cook, T. & Campbell, D. (2002). Experimental
and quasi-experimental designs for generalized causal inference.
Boston: Houghton Mifflin.
Shapiro, S. S. & Wilk, M. B. (1965). An analysis of variance test
for normality (complete samples). Biometrika, 52, 591–611.
Shaw, M. & Costonzo, P. (1982). Theories of social psychology.
New York: McGraw-Hill.
Sheth, J. N. (1967). A review of buyer behavior. Management
Science, 13, B718–B758.
Simon, H. A. & Newell, A. (1956) Models: Their uses and lim-
itations. In L.D. White (Ed.), The state of the social sciences.
(pp. 61–83). Chicago, IL: University of Chicago Press.
Sowa, J. (2000). Processes and causality. From www.jfsowa.com/-
ontology/causal.htm. Accessed August 15, 2011.
Torgerson, W. S., (1958) Theory and methods of scaling. New York:
Wiley.
Wilcox, R. (2006). Graphical methods for assessing effect size:
Some alternatives to Cohen’s d. Journal of Experimental
Education, 74, 353–367.
Wilcox, R.R. (1996). Statistics for the social sciences. New York:
Academic Press.
Wilcox, R. R. (1998). How many discoveries have been lost by
ignoring modern statistical methods. American Psychologist,
53, 300–314.
Wilcox, R.R. (2003). Applying contemporary statistical techniques.
San Diego, CA: Academic Press.
Wilcox, R.R. (2005). Introduction to robust estimation and
hypothesis testing (2nd ed.). San Diego, CA: Academic
Press.
Wilcox, R. R., Charlin, V. L., & Thompson, K. (1986). New
Monte Carlo results on the robustness of the ANOVA F, W,
and F* statistics. Communications in Statistics. Simulation and
Computation, 15, 933–944.
Yuan, K-H., & Schuster, C. (2012). Overview of statistical esti-
mation methods. In T. D. Little (Ed.), The Oxford Handbook
of Quantitative Methods, (Vol. 1, pp. 360–386). New York:
Oxford University Press.
 CHAPTER

Teaching Quantitative Psychology

6
Lisa L. Harlow

Abstract

This chapter strives to enliven quantitative psychology teaching and encourage statistical literacy.
Summaries of quantitative training, at the undergraduate and graduate levels, offer guidelines to
improve instruction and call for greater emphasis on measurement, research, and quantitative
methods. Strategies for effectively teaching quantitative psychology are suggested, including active,
hands-on learning for engaging students, e-learning and Web-based instruction, mentoring and role
models, and encouraging conceptual understanding. Quantitative students are encouraged to focus on
the nature of research questions, similarities and differences in statistical methods, and interpreting
findings with statistical tests, effect sizes, and confidence intervals. Future directions are offered
regarding model building, quantitative learning beyond the classroom through workshops, membership
in quantitative societies, and reading the quantitative literature. The reach of quantitative training
should be widened to more readily include those from disadvantaged, early career, and
under-represented groups to further strengthen the field and enlighten greater numbers about the
wonders of quantitative psychology.
Key Words: quantitative training, statistical literacy, engaging students, strategies for teaching, active
learning, mentoring, underlying concepts, widening quantitative reach

Introduction Experience and research make it all too apparent that

Readers of this volume are part of a rare and
unique subset of individuals who resonate with
the term quantitative psychology. When the topic
of psychology is discussed, whether by students,
faculty, or the general public, the qualifier “quan-
titative” does not always enter the conversation.
Students in psychology often delay taking required
statistics and research courses, short-changing their
ability to understand and develop scientific skills
necessary to open up their career options and further
the field of psychology (Rajecki, Appleby, Williams,
Johnson, & Jeschke, 2005). In more than 25 years
of teaching quantitative psychology, I have learned
not to be too surprised by cringing and flinching,
if only subtle, in students enrolled in my courses.
students often approach quantitative courses with
little interest or confidence, coupled with anxiety
andmisperceptionsaboutlearningstatisticalmaterial
(Ashcraft, & Kirk, 2001; DeVaney, 2010; Harlow,
Burkholder & Morrow, 2006; Onwuegbuzie, 2000;
Onwuegbuzie, & Wilson, 2003; Piotrowski, Bagui,
& Hemasinha, 2002). What compounds the prob-
lem is that students with high anxiety tend to have
poorer attitudes toward and lower performance in
quantitative studies (Budé, Van De Wiel, Imbos,
Candel, Broers, &Berger, 2007; Harlow, Burkholder
& Morrow, 2002; Mills, 2004; Rodarte-Luna &
Sherry, 2008; Tremblay, Gardner, & Heipel, 2000).
Fortunately, quantitative attitudes have been shown
to predict quantitative performance just as strongly as

105
pre-course quantitative skill in a class with engaging
activities to get students involved, suggesting that
it is worthwhile to try to reduce quantitative anxi-
ety by making quantitative learning more appealing
(Harlow, Burkholder, & Morrow, 2002).
Thus, it is probably not extraordinary that I find
it my greatest joy to try to engage and enlighten
students with the remarkable prospective of quan-
titative thinking and analysis. I go so far as to
argue that any student interested in psychology has
the making of a quantitative scientist, if only at
some latent, implicit level. What student has not
posed a possible theory for why something is, or
is not, a certain way? Doesn’t everyone wake up
each morning with one or more hypotheses about
how the day will emerge and which variables or fac-
tors will help bring about the desired effect, whether
small, moderate, or large? And, don’t we all think
about how much confidence we can place on our
hypotheses or expectations, depending on specific
mitigating or confounding circumstances? I believe
that we all engage in these activities, however for-
mal or informal, in which quantitative psychology
can play a pivotal role. Quantitative psychology
has the potential to empower and enlighten with
the training, skills, reasoning, and capabilities to
formalize these kinds of questions, helping us to
describe and understand the essence of myriad data
that come our way. With more advanced quantita-
tive training, we could further develop and analyze
intricate theoretical models that help explain and
predict complex processes and behaviors, integrat-
ing information necessary to inform and improve
interventions, policies, and the human condition
(Harlow, 2010; Rodgers, 2010).
In this chapter, I highlight some major issues and
topics that go into teaching quantitative psychology.
First, I provide an overview of quantitative training
to get an overarching picture of the field. Second,
I suggest a number of issues that should be taken
into account when teaching quantitative courses and
research that investigates ways to address these con-
cerns. Third, I offer several themes that run through
many statistical methods to help tie together the
numerous, seemingly isolated and obscure quanti-
tative procedures. Finally, I summarize the current
field of quantitative teaching and give recommenda-
tions for other options to supplement, enrich, and
expand statistical learning.
Overview of Quantitative Training
Aiken et al. (1990, 2008) surveyed more than
200 graduate training programs regarding statistics,
106 t e a c h i n g q u a n t i tat i v e p s yc h o l o g y
measurement, and methodology. Only 15% of the
graduate programs had a quantitative area, although
about 50% offered a minor in quantitative studies.
Almost all of the graduate programs required stu-
dents to take at least one or two quantitative courses
during their first year, although more than 25% sent
students to other departments to get this training.
Required quantitative courses usually included basic
analysis of variance (ANOVA) and some regres-
sion, with half to two-thirds of programs including
limited training in measurement or research. Com-
puter training in SPSS and SAS is often part of the
curriculum, with the possibility of EQS, AMOS,
Mplus, LISREL, and other software exposure in
more advanced courses (e.g., structural equation
modeling) in about half of the programs. Over a span
of almost two decades between surveys, quantitative
training had not improved much; over time there
was slightly more emphasis on measurement and
deterioration in coverage of research design. Much
improvement in advanced methods training is still
needed in most departments to impart the expertise
needed to effectively compete in quantitative studies
and fill needed positions in the workforce.
It is becoming readily apparent that efforts are
needed to increase the number of quantitatively
trained individuals. A recent report from an Amer-
ican Psychological Association (APA) Task Force
to Increase the Quantitative Pipeline (Aiken et al.,
2007) reported that in recent history, whereas there
were approximately two applicants for each non-
quantitative doctoral level job (e.g., cognitive, devel-
opmental, social), there were almost 2.5 jobs for
every quantitatively trained PhD student. Further,
the focus of quantitative methodology is moving
away from emphasis on learning a set of specific
methods and procedures and instead placing greater
priority on developing a broader vision of quanti-
tative science through theory building, modeling
underlying processes, and integrating information
across meta-analytic studies (e.g., Harlow, 2010;
Jaccard & Jacoby, 2009; McGrath, 2011; Rodgers,
2010; Rosenthal & DiMatteo, 2001). Consistent
with this vision is a focus on encouraging greater
statistical reasoning, thinking and literacy, rather
than rote learning (Gal, 2003; Garfield & delMas,
2010; Ridgway, Nicholson, & McCusker, 2007).
Researchers are realizing that quantitative literacy
is a needed goal in undergraduate studies, begin-
ning with the most basic, introductory statistics
courses (Ben-Zvi & Garfield, 2004; Mulhern &
Wylie, 2004, 2006; Rumsey, 2002; Watson, 2006).
Resources are becoming available to help encourage
quantitative thinking (Garfield & Ben-Zvi, 2008;
Saville, Zinn, Lawrence, Barron, & Andre, 2008;
Stark & Krause, 2009), along with a realization of
the challenges involved (Ben-Zvi & Garfield, 2004;
Sen, 2004).
Garfield and colleagues from the University of
Minnesota developed a program in Quantitative
Methods in Education that is at the forefront on
improving undergraduate statistical training. Much
of their research focuses on training statistics teach-
ers (Garfield & Everson, 2009), reforming the way
introductory statistics is taught (Garfield, Hogg,
Schau, & Whittinghill, 2002), and investigating
how to teach and learn statistics (Garfield & Ben-
Zvi, 2007). Other researchers (Friedrich, Buday,
& Kerr, 2000) also help outline the field by sur-
veying undergraduate programs across the country
with respect to quantitative training. In the next
section, I review research focusing on strategies
for teaching quantitative psychology (e.g., Gelman
& Nolan, 2002), including a set of guidelines
developed by Garfield and colleagues (Franklin &
Garfield, 2006).
Strategies for Teaching Quantitative
Psychology
There is an abundance of resources to aid instruc-
tors of statistics courses, including best practice
volumes (e.g., Dunn, Smith, & Beins, 2007;
Hulme, 2007; Hulsizer & Woolf, 2009), com-
pilations of research on teaching statistics (e.g.,
Bjornsdottir & Garfield, 2010; Zieffler, Garfield,
Alt, Dupuis, Holleque, & Chang, 2008), learn-
ing objectives for introductory statistics and research
(Tomcho, Rice, Foels, Folmsbee, Vladescu, Liss-
man, Matulewicz, & Bopp, 2009), and sugges-
tions for teaching quantitative courses (Garfield &
Everson, 2009; Ware & Johnson, 2000). In this
section, I present several strategies that have been
researched and recommended to improve statistical
literacy.
Active Learning
Research has demonstrated that actively involv-
ing students significantly improves performance in
quantitative courses (e.g., Helman & Horswill,
2002). Guidelines to improve statistical learn-
ing (e.g., encouraging statistical thinking, using
examples with technology and real data, emphasiz-
ing concepts, promoting active learning) reiterate
this approach of engaging students in the process
(Everson, Zieffler, & Garfield, 2008; Franklin
& Garfield, 2006). Humor is also a great ice-
breaker, catching students’ attention (e.g., Cobb,
1999, 2007; Zeedyk, 2006), helping to diffuse ten-
sion, and surreptitiously calling students back to
quantitative learning.
Creating a lively environment is essential for
capturing the interest of students. A number of
researchers have emphasized the importance of
hands-on and interactive learning (Dolinsky, 2001;
Kolar & McBride, 2003; Wulff & Wulff, 2004).
A simple strategy could be to invite students to
write down what is clear and not clear at the end
of each lecture, with the faculty clarifying unclear
points at the beginning of the next lecture (Harlow,
Burkholder, & Morrow, 2006). It takes little time,
and students get fairly immediate feedback on how
to clear up misunderstandings, whereas faculty get
a clearer idea of what the students understood
and what needs to be clarified. Moreover, involv-
ing students with creative examples (Chew, 2007;
Schwartz & Martin, 2004) and analyzing data (e.g.,
Nie & Lau, 2010; Watkins, Scheaffer, & Cobb,
2004) bring about more in-depth learning than tra-
ditional lecture-based approaches. Demonstrating
visual images and graphs of procedures also helps
improve understanding (e.g., Peden, 2001).
If there is more time, then students can be
arranged into small groups and be given a research
scenario in which they need to consult among them-
selves to recommend statistical procedures to address
the research question (Harlow, Burkholder, & Mor-
row, 2006). For a basic example, students could be
asked how to assess whether men and women dif-
fered on hours of exercise per week (i.e., with a two-
sample independent t-test) or whether the number
of hours worked at an outside job was related to GPA
(i.e., with a correlation). A number of researchers
have recommended engaging students in these
cooperative small groups (DaRos-Voseles, Collins,
Onwuegbuzie, & Jiao, 2008; Onwuegbuzie,
Collins, & Jiao, 2009; Peterson & Miller, 2004) or
learning communities (e.g., Barren, Benedict, Sav-
ille, Serdikoff, & Zinn, 2007) where students work
together to understand quantitative material and
immerse themselves in the process. Krause, Stark,
and Mandl (2009), on the other hand, have found
that cooperative groups did not directly improve
statistics performance, although students reported
greater perceived efficacy when working with others.
It may be that group learning is not effective for all
students, with more advanced students possibly ben-
efiting the least. For example, Harlow, Burkholder,
and Morrow (2002) found that learning activities
harlow 107
that included working in groups and with peer men-
tors was viewed more favorably when students were
more anxious and had lower confidence about quan-
titative learning. Future research could investigate
whether ability or achievement level is a moderator
for group learning and performance to clarify who
benefits from group learning.
Technology and Learning
Other research has revealed the value of tech-
nology in heightening quantitative learning using
computer-assisted analysis (Bartz & Sabolik, 2001),
Web-based tutorials (Bliwise, 2005), and specific
online learning programs such as Estimating Statis-
tics (EStat; Britt, Sellinger, & Stillerman, 2002),
Simulation-Assisted Learning Statistics (SALS; Liu,
Lin, & Kinshuk, 2010), the Utah Virtual Lab
(Malloy & Jensen, 2001), Statistical Understand-
ing Made Simple (SUMS; Swingler, Bishop, &
Swingler, 2009), or Web Interface for Statistics Edu-
cation (WISE; Berger, n.d., http://wise.cgu.edu/).
In a meta-analysis of 45 studies, there was a small-
to medium-sized Cohen’s d (i.e., 0.33) performance
benefit effect size attributed to learning statistics
with computer-assisted instruction versus learning
in a lecture-based control group that did not provide
such input (Sosa, Berger, Saw, & Mary, 2010). Thus,
students who had technology-enhanced instruction
demonstrated one-third of a standard deviation
higher performance than those students without
such advantage; further, the improvement was even
more salient when involving those who were more
advanced (i.e., graduate students) and when more
time was allotted for instruction.
Still, whereas research has demonstrated the
benefits of e-learning approaches (e.g., Fillion,
Limayem, Laferrière, & Mantha, 2008; Hanley,
2004; Sosa et al., 2010; Wender & Muehlboeck,
2003), more research is needed, as others remain
unconvinced of the merits of adding technology to
the classroom environment (e.g., Härdle, Klinke, &
Ziegenhagen, 2007). Even the value of online dis-
cussions and whether or how much faculty should
facilitate or contribute is not entirely clear (e.g.,
Mazzolini & Maddison, 2007). Instructors are
encouraged to consider literature on cognitive learn-
ing and assessment to improve Web-based materials
for students, particularly in quantitative learning
(e.g., Romero, Berger, Healy, & Aberson, 2000).
Mentors and Role Models
Mentoring can help students get the extra input
needed to understand quantitative concepts (e.g.,
108 t e a c h i n g q u a n t i tat i v e p s yc h o l o g y
Ferreira, 2001) and can help to supplement class
lectures and faculty input (e.g., Katayama, 2001).
Fortunately, graduate teaching assistants (TAs) are
often provided for undergraduate- and graduate-
level quantitative courses. When TAs are not funded,
I have found it very effective to invite one or more
top achievers from a previous semester to serve as
volunteer peer mentors or TAs, offering indepen-
dent study or teaching practicum credit. Students
in the course benefit from having a peer of similar
age demonstrating and facilitating expertise in quan-
titative methods. TAs or peer mentors gain efficacy
and greater confidence, often choosing to become
even more involved with other quantitative courses
and research to continue building their skills (e.g.,
Harlow, Burkholder, & Morrow, 2002, 2006).
Mentoring can be particularly valuable for
women and individuals from under-represented
groups who have few role models in the quanti-
tative field, for providing direction, support, and
encouragement (e.g., Kosoko-Lasaki, Sonnino, &
Voytko, 2006; Neal-Barnett, Mitchell, & Boeltar,
2002; Zirkel, 2002).
Conceptual Approach to Teaching
Perhaps the most effective idea for conveying
complex quantitative material is to focus on the
concepts rather than using a strictly mathematical
approach (e.g., Atkinson, Catrambone, & Merrill,
2003). Chiou (2009) found that encouraging stu-
dents to collaborate on conceptually mapping sta-
tistical material significantly improved performance
compared to having students complete textbook
exercises and calculations. In another study, Aber-
son, Berger, Healy, and Romero (2003) demon-
strated that an interactive approach to hypothe-
sis testing concepts was received more positively
and improved performance over traditional labo-
ratory exercises. Similarly, Meletiou-Mavrotheris
and Lee (2002) found that helping students to
understand concepts, improve statistical reason-
ing, and build intuitions about statistical ideas was
more facilitating than using a traditional teaching
approach.
Presenting and encouraging understanding of
core concepts—particularly through hands-on
engagement and research—can help foster more
in-depth insight and greater involvement in inquiry-
based future learning (Aulls & Shore, 2008; Dewey,
1997). Inviting students to seek out solutions to
quantitative research problems promotes greater sta-
tistical awareness and literacy. In the next section, I
present a set of conceptual themes that are common
to many statistical methods and that help provide
a foundation for understanding introductory and
advanced quantitative learning.
Themes that Run Through Quantitative
Psychology
Quantitative understanding is increased when
common ideas are revealed that occur in many sta-
tistical procedures. Harlow (2005) has emphasized
a number of themes that run through multivariate
methods and that can be extended to encompass
univariate methods as well. Three basic themes are
presented below to facilitate approaching, analyzing,
and interpreting quantitative methods.
Considering the Research Question
First, it is helpful to encourage students to con-
sider the kind of research question that needs to be
addressed. Group difference questions can be ana-
lyzed with basic z-tests when information is known
about the mean and the variability in the popula-
tion. For example, we could investigate whether
an exceptional students’ class evinced an IQ that
was different from the known average of 100, with
a standard deviation of 15. When only a pop-
ulation mean is known or there are two groups
involved, a t-test would be appropriate, requiring
that a researcher estimate the population standard
deviation(s) from the sample(s). For studies with
two or more groups, an ANOVA would be useful
for assessing whether there were differences among
multiple groups. For example, a teacher may want
to compare the level of interest in taking math
courses for male and female students, which could
be examined with a two-sample independent t-test.
To investigate potential differences in math inter-
est among students from three different college
majors, an ANOVA would be helpful. To exam-
ine whether several groups differ on math interest,
after accounting for the number of previous math
classes taken, analysis of covariance (ANCOVA)
could be used. Further, group differences across sev-
eral groups could be assessed on several measures
(e.g., math interest, math efficacy, and math anxiety)
using multivariate analysis of variance (MANOVA).
Similarly to the way that ANCOVA extended
ANOVA by taking into account another predic-
tor that correlated or covaried with the outcome,
a multivariate analysis of covariance (MANCOVA)
can extend a MANOVA when wanting to examine
whether several groups differ on several outcomes
(e.g., the three math attitudes suggested for the
MANOVA) after taking into account one or more
covariates such as number of previous math courses
or GPA.
For those who are not specifically interested in
mean differences across groups, a correlational ques-
tion may be asked. Researchers interested in finding
what is associated with achievement could conduct a
simple correlation analysis, assessing whether math
interest is related to college GPA. This bivariate pro-
cedure (between just two variables) could expand
to canonical correlation (CC) that allows an exam-
ination of two sets of variables. For example, a
researcher could examine whether several math atti-
tude scores (e.g., math interest, math efficacy, and
math anxiety) are related to several achievement
outcomes (e.g., GPA, the number of conferences
attended, and the number of memberships in honor
societies). Correlation could also be extended to
multiple regression (MR) to relate a linear combi-
nation of several continuous predictors (e.g., math
ability, verbal ability, previous GPA, achievement
motivation) to a single outcome such as current
GPA. If the outcome were dichotomous (e.g., suc-
cess or failure in a course or grade level), then
the merit of similar predictors could be examined
with logistic regression (LR) or discriminant func-
tion analysis (DFA), depending on whether there
was interest in conveying the odds or the degree of
correlation with the outcome, respectively. Multi-
level modeling (MLM) would be useful to predict
an outcome such as achievement, when there is
reason to believe that participants are clustered in
separate groups (e.g., classrooms, school districts).
Thus, MLM allows, and even models, heterogeneity
of variance across several groups in the data, con-
trary to more restrictive prediction methods (e.g.,
MR, LR, and DFA) that assume that data are drawn
from a homogeneous group.
Other, more advanced, research questions could
be addressed with more sophisticated methods. Fac-
tor analysis (FA) or principal components analysis
(PCA) would be useful when faced with a large
set of measures with a goal of identifying a few
key dimensions that explain the underlying struc-
ture or associations (e.g., quantitative and verbal
intelligence dimensions in an IQ test of multiple
subtests). Factor analysis would allow the delin-
eation of unique or error variance in measures before
forming factors with the variance that was in com-
mon among the measures. In contrast, PCA analyzes
all of the variance in the variables when form-
ing components. In practice, there may not be
much actual difference in results across these two
harlow 109
methods if the loadings are high for relevant vari-
ables on their respective factors, even with the initial
difference in whether unique variance is included
in the analysis (Velicer & Jackson, 1990). Struc-
tural equation modeling (SEM) could be used to
examine whether latent factors, each with several
measures, could be theoretically modeled to explain
hypothesized underlying processes. For example,
using Bandura’s (1997) social learning theory, we
could test whether different factors of intelligence
(i.e., quantitative and verbal) would predict degree
of self-efficacy in learning, which in turn could
predict an achievement factor (measured by home-
work, quiz, and exam scores). In testing the SEM,
other covariates (e.g., socioeconomic status, pre-
vious GPA) could be added as possible predictors
of the mediator (i.e., self-efficacy) to see if they
are important, or to rule them out as predictors
of achievement. Further, multiple group analyses
could examine whether such a predictive SEM of
achievement held in different groups (e.g., men ver-
sus women, different ethnicities). If findings differed
across groups, then it would indicate that the group-
ing variable moderated the prediction. Longitudinal
modeling (e.g., latent growth curve modeling) could
examine whether achievement changed over time in
mean level and in the rate of change and whether
predictors (e.g., previous GPA, IQ, gender) could
predict the level (called the intercept) and rate of
change (called the slope) across time. Other meth-
ods discussed in this volume could also be examined
to further fine tune the nature of the question being
assessed.
Therefore, the main point of the first theme (type
of research question asked) is that it is the nature
of the research and the questions asked of the data
that drive the kind of quantitative method that is
selected. There is a whole world of methods to
consider, and the choice is not so daunting when
realizing that certain methods readily lend them-
selves to different kinds of research questions (e.g.,
group difference, correlation or prediction, underly-
ing structure, longitudinal, etc.). In the next section,
similarities and differences among methods is the
second theme that is discussed. Here, it will become
apparent that although there are distinctions among
methods that make them more likely to be applied to
specific research questions, many quantitative proce-
dures share similarities in how they reveal the essence
of the data. For example, researchers who find a sig-
nificant difference in well being between groups who
exercise regularly and those who do not will also find
that there is a significant relationship between well
110 t e a c h i n g q u a n t i tat i v e p s yc h o l o g y
being and exercise. Thus, both group difference and
correlational methods can examine how much vari-
ance is shared between independent and dependent
variables. In this regard, Cohen, Cohen, West, and
Aiken (2003) have described how categorical group-
ing variables can be coded for use in correlational and
regression procedures.
Noting Similarities and Differences in
Quantitative Methods
Examining research questions and seeing specific
methods that seem to be relevant to such questions
leads into a second theme, which is to notice the
similarities and differences among various quantita-
tive procedures. Quilici and Mayer (2002) helped
students notice the underlying similarities in differ-
ent statistical word problems. This helped students
to know how to analyze specific problems by not-
ing their similarities to other statistical problems
that were addressed by certain methods. Similarly,
Derryberry, Schou, and Conover (2010) helped
students understand how to conduct rank-based sta-
tistical tests by revealing their resemblance to already
studied parametric tests.
Quantitative methods can be classified in several
ways to help delineate common aspects or distin-
guishing differences. Thus, group-difference meth-
ods (e.g., z-test, t-test, ANOVA, ANCOVA, and
MANOVA) are similar in allowing an examination
of potential mean differences between groups on
one or more outcome variables. In contrast, correla-
tional methods (e.g., Pearson’s r, CC) do not tend to
focus on means but, rather, on assessing association
between independent and dependent variables. Pre-
diction methods (e.g., MR, LR, DFA, and MLM)
all focus on predicting an outcome from a set of pre-
dictors and may have some emphasis on grouping
variables—particularly LR and DFA, which explic-
itly include a categorical dependent variable, and
MLM, which takes into account different levels or
groups in the data. However, each of these predic-
tion methods differs somewhat from ANOVA-based
methods in focusing more on weights linking inde-
pendent and dependent variables and less on mean
scores such as group averages or centroids. Dimen-
sional or structural methods (e.g., FA, PCA) are
similar by involving one or more sets of measures
with a smaller set of underlying factors or compo-
nents posited or revealed to help explain the patterns
of relationships among variables. Thus, dimensional
methods are similar to correlational methods in their
focus on associations among variables and differ-
ent from group difference methods that are more
concerned with mean differences among groups.
SEM combines the best of group difference, correla-
tion/prediction, and structural methods by allowing
an investigation of the means of latent factors across
different samples or groups, while still allowing an
examination or confirmation of the structural or
correlational relationships among variables. Longi-
tudinal methods (e.g., time series, latent growth
modeling) add the element of time to allow an
examination of temporal ordering that can help in
assessing causal precedence among variables.
The main point of this second theme is to encour-
age students to see how various quantitative methods
are similar and how they are different. Discern-
ing these distinctions and similarities will go a long
way toward highlighting the overriding features and
underlying aspects of quantitative methods and in
selecting an appropriate method to address a specific
kind of research question.
Interpreting Findings from Quantitative
Methods
Third, after identifying a research question and
noticing similarities and differences that lead to
selecting a method to analyze one’s data, it is impor-
tant to examine and interpret findings from multiple
perspectives. Although the field of quantitative
psychology has traditionally focused largely on sig-
nificance testing, there is growing emphasis on
considering other alternatives such as effect sizes
and confidence intervals (e.g., Harlow, Mulaik, &
Steiger, 1997; Kline, 2004; Wilkinson, & The Task
Force on Statistical Inference, 1999). Initially, it can
be helpful to assess whether the overall variation
in the data is significantly different from chance.
Significance tests usually involve some ratio of vari-
ances. We can think of group difference models
as explaining the ratio of how scores vary among
groups, relative to how much scores vary within each
group. This ratio can be readily recognized as the
F -test. The closer this ratio is to 1.0, the less we
are apt to consider between-groups variance as any-
thing more meaningful than the within-group error
variance. Correlational methods examine whether
the covariance between variables is salient when
contrasted with the variances within each variable.
Thus, a correlation is simply a ratio of the covariance
between variables over the product of the stan-
dardized variance (i.e., standard deviations) within
each variable. Using an analogy relating individuals,
when the covariance between two people is salient,
despite their individual variance (or uniqueness), a
meaningful relationship emerges.
After assessing significance, it is important to
examine the magnitude of an overall finding, called
an effect size (ES). For group-difference methods,
an ES can indicate the number of standard deviation
units of difference there is among group means (i.e.,
Cohen’s d ; Cohen, 1988), with small, medium, and
large effects having values of about 0.20 (almost
a quarter of a standard deviation), 0.50 (half a
standard deviation), and 0.80 (almost a full stan-
dard deviation) (Cohen, 1988). For correlational
methods, we would hope to show a meaningful rela-
tionship between pertinent variables, with values of
0.1, 0.3, and 0.5 indicating small, medium, and
large Pearson product moment correlations, respec-
tively. Particularly for prediction methods, as well
as other methods, proportions of shared variance
effect sizes (e.g., η2 or R2 ) are useful for showing
how much the independent and dependent variables
have in common, ranging in size from 0 to 1.0. Pro-
portion of variance ES values of 0.01, 0.09, and
0.25 indicate small, medium, and large univariate
effects, respectively (obtained by squaring 0.1, 0.3,
and 0.5 correlations, respectively); and 0.02, 0.13,
and 0.26 or more refer to small, medium, and large
multivariate effect sizes, respectively (e.g., Cohen,
1988; Harlow, 2005).
When interpreting ES, it is also important to pro-
vide an indication of the margin of error with con-
fidence intervals (e.g., Cumming & Fidler, 2009;
Gilliland & Melfi, 2010; Odgaard & Fowler, 2010).
Confidence intervals provide a range of values, with
narrower intervals indicating more precision for an
estimated effect. Research journals such as the Jour-
nal of Consulting and Clinical Psychology and others
are beginning to require that statistical findings be
supplemented with effect sizes and confidence inter-
vals (e.g., La Greca, 2005; Odgaard & Fowler,
2010).
Thus, the third theme, concerned with inter-
preting results, involves providing an indication as
to whether a result is significantly different from
chance, the magnitude of the effect, and the degree
of certainly about the result.
Ultimately, understanding quantitative themes
and concepts (e.g., Abelson, 1995), including
framing initial research questions; noticing sim-
ilarities and differences when selecting statistical
methods; and interpreting the significance, extent,
and precision of one’s finding will go a long way
toward moving psychology more in the forefront of
quantitative science.
harlow 111
Conclusions
This chapter features a number of issues regard-
ing the teaching of quantitative psychology. First,
research on quantitative training underscores the
need to provide greater guidelines and opportunities
for learning more about measurement, research, and
statistical methods (e.g., Franklin & Garfield, 2006;
Garfield & Everson, 2009). Job opportunities are
more available than are applicants for quantitative
careers (e.g., Aiken et al., 1990, 2008). An APA task
force on increasing the quantitative pipeline (Aiken
et al., 2007) is aimed at bringing more people into
this field to fill this need.
Second, strategies for effectively teaching quan-
titative psychology emphasize the advantages of
active learning (e.g., Onwuegbuzie, Collins, &
Jiao, 2009), technology, and Web-based instruction
(e.g., Bliwise, 2005; Fillion, Limayem, Laferrière,
& Mantha, 2008; Sosa et al., 2010), mentoring
and role models (e.g., Ferreira, 2001), and concep-
tual understanding (e.g., Mulhern & Wylie, 2004;
Swingler, Bishop, & Swingler, 2009). All of these
strategies implicitly invite students to become more
immersed in the quantitative learning process and
ultimately to become more statistically literate.
Third, several themes have been offered to bring
about greater clarity in quantitative learning (Har-
low, 2005). One of the themes includes considering
the nature of the research question. Students should
be encouraged to ask whether their research ques-
tion involves group differences, correlation, predic-
tion, underlying structure, or longitudinal studies.
Another theme encourages students to notice the
similarities and differences among quantitative pro-
cedures, as this kind of awareness is helpful in select-
ing appropriate methods to analyze data and address
research questions. Still another theme presents sug-
gestions for interpreting quantitative findings using
statistical tests, ES, and confidence intervals.
Being aware of quantitative training practices,
encouraging more engaged learning, and focusing
on conceptual thinking and underlying themes can
help us feature and spotlight quantitative psychology
as a highly worthwhile and exciting field in which
to take part.
Future Directions
A number of future directions are suggested to
point out areas in which quantitative teaching is
emerging and needs to grow. First, teaching should
focus on actively engaging students and helping
them to understand basic, underlying concepts. This
would also involve more emphasis on modeling
112 t e a c h i n g q u a n t i tat i v e p s yc h o l o g y
overarching constructs and processes rather than
limiting teaching to narrow and isolated method-
ological procedures (Embretson, 2010; McGrath,
2011; Rodgers, 2010). Building quantitative mod-
els of behavior will generate more integrated and
explanatory understanding that is necessary to keep
psychology scientifically rigorous.
Second, students, faculty, and professionals
should be encouraged to explore options for fur-
thering quantitative learning outside of the class-
room. For example, quantitative journals (e.g.,
Multivariate Behavioral Research, Psychological Assess-
ment, Psychological Methods, Structural Equation
Modeling ) as well as quantitative handbooks (e.g.,
Cooper et al., 2012), statistical dictionaries (e.g.,
Dodge, 2003; Everitt, 2002; Upton & Cook,
2008), and even online resources on quantita-
tive methods (e.g., Journal of Statistics Educa-
tion, http://www.amstat.org/publications/jse/) pro-
vide opportunities for further quantitative studies.
There are also numerous quantitative volumes that
provide input and guidelines for understanding
the main aspects of various quantitative methods
(Hancock & Mueller, 2010), including multi-
ple regression/correlation (Cohen, Cohen, West,
& Aiken, 2003), multivariate statistics (Stevens,
2009), statistical mediation (MacKinnon, 2008),
missing data analysis (Enders, 2010), and structural
equation modeling (e.g., Kline, 2011), among oth-
ers. For those interested in more advanced statistical
material, they could venture into the British Jour-
nal of Mathematical and Statistical Psychology and
Psychometrika, or even the Millsap and Maydeu-
Olivares (2009) volume on Contemporary Psychomet-
rics, realizing that these latter sources would not be
as accessible as the more translational articles in the
former outlets.
Those interested in additional training could
consider workshops (e.g., the University of Kansas
Summer Institute held by Todd Little), annual
conferences featuring quantitative presentations
(e.g., Division 5 talks at the APA convention,
Division D talks at the American Educational
Research Association convention, and presenta-
tions and workshops at the Association for Psy-
chological Science). Websites focusing on quan-
titative material (e.g., UCLA statistical website,
http://www.ats.ucla.edu/stat/, and a similar one at
Georgetown, http://statpages.org/) and online tuto-
rials in quantitative methods (Dinov & Christou,
2009; Garfield & delMas, 2010) offer additional
occasions to heighten quantitative proficiency. Fur-
ther, there are quantitative societies that provide
useful opportunities and contacts (e.g., American Active
g
Psychological Association, Division 5: Evaluation, Learnin
ning
Measurement, and Statistics; American Statisti- E-Lear
cal Association; Psychometric Society; Society for ring
Quant Mento
Mathematical Psychology, and the Society of Multi- Teaching
variate Experimental Psychology), including inter-
national statistical societies (e.g., European Associa- Research
tion of Methodology, International Association for Questions
Statistical Education, International Statistical Insti- Quant Similarities &
tute, Society for Multivariate Analysis in the Behav- Concepts Differences
ioral Sciences). Participating in quantitative forums Interpret
helps to enlarge quantitative networks, providing a Results
stronger foundation to further one’s skills. Other Quant Statistical
Third, quantitative teaching should be widened Learning Literacy
Models
to include individuals from low-income (Kitchen,
Wo
DePree, Celedón-Pattichis, & Brinkerhoff, 2007) Soc
rks
ho
Li ieti ps
and under-represented groups (Kosoko-Lasaki, Son- ter
atu es
nino, & Voytko, 2006). We also need to con- re
sider whether traditional approaches to quantitative
teaching have helped to shrink or amplify the differ- Widening
Quant Reach
ences among mainstream and marginalized students Disadvantaged
(Ceci & Papierno, 2005). Other offerings can be
Early
encouraged (e.g., Quantitative Training for Under- Care
er
Und
represented Groups) to help bring about greater erre
pres
equity and effectiveness that can open doors to ente
d
more career options in the field of quantitative sci-
ence to an even larger and more diverse group of Figure 6.1 Bringing about statistical literacy through quanti-
individuals. tative teaching, other quantitative learning, and widening the
quantitative reach
Fourth, it is important to reach out to quantita-
tive researchers who are newly graduated to encour-
age them to become involved in the larger field.
Early career psychologists make up about 10% of the technology, engineering, and mathematics that
150,000 members of the APA (2010), and Division sometimes exclude psychology. Further, research
5 (measurement, statistics and evaluation) involves reveals that students from the United States fall
approximately 1% of APA members (i.e., Gruber, in the bottom half regarding quantitative perfor-
2010). The APA Quantitative Task Force (Aiken mance when compared to other countries (Gon-
et al., 2007) helps in this regard, with a website zales, Guzmán, Partelow, Pahlke, Jocelyn, Kast-
(http://www.apa.org/research/tools/quantitative/in- berg, & Williams, 2003; McQuillan & Kennelly,
dex.aspx) that provides definitions of the areas of 2005).
focus within quantitative methods, recommenda- Figure 6.1 depicts the main focus of this chapter,
tions for preparing for quantitative training, and lists where innovative quantitative teaching (with active
of quantitative graduate programs that can help in learning, e-learning, mentoring, and concepts learn-
recruiting individuals into quantitative psychology ing), other quantitative learning (e.g., model build-
and in retaining those that are already participating. ing, workshops, quantitative societies, and quan-
Finally, continuing efforts to generate greater titative literature), and widening the quantitative
interest, participation, and performance in quan- reach (to the disadvantaged, early career individu-
titative literacy is an important goal. Unfortunately, als, and the under-represented) can help bring about
psychology is not always considered as having the greater statistical literacy. More emphasis needs to be
scientific credentials of other disciplines such as made on involving others in the marvels of quanti-
physics, chemistry, and biology (Simonton, 2004). tative psychology, whether at the undergraduate or
Particularly, recent efforts to improve technologi- graduate level, in the larger profession of psychol-
cal education in the United States (Chopra, 2010) ogy or in the general public. Ideas presented in this
have focused more on traditionally defined science, chapter, and in the larger handbook, are offered to

harlow 113
help in teaching quantitative psychology, within the
classroom, in research centers, and in every facet of
life that involves quantitative thinking.
Author Note
Lisa L. Harlow is Professor of Quantitative Psy-
chology at the University of Rhode Island. Please
address correspondence to Lisa L. Harlow, Depart-
ment of Psychology, University of Rhode Island, 142
Flagg Rd, Chafee, Kingston, RI 02881-0808.
Acknowledgments
I wish to acknowledge support from the National
Science Foundation (NSF) Grant (# 0720063)
on Quantitative Training for Underrepresented
Groups; Lisa L. Harlow, Principal Investigator (PI).
I would like to thank students, faculty and staff at
the University of Rhode Island and the Cancer Pre-
vention Research Center in Rhode Island for their
ongoing support, inspiration and encouragement.
Similarly, I would like to thank Dr. Lisa Dierker
(PI) and her colleagues at the Quantitative Analy-
sis Center at Wesleyan University. Serving on the
advisory board for their NSF Grant (# 0942246)
on An Inquiry-Based, Supportive Approach to Sta-
tistical Reasoning and Application has opened up
my thinking and those of my students on how
to productively engage students in the wonders of
quantitative inquiry.
References
Abelson, R. P. (1995). Statistics as principled argument. Hillsdale,
NJ: Erlbaum.
Aberson, C. L., Berger, D. E., Healy, M. R., & Romero, V.
L. (2003). Evaluation of an interactive tutorial for teach-
ing hypothesis testing concepts. Teaching of Psychology, 30,
75–78.
Aiken, L. S., Aguinis, H., Appelbaum, M., Boodoo, G.,
Edwards, M. C., Gonzalez, R. D., Panter, A., et al.
(2007). Report of the task force for increasing the num-
ber of quantitative psychologists. Washington DC: American
Psychological Association. Retrieved July 1, 2010 from:
http://www.apa.org/research/tools/quantitative/quant-task-
force-report.pdf.
Aiken, L. S., West, S. G., & Millsap, R. E. (2008). Doctoral
training in statistics, measurement, and methodology in psy-
chology: Replication and extension of Aiken, West, Sechrest,
and Reno’s (1990) survey of PhD programs in North America.
American Psychologist, 63, 32–50.
Aiken, L. S., West, S. G., Sechrest, L., & Reno, R. R. (1990).
Graduate training in statistics, methodology, and measure-
ment in psychology: A survey of PhD programs in North
America. American Psychologist, 45, 721–734.
APA (2010). Early career psychologists. Washington D.C.: Amer-
ican Psychological Association. Retrieved on July 1, 2010
from: http://www.apa.org/membership/index.aspx.
114 t e a c h i n g q u a n t i tat i v e p s yc h o l o g y
Ashcraft, M. H., & Kirk, E. P. (2001). The relationship among
working memory, math anxiety, and performance. Journal of
Experimental Psychology: General, 130, 224–237.
Atkinson, R. K., Catrambone, R., & Merrill, M. M. (2003). Aid-
ing transfer in statistics: Examining the use of conceptually
oriented equations and elaborations during subgoal learning.
Journal of Educational Psychology, 95, 762–773.
Aulls, Mark W., & Shore, B. M. (2008). Inquiry in education,
volume 1: The conceptual foundations for research as a curricular
imperative. New York: Taylor & Francis/Erlbaum.
Bandura, A. (1997). Self-efficacy: The exercise of control. New York:
W. H. Freeman.
Barren, K. E., Benedict, J. O., Saville, B. K., Serdikoff, S.
L., & Zinn, T. E. (2007). Innovative approaches to teach-
ing statistics and research methods: Just-in-time teaching,
interteaching, and learning communities. In Dunn, D. S.,
Smith, R. A, & Beins, B. (Eds.), Best practices for teaching
statistics and research methods in the behavioral sciences, (pp.
143–158). Mahwah, NJ: Erlbaum.
Bartz, A. E., & Sabolik, M. A. (2001). Computer and software
use in teaching the beginning statistics course. Teaching of
Psychology, 28, 147–149.
Ben-Zvi, D. & Garfield, J. (Eds.) (2004). The challenge of devel-
oping statistical literacy, reasoning, and thinking. Dordrecht,
Netherlands: Kluwer.
Berger, D. (n.d.). WISE Web Interface for Statistics Educa-
tion. Retrieved December 20, 2010 from http://wise.cgu.
edu/.
Bjornsdottir, A., & Garfield, J. (2010). Teaching bits: Statistics
education articles from 2009. Journal of Statistics Educa-
tion, 18, Retrieved May 9, 2012 from www.amstat.org/
publications/jse/v18n1/garfieldtb.pdf
Bliwise, N. G. (2005). Web-based tutorials for teaching intro-
ductory statistics. Journal of Educational Computing Research,
33, 309–325.
Britt, M. A., Sellinger, J., & Stillerman, L. M. (2002). A review
of ESTAT: An innovative program for teaching statistics.
Teaching of Psychology, 29, 73–75.
Budé, L., Van De Wiel, M. W. J., Imbos, T., Candel, M.
J. J. M., Broers, N. J., and Berger, M. P. F. (2007).
Students’ achievements in a statistics course in relation to
motivational aspects and study behaviour. Statistics Education
Research Journal [Online], 6, 5–21. Retrieved May 9, 2012
from http://www.stat.auckland.ac.nz/∼iase/serj/SERJ6(1)_
Bude.pdf
Ceci, S. J., & Papierno, P. B. (2005). The rhetoric and reality of
gap closing: When the “have nots” gain but the “haves” gain
even more. American Psychologist, 60, 149–160.
Chew, S. L. (2007). Designing effective examples and problems
for teaching statistics. In, Dunn, Dana S., Smith, Randolph
A, & Beins, Barney (Eds.), Best practices for teaching statistics
and research methods in the behavioral sciences, (pp. 73–91).
Mahwah, NJ: Erlbaum.
Chiou, C.-C. (2009). Effects of concept mapping strategy on
learning performance in business and economics statistics.
Teaching in Higher Education, 14, 55–69.
Chopra, A. (2010). Building a future for technologi-
cal innovation. White House Office of Science &
Technology Policy. Retrieved January 13, 2011 from
http://www.whitehouse.gov/sites/default/files/microsites/
ostp/CommClubr081710FINAL.pdf.
Cobb, G. (1999). Move over, Bill Nye: George Cobb
makes learning about statistics practical and fun. Mount
Holyoke College news and events. Retrieved May 9, 2012
 from http://www.mtholyoke.edu/offices/comm/csj/990409/
gummy.html.
Cobb, G. W. (2007). The introductory statistics course: A
ptolemaic curriculum? Technology Innovations in Statis-
tics Education, 1, 1?15. Retrieved May 9, 2012 from
http://escholarship.org/uc/item/6hb3k0nz# page-15
Cohen, J. (1988). Statistical power analysis for the behavioral
sciences (2nd edition). Hillsdale, NJ: Erlbaum.
Cohen, J. Cohen, P., West, S. G., & Aiken, L. S. (2003). Applied
multiple regression/correlation analysis for the behavioral sciences
(3rd edition). Mahwah, NJ: Erlbaum.
Cooper, H., Camic, P. M., Long, D. L., Panter, A. T., Rind-
skopf, D., & Sher, K. J. (Eds.). (2012). APA handbook of
research methods in psychology (Vols. 1–3). Washington, DC:
American Psychological Association.
Cumming, G., & Fidler, F. (2009). Confidence intervals: Better
answers to better questions. Zeitschrift für Psychologie/Journal
of Psychology, 217, 15–26.
DaRos-Voseles, D. A., Collins, K. M. T., Onwuegbuzie, A.
J., & Jiao, Q. G. (2008). Effect of self-perception on per-
formance of graduate-level cooperative groups in research
methodology courses. Journal of Instructional Psychology, 35,
254–259.
Derryberry, D. R., Schou, S. B., & Conover, W. J.
(2010). Teaching rank-based tests by emphasizing struc-
tural similarities to corresponding parametric tests. Journal
of Statistics Education, 18, Retrieved May 9, 2012 from
www.amstat.org/publications/jse/v18n1/derryberry.pdf.
DeVaney, T. A. (2010). Anxiety and attitude of graduate stu-
dents in on-campus vs. online statistics courses. Journal
of Statistics Education, 18, Retrieved May 9, 2012 from
www.amstat.org/publications/jse/v18n1/devaney.pdf.
Dewey, J. (1997). How we think. New York: Dover Publications.
Dinov, I. D., & Christou, N. (2009). Statistics online com-
putational resource for education. Teaching Statistics, 31,
49–51.
Dodge, Y. (Ed.) (2003). The Oxford dictionary of statistical terms.
New York: Oxford University Press.
Dolinsky, B. (2001). An active learning approach to teaching
statistics. Teaching of Psychology, 28, 55–56.
Dunn, D. S., Smith, R. A., & Beins, B. (Eds.) (2007). Best prac-
tices for teaching statistics and research methods in the behavioral
sciences. Mahwah, NJ: Erlbaum.
Embretson, S. E. (Ed.) (2010). Measuring psychological con-
structs: Advances in model-based approaches. Washington, DC:
American Psychological Association.
Enders, C. K. (2010). Applied missing data analysis. New York:
Guilford.
Everitt, B. S. (2002). The Cambridge dictionary of statis-
tics (2nd edition). Cambridge, UK: Cambridge University
Press.
Everson, M., Zieffler, A., & Garfield, J. (2008). Implement-
ing new reform guidelines in teaching introductory college
statistics courses. Teaching Statistics, 30, 66–69.
Ferreira, M. (2001). Building communities through role mod-
els, mentors, and hands-on-science. The School Community
Journal, 11, 27–37.
Fillion, G., Limayem, M., Laferrière, T., & Mantha, R. (2008).
Integrating ICT into higher education: A study of onsite
vs. online students’ and professors’ perceptions. International
Journal of Web-Based Learning and Teaching Technologies, 3,
48–72.
Franklin, C., & Garfield, J. (2006). The guidelines for assess-
ment and instruction in statistics education (GAISE) project:
Developing statistics education guidelines for grades pre K-
12 and college courses, In G.F. Burrill, (Ed.), Thinking and
reasoning with data and chance: Sixty-eighth annual NCTM
yearbook (pp. 345–375). Reston, VA: National Council of
Teachers of Mathematics.
Friedrich, J., Buday, E., & Kerr, D. (2000). Statistical train-
ing in psychology: A national survey and commentary
on undergraduate programs. Teaching of Psychology, 27,
248–257.
Gal, I. (2003). Teaching for statistical literacy and ser-
vices of statistics agencies. The American Statistician, 57,
80–84.
Garfield, J., & Ben-Zvi, D. (2007). How students learn statis-
tics revisited: A current review of research on teaching
and learning statistics. International Statistical Review, 75,
372–396.
Garfield, J., & Ben-Zvi, D. (2008). Developing students’ statistical
reasoning: Connecting research and teaching practice. New York:
Springer.
Garfield, J., & delMas, R. (2010). A web site that provides
resources for assessing students’ statistical literacy, reasoning
and thinking. Teaching Statistics, 32, 2–7.
Garfield, J., & Everson, M. (2009). Preparing teachers of
statistics: A graduate course for future teachers. Journal of
Statistics Education, 17 (2), Retrieved May 9, 2012 from
www.amstat.org/publications/jse/v17n2/garfield.html.
Garfield, J., Hogg, B., Schau, C., & Whittinghill, D.
(2002). First courses in statistical science: The status
of educational reform efforts. Journal of Statistics Educa-
tion, 10(2), Retrieved May 9, 2012 from www.amstat.
org/publications/jse/v10n2/garfield.html.
Gelman, A., & Nolan, D. (2002). Teaching statistics: A bag of
tricks. New York: Oxford University Press.
Gilliland, D., & Melfi, V. (2010). A note on confidence
interval estimation and margin of error. Journal of Statis-
tics Education, 18(1), Retrieved May 9, 2012 from
www.amstat.org/publications/jse/v18n1/gilliland.pdf.
Gonzales, P., Guzmán, J. C., Partelow, L., Pahlke, E., Jocelyn,
L., Kastberg, D., & Williams, T. (2003). Highlights from the
trends in international mathematics and science study (TIMSS).
U.S. Department of Education Institute of Education Sci-
ences, National Center for Educational Statistics.
Gruber, C. (Ed.) (2010) The score newsletter, Vol. XXXII. No. 3.
Division 5, American Psychological Association, Retrieved
July 1, 2010 from http://www.apa.org/divisions/div5/pdf/
July10Score.pdf.
Hancock, G. R., & Mueller, R. O. (Eds.). (2010). The reviewer’s
guide to quantitative methods in the social sciences. New York:
Taylor & Francis.
Hanley, G. L. (2004). E-learning and the science of instruction.
Applied Cognitive Psychology, 18, 123–124.
Härdle, W., Klinke, S., & Ziegenhagen, U. (2007). On the utility
of e-learning in statistics. International Statistical Review, 75,
355–364.
Harlow, L.L. (2005). The essence of multivariate thinking: Basic
themes and methods. Mahwah, NJ: Erlbaum.
Harlow, L. L. (2010). On scientific research: Invited com-
mentary on the role of statistical modeling and hypothesis
testing. Journal of Modern Applied Statistical Methods, 9,
348–358.
harlow 115
Harlow, L. L., Burkholder, G., & Morrow, J. (2002). Evaluating
attitudes, skill and performance in a learning enhanced quan-
titative methods course: A structural modeling approach.
Structural Equation Modeling Journal, 9, 413–430.
Harlow, L. L., Burkholder, G., & Morrow, J. (2006). Engag-
ing students in learning: An application with quantitative
psychology. Teaching of Psychology, 33, 231–235.
Harlow, L. L., Mulaik, S., & Steiger, J. (Eds.) (1997). What if
there were no significance tests? Mahwah, NJ: Erlbaum.
Helman, S., & Horswill, M. S. (2002). Does the introduction
of non-traditional teaching techniques improve psychology
undergraduates’ performance in statistics? Psychology Learning
& Teaching, 2, 12–16.
Hulme, J. (2007). Review of Handbook of demonstrations and
activities in the teaching of psychology volume 1: Intro-
ductory, statistics, research methods and history. Psychology
Learning & Teaching, 6, 164–165.
Hulsizer, M. R., & Woolf, L. M. (2009). Guide to teaching
statistics: Innovations and best practices. New York: Wiley-
Blackwell.
Jaccard, J., & Jacoby, J. (2009). Theory construction and model
building skills: A practical guide for social scientists. New York:
Guilford Press.
Katayama, A. D. (2001). Bi-modal instructional practices in edu-
cational psychology: Mentoring and traditional instruction.
Journal of Instructional Psychology, 28, 171–177.
Kitchen, R. S., DePree, J., Celedón-Pattichis, S., & Brinkerhoff,
J. (2007). Mathematics education at highly effective schools that
serve the poor: Strategies for change. Mahwah, NJ: Erlbaum.
Kline, R. B. (2004). Beyond significance testing: Reforming data
analysis methods in behavioral research. Washington, DC:
American Psychological Association.
Kline, R. B. (2011). Principles and practice of structural equation
modeling ( 3rd edition). New York: Guilford.
Kolar, D. W., & McBride, C. A. (2003). Creating problems
to solve problems: An interactive teaching technique for
statistics courses. Teaching of Psychology, 30, 67–68.
Kosoko-Lasaki, O., Sonnino, R. E., & Voytko, M. L. (2006).
Mentoring for women and underrepresented minority fac-
ulty and students: experience at two institutions of higher
education. Journal of the National Medical Association, 98(9),
1449–1459.
Krause, U,-M., Stark, R., & Mandl, H. (2009). The effects of
cooperative learning and feedback on e-learning in statistics.
Learning and Instruction, 19, 158–170.
La Greca, A. M. (2005). Editorial. Journal of Consulting and
Clinical Psychology, 73, 3–5.
Liu, T.-C., Lin, Y.-C., & Kinshuk (2010). The applica-
tion of Simulation-Assisted Learning Statistics (SALS) for
correcting misconceptions and improving understanding
of correlation. Journal of Computer Assisted Learning, 26,
143–158.
MacKinnon, D. P. (2008). Introduction to statistical mediation
analysis. New York: Taylor & Francis.
Malloy, T. E., & Jensen, G. C. (2001). Utah Virtual Lab:
JAVA interactivity for teaching science and statistics on line.
Behavior Research Methods, Instruments & Computers, 33,
282–286.
Mazzolini, M., & Maddison, S. (2007). When to jump in: The
role of the instructor in online discussion forums. Computers
& Education, 49, 193–213.
McGrath, R. E. (2011). Quantitative models in psychology.
Washington, DC: American Psychological Association.
116 t e a c h i n g q u a n t i tat i v e p s yc h o l o g y
McQuillan, L., & Kennelly, L. (2005). New study finds U.S.
math students consistently behind their peers around the world:
Findings challenge conventional wisdom about U.S. math success
in early grades. Washington, D.C.: American Institutes for
Research.
Meletiou-Mavrotheris, M, & Lee, C. (2002). Teaching students
the stochastic nature of statistical concepts in an introduc-
tory statistics course. Statistics Education Research Journal, 1,
22–37.
Mills, J. D. (2004). Students’ attitudes toward statistics: Impli-
cations for the future. College Student Journal, 38, 349–361.
Millsap, R. E., Maydeu-Olivares, A. (Eds.) (2009). The Sage
handbook of quantitative methods in psychology. Thousand
Oaks, CA: Sage Publications.
Mulhern, G., & Wylie, J. (2004). Changing levels of numer-
acy and other core mathematical skills among psychology
undergraduates between 1992 and 2002. British Journal of
Psychology, 95, 355–370.
Mulhern, G., & Wylie, J. (2006). Mathematical prerequisites
for learning statistics in psychology: Assessing core skills of
numeracy and mathematical reasoning among undergradu-
ates. Psychology Learning & Teaching, 5, 119–132.
Neal-Barnett, A., Mitchell, M., & Boeltar, C. (2002). Faculty
of color serving students, serving self: The psychology group.
Teaching of Psychology, 29, 44–45.
Nie, Y., & Lau, S. (2010). Differential relations of constructivist
and didactic instruction to students’ cognition, motivation,
and achievement. Learning and Instruction, 20, 411–423.
Odgaard, E. C., & Fowler, R. L. (2010). Confidence intervals for
effect sizes: Compliance and clinical significance in the jour-
nal of consulting and clinical psychology. Journal of Consulting
and Clinical Psychology, 78, 287–297.
Onwuegbuzie, A. J. (2000). Attitudes toward statistics assess-
ments. Assessment and Evaluation in Higher Education, 25,
321–339.
Onwuegbuzie, A. J., Collins, K. M. T., & Jiao, Q. G. (2009).
Performance of cooperative learning groups in a postgraduate
education research methodology course: The role of social
interdependence. Active Learning in Higher Education, 10,
265–277.
Onwuegbuzie, A. J., & Wilson, V. A. (2003). Statistics anxiety:
Nature, etiology, antecedents, effects, and treatments-a com-
prehensive review of literature. Teaching in Higher Education,
8, 195–209.
Peden, B. F. (2001). Correlational analysis and interpretation:
Graphs prevent gaffes. Teaching of Psychology, 28, 129–131.
Peterson, S. E., & Miller, J. A. (2004). Quality of college students’
experiences during cooperative learning. Social Psychology of
Education, 7(2), 161–183.
Piotrowski, C., Bagui, S. C., & Hemasinha, R. (2002). Devel-
opment of a measure of statistics anxiety in graduate-level
psychology students. Journal of Instructional Psychology, 29,
97–100.
Quilici, J. L., & Mayer, R. E. (2002). Teaching students to recog-
nize structural similarities between statistics word problems.
Applied Cognitive Psychology, 16, 325–342.
Rajecki, D. W., Appleby, D., Williams, C. C., Johnson,
K., & Jeschke, M. P. (2005). Statistics can wait: Career
plans activity and course preferences of American psychology
undergraduates. Psychology Learning & Teaching, 4, 83–89.
Ridgway, J., Nicholson, J., & McCusker, S. (2007). Teaching
statistics—Despite its applications. Teaching Statistics, 29,
44–48.
Rodarte-Luna, B., & Sherry, A. (2008). Sex differences in
the relation between statistics anxiety and cognitive/learning
strategies. Contemporary Educational Psychology, 33, 327–
344.
Rodgers, J. L. (2010). The epistemology of mathematical and
statistical modeling: A quiet methodological revolution.
American Psychologist, 65, 1–12.
Romero, V. L., Berger, D. E., Healy, M. R., & Aberson, C. L.
(2000). Using cognitive learning theory to design effective on-
line statistics tutorials. Behavior Research Methods, Instruments
& Computers, 32, 246–249.
Rosenthal, R. & DiMatteo, M. R. (2001) Meta-analysis: recent
developments in quantitative methods for literature reviews.
Annual Review of Psychology, 52, 59–82.
Rumsey, D. J. (2002). Statistical literacy as a goal
for introductory statistics courses. Journal of Statis-
tics Education, 10(3), Retrieved May 9, 2012 from
http://www.amstat.org/publications/jse/v10n3/rumsey2.html.
Saville, B. K., Zinn, T. E., Lawrence, N. K., Barron, K. E.,
& Andre, J. (2008). Teaching critical thinking in statistics
and research methods. In Dunn, D. S., Halonen, J. S., &
Smith, R. A. (Eds.), Teaching critical thinking in psychology: A
handbook of best practices, (pp. 149–160). New York: Wiley-
Blackwell.
Schwartz, D., & Martin, T. (2004). Inventing to prepare for
future learning: The hidden efficiency of encouraging orig-
inal student production in statistics instruction. Cognition
Instruction, 22, 129–184.
Sen, S. (2004). Teaching quantitative thinking: Comment.
Science, 304( 5670), 518.
Simonton, D. K. (2004). Psychology’s status as a scientific dis-
cipline: Its empirical placement within an implicit hierarchy
of the sciences. Review of General Psychology, 8, 59–67.
Sosa, S., Berger, D. E., Saw, A. T., & Mary, J. C. (2010).
Effectiveness of computer-assisted instruction in statistics:
A meta-analysis. Review of Educational Research, published
online August 12, 2010. DOI: 10.3102/0034654310378174
Stark, R., & Krause, U.-M. (2009). Effects of reflection prompts
on learning outcomes and learning behaviour in statistics
education. Learning Environments Research, 12, 209–223.
Stevens, J. P. (2009). Applied multivariate statistics for the social
sciences (5th ed.). New York: Routledge.
Swingler, M. V., Bishop, P., & Swingler, K. M. (2009). SUMS:
A flexible approach to the teaching and learning of statistics.
Psychology Learning & Teaching, 8, 39–45.
Tomcho, T. J., Rice, D., Foels, R., Folmsbee, L., Vladescu, J.,
Lissman, R., Matulewicz, R., et al. (2009). APA’s learning
objectives for research methods and statistics in practice: A
multimethod analysis. Teaching of Psychology, 36, 84–89.
Tremblay, P. F., Gardner, R. C., & Heipel, G. (2000). A model
of the relationships among measures of affect, aptitude, and
performance in introductory statistics. Canadian Journal of
Behavioural Science, 32, 40–48.
Upton, G. & Cook, I. (2008). A dictionary of statistics. New York:
Oxford University Press.
Velicer, W.F., & Jackson, D.N. (1990). Component analy-
sis vs. common factor analysis: Some issues in selecting an
appropriate procedure. Multivariate Behavioral Research, 25,
1–28.
Ware, M. E. & Johnson, D. E. (Eds) (2000). Handbook of
demonstrations and activities in the teaching of psychology: Intro-
ductory, statistics, research methods, and history, Vol. I (2nd
ed.). Mahwah, NJ: Erlbaum.
Watkins, A. E., Scheaffer, R. L., & Cobb, G. W. (2004). Statis-
tics in action: Understanding a world of data. New York: Key
College Publishing.
Watson, J. M. (2006). Statistical literacy at school: Growth and
goals. Mahwah, NJ: Erlbaum.
Wender, K. F., & Muehlboeck, J.-S. (2003). Animated diagrams
in teaching statistics. Behavior Research Methods, Instruments
& Computers, 35, 255–258.
Wilkinson, L., & The Task Force on Statistical Inference (1999).
Statistical methods in psychology journals: Guidelines and
explanations. American Psychologist, 54, 594–604.
Wulff, S. S., & Wulff, D. H. (2004). “Of course I’m com-
municating, I lecture every day:” Enhancing teaching and
learning in introductory statistics. Communication Education,
53, 92–103.
Zeedyk, M. S. (2006). Detective work on statistics street: Teach-
ing statistics though humorous analogy. Psychology Learning
& Teaching, 5, 97–109.
Zieffler, A., Garfield, J., Alt, S., Dupuis, D., Holleque, K.,
& Chang, B. (2008). What does research suggest about
the teaching and learning of introductory statistics at the
college level? A review of literature. Journal of Statistics
Education, 16 (2), Retrieved May 9, 2012 from http://
www/amstat.org/publications/jse/v16n2/zieffler.html.
Zirkel, S. (2002). Is there a place for me? Role models and aca-
demic identity among white students and students of color.
Teachers College Record, 104, 357–376.
harlow 117
 CHAPTER

Modern Test Theory

7
Roderick P. McDonald†

Abstract

This chapter provides a unified treatment of seven major topics in test theory. It relates Modern
Theory based on item response modeling to classical linear modeling through the use of the common
factor model. The topics include choosing a metric; measurement and measurement error (reliability);
item selection: homogeneity and dimensionality; validity; equating tests; comparing populations. The
treatment of these problems makes no distribution assumptions.
Key Words: Test theory, metri