1838
Commentary: Causal doi: 10.1093/ije/dyw229
Advance Access Publication Date: 27 January 2017
inference in epidemiology:
potential outcomes, pluralism and peer review
Douglas L Weed
DLW Consulting Services, LLC, University of Utah School of Medicine, Salt Lake City, UT 84132, USA.
E-mail: douglaslweed@aol.com
Accepted 20 July 2016
Introduction: The Debate on the Exclusivity
of Potential Outcomes
Now that two groups have challenged the idea that the poten-
tial outcomes (counterfactual) approach (POA) is the only le-
gitimate approach to causal inference in epidemiology, I fully
expect the supporters of that view to either defend their pos-
ition or offer an olive branch to appease their critics. The de-
bate is engaged and best left to those in academic towers
where these arguments mean something. In the practice of
epidemiology, despite what Vandenbroucke et al.1 and
Krieger and Davey Smith2 contend, the presumed negative
impact of POA has yet to be realized. No journal requires it.
Sometimes counterfactuals and DAGs appear and often they
do not. In the complex and ever-expanding worlds of regula-
tion and litigation, where epidemiology plays a key role, POA
is at best a whisper. Simply put, the debate about whether
POA is the only legitimate approach to causal inference in
epidemiology is as much about the power of individuals at
certain academic institutions to gain attention as it is about
the intellectual competitions that excite so-called ‘theoret-
icians’ of epidemiology. It remains to be seen if the exclusivity
claim is really a critical issue for the future of epidemiology.
Pluralism and Its Constraints
That said, there are some disturbing features of the argu-
ments by Vandenbroucke et al.1 and Krieger and Davey
Smith2 with important implications for the practice of
causal inference and for peer review. Of particular concern
are the alternative solutions (to POA) offered by these au-
thors—what they provide as proposed improvements for
causal inference. To be blunt, they advocate a type of ‘plur-
alism’, something akin to ‘anything goes’ when attempting
to assess disease causation as long as there is evidence and
judgment and their approaches fall under the rubric of ‘in-
ference to the best explanation’, at best a philosophical slo-
gan (according to its author, Lipton3).
C The Author 2017; all rights reserved. Published by Oxford University Press on behalf of the International Epidemiological Association
V
Downloaded from https://academic.oup.com/ije/article-abstract/45/6/1838/2960068
by Stockholm University Library user
on 17 May 2018
International Journal of Epidemiology, 2016, Vol. 45, No.
International Journal of Epidemiology, 2016, 1838–1840
For those of us who have participated in these sorts of
philosophical discussions—typically, philosophy in epi-
demiology—over the decades, it is not sufficient to simply
critique a method (or an approach such as POA), identify
its problems (even errors) and call it a day. To make pro-
gress (i.e. to make improvements), an alternative must be
proposed that does at least several things: it corrects the
errors in the original method, it keeps the features of the
original method that ‘work’ and it does not add new prob-
lems (errors) that make it less desirable than the original
method. With this working model of progress in mind, I
turn to the alternative approaches offered by Krieger and
Davey Smith2 and by Vandenbroucke et al.1
According to Krieger and Davey Smith, ‘there is not one
prescription for how epidemiologists can best infer caus-
ation’. Rather, we epidemiologists should be ‘open to plural-
ism about causation’ and use the approach ‘best developed
by Peter Lipton’ which will help us to ‘forge that aspire to
scope, precision, mechanism, unification, and simplicity’,
the ‘loveliest’ and not necessarily the ‘likeliest’ explanation.
Lovely explanations are a feature of Lipton’s philosophical
treatise,3 which really cannot be said to provide epidemiolo-
gists with anything resembling a practical approach to
causal inference. Krieger and Davey Smith’s contention,2
that Lipton frequently employs epidemiological examples, is
misleading. Lipton is no epidemiologist, and his intent is not
to provide a practical approach to causal inference in epi-
demiology or elsewhere. In his own words, Lipton is at-
tempting to ‘describe inductive processes’ in philosophical
terms. So we are left with what Krieger and Davey Smith2
articulate from their translation of Lipton’s effort, which is
that there are many kinds of study designs relevant and use-
ful for causal inference in epidemiology, each of which in-
volves different assumptions and biases. In their words,
there is a ‘need for causal triangulation, whereby epidemi-
ologists should employ diverse study designs, each involving
different and unrelated potential biases, and test our
 International Journal of Epidemiology, 2016, Vol. 45, No. 6
hypotheses in different populations and in different histor-
ical periods, to see if results are robust to the confounding
structures encountered and the analytic methods used’. Yes.
I agree. It is good to use more than one study design and to
examine hypotheses in more than one population, which
typically occurs in more than one historical ‘period’. Bias
and confounding are issues we need to consider, using ana-
lytical and statistical methods. Judgment is essential.
According to Vandenbroucke et al.,1 the solution to the
problem described above is also pluralism or what they
call ‘pragmatic pluralism’ which involves judgment, ‘rag-
ged evidence,’ triangulation and other ‘diverse strategies’.
They propose that ‘future epidemiologists should learn: (i)
that causal inference remains a judgment based on integra-
tion of diverse types of evidence; (ii) diverse strategies to
assess causality by ruling out alternatives, such as triangu-
lation, negative controls, and interlocking evidence from
other types of science; (iii) the elements of all types of epi-
demiological study designs, including counterfactual de-
signs; and (iv) to reflect critically on whether potential
biases matter’. Yes. I agree. Use different study designs.
Rule out alternatives, including bias and confounding (and
chance?); employ judgment. This is also not an innovative
solution to the problem.
In sum, both groups focus on judgment, and both groups
emphasize pluralism by which they mean something other
than a single approach. Remember that their arguments are
focused upon why one approach—POA—cannot be suffi-
cient. Fair enough. The POA approach isn’t enough.
But has the methodological pendulum swung too far in
the opposite direction? What these experienced epidemiolo-
gists advocate is a collection of potentially competing ideas
and vague approaches, not a prescription (to be sure), but
also not directly tied to how epidemiologists actually practise
causal inference in today’s world, as informed by a half cen-
tury of theory and practice. If a student were to ask me how
to practise causal inference based on these two papers, I
would tell them that there is no one way to do it, that you
should use different study designs, ‘triangulate’ and ‘interlock’
evidence from other sciences. Above all, use your judgment
about all that ‘ragged evidence’ and assess whether confound-
ing and bias can explain the results. Finally, consider how
‘lovely’ your explanation is. I would not be surprised if the
student would reply, ‘Yes, OK, but how do I determine if an
exposure causes a disease?’
I feel this student’s pain. There is no mention in these
two papers about the need to employ systematic review
methods, no mention of statistical significance, no mention
of meta-analysis and no mention of the tools for determin-
ing study quality or study hierarchies. Oddly enough, there
are brief mentions of Hill’s causal considerations deep in
the guts of their arguments, but no discussion and no ap-
Downloaded from https://academic.oup.com/ije/article-abstract/45/6/1838/2960068
by Stockholm University Library user
on 17 May 2018
1839
preciation for how this well-known and still relevant ap-
proach fits into the family of methods used in everyday
epidemiological practice.
Beyond the idea that there is more than one type of
study design, the pluralistic approach to causal inference in
epidemiology advocated in these two papers appears to
consist primarily of metaphors—did I mention ‘tale wag-
ging the dag?’—and judgment. This is not progress. These
authors have latched onto pluralism rather than a unilat-
eral POA approach, a simple solution to be sure. However,
in effect they have solved the problem of relying too much
upon a single approach by introducing an even more vex-
ing problem: relying upon any and all approaches as long
as they involve many different study designs or ‘triangula-
tion’ or ‘interlocking (ragged) evidence’, judgment and
lovely explanations. In the pluralistic world of
Vandenbroucke et al.1 and Krieger and Davey Smith,2 the
metaphors plus judgment approach—whether actually
based on Lipton’s philosophical treatise or not—is as insuf-
ficient as POA and potentially less helpful. In the end, the
alternatives introduce as many—perhaps more—problems
than the single problem they attempt to solve.
Judgment
As noted above, both Vandenbroucke et al.1 and Krieger
and Davey Smith2 emphasize judgment as a key compo-
nent of their pluralistic approaches. This is not a new phe-
nomenon, given that Susser4—one of the early giants of
causal inference in epidemiology—emphasized the import-
ance of judgment, and it continues to appear in recent dis-
cussions of so-called ‘weight of evidence’ approaches to
causation (Adami et al.,5 Cogliano et al.6 and Krimsky7).
But what exactly is judgment? We can agree that judg-
ment is an important part of the process of causal infer-
ence, and we can agree that we prefer those who exhibit
‘good’ or ‘sound’ judgment over those whose judgment is
questionable.8 When collecting, summarizing and inter-
preting scientific evidence, we can also agree that judgment
is applied at many steps. The problem with judgment is
that it is a complex and not wel- understood mental cap-
acity. Krimsky7 calls it a ‘black box’, a familiar term for
epidemiologists.9 At the minimum, judgment includes: sci-
entific reasoning, ethical reasoning, practical wisdom and
values. Judgment is not, therefore, a purely subjective pro-
cess, although it has a clear and significant subjective com-
ponent. Relying too much upon judgment, however, is
problematic. After all, how can we objectively determine
whose judgment is better?
Judgment may be a necessary component of the process,
but is not, by itself, sufficient for making causal claims.
Methods of causal inference and a systematically collected
 1840
body of evidence are necessary. These methods take prece-
dence over judgment in any causal assessment.
Peer review
If Vandenbroucke et al.1 and Krieger and Davey Smith2
are correct in arguing that the POA approach cannot be
the only way to assess causation, then their pluralism (plus
judgment) solution creates havoc for the peer reviewer.
How can those who have been tasked with reviewing any
submitted manuscript or published paper, that claims to
infer (or establish) or assess causation, proceed?
Given that the literature on peer review is remarkably quiet
on specifics when it comes to causal inference, I offer the fol-
lowing considerations to students and peer reviewers alike:
i. whether the authors rely primarily upon method or
primarily upon their subjective judgment in making
claims about causation;
ii. whether a method for determining causation is
described;
iii. whether the method described is one generally recog-
nized in the scientific community and referenced there;
iv. whether the authors’ description of that method is ac-
curate [i.e. whether it reasonably conforms to the de-
scriptions of that method in the published literature or
misrepresents (deviates prominently from) those same
description];
v. whether that method is appropriate for the scientific
question at hand;
vi. whether the method selected by the author was used
appropriately to interpret results.
In my experience, these six considerations cover the
bases reasonably well. Put another way, infractions of any
one of these—much less several—cast serious doubt on the
validity and reliability of any authors’ causal assessment.
Conclusion
This is not the place to fully describe the approach to causal
inference that exists in the practising community of epidemio-
logical scientists and in the community of biomedical scientists
in general. The approach—really a family of methods—has
been ignored in these papers, perhaps because it is both too
complex and too well-known. It begins with a systematic col-
lection and description of the available evidence, an assess-
ment of the quality of the individual studies (which utilize
Downloaded from https://academic.oup.com/ije/article-abstract/45/6/1838/2960068
by Stockholm University Library user
on 17 May 2018
International Journal of Epidemiology, 2016, Vol. 45, No. 6
different study designs and statistical methods), summariza-
tion of the results across studies (when appropriate) whether
by pooled analysis or meta-analysis, the application of the
Bradford-Hill viewpoints (or criteria or whatever else you
want to call them) which creates the need to repeat the process
described above in the biological-mechanistic world of toxi-
cology, and a description and critique of past reviews examin-
ing the same exposure-disease relationship. The overarching
method that subsumes all these is the general scientific
method. Viewing causal inference from the perspective of how
it is actually practised permits—indeed, invites—the inclusion
of the DAG (counterfactual) approach as one picturesque way
to describe the potential effects of exposures and confounders.
I respect the efforts of all those who work to improve
the theory and practice of causal inference in epidemi-
ology. After all, causality is the most important scientific
problem we epidemiologists aim to solve.
Conflict of interest: Dr. Weed is an independent scien-
tific consultant. No entity, whether non-profit or for-
profit, provided funding for this paper. Dr. Weed con-
ceived, designed, and wrote this paper without assistance
from any organization or individual.
References
1. Vandenbroucke JP, Broadbent A, Pearce N. Causality and causal
inference in epidemiology: the need for a pluralistic approach. Int
J Epidemiol 2016;45:1776–86.
2. Krieger N, Davey Smith G. The tale wagged by the DAG:
broadening the scope of causal inference and explanation for epi-
demiology. Int J Epidemiol 2016;45:1787–808.
3. Lipton P. Inference to the Best Explanation. 2nd edn. London and
New York: Routledge, 2004.
4. Susser M. Judgment and causal inference: criteria in epidemio-
logic studies. Am J Epidemiol 1977;105:1–15.
5. Adami HO, Berry CL, Breckenridge CB et al. Toxicology and epi-
demiology: improving the science with a framework for combin-
ing toxicological and epidemiological evidence to establish causal
inference. Toxicol Sci 2011;122:223–34.
6. Cogliano VJ, Baan RA, Straif K et al. The science and practice of
carcinogen identification and evaluation. Environ Health Perspect
2004;112:1269–74.
7. Krimsky S. The weight of evidence in policy and law. Am J Pub
Health 2005;95:S129–36.
8. Weed DL. The nature and necessity of scientific judgment. J Law
Policy 2007;15:135–64.
9. Weed DL. Beyond black box epidemiology. Am J Public Health
1998;88:12–14.