ARTICLE IN PRESS

Biological Conservation xxx (2009) xxx–xxx

Contents lists available at ScienceDirect

Biological Conservation
journal homepage: www.elsevier.com/locate/biocon

Doing more good than harm – Building an evidence-base for conservation

and environmental management
Andrew S. Pullin *, Teri M. Knight
Centre for Evidence-Based Conservation, School of the Environment and Natural Resources, Bangor University, Gwynedd, LL57 2UW, UK

a r t i c l e i n f o a b s t r a c t

Article history: The problems of environmental change and biodiversity loss have entered the mainstream political
Received 14 October 2008 agenda. Given the call from an increasingly influential environmental lobby for government and wider
Received in revised form 20 January 2009 society to make both financial and personal sacrifices to address these problems, it seems likely that con-
Accepted 24 January 2009
servation biologists and environmental managers will be asked tough questions of the general form ‘are
Available online xxxx
conservation interventions effective?’ and, ‘are they doing more good than harm?’ Science constantly
advances and must remain open to challenge, but managers and policy formers require an interim prod-
Keywords:
uct (an evidence-base) to underpin their current decision-making. The health services have been using
Systematic review
Conservation management
the objective and transparent methodology of systematic review to summarise the evidence-base relat-
Conservation policy ing to the effectiveness of interventions. Environmental management has, up until now, had no formal
Environmental evidence shared evidence-base of this kind. Reviewing recent developments in evidence-based practice, this paper
Conservation planning introduces a ‘systematic review’ section for this journal and argues that constructing an evidence-based
Adaptive management framework for environment management is possible, the challenge is scaling it up to engage the global
scientific community. We draw on the history of evidence-based healthcare, but also on the differences
between healthcare and conservation, to set out the challenges in creating a Collaboration for Environ-
mental Evidence that develops a library of systematic reviews on the effectiveness of conservation and
environmental interventions.
Ó 2009 Elsevier Ltd. All rights reserved.

1. Background movement may see them as perfectly reasonable. If we are to move

The problems of environmental change and biodiversity loss
have entered the mainstream political agenda over the last decade
and, as a community of scientists and managers seeking ways to ad-
dress these problems, our public profile is higher than ever before.
Conservation biology as a discipline is rapidly changing and expand-
ing to embrace economics and social sciences (Meine et al., 2006;
Balmford and Cowling, 2006), and conservation biologists are fre-
quently encouraged to make the discipline relevant by showing
how a healthy, biodiverse environment can improve human welfare
and wellbeing (Robinson, 2006). The message is clear; we need to
demonstrate to wider society that conserving biodiversity makes
a difference, that it’s effective in maintaining and improving quality
of our lives and those of future generations. This challenge is not
simple or easy; conservation actions cost money and therefore cost
society (Ferraro and Pattanayak, 2006). Consequently it seems valid
to ask tough questions of the general forms ‘are conservation inter-
ventions doing more good than harm?’ and ‘are conservation inter-
ventions cost effective?’ Many within the conservation movement
may raise an eyebrow at these questions but others outside the
beyond our boundaries and argue our case in wider society we must
expect this level of scrutiny. Consider the health sector; medical re-
search has moved beyond its own boundaries into social sciences,
education and environment through disciplines such as public
health and epidemiology. The health services have perhaps had less
difficulty in making their case to wider society, nobody doubts the
need for health services, yet health interventions are the subject
of considerable scrutiny concerning their effectiveness (the extent
to which a specific intervention, under normal conditions of use,
does what it is intended to do). Of course, the level of active inter-
vention in human health is partly due to the sums of money in-
vested, but one can turn this argument on its head and suggest
that, increasingly, money is invested where research has provided
evidence that interventions make a positive difference. Perhaps
more significantly, evidence has been used to show that some com-
monly used medical interventions can be ineffective and a waste of
money, for example, grommets for child hearing (Lous et al., 2005)
and some antidepressant drugs (Kirsch et al., 2008).
2. The problem of having no shared evidence-base

* Corresponding author. Tel.: +44 1248 382444. Conservation biologists share a framework of values and goals.
E-mail address: a.s.pullin@bangor.ac.uk (A.S. Pullin). A growing number of conservation organisations work in many

0006-3207/$ - see front matter Ó 2009 Elsevier Ltd. All rights reserved.
doi:10.1016/j.biocon.2009.01.010

Please cite this article in press as: Pullin, A.S., Knight, T.M. Doing more good than harm – Building an evidence-base for conservation ...
Biol. Conserv. (2009), doi:10.1016/j.biocon.2009.01.010
 ARTICLE IN PRESS
2 A.S. Pullin, T.M. Knight / Biological Conservation xxx (2009) xxx–xxx
different ways and use different mechanisms to achieve targets set
within this framework. As in the health services nobody seriously
questions the motives of the vast majority working in biodiversity
conservation, nor their commitment to doing a good job. But, just
as in the health services, without good information on what works
and what does not, and without effective monitoring and evalua-
tion, how do we know that we’re doing more good than harm?
How do we convince our critics that we are regulating our own
practice? Can we expect donors to keep funding conservation
without asking for evaluation of our projects? Do we have the data
to show we are making a difference rather than simply assuming
we are doing some good? Human health care is comparatively well
regulated, whereas anyone can claim to be doing effective conser-
vation. In an increasingly critical society, how do we keep our own
house in order, maintain credibility and differentiate between
effective and ineffective conservation programs and ultimately
make best use of limited resources?
In some cases the evidence that we are doing more good than
harm may seem obvious and straightforward, such as reversing
trends in declining species or designating protected areas that
would otherwise be damaged by human activities, whilst in others
it may well be unclear if our actions have made any difference, or
worse. Brooks et al. (2006) reviewed the effectiveness of integrated
conservation and development programmes in terms of delivering
their conservation objectives. They concluded that it is currently
not possible to provide a systematic evaluation of the effectiveness
of such programmes, largely because of the inadequacy of project
reporting.
Further, results from some individual studies of effectiveness
may be convincing, but in other cases, different studies may be
contradictory and arguments may rest on which fragment of the
evidence-base is scrutinised. For example, a good number of stud-
ies have been published on the effectiveness of engineered in-
stream structures for creating habitat for salmonid fish but they
present contradictory findings (e.g. Bayley, 2002; Roni et al.,
2002, 2005; Thompson, 2006). To demonstrate objectivity, all
available evidence needs to be considered. Stewart et al. (in press)
systematically reviewed the effectiveness of engineered in-stream
structures in creating habitat for salmonid fish. Synthesis of all the
available high-quality evidence provided evidence that these
expensive structures are ineffective in the majority of cases.
As a community of professionals, we need to critically evaluate
our actions and continuously seek to increase our effectiveness. If
we have done a lot of good things we should have the evidence
to demonstrate it. But credibility is also about being self critical
and identifying the bad as well as the good. Of course, there is a
political risk in going public about how much we know, or do
not know. Some may see knowledge gaps as weaknesses and ex-
cuses for political inaction; allowing statements such as ‘we need
more data before we act’. But contrast this with the potential ben-
efit of being able to collate and synthesise existing data in order to
show that there is consensus on an effective solution to a conser-
vation problem.
Critical evaluation of interventions can occur on a number of
different levels. The monitoring and evaluation (internal audit) of
individual projects is becoming more commonplace (Stem et al.,
2005). Much progress has been made in the specific area of adap-
tive management (Margoluis et al., 1998). The challenge is to draw
together and synthesise all the available evidence to evaluate
effectiveness and guide future decisions.
The need for a library of systematic reviews providing an evi-
dence-based framework to evaluate effectiveness and support
decision-making in conservation has been made in previous papers
(Pullin and Knight, 2001; Fazey et al., 2004; Sutherland et al.,
2004). Testing the applicability of the health services methodology
for systematic review and dissemination of evidence is ongoing but
Please cite this article in press as: Pullin, A.S., Knight, T.M. Doing more good than harm – Building an evidence-base for conservation ...
Biol. Conserv. (2009), doi:10.1016/j.biocon.2009.01.010
progress is evident in the first published systematic reviews (Tyler
et al., 2006; Davies and Pullin, 2007; Roberts and Pullin, 2007,
2008; Stewart et al., 2007a,b; Davies et al., 2008), and in the estab-
lishment of review methodology through publication of systematic
review guidelines (Pullin and Stewart, 2006). The point of this arti-
cle is to present the challenge for the next phase of its develop-
ment. In introducing the new systematic review section for
Biological Conservation, this paper argues that the methodology
of systematic review is beneficial for conservation and an evi-
dence-based framework is achievable; the challenge is scaling it
up from a pilot phase to a global collaboration that really makes
a difference to decision-making and resource allocation. We draw
on the history of the development of evidence-based medicine
and particularly the Cochrane Collaboration, but also on the differ-
ences between the disciplines of healthcare and conservation, to
set out the challenges in creating an evidence-base for conserva-
tion and environmental management.
3. What is a systematic review?
Systematic review is a methodology normally used to objec-
tively assess the effectiveness of an intervention or the impact of
an action. Systematic reviews follow a strict protocol to maximise
transparency and repeatability whilst minimizing bias. A key ele-
ment is the attempt to obtain and critically appraise all available
evidence (published and unpublished) relevant to the review ques-
tion. The methodology is particularly used to summarise, appraise
and communicate the results and implications of a large quantity
of research and information, often involving quantitative synthesis
of data to increase statistical power. Qualitative data are also fre-
quently reported using narrative syntheses.
Systematic reviews usually address a specific question contain-
ing three elements;
1. A subject (a population, species, habitat etc).
2. An intervention (directed at the subject) or other action (indi-
rectly impacting on the subject).
3. An outcome of interest (a change in state of the subject that is
measurable and of concern).
Thus a typical intervention-based question could be ‘does pop-
ulation x benefit from intervention y in terms of increases in z?’
where x is the subject, y is the conservation intervention and z is
the measurable outcome of the intervention. Alternatively a ques-
tion can seek to measure impact rather than effectiveness. An im-
pact-based question might be ‘what is the impact on population x
of action y in terms of changes in z?’ where action y is not a conser-
vation intervention but some other action such as installation of
wind farms.
The detailed methodology of systematic reviews has been de-
scribed elsewhere (Pullin and Stewart, 2006) and can be summa-
rised as;
1. Formulating a question.
2. Generating a protocol.
3. Systematic searching.
4. Study selection.
5. Methodological quality assessment (critical appraisal).
6. Data extraction.
7. Data synthesis.
8. Reporting and dissemination.
The contrast in rigour between systematic reviews and tradi-
tional narrative reviews in ecology has been demonstrated by
Roberts et al. (2006). However, it is in the health services that
 ARTICLE IN PRESS
A.S. Pullin, T.M. Knight / Biological Conservation xxx (2009) xxx–xxx
the use of systematic reviews has been most fully exploited and it
is useful to consider how this has been achieved.
4. Development of review and dissemination of evidence in
medicine: the Cochrane Collaboration and beyond
The Cochrane Collaboration was established in 1993 but its ori-
gins can be traced back to the publications of Archie Cochrane in
the early 1970s (Cochrane, 1972). It is a global, not-for-profit,
organisation that supports health care professionals to make
well-informed decisions on the effectiveness of medical interven-
tions by promoting systematic review of evidence and maintain-
ing a library of such evaluations (www.cochrane.org). Arguably,
the Cochrane Collaboration has inspired a whole range of allied re-
view and dissemination projects and so it is important to under-
stand its developmental history. Following the early work of
Cochrane, the movement seems to have gone through four key
phases:
1. The exposure of the problem – no accessible evaluations of the
effectiveness of common interventions resulting in variation in
practice across hospitals and health care districts.
2. Describing the nature and source of the problem – practitioners
have no time to access primary literature and to synthesise evi-
dence. Cochrane (1979) wrote ‘‘It is surely a great criticism of
our profession that we have not organised a critical summary,
by specialty or subspecialty, adapted periodically, of all relevant
randomised controlled trials.”
3. Establishing a solution – a collaborative movement mobilising
resources for review and dissemination of evidence on the
effectiveness of health interventions. The early development
of the Cochrane Collaboration has been described elsewhere
(Chalmers, 1993).
4. Growth of the collaboration – successful engagement of the sci-
entific and funding communities to enable a large volume of
systematic reviews to be undertaken, resulting in an effective-
ness revolution in heath care (Stevens and Milne, 1997).
4.1. Components of Cochrane
The focus of the Cochrane Collaboration is the library of system-
atic reviews (Starr and Chalmers, 2003). This resource, shared by
the whole health services community, is overseen by a steering
committee and managed day-to-day by a small Centre composed
of a Chief Executive and three administrative staff. The need for a
review may be proposed by any individual stakeholder or stake-
holder group. The review work is highly devolved in the form of
a global group of 12 ‘Cochrane Centres’ and over 50 ‘Review
Groups’. Typically the Centres support the activity of Review
Groups in their geographical area whilst also developing review
methodology and providing training. Systematic reviews are co-
ordinated by ‘Review Groups’, typically subject-based, so that each
evaluates interventions on a particular medical problem, such as
‘acute respiratory infection’. Each has a volunteer co-ordinating
editor to oversee standards (including transparency and conflict
of interest) and organise peer review. Some groups are also formed
to advance the methodology of systematic review and data synthe-
sis. In summary there are four main components;
1. The library (the output).
2. The steering group/CEO (the administration).
3. The centres (the co-ordination and support functions).
4. The review groups (the review activity).
The devolved nature of the collaboration means that the cost of
administration is low. The centres and review groups attract their
Please cite this article in press as: Pullin, A.S., Knight, T.M. Doing more good than harm – Building an evidence-base for conservation ...
Biol. Conserv. (2009), doi:10.1016/j.biocon.2009.01.010
3
own funding for review activity. Over 12,000 authors have now
contributed to the Cochrane Library.
Whilst the medical-clinical basis of Cochrane demands high
standards of data quality, mostly generated through randomised
controlled trials, many related disciplines, including public health,
social sciences and education, use more diverse quantitative and
qualitative research methods. Consequently, as the evidence-based
approach has moved into related fields, so systematic review
methodology has been adapted to cope with different types of data
and research methodologies. Integration of qualitative and quanti-
tative research data within systematic reviews is an emerging
methodology (Harden, 2006). Consequently, whilst Cochrane re-
mains a core model for developing the necessary structures to
facilitate systematic review activity, the construction of an envi-
ronmental equivalent must also take heed of broader develop-
ments in evidence-based decision-making, given the diverse
nature of environmental data.
5. The challenge of establishing an evidence-based framework
Arguably biodiversity conservation has gone through the equiv-
alent of Cochrane phases one (Pullin and Knight, 2001; Sutherland
et al., 2004; Ferraro and Pattanayak, 2006) and two (Pullin et al.,
2004; Pullin and Knight, 2005). A crucial start has been made on
phase three with the establishment of the Collaboration for Envi-
ronmental Evidence (CEE, www.environmentalevidence.org) as a
not-for-profit, ‘Cochrane style’ organisational structure, open to
contributions from all in conservation science and management.
The challenge for the conservation community is how to move from
a situation in which small units of people are testing systematic re-
view methodology, to establishing a vibrant functioning network?
The experience of the Cochrane Collaboration suggests that the
key people to mobilise are those in the scientific community. Those
who could undertake reviews need incentives (grants are already
available from a few sources see www.environmentalevidence.org)
and publishing opportunities (with impact factors) for their prod-
ucts. The latter opportunity is now realised in Biological Conserva-
tion, an international journal dealing with the conservation of
wildlife and the wise use of biological and allied natural resources.
We encourage the conservation community to consider undertak-
ing systematic reviews to create an evidence-base for their conser-
vation interventions and other human actions that have
environmental impacts. This journal now considers papers that re-
port on full systematic reviews. Supplementary material in the form
of full systematic reviews and related libraries and datasets will be
available online (e.g. through the CEE Library). The first report on a
systematic review follows this paper and should be viewed as a pio-
neering effort to develop and improve the methodology. Through
publishing papers reporting on high impact systematic reviews in
one of the foremost conservation journals, the utility of this ap-
proach will hopefully be more apparent to funders and enable us
to test whether more effective conservation and environmental
management results from investment in evidence-based practice.
6. Final points
There are many books and papers that describe environmental
science or conservation biology as emerging young disciplines or
growing subjects. What then defines the maturing phase? One pos-
sibility is the transition from a disparate community that espouses
the urgency of research and greater understanding of our actions to
one that is also able to concertedly reflect, synthesise, evaluate and
disseminate the current state of knowledge. A key indicator of
maturity may be the number of systematic reviews available to
those who must decide what action to take.
 ARTICLE IN PRESS
4 A.S. Pullin, T.M. Knight / Biological Conservation xxx (2009) xxx–xxx
Acknowledgements
Sincere thanks to all those who have so far engaged in and sup-
ported the concept of evidence-based conservation. We thank Rob
Marrs and three anonymous reviewers for their constructive com-
ments and improvements to the manuscript. This work was made
possible by grants from the UK Natural Environment Research
Council, Natural England and the UK Environment Agency.
References
Balmford, A., Cowling, R.M., 2006. Fusion or failure? The future of conservation
biology. Conservation Biology 20, 692–695.
Bayley, P., 2002. A Review of Studies on Responses of Salmon and Trout to Habitat
Change, With Potential for Application in the Pacific Northwest. Report to the
Washington State Independent Science Panel, Olympia. WA.
Brooks, J.S., Franzen, M.A., Holmes, C.M., Grote, M.N., Borgerhoff Mulder, M., 2006.
Development as a Conservation Tool: Evaluating Ecological, Economic,
Attitudinal, and Behavioral Outcomes. Systematic Review 20. Collaboration
for Environmental Evidence (<www.environmentalevidence.org>).
Chalmers, I., 1993. The Cochrane Collaboration: preparing, maintaining and
disseminating systematic reviews of the effects of health care. Annals of the
New York Academy of Sciences 703, 156–165.
Cochrane, A.L., 1972. Effectiveness and Efficiency: Random Reflections on the
Health Services. Nuffield Provincial Hospitals Trust, London.
Cochrane, A.L., 1979. 1931–1971: a critical review, with particular reference to the
medical profession. In: Medicines for the Year 2000. Office of Health Economics,
London, pp. 1–11.
Davies, Z.G., Pullin, A.S., 2007. Are hedgerows effective corridors between fragments of
woodland habitat: an evidence-based approach. Landscape Ecology 22, 333–351.
Davies, Z.G., Tyler, C., Stewart, G.B., Pullin, A.S., 2008. Are current management
recommendations for saproxylic invertebrates effective? A systematic review.
Biodiversity and Conservation 17, 209–234.
Fazey, I., Salisbury, J.G., Lindenmayer, D.B., Maindonald, J., Douglas, R., 2004. Can
methods applied in medicine be used to summarize and disseminate
conservation research? Environmental Conservation 31, 190–198.
Ferraro, P.J., Pattanayak, S.K., 2006. Money for nothing? A call for empirical
evaluation of biodiversity conservation investments. PLoS Biology 4, 482–488.
Harden, A., 2006. Extending the boundaries of systematic reviews to integrate
different types of study: examples of methods developed within reviews on
young people’s health. In: Popay, J. (Ed.), Moving Beyond Effectiveness in
Evidence Synthesis. National Institute for Health and Clinical Excellence,
London, pp. 31–40.
Kirsch, I., Deacon, B.J., Huedo-Medina, T.B., Scoboria, A., Moore, T.J., Johnson, B.T.,
2008. Initial severity and antidepressant benefits: a meta-analysis of data
submitted to the Food and Drug Administration. PLoS Medicine 5, e45.
doi:10.1371/journal.pmed.0050045.
Lous, J., Burton, M.J., Felding, J.U., Ovesen, T., Rovers, M.M., Williamson, I., 2005.
Grommets (ventilation tubes) for hearing loss associated with otitis media with
effusion in children. Cochrane Database of Systematic Reviews (1). doi:10.1002/
14651858.CD001801.pub2. Art. No.: CD001801.
Margoluis, R., Salafsky, N., Balla, A., 1998. Measures of Success: Designing, Managing
and Monitoring Conservation and Development Projects. Island Press,
Washington, DC.
Please cite this article in press as: Pullin, A.S., Knight, T.M. Doing more good than harm – Building an evidence-base for conservation ...
Biol. Conserv. (2009), doi:10.1016/j.biocon.2009.01.010
Meine, C., Soule, M., Noss, R.F., 2006. ‘‘A mission-driven discipline”: the growth of
conservation biology. Conservation Biology 20, 631–651.
Pullin, A.S., Knight, T.M., 2001. Effectiveness in conservation practice. Pointers from
Medicine and Public Health. Conservation Biology 15, 50–54.
Pullin, A.S., Knight, T.M., 2005. Assessing conservation management’s evidence-
base: a survey of management-plan compilers in the United Kingdom and
Australia. Conservation Biology 19, 1989–1996.
Pullin, A.S., Stewart, G.B., 2006. Guidelines for systematic review in conservation
and environmental management. Conservation Biology 20, 1647–1656.
Pullin, A.S., Knight, T.M., Stone, D.A., Charman, K., 2004. Do conservation managers
use scientific evidence to support their decision-making? Biological
Conservation 119, 245–252.
Roberts, P.D., Pullin, A.S., 2007. The effectiveness of management interventions used
to control ragwort species. Environmental Management 39, 691–706.
Roberts, P.D., Pullin, A.S., 2008. The effectiveness of management interventions for
the control of spartina Species: a systematic review and meta-analysis. Aquatic
Conservation: Marine and Freshwater Ecosystems 18, 592–618.
Roberts, P.D., Stewart, G.B., Pullin, A.S., 2006. Are review articles a reliable source of
evidence to support conservation and environmental management? A
comparison with medicine. Biological Conservation 132, 409–423.
Robinson, J.G., 2006. Conservation biology and real-world conservation.
Conservation Biology 20, 658–669.
Roni, P.T., Beechie, J., Bilby, R.E., Leonetti, F.E., Pollock, M.M., Pess, G.R., 2002. A
review of stream restoration techniques and a hierarchical strategy for
prioritizing restoration in Pacific Northwest watersheds. North American
Journal of Fisheries Management 22, 1–20.
Roni, P., Hanson, K., Beechie, T., Pess, G., Pollock, M., Bartley, D.M., 2005. Habitat
Rehabilitation for Inland Fisheries: Global Review of Effectiveness and Guidance
for Rehabilitation of Freshwater Ecosystems. FAO Fisheries Technical Paper 484,
FAO, Rome.
Starr, M., Chalmers, I., 2003. The evolution of The Cochrane Library, 1988–2003.
Update Software: Oxford. (<www.update-software.com/history/clibhist.htm>)
(accessed 13.10.2008).
Stem, C., Margoluis, R., Salafsky, N., Brown, M., 2005. Monitoring and evaluation in
conservation: a review of trends and approaches. Conservation Biology 19, 295–
309.
Stevens, A., Milne, R., 1997. The effectiveness revolution and public health. In:
Scally, G. (Ed.), Progress in Public Health. Royal Society of Medicine Press,
London, pp. 197–225.
Stewart, G.B., Pullin, A.S., Coles, C.F., 2007a. Poor evidence-base for assessment of
windfarm impacts on birds. Environmental Conservation 34, 1–11.
Stewart, G.B., Pullin, A.S., Tyler, C., 2007b. The effectiveness of asulam for bracken
(Pteridium aquilinum) control in the United Kingdom: a meta-analysis.
Environmental Management 40, 747–760.
Stewart, G.B., Bayliss, H.R., Showler, D.A., Sutherland, W.J., Pullin, A.S., in press.
Effectiveness of engineered and natural in-stream structure mitigation
measures for increasing salmonid abundance: a systematic review. Ecological
Applications.
Sutherland, W.J., Pullin, A.S., Dolman, P.M., Knight, T.M., 2004. The need for
evidence-based conservation. Trends in Ecology and Evolution 19, 305–
308.
Thompson, D.M., 2006. Did the pre-1980 use of in-stream structures improve
streams? A reanalysis of historical data. Ecological Applications 16, 784–
796.
Tyler, C., Pullin, A.S., Stewart, G.B., 2006. Effectiveness of management interventions
to control invasion by Rhododendron ponticum. Environmental Management 37,
513–522.