mily Sena, Gillian Currie, Hanna Vesterinen, Kieren Egan, Nicki Sherratt, Cristina Fonseca, Zsannet Bahor, J ing

Liao,
heo Hirst, Kim Wever, Hugo Pedder, Katerina Kyriacopoulou, J ulija Baginskaite, Ye Ru, Stelios Serghiou, Aaron
cLean, Catherine Dick, Tracey Woodruff, Patrice Sutton, Andrew Thomson, Aparna Polturu, Sarah MaCann, Gillian
ead, J oanna Wardlaw, Rustam Salman, J oseph Frantzias, Robin Grant, Paul Brennan, Ian Whittle, Andrew Rice,
osie Moreland, Nathalie Percie du Sert, Paul Garner, Lauralyn McIntyre, Gregers Wegener, Lindsay Thomson, David
owells, Ana Antonic, Tori OCollins, Uli Dirnagl, H Bart van der Worp, Philip Bath, Mharie McRae, Stuart Allan, Ian
arshall, Xenios Mildonis, Konstantinos Tsilidis, Orestis Panagiotou, J ohn Ioannidis, Peter Batchelor, David Howells,
anne J ansen of Lorkeers, Geoff Donnan, Peter Sandercock, Emily Sena, Gillian Currie, Hanna Vesterinen, Kieren
gan, Nicki Sherratt, Cristina Fonseca, Zsannet Bahor, J ing Liao, Theo Hirst, Kim Wever, Hugo Pedder, Katerina
yriacopoulou, J ulija Baginskaite, Ye Ru, Stelios Serghiou, Aaron McLean, Catherine Dick, Tracey Woodruff, Patrice
utton, Andrew Thomson, Aparna Polturu, Sarah MaCann, Gillian Mead, J oanna Wardlaw, RustamSalman, J oseph
antzias, Robin Grant, Paul Brennan, Ian Whittle, AndrewRice, Rosie Moreland, Nathalie Percie du Sert, Paul Garner,
auralyn McIntyre, Gregers Wegener, Lindsay Thomson, David Howells, Ana Antonic, Tori OCollins, Uli Dirnagl, H Bart
n der Worp, Philip Bath, Mharie McRae, Stuart Allan, Ian Marshall, Xenios Mildonis, Konstantinos Tsilidis, Orestis
anagiotou, J ohn Ioannidis, Peter Batchelor, David Howells, Sanne J ansen of Lorkeers, Geoff Donnan, Peter
andercock, Emily Sena, Gillian Currie, Hanna Vesterinen, Kieren Egan, Nicki Sherratt, Cristina Fonseca, Zsannet
ahor, J ing Liao, Theo Hirst, Kim Wever, Hugo Pedder, Katerina Kyriacopoulou, J ulija Baginskaite, Ye Ru, Stelios
erghiou, Aaron McLean, Catherine Dick, Tracey Woodruff, Patrice Sutton, Andrew Thomson, Aparna Polturu, Sarah
aCann, Gillian Mead, J oanna Wardlaw, RustamSalman, J oseph Frantzias, Robin Grant, Paul Brennan, Ian Whittle,
ndrew Rice, Rosie Moreland, Nathalie Percie du Sert, Paul Garner, Lauralyn McIntyre, Gregers Wegener, Lindsay
homson, David Howells, Ana Antonic, Tori OCollins, Uli Dirnagl, H Bart van der Worp, Philip Bath, Mharie McRae,
uart Allan, Ian Marshall, Xenios Mildonis, Konstantinos Tsilidis, Orestis Panagiotou, J ohn Ioannidis, Peter Batchelor,
avid Howells, Sanne J ansen of Lorkeers, Geoff Donnan, Peter Sandercock, Emily Sena, Gillian Currie, Hanna
esterinen, Kieren Egan, Nicki Sherratt, Cristina Fonseca, Zsannet Bahor, J ing Liao, Theo Hirst, Kim Wever, Hugo
edder, Katerina Kyriacopoulou, J ulija Baginskaite, Ye Ru, Stelios Serghiou, Aaron McLean, Catherine Dick, Tracey
oodruff, Patrice Sutton, Andrew Thomson, Aparna Polturu, Sarah MaCann, Gillian Mead, J oanna Wardlaw, Rustam
alman, J oseph Frantzias, Robin Grant, Paul Brennan, Ian Whittle, AndrewRice, Rosie Moreland, Nathalie Percie du
ert, Paul Garner, Lauralyn McIntyre, Gregers Wegener, Lindsay Thomson, David Howells, Ana Antonic, Tori OCollins,
i Dirnagl, H Bart van der Worp, Philip Bath, Mharie McRae, Stuart Allan, Ian Marshall, Xenios Mildonis, Konstantinos
ilidis, Orestis Panagiotou, J ohn Ioannidis, Peter Batchelor, David Howells, Sanne J ansen of Lorkeers, Geoff Donnan,
eter Sandercock, Emily Sena, Gillian Currie, Hanna Vesterinen, Kieren Egan, Nicki Sherratt, Cristina Fonseca,
sannet Bahor, J ing Liao, Theo Hirst, Kim Wever, Hugo Pedder, Katerina Kyriacopoulou, J ulija Baginskaite, Ye Ru,
elios Serghiou, Aaron McLean, Catherine Dick, Tracey Woodruff, Patrice Sutton, AndrewThomson, Aparna Polturu,
CAMARADES: Bringing evidence to translational medicine
Restoring faith in the research
enterprise a call to action
MalcolmMacleod
ProfessorofNeurologyandTranslationalNeurosciences
UniversityofEdinburgh
and
HonoraryConsultantNeurologist,NHSForthValley
CollaborativeApproachtoMetaAnalysisandReviewof
AnimalDatafromExperimentalStudies
CAMARADES: Bringing evidence to translational medicine
Faith
the substance of things hoped for, the
evidence of things not see
CAMARADES: Bringing evidence to translational medicine
Reproducibility a fourth R?
Identifiable
costs
Potential
benefits
Reduction
Replacement
Refinement
Resource
Value of
Information
CAMARADES: Bringing evidence to translational medicine
Value of information
Are our research questions important?
Do our finding describe what happens in
our highly refined model system?
Do those findings have general validity to
our target biological system?
Do we make our findings available, in
sufficient detail, for this information to be
useful to others?
CAMARADES: Bringing evidence to translational medicine
Control half dose full dose
I
n
f
a
r
c
t

V
o
l
u
m
e
0
50
100
150
200
250
300
10
-120
M 10
-60
M
CAMARADES: Bringing evidence to translational medicine
Winner of the 2012 Ignoble Prize for Neuroscience
CAMARADES: Bringing evidence to translational medicine
What happens when pharma tries to
replicate academic findings?
Bayer, Berlin
In-house target identification
and validation projects over 4
years
Womens health,
cardiovascular disease,
oncology
67 projects
Prinz et al, Nature
Reviews Drug Discovery,
2011
CAMARADES: Bringing evidence to translational medicine
CAMARADES: Bringing evidence to translational medicine
CAMARADES: Bringing evidence to translational medicine
Biomedical research:
Increasing value, reducing waste
Macleod et al, Lancet 2014
CAMARADES: Bringing evidence to translational medicine
Biomedical research:
Increasing value, reducing waste
Macleod et al, Lancet 2014
CAMARADES: Bringing evidence to translational medicine
What could be going wrong?
Randomisation and blinding
CAMARADES: Bringing evidence to translational medicine
You can usually find what youre
looking for
Group Day
1
Day
2
Day
3
Day
4
Day
5
Maze
bright
1.33 1.60 2.60 2.83 3.26
Maze
dull
0.72 1.10 2.23 1.83 1.83
+0.60 +0.50 +0.37 +1.00 +1.43
RosenthalandFode(1963),BehavSci8,1839
12graduatepsychologystudents
5dayexperiment:ratsinTmazewithdarkarmalternatingatrandom,andthe
darkarmalwaysreinforced
2groups MazeBrightandMazedull
CAMARADES: Bringing evidence to translational medicine
Blinding in disease models
Blinded assessment of behavioural outcome
No Yes
I
m
p
r
o
v
e
m
e
n
t

i
n

b
e
h
a
v
i
o
u
r
a
l

o
u
t
c
o
m
e

(
S
t
a
n
d
a
r
d
i
s
e
d

E
f
f
e
c
t

S
i
z
e
)
0.0
0.2
0.4
0.6
0.8
1.0
1.2
Multiple Sclerosis Parkinsons disease
Alzheimers disease Stroke
CAMARADES: Bringing evidence to translational medicine
Most studies are at risk of bias
Disease Randomisatio
n
(%)
Blinding
(%)
Power
calculation
(%)
Yes No Yes No Yes No
Stroke 37 63 37 63 3 97
Alzheimers 16 84 22 78 0 100
Parkinsons 16 84 15 85 0 100
Multiple
Sclerosis
9 91 16 84 0 100
Pain 12 88 26 74 0 100
Disease Randomisatio
n
(%)
Blinding
(%)
Power
calculation
(%)
Yes No Yes No Yes No
Stroke 37 63 37 63 3 97
CAMARADES: Bringing evidence to translational medicine
Across all research domains
Reportingofmeasurestoreducebiasin254reportsofinvivo,exvivoorin
vitrostudiesinvolvingnonhumananimals,identifiedfromrandomsampleof
2000publicationsfromPubMed
Ioannidis et al, 2014
CAMARADES: Bringing evidence to translational medicine
1173 publications using non
human animals, published in 2009
or 2010, from 5 leading UK
universities
an outstanding contribution to
the internationally excellent
position of the UK in biomedical
science and
clinical/translational research.
Across all institutions
Rand Blind I/E SSC
CAMARADES: Bringing evidence to translational medicine
Across the Impact Factors
Top 10% Bottom 10%
Top 10% Bottom 10%
Top 10% Bottom 10%
Top 10% Bottom 10%
CAMARADES: Bringing evidence to translational medicine
Biomedical research:
Increasing value, reducing waste
Macleod et al, Lancet 2014
CAMARADES: Bringing evidence to translational medicine
In vivo models of MS
72publicationswithnovalidoutcomes:58
reportingnoerror;7notreportingthenumber
ofanimalspergroup;2reportingnoerrororthe
numberpergroup;3withuninterpretable data;
2withnocontroldata
Pubmed search:
9726publications
1152publicationstesting
1717interventions
CAMARADES: Bringing evidence to translational medicine
Reporting research details
The Morris Water Maze in Alzheimers research
CAMARADES: Bringing evidence to translational medicine
Reporting research details
Reporting drop outs
minocycline
control
minocycline
control
CAMARADES: Bringing evidence to translational medicine
Biomedical research:
Increasing value, reducing waste
Macleod et al, Lancet 2014
CAMARADES: Bringing evidence to translational medicine
Publication bias
CAMARADES: Bringing evidence to translational medicine
Different patterns of publication
bias in different fields
outcome observed corrected
Disease
models
improvement 40% 30% Less
improvement
Toxicology
model
harm 0.32 0.56 More harm
outcome observed corrected
Disease
models
improvement 40% 30% Less
improvement
Benefit
Harm
CAMARADES: Bringing evidence to translational medicine
Excess significance in in vivo
studies
n Observed Expected
%
suppression
All 4445 1719 919 47%
AD 1054 418 86 79%
EAE 483 212 131 38%
Focal
ischemia
1403 626 370 60%
ICH 424 145 108 26%
PD 873 228 202 12%
SCI 208 90 216 76%
Implies that number of experiments actually done is 4445/919*1719, or 8314
or that even in published work only 53% of experiments are published
Tsilidas et al PLoS Biol 2013
n Observed Expected
%
suppression
All 4445 1719 919 47%
AD 1054 418 86 79%
EAE 483 212 131 38%
Focal
ischemia
1403 626 370 60%
ICH 424 145 108 26%
PD 873 228 202 12%
SCI 208 90 216 76%
n Observed Expected
%
suppression
All 4445 1719 919 47%
AD 1054 418 86 79%
EAE 483 212 131 38%
Focal
ischemia
1403 626 370 60%
ICH 424 145 108 26%
PD 873 228 202 12%
SCI 208 90 216 76%
CAMARADES: Bringing evidence to translational medicine
Biomedical research:
Increasing value, reducing waste
Macleod et al, Lancet 2014
CAMARADES: Bringing evidence to translational medicine
How big a hole does this make in
published research?
Publication Bias
Underpowered experiments
Risk of Bias
CAMARADES: Bringing evidence to translational medicine
How can we improve quality?
CAMARADES: Bringing evidence to translational medicine
If you are planning a systematic review or meta-
analysis of animal data, CAMARADES are here to
help: malcolm.macleod@ed.ac.uk