Documenti di Didattica
Documenti di Professioni
Documenti di Cultura
1017/S1368980007001334
Submitted 22 November 2006: Accepted 9 October 2007: First published online 6 December 2007
Abstract
Objective: Despite the increasing dependence on systematic reviews to summarise
the literature and to issue public health recommendations, the formal assessment
of the reliability of conclusions emerging from systematic reviews has received
little attention. The main goal of the present study was to evaluate whether two
independent centres, in two continents, draw similar conclusions regarding the
association of food, nutrition and physical activity and endometrial cancer, when
provided with the same general instructions and with similar resources.
Design: The assessment of reproducibility concentrated on four main areas:
(1) paper search and selection; (2) assignment of study design; (3) inclusion
of ‘key’ papers; and (4) individual studies selected for meta-analysis and the
summary risk estimate obtained.
Results: In total 310 relevant papers were identified, 166 (54 %) were included by
both centres. Of the remaining 144 papers, 72 (50 %) were retrieved in the
searches of one centre and not the other (54 in centre A, 18 in centre B) and
72 were retrieved in both searches but regarded as relevant by only one of the
centres (52 in centre A, 20 in centre B). Of papers included by both centres, 80 %
were allocated the same study design. Agreement for inclusion of cohort-type and
case–control studies was about 63 % compared with 50 % or less for ecological
Keywords
and case series studies. The agreement for inclusion of 138 ‘key’ papers was 87 %. Systematic literature review
Summary risk estimates from meta-analyses were similar. Endometrial carcinoma
Conclusions: Transparency of process and explicit detailed procedures are Diet
necessary parts of a systematic review and crucial for the reader to interpret Epidemiological studies
its findings. Reproducibility
Among non-communicable diseases cancer is the second years 1966 to 1975 to 9820 for 1996 to 2005 (using the
leading cause of death, estimated as responsible for 76 terms ‘cancer’ and ‘diet’ in PubMed).
million deaths in 2005, second to cardiovascular disease The World Cancer Research Fund and the American
(175 million deaths)(1). The incidence of cancer can Institute for Cancer Research (WCRF/AICR) have pub-
be reduced by 30–40 % by dietary and other lifestyle lished a second report on Food, Nutrition, Physical
changes(2). The published literature on diet and cancer Activity and the Prevention of Cancer: A Global Per-
has increased almost tenfold from 1168 articles for the spective(3), based on cancer site-specific systematic lit-
erature reviews (SLRs), to explore causal dietary, physical
y Correspondence address: British Nutrition Foundation, High Holborn
activity or nutritional links with cancer. These were car-
House, 52–54 High Holborn, London WC1 V 6RQ, UK. ried out in independent academic institutions in Europe
Number of records
9695 downloaded from
4459 6547
total* databases and
retrieved by hand
searching (duplicate
records removed)
Included (relevant)
310 epidemiological
272 204
total* studies after reading
full papers
106 38
166
relevant to centre A relevant to centre B
common
but not relevant to but not relevant to
relevant
centre B centre A
papers
54 52 18 20
in centre A’s in both centres’ in centre B’s in both centres’
search only searches but search only searches but
regarded as relevant regarded as relevant
by centre A only by centre B only
Fig. 1 Flow diagram of the number of papers found and included by each centre; *total number of unique records from both centres
Table 1 Sources of papers included as relevant by one centre, but Agreement for ‘key’ papers
not retrieved in the search of the other centre Figure 2 shows that a total number of 138 papers were
In centre A’s In centre B’s identified by either centre as ‘key’ and 120 (87 %) were
Source search only search only included by both centres.
Medline 24 8
Embase 2 4 Comparison of risk estimates
Web of Science 3 2 The centres were responsible for deciding when it was
LILACS 1 0
Pascal 2 N/A appropriate to carry out meta-analyses. The criteria used
Pre-Medline 4 0 by the centres were similar. Both included studies if risk
Old Medline 6 N/A estimates (dichotomous or quantiles) were reported with
Hand searching 19 4
Total 54 18 95 % confidence intervals. Centre A also included con-
tinuous risk estimates. Both centres used the comparison
N/A, did not search database.
of extreme categories method for meta-analysis, which
uses a risk estimate for the highest versus lowest quantile
cohort-type and case–control studies the agreement was of dietary exposure. Summary estimates for each centre
more than 60 %, whereas for case series studies it was using a random effects model for several exposures
only 14 %. The disagreement in case series classification linked to endometrial cancer in the first WCRF/AICR
was because one of the centres classified baseline data report(2) were compared. Centre A carried out study
in cohort studies of endometrial cancer survivorship as design-specific meta-analysis (with the exception of BMI
case series. where separate and combined analyses were undertaken).
1010 RL Thompson et al.
Table 2 Comparison of inclusion of papers as relevant between centres by study design
Total no. of No. (%) included as No. included as relevant by No. included as relevant by
Study design included papers relevant by both centres centre A, but not centre B centre B but not centre A
*Cohort-type includes prospective cohort, case-cohort, retrospective cohort and nested case–control studies.
13 5
120
relevant to centre A relevant to centre B
138 + common ‘key’
= but not relevant to + but not relevant to
total relevant
centre B centre A
papers
11 2 4 1
in centre A’s in both centres’ in centre B’s in both centres’
search only searches but search only searches but
regarded as regarded as
relevant by relevant by
centre A only centre B only
Fig. 2 Flow diagram of the papers considered as ‘key’ by one centre but not included by the other center
Due to the small number of cohort studies (no more than lower summary estimate and greater heterogeneity. On
two) identified for all exposures assessed apart from BMI, further evaluation, the discrepancy was related to centre
centre A performed analyses only for case–control stu- B including studies that reported animal fat as both a food
dies. Centre B combined case–control and cohort-type group and a nutrient, whereas centre A included
studies in the same analysis. Meta-analysis was not carried only animal fat as a food group. For BMI, both pooled
out for saturated fatty acids by centre A. estimates included cohort-type and case–control studies
Table 3 shows similar numbers of studies reporting risk and the summary risk estimates were close to 30 for
estimates in the direction of increased or decreased risk. both centres.
Studies included by one centre only were further eval- Although the estimates for non-starchy vegetables were
uated. Studies that did not report results as odds ratios or almost identical, significant heterogeneity was detected
relative risks were included as relevant by both centres, by the centre B analysis, but not by centre A (Table 3).
however; only one centre included these studies in Centre B included studies reporting results for green
their report tables. Eleven studies were not picked up vegetables in the meta-analysis, whereas centre A
in the searches of both centres. Where a different risk restricted analysis to total vegetables. This resulted in
estimate was used, this was due to different exposure centre B including two studies with the lowest and
definitions being used (e.g. total vegetables and cooked highest effect sizes, which increased the heterogeneity.
vegetables) or a different analysis model; for example,
one centre may have chosen an age-adjusted risk esti-
mate and the other chose the most adjusted risk Discussion
estimate. Similar numbers of studies were included in the
meta-analyses. As an initial task for the WCRF/AICR second report on
The summary estimates from both centres were very Food, Nutrition, Physical Activity and the Prevention of
similar with the exception of animal fat (Fig. 3). For Cancer: A Global Perspective(3), we conducted an
animal fat, a greater summary odds ratio (185 vs. 137) assessment of the reproducibility of conclusions from
was reported in the centre A analysis. The centre B ana- systematic reviews of epidemiological literature, using
lysis included two cohort studies in addition to the same the example of diet, nutrition, physical activity and
four case–control studies as centre A, which led to the endometrial cancer. Our findings suggested that while
Reproducibility of systematic literature reviews 1011
Summary odds ratio
No CI (17)
No CI (3)
No CI (1)
No CI (2)
No CI (1)
No CI (2)
No CI (1)
No CI (1)
No CI (8)
A
Fruit
B
heterogeneity
P value for
WCRF, World Cancer Research Fund; AICR, American Institute for Cancer Research; cc, case–control; CI, confidence interval; Cont. estimate, continuous estimate; NP, not performed.
,00001
0001
,0001
,0001
,0001
,0001
015
007
005
A
Non-starchy
vegetables
highest vs. lowest
B
category
A
Animal fats
Table 3 Results for exposures linked to endometrial cancer in the first WCRF/AICR report (pooled estimates are random effects)
(no. of cc studies)
No. of studies
meta-analysis
included in
B
36 (29)
32 (24)
9 (9)
8 (8)
7 (7)
8 (8)
4 (4)
6 (4)
4 (3)
NP
A
Body mass
the same study was used
No. of studies in which a
index
different estimate from
B
1 cohort
None
None
5 cc
4 cc
2 cc
2 cohort
6 cohort
None
23 cc
4 cc
1 cc
1 cc
5 m; 1 unclear
43 m; 1 unclear
6 k; 2 m; 1 5
7 k; 3 m; 1 5
1 k; 5 m
1 k; 5 m
35 m
11 m
1k
1k
9m
11 cohort
2 cohort
1 cohort
1 cohort
9 cohort
12 cc
11 cc
11 cc
35 cc
44 cc
9 cc
6 cc
5 cc
6 cc
6 cc
A
B
A
B
Saturated fat
Non-starchy
Animal fats