Introduction to Critical appraisals of the medical literature

Partini Pudjiastuti, Sudigdo Sastroasmoro

Department of Child Health Medical School University of Indonesia

Evidence based medicine 5 steps

Formulate question

Efficiently track down best available evidence

Critically review the validity and usefulness of the evidence

Evaluate performance Implement changes in clinical practice

Rationale
>25,000 journals worldwide >2 million published articles per year Many published articles have methodological (including statistical) flaws even in most respected journals Not all results can be applied due to many reasons, a.o. dissimilarities of study subjects with our patients Limited time of physicians: focus on articles relevant to your clinical practice

Errors and inappropriateness may occur in any part of medical research:

in choosing appropriate design in choosing population in selecting study subjects in the details of the design in intervention and measurements in use of statistical methods in analysis in interpreting statistical analysis and in writing research report

The effect of dietary habit on calcium level in pediatric patients with nephrotic syndrome J Agric Soc Sci 2006;2
r = - 0.1213, p = 0.015
There was a significant inverse relationship between the age and the total calcium level

IMPORTANT!!!
Statistical significance vs. clinical importance
Negligible clinical difference may be statistically very significant if the number of subjects >>>. e.g., difference in reduction of cholesterol level of 3 mg/dl, n1=n2 = 10,000; p = 0.00002 Large clinical difference may be statistically non- significant if the no of subjects <<<, e.g. 30% difference in cure rate, if n1 = n2 = 10, p

Clinical importance vs. statistical significan

Cholesterol level, mg/dl
Standard 300 treatment mg/dl n=10000 n=10000 New 300 treatment mg/dl 200

Clinical
197

t=

df = 9998

p = 0.00002 Statistical

Clinical importance vs. statistical significanc

Cured Standard Rx New Rx 0 3 Died 10 (100%) 7 (70%)

Absolute risk reduction = 30%

Fischer exact test: p = 0.211

Clinical Statistical

Critical appraisal (Making Reading More Worthwhile)

What is Critical Appraisal?
1. Critical appraisal = quality assessment 2. .process of weighing up evidence to see how useful it is in decision making 3. .a process of assessing the validity, reliability and usefulness of evidence 4. ..is about considering, evaluating and interpreting information in a systematic and objective way

Critically Appraise What You Read.

Separating the wheat from the chaff. Time is limited you should aim to quickly stop reading the dross. Others contain useful information mixed with rubbish.

Simple checklists enable the useful information to be identified.

Critical Appraisal Critical Thinking

Appraising the available evidence to construct clinical reasoning and to make decisions Finding strengths and limitations of written evidence

Deciding what evidence to pay attention to versus what to ignore

Why critically appraise?

Supports sound decision making based on best available evidence Helps us determine (three Rs):
How rigorous a piece of research is Validity

What the results are telling us Importance

How relevant it is to our patient Applicability

Morbidity Mortality QoL

Patient Satisfaction

Health Status

Value

Quality
=

Cost
Resources used

What is Evidence?
People disagree on what constitutes evidence Evidence generally = scientific fact Evidence - a combination of information obtained from 3 sources: research, clinical experience, and client preferences (Kitson, Harvey, & McCormack, 1998)

Why do we need evidence?

Resources should be allocated to things that are EFFECTIVE The only way of judging effectiveness is EVIDENCE In God we trust all others need evidence

Sources of Evidence
Primary sources Based on experiments and published research Secondary sources Systematic reviews Clinical guidelines Journals of secondary publication e.g. Evidence Based Medicine

Levels of evidence
1. Syst reviews of RCTs and high quality RCTs
2. Syst reviews of cohort studies, lower quality RCTs, outcomes research 3. Syst reviews of case controls, case control studies 4. Case Series 5. Expert opinion www.cebm.net

Types of Evidence
Question Types
Type of Question
Health care interventions: treatment, prevention Harm or Etiology Prognosis

Best Evidence
Quantitative: Systematic Review of RCTs or RCT Quantitative: Observational Study Cohort or Case Control Quantitative: Observational Study Cohort, Case Control

Diagnosis or Assessment
Economics Meaning

Quantitative: Comparison to Gold Standard

Quantitative: Cost-effectiveness Study Qualitative: case study, ethnography, grounded theory, phenomenologic approach

Key quality parameters

Validity

Reliability
Importance

Validity
Internal Is the study designed in such a way that I Can trust the findings?
External Is the study designed in such a way that I Can generalize the findings?

Reliability
If the study was conducted again, would the results be the same?
Usually interpreted as the accuracy of measurement.

Importance
What was the effect size or magnitude of effect?

Clinical vs. statistical significance.

Tools for Critical Appraisal

EBM simplified approach:

Are the results valid?

What are the results? Will the results help me in patient care?

V
I A

Format of research reports

Title Authors and Institutions Abstract & keywords IMRAD
Introduction Methods Results Discussion - why did I start? - what did I do? - what did I find? - what does it mean?

References

Check list for medical literature

1. 2. 3. 4. 5. Title Authors & Institutions Abstract: Structured? Informative? Abbreviation? Introduction: Length? Relevant ref? Objective? Methods:
Design, time and place Inclusion criteria Exclusion criteria Sample size, sampling method Randomization technique Intervention: masking? Outcome measurement: blinding? Primary outcome: type of variable Secondary outcome: type of variable Analysis: Clinical, statistical

Check list for medical literature

6. Results
Baseline characteristics Main outcome Secondary outcome

7. Discussion
General Strength and weakness Conclusions

8. References
Vancouver style Constant

9. Acknowledgments 10. Ethics approval 11. Conflict of interest

What to assess? (in study of cause-effect relationship)

A. General description
Type of design Target population, source population, sample Sampling method Dependent and independent variables Main results?

What to assess? (in study of cause-effect relationship)

B. Internal validity, non-causal relationship Influence of bias Influence of chance Influence of confounders

Bias
What is a bias? A process that tends to produce
results that depart systematically from the true values existing in the study population

Types of bias
1. Sample (subject selection) biases, which may result in the subjects in the sample being unrepresentative of the population which you are interested in 2. Measurement (detection) biases, which include issues related to how the outcome of interest was measured 3. Intervention (performance) biases, which involve how the treatment itself was carried out.

What to assess? (in study of cause-effect relationship)

C. Internal validity, causal relationship
Temporality (cause precedes effect) Strength of association (large difference, RR, OR, etc) or small p value or narrow confidence interval Biological gradient (dose dependence) Consistency among studies (diff. populations or designs) Specificity (certain factor results in certain effect) Coherence (does not conflict with current knowledge) Biological plausibility: can be explained with current knowledge (at least in part)

What to assess? (in study of cause-effect relationship)

D. External validity
Applicable to study subjects Applicable to source population Applicable to target population

11 items, each with 3 sections

1. Can you find this information in the paper?

2. Is the way this was done a problem?

3. Does this problem threaten the validity of the study?

11 items
1. What is the research question? 2. What is the study type? 3. What are the outcome factors and how are they measured? 4. What are the study factors and how are they measured? 5. What important confounders are considered? 6. What are the sampling frame and sampling method? 7. In an experimental study, how were the subjects assigned to groups? In a longitudinal study, how many reached final follow-up? In a case control study, are the controls appropriate? (Etc) 8. Are statistical tests considered? 9. Are the results clinically/socially significant? 10. Is the study ethical? 11. What conclusions did the authors reach about the question?

1. What is the research question?

(Is the way this was done a problem?)

Is it concerned with the impact of an intervention, causality or determining the magnitude of a health problem?
(Does this problem threaten the validity of the study?)

Is it a well stated research question/hypothesis?

2. What is the study type?

(Is the way this was done a problem?)

Is the study type appropriate to the research question?

(Does this problem threaten the validity of the study?)

If not, how useful are the results produced by this type of study?

3. What are the outcome factors and how are they measured?
(Is the way this was done a problem?) a) are all relevant outcomes assessed b) is there measurement error? (Does this problem threaten the validity of the study?) a) how important are omitted outcomes b) is measurement error an important source of bias?

4. What are the study factors and how are the measured?
(Is the way this was done a problem?) Is there measurement error? (Does this problem threaten the validity of the study?) Is measurement error an important source of bias?

5. What important potential confounders are considered?

(Is the way this was done a problem?) Are potential confounders examined and controlled for? (Does this problem threaten the validity of the study?) Is confounding an important source of bias?

6. What are the sampling frame and sampling method?

(Is the way this was done a problem?) Is there selection bias? (Does this problem threaten the validity of the study?) Does this threaten the external validity of the study?

7. Questions of internal validity

(Is the way this was done a problem?) In an experimental study, how were the subjects assigned to groups? In a longitudinal study, how many reached follow-up? In a case control study, are the controls appropriate?
of the study?

8. Are statistical tests considered?

(Is the way this was done a problem?) Were the tests appropriate for the data? Are confidence intervals given? Is the power given if a null result? In a trial, are results presented as absolute risk reduction as well as relative risk reduction? How useful are the results?

9. Are the results clinically/socially significant?

(Is the way this was done a problem?) Was the sample size adequate to detect a clinically/socially significant result? Are the results presented in a way to help in health policy decisions? (Does this problem threaten the validity of the study?) Is the study useful?

10. Are ethical issues considered?

(Is the way this was done a problem?) Does the paper indicate ethics approval? Can you identify potential ethical issues?
(Does this problem threaten the validity of the study?)

11. What conclusions did the authors reach about the study question?
(Is the way this was done a problem?) Do the results apply to the population in which you are interested? (Does this problem threaten the validity of the study?) Will you use the results of the study?

Appraisal Tools
Tools from the Critical Appraisal Skills Programme (CASP) Systematic Reviews Randomised Controlled Trials Qualitative Research Studies Cohort Studies Case-Control Studies Diagnostic Test Studies Economic Evaluation Studies Available

Study Designs Recap

Effectiveness of Therapy Risk Factors / Prognosis Randomised Controlled Trial

Cohort Study

Diagnosis Attitudes & Beliefs

Survey using Gold Standard

Qualitative (Interviews, Observations, etc)

Critical appraisal

- Valid - Important - Applicable

Methods Results Discussion

Sekuens dan hubungan subyek-predikat

50 massa pemuda dan mahasiswa berunjuk rasa memprotes pelecehan seksual yang makin marak di depan Mabes Polri Menyadari pentingnya Panduan Pelayanan Medis (PPM), dibentuklah Panitia Penyusunan SPM di RSCM

 Dalam pertemuan ilmiah yang diselenggarakan setiap tahun yang merupakan ajang untuk menyajikan perkembangan mutakhir dalam bidang ilmu penyakit dalam di tanah air tersebut menyimpulkan bahwa pertemuan tersebut disamping dilakukan oleh Universitas selayaknya juga dilakukan oleh cabang-cabang PAPDI di setiap propinsi, bahkan kalau mungkin disetiap kabupaten

Plagiarisme adalah tindakan yang dapat diartikan sebagai pencurian ide atau hasil pemikiran dan tulisan orang lain yang digunakan dalam tulisan seolah-olah ide atau tulisan orang lain tersebut adalah ide atau hasil tulisannya sendiri untuk keuntungannya sendiri sehingga merugikan orang lain baik materiil maupun non-materiil, atau plagiarisme dapat berupa pencurian sebuah kata, frase, kalimat, atau alinea, atau bahkan pencurian suatu bab dari sebuah tulisan atau buku seseorang, tanpa menyebut sumber yang dicuri. (Draft SK Rektor UI)

RCT (Pragmatic trials): Validity

Were the study participants randomized? Was the randomization technique described? Was the randomization table concealed? Were the characteristics of the subjects similar at the start of the intervention? Were all participants given equal treatment apart from the intervention? Were all relevant outcomes considered? Were the results analyzed correctly?

RCT (Pragmatic trials): Importance

Calculate: EER, CER, RRR, ARR, NNT

Success Failure

a c

b d

EER = a/(a+b) CER = c/(c+d)

RRR = (CER-EER)/CER ARR = CER-EER NNT = 1/ARR

RCT: Applicability
Were the participations similar to your patients? May be intuitively concluded or use f (factor indicating how much severe your patient compared to the study participation in terms of prognostic factor)

DIAGNOSTIC TEST: Validity

Was independent and blind comparison to gold standard applied?

Was the diagnostic test include spectrum of disease similar to your real practice?
Was the gold standard applied regardless of the diagnostic result?

Diagnostic Test: Importance

Calculate: Sensitivity, specificity, predicitive values, likelihood ratios +

+

a b

Test
Se = a/(a+c) Sp = d/(b+d) PPV = a/(a+b) NPV = d/(c+d)

d LR+ = se/(1-sp) LR - = (1-se) /sp Posttest odds = Pretest odds x LR+

Diagnostic Test: Applicability

Were the participations similar to your patients? Is the diagnostic test applicable, acceptable, and affordable in your setting? Will the result of the test help your patient?

Cohort Studies: Validity

Was the inception cohort assembled in usual point of course of the disease? Was the follow-up sufficient & complete? Were outcome criteria applied in blind fashion? Was there any validation in other group of patients? Was subgroup analysis performed after adjustment for prognostic factors?

Cohort studies: Importance

Calculate: Relative risk (RR) and 95% CI
+
+ a b

Exposure
c d

RR = a/(a+b) : c/(c+d)

RR > 1: risk factor RR < 1 : protective factor RR = 1 : not a risk factor

Cohort studies: Applicability

Were the patients similar to yours? Wil the evidence make a clinically important impact for the care of your patient?

Case control study: Validity

Was the selection of patients represent the common patients of yours? (Spectrum of the patients) Was the selection of control group sensible and reasonable? Have sufficient attempts been made to minimize recall bias?

Case control studies: Importance

Calculate: Odds ratio (OR) and 95%CI +

+ a b

Exposure
c d

OR = a/b : c/d

OR > 1 : risk factor OR < 1 : protective factor OR = 1 : not a risk factor

Case-control studies: Applicability

Were the patients similar to yours? Wil the evidence make a clinically important impact for overall care (diagnosis, treatment, prognosis) of your future patients?b

From Data to Wisdom

Data are what researchers collect Information results when data is analyzed and interpreted (EVIDENCE) Knowledge results when information is shared, acquired, and used Wisdom is the ability to make the right use of knowledge

THANKS