The EBM Cycle

1.Asking the Question 2.Searching the Literature 3.Appraising the Evidence 4.Integrating the Idea 5.Making a Decision

Components Of Critical Appraisal

Is the study valid?
VALIDITY / LACK OF BIAS

What are the results?

SIZE & PRECISION EFFECT

Will the results help me in caring for my patients?

APPLICABILITY & IMPACT

Validity vs Precision

Valid & Precise

Valid but NOT Precise

NOT Valid & NOT Precise

NOT Valid but Precise

Users Guides - Example

VALIDITY
Was there an independent blind comparison with a reference standard? Did the patient sample include an appropriate spectrum of patients to whom the diagnostic test will be applied in clinical practice? Did the results of the test being evaluated inuence the decision to perform the reference standard? Were the methods for performing the test described in sufcient detail to permit replication?

RESULTS
Are likelihood ratios for the test results presented or data necessary for their calculation provided?

APPLICABILITY
Will the reproducibility of the test result and its interpretation be satisfactory in my setting? Are the results applicable to my patient? Will the results change my management? Will patients be better off as a result of the test?

Other Available Users Guides

Article on PROGNOSIS Article on HARM Overview / Meta-Analysis Clinical Practice Guidelines Economic Analysis Clinical Decision Rule

Can be downloaded from websites: EBM Homepage JAMA

Critical Appraisal:
Articles on Therapy / Prevention
Glenn D. Marias, MD, DPBA Clinical Epidemiology

Are The Results Of The Study VALID?

1. Was the assignment of patients to treatments randomized? 2. Were all patients who entered the trial properly accounted for and attributed at its conclusion? 3. Were patients analyzed in the groups to which they were randomized? 4. Were patients, health workers, and study personnel blind to treatment? 5. Were the groups similar at the start of the trial? 6. Aside from the experimental intervention, were the groups treated equally?

What Were The RESULTS?

7.How large was the treatment effect? 8.How precise was the estimate of the treatment effect?

Will The Results Help Me In Caring For My Patient?

9.Can the results be applied to my patient care? 10.Were all clinically important outcomes considered? 11.Are the likely treatment benets worth the potential harms and costs?

First Step...
Is the research study an article on THERAPY / PREVENTION?

Article On Therapy/Prevention
Outcome (+) Disease Treatment (-) Disease Intervention

THERAPY: Evaluation of intervention/s used for curing disease / reducing S/Sx PREVENTION: Evaluation of intervention/s used for reducing risk of disease or its complications

Validity Appraisal - Therapy

Experimental group Subjects Control group Follow-up Outcome

Randomized

Randomization
Random allocation Each subject is given an equal chance of being assigned to either treatment or control group
It removes investigator bias in the allocation of subjects It produces study groups comparable with respect to known as well as unknown variables

Article on Therapy / Prevention

Experimental group (T+) Cure rate (%)

Similar in all aspects except for Tx status

Control group (T-)

Cure rate (%)

Exceptions To Randomization
Randomization is NOT possible when: No other known options for treatment Treatment of few subjects reverses uniform adverse outcomes

Ethical issues

Critical Appraisal - Therapy

1. Was the assignment of patients to treatments randomized? Answer should be YES.

Validity Appraisal - Therapy

Experimental group Subjects Control group Follow-up Outcome

Complete follow-up

Incomplete Follow-up
Reasons for losses to follow-up / dropouts:
Too Ill Already cured Migrated

Solution:
Assume WORST CASE SCENARIO Worst outcome happened to drop-outs

Example - WCS
100 subjects
4 deaths 16 lost to follow-up death rate = 4/84 = 4.8%

Worst Case Scenario

Assume the 16 lost to follow-up died death rate = 4 + 16 = 20 = 20%
84 + 16 100

When Is Drop-out Rate Signicant?

Drop-out rate > Differences in outcome rates between the 2 groups 5 and 20 rule of thumb
Fewer than 5% loss probably leads to little bias while greater than 20% loss threatens validity!

Example- Effect Of Drop-outs

Group A- 100 subjects 4 deaths Death rate= 4/100= 4% Group B- 100 subjects 7 deaths Death rate= 7/100= 7%

Death rate A < Death rate B

Group A- 100 subjects 4 deaths 50 lost to ffup (50% DO) Death rate= 4/50= 8% Group B- 100 subjects 7 deaths 2 lost to ffup (2% DO) Death rate= 7/98= 7%

Death rate A > Death rate B

Critical Appraisal - Therapy

2. Was follow-up complete? Answer should be YES.

Validity Appraisal - Therapy

Experimental group Subjects Control group Follow-up Outcome

ITT

ITT
Intent-to-treat analysis Always analyze once randomized!
DO NOT EXCLUDE FROM DATA ANALYSIS ANY SUBJECTS ONCE RANDOMIZED. Even those who did not receive assigned treatment Reasons for noncompliance is often related to prognosis REQUIREMENT FOR ITT: Outcomes of noncompliers should be known

Purpose: To preserve randomization

Example of ITT
Disease rate (compliers) 100/1000 (10%) 45/900 (5%) Disease rate (noncompliers) Disease rate Total 100/1000 (10%) 135/1000 (13.5)

Tx

1000

0/0

1000

90/100 (90%)

Example of ITT
Disease X Randomization Surgical Tx ASA Risk 3 excluded Medical Tx ASA Risk 3 included

Falsely better outcome for Surgical Tx

Critical Appraisal - Therapy

3. Were patients analyzed in the groups to which they were randomized? Answer should be YES.

Validity Appraisal - Therapy

Experimental group Subjects Control group Follow-up Outcome

Blinding

Blinding
A process wherein the researcher makes sure that the subject, investigator, &/or data collector does not know of the subjects assigned group Effect of Non-blinding:
Favor treatment group over control Overestimate the true effect

Blinding
Placebo
Indistinguishable from active treatment in appearance, taste and texture but lacking the active ingredient Automatically results in double blinding Considered BEST way of blinding

When blinding is NOT possible

Use OBJECTIVE outcome measures Survival rate, Death rates

Critical Appraisal - Therapy

4. Were patients, their clinicians, and study personnel blind to treatment? Answer should be YES.

Validity Appraisal - Therapy

Experimental group Subjects Control group Follow-up Outcome

Groups similar

Critical Appraisal - Therapy

5. Were the groups similar at the start of the trial? Answer should be YES.
Countercheck randomization Indirectly determines if sample size was adequate

If answer is NO:
Statistical analysis to adjust for differences Stratied analysis, regression

Effect of Unequal Groups at Baseline

GRP A w/ more of good prognostic factor GRP B w/ less of good prognostic factor

Better outcome

Worse outcome

Overestimate of true effect (in Grp A)

Validity Appraisal - Therapy

Experimental group Subjects Control group Follow-up Outcome

Groups treated equally

Critical Appraisal - Therapy

6. Aside from the experimental intervention, were the groups treated equally? Answer should be YES.
Countercheck to BLINDING Ensures that outcomes are actually due to the experimental intervention

Cointerventions
Treatment modalities applied to the subjects other than the treatment being studied Example
Group A: MRM Group B: CBS + Chemotherapy

COINTERVENTION

Cointerventions
GRP A: MRM Less subjects w/ chemotx GRP B: CBS More subjects w/ chemotx

Worse outcome

Better outcome

Overestimate of true effect (in Grp B)

Criteria for Validity

Blinding
Experimental group Subjects Control group Follow-up Outcome

Groups treated equally

Randomized (Groups similar)

Complete follow-up

ITT

Is The Study Valid?

Absolutely YES
Yes to all questions

Somewhat YES
Yes to rst 2 questions

NO
No to any of the rst 2 questions Article may not be worth using at all

Remember...
An ounce of critical appraisal is worth pounds of futile reading. - Isaac David Ampil II, MD

How Large Was The Treatment Effect?

Treatment T+ Compare with Non-treatment TOutcome O+

Outcome O+

Determining Risks
O+ T+ Ta c Ob d

Risk in T+ (Exptal grp)= a/a+b

Risk of developing the outcome given the exptal tx

Risk in T- (Control grp)= c/c+d

Risk of developing the outcome if exptal tx is not given

Comparing The Risks Between Groups

O+ T+ Ta c Ob d

Relative Risk (RR)

/ a + b (risk of dev. outcome given the tx) a c / c + d (risk of dev. outcome if tx is not given)

Comparing The Risks Between Groups

O+ T+ Ta c Ob d

Relative Risk Reduction (RRR) Relative Risk Increase (RRI)

(1 - RR) x 100% Complement of RR expressed in % Easier to interpret clinically than RR

Comparing The Risks Between Groups

SSI+ Ab+ AbTotal 10 20 30 SSI90 80 170 Total 100 100 200

Risk of SSI w/ Ab= 10/100= 0.1 Risk of SSI w/o Ab= 20/100= 0.2 RR= 0.1 / 0.2= 0.5 RRR= (1 - 0.5) x 100%= 50%

Comparing The Risks Between Groups

O+ T+ Ta c Ob d

Risk Difference Absolute Risk Reduction (ARR) Absolute Risk Increase (ARI)
(c / c + d) - (a / a + b)
(risk of dev outcome w/o tx) - (risk of dev outcome w/ tx)

Comparing The Risks Between Groups

O+ T+ Ta c Ob d

Number Needed to Treat (NNT)

1 / ARR Reciprocal of ARR Easier to interpret clinically than ARR

Comparing The Risks Between Groups

SSI+ Ab+ AbTotal 10 20 30 SSI90 80 170 Total 100 100 200

Risk of SSI w/ Ab= 10/100= 0.1 Risk of SSI w/o Ab= 20/100= 0.2 ARR= 0.2 - 0.1= 0.1 NNT= 1 / 0.1= 10

Interpreting The Comparisons

If outcome is CURE: RR= Outcome in exptal grp / Outcome in control grp If numerator > denominator, RR>1, tx is benecial If numerator < denominator, RR<1, tx is harmful If numerator = denominator, RR=1, tx is useless ARR= Outcome in control grp - Outcome in exptal grp If ARR > 0, tx is harmful If ARR < 0, tx is benecial If ARR = 0, tx has no effect

Interpreting The Comparisons

If outcome is DISEASE: RR= Outcome in exptal grp / Outcome in control grp If numerator > denominator, RR>1, tx is harmful If numerator < denominator, RR<1, tx is benecial If numerator = denominator, RR=1, tx is useless ARR= Outcome in control grp - Outcome in exptal grp If ARR > 0, tx is benecial If ARR < 0, tx is harmful If ARR = 0, tx has no effect

Summary Of Results
SSI+ Ab+ AbTotal 10 20 30 SSI90 80 170 Total 100 100 200

Risk of SSI w/ Ab= 10/100= 0.1 Risk of SSI w/o Ab= 20/100= 0.2 RR= 0.1 / 0.2= 0.5 RRR= (1 - 0.5) x 100%= 50% ARR= 0.2 - 0.1= 0.1 NNT= 1 / 0.1= 10

How Precise Was The Estimate Of Treatment Effect?

Concept of Condence Interval
Point estimate vs Interval estimate, eg: Estimating your age by point estimate Mean age= 48 years old Estimating your age based on interval estimate 95% CI: 32 - 60 years old 70% CI: 45 - 55 years old 99% CI: 15 - 65 years old

Interval estimate is more PRECISE than point estimate.

How Precise Was The Estimate Of Treatment Effect?

Condence Interval
Probability that the difference lies between a range of values at a given level of condence Gives an idea of the precision of the estimate The wider the interval, the less precise but the more condent you are of the estimate
95% CI: 32 - 60 y/o 70% CI: 45 - 55 y/o (more precise, less condent) 99% CI: 15 - 65 y/o (less precise, more condent)

Using CI To Determine Statistical Signicance

RR= Outcome in exptal grp / Outcome in control grp If numerator > denominator, RR>1, tx is benecial If numerator < denominator, RR<1, tx is harmful If numerator = denominator, RR=1, tx is useless

RR= 2.12 (95% CI: 0.87 - 4.86) 2.12 means tx is 2x more benecial than control 95% CI of 0.87 - 4.86 means the RR could be: <1: harmful >1: benecial The result is thus NOT STATISTICALLY SIGNIFICANT.

Using CI To Determine Statistical Signicance

RR= Outcome in exptal grp / Outcome in control grp If numerator > denominator, RR>1, tx is benecial If numerator < denominator, RR<1, tx is harmful If numerator = denominator, RR=1, tx is useless

RR= 2.12 (95% CI: 1.98 - 4.86) 2.12 means tx is 2x more benecial than control 95% CI of 1.98 - 4.86 means the RR is always benecial The result is thus STATISTICALLY SIGNIFICANT.

Using CI To Determine Statistical Signicance

ARR= Outcome in control grp - Outcome in exptal grp If ARR > 0, tx is benecial If ARR < 0, tx is harmful If ARR = 0, tx has no effect ARR= 3.12 (95% CI: -0.78 - 4.98) The ARR could be: <0: harmful >0: benecial The result is thus NOT STATISTICALLY SIGNIFICANT.

Using CI To Determine Statistical Signicance

Treatment
Surgery alone Surgery + postop chemo Surgery + postop chemo/RT Preop chemo + surgery +postop chemo

5 yr survival RR (95% CI) 0.5 (0.1 - 0.9) 2.3 (0.8 - 4.1) 3.4 (0.92 - 5.1) 14.2 (1.2 - 30.2)

Preop chemo/RT + surgery +postop chemo 32.1 (28.2 - 34.4)

Using CI To Determine Statistical Signicance

For Ratio measures (RR, OR):

Difference is not sifnicant if the interval crosses 1

For Difference measures (ARR):

Difference is not signicant if the interval crosses 0

Applicability Criteria
9. Can the results be applied to my patient care? Answer should be YES.
Patient must meet inclusion criteria and none of the exclusion criteria

Applicability Criteria
10.Were all clinically important outcomes considered? Answer should be YES.
Clinical outcomes Quality: morbidity, QOL Quantity: survival, mortality Mechanistic outcomes

vs.

Biochemical, anatomic, physiologic Serum PSA ECG ST elevation

Are The Likely Treatment Benets Worth The Potential Harm And Costs?

Harm & Cost Tx benefit NNT NNH Cost/life saved

Are The Likely Treatment Benets Worth The Potential Harm And Costs?

NNT= 1 / ARR
You need to treat # pxs to produce 1 benet Low NNT: more benecial

NNH= NNT x %side effects (in decimal)

You need to harm # pxs to produce 1 benet Low NNH: more harmful

Cost/life saved= (unit cost x #doses x duration) x NNT

You need to spend # cost to produce 1 benet

Are The Likely Treatment Benets Worth The Potential Harm And Costs?
Cost-risk benet
Need to treat __ persons to save 1 life at a cost of __ and produce __ adverse effects. Save __ lives at a cost of __ and produce __ adverse effects

Example:
NNT= 14 NNH= 7 Cost/benet= P100

You need to treat 14 persons to save 1 life at a cost of P100 but you produce 2 adverse effects

Sample C. A. T. For Treatment

Title: Search Method: Source: P: Sample here I: C: O: M: Sample here Sample here Sample here Sample here Rt Rc RR RRR Randomized? Adequate follow-up? Intention-to-treat? Blinding? Baseline traits similar? Treatment the same? ARR NNT Yes or No Yes or No Yes or No Yes or No Yes or No Yes or No

Endpoint Outcome1 Outcome2

Authors conclusion: Reviewers conclusion: Reviewer: Juan dela Cruz Date: Feb 21, 2008

CAT - Treatment
Scenario: 60 y/o M w/ T2N0M0 SCC, middle esophagus, for curative resection. Research Question: Among ptxs w/ resectable SCC of middle esophagus, will the addition of cervical & sup. mediastinal lymphadenectomy increase survival? Citation: A prospective randomized trial of extended cervical & superior mediastinal lymphadenectomy for carcinoma of the thoracic esophagus. Nishihira T, Hirayama K, Mori S. Am J Surg Vol 175; Jan 1998: 47- 51. P- ptxs < 70 yrs., good risk, w/ resectable SCC of thoracic esophagus I- Subtotal esophagectomy w/ 3-field lymphadenectomy C- Subtotal esophagectomy w/ 2-field lymphadenectomy O- 2 yr & 5 yr survival rates, recurrence rate, complication rate RCT? Y (1:1) Ff-up adeq? N; DO ctrl =23%ctrl(vs 12%) WCS prod. sl. reversal of conclusion (over) ITT? Y Blinding? Y; ptx & observer Baseline same? N; p = NS; Clin: more upper 1/3 in exptal (underest) & more lower 1/3 in exptal (overest); net eff: overest Equal tx? N; Same ff-up but more postop RT/chemotx in controlunderestimn. Rc 52% 10% Rt 34% 53% RRR 35% -4.3% RR 0.65 5.3 ARR 0.18 -0.43 NNT 6 2 p 0.192 <0.001

M- RCT Outcome 5 yr death Tracheost.

Author's conclusion: Three-field lymphadenectomy appears to prolong survival but the diff was not statistically significant. Reviewer's conclusion: (soft)- agree w/ authors but it definitely increases postop complcations Resolution of scenario: Ptx. underwent subtotal esophagectomy w/ 3-field lymphadenectomy but surgeon had to perform tracheostomy on the 7th postop day due to phrenic nerve palsy. Reviewer's name: Isaac David Ampil II, MD, FPCS Date: 07/06/00

Remember...
An ounce of critical appraisal is worth pounds of futile reading. - Isaac David Ampil II, MD

THANK YOU VERY MUCH!

Group Session

Group Session
A 61/M undergoes a colonoscopy as part of an executive check-up. Two polyps were found in the sigmoid and were removed. Histopath reveals adenomatous polyps. He asks you if there is any way to prevent the recurrence of the polyps, knowing their malignant potential. You recall having read an article in the New England Journal of Medicine on the use of Celecoxib for the prevention of adenomatous polyps.