Appraising Diagnostic Study

Muhammad Ali

Medical School
University of Sumatera Utara
 What are “tests” used for?

Log of reasons by several docs:

Diagnosis – most common
but also
Monitoring – has it changed?
Prognosis – risk/stage within Dx
Treatment planning
 Basic principles (1)
• Ideal diagnostic tests – right answers:
– (+) results in everyone with the disease and
– ( - ) results in everyone else
• Usual clinical practice:
– The test be studied in the same way it would be used in
the clinical setting
• Observational study, and consists of:
– Predictor variable (test result)
– Outcome variable (presence / absence of the disease)
 Basic principles (2)
• Sensitivity, specificity
• Prevalence, prior probability, predictive values
• Likelihood ratios
• Dichotomous scale, cutoff points (continuous scale)
• Positive (true and false), negative (true and false)
• ROC (receiver operator characteristic) curve
 EBM Issues:
• Is this evidence about the accuracy of a
diagnostic test valid?
• Does this valid evidence demonstrate an
important ability to accurately distinguish
patients who do and don’t have a specific
disorder?
• Can I apply this valid, important diagnostic
test to a specific patient?
 Critical appraisal
• Use worksheet
• Use supporting software
– CAT Maker
– Save in CAT Banks
 Is this evidence about a diagnostic test
valid?
Was there an independent, blind comparison with a
reference (“gold”) standard of diagnosis?

Was the diagnostic test evaluated in an appropriate

spectrum of patients (like those in whom we would use
it in practice)?

Was the reference standard applied regardless of the

diagnostic test result?

Was the test (or cluster of tests) validated in a second,

independent group of patients?
Does this valid evidence demonstrate an important
ability to accurately distinguish patients who do
and don’t have a specific disorder?

• Sensitivity
• Specificity
• Likelihood ratios
 • SnNout
Diagnostic test with a very high sensitivity ,
a negative result effectively rules out the
diagnosis

• SpPin
Diagnostic test with a very high specificity ,
a positive result effectively rules in the
diagnosis
 Questions to answer in applying a valid
diagnostic test to an individual patient
 Is the diagnostic test available, affordable, accurate,
and precise in our setting?

 Can we generate a clinically sensible estimate of our

patient’s pre-test probability?
 From personal experience, prevalence statistics,
practice databases, or primary studies
 Are the study patients similar to our own?
 Is it unlikely that the disease possibilities or
probabilities have changed since this evidence was
gathered?
Guides for deciding whether a screening or early
diagnostic maneuver does more good than harm:

 Does early diagnosis really lead to improved

survival, or quality of life, or both?
 Are the early diagnosed patients willing partners
in the treatment strategy?
 Is the time and energy it will take us to confirm
the diagnosis and provide (lifelong) care well
spent?
 Do the frequency and severity of the target
disorder warrant this degree of effort and
expenditure?
 Diagnostic tests …
Is not about finding absolute truth, but
about limiting uncertainty

establishes both the necessity and the

logical base for introducing
probabilities, pragmatic test-treatment
thresholds ..
Start thinking about
• what you’re going to do with the results of the
diagnostic test, and
• whether doing the test will help your patients
 DISEASE
Yes No Total
3 7 10

a+b
Yes
a b
TEST

c D
d
90

c+d
No 1 89
Total 4 96 100
a+c b+d a+b+c+d

TRADITIONAL 2x2 TABLE

 SENSITIVITY
DISEASE
Yes No Total FALSE
NEGATIVES
3 7 10

a+b
Yes
a b
TEST

c D
d
90

c+d
No 1 89
Total 4 96 100
a+c b+d a+b+c+d

SENSITIVITY
The proportion of people with the diagnosis (n=4)
who are correctly identified (n=3)
Sensitivity = a/(a+c) = 3/4 = 75%
 SPECIFICITY
DISEASE
FALSE
Yes No Total POSITIVES

3 7 10

a+b
Yes
a b
TEST

c D
d
90

c+d
No 1 89
Total 4 96 100
a+c b+d a+b+c+d

SPECIFICITY
The proportion of people without the diagnosis (n=96)
who are correctly identified (n=89)
Specificity = d/(b+d) = 89/96 = 93%
 PREDICTIVE VALUES (POS or NEG)

DISEASE
Yes No Total
3 7 10

a+b
Yes
a b
TEST

c D
d
90

c+d
No 1 89
Total 4 96 100
a+c b+d a+b+c+d

Positife or Negative predictive values PPV = 3/10 = 30%

are the proportion of disease / non
disease people to all who have NPV = 89/90=89%
positive/negative tests.
 PRE-TEST ODDS
DISEASE
Yes No Total
3 7 10

a+b
Yes
a b
TEST

c D
d
90

c+d
No 1 89
Total 4 96 100
a+c b+d a+b+c+d

In the sample as a whole, the odds of having the disease

are 4 to 96 or 4% (the PRE-TEST ODDS)
 POST-TEST ODDS
DISEASE
Yes No Total
3 7 10

a+b
Yes
a b
TEST

c D
d
90

c+d
No 1 89
Total 4 96 100
a+c b+d a+b+c+d

In the sample as a whole, the odds of having the disease

are 4 to 96 or 4% (the PRE-TEST ODDS)
In those who score positive on the test, the odds of having
the disease are 3 to 7 or 43% (the POST-TEST ODDS)
 POST-TEST ODDS
DISEASE
Yes No Total
3 7 10

a+b
Yes
a b
TEST

c D
d
90

c+d
No 1 89
Total 4 96 100
a+c b+d a+b+c+d

In the sample as a whole, the odds of having the disease are 4 to 96 or

4% (the PRE-TEST ODDS)
In those who score positive on the test, the odds of having the disease are
3 to 7 or 43% (the POST-TEST ODDS)
In those who score negative on the test, the odds of having the disease
are 1 to 89 or approximately 1%
 DIAGNOSTIC ODDS RATIO
DISEASE Potentially useful as an overall
summary measure, but only in
Yes No Total conjunction with other
measures (LR, sensitivity,
3 7 10

a+b
Yes specificity)
a b
TEST

c D
d
90

c+d
No 1 89
3
4 96 100 DOR  7
Total 1
a+c b+d a+b+c+d 89
0.429
The Diagnostic Odds Ratio is the ratio of   38.2
odds of having the diagnosis given a 0.011
positive test to those of having the
diagnosis given a negative test
 BAYES THEOREM

POST-TEST ODDS =
LIKELIHOOD RATIO x PRE-TEST ODDS
 Interpreting
Sensitivity &
Specificity

Using natural
frequencies to
calculate Bayes’
theorem

Natural frequencies provide a more graphic, easy to

understand way to portray probabilities for both
physicians and patients
In small groups what is the likelihood that this
patient has the disease?

A disease with a prevalence of 3% must

be diagnosed
There is a test for the disease
It has a sensitivity of 50% and a
specificity of 90%
For a prevalence of 3% - in every 100 patients

3 will have the disease

97 Will not have the disease

For a test with a sensitivity of 50%

For every 100

patients with the
disease 1-2 will test
positive

97 How many will not

have the disease?
For a test with a specificity of 90%

False positive rate

= 1-Specificity
(10%)

87
For those who don’t have the
disease 10 patients will go on
to test positive
Given a positive test what are my chances of disease doc?

11 ½ people have a positive test Of

which 1 ½ have will have the disease

87
13% chance of having the
disease given a positive test
Given a negative test what are my chances of not having the
disease doc?

88 ½ people
will have a negative test of
which 1 ½ will have the
disease
False reassurance rate of
1.7%

87
Describe the result to your patient

For every 100 people who have a

positive test 13 will have the
disease
What happens if I have an negative test

Negative test pretty much

rules out the disease but a
small number 1-2 people
in every 100 will still have
the disease
 Try it again
what is the likelihood that this patient has the disease?

A disease with a prevalence of 30%

must be diagnosed
There is a test for this disease
It has a sensitivity of 50% and a
specificity of 90%
 Prevalence of 30%
Sensitivity of 50%
Specificity of 90%

Disease +ve
30 15
22 positive
tests in total
100 of which 15
have the
disease
Disease -ve 70
7 About 70%
 Likelihood Ratio
Sensitivity/1-specificity
50% / 10%
=5
LR × a
b
 Likelihood Ratio
LR a
b
30 ×5 150

100

70
70
Prevalence of 30%
Sensitivity of 50%
Specificity of 90%
 Now consider the FOB screening tests

You find out that your father has undertaken the test
and has a positive result – He ask you whether he has
cancer?

Prevalence of disease is 0.3% must be

diagnosed.
Sensitivity of 50%
False positive rate 3%.
 APPLYING A DIAGNOSTIC TEST IN
DIFFERENT SETTINGS
– The Positive Predictive Value of a test will vary
(according to the prevalence of the condition in the
chosen setting)
– Sensitivity and Specificity are usually considered
properties of the test rather than the setting, and
are therefore usually considered to remain
constant
– However, sensitivity and specificity are likely to be
influenced by complexity of differential diagnoses
and a multitude of other factors
THANK YOU