Predicting treatment failure in adults and children on

antiretroviral therapy: a systematic review of the

performance characteristics of the 2010
WHO immunologic and clinical criteria
for virologic failure
George W. Rutherforda, Andrew Anglemyera, Philippa J. Easterbrookb,
Tara Horvatha, Marco Vitoriab, Martina Penazzatob and
Meg C. Dohertyb
Objective: We systematically reviewed the performance of 2010 WHO immunologic
and clinical criteria for predicting virologic failure in HIV-infected patients on anti-
retroviral therapy (ART).
Design: Systematic review.
Methods: We used Cochrane Collaboration methods. We calculated unweighted
sensitivity, specificity, positive predictive value (PPV), and negative predictive value
(NPV) of immunologic and clinical criteria for predicting virologic failure.
Results: We identified 18 studies. Sixteen assessed immunologic criteria in adults;
12 defined virologic failure as a plasma viral load of more than 50 to more than
1000 copies/ml in adults, three as viral load at least 5000 copies/ml, and two as viral
load more than 10 000 copies/ml; the sensitivity ranged from 16.8 to 54.9%, specificity
from 82.9 to 95.5%, PPV from 15.0 to 38.8%, and NPV from 90.9 to 98.6%. Seven
studies assessed clinical criteria to predict viral load of more than 50 to more than
1000 copies/ml; the sensitivity was 11.0%, specificity 90.5%, PPV 44.9%, and NPV
90.2%. Seven studies assessed clinical or immunologic criteria defining virologic failure
as viral load of more than 50 to more than1000 copies/ml; their sensitivity was 26.6%,
specificity 85.9%, PPV 49.4%, and NPV 91.1%. Four studies assessed immunologic
criteria in children; three defined virologic failure as viral load at least 5000 copies/ml
and one as viral load at least 400 copies/ml. The sensitivity ranged from 4.5 to
6.3%, specificity from 97.7 to 99.3%, PPV from 20.0 to 54.9%, and NPV from
85.5 to 91.8%.
Conclusion: The 2010 WHO clinical and immunologic criteria are insensitive and
have low PPV for predicting virologic failure. These data support the strong recom-
mendation 2013 treatment guidelines that viral load testing be used to monitor for,
diagnose, and confirm ART failure.
2014 Wolters Kluwer Health | Lippincott Williams & Wilkins

AIDS 2014, 28 (Suppl 2):S161S169

Keywords: antiretroviral therapy, CD4 T cell count, HIV, predictive value of

tests, systematic review, viral load

a
Cochrane HIV/AIDS Group, Global Health Sciences, University of California, San Francisco, San Francisco, California, USA, and
b
HIV Department, World Health Organization, Geneva, Switzerland.
Correspondence to George W. Rutherford, MD, AM, Global Health Sciences, University of California, San Francisco, 50 Beale
Street, 12th floor, San Francisco, CA 94105, USA.
Tel: +1 415 597 9108; e-mail: grutherford@psg.ucsf.edu

DOI:10.1097/QAD.0000000000000236

ISSN 0269-9370 Q 2014 Wolters Kluwer Health | Lippincott Williams & Wilkins S161
Copyright Lippincott Williams & Wilkins. Unauthorized reproduction of this article is prohibited.
S162 AIDS 2014, Vol 28 (Suppl 2)
Introduction
Viral load monitoring has become the standard of care for
monitoring the success of and diagnosing the failure of
antiretroviral therapy (ART) in high-income countries
[1] and has been explicitly recommended, when available,
by the WHO since 2010 [2]. However, in most low-
income and many middle-income countries, viral load
testing is not readily available, although some progress has
been made in recent years [35]. In settings in which
there is no access to viral load testing, clinical monitoring
alone or a combination of clinical and immunologic
monitoring is used to assess response to ART and
determine treatment failure [6].
WHO has developed simplified immunologic and clinical
criteria to diagnose treatment failure and guide switches
to second-line therapy. In 2006, there was one clinical
criterion and one immunologic criterion, which were
a new or recurrent WHO stage 4 condition, which is
not immune reconstitution inflammatory syndrome
(clinical criterion); and a fall in CD4 T cell count to
baseline or below or a 50% fall in CD4 T cell count from
on-treatment peak value or persistent CD4 T cell count
levels below 100 cells/ml in the absence of concomitant
infection that can cause a transient CD4 T cell count
decline (immunologic criterion) [6]. The guidelines
also suggested a plasma viral load threshold of
10 000 copies/ml as a threshold for defining virologic
failure [6]. In 2010, the immunologic criterion was
broadened to specify a 50% fall in CD4 T cell count
levels from on-treatment peak value or persistent CD4
T cell count levels of less than 100 cells/ml after at least
24 weeks on ART in adults and children at least 5 years
old or persistent CD4 T cell count levels of less
than 200 cells/ml or CD4 T cell count less than 10% in
24-year-old children [2]. The clinical criterion was
unchanged. The recommendations also lowered the
definition of virologic failure to a threshold viral load of
5000 copies/ml [2].
Five randomized controlled trials have assessed whether
different monitoring strategies (laboratory vs. clinical)
are associated with biological outcomes such as death
and disease progression as well as unnecessary changes
to second-line therapy [711]. Two studies, conducted
in Uganda and Zimbabwe [7,8] found that clinical
monitoring was inferior to laboratory monitoring. The
Uganda trial also compared clinical, immunologic, and
virologic monitoring with clinical and immunologic
monitoring and found no differences in mortality, disease
progression, unnecessary switches, or virologic failure [8].
Another study in Thailand found no difference in 3-year
clinical outcomes using immunologic monitoring
compared to viral load monitoring [9]. A study in
Cameroon compared clinical monitoring alone to
immunologic and virologic monitoring and found small
benefits in immunologic recovery in participants who
Copyright Lippincott Williams & Wilkins. Unauthorized reproduction of this article is prohibited.
received laboratory monitoring, but not for other
endpoints [10]. Additionally, a study in children less
than 5 years old also conducted in Uganda and Zimbabwe
found that immunologic monitoring, as opposed to
clinical monitoring alone, was associated with a small
mortality benefit after the first year of ART [11].
However, despite these findings and despite WHOs
strong recommendation [2], clinical and immunologic
monitoring continue to be used.
To inform the 2013 WHO guidelines on the use of
ART, we conducted a systematic review to assess how
accurately WHOs 2010 clinical and immunologic
criteria predicted virologic failure in studies using
different viral load thresholds in adults and children.
Methods
We specified a priori our null hypothesis that the 2010
WHO clinical, immunologic, and clinical and immuno-
logic criteria combined would be poorly predictive of
virologic failure, as defined by more than 50 copies/ml,
more than 1000 copies/ml, more than 5000 copies/ml,
and more than 10 000 copies/ml. We included studies
that assessed the performance of the 2010 WHO clinical
criterion, immunologic criteria, and the clinical and
immunologic criteria combined with respect to predict-
ing virologic failure. Studies had to report sufficient
data to calculate individual cell sizes and to allow at a
minimum the calculation of a positive predictive value
(PPV).
Search methods for identification of studies
Using Cochrane Collaboration methods [12], we
formulated a comprehensive and exhaustive search
strategy in an attempt to identify all relevant studies.
Databases searched included the Cochrane Central
Register of Controlled Trials, Excerpta Medica Database
(EMBASE), MEDLINE via PubMed, and WHOs
Global Index Medicus. The search strategy included
Medical Subject Heading (MeSH) terms and a range of
relevant keywords. The search period was from 1 January
2006 to 15 November 2012 and was iterative in that
references cited in included studies were searched for
additional references. We also included studies from the
gray literature, published in any language. Additionally,
we searched for potentially relevant abstracts presented
at key scientific conferences (the Conference on
Retroviruses and Opportunistic Infections, the Inter-
national AIDS Conference and the International AIDS
Society Conference on HIV Pathogenesis, Treatment and
Prevention) within the search period. See Appendix 1 for
our PubMed strategy, http://links.lww.com/QAD/
A493, which was modified and adapted for use with
the other databases.
 Evaluation of WHO criteria for antiretroviral treatment failure Rutherford et al.
Selection of studies
We imported search results into a bibliographic citation
management software (EndNote X4; Thomson Reuters,
New York, New York, USA) and excluded duplicate
references. Reviewing only article titles, one author (TH)
excluded all references that were clearly irrelevant. Two
authors (GWR and AA), working independently, then
reviewed the titles, abstracts, and descriptor terms of the
remaining citations to identify potentially eligible reports.
We obtained full-text articles for all references identified
that we judged potentially met inclusion criteria. GWR
and AA reviewed these full-text articles and applied
the inclusion criteria to establish each studys eligibility
or ineligibility. Our plan was to resolve any differences
of opinion through discussion and, if necessary, with a
neutral third party arbiter, but we had no disagreements.
Data extraction and management
After identifying trials for inclusion, two authors
(GWR and AA) working independently examined and
extracted data from each study. GWR and AA separately
entered these data into standardized data extraction
forms and then compared the extracted data. There were
no disagreements.
Assessment of methodological quality
We assessed methodological quality of included studies
using methods developed for assessing the quality of
studies of diagnostic test accuracy. We used the Cochrane
Collaborations 11-point adaptation [13] of the 14-point
QUADAS instrument [14], which includes items such as
representativeness of the study sample, appropriateness of
the verification procedure, blinding of test interpretation,
and missing data.
Statistical analysis and data synthesis
We conducted separate analyses for adults and children.
We calculated unweighted pooled sensitivity, specificity,
PPV, and negative predictive value (NPV) for the 2010
WHO clinical criterion, the immunologic criteria, and
either the clinical or the immunologic criteria in
combination for descriptive purposes only, as pooled
sensitivities and specificities can be misleading [15].
When there were adequate data reported within a study,
we calculated all parameters. When only true-positive
and false-positive test results were reported, we estimated
only PPVs. As there are no comparable, clear-cut clinical
criteria for treatment failure in children, we evaluated
only the performance of immunologic criteria for
predicting virologic failure. For adults, we analyzed
three different viral load thresholds: a viral load of more
than 50 to more than 1000 copies/ml, which represented
the lower limits of detection at the time the studies
were done; a viral threshold of more than 5000 copies/ml
(the 2010 WHO definition); and a viral load threshold of
more than 10 000 copies/ml (the 2006 WHO definition).
For children, we analyzed virologic failure threshold of
more than 400 copies/ml and more than 5000 copies/ml.
Copyright Lippincott Williams & Wilkins. Unauthorized reproduction of this article is prohibited.
S163
Confidence intervals (CIs) for all estimates are exact
binomial CIs [16]. We used R version 3.0.0 for the
calculations [17].
Results
Our searches initially yielded 3543 studies, of which
816 were duplicates. Of the remaining 2727 studies,
2299 were clearly irrelevant, leaving 428 studies. Of
these, we reviewed 34 (7.9%) in detail. Twenty-five
were in adults, and nine were in children. Fourteen
(56.0%) of the adult studies and four (44.4%) of
the pediatric studies met our inclusion criteria (Fig. 1,
Table 1). The adult studies were from Africa [1829],
Asia [30,31], and South America [23], and the pediatric
studies were from Africa [3234] and Asia [35].
We excluded 16 studies because performance of the
WHO criteria was not calculable from data presented
[8,3641], authors evaluated different predictor vari-
ables than the WHO criteria [4246], authors used
different outcomes such as mortality and disease
progression [4749], or authors did not define virologic
failure [50].
Of the 14 studies in adults, 13 evaluated the clinical and
immunologic criteria using lower plasma viral load values
in adults, ranging from more than 50 to more than
1000 copies/ml [1820,2231]. Three reported how
well these criteria predicted a plasma viral load of more
than 5000 copies/ml [22,26,28]; one of these provided
sufficient data to calculate PPV only [22]. Two studies
evaluated a less stringent plasma viral load threshold of
10 000 copies/ml [24,31]. Table 2 lists all study-specific
data for each criterion.
Of the studies defining virologic failure based on
threshold plasma viral loads between more than 50 and
more than 1000 copies/ml in adults, 13 reported the
performance of the 2010 WHO immunologic criteria
[1820,2231], seven the performance of the clinical
criterion [19,2224,2931], and seven the performance
of either immunologic or clinical criteria [18,2225,29
31] (Table 1). In these studies, which involved 15 581
patients, the sensitivity estimates ranged from 13 to 91%,
and the pooled sensitivity of the immunologic criteria was
54.9% (95% CI 52.956.9%). The specificity estimates
ranged from 75 to 99%, and the pooled specificity was
82.9% (95% CI 82.283.5%). The PPV estimates ranged
from 8 to 67%, whereas the pooled PPV was 38.0% (95%
CI 36.439.6%). Finally, the NPV estimates ranged from
85 to 100%, and the pooled NPV was 90.9% (95% CI
90.491.4%). The sensitivity estimates for the clinical
criterion ranged from 10 to 12%, and the pooled
sensitivity for the clinical criterion was 11.0% (95% CI
5.719.7%). The specificity estimates ranged from 84
to 96%, and the pooled specificity was 90.5% (95% CI
S164 AIDS 2014, Vol 28 (Suppl 2)

Total results
data from all three studies [22,26,28]. One study reported
n = 3543 the performance of clinical criteria and either immuno-
logic or clinical criteria [22]. In this study, the clinical
criterion had a PPV of 100% (based on a single patient),
Duplicates excluded
n = 816
and either immunologic or clinical criteria had a PPV of
70.7% (95% CI 60.279.7%) based on 92 patients.
The two studies that used a higher plasma viral load
References screened (>10 000 copies/ml) to define virologic failure assessed
by one author
n = 2727
WHO immunologic criteria alone among 3142 patients
[21,28]. They found a pooled sensitivity of 16.8%
Clearly irrelevant
(95% CI 11.324.1%, range 1023%), a specificity of
references excluded 95.5% (95% CI 94.696.2%, range 9099%), a PPV
n = 2299 of 15.0% (95% CI 10.021.7%, range 1417%), and
an NPV of 96.0% (95% CI 95.296.7%, range 9497%
References screened (Tables 2 and 4).
by two authors
n = 428 Of the four pediatric studies we identified, three used
more than 5000 copies/ml to define virologic failure
References excluded [32,34,35], and one used more than 400 copies/ml to
n = 394 define virologic failure [33]. The three studies that used
more than 5000 copies/ml to define virologic failure
Full text review by evaluated 4100 patients and found a pooled sensitivity
two authors of 4.5% (95% CI 3.16.6%, range 47%), a pooled
n = 34
specificity of 99.3% (95% CI 99.099.6%, range 99
100%), a pooled PPV of 54.9% (95% CI 40.568.6%,
References excluded
n = 16
range 49100%), and a pooled NPV of 85.4% (95% CI
84.386.5%, range 7386%). The one study that
evaluated the 2010 criteria in 2256 children using a
Studies included in
review
definition of virologic failure of more than 400 copies/ml
n = 18 found a sensitivity of 6.3% (95% CI 3.411.0%),
a specificity of 97.7% (95% CI 96.998.3%), a PPV of
Adult studies: n = 14 20% (95% CI 11.232.3%), and an NPV of 91.8%
Pediatric studies: n = 4 (95% CI 90.692.9%) [33] (Tables 2 and 4).

Fig. 1. Flowchart of study selection process.

88.292.3%). The PPVestimates ranged from 7 to 100%,
and the pooled PPV was 44.9% (95% CI 38.251.8%).
Finally, the NPVestimates ranged from 89 to 91% and the
pooled NPV was 90.2% (95% CI 88.092.1%). Using
either immunologic or clinical criteria, the sensitivity
estimates ranged from 20 to 33%, and the pooled
sensitivity was higher (26.6%, 95% CI 19.335.4%)
than estimates using clinical criterion alone, but the
specificity (85.9%, 95% CI 83.787.9%, range 8687%),
PPV (49.4%, 95% CI 45.153.6%, range 1371%), and
NPV (91.1%, 95% CI 89.292.8%, range 8792%) were
similar to those for the clinical criteria alone (Tables 2
and 3).
Among the three studies that used a viral load threshold
of more than 5000 copies/ml, the overall performance
characteristics of the immunologic criteria were a
sensitivity of 51.7% (95% CI 38.564.6%, range 28
88%), a specificity of 93.9% (95% CI 92.894.9%, range
9098%), a PPV of 27.0% (95% CI 21.633.2%, range
857%), and an NPV of 98.6% (95% CI 98.099.9%,
range 97100%); of these, only PPV was calculated using
Methodological quality of included studies
The methodological quality of the studies we reviewed is
mostly high (Appendix 2, http://links.lww.com/QAD/
A493). In nearly all studies, the sample of patients
was representative, and although different thresholds were
used, the reference standards of virologic failure were
appropriate. There is no evidence of disease progression
bias, that is, a bias arising from an inappropriately long
delay between assessing the criteria and drawing the
confirmatory tests. There is also no evidence of partial
verification bias (selective testing of patients with
the reference standard), differential verification bias
(case confirmation using different reference standards),
or incorporation bias (results of the case definition
being used as a component of the reference standard).
One study [18] reported that interpretation of test results
was not blinded. This question was not addressed in
any of the other papers, but, considering the studies
cross-sectional nature and laboratory test outcome (viral
load), the lack of blinding is unlikely to be a source of
bias. None of the included studies addressed the issue

Copyright Lippincott Williams & Wilkins. Unauthorized reproduction of this article is prohibited.
 Evaluation of WHO criteria for antiretroviral treatment failure Rutherford et al. S165

Table 1. Characteristics of included studies.

Reference Location Population N Virologic failure definition Criteria

Abouyannis et al. [18] Uganda Adults 133 >1000 copies/ml Clinical or immunologic
Barlow-Mosha et al. [32]a Uganda Children 142 >5000 copies/ml Immunologic
Chaiwarith et al. [30] Thailand Adults 327 >50 copies/ml Clinical or immunologic
Davies et al. [33] South Africa Children 2256 >400 copies/ml Immunologic
Davies et al. [34] South Africa Children 2945 >5000 copies/ml Immunologic
Hosseinipour et al. [19]a Malawi Adults 9 >400 copies/ml Clinical
Kantor et al. [20] Kenya Adults 42
400 copies/ml Immunologic
Keiser et al. [21] Africa and Adults 2009 >10 000 copies/ml Immunologic
South America
Labhardt et al. [22] Lesotho Adults 92
5000 copies/ml Clinical, immunologic,
clinical or immunologic
a
Mee et al. [23] South Africa Adults 273 10004999 copies/ml Clinical, immunologic
Mee et al. [24] South Africa Adults 319 4001000 copies/ml Clinical, immunologic
Meya et al. [25] Uganda Adults 496 >1000 copies/ml Clinical, immunologic
Moore et al. [26] Uganda Adults 1080 >5000 copies/ml Immunologic
Rawizza et al. [27] Nigeria Adults 9690 >1000 copies/ml Immunologic
Rewari et al. [31] India Adults 122
10 000 copies/ml Clinical, immunologic
Reynolds et al. [28] Uganda Adults 1133
5000 copies/ml Immunologic
van Oosterhout et al. [29] Malawi Adults 155
400 copies/ml Clinical, immunologic
Westley et al. [35] Cambodia Children 1013 >5000 copies/ml Immunologic
a
Conference abstract.

of uninterpretable test results. No study was reported to
be sponsored by industry; no authors report having
a conflict of interest. One published report [31] and
three conference abstracts reported nothing regarding
conflict of interest [19,23,32].
Discussion
We found that the 2010 immunologic and clinical
criteria for treatment failure have low sensitivity and
PPV for identifying those with virologic failure,
especially children. We also found that using a lower
threshold to define virologic failure had minimal
impact on performance of the WHO criteria. Our
findings are consistent with current guidelines not
only from WHO [51] but also from the United States
Department of Health and Human Services [52] and
the British HIV Association [53], which recommend
the use of plasma viral load monitoring to detect viro-
logic failure and to guide changes in ART. However,
we also note that there is a substantial randomized
controlled trial literature that suggests immunologic or
clinical and immunologic monitoring is not inferior
to virologic monitoring when measured against clinical
endpoints [811].
Our findings are subject to some limitations. First,
as with all systematic reviews, we are limited by the
sensitivity of our search and our ability to identify
relevant studies. We comprehensively searched four key
scientific databases and used broad inclusion criteria to
identify studies. We carefully reviewed the bibliographies
of included studies as well as abstracts from recent
conferences to assure the completeness of our search.
Second, we are potentially limited by publication bias,
but given the largely poor performance reported for
the WHO clinical and immunologic criterias ability to
predict virologic failure at any threshold, we suspect
that publication bias was not an issue in this review.
Nonetheless, we acknowledge that studies using more
recent tests, which may not as yet have been published,
might have yielded more favorable results. Additionally,
commercially sponsored studies that produced less
favorable results may not have been identified by our
searches. Third, all pooled estimates are for descriptive
purposes only and inference derived from the estimates
should be carefully considered. Pooling sensitivities and
specificities can be misleading [15], and this is especially
true with unweighted, pooled estimates, which may
be misinterpreted as weighted, pooled estimates using
random-effects or fixed-effects models. Finally, we
adopted methods developed to assess the accuracy of
diagnostic tests against a gold standard to assess the
performance of a case definition against a laboratory
outcome. However, the studies we reviewed often did
not report methodological details that inform quality
assessments of diagnostic tests [13]. Incomplete reporting
in such studies could make it difficult to discern whether
important aspects of conduct and design were adequate or
whether the reporting of such aspects was inadequate
[13]. In contrast to studies performed with the explicit
aim of assessing diagnostic test accuracy, the studies we
reviewed reported mostly retrospective data generated in
the course of clinical practice in resource-limited settings,
and, as such, explicit detail was largely missing, leading
us to infer several quality criteria. Nonetheless, we
believe that overall the literature we synthesize here is
of generally high quality.
In conclusion, our findings highlight the poor diagnostic
performance of the 2010 WHO clinical and immuno-
logic criteria for predicting virologic failure. This

Copyright Lippincott Williams & Wilkins. Unauthorized reproduction of this article is prohibited.
S166 AIDS 2014, Vol 28 (Suppl 2)

Table 2. Results of included studies by study population, criteria for assessing virologic failure, and definition of virologic failure.

Study [reference number] TP FN FP TN Sensitivity (95% CI) Specificity (95% CI) PPV (95% CI) NPV (95% CI)

Adults: immunologic criteria

Virologic failure defined as >50->1000 copies/ml
Abouyannis [18] 2 8 6 45 20 (356) 88 (7696) 25 (465) 85 (7293)
Chaiwarith [30] 4 26 31 266 13 (431) 90 (8693) 11 (327) 91 (8794)
Kantor [20] 24 18 57 (4172)
Keiser [21] 12 51 83 1863 19 (1031) 96 (9597) 13 (721) 97 (9798)
Labhardt [22] 49 26 65 (5376)
Mee 2006 [23] 7 21 13 232 25 (1145) 95 (9197) 35 (1559) 92 (8895)
Mee 2008 [24] 7 26 12 274 21 (939) 96 (9398) 37 (1662) 91 (8894)
Meya [25] 11 38 55 392 22 (1237) 88 (8491) 17 (928) 91 (8894)
Moore [26] 32 3 16 1029 91 (7798) 99 (9899) 67 (5280) 100 (99100)
Rawizza [27] 1225 872 1897 5696 58 (5661) 75 (7476) 39 (3841) 87 (8688)
Reynolds [28] 10 26 115 982 28 (1445) 90 (8891) 8 (414) 97 (9698)
Van Oosterhoutt [29] 33 43 43 (3255)
Virologic failure defined as >5000 copies/ml
Labhardt [22] 33 42 44 (3356)
Moore [26] 21 3 16 1040 88 (6897) 98 (9899) 57 (3973) 100 (99100)
Reynolds [28] 10 26 115 982 28 (1445) 90 (8891) 8 (414) 97 (9698)
Virologic failure defined as >10 000 copies/mL
Keiser [21] 6 57 29 1917 10 (420) 99 (9899) 17 (734) 97 (9698)
Reynolds [28] 18 62 107 946 23 (1433) 90 (8892) 14 (922) 94 (9295)
Adults: clinical criterion
Virologic failure defined as >5000 copies/ml
Chaiwarith [30] 3 27 13 284 10 (227) 96 (9398) 19 (446) 91 (8894)
Hosseinipour [19] 5 4 56 (2186)
Labhardt [22] 1 0 100 (17100)
Mee 2006 [23] 3 25 39 206 11 (228) 84 (7988) 7 (220) 89 (8493)
Mee 2008 [24] 4 29 27 259 12 (328) 91 (8794) 13 (430) 90 (8693)
Rewari [31] 1 1 50 (892)
Van Oosterhoutt [29] 81 35 70 (6178)
Adults: clinical or Immunologic criteria
Virologic failure defined as >5000 copies/ml
Abouyannis [18] 4 8 8 52 33 (1065) 87 (7594) 33 (1065) 87 (7594)
Chaiwarith [30] 6 24 42 255 20 (839) 86 (8190) 13 (525) 91 (8794)
Labhardt [22] 65 27 71 (6080)
Mee 2008 [24] 11 22 41 245 33 (1852) 86 (8190) 21 (1135) 92 (8895)
Meya [25] 12 37 62 385 24 (1339) 86 (8389) 16 (927) 91 (8894)
Rewari [31] 87 35 71 (6279)
Van Oosterhout [29] 89 66 57 (4965)
Children: immunologic criteria
Virologic failure defined as >400 copies/ml
Davies 2011 [33] 6 (311) 98 (9798) 20 (1132) 92 (9193)
Virologic failure defined as >5000 copies/ml
Barlow-Mosha [32] 3 38 0 101 7 (220) 100 (96100) 100 (30100) 73 (6480)
Davies 2012 [34] 18 421 19 2487 4 (26) 99 (99100) 49 (3266) 86 (8487)
Westley [35] 7 130 4 872 5 (210) 100 (99100) 64 (3189) 87 (8589)

FN, false negative; FP, false positive; NPV, negative predictive value; PPV, positive predictive value; TN, true negative; TP, true positive.

reaffirms both the importance of routine viral load unnecessary switching to second-line ART. There is a
monitoring to identify all those with virologic failure need to improve these criteria if clinical and immunologic
as well as to confirm virologic failure in those with monitoring is to continue, especially for children.
clinical and immunologic WHO criteria to avoid These conclusions were recognized in the 2013 WHO

Table 3. Ability of WHO 2010 immunologic and clinical criteria to predict virologic failure (plasma viral load threshold more than 50 to more
than 1000 copies/ml).

Pooled sensitivity Pooled specificity Pooled PPV Pooled NPV

Criterion Studies N (95% CI) (95% CI) (95% CI) (95% CI)

Immunologic 1820, 2231 15 581 54.9% (52.956.9) 82.9% (82.283.5) 38.8% (36.439.6) 90.9% (90.491.4)
Clinical 19, 2224, 2931 1047 11.0% (5.719.7) 90.5% (88.292.3) 44.9% (38.251.8) 90.2% (88.092.1)
Immunologic or 18, 2225, 2931 1583 26.6% (19.335.4) 85.9% (83.787.9%) 49.4% (45.153.6) 91.1% (89.292.8)
clinical

CI, confidence interval; NPV, negative predictive value; PPV, positive predictive value.

Copyright Lippincott Williams & Wilkins. Unauthorized reproduction of this article is prohibited.
 Evaluation of WHO criteria for antiretroviral treatment failure Rutherford et al. S167

Pooled NPV (95% CI)

(98.099.9)
(90.491.4)
(95.296.7)
(84.386.5)
(90.692.9)
treatment guidelines, which strongly recommend that
viral load testing be used to monitor for, diagnose, and
confirm ART failure [51].

98.6%
90.9%
96.0%
85.5%
91.8%
Acknowledgements
Pooled PPV (95% CI)

The authors would like to thank Alicen Spaulding, PhD,

(21.633.2)
(36.439.6)
(10.021.7)
(40.568.6)
(11.232.3)
of the University of Minnesota and Gavrilah Wells for
their able assistance in identifying studies.
Table 4. Ability of WHO 2010 immunologic criteria to predict virologic failure in adults and children on antiretroviral therapy, by viral load threshold.

27.0%
38.8%
15.0%
54.9%
20.0%

GWR and AA are the lead authors. PJE, TH, MV, MP

and MD identified studies to be screened. GWR, AA,
PJE and TH identified studies for eligibility, extracted
data and assessed the methodologic quality of included
Pooled specificity (95% CI)

studies. AA performed the analysis with assistance from

MV, MP and MD. All authors critically reviewed the
(92.894.9)
(82.283.5)
(94.696.2)
(99.099.6)
(96.998.3)

manuscript before submission.

Conflicts of interest
93.9%
82.9%
95.5%
99.3%
97.7%

The authors declare that they have no conflicts of interest.

This work was supported under a contract with the
WHO (APW 200671213).
Pooled sensitivity (95% CI)

References
(38.564.6)
(52.956.9)
(11.324.1)

(3.411.0)
(3.16.6)

1. Palmer S. Advances in detection and monitoring of plasma

viremia in HIV-infected individuals receiving antiretroviral
therapy. Curr Opin HIV AIDS 2013; 8:8792.
4.5%
6.3%
51.7%
54.9%
16.8%

2. World Health Organization. Antiretroviral therapy for HIV

CI, confidence interval; NPV, negative predictive value; PPV, positive predictive value.

infection in adults and adolescents: recommendations for

a public health approach [2010 revision]. Geneva: World
Health Organization; 2010, http://whqlibdoc.who.int/publi
cations/2010/9789241599764_eng.pdf. [Accessed 26 January
2014]
2288

3142
4100
2256
15 581

3. Chang LW, Harris J, Humphreys E. Optimal monitoring

N

strategies for guiding when to switch first-line antiretroviral

therapy regimens for treatment failure in adults and adole-
scents living with HIV in low-resource settings. Cochrane
Database Syst Rev 2010:CD008494.
4. Roberts T, Bygrave H, Fajardo E, Ford N. Challenges and
22, 26, 28
1820, 2231
21, 28
32, 34, 35
33

opportunities for the implementation of virological testing in

Studies

resource-limited settings. J Int AIDS Soc 2012; 15:17324.

5. Stevens WS, Marshall TM. Challenges in implementing HIV
load testing in South Africa. J Infect Dis 2010; 201 (Suppl 1):
S78S84.
6. World Health Organization. Antiretroviral therapy for HIV in-
fection in adults and adolescents. Recommendations for a public
health approach. [2006 revision]. Geneva: World Health Orga-
>50 to >1000 copies/ml

nization; 2006, http://www.who.int/hiv/pub/guidelines/artadul

tguidelines.pdf. [Accessed 26 January 2014]
Viral load threshold

>10 000 copies/ml

7. DART Trial Team. Routine versus clinically driven laboratory

>5000 copies/ml

>5000 copies/ml
>400 copies/ml

monitoring of HIV antiretroviral therapy in Africa (DART):

a randomised noninferiority trial. Lancet 2010; 375:123
131.
8. Mermin J, Ekwaru JP, Were W, Degerman R, Bunnel R,
Kaharuza F, et al. Utility of routine viral load, CD4 cell count,
and clinical monitoring among adults with HIV receiving
antiretroviral therapy in Uganda: randomised trial. BMJ
2011; 343:d6792.
9. Jourdain G, Le Couer S, Ngo-Giang-Huong N, Traisthit P,
Age group

Cressey TR, Fregonese F, et al. Switching HIV treatment in

Children
Children

adults based on CD4 count versus viral load monitoring:

Adults
Adults
Adults

a randomized, noninferiority trial. PLoS Med 2013; 10:

e1001494.

Copyright Lippincott Williams & Wilkins. Unauthorized reproduction of this article is prohibited.
S168 AIDS 2014, Vol 28 (Suppl 2)
10. Laurent C, Kouanfack C, Laborde-Balen G, Aghokeng AF,
Mbougua JB, Boyer S, et al. Monitoring of HIV viral loads,
CD4 cell counts, and clinical assessments versus clinical moni-
toring along for antiretroviral therapy in rural district hospitals in
Cameroon (Stratall ANRS 21220/ESTHER): a randomised non-
inferiority trial. Lancet Infect Dis 2011; 11:825833.
11. ARROW Trial Team. Routine versus clinically driven labora-
tory monitoring and first-line antiretroviral therapy strategies
in African children with HIV (ARROW): a 5-year open-label
randomised factorial trial. Lancet 2013; 381:13911403.
12. Higgins JPTT, Green S (Eds): Cochrane handbook for systematic
reviews of interventions. Chichester, UK: Wiley-Blackwell;
2008.
13. Reitsma JB, Rutjes AWS, Whiting P, Vlassov VV, Leeflang MMG,
Deeks JJ. Chapter 9: assessing methodological quality. In:
Deeks JJ, Bossuyt PM, Gatsonis C,editors. Cochrane handbook
for systematic reviews of diagnostic test accuracy. Version 1.0.0
Oxford, UK: Cochrane Collaboration; 2009. http://srdta.cochra-
ne.org/.
14. Whiting P, Rutjes AW, Reitsma JB, Bossuyt PM, Kleijnen J.
The development of QUADAS: a tool for the quality assessment
of studies of diagnostic accuracy included in systematic reviews.
BMC Med Res Methodol 2003; 3:25.
15. Gatsonis C, Paliwal P. Meta-analysis of diagnostic and
screening test accuracy evaluations: methodologic primer.
AJR 2006; 187:7181.
16. Newcombe RG. Two-sided confidence intervals for the single
proportion: comparison of seven methods. Stats Med 1998;
17:857872.
17. R Core Team. R: a language and environment for statistical
computing. Vienna, Austria: R Foundation for Statistical Com-
puting; 2013, http://www.R-project.org/. [Accessed 14 January
2014].
18. Abouyannis M, Menten J, Kiragga A, Lynen L, Robertson G,
Castelnuovo B, et al. Development and validation of systems for
rational use of viral load testing in adults receiving first-line
ART in sub-Saharan Africa. AIDS 2011; 25:16271635.
19. Hosseinipour M, van Oosterhout J, Weigel R, Mzigangira D,
Saukila N, Mhango B, et al. Validating clinical and
immunological definitions of antiretroviral treatment failure
in Malawi [Abstract WEAB101]. 4th IAS Conference on
HIV Pathogenesis, Treatment and Prevention, Sydney, Australia;
2225 July 2007.
20. Kantor R, Diero L, DeLong A, Kamle L, Muyonga S, Mambo F,
et al. Misclassification of first-line antiretroviral treatment
failure based on immunological monitoring of HIV infection
in resource-limited settings. Clin Infect Dis 2009; 49:454462.
21. Keiser O, MacPhail P, Boulle A, Wood R, Schecther M, Dabis F,
et al. Accuracy of WHO CD4 cell count criteria for virological
failure of antiretroviral therapy. Trop Med Int Health 2009;
14:12201225.
22. Labhardt ND, Lejone T, Setoko M, Poka M, Ehmer J, Pfeiffer K,
et al. A clinical prediction score in addition to WHO criteria for
antiretroviral treatment failure in resource-limited settings:
experience from Lesotho. PLoS One 2012; 7:e47937doi:
10.1371/journal.pone.0047937.
23. Mee P, Fielding K, Charalambous S, Eisenstein M, Pemba L,
Innes C, et al. Evaluation of World Health Organization criteria
for antiretroviral treatment failure in resource-limited settings
[abstract WEPE065]. XVI International AIDS Conference,
Toronto, Canada; 1318 August 2006.
24. Mee P, Fielding KL, Charalmbous S, Churchyard GJ, Grant AD.
Evaluation of WHO criteria for antiretroviral treatment failure
among adults in South Africa. AIDS 2008; 22:19711977.
25. Meya D, Spacek LA, Tibenderana H, John L, Nemugga I, Magero
S, et al. Development and evaluation of a clinical algorithm to
monitor patients on antiretrovirals in resource-limited settings
using adherence, clinical and CD4 cell count criteria. J Int AIDS
Soc 2009; 12:3.
26. Moore DM, Awor A, Downing R, Kaplan J, Montaner JS,
Hancock J, et al. CD4R T-cell count monitoring does not
accurately identify HIV-infected adults with virologic failure
receiving antiretroviral therapy. J Acquir Immune Defic Syndr
2008; 49:477484.
27. Rawizza HE, Chaplin B, Meloni ST, Eisen G, Rao T, Sankale JL,
et al. Immunologic criteria are poor predictors of virologic
outcome: implications for HIV treatment monitoring in
resource-limited settings. Clin Infect Dis 2011; 53:12831290.
Copyright Lippincott Williams & Wilkins. Unauthorized reproduction of this article is prohibited.
28. Reynolds SJ, Nakigozi G, Newell K, Ndyanabo A, Galiwongo R,
Boaz I, et al. Failure of immunologic criteria to appropriately
identify antiretroviral treatment failure in Uganda. AIDS 2009;
23:697700.
29. van Oosterhout JJG, Brown L, Weigel R, Kumwenda JJ, Mzin-
ganjira D, Saukila N, et al. Diagnosis of antiretroviral therapy
failure in Malawi: poor performance of clinical and immuno-
logical WHO criteria. Trop Med Int Health 2009; 14:856861.
30. Chaiwarith R, Wachirakaphan C, Kotarathitutium W, Prapar-
atanaphan J, Sirisanthan T, Supparatpinyo K. Sensitivity and
specificity of using CD4R measurement and clinical evaluation
to determine antiretroviral treatment failure in Thailand. Int J
Infect Dis 2007; 11:413416.
31. Rewari BB, Bachani D, Rajasekaran S, Deshpande A, Chan PL,
Srikantiah P. Evaluating patients for second-line antiretroviral
therapy in India: the role of targeted viral load testing. J Acquir
Immune Defic Syndr 2010; 55:610614.
32. Barlow-Mosha L, Mudiope P, Massavon W, Bagenda D,
Kbugho E, Etima M, et al. Validation of WHO 2010 immuno-
logic criteria in predicting pediatric first-line antiretroviral
treatment (ART) failure in ART-experienced children in
Uganda: CD4 is a poor surrogate for virologic monitoring
of pediatric ART failure [Abstract THPE062]. 19th Inter-
national AIDS Conference, Washington, DC; 2227 July
2012.
33. Davies MA, Boulle A, Eley B, Moultrie H, Technau K, Rabie H,
et al. Accuracy of immunological criteria for identifying
virological failure in children on antiretroviral therapy: the
IeDEA Southern Africa Collaboration. Trop Med Int Health
2011; 16:13671371.
34. Davies MA, Boulle A, Technau K, Eley B, Moultrie H, Rabie H,
et al. The role of targeted viral load testing in diagnosing
virological failure in children on antiretroviral therapy with
immunological failure. Trop Med Int Health 2012; 17:1366
1390.
35. Westley BP, DeLong AK, Tray CS, Sophearin D, Dufort EM,
Nerrienet E, et al. Prediction of treatment failures using
2010 World Health Organization guidelines is associated
with high misclassification rate and drug resistance among
HIV-infected Cambodian children. Clin Infect Dis 2012; 55:
432440.
36. Bisson GP, Gross R, Strom JB, Rollins C, Bellamy S, Weinstein R,
et al. Diagnostic accuracy of CD4 cell count increase for
virologic response after initiating highly active antiretroviral
therapy. AIDS 2006; 20:16131619.
37. Jourdain G, Ngo-Giang-Huong N, Le Couer S, Traisthit P, et al.
PHPT-3: a randomized clinical trials comparing CD4 vs
viral load ART monitoring/switching strategies in Thailand
[Abstract 44]. Conference on Retroviruses and Opportunistic
Infections, Boston, Massachusetts, 27 February2 March 2011.
38. Kanapathipillai R, McGuire M, Mogha R, Szumilin E,
Heinzelmann A, Pujades-Rodriguez M. Benefit of viral load
testing for confirmation of immunological failure in HIV
patients treated in rural Malawi. Trop Med Int Health 2011;
16:14951500.
39. Koethe JR, Westfall AO, Luhanga DK, Clark GM, Goldman JD,
Mulenga PL, et al. A cluster randomized trial of routine HIV-1
viral load monitoring in Zambia: study design, implementation,
and baseline cohort characteristics. PLoS ONE 2010; 5:
e9680doi: 10.1371/journal.pone.0009680.
40. PENPACT-1 (PENTA 9/PACTG 390) Study Team. First-line anti-
retroviral therapy with a protease inhibitor versus nonnucleoside
reverse transcriptase inhibitor and switch at higher versus low
viral load in HIV-infected children: an open-label, randomised
phase 2/3 trial. Lancet Infect Dis 2011; 111:273284.
41. Siberry GK, Harris DR, Oliveira RH, Krauss MR, Hofer CB,
Tiraboschi AA, et al. Evaluation of viral load thresholds for
predicting new World Health Organization stage 3 and 4 events
in HIV-infected children receiving highly active antiretroviral
therapy. J Acquir Immune Defic Syndr 2012; 60:214218.
42. Badri M, Lawn SD, Wood R. Utility of CD4 cell counts for early
prediction of virological failure during antiretroviral therapy in
a resource-limited setting. BMC Infect Dis 2008; 8:89doi:
10.1186/1471-2334-8-89.
43. Davies MA, Bolton-Moore C, Eley B, Giddy J, Keiser O, Moultrie
H, et al. Predicting 1-year mortality using CD4 percentage and
count in children on ART [Presentation]. 3rd International
Workshop on HIV Pediatrics, Rome, Italy, 1516 July 2011.
 Evaluation of WHO criteria for antiretroviral treatment failure Rutherford et al.
44. Germanaud D, Derache A, Traore M, Madec Y, Toure S, Dicko
F, et al. Level of viral load and antiretroviral resistance after
6 months of nonnucleoside reverse transcriptase inhibitor first-
line therapy in HIV-1-infected children in Mali. J Antimicrob
Chemother 2010; 65:118124.
45. Lynen L, An S, Koole O, Thai S, Ros S, De Munter P, et al. An
algorithm to optimize viral load testing in HIV-positive patients
with suspected first-line antiretroviral therapy failure in
Cambodia. J Acquir Immune Defic Syndr 2009; 52:4048.
46. Lynen L, Thai S, De Munter P, Leang B, Sokkab A, Schrooten W,
et al. The added value of a CD4 count to identify patients eligible
for highly active antiretroviral therapy among HIV-positive adults
in Cambodia. J Acquir Immune Defic Syndr 2006; 42:322324.
47. Castelnuovo B, Sempa J, Kiragga AN, Kamya MR, Manabe YC.
Evaluation of WHO criteria for viral failure in patients
on antiretroviral treatment in resource-limited settings. AIDS
Res Treatment 2011:736938doi: 10.1155/2011/736938.
48. De Luca A, Marazzi MC, Mancinelli S, Ceffa S, Altan AM,
Buonomo E, et al. Prognostic value of virological and immuno-
logical responses after 6 months of antiretroviral treatment
in adults with HIV-1 infection in sub-Saharan Africa. J Acquir
Immune Defic Syndr 2012; 59:236244.
Copyright Lippincott Williams & Wilkins. Unauthorized reproduction of this article is prohibited.
S169
49. Oliveira R, Krauss M, Essama-Bibi S, Hofer C, Harris DR,
Tiraboshci A, et al. Viral load predicts new World Health
Organization stage 3 and 4 events in HIV-infected children
receiving highly active antiretroviral therapy, independent
of CD4 T lymphocyte value. Clin Infect Dis 2010; 51:1325
1333.
50. Kiragga AN, Castelnuovo B, Kamya MR, Moore R, Manabe YC.
Regional differences in predictive accuracy of WHO immu-
nological failure criteria. AIDS 2012; 26:768770.
51. World Health Organization. Consolidated guidelines on the use
of antiretroviral drugs for treating and preventing HIV infection.
Geneva: World Health Organization; 2013, www.who.int/hiv/
pub/guidelines/arv2013. [Accessed 26 January 2014]
52. Department of Health and Human Services. Panel on Antire-
troviral Guidelines for Adults and Adolescents. Guidelines
for the use of antiretroviral agents in HIV-1-infected adults
and adolescents. http://aidsinfo.nih.gov/contentfiles/lvguide
lines/adultandadolescentgl.pdf. [Accessed 26 January 2014]
53. Williams I, Churchill D, Anderson J, Boffito M, Bower M, Cairns
G, et al. British HIV Association guidelines for the treatment of
HIV-1-postiive adults with antiretroviral therapy 2012. HIV
Med 2012; 13 (Suppl 2):185.