Model Selection and Mixed-Eects Modeling

of HIV Infection Dynamics

D. M. Bortz and P. W. Nelson

Mathematical Biology Research Group

Department of Mathematics
University of Michigan
2074 East Hall
525 East University Avenue
Ann Arbor, MI 48109-1043
USA
Abstract
We present an introduction to a model selection methodology and an application to
mathematical models of in vivo HIV infection dynamics. We consider six previously
published deterministic models and compare them with respect to their ability to
represent HIV infected patients undergoing reverse transcriptase mono-therapy. In
the creation of the statistical model, a hierarchical mixed-eects modeling approach
is employed to characterize the inter- and intra-individual variability in the patient
population. We estimate the population parameters in a likelihood function formu-
lation, which is then used to calculate information theory-based model selection
criteria, providing mathematical ranking of each of the models ability to represent
patient data. The parameter ts generated by these models, furthermore, provide
statistical support for the higher viral clearance rate c in Louie et al. (2003). Among
the candidate models, our results suggest which mathematical structures, e.g., linear
versus nonlinear, best describe the data we are modeling and illustrate a framework
for others to consider when modeling infectious diseases.
Key words: Model Selection, Mixed-eects Modeling, HIV, Information Criteria,
Parameter Estimation

Author to whom correspondence should be addressed.

Email addresses: dmbortz@umich.edu (D. M. Bortz), pwn@umich.edu,
pwn@umich.edu (P. W. Nelson).
Preprint submitted to Bulletin of Mathematical Biology August 4, 2005
1 Background
In 1995 a simple model for Human Immunodeciency Virus-1 viral loads in
the blood plasma of infected individuals was presented in Ho et al. (1995)
which revolutionized our understanding of the disease. This work was the
rst to show that the infection pathogenesis was a rapidly varying dynamical
process during which about twelve billion viral particles per day were being
produced in infected individuals. The model used to examine the data from
twenty patients was

V (t) = P cV (t) ,
where

V (t) represents the rate of change (at time t) in the time-dependent
viral concentration V (t), P the daily production rate, and c the viral clearance
rate. By assaying each patients viral loads before receiving anti-viral therapy a
steady state level for the virus was calculated, thus yielding an initial condition
for V (t). Following administration of the protease inhibitor, repeated patient
testing depicted a rapid decline in the total viral loads in the plasma. By
assuming the drug to be completely eective they were then able to identify
a specic value for c, which strongly suggested that incredibly large numbers
of virions were created every day in an HIV infected individual. A concurrent
study by Wei et al. (1995), used a dierent mathematical model, but still led to
a nearly identical conclusion, thus further supporting the high HIV production
claims.
The viral decay process is fundamentally exponential, but explicit models of
this form have a limited capacity for biological interpretation. Thus, the work
in Ho et al. (1995) was followed in 1996 by Perelson et al. (1996) in which
they extended the model to

(t) = kT
0
V
I
(t) T

(t) ,

V
I
(t) = (1 n
p
)NT

cV
I
(t) , (1)

V
NI
(t) = n
p
NT

(t) cV
NI
(t) ,
which included the dynamics of productively infected T cells T

, their rate
of decay , and the biological fact that an anti-viral therapy using a protease
inhibitor leads to the production of non-infectious viral loads V
NI
. Neither
the original model nor the data distinguished between infectious and non-
infectious viral concentrations, and thus system (1) separated the viral con-
centrations into these two compartments to aid in analysis. Clearly no drug is
completely eective, thus the ecacy of the protease inhibitor n
p
is between
zero and one, depending on its potency. If n
p
= 1, as is done in Perelson et al.
2
(1996), then all subsequent virions produced after the initiation of therapy
are non-infectious viral particles and incapable of infecting target T cells T
0
.
The decline observed in the patient data is characterized by the decay rates,
and c, and leads to estimates for the half life of productively infected T
cells and viral particles of two days and eight hours, respectively. Since this
model assumes a perfectly ecacious drug therapy (n
p
= 1), both these esti-
mates should be considered lower bounds. These publications catalyzed a large
number of studies over the next several years, with the more theoretical work
consisting primarily of variations of the model (1). It is this body of research
that we wish to address in this paper.
1.1 Subsequent Development of Mathematical Models of HIV Infection Dy-
namics
As of July 2005, there were over 2400 publications citing Ho et al. (1995) and
1050 citing Perelson et al. (1996). In many of these papers, researchers argued
over the importance of a variety of biological eects as well as for the inclu-
sion or exclusion of the corresponding representations in their mathematical
models. Following the publication of Perelson et al. (1996), additional and/or
alternative compartment formulations were proposed (Callaway and Perelson,
2002; Kramer, 1999; Murray et al., 1998; Nowak et al., 1997; Perelson and
Nelson, 1999; Staord et al., 2000; Wodarz et al., 1999) and the use of de-
lay dierential equations in modeling the eclipse phase was heavily debated
(Grossman et al., 1998; Herz et al., 1996; Lloyd, 2001; Mittler et al., 1998;
Nelson et al., 2000; Nelson and Perelson, 2002). The knowledge gained from
using models of disease pathogenesis has, in many cases, suggested novel de-
sign ideas for treatment strategies as well as laboratory experiments. In the
mid nineties, for example, several publications provided strong support for
the existence of a high rate of HIV replication and clearance in infected in-
dividuals (Ho et al., 1995; Perelson et al., 1996; Wei et al., 1995). It is now
commonly believed that in vivo, on the order of 10
10
virions are created and
then destroyed every day by the immune system (Mittler et al., 1999; Perel-
son et al., 1997; Ramratnam et al., 1999). The high replication rate implies
that the virus has an enormous number of opportunities to mutate and evolve
into a drug resistant strain. A pharmacological mono-therapy will, therefore,
eventually fail since the virus will almost certainly manifest a resistance to
any one drug. To counteract the high mutation rate, the current approach is
to simultaneously administer multiple drugs to HIV infected individuals.
In many of the aforementioned papers, the viral clearance rate was identied
by modeling the disease pathogenesis with a system of deterministic dieren-
tial equations, numerically calculating a solution, and then tting the results
with plasma viral load data (using a ordinary least squares (OLS) approach),
3
e.g., see Perelson et al. (1996, 1997); Ramratnam et al. (1999). Two statistical
issues rarely considered when studying disease pathogenesis using dynamical
systems are the modeling of variability within and between individuals as well
as the estimation of statistical evidence for the superiority of one model over
others. We will employ hierarchical nonlinear mixed-eects (NLME) modeling
approach to address the rst issue and model selection criteria for the second.
To our knowledge, the rst publication to employ NLME and basic model se-
lection in modeling in vivo HIV dynamics was Wu et al. (1998), though they
restricted their models to low-dimension, linear DEs. Aside from Wu et al.
(1998), Wu and Ding (1999), and Wu and Wu (2002b), moreover, model se-
lection methods have experienced limited use in the mathematical modeling
community in general and among HIV dynamical system modelers in partic-
ular.
2 Mixed-Eects Modeling
In the modeling of any physical system, one must choose how to address
the inherent randomness. The variability present in inorganic phenomena fre-
quently leads to singular probability densities and is typically ignored without
consequences. In biological systems, however, there are a multitude of poten-
tial sources of randomness and ignoring their impact could potentially lead
to spurious conclusions. In the development of many of the aforementioned
models, the dynamics of cellular and viral populations inside an individual
patient were implicitly assumed to be deterministic. When the statistical is-
sues were addressed, they were frequently secondary to the main conclusions of
the article.
1
We will employ a (statistically) straightforward characterization
of the inherent within- and between-patient variability. Consequently, it is im-
portant to realize that the subsequent development in this section could have
realistically been performed using alternative distributions (such as Gamma
or exponential) and approximation schemes.
We assign the sources of randomness in two stages, the rst for an individual
and the second for the overall population. For patient i, the log transform of
the n
i
-dimensional vector of viral load measurements v
i
at n
i
time-points is
modeled by
ln v
i
= lnf( +b
i
) +
i
, (2)
where the vector valued function f represents the solution to the dierential
equation, evaluated at the same n
i
timepoints, and
i
is a vector random
variable describing the measurement error in observing the viral loads.
2
The
1
Excluding the publications by H. Wu and V. DeGruttola cited in this article.
2
In a mild abuse of notation, for a vector x = (x
1
, x
2
, . . . , x
n
)
T
, we let ln x =
(ln x
1
, ln x
2
, . . . , ln x
n
)
T
.
4
vector +b
i
consists of parameters from the chosen dierential equation such
as viral clearance rates and CD4 T-cell death rates. For each patient, is
composed of q population parameters and b
i
is a vector of random variables
of perturbations away from , realized by selecting an individual patient i
from the population. This framework is known as a hierarchical nonlinear
mixed-eects model (Davidian and Giltinan, 1995; Pinheiro and Bates, 2000),
wherein is commonly called a xed eect and b
i
is a random eect. In this
paper, we will assume that the measurement errors are independent, identically
distributed (iid) log-normal, i.e.,
i
N(0,
2
I), with common variance
2
,
mitigating heteroscedastic features of the data. To test this assumptions, we
t all patients using a delayed exponential and studied the errors using a
single sample Kolmogorov-Smirnov hypothesis test. The result was that the
null hypothesis (of log-normal errors) cannot be rejected at condence levels
above 70%. On a patient by patient basis, moreover, nine out of ten reported
similar (or better) results, thus giving us further support for our assumption.
With regard to the distribution for the perturbations b
i
, the ten samples
generated in the simple case are, however, not enough to provide strong sup-
port for any distribution. In the HIV literature, several dierent distributions
have been assumed, with the most popular being a normal distribution. In
the absence of further information, we simply assumed b
i
N(0, ) with
variance/covariance matrix (as done in Wu et al. (1998)).
We assume that the individual patient pdfs are independent and consider the
following probability density function (pdf) p as a function of the viral load
data V = {v
1
, v
2
, . . . , v
M
} for all M patients
p(V, , ,
2
) =
M

i=1
p
pat
(v
i
|b
i
, , ,
2
)p
pop
(b
i
|, ,
2
)p
par
(, ,
2
) .
For simplicity, we let be a vector containing , the unique elements of
, and
2
. The rst term in the product, p
pat
(v
i
|b
i
, ), is the pdf of an in-
dividual patient, given perturbations away from the population parameters,
whereas p
pop
(b
i
|) represents the pdf of the perturbations, given population
parameters. We will simply assume the last term in the product p
par
() to be
proportional to one, as while it is possible to assert priors on , , and
2
, it
will distract from the main theme of this article.
To compute a likelihood for , , and
2
, we integrate out the marginal
distribution of the b
i
s
_
M

i=1
p
pat
(v
i
|b
i
, )p
pop
(b
i
|)db
i
., (3)
For convenience in later computations, we designate the penalized nonlinear
5
least squares term as
g(, v
i
, b
i
) = ln v
i
ln f( +b
i
)
2
2
+
2
b
T
i

1
b
i
,
setting up our approach for calculating the likelihood
L(|V) =
||
M/2
(2
2
)
(N+Mq)/2
M

i=1
_
exp
_
g(, v
i
, b
i
)
2
2
_
db
i
, (4)
where N =

M
i=1
n
i
is the total number of observations over all patients and
timepoints and q is the dimension of . The values which maximize (4) are
the Maximum Likelihood Estimators (MLE) of .
The direct evaluation of the multidimensional integral in (4) over the domain
of b
i
is challenging. To address this, we employ Laplaces approximation and
consider a Taylor series expansion of g around the best t MLEs of the per-
turbations of ,

b
i
() = arg min
b
i
g(, v
i
, b
i
)
g(, v
i
, b
i
) g(, v
i
,

b
i
()) + (b
i

b
i
())
T

2
b
g(, v
i
,

b
i
())(b
i

b
i
()) ,
noting both that the gradient of g at

b
i
() will be zero and that

b
i
() is
dependent upon , , and
2
. In other words, we minimize g for the i
th
patient
and then approximate it at the best t

b
i
() using a truncated Taylor series.
If the Hessian of g is diagonalizable, we recognize the integral as a multivariate
Gaussian, readily allowing a computation of an estimate of the integral in (4).
According to nonlinear least squares theory, the Hessian can be approximated
near

b
i
by

2
b
g(, v
i
,

b
i
())

2
b
g(, v
i
,

b
i
())
=
b
f( +

b
i
)
T

b
f( +

b
i
) +
2

1
,
and we shall employ this in our calculations. The advantage of this approach
is that it requires only that
2
b
g (and

2
b
g) be diagonalizable.
Designating as the natural log of the likelihood, our overall goal is thus to
maximize
(|V) =
N
2
ln(2
2
) +
M
2
ln ||

1
2
_
M

i=1
ln

2
b
g(, v
i
,

b
i
())

+
2
M

i=1
g(, v
i
,

b
i
())
_
,
6
over , where for patient i
lnv
i
= ln f( +b
i
) +
i
,
b
i
N(0, ) ,

i
N(0,
2
I) ,
= {, ,
2
} ,

b
i
() = arg min
b
i
g(, v
i
, b
i
, ) ,
and

2
b
g(, v
i
,

b
i
) =
b
f( +

b
i
())
T

b
f( +

b
i
()) +
2

1
.
The value of at its maximized point will subsequently be used in the next
section to calculate a selection criteria for each model.
We have chosen not to focus on the mixed-eects framework, since there is
a precedent for this formulation in HIV modeling Wu et al. (1998); Wu and
Ding (1999); Wu and Wu (2002a). In particular, we direct the interested reader
to the excellent review article (Wu, 2005). We do note, however, that these
publications do not include nonlinear dierential equation models, without
closed form solutions, in their analyses.
3 Model Selection Criteria
It is possible that a candidate model may generate an improved, but over-tted
system, due to increased degrees of freedom in the dierential equation. If the
model choice is based solely on the maximum likelihood (or least squares),
we will over-t the data. It is well known, furthermore, that the solutions
to systems of DEs of dimension three and higher can exhibit topologically
complex and even chaotic behavior. This calls into question the robustness
of rm conclusions based upon best least squares t parameters in DEs of
modest to high dimension. In the information theory and statistical inference
literature, there exist methodologies for selecting the best model from several
candidate models. From the perspective of applied mathematics, these infor-
mation criteria (IC) are measures of parametric complexity. While they do
not directly address any questions concerning the topological complexity of
the solution manifolds for the candidate DEs, we can use them to provide
statistical evidence supporting one model over others.
While diering in the derivation details, all of the considered criteria involve a
goodness-of-t term as well as a term measuring the complexity of the model.
It is therefore important to understand the utility of these criteria as well
as their limitations. Specically, since each of the three criteria we consider
7
employ dierent representations of complexity, each will rank the superiority
of candidate models according to dierent priorities.
3.1 Akaike Information Criterion
If p

is a probability density function (pdf) representing the reality of some

phenomena and p represents the pdf of a given candidate model, one way to
quantify the discrepancy between p and p

is by using the Kullback-Leibler

(KL) distance (Kullback and Leibler, 1951)
I(p

, p) = E
p
[ln(p

/p)] =
_

ln
_
p

(x)
p(x)
_
p

(x) dx,
where is the set of all realizable values for x (viral loads in our case). Clearly
p

and p need not be from the same class of functions, but we will search for
a parametrization for p, that yields a pdf p(|) as close as possible to p

(as measured by I). By assuming that a minimum for I(p

, p(|)) exists and

asymptotically expanding I around it, Akaike (1973, 1974, 1977) was able to
compute an expected estimate of I(p

, p) up to a data-dependent, additive
constant. The non-constant part of this estimate is the
Akaike Information Criterion (AIC) = 2(

|V) + 2Q, (5)

where

is the vector of Q best t Maximum Likelihood Estimator (MLE)
parameters and is the log-likelihood function, conditioned on the given data
V.
3
Since the criteria estimates the KL distance up to an additive constant,
this allows relative ranking of a set of candidate models, when all are t to
the same set of data.
Later developments and renements of the AIC include the small sample size
AIC (Hurvich and Tsai, 1989)
AIC
C
= 2(

|V) + 2Q
_
N
N Q1
_
,
and Takeuchi Information Criteria (TIC) (Takeuchi, 1976)
TIC = 2(

|V) + 2tr
_
J(

)I(

)
1
_
. (6)
Note that the trace term in the TIC contains
I(

) = E
p

2
(

|)

T
_
,
3
For the HIV data, V = {v
1
, v
2
, . . . , v
M
} and

consists of , the unique elements
of , and
2
for a total of q(q + 3)/2 + 1 = Q MLE parameters.
8
and
J(

) = E
p

_
_
_

, )
_ _

, )
_
T
_
_
,
which are the Hessian and outer product forms of the inverse Fisher Informa-
tion Matrix (IFIM), respectively.
There are, however, disadvantages to each of these criteria. The AIC is known
to over-t data and conventional wisdom suggests that we have enough obser-
vations to make the small sample correction insignicant.
4
In both of these
criteria, the model complexity measure ascribes parsimony to models with a
small number of parameters. While the TIC does include sensitivities, it is
also asymptotically unbiased, with the second term converging to 2Q in the
limit of large data. It is, moreover, also well-known for its volatility in prac-
tice (Shibata, 1989), and as depicted in the Section 5, our results illustrate
this behavior. We would prefer our metric for complexity to somehow reect
the interdependence between parameters and these criteria are, therefore, not
ideally suited for our purposes.
3.2 Information Complexity (ICOMP)
Under the assumption of a multivariate Gaussian density for both the statisti-
cal model p and reality p

, Bozdogan (1988) proposed Information Complexity

(ICOMP)
2(

|V) + Qln(

g
) , (7)
as a model selection criteria, where
a
and
g
are the algebraic and geometric
means of the eigenvalues of the variance/covariance matrix of the MLE

,
respectively. The basis for this criteria lies in Shannons concept of channel
capacity (Shannon, 1948) also known as the mutual information between the
parameters. Given the true density p

, the rst term in (7) is an unbiased

estimator of the non-constant part of I(p

, p), i.e., E
p
[ln p], and as in other
criteria, reects the goodness of t of a candidate model. The second term is
derived by considering I(

Q
i=1
p
i
, p) where the p
i
s are the marginal densities of
p with respect to the i
th
parameter. The computation of this distance reduces
to
I(
Q

i=1
p
i
, p) =
1
2
Q

i=1
ln(s
ii
)
1
2
ln |(

| , (8)
4
Accepted practice is to use the AIC
C
when observations/parameters is greater
than 40 (Burnham and Anderson, 2002). As expected, the dierences between the
AIC
C
and AIC values for our models were insignicant (results not presented).
9
where (

) is the variance/covariance matrix for the Gaussian p and s

ij
is
the (i, j)
th
element of (

) (van Emden, 1971, page 61). From a coding the-

ory perspective, one could think of (8) as the actual rate of transmission of a
signals between the random variables, i.e., and the elements of b
i
(Shannon,
1948, pages 20-2). The capacity of a noisy channel would then be calculated
by maximizing (8) over all possible sources. From a model selection perspec-
tive, a complex model would be one in which signals are accurately passed
between parameters, and several researchers, including van Emden (1971) and
Bozdogan (1988), have advocated this philosophy. Maximizing over sources is
analogous to maximizing (8) over all possible orthonormal transformations of
the coordinate axis of and b
i
. Since the eigenvalues of (

) are invariant
under similarity transformations and (

) is symmetric positive denite, the

maximal transform rotates the axes so as to equalize the s
ii
, yielding
Q
2
ln
_
tr((

))
Q
_

1
2
ln |(

)| , (9)
as a measure of the complexity. This transformation can be explicitly con-
structed employing Jacobi rotations to compute a diagonalization of result-
ing in the algebraic mean of the eigenvalues of on the diagonal (see van
Emden (1971, page 65) or Trefethen and Bau III (1997, pages 225-7)). Lastly,
it is known that asymptotically = I
1
, and thus we can compute an estimate
of the ICOMP by replacing (

) with the IFIM

ICOMP(IFIM) = 2(

|V) + Qln
_
tr
_
I(

)
1
__
ln

I(

)
1

.
A helpful interpretation of (9) is revealed by recalling that for i, j = 1, . . . , Q,
the eigenvalues {
i
}
Q
i=1
of (

) will be located in the Gerschgorin discs of

radius

i=j
|s
ij
|, centered at s
ii
. If we let e
i
be such that for each i,
i
= s
ii
+e
i
where |e
i
|

i=j
|s
ij
|, Equation (9) then equals
Q
2
ln
_
_
_
1
Q

Q
i=1

i
_

Q
i=1

i
_
1/Q
_
_
_ =
Q
2
ln
_
_
_
1
Q

Q
i=1
s
ii
_

Q
i=1
(s
ii
+ e
i
)
_
1/Q
_
_
_
=
Q
2
ln
_
_
_
1
Q

Q
i=1
s
ii
_

Q
i=1
s
ii
+

Q
i=1
_
O(e
i
)

j=i
s
jj
__
1/Q
_
_
_ ,
=
Q
2
ln
_
s
a
s
g
_

1
2
ln
_
_
1 +
Q

i=1
_
_
O(e
i
)

j=i
s
jj
s
ii
_
_
_
_
,
illustrating how the ICOMP ascribes simplicity to a model with a small num-
ber of parameters with small covariances and tight clustering of variances
(s
a
s
g
).
10
We desire a parametrically parsimonious model that not only ts the data
well, but also ascribes each parameter to a specic mathematical feature of
the data, and of the three criteria considered, the ICOMP is the criteria best
suited to our application.
3.3 Discussion
In the computation of the above criteria, I(

) and J(

) must be estimated.
The matrix I is similar to the Fisher Information matrix, but in our case, the
expectation is taken with respect to reality p

and not the candidate model p.

We will employ the empirical Hessian as an approximation
I(

)

I(

) =

2
(

|V)

T
,
and will estimate the matrix J by assuming independent random samples,
allowing the approximation
J(

)

J(

) =
N

j=1
_
_
_

, V
j
)
_ _

V
j
)
_
T
_
_
,
where j indexes over all patients and measurements.
The computation of these estimates involves calculating the sensitivity of the
viral compartments to the constitutive parameters in the dierential equation,
i.e., the derivative of state variables with respect to some chosen parameter.
These parameter sensitivities are actually solutions to sensitivity equations,
and for illustrations of derivations and analysis of sensitivity equations in the
context of HIV infection dynamics, see Bortz and Nelson (2004) or Banks
and Bortz (2005b). In all models except for the delay dierential equation,
we employed the automatic dierential software ADOL-C (Griewank et al.,
1996) to calculate both the gradients and the Hessians. The accuracy of the
elements in these matrices, therefore, is restricted only by the accuracy of the
algorithm chosen to numerically simulate a solution to the chosen dierential
equation.
As is illustrated in Banks et al. (2003) and Ciupe et al. (2005), a more conven-
tional approach to the model selection is to use the least squares functional to
calculate the statistical signicance of the introduction of more complexity into
a candidate model. The application of this type of traditional statistical anal-
ysis in the context of DEs was originally proposed in Banks and Fitzpatrick
(1990). A related technique involving statistical testing of likelihood ratios is
employed by Gorne et al. (2003) in the context of modeling B lymphocyte
development, to provide evidence for phenotypic reux between B-cells in dif-
11
ferent stages of development. Both of these techniques are, however, restricted
to a null/alternative hypothesis framework which involves binary comparisons
between nested models (where one model is a simplication of the other). Since
IC-based approaches are not restricted in this manner, they have a distinct
advantage when comparing multiple models from dierent schools of thought
and with dierent mathematical representations.
There are also manifest well-posedness questions regarding the model selec-
tion methodology we have employed. The identiability of parameters, such
as clearance and death rates is not obvious and is the focus of Sonday and
Nelson (2005). We restricted our collection of DE models to published ones
with previously proven existence and uniqueness results, and with a numer-
ical approximation f
N
to an actual solution f, with index of approximation
N, we assume that as N , f
N
f. While there are issues concerning
control of error propagation into the likelihood calculation, the well-posedness
of the forward problem is easily shown (Bortz, 2005). Since we are using a
numerical approximation to the MLE parameters

N
, the likelihood L
N
, and
the selection criteria IC
N
, however, the well-posedness of the inverse problem
is not clear. With the absence of identiability results, in order to carefully
answer the question of how (or if) IC
N
IC as N , the topology of the
parameter space will likely need to be reformulated using the Prohorov metric
as was done in Banks and Bortz (2005a). This endeavor and results concerning
the numerical stability of the likelihood approximation scheme are the focus
of Bortz (2005).
Given multiple models which t the data equally well, we can therefore, calcu-
late their IC values and rank them based on ability to encapsulate the reality of
the data. While our primary focus is to illustrate the utility of the information
theory-based model selection framework, there are other possible choices for
criteria including the Schwarz Information Criteria (SIC) (Schwarz, 1978)
5
,
the Minimum Description Length (MDL) based Stochastic Complexity (SC)
(Rissanen, 1989), and the Kullback Information Criteria (KIC) (Kim and Ca-
vanaugh, 2005) For a survey of Akaike, MDL, and Bayesian based model selec-
tion, we direct the reader to Burnham and Anderson (2002), Grunwald et al.
(2005), and Kass and Raftery (1995), respectively.
4 HIV Model Comparison
We will now utilize the mathematical theory presented in the last section to
study a collection of models for HIV pathogenesis during anti-viral drug ther-
apy. We consider the data from an experiment reported in Louie et al. (2003)
5
The SIC is also known as the Bayesian Information Criteria (BIC).
12
in which ten HIV infected patients were placed on a reverse transcriptase (RT)
inhibitor mono-therapy (Tenofovir). Measurements of their plasma HIV RNA
concentrations were recorded over three weeks and it is the rst two weeks of
this data which we propose to t.
There are many variations to (1) which could potentially be used to model
the infection dynamics and we have chosen in this section to examine and
compare ve published variations to this model. Any one of the alternative
models could be a better representation of the actual dynamics, and we are
interested in the one which best matches the dynamics as well as the statistics.
Since the experiment under consideration consists of an RT monotherapy, the
rst model we consider is

(t) = (1 n
rt
)kT
0
V (t) T

(t) ,

V (t) = NT

(t) cV (t) , (10)

where n
rt
is the ecacy of the RT inhibitor and N is the average number of
virions produced per productively infected T cell over its lifetime. We assume
that the viral concentration prior to therapy is at a steady state, and adjust
k accordingly (see Appendix A.1). As in other studies, we will focus on iden-
tifying the tted values of the c and parameters and assume known values
the remaining constants (except for the delay equation, where we also t the
delay parameter).
Our second model extends (10) to allow for a time-varying target cell pop-
ulation. It was previously assumed that the uninfected T-cell population T
0
remained constant during the rst week following the initiation of drug ther-
apy. When measuring viral loads over longer periods of time, however, this
assumption is not reasonable and the following model

(t) = (1 n
rt
)kV (t)T(t) T

(t) ,

V (t) = NT

(t) cV (t) , (11)

T(t) = S + pT(t)
_
1
T(t) + T

(t)
T
max
_
d
T
T(t) (1 n
rt
)kV (t)T(t) ,
allows the T compartment to vary in time. See Appendix A.2 for the values
of T

0
, k, and S used to ensure that

T(0) =

T

(0) =

V (0) = 0. Note that a
mathematical analysis of this system with and without the density dependent
growth term, was presented in Perelson and Nelson (1999).
Our third and fourth models retain the constant target cell population as-
sumption and instead address the eects of modeling latently infected T-cells
L. As in Perelson et al. (1997), we assume that upon infection, a certain frac-
tion f of target cells initially become latently infected and then switch to
productively infected cells T

with rate k
2
. These latently infected cells are
13
also less detectable by the immune system than productively infected T cells
and hence have a lower death rate
L
(Perelson, 2004), which we will t along
with and c. The cells in the L compartment are not considered to be true
latently infected cells and are now interpreted as simply infected cells, not
yet producing virus. For the third model

(t) = T

(t) + k
2
L(t) + (1 f)(1 n
rt
)kT
0
V (t) ,

V (t) = NT

(t) cV (t) , (12)

L(t) = f(1 n
rt
)kT
0
V (t) (
L
+ k
2
)L(t) ,
we also assume an initial steady state condition, reected in the conditions for
T

0
, k, and L
0
in Appendix A.3.
It is also possible that all infected cells must pass through the non-productive
stage (f = 1), and to address this possibility, we consider for our fourth model

(t) = T

(t) + k
2
L(t) ,

V (t) = NT

(t) cV (t) , (13)

L(t) = (1 n
rt
)kT
0
V (t) (
L
+ k
2
)L(t) ,
with conditions for T

0
, k, and L
0
in Appendix A.4.
Our fth model

(t) = T

(t) + k
2
L(t) ,

V (t) = NT

(t) cV (t) kT(t)V (t) ,

T(t) = S + pT(t)
_
1
T(t) + T

(t) + L(t)
T
max
_
d
T
T(t) (1 n
rt
)kV (t)T(t) ,
(14)

L(t) = (1 n
rt
)kT(t)V (t) (
L
+ k
2
)L(t) ,
with conditions for T

0
, k, L
0
, and S in Appendix A.5, is based on a sys-
tem originally presented in Perelson et al. (1993) and describes the dynamics
for both uninfected target cells T and latently infected cells L. This model,
furthermore, includes a nonlinear mass-action term in the

V (t) equation to
account for the loss of the virion upon infection of a target cell (kT(t)V (t)).
As mentioned previously, many researchers have argued for the inclusion of
an intra-cellular time delay between viral infection and production. Our last
dierential equation (DE)-based model is
14
Parameter : Value Description : Source
n
rt
: 0.8 RT ecacy : Nelson et al. (2001)
p : 0.03 (hr
1
) T-cell growth rate : Perelson et al. (1993)
T
0
: from data (cells/mm
3
) Initial target cell population : Louie et al. (2003)
N : 480 (virionmm
3
/(cellml)) Virions per lysing cell : Perelson and Nelson (1999)
V
0
: from data (virions/ml) Initial viral load : Louie et al. (2003)
d
T
: 0.02(hr
1
) Natural T-cell death rate : Perelson et al. (1993)
f : 0.03 Ecacy of latent infection : Perelson et al. (1997)
k
2
: 0.01 (hr
1
) Latent cell activation rate : Perelson et al. (1997)
T
max
: 1500 (mm
3
) Maximum T-Cell density : Perelson et al. (1993)
Table 1
Model Parameters.

(t) = (1 n
rt
)kT(t)V (t ) T

(t) ,

V (t) = NT

(t) cV (t) , (15)

T(t) = S d
T
T(t) (1 n
rt
)kV (t)T(t) ,
with the conditions on T

0
, k, and S dened in Appendix A.6. A similar model
was originally presented in Nelson et al. (2001) and for a full mathematical
analysis of delay dierential equation (DDE) models of HIV infection dynam-
ics, see Nelson and Perelson (2002).
5 Methods
We maximized the likelihood of each model by optimizing over the infected cell
death rates, viral clearance rates, and eclipse phase delays from each model
as well as the variances and covariances of the parameters across the patient
population. Table 1 summarizes the parameters used for all patients. We chose
to focus on the viral clearance rates and the T-cell decay rates and, therefore,
assumed constant n
rt
and N across all patients. There is evidence that these
parameters could vary across patients, but we do not address this issue here.
A sequential quadratic programming method with inequality constraints, em-
ploying a BFGS Hessian update, was used to t the population parameters
in the likelihood function. A trust-region optimization method, also with in-
equality constraints, was used to t both the parameters for individual patients
b
i
. To ensure positive deniteness of , a nonlinear constraint was employed
15
model c
L
(

) ICOMP(IFIM) TIC AIC

(10) 13.7 0.35 - - -147.5 315.6 338.4 307.0
(11) 31.2 0.35 - - -137.3 271.4 20939 286.5
(12) 25.3 0.36 0.11 - -131.2 354.7 971.2 282.4
(13) 24.7 3.6 0.38 - -132.1 313.5 402.5 284.2
(14) 20.7 0.73 0.67 - -104.6 329.4 303.5 229.2
(15) 16.7 0.36 - 0.7 -108.9 303.6 17305 304.5
Table 2
Sample mean values for the best ts for the parameters, ICOMP(IFIM), TIC, AIC
C
,
and AIC.
which penalized negative eigenvalues as well as restricted correlations to be
less than one in magnitude. Since there could be multiple local miminima for
the likelihood and NLME-based methods are know for sensitive dependence
upon initial estimates, we were careful in our choice of initial values for the
optimization. These values were computed from least squares ts (and the
sample statistics of those ts) of each deterministic model to each individual
patient. The initial estimate of
2
was calculated using the mean of the resid-
ual from tting a simple exponential decay to patient data. A termination
tolerance of 1e-3 was applied to the parameters as well as the cost functionals
of both optimization problems, yielding (for the majority of models) param-
eter values within published condence intervals. Only gradient information
was provided to the likelihood maximization, while both gradient and Hessian
information was available for ts to the individual patients.
While there exists software such as NONMEM and the S-plus function nlme
which perform the same type of likelihood maximization as we do, none utilize
advanced ODE or DDE solver or automatic dierentiation software. The solu-
tions to the ODE models were approximated using the variable-order, variable-
step multistep backwards dierentiation formula in the CVODE solver from
the SUNDIALS package (Hindmarsh et al., 2005). Solutions to the DDE model
were approximated using an explicit Runge-Kutta method of order (4)5 (with
stepsize control) due to Dormand & Prince, and described in Hairer et al.
(1993). At each timestep, the absolute error was 1e-9 while the relative error
was 1e-6 for all systems. First and second order sensitivity equations were
calculated using the ADOL-C automatic dierentiation software (Griewank
et al., 1996) for the ODEs and by hand for the DDE. The resulting equations
were solved in the same manner as the original equations.
The population parameters are summarized in Table 2, while the variances
and correlations for those parameters are in Tables 3 and 4, respectively.
16
model var(c) var() var(
L
) var()
(10) 54.8 0.013 - -
(11) 116.2 0.012 - -
(12) 0.41 0.02 0.005 -
(13) 7.23 1.83 0.024 -
(14) 5.3 0.13 0.005 -
(15) 1.2 0.01 - 1.3
Table 3
Sample variances for maximum likelihood best t values of the parameters
model corr(c, ) corr(c,
L
) corr(c, ) corr(,
L
) corr(, )
(10) 0.17 - - - -
(11) 0.3 - - - -
(12) -0.32 0.34 - -0.48 -
(13) 1.00 -0.66 - -0.67 -
(14) 0.09 0.03 - -0.26 -
(15) 0.23 - -0.48 - 0.49
Table 4
Sample correlations for maximum likelihood best t values of the parameters.
6 Results
In Table 2, our chosen criteria selects model (11) as the superior model among
these six candidate models. Recall that model (11) improves upon the simple
two compartment model of Perelson and Ho by introducing an uninfected
T-cell compartment. Admittedly, this compartment does not vary much over
the course of the experiment, yet this additional model feature is statistically
supported by the data. While the introduction of latent cell dynamics may be
biologically reasonable, the available data does not provide statistical support
for this claim. As expected, the TIC values exhibit quite a wide range of
variation. The AIC and AIC
C
(not reported here due to similarity to AIC)
criteria select model (14) as being the best, however, this ranking is driven
by the reduction in the likelihood for model (11) (note the similarity to the
likelihood column).
In Table 3, the variances for each parameter are highly dependent upon the
chosen model. In the delay model (12), for example, the variances are relatively
small. For our best (according to ICOMP(IFIM)) model (11), the variances
17
are quite large in the c estimate and relatively small in the estimate. This
suggests that while immune system ecacy may vary across patients, HIV in-
fected CD4 T-cell death rates are fairly consistent. When evaluated at

, the
Hessian of is negative denite and the gradient of is smaller than 1e-3 (in
Euclidean norm). Such a large variance, however, could suggest a misspecica-
tion in the statistical model. Accordingly, our future studies include strategies
for model diagnostics more sophisticated than rst and second order numerical
convergence criteria (Bortz, 2005).
In Table 4, we report the correlations for the parameters in all models, which
suggests widely diering correlations based on the model chosen, giving even
more credence to our claim that models must include dynamic and stochastic
components. Scientically, therefore, we focus on the correlation reported for
model (11), which suggests a weak, but positive, correlation between viral
clearance and infected cell death rate.
In Bortz and Nelson (2004), we presented a sensitivity analysis methodology
which included a principle component based analysis. In the diagonalization
of the sensitivity matrix, the eigenvector associated with the smallest (in mag-
nitude) eigenvalue represented a direction (locally) along which (c, ) values
generated similar solutions. As Figure 4 on page 1019 in Bortz and Nelson
(2004) illustrated, there is a wide range for c and a small range for over
which solutions are similar. Even though the patients in the dataset used in
that paper were undergoing protease inhibitor therapy, it is reassuring to know
that dierent analyses generate similar conclusions (both coinciding with mi-
crobiological understanding).
If we also consider the second best model (the delay model), we can comment
on a common modeling pitfall in which terms are added to a model without
fully realizing the mathematical and statistical implications. The inclusion
of a delay term is biological justied and does yield a best t with a lower
least squares value (not presented). Without accounting for variation across
patients, however, the tted rate of decay for the productively infected cells
is notably higher, and many researchers have addressed the implications of
this increase. For example, Grossman et al. (1998) argue that including a
delay in the model for the death of infected cells leads to dierent conclusions
regarding residual transmission of infection during antiviral pharmaceutical
therapy, while Lloyd (2001) argues that an absence of delays in the model
leads to grossly optimistic conclusions about treatment ecacy. Based on the
mixed eects/model selection methodology we must, however, be cautious of
claims and conclusions concerning the higher .
18
7 Conclusions
Information theory-based model selection is a powerful tool for comparing
models and modeling mechanisms. We presented an introduction to this method-
ology as well as three criteria. We discussed the relative merits of each and
chose one well suited to studying models of HIV infection dynamics during
therapy. The calculation of the model selection criteria for these nonlinear
models is nontrivial, but facilitated by the use of modern dierential equation
solver, optimization, and automatic dierentiation software.
We studied six models, previously published in the literature, and compared
the ability of each model to characterize the random as well as deterministic
features of HIV patient data from Louie et al. (2003). We concluded that it is
crucial to include target cell dynamics with density dependent growth (as in
(11)), but that a latent cell compartment is not needed over this shorter time
scale. Our results also strongly support higher estimates for the viral clearance
rate than have been used in previous publications.
It is not sucient to rely solely on biological intuition and goodness of t
criteria to develop complex models. When considering several valid, candi-
dates, the model selection approach presented here provides a framework for
constructing models supported by statistical evidence.
Acknowledgements
The research of P. W. Nelson, Ph.D., is supported in part by a Career Award
at the Scientic Interface from the Burroughs Wellcome Fund. D. M. Bortz,
Ph.D., is also supported in part from the Burroughs Wellcome Fund.
The authors wish the thank the anonymous referees for helpful insight and ex-
cellent suggestions. The authors also wish to thank A. S. Perelson (Los Alamos
National Laboratory) and K. A. Shedden (University of Michigan Statistics)
for their comments on an earlier draft of this manuscript and D. D. Ho and
M. Markowitz (Aaron Diamond Aids Research Center) for generously sharing
their data. The authors also wish to thank C. Ludwig (Technische Universitat
M unchen) for assistance in calling high performance DE solver software from
within Matlab, A. Verschild (RWTH Aachen University) for the use of the
ADiMat automatic dierentiation software, and R. Serban (Lawrence Liver-
more National Laboratory) for assistance with the SUNDIALS software.
19
A Model Parameter denitions
A.1 Model (10)
T

0
=
cV
0
N
; k =
c
NT
0
A.2 Model (11)
T

0
=
cV
0
NT
0
; k =
c
NT
0
; S =
NpT
2
0
+(pcV
0
+(d
T
p)NTmax)T
0
+V
0
Tmaxc
NTmax
A.3 Model (12)
T

0
=
cV
0
N
; L
0
=
fcV
0
N(
L
+k
2
f
L
)
; k =
c(
L
+k
2
)
T
0
N(
L
+k
2
f
L
)
A.4 Model (13)
T

0
=
cV
0
N
; L
0
=
cV
0
k
2
N
; k =

L
+k
2
T
0
k
2
N
A.5 Model (14)
T

0
=
cV
0
k
2
(Nk
2

L
k
2
)
; k =
c(
L
+k
2
)
T
0
(k
2
(N1)
L
)
; L
0
=
cV
0
k
2
(N1)
L
;
S =
1
T
max
(k
2
(N 1)
L
)
{((N 1)k
2

L
)pT
2
0
+ V
0
T
max
c(
L
+ k
2
)
+((N 1)(d
T
p)(k
2

L
)T
max
+ ( + k
2
)pcV
0
)T
0
}
20
A.6 Model (15)
T

0
=
cV
0
N
; k =
c
T
0
N
; S = d
T
T
0
+
cV
0
N
References
Akaike, H., 1973. Information theory as an extension of the maximum like-
lihood principle. In: Petrov, B. N., Csaki, F. (Eds.), Second International
Symposium on Information Theory. Akademiai Kiado, Budapest, Hungary,
pp. 267281.
Akaike, H., 1974. A new look at the statistical model identication. IEEE
Transactions on Automatic Control 19, 716723.
Akaike, H., 1977. On entropy maximization principle. In: Krishnaiah, P. R.
(Ed.), Applications of Statistics. North Holland Publishing Company, Am-
sterdam, The Netherlands, pp. 2741.
Banks, H. T., Bortz, D. M., Apr. 2005a. Inverse problems for a class of measure
dependent dynamical systems. Journal of Inverse and Ill-Posed Problems
13 (2).
Banks, H. T., Bortz, D. M., Jun. 2005b. A parameter sensitivity methodol-
ogy in the context of HIV delay equation models. Journal of Mathematical
Biology 50 (6), 607625.
Banks, H. T., Bortz, D. M., Holte, S. E., 2003. Incorporation of variability
into the mathematical modeling of viral delays in HIV infection dynamics.
Mathematical Biosciences 183 (1), 6391.
Banks, H. T., Fitzpatrick, B. G., 1990. Statistical methods for model compar-
ison in parameter estimation problems for distributed systems. Journal of
Mathematical Biology 28, 501527.
Bortz, D. M., 2005. Accurate model selection computations. In preparation.
Bortz, D. M., Nelson, P. W., 2004. Sensitivity analysis of nonlinear lumped
parameter models of HIV infection dynamics. Bulletin of Mathematical Bi-
ology 66 (5), 10091026.
Bozdogan, H., 1988. ICOMP: A new model-selection criterion. In: Bock, H. H.
(Ed.), Classication and Related Methods of Data Analysis. North Holland
Publishing Company, Amsterdam, The Netherlands, pp. 599608.
Burnham, K. P., Anderson, D. R., 2002. Model Selection and Multimodel Infer-
ence: A Practical Information-Theoretic Approach, 2nd Edition. Springer-
Verlag, New York, NY.
Callaway, D. S., Perelson, A. S., 2002. HIV-1 infection and low steady state
viral loads. Bulletin of Mathematical Biology 64, 2964.
Ciupe, S., de Bivort, B. L., Bortz, D. M., Nelson, P. W., 2005. Estimating
21
kinetic parameters from HIV primary infection data through the eyes of
three dierent mathematical models. Mathematical Biosciences. To appear.
Davidian, M., Giltinan, D. M., 1995. Nonlinear Models for Repeated Measure-
ment Data. No. 62 in Monographs on Statistics and Applied Probability.
Chapman and Hall/CRC, Boca Raton, Florida.
Gorne, M., Freedman, L., Shahaf, G., R.Mehr, 2003. Maximum likelihood
estimator and likelihood ratio test in complex models: An application to B
lymphocyte development. Bulletin of Mathematical Biology 65, 11311139.
Griewank, A., Juedes, D., Utke, J., 1996. ADOL-C: A package for the auto-
matic dierentiation of algorithms written in C/C++. ACM Transactions
on Mathematical Software 22 (2), 131167.
Grossman, Z., Feinberg, M., Kuznetsov, V., Dimitrov, D., Paul, W., 1998. HIV
infection: how eective is drug combination treatment? Immunology Today
19, 528532.
Grunwald, P. D., Myung, I. J., Pitt, M. A. (Eds.), 2005. Advances in Min-
imum Description Length : Theory and Applications. Neural Information
Processing. MIT Press.
Hairer, E., Norsett, S. P., Wanner, G., 1993. Solving Ordinary Dierential
Equations I. Nonsti Problems, 2nd Edition. Series in Computational Math-
ematics. Springer-Verlag.
Herz, A. V. M., Bonhoeer, S., Anderson, R. M., May, R. M., Nowak, M. A.,
1996. Viral dynamics in vivo: limitations on estimates of intracellular delay
and virus decay. Proceedings of the National Academy of Sciences, USA 93,
72477251.
Hindmarsh, A. C., Brown, P. N., Grant, K. E., Lee, S. L., Serban, R., Shu-
maker, D. E., Woodward, C. S., Sep. 2005. SUNDIALS: Suite of nonlinear
and dierential/algebraic equation solvers. ACM Transactions on Mathe-
matical Software 31 (3).
Ho, D. D., Neumann, A. U., Perelson, A. S., Chen, W., Leonard, J. M.,
Markowitz, M., Jan. 1995. Rapid turnover of plasma virions and CD4 lym-
phocytes in HIV-1 infection. Nature 373 (6510), 123126.
Hurvich, C. M., Tsai, C.-L., 1989. Regression and time series model selection
in small samples. Biometrika 76, 271293.
Kass, R. E., Raftery, A. E., Jun. 1995. Bayes factors. Journal of the American
Statistical Association 90 (430), 773795.
Kim, H.-J., Cavanaugh, J. E., 2005. Model selection criteria based on kullback
information measures for nonlinear regression. Journal of Statistics Planning
and Inference. To appear.
Kramer, I., 1999. Modeling the dynamical impact of HIV on the immune
system: Viral clearance, infection, and AIDS. Mathematical and Computer
Modelling 29, 95112.
Kullback, S., Leibler, R. A., 1951. On information and suciency. Annals of
Mathematical Statistics 22, 7986.
Lloyd, A. L., 2001. The dependence of viral parameter estimates on the asumed
viral load life cycle: limitations of studies of viral load data. Proceedings of
22
the Royal Society of London Series B 268, 847854.
Louie, M., Hogan, C., Hurley, A., Simon, V., Chung, C., Padte, N., Lamy, P.,
Flaherty, J., Coakley, D., Mascio, M. D., Perelson, A. S., Markowitz, M.,
2003. Determining the antiviral activity of tenofovir disoproxil fumarate
in treatment-naive chronically HIV-1-infected individuals. AIDS 17, 1151
1156.
Mittler, J. E., Markowitz, M., Ho, D. D., Perelson, A. S., 1999. Improved
estimates for HIV-1 clearance rate and intracellular delay. AIDS 13, 1415
1417.
Mittler, J. E., Sulzer, B., Neumann, A. U., Perelson, A. S., 1998. Inuence
of delayed viral production on viral dynamics in HIV-1 infected patients.
Mathematical Biosciences 152, 143163.
Murray, J. M., Kaufmann, G., Kelleher, A. D., Cooper, D. A., 1998. A model
of primary HIV-1 infection. Mathematical Biosciences 154, 5785.
Nelson, P. W., Mittler, J. E., Perelson, A. S., 2001. Eect of drug ecacy and
the eclipse phase of the viral life cycle on estimates of HIV viral dynamic
parameters. Journal of Acquired Immune Deciency Syndromes 26, 405
412.
Nelson, P. W., Murray, J. D., Perelson, A. S., 2000. A model of HIV-1 patho-
genesis that includes an intracellular delay. Mathematical Biosciences 163,
201215.
Nelson, P. W., Perelson, A. S., 2002. Mathematical analysis of delay dierential
equation models of HIV-1 infection. Mathematical Biosciences 179, 7394.
Nowak, M. A., Bonhoeer, S., Shaw, G. M., May, R. M., 1997. Anti-viral drug
treatment: Dynamics of resistance in free virus and infected cell populations.
Journal of Theoretical Biology 184, 203217.
Perelson, A. S., 2004. Personal communication.
Perelson, A. S., Essunger, P., Cao, Y., Vesanen, M., Hurley, A., Saksela,
K., Markowitz, M., Ho, D. D., May 1997. Decay characteristics of HIV-
1-infected compartments during combination therapy. Nature 387 (6629),
188191.
Perelson, A. S., Kirschner, D. E., de Boer, R., 1993. Dynamics of HIV infection
of CD4+ T-cells. Mathematical Biosciences 114, 81125.
Perelson, A. S., Nelson, P. W., 1999. Mathematical analysis of HIV-1 dynamics
in vivo. SIAM Review 41, 344.
Perelson, A. S., Neumann, A. U., Markowitz, M., Leonard, J. M., Ho, D. D.,
1996. HIV-1 dynamics in vivo: virion clearance rate, infected cell life-span,
and viral generation time. Science 271, 15821586.
Pinheiro, J. C., Bates, D. M., 2000. Mixed-Eects Models in S and S-PLUS.
Statistics and Computing. Springer-Verlag, New York, NY.
Ramratnam, B., Bonhoeer, S., Binley, J., Hurley, A., Zhang, L., Mittler,
J. E., Markowitz, M., Moore, J. P., Perelson, A. S., Ho, D. D., 1999. Rapid
production and clearance of HIV-1 and hepatitis C virus assessed by large
volume plasma apheresis. The Lancet 354, 17821785.
Rissanen, J., 1989. Stochastic Complexity and Statistical Inquiry. Vol. 15 of
23
Series in Computer Science. World Scientic, Singapore.
Schwarz, G., 1978. Estimating the dimension of a model. The Annals of Statis-
tics 6, 461464.
Shannon, C. E., 1948. A mathematical theory of communication. Bell System
Technical Journal 27, 37942 and 623656.
Shibata, R., 1989. Statistical aspects of model selection. In: Willems, J. C.
(Ed.), From data to model. Springer-Verlag, London, pp. 375394.
Sonday, B., Nelson, P. W., 2005. Identiability criteria for models of HIV. In
preparation.
Staord, M. A., Corey, L., Cao, Y., Daar, E. S., Ho, D. D., Perelson, A. S.,
2000. Modeling plasma virus concentration during primary HIV infection.
Journal of Theoretical Biology 203, 285301.
Takeuchi, K., 1976. Distribution of informational statistics and criterion of
model tting. Suri-Kagaku (Mathematic Sciences) 153, 1218.
Trefethen, L. N., Bau III, D., 1997. Numerical Linear Algebra. SIAM, Philadel-
phia, PA.
van Emden, M. H., 1971. An Analysis of Complexity. No. 35 in Mathematical
Centre tracts. Mathematisch Centrum, Amsterdam.
Wei, X., Ghosh, S. K., Taylor, M. E., Johnson, V. A., Emini, E. A., Deutsch,
P., Lifson, J. D., Bonhoeer, S., Nowak, M. A., Hahn, B. H., Saag, M. S.,
Shaw, G. M., 1995. Viral dynamics in human immunodeciency virus type
1 infection. Nature 373, 117122.
Wodarz, D., Lloyd, A. L., Jansen, V. A. A., Nowak, M. A., 1999. Dynamics
of macrophage and T cell infection by HIV. Journal of Theoretical Biology
196, 101113.
Wu, H., 2005. Statistical methods for HIV dynamic studies in AIDS clinical
trials. Statistical Methods in Medical Research 14, 122.
Wu, H., Ding, A., 1999. Population HIV-1 dynamics in vivo: Applicable models
and inferential tools for virological data from aids clinical trials. Biometrics
55, 410418.
Wu, H., Ding, A. A., de Gruttola, V., 1998. Estimation of HIV dynamic pa-
rameters. Statistics in Medicine 17, 24632485.
Wu, H., Wu, L., 2002a. Identication of signicant host factors for hiv dy-
namics modeled by nonlinear mixed-eect models. Statistics in Medicine
21, 753771.
Wu, L., Wu, H., 2002b. Missing time-dependent covariates in human immun-
odeciency virus dynamic models. Journal of the Royal Statistical Society,
Series C (Applied Statistics) 51, 297318.
24