C Basic & Clinical Pharmacology & Toxicology 2005, 96, 235241.

Printed in Denmark . All rights reserved

Copyright C
ISSN 1742-7835

MiniReview

Predicting the Outcome of Phase III Trials using Phase II

Data: A Case Study of Clinical Trial Simulation in Late Stage
Drug Development
Filip De Ridder
Biometrics & Clinical Informatics, Johnson & Johnson Pharmaceutical Research and Development (a division of
Janssen Pharmaceutica n.v.), B-2340 Beerse, Belgium
(Received July 14, 2004; Accepted October 19, 2004)

Abstract: Maximizing the likelihood of success in Phase III is the ultimate goal of the use of modelling and simulation in
the drug development process. The success in Phase III depends primarily on two questions: 1) Is the drug regimen
actually efficacious and safe in the targeted patient population?, and 2) Will the planned Phase III clinical trial(s) be
successful in demonstrating this? Traditionally, the first question is addressed in a qualitative, overall interpretation of
available study results. Integrating this information into a formal statistical model of the action of the drug, allows
running simulations to investigate the impact of uncertainties and imprecision in this knowledge. The second question is
related to having an adequately designed clinical trial. Clinical trial simulation, using a drug action model, supplemented
with appropriate models for disease progression and trial execution, allows assessing the impact of typical design features
such as doses, sample size, in-/exclusion criteria, drop-out and trial duration on the trial outcome and thus optimising
trial design. In this contribution, the use of modelling and simulation in the Phase II to Phase III transition is illustrated
using real data of a drug for symptom relief in a chronic condition. A dose-response model of the clinical response was
developed using data from Phase II. Simulations were performed to 1) generate the range of possible outcomes of ongoing
Phase III trials and compare these to the blinded data being generated from these trials; 2) assess the robustness of the
ongoing Phase III trials with respect to uncertainty of the true dose-response, patient variability in baseline severity and
drug-response, and 3) assess the likelihood of achieving a clinically relevant response with a dose lower than those included
in the trials.

The decision of entering Phase III is a very important one
in drug development in terms of the investment of time and
money. Over the last decade, the pharmaceutical industry
has encountered a lot of failures in the late stages of devel-
opment. Clinical trial simulation is recognized as one of the
emerging technologies that can contribute to a better and
more successful drug development, both from an academic
(Holford et al. 2000), industry (Goggin et al. 2003) and
regulatory perspective (Lee & Lesko 2003).
How can modelling and simulation contribute to a more
successful Phase III? In other words, how can clinical trial
simulation increase the likelihood of success in Phase III,
which traditionally, has been considered as the statistical
power of the trial. Given the trial design and assuming that
the drug, as used in the trial, has a certain effect in the
patient population studied, what is the probability of show-
ing a statistically significant benefit over a control group?
In principle, sample size calculation is easy as it depends on
three quantities: (1) the magnitude of the postulated treat-
Author for correspondence: Filip De Ridder, Johnson & Johnson,
Turnhoutseweg, B-2340 Beerse, Belgium (fax 32 1460 6124, e-mail
fdridder/prdbe.jnj.com).
ment benefit, (2) the expected variability in the population
of the outcome and (3) the power one wishes to achieve.
The first two quantities are not precisely known and depend
on a variety of other factors, including trial design features
such as doses, in-/exclusion criteria, drop-ins and drop-
outs, trial duration etc. Pharmaceutical statisticians are well
aware of these complexities and usually address it by con-
sidering the worst-case scenario. Clinical trial simulation of-
fers a more structured framework to address these ques-
tions, and also allows overcoming mathematical complex-
ities.
Before designing Phase III trials, the question needs to
be answered whether the drug is indeed active in the tar-
geted patient population at a safe dose. Traditionally, this
is done in a qualitative, overall interpretation of all available
study results. Due to the inherent exploratory nature, Phase
II trials cannot always give a straightforward answer to this
question. One important reason for this, is that Phase III
trials may differ in several respects from Phase II such
as: (1) dose range: fewer doses are used; the choice of
these doses is crucial and one of the single most important
aspects of the trial design, (2) patient population: base-
line characteristics such as disease status and severity, age,
236 FILIP DE RIDDER
medical history and (3) the definition of the primary end-
point.
Modelling and simulation can potentially provide insight
towards addressing these questions if enough knowledge is
available from Phase II and earlier, other compounds, or
literature. Often available knowledge is imprecise, but simu-
lations can assess the impact of this imprecision on model
predictions and thus provide adequate recommendations
for trial design.
An adequate model must be built to incorporate all avail-
able information, to provide a successful clinical trial simu-
lation-project to guide the Phase II Phase III transition. A
good dose-finding study can yield a good dose- or exposure
response model that can guide the dose choice. Contrary, if
Phase III is to be conducted in patients with severe disease,
but the Phase II trial data do not provide some understand-
ing of the impact of baseline severity on outcome, any simu-
lation exercise will have to depend on assumptions. The
power of clinical trial simulation in this case is the ability
to visualize the impact of these possible vague assumptions
on the trial outcome. A successful modelling and simulation
approach at the end of Phase II depends on the quality of
the available data. In this respect, modelling and simulation
activities in the earlier phases of development are of crucial
importance to guarantee that optimal data are collected
and knowledge is generated for later use (Goggin et al.
2003).
Published case studies of the application of clinical trial
simulation in late stages of development are rare. Nestorov
et al. (2001), Lockwood et al. (2003) and Mandema & Wang
(2003) provided examples of designing Phase II studies for
new compounds for pain relief. Modelling and simulation
was based on compound-specific pharmacokinetic data,
whereas the pharmacodynamic model was based on extra-
polation from other related compounds. Chabaud et al.
(2002) predicted the outcome of a Phase III trial of a new
compound, using a compound-specific pharmacokinetic/
pharmacodynamic-model as well as a mechanistic physio-
phatologic model, describing the relationship between an
intermediate endpoint measured in Phase II and the clinical
endpoint of Phase III. Veyrat-Follet et al. (2000) simulated
a Phase III oncology trial with an alternative dose intensi-
fication scheme. Kimko et al. (2000) retrospectively pre-
dicted the outcome of a Phase III trial in schizophrenia
using Phase I and Phase II data.
In this contribution, a case study of the use of modelling
and simulation in the transition from Phase II to Phase III
will be illustrated. At the time the modelling and simulation
project was initiated, a traditional Phase III programme
had started, consisting of three placebo-controlled trials
with 2 drug doses. Simulations were performed to: generate
the range of possible outcomes of the Phase III trials and
compare these to blinded data generated by the on-going
trials, assess the robustness of the ongoing Phase III trials
with respect to uncertainty of the true dose-response, pa-
tient variability in baseline severity and drug-response, and
MiniReview
assess the likelihood of achieving a clinically relevant re-
sponse with a dose lower than those included in the trials.
Material and Methods
Phase II clinical trials. A drug X was in development for the treat-
ment (symptom relief) of a chronic condition. In two placebo-con-
trolled double-blind Phase II dose-ranging trials with 5 different
doses (0.5 mg, 1 mg, 2 mg and 4 mg), patients (420 in total) were
treated during 4 weeks. In a two-week run-in period, baseline sever-
ity of the condition was assessed. Most patients completed the tri-
als. Drop out rate was only about 2.5%.
In these exploratory Phase II trials, clinical efficacy was meas-
ured in a variety of ways, including a diary capturing the occurrence
and relief of symptoms over time. These latter data were used in
the modelling project because the Phase III primary endpoint, a
percentage of responders (see below), was to be derived from these.
Phase III clinical trials. Three Phase III double-blind placebo-con-
trolled 12-week clinical trials with 2 doses (2 mg, 4 mg) were started.
In these trials, the primary endpoint was the percentage of re-
sponders. A responder was defined when the improvement of the
frequency of symptoms, averaged out over the 12-week period, ex-
ceeded a pre-defined threshold.
Based on clinical considerations, inclusion of patients was more
restrictive than in the Phase II trials, with respect to baseline sever-
ity. Patients with a more severe baseline condition were to be in-
cluded in the Phase III trials. A sample size calculation was per-
formed to achieve 90% power to find a 15% increase in the percen-
tage of responders over placebo in either one of the doses. Based
on the Phase II results, it was assumed that the placebo-response
rate was 15% and the response rate for the lower dose was at least
30%. Taking into account an expected 5% drop-out rate, about 200
patients were enrolled per arm.
All clinical trials were performed in accordance with the Declar-
ation of Helsinki.
Modelling the Phase II data. A model was developed that describes
the time course of four consecutive weekly assessments of a continu-
ous symptom score. It had the following general structure:
Symptom ScoreBaselineTreatment-effect Residual error
where
Baseline value of the symptom score at the start of the
trial (randomization);
Treatment-effect change of the symptom score due to the treat-
ment;
Residual error unexplained intra-patient residual variability,
described by a Poisson distribution.
The treatment effect was modelled as follows:
Treatment-effectPlacebo-effectTimeslope DOSE power
where
Placebo-effect change of the symptom score in the placebo-
arm of the trial;
DOSE the dose;
slope, power parameters describing the dose-response re-
lationship;
Time factor describing the change of the treatment
effect over time.
In all arms the treatment effect was highest in the first week of
treatment, and then was stable during the subsequent weeks. This
was modelled by a simple time-factor contrasting Week 1 versus
Weeks 24.
The model contained two patient-specific random effects describ-
ing inter-individual variability in the baseline symptom score and
inter-individual variability in the magnitude of the treatment-effect.
MiniReview CLINICAL TRIAL SIMULATION IN LATE STAGE DRUG DEVELOPMENT 237

Both random effects were considered as having a lognormal dis-

tribution. Exploratory data analysis showed that the change in
symptom score during treatment was dependent on the baseline
value: patients with a high baseline severity showed a bigger im-
provement during treatment. This was implemented in the model
by introducing a correlation between the random effects.
Model parameters estimates were obtained by maximum likeli-
hood using the SAS procedure NLMIXED (SAS Institute Inc.
1999). This procedure maximizes an approximated likelihood inte-
grated over the random effects using adaptive Gaussian quadrature.
The fit of the model was judged to be adequate using plots of pre-
dicted versus observed values (fig. 1), residual values versus pre-
dicted values, as well as the covariates dose and time, and quantile-
quantile plots of residuals and Empirical Bayes estimates of the
random effects, as outlined by Pinheiro & Bates (2000).

Simulating the Phase III data. In the application of simulation in

drug development it is helpful to make a distinction between popu-
lation simulation and clinical trial simulation. In a population simu-
lation, a large number (5000) of patients are simulated to obtain
a distribution of the outcome in a future population of patients.
From this distribution, several measures of interest can be derived,
such as:
O the average response;
O the proportion of patients exceeding a minimal response (% of
responders);
O between patient variability in response, which can be caused by
inter-patient variability in a variety of factors such as baseline
characteristics, pharmacokinetics and drug response.
The impact of model uncertainty can be quantified by repeating the
population simulation a large number of times (1000), each time
with a different set of model parameters. These parameter sets are
drawn from an appropriate distribution reflecting the uncertainty
of the model parameter estimates.
In a clinical trial simulation, additional sources of variability are
added to the simulation model, which are typical for the clinical
trial itself. These include sampling variability (the patients are a
sample from the population) and variability in trial conduct. A
population simulation will give insight into how well the drug
works, given at the regimen under study, in the targeted population
of interest and how sure we are of this in view of model uncertainty.
It is therefore mostly used for optimizing treatment regimens.
Clinical trial simulation, on the other hand, also reflects the im-
pact of sampling variability and trial conduct variability. It is there-
fore used to assess and optimize the performance of a clinical trial.
Fig. 2. Observed versus individual predictions of the 4-week aver-
aged symptom scores in the Phase II trials.
Results
Modelling the Phase II data.
The observed versus model predicted weekly symptom
score shows a relatively large scatter, which is due to the
Poisson nature of the residual variability (fig. 1). However,
the high within-subject variability is smoothed out to a
large extent when the observed 4-week averaged score ver-
sus the model predictions is plotted (fig. 2). When both the
observed and predicted symptom score in the Phase II trials
is converted into the Phase III endpoint (percentage re-
sponse), we see that the dose-effect relationship differs
strongly between the two trials (fig. 3). This is a conse-
quence of a different level of baseline-severity in these two
trials. In Study 1, baseline severity was more moderate as
compared to Study 2, resulting in a high placebo-response
rate and almost no apparent drug effect. The reason is that
the low baseline severity results in a large average improve-
ment, in all treatment arms. Consequently, most patients,
also those of the placebo arm, are classified as responders

Fig. 1. Observed versus individual predictions of the weekly symp- Fig. 3. Observed and model predicted % of responders in the Phase
tom scores in the Phase II trials. II trials.
238 FILIP DE RIDDER MiniReview

according to the Phase III-definition. The underlying symp-

tom score, however, showed a clear dose-response in Study
1 (data not shown). The model accommodates the observed
differences between studies through the dependency of the
drug effect on the baseline severity. Without a modelling
approach it is very difficult, if not impossible, to use the
Study 1 data to predict the outcome of the clinical response
as defined in Phase III.

Population simulation of the Phase III outcome.

Fig. 4. Predicted dose-response of response (%) in a Phase III Pa-
tient Population based on a population simulation of the Phase II
model (1000 replicates over model uncertainty, 5000 patients/dose
per replicate). The dotted lines are the 5th and 95th percentile of
the prediction distribution and represent uncertainty in the model
parameters.
Fig. 4 shows the population prediction of the percentage re-
sponse as a function of dose using the Phase II model, with
5000 patients per dose, and repeated 1000 times, each time
with a different set of parameter estimates, reflecting the
model uncertainty. The uncertainty on the population pre-
diction is small at the lower end of the dose-range and in-
creases with dose. The distribution of the plausible values in
the difference from placebo of the percentage response, given
the data and model uncertainty, is shown for 4 doses in fig. 5.

Fig. 5. Predictive distribution of the response (%) in the Phase III Patient Population when treated with 0.5 mg, 1 mg, 2 mg and 4 mg. The
insert indicates the likelihood that the improvement of the response rate exceeds a target of 10%.
MiniReview CLINICAL TRIAL SIMULATION IN LATE STAGE DRUG DEVELOPMENT 239

For a dose of 0.5 mg, the probability that the true percentage Table 1.
of responders will reach a target of at least 10% is sufficiently Probability to find a statistically significant difference with placebo
small to discard this dose a useful, whereas for 2 mg and 4 mg in the Phase III trials (based on the simulation of 1000 replicates
of the trial, either with or without taking model uncertainty into
this probability is sufficiently large to warrant inclusion of account).
these doses in Phase III. Therefore, the simulation confirms
Comparison Probability
the choice of including 2 mg and 4 mg in the Phase III trials.
For the 1 mg dose, however, the probability is intermediate 2 mg 4 mg Without model With model
uncertainty uncertainty
and the conclusion less straightforward.
Yes Yes 0.86 0.74
Clinical trial simulation of the Phase III outcome. No Yes 0.14 0.20
Yes No 0.00 0.01
After enrolment of about 50% of the targeted sample size, No No 0.00 0.06
a blinded survey of the available data was performed. As
described above, it was assumed that the placebo-response
rate was 15% and the response rate for the lower dose was at
least 30%. Based on these numbers, the anticipated blinded % response of each dose was compared with placebo using
percentage of responders was about 25%. The observed a c2-test, with a0.025 to account for multiple testing. The
value was substantially lower, only 20%. In order to better results show that the likelihood of success, that is, at least
understand this observation, blinded data were simulated one dose is found to be significantly better than placebo, is
using the Phase II model. The clinical trials were simulated sufficiently high (table 1). The likelihood that both the high
with half of the targeted sample size, and the data were and low dose will reach statistical significance is lower, but
analysed with all treatment groups pooled. The clinical trial still acceptable. For comparison, a simulation was per-
simulation shows a wide range of potential outcomes (fig. formed without model uncertainty. In this way, the likeli-
6). The range encompasses both trial-by-trial variations in hood of success was estimated to be much higher, and is
outcome, as well as model uncertainty. The simulated re- comparable to the protocol sample size calculation (90%
sults match very well with the actual blinded data result power). This was to be expected because in this approach
(20%). The anticipated value of 25% is at the upper tail the possibility that the drug-effect is smaller or the placebo-
of the distribution of simulated outcomes, and is thus not response is larger than anticipated is ignored.
consistent with the incoming blinded Phase III data, or with
the Phase II simulations. Comparison of the predicted and observed Phase III outcome.
The observed discrepancy between the observed blinded Fig. 7 compares the actual outcome of the three Phase III
response rate in the accumulating Phase III data, lead to trials with the predictive distribution derived from the Phase
the question of how robust the ongoing trials were in show- II model simulations. Overall, there was a good agreement
ing the desired efficacy. between the predicted and observed results. However, for one
To address this question, a Phase III trial simulation was trial the model slightly under predicts the effect of 2 mg,
replicated a 1000 times over model uncertainty. For each whereas for all three trials the observed response of 4 mg was
replicate, the data were analysed as the real Phase III data consistently lower than predicted. The observed Phase III
were to be analysed according to the protocol. The observed dose-response suggests that the drug-response is almost satu-

Fig. 7. Clinical trial simulation (1000 replicates, including model

uncertainty) of the dose-response of the response (%) in the Phase
Fig. 6. Clinical trial simulation (1000 replicates, including model III trails compared to the actual observed outcome in the three
uncertainty) of the response (%) in blinded data after enrolment of trials. Solid line: median. Dotted lines: 5th, 20th, 75th and 95th
50% of the patients. percentile. Each arm of a study enrolled about 200 patients.
240 FILIP DE RIDDER
rated with 2 mg. When the pooled Phase II and Phase III data
were modelled, an Emax-model provided a substantially
better fit (fig. 8). This had an important consequence on the
prediction of the effect of the 1 mg dose, which was not in-
cluded the Phase II trials. Whereas the Phase II population
simulations showed that the probability that 1 mg would
achieve a relevant increase in response over placebo was small
(0.19), a population simulation with the pooled Phase II/
Phase III model, yielded a probability that was sufficiently
high (0.85) to warrant inclusion of this dose in a clinical trial.
Discussion
During the development of a drug targeted at symptom re-
lief in a chronic condition, modelling and simulation was
applied to answer several questions, related to the design of
Phase III trials.
First, a dose-response model based on two Phase II trials
was developed. It was chosen to model the patient-specific
evolution of a symptom score over time. Although the data
that were collected in Phase III also consisted of this profile
of the symptom score over time, the primary endpoint was
the percentage of responders. A patient was classified as a
responder if the symptom improvement during treatment
exceeded a pre-defined threshold. One could argue that it
might have been simpler to directly model the dose-response
of the response rate. However, this implies dichotomizing
the underlying data, which always results in a loss of infor-
mation, which was important to avoid at the end of Phase
II when data of a relatively low number of patients were
available as compared to the end of Phase III.
The dose-response model was used to simulate the out-
come of ongoing Phase III trials. These trials had a differ-
ent design than the Phase II trials in terms of doses, base-
line severity of the condition, study duration and sample
size. The Phase II data were rich enough in terms of number
Fig. 8. Comparison the population dose-effect relationship for the
percentage of responder predicted by the Phase II model and the
pooled Phase II/Phase III model. The dots represent doses at which
a population simulation (5000 patients/dose, 1000 replicates over
model uncertainty) was performed.
MiniReview
of patients, variation in baseline severity and the dose range
studied, to allow inclusion of the effect of these design fea-
tures on the outcome into the model, and thus in the sub-
sequent Phase III simulations. Modelling allows a formal,
transparent inclusion of these factors, which is usually hard
to do in any other way.
A population simulation confirmed the choice of doses
in Phase III. Taking into account model uncertainty, it
showed that it was unlikely that a dose of 1 mg would yield
a sufficient response rate.
After 50% enrolment, the incoming blinded Phase III
data showed a response rate of 20% which was substantially
lower than the anticipated one of 25%. This number was
based on a judgment interpretation of the Phase II data,
assuming a 15% placebo-response rate and a 30% response
rate of the lower dose. It was therefore difficult to judge
the observed discrepancy. The advantage of performing a
clinical trial simulation was that the model took explicitly
the dose-response (of all doses), the correlation with base-
line severity as well as sampling variability and model un-
certainty into account. The distribution of the predicted re-
sponse in blinded Phase III data after 50% enrolment re-
flects the impact of all these factors, as well as inter-patient
variability and model uncertainty. The simulated range of
possible outcomes matched very well with the observed
value, showing that the incoming Phase III data were con-
sistent with the Phase II knowledge.
Although the response rate was lower than anticipated,
further simulation of unblinded clinical trials after full en-
rolment showed that likelihood of success was still accept-
able, also when taking model uncertainty into account.
When compared to the actual outcome of the Phase III
trials, it was clear that, although the model adequately pre-
dicted the Phase III results, the Phase III dose-response dif-
fered with respect to the prediction of the 1 mg response.
This dose was not included in the Phase III trials, a de-
cision, which was confirmed by a population simulation
based on the Phase II data. On the basis of the pooled
Phase II/Phase III model on other hand, there is a reason-
able chance for a relevant response by 1 mg. This shows
that, although the Phase II model was adequate, one cannot
expect it to capture all characteristics of drug response in
the target population and provide a perfect prediction.
In conclusion, this case study illustrates the potential
merits of modelling of clinical endpoints in the late stages
of development through both population and clinical trial
simulation for Phase III. The approach allows a more for-
mal synthesis of the data and knowledge available at the
end of Phase II, including the level of model uncertainty.
This allows studying the potential ranges of outcome in
Phase III under different scenarios, which can be used to
optimize the design of Phase III and beyond.
Acknowledgements
Part of the results presented originate from fruitful dis-
cussion with and the unpublished work of different con-
tributors: Surya Mohanty, Rene Kerstens, Achiel Van Peer,
MiniReview CLINICAL TRIAL SIMULATION IN LATE STAGE DRUG DEVELOPMENT
Vladimir Piotrovskij, Eugene Cox, Eric Snoeck and Jaap
Mandema.
References
Chabaud, S., P. Girard, P. Nony & J. P. Boissel: Clinical trial simula-
tion using therapeutic effect modeling: Application to ivabradine
efficacy in patients with angina pectoris. J. Pharmacokin. Pharm-
acodynam. 2002, 29, 339363.
Goggin, T., R. Gieschke, G. Pillai, B. Fotteler, P. Jordan & J.-L.
Steimer: Modeling and simulation of clinical trials: an industry
perspective. In: Simulation for designing clinical trials. Eds.: H.C.
Kimko & S. B. Duffull. Marcel Dekker, Inc., New York, 2003,
pp. 227244.
Holford, N. G. H., H. C. Kimko, J. P. R. Montelone & C. C. Peck:
Simulation of clinical trials. Annu. Revu. Pharmacol. Toxicol.
2000, 49, 6795.
Kimko, H. C., S. S. B. Reele, N. G. H. Holford & C. C. Peck:
Prediction of the outcome of a phase 3 clinical trials of an anti-
psychotic agent (quetiapine furmarate) by simulation with a
population pharmacokinetic and pharmacodynamic model. Clin.
Pharmacol. Ther. 2000, 68, 568577.
241
Lee, P. I. D. & L. J. Lesko: Clinical Trial Simulation (CTS): A regu-
latory clinical pharmacology perspective. In: Simulation for de-
signing clinical trials. Eds.: H. C. Kimko & S. B. Duffull. Marcel
Dekker, Inc., New York, 2003, pp. 201210.
Lockwood, P. A., J. A. Cook, W. E. Ewy & J. W. Mandema: The
use of clinical trial simulation to support dose selection: Appli-
cation to development of a new treatment for chronic neuro-
pathic pain. Pharm. Res. 2003, 20, 17521759.
Mandema, J. W. & W. Wang: Use of modelling and simulation to
optimise dose-finding strategies. In: Simulation for designing clin-
ical trials. Eds.: H. C. Kimko & S. B. Duffull. Marcel Dekker,
Inc., New York, 2003, pp. 289312.
Nestorov, I., G. Graham, S. Duffull, L. Aarons, E. Fuseau & P.
Coates: Modelling and simulation for clinical trial design involv-
ing a categorical response: a Phase II case study with sumatrip-
tan. Pharm. Res. 2001, 18, 12101219.
Pinheiro, J. C. & D. M. Bates: Mixed-effects models in S and S-
PLUS. Springer-Verlag, Inc., New York, 2000.
SAS Institute Inc. SAS/STAT Users Guide, Version 8, Cary NC:
SAS Institute Inc, 1999. 3884 pp.
Veyrat-Follet, C., R. Bruno, R. Olivares, G. R. Rhodes & P. Chai-
kin: Clinical trial simulation of docetaxel in patients with cancer
as a tool for dosage optimization. Clin. Pharmacol. Therap. 2000,
68, 677687.