ORIGINAL ARTICLE

Active Surveillance of Vaccine Safety

A System to Detect Early Signs of Adverse Events
Robert L. Davis,* Margarette Kolczak, Edwin Lewis, James Nordin, Michael Goodman,
David K. Shay, Richard Platt, Steven Black, Henry Shinefield, and Robert T. Chen

Background: There currently are no population-based systems in

the United States to rapidly detect adverse events after newly
introduced vaccines. To evaluate the feasibility of developing such
systems, we used 5 years of data from 4 health maintenance
organizations within the Centers for Disease Control and Prevention
(CDC) Vaccine Safety Datalink.
Methods: Within every year, each weeks vaccinated children were
followed for 4 weeks, and rates of adverse events were compared
with rates among children of similar ages before the introduction of
the new vaccine. We assessed risks for intussusception after rotavirus vaccination and risks for fever, seizures, and other neurologic
adverse events after the change from whole cell diphtheria-tetanuspertussis (DTPw) to acellular DTP vaccine (DTPa). We used sequential probability ratio testing, adjusted for age, sex, calendar
time, season, and HMO, and with a stopping value based on the
probability of an adverse event under the null hypothesis and under
a preset alternative hypothesis.
Results: We detected an increase in intussusception after 2589
vaccine doses of rotavirus vaccine, about the same time initial
reports of intussusception were made to the Vaccine Adverse Events
Reporting System. Decreases in risk for fever, seizures, and other
abnormal neurologic events became detectable within 12 weeks, 42
weeks, and 18 months, respectively, after the change from DTPw to
DTPa.

Submitted 21 January 2004; final version accepted 3 December 2004.

From the *Departments of Epidemiology and Pediatrics, University of
Washington, Seattle, Washington; the Vaccine Study Center, Kaiser
Permanente Northern California, Oakland, California; the Centers for
Disease Control and Prevention, Atlanta, Georgia; the Health Partners
Research Foundation, Minneapolis, Minnesota; and the Harvard Pilgrim/Harvard Vanguard, Boston, Massachusetts.
Supported by contract number 200-2002-00732 Centers for Disease Prevention and Control.
Supplemental material for this article is available with the online version
of the journal at www.epidem.com.
Correspondence: Robert L. Davis, CDC Office of Genomics and Disease
Prevention, Coordinating Center for Health Promotion, 4770 Buford
Highway, Mail Stop K89, Atlanta, GA 30341. E-mail: rad2@cdc.gov.
Copyright 2005 by Lippincott Williams & Wilkins
ISSN: 1044-3983/05/1603-0336
DOI: 10.1097/01.ede.0000155506.05636.a4

336

Conclusions: We conclude that it is feasible to develop systems for

rapid and routine population-based assessments of new vaccine
safety.
(Epidemiology 2005;16: 336 341)

ecent events in the United States have underscored the

need for surveillance systems that detect adverse events
as soon as possible after the introduction of new vaccines (eg,
rotavirus vaccine) or the reintroduction of old vaccines to
new populations (eg, smallpox vaccine).1,2 The goal of such
a monitoring system would be to speed the recognition of
increased rates of adverse events after vaccination to inform
public policy.
The discovery of intussusception after rhesus-rotavirus
vaccine (RRV) highlighted the need for this type of system.
Shortly after the introduction of RRV into clinical practice,
reports of intussusception (a rare form of bowel obstruction in
children) after vaccination were made to the Vaccine Adverse
Events Reporting System (VAERS), operated jointly by the
Centers for Disease Control (CDC) and the Food and Drug
Administration (FDA).1 Although the response to these initial
reports was rapid, months elapsed before it was possible to
estimate the risk associated with the vaccine, underscoring
the need to enhance vaccine safety monitoring in the United
States. Because VAERS is a passive system that relies on
reporting from physicians and other health care providers as
well as parents, rates of disease among vaccinated or unvaccinated subjects cannot be calculated. Thus, if only VAERS
data are available, it is difficult to make formal causality
assessments.3,4 Additionally, prelicensure trials of the rotavirus vaccine provided only limited information about the
vaccines risk for rare events because of limited study sizes.5
Although concern had been raised before licensure about
intussusception, no single trial had more than one vaccinated
child who developed intussusception, and overall the number
of vaccinated children was too small to detect the increased
risk subsequently seen. In light of these issues, the FDA held
a joint conference with the National Institutes of Health
Epidemiology Volume 16, Number 3, May 2005

Epidemiology Volume 16, Number 3, May 2005

(NIH), the CDC, and the National Vaccine Program Office to

focus on developing the capacity to more rapidly detect such
adverse events after licensure of vaccines.6
In this current study, we used data from the Vaccine
Safety Datalink to develop and test a methodology for conducting near real-time surveillance for potential vaccine adverse events. The Vaccine Safety Datalink was started in
1991 to conduct rigorous epidemiologic studies of vaccine
safety in the United States and is the largest such collaborative study in the United States.3 The Vaccine Safety Datalink,
coordinated by the CDC, includes data from 8 geographically
diverse health maintenance organizations (HMOs) and provides information on approximately 3.5% of all children
younger than the age of 6 in the United States, or 650,000
children. The Vaccine Safety Datalink has until now operated
retrospectively, adding data in yearly increments. We evaluated whether the Vaccine Safety Datalinks historical data on
vaccines and health-care encounters could be analyzed as if it
had become available on a weekly basis to provide rapid and
frequently updated assessments of vaccine safety. We then
attempted retrospectively to detect the previously recognized
rotavirus-intussusception signal7,8 and the decreased risk for
seizures and other abnormal neurologic events with the
changeover from diphtheria-tetanus-whole cell pertussis
(DTPw) to diphtheria-tetanus-acellular pertussis (DTPa) vaccine.9 These analyses allowed us to evaluate the capability of
Vaccine Safety Datalink data to meet 3 challenges facing
active surveillance: outcome definition, data management
(rapid and routine creation of analytic datasets based on
automated data), and statistical analysis of signal detection
using methods that account for multiple testing of accumulating data.

METHODS

Active Surveillance of Vaccine Safety

Each weekly cohort contained information about each

vaccinated childs age and sex, the vaccine administered that
week, and ICD9-CM diagnosis and procedure codes for the 6
weeks preceding and following the week during which immunization occurred. To protect confidentiality, subjects
were assigned unique personal identifiers in each weekly
cohort. As a result, a child would have a different identifier
for each immunization if these occurred in different weeks.
Each weeks dataset could thus be used to create a count of
the vaccinated children for that week, along with the number
diagnosed with specific medical conditions within the 30 days
after the vaccination under review.
These weekly data were partitioned into a baseline
period and a surveillance period. The baseline period was
defined as a period before the introduction of the new vaccine; during this time, the probability of a particular adverse
event after a comparison vaccine was presumed to be stable.
The surveillance period began with the introduction of a new
vaccine.
For comparison of DTPw and DTPa, the baseline
period was 2 years, from 1 January 1995 through 28 December 1996, when DTPw was in routine use. Surveillance for
adverse events after the acellular vaccine began on 9 February 1997, allowing a 6-week transition to routine use of the
acellular vaccine.
For rotavirus vaccine, we used a longer baseline period,
from 1 January 1995 through 9 January 1999, because the
rate of intussusception in children younger than 8 months old
after vaccination with routine childhood vaccines was low.
The surveillance period for intussusception following the
rotavirus vaccine began on 1 January 1999 and extended
through 6 November 1999. Rotavirus vaccine was withdrawn
during this period, and very few vaccinations were recorded
after 17 July 1999.

Overview

Classification of Outcomes

All Vaccine Safety Datalink HMOs have automated

vaccine tracking systems that track immunizations administered to members. These systems are the initial source of the
exposure data for Vaccine Safety Datalink studies. Additionally, these HMOs have the ability to capture medical care
utilization by enrolled members in hospital, emergency departments and in the outpatient clinics.

Fever, seizures, and intussusception were categorized

using standard ICD-9 codes. ICD-9 codes for additional
neurologic events were based on VAERS reports. (The codes
we used are listed in the appendix, which is available with the
electronic version of this article.) We evaluated a 3-day
interval after immunization for the occurrence of fever, seizures or other neurologic conditions following DTPw and
DTPa vaccine. For intussusception following rotavirus vaccine, we evaluated the rate of events in the 30-day time
window after vaccination. For each outcome, a person was
censored either on the date of diagnosis or on the end of the
follow-up period. These time windows were selected on our
understanding of the relationships between these conditions
and vaccination.7,9

Dataset Creation and Classification of

Exposures
We used data the HMOs had submitted to the Vaccine
Safety Datalink from 1995 through 2000. These data were
created from HMO-specific administrative datasets and include information on member demographics, vaccinations
administered, and medical care utilization.3 To simulate an
active surveillance adverse event detection system, we segmented the data into weekly cohorts of vaccinated children,
yielding 52 separate datasets for each year.
2005 Lippincott Williams & Wilkins

Statistical Analysis
Our primary goal was to detect a change in the probabilities of adverse events after the introduction of the new

337

Davis et al

rotavirus vaccine and the change from whole cell pertussiscontaining DTPw to acellular pertussis-containing DTPa.
Statistical analyses were performed using sequential probability ratio tests (SPRT).10,11 SPRT has been widely used in
industry to monitor process performance and is used for
situations where the monitoring is continuous and items or
events can be inspected one by one. Recently, the use of
SPRT in medical research has received attention, particularly
in regards to monitoring surgical failures or in monitoring
syndromic data sets for aberrations in disease rates.10,11 In
general, SPRT detects signals earlier than Shewhart p-charts
or CUSUM.12
To perform a SPRT analysis, specific values must be
formulated for the probability of a postvaccination adverse
event under the null hypothesis (p0) and under the alternative
hypothesis (p1). The null hypothesis is that the probability of an
adverse event during the surveillance period is the same as the
probability of an event during the baseline period. The alternative hypothesis is that there is a difference in the probability of
an event that is considered meaningful. The value of p0 for the
null hypothesis was calculated from the counts and events in
the baseline period, while the value of p1 for the alternative
hypothesis was based on the effect size we wanted to detect
during the surveillance period.
To assess the safety of the switch from DTPw to DTPa,
we examined fever, febrile seizures, and other neurologic
events among children younger than 24 months old. The rate
of these adverse events after DTPw vaccination was used to
establish the baseline period, and we then monitored for a
40% decrease in the probability of these events after the
changeover. To evaluate intussusception risk after rotavirus
vaccination, we monitored for a 10-fold increase in the
probability after the introduction of rotavirus vaccination
compared with the risk after other routine childhood vaccinations during the baseline period.
Our choice of threshold for rotavirus surveillance was
based on the strength of the signal seen in epidemiologic
studies.7,8 The choice of threshold for the DTPa analysis was
based on the effect size that we felt to be of public health
importance. The ability to detect a decrease of 40% would be
analogous to the ability to detect an increase of the same size,
or, in this case, a 2.5-fold increase in risk.
The SPRT method involves a likelihood ratio test to
evaluate the evidence in favor of both the null and alternative
hypotheses. The 2 likelihoods used to form the ratio are functions of the data and either p0 or p1. Using each weeks data, an
Xt is calculated from the previous weeks statistic Xt-1 plus the
log of the current weeks likelihood ratio. The log likelihood
ratio is the log of the likelihood using the current weeks data
and p1 divided by the likelihood using the current weeks
data and p0.

338

Epidemiology Volume 16, Number 3, May 2005

L(Datat ,p1)

Xt Xt1log

L(Datat ,p0)

We used the binomial likelihood function because of the

grouped data and proportions. Large values of Xt favor the
alternative hypothesis; small values of Xt favor the null
hypothesis.
The weekly values of the pair (t,Xt) were plotted on the
SPRT chart along 2 barriers. If the value of Xt crossed the
upper alternative barrier, a decision was made to accept the
alternative hypothesis, while if it crossed the lower null
barrier, the decision was made to accept the null hypothesis.
If Xt remained between the 2 barriers, no decision was made
and data continued to accumulate. The upper and lower
barriers are calculated as:

upper log

lower log

1 *

1 *

where * and * are approximately equal to the Type I and

Type II errors respectively. These error levels apply to the
entire SPRT process, not to each specific week, hence our
analyses accounted for multiple testing. The adjustment for
multiple testing is not explicit as in the Bonferroni adjustment, but rather the adjustment is incorporated into the SPRT
test in the way that the upper and lower boundaries are
calculated. These boundaries preserve the specified alpha and
beta until a final decision is reached as to whether the
hypothesis should be accepted or rejected.13
Although subjective, by necessity our rule depends on
a stopping value (in this case, a P value of approximately
0.05). The P value in this case is not being used to declare
a statistically significant relationship between some exposure
and disease, but rather is being used in the context of
surveillance to trigger an alert.
To control for potential confounders such as HMO, age,
calendar time, season, and sex, we used the risk-adjustment
methods described in Steiner et al.14 With this method, each
unique combination of risk factors forms a stratum ( j), which
will have its own baseline probability of an adverse event, poj.
These probabilities are obtained by conducting a logistic
regression analysis on the baseline data. The log likelihood
ratio then becomes the product of the likelihoods in the
various strata.

RESULTS
Descriptive Data
The analysis of DTPw and DTPa vaccines covered a
total of 156 weeks. Of these, 104 weeks of data were used for
2005 Lippincott Williams & Wilkins

Epidemiology Volume 16, Number 3, May 2005

the baseline period, with 212,634 DTPw vaccinations administered. There were 52 weeks of data used for the surveillance
period, with 63,367 DTPa vaccinations. In the baseline period, there were 40 medically attended visits for seizures, 182
for fever, and 23 for other neurologic conditions in the 30
days after vaccination. In the surveillance period, there were
11 medically attended visits for seizures, 46 visits for fever,
and 5 visits for other neurologic conditions in the 30 days
following vaccination.
The analysis of rotavirus vaccine covered a total of 253
weeks. Of these, 210 weeks of data were used for the baseline
period, with 996,733 routine childhood vaccinations administered. Forty-two cases of intussusception were recorded in the 30
days after any vaccination during the baseline period. There
were 43 weeks of data used for the surveillance period, with
26,069 rotavirus vaccinations administered; 99% of these
vaccinations were administered in the first 27 weeks of the
surveillance period. Seven cases of intussusception occurred in
the 30 days following rotavirus vaccination.

Change in Adverse Event Probabilities After

the Introduction of DTPa Vaccine
Using the risk-adjusted SPRT, we were able to rapidly
detect a 40% decrease in the clinical diagnosis of fever in the
3 days after vaccination with the changeover from whole cell
pertussis vaccine to the acellular pertussis vaccine (Fig. 1).
This signal triggered an alert approximately 12 weeks into the
surveillance period.
A 40% decrease in the rate of seizures after the
changeover from DTPw to DTPa (Fig. 2) triggered an alert
later, approximately 42 weeks after the changeover. The less
frequent occurrence of febrile seizures compared with fever
after vaccination contributed to the 30-week difference in the
detection of these 2 signals. A decrease in other neurologic
events, such as pallor or hypotonia, within the 3 days after

FIGURE 1. Decrease in fever after change from DTP to DTPa.

(SPRT assessment for 40% decrease, adjusted for season, time,
age, and HMO. Risk window 0 3 days after vaccination).
2005 Lippincott Williams & Wilkins

Active Surveillance of Vaccine Safety

FIGURE 2. Decrease in seizures after change from DTP to

DTPa, adjusted for season, time, age, and HMO. Risk window
0 3 days after vaccination.

vaccination took much longer to detect, becoming significant

1.5 years after the changeover from DTPw to DTPa (Fig. 3).

Change in Adverse Event Probabilities After

the Introduction of Rotavirus Vaccine
A positive signal for a 10-fold increased risk of intussusception triggered an alert within 10 weeks after vaccine
introduction (Fig. 4), with the second case of intussusception
among vaccine recipients. It remained at this level for the
remainder of the observation period following the fourth case
of intussusception. Chronologically, the positive signals occurred at about the same time as the first set of intussusception reports to VAERS.1

DISCUSSION
In this study we demonstrated the value of routinely
collected managed-care data for rapidly detecting predefined
signals of vaccine adverse events. We were able to find both

FIGURE 3. Decrease in neurologic events after change from

DTP to DTPa, adjusted for season, time, age, and HMO. Risk
window 0 3 days after vaccination.

339

Davis et al

FIGURE 4. Increase in intussusception after introduction of

rotavirus vaccine, adjusted for season, time, age, and HMO.
Risk window 0 30 days after vaccination among children
younger than 8 months old.

an expected signal (intussusception following RRV) and

signals for situations in which an association has been suggested by prelicensure trials (the decreased risk for fever,
seizures, and other neurologic events with DTPa).
A program such as this would add substantially to the
current U.S. vaccine safety system by complementing
VAERS and other spontaneous report systems. The specific
advantages of our system include its relatively complete
ascertainment from defined populations (permitting calculation of unbiased event rates) and its adjustment for confounders that might otherwise obscure important signals. Finally,
the statistical methods take into account the sequential, multiple nature of testing, and the systems type I and II error
rates apply to the entire process. This system can be used as
an independent means to corroborate VAERs signals, and
vice versa.
There are limitations to this study. We do not have
other data to confirm the decreased risks for fever, seizures,
and other neurologic conditions after the switch from DTPw
to DTPa presented here, although our findings are consistent
with smaller prelicensure studies.9 Also, our reliance on
automated data means that we did not know which visits were
for current, as opposed to past problems, or were for the
evaluation of symptoms found later not to represent disease.
This system relies on routinely collected automated data for
signal detection; since automated data have varying degrees
of accuracy in diagnostic coding, this approach should not be
viewed as the final confirmatory epidemiologic investigation
into potential vaccine adverse events. Our approach also is
limited to conditions that develop relatively soon after vaccination and would not be suitable for investigation of conditions with a longer induction period. Additionally, our
current approach does not account for any nonrandom distribution of onset intervals between vaccination date and the

340

Epidemiology Volume 16, Number 3, May 2005

date of adverse event onset. Nonrandom clustering of such

events might be another important signal of potential vaccine
safety concern.
This study is only the first step towards the creation of
a surveillance system for adverse events after vaccination.
The move from proof-of-principle to an active system faces a
number of challenges. Data must be made routinely available
more frequently than in the past, without appreciably increasing costs or compromising patient confidentiality or data
quality. The patient confidentiality requirement has likely
been met, as the unit of analysis in our study contained only
a weeks worth of immunization data along with the attendant
follow-up time, and we did not link identifiers across immunizations or across weeks. Hence, these datasets did not
include any patient-specific identifiers and maintained a robust degree of patient confidentiality. However, generating
and analyzing data on a weekly basis in real timewhile
feasiblewill require further work to be done smoothly.
Another challenge is to decide which adverse events to
try to monitor and the magnitude of change that is worth
detecting. Although prelicensure trials can suggest possible
associated adverse events, input from advisory groups such as
Advisory Committee on Immunization Practices and from
regulatory agencies such as the FDA will also be important.
Screening for a wide variety of possible adverse events after
vaccination would require careful interpretation of the results
because of the inevitable number of false positive signals.
Nonetheless, this active surveillance project would be flexible
enough to quickly assess new signals (that might arise from
VAERS), because of the availability of denominator datasets
and statistical programs
Future applications of this approach will focus on
surveillance of the routinely administered annual influenza
vaccine (for example, to look for serious adverse events such
as Guillain-Barre syndrome and other neurologic diseases),
and the study of new vaccines such as the live attenuated
intranasal influenza flu vaccine (licensed for healthy individuals 5 49 years) and a recently released combination DTPainactivated polio- hepatitis b vaccine. Active and prospective
surveillance analysis of Vaccine Safety Datalink data provides a
valuable, population-based early warning system to complement
VAERS in the U.S. immunization safety system.15
ACKNOWLEDGMENTS
We thank Noel Weiss for his helpful comments.

REFERENCES
1. Intussusception among recipients of rotavirus vaccineUnited States,
1998 1999. MMWR 1999;48:577581.
2. Centers for Disease Control and Prevention. Update: cardiac and other
adverse events following civilian smallpox vaccinationUnited States,
2003. MMWR 2003;52:639 642.
3. Chen RT, Glasser JW, Rhodes PH, et al. Vaccine Safety Datalink

2005 Lippincott Williams & Wilkins

Epidemiology Volume 16, Number 3, May 2005

4.
5.
6.
7.
8.
9.
10.

Project: a new tool for improving vaccine safety monitoring in the

United States. Pediatrics. 1997;99:765773.
Chen RT, Rastogi SC, Mullen JR, et al. The Vaccine Adverse Event
Reporting System (VAERS). Vaccine 1994;12:542550.
Rennels MB. The Rotavirus Vaccine Story: a clinical investigators
view. Pediatrics 2000;106:123125.
Ellenberg SS. Safety considerations for new vaccine development.
Pharmacoepidemiol Drug Saf. 2001;10:411 415.
Murphy TV, Gargiullo PM, Massoudi MS, et al. Intussusception among
infants given an oral rotavirus vaccine. N Engl J Med. 2001;344:564 572.
Kramarz P, France EK, DeStefano F, et al. Population-based study of
rotavirus vaccination and intussusception. Pediatr Infect Dis J. 2001;20:
410 416.
Decker MD, Edwards KM. Acellular pertussis vaccines. Pediatr Clin
North Am. 2000;47:309 335.
Spiegelhalter DJ. Mortality and volume of cases in paediatric cardiac

2005 Lippincott Williams & Wilkins

Active Surveillance of Vaccine Safety

11.
12.
13.
14.
15.

surgery: retrospective study based on routinely collected data. BMJ.

2002;324:261264.
Spiegelhalter DJ, Grigg O, Kinsman R, et al. Risk-adjusted sequential
probability ratio tests: applications to Bristol, Shipman and adult cardiac
surgery. Int J Quality Health Care. 2003;15:713.
Reynolds MR, Stoumbos ZG. Monitoring a proportion using CUSUM
and SPRT control charts. Frontiers in Statistical Quality Control.
2001;6:155176.
Grigg OA, Farewell VT, Spiegelhalter DJ. The use of risk-adjusted
CUSUM and RSPRT charts for monitoring in medical contents. Stat
Methods Med Res. 2003 Mar;12:147170.
Steiner SH, Cook RJ, Farewell VT, et al. Monitoring surgical performance using risk-adjusted cumulative sum charts. Biostatistics. 2000;
1,4:441 452.
Chen R, Davis RL. Vaccine safety. In: Plotkin SA, Mortimer EA, eds.
Vaccines. 3rd ed. Philadelphia: WB Saunders; 2003.

341