Focus On Quality

By Jane Metzger, Emily Welebob, David W. Bates, Stuart Lipsitz, and David C. Classen
doi: 10.1377/hlthaff.2010.0160

Mixed Results In The Safety

HEALTH AFFAIRS 29,
NO. 4 (2010): 655–663
©2010 Project HOPE—
The People-to-People Health

Performance Of Computerized Foundation, Inc.

Physician Order Entry

Jane Metzger (jmetzger2@

ABSTRACT Computerized physician order entry is a required feature for csc.com) is a principal
hospitals seeking to demonstrate meaningful use of electronic medical researcher at CSC Healthcare
in Waltham, Massachusetts.
record systems and qualify for federal financial incentives. A national
sample of sixty-two hospitals voluntarily used a simulation tool designed Emily Welebob is an
independent consultant in
to assess how well safety decision support worked when applied to Indianapolis, Indiana.
medication orders in computerized order entry. The simulation detected
David W. Bates is division
only 53 percent of the medication orders that would have resulted in chief for general internal
fatalities and 10–82 percent of the test orders that would have caused medicine at Brigham and
Women’s Hospital in Boston,
serious adverse drug events. It is important to ascertain whether actual Massachusetts.
implementations of computerized physician order entry are achieving
Stuart Lipsitz is a researcher
goals such as improved patient safety. at Brigham and Women’s
Hospital.

David C. Classen is an
associate professor of
medicine at the University of

M
any people have suggested
that electronic health rec-
ords represent essential infra-
structure for the provision of
safe health care in the United
States. For several years, the Institute of Medi-
cine, the Leapfrog Group, the National Quality
Forum, and other national groups concerned
about patient safety have recommended, in par-
ticular, widespread adoption of electronic health
records with computerized physician order
entry.1–4
Background On Decision-Support
Tools
With computerized physician order entry, phy-
sicians and other licensed clinicians write their
orders for hospitalized patients electronically.
This recommendation is based in large part on
demonstrations by pioneering organizations.
The organizations have shown that important
improvements in safety can be achieved when
rules-based decision support aids in averting
medication errors and adverse events by provid-
ing advice and warnings as physicians write or-
ders using a specially programmed computer.5–7
In this application of clinical decision support, Utah in Salt Lake City, and is
physicians are made aware of potential safety also with CSC Healthcare.
issues that can result—for example, when ampi-
cillin is given to a patient with a known allergy to
penicillin, or the dose being ordered for a pedi-
atric patient is much higher than the therapeutic
range for a child of this age and weight. Prescrib-
ing errors such as these can lead to anaphylaxis
or seizures, which are known as adverse drug
events, if the medications are actually adminis-
tered. The goal of medication safety decision
support in computerized physician order entry
is to prevent these types of serious errors as the
orders are being written.
A study demonstrated that one in ten patients
hospitalized in Massachusetts suffered an ad-
verse drug event that could be prevented by
decision-support tools in computerized physi-
cian order entry.8 The study spurred the passage
of legislation requiring Massachusetts hospitals
to implement computerized physician order
entry by 2012 as a condition of licensure.9
More recently, “meaningful use” of computer-
ized physician order entry and clinical decision
support has been singled out as a requirement
for hospitals to qualify for new financial incen-
tives. These incentives will be offered under the

APRIL 2010 29:4 H E ALT H AF FAI RS 655

Focus On Quality
health information technology (IT) stimulus
provisions of the American Recovery and Rein-
vestment Act (ARRA) of 2009.
Computerized physician order entry interacts
with other applications in the suite of digital
tools that constitute the inpatient electronic
health record (for example, to obtain infor-
mation on allergies and patients’ weight) and
is typically one of the later modules to be imple-
mented. Hospitals’ adoption of the compu-
terized physician order entry module is in-
creasing, but slowly.10,11 Several reports have
suggested that the successful application of
decision support achieved among pioneering
organizations is not being replicated and that
implementation can create new problems.12–15
This report summarizes results for sixty-two hos-
pitals across the United States that used a new
simulation tool to assess their use of medication
safety decision support in electronic health
records with computerized physician order
entry.
Study Data And Methods
History Of The Assessment Tool The impetus
for developing the assessment tool was initially
the standard developed by the Leapfrog Group.
This is an employer group that seeks to accom-
plish breakthroughs, or “big leaps,” in hospital
patient safety through a combination of public
awareness and rewards to higher-quality provid-
ers. The group selected computerized physician
order entry as one of the first three leaps in 2001.
There was accumulating evidence concerning
the frequency, tragic consequences, and finan-
cial costs of adverse drug events in hospitalized
patients—and computerized physician order en-
try and decision support had demonstrated the
ability to help avert many of them.16
The Leapfrog standard includes two elements
of meaningful use to ensure that computerized
physician order entry has been implemented in
such as way as to improve medication safety.
According to the standards, physicians and other
licensed providers must enter at least 75 percent
of medication orders using computerized entry.
Clinical decision support must also be able to
avert at least 50 percent of “common, serious
prescribing errors.”16
Clinical decision support in this setting is the
logic built into the computerized physician order
entry system that, for example, checks to see if
ampicillin has been ordered for a patient who is
known to be allergic to penicillin. This tool was
developed to specifically measure the ability of
implemented electronic health record systems
with computerized physician order entry to de-
tect and avert these common yet serious pre-
656 H E ALT H AF FAI RS APRIL 2010 29:4
scribing errors in “live” hospital settings.
The tool was intentionally designed to give
individual hospitals detailed and specific feed-
back on their performance and to give purchas-
ers, through Leapfrog, an overall score for the
hospital that can be used for benchmarking
purposes.17,18
This assessment complements efforts by the
Certification Commission for Health Informa-
tion Technology (CCHIT) to evaluate the capa-
bilities available in vendors’ electronic medical
record products “on the shelf.” It evaluates how
products were implemented and are actually
being used in hospitals.
The development of the tool was initially
funded by the Robert Wood Johnson Foun-
dation, the California HealthCare Founda-
tion, and the Agency for Healthcare Research
and Quality, and was completed in 2006. In
April 2008 the assessment was incorporated into
the Leapfrog Annual Safe Practices Survey for the
first time, and hospitals completing the assess-
ment received a feedback report. Beginning in
2009, assessment results were also factored into
determining the extent to which the computer-
ized physician order entry implementation met
the Leapfrog standard.
Design Of The Assessment Tool The assess-
ment methodology is modeled after tools that are
commonly used in other industries. It mimics
what happens when a physician writes an order
for an actual patient in the implemented elec-
tronic health record with computerized physi-
cian order entry. But it uses test patients—in
effect, fictitious patients created for purposes
of the assessment—and test orders.
A group of experts on adverse drug events, as
well as the use of decision support in computer-
ized physician order entry to decrease adverse
drug events, developed test orders that are
judged likely to cause serious harm (rather than
those with low potential for harm). The test
orders belong to the categories of adverse
drug events (such as drug-to-allergy or drug-
to-diagnosis contraindication) that prior re-
search shows cause the most harm to patients.
In most cases, they are actual orders that have
caused adverse drug events, taken from primary
adverse drug event data collection studies. The
assessment offers a one-time, cross-sectional
look at whether decision support provides advice
to a physician writing such an order.18,19
Decision support for this purpose is a set of
tools or logic that can be integrated into the
computerized physician order entry system to
suggest appropriate orders (such as a dose cal-
culator or a reminder to consider renal function)
or to critique them once they have been entered,
as through a message or an alert. The assessment
gives credit for all of these forms of decision
support as relevant advice or information.
Because of differences in the epidemiology of
preventable adverse drug events, different ver-
sions of the assessment were designed for adult
and pediatric inpatient settings.
Use Of The Assessment Tool A designated
team in the hospital performs a self-assessment
in the following fashion. First, the team down-
loads instructions and information profiles for
ten to twelve test patients. Then the team down-
loads around fifty test orders, instructions, and
observation sheets to be used in the assessment.
A participating physician enters test orders for
the test patients into the local electronic health
record and observes and notes any guidance pro-
vided by decision support, such as the calculated
dose, a message or alert displayed, and so on. The
team enters the results obtained for each test
order (decision support received or not).17 The
assessment tool instantly computes an overall
score (percentage of test orders identified), as
well as the score for the orders in each adverse
drug event category. It then displays results to
the testing team. The entire process takes no
more than six hours, and many hospitals com-
plete it more quickly.
During the development period (2002–6),
multiple testing was performed at more than
twenty-five different hospitals. The testing re-
flecting all of the leading electronic health record
and computerized physician order entry ven-
dors’ products and ensured that the test could
effectively evaluate each product. Details of the
reliability and validity of the assessment meth-
odology are available in the Online Appendix.20
Analysis
Between April and August 2008, eighty-one U.S.
hospitals completed the version of the assess-
ment for adult patients. Test orders and posted
results were reviewed, and information poten-
tially identifying patients was removed. Nine
hospitals were eliminated because registration
information indicated that computerized physi-
cian order entry was being used only in the emer-
gency department rather than more broadly in
the hospital.
In addition, ten hospitals were excluded be-
cause they exceeded a deception-analysis thresh-
old based on standard gaming detection stra-
tegies (testing irregularities such as exceeding
time limits or multiple false positive results).
A review of excluded hospitals did not reveal
results that would have skewed the findings in
this study. The resulting sample contained sixty-
two hospitals.
The categories of adverse drug events ad-
dressed by the test orders include ones for which
decision-support tools are fairly straightforward
to implement (drug-to-drug or drug-to-allergy
interactions, therapeutic duplication, inappro-
priate single dose, and inappropriate route of
administration). The assessment also included
test orders that require more effort to configure
or customize—for example, use of an inappro-
priate daily dose or weight-based dose, drug-to-
age or drug-to-diagnosis contraindication, con-
traindications based on renal status or other
metabolic abnormalities indicated by laboratory
tests, or lack of monitoring.
The framework for basic and advanced deci-
sion support was based on prior work on clinical
decision support in computerized physician or-
der entry,21 and the categories are consistent
with recent research on preventable adverse
drug events.8
Statistical Analysis
The basic unit for the analyses was the hospital.
Overall scores for test orders and the scores for
orders assigned to the “basic” group were both
found to be approximately normally distributed.
However, scores for test orders in the “advanced”
group were slightly right-skewed. Thus, we pre-
sent additional statistics to aid in interpretation.
A fuller description of our analysis is available in
the Online Appendix.20
Tests based on standard assumptions about
the population, such as through parametric
tests, gave results almost identical to those of
tests that made no such assumptions (nonpara-
metric tests). Thus, for simplicity, parametric
tests are displayed (see the Online Appendix
for details).20
The total hospital scores were found to be ap-
proximately normally distributed, as is typically
the case with normal outcomes. Thus, we as-
sumed a linear regression model with the overall
score as the dependent variable (estimating the
relationships between the outcome and covari-
ates using ordinary least squares regression).
Covariates that we identified as having a possible
influence on total hospital score were vendor
(there were nine), teaching status, hospital size
groupings (number of beds), and whether or not
the hospital was part of a health system.
We tested for appropriateness of our linear
regression model using techniques that are de-
scribed in the Online Appendix.20 We also used
standard statistical techniques to determine the
percentage of the overall variation explained by
each factor.22
We tested our model for appropriateness using
techniques that are described in the Online Ap-
pendix.20 For dichotomous outcome variables,
APRIL 2010 29:4 H E A LT H A FFA IRS 657
Focus On Quality

such as test order detected (yes, no), percentages
were calculated. However, to account for the
hierarchical nature of the order data, or the test
orders nested within hospitals, the 95 percent
binomial confidence intervals for percentages
were adjusted using the approach described by
David Williams.23
All analyses were conducted using the statis-
tical software package SAS 9.2. All tests were two-
tailed, and a p value less than 0.05 (therefore
not likely to be due to chance) was considered
statistically significant.
Study Results
The types of hospitals included in the sample
were broadly representative of larger U.S. hospi-
tals (Exhibit 1). Nearly two-thirds were teaching
hospitals. The higher representation of teaching
hospitals and lower representation of smaller
hospitals is consistent with the current pattern
of adoption of computerized physician order en-
try in hospitals.10,11 Among the sixty-two hospi-
tals, all but one reported using electronic health
record applications including computerized
physician order entry from one of seven commer-
cial vendors.
Individual Hospitals Scores for individual
hospitals ranged from 10 percent to 82 percent
of test orders detected. The scores for the top
10 percent, or six hospitals, ranged from 71 per-
cent to 82 percent. Scores for the six hospitals
with the lowest scores ranged from 10 percent to
18 percent (Exhibit 2).
Mean hospital scores (Exhibit 3) were higher
for orders that would lead to adverse drug events
that can be addressed by basic decision support
(61 percent) than for those requiring more ad-
vanced decision support (25 percent).
Aggregate Results: Adverse Drug Events
When results for all hospitals were pooled, the
adverse drug event category detected most reli-
ably was drug-to-allergy contraindication. Much
higher scores were obtained for each of the cat-
egories addressed by basic clinical decision sup-
port than for those requiring advanced tools
(Exhibit 4).
Drug-to-diagnosis contraindication includes
pregnancy, which was also analyzed separately.
These potential adverse drug events were only
detected 15 percent of the time.
The set of test orders for each hospital includes
four that are judged to result in patient fatality.
When results for this subset were analyzed, we
found that 47 percent (95 percent confidence
interval: 36.9–57.6) were not detected by the
decision support in use in these hospitals.
Although hospitals do have pharmacy and nurs-
ing review processes in place that sometimes
catch orders like these before the medication
reaches the patient, these medication orders
are far outside safe limits and would never be
appropriate physician orders.
Contributing Factors The information
available for exploring contributing factors
was limited to the vendor software solution in
use, teaching status, hospital size by number of
beds, and whether or not the hospital was part of
a health system. We assessed the relationship
between performance on the assessment and
these factors.
High-low scores for hospitals using the same

EXHIBIT 1

Characteristics Of Hospitals Participating In The Study Of Computerized Physician Order Entry (CPOE), 2008
Participating hospitals
Characteristic Number Percent All U.S. hospitals (%) Hospitals with CPOE (%)
HOSPITAL SIZE (BEDS)
<50 0 0 32.9 8.9
50–99 3 4.8 15.9 11.8
100–199 14 22.6 22.4 12.8
200–299 8 13 12.7 15.3
300–399 9 15 7.7 30.1
400–799 23 37 7.3 33.9
800+ 5 8 0.9 41.2
TEACHING STATUS

Teaching 39 63 21.7 –
Nonteaching 23 37 78.3 –
SYSTEM STATUS

Independent 15 24 45 –
Multihospital health system 47 76 55 –

SOURCES: Authors’ analysis; and Notes 10 (for hospital size data) and 11 (for teaching and system status) in text.

658 HE A LT H A FFA IR S A P R I L 2 0 10 2 9 :4
 EXHIBIT 2

Hospital Scores For Detection Of Test Orders That Would Cause An Adverse Drug Event In An Adult Patient According To
The Software Product (Vendor) Implemented
Hospital score (percent)

SOURCE Authors’ analysis.

computerized physician order entry software
product ranged by as much as 40–65 percent
(Exhibit 2). Some hospitals using each product
detected at least 50 percent of the potential ad-
verse drug events. The six top-performing hos-
pitals used six different software products: one
homegrown solution and five vendor products.
In a multiple regression model, vendor choice
was significantly correlated with performance
(p ¼ 0:009, or not likely to be due to chance).
This means that there is good statistical evidence
to suggest that choice of vendors does have some
positive effect on performance. However, vendor
choice accounted for only 27 percent of the total

EXHIBIT 3

Hospital Mean Scores For Detecting Test Orders For Adult Patients Corresponding To Adverse Drug Event Categories
Addressed By Basic And Advanced Decision Support
Percent of test orders detected
Adverse drug event categories grouped according to
level of clinical decision support Mean (SE) Median Interquartile range
Basic—relatively easy to implementa 61.4 (2.4) 61.1b 53.9–76.2
Advanced—requires more configuration or customizationc 24.8 (2.6) 18.8b 5.9–38.9
Overall score 44.3 (2.3) 41.7 31.6–57.1

SOURCE Authors’ analysis. NOTES N ¼ 62 hospitals. SE is standard error. Interquartile range is the range in which the middle 50 percent
of the observations are seen—25 percent below the median, 25 percent above. ap < 0:0001 using a paired t-test for Basic
Score = Advanced Score. bDrug-to-drug or drug-to-allergy contraindication, inappropriate single dose, therapeutic duplication, and
inappropriate route. cInappropriate cumulative (daily) dose, inappropriate dose (patient weight), age or diagnosis contraindication,
contraindication based on renal function or other condition indicated by laboratory tests, lack of monitoring.

AP R I L 2 0 1 0 2 9 :4 HE A LT H A FFA IR S 659
Focus On Quality
EXHIBIT 4
Pooled Hospital Scores For Detecting Test Orders For Adults In Various Adverse Drug Event Categories
Adverse drug event category Percent detected
a
ADDRESSED BY BASIC CLINICAL DECISION SUPPORT
Drug-allergy contraindication 83.3
Inappropriate single dose 46.4
Therapeutic duplication 54.5
Drug-drug interaction 52.4
Inappropriate route 65.3
b
ADDRESSED BY ADVANCED CLINICAL DECISION SUPPORT
Inappropriate cumulative (daily) dose 39.1
Inappropriate dosing (patient weight) 36.7
Age contraindication 14.1
Labs—creatinine 20.2
Labs—other 26.1
Drug-diagnosis contraindication 15.0
Corollary orders (monitoring) 27.0
SOURCE Authors’ analysis. NOTE N ¼ 62 hospitals. aImplementation of applicable decision support is relatively straightforward.
b
Implementation of applicable decision support requires more effort to configure or customize.
variation that we observed in performance.
Teaching status also correlated significantly
with performance (p ¼ 0:007, very unlikely to
be due to chance), accounting for 10 percent
of the observed variation in performance. But
hospital size and being part of a hospital system
did not correlate significantly with performance.
At least as far as we can detect statistically, hos-
pital size and being part of a hospital system do
not influence performance one way or the other.
We tested for interactions between vendor and
all other variables, and none were significant
(p > 0:2). Finally, although we were able to
account for only 39 percent of the observed
variation in performance (overall R2 ¼ 0:39),
standard tests (goodness-of-fit statistics and as-
sessments) indicated that our model was appro-
priate for our study, as detailed in the Online
Appendix.20
Discussion
Many of the benefits from inpatient electronic
health records and the computerized physician
order entry module in particular come from de-
cision support.24 In this study, we found wide
variation in the ability of implemented comput-
erized physician order entry decision support to
detect medication orders judged likely to cause
serious harm to adult patients.
Many hospitals performed poorly, and the
mean score was only 44 percent of potential
adverse drug events detected. However, top-
performing hospitals in this sample achieved
scores of 70–80 percent or greater. To achieve
these higher results, the hospitals have imple-
66 0 H E ALT H AF FAI RS APRIL 2010 29:4
Lower 95 percent Upper 95 percent
confidence interval confidence interval
77.7 87.8
37.6 56.6
43.7 64.9
43.4 61.3
55.7 72.5
28.9 50.4
27.9 46.4
7.9 24.0
12.9 30.1
18.7 35.1
9.9 22.1
19.7 35.7
mented advanced clinical decision support, as
well as basic tools, and their performance pro-
vides a benchmark for the continuing efforts of
lower-performing hospitals.
A comparison of aggregate scores with the
findings from a recent study of adverse drug
events, based on chart reviews in six community
hospitals not using the computerized physician
order entry module of the inpatient electronic
health record,8 showed that decision support in
the sixty-two hospitals is doing a much better job
detecting adverse drug events that occur infre-
quently than those that occur more frequently.
The categories in our study with the three high-
est detection rates—drug-to-allergy, drug-to-
drug, and duplicate medication—together only
contributed 7 percent of the adverse drug events
in the community hospital chart review study.
Fifty-five percent of the adverse drug events in
the community hospital chart review study
required considering laboratory results or pa-
tients’ age in determining medication appropri-
ateness or dosing. For these categories in
aggregate, corresponding test orders were de-
tected only 20.6 percent of the time—not very
good odds from the patients’ perspective.
Different Products In Use The high variabil-
ity found, including among hospitals using the
same electronic health record software product,
emphasizes the importance of this type of assess-
ment to gauge the extent to which decision
support is being used. In fact, this is the ration-
ale used by the Leapfrog Group and the Na-
tional Quality Forum in including the use of
the assessment tool in their recommended safe
practices.4,16
Work is now under
way to begin to
develop libraries of
decision-support rules
and practical advice.
Hospital boards and executive teams who are
investing in these systems, practicing physicians
who adopt the technology in their routine work
to improve patient safety, and external stake-
holders requiring the use of electronic health
records with computerized physician order entry
or providing incentives for adoption all need a
way to gauge progress in applying decision sup-
port to improve medication safety. This becomes
acutely important as hospitals attempt to meet
meaningful-use requirements to quality for fed-
eral stimulus incentives.
Assessing Patient Safety Given the large
investment in achieving meaningful use nation-
wide, it will be important to ascertain whether
actual implementation is achieving important
goals such as improved patient safety. To our
knowledge, this test is the first such objective
evaluation of electronic health record systems
in actual use.
The assessment measures the extent to which
clinical decision support is providing some form
of advice or an alert in response to medication
orders that would cause an adverse drug event.
The design and scope of decision-support capa-
bilities in electronic health record software prod-
ucts vary, and local hospital configuration and
customization are always required.
Although there was a relationship between the
product involved and performance on the assess-
ment, only 27 percent of the variation is associ-
ated with using different electronic health record
products. This suggests that other factors had a
bigger influence on the wide variation in perfor-
mance among the hospitals. Although results are
presented for one homegrown system, this is not
a sufficient sample from which to draw any gen-
eral conclusions.
In most hospitals, the use of decision support
grows over time. At the outset, managing it is a
new process.6 Including too much, poorly de-
signed clinical decision support initially can
even contribute to lack of computerized physi-
cian order entry adoption.25
Exploring Variability In Use Several factors
are contributing to the variability observed.
These include the completeness and ease of
applying the decision-support tool sets in the
electronic health record software; relevant
knowledge and experience within the hospital;
the availability and commitment of staff resour-
ces; and whether the use of clinical decision sup-
port began recently or has been honed over
many years.
There are multiple possible explanations for
the observed correlation between hospital teach-
ing status and performance. These include such
factors as research interest and having more staff
resources to invest. However, these could not be
explored further in this study.
For specific categories of adverse drug events,
there are also multiple possible explanations for
variability among hospitals. For drug-to-diagno-
sis contraindications, for example, many hospi-
tals are likely not yet applying decision support
because there is no constantly updated elec-
tronic problem list maintained by physicians
for patients during their hospital stay.
The assessment also does not provide insight
into whether or not the information or advice
being presented is followed in practice—another
prerequisite for improving medication safety.26
A better understanding of all of these factors is
needed so that strategies for speeding progress
can be devised.
Some reports have suggested one remedy: pro-
viding assistance to hospitals in using decision
support effectively.3,21,27 This assessment tool
provides guidance to the hospital in the form
of a scorecard. Several publications exist to guide
hospitals in their planning for and decisions
about clinical decision support.21,26,28,29 Also,
some vendors of computerized physician or-
der entry software provide clinical decision sup-
port starter sets or facilitate sharing among
customers.
Work is now under way to begin to develop
libraries of decision-support rules and practical
advice, although it is in early stages.
Study Limitations This study has several lim-
itations. Like all voluntary surveys, the study is
subject to possible response bias. It is likely that
the hospitals completing the assessment were
more interested than many of their peers were
in clinical decision support because it was
initially offered as a learning exercise instead
of being required for the annual Leapfrog survey.
Another limitation is that the sixty-two hospi-
tals in the study represent about 8 percent of U.S.
hospitals with the computerized physician order
entry module of the inpatient electronic health
record in use.10 Thus, they might not be
representative. The fact that this is the first test
of its kind means that there is no “gold standard”
AP R I L 2 0 1 0 2 9 :4 HE A LT H A FFA IR S 6 61
Focus On Quality

for comparison of these results. However, during
development of the assessment tool, the reliabil-
ity and validity were extensively evaluated and
found to be very high.
Finally, because results are self-reported, hos-
pitals may have reported better performance
than was actually assessed. However, it should
be noted that public reporting of overall scores,
which could influence hospitals’ behavior, was
not implemented until 2009, and hospitals were
asked to participate in part to aid them with
improving their decision support. To address
the potential for gaming, several standard safe-
guards were built into the test to detect patterns
of use that suggest that gaming may be going on.
Any assessment that included these character-
istics was excluded from analysis.
Conclusions
In sixty-two hospitals using an inpatient elec-
tronic health record with computerized physi-
cian order entry, we found significant vari-
ability in the use of decision support to detect
and provide advice or an alert concerning a medi-
cation order that would result in serious harm to
an adult patient. Some hospitals performed very
well, while others performed very poorly. In ad-
dition, the studied hospitals as a group were
using basic decision support far more than the
more advanced tools needed to detect types of
orders that are major contributors to adverse
drug events in chart-review studies.
These findings point to the importance of eval-
uations of the use of clinical decision support by
hospitals to help guide their continuing efforts
to improve medication safety. In addition, incen-
tives to hospitals relating to computerized physi-
cian order entry should include some type of
demonstration that clinical decision support is
actually being employed, instead of being based
solely on whether computerized physician order
entry is in use.
The broader use of this type of assessment
of meaningful electronic health record use
should be explored for other software applica-
tions used in direct clinical care. ▪

The authors thank Peter Kilbridge, Fran
Turisco, and the project expert advisers,
as well as individuals in many hospitals,
health systems, and software vendor
companies, for their involvement and
assistance in some phase of the
development and testing of the
assessment tool used in the study. They
also thank the Leapfrog Group for
access to the data for analysis. The
development of the assessment tool
used in this study was supported by
grants from the California HealthCare
Foundation, the Robert Wood Johnson
Foundation, and the Agency for
Healthcare Research and Quality
(Contract no. 290-04-0016, Subcontract
no. 6275-FCG-01). These organizations
did not sponsor and were not involved
in the reported analysis or preparation
of the manuscript. Four of the authors
(Jane Metzger, Emily Welebob, David W.
Bates, and David C. Classen) were
involved in the development of the
assessment tool. The content is solely
the responsibility of the authors and
does not necessarily represent the
official views of the funding agency.

NOTES
1 Institute of Medicine. Preventing
medication errors: quality chasm
series. Washington (DC): National
Academies Press; 2007.
2 Institute of Medicine. Crossing the
quality chasm: a new health system
for the 21st century. Washington
(DC): National Academies Press;
2001.
3 Aspden P, Corrigan JM, Wolcott J,
Erickson SM. Patient safety: achiev-
ing a new standard of care. Wash-
ington (DC): National Academies
Press; 2003.
4 National Quality Forum. Safe prac-
tices for better healthcare: a con-
sensus report [Internet]. Washing-
ton (DC): National Quality Forum;
2003 [cited 2010 Feb 21]. Available
from: http://www.ahrq.gov/qual/
nqfpract.pdf
5 Bates DW, Leape LL, Cullen DJ, Laird
N, Petersen LA, Teich JM, et al. Ef-
fect of computerized physician order
entry and a team intervention on
prevention of serious medication
errors. JAMA. 1998;280(15):1311–6.
6 Evans RS, Pestotnik SL, Classen DC,
Clemmer TP, Weaver LK, Orme JF
Jr., et al. A computer-assisted man-
agement program for antibiotics and
other anti-infective agents. N Engl J
Med. 1998;338(4):232–8.
7 Teich JM, Merchia PR, Schmiz JL,
Kuperman GJ, Spurr CD, Bates DW.
Effects of computerized physician
order entry on prescribing practices.
Arch Intern Med. 2000;160(18):
2741–7.
8 Adams M, Bates DW, Coffman G,
Everett W. Saving lives, saving
money: the imperative for comput-
erized physician order entry in
Massachusetts hospitals [Internet].
Boston (MA): Massachusetts Tech-
nology Collaborative and New
England Healthcare Institute; 2008
[cited 2010 Feb 21]. Available from:
http://web3.streamhoster.com/
mtc/cpoe20808.pdf
9 Getz L. On the right track in Mas-
sachusetts. For the Record [serial on
the Internet]. 2008 Nov 11; 20(23)
[cited 2010 Feb 21]. Available from:
http://www.fortherecordmag.com/
archives/ftr_111008p16.shtml
10 Pedersen CA, Gumpper KF. ASHP
survey on informatics: assessment of
the adoption and use of pharmacy
informatics in U.S. hospitals—2007.
Am J Health Syst Pharm. 2008;65:
2244–64.
11 American Hospital Association.
Continued progress: hospital use of
information technology [Internet].
Chicago (IL): AHA; 2007 [cited 2010
Feb 21]. Available from: http://
www.aha.org/aha/content/2007/
pdf/070227-continuedprogress.pdf
12 Nebeker JR, Hoffman JM, Weir CR,
Bennett CL, Hurdle JF. High rates of
adverse drug events in a highly
computerized hospital. Arch Intern
Med. 2005;165(10):1111–6.
13 Walsh KE, Landrigan CP, Adams
WG, Vinci RJ, Chessare JB, Cooper
MR, et al. Effect of computer order
entry on prevention of serious
medication errors in hospitalized
children. Pediatrics. 2008;121(3):
e421–7.
14 Koppel R, Metlay JP, Cohen A,
Abaluck B, Localio AR, Kimmel SE,
et al. Role of computerized physician

66 2 H E ALT H AF FAI RS APRIL 2010 29:4

 order entry systems in facilitating
medication errors. JAMA. 2005;293
(10):1197–203.
15 Han YY, Carcillo JA, Venkataraman
ST, Clark RS, Watson RS, Nguyen
TC, et al. Unexpected increased
mortality after implementation of a
commercially sold computerized
physician order entry system. Pedi-
atrics. 2005;116(6):1506–12.
16 Leapfrog Group. Fact sheet: com-
puterized physician order entry.
Washington (DC): Leapfrog Group;
2009 Mar 3 [cited 2010 Feb 21].
Available from: http://www.leap
froggroup.org/media/file/
FactSheet_CPOE.pdf
17 Metzger JB, Welebob E, Turisco F,
Classen DC. The Leapfrog Group’s
CPOE standard and evaluation tool.
Patient Safety and Quality Health-
care [serial on the Internet]. 2008
Jul/Aug: p. 22–5 [cited 2010 Feb 21].
Available from: http://www.psqh
.com/julaug08/cpoe.html
18 Kilbridge P, Welebob E, Classen DC.
Overview of the Leapfrog Group
evaluation tool for computerized
physician order entry [Internet].
Lexington (MA): Leapfrog Group
and First Consulting Group; 2001
Dec [cited 2010 Feb 21]. Available
from: http://www.leapfroggroup
.org/media/file/Leapfrog-CPOE_
Evaluation2.pdf
19 Kilbridge PM, Welebob EM, Classen
DC. Development of the Leapfrog
methodology for evaluating hospital
implemented inpatient computer-
ized physician order entry systems.
Qual Saf Health Care. 2006;15(2):
81–4.
20 The Online Appendix can be ac-
cessed by clicking on the Online
Appendix link in the box to the right
of the article online.
21 Kuperman G, Bobb A, Payne TH,
Avery AJ, Gandi TK, Burns G, et al.
Medication-related clinical decision
support in computerized provider
order entry systems: a review. J Am
Med Inform Assoc. 2007;14(1):
29–40.
22 Lin DY, Wei LJ, Ying Z. Model-
checking techniques based on cu-
mulative residuals. Biometrics.
2002;58(1):1–12.
23 Williams DA. Extra-binomial varia-
tion in logistic linear models. Appl
Stat. 1982;31;144–8.
24 Kaushal R, Shojania KG, Bates DW.
Effects of computerized physician
order entry and clinical decision
support systems on medication
safety: a systematic review. Arch In-
tern Med. 2003;163(12):1409–16.
25 Connolly C. Cedars-Sinai doctors
cling to pen and paper. Washington
Post. 2005 Mar 21. p. A01.
26 Bates DW, Kuperman GJ, Wang S,
Gandi T, Kitter A, Volk L, et al. Ten
commandments for effective clinical
decision support: making the prac-
tice of evidence-based medicine a
reality. J Am Med Inform Assoc.
2006;13:523–30.
27 Osheroff JA, Teich JM, Middleton
BF, Steen EB, Wright A, Detmer DE.
A roadmap for national action on
clinical decision support. Bethesda
(MD): American Medical Infor-
matics Association, CDS Roadmap
Steering Committee; 2006 Jun 13
[cited 2008 Sep 14]. Available from:
https://www.amia.org/files/
cdsroadmap.pdf
28 Osheroff JA, Pifer EA, Sittig DF,
Jenders RA, Teich JM. Clinical deci-
sion support implementers’ work-
book. 2nd ed. Chicago (IL): Health
Information Management Systems
Society; 2004 [cited 2010 Mar 9].
Available from: http://www
.himss.org/cdsworkbook
29 Health Information and Manage-
ment Systems Society. Improving
outcomes with clinical decision
support: an implementers’ guide.
Chicago (IL): HIMSS; 2005.
AP R I L 2 0 1 0 2 9 :4 HE A LT H A FFA IR S 6 63