The Appropriateness Evaluation Protocol is a poor predictor

of in-hospital mortality

ORegan NA, Healy L, O Cathail M, Law TW, OCarroll G, Clare J, Timmons S,

OConnor KA

Niamh Annmarie ORegan, Centre for Gerontology and Rehabilitation, School of Medicine, University
College Cork, Ireland
Liam Healy, Department of Geriatric Medicine, South Infirmary Victoria University Hospital, Cork,
Ireland
Michel Cathail, Department of Geriatric Medicine, Mercy University Hospital, Cork, Ireland
Tze Wen Law, Department of Geriatric Medicine, Mallow General Hospital, Co. Cork, Ireland
Grace OCarroll, Department of Geriatric Medicine, Mercy University Hospital, Cork, Ireland
Josie Clare, Department of Geriatric Medicine, Waterford Regional Hospital, Waterford, Ireland
Suzanne Timmons, Centre for Gerontology and Rehabilitation, School of Medicine, University College
Cork, Cork, Ireland
Kieran A OConnor, Department of Geriatric Medicine, Mercy University Hospital, Cork, Ireland

Corresponding Author: Dr. Niamh ORegan, email: niamh.oregan@ucc.ie

Abstract word count:

239

Manuscript word count:

1,741

References:

16

Author contributions
ICJME criteria for authorship:
Authors have 1) substantial contributions to conception and design, acquisition of
data, or analysis and interpretation of data; 2) drafting the article or revising it
critically for important intellectual content; and 3) final approval of the version to be
published.
ORegan NA 1,2,3; Healy L 1,2,3; O Cathail M 1,2,3; Law TW 1,2,3; OCarroll G
1,2,3; Clare J 1,2,3; OConnor KA 1,2,3; Timmons S 1,2,3.
ORegan, Clare, OConnor and Timmons: all substantially contributed to study
conception and design, data collection, statistical analysis and interpretation, drafting
the initial manuscript, and subsequent critical revision of the manuscript and final
approval.
Healy, O Cathail, Law, and OCarroll: all majorly involved in data acquisition,
critical revisions and final approval of the manuscript.

Introduction
Appropriate allocation of resources is fiscally important in delivering
healthcare. Reducing acute hospital admissions is viewed as a key method to curb
healthcare costs. Hospital utilisation review (UR) programmes have been shown to
reduce utilisation and expenditures. However, there is no evidence that UR improves
overall efficiency of resource use. Furthermore, its impact on care delivery and patient
outcomes has not been described.
Many tools have been developed to measure acute hospital bed utilisation. The
Appropriateness Evaluation Protocol (AEP), developed in 1981 by Gertman and
Restuccia is the most widely used and is regarded as superior to other tools. It was
developed as an objective, diagnosis-independent, criteria-based system to
prospectively determine appropriateness of hospital days (see figure 1). It is divided
into Admission Criteria and Day of Care Criteria, which are used to define
respectively whether a patients admission to an acute bed was appropriate; and
whether or not continuing care in that setting was required. The Admission Criteria
consists of Severity of Illness and Intensity of Service Required Criteria: a patient
must meet one of these criteria to be deemed appropriate for admission. The Severity
of Illness criteria are based on objective measures of disease severity, divided into ten
categories with six descriptive parameters (e.g. acute loss of sight or hearing), scored
as present or absent; and seven numerical physiological parameters (e.g. diastolic
blood pressure) with cut-off values beyond which results are regarded as sufficiently
abnormal to warrant admission (see figure 1). Its authors conceded that such a concise
list of criteria could never capture all circumstances requiring admission, but would
facilitate usability of the tool. Hence, they included an override facility, where if

necessary, clinical judgment could be used to overrule the decision based on the
AEPs parameters. During its development and validation (against a gold standard of
senior clinical opinion), there was good overall agreement between raters, but poor
specific agreement.
The AEP was initially developed in the U.S.A., and its original and modified
forms have since been studied and utilised in many countries, most commonly as a
tool to retrospectively ascertain levels of admission appropriateness. In Ireland, the
AEP was recently used to inform a National Acute Hospital Bed Review which stated
that on average 12% of acute hospital admissions are inappropriate.
A systematic review of methods to ascertain appropriateness of acute hospital
use, performed by McDonagh et al in 2000, declared the AEP the most valid and
reliable instrument available, but there are well-documented concerns with the use of
any tool such as this to identify inappropriate admissions. Sikka et al showed that
severity of illness tool are poorly predictive of outcomes in older patients. Hence, the
primary objective of this study was to assess the clinical utility of the AEP in
assessing appropriateness of admission, in older patients compared to younger
patients.
Methods
We used analytical retrospective cohort study design to assess admission
parameters of adult patients who died within ten days of emergency admission to four
hospitals in Cork and Waterford: Waterford Regional Hospital (WRH); Mercy
University Hospital (MUH); South Infirmary Victoria University Hospital (SIVUH);
and Mallow General Hospital (MGH). All patients had been admitted under medical,
surgical or other specialty services. Ethical approval was granted by the local research

ethics committee. The average study period was 17 months from 1st January 2009 to
31st December 2010. We used proximate death as a robust measure of illness severity.
The Hospital Inpatient Enquiry System (HIPE) identified patients who were eligible
for inclusion. We used available ED charts, paramedic documentation, and
computerised laboratory records to assign AEP criteria. Given that, in general, it is
more likely that abnormal rather than normal physiology is documented with most
consistency, missing datapoints were taken as not meeting the AEP criteria. We also
collected demographic data, details of clinical presentation and measured length of
stay before death. Cause of death was not available. Data were analysed using
Statistical Product and Service Solutions (SPSS version 18).
Results
There were 80,970 admissions in the study period and 1,416 in-hospital deaths.
Of these, 803 occurred within ten days of admission. Of the 803 requested, 585
unselected (72.9%)charts were available for review and 490 (83.8%) of these patients
were aged 65 years or older. The median age was 79 years and 50.8% were female.
The mediantime from admission to death was approximately four days in both groups
(see table 1). The primary reason for ED attendance is illustrated in table 2. Vital signs
and serum electrolytes were available in 437 / 585 patients (74.7%). ECGs were either
reported or available for review in 349 / 585 (59.7%) patients. Venous bicarbonate
and / or pH was documented in 184 / 585 (31.5%).Of 803 requested charts,218
(27.1%) were unavailable for multiple reasons (e.g. were stored off-site; damaged
during the Cork floods). The proportion of unavailable charts was spread evenly
across the four hospitals, ranging from 20.5% in SIVUH to 30.4% in WRH.

When we applied the AEP criteria, 179 of 585 patients (30.6%)were AEP noncompliant on admission: 31 / 95 (32.6%) younger patients and 148 / 490 (30.2%) of
older. Seventy-two of 179 patients(40.2%) who did not meet the admission criteria
had been coded as severely unwell on ED arrival by triage staff: 13/31 (41.9%)
younger; 59 / 148 (39.8%) older. In those who were AEP compliant, the most
common parameter reached in both age groups was abnormal diastolic blood pressure,
followed by abnormal GCS (Glasgow Coma Scale), figure 2. Derangements in serum
sodium and potassium were also very common across all age groups, but only a small
proportion of these were severe enough to meet admission criteria, figure 2. Very few
patients (42 / 585, 7.2%) met the criteria for temperature (>37.78C), but other
temperature abnormalities, including hypothermia, were common in both age groups.
Discussion
This study highlights problems with using the AEP retrospectively to assess
admission appropriateness in a severely unwell cohort. Our study hypothesis was that
the AEP may be less effective at identifying severe illness in older patients.
Worryingly, our study has illustrated its deficiency in recognising moribund younger
patients also.
Our study has some limitations. Firstly, we only reviewed ED charts and
paramedic documentation and not the complete medical charts from all admissions,
hence may have missed some important data that was documented at a later point in
admission.Additionally, due to difficulty obtaining medical records, we were unable
to review 27% of the total eligible patient charts, and given that demographic
informationpertaining to these missed cases is unavailable to us, this may be a source
of selection bias. Nonetheless, chart availability was dependent on circumstances out

of our control and all available charts were reviewed. Being retrospective and
dependent on ED charts, we were unable to discern if individual missing data points
had been measured or simply not documented. We resolved to treat missing data as
not meeting the criteria, as when documentation is concise, we concluded that it is
probably more likely to reflect abnormalities rather than normal values. Of note, bed
utilisation reviews are also performed retrospectively, and hence, encounter similar
difficulties. Importantly, in such reviews, missing datapoints are similarly considered
AEP non-compliant. Furthermore, we cannot be sure that some patients did not
succumb to nosocomial illness, as cause of death was unavailable. Given the median
time to death of four days, it is likely that most patients were severely unwell on
arrival.
To our knowledge, the only other study which describes the prognostic
significance of the AEPwas conducted by Braband and colleagues. This prospective
study of patients admitted to a Danish medical admission unit found that 38% of
admissions did not meet the AEP criteria. Additionally, of 84 in-hospital deaths,
12.4% were AEP non-compliant on admission. Of note, all of these patients died
within 10 days of admission with a median time to death 4.75 days (similar to our
study). Although a similar proportion of the Danish cohort had ECG data available
(61.3%) as had in our study (59.7%), we had more missing datapoints (25.3% versus
<10%), which may account for the difference in AEP non-compliance rates. Multiple
other scoring systems based on patient physiology have been used for decades to
predict illness severity, particularly in the intensive care setting, such as APACHE
(Acute Physiology and Chronic Health Evaluation) and SAPS (Simplified Acute
Physiologic Score), and can provide accurate predictions of inpatient mortality risk.
More recently, track and trigger systems such as the MEWS (Modified Early Warning

Score)have been shown to have utility in identifying general medical and surgical
patients at risk of rapid deterioration and critical illness. These systems, however, are
used to assess patients who have already been admitted and hence are not suitable nor
designed for ascertaining admission appropriateness.
Importantly, bed utilisation review programmes were originally designed to
concurrently screen cases requiring further clinical review. Instead they are being used
retrospectively to assess performance in acute hospitals. To us, this is a major
concern, given that almost one-third of our moribund cohort were missed by the AEP
when used retrospectively.
When we scrutinise the severity of illness criteria of the AEP, many parameter
cut-offs are startlingly exclusive. Taking sodium level for example, a patient meets the
AEP criteria with a sodium level of <123 mmol/L or >156 mmol/L, however it is
well-recognised that sodium levels between these two values can be associated with
severe illness. Moreover, it is the acuity of the sodium level changes, and not
necessarily one isolated result, that generates clinical concern. To illustrate further,
pyrexial patients are considered appropriate for admission, but the criteria neglects
hypothermia as an important parameter. The authors allowed for the instruments
exclusivity by applying a clinical override facility, but this is not often used in
practice.
Patients and referring doctors favour alternatives to acute medical care as
shown by Campbell in 2001. Unfortunately, adequate alternatives are rarely available.
Our fragmented health service in Ireland means that options vary based on geographic
location and other factors. Critics argue that if there is no available alternative, then an
admission should not be labelled avoidable .

Obviously we cannot ignore the concept of appropriateness of healthcare

resource use. It is intuitive that we should have the right services for the right
people in the right place. However, this paper highlights significant concerns with
the current methods employed to ascertain admission appropriateness, as well as the
lack of focus on safe alternatives which is central to the issue.

Funding
This study received no external funding.

Tables and figures

Table 1: Patient demographics

Total

Younger

Older

(<65

(65

n=

Age (years), median

79 (21-103)

years)
57 (21- 64)

years)
82 (65-

585

(range)
Gender, n (%)

288 (49.2)

63 (66.3)

103)
225 (45.9)

male

297 (50.8)

32 (33.7)

265 (54.1)

female
LOS (days), mean (SD)

4.52 (3)

4.14 (3)

4.59 (3)

n=

432
1
Independent samples t-test

Sig.

p=0.25
31

10

Primary reason for ED attendance

Number (% total)

Respiratory

166 (28.4)

CVS

83 (14.2)

GIT / Hepatobiliary

64 (10.9)

Sepsis

56 (9.6)

Oncology / Haematology

45 (7.7)

Stroke / SAH

45 (7.7)

Trauma / Falls

23 (3.9)

Vascular / Skin / Soft Tissue / Joint

21 (3.6)

GU / Renal

18 (3.1)

LOC / Confusion / Neurology

17 (2.9)

Other

11 (1.9)

Not documented

36 (6.2)

Table 2: Reason for attendance grouped by category

ED= Emergency Department; CVS= Cardiovascular; GIT= Gastrointestinal; SAH=

subarachnoid haemorrhage; GU= genitourinary; LOC= loss of consciousness

11

Figure 1: The Appropriateness Evaluation Protocol:

Admission Criteria, A. Severity of Illness
A.1

Sudden onset of unconsciousness or disorientation (coma or unresponsiveness)

A.2

Pulse rate:
a)
< 50 per minute
b)
>140 per minute

A.3

Blood Pressure:
a)
Systolic < 90 or > 200 mmHg
b)
Diastolic < 60 or > 120 mmHg

A.4

Acute loss of sight and hearing

A.5

Acute loss of ability to move body part

A.6

Persistent fever for > 5 days:

a)
37.78 C (100 F) orally, or
b)
38.33 C (101 F) rectally

A.7

Acute bleeding

A.8

Severe electrolyte or blood gas abnormality (any of the following):

a)
Na < 123 mmol/L
Na > 156 mmol/L
b)
K < 2.5 mmol/L
K > 6.0 mmol/L
c)
Venous bicarbonate (unless chronically abnormal) < 20 mmol/L
Venous bicarbonate (unless chronically abnormal) > 36 mmol/L
d)
Arterial pH < 7.30
Arterial pH > 7.45

A.9

Electrocardiogram evidence of acute ischaemia; must be suspicion of a new

myocardial infarction

A.10

Wound dehiscence or evisceration

12

Figure 2:Graph showing the percentage of patients (in two

age groups) who (1) met each physiological AEP criterium
for admission, and (2) who had abnormal values outside
those specified by the AEP criteria (intemperature, serum
sodium and serum potassium) (HR = heart rate, SBP = systolic
blood pressure, DBP = diastolic blood pressure, Temp =
temperature, Na = serum sodium, K = serum potassium, GCS =
Glasgow Coma Scale (i.e. criterium A1), ECG = electrocardiogram
(evidence of new ischaemia), pH = = arterial pH, Bicarb venous
bicarbonate)

13

References:
1. Wickizer TM, Wheeler JR, Feldstein PJ (1989) Does
utilization review reduce unnecessary hospital care and
contain costs? Med Care 27 (6):632-647
2. Coast J (1996) Appropriateness versus efficiency: the
economics of utilisation review. Health Policy 36 (1):69-81.
doi:0168851095007970 [pii]
3. Gertman PM, Restuccia JD (1981) The appropriateness
evaluation protocol: a technique for assessing unnecessary
days of hospital care. Med Care 19 (8):855-871
4. McDonagh MS, Smith DH, Goddard M (2000) Measuring
appropriate use of acute beds. A systematic review of
methods and results. Health Policy 53 (3):157-184.
doi:S0168851000000920 [pii]
5. Strumwasser I, Paranjpe NV, Ronis DL, Share D, Sell LJ
(1990) Reliability and validity of utilization review criteria.
Appropriateness Evaluation Protocol, Standardized
Medreview Instrument, and Intensity-Severity-Discharge
criteria. Med Care 28 (2):95-111
6. Feely E (2008) Health Service Executive Second National
Acute Hospital Bed Review.
7. Vetter N (2003) Inappropriately delayed discharge from
hospital: what do we know? BMJ 326 (7395):927-928.
doi:10.1136/bmj.326.7395.927 326/7395/927 [pii]
8. Sikka P, Jaafar WM, Bozkanat E, El-Solh AA (2000) A
comparison of severity of illness scoring systems for elderly
patients with severe pneumonia. Intensive care medicine 26
(12):1803-1810
9. Brabrand M, Knudsen T, Hallas J (2011) The characteristics
and prognosis of patients fulfilling the Appropriateness
Evaluation Protocol in a medical admission unit; a
prospective observational study. BMC health services
research 11:152. doi:10.1186/1472-6963-11-152
10. Knaus WA, Draper EA, Wagner DP, Zimmerman JE (1985)
APACHE II: a severity of disease classification system.
Critical care medicine 13 (10):818-829
11. Le Gall JR, Loirat P, Alperovitch A, Glaser P, Granthil C,
Mathieu D, Mercier P, Thomas R, Villers D (1984) A simplified
acute physiology score for ICU patients. Critical care
medicine 12 (11):975-977
12. Subbe CP, Kruger M, Rutherford P, Gemmel L (2001)
Validation of a modified Early Warning Score in medical
admissions. QJM : monthly journal of the Association of
Physicians 94 (10):521-526

14

13. Wright MM, Stenhouse CW, Morgan RJ (2000) Early

detection of patients at risk (PART). Anaesthesia 55 (4):391392
14. Subbe CP, Davies RG, Williams E, Rutherford P, Gemmell
L (2003) Effect of introducing the Modified Early Warning
score on clinical outcomes, cardio-pulmonary arrests and
intensive care utilisation in acute medical admissions.
Anaesthesia 58 (8):797-802
15. Campbell J (2001) Inappropriate admissions: thoughts of
patients and referring doctors. Journal of the Royal Society
of Medicine 94 (12):628-631
16. Beech R (2008) Identifying the Need and Scope for
Intermediate Care. In: Intermediate and Continuing Care:
Policy and Practice. Blackwell Publishing Ltd, pp 19-32.
doi:10.1002/9780470774960.ch2

15