683099

research-article2017
SJS0010.1177/1457496916683099The introduction of adult appendicitis scoreH. E. Sammalkorpi, et al.

Original Research Article

SJS
SCANDINAVIAN
JOURNAL OF SURGERY

The Introduction Of Adult Appendicitis Score Reduced

Negative Appendectomy Rate

H. E. Sammalkorpi1,2, P. Mentula1, H. Savolainen3, A. Leppäniemi1

1  Department of Gastrointestinal Surgery, Helsinki University Central Hospital, Helsinki, Finland
2  Faculty of Medicine, University of Helsinki, Helsinki, Finland
3  Department of Surgery, Kuopio University Hospital, Kuopio, Finland

Abstract

Background and Aims: Implementation of a clinical risk score into diagnostics of acute
appendicitis may provide accurate diagnosis with selective use of imaging studies. The
aim of this study was to prospectively validate recently described diagnostic scoring
system, Adult Appendicitis Score, and evaluate its effects on negative appendectomy rate.
Material and Methods: Adult Appendicitis Score stratifies patients into three groups:
high, intermediate, and low risk of appendicitis. The score was implemented in diagnostics
of adult patients suspected of acute appendicitis in two university hospitals. We analyzed
the effects of Adult Appendicitis Score on diagnostic accuracy, imaging studies, and
treatment. The study population was compared with a reference population of 829 patients
suspected of acute appendicitis originally enrolled for the study of construction of the
Adult Appendicitis Score.
Results: This study enrolled 908 patients of whom 432 (48%) had appendicitis. The score
stratified 49% of all appendicitis patients into high-risk group with specificity of 93.3%.
In the low-risk group, prevalence of appendicitis was 7%. The histologically confirmed
negative appendectomy rate decreased from 18.2% to 8.7%, p<0.001, compared to the
original dataset.
Conclusion: Adult Appendicitis Score is a reliable tool for stratification of patients into
selective imaging, which results in low negative appendectomy rate.
Key words: Appendicitis; appendicitis score; diagnosis; diagnostic score; acute abdomen; sensitivity and
specificity; adult

Correspondence:
Henna Elina Sammalkorpi, M.D. Scandinavian Journal of Surgery
Department of Gastrointestinal Surgery 2017, Vol. 106(3) 196­–201
© The Finnish Surgical Society 2017
Helsinki University Central Hospital Reprints and permissions:
PL 340 sagepub.co.uk/journalsPermissions.nav
00029 Helsinki DOI: 10.1177/1457496916683099
https://doi.org/10.1177/1457496916683099
journals.sagepub.com/home/sjs
Finland
Email: henna.sammalkorpi@hus.fi
 Adult Appendicitis Score 197

Introduction Table 1
Adult Appendicitis Score (AAS): score ⩽10 low risk of appendicitis,
The more common use of imaging studies, especially score 11–15 intermediate risk of appendicitis, and score ⩾16 high risk of
computed tomography (CT), in the diagnostic workup appendicitis.
of patients with suspected acute appendicitis has led
not only to improved diagnostic accuracy but also to Score
increased exposure to ionized radiation and diagnos-
Symptoms and findings
tic delay (1, 2).
  Pain in RLQ 2
In some institutions, imaging before surgery for
  Pain relocation 2
suspected acute appendicitis is mandatory (3). In
  RLQ tenderness Women, aged 16–49 years 1
other institutions, imaging studies are used less and
All other patients 3
with no guideline leading to unacceptably high nega-
 Guarding Mild 2
tive appendectomy rate (4). Recent evidence shows
Moderate or severe 4
that selective use of CT imaging does not increase the
Laboratory tests
rate of negative appendectomies compared to manda-
 Blood leukocyte ⩾7.2 and <10.9 1
tory CT (5, 6).
count (×109) ⩾10.9 and <14.0 2
The Implementation of a clinical risk score into the
⩾14.0 3
diagnostic pathway prevents over- and underuse of
 Proportion of ⩾62 and <75 2
imaging studies in patients with suspected acute
neutrophils (%) ⩾75 and <83 3
appendicitis (7). Scoring provides a method for select-
⩾83 4
ing high-risk patients directly into surgery, low-risk
 CRP (mg/L), ⩾4 and <11 2
patients to outpatient care, and intermediate-risk
symptoms <24 h ⩾11 and <25 3
patients to further investigations. The existing diag-
⩾25 and <83 5
nostic scores for suspected appendicitis lack either
⩾83 1
specificity or sensitivity to be used for stratification of
 CRP (mg/L), ⩾12 and <53 2
patients in routine diagnostic workup (4, 7).
symptoms >24 h ⩾53 and <152 2
We have developed a new diagnostic score, the
⩾152 1
Adult Appendicitis Score (AAS), for adult patients
with suspected acute appendicitis (8). The score takes
RLQ: the right lower abdominal quadrant; CRP: C-reactive protein.
into account the time passed between the beginning of AAS calculator: www.appendicitisscore.com.
the symptoms and diagnostics as well as the special
challenges in diagnosis of fertile-aged women. The
score was implemented into the routine clinical path- result stratifies patients into low-, intermediate-, and
way in two hospitals during the fall 2014 with the aim high-risk groups for appendicitis. The application
to validate the AAS in two relatively different hospi- thereafter suggested surgery without imaging studies
tals as a part of emergency room routines. In both for the high-risk patients, imaging for the intermedi-
study hospitals, there was underuse of imaging ate-risk patients, and no imaging studies for the low-
modalities in suspected acute appendicitis, and scor- risk patients. In case of imaging, ultrasound was
ing was implemented to reduce negative appendec- recommended as a primary method for patients
tomy rate with increased but selective use of imaging. younger than 35 years followed by CT if no diagnosis
The accuracy of the diagnosis was compared to the was established. In patients aged 35 years or older, CT
period before introducing the score. was recommended as a primary method for imaging.
In pregnant patients, magnetic resonance imaging
Methods was recommended instead of CT.
Fig. 1 illustrates a flowchart of the clinical pathway
Patients
of the study.
The data were prospectively collected in surgical After initial physical examination and laboratory
emergency departments of two university hospitals, tests, the physicians and surgeons at the emergency
the biggest (hospital A) and the smallest (hospital B), departments used the application as a diagnostic tool.
in Finland from September 2014 to May 2015. All Imaging was available at all times; the recommenda-
adult (aged 16 years or more) patients admitted to tions based on the score were not mandatory and phy-
these two health care facilities for right lower quad- sicians had the option to order imaging studies based
rant abdominal pain or suspected acute appendicitis on their clinical suspicion. However, during the study
(AA) were included in this study. The reference period, scoring was mandatory for all patients sus-
population comprised 829 patients with suspected pected of acute appendicitis.
acute appendicitis. The patient data of reference The patient data necessary for calculation of the
population were originally prospectively collected score for this study were retrieved from the scoring
in hospital A during 2011–2012 for the construction application. Further information concerning patient
of the score with detailed characterization in the characteristics, imaging, surgery, hospitalization, and
original article (8). The institutional review board final diagnoses, as well as possible complications were
approved the study design. acquired from patient databases. The medical records
AAS (Table 1) was introduced as a simple web- were reviewed after a minimum of 1 month after hos-
based application that collects the necessary informa- pital discharge for final histological diagnoses and
tion, calculates the score, and based on the scoring possible complications or failures in diagnostics.
198 H. E. Sammalkorpi, et al.

Fig. 1. Patients’ flow during the study. In case of imaging, ultrasound was recommended as a primary method for patients younger
than 35 years followed by computed tomography (CT) if no diagnosis was established. In patients aged 35 years or older, CT was
recommended as a primary method for imaging. In pregnant patients, magnetic resonance imaging was recommended instead of CT.

When performing surgery for suspected appendici-
tis, the appendix was at all times removed. The final
diagnosis of acute appendicitis was based on histo-
logical examination showing transmural infiltration of
neutrophils in the appendix with the exception of
three patients who underwent non-operative treat-
ment for appendiceal abscess in hospital B. In these
patients, the diagnosis was based on CT findings.
Complicated appendicitis was, for this study, defined
as perforation with peritonitis or abscess detected
during surgery or in three patients undergoing non-
operative treatment, abscess detected on CT.
The aim of this investigation was to validate the AAS
in two relatively different hospitals as a part of emer-
gency room routines. The diagnostic performance of
the score (specificity, sensitivity, likelihood ratios, and
diagnostic odds ratio) was calculated. The negative
appendectomy rate, perforation rate, and utilization of
imaging were compared with the reference population.
Statistical Analysis
Statistical analysis was performed using SPSS® ver-
sion 22 (IBM, Armonk, NY, USA).
women. Of these patients, 432 (47.6%) had appendici-
tis. The proportion of complicated appendicitis was 91
of 432 (21.1%). Altogether, 470 patients underwent
surgery for suspected appendicitis. Three patients
with appendiceal abscess were treated conservatively.
Other specific diagnoses were found in 207 (22.8%)
patients, whereas 292 (32.2%) patients had non-spe-
cific abdominal pain.
In the reference population, there were 829 patients
with suspected appendicitis, and 392 (47.3%) of these
patients had appendicitis. Median age was 32 (range
16–97) years, and 483 (58%) of patients were women.
Diagnostic performance of the aas
The specificity of AAS ⩾16 was 93.3% and the sensi-
tivity 49.4%. The sensitivity in patients with the score
⩾11 was 94.7%, and the specificity 60.2%. The nega-
tive predictive value of AAS (likelihood of no appen-
dicitis in the low-risk group) was 93% in this study.
For more detailed information of the performance of
the score, see Tables 2 and 3.
High Risk, Aas ⩾16

Results Of all patients with appendicitis, 213 (49%) were strat-

Patients
This study enrolled 908 patients. Median age was 31
(range: 16–86) years, and 556 (61.2%) of patients were
ified to the high-risk group. There were altogether 246
(27.1%) patients stratified to this group. Surgery for
suspected appendicitis was performed on 225 patients.
In 12 (5.3%) of these operations, no appendicitis was
 Adult Appendicitis Score 199

Table 2
Adult Appendicitis Score results and appendicitis in hospitals A and B.

All patientsa No Uncomplicated Complicated Surgery for Negative Conservative

appendicitisa appendicitisa appendicitisa suspected appendectomya treatment for
appendicitisa appendiceal abscessa

All patients 908 476 (52%) 341 (38%) 91 (10%) 470 (52%) 41 (8.7%) 3
Hospital A
  All Patients 820 439 (54%) 301 (37%) 80 (10%) 415 (51%) 34 (8.2%) 0
AAS
  ⩽10 289 269 (93%) 18 (6%) 2 (1%) 26 (9%) 6 (23.1%)  
 11–15 319 146 (46%) 142 (44%) 32 (10%) 193 (60%) 19 (9.8%)  
  ⩾16 212 25 (12%) 141 (66%) 46 (22%) 196 (93%) 9 (4.6%)  
Hospital B
  All patients 88 37 (42%) 40 (45%) 11 (13%) 55 (63%) 7 (12.7%) 3
AAS
  ⩽10 20 17 (85%) 2 (10%) 1 (5%) 4 (20%) 2 (50%) 1
 11–15 34 13 (38%) 16 (47%) 5 (15%) 21 (62%) 2 (9.5%) 2
  ⩾16 34 7 (21%) 22 (65%) 4 (12%) 29 (85%) 3 (10.3%) 0

AAS: Adult Appendicitis Score.

aValues
show number of patients (%).

Table 3
Adult Appendicitis Score in the diagnostics of acute appendicitis.

Sensitivity (95% CI) Specificity (95% CI) LR+ LR− DOR

Hospital A
  ⩾11 94.8% (92.9–96.9) 61.1% (56.7–66.1) 2.43 0.09 28.4
  ⩾16 49.1% (42.3–56.2) 94.3% (92.4–96.9) 8.60 0.54 16.0
Hospital B
  ⩾11 94.0% (87.6–100) 45.9% (29.8–64.7) 1.74 0.13 13.3
  ⩾16 52.0% (36.1–72.6) 81.1% (68.5–96.4) 2.75 0.59 4.6
All patients
  ⩾11 94.7% (92.6–97.0) 60.0% (55.7–64.7) 2.37 0.09 26.6
  ⩾16 49.4% (43.2–56.1) 93.3% (91.3–95.9) 7.37 0.54 13.6

LR+: positive likelihood ratio; LR−: negative likelihood ratio; DOR: diagnostic odds ratio; CI: confidence interval.

detected. A total of 99 (40%) patients underwent diag-
nostic imaging (Tables 2 and 3).
Intermediate Risk, Aas 11–15
In the group of possible appendicitis (AAS 11–15),
there were 353 (38.9%) patients. Of these patients, 196
(56%) had appendicitis. These patients comprised 45%
of all appendicitis patients. A total of 328 (93%) patients
underwent diagnostic imaging (Tables 2 and 3).
Low Risk, Aas ⩽10
Of all patients, 309 (34.0%) had a score of 10 or less. Of
these patients, 23 (7.4%) had appendicitis. There were
no patients with perforated appendicitis and peritoni-
tis in this group. Three patients had appendiceal
abscess. A total of 159 (51%) patients underwent diag-
nostic imaging (Tables 2 and 3).
Negative Appendectomy Rate
The overall negative appendectomy rate was 41 (8.7%)
out of 470. The lowest negative appendectomy rate 12
out of 225 (5.3%) was in the high-risk group, patients
with AAS ⩾ 16 (p < 0.001). Furthermore, 3 of these 12
patients had other disease requiring emergency sur-
gery, resulting in low rate (4%) of unnecessary opera-
tions in patients in the high-risk group. The highest
negative appendectomy rate 8 of 30 (26.7%) was in
patients with AAS ⩽ 10. Of these negative appen-
dectomies, four (50%) were performed due to false-
positive imaging results and four due to clinical
suspicion despite negative preoperative ultrasonog-
raphy (US) studies. Altogether, 13 negative appendec-
tomies were performed due to clinical suspicion either
without preoperative imaging or after negative or
inconclusive imaging in the low- and intermediate-
risk groups. In hospital A, after introduction of AAS,
the rate of negative appendectomy was 34 of 415
(8.2%), whereas before the implementation of AAS
the rate was 87 of 477 (18.2%) (p < 0.001, chi-square
test). In patients operated on without preoperative
imaging studies, negative appendectomy rate was
after the introduction of AAS 15 of 167 (9.0%). Of
these patients, two were in the intermediate probabil-
ity group, and seven in the high probability group.
Before the implementation of the score, the rate of
200 H. E. Sammalkorpi, et al.
negative appendectomy in patients without preopera-
tive imaging was 59 of 263 (18.3%) (p = 0.002, chi-
square test). The rate of complicated appendicitis was
similar in the study population 91 of 432 (21%) and in
the reference population 94 of 392 (24%), p = 0.32.
Diagnostic Imaging
In this study, 152 of 432 (35.2%) patients with AA
underwent surgery without preoperative imaging. In
the reference population, surgery was performed with-
out preoperative imaging in 67% of patients with AA.
Before the introduction of AAS, 20.9% of all patients
suspected of acute appendicitis underwent CT,
whereas during this study the proportion was 40.1%.
Discussion
This study showed that, with help of AAS, an accept-
able rate of negative appendectomy was achieved
without mandatory preoperative imaging. In this
study, the rate of negative appendectomies in hospital
A was significantly lower than before the introduction
of the score. The diagnostic performance of the score
was comparable to the performance in the original
publication describing the construction of the score
(8). After implementation of the AAS, the use of diag-
nostic imaging was increased in a controlled manner,
helping to reduce the negative appendectomy rate.
Half of patients with appendicitis were stratified to
high-risk group with the specificity of 93.3%, which is
comparable to the specificity of CT. Therefore, manda-
tory imaging before surgery offers limited benefit in
this group. In the low-risk group, there was high nega-
tive appendectomy rate, half of cases due to false-pos-
itive findings in imaging studies. Only 7% of patients
in the low-risk group had appendicitis, no patients
with peritonitis among them, suggesting that observa-
tion rather than mandatory imaging would be the best
management in these patients in case appendicitis is
clinically suspected despite the low scoring result.
This leaves only the intermediate-risk group, 39% of
all patients, to mandatory imaging.
In a recent analysis in Lancet by Bhangu et al. (9),
scoring was proposed as a step in diagnostic flowchart
of patients with suspected appendicitis. In this study,
the new AAS score was superior compared to pub-
lished validation results of Alvarado and Appendicitis
Inflammatory Responce Score (AIR) score mentioned
in the analysis by Bhangu. AIR score has excellent
specificity but lacks sensitivity in high-risk group in
validation studies. This means that only a minority of
patients with appendicitis are stratified to high-risk
group. Alvarado score has better sensitivity but lacks
specificity. The sensitivity of AIR score was 23%–33%
in previous studies compared to the sensitivity of the
new AAS of 49% in this study. AAS classified half of
all patients with appendicitis to high-risk group,
whereas in the study by Scott et al. (4) AIR recognized
30 of 132 (23%), and in the study by Kollar et al. (7) 22
of 67 (33%). The negative predictive value (likelihood
of no appendicitis in the low-risk group) was 93% in
this study and in the studies of AIR score mentioned
above, the value was 94%–95%. In the study by Kollar
et al. (7), the Alvarado score stratified 53 of 67 (79%)
patients with appendicitis to the high-risk group with
specificity of 76%. The negative predictive value of
Alvarado score was 93%. Hence, Alvarado score recog-
nizes patients with low risk of appendicitis, but does
not replace imaging in the high-risk group. Because of
lack of specificity, Alvarado score is inferior to AIR
score and, in the light of this study, the new AAS. This
study did not include direct comparison with other
scoring systems. However, in the original study of
construction of AAS, we compared the diagnostic
accuracy of AAS with AIR and Alvarado scores. In that
study, AAS was superior to both AIR and Alvarado.
A diagnostic scoring system does not replace imaging
but helps selecting the patients in need of imaging. This
is important, on one hand, because of harms of false-pos-
itive diagnosis with negative explorations and, on the
other hand, because of potential harms of imaging.
Risks of radiation are of special importance in
young patients who represent the majority of patients
with suspected appendicitis. There are estimations
that on young adults suspected of acute appendicitis,
2000 CT scans would result in at least one cancer death
(2, 10). In addition to increase in cancer risk, other
potential harms include allergic reactions to contrast
medium and contrast-induced nephropathy (11).
There are several ways to avoid excessive radiation.
Low-dose abdominal CT has equal diagnostic accu-
racy compared to standard-dose CT in diagnosis of
acute appendicitis with significantly lower radiation
dose (12). In several studies, US has been implemented
as a method to stratify patients according to risk of
appendicitis (13–15). Unfortunately, US frequently
fails to exclude appendicitis and accurately diagnose
the most common differential diagnoses (16).
Performing US on a patient with suspected AA inevi-
tably takes time and requires experienced performer.
Definition of negative appendectomy varies. Thus,
comparison of negative appendectomy rates between
different studies can be inaccurate (17). However, the
negative appendectomy rate in the study population
was comparable to the negative appendectomy rate in
studies from hospitals with mandatory preoperative
imaging (3, 6). Thus, it seems that mandatory imaging
could be restricted for patients with equivocal appen-
dicitis according to clinical scoring. In this study,
nearly two-thirds of all patients were either in the low-
risk group or in the high-risk group, where mandatory
imaging has limited benefits.
The instructions of imaging given by the scoring
system were not fully followed during the study.
Instead, the proportion of patients who underwent
diagnostic imaging was greater, and some patients
with low or intermediate score were operated on with-
out imaging. This is probably partly due to surgeons’
hesitation in trusting a newly introduced diagnostic
tool. However, no diagnostic method is infallible—a
strong clinical suspicion of either appendicitis in the
low-risk group or alternative diagnosis in all groups
should always authorize imaging. There were 13
patients in the intermediate- and low-risk groups who
underwent negative appendectomy due to clinical
suspicion either without preoperative imaging or after
negative or inconclusive imaging. If the instructions
would have been followed in these cases, the negative
appendectomy rate would have been 6.0%.
 Adult Appendicitis Score
There were 152 (35.2%) patients with AA who
underwent surgery without preoperative imaging.
Interestingly, the highest negative appendectomy rate
was in patients with initially low probability of appen-
dicitis according to scoring with high rate of false-
positive imaging study result. In order to reduce
negative appendectomy rate even more, patients with
low probability according to scoring should not
undergo diagnostic imaging nor surgery in the first
place.
This study had a potential limitation: the score was
implemented in hospital B to achieve external valida-
tion. The amount of patients enrolled in hospital B was
88 (9.7%) of 908 patients. The low number of patients
from hospital B limits the value of external validation,
and more studies outside hospital A are needed for the
external validation of the AAS.
In summary, the introduction of AAS helped to
reduce negative appendectomy rate without manda-
tory imaging in patients suspected of acute appendici-
tis. AAS is an accurate, fast, and easily applicable
method for stratifying patients according to risk of
appendicitis. AAS is available as an online calculator
on www.appendicitisscore.com.
Declaration Of Conflicting Interests
The author(s) declared no potential conflicts of interest with
respect to the research, authorship, and/or publication of
this article.
Funding
The study was financially supported by the Mary and
Georg C. Ehrnrooth’s foundation and the Martti I. Turunen’s
foundation (personal research grant for the corresponding
author).
References
1. Raja AS, Wright C, Sodickson AD et  al: Negative appendec-
tomy rate in the era of CT: An 18-year perspective. Radiology
2010;256:460–465.
2. Rogers W, Hoffman J, Noori N: Harms of CT scanning prior
to surgery for suspected appendicitis. Evid Based Med 2015;
20:3–4.
3. Lahaye MJ, Lambergts DM, Mutsaers E et al: Mandatory imag-
ing cuts costs and reduces the rate of unnecessary surgeries in
the diagnostic work-up of patients suspected of having appen-
dicitis. Eur Radiol 2015;25:1464–1470.
201
4. Scott AJ, Mason SE, Arunakirinathan M et  al: Risk stratifica-
tion by the Appendicitis Inflammatory Response score to guide
decision-making in patients with suspected appendicitis. Br J
Surg 2015;102:563–572.
5. Atema JJ, Gans SL, Van Randen A et al: Comparison of imaging
strategies with conditional versus immediate contrast-enhanced
computed tomography in patients with clinical suspicion of
acute appendicitis. Eur Radiol 2015;25:2445–2452.
6. Schok T, Simons PC, Janssen-Heijnen ML et  al: Prospective
evaluation of the added value of imaging within the Dutch
National Diagnostic Appendicitis Guideline—Do we forget our
clinical eye? Dig Surg 2014;31:436–443.
7. Kollar D, McCartan DP, Bourke M et al: Predicting acute appen-
dicitis? A comparison of the Alvarado score, the Appendicitis
Inflammatory Response Score and clinical assessment. World J
Surg 2015;39:104–109.
8. Sammalkorpi HE, Mentula P, Leppäniemi A: A new adult
appendicitis score improves diagnostic accuracy of acute appen-
dicitis—A prospective study. BMC Gastroenterol 2014;14:114.
9. Bhangu A, Soreide K, Di Saverio S et  al: Acute appendicitis:
Modern understanding of pathogenesis, diagnosis, and man-
agement. Lancet 2015;386:1278–1287.
10. Board on Radiation Effects Research: Health Risks from Expo-
sure to Low Levels of Ionizing Radiation: BEIR VII Phase 2.
National Academies Press, Washington, DC, 2006.
11. Mitchell AM, Kline JA: Contrast nephropathy following com-
puted tomography angiography of the chest for pulmonary
embolism in the emergency department. J Thromb Haemost
2007;5:50–54.
12. Kim K, Kim YH, Kim SY et al: Low-dose abdominal CT for eval-
uating suspected appendicitis. New Engl J Med 2012;366:1596–
1605.
13. Chang ST, Jeffrey RB, Olcott EW: Three-step sequential posi-
tioning algorithm during sonographic evaluation for appendici-
tis increases appendiceal visualization rate and reduces CT use.
Am J Roentgenol 2014;203:1006–1012.
14. Parker L, Nazarian LN, Gingold EL et  al: Cost and radiation
savings of partial substitution of ultrasound for CT in appen-
dicitis evaluation: A national projection. Am J Roentgenol
2014;202:124–135.
15. Wagenaar AE, Tashiro J, Wang B et al: Protocol for suspected
pediatric appendicitis limits computed tomography utilization.
J Surg Res 2015;199:153–158.
16. Shah BR, Stewart J, Jeffrey RB et  al: Value of short-interval
computed tomography when sonography fails to visualize the
appendix and shows otherwise normal findings. J Ultrasound
Med 2014;33:1589–1595.
17. Papes D, Srsen Medancic S, Antabak A et al: What is the accept-
able rate of negative appendectomy? Comment on “Prospec-
tive evaluation of the added value of imaging within the Dutch
National Diagnostic Appendicitis Guideline—Do we forget our
clinical eye”? Dig Surg 2015;32:181–182.
Received: August 5, 2016
Accepted: November 8, 2016