CLINICAL ARTICLE

Pilot project to assess and improve neurosurgery

resident and staff perception of feedback to residents for
self-improvement goal formation
Steven O. Tenny, MD, MPH, MBA, Kyle P. Schmidt, MD, and William E. Thorell, MD

Division of Neurosurgery, Department of Surgery, University of Nebraska Medical Center, Omaha, Nebraska

OBJECTIVE  The Accreditation Council for Graduate Medical Education (ACGME) has pushed for more frequent and
comprehensive feedback for residents during their training, but there is scant evidence for how neurosurgery residents
view the current feedback system as it applies to providing information for self-improvement and goal formation. The au-
thors sought to assess neurosurgery resident and staff perceptions of the current resident feedback system in providing
specific, meaningful, achievable, realistic, and timely (SMART) goals. The authors then created a pilot project to improve
the most unfavorably viewed aspect of the feedback system.
METHODS  The authors conducted an anonymous survey of neurosurgery residents and staff at an academic medical
institution to assess SMART goals for resident feedback and used the results to create a pilot intervention to address the
most unfavorably viewed aspect of the feedback system. The authors then conducted a postintervention survey to see if
perceptions had improved for the target of the intervention.
RESULTS  Neurosurgery residents and staff completed an anonymous online survey, for which the results indicated that
resident feedback was not occurring in a timely manner. The authors created a simple anonymous feedback form. The
form was distributed monthly to neurosurgery residents, neurosurgical staff, and nurses, and the results were reported
monthly to each resident for 6 months. A postintervention survey was then administered, and the results indicated that
the opinions of the neurosurgery residents and staff on the timeliness of resident feedback had changed from a negative
to a nonnegative opinion (p = 0.01).
CONCLUSIONS  The required ACGME feedback methods may not be providing adequate feedback for goal formation
for self-improvement for neurosurgery residents. Simple interventions, such as anonymous feedback questionnaires, can
improve neurosurgery resident and staff perception of feedback to residents for self-improvement and goal formation.
https://thejns.org/doi/abs/10.3171/2018.11.JNS181664
KEYWORDS  resident; feedback; goal formation

T
he Accreditation Council for Graduate Medical
Education (ACGME) requires at least semiannual
structured feedback and encourages programs to
“incorporate evaluation feedback into daily practice.”1
Feedback can encompass a wide variety of forms, includ-
ing in person, written, electronic, or video formats.2,5,9,10,​
12,​14–18,20,22–24
Although some previous studies have exam-
ined how residents view feedback,2,4,5,9–11,15,17,20,22,23,25 only
one reported study has transferred this information into a
meaningful modification of the resident feedback system.20
There are no published data to our knowledge addressing
resident and staff views on how to improve feedback for
neurosurgical training.
Feedback itself should provide not only an assessment
of the current level and state of the resident but also a guide
for moving forward.13 There are many possible ways to
help residents formulate goals for self-improvement. One
such method commonly cited in the business world is a
SMART goal. The SMART acronym stands for something
that is specific, meaningful, achievable, realistic, and time-
ly (SMART).13,19 Goals formulated by using the SMART
method have a higher likelihood of being achieved and
thus are desirable as goals along the way of resident im-
provement.13
We had anecdotally noted that some residents did not
find the formalized ACGME feedback to be as beneficial

ABBREVIATIONS  ACGME = Accreditation Council for Graduate Medical Education; PGY = postgraduate year; SMART = specific, meaningful, achievable, realistic, and
timely.
SUBMITTED  June 11, 2018.  ACCEPTED  November 21, 2018.
INCLUDE WHEN CITING  Published online March 8, 2019; DOI: 10.3171/2018.11.JNS181664.

©AANS 2019, except where prohibited by US copyright law J Neurosurg  March 8, 2019 1
Tenny et al.
as desired for assessment of current position as well as fu-
ture goal formation. Although residents receive many mi-
nor points of feedback during the day, multiple residents
voiced the desire for more formalized and broad feedback
from outside of the neurosurgical staff. We posited that we
could use the modified SMART framework for feedback
to identify and improve a current weak aspect of neurosur-
gery resident feedback. As such we set out to first survey
the neurosurgery residents and staff for their thoughts and
views of the current feedback system and then implement
a pilot project to improve the weakest element.
Methods
We conducted an anonymous online preintervention
survey of all neurosurgery residents in postgraduate year
2 (PGY-2) and above as well as all neurosurgery staff. The
PGY-2 level was chosen as these residents had finished at
least 1 complete year of residency during which they had
received assessment and feedback. The survey was creat-
ed to assess participant views of resident feedback accord-
ing to a set of modified guidelines for SMART (specific,
meaningful, achievable, realistic, and timely) goal forma-
tion by presenting the following statements for comment:
• Feedback to residents provides residents specific items
for improvement.
• Feedback to residents provides residents meaningful
items for improvement.
• Feedback to residents provides residents achievable
items for improvement.
• Feedback to residents provides residents realistic items
for improvement.
• Feedback to residents occurs in a timely fashion.
Responses were recorded according to a Likert-type
scale of strongly agree (5), agree (4), neutral (3), disagree
(2), or strongly disagree (1), with assigned points to each
category (noted in parentheses). In addition, the survey
included an additional statement for which a Likert-type
scale response was recorded, “Feedback to residents helps
the resident improve.” The survey also contained a slider
which could be set between 0 (completely unsatisfied)
and 100 (completely satisfied) in response to the follow-
ing item: “My satisfaction level with how residents receive
their feedback.” The only demographic information was
collected by asking the participants to select their sta-
tus as either neurosurgery resident or neurosurgery staff
(Supplemental Fig. 1). Residents were not informed of the
results of the preintervention survey.
Based on the preintervention survey results, we created
a pilot program to provide more timely feedback for resi-
dents over a 6-month period. The pilot project was created
so that residents would receive monthly feedback instead
of having to wait for the 6-month review. To accomplish
the feedback process each month we invited neurosurgery
residents PGY-2 and above, neurosurgery staff, and nurses
in the neuroscience ICU to complete anonymous surveys
to provide monthly feedback to the residents. Flyers with
a link to the survey were posted in the neuroscience ICU
with monthly reminders and snacks provided to the neuro-
science ICU nurses. Emails were sent monthly to all neu-
2 J Neurosurg  March 8, 2019
TABLE 1. Preintervention survey responses
Specific Meaningful Achievable Realistic Timely
Negative 3 3 2 2 8
Nonnegative 10 10 11 11 5
rosurgery residents and neurosurgery staff with the link
to the survey reminding them to fill out the survey if able.
Nurses, staff, and residents were all invited to provide
feedback to achieve a greater 360-degree feedback system
for the residents.13
The monthly feedback form was created to be short but
meaningful (Supplemental Fig. 2). The survey collected
two pieces of demographic data: the role of the evaluator
(nurse, neurosurgery resident, or neurosurgery attending)
and the specific neurosurgery resident they were provid-
ing feedback for. The survey also allowed the evaluator
to provide information of how well the resident interacted
with staff and with patients and their family members.
The survey additionally had free text boxes in which the
person completing the survey was able to identify things
the resident did well or poorly (Supplemental Fig. 2).
At the end of each month each resident received an
email containing the survey results filled out for them to
date. All individual comments were provided privately,
and the resident being evaluated was the only one viewing
the comments. In addition, all residents received an email
with any comments that were directed to all neurosurgery
residents.
After the intervention was run for 6 months, a postin-
tervention survey was conducted which was identical to
the preintervention survey, asking about SMART goals
and overall satisfaction with the resident feedback process
(Supplemental Fig. 1).
The study was reviewed by the institutional review
board and found to be exempt as it fell under the quality
improvement exemption.
Statistical Analysis
For analysis, responses were grouped as either negative
(disagree and strongly disagree) or nonnegative (neutral,
agree, and strongly agree). Groups were compared using
the Fisher exact test. Satisfaction was captured as a con-
tinuous variable (0–100) and analyzed with the Student
t-test. Significance was set at a prespecified alpha of 0.05.
Analysis was done using OpenEpi (www.OpenEpi.com).
Results
All 9 PGY-2 and later residents and 4 of 6 neurosurgery
staff members completed the preintervention survey. The
timeliness element of the preintervention survey (“Feed-
back to residents occurs in a timely fashion”) received the
lowest overall combined score of 2.6, with 8 negative and
5 nonnegative responses (Table 1), and thus timeliness be-
came the focus of the intervention.
In the intervention a total of 52 resident feedback sur-
veys were completed. Residents received between 0 and
13 feedback surveys during the intervention, with a mean

TABLE 2. Postintervention survey responses
Specific Meaningful Achievable Realistic Timely
Negative 1 1 1 1 0
Nonnegative 6 6 6 6 7
of 4.3 surveys per resident (SD 3.5). Nurses completed 35
(67%) of the surveys, neurosurgery residents 3 (6%), neu-
rosurgery attendings 10 (19%), and unspecified 4 (8%).
The postintervention survey was completed by 4 of 9
PGY-2 and later residents and 3 of 6 neurosurgery staff
(Table 2). For timeliness of resident feedback there was
a statistically significant decrease in negative perception
compared to prior to the intervention (62% prior to inter-
vention and 0% after intervention), with a p value of 0.01
(Table 3).
Discussion
We were able to successfully use a survey to assess
neurosurgery resident and staff opinions about current
resident feedback using the SMART paradigm to identify
the current weakness in the feedback system. We were
further able to create a simple and easy pilot project to
statistically significantly improve the perception of timeli-
ness of resident feedback in the postintervention survey.
The SMART paradigm for feedback is commonly used in
the business world13 but has not been previously reported
for resident feedback. There are limited published data on
using resident feedback views to improve feedback sys-
tems.20
Our data about resident views of resident feedback are
similar to the 20% dissatisfaction reported elsewhere.2
Commonly cited reasons for dissatisfaction with resident
feedback include lack of timeliness,4,5,9–12,18,20,25 poor qual-
ity,6,7,16,18 the culture of the institution,6,12,18 relationships
with those giving the feedback,6,21,23 lack of specific feed-
back,7,10,16,18,20 and sources of feedback.17,23
Various methods have been tried to address the issues
with feedback, including providing tips to staff about
feedback,3 web-based feedback, 5,20 phone app platform–
based feedback,9 cue cards for feedback,10 360-degree
feedback,17 recorded audio feedback, 22 and video review
feedback.24 However, none of the literature we could find
described using a specific methodology or framework,
such as the modified SMART paradigm, for systemati-
cally analyzing and improving resident feedback sys-
tems.
Our preintervention survey revealed that dissatisfac-
tion with the timeliness of resident feedback at our in-
stitution was similar to what has been reported for other
institutions.4,5,9–12,18,20,25 The monthly feedback system we
initiated in response to the survey improved perceptions
of both staff and residents about the timeliness of feed-
back by relying on mostly narrative feedback. Narrative
feedback provides unlimited options for how to provide
the feedback. In order to better focus feedback comments,
we prompted the reviewer to report one specific item the
resident does well and one specific item the resident can
improve. Although reviewers may be hesitant to deliver
Tenny et al.
TABLE 3. Comparison of preintervention to postintervention
changes in negative perception
Negative
Response Specific Meaningful Achievable Realistic Timely
Preintervention 23% 23% 15% 15% 62%
rate
Postinterven- 14% 14% 14% 14% 0%
tion rate
p value 0.41 0.41 0.48 0.48 0.01
blunt or directed constructive criticism,7,16,20 residents have
been shown to be able to read between the lines and com-
prehend their own strengths and weaknesses.8
Anecdotally, one of the residents in our pilot project
received mainly negative feedback and was unsure if
the feedback was useful. We discussed with this resident
the possibility that many other residents may be receiv-
ing mostly positive feedback and therefore consideration
could be given to the negative feedback this resident had
received. This suggestion prompted the resident to recon-
sider and possibly modify approaches to interactions with
nurses and others.
Our study is limited in both its size and scope. The
intervention was primarily focused on improving percep-
tions of timeliness of resident feedback while maintain-
ing some level of quality and breadth. The perception of
resident feedback may not translate into ultimate resident
improvement. Additionally, the intervention only ran for
6 months. Only about half of the residents and staff com-
pleted the postintervention survey. Although we found that
perceptions of timeliness of resident feedback improved,
we did not focus on whether this intervention changed be-
havior or outcomes as these endpoints are longer reaching
and more difficult to assess. Finally, our surveys could not
be validated due to the small sample size given the size of
the involved program.
Moving forward it would be reasonable to incorporate
more features of the feedback and assessment models
available from the business world into residency programs
to improve both the accuracy of assessments of the current
standings of residents and the formation of future goals to
facilitate the progress of residents into successful careers.
Many of the ACGME milestones for each specialty focus
on ensuring that minimum bars are being reached without
seeding lifelong improvement mentalities into residents.
Future research could apply other feedback paradigms
with some of the above-noted interventions to see if there
are more optimal feedback strategies for varying residen-
cy program types.
Conclusions
The required ACGME feedback methods may not be
providing adequate feedback for goal formation for self-
improvement for neurosurgery residents. Simple interven-
tions, such as anonymous feedback questionnaires, can
improve resident and staff perception of feedback to resi-
dents for self-improvement and goal formation.
J Neurosurg  March 8, 2019 3
Tenny et al.
References
  1. Accreditation Council for Graduate Medical Education:
ACGME Common Program Requirements. Chicago:
ACGME, 2017 (http://www.acgme.org/Portals/0/PFAssets/
ProgramRequirements/CPRs_2017-07-01.pdf) [Accessed
January 4, 2019]
  2. Angus S, Moriarty J, Nardino RJ, Chmielewski A, Rosen-
blum MJ: Internal medicine residents’ perspectives on
receiving feedback in milestone format. J Grad Med Educ
7:220–224, 2015
  3. Baker K: Clinical teaching improves with resident evaluation
and feedback. Anesthesiology 113:693–703, 2010
  4. Bentley S, Hu K, Messman A, Moadel T, Khandelwal S, St-
reich H, et al: Are all competencies equal in the eyes of resi-
dents? A multicenter study of emergency medicine residents’
interest in feedback. West J Emerg Med 18:76–81, 2017
  5. Choi HH, Clark J, Jay AK, Filice RW: Minimizing barriers
in learning for on-call radiology residents—end-to-end web-
based resident feedback system. J Digit Imaging 31:117–
123, 2018
  6. Delva D, Sargeant J, Miller S, Holland J, Alexiadis Brown
P, Leblanc C, et al: Encouraging residents to seek feedback.
Med Teach 35:e1625–e1631, 2013
  7. Donato AA, Pangaro L, Smith C, Rencic J, Diaz Y, Mensing-
er J, et al: Evaluation of a novel assessment form for observ-
ing medical residents: a randomised, controlled trial. Med
Educ 42:1234–1242, 2008
  8. Ginsburg S, van der Vleuten CP, Eva KW, Lingard L: Crack-
ing the code: residents’ interpretations of written assessment
comments. Med Educ 51:401–410, 2017
  9. Gray TG, Hood G, Farrell T: The results of a survey high-
lighting issues with feedback on medical training in the
United Kingdom and how a Smartphone App could provide a
solution. BMC Res Notes 8:653, 2015
10. Jensen AR, Wright AS, Kim S, Horvath KD, Calhoun KE:
Educational feedback in the operating room: a gap between
resident and faculty perceptions. Am J Surg 204:248–255,
2012
11. Katz-Sidlow RJ, Baer TG, Gershel JC: Providing rapid feed-
back to residents on their teaching skills: an educational
strategy for contemporary trainees. Int J Med Educ 7:83–
86, 2016
12. Kritek PA: Strategies for effective feedback. Ann Am Tho-
rac Soc 12:557–560, 2015
13. Laszlo B: Work Rules! New York: Grand Central Publish-
ing, 2015
14. Mangum R, Lazar J, Rose MJ, Mahan JD, Reed S: Exploring
the value of just-in-time teaching as a supplemental tool to
traditional resident education on a busy inpatient pediatrics
rotation. Acad Pediatr 17:589–592, 2017
15. McKendy KM, Watanabe Y, Lee L, Bilgic E, Enani G, Feld-
man LS, et al: Perioperative feedback in surgical training: A
systematic review. Am J Surg 214:117–126, 2017
16. Osmun WE, Parr J: The occasional teacher. Part 4: feedback
and evaluation. Can J Rural Med 16:96–97, 2011
17. Riveros R, Kimatian S, Castro P, Dhumak V, Honar H, Mas-
cha EJ, et al: Multisource feedback in professionalism for
anesthesia residents. J Clin Anesth 34:32–40, 2016
4 J Neurosurg  March 8, 2019
18. Sarkany D, Deitte L: Providing feedback: practical skills and
strategies. Acad Radiol 24:740–746, 2017
19. Scott SJ: S.M.A.R.T. Goals Made Simple: 10 Steps to
Master Your Personal and Career Goals. Mahwah, NJ:
Oldtown Publishing, 2014
20. Tanaka P, Bereknyei Merrell S, Walker K, Zocca J, Scotto L,
Bogetz AL, et al: Implementation of a needs-based, online
feedback tool for anesthesia residents with subsequent map-
ping of the feedback to the ACGME milestones. Anesth
Analg 124:627–635, 2017
21. Telio S, Regehr G, Ajjawi R: Feedback and the educational
alliance: examining credibility judgements and their conse-
quences. Med Educ 50:933–942, 2016
22. Topps D, Evans RJ, Thistlethwaite JE, Nan Tie R, Ellaway
RH: The one minute mentor: a pilot study assessing medi-
cal students’ and residents’ professional behaviours through
recordings of clinical preceptors’ immediate feedback. Educ
Health (Abingdon) 22:189, 2009
23. Vesel TP, O’Brien BC, Henry DM, van Schaik SM: Useful
but different: Resident physician perceptions of interprofes-
sional feedback. Teach Learn Med 28:125–134, 2016
24. Wouda JC, van de Wiel HB: The effects of self-assessment
and supervisor feedback on residents’ patient-education
competency using videoed outpatient consultations. Patient
Educ Couns 97:59–66, 2014
25. Yarris LM, Linden JA, Gene Hern H, Lefebvre C, Nestler
DM, Fu R, et al: Attending and resident satisfaction with
feedback in the emergency department. Acad Emerg Med
16 (Suppl 2):S76–S81, 2009
Disclosures
The authors report no conflict of interest concerning the materi-
als or methods used in this study or the findings specified in this
paper.
Author Contributions
Conception and design: all authors. Acquisition of data: Tenny,
Schmidt. Analysis and interpretation of data: Tenny, Schmidt.
Drafting the article: Tenny, Schmidt. Critically revising the
article: all authors. Reviewed submitted version of manuscript: all
authors. Approved the final version of the manuscript on behalf
of all authors: Tenny. Statistical analysis: Tenny. Administrative/
technical/material support: Tenny, Schmidt. Study supervision:
Thorell.
Supplemental Information
Online-Only Content
Supplemental material is available with the online version of the
article.
Supplemental Figs. 1 and 2. https://thejns.org/doi/suppl/10.​
3171/2018.11.JNS181664.
Correspondence
Steven O. Tenny: Nebraska Medical Center, Omaha, NE.