From Objectives to

Outcomes
Beth Fentress Hallmark, PhD, RN
Belmont University
Nashville, TN

Belmont University
Nashville, Tennessee
5,000 + students
College of Health Sciences
Inter-professional Education
Nursing
Accelerated, Fast track and Traditional BSN
FNP

Social Work (BSW)

Physical Therapy (DOT)
Occupational Therapy (DPT/MSOT)
Pharm D

Belmont University
Nashville, Tennessee

2- Eight bed Adult Health laboratories

8 bed Acute care lab
4 bed Peds lab
8 bed Health Assessment/OB lab
4 Inter-professional private patient areas

From Objectives to Outcomes:

Learning Objectives:
Identify the components of Healthcare
Simulation
Discuss the importance of outcomes
evaluation and challenges to traditional
assessments
Discuss the importance of validity, reliability
and feasibility as it relates to assessment
Discuss types of assessments and their
application in healthcare education

Components of Healthcare
Simulation

Jeffrey A. Groom, PhD, CRNA

Communicat
e

Healthca
re
Educatio
n

Teamwork

HOW
ARE WE
DOING?

Study Finds No Progress in

Safety at Hospitals
November 24, 2010 : NY Times: 10 North

Carolina hospitals
25.1 injuries per 100 admissions
42.7 % extra time in the hospital
2.9 % patients suffered a permanent injury
> 8 percent life-threatening
2.4 % of them caused or contributed to a
patients death
Medication errors caused problems in 162
cases.

How do we measure our

improvement?
Safe/competent practitioners: whatever the
discipline/setting.
Initial

& Continued competence

Acquisition of relevant knowledge
Development of psychomotor skills
Application of this knowledge and the skill

Current Assessments
Current methods used to measure

performance in the clinical area is difficult:

Confidentiality
Faculty to student ratio
Safety to patient
Preceptors: Valid? Reliable?
Adjunct Faculty
Tools

Model of Competence

Problem with Knowing

Knowing is measured using examinations like

NCLEX, NREMT cognitive exam, FNP

certification exam,, calculations test, etc.
Recalling basic Facts, principles and theories
Multiple choice questions and T/F questions
Test question design: Valid, reliable
Blooms taxonomy
Critical thinking questions

Problem with Knowing

Cognitive Domain
Belmont pass rate on NCLEX May 2005 is

98.6% on NP exam 100 %....

Strategies to pass these exams employed in
educational institutions.

Does this mean that each of these students

Will be prepared to care for you or your loved
ones?

Model of Competence

Problem with Knows

How
Knows How:
Application of knowledge to problem solving and
decision making (Waas, 2001)
A thought process stimulated by a problem
(Waas, 2001).
ability to solve problems, make decisions and
describe procedures (Scalese, 2008)
Case studies and essays
Multiple/multiples
Again are these students prepared to provide

safe proficient care.

Model of Competence

Shows how vs. Does

Shows How
demonstration of skills in a controlled setting

(Scalese, 2008)

Educating in these methods includes simulation

based education (SBE).

OSCE, SP, Simulations, log books, portfolios
Technical skills
Includes higher level thinking

Does
Moves from simulated environment to the real life

setting

Assessment vs.
Evaluation
Assessment and evaluation are often used

interchangeably
However for our purposes
Assessment describes the measurement of
learner outcomes
Evaluation describes the measurement of
course/program outcomes

Why do we assess learner outcomes?

Provides baseline data
Provides summative and formative

feedback
Drives learning
Allows measures of individual progress.
Encourages student reflection
Assures public that providers are competent
Licensure/credentialing requirements

Why do we evaluate our programs?

Demonstrates change and growth in

programs/courses
Identifies gaps in programs/ courses
Fundamental to outcomes- or competency-based
education
Accrediting/Credentialing facilities/programs
Allows administration to make informed allocation
decisions

Objectives/Outcomes of Program
Define outcomes based on

accrediting/professional organizations., etc.

Objectives/Outcomes leads to competency
and mastery.
Identify the Knowledge, Skills and
Attitudes/Affective Behaviors (KSA).
Curricular/Program Specific
Simulation Event Specific.
Measurable, clearly defined standards.

 change/refine

Simulation Education
Knowledge
Skills
Attitudes
Advance these throughout the curriculum via

assessment
For example: injection to team training

Preparing assessments
What should be assessed?
Every aspect of curriculum considered
Essential
Significant designated teaching time
Should be consistent with learning outcomes

that are established as the competencies

students should master/perform at a given
phase of study

Use of Assessment in
Simulation

Formative
or
Summativ
e
Rosen, MA et al. Measuring Team Performance in Simulation-Based Training: Adopting Best Practices for Healthcare.
Sim Healthcare 3:2008;3341.

Assessment
Formative Assessment
Lower stakes assessment
One of several assessments over time of course or

program
May be evaluative, diagnostic, or prescriptive
Often results in remediation or progression to next
level

Summative Assessment
Higher stakes assessment
Generally final course or program assessment
Primary purpose is performance evaluation

Assessments - peer
Enables learners to hone their skills in

their ability to work with others and

professional insight
Enables faculty to obtain a view of
students they do not see
An important part of peer assessment is
for students to justify the marks they
award to others
Justification can also be used as a
component when faculty evaluates
attitudes and professionalism.

Assessments - standard
setting
Should be set to determine competence
Enables certification to be documented,

accountable and defensible

Appropriately set standards for an
assessment will pass those students who
are truly competent
Standards should not be two low (false

positives) to pass those who are incompetent,

nor too high (false negative) to fail those who
are competent.

Assessments - standard
setting
Standards should be set around a core

curriculum that includes the knowledge, skills

and attitudes required of all students

When setting a standard the following should

be considered:

Must reflect the core curriculum

High standard in the core components of the

curriculum
Demonstrate mastery at each phase

Performance
Assessment
Basic to performance

Do they know it and know how?

Competence

Can they do it?

Performance

Do they do it?

Assessing Simulation
Documenting Data:
Live, video recording, Software logging systems
Logistics of documenting Data:
AV annotation via logging, pencil paper (wipe
off cards), scantron, PDA/handheld/TabletPC

Assessors:

Instructors, Observers, SIM/Patients, Peers,

Participants

Choosing appropriate assessment

methods/tools
When choosing the assessment instrument,

the following should be answered:

Is it valid
Is it reliable
Is it feasible

Assessments - validity
Are we measuring what we are supposed to

be measuring?
Use the appropriate instrument for the
knowledge, skill, or attitude you are testing
The major types of validity should be
considered (content, predictive, and face)

Assessments - reliability
Does the test consistently measure what it

is supposed to be measuring
Types of reliability:
Inter-rater (consistency over raters)
Test-retest (consistency over time)
Internal consistency (over different items/forms)

Assessment Tools

Tools should measure KSA within the domains

that you are measuring

Cognitive
Psychomotor
Affective

Do these domains occur alone? Or

simultaneous?
Simulation offers the ability to assess each of

these domains an application of the cognitive

domain while performing psychomotor skills as
the student demonstrates how they have
internalized values, attitudes and beliefs.

Where did I start?

Low hanging Fruit
TASTED GREAT!!
Self- reported
Confidence
Increased critical thinking
Satisfaction
Situational Awareness

Where should you start?

Tools developed for your OBJECTIVES!
To measure clinical judgment
Use a tool developed for this.
Lasater (2007).
Adds to reliability and validity
May combine instruments
What about the tool you use for clinical evaluation?
Is it reliable? Valid? Who developed it? Have you had

consistency issues with tool/students in clinical?

Does it measure what you really want it to?

Assessments - feasibility
Is the administration of the assessment

instrument feasible in terms of time and

resources?

Time to construct?
Time to Score ?
Ease of interpreting the score/producing

results ?
Practical given staffing/organization ?
Quality of feedback ?
Learner takeaway ?
Motivate Learner ?

Practicality
Number of students to be assessed
Time available for the assessment
Number of staff available
Resources/equipment available
Special accommodations

Examples of Tools
Kardong-Edgren, S., Adamson, K.A., Fitzgerald,

C. (2010, January). A review of currently

published evaluation instruments for human
patient simulation. Clinical Simulation in
Nursing, 6(1), e25-e35.
Doi:10.1016/jecns.2009.08.004.

Exercise
Lets try it: OUTLOUD
Groups
Hospital
Emergency
Nursing education
Safe Medication Administration
How does this link to the programmatic

outcomes and then with your course?

Safe Medication
Administration
How are you measuring this now?
Summative
Formative
Knowledge (Cognitive exams)
Skills/Psychomotor (lab check off)
Attitudes/Affective (what would you examine

or are you examining here?) Likert

satisfaction? Self Confidence?

Model of Competence

Knowing: Safe Med Administration

Each group write a high level MC question for

Safely Administering a specific medication

(Choose One).

Is the student who answers this question safe

and competent?

Model of Competence

Knows How: Safe Med Administration

Write a short case related to giving the same

medication?
What components must the student tell the grader?
How to administer the med?
Side Effects?
Teaching ?
What Else will we measure?
For the student who reaches all of these assessment

criteria Are they competent and safe to give the

medication?

Model of Competence

Shows How: Safe Med Administration

Take the case above and the objectives and

apply to a simulation.
This can be simple or advanced incorporating
teamwork/communication/ high acuity.
What KSAs are required?
Which student do you want taking care of

you?

References
Decker, S., Sportsman, S., Puetz, L., Billings, L. (2008). The Evolution of

Simulation and Its Contribution to Competency. The Journal of Continuing

Education in Nursing, 39 (2), 74-80.
Groom, J.A. (2009). Creating new solutions to the simulation puzzle. Society
for Simulation in Healthcare, 4 (3), 131. DOI:
10.1097/SIH.0b013e3181b3e4c3
Kardong-Edgren, S., Adamson, K.A., Fitzgerald, C. (2010, January). A review
of currently published evaluation instruments for human patient simulation.
Clinical Simulation in Nursing, 6(1), e25-e35. Doi:10.1016/jecns.2009.08.004.
Mckimm, J. (2010).University of Auckland; Visiting Professor of Healthcare
Education and Leadership, University of Bedfordshire and Honorary Professor
in Medical Education, Swansea University. Retrieved from
http://www.faculty.londondeanery.ac.uk/e-learning/setting-learning-objective
s/learning-objectives-and-learning-outcomes
.
Ross J. Scalese, MD, Vivian T. Obeso, MD, and S. Barry Issenberg, MD.
(2009).Simulation Technology for Skills Training and Competency Assessment
in Medical Education. Journal of General Internal Medicine, Journal of General
Internal Medicine , 23(1), 46-49, DOI: 10.1007/s11606-007-0283Wass, V., Van der Vleuten, C., Shatzer, J., & Jones, R. (2001). Assessment of
clinical competence. Lancet, 24;357(9260):945-9.