Methods for Educational Evaluation: using Stake's Countenance model

of evaluation, and Alexander and Hedberg's evaluation framework

Professor Kerry Shephard, Director HEDC
Resource page to support an HEDC Research Presentation 17th August 2006

Aims for this presentation - two directions

Describe one case study of an educational intervention but
Emphasise the research approaches adopted

Comments on scientific approaches

experimental science, fundamental laws, theory, hypothesis, experiment with controls,
intervention, prediction, measurement, statistical interpretation, support or refute theory,
add to fundamental laws repeatable.

Comments on social science research approaches

fewer fundamental laws, theories are less likely to be grounded in these laws,
experiments more difficult, high variability and dependent on individual behaviour, less
opportunities for controls, may not be repeatable, much research is qualitative, quasiexperimental, some is essentially a-theoretical and pragmatic in nature (what hope for
pedagogy as the new science of learning and teaching?).

Educational research a long history of dispute and an uneasy way forward?

behavioural psychology and quantitative research methods seeking explanation,
predictive enquiry and generalisation vs. qualitative exploration (Cronbach, 1975) to know
man as he is no mean aspiration).
leading to an educational research methodological mosaic (Shulman, 1981)

What approaches are appropriate in educational research?

Confirmatory, hypothesis testing, predictive; exploratory, hypothesis-generating,
descriptive, interpretive curiosity is a central driver much in common with scientific
method interventions are common research tools.

What happens if ?
we increase the pressure of a
discrete volume of gas?

What happens if ?

Models/theories and laws

General Gas Laws

Models/theories and laws

Prediction
Volume decreases

Prediction

we increase the pressure of a

discrete volume of gas?

General Gas Laws

Volume decreases

we remove predators from a

balanced ecosystem?

Lotka-Volterra theoretical
model

Numbers of prey rise

What happens if ?
we increase the pressure of a
discrete volume of gas?

Models/theories and laws

General Gas Laws

Prediction
Volume decreases

 we remove predators from a

balanced ecosystem?

Lotka-Volterra theoretical
model

Numbers of prey rise

we introduce online video as

learning resources to first year
student nurses?

Behaviourist models;
conditions for learning/
Constructivist models; how
learners construct meaning/ ?

New resources motivate some

students to learn/motivated
students construct knowledge/
(some staff will not find time)

Designing the intervention in a way that will yield data that will allow us to improve
our understanding of the system and its fundamental processes

The Project
LIFESIGN, streaming video, University of Southampton UK School of Nursing and
Midwifery, UK desperate for more nurses, individual learning routes, wide range of learning
resources, video embedded with activities in online resources, IT support/educational
developer support, evaluation incorporated into project design.

Research Approaches experimental design?

LIFESIGNs evaluation methodology is based on that of Alexander and Hedberg (1994) and
emphasises the role of evaluation in all aspects of an educational innovation; from design
to post-mortem. In that sense it has affinity with action-research and Rothman and
Friedmans Action Evaluation (Friedman and Rothman, 2002) and encourages the
participation of stakeholders in a continual process.
Design

Development

Stage

Design of

evaluation
programme for
each intervention
or change

Implementation

Focus
Analyse
existing
element of
taught
programme

Institutionalisation

Purpose of the
evaluation

Methods

What could be done better Self, peer, expert review, educational

using streaming video?
literature, survey methods; focus
group, user- requirement analysis,
interviews, confidence logs, attitude
surveys.

Stage

Focus

Purpose of the
evaluation

Methods

Identify source of
information and
support

Optimise use
of existing
video
resources and
identify need
for rights
clearance,
conversion to
streaming
video and new
footage.

To identify the video

resources available to
support the indicated
curriculum element. To
identify need for new
footage and initiate
process of production
where necessary.

Agree the scope of

the evaluation

Design
innovation

What innovation is likely to Design team

bring about the desired
outcome?

Specify area of
interest
Identify
stakeholders

Agree time-scales
Specify questions

Advice from all LIFESIGN partners

(UWIC/ Southampton/Other). For each
expression of interest the LIFESIGN
partnership should produce a list of
video resources that will be useful,
indicating some idea of rights
clearance applicable to the footage
and information and advice on
conversion to streaming format.

Specification of How feasible is the

Feasibility analysis by prototyping,
innovation
proposed innovation and
storyboarding, peer/expert review
how likely is it to bring
about the desired change?

Stage

Development

Focus

Purpose of the
evaluation

Methods
Observation, user-tracking, student
and peer interviews, focus group.

Formative
monitoring of
changing
learning
environment

Is the innovation
accessible and functional?

Formative
monitoring of
changing
learning
process

Is the innovation
influencing the learning
process?

Student and peer interviews, focus

group. Reflective journals. Confidence
logs. Attitude surveys. Peer and expert
review.

Is the innovation
producing the intended
learning outcome?

Outcome-relevant assessment (do

students pass their assessments
well?). Survey methods; focus group,
interviews, confidence logs, attitude
surveys.

Implementation Summative
evaluation of
learning
outcome
Summative
evaluation of
the validity of
the innovation

Peer and expert review.

Is the innovation
Peer and expert review. Evidence of
appropriate in this setting? integration of the innovation into the
whole curriculum.

Stage

Dissemination

Focus

Purpose of the
evaluation

Methods

Impact
evaluation

Is the learning transferred

beyond the immediate
context of the innovation?

Evidence of beneficial learning in other

areas of the curriculum (transfer to
the workplace?) via peer and expert
review, focus groups. Indirect
indicators such as progress and
retention rates?

Maintenance
evaluation

Is the innovation
sustainable and
transferable?

Peer and expert review of benefits in

relation to cost and educational and
funding policies of the institution.

Research Approaches data analysis?

Huge amount of data how can we best make sense of it?
Some is quantitative; numbers of students who were confident that they learned from the videos
Some is qualitative; focus group outcomes, reflective statements
Some is good data; tracking access, observation of student behaviour
Some is not so good; results influenced by online video going offline

Stake (1967) Countenance model of evaluation

intentions

observations

antecedents
transactions
outcomes

Intentions relate strongly to Predictions; Observations to Results. This approach allows a

wide range of data to be assimilated into an analysis.

How did Stakes model help us to analyse this data?

intentions

observations

issues

antecedents Staff and

students
would
acquire skills

Not all had

these skills or
chose to
develop them

Time

transactions All students

would have
access to
video

IP address
restrictions
denied access
to some

Copyright

outcomes

About half
used video
and one third
enjoyed it

What of the
others?

Most
students
would use
video

Conclusions
Educational research can make use of a wealth of qualitative and quantitative data and a
mosaic of research approaches. In general its approaches are insufficiently scientific to
yield or build generalisable theories, but often, that is not their aim.
References/resources
Alexander S and Hedberg J 1994 Evaluating technology based learning; Which model? In: K Beatie, C.
McNaught and S. Wills (eds) Multimedia in education; designing for change in teaching and learning.
Amsterdam. Elsevier p 233-244

Cronbach L (1975) Beyond the two disciplines of scientific psychology. American Psychologist, 30, 116-127
Friedman V J and Rothman V 2002 Action evaluation for knowledge creation in social-education programs.
http://www.aepro.org/inprint/papers/knowledge.html
Green S, Voegeli D, Harrison M, Phillips J, Knowles J, Weaver M and Shephard K. (2003) Evaluating the
use of streaming video to support student learning in a first-year life sciences course for student nurses.
Nurse Education Today 23, 255-261
Green S, Weaver M, Voegeli D, Fitzsimmons D, Knowles J, Harrison M and Shephard K (2006) The use of
a Virtual Learning Environment (Blackboard 5) to support the learning of pre-qualifying nursing students
undertaking an anatomy and physiology module. Nurse Education Today 26 (5) 388-395
LIFESIGN 2002 The LIFESIGN Project. www.lifesign.ac.uk
Shulman L S (1981) Disciplines of Inquiry in Education: An Overview. Educational Researcher 10 (6) 5-12
Stake R E 1967 The countenance of educational evaluation. Teachers College Record 68 (7): 523-540.
Return

Bookmarked Resources
Session Outline

Module title & Code

Level & year
Session title
Lecturer

Foundation in life sciences (LS01)

Level 1, year 1
The Brain in situ
Jess Knowles & Jess Knight

Aim of session
Learning Outcomes
Learning & teaching
strategies
Related learning
activities
Indicative content
Key references &
other learning
resources

to introduce the structure of the brain

At the end of the session & following associated
study each student will be able to describe the
basic anatomy & physiology of the brain
Directed learning session
4 lectures on the nervous system
The nervous system
Recommended & key text for LS01 or a physiology
book of your choice.

WARNING!! This video contains very graphic illustrations of a human brain being dissected. This
video is not essential viewing for this module if you are likely to be disturbed by viewing this
type of material. The material in your course textbook covers the topics illustrated.
The Brain in situ
This page provides access to segments of the Shotlist video; The Brain in situ.
The narrator is Professor Susan Standring of the United Medical & Dental Schools, London. Professor
Standring gives a guided tour of the main structures of the human brain, using museum specimens and
a brain seen in situ and removed from a cadaver. A skeleton, specimen brain and spinal chord and
sagittal section of a head and neck are used to demonstrate the location of the brain and show some
features of the spinal chord such as dorsal root ganglia. She then watches as the skull cap is removed

from a cadaver to reveal the brain in situ. Once the dura mater is removed she points out features
such as the lobes, arachnoid granulations, vascular system and optic nerve. The brain is then removed
to reveal the ventral surface, including the cerebellum, pons and medulla. We see the brain being
hung to fix in a bucket of fixative. With the brain removed, she examines the cranial cavity, showing
features such as the crista galli, pituitary stalk, cranial nerves, foramen magnum, medulla and some
cerebral-spinal fluid. She then discusses the ventricular system, explaining the role of cerebro-spinal
fluid in protecting the brain and showing where the ventricular system is to be found using a
horizontal section through the brain. A resin cast of the ventricular system shows its structure. Finally
Prof. Standring explains why she feels study of the brain and nervous system is important. The
complete video plays for nearly 20 minutes.
Click on the blue hyperlinks below to view the video or short segments as directed.
1 Watch Part 1 of the video; the brain in situ
Watch Part 2 of the video; the brain removed.
Note: This video shows detailed shots of work on a cadaver and a brain, which may disturb some
viewers. The cadaver is covered up below the skullcap at all times, making identification impossible.
2 Focus on shorter segments of the video and use the indicated learning tools to direct your
learning.
2.i.

Introduction

2.ii.

General anatomy

Learning Tool A; draw your own labelled-diagram of a side-view of the brain.

2.iii. Meninges, brain gyri and general brain anatomy
Learning Tool B; As structures are identified in the video, record their names and then spend some
time, using the recommended text for this module, to identify their functions. Look particularly at
Meninges, Lobes, Olfactory Tract.
2.iv. Cranial nerves
Learning Tool C; Identify all 12 pairs of cranial nerves and list their functions.
2.v.

Summary of brain structure

Learning Tool D; As structures are identified in the video, record their names and then spend some
time, using the recommended text for this module, to identify their functions. Look particularly at
Cerebellum, Brainstem, Pituitary.
2.vi. Ventricular system
Learning Tool E; write a short summary of the structure and function of the ventricular system and
cerebro-spinal fluid.
2.vii. Summary and conclusions

Learning Tool F; write short notes to describe why you think that it is important to study the
structure, functioning and pathology of the human nervous system.
Please remember to use the MCQ to test your learning.
Return

Bookmarked Resources 2
Each session was associated with a MCQ designed primarily to allow students to
formatively assess their own progress. The three Directed Learning Sessions that used
streaming video all included three non-scored 'survey questions' designed to ascertain
students access to streaming video; and enjoyment of, and learning-confidence in the way
that it was used. The three questions were;

Was the streamed video easy to access? (yes, no, don't know, no response)
Did you enjoy using this learning resource? (yes, no, don't know, no response)
How confident are you that you have learned from this learning resource. (very, not very,
don't know, no response)

Blackboard's on-line grade-book gave access to the responses of each student.

Return
Bookmarked Resource 3

One quote from a reflective email will illustrate the point and provides information that
relates to the data on student access to, and use of, the resources. I spoke to my class
yesterday and asked specifically about the video. Those students who were able to access
it thought it was very useful.... (but). It was commonly reported as fragmented, far too
slow or it crashed. About 50% of my class said they tried to access it and gave up...that's
not good.
Return