Methods for Educational Evaluation: using Stake's Countenance model of evaluation, and Alexander and Hedberg's evaluation framework

Professor Kerry Shephard, Director HEDC Resource page to support an HEDC Research Presentation 17th August 2006

Aims for this presentation - two directions Describe one case study of an educational intervention but Emphasise the research approaches adopted

Comments on scientific approaches experimental science, fundamental laws, theory, hypothesis, experiment with controls, intervention, prediction, measurement, statistical interpretation, support or refute theory, add to fundamental laws repeatable.

Comments on social science research approaches fewer fundamental laws, theories are less likely to be grounded in these laws, experiments more difficult, high variability and dependent on individual behaviour, less opportunities for controls, may not be repeatable, much research is qualitative, quasiexperimental, some is essentially a-theoretical and pragmatic in nature (what hope for pedagogy as the new science of learning and teaching?).

Educational research a long history of dispute and an uneasy way forward? behavioural psychology and quantitative research methods seeking explanation, predictive enquiry and generalisation vs. qualitative exploration (Cronbach, 1975) to know man as he is no mean aspiration). leading to an educational research methodological mosaic (Shulman, 1981)

What approaches are appropriate in educational research? Confirmatory, hypothesis testing, predictive; exploratory, hypothesis-generating, descriptive, interpretive curiosity is a central driver much in common with scientific method interventions are common research tools.

What happens if ? we increase the pressure of a discrete volume of gas?

Models/theories and laws General Gas Laws

Prediction Volume decreases

What happens if ? we increase the pressure of a discrete volume of gas? we remove predators from a balanced ecosystem?

Models/theories and laws General Gas Laws Lotka-Volterra theoretical model

Prediction Volume decreases Numbers of prey rise

What happens if ? we increase the pressure of a discrete volume of gas? we remove predators from a balanced ecosystem? we introduce online video as learning resources to first year student nurses?

Models/theories and laws General Gas Laws Lotka-Volterra theoretical model Behaviourist models; conditions for learning/ Constructivist models; how learners construct meaning/ ?

Prediction Volume decreases Numbers of prey rise New resources motivate some students to learn/motivated students construct knowledge/ (some staff will not find time)

Designing the intervention in a way that will yield data that will allow us to improve our understanding of the system and its fundamental processes

The Project LIFESIGN, streaming video, University of Southampton UK School of Nursing and Midwifery, UK desperate for more nurses, individual learning routes, wide range of learning resources, video embedded with activities in online resources, IT support/educational developer support, evaluation incorporated into project design.

Research Approaches experimental design? LIFESIGNs evaluation methodology is based on that of Alexander and Hedberg (1994) and emphasises the role of evaluation in all aspects of an educational innovation; from design to post-mortem. In that sense it has affinity with action-research and Rothman and Friedmans Action Evaluation (Friedman and Rothman, 2002) and encourages the participation of stakeholders in a continual process. Design Development Implementation Institutionalisation

Stage

Focus

Design of
evaluation programme for each intervention or change

Specify area of interest Identify stakeholders Identify source of information and support Agree the scope of the evaluation Agree time-scales Specify questions

Analyse existing element of taught programme Optimise use of existing video resources and identify need for rights clearance, conversion to streaming video and new footage. Design What innovation is likely to Design team innovation bring about the desired outcome? Specification of How feasible is the Feasibility analysis by prototyping, innovation proposed innovation and storyboarding, peer/expert review how likely is it to bring about the desired change?

Purpose of the Methods evaluation What could be done better Self, peer, expert review, educational using streaming video? literature, survey methods; focus group, user- requirement analysis, interviews, confidence logs, attitude surveys. To identify the video Advice from all LIFESIGN partners resources available to (UWIC/ Southampton/Other). For each support the indicated expression of interest the LIFESIGN curriculum element. To partnership should produce a list of identify need for new video resources that will be useful, footage and initiate indicating some idea of rights process of production clearance applicable to the footage where necessary. and information and advice on conversion to streaming format.

Stage

Focus

Formative monitoring of changing learning environment Formative monitoring of changing learning process Summative Implementation evaluation of learning outcome

Development

Purpose of the evaluation Is the innovation accessible and functional?

Methods Observation, user-tracking, student and peer interviews, focus group. Peer and expert review.

Is the innovation influencing the learning process? Is the innovation producing the intended learning outcome?

Student and peer interviews, focus group. Reflective journals. Confidence logs. Attitude surveys. Peer and expert review.

Dissemination

Summative evaluation of the validity of the innovation Impact evaluation

Outcome-relevant assessment (do students pass their assessments well?). Survey methods; focus group, interviews, confidence logs, attitude surveys. Is the innovation Peer and expert review. Evidence of appropriate in this setting? integration of the innovation into the whole curriculum. Is the learning transferred Evidence of beneficial learning in other beyond the immediate areas of the curriculum (transfer to context of the innovation? the workplace?) via peer and expert review, focus groups. Indirect indicators such as progress and retention rates?

Stage

Focus Maintenance evaluation

Purpose of the evaluation Is the innovation sustainable and transferable?

Methods Peer and expert review of benefits in relation to cost and educational and funding policies of the institution.

Research Approaches data analysis? Huge amount of data how can we best make sense of it? Some is quantitative; numbers of students who were confident that they learned from the videos Some is qualitative; focus group outcomes, reflective statements Some is good data; tracking access, observation of student behaviour Some is not so good; results influenced by online video going offline

Stake (1967) Countenance model of evaluation

intentions observations

antecedents transactions outcomes

Intentions relate strongly to Predictions; Observations to Results. This approach allows a wide range of data to be assimilated into an analysis.

How did Stakes model help us to analyse this data?

intentions antecedents Staff and students would acquire skills transactions All students would have access to video outcomes Most students would use video

observations issues Not all had Time these skills or chose to develop them IP address restrictions denied access to some About half used video and one third enjoyed it Copyright

What of the others?

Conclusions Educational research can make use of a wealth of qualitative and quantitative data and a mosaic of research approaches. In general its approaches are insufficiently scientific to yield or build generalisable theories, but often, that is not their aim. References/resources
Alexander S and Hedberg J 1994 Evaluating technology based learning; Which model? In: K Beatie, C. McNaught and S. Wills (eds) Multimedia in education; designing for change in teaching and learning. Amsterdam. Elsevier p 233-244 Cronbach L (1975) Beyond the two disciplines of scientific psychology. American Psychologist, 30, 116127 Friedman V J and Rothman V 2002 Action evaluation for knowledge creation in social-education programs. http://www.aepro.org/inprint/papers/knowledge.html Green S, Voegeli D, Harrison M, Phillips J, Knowles J, Weaver M and Shephard K. (2003) Evaluating the use of streaming video to support student learning in a first-year life sciences course for student nurses. Nurse Education Today 23, 255-261 Green S, Weaver M, Voegeli D, Fitzsimmons D, Knowles J, Harrison M and Shephard K (2006) The use of a Virtual Learning Environment (Blackboard 5) to support the learning of pre-qualifying nursing students undertaking an anatomy and physiology module. Nurse Education Today 26 (5) 388-395 LIFESIGN 2002 The LIFESIGN Project. www.lifesign.ac.uk

Shulman L S (1981) Disciplines of Inquiry in Education: An Overview. Educational Researcher 10 (6) 5-12 Stake R E 1967 The countenance of educational evaluation. Teachers College Record 68 (7): 523-540. Return

Bookmarked Resources
Session Outline

Module title & Code Level & year Session title Lecturer Aim of session Learning Outcomes Learning & teaching strategies Related learning activities Indicative content Key references & other learning resources

Foundation in life sciences (LS01) Level 1, year 1 The Brain in situ Jess Knowles & Jess Knight to introduce the structure of the brain At the end of the session & following associated study each student will be able to describe the basic anatomy & physiology of the brain Directed learning session 4 lectures on the nervous system The nervous system Recommended & key text for LS01 or a physiology book of your choice.

WARNING!! This video contains very graphic illustrations of a human brain being dissected. This video is not essential viewing for this module if you are likely to be disturbed by viewing this type of material. The material in your course textbook covers the topics illustrated. The Brain in situ This page provides access to segments of the Shotlist video; The Brain in situ. The narrator is Professor Susan Standring of the United Medical & Dental Schools, London. Professor Standring gives a guided tour of the main structures of the human brain, using museum specimens and a brain seen in situ and removed from a cadaver. A skeleton, specimen brain and spinal chord and sagittal section of a head and neck are used to demonstrate the location of the brain and show some features of the spinal chord such as dorsal root ganglia. She then watches as the skull cap is removed from a cadaver to reveal the brain in situ. Once the dura mater is removed she points out features such as the lobes, arachnoid granulations, vascular system and optic nerve. The brain is then removed to reveal the ventral surface, including the cerebellum, pons and medulla. We see the brain being hung to fix in a bucket of fixative. With the brain removed, she examines the cranial cavity, showing features such as the crista galli, pituitary stalk, cranial nerves, foramen magnum, medulla and some cerebral-spinal fluid. She then discusses the ventricular system, explaining the role of cerebro-spinal fluid in protecting the brain and showing where the ventricular system is to be found using a horizontal section through the brain. A resin cast of the ventricular system shows its structure. Finally Prof. Standring explains why she feels study of the brain and nervous system is important. The complete video plays for nearly 20 minutes. Click on the blue hyperlinks below to view the video or short segments as directed.

1 Watch Part 1 of the video; the brain in situ Watch Part 2 of the video; the brain removed. Note: This video shows detailed shots of work on a cadaver and a brain, which may disturb some viewers. The cadaver is covered up below the skullcap at all times, making identification impossible. 2 Focus on shorter segments of the video and use the indicated learning tools to direct your learning. 2.i. 2.ii. Introduction General anatomy

Learning Tool A; draw your own labelled-diagram of a side-view of the brain. 2.iii. Meninges, brain gyri and general brain anatomy Learning Tool B; As structures are identified in the video, record their names and then spend some time, using the recommended text for this module, to identify their functions. Look particularly at Meninges, Lobes, Olfactory Tract. 2.iv. Cranial nerves Learning Tool C; Identify all 12 pairs of cranial nerves and list their functions.

2.v.

Summary of brain structure

Learning Tool D; As structures are identified in the video, record their names and then spend some time, using the recommended text for this module, to identify their functions. Look particularly at Cerebellum, Brainstem, Pituitary. 2.vi. Ventricular system Learning Tool E; write a short summary of the structure and function of the ventricular system and cerebro-spinal fluid. 2.vii. Summary and conclusions Learning Tool F; write short notes to describe why you think that it is important to study the structure, functioning and pathology of the human nervous system. Please remember to use the MCQ to test your learning.
Return

Bookmarked Resources 2

Each session was associated with a MCQ designed primarily to allow students to formatively assess their own progress. The three Directed Learning Sessions that used streaming video all included three non-scored 'survey questions' designed to ascertain students access to streaming video; and enjoyment of, and learning-confidence in the way that it was used. The three questions were;

Was the streamed video easy to access? (yes, no, don't know, no response) Did you enjoy using this learning resource? (yes, no, don't know, no response) How confident are you that you have learned from this learning resource. (very, not very, don't know, no response)

Blackboard's on-line grade-book gave access to the responses of each student.

Return Bookmarked Resource 3

One quote from a reflective email will illustrate the point and provides information that relates to the data on student access to, and use of, the resources. I spoke to my class yesterday and asked specifically about the video. Those students who were able to access

it thought it was very useful.... (but). It was commonly reported as fragmented, far too slow or it crashed. About 50% of my class said they tried to access it and gave up...that's not good.
Return