INNOVATIONS ENHANCING STUDENT LEARNING

INNOVATION: Certainty-based online MCQs for the facilitation of self-reflection in nursing students
Simon Coghlan

What is it?
This innovation involves a specialised extension of the bog-standard multiple choice question (MCQ), and is designed with the modest intention of beginning the process of self-reflective learning. Standard MCQs are designed to test a students understanding of a body of subject matter. They do not necessarily merely test the recall of the student, but are able also test a students ability to analyse and evaluate information or ideas (even if they are hardly an exhaustive means of assessment). However, standard MCQs do not especially develop the skill of self-reflection on levels of understanding and competence with content and concepts. In contrast, this new form of MCQ explicitly encourages students to specifically examine their level of learning and confidence in that learning. In so doing, it initiates them into the skill and lifelong habit of reflective learning. As I will later explain, this approach is intended merely as an introduction to the skill of self-reflection.

How does it work?

Students are first taught a component of the course, such as via an online module. The MCQs are then delivered this can also be done online as an e-learning activity and incorporated into the modules. The online questions can be delivered at a specified time to a large number of students and used as either a formative or summative assessment task. Because it is assessed, all students have a reason to participate. The special addition to the standard MCQs is that students are asked to rate how certain they are about their answer on a three-point certainty scale of low, medium and high. If their answer was correct, they get 3 marks for being certain, 2 marks for a medium level of confidence, and 1 mark for a low level of confidence. But if their response was incorrect, students lose 6 marks for ticking the high certainty box, 2 marks for medium certainty, and zero marks for low level certainty. Students are then emailed the breakdown of their test results so that they can see how their answers match up with their level of confidence or certainty for each question. The overall result is represented in a composite score. The approach of employing relative weighting of levels of certainty has a basis in probabilistic decision theory (Gardner-Medwin & Curtin, 2007). This allows both for the effect of guesswork to be minimised and for the certainty-criteria to accurately reflect the students

known understanding of the subject material.

Who else is using it?

The use of certainty based marking has been occurring for over 10 years at University College of London and Imperial College London, with reportedly good success and positive student and teacher feedback (Gardner-Medwin & Curtin, 2007). In Australia, it has been trialled at universities in Melbourne, Sydney, and Tasmania Khanal & Zuo, 2012).

Why is it useful in teaching?

From a purely practical point of view, online MCQs can be marked relatively quickly, which allows teachers to economise on time while assessing large numbers of students. The addition of the certainty-weighting does not hugely extend marking time. Another benefit is that the weighting factor helps to correct for guesswork in MCQs. More relevant for our purposes, however, is the fact that it can aid in self-reflection. When answering multiple choice questions, students may have a mistaken level of certainty or confidence about their answers they may be over-confident or overly diffident. This can occur for personality-based reasons, or simply because they have not correctly or adequately come to terms with the subject matter. Forcing students to reflect upon their level of understanding can help them to identify areas of the course that they are either understand well or know poorly. This is especially useful in a formative assessment setting, because it affords the opportunity to revise and learn in greater depth. In other words, a metacognitive approach, which gives pupils insight into their present state of knowledge and understanding, is an aid to deep, as opposed to superficial, learning. Moreover, the certainty-based approach is designed to support the skill and habit of self-reflective learning. This is a desirable skill to have in its own right, not merely as an aid to learning a particular bit of subject matter. Clearly, it is a cognitive skill that is highly relevant to medical professions, including nursing. This profession requires ongoing learning and continual adjustment of theory to practice. The present approach encourages the student to ask themselves Do I really understand this? How sure am I that I am right or wrong?. The student may then come to see more clearly that they were previously too confident or too unconfident, and to take action accordingly. Because this is a skill, it can be taken on even by students who as a matter of personality or disposition tend to misrecognise their level of ability and understanding. And because this approach requires an explicit and conscious metacognitive effort to self-scrutinise, rather than the infrequent, haphazard and largely unconscious monitoring of self, it helps to build the habit and skill of reflective learning. Such reflection, as Schon (1983) recognised, can be reflection in action or reflection on action. As can be seen, the weighted MCQs call upon students to question their understanding during the test and also before and after the test. Finally, the certainty-based approach can be incorporated into any MCQ test and, if its uptake continues, may indeed come to replace the simple MCQ format.

Innovations enhancing student learning

How can it support students?

The student cohort of concern is ESL International (principally Asian). In the website project, we have identified some learning issues most relevant to this group. Of particular interest is the apparent fact that some international students feel uncomfortable sharing their experiences and are less used to expressing and identifying their feelings in the learning process. They are, perhaps, also less comfortable with being uncertain in situations that are vague and messy. Their learning styles often teacher-centred and focussed on problem solving rather than on feelings and self-reporting - may make them more reluctant to engage with the process of reflective learning. Now, it must be acknowledged that weighted MCQs do not squarely engage with messy real life situations in the way that, say, reflective journaling, does. However, the certainty-based MCQ approach is offered simply as an introduction to the process of self-reflecting upon knowledge and feelings Students are encouraged to pose to themselves questions like: do a feel sure about this or am I uneasy?; am I competent at correctly applying my knowledge?; is my natural diffidence/tentativeness/modesty/overconfidence/impulsiveness an impediment to good learning and practice?, etc. I would suggest that the present setting for such questions a relatively safe method of entering into the reflective state of mind, a state of mind that will later be applied to real life scenarios. The e-learning MCQ format is one that Asian ESL students will likely be familiar and comfortable with note, for example, that such students may tend to be visual learners and to prefer written material - and the feedback at this early stage is through emailed results rather than through group sessions, face-to-face discussions (etc.), some of which may be more anxiety-provoking. Once the students have become somewhat familiar with the explicit metacognitive approach (albeit in a limited way), they may then be more comfortable in moving to some untidier and more complex life situations that do require a higher level of self-analysis and self-evaluation.

Further reading
Gardner-Medwin, A. R., & Curtin, N. A. (2007). Certainty-Based Marking (CBM) For Reflective Learning And Proper Knowledge Assessment. REAP Int. Online Conf. on Assessment Design for Learner Responsibility. Retrieved from http://www.reap.ac.uk/reap/reap07/Portals/2/CSL/t2%20%20great%20designs%20for%20assessment/raising%20students%20metacognition/Certainty_based_marking_for_reflective_learning_and_knowledge_a ssessment.pdf

Innovations enhancing student learning

4 Khanal, S & Zuo, Y. (2012). An innovative approach to facilitate critical thinking and reflective learning in prescribing and therapeutics e-learning. Retrieved from http://www.ascilite.org.au/conferences/wellington12/2012/images/custom /khanal,_santosh_-_an_innovative_approach.pdf Schn, D. (1983). The Reflective Practitioner, How Professionals Think In Action. Basic Books. Zhenhui, R. (2001). Matching Teaching Styles with Learning Styles in East Asian Contexts. The Internet TESL Journal, Vol. VII, No. 7, July. Retrieved from http://iteslj.org/Techniques/Zhenhui-TeachingStyles.html

Innovations enhancing student learning