Recommendation for Question Bank for the College of Medicine, University of Saskatchewan

Date: March 1, 2011

Submitted to

Dr. Gary Linassi, Assistant Dean

by
Dr. Kalyani Premkumar, MBBS MD MSc (Med Ed) PhD
Associate Director, Educational Support & Development Unit,
College of Medicine, University of Saskatchewan

Current Situation

The College of Medicine currently has a question bank that consists of approximately 8366 questions,
stored in Lxr software, managed by an administrative assistant. These questions are primarily multiple
choice questions requiring students to choose one best answer. The bank also includes a few questions
that are of short answer type, true/false and matching. The questions currently available test
information only relating to 2 (the Form and Function of the Human Body Course and Systems Course)
of the approximately 31 courses taken in undergraduate medicine.

While 8366 questions for two courses may seem like an impressive number, the quality of our question
bank is questionable for the following reasons:

many of the questions are duplicates or triplicates (the same question often gets uploaded as a
new question when coordinators do not flag it as a repeat question when they submit questions
to the administrator -- who is not a content expert)

Many questions are fraught with technical errors (possibly due to lack of training in item writing)

The questions are not amenable to searching (the keywords assigned often only relate to the
course it belongs to and/or a broad topic)

Keywords assigned to some of the questions are not consistent; for example ECG; EKG;
electrocardiogram may all be used. In such cases, all questions relating to the same topic are not
identified

Reuse of questions, year after year, has resulted in student-created archived banks that gets
passed on from class to class

1
 By allowing students to review their questions and answers given in weighted exams in the past
years, students in subsequent years have been able to compile a question bank. This has led to
memorisation of answers rather than mastery of the content, thus compromising the reliability
and validity of the test results

For other courses of the undergraduate curriculum, individual coordinators have their own question
banks in their hard drive. They administer the examinations themselves and provide the grades to the
administrators. Similarly, OSCEs and other forms of assessment in the clinical courses are not managed
by a centralised body nor are they stored in a centralized place.

The process used by Coordinators for choosing questions in specific exams is not available. The basis
and rationale for discarding questions in a given examination is also not transparent. To my knowledge,
the only course that has a pre and post-exam meeting to review examinations is the Form and Function
Course.

As a college, we have not identified a mechanism for determining the pass-fail grade.

The problem

Because of the current situation, a number of problems exist and there is potential for problems to arise
in the future:

- We dont have a defense for the process used for choosing questions

- We do not have a rationale for determining pass/fail grade

- We are not fully utilizing the item analysis, discriminative and difficulty indices as evidence for
identifying the quality of test items

- Every course does not use an examination blueprint prior to creating examinations

- The topics tested in the exams are determined by what is covered by instructors, not by the
objectives determined for the course

- Because of the fact that exams test topics covered by instructors, students tend to use only
their class notes while preparing for examinations (many students do not buy textbooks; nor do
they use other resources for studying)

- Topics not covered face-to-face are not tested; this leads to gaps in the knowledge of students;
this in turn leads to poor performance in licensing exams

- Since there is a parade of instructors (specialists and others) in each course, the curriculum
seems to be determined by what individual instructors decide to teach on a particular topic. This
situation is perpetuated as, the exam is compiled by asking each instructor to submit a few
questions.

2
Rationale for Question Bank

Some of the existing problems may be mitigated by acquiring a question bank. By adopting a question
bank shared by other medical schools, technically correct questions can be used; coordinators are
forced to create an examination blue-print; there is a possibility of benchmarking our students
performance against students of other medical schools; there is greater scope for providing students
with formative assessment; students are exposed to questions similar to what they are likely to see in
licensing examinations; many question banks provide guidelines for pass/fail grades that are derived
from universally accepted methods. As the questions in examinations are based on an examination
blueprint, it will force instructors to teach to MCC objectives rather than to what is of interest to them
individually.

The Needs of College of Medicine, University of Saskatchewan

Based on our current situation, we need a question bank that

- provides access to both formative and summative questions

- allows blueprinting of exams
- has a large number of questions
- has minimal additional commitments (we do not have a collection of quality formative and
summative questions; we do not have the resources to create a large number of high quality
questions for submission every year)
- is comparable in cost to other available question banks

Process Used to Identify a Suitable Question Bank

All Canadian medical schools were contacted in addition to a number of other US and international
medical schools. A general call for information relating to question banks was sent out on two medical
education list-serves. Individuals in specific medical schools that use a particular question bank were
also contacted to find out their experience with that question bank. The contact person for individual
question banks was also contacted by email/telephone. As well, websites describing individual question
banks were scrutinized. A comparison was then made of the four commonly used question banks (see
Table ). (I can provide further details of each of the products on request)

IDEAL - http://www.hkwebmed.org/idealweb/index.html
Exam Master - http://www.exammaster.com/
NBME - http://www.nbme.org/
IPPT - http://ipptx.org/

3
 Product IDEAL Exam Master NBME IPPT

1. Basic science questions? Yes Yes Yes Yes

2. Clinical questions? Yes Yes Yes Yes

3. Number of questions in the 22,000 summative; 23,000+ 10,850 2,291 (+ 450 for 2011)
bank? 6500 formative

4. Formative assessment. Yes Yes User dependent Yes

5. Summative assessment. Yes Yes User dependent Used as part of summative assessment

6. Search capabilities (subject; Yes Yes Various ways to filter items Yes limited
disease group; any other key
words?)

7. Blueprinting (can be done Schools choose Faculty create exams; Done within the tool once Each partner defines its own test
internally or only by external questions as they can edit questions approved, user generates blueprint currently all IPPT progress
means; e.g. do you choose the wish. test from it tests make use of the same blueprint
questions for us?)

8. Potential growth of questions 3000 3500 items Dependent on authors 3,500 4,000 Dependent on partner contribution
per year.

9. Types of questions (MCQ; short Most are A-type, then MCQ; faculty can modify MCQ, A-Type and R-Type. Most are clinical stem, 5-option,
answers etc.) that can be used. R-Type; 500 OSCE to create other types Clinical vignette (images, best/correct response
stations drawing, experimental)

10. Security. Requirements of Lock-down browser Yes from NBME and user Yes access is by username and
schools using the school password
banks

11. Our additional commitment All existing high Agree to terms of Not currently Encouraged, but not required,
(i.e. additional questions to be quality exam items - licensing agreement contribution
supplied etc.). 150 A-type or
equivalent

12. Cost. $6000 U.S. annually $2995 (Student version); $1,500 annually plus cost Entry fee - $5,000; annual membership
(likely $8000 after $5995 (Faculty) 1st per examinee (varies) - $5,000 Cdn.
June 30, 2011) year introductory offer

13. Adoption (i.e. number of 28 Members (3 in List is on 44 Schools 35 Currently 6 current members
schools that have adopted Canada www.exammaster.com/i
Canadian; US; UK; Worldwide). mages/pdf/client-list.pdf

> 14. Internal control (you can
mark using a scale how much
control does the institution using
the exam bank have in terms of
marking?)
Use banks whenever Faculty control of Does not weight items; item May request different scoring methods
and however you wish questions score statistics provided during but online application scores all tests
(access restrictions on reports are generated construction; can flag items the same way
summative to ensure automatically and create notes
security)

15. Reports (item analysis; Yes, for both selected Yes. Not for comparison Yes, a variety of item Not generally. However, reports
individual student analysis; and constructed to other schools analysis is provided provide students and managers
comparison with other responses performance relative to peers
Schools?)

4
Findings

Many of the Canadian medical schools (e.g. Univerisity of Manitoba; McGill; Dalhousie) continue to use
their own banks. Some schools (e.g. U of Western Ontario) use external banks for student use. A large
number of US, UK, Europe and Australian schools use external question banks, such as those listed
above.

In addition to question banks that cater to all subjects in the undergraduate curriculum, certain
Associations (e.g., Pathology; Obstetric & Gynaecology; Pediatrics) have developed questions banks for
their own disciplines.

During my research, I discovered that our College has been a subscriber of the National Board of
Medical Examiners (NBME) question bank since 2008. However, this bank has been used sparingly. We
already have a Chief Exam Proctor in place. Surgery and Medicine have experimented with the bank
twice (using 1 and 2 undergraduate examinees respectively). Internal Medicine is hoping to use this
question bank for a few more clerks in March 2011. From March 2011, Regina campus has been
established as a satellite site for conducting NBME exams.

Recommendations

Based on research, the following are my recommendations:

1. Continue to subscribe to NBME and use this bank for summative examinations. Apart from the
annual subscription cost, a fee per student is required. The cost is comparable to other banks.
NBME has a large number of questions that can be used throughout the curriculum. Currently,
most of our faculty are unaware that this bank is available. We need to aggressively market and
train coordinators to use this question bank. The drawback is that there are no practice exams
for students.

2. Subscribe to the ExamMaster Online, for students, for one year. This version allows students to
practice and review questions. The questions have explanations for answers as well. The cost is
$50 per student. I suggest that we initially provide access to 3rd and 4th year students, to help
them prepare for their licensing exams. This may be piloted for one year and then discontinued,
based on usage and feedback from students.

3. Ask Obstetrics & Gynaecology and Pathology faculty if they would like to use the question bank
created by their respective associations (Association of Professors of Gynecology and Obstetrics
APGO; Group for Research in pathology education GRIPE). The cost of the pathology bank is
$900. The questions are provided as a DVD and instructors have the option to choose from the
bank for both formative and summative exams. If the faculty is interested, subscribe to the
question bank for one year and then revisit based on feedback. The cost of APGO has to be
determined.
5
4. Encourage Pediatrics to continue to use CLIPP (Computer-assisted Learning in Pediatrics
Program) during paediatrics rotation.

5. Renew our licence to ACDET (http://acdet.com/) the web-based differential diagnosis tutor
KBIT. This tutorial has 33 presenting problem lists (e.g. cough; congestive heart failure; dyspnea
etc.). Based on level of training, students can be assigned specific modules to complete. The cost
is based on per student usage. 2nd and 3rd year students may be given access to facilitate critical
reasoning and critical judgement skills. A few years ago we did have the license. Since its
availability was not made known to all faculty, it was not fully utilized. There is evidence to show
that this tutor improves clinical reasoning skills. I suggest that we obtain a license for one year.
We need to provide access to the software and encourage systems coordinators to assign
specific modules during training.

6. After a few years, strive to join the IDEAL consortium. One of the advantages of this consortium
is that it allows institutions to access both summative and formative exams. There is also more
internal control of examinations, as compared to NBME. Currently, we will not be admitted into
the consortium as we do not have our own high quality question bank. In addition, we do not
have the potential to create 150 high quality questions per year.

7. Provide resources to ensure that our current question bank is screened for technical quality,
properly tagged with key words, and screened for duplication before it is moved over from lxr
to Question Mark (exam management software). It is important to have a proper system put
in place in order to maintain our question bank.

8. Train faculty in exam creation, examination blueprinting and assessment in general.

9. Create a college-wide policy for administering weighted examination, in face-to-face and in a

distributed setting.

Acknowledgement

I would like to thank Mr. Sean Polreis and Ms. Janice Cruise of Educational Support &
Development Unit for help with the research.