Evaluation of Student

Learning:
Test Construction & Design
Mohamed Nadzri Mohd Yusoff (Hj.)

CURRICULUM VITAE
Mohamed Nadzri bin Mohd . Yusoff (Hj.)
MSc.Medical Imaging Science (Aus)

Address
Telephone No.
Email

:
:
:

Lot.6471-B, Jalan Teratai, Kg.Paya Jaras Hilir, 47000 Sg Buloh, Selangor D. E

012 4558130 (HP)
nadzy65uitm@salam.uitm.edu.my
nadzy65uitm@puncakalam.uitm.edu.my

Website

https://sites.google.com/site/nadzy65/home/obe

Current
Employer

Universiti Teknologi MARA (UiTM), Fakulti Sains Kesihatan , UiTM Kampus Puncak Alam

Senior Lecturer (UiTM),

PSMB Certified Trainer
MQA Training Centre Trainer
MQA Panel of Assessor

OBE PROCESS
Area
8

Curriculum

Curriculum REVIEW
CQI Report

Planning
(1)

Area
9

MQF 1

Program Outcomes (PO)

MQF 2

Area
2

Curriculum

Evaluate / analyse

Evaluatin
g (4)

Program
Standards

PEO

Academic Level

Closing the Loop (CDL)

Area
7

Area
1

Developin
g (2)

Course Outcomes (CO)

Area
3

TL Activities

Assessments

Entrance/Exit Surveys

Lesson Plan
Marking Scheme Rubrics
Student Portfolio
Blooms
Formative AssessmentTaxonomy

Summative Assessment

SLT

MQF 3

Test Blueprint

Implementin
g (3)
Evidences

Credit Hour

Curriculum

Documenting

Area
6

Area
5

Blooms
Taxonomy

Area
4

Blooms
Taxonomy

Assessment, in general, serves the purposes to:

I. promote learning;
II. measure performance, by awarding grades which
indicate whether and how well a particular student has
attained the stated LOs;
III. determine whether a particular student is sufficiently well
prepared in a subject area to proceed to the next level
of instruction;
IV. provide feedback to students which indicates levels of
attainment and diagnoses misunderstandings and
learning difficulties; and
V. provide feedback to teaching staff to identify and
diagnose ineffective teaching methods/ techniques.

Guiding Principles for

Evaluation
Evaluation should relate directly to learning
outcomes
Each evaluation activity should be designed to
promote student growth
The actual activity should be useful practice in itself
Feedback should be useable by the student

Multiple evaluation strategies should be provided

to master achievement of
outcomes/competencies
Student should clearly understand the methods of
evaluation for test or activity

What to assess?

What to assess?
Learrning Outcome
(LO)

Teaching and
Learning
Strategy

Assessment
Strategy

LO1
LO2

knowledge;

Lecture, Tutorial

Written Tests, Quiz

practical skills;

Practical,
Demonstration

Practical Tests

LO3

problem solving and scientific

skills;

Case Study, Project,

Tutorial, Group Work

Presentation, Essay

LO4

communication skills;

Case Study, Project,

Tutorial,

Presentation

LO5

social skills, team skills and

responsibilities;

Case Study, Project,

Tutorial, Group Work

Presentation, Project

LO6

values, attitudes and

professionalism;

Case Study, Project,

Tutorial, Discussion

Witten, Presentation,
Project

LO7

information management and

lifelong learning skills;

Case Study, Project,

Group Work

Project, Portfolio

LO8

managerial and entrepreneurial

skills.

Case Study, Project,

Project, Industrial
Attachment

LO9

leadership

Case Study, Project,

Group Work,

Project, Industrial
Attachment

Questions to Ask yourself in

Designing a Test
What outcomes will (should) I be testing?

LO3

Refer to CLOs : What PLO is the CLO

addressing?
Eg: Organise in groups to study and present
the application of advanced manufacturing and
operation analysis methods based on a given
topicssolving
(C5,
P5, PLO3,
PLO4, PLO5)
problem
andA4,
scientific
Case Study,
Presentation, Essay
skills;

Project, Tutorial,
Group Work

LO4

communication skills;

Case Study,
Project, Tutorial,

Presentation

LO5

social skills, team skills and

responsibilities;

Case Study,
Project, Tutorial,
Group Work

Presentation,
Project

Questions to Ask yourself in

Designing a Test
What types of items will be included in the
test?
MCQS, Essays?
E.g: will be able to define the basic principles
of (C1, PLO1)
C1: MCQs, Short Answer
E.g: .will be able to analyse a case based on
the theory of.. (C4, A4, PLO3)
C4: Case study, Assignment, Project

Questions to Ask yourself in

Designing a Test
How long will the test be in terms of time
and number of items?
1 MCQ = 70 90 seconds, @ 40-50 MCQs = 60
mins
Short answer (3-5 marks) = 5-10 minutes
Essay (20-25 marks) = 30 45 minutes
@ 100 marks = 120 mins
@ 1 mark = 72 seconds

Questions to Ask yourself in

Designing a Test
How much will each outcomes be worth in
terms of weighting and number of items?

Tests as Diagnostic Tools

Students demonstrate learning
Instructor effectiveness modify teaching
strategies or activities
Assignment of letter grades

Different Types of Tests &

Learning
Paper & Pencil/Online Testing
Limited Choice Questions (MC, T/F, Matching)
Open-Ended Questions (Short Answer, Essay)

Performance Testing
Acquisition of skills that can be demonstrated through
action (e.g Practical Tests, OSCE)

Translating Course Outcomes/

Competencies into Test Items
Syllabus
what was taught/weight areas to be tested

Creating a Test Blueprint

Blueprint- this is the test plan, i.e., which questions, test what
concept
Plotting the outcomes/competencies against some hierarchy
representing levels of cognitive difficulty or depth of processing

Thinking Skills
What level of learning corresponds to the course
content
Blooms Taxonomy of Educational Objectives
Knowledge
Comprehension
Application
Analysis
Synthesis
Evaluation

Practical Considerations
Representative sample of the course
content, not random purposeful; based
on blueprint
Representative sample of skill or cognitive
levels across content
Analyse results by level AND content area

Planning a Test
Step 1:
Outline learning outcomes or major concepts
to be covered by the test
Test should be representative of outcomes and material
covered
Major student complaint: Tests dont fairly cover the
material that was supposed to be canvassed on the test.

Planning a Test
Step 2:
Refer to test blueprint (from course
syllabus)
Step 3:
Create questions based on blueprint
Match the question type with the appropriate level of
learning

Planning a Test
Step 4:
For each check on the blueprint, jot
down 3-4 alternative question on
ideas and item types which will get at
the same outcomes
Step 5:
Organise questions and/or ideas by
item types

Planning a Test
Step 6:
Eliminate similar questions
Step 7:
Walk away from this for a couple of
days

Planning a Test
Step 8:
Reread all of the items try doing
this from the standpoint of a student
Step 9:
Organise questions logically

Planning a Test
Step 10:
Time yourself actually taking the test
and then multiply that by about 3-4
depending on the level of students
Step 11:
Analyse the results (item analyses)

Question Arrangement on a
Test
Group by question type
Common instructions will save reading time

Limit the number of times students have to

change frame of reference
Patterns on test must be logical
Arrange from a content standpoint
Keep similar concepts together

Group by difficulty (easy to hard)

Selecting the Right Type of

Evaluation
How do you know what type of question to use
and when?
It depends on the skill you are testing.

Evaluation should always match as closely as

possible the actual activity youre teaching.
Examples: Teaching Rad.Tech, should evaluate
performance
If testing ability to describe procedure, better give an
essay.
Testing reading (i.e knowledge) MC, TF
Wouldnt use MC to test reflective practice, critical
analysis, comparative imaging, image critique

Question Types vs . Cognitive Levels

of Learning
Knowledge
Comprehension

Application

Analysis
Synthesis
Evaluation

Multiple Choice
(MC)
True/False (TF)
Matching
Completion
Short Answer

MC
MC
Short
Short
Answer
Answer
Problems
Essay
Essay
Performanc
e

Constructing the Test

Types of Test Questions:
Multiple-Choice Items
True-False Items
Matching Items
Fill-In, Completion or Short-Answer Items
Essay Questions

Multiple Choice Items

Advantages:
Extremely versatile- can measure the higher level
mental processes (application, analysis, synthesis and
evaluation)
A compromise between a short answer/essay and T/F
item
Can cover a wide range of content can be sampled by
one test

Disadvantages
Difficult to construct plausible alternative responses

Multiple Choice Items

Four Basic Types

Question Type
Incomplete Statement Type
Right Answer Type
Best Answer Type

Which Type is Best?

Question Type vs. Incomplete Statement
Right Answer vs. Best Answer Type

Multiple Choice Items

Writing the stem first:
Be sure the stem asks a clear question
Stems phrased as questions are usually easier
to write
Stems should not contain a lot of irrelevant
info.
Appropriate reading level/terms
Be sure the stem is grammatically correct
Avoid negatively stated stems

Multiple Choice Items

Writing the correct response
Use same terms/reading level
Avoid too many qualifiers
Assign a random position in the answer sequence

Read the stem and correct response

together
Generate the distractors/alternative
responses

Multiple Choice Items

Other Tips for Constructing MC Items:
Items should have 4-5 alternatives.
Stem should present a single, clearly
formulated problem
Simple, understandable, exclude extraneous
words from both stem and alternatives
Include in the stem any word that are repeated
in each response
Avoid All of the above (can answer based on
partial information)
Avoid None of the above

Multiple Choice Items

Other Tips for Constructing MC Items:
Alternative responses/distractors should be
plausible and as homogeneous as possible
Response alternatives should not overlap
Two synonymous terms (arithmetic average/mean)

Multiple Choice Items

Other Tips for Constructing MC Items:
Avoid double negatives
None of the following are part of the brain except
which one?

Emphasise negative wording

Each item should be independent of other
items in the test
Information in the stem of one item should NOT help
answer another item.

Multiple Choice Items

The most common mistakes made when writing
multiple choice questions are:
1.Asking vague questions
2.Asking two questions at once
3.Asking questions only at the Knowledge level
4.Making more than one plausible answer

#1 : Asking Vague Questions

Example of a typical vague multiple choice
question:
Which of the following was the primary economic
problems from the last recession?
a)
b)
c)
d)

Job losses
Strict monetary policy
Weakness of the ringgit
Consumer confidence loss

Fixing Vague Questions

Why was that a vague question?
1. When was the last recession?
2. Was there just one primary problem?
3. Where was the recession? In Malaysia or
elsewhere?
*** Quiz:
At which level on Blooms Taxonomy is this
question?

Fixing Vague Questions

Answer:
At the knowledge level IF you or the text
specifically said this information.
At the comprehension level if the student had to
make a conclusion.
How could you make the question less vague?

Fixing Vague Questions

According to class lecture, the economic problem
that contributed most to the recession in Malaysia
in 1987 was:
a)Job losses
b)Strict monetary policy
c)Weakness of the dollar
d)Consumer confidence loss

#2: Asking Two Questions at Once

Sometimes a test writer asks two questions at once.
This confuses the student. Example:
When exercising, what is the initial source of
energy and where is it stored?
a)Glycogen
b)Fat
c)Carbohydrates
d)Gatorade

#2: Asking Two Questions at Once

You COULD put the answers to both questions in
your answer list, but it is still confusing for the
students:
When exercising, what is the initial source of energy
and where is it stored?
a)Glycogen, muscle
b)Fat, thighs
c)Carbohydrates, muscle
d)Gatorade, stomach

#2: Asking Two Questions at Once

A better solution would be to ask two separate
questions:
When exercising, what is the initial source of
energy?
Where is the initial source of energy used when
exercising stored?

#3: Questions at the Knowledge Level

It is easiest to write multiple choice questions at the
knowledge level because you can take a fact and
convert into a question.
HOWEVER,
most instructors want to test at higher levels of
learning than just memory.
What can you do?

Questions at Other Levels

To write multiple choice questions at higher levels,
you can:

Have students read a passage and answer

questions requiring them to comprehend,
analyze, or synthesize information (e.g IELTS)

Set up a hypothetical situation and ask students

to answer several questions about it.

Example Question at a Higher Level

Johan and Joanna were arguing over the furniture
while they were settling their divorce agreement.
Joanna said to Johan, I never liked that chair
anyway. At which of Knapps relational
conversation levels is this statement?
Integrating
Differentiating
Sustaining
Initiating

Example Question at a Higher Level

This question was at the Application level.
Why?
Because it required a student to understand the
relationship levels and apply a statement to that
typology.

#4: Making More than Two Plausible Answers

One of the worst mistakes in writing multiple choice
questions is having more than one plausible answer.
Students will continue to try to get points from
instructors who have more than one plausible
answer to questions.
How do you avoid this problem?

#4: Making More than Two Plausible Answers

When writing answers, have one that is clearly
correct. The others need to sound plausible but
definitely be WRONG.
Sometimes faculty try to trick students with
answers that seem very similar to correct ones.
Dont fall into that trap or you will have students
arguing with you.
Make sure all wrong answers are defensible and
have real reasons for being wrong.

True-False Test Items

Best suited for testing 3 kinds of
info.:
Knowledge level learning
Understanding of misconceptions
When there are two logical responses

True-False Test Items

Advantages:
Sample a large amount of learning per
unit of student testing time

Disadvantages:
Tends to be very easy
50-50 chance of guessing
Tends to be low in reliability

True-False Test Items

Tips for constructing True-False Items
Avoid double negatives
Avoid long or complex sentences
Specific determiners (always, never, only, etc.)
should be used with caution
Include only one central idea in each
statement
Avoid emphasizing the trivial
Exact quantitative (two, three, four) language
is better than qualitative (some, few, many)
Avoid a pattern of answers

Objective Test Item

Analyses
Evaluating the Effectiveness of Items.
Why?
Scientific way to improve the quality of tests
and test items
Identify poorly written items which mislead
students
Identify areas (competencies) of difficulty

Item analyses provided info. on:

Item difficulty
Item discrimination
Effectiveness of alternatives in MC Tests

Short-Answer Items
Two Types: (Question and Incomplete Statement)
Advantages:
Easy to construct
Excellent format for measuring who, what, when, and
where info.
Guessing in minimized
Student must know the material- rather than simply
recognize the answer

Disadvantages:
Grading can be time consuming
More than one answer can be correct

Short Answer Items

Tips for Constructing Short Answer Items
Better to supply the term and require a definition
For numerical answers, indicate the degree of
precision expected and the units in which they are to
be expressed.
Use direct questions rather than incomplete
statements
Try to phrase items so that there is only one possible
correct response
When incomplete statements are used, do not use
more than one blank within an item.

Essay Questions
Types of Essay Questions
Extended Response Question
Great deal of latitude on how to respond to a question.
Example: Discuss imaging goals.

Restricted Response Question

More specific, easier to score, improved reliability and
validity
Example: Compare and contrast the relative advantages
or disadvantages of MRI in head trauma with respect to:
diagnostic accuracy, image quality, dose

Essay Items
Advantages:
Measures higher learning levels (synthesis,
evaluation) and is easier to construct than an
objective test item
Students are less likely to answer an essay
question by guessing
Require superior study methods
Offer students an opportunity to demonstrate
their abilities to:
Organize knowledge
Express opinions
Foster creativity

Essay Items
May limit the sampling of material
covered
Tends to reduce validity of the test

Disadvantages

Subjective unreliable nature of scoring

halo effect good or bad students previous
level of performance
Written expression
Handwriting legibility
Grammatical and spelling errors

Time Consuming

Essay Questions
Give students a clear idea of the scope & direction
intended for the answer
Might help to start the question with the description of the
required behavior (e.g., compare, analyze)

Appropriate language level for students

Construct questions that require students to
demonstrate a command of background info, but
do not simply repeat that info.
If question calls for an opinion, be sure that the
emphasis is not on the opinion but on the way its
presented or argued.
Use a larger number of shorter, more specific
questions rather than one or two longer questions
so that more information can be assessed.

Essay Questions
You might
Give students a pair of sample answers to a question of
the type you will give on the test.
Sketch out a rubric (grading scheme) for each question
before reading the papers OR randomly select a few to
read and make up the grading scheme based on those
answers
Give students a writing rubric
Detach identifying information and use code numbers
instead to avoid letting personality factors influence you.
After grading all the papers on one item, reread the first
few to make sure you maintained consistent standards
Be clear to student the extend to which factors other than
content (e.g., grammar, handwriting, etc.) will influence
the grade.

Essay Questions
Tips for constructing Essay Questions
Provide reasonable time limits for each question
thinking and writing time

Avoid permitting students a choice of questions

Will not necessarily get a representative sample of
student achievement. Only be requiring all students to
answer all questions can their achievement be
compared

A definite task should be put forth to the student

Critical words: compare, contrast, analyze, evaluate, etc.

Scoring Essay Items

Write an outline of the key points (use outline to
design a rubric)
Determine how many points are to be assigned to
the question as a whole and to the various parts
within it.
If possible, score the test without knowledge of
the students name
Face Sheet

Score all of the answers to one question before

proceeding to the next question
Consistent standard

Scoring Essay Exams

If possible, score each set of answers
within the same time frame
Handwriting, spelling & Neatness
Two separate grades?
Mastery of material
Other

Alternative Methods of
Assessment

Research/Term Papers
Research Reviews
Reports
Case Studies
Portfolios
Projects
Performances
Peer evaluation
Mastery
Simulations

Cheating
Preventing Cheating
Reduce the pressure (multiple
evaluations)
Make reasonable demands
(length/content of exam)
Use alternative seating
Use alternative forms
Be cautious with extra copies

Using Assessment & Evaluation to

Improve Student Learning Outcomes

Providing feedback to student

Closing the assessment & evaluation
loop
Maximising student learning

Food for thought?

How do you determine the weightage
of your assessment? (e.g. Test =
10%, Assignment=10%)
How do you determine the duration
of your final examination? (e.g. 2hrs,
2.5 hrs, 3 hrs.)