1

Criterion-Referenced Assessment












Prepared by:

Dr Rob Connoley
Educational Developer/Lecturer Teaching & Learning
Faculty of Business & Law
Deakin University

Version 1 - November 2004


2
Introduction

The main purpose of this paper is to explain the concept of criterion-referenced
assessment and how it differs from norm-referenced assessment. Criterion-referenced
assessment is increasingly employed in education practice, particularly in higher
education. This is because of its focus on measuring achievement against learning
objectives and on assessing changes in performance as a consequence of the student's
learning experience. The following describes the concept of criterion-referenced
assessment, how it differs from norm-referenced assessment and suggests how you
can use rubrics to assist in assessing criterion-referenced assessment activities. You
may recall that the previous two papers dealt with the issues of writing learning
objectives and formulating learning and assessment activities to meet your learning
objectives. Thus, this current paper can be seen as a logical extension to those two
papers.

Learning objectives

After completing this training package, you should be able to:

Define criterion-referenced assessment.
Distinguish between criterion-referenced assessment and norm-referenced
assessment.
Explain how to prepare criterion-referenced assessment tasks.
Describe how a rubric can be developed to present assessment criteria and
performance standards to students.
The increasing attention on assessment practices

The focus on assessment practices in higher education has grown over the past few
years. For instance, in a 2002 study of assessment in Australian universities prepared
by the Centre for the Study of Higher Education (CSHE), assessment was considered
a central element in the overall quality of teaching and learning. A brief review of
some university teaching and learning sites has also revealed most have publications
assisting teaching staff on assessment practices. In addition, the Federal Government
has established The Carrick Institute for Learning and Teaching in Higher Education
for the enhancement of learning and teaching in Australian higher education
providers. One of the Institutes responsibilities will be to improve assessment
practices throughout the sector. Furthermore, the University College Dublin reported
that there have been major changes internationally in assessment practices over the
past 20 years. Some of the major changes noted included the replacement of written
exams with more continuous assessment and coursework; a move towards more
student involvement and choice in assessment; more explicit course outlines about the
expectations in assessment; more frequent use of group assessment; an understanding
of process is now seen as, at least, equally important to a knowledge of facts; and a
greater focus on what the student will learn rather than what the teacher plans to
teach.


3
Types of assessment: Norm-referenced assessment
and Criterion referenced assessment

Both Norm and Criterion-Referenced assessment refer to the 'type' of assessment.
Norm-referenced assessment has been the most common type of assessment used in
higher education. A norm-referenced assessment makes judgements about people,
expresses students scores in rank order, based on a distribution of scores. For
instance, the top 10 per cent of the class are awarded high distinction, the next 15 per
cent distinction etc. Thus, it relates the achievement of a particular student to the
achievements of other students after the teaching is over.

Although norm-referenced assessment may be more common in higher education,
there is a growing emphasis toward the implementation of criterion-referenced
assessment. Criterion-referenced assessment makes judgements about performance,
rather than on people. It assessed the extent to which a student has achieved the
intended learning objectives and performance outcomes of a subject. Thus, student
performance is compared to a previously specified standard of achievement (criteria)
and not relative to other students. Under this type of assessment, teachers and
students can see where students are succeeding and where they are not. The criteria
are usually set before teaching has taken place. It may be necessary to adjust if set
inappropriately high or low before reporting final grades (Biggs 2003, p.145).

How to apply criterion-referenced assessment

The logic of criterion-referenced assessment is say what you want students to be able
to do (see learning objectives), teach them to do it (through lectures, tutorials, and
learning activities), and then see if they can do it (Biggs 2003, p.144). Thus, it is
about alignment. For criterion-referenced assessment to work, it is necessary to be
clear about what your students should be learning in terms of qualities or performance
criteria that define the grading categories and then to devise assessment tasks that will
tell you how well students meet the criteria (Biggs 2003, p.145). The first task is a
matter of setting learning objectives. As noted in the first training package, learning
objectives are statements of what you expect your students to learn as a result of
studying your unit. In other words, what should they be able to do at the end that
they could not do before they commenced the unit. It is a focus on student
performance not on how you teach! The use of action verb within Blooms taxonomy
that has been used to write a learning objective statement inherently establishes the
level of performance you are seeking in respect of that objective.

Criterion referenced assessment is carried out against previously specified yardsticks
('criteria'). Where a grade is assigned, it is assigned on the basis of the performance
standard the student has achieved on each of the criteria.

Criteria are the dimensions along which you judge how well a student has carried out
the assessment task (yardsticks). For example, an essay may be judged on the
following criteria: knowledge of subject material (*), the appropriateness of the
literature cited, the quality of the argument, and the quality of expression.


4
Oral presentations may be judged on the following criteria: knowledge of subject
material, ability to answer or respond to questions, structure of presentation, use of
audio/visual material, pace and timing, and delivery style.

(*)The subject material criteria need to be developed from the learning objective. For
example: if the essay was based on the learning objective: the student is able to
explain the role of the Reserve Bank, then the criteria for assessing a students
achievement may include:

The students ability to state the roles accurately.
The students ability to state the roles in their own words.
The students ability to apply role to an example.


Performance standards are the actual levels which might be achieved by students
against the criteria (measurements along the yardsticks). For example, for each of the
criteria noted above a range of standards could be developed ranging from poor to
excellent. In the above example (*) the standards for each criterion could be: one,
two or three roles stated accurately; one, two or three roles paraphrased accurately;
and one, two, or three roles illustrated by an example. To help develop performance
standards, you will need to write down some ideas as to what is the minimum level of
performance to be met. You should then describe the sort of performance that you
could best hope to expect in this level unit and for these students. Appendix 1
provides a sample of this for developing performance standards for essays. Having
defined these limits, you should then fill in the remaining categories in between. This
could then be presented in a rubric (see next section).

The University of Queenslands Teaching and Educational Development Institute has
a paper on Assessment criteria, standards and marking criteria at
http://www.tedi.uq.edu.au/Teaching/assessment/criteria.html. This paper is a good
source of reference discussing criteria and standards. You may also wish to examine
the following site, which has generic descriptors whenever one is interested in
checking the quality of reasoning about a problem, issue, or situation:
http://www.criticalthinking.org/university/unistan.html. In addition, a further site
provides a suggestion of criteria as they might be applied to essays, whether submitted
in tutorials or completed as part of an examination. To obtain a particular class of
assessment an essay does not have to fulfil all the criteria listed for that class - for
example a beautifully written essay can still fail for being irrelevant - but the list helps
to show what is looked for at university level: see:
http://www.shef.ac.uk/geography/ug/handbook/assess/marking.html.

You should provide students with an explanation of the criteria which are to be
used in judging their attempts and the relative importance of the various
criteria. You should also provide an indication of the expected standards of
performance and grading scale.

5
How to present assessment criteria

A rubric, or scoring matrix, is a very useful device for presenting criterion-referenced
assessment tasks. A rubric is designed to make grading easier and faster, as well as to
provide more useful feedback to students. Especially when grading projects, rubrics
offer an easy, fast, and consistent method for grading.
What is a rubric?

A rubric is a printed set of scoring guidelines (criteria) for evaluating work (a
performance or a product) and for giving feedback. It is a scoring guide or a set of
characteristics that are used to judge student performance.

A rubric is usually handed out before the assignment begins in order to get students to
think about the criteria on which their work will be judged. It is suggested that
rubrics improve students' end products and therefore increase learning. When teachers
evaluate papers or projects, they know implicitly what makes a good final product and
why. When students receive rubrics beforehand, they understand how they will be
evaluated and can prepare accordingly. Sharing the rubric and going over it step-by-
step will help students to understand the standards by which their work will be
judged.
The advantages of using rubrics

The following are some advantages to using rubrics:
Teachers can increase the quality of their direct instruction by providing focus,
emphasis, and attention to particular details as a model for students.
Students have explicit guidelines regarding teacher expectations.
Students can use rubrics as a tool to develop their abilities.
Teachers can reuse rubrics for various activities. An established rubric can be
used or slightly modified and applied to many activities. For example, the
standards for excellence in a writing rubric remain constant throughout the
year; what does change is students' competence and your teaching strategy.
Because the essentials remain constant, it is not necessary to create a
completely new rubric for every activity.
How to create a rubric

A rubric is constructed by establishing a set of characteristics which are arranged in
levels, indicating the degree to which a standard has been met. Criteria define
descriptors of ability at each level of performance. Levels referred to are proficiency
levels which describe a continuum from excellent to unacceptable.

Learning to create rubrics is like learning anything valuable. It takes an initial time
investment. Once the task becomes second nature, it actually saves time while
creating higher quality student outcomes. The following questions will help you get
started:

6

Determine the concepts to be taught. What are the essential learning
objectives?

Choose the criteria to be evaluated. Criteria are the dimensions along which
you will judge how well a student has carried out the assessment task. Name
the evidence to be produced and indicate relevant weighting. Weighted rubrics
are useful for explicitly describing to students what concepts take priority over
others.

Develop a grid, which is the physical layout of the rubric. Plug in the concepts
and criteria.

Share the rubric with students before they begin the assessment task.

Evaluate the end product. Compare individual students' work with the rubric
to determine whether they have mastered the content.

A rubric template is shown in appendix 2. Two samples are also provided (see
appendices 3 and 4).

The following site provides a means of building your rubric: http://landmark-
project.com/classweb/tools/rubric_builder.php3 [Accessed 2 November 2004]

The following site requires you to log on first prior to allowing you to build a rubric:
http://rubistar.4teachers.org/index.php [Accessed 2 November 2004]

Summary

It is important that you should aim to fully align your units objectives, teaching
methods, assessment procedures, and educational climate. For this paper it is
important to remember the distinction between criterion-referenced assessment and
norm-referenced assessment as types of assessment. In addition, and reinforcing the
work of the two previous training packages, you should aim to describe all assessment
tasks. This means you should:

describe what learning objectives will be measured by the particular
assessment task;
provide students with clear instructions about the assessment tasks;
describe the conditions under which the students will take the assessment
(such as when, where and how); and
provide the scoring criteria (such as a rubric) that will be used.

Further information about how to develop criterion-referenced assessment can be
obtained from the University of Queenslands Teaching and Educational
Development Unit (TEDI), which has some guidelines and templates on how to
implement criterion-referenced assessment.


7
References

Biggs, J. (2003). Teaching for quality learning at university : What the student does.
Second Edition, Society for Research into Higher Education : Open University Press,
Buckingham.

Centre for the Study of Higher Education. 2002. Assessing Learning in Australian
Universities: Ideas, strategies and resources for quality in student assessment, A
national project by Richard James, Craig McInnis and Marcia Devlin,
www.cshe.unimelb.edu.au/assessinglearning [accessed 8th October 2004]

Centre for Teaching and Learning, University College Dublin, Good practices in
student assessment, http://www.ucd.ie/teaching/assess/as6.htm [Accessed 20 October
2004]

Department of Education, Science and Training. 2004. Fact Sheet: Promoting
Excellence in Learning and Teaching, March

Family Education Network. Creating Rubrics,
http://www.teachervision.fen.com/lesson-plans/lesson-4521.html?detoured=1
[Accessed 2 November 2004]

Illinois Online Network (ION),
http://www.ion.illinois.edu/IONresources/assessment/rubric.asp
[Accessed 11th October 2004]

Tognolini, J. 2001. Educational Testing Centre. University of New South Wales
http://tlcweb.np.edu.sg/lt/articles/assessmentTognolini.htm, [Accessed 28 October
2004]

University of Queensland, Teaching and Educational Development Unit,
http://www.tedi.uq.edu.au/Teaching/assessment/cra.html, [Accessed: 19 October
2004], http://www.tedi.uq.edu.au/Teaching/assessment/designing.html, [Accessed 19
October 2004]; http://www.decs.act.gov.au/bsss/critref.htm
[Accessed 20 October 2004]

University of Tasmania,
http://www.utas.edu.au/tl/supporting/assessment/judgement.html, [Accessed 20
October 2004]


8
Assistance

Assistance is available on the development of criterion-referenced assessment. My
contact details are:

Rob Connoley
Educational Developer/Lecturer in Teaching & Learning
Faculty of Business and Law
Deakin University
Waurn Ponds VIC 3217

Email: rob.connoley@deakin.edu.au

Phone: X724755
Rm: ib3.105


9
Appendix 1: Criteria for marking essays and sample
performance standards

Criteria Sample performance standards
1. Addressing the
question
The relevance of the content of the essay to the question or
title set.
Good essays select relevant material (knowledge,
concepts, interpretation, theoretical models, others
perspectives).
Better ones make it clear throughout how the material
is relevant to the question.
2. Using evidence



The use of externally sourced material, such as research
findings, facts, quotations, or other forms of information.
Good essays include information from outside sources
that backs up the points made in the essay.
Better ones explicitly highlight or interpret the
evidence to support a more general claim or idea or
point being made in the essay.
3. Developing
argument



The construction of a coherent and convincing set of
reasons for holding a particular point of view; the
following of an analytical path leading from a starting
point to a concluding point.
Good essays contain expressions of positions on the
issues raised by the essay.
Better ones develop arguments throughout the essay,
with each element building on the last.
4. Critical evaluation



Determining the value, significance, strengths and/or
weaknesses of something (e.g., research findings, theory,
methodological approach, policy, anothers argument or
interpretation).
Good essays contain evaluative assertions or
descriptive points about the strengths and weaknesses
of elements referred to in the essay.
Better ones contain systematic, reasoned explanations
for the evaluative points being made.
5. Structuring


The formal arrangement of the essay content into
paragraphs.
Good essays have clearly recognisable introductory
and concluding paragraphs, and paragraphs in the main
body of the essay each have a clear, single concept or
point as their main focus.
Better essays have a paragraph structure that supports
the development of ideas within the essay, so that the
structure of the essay is linked to the developing
argument.

10
6. Use of language



The use of words, grammar, and punctuation to formulate
an utterance appropriate to the purpose and context.
Good essays are free from errors in spelling,
punctuation and grammar, and would be acceptable
pieces of writing in the wider world.
Better essays adopt academic styles and conventions,
and approximate to the appropriate academic register.
Source:
http://www.assessmentplus.net/web%20version%20of%20staff%20workshop%20mat
erials/Core%20criteria%20descriptions%20(web).doc

[Accessed 2 November 2004]

11
Appendix 2: A rubric template

(Describe here the task or performance that this rubric is designed to evaluate.)
Beginning
1
Developing
2
Accomplished
3
Exemplary
4
Score

Criteria
(Name)

Description
of identifiable
performance
characteristics
reflecting a
beginning
level of
performance.
Description
of identifiable
performance
characteristics
reflecting
development
and
movement
toward
mastery of
performance.
Description of
identifiable
performance
characteristics
reflecting
mastery of
performance.
Description
of identifiable
performance
characteristics
reflecting the
highest level
of
performance.


Criteria
(Name)



Description
of identifiable
performance
characteristics
reflecting a
beginning
level of
performance.
Description
of identifiable
performance
characteristics
reflecting
development
and
movement
toward
mastery of
performance.
Description of
identifiable
performance
characteristics
reflecting
mastery of
performance.
Description
of identifiable
performance
characteristics
reflecting the
highest level
of
performance.


Criteria
(Name)



Description
of identifiable
performance
characteristics
reflecting a
beginning
level of
performance.
Description
of identifiable
performance
characteristics
reflecting
development
and
movement
toward
mastery of
performance.
Description of
identifiable
performance
characteristics
reflecting
mastery of
performance.
Description
of identifiable
performance
characteristics
reflecting the
highest level
of
performance.


Criteria
(Name)


Description
of identifiable
performance
characteristics
reflecting a
beginning
level of
performance.
Description
of identifiable
performance
characteristics
reflecting
development
and
movement
Description of
identifiable
performance
characteristics
reflecting
mastery of
performance.
Description
of identifiable
performance
characteristics
reflecting the
highest level
of
performance.


12
toward
mastery of
performance.

Criteria
(Name)


Description
of identifiable
performance
characteristics
reflecting a
beginning
level of
performance.
Description
of identifiable
performance
characteristics
reflecting
development
and
movement
toward
mastery of
performance.
Description of
identifiable
performance
characteristics
reflecting
mastery of
performance.
Description
of identifiable
performance
characteristics
reflecting the
highest level
of
performance.


Source: Adapted from :
http://edweb.sdsu.edu/triton/july/rubrics/Rubric_Template.html

[Accessed 2 November 2004]



13
Appendix 3: A sample rubric - Essay


Standard/Benchmark:

Effective Communicator: Students write for a variety of audiences and purposes (i.e.
narrative, persuasive, and informative forms) using well organises paragraphs with
adequate supporting evidence.

Performance Task:
Write an essay on an assigned topic or a topic of your choice.
Performance
Element
Excellent Good Average Poor Unacceptable
Idea
Development
(20 points)
Thesis is
present and
directly
supported
throughout
essay. (18-20
points)
Thesis is
present and
most points
supported. (16-
17 points)
Thesis is
present but
vague; support
may stray from
thesis. (14-15
points)
Thesis is vague
with little
support or
strays from
thesis. (12-13
points)
Lacks thesis or
organizing
idea. (0-11
points)
Elaboration
(50 points)
Well written,
fully elaborates
points
addressed with
clear, accurate,
and detailed
information to
support thesis.
(45 - 50 points)
Well written,
most points
elaborated with
clear and
detailed
information to
support thesis;
may contain
minor factual
errors. (40 - 44
points)
Adequately
written essay;
some points
elaborated;
may contain
factual errors
or irrelevant
information.
(35 - 39 points)
Awkward
writing style,
points are
general, factual
errors present,
and may stray
from thesis. (30
- 34 points)
Poor writing
style with little
or no specific
details, off
topic, factual
errors present.
(0 - 29 points)
Organization
(20 points)
Clearly
organized and
remains
focused. (18 -
20 points)
Good
organization
with few
statements out
of place. (16 -
17 points)
Organization
present but
awkward; may
contain several
statements out
of place,
problems
existing with
introduction or
conclusion;
lack of topic
sentences. (14 -
Organization is
confusing to
the reader.
Organization
may lack
proper
paragraph
construction,
transitions,
topic sentences,
organizing
support, or
Little or no
structure
present. Essay
rambles
aimlessly from
topic to topic or
is completely
off topic. (0 -
11 points)

14
15 points) relevant
information.
(12-13 points)
Mechanics
(10 points)
Possesses no
critical errors
interfering with
comprehension.
(9 - 10 points)
Possesses 1-2
critical errors
interfering with
comprehension.
(8 points)
Possesses 3-4
critical errors
interfering with
comprehension.
(7 points)
Possesses 5-6
critical errors
interfering with
comprehension.
(6 points)
Possesses 7 or
more errors
interfering with
comprehension.
(0 - 5 points)


Source: http://www.rubrics.com/rubrics/Essay.html
[Accessed 2 November 2004]



15
Appendix 4: A sample rubric interpersonal
communication skills

Standard/Benchmark:

Effective Communicator: Exhibits effective interpersonal
communication skills.

Performance Task:
Conduct a discussion with someone else on a topic of interest.
Performance
Element
Level 4 Level 3 Level 2 Level 1
Listening Is always
attentive to
others when
they speak.
Is attentive to
others when
they speak; is
occasionally
distracted by
others.
Is occasionally
inattentive to
others when
they speak;
distracts others'
listening.
Is
consistently
inattentive
to others
when they
speak for a
wide variety
of reasons.
Interruptions Never
interrupts
when others
are talking.
Interrupts
others with
their
permission;
interruption
helps delivery
of message.
Interrupts
others without
their
permission;
interruption
interferes with
delivery of
message and
may or may not
cause speaker
problems.
Is
consistently
disruptive to
others;
interruptions
interferes
with
delivery of
message and
clearly
causes
speaker
problems.
Eye Contact Faces and
maintains eye
contact with
those who are
speaking.
Faces and
maintain eye
contact with
those who are
speaking; eye
contact may be
distracting to
those who are
speaking.
Faces or
maintains eye
contact with
the speaker;
occasionally
loses or
removes eye
contact,
distracting the
speaker.
Does not
face or
maintain
eye contact
with the
speaker;
annoys
speaker.

16
Information
Seeking
Questions
Asks questions
when he or she
does not
understand
what is being
said; questions
are pertinent to
main ideas.
Asks questions
when he or she
does not
understand
what is being
said; some
questions may
not be pertinent
to main ideas.
Asks questions
when he or she
does not
understand
what is being
said; many
questions are
not pertinent
and do not
allow the
speaker to
effectively
communicate
main ideas.
Exhibits
little or no
interest in
information
being
shared; no
information
seeking
questions
are asked.
Voice Always speaks
in an
understandable
voice, using
clear tone,
enunciation,
and reasonable
pace; message
is clearly
received.
Speaks in a
voice which is
generally
understandable;
sometimes
tone,
enunciation or
pace interferes
with message;
adjusts delivery
when listener
does not
understand.
Speaks in a
voice which is
generally
understandable;
sometimes
tone,
enunciation or
pace interferes
with message;
needs listener
requests to
modify
delivery of
message.
Speaks in a
voice which
is frequently
difficult to
understand
due to poor
tone,
enunciation
or pace even
after listener
requests to
repeat
message.
Understandability Uses words,
terms, and
examples
which others
clearly
understand.
Uses
understandable
words and
terms, but
needs examples
to clarify major
ideas.
Uses confusing
words and
terms; uses
examples
which, to a
limited degree,
help listener
understand.
Uses
confusing
words,
terms,
and/or
examples
which
listener does
not
understand.
Empathy Puts self in
others' shoes;
accepts and
understands
the feelings
and
motivations of
others; takes
steps to
Puts self in
others' shoes;
accepts others'
feelings, but
does not
understand
others'
motivations; is
interested in
Accept others'
feelings; does
not understand
others'
motivations
and shows no
or little interest
in seeking to
understand
Finds it
difficult to
accept
others'
feelings;
does not
understand
others'
motivations;

17
deepen
understanding.
seeking deeper
understanding,
but does not
take active
steps.
others. shows no or
little interest
in others

Source: http://www.rubrics.com/rubrics/Interpersonal_Communication_Skills.html

[Accessed 2 November 2004]