M edical Teacher, Vol. 22, No.
2, 2000
AMEE GUIDE
AMEE Guide No. 18: Standard setting in student
assessment
MIRIAM FRIEDMAN BEN-DAVID
Centre for Medical Education, University of Dundee, UK
Licensure, credentialling and academ ic institutions
Med Teach Downloaded from informahealthcare.com by Universitetssykehuset Nord-Norge on 02/28/13
SUM M AR Y
are seeking new innovative approa ches to the assessm ent of profes-
sional competence. Central to these recent initiatives is the need to
determine standards of perform ance, which separate the com petent
from the non-com petent candidate. Setting standards for perform -
ance assessm ent is a relatively new area of study. Consequently,
there is no one recom mended approa ch to setting standards. The
goal of this guide is to fam iliarize the reader w ith the fram ework,
pr inciples, key concepts and practical considerations of standard
setting approa ches and to enable the reader to m ake `educated’
choices in selecting the most appropr iate standard setting approach
for their testing needs.
For personal use only.
W hy should we use standard-setti ng procedures?
In designing assessm ent tasks, test developers incorporate
meaningful and essential perform ance criteria designed to
provide evidence that candidates have successfully completed
the task. Ideally, candidates should demonstrate mastery of
com petence by responding correctly to the task criteria and
by achieving the maximum scoring points. However, in
reality, candidates may demonstrate a variety of perform-
ance pro® les that m ay range from non-competent through
minimally competent to fully com petent. A requirement of
mastery approach to performance (fully com petent) for
passing the test may appear unrealistic in most situations,
owing to the complex nature of a medical task and measure-
m e nt er ro r. C onsequ en tly, te st de velo per s nee d an
educational tool by which they determine the cut-off point
on the scoring scale that separates the non-competent from
the competent. The traditional approaches in de® ning such
a cut-off point (i.e. responding correctly to 70% of the
items) do not provide robust and valid evidence for pass/fail
decisions. The problem intensi® es when the test results hold
serious promotion or career consequences for the candidates.
These are the questions the standard-setting guide will
address by providing both theoretical and practical guidelines
to the reader.
The goal of this guide
Historically, standard-setting methods were most commonly
employed on multiple-choice and other written examinations
(Hambleton, 1995; Jaeger et al., 1996). During the last decade,
with the emergence of performance assessment methods in
general, and credential examinations in the professions in
particular, there was a need to re-examine the ® tness of existing
120 ISSN 0142-159X (print)/ISSN 1466-187X (online)/00/020120-11 ½
methods of standard setting for the complex nature of perform-
ance assessment. A number of standard-setting approaches
are currently available for both written and performance tests.
Information through recent research studies may assist the
reader to formulate an `educated’ choice of a particular
method. Some assessment methods or procedures may better
® t one standard-setting approach vs. another. Therefore, the
present guide will describe a number of approaches and will
focus in detail on one commonly used approach, the `M odi-
® ed Angoff’ , as an example and a guide for a standard-setting
framework. The principles, key concepts and the practical
considerations outlined in this guide may serve as common
consideration for most standard-setting approaches. Whilst
setting standards on a written examination is essential for
good testing practice, owing to rapid developments in testing
of professionals, special attention will be given in this guide to
performance assessment. The practical example will outline a
process of setting standards for performance assessment.
However, it is important that the reader distinguishes between
the main characteristics of written examination and perform-
ance assessment. A section of this guide will highlight some
questions concerning the differences between the two modes
of assessment, with regard to standard-setting procedures.
The m edical education ® eld will bene® t greatly from
ne w in novative ap pro ac hes to setting sta nd ards fo r
performance-assessment m easures. It is hoped that the
present guide will stimulate the reader to explore new ways
of setting standards for minimally competent candidates as
well as for fully competent candidates.
K ey concepts
T he goal of this guide
To fam iliarize th e reader with th e fram ewo rk, th e
principles, key concepts and practical considerations of
standard setting approaches and to enable the reader to
make `educated’ choices in selecting the most appropriate
standard setting approach for their testing needs.
Correspondence: Dr M iriam Friedman Ben-David, C entre for M edical Educa-
tion, U niversity of Dundee, Tay Park H ouse, 484 Perth Road, Dundee D D2
1LR , UK . Tel: + 4 4 (0 )1 382 6 319 76; f ax: + 44 (0 )13 82 645748; em ail:
mfbendavid@ aol.com
2000 Taylor & Francis Ltd

Norm-referenced vs. criterion-referenced standards
In credentialling examinations where the stakes are high,
standard-setting approaches may differ in their orientation of
norm referenced vs. criterion referenced. Credentialling
organizations may use norm-referenced orientation, in which
the standard is based on performance of an external large
representative sam ple (norm group) equivalent to th e
candidates taking the test. The criterion-reference orientation
links the standard to the content of the competence level
under consideration. The norm-referenced standard will be
somewhat unstable and will shift according to the perform-
ance of the norm group, as large as it may be. The criterion-
referenced standard is a ® xed standard that may undergo
periodic re-evaluation in view of shifts or trends in candidates’
performance over time (Nungester et al ., 1991). The norm-
referenced approach, employing a group-referenced standard,
Med Teach Downloaded from informahealthcare.com by Universitetssykehuset Nord-Norge on 02/28/13
may result in reasonable standards providing the group is
representative of the candidates’ population, is heterogeneous
and is large. Shift of the standard over time is a concern.
At the school level, relative (norm) vs. absolute (criterion)
standards are also considered (Muijtjens et al ., 1998). A
relative standard can be set at the m ean performances of the
candidates, or by de® ning the units of standard deviation
from the mean. Percentile ranks, as well as the m edian
point, could also be set as norm-referenced standards. These
standards may vary from year to year due to shifts in the
ability of the g roup and m ay result in a ® xed annual
For personal use only.
percentage of failing students, if the scores maintain a normal
distribution across administrations. An absolute standard
stays the sam e over multiple administrations relative to the
content speci® cations of the test. The failure rate m ay vary
due to changes in the group’s ability, from one administra-
tion to the other.
Nedelsky stated:
The passing score is to be based on the instructor’ s
ju dg e m e nt of what con stitutes an ad equ ate
achievement on the part of the student and not on
the perform ance by the student relative to his/her
class or to any other particular group of students.
In th at sen se the stan dard to be used for
determining the passing score is absolute. (Nedel-
sky, 1954, p. 3)
A standard is de® ned as absolute ª if it can be stated in
term s of the knowledge and skills a student must possess in
order to pass the courseº (Nedelsky, 1954, p.4).
Com pensator y vs. conjunctive standards
Weaknesse s of relative/nor m -referen ced standards
x Standards are not content related
x A ® xed number of candidates may fail each year
x Examinees’ ability may in¯ uence the standard
x Standard is not known in advance
x Diagnostic feedback relative to performance is unclear
An example of an objective structured clinical examination
(OSCE) will help clarify the difference between compensa-
tor y and conjunctive standards. Let us assum e we are
required to set standards on a 15 stations standardized
AM EE G uide No. 18
patients examination. Each station measures three skills:
history, physical exam ination and com munication. Each
station contains separate checklist items (criteria) for each
skill component. Let us further assume that each station
varies signi® cantly in difficulty in relation to history and
physical exam ination scores. However, perform ance on
comm unication skills does not vary substantially from one
station to anothe r, and a con stant difficu lty level is
maintained for all stations.
In any given station, all candidates’ positive responses
per skill are summarized and divided by the highest possible
score and multiplied by 100 to produce a station skill
percentage score. The total test score is calculated by
averaging the station scores across the 15 stations. If the
standard is set on the total test score, which is the average
performance across stations, the skill standard will constitute
a com pensatory standard . This method of scoring permits
candidates to compensate for relatively poor skill perform -
ance on som e stations (Norcini et al ., 1993). Any combina-
tion of performance (skill scores) across the stations is
acceptable, as long as the examinee exceeds the skill perform-
ance standard for the total test (Hambleton, 1995).
Test developers or decision makers may, however, insist
that a number of stations should be com petently m anaged
before a passing score is warranted. This m ethod, which
explicitly de® nes the num ber of stations that m ust be
successfully managed in order to pass the test, constitutes a
conjunctive standard . Sim ilarly, test developers may decide
that candidates must exceed separately each skill standard
(history, physical examination and communication) at the
total test level. This approach will also constitute a conjunc-
tive standard, which does not allow candidates to compensate
for relatively poor performance on one or more skills. For
exam ple, poor perform ance on history should not be
compensated by high performance on physical examination
and vice versa. However, multiple conjunctive standards
may produce multiple failures. In a long OSCE examina-
tion, even the better students may fail a station because of
measurement error (Hambleton, 1995). In an OSCE a single
station score is considered an unreliable score, mostly as a
result of case speci® city and sampling error (C.P.M . Van
der Vleuten, G.R. Norman & E. De Graaf, Pitfalls in the
Pursuit of O bjectivity, personal com munication, 1990).
Acceptable test reliabilities are attained with careful choice
of optimal number of stations, which will allow generaliza-
tion to other clinical cases. Standards should be set on reli-
able scores to avoid problems of decisions inconsistencies
and candidates’ misclassi® cations (Jaeger et al ., 1996). Thus,
conjunctive standards set on scores with low reliability at
the station or test level may result in wrong pass/fail deci-
sions due to m easurem ent error. Issues of resitting the
examinations due to m ultiple failures and the logistics and
cost involved sh ould be considered carefully, prior to
selecting a conjunctive approach. The degree of relationship
am ong the test skill components m ay further assist in
conjunctive vs. compensatory decisions. The higher the
correlation among the test components the greater the
inclinations towards a com pensatory standard. Highly
correlated test components may imply a common construct
or dimension of perform ance, in which performance on one
component impacts on performance on the other. The
121
 M . Fr iedman B en-D avid
aggregation of related com ponent scores into one dimen-
sion score is the basis for a compensatory pass/fail standard.
If there is no or little relationship among test components
the conjunctive methods could be justi® ed.The com munica-
tion skill in this example m ay call for a compensatory
standard to be set for the total test and not separately for
each station since skill scores do not vary signi® cantly from
station to station. However, the history and physical examina-
tion standards will be set separately for each station and it is
the decision of the test developers whether the total test
standard for each skill will be conjunctive or com pensatory.
For example, in an OSC E where history and physical
examination test component scores are highly correlated,
both com ponents m ay be regarded as one dimension of
data-gath ering skills. Setting standards on a com bined score
of data gathering constitutes a compensatory approach by
Med Teach Downloaded from informahealthcare.com by Universitetssykehuset Nord-Norge on 02/28/13
which an examinee m ay com pensate for poor physical
exam ination skills with good history skills. In another
example the two skills components may be uncorrelated but
a combined score presents higher reliability compared with
the two unreliable, separate component scores. In such a
case psychometricians will explore issues of misclassi® ca-
tion in the aggregated approach (compensatory) vs. the
separated com ponent approach (conjunctive) (Jaeger et al .,
1996). A conjunctive standard allows diagnostic feedback
to candidates, since each skill component is considered
separately. In the compensatory approach information about
test com ponents is aggregated by setting one standard.
For personal use only.
Consequently, perfor m ance feedback on sepa rate test
com ponents may not be available.
Standard-setti ng m ethods
C onjun ctive standard considerations
x Reliability of the test components at the station or
skills level might be low
x Long tests may produce multiple failures at the station
level
x Standards on multiple-test components may also result
in multiple failures
Com pensator y standard considerations
x What are the consequences for compensating for poor
performance?
x Loss of diagnostic feedback
Test-centred m odels
Jaeger (1989) refers to test-centred models in which the
judges set standards by reviewing the test items and provide
judgements as to the `just adequate’ level of performance
on these item s. The following descriptions of the test-
centred m ethod accentuate the different ways for judges to
provide their ratings. The Angoff (1971) model employs a
test-centred approach and is known for its wide use in
educational testing and performance assessment.The present
guide will use the Angoff approach as a case study for
setting standards.
Ebel’ s (1972) approach requests judges to categorize the
122
item s in a test employing a num ber of categories, for
example: three levels of difficulty and four levels of relevance
to the decision to be m ade. In a m odi® ed Ebel procedure,
judges categ orize item s as `E ssential’ , `Im por tant’ or
`Indicated’ (Case & Swansen, 1998). After classifying the
items into each category, judges then decide on the propor-
tion of item s in each category that a hypothetical group of
borderline exam inees could resp ond to correctly. The
number of items in each category is multiplied by the propor-
tion of item s in that category that would be answered
correctly by a borderline candidate, and these products are
summed to produce a test standard.
The Nedelsky (1957) approach was originally designed
for m ultiple-choice items. For each item, the judges decide
on how m any of the distractors (response options) a
minimally competent exam inee would recognize as being
incorrect.
The m inimal pass level for the item is computed as one
over the number of distractors (options) rem aining after the
incorrect options are rem oved. For example, on a ® ve-
option item , if a judge decides that the minim ally com petent
exam inee will recognize two options as being incorrect, the
minimal pass level (MPL) for this item will be 0.33, since
three options remain after the two incorrect options were
removed. The M PL is then summ ed over all items in the
test to produce a passing score for the total test.
Jaeger’s method (1982, 1989) differs from most other
test-centred methods in a number of important ways (Kane,
1994), all of which tend to emphasize the role of standard
setting as the development of policy rather than as a technical
approach to estimating a passing score. Jaeger (1989)
emphasizes the importance of recognizing the need to
ª sample all populations that have a legitimate interest in the
outcom es of com petency testingº . The political aspects
involved in many testing contexts is emphasized as well as
the need to incorporate m ultiple points of view. Most of the
other m ethods employ a single panel of expert judges. The
focus in Jaeger’ s method is on the passing examinees rather
than on the borderline or the minimally com petent. Judges
are asked to rate `yes’ or `no’ for each item, whether each
exam inee who passes the test should be able to answer the
item correctly. This procedure is employed within a number
of expert panel groups. Each panel will receive input from
the other panel’ s ratings and will be presented with actual
performance data. Judges are then asked to provide second
ratings or more, while reconsidering their judgem ents in
view of the new information. In such iterative procedures,
considerations of the validity of the standards are built into
the process (Kane, 1994).
Examinee-centred m odels
In the examinee-centred approaches panellists m ake pass/
fail decisions by identifying a point on the score scale that
would be most consistent with the test purpose (Kane,
1994).
In the Borderline-Group method (Livingston & Zieky,
1982), the judges may use different approaches to identify
an actual (not hypothetical) borderline group. The level of
achievem ent of this group is around th e perform ance
standard. The judges may use their experience or other
methods of assessm ent to identify the group. The median
score for this group could be used as the passing score. The

scores of the borderline group should cluster together to
produce a reasonable standard. If the scores are spread, the
method m ay not be applicable.
In the Contrasts by Group approach (Livingston & Zieky,
1982), the panellists sort the examinees into two groups:
competent and not competent. The judgement is based on
characteristics of the examinees relative to the task other
than the test scores (i.e. the test scores are not known to the
panellist during the sorting process). After the sorting is
completed, the score distributions for the competent and
not competent groups are plotted. Comm only, the point of
intersection of the two distributions could be considered as
the passing score (Clauser et al. , 1996; Burrows et al ., 1999).
However, the overlapping areas of the distribution would
indicate the degree of separation between the two groups.
The larger the overlapping areas, the smaller the separation
Med Teach Downloaded from informahealthcare.com by Universitetssykehuset Nord-Norge on 02/28/13
between the two groups. Ideally the chosen passing score
should maximize discrimination between the competent and
the non-competent exam inees, thus the distributions should
indicate minimal overlap.
Relative/absolute com prom ise standards: the H ofstee m ethod
This is a standard-setting approach that incorporates the
advantages of both relative and absolute standard setting
procedures (de Gruijter, 1985). Panellists review the test
materials and are asked to provide four values:
For personal use only.
(1) lowe st ac cepta ble percenta ge of failing e xa m inee
(minimum failure rate);
(2) hig hest acceptable percentage of failing exa m inee
(maximum failure rate);
(3) lowest score which allows a candidate to pass (minimum
passing point);
(4) highest score required for a candidate to pass (maximum
passing point).
The median values of the group of judges are plotted for
each value.
In Figure 1, the m inim um failure rate is 0, the maximum
failure rate is 20%, the m inimum passing point is 50%, and
the m aximum passing point is 60%. The (actual perform -
ance) test scores curve is plotted, which indicates the failure
rate as a function of the passing score. The intersection of
test scores curve with the diagonal line drawn from upper
left to lower right is the cut-off point (just above 55%
correct).
F igure 1.
AM EE G uide No. 18
M odi® ed Angoff
The Angoff standard-setting approach is one of the most
widely used in medicine (Cizek, 1996). In its purest form the
Angoff method is a judgemental approach in which a group of
expert judges makes estimates about how borderline candidates
would perform on items in the examination (Livingston &
Zieky, 1982; Berk, 1986), i.e. the proportion of borderline
examinees who will answer an item correctly.This is equivalent
to estimating the candidate’s likelihood of answering a number
of items correctly (Sizmur, 1997). Estimates are averaged
over judges and summed over items to create a standard
(cut-off score). A modi® cation adds to the process by the
provision of data about difficulty of the items, based on actual
performance data. The focus on the borderline group should
enhance maximum discrimination at the borderline level.
The Angoff method depends on expert judges’ (panel-
lists’ ) degree of fam iliarity with the minimally competent
test-taker (the hypoth etically borderline group), who is
neither quali® ed nor unquali® ed to pass the test (Norcini et
al ., 1993). The panellists are asked to m ake judgem ents
about that candidate’s likelihood to respond correctly to
each of the test items. This task is highly dependent on the
capacity of the panellists to understand the m eaning of a
borderline candidate’s characteristics. The panellist also has
to understand how the borderline candidate will respond to
the required performance task. The panellist must also be
fam iliar with the expected level of examinee perform ance as
de® ned by the purpose of the test, and be able to understand
the degree of difficulty of the task. Research with judges
showed that the standards that they demanded in theory
were extremely stringent and different from those they actu-
ally used in practice. In general, judges have a tendency to
produce very high standards (Wolf & Silver, 1986).
This guide for standard-setting approaches will focus on
one m odi® ed version of the Angoff approach. In reviewing
the scoring sheet of an O SCE station, panellists will be
asked to deter m ine the num ber of scoring points an
individual borderline candidate will answer correctly in order
to pass the station. This process will be done separately for
each of the station’s skill components: history and physical
ex am ination. A separate process will be outline d for
comm unication skills.
M odi® ed A ngoff
x Test-centred approach
x Criterion referenced
x Relies on expert (panellist) judgements
x Requires understanding of borderline characteristics
x Actual performance data are provided to panellists as
an additional information source
Selection of panellists
The importance of panellist selection for the creation of a
d efen sible stan dard -setting proc ess can not be ove r-
em phasized. H am bleton (1995) describes a defensible
process which is based on: selection of appropriate panel-
lists, excellent panellist training and a well-planned and
systematic standard-setting process, which provides am ple
123
 M . Fr iedman B en-D avid
opportunity for discussion and deliberations among panel-
lists. The reliability of the performance standard will be
re¯ ected by the degree of agreement among panellists and
a reasonab le perfor m ance standard will coincide with
expectations about the speci® ed professional performance.
M essick (1989, p. 15) refers to test validity as the ª infer-
ences and actions based on test scoresº . The standard-
setting procedures are in fact enhancing the validity of the
test by establishing , through a defensible process, the
meaning and inferences that we m ake on the basis of test
scores (W iliam , 1996). Thus, standard-setting procedures
lie at the very heart of the validity of the test. A defensible
approach to standard-setting procedures is essential to
support the appropriateness of standard inferences and the
consequential decisions based on those inferences.
Jaeger (1991) identi® es the quali® cations of panellists.
Med Teach Downloaded from informahealthcare.com by Universitetssykehuset Nord-Norge on 02/28/13
Among other things, they should be experts excelling in
their specialization, able to conceptualize and problem-
solve in their specialty and able to employ self-monitoring
skills. Jaeger’ s criteria for panellist selection may further be
developed to include quali® cations relevant to test speci® ca-
tions, purpose of the test and standard-setting methods
employed. For example, in setting standards for a 15 station
OSCE examination for fourth-year medical students who
are required to pass the examination for graduation quali® ca-
tions, ad dition al pan ellist charac ter istics sh ould be
considered. Panellists must be fam iliar with performance of
fourth-year students, should be familiar with the OSCE
For personal use only.
method of assessment, should have experience as clinical
teachers and should have knowledge of the next phase of
training the graduates are expected to enter.
The developers of tests m ust create their own list of
panellists’ characteristics, which will speci® cally address the
purpose of the examination. In a high-sta kes exam ination,
panellists who are recognized for contribution in the ® eld
may be appropriate. In addition, an adequate mix of gender,
age, seniority, academic vs. community experiences and other
relevant attributes will further establish the defensibility of
the standard-setting process. M ultiple panellist groups for
rating the same stations will further add to the defensibility
of the process by demonstrating consistency of standards
across occasions and across panellists. The larger the panel-
list sample the m ore stable will be the resulting standard
(Jaeger et al ., 1996). However, one needs to consider how
manageable a large size group would be. Jaeger et al. (1996)
propose an estimation method for panellist group size by
considering the standard error of measurement of the assess-
ment tool, and the standard deviation of the panellist ratings.
Panellist s sho uld be:
x Experts in the related ® eld of examination
x Fam iliar with the examination m ethods
x Good problem solvers
x Fam iliar with level of candidates
x Interested in education (teachers)
W ritten vs. perform ance standards
M ost of the standard-setting procedures employed for
performance assessment were ® rst applied to written tests.
124
Since multiple-choice examinations are the most commonly
used form of written tests, it may be helpful to consider
differences between multiple-choice examinations (MCQ) as
a form of written test and an OSCE as a form of performance
test. Let us assume that the MCQ examination contains 400
items, sampled across domains. The OSCE test contains 15
standardized patient-based stations sampled across speciality
areas. Items on both forms of assessment are either a multiple-
choice question or an OSCE checklist criterion.The following
statements highlight the differences between the two forms of
assessment and questions are raised for further consideration
with regard to the modi® ed Angoff approach.
(1) The traditional M CQ items are written independently
of each other. In contrast, a station item is dependent
on the other items in the checklist.Thus, panellist ratings
for MCQs are provided separately for each item, whereas
in the OSCE example items are either rated independ-
ently or as `sets’ .
Q uestion:
In which way does the rating of individual items vs. sets
of items impact on the estimated standard?
(2) While estimating the difficulty of an MCQ item, panel-
lists consider the likelihood of candidates getting an
isolated know ledge com ponent right. In the O SCE
example the estimation is applied to a `set’ of items which
represent a `dimension’ of professional competence.
Q uestion :
Are the two estimation procedures similar? Which estima-
tion task is more difficult? Does the multidim ensional
aspect of an OSCE station violate the unidim ensional
assumptions that underlie standard setting procedures?
(3) The large number of M CQ items (400) challenges the
generalizability of the num ber of OSCE stations and
the broad nature of the tests.The complexity of perform-
ance tasks, in terms of resources and time, limits the
test to a m anageable size.
Q uestion:
Does the standard constitute a generalizable pass/fail
decision?
(4) In multiple-choice questions the comm on response is
one correct answer. In an OSCE station the response
m ay represent different degrees of success on a scale
continuum.
Q uestion:
Does the graded response key introduce further difficul-
ties in setting standards on performance tasks?
The above are only a few differences between M CQs and
OSCE items. Since traditional standard-setting m ethods

are applied to performance assessment, the differences
should be considered for the establishm ent of a valid
stan dard.
E ducational b ene® ts of standard setting
Faculty developm ent
Standard-setting procedures could be employed as a form
of faculty developm ent. The use of orientation materials
such as videos, test forms, performance data etc. allows
faculty to experience ® rst hand information on candidates’
perform ance on the task. In some instances, faculty may
observe for the ® rst time videoed perform ance of students,
containing criteria for acceptable performance. They may
consider the information as feedback for their own clinical
teaching. They may com pare poor and excellent candidates’
Med Teach Downloaded from informahealthcare.com by Universitetssykehuset Nord-Norge on 02/28/13
perform ance to their expectations. Som e may not `believe’
that the poor performer is actually enrolled in their medical
school.The standard-setting process provides a reality check,
which is not available otherwise owing to the low incidence
of direct clinical observation in medical schools. They may
identify the existence of persisting weaknesses, for exam ple
bedside m anner. It is not uncommon to hear faculty say, ª I
am going back to the department to change the way I teach
bedside mannerº .
Quality control of test m aterials
For personal use only.
The process of exposing faculty to test m aterials, scoring
policy and pro® les of scored performance constitutes a
scrutinized quality-control procedure. Panellists in the
process of reviewing test materials identify inappropriate
item s, which are either ambiguous or irrelevant. They may
® nd tasks for which the criteria are incomplete. As an external
group of experts, they will compare video performance with
the actual scoring results, and will comment on discrepan-
cies. Satisfying results of standard-setting procedures suggest
that test developers should submit test materials for which
they have 100% con® dence in its quality and its scoring
mechanism. Otherwise, they may lose panellists’ trust in the
test. In actual standard-setting procedures panellist are asked
to refrain from comments on the design of the station or the
scoring policy. However, mistakes and inconsistencies, if
present, create serious complications in the process of setting
stan dards.
Practical steps of the M odi® ed A ngoff A pproach
Step 1: General orientation
The facilitator presents to panellists a general overview of
the OSCE test, the duration of each station, test components,
scoring methods and any other information from which the
panellists may bene® t. Central to the orientation session is
a statement regarding the purpose of the test. In the O SCE
example, the purpose may read as follows:
The g raduates of this m edical school should
demonstrate m inimal competence in history, physical
examination and doctor/patient communication,
fo r th e purpo se of e nte rin g a su per v ised
Pre-Registration House Officer (PRHO ) training
programme in the UK!
AM EE G uide No. 18
The key words are `minimal competence’ and `supervised
programme in the UK’ . This statement of purpose will
guide panellists in their estimates as to what is expected in
the UK concerning PRHO entry level skills, as well as what
is expected of the medical school graduates at their exit
point. The minim ally competent concept and the continuing
supervised training may, for example, direct panellists to
consider conjunctive vs. compensatory standards. Knowing
that candidates are expected to continue with their training
in a supervised mode, and that poor perform ance in som e
areas will be further developed in the next phase of training,
allows a compensatory approach. Alternatively they may
consider that basic minimal competence in each component
is essential for better future practice and thus a conjunctive
approach is warranted.
In some instances, where faculty is not fam iliar with the
OSCE stations, a mini OSCE is set as part of the standard-
setting orientation procedure. Faculty may play the role of
examinees, or examiners, by observing each other. This is
done to avoid overestimation or underestimation of the
station difficulty. Their own level of performance as experts
may serve as a `ceiling’ effect for the standard-setting ratings.
However, panellists should be cautioned that as experts
they m ight perform short cuts, which will result in a lower
score for the station. The scores they achieved in the mini
OSCE could serve as good points of discussions regarding
the panellists’ level of perform ance compared with the
expected level of the candidate’s performance.
Step 2: Orientation to a `practice’ station
An example of the practice station is shown below, and the
corresponding checklists are shown in Figures 2 and 3. The
practice station is divided into two stations, history and
physical. In each station an examiner observes the candidate
1
and rates his/her performance on the checklist items.
Instructions for C andidates
H istor y taking
M r/M rs J is a patient who com plains of easy bruising.
You have four and one half m inu tes to take relevant
history from her.You will be asked to present a sum mary
of your ® ndings at the next station, including a working
diagnosis.
Instructions for C andidates
Physical exam ination
At this station there is an observer and a patient. The
patient has arterial peripheral vascular disease. Carry
out an exam ination of the circulation of the lower
ex trem ities of the patient. W hile doing this, please
explain to the observer what you are doing.
Test developers present the `practice’ stations to panellists.
The practice stations materials are discussed. At the end of
the presentation, panellists are asked to make judgements as
to how m any item s should be answered correctly by the
borderline candidate in order to pass the `practice’ stations.
125
 M . Fr iedman B en-D avid

E xam iner’s checklist

Please ask the student for an identity label to place on the checklist

Using the marking schedule given below, allocate a mark by (± ) in the appropriate column to the
right of the sheet as carried out by the candidate.
0+ Not done / ½= Done poorly/1= Done satisfactorily
Explanation to patient 0 ½ 1
Position patient on couch 0 ½ 1
Inspection for evidence of skin changesÐ pallor 0 ½ 1
Inspection for evidence of skin changesÐ hair loss, dryness etc 0 ½ 1
Palpation for temperature changes 0 ½ 1
Palpation of pulses
Femoral 0 ½ 1
Popliteal 0 ½ 1
½
Med Teach Downloaded from informahealthcare.com by Universitetssykehuset Nord-Norge on 02/28/13

Posterior tibial 0 1
Dorsalis pedis 0 ½ 1
Capillary re® ll (time) 0 ½ 1
M ax 10 m arks

Figure 2. Practice station: History checklist.

E xam iners’ checklis t

Please ask the student for an identity label to place on the checklist
For personal use only.

Students have been given the following instructions

M r/M rs J is a patient who complains of easy bruising. You have four and a half m inutes to take a
relevant history. You will be asked to present a summary of your ® ndings at the next station, including
a working diagnosis.
Please ask the student to summarize the history he/she has taken at the last station.
Note: prompting halves the m arks.
Use the marking schedule given below and tick the item in the appropriate box.
Not done Prom pted N ot prom pted

Name and age of patient 0 ½ 1

Duration of bruising 0 1 2
Site of bruising 0 ½ 1
Precipitating factors (trauma/spontaneous) 0 1 2
Other bleeding sites (guts, urine, joints, menorrhagia) 0 1 2
Severity of bleeding 0 ½ 1
Previous operations 0 ½ 1
Previous dental extraction’ 0 ½ 1
Family history 0 1 2
Drugs aspirin 0 ½ 1
Anticoagulants 0 ½ 1
Other 0 ½ 1
Sequence 0 1 2
Working diagnosis 0 1 2

F igure 3. Practice station: Physical exam ination checklist.

The `practice’ orientation materials may include:
(1) a full description of the stations;
(2) history and physical examination checklists (Figures 2
and 3);
(3) videotapes of one low performer and one high performer
for the practice stations;
(4) the panellist will have a blank checklist for the two
component skills while viewing the video;
(5) the actual skill score will be presented to the panel-
lists following the com pletion of each video perform -
ance.
The low vs. the high performers assist the panellist to form
a range of possible performance pro® les for the practice
stations. The orientation may contain other com ponents
according to the nature of the test. A careful process of
orientation may assist the panellists in arriving at `realistic’

126
 AM EE G uide No. 18

judgem ents, and further facilitates a defensible standard-
setting process.
Step 3: Characteristics of borderline candidates
The importance of understanding the characteristics of
borderline candidates has already been emphasized. In the
m odi® ed A ng off approach th e bo rd erline c andid ate s
constitute a hypothetical group. Therefore, it is essential to
introduce a process by which the panellists will increase
their understanding of the borderline characteristics.
This process becomes more meaningful following the
orientation steps in which panellists viewed performance
pro® les as they understand the purpose of the test and are
familiar with the test m aterials. During this step, a facili-
tator poses the question to the panellists:
Med Teach Downloaded from informahealthcare.com by Universitetssykehuset Nord-Norge on 02/28/13
rating form will be used for the entire standard-setting
procedure. Two separate forms will be distributed, namely
(1) history and (2) physical exam ination. Each form will
specify in the left-hand column the alphabetical order of the
stations to be rated. The next column will outline the
maximum number of points available for the station. The
next two columns are provided for the ® rst and second
ratings.
In the ® rst ratings, panellists are asked to enter their
individual judgement as to how many item s will be answered
correctly by a borderline exam inee in order to pass the
history skill component in this station. (Note: it is recom -
mended that the number of items will constitute the ratings
and not percentages of items. Experience has shown that
panellists m ay be in¯ uenced by traditional concepts of
acceptable standards expressed in percentages [for example,

Please indicate the characteristics of borderline
candidates that are relevant to the skills measured
in this test. These are fourth-year medical students
who are neither unquali® ed nor quali® ed to pass
the test. These candidates’ scores lie around the
estimated cut-off point. It is important that we
characterize the group in order to estimate and
make judgements about their ability to demonstrate
minimal competence in the practice station.
Panellists may then write independently the characteristics
of the borderline group per skill component. The panellists’
statements are then posted and the facilitator discusses with
For personal use only.
70% vs. 50%]. Thus, it is m uch `safer’ to count the items,
and provide raw num bers as estim ates.) Following the
completion of the ® rst rating, the facilitator presents all
panellists’ ratings on the board by assigning a num ber to
each panellist.The panellists discuss their ratings. The facili-
tator encourages panellists with the highest and lowest ratings
to re¯ ect on their judgements. The facilitator will average
the ratings of the panellists to produce a cut-off raw score
for history in the practice station.
Provision of actual perform ance data

A history score distribution for the practice station is

the panellists each statem ent. Argum ents and disagree-
presented to panellists (Figure 5). The score distribution is
ments are clari® ed and the group reaches a consensus as to
generated from previous adm inistrations of the station to a
what wou ld be an appro pr iate list of borderline
sim ilar c ohor t of four th -year m ed ical stud en ts. The
characteristics per skill component. This process is essential
characteristics of the cohort such as gender, year, purpose
to facilitate further the estimates and judgements of the
of administration etc. should be presented to the panellists.
panellists and to support the defensibility of the process.
Knowledge of the similarities or differences between the
cohort and the potential candidates allows panellists to
Step 4: Panellists provide ratings
evaluate the appropriateness of the actual performance data.
A rating form is distributed to panellists (Figure 4). The The distribution outlines the cum ulative num bers and

Panellist

History

Station Max. pts 1st ratings 2nd ratings

A (Practice) 20 Ð Ð Ð Ð Ð Ð Ð Ð Ð Ð
B 28 Ð Ð Ð Ð Ð Ð Ð Ð Ð Ð
C 28 Ð Ð Ð Ð Ð Ð Ð Ð Ð Ð
D 30 Ð Ð Ð Ð Ð Ð Ð Ð Ð Ð
E 28 Ð Ð Ð Ð Ð Ð Ð Ð Ð Ð
F 34 Ð Ð Ð Ð Ð Ð Ð Ð Ð Ð
G 31 Ð Ð Ð Ð Ð Ð Ð Ð Ð Ð
H 28 Ð Ð Ð Ð Ð Ð Ð Ð Ð Ð
I 28 Ð Ð Ð Ð Ð Ð Ð Ð Ð Ð
J 30 Ð Ð Ð Ð Ð Ð Ð Ð Ð Ð
K 28 Ð Ð Ð Ð Ð Ð Ð Ð Ð Ð
L 34 Ð Ð Ð Ð Ð Ð Ð Ð Ð Ð
M 31 Ð Ð Ð Ð Ð Ð Ð Ð Ð Ð
N 28 Ð Ð Ð Ð Ð Ð Ð Ð Ð Ð
O 28 Ð Ð Ð Ð Ð Ð Ð Ð Ð Ð
P 30 Ð Ð Ð Ð Ð Ð Ð Ð Ð Ð

Figure 4. Panellist rating form.

127
 M . Fr iedman B en-D avid

H istor y scores

Score No. Cum.No. % Cum .%

* 0 2 2 1.3 1.3
1 0 2 0.0 1.3
2 0 2 0.0 1.3
3 0 2 0.0 1.3
4 0 2 0.0 1.3
* 5 1 3 0.6 1.9
6 0 3 0.0 1.9
* 7 1 4 0.6 2.6
* 8 1 5 0.6 3.2
* 9 2 7 1.3 4.5
***** 10 7 14 4.5 9.1
***** 11 8 22 5.2 14.3
Med Teach Downloaded from informahealthcare.com by Universitetssykehuset Nord-Norge on 02/28/13

********* * 12 16 38 10.4 24.7

********* ** 13 17 55 11.0 35.7
********* ****** * 14 25 80 16.2 51.9
********* ******* * 15 26 106 16.9 68.8
******** 16 13 119 8.4 77.3
********* ***** 17 22 141 14.3 91.6
****** 18 10 151 6.5 98.1
** 19 3 154 1.9 100.0
20 0 154 0.0 100.0
Notes: Total number = 154. Scores are rounded to the next whole number

F igure 5. Practice station.

For personal use only.

percentages of students who got one item correct, two item s
correct, three items correct etc. The facilitator indicates the
percentages of students who might fail the history skill
com ponent of the station if the panellist average ratings are
applied to the distribution as a cut-off score. A discussion
should revolve around the issue of the `consequential data’ ,
i.e. percentage failure. Are panellists surprised? Should they
expect a lower or a higher num ber of failures? Does the
percentage failure seem to coincide with their experience?
The purpose of the actual perform ance data is to orient the
panellists to `realistic’ norm ative information. This provides
another source of inform ation, which helps panellists adjust
their ratings, when they are asked to attempt a second rating
on the form (see Figure 4).
As stated earlier, the higher the agreem ent among panel-
lists, the m ore reliable is the standard. Therefore, it is hoped
that in the second ratings, the panellists will adjust the
scores in view of their peers’ ratings and the actual perform-
ance data. It is also hoped that the standard deviation of the
ratings will decrease at the second ratings. Panellists provide
their second ratings, which are posted on the board by the
facilitator. A ® nal cut-off score is calculated by averaging all
the ratings. The same process is repeated for the physical
examination checklist with a separate rating form for physical
examination.
The same process is repeated for each station and its skill
com ponent in the test. This includes high and low video
perform ances, orientation to the checklist and to the scoring.
To maximize the outcome of the standard-setting procedures
it is possible to divide, after the orientation, a large group of
panellists (i.e. 18) into three groups of six each. The groups
will set standards on different stations but one or two stations
will be rated by all. This is done to exam ine the consistency
of ratings among panellist groups.
Ratings of comm unication skills
As stated earlier, the comm unication skill score did not vary
substantially from one station to the other. Therefore, setting
a communication standard by employing a holistic approach
may be appropriate for this particular situation. Panellists
do not consider a standard for each station, but rather set a
point of `how much is enough’ on the scoring scale for the
total test.
Orientation . Panellists are presented with test m aterials
relevant to the com munications com ponent of the test. This
includes the theoretical framework, the criteria, the training
materials for the assessors (in this case the standardized
patients) and the scoring scale. Panellists will view vide-
otapes of a range of perform ances (high, middle and low)
and the application of the scoring scheme to the perform-
an ce will be dem on strate d. T he bord erline g roup
characteristics are discussed and listed with relation to the
communication skills. Panellists will then be asked to state
what will be the point on the scale at which the borderline
candidate will demonstrate minimal competence for passing
a station with no reference to a particular station. Inherent
in this standard-setting approach is the com pensator y orienta-
tion. The stability or consistency of the communication
skills com ponent over stations (for example, Cronbach a
reliability 0.85± 0.90) justi® es such an approach. In this case
panellists are not concerned with com pensating poor
performance in one station with high perform ance on the
other since performance is generally stable over stations.
The ® rst ratings are presented and averaged across panel-
lists. Actual performance data are presented and discussion
revolves around the rate of failure relative to the established
standard. Panellists are then asked to provide a second
rating. The second rating will constitute the ® nal standard

128

for th e com m unication skills com ponent. C andidates’
communication scores for each station will be averaged to
produce a total test comm unication score. Candidates who
exceed the standard will pass the communication component
of the test.
E valuation
T he evaluat ion co m po ne nt of th e sta n dard -sett ing
procedures is important from a number of perspectives:
(1) To support the validity of panellists’ inferences and
actions by demonstrating a defensible process.
(2) To demonstrate the reliability of the ratings by indicating
a decrease in the variability of ratings. This means that
more consensus was gained through discussions of
ratings and feedback from the other panellists. During
Med Teach Downloaded from informahealthcare.com by Universitetssykehuset Nord-Norge on 02/28/13
the process of discussions, the panellists identify their
own stringent or lenient tendencies and m ay correct
their ratings in the iterative process of multiple ratings.
(3) To ® nd out the effect of the normative inform ation on
the panellists’ ratings and ensure that the criterion-
references approach did not shift to a norm-referenced
approach owing to high in¯ uence of the normative data.
(4) To ® nd out the effectiveness of the orientation material
and process for determining the extent to which the
panellists changed their previous conceptions as a result
of their orientation experience.
(5) To determine the extent to which the panellists were
For personal use only.
con® dent in the process and the resulting standards.
Panellists’ con® dence in the standard is another indica-
tion of the validity of the process. A broad representa-
tive sample of panellists constitutes collective expert
judgements for justifying a standard. This is another
aspect that supports the defensibility of the standard-
setting approach.
(6) To obtain feedback from panellists as to how the process
might be improved.This is an important aspect of evalu-
ation since standard-setting approaches in performance
assessment in the medical profession are still in an
evolutionary stage. Any feedback on the improvem ent
of the process may assist in the construction of improved
approaches to setting standards. As Linn et al. (1982)
state: ª Standard setting m ethods depend on human
judgements and there is no single perfect method of
setting themº .Therefore any improvement in the process
will maximize the bene® ts.
Evaluation materials should include data on the ® rst and
se cond ratings o f th e panellists for e ach of th e test
com ponents rated, which should demonstrate increased
consensus of raters. It should also include a questionnaire
adm inistered to panellists at the end of the standard-setting
process. Test developers should design the questionnaire
according to the test needs with consideration of the six
points stated above. In high-stakes examinations the aspect
of evaluation could not be overemphasized. However, even
in other settings such as medical school or courses the
defensibility of the process must be demonstrated through
the evaluation process.
C oncluding com m ents
· M uch work is still needed to establish effective standard-
setting procedures. Often, these procedures are chal-
AM EE G uide No. 18
lenged for their arbitrariness and for the unstable standards
over occasions, over panellists and over methods (Jaeger
et al., 1996). However, the growing need for establishing
standards of candidates’ performance generates innova-
tive and creative procedures for standard setting, mainly
for performance assessment.
· The length of procedures (for the O SCE it may take two
days of panellists’ work), should also be considered and
ways to shorten the process are needed.
· The internal process of standard setting m ay incorporate
a built-in mechanism to identify test com ponents that
`behave’ differently or may result in `odd’ standards. This
could serve as an opportunity to identify test materials
with `¯ aws’ . Comparison with the other test components
may assist in rectifying the problem (Clauser et al ., 1996).
· Further consideration m ust be given to fully compensa-
tory models (Clauser et al ., 1996; Burrows, 1999) in which
test items or components are averaged to produce a test
standard. Is the averaging an adequate procedure? Is the
total test standard a simple average of the test component
standards, or should it be a weighted average, or any
other form of aggregation?
· Obtained standards should be checked against other
information available on the test taker to ensure the validity
and the reasonableness of the standard.
· Effective methods of training panellists to recognize
borderline characteristics are essential if the Ango ff
approach is widely used.
· The more standard-setting procedures are applied to a
variety of tests, the more we will enhance the practice of
high-quality testing, and the higher will be the confidence
in the testing of professional competences.
· As Jaeger et al. (1996, p. 80) state: ª the state of the art, ( of
standard setting for performance assessm ent ), is far from a
state of grace. Much work remains to be done.º
N otes on contributor
M IR IAM F RIEDM AN B EN -D AVID , form erly Co-Director of the Clinical
Skills Certi® cation Program at the Educational Comm ission for
Foreign M edical Graduates, is currently a visiting professor at the
Centre for M edical Education at the U niversity of Dundee. H er
m ain area of research is the evaluation of physician com petences, in
undergraduate and postgraduate education.
N otes
[1] The practice case was drawn from the University of
Dundee Medical School Clinical Skills Centre O SCE
bank.
R efe rences
A NG OF F , W.H . (1971 ) Scales, norms, and equivalent scores, in: R.L.
T HOR NDIKE (E d.) Educational M easurement, 2nd edn, pp. 508± 600
(Washington, DC, American Council on Education).
B E RK , R .A . (1 986 ) C onsum ers’ g uide to se tting pe rform ance
standards on criterion reference tests, Review of Educational Research,
56, pp. 137± 172 .
B U RROW S , P.J., B ING HAM , L. & B RAILOV SKY, C.A. (1999 ) A m odified
contrasting groups m ethod used for setting the passm ark in a sm all
scale standardised patient examination, A dvances in H ealth Sciences
Education, 4, pp. 145± 154.
C ASE , S.M . & S W ANSEN , D.A. (1998 ) Constr ucting W ritten Test Ques-
tions in the B asic Sciences Philadelphia, PA, National Board of
M edical Exam iners, 3750 M arket Street, PA 19104) .
129
 M . Fr iedman B en-D avid
C IZEK , G.J. (1996 ) Standard setting guidelines, Educational M easure-
ment Issues and Practice, 15, pp. 12± 21.
C LAU SER , B.E ., C LYM AN , S.G., M ARG OLIS , M .J. & R OSS , C.P. (1996 )
Are fully com pensatory models appropriate for setting standards
on perform ance assessm ent of clinical skills?, Academic M edicine,
71, pp. 590± 592.
D E G R U IJT ER , D. (1985 ) C om prom ise m odels for establish ing
exam ination standards, Journal of Educational M easurement, 22, pp.
263± 269.
E BEL , R.L. (1972 ) Essentials of Education al Measurement (Englewood
Cliffs, NJ, Prentice-H all).
H AM BLETON , R.K. (1995 ) Setting standard on perform ance assess-
ments promising new methods and technical issues, paper presented
at the m eeting of Am erican Psychological Association, New York,
August 1995.
JA EG E R , R.M . (1982 ) An interactive structures judgem ent p rocess
for es tab lishing standards on com p etency tests: theor y and
application, E ducational E valuati on and Policy A nalysis, 4, pp.
461± 476.
Med Teach Downloaded from informahealthcare.com by Universitetssykehuset Nord-Norge on 02/28/13
JAEG ER , R.M . (1989 ) Certi® cation of student com petence, in: R.L.
L IN N (Ed.) Educational Measurement, 3rd edn, pp. 485± 514 (N ew
York, American Council on Education and Macm illan).
JAEG ER , R.M . (1991 ) Selection of judges for standard setting, Educa-
tion M easurement Issues and Practice, 10(2), pp. 3± 6.
JAEGER , R.M ., M ULLIS, I.V.S., B OURQU E , M.L. & S H AKRANI , S. (1996)
Setting perform ance standards for perform ance assessm ent: som e
fundam ental issues, current practice, and technical dilem m as, in:
Technica l Issues in Large-Scale Performance Assessment, NCES 96± 802
(U S D epartm ent of Education, Office of Educational Research
and Improvement).
K AN E , M . (1994 ) Validating the perform ance standards associated
For personal use only.
130
w ith passing scores, Review of Educational Research, 64(3), pp.
425± 461.
L IN N , R .L., M A D AU S , G .F. & P E D U L LA , J.J. (1982 ) M inimum
competency testing cautions on the state of the art, Am er ican Journal
of Education, 91, pp. 1± 35.
L IVINGSTON , S.A & Z IEKY , M .J. (1982 ) Passing Scores: A Manual for
Setting Standards of Performance on Education al and Occupatio nal
Tests (Princeton, NJ, Educational Testing Services).
M ESSIC K , S. (1989) Validity in Education al M easurement, 3rd edn, Ed.
R.L . L IN , pp. 3± 105 (N ew York, Am erican Council on Education/
Macm illan).
M U IJTJENS , A.M .M ., H OOG ENBO OM , R.J.I., VERW IJNEN , G.M . & VAN
D ER V LEU TEN , C.P.M. (1998 ) Relative or absolute standards in
assessing medical knowledge using progress tests, Advances in Health
Sciences Education, 3, pp. 81± 87.
N EDELSK Y , L. (1954 ) Absolute grading standards for objective tests,
Education al and Psychological M easurement, 14, pp. 3± 19.
N ORCINI , J.J., S TILLM AN , P.L., S U TNICK , A.I., R EG AN , M.B., H ALEY,
H.L.,W ILLIAM S , R.G. & F R IEDM AN , M. (1993 ) Scoring and standard
setting with standardised patients, Evaluation and The Health Profes-
sions, 16, pp. 322± 332 .
N U NGESTE R , R.J., D ILLON , G.F., S WANSON , D.B., O R R , N. A. &
P OW E LL , R .D. (1991 ) Standard setting p lans for the N BM E
Comprehensive Part I and Part II examinations, Academic M edicine,
66(8), pp. 429± 433.
S IZM U R , S. (1997 ) Look back in Angoff: a cautionary tale, B ritish
Education al Research Jour nal, 23, pp. 3± 13.
W ILIAM D. (1996 ) M eanings and consequences in standard setting ,
Assessment in Education, 3, pp. 287± 296.
WO LF , A. & S ILVER , R. (1986) Work-based Lear ning:Trainee Assessment
by Super visors, MSC R&D Series No. 33 (Sheffield).