Documenti di Didattica
Documenti di Professioni
Documenti di Cultura
Raja C. Bandaranayake
DEFINITIONS
Standard setting is a process of determining how much is good enough. The standard or criterion level of performance is a point on the scale of measurement at which separation of competence and incompetence occurs. Cut-score, cut-off score or passing score represents this standard on a given test for making decisions pertaining to the purpose for which the test was conducted, e.g., to certify competence.
ERROR IN MEASUREMENT
True score is a conceptual measure indicating true extent of competence in a given subject, e.g., Anatomy.
Observed score is the score assigned as a result of taking a test, say in Anatomy. The difference between true and observed scores is indicative of the amount of error in the measurement. The reliability of a test and the associated standard error of measurement are estimates of the amount of error in the measurement.
DECISION ERRORS
False positive: passing an incompetent examinee False negative: failing a competent examinee
NORM- & CRITERION-REFERENCED STANDARDS NORM-REFERENCED Relative CRITERIONREFERENCED Absolute Not related to peer performance Standard set prior to exam Referenced to a defined level of performance
Based on peerperformance
Varies with each group Cut-off point not related to competence
2. Examinee-centred methods
3. Compromise methods
Provide flexibility for adjusting the standard based on the examinees performance on the test.
Example
Consider N judges and n MCQ items of 1 in 5 type Judge A identifies 2 options in item 1 as those which a minimally competent examinee should eliminate as incorrect. MPL for that item for Judge A [MPLA1] = 1/(5-2) = 1/3 Similarly, in item 2 he identifies 3 options, giving an MPLA2 = 1/(5-3) = 1/2 He repeats this process for each item. The exam MPL for Judge A [MPLA] = MPLA1 +MPLA2 + MPLA3 + .MPLAn
The MPL for Judge A [MPLA] is then: (1500 + 1600 + 500 + 300 + 2000 + 1000 + 300 + 600 + 1000 + 50 + 0) / 200 = 56.25 % Similarly, the MPL for Judges B [MPLB], C [MPLc], D [MPLD] ..N [MPLN] are determined. The MPL for the exam (cut-off score) is: (MPLA+ MPLB+ MPLc+ MPLD + ..MPLN) / N
ESSENT.
6x 100% = 600
12 x 80% = 960
7 x 50% = 350
IMPORT.
12 x 80% = 960
24 x 60% = 1440
19 x 40% = 760
ACCEPT.
5 x 60% = 300
12 x 50% = 600
3 x 10% = 30
MPL:
HOFSTEE METHOD
fmax
Failure Rate%
20
15
fmin
10
35
40
45
50
Cut-off score(%)
cmin
cmax
Example
HOFSTEE METHOD
A plot of cut-off scores for a given exam against resulting failure rates is given
CUT-OFF SCORE FOR 1 IN 5 MCQ [FRACS PART 1] Probability of guessing (=1 in 5) = 20%
= 20%
=100% = 20% to = 60%
= 5%
56.7%
Average exam mean score over last 4 years: 59.4% Thus mean score in this exam is: Assuming this candidate group is of same standard as in last 4 yrs, this exam is: 2.7% lower
2.7% harder
2.0% higher
2.0% better
CUT-OFF SCORES: MARKER QUESTIONS contd. 4. Determining cut-off score The cut-off level for an average exam is: Thus the cut-off level for this exam should be (65 4.7)% = 65.0%
60.3%
HOFSTEE CURVE
20
Failure Rate%
15
10
55
60
65
70
Cut-off score(%)