Evaluation

By Dr.Jitendra Ahuja

Evaluation

Definition Evaluation is a systematic process of determining the extent to which the predetermined objectives are achieved (by the student). It includes qualitative and quantitative description (measurement) of desirability of that behavior. Evaluation= Measurement + Value judgment

Evaluation
2. PURPOSE-Evaluation plays an important role in education. It helps in: Selection of students Pre-asessrnent of the learners educational needs (Pre-test) Monitoring the learning progress - feed back to the student (Formative Test) Feed back to the teacher -. learning difficulties especially what part of instructions students have not understood (Diagnostic Test) Modification of learning activities Determining success or failure ( Summative Test) Administration - for certifying (Summative Test) For guidance and counseling. As an external motivator for learning Determining programme effectiveness

Evaluation-Types of evaluation procedure

FORMATIVE (Diagnostic) ASSESSMENT

Purpose: To detect specific strengths and weaknesses of individual students. Frequency: Programmed during or at the end of each unit of instruction. Area covered: A single unit of instruction or a number of units. Administrative utility: Advisory only, not always taken into account for permanent record. Feed back to the students: Done immediately to enable students to identify what they have learnt or failed to learn. Feed back to the faculty: Errors committed by more than a few students immediately identify the area of weakness in instruction.

Evaluation-Types of evaluation procedure

B. SUMMATIVE (Certifying) ASSESSMENT Purpose: To determine the overall achievement of a satisfactory level of competence. Frequency: Only at the point of promotion, graduation or licensure. Area covered: Course content. Administrative utility: Goes into permanent record. Feedback to students: Only intimation regarding pass or failure. Feedback to faculty: Overall passes or failures.

Evaluation-Types of evaluation procedure

Test Test is an instrument containing a set of questions, exercises or other means of measuring the characteristics such as knowledge, skills, intelligence, aptitude, etc., of an individual or a group of individuals. 2 Types Norm-Referenced Tests Criterion Referenced Tests (CRT)

Evaluation-Norm-Referenced Tests

Standardized tests compare students performance to that of a norm or sample group who are in the same grade, the purpose is to discriminate between the high-achievers and the low achievers. It assesses the ability of one against the standard norm of achievement of one s fellow testers. Students performance is communicated in percentile ranks, grade- equivalent scores, normal-curve equivalents, scaled scores. Relative Interpretation. E.g. Competitive or entrance examination.

Evaluation-Criterion Referenced Tests

(CRT) It determines an individual status with refer to well defined criterion behavior. It is an attempt to interpret test results in terms of clearly defined learning outcomes which serve as referents (criteria). A CRT compares the learners performance with a set standard of performance OR A criterion-referenced test is used to ascertain an individuals status with respect to a defined achievement domain. Absolute interpretation- e.g. 50% is set as passing standard tor evaluation in medicine

Evaluation-Important characteristics of the

test

I. Validity The extent to which a test serves the purpose for which it has been prepared is called its validity. It is the degree to which a test measures what it is intended to measure. Validity pertains to test result. It is a matter of degree and is specific to some particular subject. It depends upon the extent to which the content of the test samples the subject matter and behavior under consideration.

Validity is accuracy
Archer 1 hits bulls eye every time

Archer 2 hits outer ring in same spot every time

Both archers are equally reliable Validity = quality of archers hits

Validity is accuracy
Archer 1 hits bulls eye every time

Archer 2 hits outer ring in same spot every time

Both archers are equally reliable Validity = quality of archers hits

Reliability

2. Reliability Next to validity, reliability is the most important characteristic of evaluation results. Reliability is the extent to which results obtained through a test remain constant over a period of time or in different situations for the same sample. It refers to the consistency with which an instrument measures the variable.

Reliability as Consistency
Archer 1 hits bulls eye every time.

Archer 2 hits outer ring in same spot every time.

Both archers are equally reliable.

Reliability and Validity

Reliable and Valid

Reliable, not valid

Not reliable, not valid

Evaluation-Objectivity

Objectivity in scoring may be considered as consistency in scoring by different scorers i.e. uniformity in scoring. It is the degree of concordance between the judgments of independent and competent examiners as to what constitutes a good answer for each question.

Evaluation- Methods of evaluation

a) Written test b)Practical c) Clinical d)Oral e) Observation f) Thesis g) Projects

Evaluation- Selection of evaluation

methods
Cognitive domain: Skills: Psychomotor domain Practical Clinical
Direct unobtrusive observation Simulated patients and patient interviews

Affective Domain

Written test Viva-oral test

Thesis writing

Situational assessmentsimulated situations or mock exercises

Project

Evaluation- the types of evaluation tools

according to domains/skills.
Activities Intellectual Skills Tools suggested EQ. SAQ. MCQs (testing higher abilities and problem solving OSCE/OSPE OSCE/Structured case viva OSCE. Structured viva OSCE Stations, Simulated cases (real/paper), checklist. logbooks

Procedural skills Clinical skills Communication skillsAttitude, Behavioral and Ethics

Evaluation- ORAL EXAMINATION

1. 2. 3. 4. 5. 6.

Purpose: To permit the student through his answers to questions put to him orally, to demonstrate his knowledge and understanding in his subject of study, as well as his thinking and problems solving ability. Disadvantages Inadequate standardization . Insufficient objectivity and reproducibility of result. Possible abuse of personal contact with examiner and probable cueing. Undue influence of irrelevant factors Few trained examiners available Excessive cost in terms of professional time in relation to the limited value of the information obtained.

Evaluation- ORAL EXAMINATION

1. 2. 3. 4. 5.

Advantages
Direct personal contact with the candidates. Opportunity to take into account mitigating circumstance. Flexibility in moving from strong to weak areas. Opportunity to ask candidate how he arrived at the answer. Opportunity for simultaneous assessment by two examiners.

Evaluation- Practical examination

Purpose: To reveal what the examinee can do as distinct from what he says he can do. Disadvantages

Insufficiently standardized conditions whether in laboratory experiments using animals or in bedside examination with patients of varying degrees of cooperativeness. Insufficient objectivity and intrusion of irrelevant factors Limited feasibility for large groups. Difficulties in arranging for examiners to observe candidates demonstrating the skills to be tested.

Evaluation- Practical examination

Advantages 1. Opportunity to test skills involving all the senses with observation of performance by examiner 2. Opportunity to confront the candidate with new problems both in the laboratory and at the bedside. to test his investigative ability . 3. Opportunity to observe and test attitudes and responsiveness to the total situation. 4. Opportunity to test the ability of student to communicate with the patient, to discriminate between important and trivial issues, to arrange and display the data.

Evaluation- Essay type examination

Purpose: To permit different examiners independently to arrive at the same or very similar grades for each examination question. Disadvantages Severe limitation of the area of students achievement that can be sampled. Subjective evaluation and low Reliability

Evaluation- Essay type examination

Advantages Opportunity to test notonly a candidates store of information. Also his ability to organize ideas and express them in own language.

Evaluation- Objective examinations

Most common type are the multiple-choice questions, consisting of several items (questions). A stem either in the form of a direct question or an incomplete sentence and a number of responses, one of which is the best answer; the other answers are referred to as distracters. Disadvantages 1. Construction. is time consuming if arbitrary and ambiguous questions are to be avoided, 2. Necessity of making allowance for positive scores that may be achieved by guessing. 3. Prejudice among some teachers against this type of examinations. 4. Clues are provided that are unavailable in practice.

Evaluation- Objective examinations

Advantages
1. 2.

3.
4. 5.

Significant increase in the range and variety of facts that can be sampled in a given time. Opportunity to test the candidate at the desired level by varying the difficulty of the questions and in the case of MCQ, including as possible answers, misconceptions common at his level of training. Opportunity to obtain detailed feedback for both student and facility Very economical for large groups. The standards of scoring can be kept constant for many years.

Evaluation- Observational reports on students

performances:

Purpose: This type of test serves dual purpose of identifying those with exceptional abilities and / or revealing those with persistent intractable deficiencies. Disadvantages The examiner acts both as an observer and a judge. Extended contact with student required for a valid estimate of his performance.

Evaluation- Observational reports on student

performances:

Advantages Opportunity to obtain full and usually more valid information about candidate. By pooling the reports can be made more reliable.

Evaluation- Thesis and research projects:

Purpose: This method of assessment is designed primarily to provide information on the students ability to co11e information to put it in order. He is normally expected to work independently and at his own convenience. However, we believe that it is not appropriate for all students and should be treated as an optional type of examination. Therefore, we feel that a list of advantages and disadvantages can be omitted.

Thank you