Sei sulla pagina 1di 31

Human Resource Development: Principles and Practice By Brian Delahaye

13.1
CHAPTER 13: EVALUATION
MISCONCEPTIONS ABOUT EVALUATION
ASSESSMENT OF LEARNING
KIRKPATRICKS FOUR LEVELS
THE PRESAGE FACTORS
THE SCIENTIFIC MODELS
COST-BENEFIT ANALYSIS
THE EVALUATION PLAN
THE EVALUATION REPORT
THE LEGITIMATE SYSTEM
THE SHADOW SYSTEM
THE NEED FOR DIALECTIC THINKING
Human Resource Development: Principles and Practice By Brian Delahaye
13.2
CHAPTER 13: EVALUATION
MISCONCEPTIONS ABOUT EVALUATION
ASSESSMENT OF LEARNING
KIRKPATRICKS FOUR LEVELS
THE PRESAGE FACTORS
THE SCIENTIFIC MODELS
COST-BENEFIT ANALYSIS
THE EVALUATION PLAN
THE EVALUATION REPORT
THE LEGITIMATE SYSTEM
THE SHADOW SYSTEM
THE NEED FOR DIALECTIC THINKING
Human Resource Development: Principles and Practice By Brian Delahaye
13.3
MISCONCEPTIONS ABOUT
EVALUATION
While evaluation is seen as professionally and theoretically
desirable, some have doubts about its worth.
1. It is too costly. This misconception is based on a very short-
term view of costs. The cost of risk is also part of the equation.
These risks include not knowing if mistakes were made and not
knowing how to avoid these mistakes in the future.

2. The measures are not exact. This misconception does not
recognise that any complex decision is based on less than perfect
information.

3. It is too difficult. True, most evaluation recommendations
apply more obviously to the legitimate system. However, the
developmental efforts of the shadow system can be evaluated.
Human Resource Development: Principles and Practice By Brian Delahaye
13.4
THE ROLE OF EVALUATION
The role of the evaluation stage is four-fold:

1. to measure what change has occurred

2. to improve the other three stages investigation, design and
implementation

3. to see if the change is attributable to the learning episode

4. to see if the amount of change is worthwhile.
Human Resource Development: Principles and Practice By Brian Delahaye
13.5
CHAPTER 13: EVALUATION
MISCONCEPTIONS ABOUT EVALUATION
ASSESSMENT OF LEARNING
KIRKPATRICKS FOUR LEVELS
THE PRESAGE FACTORS
THE SCIENTIFIC MODELS
COST-BENEFIT ANALYSIS
THE EVALUATION PLAN
THE EVALUATION REPORT
THE LEGITIMATE SYSTEM
THE SHADOW SYSTEM
THE NEED FOR DIALECTIC THINKING
Human Resource Development: Principles and Practice By Brian Delahaye
13.6
ASSESSMENT OF LEARNING
Assessment addresses the question What changes have occurred
in the mind of the learner as a result of the learning episode?. To
measure this change, the learning has to be converted into a
behaviour that can be observed and this has some weaknesses:

e The behaviour is only a sample of the learning.
e The behaviour represents explicit knowledge only.
e The learning generation processes may take some time.
e The test has to tap into the appropriate knowledge.
e The test has to tap into the same potential knowledge each
time.
There are several issues that need to be considered when planning
assessment.
Human Resource Development: Principles and Practice By Brian Delahaye
13.7
TYPES OF ASSESSMENT
Assessment can be divided into six types:

> skill tests are used for procedural skills
> objective written tests e.g. multiple choice
> subjective written tests e.g. essays
> performance tests used to assess complex processes. The
examiner may asses the process itself or the product often using a
pre-designed observation form (see Fig.13.1 on page 346).
> learning diaries Journals written by the learner; good for
assessing externalisation
> portfolio assessment collections of the learners work; takes
a long term view, focuses on quality and the learner reflects on
own work.
Human Resource Development: Principles and Practice By Brian Delahaye
13.8
THE MEANING OF SCORES
When a test has been examined and given a quantitative score the
result is called a raw score.

This raw score can be interpreted in a number of ways:
_ criterion-referenced scoring the learners score is compared
to a pass mark
_ norm-referenced scoring the raw score is compared to the
average of a nominated group
_ formative assessment the score is used for developmental
feedback
_ summative assessment the score is used for evidence of
learning.
Human Resource Development: Principles and Practice By Brian Delahaye
13.9
OVERLAP OF THE FOUR HRD
PROCESSES
The assessment of learning plays a dual role being critical to
both the implementation stage and the evaluation stage of HRD.

In the implementation stage, assessment provides dynamic
feedback. Firstly, this feedback is often immediate. Secondly, it
provides irrefutable evidence that the learner finds difficult to
ignore.

In the evaluation stage, assessment provides initial evidence of
the success or otherwise of the learning experience. This
feedback may be taken into account immediately (e.g. the HR
developer changes the learning processes) or later (e.g.
recommendations in the Evaluation Report).
Human Resource Development: Principles and Practice By Brian Delahaye
13.10
CHAPTER 13: EVALUATION
MISCONCEPTIONS ABOUT EVALUATION
ASSESSMENT OF LEARNING
KIRKPATRICKS FOUR LEVELS
THE PRESAGE FACTORS
THE SCIENTIFIC MODELS
COST-BENEFIT ANALYSIS
THE EVALUATION PLAN
THE EVALUATION REPORT
THE LEGITIMATE SYSTEM
THE SHADOW SYSTEM
THE NEED FOR DIALECTIC THINKING
Human Resource Development: Principles and Practice By Brian Delahaye
13.11
KIRKPATRICKS FOUR LEVELS
Kirkpatricks model is based on four levels of ascending order:

1. reaction measures the reactions of the learners to the
learning episode
2. learning measures the knowledge gained by the learners.
Has been discussed under Assessment of Learning.
3. behaviour examines the change of behaviour of the learner
on the job
4. results measures the impact of the learning on the
organisation as a whole

Several comments can be made about Kirkpatricks model.

Human Resource Development: Principles and Practice By Brian Delahaye
13.12
COMMENTS ON KIRKPATRICKS
MODEL
The strengths of Kirkpatricks model include:
` simple and seemingly robust model
` recognises the impact of a learning episode should extend
beyond the individual learner
` provides an easily remembered checklist
` each level provides a unique examination of HRD.

The criticisms of Kirkpatricks model include:
the levels are not co-dependent
the model concentrates on processes and outputs, not inputs
the initial level is easier but measurement of the deeper levels
are more susceptible to contamination.
Human Resource Development: Principles and Practice By Brian Delahaye
13.13
CHAPTER 13: EVALUATION
MISCONCEPTIONS ABOUT EVALUATION
ASSESSMENT OF LEARNING
KIRKPATRICKS FOUR LEVELS
THE PRESAGE FACTORS
THE SCIENTIFIC MODELS
COST-BENEFIT ANALYSIS
THE EVALUATION PLAN
THE EVALUATION REPORT
THE LEGITIMATE SYSTEM
THE SHADOW SYSTEM
THE NEED FOR DIALECTIC THINKING
Human Resource Development: Principles and Practice By Brian Delahaye
13.14
THE PRESAGE FACTORS
Brinkerhoffs model proposes six levels:

Stage I: Evaluate needs and goals ( this is similar to the HRDNI).
Stage II: Evaluate the HR design
Stage III: Evaluate implementation
Stage IV: Evaluate learning (similar to Kirkpatricks level 1)
Stage V: Evaluate usage and endurance of learning (similar to
Kirkpatricks level 3)
Stage VI: Evaluate pay-off (similar to Kirkpatricks level 4).

Brinkerhoff, then, adds two that occur prior to the learning
episode evaluate HR design and evaluate implementation.
These are called presage factors.
Human Resource Development: Principles and Practice By Brian Delahaye
13.15
THE PRESAGE FACTORS (Continued)
Stage II, evaluating the HR design focuses on the quality of the
HR plan (i.e. the training or workshop program).

A stage II evaluation should occur when:
the design is unique or experimental
the costs of mounting the program are high
the HRD issues are crucial
the participant groups are volatile, influential or demanding.

The design should be reviewed by a variety of stakeholders.
Human Resource Development: Principles and Practice By Brian Delahaye
13.16
THE PRESAGE FACTORS (Continued)
Stage III, evaluating during the implementation provides data that
will help shepherd the learning event to a successful conclusion.

This highlights the covert activity of the conducting HR
developer where the program is continually monitored to
compare reality with the program plan.

The HR developer can also be monitored, usually by another HR
developer. However, the big brother syndrome has to be
avoided.
Human Resource Development: Principles and Practice By Brian Delahaye
13.17
TIME OUT
To this stage of the discussion, we have examined assessment of
learning, Kirkpatricks model and Brinkerhoffs model. This has
answered two of the roles of evaluation:

1. to identify what change has occurred
2. to improve the other three stages.

The other two roles will now be addressed:

3. to see if the change is attributable to the learning episode. This
will be examined by looking at the SCIENTIFIC MODELS.
4. to see if the amount of change was worthwhile. This will be
examined by looking at COST BENEFIT ANALYSIS.
Human Resource Development: Principles and Practice By Brian Delahaye
13.18
CHAPTER 13: EVALUATION
MISCONCEPTIONS ABOUT EVALUATION
ASSESSMENT OF LEARNING
KIRKPATRICKS FOUR LEVELS
THE PRESAGE FACTORS
THE SCIENTIFIC MODELS
COST-BENEFIT ANALYSIS
THE EVALUATION PLAN
THE EVALUATION REPORT
THE LEGITIMATE SYSTEM
THE SHADOW SYSTEM
THE NEED FOR DIALECTIC THINKING
Human Resource Development: Principles and Practice By Brian Delahaye
13.19
THE SCIENTIFIC MODELS
The scientific models are based on the experimental methods
used in research laboratories and are used to demonstrate
causality. They also assume that changes can be measured.

Post-test = learning experience/evaluation
Pre-test - Post-test = evaluation/learning experience/evaluation
Time series evaluation = pre-test/pre-test/pre-test/learning
experience/post-test/post-test/post-test
Control group =
Experimental group pre-test/learning experience/post-test
Control group pre-test/usual duty/ post-test
The Solomon four uses three control groups and the
experimental group

The HR developer has to compare the costs of the designs
compared to the expected benefits gained.
Human Resource Development: Principles and Practice By Brian Delahaye
13.20
CHAPTER 13: EVALUATION
MISCONCEPTIONS ABOUT EVALUATION
ASSESSMENT OF LEARNING
KIRKPATRICKS FOUR LEVELS
THE PRESAGE FACTORS
THE SCIENTIFIC MODELS
COST-BENEFIT ANALYSIS
THE EVALUATION PLAN
THE EVALUATION REPORT
THE LEGITIMATE SYSTEM
THE SHADOW SYSTEM
THE NEED FOR DIALECTIC THINKING
Human Resource Development: Principles and Practice By Brian Delahaye
13.21
COST-BENEFIT ANALYSIS
Cost-benefit analysis (CBA) is used to see if the amount of
change was worthwhile. It is based on the premise:
~ identify the costs, in dollar terms, for the learning experience
~ identify the the benefits accruing from the learning
experience, in dollar terms
~ the ratio between the costs and benefits should be in favour
of the benefits.

There are two problems often encountered:
converting the benefits into monetary values
deciding on the cut-off points.

There are two good reasons for conducting a CBA. Firstly, the
survival of the HR section may depend on proving the value of
development. Secondly, it helps in the decision of selecting the
learning programs that have the most impact.
Human Resource Development: Principles and Practice By Brian Delahaye
13.22
CHAPTER 13: EVALUATION
MISCONCEPTIONS ABOUT EVALUATION
ASSESSMENT OF LEARNING
KIRKPATRICKS FOUR LEVELS
THE PRESAGE FACTORS
THE SCIENTIFIC MODELS
COST-BENEFIT ANALYSIS
THE EVALUATION PLAN
THE EVALUATION REPORT
THE LEGITIMATE SYSTEM
THE SHADOW SYSTEM
THE NEED FOR DIALECTIC THINKING
Human Resource Development: Principles and Practice By Brian Delahaye
13.23
THE EVALUATION PLAN
Planning for evaluation commences during the design stage. This
ensures that the appropriate evaluation occurs at the appropriate
time. It also allows the coordination of developmental and
judgemental evaluation. Planning or evaluation should include:
develop the assessment for developmental purposes first
incorporate this into the evaluation plus any further
assessment required
decide what presage variables will be evaluated and when
incorporate appropriate HRDNI investigation instruments
design daily and course/workshop reaction sheets
design pre-test and post-test instruments, if appropriate
identify and plan methods to be used at Stage V and VI
if CBA is to be used, arrange collection of costs and benefits
prepare a budget
send evaluation plan to staff who are affected.
Human Resource Development: Principles and Practice By Brian Delahaye
13.24
CHAPTER 13: EVALUATION
MISCONCEPTIONS ABOUT EVALUATION
ASSESSMENT OF LEARNING
KIRKPATRICKS FOUR LEVELS
THE PRESAGE FACTORS
THE SCIENTIFIC MODELS
COST-BENEFIT ANALYSIS
THE EVALUATION PLAN
THE EVALUATION REPORT
THE LEGITIMATE SYSTEM
THE SHADOW SYSTEM
THE NEED FOR DIALECTIC THINKING
Human Resource Development: Principles and Practice By Brian Delahaye
13.25
THE EVALUATION REPORT
Once the evaluation has been completed a report should be
completed for the key stakeholders. The evaluation plan has a
communication role, decision making role and becomes an
historical document. This report should include:
an executive summary
a findings/recommendations section
a contents list
the main body, which should include:
> reasons for the evaluation
> list of personnel involved
> discussion of the various types of evaluation undertaken
> a discussion on the findings, the options and the
conclusions
> a list of, and discussion of, the recommendations
appendices.
Human Resource Development: Principles and Practice By Brian Delahaye
13.26
CHAPTER 13: EVALUATION
MISCONCEPTIONS ABOUT EVALUATION
ASSESSMENT OF LEARNING
KIRKPATRICKS FOUR LEVELS
THE PRESAGE FACTORS
THE SCIENTIFIC MODELS
COST-BENEFIT ANALYSIS
THE EVALUATION PLAN
THE EVALUATION REPORT
THE LEGITIMATE SYSTEM
THE SHADOW SYSTEM
THE NEED FOR DIALECTIC THINKING
Human Resource Development: Principles and Practice By Brian Delahaye
13.27
THE LEGITIMATE SYSTEM
Most of what has been discussed in this chapter applies directly
to the legitimate system. The methods mainly rely on negative
feedback systems. The methods also emphasise the legitimate
systems focus on the efficient use of resources and the immediate
survival of the organisation.

For the legitimate system, one of the advantages of competency-
based learning has been the recognition of the strong link
between learning in a planned program and the application of
that learning in the workplace.

It should also be recognised that, if the assessment process fails
to detect a lack of comprehension on the part of the learners, the
HR developer could be held liable.
Human Resource Development: Principles and Practice By Brian Delahaye
13.28
CHAPTER 13: EVALUATION
MISCONCEPTIONS ABOUT EVALUATION
ASSESSMENT OF LEARNING
KIRKPATRICKS FOUR LEVELS
THE PRESAGE FACTORS
THE SCIENTIFIC MODELS
COST-BENEFIT ANALYSIS
THE EVALUATION PLAN
THE EVALUATION REPORT
THE LEGITIMATE SYSTEM
THE SHADOW SYSTEM
THE NEED FOR DIALECTIC THINKING
Human Resource Development: Principles and Practice By Brian Delahaye
13.29
THE SHADOW SYSTEM
Evaluation in the shadow system is a very delicate affair. The
use, by a third party, of the methods described in this chapter
results in negative feedback loops and this kills the creativity
needed in the shadow system.

The main evaluation method for the shadow system is the use of
values systems. The manger instills a particular value in the
system and checks for evidence for this value.

The actors in the shadow system, however, should use the
evaluation methods on themselves and their own endeavours.
Human Resource Development: Principles and Practice By Brian Delahaye
13.30
CHAPTER 13: EVALUATION
MISCONCEPTIONS ABOUT EVALUATION
ASSESSMENT OF LEARNING
KIRKPATRICKS FOUR LEVELS
THE PRESAGE FACTORS
THE SCIENTIFIC MODELS
COST-BENEFIT ANALYSIS
THE EVALUATION PLAN
THE EVALUATION REPORT
THE LEGITIMATE SYSTEM
THE SHADOW SYSTEM
THE NEED FOR DIALECTIC THINKING
Human Resource Development: Principles and Practice By Brian Delahaye
13.31
THE NEED FOR DIALECTIC
THINKING
Some of the misperceptions about evaluation can be explained
by the concept of dialectic thinking (see Chapter 2).

Evaluation has a number of opposing characteristics:
evaluation can be both developmental and judgemental
the more objective the measure used, the less rich the insights
and vice versa
liberal education vs. vocational education.

Evaluation serves and supports a number of dual roles. Balancing
these conflicting roles is the new challenge as knowledge is a
valuable but delicate resource.