KNOWLEDGE MANAGEMENT INITIATIVES

BVICL/Jul’13/wk-27

What is training effectiveness and how to measure it?

Each year globally many millions of dollars are spent on training. It would seem reasonable to establish just
how effective this training has been if only to help justify the budget. Evaluation techniques are not new,
indeed the most influential early work on evaluation was performed in the USA during the 1940s by Tyler.
Probably the most important aspect of Ralph W Tyler’s work was the realisation of the importance of
objectives in designing an effective school curriculum. This early methodology can be defined as the
scientific/experimental approach.

The model emphasised the importance of knowing the educational standards of the individuals in the
control group before the new education initiative took place and then determining the change that had
taken place by measuring the difference in attainment after the educational intervention was completed. In
other words the aim was to determine the effectiveness of the training by scientific means on the level of
performance of the individual. Although there have been some critics of this methodology it has proved to
be valuable in that organisations have been able to quantify the relationship between their corporate
objectives (often expressed as key performance indicators (KPIs) today) and the training aims and
objectives.

The main criticism of the scientific/experimental approach is that the methodology is unable to take into
account the unexpected or unintentional learning that can take place within a dynamic and changing
learning environment. Those readers who can remember reading about the original, and now famous
Hawthorne experiments to determine the effects of illumination levels on production held at the Western
Electric plant in Cicero, Illinois in the 1920’s and the later experiments also conducted by the National
Research Council showed that human behaviour is indeed difficult to quantify exactly. Nevertheless the
greater attention that was given to training methodology and in particular the effectiveness of management
development during the 1960s and 1970s has led to much better understanding and the emergence of
systems evaluation methodology.

As we will see this approach can be divided into two main schools; the narrow focused approach being
attributed to Donald L Kirkpatrick in the early 1960s and the somewhat broader model advocated by P.
Warr et al (The Evaluation of Management Training) known as the CIRO model. The CIRO model looked at
context evaluation, input evaluation, reaction evaluation and outcome evaluation.

1|Page

(This is a small endeavour to enable our auditing competence

and
to be used for internal use only )
 KNOWLEDGE MANAGEMENT INITIATIVES

BVICL/Jul’13/wk-27

Donald L Kirkpatrick’s "Four steps to measuring training effectiveness”

This model uses four separate stages for the evaluation of the effectiveness of a training program.

The four stages are:-

 Reaction
 Learning
 Behaviour
 Results

Level 1

The first stage is about the reaction of the trainee to the training. This sort of measurement is concerned
with how the trainees "feel” about the course. The usual course feedback sheets are an example of the
Kirkpatrick level 1 evaluation. Most organisations do not do any more than this type of measurement and
analysis. The drawback is that we do not really know if the trainee has actually learnt anything. What really
seems to be being asked of the trainee was how "happy” were you with the course; hence the somewhat
derogatory description that is often applied to the "happy sheets”! We will now look at the next level of
evaluation concerned with Learning.

Level 2

Things can be improved by using a pre-test and post-test and comparing the results. The questions need to
be objective and closely related to the course objectives (more about that later). In this way we can
determine if the training actually delivered knowledge and this was understood by the trainees at the time.
An organisation that does this can be confident that the trainee has actually learnt something at that time.
Why do I make the point that we have to make the measurement and consider the learning at a certain
time? Well because we do not know if the learning has had time to be internalised and become "concrete”.
All too often trainees (and delegates at conferences for example) will have difficulty remembering what was
in the course or seminar they attended a few days later let alone months later. There are various
techniques that we can employ to improve the level of recall but basically "if we don’t use it we will lose it”!

Level 3

This is concerned with "behaviour”. By that we mean the measurable change in an individual as a result of
their attendance on the training course. This is, in my opinion, the least we should be expecting from any

2|Page

(This is a small endeavour to enable our auditing competence

and
to be used for internal use only )
 KNOWLEDGE MANAGEMENT INITIATIVES

BVICL/Jul’13/wk-27
training program. After all what is the point of spending money and using resources if the training does not
effect some measurable change in the behaviour of the trainee?

Level 4

Kirkpatrick is now concerned with the training to determine if it has actually been translated into tangible
benefits to the organisation. Quite simply has productivity and or quality been improved? Have the number
of accidents or incidents been reduced? Has plant availability and or plant utilisation been improved? Has
the morale of the workforce changed for the better? These are metrics which really have an impact on the
"bottom line” and for that reason feature in the companies balance sheets and KPIs. We have to ask
ourselves is this not the real reason for training? Training has to make a real difference in performance and
effectiveness; this is tied closely to competence. Training has be proven to deliver results and be cost
effective. It might sound simple but it is not for most organisations. The reason is that most do not have in
place any system for measuring the improvement in competence of the individual; let alone a systematic
approach to identifying the most effective means of assisting the individual to becoming competent.

Cost effective training

If we all now accept that the majority of organisations carry out training to change the behaviour of the
workforce and improve their performance in the workplace we also must accept that this cannot be
achieved without a clear plan and strategy.

This rational approach to training and development is best demonstrated by the Competence Profile(s) or
Competence Map(s) for a particular job(s) in the organisation(s). If the approach in constructing these has
been methodical we will have at our disposal a database of learning objectives.

I call these TCOs or terminal competence objectives. These TCOs must relate to the needs of the
organisation and will contain the "underpinning” knowledge required to understand the tasks or duties the
employee must perform as part of his/her duties as well as the practical skills necessary.

In this way we are able to ensure that the training is appropriate to the individual at that time in their
development and truly meets both their needs and those of the organisation.

If we use these TCOs to develop the training program (whether it is a classroom based course or a
workshop activity) we will have a course "tailored” to the needs of the organisation and furthermore we will
be able to measure the improvement in performance and contribution to the team’s KPIs.

3|Page

(This is a small endeavour to enable our auditing competence

and
to be used for internal use only )
 KNOWLEDGE MANAGEMENT INITIATIVES

BVICL/Jul’13/wk-27
Evidence or proof?

A goal can be considered to be something that the organisation strives to achieve through the meeting of
specific objectives. Achieving the individual objectives in a methodical and logical way effectively maps out
the process towards the eventual achievement of the goal. If we look at an important KPI for any
organisation it might be expressed as "zero lost time injuries”; in other words no employee being away from
work for more than 3 days because of an injury sustained while in the company or performing work related
duties (like driving a truck for example). If a company with over a 1,000 employees for example achieves a
figure of say 6,000,000 man hours without a LTI then clearly their HSE systems must be working. This can
be translated into real company benefits and major savings to the organisation. There are many tangible
benefits and other benefits which are very important but more difficult to quantify.

Benefits can include:-

 Better productivity
 Lower insurance premiums
 Improved morale
 Enhanced reputation
 Reduced stress
 No legal costs
 No compensation cases
 Lower staff turnover
 Employer of choice

Input, Process and Output

I have presented an argument for mapping out the competences required to perform a job and to linking
these competences to the delivery of training interventions. In effect this means that the strategic goals of
the organisation must be cascaded down to team level within the organisations. In time new competences
will be required for team members as the industry "drivers” like new technology and competition
encourages or perhaps forces the organisation to adopt new work systems and processes. Some leading
companies recognised this change model and one (IBM) introduced the IPO paradigm. The concern being
is training delivering the desired results within the organisation? The IPO model was designed to confront
and solve problem. In this model a distinction is made between output (short term benefits) and outcomes
(benefits which are longer term but actually determine the availability of future training resources). The
following quote is particularly relevant here "The ultimate payoff or added value of an employee’s learning
experience is how well he or she performs on the job.” David S Bushnell writing about the "Input, Process,
Output: A model for Evaluating Training” 1990 Training and Development Journal.

4|Page

(This is a small endeavour to enable our auditing competence

and
to be used for internal use only )
 KNOWLEDGE MANAGEMENT INITIATIVES

BVICL/Jul’13/wk-27
Flow chart of input-output approach to training evaluation

Conclusion

We have seen that evaluation of training interventions such as conventional courses are generally a
neglected area as regards evaluation of their effectiveness. In general the "happy sheet” type of evaluation
is ineffective as it is inclined to be subjective, and influenced by considerations which are actually outside
the course itself. That is the course content could be poor or inappropriate but was delivered by a good
instructor who managed the group well and they actually "enjoyed” the experience but learnt little!

Instead I have suggested that we owe it to the organisation to identify the competences that are really
required by the individual to do his or her job well. These competences are the skills and underpinning
knowledge which should be part of the company database and accessible through its Competence
Management System.

The training intervention has to be designed to meet the needs of the organisation by satisfying the
competence development requirements of the individual. By achieving this goal we have a rational and
justifiable case for training and we are able to prove that training really does deliver tangible benefits, not
least to the "bottom line”.

Source:

5|Page

(This is a small endeavour to enable our auditing competence

and
to be used for internal use only )