Home > Our News and Articles > How to measure training effectiveness

How to measure training effectiveness

Each year globally many millions of dollars are spent on training. It would seem reasonable to establish just how effective this training has been if only to help justify the budget. Evaluation techniques are not new, indeed the most influential early work on evaluation was performed in the USA during the 1940s by Tyler. Probably the most important aspect of Ralph W Tylers work was the realisation of the importance of objectives in designing an effective school curriculum. This early methodology can be defined as the scientific/experimental approach. The model emphasised the importance of knowing the educational standards of the individuals in the control group before the new education initiative took place and then determining the change that had taken place by measuring the difference in attainment after the educational intervention was completed. In other words the aim was to determine the effectiveness of the training by scientific means on the level of performance of the individual. Although there have been some critics of this methodology it has proved to be valuable in that organisations have been able to quantify the relationship between their corporate objectives (often expressed as key performance indicators (KPIs) today) and the training aims and objectives. The main criticism of the scientific/experimental approach is that the methodology is unable to take into account the unexpected or unintentional learning that can take place within a dynamic and changing learning environment. Those readers who can remember reading about the original, and now famous Hawthorne experiments to determine the effects of illumination levels on production held at the Western Electric plant in Cicero, Illinois in the 1920s and the later experiments also conducted by the National Research Council showed that human behaviour is indeed difficult to quantify exactly. Nevertheless the greater attention that was given to training methodology and in particular the effectiveness of management development during the 1960s and 1970s has led to much better understanding and the emergence of systems evaluation methodology. As we will see this approach can be divided into two main schools; the narrow focused approach being attributed to Donald L Kirkpatrick in the early 1960s and the somewhat broader model advocated by P. Warr et al (The Evaluation of Management Training) known as the CIRO model. The CIRO model looked at context evaluation, input evaluation, reaction evaluation and outcome evaluation. Donald L Kirkpatricks "Four steps to measuring training effectiveness This model uses four separate stages for the evaluation of the effectiveness of a training program. The four stages are:

Reaction Learning

Behaviour Results

Level 1 The first stage is about the reaction of the trainee to the training. This sort of measurement is concerned with how the trainees "feel about the course. The usual course feedback sheets are an example of the Kirkpatrick level 1 evaluation. Most organisations do not do any more than this type of measurement and analysis. The drawback is that we do not really know if the trainee has actually learnt anything. What really seems to be being asked of the trainee was how "happy were you with the course; hence the somewhat derogatory description that is often applied to the "happy sheets! We will now look at the next level of evaluation concerned with Learning. Level 2 Things can be improved by using a pre-test and post-test and comparing the results. The questions need to be objective and closely related to the course objectives (more about that later). In this way we can determine if the training actually delivered knowledge and this was understood by the trainees at the time. An organisation that does this can be confident that the trainee has actually learnt something at that time. Why do I make the point that we have to make the measurement and consider the learning at a certain time? Well because we do not know if the learning has had time to be internalised and become "concrete. All too often trainees (and delegates at conferences for example) will have difficulty remembering what was in the course or seminar they attended a few days later let alone months later. There are various techniques that we can employ to improve the level of recall but basically "if we dont use it we will lose it! Level 3 This is concerned with "behaviour. By that we mean the measurable change in an individual as a result of their attendance on the training course. This is, in my opinion, the least we should be expecting from any training program. After all what is the point of spending money and using resources if the training does not effect some measurable change in the behaviour of the trainee? Level 4 Kirkpatrick is now concerned with the training to determine if it has actually been translated into tangible benefits to the organisation. Quite simply has productivity and or quality been improved? Have the number of accidents or incidents been reduced? Has plant availability and or plant utilisation been improved? Has the morale of the workforce changed for the better? These are metrics which really have an impact on the "bottom line and for that reason feature in the companies balance sheets and KPIs. We have to ask ourselves is this not the real reason for training? Training has to make a real difference in performance and effectiveness; this is tied closely to competence. Training has be proven to deliver results and be cost effective. It might sound simple but it is not for most organisations. The reason is that most do not have in place any system for measuring the improvement in competence of the individual; let alone a

systematic approach to identifying the most effective means of assisting the individual to becoming competent. Cost effective training

If we all now accept that the majority of organisations carry out training to change the behaviour of the workforce and improve their performance in the workplace we also must accept that this cannot be achieved without a clear plan and strategy. This rational approach to training and development is best demonstrated by the Competence Profile(s) or Competence Map(s) for a particular job(s) in the organisation( s). If the approach in constructing these has been methodical we will have at our disposal a database of learning objectives. I call these

Evidence or proof? A goal can be considered to be something that the organisation strives to achieve through the meeting of specific objectives. Achieving the individual objectives in a methodical and logical way effectively maps out the process towards the eventual achievement of the goal. If we look at an important KPI for any organisation it might be expressed as "zero lost time injuries; in other words no employee being away from work for more than 3 days because of an injury sustained while in the company or performing work related duties (like driving a truck for example). If a company with over a 1,000 employees for example achieves a figure of say 6,000,000 man hours without a LTI then clearly their HSE systems must be working. This can be translated into real company benefits and major savings to the organisation. There are many tangible benefits and other benefits which are very important but more difficult to quantify. Benefits can include:

Better productivity Lower insurance premiums Improved morale Enhanced reputation Reduced stress No legal costs No compensation cases Lower staff turnover Employer of choice

Input, Process and Output I have presented an argument for mapping out the competences required to perform a job and to linking these competences to the delivery of training interventions. In effect this means that the strategic goals of the organisation must be cascaded down to team level within the organisations. In time new competences will be required for team members as the industry "drivers like new technology and competition encourages or perhaps forces the organisation to adopt new work systems and processes. Some leading companies recognised this change model and one (IBM) introduced the IPO paradigm. The concern being is training delivering the desired results within the organisation? The IPO model was designed to confront and solve problem. In this model a distinction is made between output (short term benefits) and outcomes (benefits which are longer term but actually determine the availability of future training resources). The following quote is particularly relevant here "The ultimate payoff or added value of an employees learning experience is how well he or she performs on the job. David S Bushnell writing about the "Input, Process, Output: A model for Evaluating Training 1990 Training and Development Journal. Flow chart of input-output approach to training evaluation

Conclusion We have seen that evaluation of training interventions such as conventional courses are generally a neglected area as regards evaluation of their effectiveness. In general the "happy sheet type of evaluation is ineffective as it is inclined to be subjective, and influenced by considerations which are actually outside the course itself. That is the course content could be poor or inappropriate but was delivered by a good instructor who managed the group well and they actually "enjoyed the experience but learnt little! Instead I have suggested that we owe it to the organisation to identify the competences that are really required by the individual to do his or her job well. These competences are the skills and underpinning knowledge which should be part of the company database and accessible through its Competence Management System.

The training intervention has to be designed to meet the needs of the organisation by satisfying the competence development requirements of the individual. By achieving this goal we have a rational and justifiable case for training and we are able to prove that training really does deliver tangible benefits, not least to the "bottom line. In this global marketplace it is essential to maintain a competitive advantage through the workforce; this is best achieved by an effective Performance Management system underpinned by a robust Competence Management System such as Sentrico.
Button