Training:

What is the Impact

on the Bottom Line?

Robert J. Gray
QA 7504
September 22, 2001
1

Order of Presentation
Introduction
The Need to Evaluate Training from a Corporate
Perspective
What is e-Learning?
The State of the e-Learning Business
Evaluating the Effectiveness of Training
Kirkpatrick Model
Phillips Model

Applications of e-Learning Evaluation Using ROI

Conclusion

Introduction
The New Corporate Realities of the Global
Economy

International competition
New technology
Emphasis on productivity, performance, and return
on investment
Needs of the new learner are challenged
New leaders with expanded competencies
Rethinking, restructuring, and reengineering of
education and training is required

Introduction
Corporations spend millions of dollars for
training
Corporations are concerned about
profitability
Training departments are cost centers
Training gets cut during expense reduction
programs
Imperative to quantify the value of training

The Need to Evaluate Training

from a Corporate Perspective
$58.6B budgeted for training in 1997
Top challenge determining the impact
of training on financial performance
How are you helping the business?

The Need to Evaluate Training

from a Corporate Perspective
The Language Barrier
Training Focus
Training content

Business Executive
Focus
Business results

Staying on budget

Maximizing profit

Building
competencies
The latest training
techniques

Solving customer
problems
The latest
competitive threat

The Need to Evaluate Training

from a Corporate Perspective
Change of Focus
Customers

What are they trying to

accomplish?
Products
What solutions do customers
demand?
Operational How do we ensure high level of
Efficiency
productivity?
Measurement How do we track our performance,
return, opportunities to improve?
7

The Need to Evaluate Training

from a Corporate Perspective
Paradigm Shift
From

To

Event oriented

Process oriented

Domain is the classroom

Domain is the organization

Focus on learners

Focus on workplace

Analyzes instructional needs

Analyzes performance needs

Develops training
interventions
Implements training solutions

Develops performance
interventions
Implements performance
solutions
Evaluates bottom-line results

Evaluates learning and

behavior

What is e-Learning?
Enterprise Learning?
Electronic Learning?
Everywhere Learning?
Excellent Learning?
Experience Learning?

What is e-Learning?
Characteristics

Dynamic
Up-to-date and relevant
Real-time learning environment
Highly available
Flexible
Learner control

10

State of the
e-Learning Business
Web-Based Education Commission Report to the
President and the Congress
Recommendations
Make Internet available and affordable
Provide training and support to educators and
administrators
New research framework on how people learn
High quality online educational content
Sustain funding
e-Learning agenda

11

State of the
e-Learning Business
U.S. Corporate e-Learning Market
1997 - 2003E
$11,415

Dollars in Millions

$12,000
$10,000

$7,113

$8,000
$6,000

$4,053

$4,000
$2,000

$234 $550

$2,222
$1,114

$0
1997 1998 1999 2000 2001 2002 2003
Year
12

State of the
e-Learning Business
e-Learning Projected Market Share
90%
Percent Market Share

80%
70%
60%

e-Learning

50%
40%

Classroom
Learning

30%
20%
10%
0%

1999

2003
Year
13

State of the
e-Learning Business

Revenue in billions

$7.0

Growth of the U.S. Internet-Based Training

Market by Training Product
$6.1

$6.0

$5.3

$5.0
$4.0

$4.1

$3.0

$3.1

$2.7

$2.0

$1.7

$1.0
$0.0

$0.9
$0.4
$0.1

1998

$0.2

1999

IT Training
Soft Skills Training

$1.4

$0.6

2000

2001

Year

2002

2003
14

Evaluating the
Effectiveness of Training
Level
1 Reaction

The Kirkpatrick Model

Type of
Focus
Measure

Satisfaction

2 Learning Learning,
Attitudes
3 Behavior Behavioral
change
4 Results Results

Learner satisfaction with the

program
Principles, facts, techniques
Changes in job behavior
Tangible results cost,
quality, quantity

15

Evaluating the
Effectiveness of Training:
TM
The ROI Process
EVALUATION
PURPOSES

TABULATING
PROGRAM
COSTS

EVALUATION
INSTRUMENTS

COLLECTING
POST PROGRAM
DATA

ISOLATING
THE EFFECTS
OF THE
PROGRAM

EVALUATION
TIMING
EVALUATION
LEVELS

CONVERTING
DATA TO
MONETARY
VALUE

CALCULATING
THE RETURN ON
INVESTMENT

IDENTIFYIN
G
INTANGIBLE
BENEFITS16

Evaluating the Effectiveness

of Training: The ROI ProcessTM
Data Collection Tools and Techniques
Surveys

Tool

Purpose

Questionnaires

Determine degree of use of various

aspects
Specific applications used

On-the-job observation

Actual application and use

Interviews

Extent of utilizations

Focus Groups

Group application of learning

Program assignments

How skills and knowledge are utilized

Action plans

Obtain evidence of success

Performance contracts

Document expectations

Follow-up session

Obtain further input

Performance monitoring

Examine operational data

17

Evaluating the Effectiveness

TM
of Training: The ROI Process
Isolating the Effects of Training
Method

Purpose

Control Group

Performance comparison

Trend Lines

Comparison to expectations

Forecasting Model

Comparison to forecasts

Participants Estimate

Learners estimate percentage improvement

Supervisor Estimate

Supervisor estimates percentage improvement

Management Estimate

Managers estimate percentage improvement

Experts Estimate

Subject Matter Expert estimates percentage

improvement
Subordinates estimate percentage improvement

Subordinates Identify
Changes in Work Climate
Identification of Other
Influencing Variables
Customer input

Factor out non-training variables

Influence on purchase decisions

18

Evaluating the Effectiveness

of Training: The ROI ProcessTM
Converting Data to Monetary Values
Tool

Purpose

Convert data to profit

contribution or cost savings
Calculate cost of quality

Determine unit contribution to profit or

cost reduction
Convert quality improvement into dollars

Use wages and benefits for the

value of time
Historical tools

Convert time saved into dollars

Experts

Provide estimates

External databases

Research, government, industry estimates

Participants estimates

Learner estimates based in experience

Supervisors estimates

Supervisor estimates based on observation

Manager estimates

Manager estimates based on observation

HRD staff estimates

HR professionals estimates based on

observation

Can be used for specific variables

19

Evaluating the Effectiveness

TM
of Training: The ROI Process
Cost Calculation Elements
Program design and development
Program materials
Instructor preparation and delivery time
Facilities
Travel, lodging, meals for participants
Salaries and benefits
Administrative and overhead costs
Needs assessment and evaluation costs

20

Evaluating the Effectiveness

TM
of Training: The ROI Process
Formula 1: Cost/Benefit Ratio (CBR)
CBR = Program Benefits/Program Costs
Formula 2: Return on Investment Percentage (ROI%)
ROI(%) = Net Program Benefits
Program Costs

X100

21

Evaluating the Effectiveness

TM
of Training: The ROI Process
Soft Benefits
Increased job satisfaction
Increased organizational commitment
Improved teamwork
Improved customer service
Reduced complaints
Reduced conflicts

22

Evaluating the Effectiveness

TM
of Training: The ROI Process
Issues with the ROI ProcessTM
Use of estimates
Soft data
Intangible benefits
Use of industry studies and vendor
information
Ability to isolate training effects
Lack of standards
23

Evaluating the Effectiveness

TM
of Training: The ROI Process
Caveat Emptor
Estimating costs of training projects is a black art only slightly more
precise than reading tea leaves or bird entrails.
These spreadsheets offer a frameworkor just a framework for a
frameworkfor estimating costs.
The only thing we guarantee about them is that they are wrong for your
particular project. But they can suggest ways you can create spreadsheets
to estimate costs for your own project.

24

Applications of e-Learning
Evaluation using ROI
Internal Benefits
Which of the following benefits does your company expect
to gain from using e-Learning for training internally?
Le ss trav el time and costs
Better conve nie nce and se rvice
M ore training about company products and
procedures
M ore access to company information
Better-quality training
Training outside work hours
New custome rs
New re ve nue source
0%

10%

20%

30%

40%

50%

60%

70%

Percentage response

80%

90%

100%

25

Applications of e-Learning
Evaluation using ROI
Early Returns
When do you expect to see a
return on investment from e-Learning?

More than 2 years

6%
1 year to 2 years
15%

Already using ROI

46%

6 months to 1 year
17%
3 to 6 months
8%

Within 3 months
8%

26

Applications of e-Learning
Evaluation using ROI
Cisco Systems Task Force Key Metrics Categories
Satisfaction
Penetration and Usage
Effectiveness
Relevance
Cost

27

Evaluation using ROI

Effectiveness Measures
Pre-performance and post-performance per
offering
With and without intervention performance
Time to completion
Time to competency
Change in ability to perform job
responsibilities against time spent learning

28

Applications of e-Learning
Evaluation using ROI
Do the Cisco measures evaluate ROI?
Pre-test and post-test scores per offering
NO
Time to completion
NO
Time to competency
NO
Change in ability to perform job responsibilities
against time spent learning
YES
29

Applications of e-Learning
Evaluation using ROI
Six Significant Research Findings about e-Learning
ROI impact analysis driven by clients funding
projects
Level 1 Reaction higher for classroom
instruction
e-Learning as effective as face to face for
Level 2 - Learning
Levels 1 through 5 can be applied to e-Learning
ROI studies focus on cost reduction alone
Evaluation can result in savings of money and
time

30

Applications of e-Learning
Evaluation using ROI
Phillips - Issues with e-Learning evaluation
Difficult to gather data from remote
learners
Lack of a framework for data collection
Lack of standards
Process models
Operating standards
Approaches to implementation
31

Conclusion
ROI evaluation is in its early stages
Use is somewhat limited
Difficult to obtain data
Use of soft data
Lack of standardized methods and
criteria
Difficult to integrate data from many
sources
Companies are skinning their knees
32

Conclusion
New evaluation methods
Time to competency
Time to market
Time to understanding

33

Conclusion
Success Stories
Reduced injuries and
damages
Reduced time to train
Reduced time develop new
hires
Reduced distribution time
of training materials
Reduced time to market
Reduced certification time

Created online certification

programs
Used website learning
Used online discussion topics
Taught sales skills and
product knowledge online
Developed online resource
center
Reduced training expenses
34

Conclusion
e-Learning is growing rapidly
An attractive solution to reduce costs
Bottom-line benefits have not been
conclusively demonstrated
Need standard tools and
methodologies

35

Conclusion
Undeniably, e-Learning cuts the costs of travel,
facilities, administrative overhead, duplication of
effort, and more importantly, the opportunity cost
of people away from the job in times of great need.
There's no doubt that e-Learning can be rolled out
fast. The time required to roll out a new product
globally can shrink from months to hours. But all
the cards aren't in yet.
e-Learning is too new to have produced hard
evidence of learning gains. e-Learning's top-line
upside is speculative; its bottom-line savings are on
36
more solid ground.

Questions?

37