The CIPP Approach to Evaluation

Owen Perkins
June 25, 2013

Management-Oriented Evaluation Approach

Meant to serve decision makers at all levels Good decision making is dependent on evaluative information A systems approach to evaluation Decisions are made about inputs, processes, and outputs Approach was a shift from program objectives to the decision(s) of program managers Evaluation of the decision(s) the administrator(s) must make

The CIPP Evaluation Model

Daniel Stufflebeam, supported a decision-oriented evaluation approach to assist administrators make good decisions His approach was developed in the 1960s His Evaluation method proposed that administrators face four kinds of decisions Context evaluation Input Evaluation Process evaluation Product evaluation Emphasis is on decision making by program administrators

Context Evaluation
Determine needs to define objectives of the program Asks what should we do? Collect data to determine goals of program Example: A standardized test is given to every student in 6th grade across a state. What are the objectives of this program? Raise level of education, standard to evaluate teacher performance, etc.
4

Input Evaluation
Determine resources, alternative strategies, and determine plan that has best potential to meet the needs of the program Asks how should we do it? Involves determining the steps and resources needed to accomplish goal or objective

Process Evaluation
To evaluate how well the plan has been implemented Asks are we doing it as planned? Continuously monitor the program, allows decision maker to see conflicts, strengths and weaknesses of program and materials, how well it is following the established plan, etc.

Product Evaluation
Determine program attainments What results were obtained? How well were needs reduced? How did the program work? Compare actual outcomes to anticipated outcomes Should the program being evaluated be continued, modified or dropped?

Stufflebeam suggested evaluators follow these general steps: 1. Focus the evaluation 2. Collection of Information 3. Organization of Information 4. Analysis of Information 5. Reporting of Information 6. Administration of the Evaluation
8

Focusing the Evaluation

Identify the level of decision making being addressed For each level of decision making analyze potential decisions Define criteria for each decision situation by specifying variables for measurement and standards for use in the judgment of alternatives Define policies within which the evaluator must operate

Collecting Information
Specify the source of information to be collected Specify the instruments and methods for collecting the needed information Specify the sampling procedure to be used Specify the conditions and schedule for information to be collected

10

Organization of Information
Provide a format for the information that is to be collected Designate a means for performing the analysis

11

Analysis of Information
Select the analytical procedures to be used Designate a means for performing the analysis

12

Reporting of Information
Define the audiences for the evaluation reports Specify a way to present information to audiences Develop format for evaluation reports Schedule the reporting of information
13

Administration of the Evaluation

Summarize the evaluation schedule Evaluate the potential of the evaluation design for providing information that is valid, reliable, credible, timely, and pervasive Specify and schedule means for periodic updating of the evaluation design Provide a budget for the total evaluation program

14

Strengths and Weaknesses of CIPP

Holistic approach:
Broad, vague, and summative Good or bad?

Idealized notion of process and program:

What the process should be rather than its actual form

Too top-down and managerial in approach

15

Imprisonment and Career Development:

An Evaluation of a Guidance Program for Job Finding
By: Gemma Filella-Guiu and Angel Blanch-Plana

16

Article Abstract
This study describes the evaluation results of PORO; a three level occupational guidance program Designed to improve offenders job finding possibilities and to reduce reoffending after imprisonment Used Stufflebeams CIPP model, considering the efficiency, effectiveness and impact of the program Evaluation outcomes and practical suggestions for career development applications addressed to prison populations are reported
17

PORO Definition
The Guidance Program for Job Search (PORO)
Go figure?

Wikipedia: Poro is a fraternal secret society of Sierra Leone and Liberia

18

Program Description
All internees (prisoners) are categorized by their individual judicial situation PORO comprises three levels of guidance to internees:
1. Internees given professional and occupational training in prison 2. Internees given ordinary parole permits 3. Has two distinct sections:
Internees go into community on daily basis Probation and freed internees in community

19

CIPPs Use in Study

Context Evaluation is the delineation and specification of the programs environment, the population and sample of individuals to be used, and the program objectives Input Evaluation identifies and assesses system capabilities, determines methods that meet objectives and needs, and studies the relations between cost and potential benefits of each proposed method Process Evaluation helps to provide feedback to the teachers on progress of the program Product Evaluation allows decisions to be taken regarding the continuity, termination or modification of the program
20

Context Evaluation
What should we do? Collect data to determine goals of program

21

Context Evaluation: Collect Data

145 people in the experimental group 51 in the control group (N = 196) 11% females and 89% males Between 25 and 35 year old Serving sentences of less than 4 years. Specialties taken in the occupational courses were car mechanics, graphic arts, industrial dressmaking, building, electricity, binding, carpentry and computing

22

Context Evaluation: Determine Goals

The main objectives of this intervention were:
a) To improve individual competence in job searching b) To facilitate the acquisition of useful job search knowledge and skills c) To increase professional self-esteem

The accomplishment of these specific aims is as fundamental for enabling ex-convicts to adjust to the job market and the community on a long-term basis
23

Input Evaluation
How should we do it? Determine resources, alternative strategies, and determine plan that has best potential to meet the needs of the program

24

Input Evaluation: Determine plan that has best potential to meet the needs of the program
The contribution of an expert group was requested to assess the design and methodological validity of PORO in order to achieve the best possible adaptation to a confinement context The potential benefits that might be obtained were rewarding enough considering the assignment of operational, material and human resources
= Cheap Program

25

Process Evaluation
Are we doing it as planned? To evaluate how well the plan has been implemented

26

Process Evaluation: How well has the plan been implemented?

Feedback from all course teachers and 52 randomly selected participants on the program were used to evaluate the process dimension No serious difficulties arose during the teaching program on the majority of occupational courses Internees were not very enthusiastic about the program in its initial stages. However, in the end, this lack of motivation was gradually overcome during the program development, final opinions being highly positive
27

Product Evaluation
What results were obtained? How well were needs reduced? How did the program work? Determine program attainments

28

Product Evaluation: Determine Program Attainments

Increase in job knowledge and skills scores Increase in professional self-esteem scores Based on the findings it should be expected that other internees would reach these same employment rates when their sentences have been served The program had a high level of efficiency:
The material, human and operational resources used in the [program] development did not pose any additional expense

The program did not have any adverse effects:

Undesirable contingencies were not noted once the investigation finished, nor were unforeseen relevant consequences due to the occupational guidance [program] detected

29

Conclusion
The resultsgive further support to the idea that career intervention addressed at imprisoned people in adverse social and economic circumstancesis of paramount importance for improving the employment possibilities of people whose [behavior] and lack of skills are troublesome to them and to society
30

Conclusion, contd
This case study is an excellent example of the CIPP evaluation approach Each step of the approach was carefully considered

31

The End
GO HOKIES!
June 25, 2013