MONITORING POLICY OUTCOMES DDA 326: Public Policy Analysis

Discussant: Samuel O. Parami, MM., MPD.

Presentation Outline
A. Introduction B. Monitoring in Policy Analysis 1. Sources of Information 2. Types of Policy Outcomes 3. Types of Policy Actions 4. Definitions and Indicators C. Approaches to Monitoring 1. Social Systems Accounting 2. Social Experimentation 3. Social Auditing 4. Research & Practice Synthesis D. Techniques for Monitoring

The Power of Measuring Results

If you do not measure results, you can not tell

success from failure If you can not see success, you can not reward it If you can not reward success, you are probably rewarding failure If you can not see success, you can not learn from it If you can not recognize failure, you can not correct it If you can demonstrate results, you can win public support
Adapted from Osborne & Gaebler, 1992

Introduction

The consequences of policy actions are never

fully known in advance and, for this reason, it is essential to monitor policy actions after they have occurred.

In fact, policy recommendations may be

viewed as hypotheses about the relationship between policy actions and policy outcomes: if action A is taken at time t1, outcome O will result at time t2. 4

A. Monitoring in Policy Analysis

Monitoring is the policy-analytic procedure

used to produce information about the causes and consequences of public policies.

Monitoring, since it permits analysts to

describe relationships between policyprogram operations and their outcomes, is the primary source of knowledge about policy implementation.

Monitoring is primarily concerned about with

5 establishing factual premises about public policy.

A. Monitoring in Policy Analysis

Monitoring performs at least four major

functions:
1. Compliance: monitoring helps determine whether the actions of program administrators, staff, and other stakeholders are in compliance with standards and procedures imposed by legislatures, regulatory agencies, and professional bodies. 2. Auditing: monitoring helps determine whether resources and services intended for certain target groups and beneficiaries have actually reached them.

A. Monitoring in Policy Analysis

Monitoring performs at least four major

functions:
3. Accounting: monitoring produces information that is helpful in accounting for social and economic changes that follow the implementation of broad sets of public policies and programs over time. 4. Explanation: monitoring also yields information that helps to explain why the outcomes of public polices and programs differ. 7

A. Monitoring in Policy Analysis

1. Sources of information.

Information must be:

a. Relevant.

Macronegative versus micropositive.

b. Reliable.

Observations precise and dependable.

c. Valid.

Information about policy outcomes should

measure what we think it is measuring.

A. Monitoring in Policy Analysis

1. Sources of information.

Information on policy outcomes.

Current Opinion

Historical Statistics of the United States

Statistical Abstract of the United States

County and City Data Book

United States Census of Population by States

Congressional District Data Book

Census Use Study

Social and Economic Characteristics of Students Educational Attainment in the United States

National Opinion Research Center General Social Survey

Survey Research Center National Election Studies National Clearinghouse for Mental Health Information National Clearinghouse for Drug Abuse Information

Current Population Reports

A. Monitoring in Policy Analysis

1. Sources of information.

Information on policy outcomes.

National Clearinghouse for Drug Abuse Information

Current Population Reports

The Social and Economic Status of the Black Population in the United States
Female Family Heads Monthly Labor Review Handbook of Labor Statistics

National Criminal Justice Reference Service

Child Abuse and Neglect Clearinghouse Project National Clearinghouse on Revenue Sharing Social Indicators

Congressional Quarterly

State Economic and Social Indicators

10

A. Monitoring in Policy Analysis 1. Sources of information.

When information is not available from
existing sources, monitoring may be carried out by some combination of

questionnaires, interviewing, field

observation, and the use of agency records. 11

A. Monitoring in Policy Analysis

2. Types of policy outcomes.

a.

Consequences:

Policy outputs the goods, services, or

resources received by target groups and beneficiaries.

Policy impacts actual changes in behavior

and attitudes that result from policy outputs.

b.

Populations:

Target groups individuals, communities, or

organizations on whom policies and programs are designed to have an effect.

12

Beneficiaries groups for whom the effects of

policies are beneficial or valuable.

A. Monitoring in Policy Analysis 3. Types of policy actions. Policy actions have two major purposes. a. Regulation actions designed to ensure compliance with certain standards or procedures. b. Allocation actions that require inputs of money, time, personnel, and equipment.

Regulative and allocative actions may have consequences that are distributive or redistributive.
13

A. Monitoring in Policy Analysis

14

4. Types of policy actions. a. Policy inputs the resources time, money, personnel, equipment, and supplies used to produce outputs and impacts. b. Policy processes the administrative, organizational, and political activities and attitudes that shape the transformation of policy inputs into policy outputs and impacts.

A. Monitoring in Policy Analysis

POLICY ACTIONS ISSUE AREA Inputs
Criminal Justice. Expenditures for salaries, equipment, maintenance. Expenditures for sanitation workers and equipment.

POLICY OUTCOMES Outputs Impacts

Criminals convicted per 100,000 known crimes. Cleanliness of streets.

Processes

Illegal arrests Criminals as percentages arrested per of total arrests. 100,000 known crimes. Morale among workers. Total residences served. Welfare cases per social worker.

Municipal Services.

Social welfare. Number of Rapport with social workers. welfare recipients.

15

Standard of living of dependent children.

A. Monitoring in Policy Analysis

4. Definitions and indicators.

Variables versus constants. Definitions.

Constitutive definitions gives meaning to

words used to describe variables by using synonymous words.

Provide no concrete rules or guidelines for

actually monitoring changes.

Operational definitions gives meaning to a

variable by specifying the operations necessary to experience and measure it.

16

Indicators directly observable characteristics.

B. Approaches to Monitoring TYPE OF INFORMATION REQUIRED

Available and/or new information New information

APPROACH
Social systems accounting Social experimentation

TYPES OF CONTROL
Quantitative Direct manipulations and quantitative Quantitative and/or qualitative Quantitative and/or qualitative

Social auditing Research and practice synthesis

New information

Available information

17

B. Approaches to Monitoring
MANIPULABLE ACTIONS POLICY INPUTS In1 In2 . . . Ini POLICY PROCESSES P1 P2 . . . Pj CONTROLLED OUTCOMES POLICY OUTPUTS O1 O2 . . . Om POLICY IMPACTS Im1 Im2 . . . Imn

PC1 PC2 . . . PCp PRECONDITIONS

E1 E2 . . . Eq UNFORESEEN EVENTS

SES1 SES2 . . . SESr SIDE-EFFECTS AND SPILLOVERS UNCONTROLLED EFFECTS

UNMANIPULABLE CAUSES

18

1. Social Systems Accounting

An approach and set of methods that permit

analysts to monitor changes in objective and subjective social conditions over time.

19

1. Social Systems Accounting

The major analytic element of social

systems accounting is the social indicator.

Statistics that measure social

conditions and changes therein over time for various segments of a population. By social conditions, we mean both the external (social and physical) and the internal (subjective and perceptional) contexts of human existence in a given society.

20

1. Social systems accounting Representative social indicators

AREA INDICATOR
Persons in state mental hospitals

Health and illness

Public safety
Education Employment Income

Persons afraid to walk alone at night

High school graduates aged 25 and older Labor force participation by women Percent of population below the poverty line

Housing
Leisure and recreation Population Government and politics Social values and attitudes Social mobility

Households living in substandard units

Average annual paid vacation in manufacturing Actual and projected population Quality of public administration Overall life satisfaction and alienation Change from fathers occupation Air pollution index Scientific discoveries

21

Physical environment Science and technology

1. Social Systems Accounting

Must assume that the reasons for

indicator change are related to policy actions. This is not always the case.

Advantages.
Identify areas of insufficient information. When indicators provide reliable
information, possible to modify policies.

22

1. Social Systems Accounting

Limitations. Choice of indicators influenced by values

of analysts.

Social indicators frequently do not cover

manipulable policy actions.

Most social indicators are based available

data about objective (rather than subjective) social conditions.

Social indicators provide little information

23 about how inputs are transformed into outcomes.

1. Social Systems Accounting

24

1. Social Systems Accounting

25

1. Social Systems Accounting

Model Summarya Breaks in disaster patterns 1950 - 1971 1972 - 1988 1989 - 2006 a Predictors: (Constant), Number of years *p<.05; **p<.01; ***p<.005
Coefficientsa Unstandardized Coefficients B Std. Error 13.516 2.561 .340 .243 71.419 11.060 -1.551 .403 -6.745 22.133 1.218 .494 Standardized Coefficients Beta .322 -.705 .525

R R Square Sig. 0.322 0.103 0.705 0.497 *** 0.525 0.275 *

Breaks in disaster patterns 1950 - 1971 1972 - 1988 1989 - 2006

Model 1 1 1

(Constant) Number of years (Constant) Number of years (Constant) Number of years

t 5.277 1.400 6.458 -3.849 -.305 2.465

Sig. .000 .179 .000 .002 .764 .025

a. Dependent Variable: Total Disaster Declarations

26

1. Social Systems Accounting

27

2. Social Experimentation

Use of social indicators often leads to random

innovation rather than systematic change.

Social experimentation is the process of

systematically manipulating policy actions in way that permits more or less precise answers to questions about the sources of change in policy outcomes.

28

2. Social Experimentation

Characteristics and procedures.

Direct control of experimental treatment. Comparison (control) groups. Random assignment.

Goal: internal validity the capacity of

experiments and quasi-experiments to make valid causal inferences about the effects of actions on outcomes.
29

2. Social Experimentation

Threats to internal validity.

History. Maturation. Instability. Instrumentation and testing. Mortality. Selection. Regression artifacts.
30

2. Social Experimentation

Social experimentation is weakest

in the area of external validity or generalizability.

Neglects policy processes.

31

3. Social Auditing

One of the limitations of social systems

accounting and social experimentation is that both approaches neglect or oversimplify policy processes.

32

3. Social Auditing

Social auditing explicitly monitors

relations among inputs, processes, outputs, and impacts in an effort to trace policy inputs from disbursement to final recipient.

Social auditing helps to determine

whether ineffective outcomes are the result of inadequate inputs or processes that divert resources or services from intended target groups or beneficiaries.

33

3. Social Auditing

Processes monitored of two

main types:

Resource diversion (from target

or beneficiary groups).

Resource transformation
(changes in meaning of policy actions from administrator to recipient).

34

Social Audit Steps

The three phases of a social audit

Phase 1: Design and data collection

a. Clarify the strategic focus

b. Design instruments, pilot test

c. Collect information from households and key informants in a panel of representative communities
35

Social Audit Steps

Phase 2: Evidence-based dialogue and

analysis
a. Link household data with information from public services

b. Analyze findings in a way that points to action

c. Take findings back to the communities for their views about how to improve the situation 36 d. Bring community members into discussion of evidence with service providers/planners.

Social Audit Steps

Phase 3: Socialization of evidence for

public accountability

Work-shopping Communication strategy Evidence-based training of planners

and service providers

Media training Partnerships with civil society

37

4. Research and practice synthesis.

An approach to monitoring that involves the

systematic compilation, comparison, and assessment of the results of past efforts to implement public policies.

Two primary sources of available information.

a. Case studies of policy formulation and implementation. b. Research reports that address relations among policy actions and outcomes. 38

4. Research and practice synthesis.

Two methods.
a. Case survey method a set of procedures used to identify and analyze factors that account for variations in the adoption and implementation of policies. Requires case coding scheme, a set of
39

categories that capture key aspects of policy inputs, processes, outputs, and impacts.

Research and practice synthesis

Two methods.
b.
Research survey method a set of procedures used to compare and appraise results of past research on policy actions and outcomes.

Yields empirical generalizations about sources

of variation in policy outcomes, summary assessments of the confidence researchers have in these generalizations, and policy alternatives or action guidelines that are implied in these generalizations.

Requires the construction of a format for

extracting information about research reports.

40

4. Research and practice synthesis

Advantages.
a. Comparatively efficient way to compile and appraise an increasingly large body of cases and research reports on policy implementation.
b. The case survey method is one among several ways to uncover different dimensions of policy processes that affect policy outcomes. c. The case survey method is a good way to examine both objective and subjective conditions.

41

4. Research and practice synthesis

Limitations. All related to reliability and validity of

information.

42

4. Research and Practice Synthesis

Example

David H. Greenberg; Charles Michalopoulos; Philip K. Robins.

Do Experimental and Nonexperimental

Evaluations Give Different Answers about the Effectiveness of Government-Funded Training Programs?

43

4. Research and Practice Synthesis: Example

Abstract This paper uses meta-analysis to investigate

whether random assignment (or experimental) evaluations of voluntary government-funded training programs for the disadvantaged have produced different conclusions than nonexperimental evaluations.

Information includes several hundred

estimates from 31 evaluations of 15 programs that operated between 1964 and 1998. 44

4. Research and Practice Synthesis: Example

The results suggest that experimental

and nonexperimental evaluations yield similar conclusions about the effectiveness of training programs, but that estimates of average effects for youth and possibly men might have been larger in experimental studies.
45

4. Research and Practice Synthesis: Example

The results also suggest that variation

among nonexprimental estimates of program effects is similar to variation among experimental estimates for men and youth, but not for women (for whom it seems to be larger), although small sample sizes make the estimated differences somewhat imprecise for all three groups.
46

C. Techniques for Monitoring

INTERRUPTED TIMESERIES ANALYSIS REGRESSION DISCONTINUITY ANALYSIS

APPROACH

GRAPHIC DISPLAYS

TABULAR DISPLAYS

INDEX NUMBERS

CONTROLSERIES ANALYSIS

Social systems accounting Social auditing Social experimentation

x
x

x
x

x
x

x
x

x
x

o
x

Research and practice synthesis

47