Future Trends in Evaluation

Live Webinar 1st July 2010

This series of webinars are based on the book published by UNICEF in partnership with key international institutions
Authors:
40 global evaluation leaders Partnership:
UNICEF, WB, UNDP, WFP, UNIFEM, IDEAS, IOCE, DevInfo

Available for free download at www.mymande.org

2010 Monthly webinars:

Saraswathi Menon
Finbar OBrien Michael Quinn Patton Marco Segone

The Role of the United Nations in Fostering 22nd June 2010 National Ownership and Capacities in Evaluation
Future trends in evaluation. Moving from policies to results by developing national capacities for countryled monitoring and evaluation systems Evaluating policies and their results The role of policy analysis in over-coming the implementation challenge Human Rights and gender in evaluation The professionalization of evaluation Professionalizing development evaluators. Travelling the road to results Joint evaluation of the role and contribution December 2010 of the UN system in South Africa. Lessons learned October 2010 November 2010 September 2010 1st July 2010

Caroline Heider Craig Russon

Belen Sanz Jean Quesnel

Linda G. Morra Imas

Oscar A. Garcia Angela Bester

2011 Monthly webinars:

Michael Bamberger Institutionalizing impact evaluation. A key January 2011 element in strengthening country-led monitoring and evaluation systems Exploring effective strategies for facilitating evaluation capacity development Use of evaluation training in evaluation capacity building Utilization-focused evaluations Empowerment evaluation Collaborative evaluation. Creating environments to engage stakeholders in evaluation March 2011 April 2011 February 2011 Hallie Preskill

Alexey Kuzmin

Michael Quinn Patton David Fetterman Rita OSullivan (TBC)

Indran Naidoo

The monitoring and evaluation in South Africa. Many purposes, multiple system
Building a results-based management and evaluation system in Colombia National monitoring and evaluation system in Sri Lanka. Experiences, good practices, challenges and the way forward Policies, institutions and personalities. Lessons from Ugandas experience in monitoring and evaluation

May 2011

Manuel Fernando Castro Diego Dorado (TBC) Velayuthan Sivagnanasothy

June 2011

July 2011

David Rider Smith

September 2011

The recording of the previous webinars are available at www.mymande.org

The recording of the previous webinars are available at www.mymande.org

The recording of the previous webinars are available at www.mymande.org

Agenda
9h30 9h35 9h35 9h50 Welcome and introduction Michael Quinn Patton, Founder and Director, Utilizationfocused evaluation, and former President of the American Evaluation Association Marco Segone, Systemic management, UNICEF Evaluation Office

9h50 10h05

10h05 10h25 Questions and Answers Moderator: Abigail Taylor, Knowledge Management Specialist, UNICEF Evaluation Office 10h25 10h30 Wrap-up

Questions and Answers

Keynote speakers
QUINN PATTON, Michael former President of the American Evaluation Association and author of six evaluation books including Utilization-Focused Evaluation; Developmental Evaluation: Applying Complexity Concepts to Enhance Innovation and Use; and Qualitative Research and Evaluation Methods.

Trends in EVALUATION
UNICEF Webinar
Michael Quinn Patton

Michael Q. Patton,

Evolving Understandings
I keep changing what I said. Any person who is intellectually alive changes his ideas. If anyone at a university is teaching the same thing they were teaching five years ago, either the field is dead, or they havent been thinking. Noam Chomsky
The Professor Provaocateur, The New York Times Magazine, Nov. 2, 2003: 13.

Michael Q. Patton,

TRENDS
1. Globalization of the Profession IOCE IDEAS
Challenge: Non-professionals doing evaluation Diversity of evaluation
Michael Q. Patton,

Evaluation Standards and Culture

Utility Feasibility Propriety Accuracy
NDE: International Perspectives on Evaluation Standards, No. 104, 2004.

Michael Q. Patton,

1. Globalization of the Profession

2. Evaluation as a Transdiscipline Evaluative Thinking IDRC

Michael Q. Patton,

1. Globalization of the Profession 2. Evaluation as a Transdiscipline

3. Increased political interest in accountability, performance indicators, and transparency

Michael Q. Patton,

SPEAKING TRUTH TO POWER

19

20

General oe9S6Sgf result ACfU3U1E PP1 R Search Books

S h a k e H a n d s w i P t r A h e b v o t i u h e t e w t h D t i e h s v i i s b l o b o B o k y o k R o m o D a l l a i r e

This is a preview. The total pages displayed will be limited. L

21

Trends
4. Growing emphasis on evaluation capacity-building and skill development.

Essential skills beyond just methods Opportunities: IPDET

The Evaluators Institute Professional associations

Michael Q. Patton,

Trends
5. Continuing debate about what constitutes methodological rigour

Michael Q. Patton,

GOLD STANDARD DEBATE: METHODOLOGICAL APPROPRIATENESS

not Methodological orthodoxy or rigidity

Mixed Methods
Matching methods to evaluation questions, purposes, and resources;
Contingency-based evaluation

Michael Q. Patton,

Trends
6. Systems thinking and complexity concepts as frameworks for evaluation

Michael Q. Patton,

AEA monograph: Using Systems Concepts in Evaluation edited by Bob Williams and Iraq Iman

Michael Q. Patton,

Seeing Through A Complexity Lens

Michael Q. Patton,

Developmental Evaluation:
Applying Complexity Concepts to Enhance Innovation and Use
Michael Quinn Patton 2010, Guilford Press

29

Complexity Concepts
Emergence Openness Dynamic High uncertainty Nonlinear: Small actions, large impacts Co-evolution Adaptation Unanticipated consequences

Michael Q. Patton,

Unanticipated developments as an expected aspect of innovation and

a focus and challenge for evaluation.

Michael Q. Patton,

Beyond just formative and summative to Developmental Evaluation

Michael Q. Patton,

Development
not just Improvement. Core questions: What is developing? What is being developed?
Michael Q. Patton,

Challenge
Matching the evaluation process and design to the nature of the situation:

Contingency-based Evaluation

Michael Q. Patton,

Contingency-based Evaluation
Situational analysis & responsiveness Context sensitivity Clarify and focus on intended users: stakeholder analysis Clarify and focus on intended uses Methodological appropriateness Criteria for evaluating the evaluation: credibility, meaningfulness, utilty

Michael Q. Patton,

Summary: Six Trends

1. Globalization of the profession

2. Evaluation as a transdiscipline. 3. Increased political importance of M & E 4. Growing emphasis on capacity-building and essential skills 5. Debate about methods: Gold standard debate 6. Using systems thinking and complexity concepts

Michael Q. Patton,

37

References
Developmental Evaluation: Applying Complexity Concepts to Enhance Innovation and Use. Michael Quinn Patton, 2010, Guilford Press. http://www.guilford.com/cgibin/cartscript.cgi?page=pr/patton.htm&dir=resear ch/res_eval&cart_id=435936.20323 Utilization-Focused Evaluation, 4th ed., Michael Quinn Patton, Sage Publications, 2008.
Michael Quinn Patton

References
Future Trends in Evaluation. Pp. 44-57 in From Policies to Results: Developing capacities for country M & E systems, Marco Segone (ed.) UNICEF, 2010.

Michael Q. Patton,

Keynote speakers
SEGONE, Marco is responsible for the decentralized evaluation portfolio at the UNICEF Evaluation Office, serves as a Senior Advisor to the IOCE Board and was IOCE Vice-President.

National evaluation capacity development

Key elements for a conceptual framework

Marco Segone, Systemic Strengthening, UNICEF Evaluation Office

WHY

In line with new international development paradigm

National ownership and capacity development: the key ingredients to development

Managing for results

Mutual accounta bility

Harmoni zation

Alignment

Ownership

Paris Declaration on Aid Effectiveness

 the UN General Assembly

national Governments have the primary responsibility for coordinating external assistance and evaluating its contribution to national priorities mandated the United Nations system to promote national ownership and capacity development, and to make system-wide progress in harmonizing evaluation practices

UNICEF

 and UN Agencies Boards request.

2009 UNICEF ExBoard:

emphasizes that programme countries should have greater ownership and leadership in the evaluation of all forms of assistance underlines the importance of increasing the participation of national counterparts and strengthening national capacity in evaluation encourages UNICEF to use national evaluation systems, where available, to continue to incorporate capacity-building mechanisms into programme design and implementation and ensure that evaluations are responsive to national demand

UNICEF

HOW

A Systemic approach to Capacity Development.

Individual Level
(skills, knowledge, experience)

UNICEF

Individual Level
Demand side
Capacity to strategically plan evaluations, and to identify the key evaluation questions Capacity to manage evaluation for independence and credibility Capacity to use evaluation

Supply side (capacity to conduct an evaluation):

Behavioural independence Independence of mind & integrity Knowledge and respect of evaluation standards Agreed evaluation processes & products Professional competences Formal education (Masters) Specialized training Professional Conferences and meeting On the job training (country-led evaluations) Community of Practices and networking

A Systemic approach to Capacity Development.

Institutional Level
(policies, procedures, frameworks)

Individual Level
(skills, knowledge, experience)

UNICEF

Institutional Level
Evaluation culture
Set of values and attitudes supporting evaluative (critical) thinking within an organization Institutional commitment to learning from evaluation, support evidence-based policy debate and demand for accountability. Individual more self-directed learners and use information to act; take higher risks but also develop a greater sense of personal accountability and responsibility; consult, coach, and support each other more. Protective culture (Remove repercussions on careers) Understanding of the foundations and principles of Monitoring and Evaluation Evaluation policies Independence & adequacy of budget Institutional endorsement of standards In-built Quality Assurance systems

Institutionalizing independence, credibility and utility

Work programme and budget

Conduct of evaluations

A Systemic approach to Capacity Development.

Institutional Level Enabling Environment

(policies, legislation, power relations, social norms) (policies, procedures, frameworks)

Individual Level
(skills, knowledge, experience)

UNICEF

Enabling Environment
Public administration committed to manage for results and accountability
Transparency Results-based public budgeting Evidence-based policy making

Strong civil society

rights holders able to demand for and monitor quality of public services

Strong national evaluation association

foster indigenous demand for M&E, and strengthen indigenous supply

We need a mind shift to do things differently...

We should always aim at strengthening National ownership and leadership, and not undermining it: Selecting topics of mutual interest Implement evaluations jointly with Governments, as a first step towards country-led evaluation Hiring local experts to the maximum extent possible Not assuming there are weak evaluation capacities, even if none is immediately apparent Co-ordinating with other UN agencies and key international stakeholders Invest in the country-led process, even if it may require additional efforts and less control

Questions and Answers

The discussion goes on at www.mymande.org

The discussion goes on at www.mymande.org

The discussion goes on at www.mymande.org

Next webinar, September, same time

Caroline Heider Director of the Office of Evaluation of the World Food Programme (WFP) and Vice Chair of the UN Evaluation Group

Craig Russon Senior Evaluation Officer, International Labour Organization (ILO)