Monitoring, Evaluating, and Improving: An Evidence-Based Approach to Achieving Development Results that Matter!

MONITORING, EVALUATING, AND IMPROVING

An Evidence-Based Approach to Achieving Development Results that Matter!

Dr. Samuel Taddesse

Copyright © 2017 Dr. Samuel Taddesse.

This book is a work of non-fiction. Unless otherwise noted, the author and the publisher make no explicit guarantees as to the accuracy of the information contained in this book and in some cases, names of people and places have been altered to protect their privacy.

All rights reserved. No part of this book may be reproduced, stored, or transmitted by any means—whether auditory, graphic, mechanical, or electronic—without written permission of the author, except in the case of brief excerpts used in critical articles and reviews. Unauthorized reproduction of any part of this work is illegal and is punishable by law.

ISBN: 978-1-4834-7108-2 (sc)

ISBN: 978-1-4834-7107-5 (e)

Library of Congress Control Number: 2017909718

Because of the dynamic nature of the Internet, any web addresses or links contained in this book may have changed since publication and may no longer be valid. The views expressed in this work are solely those of the author and do not necessarily reflect the views of the publisher, and the publisher hereby disclaims any responsibility for them.

Any people depicted in stock imagery provided by Thinkstock are models, and such images are being used for illustrative purposes only.

Certain stock imagery © Thinkstock.

Lulu Publishing Services rev. date: 6/30/2017

CONTENTS

Acknowledgement

Introduction

Acronyms and Initialisms

CHAPTER 1 Anatomy of a Robust M&E System

Overview

What Is a Project M&E System?

How Does an M&E System Help?

Components of a Robust M&E System

Participatory Deliberative Discussion and Decision Making

M&E Core Values and Code of Ethics

Conclusion

CHAPTER 2 Component 1: Diagnostic Assessment

Overview

The Power of DA

Conclusion

CHAPTER 3 Component 2: The Results Framework

Overview

The Results Framework

Articulating the Desired Results Reflected in the RF

Critical Assumptions

Conclusion

CHAPTER 4 Component 3: Selecting Project Activities

Overview

Project Activity Selection, Implementation, and Tracking

Conclusion

CHAPTER 5 Component 4: Monitoring and Reporting Project Progress

Overview

Defining Performance Indicators

Collecting Indicator Data

Data Collection Tools

Steps for Collecting Data

Assessing Progress

Putting It All Together

Conclusion

CHAPTER 6 Component 5: Stocktaking, Lessons Learning, and Improving

Overview

Elements of the SLLI Exercise

Conclusion

CHAPTER 7 Putting It All Together: Building a Cost-Effective M&E System

Overview

Steps for Building a Robust and Cost-Effective M&E System

Designing an Appropriate Performance Measurement Method

Conclusion

CHAPTER 8 Hitting the Ground Running

Overview

Funding the M&E Work Stream

Conclusion

CHAPTER 9 In Summary

APPENDIX 10 Principles of Value for Money Analysis

APPENDIX 11 An Approach for Assessing Organizational Capacity

APPENDIX 12 Template for Summarizing Diagnostic Analysis Findings

APPENDIX 13 Results Framework Template

APPENDIX 14 Project Activity Matrix

APPENDIX 15 Results Performance Management Plan Template

APPENDIX 16 Stocktaking and Lesson-Learning Template

APPENDIX 17 Stakeholders Analysis Template

Glossary

References

ACKNOWLEDGEMENT

The drafting of this book began in the 1990s when I was engaged in the evaluation of different USAID-funded programs around the world. The foundation of my M&E knowledge began developing when I worked as a consultant for Management Systems International (MSI) to work on the PRISM project. The primary task of PRISM was to assists USAID Missions develop strategic objectives, intermediate results and performance indicators for their Country Strategic Assistance Programs. The 1993 Government Performance Reporting Act (GPRA) promoted and fostered managing for results principles. In the process, the M&E practice required identification of measurable results and the use of quality performance indicators for which data was collected through observations, surveys, focus group discussions and key informant interviews.

Since 1989 I have conducted reviews, assessments, and evaluations of different projects in different sectors and countries for numerous consulting firms. The ability to work across different sectors including economic growth, agriculture, health, basic and higher education, transportation, water and environment protection has rounded out my knowledge of problem analysis and managing for results. Also, the various discussions and feedbacks received from evaluation team members, USAID and contract managers were invaluable. I wish to thank all my employers and the team members I worked with, who are too numerous to name. I also thank Lulu Publishing for providing professional editorial services.

Finally, I would like to thank my wife Menbere Tsheay for being my sounding board and for her patience allowing me a quiet time to work on this book. I also would like to thank my son David Samuel Taddesse for his graphics expert assistance. Needless to say, I take full responsibility for errors and omissions. Readers comments are most welcome. Please send your comments and feedback or questions via my email address: samtad007@gmail.com.

INTRODUCTION

This book is about achieving and measuring results. It presents an approach to understanding a development problem: What results are needed to resolve the issue, and how will success be measured? It advocates identifying the root causes of a problem and dealing with them to address the development challenges for better human outcomes.

Moreover, the book makes a case for inclusion of stakeholders, including policy makers and people affected by the development challenges, as key participants in the problem-analysis and problem-solving processes. The book also provides extensive discussion of how to define and measure indicators and how to select and use appropriate data sources and data collection methods and tools, taking data quality into account.

Around the world, nations are faced with income inequality that has exacerbated poverty and civil conflict; wrongheaded economic development policies; and natural disasters that continue to displace and decimate communities and cause untold misery. Since the end of World War II, multilateral organizations, bilateral donors, and national governments have spent billions of dollars each year to address these challenges. There has been progress in solving some of them; however, many interventions have failed because of a lack of proper understanding of the root causes and the implementation of faulty policies and programs.

Many national and donor programs only attempt to treat the symptoms, and as a result, the same socioeconomic and political challenges keep recurring. Because of tight time frames and budgets, the international development experts who are contracted to deal with these problems are expected to hit the ground running without a clear understanding of the challenges they are supposed to help solve. The outcomes of such interventions have been very disappointing.

For example, in a recent donor program I was involved with, the team was asked to address tax compliance issues as well as ethics, corruption, and mismanagement of public resources. Instead of spending time understanding the tax compliance and corruption issues, the team began to implement activities the team leader thought would show that the program was doing something. Some of the activities executed, while useful, did not address the core issues the customers wanted addressed. By the middle of the program’s life, tax compliance had not improved, and corruption in the government and private sectors continued as usual. Failure to understand the root causes of the tax compliance problem and the high level of corruption prevented the program from designing effective interventions.

A related programmatic issue is the practice of trying to adapt an intervention from one continent to another and from one country to another—misguided by best practice theory, as propagated by the International Monetary Fund (IMF), the World Bank, United Nations Development Program (UNDP), and many bilateral programs. Poverty alleviation strategies that worked in Country A do not necessarily work in Country B, because of the differing context and circumstances of the two countries. Socioeconomic interventions must be tailored to the unique environment of the target country and community.

In many situations, social and economic challenges are expressed in general terms and fail to articulate real problems and desired outcomes. The desired results are articulated without feedback from the affected population, leading to the implementation of ineffective interventions. Interventions to resolve socioeconomic challenges are designed blindly, without supporting valid empirical data and information. For example, a central planner in one drought- and famine-stricken country sent out a policy directive to plant trees on the hillside and direct the rainwater overflow toward the river. The policy directive could not be followed, as there was no hill or river in the district. The reforestation program could not be implemented.

Often, after an intervention has been implemented, policy makers and public officials rush to claim success without thoroughly examining the evidence. In effect, the practice of monitoring and evaluation (M&E) is given lip service. Policy makers and civil servants have an aversion to M&E instead of valuing it as a tool for sound decision making.

In this book, I present an M&E approach for dealing with social, economic, and political challenges holistically. Applying the approach will minimize the pitfalls and maximize the chances of successfully resolving the socioeconomic and political problem. The book draws on my thirty-five years of work experience with government and donor programs.

Structure of the Book

Chapter 1 introduces a robust M&E system; chapters 2, 3, 4, 5, and 6 discuss each component of a robust M&E system; and chapter 7 puts it all together. Chapter 8 discusses how project contractors can quickly start up an effective M&E system in the field, and chapter 9 summarizes the discussion presented in the book. Several appendices are attached. The book also includes an extensive glossary of terms used in the M&E practice area and a list of relevant references.

ACRONYMS AND INITIALISMS

Chapter 1

ANATOMY OF A ROBUST M&E SYSTEM

This chapter highlights key aspects of a robust monitoring and evaluation system and provides a basic code of ethics for M&E practitioners. For several years, the American Evaluation Association has debated the issue of how to prevent or minimize the distortion of evaluation and assessment findings and conclusions by hidden agendas of evaluators as well as of funding agencies and national governments. The outcome of this discussion has been the adoption of a code of ethics and guiding principles for evaluators.

Overview

Each year, billions of dollars are spent to develop, implement, and use monitoring, evaluation, and reporting systems to improve development outcomes and service-delivery performance to citizens. Under the Government Performance and Results Act (GPRA) of 1993, federal agencies in the United States are required to plan, set objectives and performance targets, and link results to resources.¹ In 2010, the US Congress passed the GPRA Modernization Act, which requires federal agencies to identify and describe major management challenges and how they plan to address them in their agency performance plans.²

Major management challenges include programs or management functions that are vulnerable to waste, fraud, abuse, mismanagement, and other problems that seriously affect the ability of an agency to carry out its mandate effectively. In its June 2016 report, the Government Accountability Office, after reviewing twenty-four agencies, found seven management functions cited frequently as major challenges:

1. Acquisition and procurement

2. Contract management and contractor oversight

3. Cybersecurity

4. Financial management

5. Human capital management

6. Addressing improper payments

7. Real property and asset management

Box 1.1: The Power of Results-Based M&E

What gets measured gets done:

If you don’t measure results, you can’t tell success from failure.

If you can’t see success, you can’t reward it.

If you can’t reward success, you’re probably rewarding failure.

If you can’t see success, you can’t learn from it.

If you can’t recognize failure, you can’t correct it.

If you can demonstrate results, you can increase the chances of winning future projects.

Source: Adapted from David Osborne and Ted Gaebler, Reinventing Government.

Addressing such challenges enhances the effectiveness and efficiency of these agencies, resulting in improved public services.³ At the subnational level, state agencies and local governments also use M&E tools to improve basic public-service delivery. Similarly, across the world, national and subnational governments are increasingly using M&E tools to identify and resolve major management challenges and thereby strengthen and improve public-service