Multilateral Effectiveness and Financing Meeting June 20th London A Selection of key papers

Multilateral Effectiveness and Financing Meeting, 20 June 2007, London 1

Foreword by Mark Lowcock, Director General, Policy and International, Department for International Development (DFID)
International organisations have a critical role to play in helping developing countries achieve the Millennium Development Goals (MDGs). Three institutions the European Commission, the World Bank and the United Nations already account for around 30% of global aid. Multilaterals have a strong comparative advantage in the fight against poverty because of the global reach of their operations, unique legitimacy, breadth and depth of knowledge and expertise of staff. The UK is strongly committed to working through the international system to reduce poverty in developing countries. In 2005/06 the UK channelled 38% of our aid through multilateral organisations. This is likely to increase in the future. As levels of aid to multilaterals increases, we shall need better evidence to demonstrate that this is contributing to improved results on the ground. This is important both for accountability purposes and to promote ongoing multilateral reforms. It will also enable donors to make more informed resource allocation choices between agencies. At present, there are no common reporting standards for multilaterals and the volume and quality of published data on performance varies immensely. Partly in response to this, a number of instruments have been developed to assess different aspects of multilateral effectiveness. For example, the annual survey of the Multilateral Organisation Performance Assessment Network (MOPAN) looks at multilateral performance at country level as assessed by a group of ten bilateral donors. The Paris baseline survey also monitors multilateral performance against key indicators contained in the Paris Declaration on Aid Effectiveness. In addition, several bilateral donors, including DFID, have developed their own methodologies to assess multilateral effectiveness. On June 20 2007, DFID hosted a meeting of bilateral donors to discuss multilateral effectiveness and financing issues. The purpose of this meeting was threefold: (i) to share experience, tools and methodologies for assessing multilateral effectiveness; (ii) (iii) to explore the potential for streamlining some of the various tools into a joint approach; and to discuss the possibility of developing clearer criteria for making decisions on aid allocations between multilaterals

This package is the main output from that meeting and includes summaries of the key presentations on:
Multilateral Effectiveness and Financing Meeting, 20 June 2007, London

1. DFIDs Multilateral Development Effectiveness Summaries (MDES); 2. The Swedish Ministry of Foreign Affairs: Framework for Assessing Relevance and Effectiveness; 3. A Canadian International Development Agency (CIDA) study on a comparative analysis of assessment tools and development of a joint approach; 4. DFIDs multilateral allocative efficiency methodology; and 5. An Overseas Development Institute (ODI) survey of partner country perceptions of multilateral performance We hope that these materials can make a useful contribution to the emerging discussion around multilateral effectiveness. This is still work in progress. Future challenges include refining and harmonising existing multilateral assessment methodologies; engaging multilaterals and partner countries in a substantive dialogue around effectiveness issues; and developing a more joined up international approach to resource allocations. Beyond the goal of increasing individual agency effectiveness, there is also a need to develop a shared vision for the overall international development architecture. In recent years, the global aid system has seen a proliferation, fragmentation and verticalisation of institutions. The average number of donors in each low income country has risen from around 12 in the 1960s to more than 30 now. In 2004, the total number of donor activities was estimated at 60,000, compared to 20,000 in the late 1990s. Changes in the multilateral system with the establishment of single purpose multilateral funds for health and education, together with the emergence of new bilateral donors and Foundations, lie behind this. The effects have been to increase transactions for recipients and reduce coherence. This is particularly problematic for those low income countries which are highly reliant on aid resources and have the weakest administrative capacity to manage donors. Without fundamental system reform, there is risk that the international system will be unable to meet the significant global challenges which we currently face, for example in relation to conflict, fragile states, HIV/AIDS and climate change. The challenge for the future will be to promote an international system fit for the 21st century. This should include a clear vision of the respective roles, responsibilities and comparative advantages of multilateral and bilateral actors in delivering international development goals. DFID looks forward to working with others over the coming period to progress this important agenda. Mark Lowcock

Multilateral Effectiveness and Financing Meeting, 20 June 2007, London

Contents
Page 5 11 14 Department for International Development: Multilateral Development Effectiveness Summaries Methodology Note Swedish Ministry of Foreign Affairs: Framework for Assessing Relevance and Effectiveness Canadian International Development Agency: Assessing Multilateral Effectiveness: A Comparative Analysis of Data Collection Tools Department for International Development: Bilateral and Multilateral Allocative Efficiency Methodology Note Overseas Development Institute: Recipient Country Stakeholder Perceptions of Multilateral Donor Effectiveness

17 22

Multilateral Effectiveness and Financing Meeting, 20 June 2007, London

1. Multilateral Development Effectiveness Summaries

Presented by Department for International Development (DFID) Introduction
1. This note explains the methodology the Department for International Development (DFID) is using to measure the effectiveness of multilateral organisations. Just as there is no universal definition of multilateral effectiveness there is no one way for organisations to achieve it and no one way to measure it.

Current Approaches to Measuring Multilateral Effectiveness

2. There are a number of tools to measure multilateral effectiveness. Each concentrates on a different part of the results chain and considers the issues from the perspective of different stakeholders. Figure 1 shows how some of these tools fit within the monitoring or evaluation process.
Figure 1: Different Aspects of Measuring Effectiveness Allocation Models Factor Annual Progress Reports

One

Data Factor Collection Two

Surveys Censuses Administrative Factor data Three

Performance Monitoring
Inputs Outputs Outcomes Impact Donor Behaviour

Evaluation
Projects Programmes Themes Policy instruments Institutions Interventions

RESULTS

Paris Declaration

MEFF, MOPAN Impact Evaluation

3. Some of the main tools used to assess multilateral effectiveness are:

Multilateral

Effectiveness Financing Framework (MEFF) developed by DFID in 2004, this looks at whether an organisation incorporates a results focus into 5

Multilateral Effectiveness and Financing Meeting, 20 June 2007, London

all its business processes and continually uses results to improve its performance.

Multilateral Organisations Performance Assessment

Network (MOPAN) this looks at multilateral performance at country level as assessed by a group of eight bilateral donors, including DFID. indicators and targets developed by OECD Development Assistance Committee (DAC) to make aid more effective.

Paris Declaration (PD) on Aid Effectiveness a set of

4. None of these tools encompasses the full range of information which is needed to measure effectiveness although they each provide useful information from particular perspectives.

A New Approach
5. Rather than attempting to create a new all encompassing single tool to measure agency effectiveness, DFID have devised a new system, based on a Balanced Scorecard Approach, which attempts to collate existing information and to present it within a coherent framework. A standard Balanced Scorecard Approach recognises the need for organisations to consider a wider range of information beyond the financial measures they traditionally used. 6. DFID has adopted the Balanced Scorecard to reflect its own business needs and uses this approach in its Quarterly Management Report. In particular it recognises the importance of partnerships and relationships in DFIDs work (see Figure 2). Figure 2: DFID Quarterly Management Board Report Framework
Delivery of Results
Are we delivering against our PSA targets?

Managing Resources

How well are we planning and managing our resources?

DFID Mission: Poverty reduction through partnerships

Managing External Relationships

Are we managing our relationships effectively?

Building for the Future

Are we delivering our people and organisation for the future?

Multilateral Effectiveness and Financing Meeting, 20 June 2007, London

DFIDs Multilateral Development Effectiveness Summaries

7. In developing a new framework for considering multilateral effectiveness DFID has adapted the Balanced Scorecard Approach and tailored it to meet the specific context in which multilateral organisations operate. This is the Multilateral Development Effectiveness Summary (MDES). 8. The summary looks at four broad areas of effectiveness: managing resources, building for the future, partnerships and country/global performance. Each area in turn affects the next one hence they are shown in Figure 3 as a continuous circle. While the summary gives a comprehensive picture of the overall effectiveness of the organisation, it is not intended to directly measure actual development results on the ground.
Figure 3: Multilateral Development Effectiveness Framework

Country/ Global Results Partnerships

Effectiveness Effectiveness

IMPACT

Building for the Future

9. The MDES are short publications which cover the following four areas of the organisation:

Building for the Future - How is the organisation building

for the future through sharing information, learning and innovating? This area considers whether the organisation is committed to a culture of continual learning and improvement, how it manages information and its results and whether it is investing in its staff.

Managing Resources - How is the organisation managing its activities and processes? This area looks at how the organisation uses its financial and human resources. It includes indicators such as disbursement ratios, resource
Multilateral Effectiveness and Financing Meeting, 20 June 2007, London

Managing

Resources

allocation criteria, staff recruitment, postings and promotions processes and degree of decentralisation.

Partnerships - How is the organisation engaging with other development partners? This area looks at how the organisation works with other development partners including developing country governments themselves. It encompasses Paris Declaration Indicators on alignment and harmonisation. Country/Global Performance - How well is the organisation performing either at the country level and/or at the global level? The final area considers any information which is available on actual performance either at the country level or at the global level. This includes results of any evaluations, by the organisation themselves or by other organisations, information on portfolio quality, any data on programme targets and any other indicators on outcomes and impact.

10. Each of these four areas is considered equally important and only by considering the four areas together can one begin to understand how effective the organisation is. The inclusion of an area on building for the future helps us attempt to measure how well an organisation is placed to perform in the future. Within each of the four areas we have attempted to collate a number of indicators under broad headings as shown in Annex 1. The headings and indicators are not designed as an exhaustive list. Rather they should provide some suggestions on the types of indicators which can be considered under each heading. 11. In the future, DFID is thinking of collating information for each of these areas, summarising and possibly assigning a traffic light based on assessment, as shown in Figure 4. The traffic light assessments should be based on objective evidence but they will also rely on judgement depending on what evidence is available for that particular organisation.
Figure 4: Traffic Light Assessment

Multilateral Effectiveness and Financing Meeting, 20 June 2007, London

Purpose of the Multilateral Development Effectiveness Summaries

12. The Multilateral Development Effectiveness Summaries give users a general overview of an organisations effectiveness. They include basic data on the organisation as well as a summary of the assessment and the context in which the organisation is working. The summaries are aimed at providing senior managers and staff working on multilaterals with a concise summary of all available information on the effectiveness of a particular multilateral organisation. 13. The information is not designed to produce a ranking of multilateral organisations by their effectiveness or to provide a single basis for expenditure decisions as there are other considerations which would also feed into this type of decision, such as the scale, scope, coverage and risk involved with each particular organization. Rather the information should be used to help to inform decision making and to ensure senior managers have a consistent picture of each multilateral.

Limitations to the Multilateral Development Effectiveness Summaries

14. The multilateral development effectiveness summaries are a tool to present the latest available information on the effectiveness of an organisation. They rely on the collation of existing data. As such there are issues of comparability both across organisations and also across time. One organisation may collect a lot more information than another which may influence the overall picture of its effectiveness. The quality of data will vary considerably across organisations. The data which does exist may not always be comparable across organisations as it may be collected in different ways and using different definitions. 15. For these reasons the multilateral development effectiveness summaries should not be used to rank organisations. Rather they provide a snapshot of existing information, collated into a coherent framework.

Multilateral Effectiveness and Financing Meeting, 20 June 2007, London

Examples of MDES Indicators

Commitment to Continual Improvement Commitment to Continual Does the Board require management to act on performance results (MEFF 1Ac)? Improvement Building Knowledge and Lesson Learning Lesson Learning Does the organisation have adequate mechanisms for spreading lesson learning across the office (MEFF 7Ac)? RBM System To what extent does this organisation manage by results? Staff Development Training & Development What is the average number of days training per staff member? Staff satisfaction What are staff resignation rates? Does the organisation have IiP accreditation or similar? What is the level of staff satisfaction (e.g. % of staff proud to work for the organisation)? Corporate Strategy Strategy Is the organisation's corporate strategy based on a clear definition of mandate and comparative advantage (MEFF 2Aa)? Resource Management Disbursement Administrative Efficiency Aid Allocation Staff Management Recruitment & postings Operational Management Decentralisation Programme Efficiency Voice Voice Civil Society Partnership Behaviour Partnership Behaviour Alignment Alignment What is the disbursement ratio i.e. commitments versus disbursements (Paris Declaration Indicator 7)? What is the administrative efficiency ratio? How well is the organisation's resource allocation criteria aligned with its corporate strategy and comparative advantage? To what extent is staff recruitment, postings and promotions meritocratic and transparent (MEFF 6Aa)? Is the organisation sufficiently decentralised to enable it to respond flexibly to country demand (MEFF 4Ba)? What is the speed of disbursements (e.g. time taken from project approval to implementation)? What mechanisms exist for developing countries to influence the strategy of the multilateral? How actively is this organisation promoting the participation of civil society (MOPAN Support to Civil Society)? What mechanisms are in place to seek feedback on partnership behaviour and what do the results show? To what extent does this organisation promote or enable government ownership throughout the project/programme cycle (MOPAN Capacity Development and Technical Advice Qu.)? What % of aid flows to government sector is reported on national partners budgets (PD 3) Is the number of Project Implementation Units decreasing or non-existent? (MEFF 4Bb, PD 6)? What % of TC flows are provided through coordinated programmes consistent with partners' national development strategies (Paris Declaration Indicator 4)? Do they use countries own public financial management and procurement systems (PD 5a & 5b)? In what ways has this organisation been aligning its strategy/programme/projects with national strategies (MOPAN Alignment with National PRS, Policies and Procedures Qu.)? Harmonisation Coordination Information Sharing Harmonising Systems To what extent does this organisation participate in local donor coordination activities such as donor working groups (MOPAN Inter-Agency Coordination)? To what extent does this organisation share information with other donors (MOPAN Inf. Sharing Qu.) What evidence is there of harmonising procurement and consulting services procedures, disbursement policies and evaluation practices (e.g. % of donor missions and country analytical work which is joint PD 10a and 10b)? What information is available on the organisations' performance at country level? What evidence is there of the independence, credibility and utility of the organisations' own evaluations (e.g. Operations Evaluations Departments, DAC Peer Reviews, NGO reports)? What impact is the organisation having at the country/global level? What % of projects/programmes met their targets? How does this vary across sectors, regions and countries?

PARTNERSHIPS COUNTRY/GLOBAL RESULTS

MANAGING RESOURCES

BUILDING FOR THE FUTURE

Use of Country Systems

Country/Global Results Evaluation

Results Portfolio Quality Quality of projects and programmes

Multilateral Effectiveness and Financing Meeting, 20 June 2007, London 10

2. Swedish Framework for Assessing Relevance and Effectiveness

Presented by the Swedish Ministry for Foreign Affairs Introduction
1. In 2006 Sweden channelled 13 billion Swedish kronor or 50% of its development budget through multilateral organisations (MOs). As with other donors, this has generated increased demands for improved monitoring and accountability of multilateral performance. In April 2007 the Swedish government adopted a Strategy for Multilateral Development Cooperation. The strategy gives normative guidance on defining and developing criteria for priorities, financial principles and methods for strategic influence on multilateral work. 2. The strategy proposes that relevance and effectiveness should be the main criteria in assessing and deciding on aid to all multilateral channels. Relevance means, firstly, the compatibility of the activities with Swedish development goals and, secondly, the role of the organisation in the international multilateral architecture. Effectiveness means, firstly, whether the organisation contributes to the relevant goals set and, secondly, whether the activities are organised so as to lead to results and employ aid resources effectively. 3. The Ministry for Foreign Affairs developed a pilot framework for assessing relevance and effectiveness of the multilateral organisations. 27 multilateral organisations (UN organizations, development banks, EU and vertical funds) were assessed during this first cycle.

Purpose
4. The assessment framework serves two main purposes, namely to: Promote strategic dialogue with the MOs, and provide the basis for developing specific organizational strategies. Guide decisions on allocations of funds to MOs,

Multilateral Effectiveness and Financing Meeting, 20 June 2007, London 11

Design Characteristics
5. The overall objective of the organisation performance assessment model is to provide a snap-shot of where the organisation is at and in what direction it is moving. To do this an assessment template or scorecard has been developed, which includes five separate sections covering: (i) (ii) (iii) Relevance of goals to the main priorities set out in Swedens Policy for Global Development Relevance of the organization to the international aid architecture. Internal effectiveness e.g. the organizational structure, results-based management focus, monitoring and evaluation systems, and transparency/disclosure policy, including reporting on auditing. External effectiveness e.g. selectivity, cooperation with others (i.e. in line with the Paris declaration), and the degree of focus at country level. Reform potential and other expected changes (or trends).

(iv)

(v)

6. Against each of these criteria a number of qualitative and quantative questions are asked in order to build up an overall picture of the relevance and effectiveness of the organizations concerned.

Methodology
7. The process for conducting the assessment during 2007 was led by a project team in the Swedish Ministry for Foreign Affairs, and for each organization a special sub-team was set up with staff from the Swedish International Development Cooperation Agency (Sida), other ministries, embassies, constituency offices and EU/UN representations, all working in different capacities with the specific organization. The assessment was made jointly by the subgroup. The time-frame for completing the assessments was fairly short (four weeks). 8. Both internal and external sources of information were used in compiling the assessment. A reference group was also established for quality assurance purposes, which reviewed the quality of assessments at the end of the process.

Multilateral Effectiveness and Financing Meeting, 20 June 2007, London 12

Relevance to resource allocation

9. According to the strategy, a number of core principles shall apply to the financing. These include: Non-earmarked contributions Predictable long-term financing Thematic Financing (vertical funds) Multi/bilateral support through country programmes Burden sharing

10. The assessment framework has provided further information on which to base decisions around resource allocations to multilaterals. Generally, the following principles are likely to be applied in the future: Enhanced effectiveness and relevant = increased contributions, non-earmarked and multi-year contributions High effectiveness and relevant = unchanged contributions, non-earmarked and multi-year contributions Low effectiveness but relevant = reduced contributions and short-term financing Not relevant = reduced contributions and possible phasing out

Lessons Learned
11. A number of lessons have been learnt from the initial phase of the Swedish assessment process. For example: (i) (ii) (iii) (iv) It is difficult to directly compare multilateral organisations which have different roles and mandates within the international system Assessments of this type make an important contribution to organisational learning They also provide a good basis for dialogue with multilaterals, organisational strategies and budget allocations There is limited knowledge/evidence on the overall internal effectiveness of multilateral organisations

Multilateral Effectiveness and Financing Meeting, 20 June 2007, London 13

3. Assessing Multilateral Organisation Effectiveness: A Comparative Analysis of Data Collection Tools

Presented by Canadian International Development Agency (CIDA) Introduction
1. The Multilateral Organization Performance Assessment Network (MOPAN) is a network of ten like-minded donor countries1. At MOPAN meetings in 2006 there was consensus that members required further information on the effectiveness of individual multilateral organisations (MOs) beyond the Annual MOPAN Survey2. This was in order to meet growing demands for accountability, and to make more informed policy choices as multilateral budgets increased. 2. The members therefore established a Working Group to begin moving towards a common approach for the assessment of multilateral effectiveness. CIDA has led the initial phase of work, which has included a comparative analysis of existing bilateral donor assessment approaches3 and recommendations on how a more harmonised approach might be developed and applied in the future.

Main Findings
3. The main findings from the comparative analysis of existing bilateral assessment approaches were as follows: The main drivers for assessing multilateral effectiveness are to (i) increase accountability (ii) promote organisational learning and change, and (iii) inform resource allocation decisions
1

Current MOPAN members include: Austria, Canada, Denmark, Finland, France, The Netherlands, Norway, Sweden, Switzerland, United Kingdom

MOPAN conducts an annual perception survey of 3 multilateral organisations in 8-10 countries. In each country, MOPAN members complete questionnaires which feed into a country report and an overall synthesis report for the agencies concerned. The focus of the Survey is on multilateral partnership behaviour towards national stakeholders (governments, NGOs, private sector) in developing countries as well as towards other international development agencies. The Survey is not an evaluation and does not cover actual development results on the ground.

A total of seven bilateral donor tools were assessed as part of this work: Performance Management Framework (Danida); Scorecard/Multilateral Monitoring System (Netherlands MFA); Framework for Assessing Relevance and Performance (Sweden MFA); Multilateral Evaluation Relevance and Assessment System (CIDA); Multilateral Effectiveness Framework (DFID); Results-Based Management Assessment Program (Danida); Multilateral Effectiveness Summary Balanced Scorecard (DFID).
3

Multilateral Effectiveness and Financing Meeting, 20 June 2007, London 14

 Comparison of the design characteristics reveals a common set of four criteria for assessing effectiveness: domestic policy relevance, international relevance, internal performance and external performance. The seven multilateral assessment tools reviewed included over 250 indicators of effectiveness, many of which are overlapping and duplicatory. A few information gaps related to corporate indicators, quality assurance processes, and partner government perspectives. There is a need for greater transparency in conducting effectiveness assessments, particularly where these are linked to resource allocation decisions Comparison of the indicators reveals a common frame of reference focused on strategic, operational, relationship and knowledge management. Good potential exists to develop a common donor approach based around these core dimensions of effectiveness.

Common Approach: Key Principles

4. In developing a common approach five key design principles (based on agreed Managing for Development Results principles) have been proposed. The assessment approach should: i) Generate relevant and credible information to support bilateral donors in fulfilling their governance responsibilities for holding multilateral organizations to account. ii) Generate useful information to support multilateral organizations in their efforts to foster a results-based management culture and develop the requisite monitoring and evaluation systems to facilitate organisational learning, improve development effectiveness and produce credible performance reports iii) Should apply the keep it simple principle, keeping the initial number of indicators reasonable and designing the data collection tools with a view to facilitating data analysis and interpretation. iv) Should be designed as a collaborative management exercise, as opposed to a bilateral donor measurement exercise. v) Should be designed to engage multilateral organisations in a learning and improvement process.

Multilateral Effectiveness and Financing Meeting, 20 June 2007, London 15

Developing a Common Approach: Recommendations

5. This study proposes the use of a Balanced Scorecard as a common approach to assessing multilateral organisation effectiveness. The Balanced Scorecard below is based on a common frame of reference focused on strategic, operational, relationship and knowledge management categories derived from the concept mapping exercise undertaken as part of the comparative analysis. To further develop the balanced scorecard, MOPAN Members will have to agree on a core set of common indicators, which will build on the strengths of the existing assessment approaches, and modify them as required to collect data from at least three data sources representing different perspectives on the same indicators of interest (e.g. donor, client and multilateral organisation perspectives). When selecting the core set of performance indicators, they will also have to forego their stakeholder role in favour of a shareholder role, thereby putting the multilateral organisation in the foreground, while the uniquely national development policy objectives of each bilateral donor country are put in abeyance. 6. A second phase of work is now underway. This will seek to agree a core set of indicators to populate the scorecard, as well as a commonly agreed process for collecting the relevant data.

The MOPAN Balanced Scorecard

STRATEGIC MANAGEMENT
Governance Corporate Strategy Country-Level Strategies Corporate Indicators/Targets

KNOWLEDGE MANAGEMENT
Organisational Stewardship Performance Monitoring Evaluation Management Performance Reporting Lessons Learned

OPERATIONAL MANAGEMENT MfDR RELATIONSHIP MANAGEMENT

Donor Harmonisation w/ Multilaterals and Bilaterals Alignment with Country Partners Financial Resources Human Resources Quality @ Entry Country Portfolio

June 3, 2007

16

Multilateral Effectiveness and Financing Meeting, 20 June 2007, London 16

4. DFID Multilateral Allocative Efficiency: Methodology note

Presented by DFID Background
1. DFIDs programme expenditure is set to increase significantly as the UK continues to make progress towards its commitment to meet the UNs oda/GNI target of 0.7% and the commitments and expectations resulting from the Commission for Africa, the 2005 Gleneagles G8 Summit and DFIDs third White Paper Making governance work for the poor. Within this context, the Value for Reform programme has taken a robust and critical look at how and where the aid budget is deployed with a view to improving programme efficiency and effectiveness. 2. This note provides an explanation of the methodology used in DFID to measure non-cashable efficiency savings.

Efficiency Methodology
3. Allocative efficiency for the bilateral programme is about choosing the country allocation scenarios which would maximise the highest number of poor people lifted out of poverty for a given budget. In the multilateral case, allocative efficiency is defined as the extent to which a portfolio of multilateral investments and their corresponding country allocations, have greater poverty impact (measured by the number lifted out of poverty) than a set of previous decisions made on the multilateral institutions. 4. The methodology adopted here is based on a Collier Dollar model of poverty efficiency modified for DFIDs purposes. The model basically looks at an aid growth equation which shows the marginal and average impact of aid on growth. The growth is then linked to poverty reduction by using an average poverty elasticity of growth of 2 (a 1% increase in GNI leads to a 2% fall in the poverty headcount). The size of this impact then is dependent on the aid flows to a particular country, the level of poverty and the quality of policies (as proxied by CPIA). 5. Poverty efficiency is maximised by reallocating aid to equalise the marginal efficiency in each country. The marginal efficiency estimates derived from the model determine how much extra growth and therefore, how much extra poverty reduction we will achieve by scaling up aid to a particular country. Thus, we can then estimate for each country the number of poor people that are
Multilateral Effectiveness and Financing Meeting, 20 June 2007, London 17

removed from poverty for a given aid allocation and sum this up over the aid programme and alternative portfolio of investments scenarios. The efficiency savings figures are derived by taking the percentage of the additional people lifted out of poverty from a poverty efficient scenario multiplied by the total budget.

Models shortcomings
6. Note that the model is imperfect and is highly dependent both on the parameters of the aid growth relationship and on the available data which results in a large margin of error. Some of the shortcomings of the model are outlined below: This model only compares bilateral and multilateral investments according to how poverty efficient the allocations are; no other information is taken into account in determining the efficiency savings. The model derives the marginal efficiency (ME) estimates for each country based on a dataset of global aid in 2002. Updating the data for subsequent years would result in different ME estimates and different efficiency savings figures. Changing the parameters of the aid growth relationship will yield a different set of marginal and average efficiency estimates of the aid impact on poverty reduction. 20% of current aid goes to countries where there are data gaps which means we cannot calculate a poverty efficient allocation to these countries. This model is about allocative efficiency only. The other determinant of aid effectiveness the quality of the aid itself is assumed to be the same across institutions and across time. The model currently only has information of historical and current disbursement patterns for 9 or 10 of the multilateral institutions which means not all of the positive choices made on portfolio investments will be wholly reflected in the efficiency savings. The model does not incorporate any further efficiency gains that might be achieved from technical efficiency gains or additional information that may influence the allocation decision.

Bilateral efficiency
7. The efficiency programme for our bilateral allocations compares a baseline case of proportionate scale up from 2007/08 each year to 2010/2011 to a more poverty efficiency scenario. The efficiency savings measure is calculated by taking extra people
Multilateral Effectiveness and Financing Meeting, 20 June 2007, London 18

lifted out of poverty permanently as a percentage of the total and translating that into an efficiency savings figure. 8. Using this methodology, an efficiency savings of 100 million has corresponding number of people lifted out of poverty permanently. This means that by allocating in a more poverty efficient way, we can save 100 million or lift an additional 100,000 people permanently. Efficiency savings figures can also be cumulated over multi-year budgeting cycles because the additional people we lift out poverty in each year is a permanent effect.

Multilateral efficiency
9. The efficiency programme we are expecting to deliver through the multilateral portfolio will be calculated in three ways: quantitatively based on a Collier Dollar global ideal framework, other quantitative estimates for individual institutions and finally by bringing in qualitative effectiveness information of institutions comparative advantage. 10. The baseline scenario for the multilaterals, unlike in the bilateral programme, cannot simply be based on a proportional scale up from 07/08 as there are other factors influencing the decisions around the multilaterals. The baseline is based on a set of low case assumptions and excludes any investments to meet White Paper 3, and spending decisions made in the last 12 months. Therefore, the annual efficiency savings figures represent the extent to which a portfolio of multilateral investments and their current and historical country disbursement patterns is more poverty efficient than a set of previous decisions made individually on the multilateral institutions. 11. We have looked at the allocative efficiency implications of several multilateral spending scenarios. Here again, the efficiency figures we derive reflect that by making portfolio choices in a more efficient way saves DFID x million or lifts an additional x thousand people out of poverty permanently.

Relative efficiency by institution

12. Other qualitative information should be used in making portfolio choices between the institutions, including institutions relative contribution to sectoral goals. This note also includes allocative efficiency estimates of the major institutions according to their global aid ideal. The calculations are based on an assumption that were there to be an additional 100 million above the preferred budget scenario, how would investing all of it in one
Multilateral Effectiveness and Financing Meeting, 20 June 2007, London 19

institution compare with investing it proportionately through all the institutions. 13. These estimates could be seen as the opportunity costs of different scenarios of scaling up within a growing envelope. Therefore, an efficiency savings of 17 million for IDA means that by investing the additional 100 million in IDA above that which has been allocated under the preferred spending scenario, we could spend 17 million less to lift an additional 15,600 people versus investing it through all the institutions in proportionate amounts to the preferred scenario allocations. By the same token, investing the 100 million in the EC, means we need to spend an additional 41 million pounds to lift the same number of people out of poverty as we would by investing the additional 100 million proportionately to all the institutions. 14. Again, these efficiency savings are based on the Collier Dollar model and the same margins of error exists. It should be noted that these figures demonstrate at the margin what the allocative efficiency savings would be. In deciding portfolio balance, there will be many other efficiency and effectiveness considerations central to that decision.

Multilateral Effectiveness and Financing Meeting, 20 June 2007, London 20

Relative efficiency by institution

Rating 2010/2011 AfDF 38,300 EDF FTI 15,600 IDA GFATM 2,100 UNDP 1,100 Other UN -20,500 AsDF -37,700 EC DFID bilateral Weak 33,600 36.89 -41.40 -22.50 Medium -14,900 -16.33 1.16 2.28 Strong 7,000 7.74 17.19 Very strong 27,100 29.78 42.04 Additional Number of people lifted out of poverty 41,400 Efficiency savings ( m) 45.52

Multilateral Effectiveness and Financing Meeting, 20 June 2007, London 21

5. Recipient country stakeholder perceptions of multilateral donor effectiveness

Presented by Overseas Development Institute (ODI)1
Key Points The perceptions of 261 senior recipient country stakeholders about the performance of a range of multilateral organisations are a challenge for donors. Respondents and donors appear to use different sets of criteria to assess overall effectiveness. Perceptions of governance and ownership appear to affect recipient preferences about which multilateral organisations should disburse additional sums of ODA 1. This briefing paper describes the results from a pilot project aimed at seeking the views and perceptions of key stakeholders in recipient countries on the performance of a range of multilateral organisations and their preferences for which organisations should disburse additional aid. 2. Despite the dip in aid volumes in 2005, OECD/DAC predictions still suggest that aid volumes will rise to around $130 bn by 20102. At the same time, a number of bilateral donors are reducing costs by holding or even cutting staff numbers. These donors are therefore seriously considering increasing the aid they channel through multilateral organisations, because they will find it difficult to spend the additional money bilaterally. In addition, there may be a number of positive effects of increasing disbursements through the multilateral system such as reducing procurement costs and streamlining political conditionality. 3. If bilateral donors are to increase their spending through the multilateral system substantially, they will need to demonstrate that the money is being spent well. This will require them to have a better understanding of the effectiveness of multilateral organisations. There have been a number of attempts to assess the effectiveness of donors, including multilateral agencies. Some of these use results-based management toolkits, a few examine organisational policies and processes, but most are based on
1

This paper is taken from ODI briefing paper of same title. OECD/DAC April 2007, http://www.oecd.org/dataoecd/13/26/38341348.pdf

Multilateral Effectiveness and Financing Meeting, 20 June 2007, London 22

perception data, usually of staff from donor governments and agencies3. A few, like the Strategic Partnership for Africa (SPA) Survey and the OECD/DAC's Paris Baseline Survey also seek the views of recipient government officials. Until now there have been no systematic attempts to seek the views of a wider range of stakeholders in recipient countries, a gap that is all the more surprising considering the range of stakeholders that interacts with donors in-country and is responsible for implementing donor programmes.

The Study: seeking stakeholder perceptions

4. Stakeholders from six countries gave their views and perceptions about various dimensions of effectiveness of seven organisations. The six countries were: Bangladesh, Ghana, India, South Africa, Tanzania and Zambia. The seven organisations were: the African and Asian Development Banks, the European Commission, the Global Fund for AIDS, TB and Malaria, United Nations Childrens Fund (UNICEF), United Nations Development Fund (UNDP) and the World Bank. Locally based country coordinators administered a questionnaire seeking views from five different stakeholder groups; Figure 1 shows the total number of respondents by country, while Figure 2 shows the total number of respondents by stakeholder group. In total, despite a very short data collection period, 261 well-informed and senior individuals answered the questionnaire and contributed over 2300 text comments which provide valuable context and additional insight.

35

55

Bangladesh Ghana Tanzania India 49 South Africa Zambia

35

38 49

Figure 1: Number of respondents by country, total 261

Donor attempts to assess donor effectiveness where the results are publicly available include: The Multilateral Organisation Performance Assessment Network (MOPAN) Survey, the OECD/ DAC Paris Baseline Survey, the Performance Management Framework (Denmark), and the Strategic Partnership for Africa survey.

Multilateral Effectiveness and Financing Meeting, 20 June 2007, London 23

Business 43 53 Civil Servants 44 52 Civil Society Govt. Ministers MPs

69

Figure 2: Number of respondents by stakeholder group, total 261

6. Much of the attention of donor and recipient governments has been focused on assessing progress against the Paris Declaration as a key measure of effectiveness. The survey sought respondents perceptions of multilateral organisation (MO) performance against three elements of the Declaration. 7. In addition, the project developed its own effectiveness criteria. This exercise drew on two separate projects which brought together recipient country stakeholders, one coordinated by the Commonwealth Secretariat together with la Francophonie, and the second by Debt Finance International. Respondents were asked to rank fifteen of the most important criteria from the two exercises in order to see if recipient stakeholders take into account a broader range of factors as they think about organisational effectiveness. They were then asked to rank the performance of the MOs against each of the criteria. Finally, the survey asked respondents to rank the MOs in the order in which they would prefer them to act as disbursement channels for any additional ODA flows.

The Results
8. Figure 3 summarises the results. Each bar is split into four elements. The top three elements show the ratings given to the MOs for the three Paris Declaration indicators. The bottom element shows the scores received by the MOs when respondents were asked to rate overall effectiveness. Ratings for each element were on a five point scale giving an overall maximum potential score of twenty. The crosses on Figure 3 shows respondents preferences for which organisations should disburse additional ODA. The maximum potential score is three.

Multilateral Effectiveness and Financing Meeting, 20 June 2007, London 24

Large scale funders

Specialised Agencies

20.00
Multilateral Organisation Effectiveness Ratings (maximum score 20)

3.00
Respondent Preferences for Disbursement (maximum score 3)

18.00 16.00 14.00 12.00 10.00 8.00 6.00 4.00 2.00 0.00
EF P W B EC As D Af D N IC U U G FA N TM B B D

Overall development effectiveness Harmonises with other donors Aligns with government priorities Promotes government ownership of development process Respondent disbursement preferences

2.00

1.00

0.00

Figure 3: Multilateral organisation effectiveness ratings and respondents preferences for disbursement of additional aid flows. 9. Respondents have a clear idea about the elements that make up an effective donor. The fifteen additional effectiveness criteria can be split into two groups: criteria relating to the way donors provide funds, and criteria relating to donors policies and procedures. Respondents ranking of the criteria, see box 1, suggests that the criteria relating to policies and procedures are more important; six of eight criteria are rated by 70% or more of those who answered the question as highly important. In comparison, only two of the seven funding criteria are rated as high as this. Respondents were also offered the opportunity to add additional criteria that they believe to be important to donor effectiveness. Few did this, suggesting that the list presented in the survey was generally seen to be adequate.

Multilateral Effectiveness and Financing Meeting, 20 June 2007, London 25

Funding Criteria - Disburses funds quickly - Flexible in the types of funding provided - Makes long-term commitments - Provides predictable funding - Provides funds with low conditionality - Provides untied aid - Provides highly concessional funds

Policies and Procedures Criteria - Facilitates the participation of stakeholders in its work - Is transparent in the way it makes funding decisions - Is cost-effective - Undertakes constructive policy dialogue - Monitors and evaluates its work effectively - Programmes and projects aligned to governments development priorities - Harmonises with the procedures of other donors - Uses government procurement procedures

These criteria are presented in the order they were ranked by respondents, highest at the top. Box 1: Important additional criteria affecting donor effectiveness 10. Respondents perceive there to be little difference in the performance of multilateral organisations against the Paris Declaration. In aggregate there is limited differentiation between respondents ratings of the different organisations against three of the Paris Declaration indicators, ownership, harmonisation and alignment. Where there are differences, the United Nations (UN) agencies tend to be rated higher than the Banks and European Commission, though this is country specific. 11. Respondents rank the Asian Development Bank, UNDP and World Bank highest for both the funding and the policies and procedures criteria. In contrast to the ratings received by the MOs against the Paris Declaration indicators, their rankings against the fifteen additional effectiveness criteria show more differentiation. Asian Development Bank, UNDP and World Bank receive a greater number of highest place rankings than the other organisations in the survey. The European Commission receives only three highest place rankings, the African Development Bank one and the Global Fund for Aids, TB and Malaria receives none. 12. Respondents perceive there to be no difference in the overall effectiveness of different multilateral organisations. Figure 3 shows that respondents give equal ratings, in the range 3.30 3.52, for overall effectiveness to the seven organisations studied. This finding is surprising for two reasons. First as described above, there is clear differentiation in respondents rankings of the MOs against the fifteen effectiveness criteria. Second, the results of other methodologies assessing MOs find differences in their

Multilateral Effectiveness and Financing Meeting, 20 June 2007, London 26

effectiveness. Despite the limited differentiation against the Paris Declaration indicators, it might be expected, therefore, that the different MOs would receive different ratings for their overall effectiveness. That they do not suggests that the respondents took into account other factors as they assessed multilateral organisation effectiveness. 13. Respondents have complex views about the factors that influence the effectiveness of different multilateral organisations. Analysing the data suggests that respondents take different factors into account for different MOs when they assess effectiveness. For example, for the majority of organisations in the study, there is an association, although weak, between the promotion of government ownership and rankings of overall effectiveness, i.e. respondents giving higher ratings for the extent to which an organisation promotes government ownership are also more likely to give it a higher rating for overall effectiveness. A number of associations between the fifteen effectiveness criteria and overall effectiveness were also found. But these associations were all weak, raising the possibility that respondents were taking into account a range of other factors as they thought about MO effectiveness. Respondent open-ended comments on the questionnaire provide further insight into the factors they appear to be taking into account when assessing overall effectiveness. For example, the focus of activities appears to be important. UNICEF and the Global Fund for Aids, TB and Malaria are valued for their health focus; the two regional development banks, the World Bank, and, to a lesser extent the European Commission, for their focus on productive sectors and infrastructure; and UNDP for its technical assistance. The scale of activities also appears to be a factor, as the Banks and European Commission receive more positive comments about the amount of money they can disburse, while the UN agencies receive almost exclusively negative comments in this regard.

14. Respondents preferences for which multilateral organisation should disburse additional aid appear to be independent of their ratings of effectiveness. Figure 3 shows that, unlike the ratings given for overall effectiveness, respondents preferences for which of the MOs should disburse additional ODA are clearly differentiated. In aggregate, respondents rank UNDP as first preference for disbursing additional aid. Comparing the multilateral organisations that can disburse significant sums of extra financing, stakeholders preferred the two Regional Development Banks to the World Bank and European Commission. Analysing the data confirms that respondent preferences for disbursement are only weakly correlated with the overall effectiveness, the Paris Declaration and

Multilateral Effectiveness and Financing Meeting, 20 June 2007, London 27

other effectiveness indicators explored in the study. This suggests that other factors also influence disbursement preferences. 15. Governance appears to be an important influence on disbursement preferences. Overall the African Development Bank is rated poorly against most of the fifteen effectiveness indicators. It is surprising therefore that it is preferred over both the European Commission and World Bank as a disbursement channel for additional aid in three of the four African countries studied. Respondent comments suggest that one key reason for this could relate to the more equitable governance structure of the Bank. This supports the hypothesis that perceptions of ownership of the institution and its policies can trump perceptions about its effectiveness. 16. Country context is important. The comments made by respondents on the questionnaire provide a helpful insight into why they gave the different MOs particular scores. A number of the comments suggest that a broader context is important in determining stakeholder perceptions of effectiveness. These contextual factors will vary across countries but may include the history of the organisation in that country, the visibility of its activities, the type of activities it undertakes, as well as personal factors such as the performance of the head of the agency in country. 17. The pilot study has demonstrated that key stakeholders in aid recipient countries want to have their voices heard. It invited the views of senior individuals from five respondent groups. The individuals invited to complete the questionnaire are all very busy and at the beginning of the project there was a real fear that they would not find the time to complete a questionnaire. In the end, the majority of respondents also contributed textual comments, totalling over 2300.

Multilateral Effectiveness and Financing Meeting, 20 June 2007, London 28

Box 2: Comments about multilateral organisation effectiveness

The Global Fund for Aids, TB and Malaria and UNICEF's funding especially as regards to HIV/AIDS and children, its effectiveness can be seen through activities currently implemented in these two areas (Tanzania Civil Service) The World Bank and African Development Bank are likely to favour the productive sectors for growth promotion. (Tanzania Civil Society) The UN and European Commission offer more than just funding, they build capacity in government and civil society. They offer a package (South Africa Parliament) The World Bank with its high lending of fund is relatively effective than UNDP which with less funds is content to publish more reports(Business India). UNICEF [] releases funds timely (South Africa Government) The European Commission is very bureaucratic and takes a long time when contracts are signed and when it is implemented. UNDP and World Bank are much better, even though there are delays also (Ghana Business) The World Bank's loan disbursement policy is guided by the head office. Their prescription never give results for the LDCs. (Bangladesh Business) The World Bank is sometimes bogged down with inflexible conditionalities. (Tanzania Business) The World Bank gives loans which have to be paid back at some future date. We need more grants. (Zambia Business) The African Development Bank is directly accountable to African countries (Tanzania Government) The African Development Bank is a regional organisation and is more in touch with continent issues and has a stake in its development. (South Africa Government) I believe in the potential of the UN system and its political accountability. (Ghana Civil Society) Organisations such as the European Commission hide behind their regulations forcing recipients to use their procedures arguing that these cannot be changed quickly because of their bureaucracy etc. These regulations impede efforts at attaining aid effectiveness. (South Africa Government)

What next?
18. If donors want to ensure that they have a full picture of MO effectiveness, then systematic studies of stakeholder perceptions will become increasingly important. The result that the survey respondents perceive no significant difference in overall effectiveness is, at first sight, surprising. Other assessments of effectiveness might lead to the expectation that the World Bank, for example, would rate higher than it does. The results from the pilot project suggest that donor governments conceive and assess effectiveness using a different range of factors and criteria to those used by stakeholders in recipient countries. Bilateral donors place more emphasis on outputs and outcomes, while recipient
Multilateral Effectiveness and Financing Meeting, 20 June 2007, London 29

stakeholders appear to place at least as much importance on ownership and governance. Given the clear consensus that ownership of the development process is critical to aid effectiveness, understanding stakeholder perceptions must be a critical element of any overall assessment of donor effectiveness. This difference in emphasis on the range of factors which could be taken into account while assessing effectiveness suggests that a single tool is not a viable way to assess donor effectiveness. Instead a suite of tools will be needed to capture the various dimensions of effectiveness necessary to obtain a complete view. These would include results based management tools, assessments of organisational processes, alongside different stakeholder assessments. Donors, recipient governments and key stakeholders will then need to use the range of evidence generated to form balanced judgements about which multilateral organisations to fund and engage with. 19. If donor governments wish to take stakeholder preferences for multilateral funding channels into account, then the results suggest that respondents have clear preferences. Survey respondents rated the African Development Bank relatively poorly against the fifteen effectiveness criteria in the survey, yet they prefer it as a disbursement channel to both the European Commission and World Bank. Comments drawn from the stakeholders suggest that perceptions of ownership and governance are, at least in part, responsible for these preferences. Yet, donors will find it hard to justify significantly increasing funding through the African Development Bank in the immediate future given the relatively mixed results from the various attempts to assess its effectiveness. This suggests that bilateral donors need to work harder to assist the African Development Bank to build its capacity and increase its effectiveness in order to be able to provide bigger volumes of funding to it as the institution which stakeholders appear to feel the greatest sense of ownership. 20. If the views and perceptions of recipient stakeholders become part of bilateral donor assessments of effectiveness then multilateral organisations will need to pay closer attention to issues of governance and ownership. The issue of country ownership will continue to be important. As highlighted above, it is likely that the views of recipient stakeholders from inside and outside of government about donor performance will therefore become more important and methodologies of this type will be increasingly employed by donors assessing MO effectiveness as part of disbursement decisions. Given that governance and ownership appear to play an important role in shaping stakeholders views, multilateral organisations wishing to improve their ratings will need to pay closer attention to these issues.

Multilateral Effectiveness and Financing Meeting, 20 June 2007, London 30