M&amp E Handbook 2006

47 Temple Street, Pietermaritzburg 3201 P O Box 157, Pietermaritzburg 3200 E-mail: info@cindi.org.
.za Tel: +27 33 345 7994 Fax: +27 33 345 7272 Website: www. cindi.org.za NPO 011-496 PBO 930008976
HANDBOOK on
Monitoring and Evaluation

for the CINDI Network
Compiled by Liz Clarke, CINDI M&E Advisor
March 2006
The development of this programme and document was made possible through the generosity of the Rockefeller Brothers Fund
Contents
1 2 3 4 5 6 7 8 9 Background ........................................................................................................................................ 3 The CINDI M&E journey .................................................................................................................... 3 The current buzz around M&E............................................................................................................ 5 From programme cycle to results based M&E ..................................................................................... 8 Mapping the journey information needs and indicators ................................................................. 13 Tools for gathering information ......................................................................................................... 16 Critical reflection: thinking through the journey ................................................................................ 21 Disseminating information ................................................................................................................ 24 Measuring, demonstrating or assessing impact? ................................................................................ 26
10 The basics of evaluation ................................................................................................................... 31 11 Developing an M&E Framework for CINDI Network ........................................................................ 35 12 Acknowledgements and References .................................................................................................. 36
Annex 1: Glossary ................................................................................................................................. 39 Annex 2: Ten criteria for assessing indicators ......................................................................................... 43 Annex 3: Draft M&E Framework ............................................................................................................ 44 Annex 4: UNICEF Framework ............................................................................................................... 59
How this Handbook has been put together ...

The different sections draw on information that was distributed at learning sessions to promote understanding of M&E concepts and processes. Some of the issues raised by participants are also included as there was a genuine effort to make the system relevant to the CINDI Network and the environment in which it works.
1. Background
Children in Distress (CINDI) is a network of over 100 concerned organisations seeking to effectively respond to the growing numbers of children affected by HIV and AIDS. CINDIs vision is to be a multi-sectoral well-resourced network of civil society capable of implementing diverse, effective, sustainable care and preventative programmes for children affected or orphaned by HIV and AIDS. The networks mission is to foster among partners a spirit of Ubuntu the principal of caring for each others well being with an attitude of mutual support. CINDI Members: acknowledge the AIDS epidemic is a problem too large for any individual, agency or government department to meet; have found a working framework for their alliance; believe children need protection but fare best within their own families, communities and circles of friends; understand extended families, caregivers and communities need external help if they are to be responsible for meeting the needs of vulnerable children ; emphasise development, not charity.
As a collection of independent initiatives, CINDI Members retain individual character, flexibility and the ability of individuals to respond to individual children, but are strengthened and guided by their link to the Network. Their aim is to identify and help children in distress. The scale of the epidemic and the poverty surrounding it is overwhelming. In terms of the scale of need available financial support is grossly inadequate. Yet the innovative elements of CINDIs work are important guides to the challenge of serving affected children in a manner that is acceptable to communities and sustainable for all the years it will take to raise a generation of children made extremely vulnerable by HIV and AIDS.
2. The CINDI monitoring and evaluation (M&E) journey

The design of the M&E framework is an initiative that follows on from an initial 2 year project funded by the Rockefeller Brothers Fund (RBF) which assisted Members with organisational development, strategic planning and monitoring and evaluation. This funding coincided with an allocation of funding from Irish Aid (formerly known as Development Cooperation Ireland) for operationalising the CINDI Network Office and supporting selected projects undertaken by nine CINDI Members. A further allocation of funding for 2005-2006 was made to facilitate the development of an M&E framework and support the training of approximately 20 Members of the CINDI Network. Eleven member organisations mostly funded by Irish Aid participated in the programme. The exercise was planned and implemented as a participatory process so that participating organisations help shape the framework to ensure it addresses their needs and reflects the realities of the external environment in which organisations work. From the visits
Handbook on monitoring and evaluation for the CINDI Network
to the organisations during the inception phase of the project it seemed that some tools had been developed to monitor programmes. Some of the tools had been developed by trial and error while other tools were borrowed from other programmes. The stipulated reporting requirements of the donors also seemed to fulfil an important monitoring role. Thus the development of the CINDI M&E framework has not happened in a vacuum but builds on experience already gained. The Handbook covers the different elements of the framework that were covered by the learning sessions as well as a proposed results based framework drawn up by participants. Participating organisations shared valuable information on a number of challenges that the development of an M&E system would need to contend with. These are summarised below:
2.1. User friendliness

Organisations generally felt the monitoring tools developed should be user friendly and not riddled with technical M&E jargon. It was felt that some of the terminology in the literature is very abstract and the tendency to strive towards alignment with corporate values was often at the expense of ensuring key development concepts particularly relating to the well being of children and families. Critical issues of well being are not easily measured or accommodated in log frames and reporting formats.
2.2. Data and the location of the data base

Many organisations wanted to have instant access to a very wide range of data as they believed this would make monitoring and evaluation easier and would also help them with funding proposals. However, it was conceded that the capacity (particularly IT skills) to sustain such a system is currently not available in all organisations. It was felt that the CINDI Network Office was an ideal location for such a centralised data base. Critical issues in the design of M&E framework is deciding on what data is needed, how it is collected, how it is organised, how it is stored and how it will be updated and accessed.
2.3. Monitoring non quantitative aspects of the work

Many organisations are working with children and families and are endeavouring to provide holistic care for them. This calls for specialised qualitative techniques of monitoring well being, growth and development which do not translate easily on to the usual M&E formats. Furthermore, if the M&E system is to provide a tool for empowerment of communities and capacitation of CBOs and other community structures innovative tools that are user friendly for people with low levels of literacy must be developed.
2.4. Linking monitoring and evaluation to planning

Organisations that had developed comprehensive strategic/operational plans seemed to have made the most progress with the regular monitoring of their work. It seemed much easier for them to meet with reporting requirements and also to reflect on their own progress meaningfully. Those organisations that had grasped and implemented such planning and monitoring systems seemed to enjoy working with them.
2.5. Gaps between management and fieldworkers

There sometimes seemed to be a gap between fieldworkers who contended on a daily basis with a lot of difficulties in their work and managers who wrote reports and focussed mostly on the outputs which did not always represent the realities in the external envi-
ronment. Such a gap may also exist between NGOs and the CBOs they mentor. There seems to be a we and they mindset which may result in the monitoring tools having limited value as an instrument of planning and change.
2.6. Issues of capacity

The AIDS pandemic, growing poverty, unemployment and bottlenecks, inefficiency and corruption in the public sector programmes have combined to put a great deal of strain on organisations that are trying to maintain high standards while at the same time expand their work. NGOs have difficulty retaining the services of professional people because salaries are not really competitive. Organisations want to see the M&E system set up and working but do not want too many additional demands on their time to be taken up servicing the system.
2.7. Involvement of beneficiaries in M&E

In some programmes, beneficiaries e.g. families, are involved in the assessment of the programmes. Some organisations wanted to try to expand the involvement of participants in the monitoring and assessment of the programmes.
2.8. Gender equality

Most organisations had complied with the requirement that a Gender Policy be put in place. However, there was a view that it had been imposed and was not necessarily related to the realities within which most organisations work. Generally the international best practice of integrating a gender perspective into both the internal and external work of the organisation was not really understood or was viewed as unnecessary because of how some organisations are already structured (e.g. most home based care workers are women).
3. The current buzz around M&E

3.1. The buzz around M&E
In recent years there has been an upsurge of thinking around monitoring and evaluation. With the adoption of the New Public Management Framework during the 1990s, corporate values and performance management frameworks were increasingly adopted by government and development organisations and eventually this became a requirement by funders of NGOs. Project management, financial management and organisational development have become fields of specialised knowledge and the expectation is that NGOs reflect these concepts in their management and work. It was assumed that rigorous monitoring and evaluation would ensure government officials, development managers and NGOs improve service delivery, the allocation of resources and planning. It would also ensure greater accountability in the use of donor resources. Almost all major initiatives such as those supported by the Global Fund, and international frameworks for responding to HIV and AIDS and poverty have very elaborate M&E systems. M&E has thus become a highly specialised field forming an integral part of strategic planning and programme management. However, it is still viewed by some as a stand alone which must meet funders reporting requirements. For many smaller NGOs with limited capacity the new M&E requirements have proved some-
what daunting. For CBOs and community structures working in lesser developed areas the challenge of meeting M&E requirements is a critical one as they are already being excluded from accessing much needed funding because of their low levels of capacity. Amongst those development workers who understand and support people centred development there is a growing uneasiness with some of the rigorous reporting requirements of funders. The concerns arise from the shift to a preoccupation with service delivery and product at the expense of development process which shifts power relations in the favour of the less powerful. In the learning sessions there was much discussion about the need to create opportunities for local community structures to fulfil a more empowered role in the response to HIV and AIDS and poverty. It is hoped that the framework will ultimately reflect not only an M&E system that improves accountability and informs planning but that also alters the power relations within the CINDI Network and mobilises and integrates CBOs and local community structures into the mainstream of the response to HIV and AIDS and poverty, especially from the point of view of children in KwaZulu-Natal.
3.2. Defining M&E

Definitions of monitoring and evaluation differ widely but in the context of the work of the CINDI Network the following definitions capture the scope of the M&E Framework. Monitoring refers to the systematic and continuous collecting and analysing of information about the progress of a project over time which helps an organisation assess its effectiveness in achieving its goals. Information gained from the monitoring process should be integrated into the planning process. Evaluation refers to a systematic assessment of a planned, ongoing or completed project which aims to answer specific management questions. Information gathered should help organisations assess what has been learnt which can inform future planning and actions. Issues of efficiency, effectiveness, sustainability and relevance of an organisations programme objectives are commonly central to evaluation exercises.
3.3. The M&E System1

M&E is no longer an add on to a project or programme but requires the development of a system and an M&E Plan. Setting up the system requires the following steps: 1. Defining the purpose and scope of the M&E system What is the purpose of the M&E system and how comprehensive should it be? 2. Identifying performance questions What do we need to know to monitor and evaluate interventions and to manage projects well?
1 The design of the framework draws heavily on the work of IFAD as outlined in IFAD. 2002. IFAD Managing for Impact in Rural Development. Rome. Italy
3. Planning information gathering and organisation How will the required information be gathered and organised? 4. Planning critical reflection processes and events How will we analyse the information and use it to make improvements? 5. Planning for quality communication and reporting How and to whom do we want to communicate what in terms of our project activities and processes?
6. Planning the development of the necessary capacities What is needed to ensure our M&E system actually works?
3.4. Different M&E Models

3.4.1. Traditional M&E
Traditionally M&E focussed on inputs (resources) and outputs (goods and services) Data collection was haphazard and there was no systematic and long term plan.
3.4.2. Results Based M&E

The shift has been away from inputs and outputs towards outcomes and impacts. This shift comes about as a result of the growing complexity of development issues, growing competition for scarce resources and a growing need to integrate with higher level strategic objectives e.g. the Millennium Development Goals. However, measuring/assessing outcome and impact is still an area in which new thinking is emerging all the time
3.4.3. Participatory M&E

As a result of HIV and AIDS and poverty, the situation in KwaZulu-Natal requires continual commitment to scaling up programmes especially in lesser developed rural areas and poor urban informal settlements. This calls for the greater involvement of people within these communities in all aspects of project planning and development, monitoring and evaluation. Increasingly participatory methodologies are being used to ensure local communities are fully engaged in development activities that affect themselves. Conventional M&E focuses on measurement of results of activities such as service delivery, information dissemination, behaviour change etc. Participatory M&E focuses on processes and results such as inclusion, collaboration, collective action. It encourages dialogue at local level and attempts to transform local people from being passive beneficiaries to active participants.
3.4.3. Developmental M&E

Developmental M&E places great emphasis on changes within people as indicators of development. These changes are critical to development but cannot be easily measured or log framed. A couple of important areas of change in people are listed below: Relationships More solidarity openness and trust Mutual co-operation Integrity and credibility Understanding Understand their world internal and external Understand their needs Ability to reflect and learn from experience
3.4.4. Integrating different approaches

The CINDI M&E Framework is strongly driven by the Results Based Model but if it is to ensure it promotes development and empowerment it needs to integrate aspects of participatory and developmental M&E. This will be crucial to building capacity of the CBOs and scaling up.
4. From programme cycle to results based M&E

4.1. The Programme Cycle
Initially the Programme Cycle formed the basis of conceptualising project and programme work. It was included in the learning sessions to help participants understand how monitoring and evaluation was originally conceptualised in the programme cycle. The main phases of the programme cycle included the following: 1. Needs assessment which is essentially an exercise to get information about community problems. This information has generally made some assumptions about how the needs/problems could be addressed. This was later expanded to being a situation analysis which included needs and resources. 2. Planning uses those assumptions to draw a programme of activities. This later gave way to strategic and operational planning which is an integral part of current programme development 3. Implementation refers to the follow through of the activities in accordance with the planning. Implementation is now expressed in operational or action plans which commonly outline activities, time frames, responsibilities and budgets for the achievement of various objectives. 4. Monitoring and review and evaluation are processes which originally assessed the use of inputs and successful completion of activities (outputs) in accordance with the plan.
Needs assessment
Review and Evaluation Planning
Monitoring Implementation
The Programme Cycle really represents a mechanism to ensure that service providers: think about what they are trying to achieve before they start develop the most effective plan of action to address their goals ensure that they monitor how effective they are being reflect on what learnings come out of the programme Monitoring and evaluation was kept simple and personal and staff judgement, anecdotal data or haphazardly collected field data was for the most part good enough for most programme managers and funders. (Indeed that is how most government departments worked as well)
What did the participants say during the learning session?

The focus of discussion was on who should be involved in the different phases of the project cycle. The following views were expressed: Ideally beneficiaries, implementers and funders should be involved in all the stages but this will differ from organisation to organisation and from programme to programme Questions were raised about how practical it is to involve all the stakeholders as it could slow the project process There was a strong feeling that funders need to have an understanding of what problems are being experienced by the programme implementing agencies so that they did not have unrealistic expectations There was the realisation that there is a need to be more structured and improve the quality of work.
4.2. Results Based M&E2

Almost all major donors and development organisations use the results based framework for M&E. Results commonly refer to the measurable or describable consequences of a project, programme or activity deriving from a cause-and-effect relationship.
4.2.1. The results based framework is based on assumptions

The results framework works on the basis of a cause-and-effect linkage which the organisation believes exists. This linkage should derive from an in depth understanding of the external environment including the social, political, economic and cultural context within which a programme is being developed as these may well affect development outcomes. Developing a results framework helps identify linkages and important external factors that might influence outcomes and require monitoring.
4.2.2. Levels of Results

Results can be considered as short, intermediate and long term. Within the results based framework there are four levels of results that can be summarised as follows:
2 Liberal use has been made of material from PACT. 2005. Building Monitoring, Evaluation, and Reporting Systems for HIV/AIDS Programmes.
1. Inputs and processes/activities This refers to the resources allocated and processes used to carry out an activity, project or programme. Inputs commonly include resources such as funds, equipment, training aids etc. Processes refer to the activities used to achieve the objectives such as training, capacity building, service provision, information dissemination etc. For example: A project may have an objective of training home based care workers. The funds and training aids are the input and the actual training is the process/ activity. Inputs usually produce results immediately (01 years). 2. Outputs This refers to the consequences of the inputs and processes. For example: The outputs of the above project may be 10 home based care workers having an expanded understanding of issues relating to those in need of home based care. Depending on the nature of the project outputs usually reflect a result achieved in a relatively short time period (02 years). 3. Outcomes This refers to recognizable changes in conditions in the targeted area. Outcomes may reflect behavior or economic change, or expanded service delivery and help us assess whether the activities have contributed toward development outcomes. Outcomes go further than outputs and attempt to answer the question so what? In this way one gets some understanding of the longer term consequences of the outputs. For example: 10 people have increased knowledge as a result of their training as home based care workers. So what? What difference has the training made to those infected or affected by HIV and AIDS? What changes has this brought in the targeted area? Outcomes usually reflect a result achieved over an intermediate time period (25 years). 4. Impacts Impacts are the overall and long-term effects of an intervention. Impacts are the ultimate result probably attributable to interventions over an extended period. For example: After 5 years the 10 home based care workers may have left the programme and thus the long term impact may be negligible alternatively they may have formed the nucleus of a comprehensive programme which contributed to improving the quality of life of those affected and infected by HIV and AIDS. Depending on the nature of the programme impacts usually reflect a result achieved over a longer time period (5-10+ years.)
10
4.3. The results chain

Inputs/Processes
which leads to ...
Outputs
which leads to ...
Outcomes
(So what?) which leads to ...
Impacts
Resources and processes utilised to produce a result Inputs commonly include: Staff Funds Equipment Facilities Training materials Processes/Activities: Training Mentoring Community mobilisation Establishment of services Establishment of networks Research
Short term change effects and results
Intermediate change effects and results
Long term change effects and results
Outputs commonly include: Knowledge, awareness understanding change Improved or expansion of access to services, programmes, networks, materials, information Quality change (improved programmes or services)
Outcomes commonly include: Increased coverage of target populations by interventions leading to: Behaviour change such as improved HIV and AIDS prevention practice Improvement in economic situation of children/ caregivers
Impacts commonly include: Quality of life change Overall health status changed such as increased survival Political change human rights policies implemented Socio/cultural/ economic and empowerment change, Resource management change
What participants said during the learning session

Participants felt the Result Based Management Framework is logical, focussed and future oriented. While the Logical Framework was currently used by many funders and participants it was felt that it had many weaknesses. There was a concern with regard to the time frame around impact. Some interventions may be able to demonstrate impact in less than five years but there were reservations about the longer term impacts. Participants felt that there will be a need to re-evaluate or re-assess the changes. Some organisations providing care for children were concerned that it would only be in adolescence and early adulthood that one would be able to truly assess the impact of the care received in childhood. Some of the terms seem to be used interchangeably and some of the funders including the government departments seem to use different terminology altogether. Consensus was reached that there is a need to have a system that works for CINDI Members and meets the requirements of funders.
11
Reflections on a CINDI Big Issue!

The issue of collaboration CINDIs mission is to foster a spirit of Ubuntu amongst CINDI Members, encourage collaboration, coordinate research and unlock resource opportunities for the benefit of all Members Participants raised questions about the meaning of collaboration What is collaboration? Are we collaborating? How do we collaborate? How do we know if we are really collaborating?
Collaboration defined Collaboration basically means working together. At a community level collaboration is an essential part of achieving project goals. Individuals and organisations rely on collaborative efforts and resources of others to achieve meaningful and sustainable outcomes. Collaboration can also be an outcome in itself, as it can be a key element for creating social infrastructure that is sustainable beyond the duration of the project. The main activities in the context of CINDI that facilitate collaboration include: Networking and facilitating dialogue to ensure common understanding. Disseminating information and creating a base for support. Cooperation. Aligning needs to limit duplication, clarifying roles, modifying activities to ensure tasks are completed and goals achieved. Coordination. Sharing resources, formalised links, group decision making such as the clusters, remaining focused on CINDI priorities. Coalition. Shared resources from the existing systems, formalised agreements, Shared decision-making and the development of new resources. Multi-sector collaboration. Shared vision, risks and rewards, highly developed communication.
12
5. Mapping the journey information needs and indicators

5.1. Why are indicators important?
Indicators are quantitative or qualitative factors that provide a simple and reliable means to measure achievement, to reflect the changes connected to an intervention, or to help assess the performance of a development actor. They can also serve as an early warning system to indicate future outcomes and the need for corrective action.
5.2. Counting measures versus proxy indicators

5.2.1. Indicators as counting measures
An indicator is ideally meant to provide evidence of change in conditions in a relatively simple form of counting measures. For example, number of home-based care workers trained, number of families being supported with food parcels, etc.
5.2.2. Proxy indicators

Alternate or indirect measures are sometimes used to stand in for another indicator when using counting measures is too difficult, time consuming, or sensitive. For example, the number of people undergoing VCT could be seen as an indirect measure of awareness. These are referred to as proxy indicators.
5.2.3. Indexed measures

Some results are complex enough or have enough identifiable steps that they lend themselves to being measured by an index. An index value is achieved by developing a scale of results and assigning points to the level of completion of each, and then adding the total to determine the level of measurement. For Example: Indicator: Level of CBO Capacity in Project Planning Directions: Score the following statements on a scale of 1 to 5, with 1 indicating little or no capacity and 5 indicating extensive capacity. Average the score to determine the organisations project planning capacity. CBO can formulate a project plan 1. Project planning is done in a participatory manner. 2. Project plan is available in writing with appropriate levels of formats and details. 3. Project plan outlines achievable and appropriate goals, objectives and activities. 4. Project plan reflects appropriate linkages with other key stakeholders. 5. Project plan includes a budget appropriate to the scope of the project. 6. Project reflects a basic plan for monitoring and evaluation. Indexed measures are very useful in many field programmes when a complex series of activities that jointly respond to an anticipated result for instance, quality of life of vulnerable children and quality of patient care need to be measured.
13
5.3. Steps in selecting indicators

Developing appropriate and useful indicators is a fairly straightforward process, but requires collaboration and consensus-building and needs to be interactive. The following suggestions can be helpful in the selection process. Although presented as sequential steps, some can be effectively undertaken simultaneously. Step 1: Clarify the results statements; identify what needs to be measured. Good indicators start with good results statements that people can understand and agree upon. In developing a good results statement the following is important: Desired results should be carefully considered What is expected to change a situation, a condition, the level of knowledge, attitude, behaviour? Is change expected to occur amongst individuals, families or communities? Before developing indicators be clear about the relationship between the activities and their intended results.
In the context of poverty and HIV and AIDS in KZN there are a number of factors that can impact on results. Step 2: Create an initial list of possible indicators by: Internal brainstorming Identifying and building on the experience of other similar organisations Consulting with beneficiaries and experts Identifying existing secondary data sources (data collected by someone else that you use to measure a result). Key data should be available from the local municipality and Statistics South Africa. Referring to those developed by the CINDI Network and existing indicators developed by UNICEF, MDG, Global Fund.
Step 3: Select the best indicators. Consider carefully what the indicators should be achieving and narrow the list to the final indicators that will be used in the monitoring system. Remember to take into consideration the realities of the environment you work in and the costs and time involved in gathering information. Limit the number of indicators used to track each objective or result to two or three. Select only those that represent the most basic and important dimensions of your objectives. Step 4: Draft indicator protocols. Protocols are instruction sheets that capture the reason for selecting indicators, describe the indicator in precise terms, and identify the plans for data collection, analysis reporting and review. Step 5: Collect baseline data. The baseline is a record of what exists in an area prior to an action. The baseline values establish the starting point from which change can be measured. Step 6: Refine indicators and protocols and finalize your selection. Based on initial data collection efforts, refine your indicators and/or your data collection instructions.
14
What participants said and did during the learning session

1. The participants drew up a list of indicators for the CINDI Network. These were aligned to the specific needs of CINDI, as well as making use of the UNICEF Framework for responding to children in a world of AIDS. 2. There was a lot of discussion around the indicators particularly around the issue of improved quality of life which is summarised below: Improved educational status of children should include additional indicators about absenteeism and age for grade. Emotional well being of vulnerable children should include an indicator about children having access to a caring, consistent significant other. Improved protection of vulnerable children should include an indicator about the proportion of children aware of childrens rights and available resources. The issue of spiritual and cultural well being as an integral part of quality of life was discussed at length. Some of the points put forward include: African culture is in such a state of flux that it would be difficult to identify specific patterns of behaviour and label them as cultural. Self esteem and a sense of belonging were closely linked with cultural attributes derived from the immediate social environment Spirituality deriving from whatever belief system helped people cope when the going was tough and therefore was very important What was critical was that children growing up should exercise informed options which would sometimes embrace western culture and sometimes embrace traditional culture. The question was who would help children understand the options? It was felt that the output indicator increased quality of service and care should also be represented as an outcome. 3. The indicators outlined should also be linked with the work of the five CINDI Network Clusters i.e. the Psychosocial Cluster, the Community Development Cluster, Children in Care Cluster, School and Youth Development Cluster, and the Home Based Care Cluster.
15
6. Tools for gathering information3

3 This section is adapted from the work of IFAD as outlined in IFAD. 2002. IFAD Managing for Impact in Rural development. Rome. Italy
There are a number of tools that can be used to gather information for monitoring and evaluation purposes. Some tools are designed to be used to assess information relating to specific focus areas. The table below outlines a number of tools and the specific focus areas in which they are most appropriate. Their purpose and use in a monitoring and evaluation context including how they may be useful in the context of CINDI is set out below. The steps for using these tools are also set out.
Method
Purpose and possible relevance to CINDI Network
Steps
Core M&E Methods The 10 methods that follow reflect the standard core methods commonly used in M&E exercises Stakeholder Analysis In the context of M&E this helps decide who should be involved in the design of an M&E plan. It is helpful in providing a foundation and strategy for participation throughout the project. In the context of CINDI it is critical to identify the range of stakeholders that need to be included in the planning, implementation and M&E. It is particularly important to include project beneficiaries and representatives from local community structures. Documentation Review This helps to understand the historical evolution and performance of a project. In this way important baseline information can be obtained which helps to inform the development of an M&E system. The changing role of CINDI in response to a changing external environment highlights the need for on going transformation and alignment which is captured in the documents. Biophysical measurements This helps measure physical changes related to any indicator e.g. weight for age etc. From an M&E perspective it can provide statistically reliable information for measuring change and impact. In the CINDI context the use of health related indicators are important for monitoring the progress of vulnerable children. Direct observation To obtain useful and timely information by observing what people do or behaviours. From an M&E point of view this method is often used to complement collected data, understand context and help explain results. In the CINDI context this is an important tool for psycho-social assessments and the work of the Psychosocial Cluster 1. Clarify the main purpose of the stakeholder analysis 2. Decide on criteria for including stakeholders 3. List all organisations and individuals that fit your criteria 4. Reach agreement on how to involve people
1. Be sure about which issues you want to understand better 2. List all possible sources of existing information 3. Prioritise those most likely to provide the required information 4. Collect documentation and check reliabilityanalyse it in terms of the questions you were trying to answer 5. Identify other collection methods to address gaps 1. Be sure about the information being provided by the indicator 2. Ensure availability and standardisation of equipment 3. By comparing recorded data, comparisons can be made over a period of time 4. The Road to Health cards for under fives from public sector clinics are useful in monitoring health related progress 1. Agree on guidelines for what needs to be observed 2. Choose an appropriate observer or observers 3. If necessary train in observational skills 4. Collect and record data as agreed 5. Organise time to discuss observations with relevant stakeholders
16
Cost-Benefit Analysis
To provide a format for assessing the range of benefits and costs surrounding a decision. In the M&E context this method is commonly used to evaluate a project by comparing final costs and benefits against those proposed in the design of the project. In the context of CINDI it is important for assessing the value of training and monitoring costs of services.
1. You may need someone with the relevant financial background 2. List all project activities 3. Calculate all project costs for the period 4. For each project activity estimate the benefits which may continue into the future 5. If possible include beneficiaries so that they too can comment on outcomes 1. Agree on purpose and information needs of the questions 2. Decide whether a questionnaire (just a set of questions) or survey (which includes questions and observations is required) 3. Ensure questions are focused and well formulated 4. Agree on target group and number of respondents to be included in sample 5. Pretest questionnaire to ensure they are appropriate and give required information 6. Collect and use information
Questionnaires and Surveys
To gain data from a large number of people in a structured way. Commonly used in M&E exercises as they allow for focussed data collection around indicators. Potentially a very important tool for monitoring and evaluating programmes within the CINDI Network.
Semi-structured interviews
To gain information face to face from an individual or small group using a series of broad questions to guide the conversations but allowing for new ones to arise during the course of the discussion. From an M&E perspective this tool helps to develop an in depth understanding of qualitative issues. Semi-structured interviews could be very useful when trying to understand attitudes, behaviours and beliefs. They can be used in conjunction with other methods. They therefore could be a useful tool for the CINDI Network.
1. Define the purpose and information needs of the inquiry and formulate an interview checklist of open-ended questions. The questions should encourage expression of opinions 2. Agree on who and how many should be interviewed 3. Train people to understand the purpose and develop the proper skills 4. Pretest the interview questions 5. Undertake the interviews and analyse outcomes
Case Studies
To document life stories or sequence of events over a period of time related to person, location, and organisation in order to understand impact. From an M&E perspective they help to give a face to data and help to reflect context of some of the data. A very important tool for CINDI to contextualise local realities. Members should attach case histories to reports to give an idea of process and context remembering to respect confidentiality at all times.
1. Define the purpose and precise information needs of the case study 2. Decide how individuals, households, organisations etc will be selected 3. Decide how the information will be obtained from households, individuals etc. 4. Develop question checklist to guide information collection 5. Collect information and compile case history
Discussion Methods for Groups Group based methods are important when working with participatory projects. Basically the tools are used to encourage discussion and sharpen the focus on issues. In this way they can help an organisation gain an understanding of community perspectives and build these into M&E frameworks. Focus Groups To use group discussion to collect general information, clarify details or gather opinions in a smaller group situation. In the M&E context they are useful for assessing opinions about change, or assessing the quality of project services or providers and identifying areas of improvement. 1. Determine the participants (4-8 people is ideal) which can be homogeneous or heterogeneous 2. Present the group with a broad question e.g. what impact is the CINDI mentoring programme having 3. Discuss the question for the time agreed beforehand (one or two hours maximum) with minimal intervention from the facilitator
17
Focus Groups (continued)
Focus groups can be very important in assessing perceived outcome and impact. The tool can be used to help get information during the planning stages of an intervention. For this reason in the context of CINDI they can used to assess longer term outcomes and impact of programmes. To identify the strengths, weaknesses, opportunities and threats in relation to a project or group and how such an assessment will change over time. From an M&E perspective this method is useful when qualitatively assessing the services provided by a service provider, exploring relationships etc. This tool can help CINDI when involving stakeholders in participatory processes.
4. Take detailed notes of the discussion. Focus groups are best if facilitated in pairs one facilitates and one takes notes 5. To ensure reliability it is suggested that different focus group discussions be held around the same topic
SWOT Analysis (Strengths, Weakness, Opportunities, Threats)
1. The group defines, discusses, and records as many factors as possible under the various headings 2. One way of using the headings is as follows: Strengths: Aspects that are working well in a project Weaknesses: Aspects that have not worked so well in a project Opportunities: How to build on strengths and overcome weaknesses Threats: The aspects that constrain or threaten the range of opportunities 1. Start by asking people through meetings held with individuals, households, communities etc how they would like to be in the future in response to certain focus areas 2. In practical terms ask for people to reflect on an individual basis for about 15 minutes and then get sharing in sub groups so that some consensus emerges 3. The dreams can be written down or represented as a symbol. Dreams can be specified with clear time frames for achievement 4. Once articulated dreams can be expressed as indicators 5. The discussion can be repeated every 6-12 months as a means of monitoring progress 1. Choose the central theme whether it is an M&E indicator or question or scenario to be dramatised 2. Decide who is to work on a drama piece or get different groups e.g. women, young people, elderly each to present their views on a topic like the prevention of HIV and AIDS 3. Participants should construct their drama, in which they present their opinions and thoughts on the topic being discussed 4. Facilitators should record key issues in the performance 5. The group then discusses the issues emerging from the play and conclusions drawn
Dreams realised or visioning
To have informed discussion about peoples dreams or shared visions for the future of the project or other activity. From an M&E perspective this is a good method for identifying if primary stakeholders feel their well being is increasing or not or whether a project is based on peoples visions for development. This is a powerful tool for capturing community opinions and understandings, planning and developing a M&E system all in one. In the CINDI context this tool is very relevant and can be used to help plan projects.
Drama or role plays
To encourage groups of people to enact scenes from their lives concerning perceptions, issues and problems that have emerged relating to a project intervention. From an M&E perspective they can also help a group identify possible indicators for a project. A very valuable tool when working with rural people who seem not to articulate problems as well as they can perform them! Drama and role plays can often create a non confrontational way of dealing with differences e.g. generational and gender. In the CINDI context this tool is valuable means of capturing local peoples perceptions of development challenges.
Methods for Spatially-Distributed Information These methods deal with information that has some level of geographic distribution which could relate to health, education or economic issues. Maps can help locate biophysical, economic and social indicators. They can be very elementary such as drawn in sand at a neighbourhood meeting or make use of GPS systems. (CINDI Members should bear in mind Valley Trusts offer to integrate information into their GIS system) Sketch mapping To provide a visual representation in a particular geographical context based on stakeholders perceptions of any focus issue or indicator that is being monitored and evaluated. 1. Ask participants to draw boundaries of the area under discussion. This can be represented on paper or using local materials such as sticks, stones seeds etc.
18
Sketch mapping (continued)
Perhaps in the HIV and AIDS context it is important to map areas that are not being adequately served or lack water and sanitation. In the context of CINDI this tool maybe difficult to implement and maintain but it is important that projects reach the most vulnerable households and communities.
2. Participants should draw the main features such as roads, rivers etc. and reflect what is of interest to them or significant to the development effort. 3. Once a base map is in place and translated into a paper map it can be used for social mapping of households and the monitoring and evaluation of the spread of programmes 1. After noting topics and indicators to be observed decide who could provide relevant information 2. If a map is available use it to identify what the route will be. (Can be an hour or a day depending on the size of the area) 3. Already available information should be used to form the basis of observations and measurements during the walk 4. As the walk proceeds the observations should be probed to give further insights and should be recorded 5. Record what has been seen and discussed on a schematic diagram to be used for future monitoring transect walks 6. The frequency of the walks can vary to provide a means of monitoring and evaluation 7. Compare the different observations over time to give an indication of progress
Transects
To undertake a structured walk through an area to observe particular indicators (such as places that present health and safety risks or difficulty in accessing services). In the context of CINDI this can really help NGOs and CBOs to develop a common understanding of local situations. It has the potential of being an important monitoring and evaluation tool. It can also help service providers develop a common understanding of issues with local participants.
Methods for Time Based Patterns of Change These methods help us understand changes related to specific blocks of time e.g. September of one year to September of another year or simply critical events that have occurred during the time span of a project. Diaries To help record events, facts, reactions and/or opinions over time by individual stakeholders or groups. From an M&E perspective, this method is useful for capturing details that might otherwise be missed and that might explain the context in which change occurred. The method might promote understanding of how a change came about or focus on specific performance areas or indicators. In the context of CINDI this tool is probably a useful means of gathering information to support statistical information especially at local community level. Historical trends and timelines To obtain historical understanding of sequential changes that have occurred, relating to particular points of interest. From an M&E perspective this could focus on specific indicators which can be used as triggers in discussions to assess certain changes that can be attributed to project activities. In the context of CINDI this can be a very helpful tool to monitor community perceptions of change and historical aspects of problems. Particularly in relation to the changing environment in which CINDI works this could be very helpful e.g. changes in attitude to VCT following the introduction of ARVs. 1. Introduce the diary early in the life of the project 2. The form and focus of the diary and who records the entries 3. The diaries can be individually written documentation or based on group discussion 4. Diaries can then be used to stimulate discussion and identify changes that might be particularly significant 5. Can be used in conjunction with other statistical methods 6. Reviewing the diary can also be a helpful means of supervision 1. Agree on events/indicators important to the situation at hand. (e.g. the prevalence of AIDS) 2. On a large sheet of paper draw rows and columns to make a matrix. List dates at the top e.g. 1 year ago, 5 years ago, 10 years ago 3. Write in key events/influences that might have impacted on the spread of the disease 4. Work with groups to fill in the information and use it for discussion 5. You might add a fourth column and label it the future in which people identify what they would like to see
19
Methods for Analysing Linkages and Relationships Fundamental for all projects is an understanding of changes in relationships and linkages between groups, such as primary stakeholders and organisations and also between issues, activities, causes and effects. This cluster of methods provides ideas on how to analyse such issues by using different visualisation techniques. (Perhaps very important for CINDI in relation to their members and their members in relation to CBOs). Rich Pictures or Mind maps To make a pictorial representation of the elements that need to be considered or are important to a particular (project) situation, including stakeholders and issues, and the interactions and connections between them. From an M&E perspective, a rich picture can help identify what aspects of a situation need to be monitored, which change indicators to track and/or which key stakeholders need to be included in the M&E effort. In the context of CINDI this may be a useful tool for analysing relationships within the CINDI Network. 1. Using a large sheet of paper and symbols, pictures and words, draw a rich picture (or mind map) of the situation (project/group) that you wish to evaluate. This is best done with about 4 to 8 people and it takes a half to 2 hours 2. Start by asking people to note all the physical entities involved, for example, the critical people, organisations or aspects of the landscapes 3. Ask people to present their rich picture by describing the key elements and key linkages between them 4. If there is more than one group, compare their pictures and cluster the ideas that are similar and those that diverge. In this way you can identify the most important issues to discuss, such as critical topics to focus on in an evaluation, possible indicators or key stakeholders to include in M&E 1. Make sure that the topic being assessed is clear. For example, the idea of the capacity of an organisation must be very clear and understood by all of the participants. Have the participants agree on which criteria to use to assess the quality of the topic. These are, in fact, the indicators 2. The selected indicators are arranged in the form of a wheel, with each indicator being one spoke as on a bicycle wheel. The spokes are spaced equidistant to one another. The indicators can be represented by words or symbols 3. Next, participants agree on how to rank each indicator from 0 as the lowest/worst level to 100 (or 25, 10, etc.) representing the highest/ best level. It does not matter if 0 is on the outer edge of the wheel and 100 in the centre or the other way around, as long as all the spokes on the wheel are the same 4. Once the wheel has been made, assess each indicator. If doing this with a group, then there will need to be consensus on the final score (or an average figure). Indicate the place on the spoke that corresponds with the final score given. Then join all the scores, which are marked as points on the spokes, to show what ends up looking like a spider web. A look at the spider web gives a quick overview of key weaknesses and key strengths. The weaker aspects of the issue being assessed are those that have scores closest to 0 5. Previously made wheels can be revisited at subsequent monitoring sessions in order to compare how the situation changes over time
M&E Wheel (Spider Web)
To provide a visual index that helps in assessing the issue being monitored or evaluated in terms of its ideal, or in comparing two or more monitoring sites and how they change over time. This method can also be used to measure how well a project is meeting anticipated targets, or how an organisations capacity changes over time. From an M&E perspective, the spider web provides a visual means of measuring changes in ratings on chosen indicators. In the context of CINDI this tool could be useful for analysing transformation and relationships within the CINDI Network. See example.
Assessment of Organisational Competence of CBO Project Management 20 10 0
Capacity development Fundraising Communication
Financial management Community linkages Alignment
M&E Areas of organisational competence Organisational score
20
7. Critical reflection: thinking through the journey

7.1. What is critical reflection?
Critical reflection means creating the time and space in a project to reflect on experiences and data and look for new insights into the situations the interventions are addressing. All too often the commitment is to collecting, processing and reviewing of data that is part of the M&E system. Critical reflection goes beyond all the data and reports and asks the question, What is really happening in this project? Other questions which can be asked are: Why is it happening? What are the implications for the project? What do we need to do next? Some of the issues that are of particular importance in critical reflections include: What assumptions were made about the project during the planning stages? Are they still valid? Are the interventions having the predicted outcome? If not, why not?
7.2. The importance of critical reflection for M&E

M&E is often viewed as primarily an administrative/bureaucratic obligation to meet the needs of the funders. It is not often viewed as an exercise to benefit the project. Commonly there is little appreciation of what can be gained from the frank analysis of projects with disappointing outcomes. This is of particular value when further planning is being considered. Changing attitudes towards the value of M&E for projects and project staff requires a commitment to building capacity, sensitisation to the realities in the internal and external environment and putting in place effective incentives. Project stakeholders will improve their interventions by regularly setting aside time to reflect on the data and acknowledge the lessons that are being learnt.
7.3. Process aspects of critical evaluation

Project M&E requires planning a series of reflective events including regular staff meetings and informal sessions to more formal supervision sessions and mid term reviews. The events whether self organised or formally organised should happen alongside the data gathering exercises and make use of some of the data gathered. The learning will evolve as participants explore important issues and question assumptions in a group setting.
7.4. Importance of individual reflection

While group processes are important, individual reflection should not be overlooked which includes some of the following activities: Review and reflection of job descriptions Encouraging reporting that reflects their opinions particularly in relation to achievements, problems and proposed solutions,
21
Regularly asking project stakeholders their views Providing constructive feedback Seeking feedback from the people you deal with if you are a manager Valuing field visits and exchange Rewarding critical reflection
7.5. Capturing lessons learned with stakeholders

In the current climate wherein a number of new approaches are being explored there are a lot of opportunities to reflect on the lessons learnt which it is important to recognise and document. When formulating a lesson learnt consider the following: Include a generalised principle that can be applied to other situations Explain the lesson in the context of the project Justify the lesson by means of providing evidence of the learning Check the lesson is not too general or too specific to be useful Lessons learnt should be documented with at least five elements: The theme of the lesson learned What was the original understanding or assumption What is the revised understanding or assumption One or two examples to demonstrate the revised understanding How the project obtained this insight
7.6. Developing a plan for critical processes and events

Critical processes and events should be planned throughout the life cycle of the project and a plan for such events should form part of the M&E system.
Critical reflection Processes of Events Participatory review of project strategy Purposes and description Update the situation analysis, revise problems/ visions and assumptions Whom to Involve Timing
Representatives of intended primary stakeholders, project staff and facilitator Representatives of intended primary stakeholders, project staff and facilitator
Three day workshop at start up
Development of M&E Plan with stakeholders
Assess different information needs, take note of who is doing what agree on priorities, revisit indicators, agree on responsibilities Discussion of successes and problems
Four or five full day meetings during the first six months of the project One day meeting every three months
Quarterly progress reviews by project management
Senior project management staff and partner organisations
22
Field visit
First hand observation of what is happening in the field, informal discussions about how activities are being implemented. Summary of key successes and challenges. Ideas for changing project outputs and assumptions and review of implications of changes Focussed discussion about the strategy and key activities and revision of priorities and identification of lessons learned Explain the purpose of the session, agree on what the project and stakeholders would like to get out of the session and what preparation is required by whom
Field staff, supervisors of field staff, and project manager Representatives of intended primary stakeholders, staff of implementing partners, all project staff Key stakeholders of the project, intended primary stakeholders, field and project and management staff. Key stakeholders of the project component, intended primary stakeholders, implementing partners field and senior project staff
Weekly visit for field staff, monthly for project director Once a year
Annual project review
Periodic review workshops of key project components
Once a year in the first two years, after that once every two years One month prior to supervision mission.
Preparation for supervision session
What participants said:

1. Participants acknowledged that critical events seemed to be a very important of way of bringing depth to monitoring and evaluation efforts. It was felt that sometimes organisations embarked on strategies without fully understanding the implications and this resulted in difficulties and miscommunication. 2. There is a need to develop a plan for critical reflection processes or events as an integral part of M&E activities within the CINDI Network. 3. Potential funders, managers from the CINDI Network Office and CEOs of member organisations needed to visit programmes on the ground so that they could have first hand experience of the realities and constraints that organisations and field workers are faced with. 4. Participants also felt that when issues are not being resolved and create tension within the work setting it was felt that well planned critical reflection processes could help to assess progress and identify challenges. It was felt that some of the tensions between CBOs and NGOs and some of the issues around scaling up would benefit from a series of critical reflection processes.
23
8. Disseminating information
8.1. Importance of communicating M&E findings
Generally speaking two sets of information will need to be communicated: A wide range of stakeholders including participants should have the opportunity to give feedback on M&E findings. This will help to check accuracy and increase ownership of the M&E exercise. Once there is consensus on the findings they can be communicated to funders, project beneficiaries, government departments and other projects. In this way not only funders needs for accountability will be met but also the findings will fulfil an advocacy role.
8.2. Developing a strategy to communicate M&E findings

Know your audience and ensure your presentation is appropriate to the level of the audience Build a communication plan into your M&E system Ensure the resources are in your budget for a communication plan.
Things to think about when disseminating information Ensure your message is clear. Different stakeholders will inevitably focus on different types of information. Agree on frequency for communicating information. Try to organise feedback sessions for key meetings at the different levels. Ensure Timeliness. Ensure feedback sessions are timely so that issues can be dealt with when there is still a level of momentum in project activities. Consider location. Think about the various opportunities for communication at the different levels. Make it a point to ensure information reaches people so that they can give feedback. Make effective use of graphic information to facilitate analysis Visually presented information is easier to understand. When addressing boards and funders learn to use Power Point. At local level, maps and other visual aids may help with analysis and communicate findings Keep focussed on your task. Feedback sessions can easily become a talking event with no real outcomes. Plan the event carefully to include clarifications, additional insights, conclusions and action steps. Make it sincere and thought provoking not just a glib exercise.
24
8.3.
Using different media to communicate findings
Written Reporting Remember to put a human face on impact reports by including stories from people themselves. Work out a reporting strategy that fits the organisation and the level of the field workers. Templates are helpful but they should be drawn up with the local realities in mind. Think through the information that you need to know and that which would be nice to know and keep it simple. Dont let the reporting become the main thrust of the project and an end in itself. Oral Presentations Important information is obtained through personal contacts and oral presentations. Do not underestimate their value and ensure time is given for discussion and feedback. Visual presentations Visual displays such as graphs, charts and maps help illustrate data presented in oral presentations. Photographs, video clips etc also help to convey a picture of what has been done. At local level dramatic presentations can also be a powerful means of communicating issues around a project.
The CINDI Network and communication

The issue of communication 1. Remember the CINDI Web Site is there and has approximately 5000 page requests a month. It thus provides a very effective means of communicating your work and sharing experiences. 2. The CINDI Network relies heavily on electronic communication which is making it very difficult for the CBOs who have no access to the internet to keep up with developments. This remains a critical communication challenge for the CINDI Network. 3. To translate materials into isiZulu or not is a big CINDI issue! There is a view that people at local community level are excluded from broadening their knowledge and expressing their own views because of their difficulties in expressing themselves in English. For this reason materials and information must be available in isiZulu. There is another view that people must get to grips with communicating in English which appears to be the language of globalisation. Costs of translation are prohibitively expensive and with increasing access to education within a few years lack of fluency in English will be a thing of the past. The debate continues ...
25
9. Measuring, demonstrating, or assessing impact?4

9.1. Introduction and definitions
As has been pointed out in the learning sessions the focus of M&E in recent years has shifted from inputs and outputs to impact giving rise to the results based approach to M&E. The shift to assessing impacts has been a feature of both the public and private sector. In terms of the various frameworks for strategic planning and M&E most people are generally comfortable with some of the terminology. The terms inputs, activities and outputs for example, are all fairly well understood. Monitoring these dimensions of a project or activity is fairly easy as it is basically a counting exercise and results are commonly expressed in numbers. Outcomes refer to the benefits or changes for project participants. They are harder to measure as they are less tangible and not readily countable. Some of the outcomes of projects within the CINDI Network might include, for example, improved care of children orphaned or made vulnerable by HIV and AIDS, or increased understanding and capacity amongst youth to prevent the spread of HIV and AIDS etc. Outcomes are commonly set out in an organisations objectives. Impact is an even more difficult to define and measure. Commonly the term is used as a verb meaning to have an effect on something e.g. How many children does the CINDI Network impact on? However, as an M&E term and noun meaning effect it is used to describe the all important longer term (hopefully developmental) effect of an intervention. One definition that has been compiled describes impact as any change resulting from an activity, project or organisation. It includes intended as well as unintended effects, negative as well as positive and long term as well as short term. This very broad definition will probably need narrowing down to make it operational in a given context.
9.2. Measuring or demonstrating impact?

As has been mentioned earlier the shift to funders wanting evidence of impact is still fairly new and the topic of much debate. The aim is to ensure efficient resource allocation and utilisation. The emphasis on measurement usually comes as part of a top down approach with funders, governments and the media wanting measurable results as a means of assessing performance.
4 Much of this section has been adapted from Measuring Impact: Case Studies of impact assessment in small and medium sized voluntary organisations. The UK Voluntary Sector research Group. May 2003
Demonstrating impact is more likely to reflect a more community driven bottom up approach whereby stakeholders including project participants assess what has been achieved towards a set of goals with a certain level of resources. If efficient resource allocation is the most critical issue to the assessment of impact then demonstrating impact may not succeed in proving this point.
26
9.3. Some of the difficulties with current approaches

There are a number of difficulties with current approaches to measuring impact including the following: There are no tools that can assess all aspects of impact as usually a framework is set at the beginning of the process resulting in unanticipated impacts not being captured. The short time scales and the expectation that organisations can report on impact as soon as a project is completed makes it difficult to assess long term impact. The difficulty in proving a causal relationship between a particular activity and a particular outcome especially when measuring long term impact. Organisations working with prevention programmes will have a lot of difficulty proving positive impacts when success for them is something not happening e.g. prevention of drug addiction, teenage pregnancy etc. It is very difficult to measure/demonstrate or assess impact if there has not been a base line survey and generally this is not easy for smaller resource constrained organisations. There is still a negative attitude towards M&E. Funders expectations are often unrealistic which leads to reports being drawn up to suit the funder rather than to assess progress for the sake of the project. Generally there are difficulties in reflecting unanticipated impact with the result that these tend to be ignored or denied, e.g. the widespread fraud around the child care grant. Impact assessment is generally a costly and time consuming exercise which creates problems for small organisations. Current approaches seem geared primarily to quantitative data. There is often difficulty with qualitative approaches and much of the work undertaken by CINDI organisations does not readily translate into quantitative data.
9.4. Dimensions of impact

Impact has a number of dimensions as indicated below: Impact can have economic, social or political dimensions It can be individual or collective It can have a geographic and specific locality focus It can be time specific It can have intended and unintended dimensions It can be short term or long term.
27
9.5. Examples of basic models of impact evaluation5

Model 1. Randomized pretest post-test evaluation. Design Subjects (families, schools communities etc) are randomly assigned to project and control groups. Questionnaires or other data collection instruments are applied to both groups before and after the project intervention. Additional observations may be made during project implementation. Comment 1. Some of our communities have had enough of being part of studies and research. It may be difficult to sell this model 2. May be costly and time consuming 3. The debate will be how much time needs to elapse between programme intervention and impact? 4. Nevertheless if resources were available it could be used as a model for impact evaluation Similar concerns as above. Can you just set up control and experimental groups as required in our communities?
2. Quasiexperimental design with before and after comparisons of project and control populations. 3. Rapid assessment ex-post impact evaluations
If randomisation is not possible then a control group can be set up which matches the project group
Some evaluations only study groups affected by the project while others include matched control groups. Participatory methods can be used to allow groups to identify changes resulting from the project, who has benefited and who has not, and what were the projects strengths and weaknesses. Triangulation is used to compare the group information with the opinions of key informants and information available from secondary sources. Case studies on individuals or groups may be produced to provide more in-depth understanding of the processes of change.
Could be used to assess wide range of programmes relevant to the CINDI Network
5 Adapted from Monitoring & Evaluation: Some tools, Methods and Approaches. World Bank. 2004
28
9.6. Some of the tools used in assessing impact

Questionnaires and Surveys These can designed to collect standardised information. They can serve as an important means of obtaining baseline information as well as changes over a period of time. Impact Indicators Indicators to demonstrate impact can be designed to assess psycho-social impact and the impact of training e.g. I feel less isolated in the community or I am better able to cope with my problems. Participatory approaches Participatory approaches are those that engage participants and other stakeholders in the design and implementation of M&E exercises. They often generate a real sense of ownership of the M&E processes. They are an important means of building capacity and empowering poor people. The disadvantage is that they are sometimes not considered to be objective and can be manipulated by powerful stakeholders. Rapid appraisal methods Rapid appraisals are quick and usually low cost methods to gather the views and feedback on the impact of a project. In this way information can be obtained to guide further policy decisions. They methods allow flexibility to explore new ideas. They tend to represent very localised views and are viewed as less credible and reliable than formal surveys.
9.7. Basic learnings from various case studies

Set realistic goals! One can be tempted to try to measure/demonstrate/assess every aspect of an organisations work. However, this is an almost impossible exercise as there is no single tool that can be used and assessing impact is a costly business. Thus it is important to be selective about which impacts you want to measure and it is helpful to ask yourself a few questions: What are the reasons for assessing impact? If it is for a funder what is it they want to know? Is it for internal learning so as to effect improvements? Is it necessary to measure it or will it be more meaningful to reflect it?
Such guiding questions can help with the design. What has worked well?6 Following the documenting of best practice in a number of case studies the following observations were made about what works well. One thing that worked well was that impact assessment was a process that contributed to capacity building. By using a participative approach we were able to align our aims with those of the major stakeholders, which meant they were willing to take part in the process. Bringing people together in focus groups promoted the development of inter-organisational working, reflective learning and strategic planning. Therefore, undertaking
6 Summary drawing from four case histories in Measuring Impact: case studies of impact assessment in small and medium sized voluntary organisations. The UK Voluntary Sector Research Group. May 2003.
29
an impact assessment was potentially a very useful process. Think of an organisation as part of a system or network, taking into account all stakeholders and activities carried out by the organisation. This will help to highlight impactees, and prevent you from thinking about the intended beneficiary group as the only people who are impacted on; If interviews are being used, keep the structure flexible, allowing unanticipated impacts to be uncovered; Use visual aids such as a network map which represents where the organisation sits in relation to its stakeholders. This is a useful prompt during interviews; Be realistic about what is to be achieved with the time and resources available.
9.8. Learning for the CINDI Network

Dont fall into traps trying to measure unmeasurable or very complex impact! Be sure when planning a project what impact you want to achieve and how you will assess/demonstrate/measure that impact! Be aware of and deal honestly with unintended outcomes. Try to work hard at getting baseline data so that you can make comparisons. Always keep things realistic. The more participatory M&E actions are the greater ownership and empowerment there will be as long as organisations do not become a target of funding related power games and manipulation.
30
10. The basics of evaluation7

10.1. What is evaluation?
Earlier evaluation was described as: ... a systematic assessment of a planned, ongoing or completed project which aims to answer specific management questions. Information gathered should help organisations assess what has been learnt which can inform future planning and actions. Issues of efficiency, effectiveness, sustainability and relevance of an organisations programme objectives are commonly central to evaluation exercises. Evaluation is most helpful when an organisation wants to assess results on a deeper level and try to understand the underlying reasons why change is occurring or not occurring in the field. This information is then used to further develop both the activities and the conceptual (results) framework. Evaluation focuses on why results are or are not being achieved, on unintended consequences, or on issues of interpretation, relevance, effectiveness, efficiency, impact, or sustainability.
10.2. Types and purposes of evaluation

1. Formative Evaluation: Formative evaluation normally takes place at the concept and design phase of a programme. Formative evaluations help organisations understand the following: The operational setting of a potential programme What the focus of an intervention or programme should be What exactly is required in terms of inputs and activities Where the geographical focus of the programme should be Who should be involved in the intervention How the intervention will be carried out. How the intervention should be managed
Formative evaluation provides the information needed to define realistic goals, objectives, and strategies for a programme. Methods to undertake formative evaluation may include: Reviews of existing information and knowledge (literature reviews and discussion with potential beneficiaries and partners) Focus group or small group discussions with relevant stakeholders Individual in-depth interviews or short surveys 2. Process Evaluation Process evaluations take place once activities are underway and focus on tracking the efficiency of a given programme or organisation. Process evaluations focus on: Providing information relating to what extent planned activities are being implemented, How well services are being provided, in what timeframe, at what cost, and with what result.
7 Much of this section derives from the following publication: PACT. 2005. Building Monitoring, Evaluation, and reporting Systems for HIV/AIDS Programmes.
31
Analyse how efficiently inputs (money, time, equipment, personnel, etc.) are being used in creation of outputs (products, results, etc.).
Process evaluations help organisations analyse what they planned to do versus what they actually are achieving and are used to adjust implementation strategies. Process evaluations are often conducted informally (staff meetings, etc.) at regular intervals during the programme year to assess progress toward achieving the results. They need to be based on performance data (results from indicator data collection) as well as staff observation of projects and programmes. Methods for process evaluation include: Reviews of service records and regular reporting systems Key stakeholders interviews including project participants Direct observations Population-based surveys, etc. 3. Mid-Term Evaluations Most organisations are also familiar with mid-term evaluations, which are process evaluations conducted halfway through a strategic cycle. Mid-term evaluations may be required or planned for some projects but should also be considered as a way to review programme or strategic plans. Mid-term evaluations usually: Assess the organisations progress in implementing activities Assess progress toward achievement of objectives or benchmarks Assess whether interventions and budgets are sufficient to reach desired results Identify barriers to achievement of results, objectives, and activities Identify opportunities, unanticipated accomplishments, or innovation Specify course correction or changes required
Mid-term evaluations usually suggest actions to bring mid-course adjustments in the last half of the programme. Mid-term evaluations should endeavour to be participatory in nature and include stakeholders in the analysis of programmes. 4. Effectiveness Evaluation Effectiveness or impact evaluations normally take place toward the end of a programme intervention and focus on assessing the overall outcomes and impacts attained. Effectiveness evaluations focus on questions pertaining to what results have been achieved, what short-term and intermediate effects were observed as a result of programme effort, and what the outcomes mean. Does the programme make a difference towards the larger development impact sought? 5. Final Evaluations Most organisations are familiar with final evaluations, which are normally effectiveness evaluations. Final evaluations may be required or planned for some projects but should also be considered as a way to review and update your strategic plan. Final evaluations normally: Assess why progress toward planned results has been positive or negative Test the validity of assumptions underlying a results framework
32
Assess how well needs of different project participants were met (e.g., by gender, age, ethnic group) Identify and analyse unintended consequences and effects of activities Assess sustainability of activities and their results Identify lessons learned that may be useful elsewhere and/or by others Where possible final evaluations should be participatory in nature and include stakeholder (target population) analysis of programmes.
Methods for effectiveness evaluation include: Reviews of service records and regular reporting systems Key stakeholder interviews Direct observations Population-based surveys, longitudinal studies, review of secondary data, etc.
10.3 Developing a learning agenda and evaluation schedule

Irrespective of the type of evaluation format it has planned to use, an organisation initially needs to identify the questions it wants to ask and when it wants to collect the data in other words, develop a learning agenda. Developing a learning agenda includes: 1. Determining what components of your programme you want to learn about (identifying what needs to be evaluated). In deciding what needs to be reviewed check out the implementation plan, deliverables and results framework and identify key components, sub-components, or aspects of the programme which will be analysed to assess the organisations effectiveness in achieving their goals. 2. Clarifying what exactly you want to learn about each component identified. To help determine what question to ask review the planned deliverables, results frameworks and indicators. Some questions that might be asked include: What do you want to know about this issue? What was changed as a result of the programme? How do targeted stakeholders perceive the programmes? What sort of reach does the organisation have? How was the target population affected? How much money was spent? 3. Identifying how the data will be obtained? What data is available to help analyse this issue and what data is needed to be able to answer the questions? For example, what is needed to get the needed data a focus group discussion, a research consultant, a staff meeting, or use data from specific indicators? 4. Identifying who should be involved in answering the questions and in participating in analysis of the answers. 5. Determining deadline dates for obtaining the data and conducting the analysis. Is the information needed every month, each quarter, at the end of the project?
33
6. Identifying how to document what has been learnt, disseminate findings, and adapt programme activities and/or update underlying premises or results frameworks, thus altering the programme design. 7. Ensuring evaluation and learning processes are participatory. Stakeholder involvement in learning efforts usually promotes a sense of partnership among all the key people and/or groups interested in the organisation and enables a deepened evaluation of intended and unintended impacts.
10.4. Attributing impact to programme effort

Attributing impact to particular interventions presents many challenges particularly in the external environment in which CINDI Member organisations work as often there are a range of service providers. Under these circumstances it almost impossible to scientifically attribute change to programmes at outcome and impact level. So what are the issues that are important when evaluating outcome and impact? In evaluating impact it is more important to evaluate programmes in terms of improvement in the knowledge, capacity, reach, access, and service quality of the programmes. Very importantly an organisation needs to analyse the underlying assumptions it made when selecting the activities and assess whether they are still valid based on the implementation and evaluation efforts. Capturing best practices at this level remains a critical issue for improving responses to children living in a world of AIDS internationally.
34
11. Developing an M&E plan for the CINDI Network

11.1. Developing a M&E plan
The template for the M&E framework follows and was drawn up collectively often with a great deal of discussion. This now needs to be operationalised in collaboration with the various organisations. For the programmes funded by Irish Aid the next steps would be to: Align the proposed framework with projects already in place Formulate a results framework as an M&E matrix for the project Revisit the suitability of the indicators and collaboratively work towards protocols for the various indicators Make decisions on reporting formats and requirements Make decisions on what data is needed and how it will be collected and stored Work towards integrating the M&E framework into strategic planning initiatives.
11.2. Understanding M&E needs of the various organisations

It became evident during the learning sessions that different organisations have different needs in accordance with their divergent roles. Some organisations are providing institutional care for children or training for youth and therefore the wide range of indicators is not really applicable. Organisations are thus likely to select indicators that are most relevant to the work they do and probably will include additional indicators relevant to their scope of work.
11.3. Selecting monitoring tools and indicators

It will be important to try and use the same range of monitoring tools and also to be innovative and use some new tools. In view of the need to strengthen structures at community level there should be a commitment to using tools that are participatory and that engage beneficiaries.
11.4. Baseline studies and creation of a data base

There is a clear need for base line studies to be undertaken in a number of areas that member organisations attempt to address in their work. Some of this data is already available in other data bases. It will be helpful if there is common agreement on the information needed. Furthermore, the setting up of a data base within the CINDI Network office will be critical to the successful operationalisation of the M&E framework.
11.5. The M&E framework as a pilot programme

At this point the M&E should be considered a pilot programme so that difficulties can be quickly identified and addressed. There must be a willingness to reflect on what is happening and make adjustments where necessary.
35
12. Acknowledgements and References

Acknowledgements
1. The support of the organisations that participated in the learning sessions and the drafting of the M&E framework is much appreciated. Their inputs helped shaped the M&E framework and ensured it is rooted in the realities in which organisations work. The organisation and main participating member(s) are listed below: CINDI Network Office Gods Golden Acre Kenosis LifeLine Msunduzi Hospice PMB Child Welfare Society Rob Smetherham BSC Project Gateway SINANI Umvoti AIDS The Valley Trust Youth for Christ KZN Sixolile Ngcobo and Stellar Zulu Rosetta Heunis Elke Carrihill Debbie Harrison Elmarie Wichmann, Zo Finca and Judy Coghlan Julie Todd Lulu Hlophe Suzanne Clulow Dominique Mamet Philani Madi Fisani Dladla Kanthi Raidoo
2. Permission from International Fund for Agricultural Development (IFAD) to make use of material from their very comprehensive manual Managing for Impact in Rural Development is acknowledged with thanks. 3. The funding allocation from Rockefeller Brothers Fund made this initiative possible and is much appreciated.
References
1 2 3 4 5 6 7 Baker JL. 2000. Evaluating the Impact of Development Projects on Poverty. World Bank. Washington DC/ Connexions. October 2001. A Little Book of Evaluation Core Initiative. November 2004. Participatory Monitoring and Evaluation of Community and Faith based Programmes Coupal, Franoise. July 2001. Results-based Participatory Monitoring & Evaluation Davies Rick, 2001. Monitoring and Evaluating NGO Achievements. Cambridge, UK. Department of Health. September 2004. Monitoring and Evaluation Framework. Family Health International. September 2004. Monitoring HIV/AIDS Programmes. Participant Guide.
36
8 9
Global Fund. June 2004. Monitoring and Evaluation Toolkit. HIV/AIDS Tuberculosis and Malaria Gosling L. Edwards, M. 1999. Development manual 5. Practical Guide to Assessment, Monitoring and Evaluation. Save the Children.
10 IFAD. 2002. IFAD Managing for Impact in Rural development. Rome. Italy 11 IPPF. December 2002. Guide for designing Results-Oriented Projects and writing Successful Proposals. 12 Munt. R. Building Collaboration. Australian Institute of Family Studies. 13 NCVO. May 2003. Measuring Impact. Case studies of impact assessment in small and medium sized voluntary organisations. The UK Voluntary Sector Research Group 14 NCVO. May 2003. Measuring Impact. A Guide to resources. The UK Voluntary Sector Research Group 15 Open Society Institute. October 2001. MAPA Project. A practical guide to Integrated project Planning and Evaluation 16 PACT. 2005. Building Monitoring, Evaluation, and reporting Systems for HIV/ AIDS Programmes. 17 Simister N. January 2000. Laying the foundations: The role of data collection in the Monitoring Systems of development NGOs. Occasional Paper 01/00 Centre for Development Studies. University of Bath. 18 Simister N. March 2005. RAISA 11 Monitoring and Evaluation System. INTRAC 19 Taylor J & Soal S. 2003. Measurement in Developmental Practice. From the Mundane to the Transformational. CDRA. Cape Town. 20 UNDP 2002. Handbook on Monitoring and Evaluating for results. Evaluation . Office 21 UNICEF (and others). The framework for the protection, care and support of orphans and vulnerable children in a world with HIV and AIDS. July 2004 22 WHO. 2004. A Guide To Monitoring And Evaluating HIV/Aids Care And Support. National Programmes 23 World Bank. 2004. Monitoring and Evaluation: Some Tools, Methods, & Approaches.
37
38
Annex 1
Annex 1
GLOSSARY8
Adaptive management: An approach to monitoring, evaluation, and management decision-making that involves a cycle of planning, implementation, monitoring, research, and subsequent re-examination of management decisions based on underlying premises and in conjunction with new information. In its simplest form, adaptive management is action in response to learning. Baseline: A record of what exists in an area prior to an action. The baseline values establish the starting point from which change can be measured. Critical assumptions: Conditions outside the control of an organisation that are likely to affect results and that are assumed will or will not take place. Some examples might be the continued existence of a positive policy environment, a relatively free press, absence of severe drought, or a peaceful election process. Data analysis: Concise description of how performance data for individual indicators or groups of related indicators will be analysed to determine progress on results. Data analysis techniques and data presentation formats are identified. Data collection method: The approach to data collection, both primary and secondary, taken by the organisation for each indicator. Primary data refers to data collected specifically within the context of the programme. Secondary data refers to data collected by another source for some other purpose. Data source: The source is the entity from which the data is originally obtained. Data sources may include field staff or clinics, partner organisations, government departments, international organisations, donors, NGOs, private firms, contractors, or activity implementing agencies. Development objective: The overall and long-term effect of an intervention. It is the highest level of impact anticipated. Disaggregated: How data are to be separated to improve the breadth of understanding of results reported. Typical ways to disaggregate data include geographic location, gender, and age. Effectiveness: Measures the degree to which results/objectives have been achieved. An effective organisation is one that achieves its results and objectives. Effectiveness or impact evaluation: Analysis focused on questions pertaining to what results have been achieved, what short-term and intermediate effects were observed as a result of programme effort, and what the outcomes mean. Does the programme make a difference towards the larger development impact sought? Normally, this takes place toward the end of a programme intervention and focuses on assessing the overall outcomes and impacts attained. Efficiency: Measures how productively inputs (money, time, equipment, personnel, etc.) were used in the creation of outputs (products, outcomes, results). An efficient organisation is one that achieves its objectives with the most resourceful expenditures of resources.
8 Adapted from PACT. 2005. Building Monitoring, Evaluation, and Reporting Systems for HIV/AIDS Programmes.
39
Annex 1
Evaluation: A systematic process of collecting and analysing information to assess the effectiveness of the organisation in the achievement of goals. Evaluation provides regular feedback that helps an organisation analyse the consequences, outcomes, and results of its actions. Evaluation also provides regular feedback that helps organisations assess their relevance, scope, and sustainability. In its simplest terms, evaluation is the collection and analysis of information to assess the impact of the organisations work. Formative evaluation: Analysis focused on helping organisations identify and understand the operational setting of a potential programme and determine if an intervention is required (and where), what exactly is required, who should be involved in the intervention, and how the intervention should be carried out. Normally, this takes place at the beginning of (or prior to) a programme in the concept and design phase. Formative evaluation provides the information needed to define realistic goals, objectives, and strategies for a programme. Frequency of data collection: How often data is to be collected. The frequency of monitoring will depend on the variables being investigated. Depending on the performance indicator, it may make sense to collect data on a quarterly, annual, or less frequent basis. When planning the frequency and scheduling of data collection, it is important to consider managements needs for timely information for decision-making. Impact-level results: The overall and long-term effects of an intervention. Impacts are the ultimate result attributable to a development intervention over an extended period, such as improvement in food security or higher standards of living. Impacts usually reflect a result achieved over a longer time period (510+ years). Indexed measure: Derived by measuring a compilation of discreet variables across a scale (e.g. key elements or a series of steps are identified, points assigned as to the level of completion of each, and then the total added to determine the level of measurement). Indicator: What will be measured to know if conditions have or have not changed? A performance indicator is a quantitative or qualitative dimension or scale to measure programme results against a strategic objective or a programme outcome. A performance indicator should be a precise, direct measure of the relevant objective. It should be practical; that is, data is available or can be generated and disaggregated where possible and appropriate. Input- and process-level results: The resources and methods employed to conduct an activity, project, and/or programme. Inputs can be physical (e.g., equipment rental or purchase), material (e.g., supplies and provisions), human (labour costs such as salaries for technical assistance, staff, etc.) or financial (such as travel costs, per diem costs, direct and indirect costs). Processes are the methods or course of action selected to conduct work (e.g., training, capacity building, service provision, message promotion). Learning agenda: The programme and processes an organisation identifies to ensure it answers key questions it wants to pose or evaluate. Normally, organisations determine what components of their programmes they want to evaluate or learn about; clarify what exactly they want to learn (pose questions they want to answer or evaluate); identify how they will obtain the data; identify who will be involved in answering the questions and in participating in analysis of answers and when; and identify how they will document what was learned, disseminate findings, and adapt programme activities and or update underlying premises or results frameworks.
40
Annex 1
Monitoring: A systematic process of collecting and analysing information to track the efficiency of the organisation in achievement of goals. Monitoring provides regular feedback that helps an organisation track costs, personnel, implementation time, organisational development, and economic and financial results to compare what was planned to actual events. In its simplest terms, monitoring is collection and analysis of information to track what is going on. Monitoring, evaluation, and reporting plan: A comprehensive performance-monitoring plan is designed to track programme/project efficiency and effectiveness/impacts in all the programme/project phases. The variables to be tracked are carefully selected and they must be good measures of the anticipated changes. The monitoring plan describes all the indicators to be monitored, the units of measurement, data sources, methodology of data collection, monitoring frequency, responsibility, baseline values, and targets set within the planning horizon. Outcome-level results: Broad changes in development conditions. Outcomes help answer the so what? question: we trained 100 people and increased their knowledge but did they or did they not change their behavior? Outcomes often reflect behavior or economic changes and help analyse how activities and projects scale up or contribute toward development outcomes. Outcomes usually reflect a result achieved in an intermediate time period (25 years). Output-level results: Information, products, or results produced by undertaking activities or projects. Outputs relate to completion of activities and are the type of results over which managers have a high degree of influence. Outputs reflect what the organisation hoped to produce from a particular input (or set of inputs). For example: if the process is to train people, people trained is the result at the input/process level while knowledge level increased is the result at an output level. The assumption being that if people are trained, they will increase their knowledge on a given subject. Outputs usually reflect a result achieved in a relatively short-time period (02 years). Process evaluations: Analysis focused on providing information relating to what extent planned services are being realized, how well services are being provided, in what timeframe, at what cost, and with what result. Process evaluations analyse how efficiently inputs (money, time, equipment, personnel, etc.) are being used in creation of outputs (products, results, etc.). Process evaluations help organisations analyse what they planned to do versus what they are actually achieving and are used to make adjustments or refinements in tactics or implementation strategies. Protocols: Instructions that capture the reason for selecting indicators, describe the indicator in precise terms, and identify the plans for data collection, analysis reporting, and review. This information is documented not only to clarify and articulate what the indicators mean but also to provide the organisation with the means to collect data over the life of the project and beyond, as required. Protocols also help ensure the reliability of indicators as they provide critical information to help different people repeatedly measure the indicator with the same precision. Proxy indicators or proxy measures: Alternate or indirect measures used to stand in for another indicator when obtaining direct information is too difficult, time-consuming, or sensitive. For example, the number of people trained could be seen as an indirect measure for level of knowledge, if we assume that when people are trained, their level of knowledge always increases. Another example is the household consumption of maize, which could be a proxy indicator for household income if it is known that maize consumption always rises with income gain.
41
Annex 1
Reporting: Systematic and timely provision of essential (useful) information at periodic intervals. Reporting provides regular feedback that helps organisations inform themselves and others (stakeholders, partners, donors, etc.) on the progress, problems, successes, and lessons of programme implementation. Result: A consequence of a particular activity, project, or programme that an organisation can effect and for which it is willing to be held accountable. A change in condition attributable in whole or part to an organisation. Results-based monitoring: Sometimes referred to as performance monitoring or outcome monitoring. In addition to tracking general project implementation information (such as how much money was spent on an activity), a monitoring, evaluation and reporting system also measures how an organisations processes, products, and services contribute to broader development objectives. Results framework or results chain: Tabular or log frame presentation of the short-, intermediate-, and long-term results anticipated (inputs, outputs, outcomes, and impactlevel results). The results chain/framework describes the development hypothesis of an organisation, illustrating the cause-and-effect linkage that it believes to exist. The framework is an if then approach to planning and implementation. If we improve the message promotion then people will improve their knowledge about HIV and then people will change their behaviors and then HIV will be reduced. Results in the framework are based on critical assumptions. Secondary data: Data not collected by an organisation but used by that organisation to monitor results. Target: Magnitude or level of outputs expected to be achieved. Targets are values against which the actual programme/project achievements are measured. They should be realistic and quantitative statements of expected outcomes. If the targets are qualitative, there is need for a detailed statement of the expected state of affairs at the end of a planning period. Triangulation: Use of variety of sources, methods or team members to cross check and validate data and information to limit biases Unit of measurement: The precise parameter used to describe the magnitude or size of the indicator.
42
Annex 2
Annex 2
Ten Criteria for Assessing Indicators9

1. Practical: Data can be collected on a timely basis and at reasonable cost. A general rule is to plan on allocating 3 to 10 percent of total programme resources for monitoring and evaluation. 2. Reliable: Data can be measured repeatedly with precision by different people. 3. Relevant attributable at least in part to your organisation: A result is caused to some extent by grant-sponsored activities. Attribution exists when the links between the outputs produced by grant-financed activities and the results being measured are clear and significant. 4. Useful to management: Information provided by the measure is critical to decision-making. Avoid collecting and reporting information that is not used to support programme management decisions. 5. Direct: The indicator closely tracks the result it is intended to measure. An indicator should measure as closely as possible the result it is intended to measure. 6. Sensitive: Indicators serve as an early warning of changing conditions. A sensitive indicator will change proportionately and in the same direction as changes in the condition or item being measured, thus sensitive proxy indicators can be used as an indication (or warning) of results to come 7. Responsive: Indicators should reflect change as a result of project activities and thus indicators reflect results that are responsive to management control. 8. Objective: An objective indicator should be operationally precise with no ambiguity about what is being measured. Thus there should be general agreement on the interpretation of the results. 9. Capable of being disaggregated: Disaggregating data by gender, age, location, or some other category is often important from a management or reporting point of view. Experience shows that development activities often require different approaches for different groups and affect those groups in different ways. 10. Measurable: Ideally it should be quantifiable and measured by some scale. However, even when effective quantitative indicators are being used, qualitative indicators can supplement them to provide a richness of information that brings a programmes results to life.
9 From: PACT. 2005. Building Monitoring, Evaluation, and reporting Systems for HIV/AIDS Programmes.
43
Annex 3
Annex 3
M&E Framework for the CINDI Network

The draft framework attempts to capture a number of issues that were raised by members during the course of the learning sessions as well in the external evaluation completed earlier this year. From the discussions it seemed that the indicators needed to address two main objectives for organisations within the CINDI Network. The content and issues around these two objectives are outlined below: Objective 1: Enhancing the effectiveness of the CINDI Network Members to improve the quality of life of children in the context of HIV/AIDS through collaboration, transformation, and alignment with key provincial, national and global initiatives The issues here relate to the strengthening of the CINDI Network Members and include the following: Transformation and Empowerment issues Increased funding to support community based services Increased representivity of communities on governance structures Increased capacity to support empowerment of CBOs and local structures Increased language accessibility of documents Increased youth participation Alignment issues Improved alignment and integration with Millennium Development Goals Improved alignment and integration with National HIV/AIDS Plan Objective 2: Supporting community based interventions that enhance the quality of life of children in the context of HIV/AIDS The issues here relate to the programmes of the organisations. Some of the indicators are cross cutting for all clusters while others relate to specific clusters although it is understood that there may be overlaps. Cross cutting issues Strengthening of CBOs Improvement of quality of care Increased knowledge of prevention and transmission of HIV/AIDS Promotion of awareness and appropriate response to child abuse Cluster based issues organised in terms of the various clusters: Community Development Psychosocial Children in care Home based care Youth and Schools
44
Objective 1: Enhancing the effectiveness of the CINDI Network Members to improve the quality of life of children in the context of HIV/AIDS through collaboration, transformation, and alignment with key provincial, national and global initiatives Output Indicators Number of organisations supported with funding Programme information: Follow up on funding allocations Improved capacitation of community structures as a result of increased funding Baseline information: Current funding allocations Outcomes Outcome Indicators Information and Organisation Tools Tools: Surveys, questionnaires and maps How often: Updated quarterly Tools: Surveys, cost benefit, questionnaires, stakeholder analysis, focus groups How often: Updated quarterly
Inputs/Activities
Outputs
Inputs
Funding for programmes, materials and facilities Number of community representatives on governing structures Improved representivity of communities served in governance structures of organisations Proportion of CINDI Members with increased representivity of communities served in governance structures of organisations
Increased funding to support community based structures
Proportion of NGOs, CBOs and community structures having improved capacity as a result of increased funding for approved projects
Activities

Baseline information: Baseline reports on current training and mentoring arrangements with regard to governance structures, linkages with community groups, linkages with major provincial, national and global initiatives Programme information: Proportion of CINDI members organisations with significant number of representatives on governance structures Number of community empowerment programmes being implemented Improved capacity to support and empower community based structures Proportion of community structures able to plan implement, and monitor community based programmes Baseline information: Capacity profiling community based structures Programme Information: Proportion of CBOs able to meet capacitation criteria. Tools: Reports, the wheel, focus groups, questionnaires, time lines How often: Information collected quarterly Proportion of information sheets and learning materials available in English and isiZulu Baseline information : Information on current levels of accessibility of available information Programme information10 Tools: Reports and focus groups How often: Information collected quarterly. Data base updated quarterly Number of key pamphlets and learning materials available in English and isiZulu and utilised Improved accessibility to key information in written form
Training, mentoring programmes for key stakeholders
Increased representivity of communities served in governance structures of organisations
Increased capacity to support and empower community based structures
Written information in accessible form
10 There is a concern about the cost of producing documents in isiZulu. Appropriate translation of documents cannot be done by anybody it is specialized and time consuming
Annex 3
45
Annex 3
46
Output Indicators Number of formalised linkages with community groups Programme information: Proportion of community structures linked Improved formalised linkages with community groups Proportion of community based stakeholders with formalised links to the CINDI Network and other key stakeholders Baseline information: Current number of linked organisations Outcomes Outcome Indicators Information and Organisation Tools Tools: Reports, focus groups and questionnaires How often: Information collected quarterly. Data base updated quarterly Tools: Reports and questionnaires How often: Built into strategic plans annually. Data base updated quarterly Programme information: proportion of programmes linked with MDG in programme objectives Baseline information: Number of programmes reflecting rights based approach Programme information: Proportion of programmes reflecting rights based approach Tools: Reports and questionnaires, wheel How often: Built into strategic plans annually. Data base updated quarterly Baseline information: Number of Organisations reflecting alignment with national HIV/ AIDS plan Programme information: Proportion of programmes reflecting national HIV/AIDS plan Tools: Reports and questionnaires How often: Built into strategic plans annually Number of MDGs integrated into programme plans Improved alignment with and integration of MDGs in programme objectives Proportion of programmes reflecting Millennium Development Goals Baseline information: Number of programmes aligned with MDG goals Number of programmes reflecting a rights based approach Improved alignment with and integration of rights based approach in programme objectives Proportion of programmes reflecting a rights based approach Number of programmes reflecting National HIV and AIDS Programme goals Improved alignment with National HIV and AIDS Programme Proportion of programmes reflecting alignment with National HIV and AIDS Programme
Inputs/Activities
Outputs
Activities
Training, mentoring programmes for key stakeholders
Increased number of formalised linkages with community groups
Increased alignment with and integration of MDGs in programme objectives
Increased alignment with and integration of rights based approach in programme objectives
Increased alignment with and integration of National HIV and AIDS Programme goals (See www.doh.gov.za)
Objective 2: Supporting community based interventions that enhance the quality of life of children in the context of HIV/AIDS Cross cutting all clusters Output indicators Number of organisations supported with funding Baseline information: on funding allocations Programme information: proportion of targeted CBOs being supported Baseline information: Baseline reports on current training and mentoring arrangements with regard to supporting community based initiatives that enhance the quality of life of children Programme information: Proportion of CBOs and community structures able to manage and support programmes for targeted childrens Proportion of CBOs and community structures being supported with funding for approved projects Baseline information: Prevailing standards to care e.g. competencies of child care workers, home based care workers etc. as current resource support. Programme information: Standards of care following exposure to training, mentoring and improved resource support Number of programmes reflecting parameters for increased quality of care Improved quality of care in targeted programmes Proportion of targeted programmes reflecting increased quality of care Tools: Surveys, cost benefit, questionnaires, stakeholder analysis, focus groups How often: Updated quarterly Improved support for community based structures Proportion of CBOs and community structures being supported with funding for approved projects Proportion of CBOs and community structures able to manage and support programmes for targeted children. Outcomes Outcome indicators Information Tools Tools: Surveys, questionnaires and maps How often: Updated quarterly Tools: Surveys, cost benefit, questionnaires, stakeholder analysis, focus groups How often: Updated quarterly
Inputs/Activities
Outputs
Inputs:
Funding for programmes, training, materials facilities and equipment etc. Number of organisations having effective leadership structures and programmes of action in place. Strengthened networks of community representative stakeholders able to holistically support targeted children at local level
Increased funding to support community based structures.
Activities:
Handbook on monitoring and evaluation for the CINDI Network Number of organisations supported with funding Improved support community based structures
Training, production of training materials, mentoring to strengthen networks of support
CBOs and community structures with increased competencies in basic organisational skills
Inputs:
Funding for programmes, training, materials facilities and equipment etc.
Increased funding to support community based structures to improve quality of care
Activities: Training, production of training materials to increase quality of care
Activities including training, mentoring and support to improve quality of care
Annex 3
47
Annex 3
48
Output indicators Decrease in level of HIV/ AIDS prevalence in communities served by CINDI Network Programme information: Estimated levels of HIV prevalence in targeted communities. Decrease in level of HIV/AIDS prevalence in communities served by CINDI Network Baseline information: Current estimated levels of HIV prevalence in targeted communities. Outcomes Outcome indicators Information Tools Tools: Surveys, questionnaires, focus groups, mapping, time line How often: Updated quarterly Increased number of targeted children and child care providers having increased knowledge regarding prevention and transmission of HIV and AIDS Increased awareness of and appropriate responses to child abuse in targeted areas Improved levels of awareness of and appropriate responses to child abuse in targeted communities Proportion of targeted areas reflecting decrease in levels of child abuse. Proportion of child abuse cases appropriately responded to Baseline information: Current levels of reported child abuse. Programme information: Follow up on levels of reported child abuse Tools: Surveys, questionnaires, focus groups, mapping, time line How often: Updated quarterly
Inputs/Activities
Outputs
Inputs: Funding for programmes, training, materials facilities and equipment etc
Activities: Training, production of training materials to increase knowledge and understanding of HIV/AIDS prevention, transmission and care
Number of targeted children having increased knowledge regarding prevention and transmission of HIV and AIDS
Activities: Training, production of training materials to increase awareness of child abuse
Number of sessions to promote awareness of and appropriate response to child abuse prevention and treatment (aligned with national and international legislation)
Psychosocial cluster Primary Goal: To build individual psychological and social resources to increase and strengthen their capacity to cope Output indicators Number of organisations supported with funding Programme information: Guidance from relevant organisations Improved emotional well being of targeted children Proportion of children with improved emotional well being (through accessing psychosocial care) including: 1. decrease in depression 2. decrease in psychosomatic and anxiety related symptoms, 3. increase in perception of social support Proportion of targeted children having access to a caring and consistent significant other (adult) Number of targeted households ensuring children attend school Improved educational levels of targeted children Proportion of targeted children who have completed their school year Proportion of children whose grades are appropriate for their age Proportion of children whose absenteeism from school has not exceeded 15 school days Baseline information: Number of children from targeted homes that complete the school year Number of children whose grades are appropriate for age Programme information: Follow up information on number of children from targeted homes that complete the school year and grades are appropriate for age Tools: Reports, surveys and focus groups How often: Updated quarterly Baseline information: Guidance from relevant organisations Outcomes Outcome indicators Information Tools Tools: Reports, surveys, questionnaires, focus groups, How often: Updated quarterly
Inputs/Activities
Outputs
Inputs: Funding for programmes, training, materials facilities and equipment etc Increased number of community members able to provide psychosocial support at local community level Increased availability of user friendly training materials
Increase in number of programmes financed
Trained community members able to provide psychosocial support at local community level
Activities: Training, production of training materials, Facilitation of support for programmes which build psycho social support
Development of user friendly training materials
Increased access to education facilities
Annex 3
49
Annex 3
50
Output indicators Outcomes Outcome indicators Information Tools Tools: Reports and surveys How often: Updated quarterly Number of targeted families making decisions about succession planning for future care of children and assets Improved support and utilisation of succession planning for future care of children and assets Proportion of targeted families making provision for succession planning for care of children and assets. Baseline information: Number of targeted families in which decisions have been made about succession planning for children and care of assets. Programme information: Follow up information on number of targeted families in which decisions have been made about succession planning and care of assets. Proportion of targeted child carers who have been exposed to child care enhancement training. Baseline information: Assessment of current child care capacity of targeted care giver carers. Programme information: Follow up information on assessment of child care capacity of targeted care giver carers Tools: Reports and surveys and focus groups How often: Updated quarterly Number of targeted child carers exposed to child care enhancement training Improved support for child care capacities
Inputs/Activities
Outputs
Increased support and utilisation of succession planning for future care of children and assets
Increased support for enhancement of care givers child care capacities
Community Development Cluster Primary goal: To achieve long term sustainable development whereby the community is able to care for their own needs and the needs of children Output indicators Number of organisations supported with funding to improve economic standing of targeted households Programme information: Follow up information on basic income and economic activities of households in targeted areas Improved economic standing of targeted children and/or their care providers Baseline information: Basic income and economic activities of households in targeted areas Outcomes Outcome indicators Information Tools Tools: Reports, surveys, questionnaires, stakeholder analysis, focus groups, mapping How often: Updated quarterly
Inputs/Activities Proportion of targeted children who have improved their economic standing (through a government grant, successful income generation programme, housing, scholarship)
Outputs
Increased funding to support community based structures
Handbook on monitoring and evaluation for the CINDI Network Improved health of targeted children Proportion of targeted children whose clinic charts reflect: height and weight appropriate for age have improved their nutritional well being are up to date with immunisations Number of households reflecting increased knowledge of health issues and increased access to health care facilities Baseline information: Baseline reports on relevant health data of households in targeted areas Programme information: Follow up information on relevant health data of households in targeted areas. Tools: Reports, surveys, questionnaires, stakeholder analysis, focus groups, mapping How often: Updated quarterly
Activities: Training, production of training materials, Facilitation of support for programmes which address basic needs and services for children
Increased knowledge of health issues and access to health care facilities
Annex 3
51
Annex 3
52
Output indicators Number of targeted households ensuring children attend school Proportion of children whose grades are appropriate for their age Proportion of children whose absenteeism from school has not exceeded 15 school days Proportion of targeted households were care givers make progress in terms of functional literacy levels Baseline information: Number of care givers from targeted homes that advance their levels of functional literacy in the prescribed time Programme information: Follow up information on number of care givers from targeted homes that advance their levels of functional literacy in the prescribed time Proportion of targeted families in which all births have been registered. Proportion of affected families accessing death certificates. Baseline information: Number of children in targeted households that have been registered. Number of caregivers that have access to key death certificates Programme information: Follow up information on Number of children in targeted households that have been registered. Number of caregivers that have access to key death certificates Tools: Reports and surveys How often: Updated quarterly Programme information: Follow up information on number of children from targeted homes that complete the school year Tools: Reports, surveys and focus groups How often: Updated quarterly Number of children whose grades are appropriate for age How often: Updated quarterly Improved educational levels of targeted children Proportion of targeted children who have completed their school year Baseline information: Number of children from targeted homes that complete the school year Tools: Reports, surveys and focus groups Outcomes Outcome indicators Information Tools Number of targeted households where adult care givers have access to functional literacy training Improved levels of functional literacy amongst care givers in the targeted households Number of targeted families acquiring required documentation including Master Birth Certificate Improved registration of births and deaths
Inputs/Activities
Outputs
Increased access to education facilities
Increased levels of functional literacy
Increased awareness and support to targeted families on processes involved in acquiring essential documentation
Increased support for improved physical environment of targeted households including shelter water and sanitation Programme information: Follow up on number of targeted families that have adequate shelter, water and sanitation Baseline in formation: Number of targeted households who do not adequate access to food. Programme information: Updated information on number of households that do not have adequate access to food. Number of targeted households with protection plan in place Number of households aware of childrens rights and local resources Proportion of targeted households aware of childrens rights and accessing available support when required Improved protection of targeted children (reduced exploitation) Proportion of targeted children and households with protection orientated intervention in place Baseline information: Information on number of children being served Programme information: Proportion of children being served How often: Updated quarterly Number of targeted households that have improved food security Improved food security within targeted families Proportion of targeted families who have adequate food
Number of targeted households that are able to improve their physical environment in terms of shelter, sanitation and water
Improved physical environment for targeted children (shelter, water and sanitation)
Proportion of targeted children having adequate shelter and access to potable water and sanitation
Baseline information: Number of targeted families that do not have adequate shelter, water or sanitation
Tools: Reports, surveys, questionnaires, focus groups, mapping, wheel
Increased food security for targeted households
Tools: Reports, surveys, questionnaires, focus groups, mapping, wheel How often: Updated quarterly Tools: Reports, surveys, questionnaires, focus groups How often: Updated quarterly
Increased protection of targeted children. Increased awareness of childrens rights and available resources.
Annex 3
53
Annex 3
54
Output indicators Number of home based care workers trained in compliance with national requirements. (52 day training) Programme information: Proportion of targeted families being reached Improved home based services to clients in need of care Proportion of targeted households in which home based care services are provided on a regular basis Baseline information: Information on number of families needing home based care services Outcomes Outcome indicators Information Tools Tools: Reports, surveys, questionnaires, focus groups, mapping How often: Updated quarterly Number of home based care workers integrated into continuum of care Improved linkages with health care networks Proportion of home based care workers that are formally part of a continuum of care Baseline information: Information on number of home based care workers linked to continuum of care Programme information: Proportion of home based care workers linked to continuum of care Proportion of home based care workers provided with adequate resources to care for families Baseline information: Information on current basic resources provided to home based care workers Programme information: Information of resources provided to home based care workers Tools: Reports, surveys, questionnaires, focus groups, mapping How often: Updated quarterly Tools: Reports, surveys, questionnaires, focus groups, mapping How often: Updated quarterly Number of home based care workers provided with adequate material resources to provide essential care Improved quality of care for families
Home Based Care Cluster: Primary Goal: To build individual psychological and social resources and offer basic health care to increase and strengthen the capacity of those living with life threatening and chronic illness to cope.
Inputs/Activities
Outputs
Inputs: Funding for programmes, training, materials, facilities and equipment etc.
Activities: Training, production of training materials,
Increased provision of home based community care and development including training, provision of resources, integration with continuum of care, regular monitoring and supervision
Facilitation of support for community based programmes which address basic needs and services for clients in need of home based care.
Number of home based care workers receiving regular monitoring and supervision Programme information: Information on supervision and support plans for home based care workers Baseline information: Information on current treatment support within targeted households. Programme information: Information on treatment support within targeted households as programme activity How often: Updated quarterly
Improved supervision and support for home based care workers
Proportion of home based care workers receiving regular supervision and support
Baseline information: Information on current basic on supervision and support plan for home based care workers
Tools: Reports, questionnaires
Increased treatment support in relation to TB treatment and ARVs for adults and children
Tools: Reports, questionnaires How often: Updated quarterly
Number of targeted households in which adults and children receive regular treatment support with ARVs or TB medication
Improved treatment support in targeted households in which adults and children receive regular treatment support with ARVs or TB medication Improved care for children in targeted homes Proportion of targeted homes with succession and positive care programme for children
Proportion of targeted households in which patients receive regular treatment support with ARVs or TB medication
Handbook on monitoring and evaluation for the CINDI Network Baseline information: Information on succession planning and positive care programme in targeted households Programme information: Information on succession planning and positive care programme in targeted households Tools: Reports, surveys, questionnaires, focus groups, mapping How often: Updated quarterly Number of children in targeted families in which succession planning, and positive care programme are being implemented
Increased support for children in families in which client lives
Annex 3
55
Annex 3
56
Output indicators Improved knowledge and compliance with regulations of care of children in facilities Programme information: Follow up information on number of targeted facilities and projects reflecting knowledge and compliance with regulations of Child Care Act Baseline information: Number of voluntary workers screened by means of references Programme information: Follow up information on number of workers screened by means of references Proportion of targeted children who have completed their school year Proportion of children whose grades are appropriate for their age Proportion of children whose absenteeism from school has not exceeded 15 school days Baseline information: Number of children from targeted homes that complete the school year Number of children whose grades are appropriate for age Programme information: Follow up information on number of children from targeted homes that complete the school year and whose age for grade is appropriate Tools: Reports, questionnaires and focus groups How often: Updated quarterly Baseline information: Number of targeted facilities and projects reflecting knowledge and compliance with regulations of Child Care Act appropriate for age Outcomes Outcome indicators Information Tools Tools: Reports, questionnaires and focus groups How often: Updated quarterly Increase in number of residential and community based care facilities for children aware of and implementing regulations of the Child Care Act Proportion of targeted child care facilities and programmes that reflect improved knowledge and compliance with Child Care Act Increase in number of voluntary workers which have been screened by means of references Number of targeted children having access to education Improved educational levels of targeted children Improved screening of voluntary workers by means of references Proportion of voluntary workers screened by means of references Tools: Reports, Surveys and focus groups How often: Updated quarterly
Children in Care Cluster: Primary Goal: To support projects providing care to children which focus on keeping children in their community of origin.
Inputs/Activities
Outputs
Inputs: Funding for programmes, training, materials facilities and equipment etc.
Increased compliance with all aspects of regulations in terms of Child Care Act
Activities: Holistic residential care of children,
Supervised community care of children. Training, production of training materials,
Increased screening of volunteer and staff by means of references
Facilitation of support for community based programmes which address basic needs and services for children
Increased number of targeted children in care having access to education
Increased number of targeted children in care having access to basic health care height and weight appropriate for age improved nutritional well being immunisations are up to date Number of targeted children with positive sense of self identity Number of targeted children able to identify adult significant other Improved care and appropriate discipline for children Improved emotional well being Proportion of targeted children able identify significant adult other Improved sense of positive self identity. Proportion of targeted children with improved sense of positive Baseline information: Nutritional status of children Programme information: Follow up information on nutritional status of children Baseline information: Number of children able to identify significant adult other Programme information: Follow up information on Baseline information: Current understandings of discipline and care Programme information: Follow up information on number of children able to identify significant adult other understandings of discipline and care Programme information: Follow up information on relevant health data of households in targeted areas. How often: Updated quarterly
Number of targeted children having access to basic health care
Improved health of targeted children
Proportion of targeted children whose clinic charts reflect:
Baseline information: Baseline reports on relevant health data of households in targeted areas
Tools: Reports, surveys, questionnaires, stakeholder analysis, focus groups, mapping
Increased number of targeted children with positive sense of self identity
Tools: Reports, Surveys and focus groups How often: Updated quarterly Tools: Reports, Surveys and focus groups How often: Updated quarterly Tools: Reports, Surveys and focus groups How often: Updated quarterly
Handbook on monitoring and evaluation for the CINDI Network Number of care givers able to describe and implement minimal legislative requirements relating to discipline Proportion of targeted care givers able to describe and implement minimal legislative requirements relating to discipline
Increased number of children able to identify significant adult other
Increased number of care workers able to describe and implement minimal legislative requirements relating to discipline
Annex 3
57
Annex 3
58
Output indicators Number of targeted youth with increased knowledge and skills relating to leadership and life skills Improved life skills and leadership capacity Programme information: Follow up information on leadership and life skills knowledge and practice of targeted youth. Proportion of targeted youth with increased knowledge and skills relating to leadership and life skills Baseline information: Current leadership and life skills knowledge and practice of targeted youth. Outcomes Outcome indicators Information Tools Tools: Reports, questionnaires and focus groups How often: Updated quarterly Number of targeted youth with increased knowledge of HIV and AIDS, reproductive health and other youth related threats Improved capacity to prevent HIV and AIDS and awareness of reproductive health and other youth related threats Proportion of targeted youth with Improved capacity to prevent HIV and AIDS and awareness of reproductive health and other youth related threats Baseline information: Current levels of knowledge of HIV and AIDS, reproductive health and other youth related threats. Programme information: Follow up information on levels of knowledge of HIV and AIDS, reproductive health and other youth related threats Proportion of targeted victims of rape and abuse that received support. Baseline information: Current levels of support to victims of rape and abuse Programme information: Follow up information on levels of support to victims of rape and abuse Tools: Reports, questionnaires and focus groups How often: Updated quarterly Number of victims of rape and abuse receiving support Improved support for victims of rape and abuse Tools: Reports, Surveys and focus groups How often: Updated quarterly
School and Youth Development Cluster Primary goal: To prevent HIV and AIDS through leadership development and youth appropriate responsibility for reproductive health
Inputs/Activities
Outputs
Activities: Facilitation and training for life skills and leadership.
Increased life skills and leadership capacity amongst targeted young people
Production of training materials,
Dissemination of information on HIV and AIDS, reproductive health, and other youth related challenges
Support to youth victims of rape and abuse
Increased knowledge of HIV and AIDS, reproductive health and other youth related threats
Increased support to survivors of rape and abuse
Annex 4
Annex 4
UNICEFs framework for the protection, care and support of orphans and vulnerable children living in a world with HIV & AIDS13
Five key strategies
1. Strengthen the capacity of families to protect and care for orphans and vulnerable children by prolonging the lives of parents and providing economic, psychosocial and other support. 2. Mobilise and support communitybased responses 3. Ensure access for orphans and targeted children to essential services, including education, health care, birth registration and others. 4. Ensure that governments protect the most targeted children through improved policy and legislation and by channeling resources to families and communities.
13 UNICEF (and others) The framework for the protection, care and support of orphans and vulnerable children in a world with HIV and AIDS. July 2004
5. Raise awareness at all levels through advocacy and social mobilization to create a supportive environment for children and families affected by HIV and AIDS. The actions described below are intended to help shape an effective response to the growing crisis. Conduct a collaborative situation analysis
Improve household economic capacity
Engage local leaders in responding to the needs of vulnerable community members Organise and support activities that enable community members to talk more openly about HIV and AIDS Organise cooperative support activities
Increase school enrolment and attendance Ensure birth registration for all children
Adopt national policies, strategies and action plans Enhance government capacity
Provide psychosocial support to affected children and their caregivers
Mobilise influential leaders to reduce stigma, silence and discrimination
Strengthen and support child-care capacities? Support succession planning
Provide basic health and nutrition services Improve access to safe water and sanitation Ensure that judicial systems protect targeted children?
Ensure that resources reach communities Develop and enforce a supportive legislative framework Establish mechanisms to ensure information exchange and collaboration of efforts
Strengthen and support social mobilization activities at the community level
Promote and support community care for children without family support
Prolong the lives of parents
Strengthen young peoples life skills
Ensure placement services for children without family care Strengthen local planning and action
59

M&amp E Handbook 2006

Caricato da

Informazioni sul documento

Descrizione originale:

Titolo originale

Copyright

Formati disponibili

Condividi questo documento

Condividi o incorpora il documento

Opzioni di condivisione

Hai trovato utile questo documento?

Questo contenuto è inappropriato?

Copyright:

Formati disponibili

M&amp E Handbook 2006

Caricato da

Copyright:

Formati disponibili

47 Temple Street, Pietermaritzburg 3201 P O Box 157, Pietermaritzburg 3200 E-mail: info@cindi.org.

Monitoring and Evaluation

How this Handbook has been put together ...

2. The CINDI monitoring and evaluation (M&E) journey

Handbook on monitoring and evaluation for the CINDI Network

2.1. User friendliness

2.2. Data and the location of the data base

2.3. Monitoring non quantitative aspects of the work

2.4. Linking monitoring and evaluation to planning

2.5. Gaps between management and fieldworkers

Handbook on monitoring and evaluation for the CINDI Network

2.6. Issues of capacity

2.7. Involvement of beneficiaries in M&E

2.8. Gender equality

3. The current buzz around M&E

Handbook on monitoring and evaluation for the CINDI Network

3.2. Defining M&E

3.3. The M&E System1

Handbook on monitoring and evaluation for the CINDI Network

3.4. Different M&E Models

3.4.2. Results Based M&E

3.4.3. Participatory M&E

3.4.3. Developmental M&E

Handbook on monitoring and evaluation for the CINDI Network

3.4.4. Integrating different approaches

4. From programme cycle to results based M&E

Review and Evaluation Planning

Handbook on monitoring and evaluation for the CINDI Network

What did the participants say during the learning session?

4.2. Results Based M&E2

4.2.1. The results based framework is based on assumptions

4.2.2. Levels of Results

Handbook on monitoring and evaluation for the CINDI Network

Handbook on monitoring and evaluation for the CINDI Network

4.3. The results chain

Short term change effects and results

Intermediate change effects and results

Long term change effects and results

What participants said during the learning session

Handbook on monitoring and evaluation for the CINDI Network

Reflections on a CINDI Big Issue!

Handbook on monitoring and evaluation for the CINDI Network

5. Mapping the journey information needs and indicators

5.2. Counting measures versus proxy indicators

5.2.2. Proxy indicators

5.2.3. Indexed measures

Handbook on monitoring and evaluation for the CINDI Network

5.3. Steps in selecting indicators

Handbook on monitoring and evaluation for the CINDI Network

What participants said and did during the learning session

Handbook on monitoring and evaluation for the CINDI Network

6. Tools for gathering information3

Purpose and possible relevance to CINDI Network

Handbook on monitoring and evaluation for the CINDI Network

Questionnaires and Surveys

Handbook on monitoring and evaluation for the CINDI Network

Focus Groups (continued)

SWOT Analysis (Strengths, Weakness, Opportunities, Threats)

Dreams realised or visioning

Drama or role plays

Handbook on monitoring and evaluation for the CINDI Network

Sketch mapping (continued)