PRESENTATION OVERVIEW

Introductions What is evaluation? Why conduct evaluation? Common fears & concerns Who conducts evaluations? Methods Examples Stages What happens after the evaluation? BREAK Group exercise

WHAT IS EVALUATION?

WHAT IS EVALUATION?
A systematic method of collecting, and using information to answer questions about community projects and initiatives

Ideally, evaluation should involve procedures that are: Useful Feasible Ethical Accurate

WHAT IS EVALUATION?

WHY CONDUCT EVALUATION?

Promote understanding Evaluation can help us understand how community initiatives develop, offering lessons other groups can profit from
Improve work Evaluation feedback can improve community work by encouraging continuous adjustments of programs, policies, and interventions Community involvement and empowerment Involving community members in evaluation, can lend people who haven't had a voice gain the opportunity to better understand and improve local efforts

Accountability Evaluation can help hold groups accountable to the community and to the funders. It can also help hold funders accountable to the communities that they serve Evidence based practice

EVIDENCE BASED PRACTICE

The term evidence based practice has roots in clinical and medical practice

Its basic principles are that all practical decisions made should be: Based on evidence/research That evidence based studies are selected and interpreted according to specific norms and criteria
Better than using intuition/common sense/personal experience? How do you do anything innovative or different?

EXAMPLE OF EVIDENCE BASED PRACTICE

FEARS & CONCERNS

COMMON FEARS & CONCERNS

Evaluation is costly Evaluation is time consuming

Our work will be judged

We will be compared unfavourably to other community initiatives Someone will come in from the outside who knows nothing about the history of our organisation or what it means to our community What if the news is bad? Findings are limited or not as pronounced as expected

WHO CONDUCTS EVALUATIONS?

Outside evaluators Academic researchers Community researchers Pros and cons In-house evaluators Pros and cons

METHODS USED IN EVALUATION

Randomized control trials
Surveys Custom designed Standardized measures Analysis of existing procedures or data Qualitative research Interviews Focus groups Community based participatory research (CBPR)

EXAMPLES OF EVALUATION
Large Scale Evaluations The National Evaluation of Systems of Care Targeted Evaluations The Evaluation of Preparing for Life Participatory Research PhotoVoice example

EXAMPLES OF EVALUATION: NATIONAL EVALUATION OF SYSTEMS OF CARE

EXAMPLES OF EVALUATION: EVALUATION OF PREPARING FOR LIFE

EXAMPLES OF EVALUATION: PHOTOVOICE HOMELESS PROJECT

This PhotoVoice study gave a sample of homeless women in central Auckland a camera with which to photograph their lives in order to voice their felt health needs as advocates and agents for positive change Interviews explored the meanings given to street lives captured in the photographs and reveal threats to the womens mental health and worsening addictions The women perceived social services as helping them survive and support their health, but not ending their homelessness. They identified their own leaders who could link with state housing services to implement and evaluate new homelessness programmes

(Bukowski & Buetow, 2011)

STAGES OF EVALUATION
Identify key stakeholders and what they care about Describe the program or initiatives framework or logic model

Focus the evaluation design - what the evaluation aims to accomplish, how it will do so, and how the findings will be used
Gather credible evidence- decide what evidence is, and what features affect credibility of the evaluation Make sense of the data and interpret conclusions Use the information to celebrate, make adjustments, and communicate lessons learned

IDENTIFY KEY STAKEHOLDERS

IDENTIFY KEY STAKEHOLDERS

Those involved in program operations Funders Collaborators Staff & administrators Those served or affected by the program Clients Family members Neighbourhood organizations Elected officials Sceptics & opponents

DESCRIBE THE PROJECT

Need Nature and magnitude of the problem or opportunity Individuals served Expected effects What the project should accomplish Activities What the project does Resources Time, talent, technology, equipment, information, money, etc. Context History, geography, politics, social and economic conditions, and previous efforts

Logic Model

BEFORE YOU EVALUATE, BE CLEAR ON WHAT YOU DO!

FOCUS THE EVALUATION DESIGN

Articulate a purpose/ intent Identify who will receive findings

Plan how the information will be applied

Focus on what specific questions will be answered Decide what methods will be utilized

Articulate who will be responsible for what aspect of the evaluation

GATHER CREDIBLE EVIDENCE

Sources - People, documents, or observations that provide information for the evaluation

Quality - Appropriateness and integrity of information gathered in the evaluation

Quantity - Amount of information gathered in the evaluation Logistics - Methods, timing, and physical infrastructure for gathering and handling information

INTERPRET THE DATA COLLECTED

Standards - The values held by stakeholders that provide the basis for interpreting the data

Analysis - Detects patterns in information gathered

Interpretation - The effort of figuring out what the findings mean Recommendations - Actions resulting from the information gathered

COMMUNICATE FINDINGS
Feedback - Communication among all parties to the evaluation Creates an atmosphere of trust among stakeholders Keeps an evaluation on track Follow-up - Technical and emotional support Dissemination - Process of communicating procedures, findings & lessons learned from an evaluation to relevant audiences

TYPES OF DISSEMINATION
Reports Presentations Participant stories Media outlets

Who is your audience? Funders Colleagues Community Government Officials

SO YOUVE EVALUATED, NOW WHAT?

Organize and review evidence of progress and impact System change Community/participant improvement Communicate and use early and on-going indicators of progress to assess and improve programme or initiative Assess findings with partners to determine next steps for continuous improvement.

BREAK TIME!

LETS GIVE IT A TRY

Problem statement What issue/problem/concern does your project address?

Goal of project How does your project impact this issue?

Outcomes of project What are the expected results of your project? One specific question youd like answered - Put in question form one specific item you would like answered about your project. How will you answer this question ? Brainstorm possible ways to evaluate this particular aspect of your project.

RESOURCES & CONTACT INFORMATION

The Community Toolbox http://ctb.ku.edu/ The Center for Disease Control and Prevention http://www.cdc.gov/eval/framework/index.htm

Eylin Palamaro Munsell, PhD

Geary Institute, University College Dublin Phone: +353 1 7164613 eylin.palamaromunsell@ucd.ie