building a framework for evaluation in community settings_280912.pdf

7/29/2019 Building a Framework for Evaluation in Community Settings_280912.pdf

1/32


2/32

PRESENTATION OVERVIEW

Introductions What is evaluation? Why conduct evaluation? Common fears & concerns Who conducts evaluations? Methods Examples Stages What happens after the evaluation?BREAK Group exercise


3/32

WHAT IS EVALUATION?


4/32

WHAT IS EVALUATION?

A systematic method of collecting, and using information toanswer questions about community projects and initiatives

Ideally, evaluation should involve procedures that are: Useful

Feasible

Ethical

Accurate


5/32

WHAT IS EVALUATION?


6/32

WHY CONDUCT EVALUATION?

Promote understanding Evaluation can help us understand how community initiatives develop, offering lessons other

groups can profit from

Improve work Evaluation feedback can improve community work by encouraging continuous adjustments

of programs, policies, and interventions

Community involvement and empowerment

Involving community members in evaluation, can lend people who haven't had a voice gainthe opportunity to better understand and improve local efforts

Accountability Evaluation can help hold groups accountable to the community and to the funders. It can

also help hold funders accountable to the communities that they serve

Evidence based practice


7/32

EVIDENCE BASED PRACTICE

The term evidence based practice has roots in clinical andmedical practice

Its basic principles are that all practical decisions made shouldbe:

Based on evidence/research

That evidence based studies are selected and interpreted

according to specific norms and criteria

Better than using intuition/common sense/personal experience? How do you do anything innovative or different?


8/32

EXAMPLE OF EVIDENCE BASED PRACTICE


9/32

FEARS & CONCERNS


10/32

COMMON FEARS & CONCERNS

Evaluation is costly Evaluation is time consuming Our work will be judged We will be compared unfavourably to other community initiatives Someone will come in from the outside who knows nothing about

the history of our organisation or what it means to our community What if the news is bad?

Findings are limited or not as pronounced as expected


11/32

WHO CONDUCTS EVALUATIONS?

Outside evaluators Academic researchers

Community researchers

Pros and cons

In-house evaluators Pros and cons


12/32

METHODS USED IN EVALUATION

Randomized control trials Surveys

Custom designed

Standardized measures

Analysis of existing procedures or data Qualitative research

Interviews

Focus groups

Community based participatory research (CBPR)


13/32

EXAMPLES OF EVALUATION

Large Scale Evaluations The National Evaluation of Systems of Care

Targeted Evaluations The Evaluation of Preparing for Life

Participatory Research PhotoVoice example


14/32

EXAMPLES OF EVALUATION:

NATIONAL EVALUATION OF SYSTEMS OF CARE


15/32


EVALUATION OF PREPARING FOR LIFE


16/32


PHOTOVOICE HOMELESS PROJECT

This PhotoVoice study gave a sample of homeless women in central Auckland acamera with which to photograph their lives in order to voice their felt healthneeds as advocates and agents for positive change Interviews explored the meanings given to street lives captured in thephotographs and reveal threats to the womens mental health and worseningaddictions The women perceived social services as helping them survive and support theirhealth, but not ending their homelessness. They identified their own leaders who could link with state housing services toimplement and evaluate new homelessness programmes

(Bukowski & Buetow, 2011)


17/32

STAGES OF EVALUATION

Identify key stakeholders and what they care about Describe the program or initiatives framework or logic model Focus the evaluation design - what the evaluation aims to

accomplish, how it will do so, and how the findings will be used Gather credible evidence- decide what evidence is, and what

features affect credibility of the evaluation Make sense of the data and interpret conclusions Use the information to celebrate, make adjustments, and

communicate lessons learned


18/32

IDENTIFY KEY STAKEHOLDERS


19/32

IDENTIFY KEY STAKEHOLDERS

Those involved in program operations Funders Collaborators

Staff & administrators

Those served or affected by the program Clients

Family members

Neighbourhood organizations

Elected officials

Sceptics & opponents


20/32

DESCRIBE THE PROJECT

Need

Nature and magnitude of the problem or opportunity Individuals served

Expected effects What the project should accomplish

Activities What the project does

Resources Time, talent, technology, equipment, information, money, etc.

Context History, geography, politics, social and economic conditions, and previous efforts

Logic Model


21/32

BEFORE YOU EVALUATE,

BE CLEAR ON WHAT YOU DO!


22/32

FOCUS THE EVALUATION DESIGN

Articulate a purpose/ intent Identify who will receive findings Plan how the information will be applied Focus on what specific questions will be answered Decide what methods will be utilized Articulate who will be responsible for what aspect of the

evaluation


23/32

GATHER CREDIBLE EVIDENCE

Sources - People, documents, or observations that provideinformation for the evaluation

Quality - Appropriateness and integrity of information gathered inthe evaluation

Quantity - Amount of information gathered in the evaluation Logistics - Methods, timing, and physical infrastructure forgathering and handling information


24/32

INTERPRET THE DATA COLLECTED

Standards - The values held by stakeholders that provide thebasis for interpreting the data

Analysis - Detects patterns in information gathered Interpretation - The effort of figuring out what the findings mean Recommendations - Actions resulting from the information

gathered


25/32

COMMUNICATE FINDINGS

Feedback - Communication among all parties to the evaluation Creates an atmosphere of trust among stakeholders Keeps an evaluation on track

Follow-up - Technical and emotional support Dissemination - Process of communicating procedures, findings

& lessons learned from an evaluation to relevant audiences


26/32

TYPES OF DISSEMINATION

Reports Presentations Participant stories Media outlets

Who is your audience? Funders

Colleagues

Community

Government Officials


27/32

SO YOUVE EVALUATED, NOW WHAT?

Organize and review evidence of progress and impact System change Community/participant improvement

Communicate and use early and on-going indicators of progressto assess and improve programme or initiative

Assess findings with partners to determine next steps forcontinuous improvement.


28/32


29/32

BREAK TIME!


30/32

LETS GIVE IT A TRY

Problem statement What issue/problem/concern does yourproject address?

Goal of project How does your project impact this issue? Outcomes of project What are the expected results of your

project? One specific question youd like answered - Put in question formone specific item you would like answered about your project. How will you answer this question ? Brainstorm possible ways to

evaluate this particular aspect of your project.


31/32


32/32

RESOURCES & CONTACT INFORMATION

The Community Toolboxhttp://ctb.ku.edu/The Center for Disease Control and Preventionhttp://www.cdc.gov/eval/framework/index.htm

Eylin Palamaro Munsell, PhDGeary Institute, University College DublinPhone: +353 1 [email protected]
http://ctb.ku.edu/http://www.cdc.gov/eval/framework/index.htmhttp://www.cdc.gov/eval/framework/index.htmhttp://www.cdc.gov/eval/framework/index.htmhttp://ctb.ku.edu/http://ctb.ku.edu/

building a framework for evaluation in community settings_280912.pdf

Documents