too expensive too complicated too time consuming

Download Too expensive Too complicated Too time consuming

Post on 12-Jan-2016

217 views

Category:

Documents

2 download

Embed Size (px)

TRANSCRIPT

&%$#@&%$!!

&%$#@&%$!!Evaluation is NOT a Dirty Word

Kathleen Dowell, Ph.D.EvalSolutions

Epilepsy Foundation: Best Practices InstituteSeptember 29, 2012Denver, Colorado1Too expensive

2Too complicated

3Too time consuming

4Not a priority

5Just dont know where to start

6BarriersLack of research/statistics skills

Lack of timeLack of resourcesOther prioritiesLack of incentiveFearDont see valueDont let barriers get in way small scale, low budget evaluations can yield useful informationBy the end of todays session, I hope you will rethink (at least a little bit) your view about evaluation and cross at least one of these barriers off your list of reasons why not to do evaluationmaybe, just maybe consider taking on some evaluation activities at your organizationEvaluation does not have to be expensive, complicated, time consumingand its one of the most valuable things you can do to improve your program and services7What is Evaluation?The process of determining the merit, worth, or value of a program (Scriven, 1991)

8What is Evaluation?Systematic inquiry that describes and explains, policies and programs operations, effects, justifications, and social implications (Mark, Henry, & Julnes, 2000)

9What is Evaluation?The systematic application of social research procedures for assessing the conceptualization, design, implementation, and utility of social intervention programs (Rossi & Freeman, 1989)

10In simpler terms..

Collection of information to determine the value of a program

eVALUation

By value I dont mean moneyI mean value in terms of improved knowledge, attitudes, skills, behaviors, quality of life, etc., real impacts on the people who participate in your programswhat is the value to your clients, patients, etc.11Evaluation is NOT.AuditingPersonnel assessmentMonitoring (although this can be part of an evaluation process)Used to end or shut down programs

The goal is not to make sure you are dotting your Is and crossing your Ts or to make sure that you are spending your money according to agreements with your funders; its also not meant to be an evaluation of you or your staff its called program evaluation because it is the program being evaluated (the impact of the program), not the staff who are being evaluatedProgram evaluators are not meant to look over your shoulder and monitor your activities and punish you if you are not doing what youre supposed to be doingMost importantly, the goal of evaluation is not to determine if a program should live or die evaluation is meant to collect information to help stakeholders figure out how to make their programs the best that they can beAbove all, evaluation id not done TO you, it should be done WITH you12Evaluation Myth #1Evaluation is an extraneous activity that generates lots of boring data with useless conclusions

That may have been true in the past when evaluations were focused heavily on very strict research methodologies and collected data on everything possible were not focused on specific evaluation questions -- often evaluations didnt provide and useful conclusions about program improvementThat has changed and the evaluation field has moved toward focusing efforts on only those things that you need and want to know about your program, using multiple and varied methods and the focus is on providing recommendations that are specific to how programs can improveFinally, evaluation should never be extraneous it should be integrated into your program operations13Evaluation Myth #2Evaluation is about proving the success or failure of a program

Ive said this before evaluation is not about success versus failure; good versus bad; It is about finding out what works within a program, what is not working so well, whether programs are achieving the outcomes they want to achieve, and if they are not, why not? Then we can use this information to pinpoint areas where the program can be changed to make it better14Evaluation Myth #3Evaluation is a unique and complex process that occurs at a certain time in a certain way, and almost always includes the use of outside experts.

Let me try to address the issues noted here. Not so. Many evaluations are done in house by program staff who may or may not be dedicated solely to evaluation tasks.There are as many different ways of evaluating programs as there are programs to be evaluated.There is no certain time or certain way that an evaluation must be done, as there are many ways to evaluate that range from very simple to quite complex. Sometimes simpler can be better. To be most effective, evaluation should be an ongoing process, not something that happens at a specific time

15How Can Evaluation Help You?Demonstrate program effectiveness or impacts

16How Can Evaluation Help You?Demonstrate program effectiveness or impacts Better manage limited resources

17How Can Evaluation Help You?Demonstrate program effectiveness or impacts Better manage limited resourcesDocument program accomplishments

18How Can Evaluation Help You?Demonstrate program effectiveness or impacts Better manage limited resourcesDocument program accomplishments Justify current program funding

19How Can Evaluation Help You?Demonstrate program effectiveness or impacts Better manage limited resourcesDocument program accomplishments Justify current program fundingSupport need for increased funding

20How Can Evaluation Help You?Demonstrate program effectiveness or impacts Better manage limited resourcesDocument program accomplishments Justify current program fundingSupport need for increased fundingSatisfy ethical responsibility to clients to demonstrate positive and negative effects of participation

21How Can Evaluation Help You?Demonstrate program effectiveness or impacts Better manage limited resourcesDocument program accomplishments Justify current program fundingSupport need for increased fundingSatisfy ethical responsibility to clients to demonstrate positive and negative effects of participationDocument program development and activities to help ensure successful replication

22UltimatelyTo improve program performance which leads to better value for your resources

23No Evaluation Means.No evidence that your program is working or how it works

24No Evaluation Means.No evidence that your program is working or how it works

Lack of justification for new or increased funding25No Evaluation Means.No evidence that your program is working or how it works

Lack of justification for new or increased fundingNo marketing power for potential clients26No Evaluation Means.No evidence that your program is working or how it works

Lack of justification for new or increased fundingNo marketing power for potential clientsLack of credibility27No Evaluation Means.No evidence that your program is working or how it works

Lack of justification for new or increased fundingNo marketing power for potential clientsLack of credibilityLack of political and/or social support28No Evaluation Means.No evidence that your program is working or how it works

Lack of justification for new or increased fundingNo marketing power for potential clientsLack of credibilityLack of political and/or social supportNo way to know how to improve29Program Life Cycle

But its critical to begin thinking about evaluation when you start the program development process this allows you to make sure that you identify appropriate outcomes, data collection methods, ect. Before the program gets under way30Basic TerminologyTypes of EvaluationOutcome (summative)Process (formative)

31Basic TerminologyTypes of EvaluationOutcome (summative)Process (formative)

OutcomesWhat you want your program participants to achieve common outcomes are related to knowledge, attitudes, skills, behaviors, intentions, health indicators are also common for health-relate programs32Basic TerminologyTypes of EvaluationOutcome (summative)Process (formative)

OutcomesIndicatorsIndicators are how you will measure your outcomes. For example, if you are measuring participants knowledge of how to treat epilepsy, one indicator might be their score on a test of epilepsy knowledge; or if your outcome is that caretakers have more positive attitudes toward people with epilepsy, the indicator might be their score on an attitude questionnaire33Basic TerminologyTypes of EvaluationOutcome (summative)Process (formative)

OutcomesIndicatorsMeasures34Basic TerminologyTypes of EvaluationOutcome (summative)Process (formative)

OutcomesIndicatorsMeasuresBenchmarks35Basic TerminologyTypes of EvaluationOutcome (summative)Process (formative)

OutcomesIndicatorsMeasuresBenchmarksQuantitative vs. qualitative36Evaluation Process

37Engage StakeholdersThose involved in program design, delivery, and/or fundingThose served by the programUsers of the evaluation results

How do you do that?? Engaging stakeholders is critical so that your evaluation gets done and gets used need buy in to get program staff to help with data collection, need support from funders to provide funds for evaluation, Need buy in from participants so that they will participate in data collection efforts, engage users who can tell you what they need to know about your programHow do you engage them? Tell them about your evaluation and ask for their input what do they want to know about your program? What questions would they ask? Can they help you with any aspects of the evaluation? Do they have any ideas that can help you focus your evaluation and give them the chance to be involved?38Clearly Define ProgramResources, activities, outcomesContext in which program operatesLogic modelExplicit connections between how and whatHelps with program impro