1
Qualitative Evaluation TermsCoding/categorization The process of condensing
qualitative data through the identification of common themes.
Data Matrix or Display Grouping data within common themes in order to identify patterns in the data.
StorytellingUsing quotes and detailed description to capture the “story” or essence of a program or event.
2
Analyzing Quantitative Data
Can be done by program staff with the help of a spreadsheet (such as Excel).
Quantitative data analysis consists of:
Complex manipulations of the data designed to understand causal or correlational effects of program interventions.
Simple calculations yielding factual information on attendance, usage, changes in performance, and/or changes in knowledge or attitudes (pre/post test).
Best done by an evaluation contractor or trained researcher.
Or
3
4. Analyze your Data • Quantitative Data
Analysis–Analysis of numbers. Best presented in the form of pictures – graphs and charts.
• Qualitative Data Analysis– Analysis of words and pictures. Best presented as “word stories” or “video stories.”
11 22 33
55 66
4
2. Gather First Impressions
Guiding Questions:• What did you talk about
with interviewees? • What are your initial
theories of what is working and what is not?
• Did anything surprise you?
11 33
44 55 66
Discuss the quality and breadth of data with data collection team.
5
3. Organize and “Clean” Your Data
• Examine all the data related to each research question separately. Ideally, there is more than one data source for each question.
• Throw out surveys or
written responses that are less than halfway completed.
11 22
44 55 66
6
Overview of Data Analysis Process
1. Form a data collection team
2. Gather first impressions of data
3. Organize and “Clean” your data
4. Analyze data 5. Prepare findings & recommendations
6. Discuss emerging findings with key stakeholders
7
Tips for Designing & Choosing Methods
Consider your resource requirements.
Determine the appropriateness of methods for respondents, organizational approach, and setting.
Ensure that new data gathered fits well with existing data sources.
Determine that methods fit with evaluation questions.
Select methods that are easy to use.
8
How, What, Why? Choosing Methods (cont’d)
2. Identify a desired outcome/output and choose a method to measure it. Attendance: sign-in sheets, intake forms, etc. Writing skills: writing samples Knowledge of community: survey of youth knowledge
3. More than one method enhances credibility. Gathering multiple sources of evidence provides opportunities to include different perspectives.
Surveys and interviews: Surveys provide numerical account of youth experience and outcomes. Interviews highlight youth voices and stories.
9
How, What, Why? Choosing Methods
1. Consider evaluation questions. Your choice of tools should be driven by the types of questions you want answered (how, what, how many, why).
Why questions are best addressed through interviews and focus groups.
How
What
How Many
Why
How questions help you understand how something happened or the process of implementing a program. Interviews and focus groups.
What questions help you document what program staff have done and what participants have experienced. All methods.
How many questions are best addressed through surveys, activity logs, intake data, etc.
10
ImplementationQuality & Quantity
OutcomesEffectiveness, Magnitude & Satisfaction
SettingAssumptions
Connecting Logic Model to Evaluation Questions
ContextContext
InputsInputsActivitiesActivities OutputsOutputs Short-term
Outcomes
Short-term Outcomes
Intermediate Outcomes
Intermediate Outcomes
Long-term Outcomes
Long-term Outcomes
Adopted from WKKF Logic Model Development Guide p.36
What aspects of our situation most shaped our ability to do the work we set out to do in our community?
What did our program accomplish in the community?
What resulted from our work in the community?
What have we learned about doing this kind of work in a community like ours?
11
Identify Evaluation Questions
• Evaluation questions focus on understanding how your program will meet its intended goals. Good questions enable you and others to get the answers you need to tell your story.
• Pick a program activity that you want to evaluate. • Decide whether you want your questions to focus on the
processes or outcomes of your work.• Use simple, concrete language that focuses on what you need
to know.
12
Identify strengths and weaknesses of program activities.
Benefits of EvaluationInform practice to improve program.Inform practice to improve program.
Build organizational capacity.Build organizational capacity.
Inform and refine community change efforts.Inform and refine community change efforts.
Enhance personal growth and development among staff and youth participants.
Enhance personal growth and development among staff and youth participants.
Provide evidence of program effectiveness.Provide evidence of program effectiveness.
Improve ability to plan and implement programs/campaigns. Document program progress toward meeting goals.
Identify unmet community needs and assess impacts of social change efforts.
Provide feedback to staff/participants on their work. Recognize accomplishments and provide suggestions for improvement.
Report to community and funders about program effectiveness.