1

Qualitative Evaluation TermsCoding/categorization The process of condensing

qualitative data through the identification of common themes.

Data Matrix or Display Grouping data within common themes in order to identify patterns in the data.

StorytellingUsing quotes and detailed description to capture the “story” or essence of a program or event.

2

Analyzing Quantitative Data

Can be done by program staff with the help of a spreadsheet (such as Excel).

Quantitative data analysis consists of:

Complex manipulations of the data designed to understand causal or correlational effects of program interventions.

Simple calculations yielding factual information on attendance, usage, changes in performance, and/or changes in knowledge or attitudes (pre/post test).

Best done by an evaluation contractor or trained researcher.

Or

3

4. Analyze your Data • Quantitative Data

Analysis–Analysis of numbers. Best presented in the form of pictures – graphs and charts.

• Qualitative Data Analysis– Analysis of words and pictures. Best presented as “word stories” or “video stories.”

11 22 33

55 66

4

2. Gather First Impressions

Guiding Questions:• What did you talk about

with interviewees? • What are your initial

theories of what is working and what is not?

• Did anything surprise you?

11 33

44 55 66

Discuss the quality and breadth of data with data collection team.

5

3. Organize and “Clean” Your Data

• Examine all the data related to each research question separately. Ideally, there is more than one data source for each question.

• Throw out surveys or

written responses that are less than halfway completed.

11 22

44 55 66

6

Overview of Data Analysis Process

1. Form a data collection team

2. Gather first impressions of data

3. Organize and “Clean” your data

4. Analyze data 5. Prepare findings & recommendations

6. Discuss emerging findings with key stakeholders

7

Tips for Designing & Choosing Methods

Consider your resource requirements.

Determine the appropriateness of methods for respondents, organizational approach, and setting.

Ensure that new data gathered fits well with existing data sources.

Determine that methods fit with evaluation questions.

Select methods that are easy to use.

8

How, What, Why? Choosing Methods (cont’d)

2. Identify a desired outcome/output and choose a method to measure it. Attendance: sign-in sheets, intake forms, etc. Writing skills: writing samples Knowledge of community: survey of youth knowledge

3. More than one method enhances credibility. Gathering multiple sources of evidence provides opportunities to include different perspectives.

Surveys and interviews: Surveys provide numerical account of youth experience and outcomes. Interviews highlight youth voices and stories.

9

How, What, Why? Choosing Methods

1. Consider evaluation questions. Your choice of tools should be driven by the types of questions you want answered (how, what, how many, why).

Why questions are best addressed through interviews and focus groups.

How

What

How Many

Why

How questions help you understand how something happened or the process of implementing a program. Interviews and focus groups.

What questions help you document what program staff have done and what participants have experienced. All methods.

How many questions are best addressed through surveys, activity logs, intake data, etc.

10

ImplementationQuality & Quantity

OutcomesEffectiveness, Magnitude & Satisfaction

SettingAssumptions

Connecting Logic Model to Evaluation Questions

ContextContext

InputsInputsActivitiesActivities OutputsOutputs Short-term

Outcomes

Short-term Outcomes

Intermediate Outcomes

Intermediate Outcomes

Long-term Outcomes

Long-term Outcomes

Adopted from WKKF Logic Model Development Guide p.36

What aspects of our situation most shaped our ability to do the work we set out to do in our community?

What did our program accomplish in the community?

What resulted from our work in the community?

What have we learned about doing this kind of work in a community like ours?

11

Identify Evaluation Questions

• Evaluation questions focus on understanding how your program will meet its intended goals. Good questions enable you and others to get the answers you need to tell your story.

• Pick a program activity that you want to evaluate. • Decide whether you want your questions to focus on the

processes or outcomes of your work.• Use simple, concrete language that focuses on what you need

to know.

12

Identify strengths and weaknesses of program activities.

Benefits of EvaluationInform practice to improve program.Inform practice to improve program.

Build organizational capacity.Build organizational capacity.

Inform and refine community change efforts.Inform and refine community change efforts.

Enhance personal growth and development among staff and youth participants.

Enhance personal growth and development among staff and youth participants.

Provide evidence of program effectiveness.Provide evidence of program effectiveness.

Improve ability to plan and implement programs/campaigns. Document program progress toward meeting goals.

Identify unmet community needs and assess impacts of social change efforts.

Provide feedback to staff/participants on their work. Recognize accomplishments and provide suggestions for improvement.

Report to community and funders about program effectiveness.