making it count! program evaluation for youth-led initiatives

Making it Count!Program Evaluation For Youth-Led Initiatives

What Is Evaluation?

• Evaluation is a systematic determination of a subject's merit, worth and significance, using criteria governed by a set of standards.

• Wikipedia

In plain language, evaluation measures the value of something against the goals we identified. In community development, this often means we measure to determine if we are making the changes we hoped in our participants and our communities.

Why Evaluate?

• For our learning! • We know what works, what doesn’t and can make revisions to our

programs and services• This is your time to experiment, practice, learn and grow—this kind of

learning will help prepare you for your future!

• To be accountable!• Funders (and the public) want to know what was done and accomplished

with the money provided• You can show your community what has been achieved and the progress

that’s been made

• You know you’re making a difference—and you can prove it!

MAKING CHANGE Defining Your Outcomes

Outcomes

What are outcomes?

• Outcomes are changes in behaviour, attitude, skills, condition, knowledge.

• We can see outcomes in our community, our participants or clients, and even in ourselves.

Questions to Consider

To identify your outcomes, think about:

What difference do you WANT to make in your community with your programs and services?

What changes do you hope or expect to see in your clients or participants?

What changes do you hope or expect to see in yourself, your staff, and your volunteers?

Examples of Outcomes

• Youth feel a sense of increased safety• Increased participation of youth in community events• Youth experience increased access to opportunities• Youth increase participation in civic activities (ex.

Voting)• Decrease in violence involving youth

Measuring Outcomes

• Outcomes, like change, take time; some of the outcomes you identify may not happen for a long time and therefore may be difficult to measure, manage or track

• Think about shorter term outcomes you might be able to measure more easily, like changes in knowledge:• The youth participants have an increased understanding of

sexual health• The youth participants feel an increased sense of self-

confidence

• Outputs are different than outcomes. While outcomes talk about changes we want to see in participants and community, outputs talk about hard numbers and deliverables.

• Output: the immediate results of our program activities and services, usually numeric, or the deliverables (hard product) that we create as a result of our activities.

Examples of Outputs

Activities Outputs

Host employment workshops # of workshops# of attendees per workshop# of total attendees1 toolkit1 workshop guide

Host budgeting/financial management workshops

# of workshops# of attendees per workshop# of total attendees1 toolkit1 workshop guide

Measuring Outputs

• Measuring outputs is generally far easier than measuring outcomes

• Often outputs are measured through simple tracking devices, like attendance, or the completion of a deliverable

Indicators /Measures

• Indicators are the measurements that you’ll track to determine if you’ve achieved your outcomes. There are generally two types:• Qualitative: these measures do not involve numbers, and instead

focus on descriptions of changes, testimonials, and stories of change 

• Quantitative: these are numerical measures (numbers! Numbers! Numbers!) that show the changes that have been made

Have Indicators that capture both kinds of information

Indicators

Qualitative Quantitative Purpose To provide context,

detail, and description around the changes your program or services are creating.

To provide clear data (proof) that demonstrates changes that your program or services are creating.

Benefit Creates a human element that allows people to connect to the story. Paints a picture, tells a story that’s easily understood.

Demonstrates change beyond one person or one story.Shows the bigger picture.

Disadvantages Can be seen as an individual success that doesn’t show the whole story.

Misses the human impact element—doesn’t tell a story or create empathy.

Indicators

• Indicators can be hard to develop or select. Try thinking about:• What measurement will prove that we’ve achieved our

outcome?• What information do we already collect or what information

is collected elsewhere that we can use?• How can you collect the information you need?

Examples of Indicators

Outcome IndicatorThe youth participants have an increased understanding of sexual health

-increased # of youth can identify sexual health resources in their communities-increased # of youth report using condoms with other forms of protection…

The youth participants feel an increased sense of self-confidence

-increased # of youth indicate they have more hope about their future-increased # of youth are entering/using spaces that they wouldn’t have before…

Program Planning

In order to evaluate at the end of a program, you need to think about evaluation right from the beginning! Program planning is when you outline your outcomes, outputs, and indicators, among other things.

When you begin to plan a program, take into consideration:• Your community’s likes, wants, needs, assets• Your organization’s mission, strengths • The outcomes you want to achieve• The activities that will allow you to achieve your outcomes

Types of Evaluation

• Evaluation can be simple or it can be very complex. This may depend on:• The type of program• The length of the program • The amount of funding/resources you have• The outcomes you identified • The activities or services you’re offering

The Fundamentals

• At it’s core, evaluation is about collecting feedback from your participants about their experience using your programs or services, and the changes that may (or may not) have come about because of them.

• Even with minimal resources, you can do the following:

1. Get “baseline” data: This means knowing where your participants are at when they start coming to your program or using your services. This can be a simple intake process, a survey on the very first day, or asking a few key questions to the group that help you understand what they understand about the program topic.

2. Take attendance: Know how many program activities were held or services offered, know how many people used them and who used them.

3. Get feedback: After every program activity or service, ask the participants what they thought—a survey, a few questions to the group—ask about the quality of the program or service, but also ask them about what they learned, what they still don’t understand, and how the program can be better.

4. Find out what’s changed: At the end of the program or service offering, ask the same questions you asked to get your baseline data. Make note of what’s changed in the knowledge, attitude, and behaviours of your participants. Host focus groups or ask for testimonials to get stories. Make sure your participants know how you’re going to use what they say (ex. Put it on the website, share it in a grant application….)

RBA: Population Accountability

• Results-based accountability talks about some BIG outcomes, that it calls “population accountability” • It refers to changes in the well-being of WHOLE populations

(beyond participants) over time

• The challenge is you need to be able to a.) measure over a longer period of time and b.) demonstrate that your programs and services made the change

• While this is important, it can be difficult to apply

RBA: Performance Accountability

• RBA also talks about the value of a program or service to participants through something called “performance accountability”

• It’s based on two simple questions:How much did we do?How well did we do it?

How much did we do?

How welldid we do it?

Is anyonebetter off?

Quantity Quality

Eff

ect

Eff

ort

#%

Stories

(Outputs) (Response to activities and processes)

(Outcomes)

Young Women’s Sexual Health Program

Is anyone better off?

-# of workshops

-# of field trips

-# of new sexual health resources shared

-participants’ feedback on value of workshops and field trips, and shared resources

-# of participants accessing new sexual health resources

-# of participants who feel more confident about sexual health

-# of or % of participants who utilize protection/contraception

-participants feel empowered about their own sexual health

Quantity: How much did we do?

Quality: How well did we do it?

Eff

ect

E

ffor

t

Tools for Measurement

• Once you identify what you want to measure, you need to identify and create or find the tools to measure.• What is the best way to gather the information you need?• What tools are appropriate for your participants and

community?• How will you get the best and richest information?

Tools for Measurement:

• Surveys• Focus Groups• Interviews• Art Projects • Census Data (identify broader changes)

An Evaluation Plan

Activity, Output, or OutcomeIndicator Baseline Targets Data

SourceCollection Methods

Frequency Responsibility

What you’re going to measure

Where your target population is at the beginning

How much you expect they will change or achieve

What will you use to collect the data you need? When and how will you collect it?

Who is in charge of implementing this plan?

-# of youth accessing sexual health resources

-2 of 15 participant accessed sexual health resources

-10 of 15 participant accessing sexual health resources

-youth participants will share resource usage with us-survey-start of program, at midway point, at end of program

-program coordinator

Quiz Time! What did you learn?

1. What is the purpose of evaluation in community development?

2. What is the difference between outcomes and outputs?3. What kind of indicators exist? What’s the value of

each?4. What does results-based accountability’s performance

accountability model measure?5. What are some tools you can use to collect

measurements?