measure what matters karen macdonald, bgca midwest leadership conference october 16, 2009
TRANSCRIPT
Measure What Matters
Karen MacDonald, BGCA
Midwest Leadership Conference
October 16, 2009
“Not everything that counts can be counted, not
everything that can be counted counts.”
“In God we trust, all others bring data.”
“Data is the new sexy!”
Why is Outcome Measurement important?
• Era of accountability
• Doing good is not enough
• Stakeholders are asking more difficult
questions
The Importance of Measurement
Formal Program Evaluation Outcome Measurement
Does the program work? How well is it working?
• Proving • Improving
• Research tool • Management tool
• Applies to programs • Applies across org
• Analyzing • Learning
• Theory of Change • Best Practices
Program Evaluation vs Outcome Measurement:
Key Elements of Impact
• Safe, positive environment• Fun• Supportive relationships• Opportunities and expectations• Recognition
Making Every Day Count
• Longitudinal Study • Tracked 7th and 8th graders
for 30 months• Linked levels of
attendance at Boys & Girls Clubs to outcomes
• Finding – MORE IS BETTER
Outcome (All changes are positive changes)
Minimum Level of Attendance Over 30-Months Related to Positive Change in
Outcome
52 Days 122 Days 244 Days 366 Days
School effortConfident in abilityIntegrityCommunity service Initiate marijuana use Future connectedness
ShynessAggressionInitiate weapon carryingStopped by policeInitiate drinking Initiate Smoking
Number times skippedInitiate sexual activity
Negative peers
Four things led to teen participation…
1. Fully engaged in a variety of activities
2. Friends came to the Club3. Involved in leadership roles4. First became involved as
preteens or even younger
Making Every Day Count affirms…
• Attendance matters• Participation matters• Fun matters• Relationships matter• Programs matter
Annual VisitsDefinition: The number of times a member
attends the Club over the course of the year.
• It’s important to track this metric.• 52 - 104 times per year is the baseline• Increase attend levels of all members
regardless of current attendance patterns
A big thank you to…• Boys & Girls Clubs
of Greater Milwaukee
• Boys & Girls Clubs of Omaha
Planning a Measurement Strategy:Key Steps
• Identify key organizational priorities and select
key indicators
• Track attendance and program participation• Locate and collect data• Understand and use data• Communicate results
•Review sources that outline your priorities
including your strategic plan
•Review the needs of your community
•Involve your board by asking questions
Identify Organizational Priorities
Success Equation Components
A + B + C = D A + B + C = D
What are you ultimately trying to accomplish?
Priority OutcomesWhat changes in behavior or condition are required?
IndicatorsHow do we measure progress
against our outcomes?
Key StrategiesWhich
programs or activities will
drive results?
© 2008 Mission Measurement LLC – as adapted from Benchmarking for Nonprofits, by Jason Saul
• Use your success equation and priority outcomes as your guide
• Start with the basics: attendance and program participation
• Continue with more complex indicators: changes in knowledge/attitude and changes in behavior
• Use the Outcomes Framework as a guide
• Select indicators that are feasible to collect and credible to communicate
Select Indicators to Track Progress
•Practical – Can you realistically capture this data without a lot of cost or difficulty?
•Actionable – Will the data allow you to make decisions in a timely manner?
•Relevant – Does it answer the “so what” question?
•Controllable – Is it within your sphere of influence or control?
Characteristics of Good IndicatorsCharacteristics of Good Indicators
• more time spent going to the Club was linked to positive change
• research supports the importance of regular attendance and the links to positive outcomes.
• at least 52 times per year is a baseline • Clubs should strive to increase the attendance levels of all
members • Average daily attendance is an important measure of
Club’s operational capacity, but does not measure individual members and is therefore not an indicator of changes for youth.
Track Attendance and Program Participation
Locate and Collect DataLocate and Collect Data• Collect on a program-by-program basis• Leverage existing tools, i.e. test or surveys• Use Club-wide surveys when warranted• Tests or Surveys can be administered by hand or electronically• Results from the tests or surveys can be tracked in the member’s management system.
Outcome Indicator Indicator Definition Location of data Frequency Collect Where
Report
Academic Achievement
Indicator 1
Indicator 2
Indicator 3
Locate and Collect DataLocate and Collect Data
• Improve program and/or youth results • Share progress and success
• Share results alone • 50% of members participated in community
service.• Share results compared past performance
(growth/change)• Participation in community service activities has
increased by 25%.
Understand and Use ResultsUnderstand and Use Results
Indicator Sample Impact Statement Links to External Research
Number of volunteer hours completed by Club members
75% of Boys & Girls Clubs of Main Street members participated in volunteer service activities in 2008 and contributed a total of 1,000,000 hours of service.
Young people who volunteer are likely to do better in school, graduate, maintain positive attitudes towards society and avoid risky behaviors (4)
Simply based on regular member attendance and program participation, Clubs can demonstrate the impact they are having by linking participation to external research on positive activities:
• Explore and define how measurement can and will be used
• Setting the right tone and creating ownership and buy-in
• Setting the process in motion • Identify the processes and plans needed to bring
measurement to life
Engage your StaffEngage your Staff
Available Resourceswww.bgca.net/demonstratingimpact
• Demonstrating Impact Guide• Outcome Measurement Framework• BGCA - Public/Private Ventures Study• Distance Learning courses
Abigail Diner, [email protected] MacDonald, [email protected]
Romero Brown, [email protected]