operational excellence: outcomes evaluation system impact and outcomes breakout session june 14 th,...

Operational Excellence: Outcomes Evaluation System

Impact and Outcomes Breakout SessionJune 14th, 2011

Sandra LaFleur, Senior Director of Youth Outcomes

2011 National Conference • Dallas, Texas • June 14 - 16

Purpose & Desired Outcomes

• Review critical components of BBBS’ Outcome Evaluation System (OES)

• Highlight latest data analyses / results

• Reinforce focus on and clarify understanding of the OES

• Others?

2011 National Conference • Dallas, Texas

Today’s framing…• Understand it!

– The rationale for it– The connection to both the past and the present

• Do it!– Are you using RR, SoR, YOS?– How are you tracking it?

• Use it!– Case for support– Indications for improvement

Big Brothers Big Sister’s Outcomes Evaluation System (OES)

Click to add text


BBBS’ collective, evidenced-based approach to mentoring–a professional system that changes the lives of our children for the better, forever.

• Builds upon our two P/PV study findings that indicate specific areas where our work yields impact

• Builds upon BBBS’ ability to institute common service delivery methodologies and measures through AIM

• Gives us real-time, agency-specific information on our progress at achieving critical youth outcomes in addition to our nationwide reach


Connecting Research to YOS Survey

Leveraging long-standing research to account for real-time progress in our youth across the Big Brothers Big Sisters network


We didn’t get here overnight… BBBS has developed Unmatched Capabilities to Link Research & Process to Drive Outcomes on a National Scale


Components of BBBS’ Outcomes Evaluation System (OES)

Eight (8)Measurable

Outcome Areas

Focused on:

Community Impact

Future:

Additional 3rd party research

Future: Targeted hard data collection

Youth Outcomes:YOS survey

Match QualityStrength (SOR survey)

Outcomes/AccountabilitiesEducational success

Avoidance of risky behaviorAspirations, confidence, relationships

Match QualityLength (RR=retention rate)

Community Impact


Research…to Outcomes…to Accountabilities


Reflect…

• Does your board know this information?• Does your agency leadership understand

this information?• Does your program staff recognize their

connection to this?


MATCH SUPPORT Match Support Complete 2011 92.0% 96 % 95.5% 89.7% 6.3%

Match Support On Time 2011 71.0% 84.7 % 84.9% 66.5% 18.2%

% Completion SOR/SITE 3 MO 2011 70.0% 75% 100% 52.0% %

% Completion SOR/COMM 3 MO 2011 70.0% 64% 59.0% 46.0% %

% Completion of YOS/Baseline 2011 100% 97.9% 91.7% 92.2 % 6.2%

% Completion of YOS/Annual 2011 70% 52.9% 65.0% 0 % %

Average Match Length SITE 2011 13.0mo 9.3 9.3 mo 11.5 mo -19.1 mo

Average Match Length COMM 2011 21.5mo 16.7 mo 18.7 mo 17.3 mo -3.2 mo

12 Month Retention COMM 2011 61.1% 56.6 % 49.2% 15.1% 41.5%

12 Month Retention SITE 2011 35.0% 37.3 % 36.1% 11.8% 25.5%

TEAM

Total Youth Served 2011 16391084/680CB

248SB1028/727CB

301SB1,225/704 CB

310SB -10.5%

Closures 2011 40% 12.3% 8.6% 12.4% -1.1%


SoR3 month vol compliance

YOSfollow-up

compliance

REGIONS 2010 Q1 2011 2010 Q1 2011

NE 45% 64% 45% 50%

SE 61% 75% 59% 66%

MW 55% 61% 63% 52%

SW 63% 78% 58% 60%

W 57% 70% 57% 53%


Reflect…• Are you currently tracking your agency’s

performance on 12 month RR, SoR compliance, and YOS compliance?

• Do you share these numbers with the whole staff?

• Are these goals broken down into individual goals for appropriate staff?

• Are they held accountable for performance on those individual goals critical to OES?


Developing Your Case for Support


Outcomes Data – More Terms

Correlates to the three accountability areas identified in the BBBSA 2011-2014 Strategic Direction. Your local needs may differ or may require a focus in one particular area.

Strategic Outcomes Areas

Also called Outcome categories (8) and sub-categories on the YOS Score and Outcome reports. Outcomes Measures

Refers to the number of youth whose scores indicate improvement (NOT the percentage change from baseline score to follow-up score)

Percentage of Youth Improving

Percentage of Youth Staying the Same

Statistically Significant

Refers to the number of youth whose scores indicate no change. For example, staying the same in a measure that tends to decline as a child enters teen years could be meaningful.

A statistical significance test determines whether a change in scores is due to chance. For example, if the p value =.05, there is only a 5% likelihood that the change is due to chance


Preliminary YOS Data: Early evidence of positive youth outcomes

YOS Scales CB SBSocial acceptance * *

Scholastic competency * *

Grades *

Educational expectations * *

Attitudes towards risk * (*)

Parental trust * *

Truancy

Special adult * *


Preliminary YOS Data: Early evidence of positive youth outcomes SUB SCALES

YOS Scales CB SBFinish High School (Educ Exp) * *

Go to college *

Finish college *

Drugs *

Hitting *

Breaking rules in school *

Being late to school *


Preliminary YOS Data: SPECIAL POPULATIONS

YOS Scales CBM SBM CBM Children w/ incarcerated

parent

SBM Hispanic Girls

SBM Hispanic

Boys

Social acceptance * * * * *

Scholastic competency

* * * * *

Grades * *

Educational expectations

* * * *

Attitudes towards risk

* (*) *

Parental trust * * *

Truancy

Special adult * * * * *


Preliminary YOS Data: (Future Analysis) SPECIAL POPULATIONS

YOS ScalesFoster Care

Youth

Juvenile Justice

referrals

CBM African

American

SBM African

American

CBM Native

American

SBM Native

American

Social acceptance

Scholastic competency

Grades

Educational expectations

Attitudes towards risk

Parental trust

Truancy

Special adult


Predicting match closures with SoR…• when controlling for background variables, both 3 month volunteer and child SoR scores are predictive of match closures• Vol SoR scale ranking:

•Confidence•Lack of Frustration•Closeness•Connectedness

•Child SoR scale ranking: •Closeness•Lack of Disappointment•Coping


Revisiting SoR Cut Points

Volunteer 3 month SoR Score > 3.95

43% more likely to “survive” than matches below this level

Child 3 month SoR Score > 4.65

39% more likely to “survive” than matches below this level


Reflect…• How successfully is your agency leveraging your

OES data in building your case for support?• Are your OES performance results used to

discuss improvements or enhancements to program operations?


Next steps…• Assess your obstacles and your facilitators• Talk with your agency team – build your

action plan

operational excellence: outcomes evaluation system impact and outcomes breakout session june 14 th,...

Documents

critical youth outcomes

drive outcomes

outcomes breakout sessionjune

bbbs ability

overnight bbbs

completion sorcomm

completion of yosannual201170

completion sorsite