leading indicators: evaluation for site-level improvement …...leading indicators: evaluation for...
TRANSCRIPT
![Page 1: Leading Indicators: Evaluation for Site-Level Improvement …...Leading Indicators: Evaluation for Site-Level Improvement and System-Level Planning Samantha Sugar Research Associate/Analyst](https://reader034.vdocuments.mx/reader034/viewer/2022051810/60193b847593f43e2a39f87d/html5/thumbnails/1.jpg)
Leading Indicators: Evaluation for Site-Level Improvement and System-Level Planning Samantha Sugar Research Associate/Analyst David P. Weikart Center for Youth Program Quality Charles Smith Vice President for Research Forum for Youth Investment Executive Director David P. Weikart Center for Youth Program Quality April 18, 2012
#readyby21
![Page 2: Leading Indicators: Evaluation for Site-Level Improvement …...Leading Indicators: Evaluation for Site-Level Improvement and System-Level Planning Samantha Sugar Research Associate/Analyst](https://reader034.vdocuments.mx/reader034/viewer/2022051810/60193b847593f43e2a39f87d/html5/thumbnails/2.jpg)
Agenda • Welcome • Opening Activity • Important Questions
– The Why – The What – The How
• Site-Level Improvement • System-Level Planning
– The Where • Looking Forward/Next Steps • Questions
![Page 3: Leading Indicators: Evaluation for Site-Level Improvement …...Leading Indicators: Evaluation for Site-Level Improvement and System-Level Planning Samantha Sugar Research Associate/Analyst](https://reader034.vdocuments.mx/reader034/viewer/2022051810/60193b847593f43e2a39f87d/html5/thumbnails/3.jpg)
Objectives
• Learn how collecting program data from staff, youth and parents help to tell the overall story about Quality for a single site.
• See how this data can be used to identify low capacity programs and support resources targeting decisions.
![Page 4: Leading Indicators: Evaluation for Site-Level Improvement …...Leading Indicators: Evaluation for Site-Level Improvement and System-Level Planning Samantha Sugar Research Associate/Analyst](https://reader034.vdocuments.mx/reader034/viewer/2022051810/60193b847593f43e2a39f87d/html5/thumbnails/4.jpg)
Opening Activity Effec%veperformancedatadescribesbehaviorsandcondi%onsinawaythatis:
a.Timely–Dataisavailableinreal%measeventsoccurorjusta?ercomple%on
b.Objec%ve–Dataisfocusedonbehaviorsandcondi%onsthatcanbeiden%fiedthroughobserva%onandeasilynamedinrela%ontoprac%ce
c.Reliable–Dataisseenaspreciseandfactualbyallduetostandardiza%onofmeasures/methods
d.Sensi%ve–Datadescribesbehaviorsandcondi%onsthatarelikelytochangeinresponsetointerven%onandchangecanbecapturedonthemeasures
e.Valid–Datadescribesbehaviorsandcondi%onsthoughttobealinkinacausalchainofeventsdesiredbytheactorsinvolved
f.Feasible‐Theminimumdatanecessaryarecollectedusingtypicalcommunityresources
g.Mul%‐Purpose–Astheyoccur,BOTHdatacollec%onanddatainterpreta%onprocessespromotelearningandcoordina%onamongactorsintheorganiza%on
h.Mul%‐level–Datadesignedforusebyindividualunits(staff/sites)canbeaggregatedacrossindividualunitstoassesscollec%veperformance
Op%malCharacteris%csofPerformanceData
![Page 5: Leading Indicators: Evaluation for Site-Level Improvement …...Leading Indicators: Evaluation for Site-Level Improvement and System-Level Planning Samantha Sugar Research Associate/Analyst](https://reader034.vdocuments.mx/reader034/viewer/2022051810/60193b847593f43e2a39f87d/html5/thumbnails/5.jpg)
Quality of Instruction (Point of Service Setting)
YouthVoiceandProgramGovernanceStructures
ENGAGEMENT
INTERACTION
SUPPORTIVEENVIRONMENT
SAFEENVIRONMENT
Higherorderengagementthroughchoice,planning,andreflec4on.
Peerinterac4onthroughgroupingandcoopera4velearning.
Suppor4veenvironmentthroughwelcoming,conflictresolu4on,ac4velearning,andskillbuilding.
Physicalandemo4onalsafetyisprovided.
Why Were the Leading Indicators Developed?
![Page 6: Leading Indicators: Evaluation for Site-Level Improvement …...Leading Indicators: Evaluation for Site-Level Improvement and System-Level Planning Samantha Sugar Research Associate/Analyst](https://reader034.vdocuments.mx/reader034/viewer/2022051810/60193b847593f43e2a39f87d/html5/thumbnails/6.jpg)
Continuous Improvement Practices by Site Teams (Organizational Setting)
StandardizedAssessmentofInstruc4on
Team‐basedPlanningwithData
CoachingandPerformanceFeedback
TrainingforInstruc4onalSkills
Why Were the Leading Indicators Developed?
![Page 7: Leading Indicators: Evaluation for Site-Level Improvement …...Leading Indicators: Evaluation for Site-Level Improvement and System-Level Planning Samantha Sugar Research Associate/Analyst](https://reader034.vdocuments.mx/reader034/viewer/2022051810/60193b847593f43e2a39f87d/html5/thumbnails/7.jpg)
Objec4veData
MeaningfulInforma4on
Ac4on/Exper4se
ImprovedOutcomes
LowerStakesAccountabili4es
Interpre4veCommunity
• TeamSelfAssessment• Reviewexternalscores
TeamPlanningandImplemen4ng
• Improvementplanning• Performancecoaching
HigherStakesAccountabili4es
Why Were the Leading Indicators Developed?
![Page 8: Leading Indicators: Evaluation for Site-Level Improvement …...Leading Indicators: Evaluation for Site-Level Improvement and System-Level Planning Samantha Sugar Research Associate/Analyst](https://reader034.vdocuments.mx/reader034/viewer/2022051810/60193b847593f43e2a39f87d/html5/thumbnails/8.jpg)
Why Were the Leading Indicators Developed?
PointofServiceSeOng
Organiza4onalSeOng
PolicySeOng
Con%nuousImprovementPrac%cesforSiteTeams
QualityInstruc%on&ProximalChildOutcomes
LowStakesAccountabilityandSupports
TheoryofChange:Mul%pleLevelsofSe]ng
![Page 9: Leading Indicators: Evaluation for Site-Level Improvement …...Leading Indicators: Evaluation for Site-Level Improvement and System-Level Planning Samantha Sugar Research Associate/Analyst](https://reader034.vdocuments.mx/reader034/viewer/2022051810/60193b847593f43e2a39f87d/html5/thumbnails/9.jpg)
Effec%veperformancedatadescribesbehaviorsandcondi%onsinawaythatis:
a.Timely–Dataisavailableinreal%measeventsoccurorjusta?ercomple%on
b.Objec%ve–Dataisfocusedonbehaviorsandcondi%onsthatcanbeiden%fiedthroughobserva%onandeasilynamedinrela%ontoprac%ce
c.Reliable–Dataisseenaspreciseandfactualbyallduetostandardiza%onofmeasures/methods
d.Sensi%ve–Datadescribesbehaviorsandcondi%onsthatarelikelytochangeinresponsetointerven%onandchangecanbecapturedonthemeasures
e.Valid–Datadescribesbehaviorsandcondi%onsthoughttobealinkinacausalchainofeventsdesiredbytheactorsinvolved
f.Feasible‐Theminimumdatanecessaryarecollectedusingtypicalcommunityresources
g.Mul%‐Purpose–Astheyoccur,BOTHdatacollec%onanddatainterpreta%onprocessespromotelearningandcoordina%onamongactorsintheorganiza%on
h.Mul%‐level–Datadesignedforusebyindividualunits(staff/sites)canbeaggregatedacrossindividualunitstoassesscollec%veperformance
Op%malCharacteris%csofPerformanceData
Why Were the Leading Indicators Developed?
![Page 10: Leading Indicators: Evaluation for Site-Level Improvement …...Leading Indicators: Evaluation for Site-Level Improvement and System-Level Planning Samantha Sugar Research Associate/Analyst](https://reader034.vdocuments.mx/reader034/viewer/2022051810/60193b847593f43e2a39f87d/html5/thumbnails/10.jpg)
LeadingIndicator5.1–
FamilySa4sfac4on
LeadingIndicator4.2–AcademicEfficacy
LeadingIndicator4.1–Socioemo4onalDevelopment
LeadingIndicator2.2–
EngagingInstruc4on
LeadingIndicator1.1–StaffingModel
LeadingIndicator3.4–CommunityResources
LeadingIndicator3.3–
SchoolAlignment
LeadingIndicator3.2–
FamilyEngagement
LeadingIndicator2.1–AcademicPress
LeadingIndicator1.2–
YouthGovernance
LeadingIndicator3.1–SystemNorms
LeadingIndicator1.4–Enrollment
Policy
LeadingIndicator1.2–Con4nuousImprovement
PointofServiceSeOng
Organiza4onalSeOng
PolicySeOng
Con%nuousImprovementPrac%cesforSiteTeams
QualityInstruc%on&ProximalChild
Outcomes
LowStakesAccountabilityandSupports
Why Were the Leading Indicators Developed?
![Page 11: Leading Indicators: Evaluation for Site-Level Improvement …...Leading Indicators: Evaluation for Site-Level Improvement and System-Level Planning Samantha Sugar Research Associate/Analyst](https://reader034.vdocuments.mx/reader034/viewer/2022051810/60193b847593f43e2a39f87d/html5/thumbnails/11.jpg)
Why Were the Leading Indicators Developed?
• Isn’t this just more data?
![Page 12: Leading Indicators: Evaluation for Site-Level Improvement …...Leading Indicators: Evaluation for Site-Level Improvement and System-Level Planning Samantha Sugar Research Associate/Analyst](https://reader034.vdocuments.mx/reader034/viewer/2022051810/60193b847593f43e2a39f87d/html5/thumbnails/12.jpg)
13 composite measures categorized into five different contexts:
- Organizational Context - Instructional Context - External Relationships - Youth Characteristics - Family Satisfaction
What Are the Leading Indicators? WhataretheLeadingIndicators?
Wheredidtheycomefrom?
- Grantee Director/Site Coordinator Surveys - Afterschool teacher/Youth Worker Surveys
- Youth Surveys (grades 4-12)
- Parent Surveys
- PPICS data
Howdowemeasurethem?
- Youth Program Quality Intervention (YPQI) - California Outcomes Measures (Vandell)
- PPICS data
![Page 13: Leading Indicators: Evaluation for Site-Level Improvement …...Leading Indicators: Evaluation for Site-Level Improvement and System-Level Planning Samantha Sugar Research Associate/Analyst](https://reader034.vdocuments.mx/reader034/viewer/2022051810/60193b847593f43e2a39f87d/html5/thumbnails/13.jpg)
• Sample Report What Are the Leading Indicators?
![Page 14: Leading Indicators: Evaluation for Site-Level Improvement …...Leading Indicators: Evaluation for Site-Level Improvement and System-Level Planning Samantha Sugar Research Associate/Analyst](https://reader034.vdocuments.mx/reader034/viewer/2022051810/60193b847593f43e2a39f87d/html5/thumbnails/14.jpg)
What Are the Leading Indicators?
A
C
B
• Itemsaresimplythequestionsthatweaskonthesurveys,demographicandenrollmentdata,orYouthPQAscores.OnFigure1below,theItemscorrespondwithletterA. • ScalesaremadeupofgroupingsofdifferentItemsthatgotogetherwell.AScaleisdesignatedbyletterBinFigure1below. • LeadingIndicatorsaremadeupofgroupingsofdifferentScales,muchliketheScalesthemselvesaremadeupofItems.Intheexamplebelow,the“Accountability”Scale(alongwith“Collaboration”)makeuptheLeadingIndicator3.1–SystemNorms,whichisrepresentedbyletterCinFigure2below.[JB1] • Finally,alloftheLeadingIndicatorsaregroupedintoLiveoverarchingDomainsbasedonthecontextthattheyrepresent.TheseDomainsarecolor‐codedforeasydistinction,andinclude:OrganizationalContext(red),InstructionalContext(green),ExternalRelationships(blue),YouthCharacteristics(purple)andParentSatisfaction(brown).
Figure1
Figure2
![Page 15: Leading Indicators: Evaluation for Site-Level Improvement …...Leading Indicators: Evaluation for Site-Level Improvement and System-Level Planning Samantha Sugar Research Associate/Analyst](https://reader034.vdocuments.mx/reader034/viewer/2022051810/60193b847593f43e2a39f87d/html5/thumbnails/15.jpg)
Effec%veperformancedatadescribesbehaviorsandcondi%onsinawaythatis:
a.Timely–Dataisavailableinreal%measeventsoccurorjusta?ercomple%on
b.Objec%ve–Dataisfocusedonbehaviorsandcondi%onsthatcanbeiden%fiedthroughobserva%onandeasilynamedinrela%ontoprac%ce
c.Reliable–Dataisseenaspreciseandfactualbyallduetostandardiza%onofmeasures/methods
d.Sensi%ve–Datadescribesbehaviorsandcondi%onsthatarelikelytochangeinresponsetointerven%onandchangecanbecapturedonthemeasures
e.Valid–Datadescribesbehaviorsandcondi%onsthoughttobealinkinacausalchainofeventsdesiredbytheactorsinvolved
f.Feasible‐Theminimumdatanecessaryarecollectedusingtypicalcommunityresources
g.Mul%‐Purpose–Astheyoccur,BOTHdatacollec%onanddatainterpreta%onprocessespromotelearningandcoordina%onamongactorsintheorganiza%on
h.Mul%‐level–Datadesignedforusebyindividualunits(staff/sites)canbeaggregatedacrossindividualunitstoassesscollec%veperformance
Op%malCharacteris%csofPerformanceData
What Are the Leading Indicators?
![Page 16: Leading Indicators: Evaluation for Site-Level Improvement …...Leading Indicators: Evaluation for Site-Level Improvement and System-Level Planning Samantha Sugar Research Associate/Analyst](https://reader034.vdocuments.mx/reader034/viewer/2022051810/60193b847593f43e2a39f87d/html5/thumbnails/16.jpg)
How Have the Leading Indicators Been Used? OklahomaExemplar–SystemCharacteris4cs
• History of the system • Integration of QIS and required evaluation
efforts • 75 grantees in the first year, 77 this year • Timeline • Data Collection Methods • Outputs
![Page 17: Leading Indicators: Evaluation for Site-Level Improvement …...Leading Indicators: Evaluation for Site-Level Improvement and System-Level Planning Samantha Sugar Research Associate/Analyst](https://reader034.vdocuments.mx/reader034/viewer/2022051810/60193b847593f43e2a39f87d/html5/thumbnails/17.jpg)
Site-Level Improvement
![Page 18: Leading Indicators: Evaluation for Site-Level Improvement …...Leading Indicators: Evaluation for Site-Level Improvement and System-Level Planning Samantha Sugar Research Associate/Analyst](https://reader034.vdocuments.mx/reader034/viewer/2022051810/60193b847593f43e2a39f87d/html5/thumbnails/18.jpg)
• The report is… – A tool to help you identify the
strengths of your program – A tool to help you identify the
weaknesses of your program
• The report is not… – A mechanism to induce
evaluative comparisons or competitions across grants
– Something to be scared of
Site-Level Improvement
![Page 19: Leading Indicators: Evaluation for Site-Level Improvement …...Leading Indicators: Evaluation for Site-Level Improvement and System-Level Planning Samantha Sugar Research Associate/Analyst](https://reader034.vdocuments.mx/reader034/viewer/2022051810/60193b847593f43e2a39f87d/html5/thumbnails/19.jpg)
• Get a feel for the layout of the report • Study the graphs - In what areas are you
doing comparatively well? In what areas does it look like your site could improve?
• Celebrate your strengths • What could you work on? • Do some thinking • Prepare to make a plan!
Howtoreadandinterpretyourreport
Site-Level Improvement
![Page 20: Leading Indicators: Evaluation for Site-Level Improvement …...Leading Indicators: Evaluation for Site-Level Improvement and System-Level Planning Samantha Sugar Research Associate/Analyst](https://reader034.vdocuments.mx/reader034/viewer/2022051810/60193b847593f43e2a39f87d/html5/thumbnails/20.jpg)
Priority Assessment Form: Leading Indicators
1. Create the story of your data (column one) – What is the message or story of your data?
What do the numbers tell you? – What’s missing from the data? What
important things about program quality do not come through?
– Where are the gaps between what you want to provide and what the data says you’re providing?
2. Brainstorm ideas for improvement (column two)
Site-Level Improvement
![Page 21: Leading Indicators: Evaluation for Site-Level Improvement …...Leading Indicators: Evaluation for Site-Level Improvement and System-Level Planning Samantha Sugar Research Associate/Analyst](https://reader034.vdocuments.mx/reader034/viewer/2022051810/60193b847593f43e2a39f87d/html5/thumbnails/21.jpg)
![Page 22: Leading Indicators: Evaluation for Site-Level Improvement …...Leading Indicators: Evaluation for Site-Level Improvement and System-Level Planning Samantha Sugar Research Associate/Analyst](https://reader034.vdocuments.mx/reader034/viewer/2022051810/60193b847593f43e2a39f87d/html5/thumbnails/22.jpg)
The profiles (clusters) in Figure A-8 may be interpreted as follows. • Cluster 3: “High quality”
Thirty-three percent of grantees fall into Cluster 3, where programs show high quality in all areas.
• Cluster 2: “High with low -- growth/mastery and family communication” This cluster represents 24% of grantees. These programs show relatively high quality in most areas, but low school alignment and parent communication.
• Cluster 1: “Medium” Cluster 1 represents 21% of grantees. These programs have medium levels of supervision quality, high academic press & school alignment, low program quality (growth & mastery), and low family communication.
• Cluster 4: “Low with high -- school alignment & family communication” Four percent of grantees fall into Cluster 4, where programs show low quality in supervision, growth & mastery, and academic press, but high school alignment & family communication.
• Cluster 5: “Low quality” Eleven percent of grantees fall into Cluster 5, where programs appear to demonstrate low quality in all areas.
System-Level Planning
![Page 23: Leading Indicators: Evaluation for Site-Level Improvement …...Leading Indicators: Evaluation for Site-Level Improvement and System-Level Planning Samantha Sugar Research Associate/Analyst](https://reader034.vdocuments.mx/reader034/viewer/2022051810/60193b847593f43e2a39f87d/html5/thumbnails/23.jpg)
Intermediate & Academic Outcomes
• Table A-8 provides means of the satisfaction variables across cluster groups. The omnibus ANOVA indicates that the clusters produce significantly different scores in the Intermediate Outcomes set (staff job satisfaction, parent satisfaction, and parent reports of the program supporting academics). The highest quality group of grantees/sites (Cluster 3) produces the highest staff satisfaction, youth engagement, and academic efficacy, whereas the lowest quality group of grantees/sites (Cluster 5) exhibits the lowest or nearly the lowest in each area.
Table A-8 Means Scores for Intermediate Outcomes by Level of Quality
System-Level Planning
Cluster Staffjobsa%sfac%on
(S) ParentSa%sfac%on
(P) YouthEngagement
(Y) Homework
comple%on(Y) Prog.Supportsacademics(P)
3:Highquality 4.5 4.7 4.0 4.4 4.2
2:Highwithlow 4.2 4.7 3.8 4.0 3.9
1:Mediumquality 4.2 4.6 3.8 4.3 3.8
4:Lowwithhigh 3.5 4.8 3.7 4.2 4.2
5:Lowquality 3.6 4.5 3.7 4.2 3.6 Omnibusdifferenceacrossclusters(ANOVAF)
8.3***
2.6*
1.4
2.3+
5.9***
![Page 24: Leading Indicators: Evaluation for Site-Level Improvement …...Leading Indicators: Evaluation for Site-Level Improvement and System-Level Planning Samantha Sugar Research Associate/Analyst](https://reader034.vdocuments.mx/reader034/viewer/2022051810/60193b847593f43e2a39f87d/html5/thumbnails/24.jpg)
Where Have the Leading Indicators Been Used?
![Page 25: Leading Indicators: Evaluation for Site-Level Improvement …...Leading Indicators: Evaluation for Site-Level Improvement and System-Level Planning Samantha Sugar Research Associate/Analyst](https://reader034.vdocuments.mx/reader034/viewer/2022051810/60193b847593f43e2a39f87d/html5/thumbnails/25.jpg)
Looking Forward/Next Steps
• Further Validation Work – Exploration of Leading Indicators Framework –
Theoretical and Statistical
• Expansion to New Networks • Integration of Quality Improvement Systems
![Page 26: Leading Indicators: Evaluation for Site-Level Improvement …...Leading Indicators: Evaluation for Site-Level Improvement and System-Level Planning Samantha Sugar Research Associate/Analyst](https://reader034.vdocuments.mx/reader034/viewer/2022051810/60193b847593f43e2a39f87d/html5/thumbnails/26.jpg)
Questions?
![Page 27: Leading Indicators: Evaluation for Site-Level Improvement …...Leading Indicators: Evaluation for Site-Level Improvement and System-Level Planning Samantha Sugar Research Associate/Analyst](https://reader034.vdocuments.mx/reader034/viewer/2022051810/60193b847593f43e2a39f87d/html5/thumbnails/27.jpg)
Thank You!