overview of outcome measures online toolbox
DESCRIPTION
afterschooloutcomes.org. Overview of Outcome Measures Online Toolbox. Deborah Lowe Vandell, PhD. Principal Investigator Professor and Founding Dean School of Education University of California, Irvine. Project Researchers: Valerie Hall, Ph.D. Pilar O ’ Cadiz , Ph.D. - PowerPoint PPT PresentationTRANSCRIPT
Overview of Outcome Measures Online ToolboxDeborah Lowe Vandell, PhD. Principal Investigator
Professor and Founding DeanSchool of EducationUniversity of California, Irvine
afterschooloutcomes.org
Project Researchers:Valerie Hall, Ph.D.
Pilar O’Cadiz, Ph.D.Andrea Karsh, M.A.
Background and Rationale for Online Toolbox Research indicates high quality afterschool programs can have
beneficial effects on students both socially and academically The Afterschool Outcome Measures Project at UC Irvine
developed an Online Toolbox in response to the State’s legislative mandate for alternative accountability measures to standardized test scores for publically funded afterschool programs in California*
Online Toolbox includes reliable and valid measures of Positive Behavior Change and Skill Development for use by diverse afterschool programs
Field Test of online toolbox at over 200 afterschool and summer learning program sites
Online Toolbox Statewide Implementation in Fall 2012/ Spring 2013
Development and Testing of Outcome MeasuresTHREE PHASES
PHASE 1—INSTRUMENT DEVELOPMENT (2008-2009) Literature review Survey of the field (expected outcomes and current practices) Selection of measures with input from the CDE and
Accountability and Outcomes Subcommittee (representation from diverse programs)
PHASE 2—PILOT STUDY (2009-2010) Pre-piloting at 9 sites to test procedures and instructions & to
determine feasibility of online administration of surveys Piloting at 28 sites to test protocols under varying conditions:
• Range of program size, geographic location, and level of operation• Sites with/without internet access
PHASE 3—FIELD STUDY (2010-2011) Test of measures & online administration across the State of
California• 196 field test sites representing diverse program sizes and
populations served by 29 programs in 9 regions statewide
NEXT PHASE: Statewide Implementation of Online
Toolbox for California ASES & 21stCCLC
Programs Opportunity to participate open to all publically funded afterschool programs in California:
Afterschool Education and Safety Program (ASES) 21st Century Community Learning Center (21stCCLC)
Up to 1000 sites statewide in 2012-2013
Up to 2000 sites to participate by 2014
Representation of diverse programs across 11 regions in the state
WHAT MEASURES ARE IN TOOLBOX ?
WHO TAKES THE SURVEYS?
HOW DO PROGRAMS ADMINISTER SURVEYS?
WHEN DO ASSESSMENTS TAKE PLACE?
WHY DO IT?
Outcome Measures Online Toolbox
Overview of Outcome Measures Web-based surveys of student performance collected from three
respondents: 1. Students (grades 3-12)2. Program staff3. Classroom teachers
PRE and POST Surveys collected at TWO time points, for example: Time 1 (early Fall/Winter) & Time 2 (Spring) Beginning of Summer & End of Summer
Technical Assistance provided through: Online Instructions and Resources Email and Telephone Communication
Individualized reports are provided to all participating programs. These reports enable programs to compare their students’
skills and behaviors over time, and to compare their program and site-level outcome scores to aggregate scores across the State.
Description of Measures
STUDENT SELF-REPORTS Skill Development •Work Habits•Reading Efficacy •Math Efficacy•Science Efficacy
Positive Behavior Change• Social Competencies• Reductions in Misconduct
Program Quality (post only)• Affiliation with Peers• Relations with Program Staff• Interest in Program Activities
PROGRAM STAFF & CLASSROOM TEACHER REPORTS—of Student Performance
Skill Development• Work Habits• Task Persistence• Academic Performance (classroom
teachers only)
Positive Behavior Change• Reductions in Aggressive Behaviors • Prosocial Behavior with Peers• Social Skills
Programs have option to administer one, two, or three surveys:
(1) Student (grades 3-12); (2) Program Staff; (3) Classroom Teachers
Field Test Key Findings
The Online Toolbox is an easy-to-use, inexpensive method of assessing skill development and positive behavior.
Elementary and middle school students reported having high quality experiences at their ASES programs.
Quality of experiences at ASES programs is linked to students’ skill development and positive behaviors.
Significant improvements in student performance were found between the fall and spring assessments.
SIGNIFICANT FINDINGS 2010-2011 Field Test
PRE-POST OUTCOMES in MATCHED SAMPLEStaff Reports of Elementary Students Increase in Prosocial Behavior **
Staff Reports of Middle School Students Increase in Prosocial Behavior **
Teacher Reports of Elementary Students Increase in Academic Performance ** Decrease in Aggressive Behavior**
Student Reports of Quality Program Experience Positive student reports of program experience linked
to positive results in ALL student outcomes** p<.01
Staff Reports on Student Prosocial Behavior
Increase in pre-post scores—matched sample
Fall Spring1.34
1.35
1.36
1.37
1.38
1.39
1.4
1.41
1.42
1.43
1.37
1.42
Fall Spring1.22
1.24
1.26
1.28
1.3
1.32
1.34
1.36
1.38
1.4
1.28
1.39
Elementary School Students Middle School Students
SIGNIFICANT FINDINGS
Teacher Reports on Elementary Students
Fall Spring2.6
2.7
2.8
2.9
3
3.1
3.2
2.79
3.13
Fall Spring0.1
0.15
0.2
0.25
0.3
0.35
0.4
0.35
0.28
Improvements in Academic Performance
Reductions in Aggressive Behavior
SIGNIFICANT FINDINGS
Program Quality Distribution
Staff & Activities Peer Affiliation
Low Fair Good Excellent0.0%
5.0%
10.0%
15.0%
20.0%
25.0%
30.0%
35.0%
40.0%
45.0%
5.5%
25.9%
34.3% 34.3%
9.9%
39.3%
31.9%
18.9%
Elementary Middle
Low Fair Good Excellent0.0%
5.0%
10.0%
15.0%
20.0%
25.0%
30.0%
35.0%
40.0%
45.0%
5.4%
24.8%
30.1%
39.7%
7.1%
32.6% 32.8%
27.4%
Elementary Middle
Student reports of higher quality experiences with staff & activities are linked to greater Math Efficacy
Student reports of higher quality experiences with peers are linked to less Aggressive Behavior
Quality to Outcomes
Low Quality Mid Quality High Quality0.10
0.20
0.30
0.40
0.50
0.60
0.70
0.60
0.45
0.34
0.64
0.53
0.43
Elementary Middle
Low Quality Mid Quality High Quality2
2.2
2.4
2.6
2.8
3
3.2
3.4
3.6
3.8
3.13
3.49
3.63
2.73
3.20
3.47
Elementary Middle
How does it work?http://afterschooloutcomes.org/
EASY TO ADMINISTERONLINE SURVEYS
Step-by-Step Instructions available to download from website
To APPLY to participate in
Statewide ImplementationCLICK HERE
Sign-up & Instructions on Website
To download Instructions for Administering
SurveysCLICK HERE
Program Staff & Teacher SurveysReports on Students in Afterschool Program
Program Staff and Classroom teachers are assigned individual access codes
Use both individual access code and the ID code for each student for whom they complete a report
Each student report takes 3-5 minutes to complete. Student reports may be completed in multiple
sessions. For example: A staff completing reports for 20 students can do four reports a day (a 20-minute session) on five different days.
Why the Online Toolbox? Provides a battery of valid, reliable, and easy to
administer youth outcome measures
Expands accountability tools beyond standardized test scores and other academic measures
Value in tracking individual student performance over time with matched pre-post surveys, providing more powerful & interpretable data• MATCHED = Respondent data are matched for both pre and post
survey administrations (i.e. same students, staff and/or teachers in fall and spring)
Information can be useful to guide program improvement based on student reports of program experience AND on student performance
For more information on measures:Forum for Youth
Investment publication (September 2011)
• Review of measures in Toolbox
with a technical appendix• Down Load at:
forumfyi.orgOr, link from our website:
http://afterschooloutcomes.org/
Contact Info:EMAIL: [email protected]