Presentation to the Presentation to the

Before and Afterschool Advisory Committee Before and Afterschool Advisory Committee

California Department of EducationCalifornia Department of Education

January 11, 2011January 11, 2011

Pilot Findings and Field Test:Pilot Findings and Field Test:An Update An Update

California Afterschool Outcome Measures ProjectCalifornia Afterschool Outcome Measures Project

Last report to the Advisory Committee —Last report to the Advisory Committee —February 24, 2010 February 24, 2010

Pre-Pilot activities and Key Findings:Pre-Pilot activities and Key Findings: Completed pre-pilot test of initial versions of measures Completed pre-pilot test of initial versions of measures

and online administrationand online administration Measures performed well in online format; preliminary Measures performed well in online format; preliminary

outcomes comparable to previous studiesoutcomes comparable to previous studies Minor modifications made to measures and Minor modifications made to measures and

procedures based on pre-pilot findingsprocedures based on pre-pilot findings• Changes to wording in items, scales removed and addedChanges to wording in items, scales removed and added• Survey administration instructions revised—clear step-by-Survey administration instructions revised—clear step-by-

step step • Special instructions created to support ELL students and Special instructions created to support ELL students and

slow readersslow readers

Highlights of Today’s ReportHighlights of Today’s Report

Phase 2—Report on the Pilot Phase 2—Report on the Pilot TestTest Creation of project websiteCreation of project website Results of the spring 2010 pilot workResults of the spring 2010 pilot work

Phase 3—Update on the Field Phase 3—Update on the Field TestTest Fall survey administration experienceFall survey administration experience Dissemination plansDissemination plans

Phase 2: PILOT PHASEPhase 2: PILOT PHASE

Pilot DataPilot DataNumber of Respondents & Time to complete Number of Respondents & Time to complete

Survey Survey 1,656 Students completed surveys1,656 Students completed surveys

• 1,175 Elementary; 481 Middle1,175 Elementary; 481 Middle• On average less than 10 minutes to complete survey (observed)On average less than 10 minutes to complete survey (observed)• 51% elementary students report they took less than 10 minutes51% elementary students report they took less than 10 minutes• 61% middle school students report taking less than 10 minutes61% middle school students report taking less than 10 minutes

105 Staff completed 769 student reports105 Staff completed 769 student reports• 54% report taking less than 5 minutes per student report54% report taking less than 5 minutes per student report• 46% report taking 5 to 9 minutes46% report taking 5 to 9 minutes

70 Classroom Teachers completed 535 student reports70 Classroom Teachers completed 535 student reports• 81 % report completing each student report in less than 5 minutes81 % report completing each student report in less than 5 minutes• 18% report taking 5-10 minutes per report18% report taking 5-10 minutes per report• 1% took more than 10 minutes 1% took more than 10 minutes

FIELD TESTFIELD TEST (2010-2011) (2010-2011)FIELD TESTFIELD TEST (2010-2011) (2010-2011)Overview of key elements: Overview of key elements:

Participation open to all programs in the stateParticipation open to all programs in the state

Outreach effort to all regionsOutreach effort to all regions::• Presented on Field Test at Regional Lead Meeting (October 13, 2010)Presented on Field Test at Regional Lead Meeting (October 13, 2010)• Sent Field Test materials electronically to RLs for dissemination to all granteesSent Field Test materials electronically to RLs for dissemination to all grantees

Web-based surveys for students, program staff, and Web-based surveys for students, program staff, and classroom teachers completed in fall 2010 and spring classroom teachers completed in fall 2010 and spring 2011 2011

• Programs select which surveys to administerPrograms select which surveys to administer

Confidential summary report of survey resultsConfidential summary report of survey results• students’ scores of positive behavior change and skill development provided to students’ scores of positive behavior change and skill development provided to

participating programsparticipating programs

All materials and instructions available on Project All materials and instructions available on Project websitewebsite

Technical assistance provided to programsTechnical assistance provided to programs• Support provided by email and telephone during pre and post survey administrations Support provided by email and telephone during pre and post survey administrations

& in interpreting their scores at the end of Field Test & in interpreting their scores at the end of Field Test

Measures modified based on Pilot results and feedback Measures modified based on Pilot results and feedback from fieldfrom field

FIELD TESTFIELD TEST (2010-2011) (2010-2011)Outcome MeasuresOutcome Measures

StudentStudent—Elementary & Middle School——Elementary & Middle School—(5 subscales)(5 subscales)• Work HabitsWork Habits• MisconductMisconduct• Reading Efficacy Reading Efficacy • Math EfficacyMath Efficacy• Social EfficacySocial Efficacy

Staff (5 subscales)Staff (5 subscales)• Work HabitsWork Habits• Aggressive Behavior with PeersAggressive Behavior with Peers• Prosocial Behavior with PeersProsocial Behavior with Peers• Task PersistenceTask Persistence• Social SkillsSocial Skills

TeacherTeacher (6 subscales)(6 subscales)• Same as Staff plus Mock Report CardSame as Staff plus Mock Report Card

FIELD TEST FINDINGS FIELD TEST FINDINGS (Fall 2010 – Spring 2011)(Fall 2010 – Spring 2011)

Diverse programs across the state participate in fall 2010 Diverse programs across the state participate in fall 2010 field test field test • 156156 Sites (elementary & middle school) Sites (elementary & middle school)

• 2323 Programs (10 rural, 8 urban, 5 suburban) Programs (10 rural, 8 urban, 5 suburban)

• 8 8 Regions representedRegions represented

Pre-Surveys Completed Pre-Surveys Completed (November 1 through December 17) (November 1 through December 17) • 3934 3934 elementary studentselementary students• 1847 1847 middle school studentsmiddle school students• 360 360 program staff completed program staff completed 4611 4611 student reportsstudent reports• 29 29 classroom teachersclassroom teachers completed completed 185 185 student reportsstudent reports

Initial feedback from Field Test sites positiveInitial feedback from Field Test sites positive• Instructions clearInstructions clear• Online administration easyOnline administration easy• Happy with the quality of technical support Happy with the quality of technical support

Field Test psychometrics consistent with Pilot results and Field Test psychometrics consistent with Pilot results and previous studiesprevious studies

FIELD TEST FINDINGS FIELD TEST FINDINGS Time to Complete Survey Time to Complete Survey

Elementary and Middle School Students Reported Time to Complete Survey

Pilot Field Test

Elementary Students N=1,175 N=3934

< 5 minutes 23% 30%

5-9 minutes 28% 27%

10-15 minutes 16% 10%

More than 15 minutes 2% 3%

Don’t Know 31% 30%

Middle School Students

N=481 N=1847

< 5 minutes 24% 37%

5-9 minutes 38% 33%

10-15 minutes 8% 5%

More than 15 minutes 3% 2%

Don’t Know 27% 23%

FIELD TEST FINDINGS FIELD TEST FINDINGS Time to Complete Survey Time to Complete Survey

Pilot Field Test

Teachers N = 70 N = 29

< 5 minutes 81% 50%5-9 minutes 18% 46%

10 minutes or more 1% 4%

Staff N = 105 N = 360

< 5 minutes 54% 61%

5-9 minutes 46% 31%

10 minutes or more — 4%

Classroom Teacher and Staff Reported Time to Complete Each Student Report

NEXT STEPSNEXT STEPS Can programs sign up now? ign up now?

• Yes, but they will not get a report of pre/post change scores, only Yes, but they will not get a report of pre/post change scores, only the results of the spring survey administration.the results of the spring survey administration.

Provide support to new participants Provide support to new participants enlisting in Field Test for spring 2011enlisting in Field Test for spring 2011

Contact current Field Test participants to Contact current Field Test participants to prepare for spring survey administration prepare for spring survey administration (April - May 2011)(April - May 2011)

Dissemination of findings in summer 2011 Dissemination of findings in summer 2011 in collaboration with California Afterschool in collaboration with California Afterschool Network Regional Leads and CDENetwork Regional Leads and CDE