Copyright © 2012

American Institutes

for Research

All rights reserved.

 CASEL/NoVo Collaborating Districts Initiative for Social and Emotional Learning:

Cross-District Evaluation Report

David Osher, Kimberly Kendziora, Lawrence B. Friedman, and the AIR CDI Evaluation Team

November 8, 2012

2

Key Questions: Overall CDI Evaluation

• What are the outcomes in districts? In schools?• How did the districts implement SEL over time?• What factors influence implementation quality?• To what extent and in what respects are

intended student-level outcomes realized?• How are realized student outcomes associated

with the school- and district-level implementation of the CDI?

3

Key Questions: Focus of this Presentation

• What is the progress of district activities and outcomes as summarized by rubric scores? 2011 and 2012 scores for Cohort 1 2012 scores for Cohort 2

• Student social and emotional competence at baseline

• Challenges and solutions• Next steps

4

Methods•Two cohorts of districts:

• Cohort 1 = Anchorage, Austin, and an R&D district, Cleveland• Cohort 2 = Chicago, Nashville, Oakland, Sacramento,

Washoe County

•District interviews 97 stakeholders across the 8 districts were interviewed

•Data examined Planning documents, communication materials, handouts,

district reports, consultant input Districtwide staff SEL surveys Student data, climate data, student social and emotional skills

ratings

•Triangulation of data District rubric scores and district reports

5

Cohort 1 Districts: Scores from 2011 and 2012

Vis

ion

Cen

tral

off

ice

expe

rtis

e

PD

pro

gram

s

Alig

n re

sour

ces

Com

mun

icat

ions

SE

L st

anda

rds

for

PK

–12

Evi

denc

e-ba

sed

prog

ram

s

Inte

grat

e S

EL

with

oth

er in

itiat

ives

Con

tinuo

us im

prov

emen

t

Pos

itive

clim

ate

Sta

keho

lder

com

mitm

ent 0

0.5

1

1.5

2

2.5

3

3.5

Rubric Scores for Cohort 1 Districts: 2011 (Planning Phase) and 2012 (Implementation Year

1)

2011

2012

6

Cohort 2 Districts: Planning Phase Rubric Scores

Nee

ds &

res

ourc

es

Vis

ion

Cen

tral

off

ice

expe

rtis

e

PD

pro

gram

s

Alig

n re

sour

ces

Com

mun

icat

ions

SE

L st

anda

rds

for

PK

–12

Evi

denc

e-ba

sed

prog

ram

s

Inte

grat

e S

EL

with

oth

er in

itiat

ives

Con

tinuo

us im

prov

emen

t

Pos

itive

clim

ate

Sta

keho

lder

com

mitm

ent

Rol

es &

res

pons

ibili

ties

0

0.5

1

1.5

2

2.5

3

2012 Rubric Scores: Cohort 2

7

Grade 7 Students’ Social and Emotional Competence: Baseline

Self-a

waren

ess

Self-m

anag

emen

t

Social

awar

enes

s

Relatio

nship

skil

ls

Resp

Decisi

on-m

aking

0%

20%

40%

60%

80%

100%

23%18% 22%

15% 17%

29%33%

33%

29%30%

28% 31% 28%38%

38%

21% 17% 18% 17% 14%

85th percentile or above

50th to 84th percentile

16th to 49th percentile

15th percentile or below

Caution: Data are combined across all districts and all survey types. Data do not represent any one district. Reliability for some scales was low.

8

Grade 10 Students’ Social and Emotional Competence: Baseline

Self-a

waren

ess

Self-m

anag

emen

t

Social

awar

enes

s

Relatio

nship

skil

ls

Resp

Decisi

on-m

aking

0%

20%

40%

60%

80%

100%

6%13%

8% 8% 9%

29%

40%

35%30% 26%

39%

31%

36% 48%45%

26%16%

21%15%

20%

85th percentile or above

50th to 84th percentile

16th to 49th percentile

15th percentile or below

Caution: Data are combined across all districts and all survey types. Data do not represent any one district. Reliability for some scales was low.

9

Findings about Student Social and Emotional Competence Across All Districts at Baseline

Results for seventh grade students• Females rate themselves more highly then

males for self-management, social awareness, and relationship skills

• For self-awareness and social awareness, American Indian and White students rated themselves the highest, followed by multi-racial, Native Hawaiian, Black, Hispanic, and Asian students, and finally Alaska Native students.

10

Findings about Student Social and Emotional Competence Across All Districts at Baseline

Results for tenth grade students• Females rate themselves more highly then

males for responsible decision making• American Indian students rated themselves

higher than all other racial groups (Native Hawaiian and Alaska Native students were not included because n=14 and n=5).

11

Sample Grade 7 Subgroup Results: Social Awareness

Total

Male

Female

Whit

e

Africa

n Am

erica

n

Hispan

icAsia

n

Amer

ican

India

n

Alaska

Nat

ive

Native

Haw

aiian

Mult

i-rac

ial0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

0.2150265038027190.2113821138211380.218823529411765

0.09169407894736850.1420798065296250.168535507985049

0.330453563714903

0.0892857142857142

0.636363636363637

0.3076923076923080.259567387687188

0.333026042867020.3771454381210480.287058823529412

0.315378289473685

0.4519347037484890.351342167855929

0.265658747300216

0.321428571428572

0.181818181818182

0.179487179487180.316139767054909

0.2756395482830140.258355916892502

0.293647058823529

0.343338815789474

0.271765417170496

0.3098878695208970.252699784017279

0.357142857142857

0.136363636363636

0.307692307692308

0.26955074875208

0.1763079050472460.1531165311653120.200470588235294

0.249588815789474

0.1342200725513910.1702344546381240.1511879049676030.232142857142857

0.0454545454545455

0.2051282051282050.154742096505824

85th percentile or above

50th to 84th percentile

16th to 49th percentile

15th percentile or below

Caution: Data are combined across all districts and all survey types. Data do not represent any one district. Reliability for some scales was low.

12

Sample Grade 10 Subgroup Results: Responsible Decision Making

Total

Male

Female

Whit

e

Africa

n Am

erica

n

Hispan

icAsia

n

Amer

ican

India

n

Mult

i-rac

ial0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

0.06099058534588620.06350550381033020.05863708399366090.07217847769028870.103833271306290.07301107754279960.06164383561643830.10383327130629

0.0616113744075829

0.2906262791649610.3048264182895850.277337559429477

0.3530183727034120.353181987346483

0.3444108761329310.287671232876712

0.3531819873464830.364928909952607

0.3888661481784690.3759525825571550.40095087163233

0.3293963254593180.3066617045031630.3625377643504530.452054794520548

0.3066617045031630.350710900473934

0.2595169873106840.255715495342930.2630744849445330.2454068241469820.2363230368440640.2200402819738170.1986301369863010.2363230368440640.222748815165877

85th percentile or above

50th to 84th percentile

16th to 49th percentile

15th percentile or below

Caution: Data are combined across all districts and all survey types. Data do not represent any one district. Reliability for some scales was low.

13

Strengths Across Districts

• Even in a climate of fiscal austerity, districts have still managed to support SEL.

• All districts have expressed a desire to go to scale with SEL

• Willingness to take on SEL standards seriously• Focus on professional development• Efforts at aligning foundational efforts and early and

intensive interventions• Insight into the relationship between SEL and deep

learning  and the whole child • Insight into the relationship between SEL and

disparities in discipline, attendance, and learning

14

Challenges: Multiple Initiatives

• In every district there was concern expressed about: the rapid pace of program turnover being overburdened with too many initiatives at

once “innovation fatigue”

15

Challenges: Understanding

• SEL is sometimes: seen as “soft” by some educators regarded as a distraction confused with PBIS, character education,

student support, and other initiatives

16

Challenge: Implementation

• School autonomy and variation present challenges for supporting high-quality SEL implementation

• Linking elementary-middle-high school SEL has been a challenge

17

Challenge: Turnover

• Who is the champion for the CDI? How to ensure sustainability when there is turnover in key staff?

18

Challenge: SEL Metrics

• Strategic use of SEL metrics can support high-quality SEL implementation

• Some districts do well with data use, but don’t have SEL-related metrics available

• Some districts have SEL metrics available, but have not systematically supported use of those data

19

CDI Evaluation Activities 2012–13

• Site visits in spring Interviews with key district stakeholders SEL lead / team phone interviews Document review: focus on evidence of CDI

implementation

• Districtwide staff SEL survey Central office: relevant staff Schools: Principals, teachers, student support staff

• Teacher ratings of Grade 3 students’ SEL• Grade 7 & 10 student self-report of SEL skills• No school case studies

20

Next Steps

• AIR evaluation team will contact districts to develop/revise data collection plans for 2012–13 by the end of December

21

David Osher, Ph.D.

Phone: 202-403-5373

Email: dosher@air.org

1000 Thomas Jefferson St., NW

Washington, DC 20007

Website: www.air.org