sree annual conference march 6, 2010
DESCRIPTION
SREE Annual Conference March 6, 2010. Using RTCs to determine the impact of reading interventions on struggling readers Newark Public Schools Jennifer Hamilton, Senior Study Director [email protected] Matthew Carr, Analyst [email protected]. Overview of Presentation. - PowerPoint PPT PresentationTRANSCRIPT
SREE ANNUAL CONFERENCEMARCH 6, 2010
Using RTCs to determine the impact of reading interventions on
struggling readers
Newark Public Schools
Jennifer Hamilton, Senior Study Director
Matthew Carr, Analyst
Overview of Presentation Context – Striving Readers in Newark, NJ Fidelity of implementation
Adherence Exposure
Discussion For more information…
2
Context – Newark, NJ
35% children living in poverty (compared to 18% nationally) Largest school district in the state of NJ
A ‘district in need of improvement’ for last 4 years State took over the district in 1995 (limited control given back in
2008) Only ~ 50% of students in grades 6, 7, & 8 are proficient readers
3
Importance of Fidelity
Fidelity is the extent to which the intervention as implemented is faithful to the pre-stated
model.
Little black dress is in; little black box is out Internal validity - Helps to explain failure External validity - Helps to make treatment
more stable and replicable (treatment has to be well defined)
Helps ensure treatment is absent from control condition
4
Components of Fidelity - Theory of Change
Adherence
Exposure
5
Establishing Fidelity (Adherence)
4-steps: (1) identify, (2) measure, (3) score, (4) analyze
Step 1: Identify critical components Adaptation issue
Step 2: Measure Multiple sources of data, range of methodologies
Extant data (training receipt, class size, SRI, computer use)
Classroom observations Practical considerations - $$$$
Qualifications of data collection staff Number of points in time (cost)
6
Establishing Fidelity (Adherence)
Step 3: Score Assign sub-scores Number of sessions per week using instructional software
Combine to a single score Equal weighting
7
Fidelity
% of classrooms
High 18.2%
Adequate
36.4%
Low 18.2%
Very Low
27.3%
Newark - Single Adherence Score
Year 1 = 88% Year 2 = 82% Year 3 = 89%
8
Establishing Fidelity (Adherence)
Step 4: Analysis Descriptive
But profoundly unsatisfying, given all the effort and expense
Generally, should not be used as a mediating variable Fidelity usually related to error term as well as outcome
Error term contains unmeasured factors, such as teacher quality/charisma and student engagement
Non-experimental/exploratory Fidelity as a predictor (with lots of covariates) Correlational
9
Newark- Descriptive Adherence Data
10
Exposure
You are here
11
Exposure
Student Receipt of Intervention -- Components
Attrition Attendance No-Shows
12
Exposure - Attrition
WWC (2008) Benchmarks for attrition tolerance
13
Newark
19.6% overall
5.6% differential
Exposure - Attendance
Number of unexcused absences by analytic group
14
Group 1
Group 2
Group 3
Group 4
Group 5
Treatment
24.20 23.25 21.57 27.09 23.90
Control 24.15 23.30 21.40 27.12 23.80Group 1 = 1 year of potential exposure (6,7, 8 year 1; 6 year 2)Group 2 = 1 year of potential exposure - 6th graders only (years 1,2)Group 3 = 2 years of potential exposure – 7th graders only (year 2)Group 4 = 2 years of potential exposure – 8th graders only (year 2)Group 5 = 2 years of potential exposure – 7th + 8th graders (7,8 year 2)
No significant differences b/t Treatment and Control students
Exposure – No-Shows
Intention to Treat (ITT) vs. Treatment on the Treated (TOT) Removing T students who didn’t receive T would bias
the data But keeping them in underestimates effects Issue of real world implementation vs. ideal
implementation
Policymakers want to know TOT, Researchers need to report ITT
Solution – The Bloom Adjustment
15
The Bloom Adjustment
Adjusts the effects of an intervention upwards by the treatment group no-show rate
AllSubjectEffect = γ*NoShowEffect + (1-γ)TreatSubjectEffect Assuming the effect
per no-show is zero, then:
AS = γ * 0 + (1- γ)TS
AS = (1- γ)TS Therefore:
TS = AS / (1- γ)
16
17
Example: Striving Readers
Student sample divided into 5 analytic groups
Year 1 (06-07)
Year 2 (07-08)
6 7 8 6 7 8
(1) 1 year of exposure (n=1,772)
X X X X
(2) 1 year of exposure (n=904)
X X
(3) 2 years of exposure (n=444)
X
(4) 2 years of exposure (n=373)
X
(5) 2 years of exposure (n=817)
X X
Striving Readers Example
ITT effect sizes compared to Bloom Adjusted (year 2)
Subgroup Test DomainNo-Show
RateITT
Effect Size
Bloom Effect Size
Male (2 yrs 7th) Vocabulary 9% 0.34 0.37 Male (2 yrs 7+8) Vocabulary 8% 0.23 0.25 Hispanic (2 yrs 8th) Language arts 8% 0.47 0.51 Special Education (2 yrs 7+8)
Comprehension 7% 0.24 0.25
Special Education (2 yrs 7th)
Comprehension 5% 0.37 0.39
18
Review
Adherence = receipt of materials + accurate delivery 4 steps: identify, measure, score, analyze
Receipt Attrition Attendance No Shows – Bloom Adjustment
19
For more information…
Bloom, H. (1984). Accounting for No-Shows in experimental evaluation designs. Evaluation Review, 8, 225-246.
Durlak, J.A., & DuPre, E.P. (2008). Implementation matters: A review of research on the influence of implementation on program outcomes and the factors affecting implementation. American Journal of Community Psychology, 41, 327-350.
Hill, L.G., Maucione, K,. & Hood, B.K. (2007). A focused approach to assessing program fidelity. Prevention Science, 8, 25-34.
Mowbray, C. Holter, M. Teague, G., & Bybee, D. (2003). Fidelity Criteria: Development, Measurement, and Validation. American Journal of Evaluation, 24, 315-340.
What Works Clearinghouse. (2008). WWC Procedures and Standards Handbook. Available online at http://ies.gov/ncee/wwc/references
20
21
On the Web
Department of Elementary and Secondary Education Striving Readers webpage
http://www.ed.gov/programs/strivingreaders/index.html