This article was downloaded by: [Florida State University]On: 13 November 2014, At: 06:34Publisher: RoutledgeInforma Ltd Registered in England and Wales Registered Number: 1072954Registered office: Mortimer House, 37-41 Mortimer Street, London W1T 3JH,UK

Journal of Education forStudents Placed at Risk(JESPAR)Publication details, including instructions forauthors and subscription information:http://www.tandfonline.com/loi/hjsp20

Educating Students Placed AtRisk: Evaluating the Impactof Success for All in UrbanSettingsMarco A. Munoz & Dena H. DossettPublished online: 16 Nov 2009.

To cite this article: Marco A. Munoz & Dena H. Dossett (2004) Educating StudentsPlaced At Risk: Evaluating the Impact of Success for All in Urban Settings, Journalof Education for Students Placed at Risk (JESPAR), 9:3, 261-277, DOI: 10.1207/s15327671espr0903_3

To link to this article: http://dx.doi.org/10.1207/s15327671espr0903_3

PLEASE SCROLL DOWN FOR ARTICLE

Taylor & Francis makes every effort to ensure the accuracy of all theinformation (the “Content”) contained in the publications on our platform.However, Taylor & Francis, our agents, and our licensors make norepresentations or warranties whatsoever as to the accuracy, completeness,or suitability for any purpose of the Content. Any opinions and viewsexpressed in this publication are the opinions and views of the authors, andare not the views of or endorsed by Taylor & Francis. The accuracy of theContent should not be relied upon and should be independently verified withprimary sources of information. Taylor and Francis shall not be liable for anylosses, actions, claims, proceedings, demands, costs, expenses, damages,and other liabilities whatsoever or howsoever caused arising directly orindirectly in connection with, in relation to or arising out of the use of theContent.

This article may be used for research, teaching, and private study purposes.Any substantial or systematic reproduction, redistribution, reselling, loan,sub-licensing, systematic supply, or distribution in any form to anyone isexpressly forbidden. Terms & Conditions of access and use can be found athttp://www.tandfonline.com/page/terms-and-conditions

Dow

nloa

ded

by [

Flor

ida

Stat

e U

nive

rsity

] at

06:

34 1

3 N

ovem

ber

2014

Educating Students Placed At Risk:Evaluating the Impact of Success for All

in Urban Settings

Marco A. MuñozDena H. Dossett

Jefferson County Public Schools

Schools across the nation are implementing reform models intended to enable allchildren to meet state-established academic standards. One of the nation’s compre-hensive school reform models is Success for All (SFA). This study assessed the im-pact of the program on elementary school students in standardized reading scores aswell as attendance and disciplinary measures when compared to control students.Program and control schools were further compared using teacher, student, and par-ent perception data on school climate and job satisfaction scales. Significant effectswere seen on some but not all measures, but the consistent direction and magnitude ofthe effects showed benefits for SFA students when compared to control students. Im-plications for educational policy and practice are discussed.

Over the last two decades, numerous national studies and reports have documentedboth the struggles and failings of public education. Educators, policymakers, and re-searchers alike concluded that a large number of schools, particularly in high-pov-erty urban centers, were ineffective at meeting the needs of diverse student popula-tions (Balfanz & Legters, 2001; Cooper, 1998). In an effort to assist schools inmaking curriculum changes, aid in instructional delivery, and strengthen the organi-zational structure of the schools, an abundance of schoolwide reform models haveemerged (Herman et al., 1999). If educators have learned anything about school re-form, it is that a piecemeal approach to changing poor classroom practice is a losingbattle.Acollectionof isolatedprogramsdoesnotadduptoschoolwide improvement(Northwest Regional Educational Laboratory, 2001).

JOURNAL OF EDUCATION FOR STUDENTS PLACED AT RISK, 9(3), 261–277Copyright © 2004, Lawrence Erlbaum Associates, Inc.

Requests for reprints should be sent to Marco A. Muñoz, Jefferson County Public Schools, Ac-countability, Research, and Planning Department, 3332 Newburg Road, Louisville, KY 40218.E-mail: mmunoz2@jefferson.k12.ky.us

Dow

nloa

ded

by [

Flor

ida

Stat

e U

nive

rsity

] at

06:

34 1

3 N

ovem

ber

2014

The comprehensive approach to schoolwide improvement is not new. A line ofinquiry referred to as effective school research has been at the forefront of the com-prehensive approach to school improvement. These studies have attempted toidentify the characteristics of schools that make them instructionally effective fordisadvantaged students (Brookover, Beady, Flook, Schweitzer, & Wisenbaker,1979; Clark, Lotto, & McCarthy, 1980; Edmonds, 1979; Weber, 1971; for a re-view, see Teddlie & Reynolds, 2000).

One of the nation’s comprehensive school reform models is Success For All(SFA). SFA is designed for elementary schools (PreK–6) to ensure that every childlearns to read in the early grade levels (Slavin et al., 1996). The primary goal ofSFA is to prevent reading problems and to intervene swiftly and intensively ifproblems do appear (Northwest Regional Educational Laboratory, 2001). Accord-ing to Madden and Slavin (1999), the first goal of reform should be to ensure thatevery child, regardless of home background, language, or learning style, achieveshis or her fullest potential.

In the SFA program’s organizational structure, students are grouped accordingto reading level for one 90-min reading period per day; for the rest of the day, theyare assigned to regular age-grouped grades. At 8-week intervals, teachers evaluatethe performance of students to (a) determine who requires reading tutors, (b) reas-sign reading groups, (c) suggest other adaptations in students’ programs, and (d)identify students who need other types of assistance (e.g., screening for vision andhearing problems).

The model also incorporates a Family Support Team (FST) that encourages pa-rental participation and involvement in the school. SFA schools have a program fa-cilitator who is responsible for assisting teachers with classroom management andthe implementation of the curriculum. Teachers receive 3 days of in-service train-ing at the beginning of the school year. Throughout the year, the program facilita-tor and technical support staff provide professional development on topics such asclassroom management, instructional pace, and cooperative learning.

REVIEW OF RESEARCH ASSOCIATED WITH SFA

Established in 1987 by Robert Slavin and his colleagues at Johns Hopkins Univer-sity, SFA has been implemented in about 15,000 schools in 48 states throughoutthe nation. It is considered one of the most successful and extensively researchedwhole-school change models (Borman & Hewes, 2002; Cooper, 1998; Cooper,Madden, & Slavin, 1997; Madden & Slavin, 1999). Developers’and third-party re-search work has been done in Memphis, TN (Ross, Sanders, & Wright, 1998); FortWayne, IN (Ross, Smith, & Casey, 1997); Flint, MI (Madden & Slavin, 1999); andBaltimore, MD (Slavin, Madden, Dolan, Wasik, Ross, & Smith, 1994).

Madden, Slavin, Karweit, Dolan, and Wasik (1993) investigated the longitudi-nal effects of SFA on inner-city elementary schools in Baltimore. Their purpose

262 MUÑOZ AND DOSSETT

Dow

nloa

ded

by [

Flor

ida

Stat

e U

nive

rsity

] at

06:

34 1

3 N

ovem

ber

2014

was to identify outcomes of student reading achievement and other outcomes in el-ementary schools serving large numbers of disadvantaged students. A total of fiveSFA schools were studied over a 3-year period with comparisons with matchedstudents in matched schools. Control schools were matched based on the percent-age of students receiving free lunch, their historical achievement level, and otherfactors. Within each matched school, students were individually matched on stan-dardized achievement test scores from the spring before SFA implementation. Thereading analyses found strong positive effects on administered reading measuresin most schools for students who had been in the program since the first grade.

Ross et al. (1997) examined the impact of SFA on school performance acrossthree grades on both individually administered and state-mandated standardizedreading tests. The study used program and matched-control students. Findings in-dicated more positive effects for SFA students, some decline in effects over time,and greater effects for minority than for nonminority students.

More recently, Borman and Hewes (2002) conducted an investigation of thelong-term outcomes of SFA. Sustained effects on student achievement were pro-posed and investigated in treatment (n = 1,310) and control (n = 1,730) student sec-ondary data from the Baltimore City Public School System. The treatment studentsincluded all students from the original SFA elementary schools; the control stu-dents were similar on demographic characteristics (i.e., gender, race, free or re-duced-price lunch, and age). Three dependent variables were of primary interest:(a) achievement, (b) grade-level progression (or retention), and (c) special educa-tion placements. Findings indicated that there were statistically significant differ-ences between SFA and control students. When controlling for kindergarten pre-test differences, SFA students had higher eighth grade Comprehensive Test ofBasic Skills (CTBS) reading and math scores, in comparison to control students.The findings showed statistically significant differences between SFA and controlstudents with SFA students spending fewer years enrolled in special education andexperiencing a lower number of retentions.

CURRENT STUDY

Kentucky is a state known for its innovative educational reform environment. In1990, the Kentucky General Assembly passed the Kentucky Educational ReformAct (KERA), which mandated a complete restructuring of the public elementaryand secondary school system in the areas of finance, governance, curriculum,and assessment (Pankratz & Petrosko, 2000). Kentucky’s structure includes es-sential elements of current accountability systems, namely: (a) rigorous contentstandards apply to all students; (b) student achievement is assessed; (c) profes-sional development is aligned with standards and assessment; (d) results are re-ported publicly; and (e) results lead to rewards, sanctions, and targeted assis-tance (Watts, Gaines, & Creech, 1998).

EDUCATING STUDENTS PLACED AT RISK 263

Dow

nloa

ded

by [

Flor

ida

Stat

e U

nive

rsity

] at

06:

34 1

3 N

ovem

ber

2014

The school district in Kentucky serving as the research site for this study hasmore than 93,000 students in Grades K–12 and approximately 150 school sites.The school district serves students from urban and suburban neighborhoods andthe majority of students qualify for the national free or reduced-price lunch pro-gram. The district has a student assignment plan based on managed choice, whichfacilitates the racial disaggregation of its schools by providing students with trans-portation from their home neighborhoods to other parts of the district.

This third-party investigation contributes to research on SFA by using amixed-method approach in assessing a large number of school-related variables.The study’s primary objective was to determine the impact of the program onschool, student, teacher, and parent variables after 3 years of implementation. Thefollowing overarching research questions guided the study: Does SFA impact stu-dents’ reading achievement? Does SFA result in improved student attendance?Does the model result in a decrease in discipline problems? Finally, what effectsdoes SFA have on teacher, student, and parent perceptions of school climate, edu-cational quality, and job satisfaction?

Because SFA is primarily a reading program implemented during school hours,the analysis of the impact of the program in reading scores was an essential compo-nent of the study. Supplementary variables such as attendance, suspensions, andschool climate were assessed to gain an understanding of the systemic nature ofchange that SFA creates when implemented in public schools. SFA is a programthat is comprehensive in nature, because it incorporates curriculum, assessment,individualized assistance, and family support. Given the multifaceted nature of theprogram, the framework of biosocial developmental contextualism (Ramey &Ramey, 1998) provided a rationale for assessing the supplementary variables. Thistheory hypothesizes that intensive, high-quality, ecologically pervasive interven-tions are likely to provide enduring cognitive benefits to students. In contrast, frag-mented and weak efforts in educational interventions are not likely to succeed. TheRamey and Ramey framework has proved valuable in understanding the SFA pro-gram in prior research (Borman & Hewes, 2002).

METHOD

Participants

The sample included three treatment elementary schools and three matched con-trol schools. The three schools became involved with SFA as part of their effort toincrease student achievement and had been implementing SFA for 3 years (fromschool year 1999–2000 to school year 2001–2002).

Table 1 illustrates the key characteristics by which the treatment and controlschools were matched, including the aggregated data for both treatment and con-

264 MUÑOZ AND DOSSETT

Dow

nloa

ded

by [

Flor

ida

Stat

e U

nive

rsity

] at

06:

34 1

3 N

ovem

ber

2014

trol schools and the district average at the baseline year (school year 1998–1999).The urban district that served as the research site educates a high percentage ofat-risk urban students with high poverty levels. Table 1 shows that the treatmentand control schools had a larger percentage of at-risk student populations than theschool district as a whole.

Table 2 summarizes the baseline data for the student-level samples. These sam-ples only include third graders at the treatment and control schools who also hadmembership in the schools for 3 years.

EDUCATING STUDENTS PLACED AT RISK 265

TABLE 1School-Level Matching Data for Treatment and Control Schools at

Baseline Year (1998–1999)

CTBSa

School NPoverty

(%)Mobility

(%)Attendance

(%)ECE(%)

SingleHousehold (%) N M

School A 718 86.0 17.89 92.8 8.32 63.5 140 30.1Control A 462 88.8 18.93 94.1 6.73 66.0 75 38.2School B 656 81.1 16.27 94.3 6.27 71.7 98 40.8Control B 486 83.2 10.98 94.3 7.79 60.2 83 43.1School C 406 75.3 15.90 93.3 7.09 66.8 68 40.1Control C 494 73.5 16.93 94.3 5.35 68.8 103 36.3Treatment schools 593 80.8 16.69 93.5 7.23 67.3 306 37.0Control schools 481 81.8 15.61 94.2 6.62 65.0 261 39.2District 536 56.0 10.37 95.0 6.63 49.4 7359 47.7

Note. Poverty was operationalized as participation in the free/reduced-price lunch program. ECE =Exceptional Child Education (students with disabilities); CTBS = Comprehensive Test of Basic Skills.

aCTBS is only administered to third graders in the state that served as the research site.

TABLE 2Baseline Data for the Individual Student-Level Analytical Sample (2000–2001)

Success for All Control

M SD M SD

Reading Diagnostic Pretest 39.52 16.76 35.49 15.64Female 0.55 0.5 0.55 0.5Minority 0.54 0.5 0.59 0.49Free/reduced-price lunch 0.86 0.35 0.83 0.37Single-parent home 0.73 0.44 0.72 0.45

Note. The Reading Diagnostic Pretest is the Stanford Diagnostic Reading Test (SDRT); it is ad-ministered to second graders at the beginning of the school year. Only students with complete demo-graphic and testing data were included in the analysis. An aggregated matching procedure was utilized.

an = 163. bn = 132.

Dow

nloa

ded

by [

Flor

ida

Stat

e U

nive

rsity

] at

06:

34 1

3 N

ovem

ber

2014

Instrumentation

All data from the 1998–1999 to 2001–2002 school years were abstracted fromcomputerized files provided by the school district that served as the research site.1998–1999 was used as the baseline year. Data from the state assessment system(i.e., CTBS) and from the school district (i.e., the Stanford Diagnostic ReadingTest [SDRT]) were utilized in this study.

The primary dependent variable used in this study was the CTBS Normal CurveEquivalent (NCE) scores in reading (Kramer, Conoley, & Murphy, 1992). NCEscores range from 1 to 99 with an average of 50 and a standard deviation of 21;these scores compare the students’ performance to a national norm group. TheCTBS is a standardized achievement test that was group-administered at the end ofthe school year. The CTBS includes reading, language arts, and mathematicssubtests. The Level 13 test is only administered to third graders and has 30 multi-ple-choice items in reading.

The independent variable was membership in treatment and control groups andthe SDRT served as a covariate. The SDRT (Kramer et al., 1992) is a standardizeddiagnostic test that was administered to second-grade students during the first fewweeks of the school year. Only students who had both the diagnostic and theachievement test participated in this study.

Because SFA is a reading curriculum based on research on effective practices(Madden & Slavin, 1999), only the reading indexes were abstracted for the pur-poses of this study. Noncognitive indicators such as attendance and suspensionswere also utilized as dependent variables. In an attempt to capture teacher, student,and parent perceptions, data were collected from the district-wide comprehensivesurveys. The surveys contained different subscales, such as school climate (23items) and educational quality (16 items), with Likert-type response scales rangingfrom 1 (strongly disagree) to 5 (strongly agree). Teachers had an additional scaleassociated with job satisfaction (7 items).

A total of 115 teachers, 667 students, and 867 parents completed the instruments.Thereliabilityanalyses,usingCronbach’salpha,wereconductedon thesubscalesofthe teacher (.93 on school climate, .94 on educational quality, and .95 on job satisfac-tion), student (.84 on school climate and .88 on educational quality), and parent (.94onschoolclimateand .97oneducationalquality) surveyson thebaselineyear.Thesereliability analyses yielded alpha coefficients well beyond the recommended mini-mumacceptable levelof .60(Nunnally,1994) inall thescalesof thevarioussurveys.

Design and Procedure

A mixed-method design involving qualitative and quantitative research proveduseful in understanding the impact of the SFA program in the large urban districtthat served as the research site for this study. Few studies have involved the anal-

266 MUÑOZ AND DOSSETT

Dow

nloa

ded

by [

Flor

ida

Stat

e U

nive

rsity

] at

06:

34 1

3 N

ovem

ber

2014

ysis of the SFA program from the perspective of multiple stakeholders (i.e.,teachers, parents, and students). Greene and McClintock (1985) argued thatmixed-method designs can be used to enhance and clarify the results from onemethod with the results from the other.

The mixed-method design employed in this study used a questionnaire aboutstudents’, teachers’, and parents’ perceptions of school climate and educationalquality combined with more quantitative measures (e.g., standardized test scores,attendance, suspensions). The analyses were conducted separately and integrationof their results occurred during the interpretation and reporting phases of the study.Furthermore, because results convergence was hypothesized, perception data werebrought in to support quantitative findings.

The impact evaluation design employed a matched pre- and posttest design witha control group (Cook & Campbell, 1979; Rossi, Freeman, & Lipsey, 1999). Thedata at the school level for both treatment and control schools were analyzed usingdescriptive statistics. School pair factors were not incorporated into the analysisdue to the limited number of students at the participating schools; this fact limitedthe statistical power needed for a more refined analytical design at the school level.

Due to the inability to assign randomly to treatment and control groups, atwo-level matching procedure was used to increase the internal validity of the study(Cook & Campbell, 1979). The first level of matching was at the school level and in-volved checking for similarity in terms of poverty (participation in the free or re-duced-price lunch program), race, single-parent households, gender, and SDRTscores. Only those third graders in these schools who had both SDRT and CTBSreading scores, in turn, served as the basis for the second level of the aggregatedmatching procedure (Rossi et al., 1999), which took place at the student level. Treat-mentandcontrol studentswerematchedonfourdemographicvariables,namelyfreeor reduced-price lunch, race, single-parent households, and gender. Given the cate-gorical nature of the variables, the matching procedure was checked usingchi-squares. No statistically significant differences were found on free or re-duced-price lunch (χ2 = .37, p > .05), race (χ2 = .77, p > .05), single-parent household(χ2 = .01, p > .05), and gender (χ2 = .11, p < .05). This confirmed that the aggregatedmatchingprocedurewassuccessful inavoiding theneed forusingstatistical controls(i.e., covariates) beyond previous differences on test scores (t = 2.12, p < .05).

The data were first examined to test the statistical assumptions (e.g., distribu-tional assumptions of the outcomes, homogeneity of variance, examination ofoutliers) of the desired analysis procedure (Tabachnik & Fidell, 1996). Becausethe data met the requirements for use with General Linear Modeling, the data atthe student level were analyzed using a comparison design that involved the useof the analysis of covariance (ANCOVA), with the treatment condition as the be-tween-subject factor and the pretest scores as the covariate. In addition, eachmean difference between SFA and control students was divided, or standardized,by the pooled posttest standard deviation for the outcome (Hedges & Olkin,

EDUCATING STUDENTS PLACED AT RISK 267

Dow

nloa

ded

by [

Flor

ida

Stat

e U

nive

rsity

] at

06:

34 1

3 N

ovem

ber

2014

1985). The resulting standardized differences, or effect sizes, provided summa-ries of the magnitude of each effect and were interpreted as the number of stan-dard deviation units separating SFA from control students on the outcomes.

Thecomprehensive survey instrumentswereadministeredyearlyusingScantronforms. The teacher, student, and parent surveys were distributed and completed bythe schools in the fall semester. Students were given the parents’ surveys to takehome with them and return to school. Staff forms remained confidential. The com-bined response rates for the baseline year (1998–1999) and after 3 treatment years(2001–2002) varied among teachers, students, and parents. The response rates forthe teacher (68.8%) and student (68.5%) surveys were considered “good” (Babbie,1989), whereas the parent response rate was less than optimal (41.9%). The percep-tion data were analyzed using inferential statistics. Dependent-sample t tests wereused to analyze the perception data related to teachers, students, and parents.

RESULTS

Research Question 1: Does SFA affect overall student achievement as measuredon standardized reading test scores?

Table 3 identifies the CTBS reading scores for both the treatment and controlschools from 1998–1999 to 2001–2002. The treatment schools doubled the gainswhen compared to the control schools in the CTBS reading test scores. The aggre-gated treatment school data show a continuous increase in CTBS reading scoresfrom the baseline year to the third year of SFA implementation (4.4 NCE gain); incontrast, the aggregated control schools’ data showed a gain of 2.3 points. Two out

268 MUÑOZ AND DOSSETT

TABLE 3School-Level CTBS Reading NCE Scores for Treatment and Control Schools

1998–1999 1999–2000 2000–2001 2001–2002PointGainsSchool N M N M N M N M

School A 140 30.1 126 38.1 121 34.6 115 35.1 5.0Control A 75 38.2 82 38.0 85 39.4 74 39.0 0.8School B 98 40.8 109 38.3 117 47.0 106 51.3 10.5Control B 83 43.1 69 40.4 86 44.2 67 43.4 0.3School C 68 40.1 65 43.0 76 39.9 52 37.9 –2.2Control C 103 36.3 96 36.9 92 39.4 94 42.1 5.8Aggregated Treatment 306 37.0 300 39.8 314 40.5 273 41.4 4.4Aggregated Control 261 39.2 247 38.4 263 41.0 235 41.5 2.3

Note. Gains = completed third year of implementation (2000–2001) minus baseline year(1998–1999). Aggregated results are means of the school-level means. CTBS = Comprehensive Test ofBasic Skills; NCE = Normal Curve Equivalent.

Dow

nloa

ded

by [

Flor

ida

Stat

e U

nive

rsity

] at

06:

34 1

3 N

ovem

ber

2014

of three treatment schools showed an increase on the CTBS scores during the firstyear of implementation of the model (1999–2000). Only one treatment schoolshowed gains in the second year (2000–2001). Two out of three treatment schoolsshowed gains in the third year (2001–2002).

The ANCOVA test was used to analyze the student level sample (N = 295) andindicated that, after adjustment by the covariate, the effect of the program re-mained significant, F(1, 294) = 4.71, p < .05, ES = .11. SFA students (M = 46.4, SD= 1.2) had higher adjusted mean scores than the control group (M = 43.3, SD = 0.9)on CTBS reading scores.

Research Question 2: Does SFA result in improved school student attendance?

Table 4 identifies the school-level attendance for the treatment and controlschools. At the school level, the aggregated treatment and control school data showsimilar trends in attendance. Overall, both treatment and control schools made gainsin the 3 years of SFA implementation since the baseline year (gains of 1.2 and .7, re-spectively);however,SFAschoolsnearlydoubled thegainsof thecontrol schools.

Research Question 3: Does the model result in a decrease in discipline problems?

From the baseline year (1998–1999) to the third year of SFA implementation(2001–2002), the aggregated treatment schools notably decreased the number ofout-of-school suspensions when compared to the control schools. Despite the de-cline, the treatment schools had a higher number of out-of-school suspensions. Amajor increase in out-of-school suspensions was observed during the first year ofSFA implementation, particularly in one of the schools. The out-of-school suspen-

EDUCATING STUDENTS PLACED AT RISK 269

TABLE 4School-Level Attendance Percentages for Treatment and Control Schools

1998–1999 1999–2000 2000–2001 2001–2002PointGainSchool N M N M N M N M

School A 718 92.8 711 93.1 602 93.0 593 93.9 1.1Control A 462 94.1 403 93.6 460 94.4 413 94.4 0.3School B 656 94.3 686 93.3 656 95.1 664 95.6 1.3Control B 486 94.3 406 93.3 480 94.9 478 95.4 1.1School C 406 93.3 402 93.5 405 93.8 365 94.7 1.4Control C 494 94.8 488 94.8 491 94.8 480 95.4 0.6Aggregated treatment 1780 93.5 1799 93.3 1663 94.0 1622 94.7 1.2Aggregated control 1442 94.4 1297 93.9 1431 94.7 1371 95.1 0.7District 43345 92.7 42743 93.1 46607 93.4 46327 94.1 1.4

Note. School attendance rates = [(membership to school – absences)/membership to school] ×100. Gains = completed third year of implementation (2001–2002) minus baseline year (1998–1999).Aggregated results are means of the school-level means.

Dow

nloa

ded

by [

Flor

ida

Stat

e U

nive

rsity

] at

06:

34 1

3 N

ovem

ber

2014

sion incidences remained higher for the program schools in the second year of SFA.The third year of SFA showed a remarkable decrease in the number of out-of-schoolsuspensions at the treatment schools. Both treatment and control schools had ap-proximately a 50% reduction in out-of-school suspensions from the baseline to thethirdyear.Table5displaysdisciplinedata forboth treatmentandcontrol schools.

Research Question 4: What effects does SFA have on teacher, student, and parentperceptions as measured by the comprehensive survey?

Teachers in the SFA schools showed larger improvements in their perceptionsof educational quality and job satisfaction than teachers in the control schools (seeTable 6). Teachers in the treatment schools gave higher ratings in the areas of edu-cational quality, t(53) = 2.3, p < .05, and job satisfaction, t(53) = 2.1, p < .05, after 3years of SFA implementation when compared to baseline measures. For instance,job satisfaction showed a statistically significant increase in the treatment schools,t(53) = 2.1, p < .05, but not in the control schools, t(53) = .78, p >.05. Two of thecontrol schools showed decreases in teachers’ perceptions of school climate andeducational quality over time.

Students in the treatment schools demonstrated higher ratings of school cli-mate, t(309) = 1.98, p = .05, and educational quality, t(309) = 1.97, p = .05, afterSFA implementation than at the baseline year. Although students in the controlschools gave higher ratings during the 2001–2002 school year than did students inthe treatment schools, their ratings did not reach statistically significant differ-ences in school climate, t(231) = .77, p > .05, and educational quality, t(231) = .43,p > .05. In general, students in the SFA schools showed larger improvements whencompared to their baseline year (see Table 7).

270 MUÑOZ AND DOSSETT

TABLE 5Summary of School-Level Out-of-School Suspension Incidences for

Treatment and Control Schools

1998–1999 1999–2000 2000–2001 2001–2002

GainsSchool N Sum N Sum N Sum N Sum

School A 718 27 711 58 602 27 593 15 +12Control A 462 6 403 27 460 10 413 5 +1School B 656 6 686 1 656 3 664 4 +2Control B 486 0 406 1 480 2 478 3 –3School C 406 16 402 8 405 22 365 7 +9Control C 494 16 488 5 491 6 480 3 +13Treatment schools 1780 49 1799 67 1663 52 1622 26 +23Control schools 1442 22 1297 33 1431 18 1371 11 +11

Note. Sum = total number of out-of-school suspensions per school. Gains = completed third yearof implementation (2001–2002) minus baseline year (1998–1999). Aggregated results are means of theschool-level means.

Dow

nloa

ded

by [

Flor

ida

Stat

e U

nive

rsity

] at

06:

34 1

3 N

ovem

ber

2014

Parents in the SFA schools showed greater improvements in their perceptions ofschool climate than control schools. Parents whose children attended the treatmentschools demonstrated statistically significant higher ratings of school climate,t(363) = 7.54, p < .05, and educational quality, t(363) = 6.30, p < .05, after SFA im-plementation than at the baseline year (see Table 8).

EDUCATING STUDENTS PLACED AT RISK 271

TABLE 6Teacher Perceptions on School Climate, Educational Quality, and Job

Satisfaction for Treatment and Control Schools

School Climate Educational Quality Job Satisfaction

1998–1999 2001–2002 1998–1999 2001–2002 1998–1999 2001–2002

School N M SD M SD M SD M SD M SD M SD

School A 26 4.3 –0.5 4.4 –0.5 3.6 –1.2 4.4 –0.6 3.9 –1.4 4.6 –0.5Control A 4 4.5 –0.7 3.5 –0.7 4 –1.4 3.5 –0.7 5 0 4.5 –0.7School B 33 4.1 –0.7 4.3 –0.6 4.2 –0.8 4.3 –0.7 4.5 –0.6 4.4 –0.7Control B 12 4 –0.7 3.6 –0.9 3.9 –0.8 3.8 –0.7 4.4 –0.5 4.4 –1.1School C 12 3.8 –0.9 4.1 –0.7 3.6 –1 4 –0.8 4 –0.8 4.6 –0.7Control C 21 4 –0.8 4.2 –0.7 3.9 –0.9 4 –0.7 4.3 –0.6 4.5 –0.5Aggregated

treatment71 4.1 –0.7 4.3 –0.6 3.9 –1 4.3 –0.7 4.1 –1 4.5 –0.6

Aggregatedcontrol

37 4 –0.8 4 –0.8 3.9 –0.9 3.9 –0.7 4.4 –0.6 4.5 –0.7

Note. Aggregated results are means of the school-level means.

TABLE 7Student Perceptions on School Climate and Educational Quality Measures

for Treatment and Control Schools

School Climate Educational Quality

1998–1999 2001–2002 1998–1999 2001–2002

School N M SD M SD M SD M SD

School A 146 3.9 –0.8 4.1 –0.8 4.3 –0.8 4.4 –0.8Control A 98 4.1 –0.8 4.2 –0.5 4.3 –0.9 4.5 –0.5School B 141 4.4 –0.6 4.3 –0.7 4.6 –0.6 4.6 –0.6Control B 94 4.3 –0.8 4.2 –0.7 4.5 –0.7 4.5 –0.5School C 62 3.8 –0.8 4.3 –0.7 4.1 –0.9 4.4 –0.9Control C 126 4.4 –0.7 4.3 –0.7 4.6 –0.6 4.6 –0.6Aggregated for treatment group 349 4.1 –0.8 4.2 –0.7 4.3 –0.8 4.5 –0.8Aggregated for control group 318 4.3 –0.7 4.3 –0.7 4.5 –0.7 4.6 –0.6

Note. Aggregated results are means of the school-level means. Only fourth and fifth graders com-pleted the survey.

Dow

nloa

ded

by [

Flor

ida

Stat

e U

nive

rsity

] at

06:

34 1

3 N

ovem

ber

2014

In summary, the perceptions of all three key stakeholder groups—teachers, stu-dents, and parents—showed improvements in the areas of school climate and edu-cational quality from the baseline year to after the conclusion of the third year oftreatment. In general, the gains or improvements were greater for the treatmentschools than those of the control schools, especially for teachers.

DISCUSSION

In an effort to assist schools in making instructional and organizational changes, anabundance of schoolwide reform models have emerged. With more than a decadeof research documenting its impact on public education, SFA is considered a toolfor urban school reform. The goal of SFA is to restructure elementary schools toensure that every child learns to read in the early grades. The idea is to preventreading problems from appearing in the first place and to intervene speedily and in-tensively when problems do appear.

Previous third-party evaluations (Ross, Nunnery, & Smith, 1996; Ross et al.,1998) of SFA schools in districts across the nation have shown that the program in-creased student reading performance and other school-related measures. SomeSFA research has been criticized in the past. For instance, Pogrow (1999) arguedthat sources of bias include (a) the use of tests that the SFA curriculum may begeared to and that control schools do not align with, (b) the fact that the vendor se-lects the test, (c) school sampling bias, (d) student sampling criteria, and (e) thefact that SFA relies primarily on individually administered tests, whereas studentsare tested in the “real world” with group-administered tests. This study addresses

272 MUÑOZ AND DOSSETT

TABLE 8Parent Perceptions on School Climate and Educational Quality Measures

for Treatment and Control Schools

School Climate Educational Quality

1998–1999 2001–2002 1998–1999 2001–2002

School N M SD M SD M SD M SD

School A 207 4 –0.8 4.4 –0.7 4.1 –0.8 4.4 –0.7Control A 166 4 –0.6 4.4 –0.7 3.9 –0.6 4.4 –0.6School B 141 3.9 –0.7 4.5 –0.6 4 –0.7 4.6 –0.6Control B 121 4.2 –0.7 4.4 –0.6 4.2 –0.7 4.3 –0.7School C 104 4.1 –0.7 4.4 –0.7 4.2 –0.7 4.4 –0.7Control C 91 4.3 –0.7 4.5 –0.8 4.2 –0.7 4.5 –0.7Aggregated for treatment group 452 4 –0.8 4.4 –0.7 4.1 –0.8 4.4 –0.7Aggregated for control group 378 4.2 –0.7 4.4 –0.7 4.1 –0.7 4.4 –0.7

Note. Aggregated results are means of the school-level means.

Dow

nloa

ded

by [

Flor

ida

Stat

e U

nive

rsity

] at

06:

34 1

3 N

ovem

ber

2014

some of these sources of bias by using valid and reliable group-administered tests,namely the CTBS (i.e., achievement test) and SDRT (i.e., diagnostic test).

This evaluation of the SFA program was conducted at both the school and thestudent level. The school-level evaluation showed that the model has a positive ef-fect on CTBS scores and attendance, but that this effect varied across schools. Thetreatment schools that have properly implemented SFA have continuously in-creased their CTBS scores in comparison to the control schools. In terms of atten-dance, gains are higher for the treatment schools when compared to the controlschools. In terms of suspensions, a notable decrease was observed in the treatmentschools when compared to the control schools. At the student level, the resultsshowed a reading effect in the SFA students when compared to the control stu-dents. The standardized differences indicated that SFA produced an effect size of.11. The magnitude of the SFA reading effect size exceeded 1/10 of one standarddeviation. These results suggested that the cognitive effects of the program, espe-cially in its target subject matter of reading, attained both statistical and practicalsignificance. It can be hypothesized that the connection between SFA and CTBSlies in the basic skills that are tested in CTBS and the skills taught in the SFA cur-riculum. Teachers, students, and parents at SFA schools showed larger improve-ments in their perceptions of school climate, educational quality, and job satisfac-tion when compared to those of the control schools.

These findings agree with the recent research findings of other third-partymeta-evaluators. Herman et al. (1999) conducted a study that evaluated the re-search base underlying the most widely used schoolwide programs. Herman et al.concluded that SFA is one of three programs with strong research supporting its ef-fectiveness. More recently, a meta-analysis conducted by Borman, Hewes, Over-man, and Brown (2003) found that SFA is one of the three models meeting thehighest standards of evidence on improving student test scores across varying con-texts and study designs.

The framework of biosocial developmental contextualism (Ramey & Ramey,1998) provides an avenue for understanding the SFA program benefits (Borman &Hewes, 2002). The framework highlights issues such as (a) developmental timing,(b) program intensity, (c) direct provision of learning experiences, (d) programbreadth and flexibility, (e) individual differences in program benefits, and (f) envi-ronmental maintenance of development.

The overall positive outcomes of SFA might be associated with multiple fac-tors. SFA is a highly scripted program supported by on-site facilitators who pro-vide ongoing professional development. SFA emphasizes attention to the criticalfirst grade of elementary school (Wasik & Slavin, 1993): Certified teacher tutors(not teacher aides) work intensively with early identified academically at-risk stu-dents and provide them with direct learning experiences. The program is compre-hensive in nature and includes continuous assessment, individualized assistance,and family support.

EDUCATING STUDENTS PLACED AT RISK 273

Dow

nloa

ded

by [

Flor

ida

Stat

e U

nive

rsity

] at

06:

34 1

3 N

ovem

ber

2014

Based on site visits and conversations with staff associated with the treatmentschools, it can be speculated that SFA produced these types of school outcomes inthe district being studied because of its quality of implementation, strong externaltechnical assistance, and district-level support. The entire faculty and staff of theSFA-implementing schools share a sense of commitment to continuously improvetheir implementation of the SFA model. The good quality of implementation in thedistrict that served as a research site has been associated with the large degree ofteacher buy-in and involvement with the comprehensive school reform change(Sarason, 1990, 1996). The service providers have made available technical assis-tance that involves (a) regular visits to each participating school and (b) annual re-ports of implementation fidelity. The district provides multiple opportunities forprofessional development; the schools fully utilized the professional developmentdays to learn about the research-based approaches to teaching and learning.

Principals of the SFA schools in the district under consideration protect the unin-terrupted 90-min literacy block and monitor the implementation of SFA at the class-room level. Every student in the SFA schools receives 90 min of reading instructiondaily. The facilitators provide follow-up support to address specific classroom-levelissues associated with program implementation fidelity. Teachers at each of the par-ticipating schools voted to implement the program, and that has given them high lev-els of empowerment (Fullan, 1991, 1993). Change has not been imposed on theteachers of the treatment schools through a top-down mandate (Bailey, 2000).

The researchers observed that student engagement in reading was high.Teacher support is strong in the treatment schools because teachers are seeingpositive academic testing results. Teachers are in the process of internalizing theconcept of using research-based strategies to improve their instruction. Further-more, teachers in the treatment schools are acquiring valuable skills for teachingreading while improving collaboration between teachers and administrators(Datnow & Castellano, 2000a).

Educational innovations are infrequently implemented in the classroom as ex-actly as developers intend them to be (Elmore & Sykes, 1992). During site visitsconducted at the treatment schools, veteran teachers expressed their concernsabout the scripted nature of the program. Teachers complained about instructionalstrategies that did not fit with their preferred way of teaching. This finding pointsto the importance of understanding program implementation fidelity and promot-ing long-term teacher empowerment for change (Datnow & Castellano, 2000b).

Parent and community involvement, however, have proven to be a challengingarea for SFA. Some parents have not accepted the idea of reading with their childfor 20 min a day. Volunteers from the community are coming to the schools andreading with those children whose parents are not reading to them at home. TheFST has been helpful in this component of the program.

Although clearly encouraging, the findings of this mixed-method study are lim-ited. This study was not conducted as a randomized trial, which limits our ability toreach firm causal conclusions. Due to the strong matching procedure and statistical

274 MUÑOZ AND DOSSETT

Dow

nloa

ded

by [

Flor

ida

Stat

e U

nive

rsity

] at

06:

34 1

3 N

ovem

ber

2014

controls, the study still had strong internal validity and differential group changemay be attributed more to the interventions than to potentially confounding inter-vening factors (Cook & Campbell, 1979). Incomplete diagnostic testing data andmobility at the student level resulted in shrinkage of the sample size for both thetreatment and the control group. As a consequence, it weakened the representative-ness of the sample (i.e., threat to external validity). Due to the low response rate,the parent perception findings should not be generalized to other settings.

Future third-party researchers need to address the impact of the program onhigher order skills assessed using open-ended questions. This is important in lightof the value placed on the open-ended responses in the Commonwealth Account-ability Testing System (CATS) used in Kentucky. KERA, one of the nation’s mostcomprehensive and longest running statewide school reform programs, placeshigh value on the CATS test, which is used to assess specifically the educationalstandards delineated in the Kentucky Academic Expectations and the Core Con-tent for Assessment (Pankratz & Petrosko, 2000).

Finally, given the dynamic school characteristics and internal factors affectingmodel implementation, further analysis using qualitative methods might clarify el-ements related to implementation quality. Additional qualitative studies mighthelp in the process of understanding important elements such as ownership amongveteran and novice teachers, levels of adaptation at the classroom level, and thetrade-off between the scripted nature of the program and the need for more auton-omy and creativity. These are important elements in the discussion of any exter-nally developed comprehensive reform model.

REFERENCES

Babbie, E. (1989). The practice of social research. Belmont, CA: Wadsworth.Bailey, B. (2000). The impact of mandated change on teachers. In A. Hargreaves & N. Bascia (Eds.), The

sharpedgeofchange:Teaching, leading,andtherealitiesof reform (pp.112–128).London:Falmer.Balfanz, R., & Legters, N. (2001, January). How many central city high schools have a severe drop-

out problem, where are they located, and who attends them? Paper presented at a forum convenedby The Civil Rights Project at Harvard University’s Graduate School of Education and Achieve,Inc., Cambridge, MA.

Borman, G. D., & Hewes, G. M. (2002). The long-term effects and cost-effectiveness of Success forAll. Educational Evaluation and Policy Analysis, 24, 243–266.

Borman, G. D., Hewes, G. M., Overman, L. T., & Brown, S. (2003). Comprehensive school reform andachievement: A meta-analysis. Review of Educational Research, 73, 125–230.

Brookover, W. B., Beady, C., Flook, P., Schweitzer, J., & Wisenbaker, J. (1979). School social systemsand student achievement: Schools can make a difference. New York: Praeger.

Clark, D. L., Lotto, L. S., & McCarthy, M. M. (1980). Factors associated with success in urban elemen-tary schools. Phi Delta Kappan, 61, 469–470.

Cook, T. D., & Campbell, D. T. (1979). Quasi-experimentation: Design and analysis issues for fieldsettings. Chicago: Rand McNally.

Cooper, R. (1998). Socio-cultural and within-school factors that affect the quality of implementation ofschoolwide programs. Baltimore: Center for Research on the Education of Students Placed At Risk.

EDUCATING STUDENTS PLACED AT RISK 275

Dow

nloa

ded

by [

Flor

ida

Stat

e U

nive

rsity

] at

06:

34 1

3 N

ovem

ber

2014

Cooper, R., Madden, N. A., & Slavin, R. E. (1997). Success for All: Exploring the technical, normative,political, and socio-cultural dimensions of scaling-up. Baltimore: Center for Research on the Educa-tion of Students Placed At Risk.

Datnow, A., & Castellano, M. (2000a). An “inside” look at Success for All: A qualitative study of imple-mentation and teaching and learning. Baltimore: Center for Research on the Education of StudentsPlaced At Risk.

Datnow, A., & Castellano, M. (2000b). Teachers’responses to Success for All: How beliefs, experiences,and adaptations shape implementation. American Educational Research Journal, 37, 775–799.

Edmonds, R. (1979). Effective schools for the urban poor. Educational Leadership, 37, 15–24.Elmore, R., & Sykes, G. (1992). Curriculum policy. In P. Jackson (Ed.), Handbook of research on cur-

riculum (pp. 185–215). New York: McMillan.Fullan, M. (1991). The new meaning of educational change. New York: Teachers College Press.Fullan, M. (1993). Change forces: Probing the depths of educational reform. London: Falmer.Greene, J. G., & McClintock, C. (1985). Triangulation in evaluation: Design and analysis issues. Evalu-

ation Research, 9, 523–547.Hedges, L. V., & Olkin, I. (1985). Statistical methods for meta-analysis. New York: Academic.Herman, R., Aladjem, D., McMahon, P., Masem, E., Mulligan, I., O’Malley, A. S., Quinones, S.,

Reeve, A., & Woodruff, D. (1999). An educators’ guide to schoolwide reform. Washington, DC:American Institute for Research.

Kramer, J. J., Conoley, J. C., & Murphy, L. L. (1992). The eleventh mental measurements yearbook.Lincoln: University of Nebraska.

Madden, N. A., & Slavin, R. E. (1999). Success for All/Roots & Wings: Summary of research onachievement outcomes. Baltimore: Center for Research on the Education of Students Placed At Risk.

Madden, N. A., Slavin, R. E., Karweit, N. L., Dolan, L. J., & Wasik, B. A. (1993). Success for All: Lon-gitudinal effects of a restructuring program for inner-city elementary schools. American EducationalResearch Journal, 30, 123–148.

Northwest Regional Educational Laboratory. (2001). Catalog of school reform models. Retrieved No-vember 13, 2003, from http://www.nwrel.org/scpd/ http://www.nwrel.org/scpd/catalog/index.shtml

Nunnally, J. C. (1994). Psychometric theory. New York: McGraw-Hill.Pankratz, R. S., & Petrosko, J. M. (2000). All children can learn: Lessons from the Kentucky reform ex-

perience. San Francisco: Jossey-Bass.Pogrow, S. (1999). Consistent large gains and high levels of achievement are the best measures of pro-

gram quality: Pogrow responds to Slavin. Educational Researcher, 28(8), 24–31.Ramey, C. T., & Ramey, S. L. (1998). Early intervention and early experience. American Psychologist,

53, 109–120.Ross, S. M., Nunnery, J., & Smith, L. J. (1996). Evaluation of Title I reading programs. Memphis, TN:

University of Memphis Center for Research in Educational Policy.Ross, S. M., Sanders, W. L., & Wright, S. P. (1998). An analysis of Tennessee Value Added Assessment

(TVAAS) performance outcomes of Roots and Wings schools from 1995–1997. Memphis, TN: Uni-versity of Memphis.

Ross, S. M., Smith, L. J., & Casey, J. P. (1997). Preventing early school failure: Impacts of Success forAll on standardized test outcomes, minority group performance, and school effectiveness. Journal ofEducation for Students Placed at Risk, 2, 29–53.

Rossi, P. H., Freeman, H. E., & Lipsey, M. W. (1999). Evaluation: A systematic approach (6th ed.).Newbury Park, CA: Sage.

Sarason, S. (1990). The predictable failure of educational reform. San Francisco: Jossey-Bass.Sarason, S. (1996). Revisiting the culture of the school and the problem of change. New York: Teachers

College Press.Slavin, R. E., Madden, N. A., Dolan, L. J., Wasik, B. A., Ross, S., & Smith, L. (1994). “Whenever and

wherever we choose…”: The replication of Success for All. Phi Delta Kappan, 75, 639–647.

276 MUÑOZ AND DOSSETT

Dow

nloa

ded

by [

Flor

ida

Stat

e U

nive

rsity

] at

06:

34 1

3 N

ovem

ber

2014

Slavin, R. E., Madden, N. A., Dolan, L. J., Wasik, B. A., Ross, S., Smith, L., & Dianda, M. (1996). Suc-cess for All: A summary of research. Journal of Education for Students Placed At Risk, 1, 41–76.

Tabachnik,B.G.,&Fidell,L.S. (1996).Usingmultivariatestatistics (3rded.).NewYork:HarperCollins.Teddlie, C., & Reynolds, D. (2000). International handbook of school effectiveness research: An inter-

national survey of research on school effectiveness. London: Falmer.Wasik, B. A., & Slavin, R. E. (1993). Preventing early reading failure with one-on-one tutoring: A re-

view of five programs. Reading Research Quarterly, 28, 178–200.Watts, J. A., Gaines, G. F., & Creech, J. D. (1998). Getting results: A fresh look at school accountability.

Atlanta, GA: Southern Regional Education Board.Weber, G. (1971). Inner-city children can be taught to read: Four successful schools. Washington, DC:

Council for Basic Education.

EDUCATING STUDENTS PLACED AT RISK 277

Dow

nloa

ded

by [

Flor

ida

Stat

e U

nive

rsity

] at

06:

34 1

3 N

ovem

ber

2014