rate of improvement calculation and decision making caitlin s. flinn, eds, ncsp andrew e. mccrea,...
TRANSCRIPT
Rate of ImprovementRate of Improvement Calculation and Decision MakingCalculation and Decision Making
Caitlin S. Flinn, EdS, NCSPCaitlin S. Flinn, EdS, NCSP
Andrew E. McCrea, MS, NCSPAndrew E. McCrea, MS, NCSP
Why we’re here…
While there exists a wealth of convincing research supporting the implementation of a response-to-intervention (RtI) framework, there are many questions yet to be empirically answered.
Within multi-tiered model of assessment and instruction/intervention, how do we know whether a student is learning?
Measuring Learning
Class tests Quizzes Assignment/homework completion and accuracy Ask students questions in class Grades/report cards State/local assessments Universal screening, benchmark assessments Progress monitoring
With Progress Monitoring Data…
How do we know if a student is learning?Look at the data points
Where are they on the graph? Are the data points getting closer to the goal or
benchmark?
Is there a way to measure growth?Make an aimline toward goalLook to see where data points are compared to
aimlineCalculate Rate of Improvement (RoI)
Today’s Objectives
Explain what RoI is, why it is important, Explain what RoI is, why it is important, and how to compute it.and how to compute it.
Establish that Simple Linear Regression Establish that Simple Linear Regression should be the standardized procedure for should be the standardized procedure for calculating RoI. calculating RoI.
Discuss how to use RoI within a problem Discuss how to use RoI within a problem solving/school improvement model.solving/school improvement model.
RoI Definition
Rate of Improvement can be described Rate of Improvement can be described algebraically as the slope of a algebraically as the slope of a lineline
Slope is defined as: the vertical change Slope is defined as: the vertical change over the horizontal change on a Cartesian over the horizontal change on a Cartesian plane. (x-axis and y-axis graph)plane. (x-axis and y-axis graph)Also called: Rise over runAlso called: Rise over runFormula: m = (yFormula: m = (y2 2 - y- y11) / (x) / (x2 2 - x- x11))Describes the steepness of a line (Gall & Gall, Describes the steepness of a line (Gall & Gall,
2007)2007)
RoI Definition
Finding a student’s RoI is determining the Finding a student’s RoI is determining the student’s learningstudent’s learningCreating a line that fits the data points, a Creating a line that fits the data points, a
trendlinetrendlineTo find that line, we use: To find that line, we use:
Linear regressionLinear regressionOrdinary Least SquaresOrdinary Least Squares
How does Rate of Improvement Fit into the
Larger Context?
School Improvement/Comprehensive School Reform
Response to Intervention
Dual Discrepancy: Level & Growth
Rate of Improvement
School Improvement/ Comprehensive School Reform
Grade level content expectations (ELA, Grade level content expectations (ELA, math, science, social studies, etc.).math, science, social studies, etc.).
Work toward these expectations through Work toward these expectations through classroom instruction.classroom instruction.
Understand impact of instruction through Understand impact of instruction through assessmentassessment..
Assessment
Formative Assessments/High Stakes Formative Assessments/High Stakes TestsTestsDoes student have command of content Does student have command of content
expectation (standard)?expectation (standard)?Universal Screening using CBMUniversal Screening using CBM
Does student have basic skills appropriate for Does student have basic skills appropriate for age/grade?age/grade?
Assessment
QQ: For students who are not proficient on : For students who are not proficient on grade level content standards, do they grade level content standards, do they have the basic reading/writing/math skills have the basic reading/writing/math skills necessary?necessary?
AA: Look at Universal Screening; if above : Look at Universal Screening; if above criteria, intervention geared toward content criteria, intervention geared toward content standard, if below criteria, intervention standard, if below criteria, intervention geared toward basic skill. geared toward basic skill.
Progress Monitoring
Frequent measurement of knowledge to Frequent measurement of knowledge to inform our understanding of the impact of inform our understanding of the impact of instruction/intervention.instruction/intervention.
Measures of basic skills (CBM) have Measures of basic skills (CBM) have demonstrated reliability & validity demonstrated reliability & validity (see table (see table
at www.rti4success.org).at www.rti4success.org).
Classroom Instruction (Content Expectations)
Measure Impact (Test)
Proficient! Non Proficient
Content Need? Basic Skill Need?
InterventionProgress Monitor With CBM
Rate of Improvement
InterventionProgress Monitor
If CBM is Appropriate Measure
Use Diagnostic Test to Differentiate
So… Rate of Improvement (RoI) is how we understand Rate of Improvement (RoI) is how we understand
student growth (learning).student growth (learning). RoI is reliable and valid (psychometrically RoI is reliable and valid (psychometrically
speaking) for use with CBM data.speaking) for use with CBM data. RoI is best used when we have CBM data, most RoI is best used when we have CBM data, most
often when dealing with basic skills in often when dealing with basic skills in reading/writing/math.reading/writing/math.
RoI can be applied to other data (like behavior) RoI can be applied to other data (like behavior) with confidence too!with confidence too!
RoI is not yet tested on typical Tier I formative RoI is not yet tested on typical Tier I formative classroom data.classroom data.
RoI is usually applied to…
Tier One students in the early grades at Tier One students in the early grades at risk for academic failure (low green kids).risk for academic failure (low green kids).
Tier Two & Three Intervention Groups.Tier Two & Three Intervention Groups.Special Education Students (and IEP Special Education Students (and IEP
goals)goals)Students with Behavior PlansStudents with Behavior Plans
RoI Foundations
Deno, 1985Curriculum-based measurement
General outcome measuresTechnically adequateShortStandardizedRepeatableSensitive to change
RoI Foundations
Fuchs & Fuchs, 1998Hallmark components of Response to
InterventionOngoing formative assessmentIdentifying non-responding studentsTreatment fidelity of instruction
Dual discrepancy modelOne standard deviation from typically
performing peers in level and rate
RoI Foundations
Ardoin & Christ, 2008Slope for benchmarks (3x per year)More growth from fall to winter than
winter to springMight be helpful to use RoI for fall to
winterAnd a separate RoI for winter to spring
RoI Foundations
Fuchs, Fuchs, Walz, & Germann, Fuchs, Fuchs, Walz, & Germann, 19931993Typical weekly growth rates in oral Typical weekly growth rates in oral
reading fluency and digits correctreading fluency and digits correctNeeded growth to remediate skillsNeeded growth to remediate skills
Students who had 1.5 to 2.0 times the Students who had 1.5 to 2.0 times the slope of typically performing peers were slope of typically performing peers were able to close the achievement gap in a able to close the achievement gap in a reasonable amount of timereasonable amount of time
RoI Foundations
Deno, Fuchs, Marston, & Shin, 2001Slope of frequently non-responsive children
approximated slope of children already identified as having a specific learning disability
How many data points?
10 data points are a minimum requirement for a reliable trendline (Gall & Gall, 2007)Is that reasonable and realistic?
How does that affect the frequency of administering progress monitoring probes?
How does that affect our ability to make instructional decisions for students?
How can we show RoI? Speeches that included visuals, especially in Speeches that included visuals, especially in
color, improved recall of information (Vogel, color, improved recall of information (Vogel, Dickson, & Lehman, 1990)Dickson, & Lehman, 1990)
““Seeing is believing.” Seeing is believing.” Useful for communicating large amounts of Useful for communicating large amounts of
information quicklyinformation quickly ““A picture is worth a thousand words.”A picture is worth a thousand words.” Transcends language barriers (Karwowski, Transcends language barriers (Karwowski,
2006)2006) Responsibility for accurate graphical Responsibility for accurate graphical
representations of datarepresentations of data
Skills for Which We Compute RoI
Reading Oral Reading Fluency Word Use Fluency Reading Comprehension
MAZE Retell
Early Literacy Skills Initial Sound Letter Naming Letter Sound Phoneme Segmentation Nonsense Word
Spelling Written Expression Behavior
Math Math
Computation Math Facts Early Numeracy
Oral Counting Missing Number Number
Identification Quantity
Discrimination
Guidelines?
Visual inspection of slopeVisual inspection of slope
Multiple interpretationsMultiple interpretations
Instructional servicesInstructional services
Need for explicit guidelinesNeed for explicit guidelines
Ongoing Research
RoI for instructional decisions is not a perfect RoI for instructional decisions is not a perfect processprocess
Research is currently addressing sources of Research is currently addressing sources of error:error:Christ, 2006: standard error of measurement for Christ, 2006: standard error of measurement for
slopeslopeArdoin & Christ, 2009: passage difficulty and Ardoin & Christ, 2009: passage difficulty and
variabilityvariabilityJenkin, Graff, & Miglioretti, 2009: frequency of Jenkin, Graff, & Miglioretti, 2009: frequency of
progress monitoringprogress monitoring
Future Considerations
Questions yet to be empirically answeredQuestions yet to be empirically answeredWhat parameters of RoI indicate a lack of RtI?What parameters of RoI indicate a lack of RtI?How does standard error of measurement How does standard error of measurement
play into using RoI for instructional decision play into using RoI for instructional decision making?making?
How does RoI vary between standard How does RoI vary between standard protocol interventions?protocol interventions?
How does this apply to non-English speaking How does this apply to non-English speaking populations?populations?
How is RoI Calculated? Which way is best?
Multiple Methods for Calculating Growth
Visual Inspection ApproachesVisual Inspection Approaches ““Eye Ball” Approach Eye Ball” Approach Split Middle ApproachSplit Middle ApproachTukey Method Tukey Method
Quantitative Approaches Quantitative Approaches Last point minus First point ApproachLast point minus First point ApproachSplit Middle & Tukey “plus”Split Middle & Tukey “plus”Linear Regression ApproachLinear Regression Approach
The Visual Inspection Approaches
8
10
7
17
14
11
19
14
0
2
4
6
8
10
12
14
16
18
20
1 2 3 4 5 6 7 8
Eye Ball Approach
Split Middle Approach
Drawing “through the two points obtained Drawing “through the two points obtained from the median data values and the from the median data values and the median days when the data are divided median days when the data are divided into two sections” into two sections”
(Shinn, Good, & Stein, 1989).(Shinn, Good, & Stein, 1989).
Split Middle
8
10
7
17
14
11
19
14
0
2
4
6
8
10
12
14
16
18
20
1 2 3 4 5 6 7 8
X(9)
X(14)
X (9)
Tukey Method
Divide scores into 3 equal groupsDivide scores into 3 equal groupsDivide groups with vertical linesDivide groups with vertical lines In 1In 1stst and 3 and 3rdrd groups, find median data groups, find median data
point and median week and mark with an point and median week and mark with an “X”“X”
Draw line between two “Xs”Draw line between two “Xs”
(Fuchs, et. al., 2005. Summer Institute Student progress monitoring for math. (Fuchs, et. al., 2005. Summer Institute Student progress monitoring for math. http://www.studentprogress.org/library/training.asp) http://www.studentprogress.org/library/training.asp)
Tukey Method
8
10
7
17
14
11
19
14
0
2
4
6
8
10
12
14
16
18
20
1 2 3 4 5 6 7 8
X(8)
X(14)
The Quantitative Approaches
Last minus First
Iris Center: last probe score minus first Iris Center: last probe score minus first probe score over last administration period probe score over last administration period minus first administration period.minus first administration period.
Y2-Y1/X2-X1= RoIY2-Y1/X2-X1= RoI
http://iris.peabody.vanderbilt.edu/resources.htmlhttp://iris.peabody.vanderbilt.edu/resources.html
8
10
7
17
14
11
19
14
0
2
4
6
8
10
12
14
16
18
20
1 2 3 4 5 6 7 8
(14-8)/(8-0)=0.75
Last minus First
Split Middle “Plus”
8
10
7
17
14
11
19
14
0
2
4
6
8
10
12
14
16
18
20
1 2 3 4 5 6 7 8
X(9)
X(14)
(14-9)/8=0.63
Tukey Method “Plus””
8
10
7
17
14
11
19
14
0
2
4
6
8
10
12
14
16
18
20
1 2 3 4 5 6 7 8
X(8)
X(14)
(14-8)/8=0.75
8
10
7
17
14
11
19
14
y = 1.1429x + 7.3571
0
2
4
6
8
10
12
14
16
18
20
1 2 3 4 5 6 7 8
Linear Regression
RoI Consistency?
Any Method of Any Method of Visual InspectionVisual Inspection
??????
Last minus FirstLast minus First 0.750.75
Split Middle “Plus”Split Middle “Plus” 0.630.63
Tukey “Plus”Tukey “Plus” 0.750.75
Linear Linear RegressionRegression
1.101.10
RoI Consistency?
If we are not all using the same model to If we are not all using the same model to compute RoI, we continue to have the same compute RoI, we continue to have the same problems as past models, where under one problems as past models, where under one approach a student meets SLD criteria, but approach a student meets SLD criteria, but under a different approach, the student does under a different approach, the student does not. not.
Hypothetically, if the RoI cut-off was 0.65 or Hypothetically, if the RoI cut-off was 0.65 or 0.95, different approaches would come to 0.95, different approaches would come to different conclusions on the same student. different conclusions on the same student.
RoI Consistency?
Last minus First (Iris Center) and Linear Last minus First (Iris Center) and Linear Regression (Shinn, etc.) only quantitative Regression (Shinn, etc.) only quantitative methods discussed in CBM literature.methods discussed in CBM literature.
Study of 37 at risk 2Study of 37 at risk 2ndnd graders: graders:
Difference in RoI b/w LmF & LR Difference in RoI b/w LmF & LR MethodsMethods
Whole YearWhole Year 0.26 WCPM0.26 WCPM
FallFall 0.31 WCPM0.31 WCPM
Spring Spring 0.24 WCPM0.24 WCPMMcCrea (2010) Unpublished dataMcCrea (2010) Unpublished data
Technical Adequacy
Without a consensus on how to compute Without a consensus on how to compute RoI, we risk falling short of having RoI, we risk falling short of having technical adequacy within our model. technical adequacy within our model.
So, Which RoI Method is Best?
Literature shows that Linear Regression is Best Practice
Student’s daily test scores…were entered into a Student’s daily test scores…were entered into a computer program…The data analysis program computer program…The data analysis program generated generated slopes of improvementslopes of improvement for each level for each level using an using an Ordinary-Least SquaresOrdinary-Least Squares procedure procedure (Hayes, 1973) and the line of best fit. (Hayes, 1973) and the line of best fit.
This This procedure has been demonstrated to procedure has been demonstrated to represent CBM achievement data validly within represent CBM achievement data validly within individual treatment phasesindividual treatment phases (Marston, 1988; (Marston, 1988; Shinn, Good, & Stein, in press; Stein, 1987).Shinn, Good, & Stein, in press; Stein, 1987).
Shinn, Gleason, & Tindal, 1989Shinn, Gleason, & Tindal, 1989
Growth (RoI) Research using Linear Regression
Christ, T. J. (2006). Short-term estimates of growth using Christ, T. J. (2006). Short-term estimates of growth using curriculum based measurement of oral reading fluency: curriculum based measurement of oral reading fluency: Estimating standard error of the slope to construct confidence Estimating standard error of the slope to construct confidence intervals. intervals. School Psychology ReviewSchool Psychology Review, 35, 128-133., 35, 128-133.
Deno, S. L., Fuchs, L. S., Marston, D., & Shin, J. (2001). Using Deno, S. L., Fuchs, L. S., Marston, D., & Shin, J. (2001). Using curriculum based measurement to establish growth standards curriculum based measurement to establish growth standards for students with learning disabilities. for students with learning disabilities. School Psychology School Psychology ReviewReview, 30, 507-524., 30, 507-524.
Good, R. H. (1990). Forecasting accuracy of slope estimates for Good, R. H. (1990). Forecasting accuracy of slope estimates for reading curriculum based measurement: Empirical evidence. reading curriculum based measurement: Empirical evidence. Behavioral AssessmentBehavioral Assessment, 12, 179-193., 12, 179-193.
Fuchs, L. S., Fuchs, D., Hamlett, C. L., Walz, L. & Germann, G. Fuchs, L. S., Fuchs, D., Hamlett, C. L., Walz, L. & Germann, G. (1993). Formative evaluation of academic progress: How much (1993). Formative evaluation of academic progress: How much growth can we expect? growth can we expect? School Psychology ReviewSchool Psychology Review, 22, 27-48., 22, 27-48.
Growth (RoI) Researchusing Linear Regression
Jenkins, J. R., Graff, J. J., & Miglioretti, D.L. (2009). Jenkins, J. R., Graff, J. J., & Miglioretti, D.L. (2009). Estimating reading growth using intermittent CBM Estimating reading growth using intermittent CBM progress monitoring. progress monitoring. Exceptional ChildrenExceptional Children, 75, 151-163., 75, 151-163.
Shinn, M. R., Gleason, M. M., & Tindal, G. (1989). Shinn, M. R., Gleason, M. M., & Tindal, G. (1989). Varying the difficulty of testing materials: Implications for Varying the difficulty of testing materials: Implications for curriculum-based measurement. curriculum-based measurement. The Journal of Special The Journal of Special EducationEducation, 23, 223-233., 23, 223-233.
Shinn, M. R., Good, R. H., & Stein, S. (1989). Shinn, M. R., Good, R. H., & Stein, S. (1989). Summarizing trend in student achievement: A Summarizing trend in student achievement: A comparison of methods. comparison of methods. School Psychology ReviewSchool Psychology Review, 18, , 18, 356-370.356-370.
So, Why Are There So Many Other RoI Models?
Ease of application Ease of application Focus on Yes/No to goal acquisition, not Focus on Yes/No to goal acquisition, not
degree of growthdegree of growthHow many of us want to calculate OLS How many of us want to calculate OLS
Linear Regression formulas (or even Linear Regression formulas (or even remember how)?remember how)?
Pros and Cons of Each Approach
ProsPros ConsCons
Eye BallEye Ball Easy Easy
UnderstandableUnderstandable
SubjectiveSubjective
Split Split Middle & Middle & TukeyTukey
No software neededNo software needed
Compare to Compare to Aim/Goal lineAim/Goal line
Yes/No to goal Yes/No to goal acquisitionacquisition
No statistic No statistic provided, no provided, no idea of the idea of the degree of degree of growth growth
Pros and Cons of Each Approach
ProsPros ConsCons
Last minus Last minus FirstFirst
Provides a growth Provides a growth statisticstatistic
Easy to compute Easy to compute
Does not consider Does not consider all data points, all data points, only twoonly two
Split Middle & Split Middle & Tukey “Plus”Tukey “Plus”
Considers all data Considers all data points.points.
Easy to computeEasy to compute
No support for No support for “plus” part of “plus” part of methodology methodology
Linear Linear RegressionRegression
All data points All data points
Best Practice Best Practice
Calculating the Calculating the statistic statistic
An Easy and Applicable Solution
Get Out Your Laptops!
Open Microsoft ExcelOpen Microsoft Excel
I loveROI
Graphing RoIFor Individual Students
Programming Microsoft Excel to Programming Microsoft Excel to Graph Rate of Improvement: Graph Rate of Improvement:
Fall to WinterFall to Winter
Setting Up Your Spreadsheet
In cell A1, type In cell A1, type 3rd Grade ORF3rd Grade ORF In cell A2, type In cell A2, type First SemesterFirst Semester In cell A3, type In cell A3, type School WeekSchool Week In cell A4, type In cell A4, type BenchmarkBenchmark In cell A5, type the Student’s Name In cell A5, type the Student’s Name
(Swiper Example)(Swiper Example)
Labeling School Weeks
Starting with cell B3, type numbers Starting with cell B3, type numbers 11 through through 1818 going across row 3 going across row 3 (horizontal).(horizontal).
Numbers 1 through 18 represent the Numbers 1 through 18 represent the number of the school week.number of the school week.
You will end with week 18 in cell S3.You will end with week 18 in cell S3.
Labeling Dates
NoteNote: You may choose to enter the date of : You may choose to enter the date of that school week across row 2 to easily that school week across row 2 to easily identify the school week.identify the school week.
Entering Benchmarks(3rd Grade ORF))
In cell B4, type In cell B4, type 7777. . This is your fall This is your fall benchmark.benchmark.
In cell S4, type In cell S4, type 9292. . This is your winter This is your winter benchmark.benchmark.
Entering Student Data (Sample)
Enter the following Enter the following numbers, going numbers, going across row 5, under across row 5, under corresponding week corresponding week numbers.numbers.
Week 1 – 41Week 1 – 41 Week 8 – 62Week 8 – 62 Week 9 – 63Week 9 – 63 Week 10 – 75Week 10 – 75 Week 11 – 64Week 11 – 64
Week 12 – 80Week 12 – 80 Week 13 – 83Week 13 – 83 Week 14 – 83Week 14 – 83 Week 15 – 56Week 15 – 56 Week 17 – 104Week 17 – 104 Week 18 – 74Week 18 – 74
*CAUTION*
If a student was not assessed during a If a student was not assessed during a certain week, leave that cell blankcertain week, leave that cell blank
Do not enter a score of Zero (0) it will be Do not enter a score of Zero (0) it will be calculated into the trendline and calculated into the trendline and interpreted as the student having read interpreted as the student having read zero words correct per minute during that zero words correct per minute during that week. week.
Graphing the Data
Highlight cells A4 and A5 through S4 and Highlight cells A4 and A5 through S4 and S5S5
Follow Excel 2003 or Excel 2007 Follow Excel 2003 or Excel 2007 directions from heredirections from here
Graphing the Data
Excel 2003Excel 2003 Across the top of your Across the top of your
worksheet, click on worksheet, click on “Insert”“Insert”
In that drop-down In that drop-down menu, click on “Chart”menu, click on “Chart”
Excel 2007Excel 2007 Click Click InsertInsert Find the icon for Find the icon for LineLine Click the arrow below Click the arrow below
LineLine
Graphing the Data
Excel 2003Excel 2003 A Chart Wizard A Chart Wizard
window will appearwindow will appear
Excel 2007Excel 2007 6 graphics appear6 graphics appear
Graphing the Data
Excel 2003Excel 2003 Choose “Line”Choose “Line” Choose “Line with Choose “Line with
markers…”markers…”
Excel 2007Excel 2007 Choose “Line with Choose “Line with
markers”markers”
Graphing the Data
Excel 2003Excel 2003 ““Data Range” tabData Range” tab ““Columns”Columns”
Excel 2007Excel 2007 Your graph appearsYour graph appears
Graphing the Data
Excel 2003Excel 2003 ““Chart Title”Chart Title” ““School Week” X AxisSchool Week” X Axis ““WPM’ Y AxisWPM’ Y Axis
Excel 2007Excel 2007 Change your labels by Change your labels by
right clicking on the right clicking on the graphgraph
Graphing the Data
Excel 2003Excel 2003 Choose where you Choose where you
want your graphwant your graph
Excel 2007Excel 2007 Your graph was Your graph was
automatically put into automatically put into your data spreadsheetyour data spreadsheet
Graphing the Trendline
Excel 2003Excel 2003Right click on any of the Right click on any of the studentstudent data points data points
Excel 2007Excel 2007
Graphing the Trendline
Excel 2003Excel 2003Choose “Linear”Choose “Linear”
Excel 2007Excel 2007
Graphing the Trendline
Excel 2003Excel 2003Choose “Custom” and check box next to Choose “Custom” and check box next to
“Display equation on chart”“Display equation on chart”
Excel 2007Excel 2007
Graphing the Trendline
Clicking on the equation highlights a box Clicking on the equation highlights a box around itaround it
Clicking on the box allows you to move it Clicking on the box allows you to move it to a place where you can see it betterto a place where you can see it better
Graphing the Trendline
You can repeat the same procedure to You can repeat the same procedure to have a trendline for the benchmark data have a trendline for the benchmark data pointspoints
Suggestion: label the trendline Suggestion: label the trendline Expected Expected ROIROI
Move this equation under the firstMove this equation under the first
Individual Student Graph:Fall to Winter
y = 2.5138x + 42.113
y = 0.8824x + 76.118
0
20
40
60
80
100
120
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18
Benchmark
Student: Sw iper
RoI
Linear (Benchmark)
Individual Student Graph
The equation indicates the slope, or rate of The equation indicates the slope, or rate of improvement. improvement.
The number, or coefficient,The number, or coefficient, beforebefore "x" is "x" is the average improvement, which in this the average improvement, which in this case is the average number of words per case is the average number of words per minute per week gained by the student. minute per week gained by the student.
Individual Student Graph
The rate of improvement, or trendline, is The rate of improvement, or trendline, is calculated using a linear regression, a calculated using a linear regression, a simple equation of least squares. simple equation of least squares.
To add additional progress To add additional progress monitoring/benchmark scores once you’ve monitoring/benchmark scores once you’ve already created a graph, enter additional already created a graph, enter additional scores in Row 5 in the corresponding scores in Row 5 in the corresponding school week.school week.
Individual Student Graph
The slope can change depending on The slope can change depending on which week (where) you put the which week (where) you put the benchmark scores on your chart. benchmark scores on your chart.
Enter benchmark scores based on when Enter benchmark scores based on when your school administers their benchmark your school administers their benchmark assessments for the most accurate assessments for the most accurate depiction of expected student progress.depiction of expected student progress.
Programming ExcelFirst Semester
Calculating Needed RoI
Calculating Benchmark RoI
Calculating Student’s Actual RoI
Quick Definitions
Needed RoIThe rate of improvement needed to “catch” up
to the next benchmark.Benchmark RoI
The rate of improvement of typically performing peers according to the norms
Student’s Actual RoIBased on the available data points, this is the
student’s actual rate of improvement per week
Calculating Needed RoI In cell T3, type Needed RoI Click on cell T5 In the fx line (at top of sheet) type this formula
=((S4-B5)/18) Then hit enter Your result should read: 2.83333... This formula simply subtracts the student’s
actual beginning of year (BOY) benchmark from the expected middle of year (MOY) benchmark, then dividing by 18 for the first 18 weeks (1st semester).
Calculating Benchmark RoI
In cell U3, type Benchmark RoIClick on cell U4 In the fx line (at top of sheet) type this
formula =SLOPE(B4:S4,B3:S3)Then hit enterYour result should read: 0.8825...This formula considers 18 weeks of
benchmark data and provides an average growth or change per week.
Calculating Student Actual RoI
Click on cell U5 In the fx line (at top of sheet) type this
formula =SLOPE(B5:S5,B3:S3)Then hit enterYour result should read: 2.5137...This formula considers 18 weeks of
student data and provides an average growth or change per week.
Graphing RoIFor Individual Students
Programming Microsoft Excel to Graph Rate of Improvement:
Winter to Spring
Setting Up Your Spreadsheet
In cell A1, type 3rd Grade ORF In cell A2, type Second Semester In cell A3, type School Week In cell A4, type Benchmark In cell A5, type the Student’s Name
(Swiper Example)
Labeling School Weeks
Starting with cell B3, type numbers 1 through 18 going across row 3 (horizontal).
Numbers 1 through 18 represent the number of the school week.
You will end with week 18 in cell S3.
Labeling Dates
Note: You may choose to enter the date of that school week across row 2 to easily identify the school week.
Entering Benchmarks(3rd Grade ORF)
In cell B4, type 92. This is your fall benchmark.
In cell S4, type 110. This is your winter benchmark.
Entering Student Data (Sample)
Enter the following numbers, going across row 5, under corresponding week numbers.
Week 1 – 74 Week 3 – 85 Week 4 – 89 Week 5 – 69 Week 6 – 85
Week 7 – 96 Week 8 – 90 Week 9 – 84 Week 10 – 106 Week 11 – 94 Week 15 – 100
*CAUTION*
If a student was not assessed during a certain week, what do you put in that cell?
Why?
Graphing the Data
Highlight cells A4 and A5 through S4 and S5
Follow Excel 2003 or Excel 2007 directions from here
Graphing the Data
Excel 2003 Across the top of your
worksheet, click on “Insert”
In that drop-down menu, click on “Chart”
Excel 2007 Click Insert Find the icon for Line Click the arrow below
Line
Graphing the Data
Excel 2003 A Chart Wizard
window will appear
Excel 2007 6 graphics appear
Graphing the Data
Excel 2003 Choose “Line” Choose “Line with
markers…”
Excel 2007 Choose “Line with
markers”
Graphing the Data
Excel 2003 “Data Range” tab “Columns”
Excel 2007 Your graph appears
Graphing the Data
Excel 2003 “Chart Title” “School Week” X Axis “WPM’ Y Axis
Excel 2007 Change your labels by
right clicking on the graph
Graphing the Data
Excel 2003 Choose where you
want your graph
Excel 2007 Your graph was
automatically put into your data spreadsheet
Graphing the Trendline
Excel 2003Right click on any of the student data points
Excel 2007
Graphing the Trendline
Excel 2003Choose “Linear”
Excel 2007
Graphing the Trendline
Excel 2003Choose “Custom” and check box next to
“Display equation on chart”
Excel 2007
Graphing the Trendline
Clicking on the equation highlights a box around it
Clicking on the box allows you to move it to a place where you can see it better
Graphing the Trendline
You can repeat the same procedure to have a trendline for the benchmark data points
Suggestion: label the trendline Expected ROI
Move this equation under the first
Individual Student Graphy = 1.8872x + 74.81
y = 1.0588x + 90.941
0
20
40
60
80
100
120
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18
Benchmark
Student: Sw iper
Rate of Improvement
Expected RoI
Challenge!
What was the first equation?What is the slope of that equation?
What was the second equation?What is the slope of that equation?
Describe the achievement gap at the end of the school year.
Programming ExcelSecond Semester
Calculating Needed RoI
Calculating Benchmark RoI
Calculating Student’s Actual RoI
Calculating Needed RoI In cell T3, type Needed RoI Click on cell T5 In the fx line (at top of sheet) type this formula
=((S4-B5)/18) Then hit enter Your result is _____ ? This formula simply subtracts the student’s
actual middle of year (MOY) benchmark from the expected end of year (EOY) benchmark, then dividing by 18 for the first 18 weeks (1st semester).
Calculating Benchmark RoI
In cell U3, type Benchmark RoIClick on cell U4 In the fx line (at top of sheet) type this
formula =SLOPE(B4:S4,B3:S3)Then hit enterYour result should read: ____?This formula considers 18 weeks of
benchmark data and provides an average growth or change per week.
Calculating Student Actual RoI
Click on cell U5 In the fx line (at top of sheet) type this
formula =SLOPE(B5:S5,B3:S3)Then hit enterYour result should read: 1.89This formula considers 18 weeks of
student data and provides an average growth or change per week.
Assuming Linear Growth…Assuming Linear Growth…
……Finding Curve-linear GrowthFinding Curve-linear Growth
Why Graph only 18 Weeks at a Time?
Non-Educational Example of Curve-linear Growth
Weight Loss Chart
200
197.5
193
189.5
186
184182.5
181179.5
178
165
170
175
180
185
190
195
200
205
1 2 3 4 5 6 7 8 9 10
Weeks
Wei
ght
10 Week RoI = -2.5First 5 Weeks RoI = -3.6Second 5 Weeks RoI = -1.5
Academic Example of Curvilinear Growth
0
10
20
30
40
50
60
70
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36
Weeks
WC
PM
BOY to MOY = 1.60
MOY to EOY = 1.19
BOY to EOY = 1.35
McCrea, 2010
Looked at Rate of Improvement in small Looked at Rate of Improvement in small 22ndnd grade sample grade sample
Found differences in RoI when computed Found differences in RoI when computed for fall and spring:for fall and spring:
Ave RoI for fall:Ave RoI for fall: 1.47 WCPM1.47 WCPMAve RoI for spring:Ave RoI for spring: 1.21 WCPM1.21 WCPM
Ardoin & Christ, 2008
Slope for benchmarks (3x per year)Slope for benchmarks (3x per year)More growth from fall to winter than winter More growth from fall to winter than winter
to springto spring
Christ, Yeo, & Silberglitt, in press
Growth across benchmarks (3X per year)Growth across benchmarks (3X per year)More growth from fall to winter than winter More growth from fall to winter than winter
to springto springDisaggregated special education Disaggregated special education
populationpopulation
Graney, Missall, & Martinez, 2009
Growth across benchmarks (3X per year)Growth across benchmarks (3X per year)More growth from winter to spring than fall More growth from winter to spring than fall
to winter with R-CBM.to winter with R-CBM.
Fien, Park, Smith, & Baker, 2010
Investigated relationship b/w NWF gains Investigated relationship b/w NWF gains and ORF/Comprehensionand ORF/Comprehension
Found greater NWF gains in fall than in Found greater NWF gains in fall than in spring.spring.
DIBELS (6th) ORF Change in Criteria
Fall to Fall to WinterWinter
Winter to Winter to SpringSpring
22ndnd 2424 2222
33rdrd 1515 1818
44thth 1313 1313
55thth 1111 99
66thth 1111 55
AIMSweb Norms
Based on 50Based on 50thth PercentilePercentile
Fall to WinterFall to Winter Winter to Winter to SpringSpring
11stst 1818 3131
22ndnd 2525 1717
33rdrd 2222 1515
44thth 1616 1313
55thth 1717 1515
66thth 1313 1212
Speculation as to why Differences in RoI within the Year
Relax instruction after high stakes testing in Relax instruction after high stakes testing in March/April; March/April; a PSSA effecta PSSA effect. .
Depressed BOY benchmark scores due to Depressed BOY benchmark scores due to summer break; summer break; a rebound effecta rebound effect (Clemens). (Clemens).
Instructional variablesInstructional variables could explain differences could explain differences in Graney (2009) and Ardoin (2008) & Christ (in in Graney (2009) and Ardoin (2008) & Christ (in press) results (press) results (Silberglitt).Silberglitt).
VariabilityVariability within progress monitoring probes within progress monitoring probes ((Ardoin & Christ, 2008) Ardoin & Christ, 2008) (Lent).(Lent).
ROI as a Decision ToolROI as a Decision Tool
within a Problem-Solving Modelwithin a Problem-Solving Model
Steps
1.1. Gather the dataGather the data
2.2. Ground the data & set goalsGround the data & set goals
3.3. Interpret the dataInterpret the data
4.4. Figure out how to fit Best Practice into Figure out how to fit Best Practice into Public EducationPublic Education
Step 1: Gather Data
Universal Screening Universal Screening
Progress MonitoringProgress Monitoring
Common Screenings in PA
DIBELSDIBELSAIMSwebAIMSwebMBSPMBSP4Sight4SightPSSAPSSA
Validated Progress Monitoring Tools
DIBELSDIBELSAIMSwebAIMSwebMBSPMBSPwww.studentprogress.org www.studentprogress.org
Step 2: Ground the DataStep 2: Ground the Data
1) To what will we compare our 1) To what will we compare our student growth data?student growth data?
2) How will we set goals?2) How will we set goals?
Multiple Ways toLook at Growth
Needed Growth Needed Growth Expected Growth & Percent of Expected GrowthExpected Growth & Percent of Expected Growth Fuchs et. al. (1993) Table of Realistic and Fuchs et. al. (1993) Table of Realistic and
Ambitious GrowthAmbitious Growth Growth Toward Individual Goal*Growth Toward Individual Goal*
*Best Practices in Setting Progress Monitoring Goals for Academic Skill *Best Practices in Setting Progress Monitoring Goals for Academic Skill Improvement (Shapiro, 2008)Improvement (Shapiro, 2008)
Needed Growth
Difference between student’s BOY (or Difference between student’s BOY (or MOY) score and benchmark score at MOY MOY) score and benchmark score at MOY (or EOY).(or EOY).
Example: MOY ORF = 10, EOY Example: MOY ORF = 10, EOY benchmark is 40, 18 weeks of instruction benchmark is 40, 18 weeks of instruction (40-10/18=1.67). Student must gain 1.67 (40-10/18=1.67). Student must gain 1.67 wcpm per week to make EOY benchmark.wcpm per week to make EOY benchmark.
Expected Growth
Difference between two benchmarks.Difference between two benchmarks.Example: MOY benchmark is 20, EOY Example: MOY benchmark is 20, EOY
benchmark is 40, expected growth (40-benchmark is 40, expected growth (40-20)/18 weeks of instruction = 1.11 wcpm 20)/18 weeks of instruction = 1.11 wcpm per week.per week.
Tigard-Tualatin School District (www.ttsd.k12.or.us)
Looking at Percent of Expected Growth
Tier I Tier II Tier III
Greater than 150%
Between 110% & 150%
Possible LD
Between 95% & 110%
Likely LD
Between 80% & 95%
May Need More
May Need More
Likely LD
Below 80% Needs More Needs More Likely LD
Fuchs, Fuchs, Hamlett, Walz, & Germann Fuchs, Fuchs, Hamlett, Walz, & Germann (1993)(1993)
Oral Reading Fluency Adequate Response Table
Realistic Growth
Ambitious Growth
1st 2.0 3.0
2nd 1.5 2.0
3rd 1.0 1.5
4th 0.9 1.1
5th 0.5 0.8
Fuchs, Fuchs, Hamlett, Walz, & Germann Fuchs, Fuchs, Hamlett, Walz, & Germann (1993)(1993)
Digit Fluency Adequate Response Table
Realistic Realistic GrowthGrowth
Ambitious Ambitious GrowthGrowth
11stst 0.30.3 0.50.5
22ndnd 0.30.3 0.50.5
33rdrd 0.30.3 0.50.5
44thth 0.750.75 1.21.2
55thth 0.750.75 1.21.2
If Local Criteria are Not an Option
Use norms that accompany the measure Use norms that accompany the measure (DIBELS, AIMSweb, etc.).(DIBELS, AIMSweb, etc.).
Use national norms.Use national norms.
Making Decisions: Best Practice
Research has yet to establish a blue print Research has yet to establish a blue print for ‘grounding’ student RoI data. for ‘grounding’ student RoI data.
At this point, teams should consider At this point, teams should consider multiple comparisons when planning and multiple comparisons when planning and making decisions. making decisions.
Making Decisions: Lessons From the Field
When tracking on grade level, consider an When tracking on grade level, consider an RoI that is 100% of RoI that is 100% of expectedexpected growth as a growth as a minimum requirement, consider an RoI minimum requirement, consider an RoI that is at or above the that is at or above the neededneeded as optimal. as optimal.
So, 100% of expected and on par with So, 100% of expected and on par with needed become the limits of the range needed become the limits of the range within a student should be achieving.within a student should be achieving.
Is there an easy way to do all of this?
Oral Reading Fluency01/15/09 01/22/09 01/29/09 02/05/09 02/12/09 02/19/09 02/26/09 03/05/09 03/12/09 03/19/09 03/26/09 04/02/09 04/09/09 04/16/09 04/23/09 04/30/09 05/07/09 05/14/09
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18
Benchmark 68 90 1.29
Aiden 61 40 52 60 71 95 1.61 2.17 167%
Ava 49 43 49 77 57 54 87 92 2.28 2.76 213%
Noah 49 48 45 69 61 54 84 2.28 2.01 156%
Olivia 65 49 57 70 79 83 1.39 1.50 116%
Liam 55 53 36 54 70 83 1.94 1.58 122%
Hannah 59 54 64 69 52 60 82 1.72 1.20 93%
Gavin 64 40 67 68 84 79 1.44 1.66 129%
Grace 53 48 46 60 74 79 2.06 1.76 136%
Oliver 50 44 46 68 51 51 57 78 2.22 1.45 112%
Peyton 63 50 47 58 75 77 1.50 1.12 87%
Josh 49 38 49 55 48 36 67 77 2.28 1.62 125%
Riley 42 49 54 69 67 50 76 2.67 1.76 136%
Mason 53 53 50 64 60 74 2.06 1.17 91%
Zoe 34 38 42 68 55 51 58 3.11 1.44 111%
Ian 41 31 45 49 47 30 46 2.72 0.24 19%
Faith 29 36 35 36 36 29 45 44 3.39 0.75 58%
David 30 23 44 52 43 19 63 38 3.33 0.79 61%
Alexa 18 19 25 33 33 23 28 37 4.00 0.94 73%
Hunter 23 23 24 48 38 32 34 3.72 0.75 58%
Caroline 28 20 28 40 37 19 25 30 3.44 0.02 2%
** Actual RoI based on linear regression of all data points
Benchmarks based on DIBELS Goals
Expected RoI at Benchmark Level
(Fuchs, Fuchs, Hamlett, Walz, & Germann 1993)
% of Expected RoI
Needed RoI* Actual RoI**
Realistic Grow thAmbitious Grow th
Oral Reading Fluency Adequate Response Table
1st Grade
2nd Grade
* Needed RoI based on difference betw een w eek 1 score and Benchmark score for w eek 18 divided by 18 w eeks
0.5
3rd Grade
4th Grade
5th Grade
0.9
0.8
2.0
1.5
1.0
3.0
2.0
1.5
1.1
1/14/2011 1/121/2011 1/28/2011 5/14/20111 2 3 18
Benchmark 68 90 1.29Student 22 27 56 3.78 1.89 147%
Needed RoI Actual RoI% of Expected
RoI
Access to Spreadsheet Templates
http://sites.google.com/site/http://sites.google.com/site/rateofimprovement/homerateofimprovement/home
Click on Charts and Graphs.Click on Charts and Graphs.Update dates and benchmarks.Update dates and benchmarks.Enter names and benchmark/progress Enter names and benchmark/progress
monitoring data.monitoring data.
What about Students not on What about Students not on Grade Level?Grade Level?
Determining Instructional Level
Independent/Independent/InstructionalInstructional/Frustrational /Frustrational Instructional often b/w 40Instructional often b/w 40thth or 50 or 50thth
percentile and 25percentile and 25thth percentile. percentile.Frustrational level below the 25Frustrational level below the 25thth
percentile.percentile.AIMSweb: Survey Level Assessment AIMSweb: Survey Level Assessment
(SLA).(SLA).
Setting Goals off of Grade Level
100% of expected growth not enough.100% of expected growth not enough.Needed growth only gets to instructional Needed growth only gets to instructional
level benchmark, not grade level.level benchmark, not grade level.Risk of not being ambitious enough.Risk of not being ambitious enough.Plenty of ideas, but limited research Plenty of ideas, but limited research
regarding Best Practice in goal setting off regarding Best Practice in goal setting off of grade level.of grade level.
Possible Solution (A)
Weekly probe at instructional level and Weekly probe at instructional level and compare to expected and needed growth compare to expected and needed growth rates at instructional level.rates at instructional level.
Ambitious goal: 200% of expected RoI Ambitious goal: 200% of expected RoI
Oral Reading Fluency
01/15/10 01/22/10 01/29/10 02/05/10 02/12/10 02/19/10 02/26/10 03/05/10 03/12/10 03/19/10 03/26/10 04/02/10 04/09/10 04/16/10 04/23/10 04/30/10 05/07/10 05/14/10Needed RoI* Actual RoI**
% of Expected RoI
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18
5th Grade 115 124 0.53
104 134 128 143 1.11 2.15 407%
104 92 101 115 116 129 108 121 1.11 1.43 271%
94 108 95 121 135 126 109 113 1.67 1.32 248%
99 66 100 83 92 107 109 93 1.39 0.89 168%
110 97 124 113 131 132 142 0.78 2.33 440%
96 74 108 62 107 92 94 95 1.56 0.39 74%
73 93 79 114 111 112 87 116 2.83 2.03 383%
102 112 122 103 118 135 105 112 1.22 0.51 97%
111 AB 121 121 134 131 99 120 0.72 0.11 21%
110 90 118 103 119 121 122 119 0.78 1.21 228%
107 91 105 90 79 89 90 120 123 119 0.94 1.22 231%
4th Grade 105 118 0.76
60 72 43 59 76 57 47 65 4.14 -0.06 -8%
80 81 92 120 137 110 2.71 4.94 646%
3rd Grade 115 124 0.53
55 57 66 76 66 47 66 6.27 0.49 93%
85 72 84 92 94 82 76 82 3.25 -0.08 -15%
79 81 91 70 65 73 3.21 -1.41 -267%
70 104 79 79 68 4.50 -1.45 -274%
2nd Grade 68 90 1.29
74 63 83 84 71 74 86 82 77 91 0.94 1.15 89%
91 70 104 -0.06 2.21 171%
Possible Solution (B)
Weekly probe at instructional level for Weekly probe at instructional level for sensitive indicator of growth.sensitive indicator of growth.
Monthly probes (give 3, not just 1) at Monthly probes (give 3, not just 1) at grade level to compute RoI.grade level to compute RoI.
Goal based on grade level growth (more Goal based on grade level growth (more than 100% of expected).than 100% of expected).
Step 3: Interpreting GrowthStep 3: Interpreting Growth
What do we do when we do not get the growth we want?
When to make a change in instruction and When to make a change in instruction and intervention?intervention?
When to consider SLD?When to consider SLD?
When to make a change in instruction and intervention?
Enough data points (6 to 10)?Enough data points (6 to 10)?Less than 100% of expected growth.Less than 100% of expected growth.Not on track to make benchmark (needed Not on track to make benchmark (needed
growth).growth).Not on track to reach individual goal.Not on track to reach individual goal.
When to consider SLD??
Continued inadequate response despite: Continued inadequate response despite: Fidelity with Tier I instruction and Tier Fidelity with Tier I instruction and Tier
II/III intervention.II/III intervention. Multiple attempts at intervention.Multiple attempts at intervention. Individualized Problem-Solving Individualized Problem-Solving
approach.approach.
Evidence of dual discrepancy… Evidence of dual discrepancy…
05/14/09
18
90 1.29
95 1.61 2.17 167% Keep On Truckin92 2.28 2.76 213% Keep On Truckin84 2.28 2.01 156% 83 1.39 1.50 116% 83 1.94 1.58 122% 82 1.72 1.20 93% 79 1.44 1.66 129% 79 2.06 1.76 136% 78 2.22 1.45 112% 77 1.50 1.12 87% 77 2.28 1.62 125% 76 2.67 1.76 136% 74 2.06 1.17 91% 58 3.11 1.44 111% 46 2.72 0.24 19% BIG PROBLEMS44 3.39 0.75 58% BIG PROBLEMS38 3.33 0.79 61% BIG PROBLEMS37 4.00 0.94 73% BIG PROBLEMS34 3.72 0.75 58% BIG PROBLEMS30 3.44 0.02 2% BIG PROBLEMS
Dual Discrepancy?% of Expected RoI
Needed RoI* Actual RoI**
85% - 125%
<85%
Growth Criteria
>125%
Three Levels of Examples
Whole ClassWhole ClassSmall GroupSmall Group Individual Student Individual Student
- Academic Data- Academic Data
- Behavior Data- Behavior Data
Whole Class ExampleComputation
01/15/10 01/22/10 01/29/10 02/05/10 02/12/10 02/19/10 02/26/10 03/05/10 03/12/10 03/19/10 03/26/10 04/02/10 04/09/10 04/16/10 04/23/10 04/30/10 05/07/10 05/14/10
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18
50th P ercentile 25 31 0.35
25th P ercentile 19 23 0.24
Student 6.5 9 8 8.5 5.5 11 13 1.72 0.61 173%
Student 6 7.5 8.5 5 11 11.5 1.72 0.57 161%
Student 4.5 5.5 6.5 9.5 10.5 1.72 1.06 300%
Student 13 8 9.3 8 5.6 9.6 9.6 1.72 -0.23 -66%
Student 8 10.5 10.5 5.6 9.3 9 1.72 -0.03 -7%
Student 8.5 5.5 9 8 4 8 9 1.72 0.07 21%
Student 6.5 5.5 6 10.5 9 1.72 0.43 122%
Student 6.5 9 4.5 6 8 1.72 0.07 20%
Student 8 10.5 4.5 6.5 4 7 1.72 -0.25 -71%
Student 9 10 5.6 6.6 5 4.6 6.6 1.72 -0.42 -119%
Student 8 8 8.5 4 8 6.6 1.72 -0.18 -51%
Student 9 4.5 4.5 4 3.5 3.5 6.5 1.72 -0.24 -67%
Student 6.5 5 6.5 9 7.5 6.5 1.72 0.09 26%
Student 5.5 3 8 4 6.5 6.3 1.72 0.19 55%
Student 7.5 10 6.6 3.3 3 6.3 1.72 -0.46 -130%
Student 5 5.5 6.5 6 5 6 1.72 0.04 11%
Student 5 4 8 8.5 10 8 6 1.72 0.25 71%
Student 4.5 6 5 2.5 3.5 5.5 1.72 -0.03 -8%
Student 5.5 4.5 10.5 11 5 5.3 1.72 -0.14 -40%
* Needed RoI based on difference betw een w eek 1 score and Benchmark score for w eek 18 divided by 18 w eeks
** Actual RoI based on linear regression of all data points
Percentiles based on AIMSw eb Grow th Tables
Expected RoI at 50th PercentileExpected RoI at 25th Percentile
Needed RoI* Actual RoI** % of Expected RoI
Digit Fluency Adequate Response Table
Realistic Grow thAmbitious Grow th
1st Grade 0.3 0.5
2nd Grade 0.3 0.5
3rd Grade 0.3 0.5
(Fuchs, Fuchs, Hamlett, Walz, & Germann 1993)
4th Grade 0.75 1.2
5th Grade 0.75 1.2
3rd Grade Math Whole Class
Who’s responding? Who’s responding? Effective math Effective math
instruction? instruction? Who needs more?Who needs more?
N=19N=194 > 100% growth 4 > 100% growth 15 < 100% growth15 < 100% growth9 w/ negative 9 w/ negative
growthgrowth
Small Group ExampleOral Reading Fluency
09/11/09 09/18/09 09/25/09 10/02/09 10/09/09 10/16/09 10/23/09 10/30/09 11/06/09 11/13/09 11/20/09 11/27/09 12/04/09 12/11/09 12/18/09 01/01/10 01/08/10 01/15/10
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18
Benchmark 44 68 1.41
Student 35 39 41 45 42 45 52 57 62 1.83 1.49 106%
Student 28 38 42 40 50 55 64 72 74 2.22 2.77 196%
Student 26 28 32 31 27 29 35 34 38 2.33 0.57 41%
Student 31 35 39 45 42 47 53 58 65 2.06 1.90 135%
Student 40 44 38 48 52 64 72 74 78 1.56 2.62 186%
** Actual RoI based on linear regression of all data points
Benchmarks based on DIBELS Goals
Expected RoI at Benchmark Level
(Fuchs, Fuchs, Hamlett, Walz, & Germann 1993)
5th Grade
* Needed RoI based on difference betw een w eek 1 score and Benchmark score for w eek 18 divided by 18 w eeks
% of Expected RoI
2nd Grade
0.9 1.1
3rd Grade
4th Grade
0.5 0.8
1.5 2.0
1.0 1.5
Needed RoI* Actual RoI**
2.0 3.0
Oral Reading Fluency Adequte Response Table
1st Grade
Realistic Grow thAmbitious Grow th
Intervention Group
Intervention working for how many?Intervention working for how many?Can we assume fidelity of intervention Can we assume fidelity of intervention
based on results?based on results?Who needs more?Who needs more?
Individual Kid Example 2nd Grade Reading Progress
44
68
90
31
56
45
53 53
4846
60
74
79
y = 1.5333x + 42.8
y = 0.9903x + 36.873
0
10
20
30
40
50
60
70
80
90
100
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
09/12/08 09/19/0809/26/0810/03/08 10/10/08 10/17/08 10/24/08 10/31/08 11/07/08 11/14/08 11/21/08 11/28/08 12/05/08 12/12/08 12/19/08 01/16/09 01/23/0901/30/0902/06/0902/13/0902/20/0902/27/0903/06/0903/13/0903/20/0903/27/0904/03/0904/10/0904/17/0904/24/0905/01/09
Wo
rds
Re
ad
Co
rre
ct
Pe
r M
inu
te
Benchmark Linear (Benchmark) Linear
Individual Kid
Making growth?Making growth?How much (65% of expected growth).How much (65% of expected growth).Atypical growth across the year (last 3 Atypical growth across the year (last 3
data points). data points). Continue? Make a change? Need more Continue? Make a change? Need more
data?data?
RoI and Behavior?RoI and Behavior?
Percent of Time Engaged in Appropriate Behavior
y = 2x + 22
y = 3.9x + 19.8
y = 7.2143x - 1.5
0
10
20
30
40
50
60
70
80
90
100
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18
Per
cen
t
Baseline Condition 1 Condition 2 Linear (Baseline) Linear (Condition 1) Linear (Condition 2) Linear (Condition 2)
Step 4: Figure out how to fit Step 4: Figure out how to fit Best Practice into Public Best Practice into Public
EducationEducation
Things to Consider
Who is At-Risk and needs progress Who is At-Risk and needs progress monitoring?monitoring?
Who will collect, score, enter the data?Who will collect, score, enter the data?Who will monitor student growth, when, Who will monitor student growth, when,
and how often?and how often?What changes should be made to What changes should be made to
instruction & intervention? instruction & intervention? What about monitoring off of grade level?What about monitoring off of grade level?
Who is At-Risk and needs progress monitoring??
Below level on universal screeningBelow level on universal screeningEntering 4Entering 4thth Grade Example Grade Example
DORF DORF (110)(110)
ISIP ISIP TRWM TRWM
(55)(55)
4Sight 4Sight (1235)(1235)
PSSA PSSA (1235)(1235)
Student AStudent A 115115 5858 12551255 12321232
Student BStudent B 8585 4848 12161216 11261126
Student CStudent C 7272 3535 10561056 10481048
Who will collect, score, and enter the data?
Using MBSP for math, teachers can Using MBSP for math, teachers can administer probes to whole class.administer probes to whole class.
DORF probes must be administered one-DORF probes must be administered one-on-one, and creativity pays off (train and on-one, and creativity pays off (train and use art, music, library, etc. specialists).use art, music, library, etc. specialists).
Schedule for progress monitoring math Schedule for progress monitoring math and reading every-other week.and reading every-other week.
Week 1 Week 2
Reading Math Reading Math
1st X X
2nd X X
3rd X X
4th X X
5th X X
Who will monitor student growth, when, and how often?
Best Practices in Data-Analysis Teaming Best Practices in Data-Analysis Teaming (Kovaleski & Pedersen, 2008)(Kovaleski & Pedersen, 2008)
Chambersburg Area School District Elementary Chambersburg Area School District Elementary Response to Intervention Manual (McCrea et. Response to Intervention Manual (McCrea et. al., 2008)al., 2008)
Derry Township School District Response to Derry Township School District Response to Intervention Model Intervention Model (http://www.hershey.k12.pa.us/56039310111408/lib/56039310111408/_files/Microsoft_Word_-(http://www.hershey.k12.pa.us/56039310111408/lib/56039310111408/_files/Microsoft_Word_-_Response_to_Intervention_Overview_of_Hershey_Elementary_Model.pdf)_Response_to_Intervention_Overview_of_Hershey_Elementary_Model.pdf)
What changes should be made to instruction & intervention?
Ensure treatment fidelity!!!!!!!!Ensure treatment fidelity!!!!!!!! Increase instructional time (active and Increase instructional time (active and
engaged)engaged)Decrease group sizeDecrease group sizeGather additional, diagnostic, informationGather additional, diagnostic, informationChange the intervention Change the intervention
Final Exam…
Student Data: 27, 29, 26, 34, 27, 32, 39, Student Data: 27, 29, 26, 34, 27, 32, 39, 45, 43, 49, 51, --, --, 56, 51, 52, --, 57.45, 43, 49, 51, --, --, 56, 51, 52, --, 57.
Benchmark Data: BOY = 40, MOY = 68.Benchmark Data: BOY = 40, MOY = 68.What is student’s RoI?What is student’s RoI?How does RoI compare to expected and How does RoI compare to expected and
needed RoIs?needed RoIs?What steps would your team take next?What steps would your team take next?What if Benchmarks were 68 and 90 What if Benchmarks were 68 and 90
instead?instead?
The RoI Web SiteThe RoI Web Site
http://sites.google.com/site/rateofimprovement/http://sites.google.com/site/rateofimprovement/ Download powerpoints, handouts, Excel graphs, Download powerpoints, handouts, Excel graphs,
charts, articles, etc.charts, articles, etc.
Caitlin Flinn BennyhoffCaitlin Flinn Bennyhoff [email protected] [email protected]
Andy McCreaAndy McCrea [email protected] [email protected]
Matt FerchalkMatt Ferchalk
[email protected]@norleb.k12.pa.us