avoiding paralysis by analysis: part 2 progress · pdf fileavoiding paralysis by analysis:...
TRANSCRIPT
Avoiding Paralysis by Analysis: Part 2 Progress Monitoring
ND RTI Conference June 18-20, 2012
Dr. Lisa H. Stewart [email protected]
“Real” Data thanks to several local RTI coaches and teams – Thank You!
Is this progress monitoring? Why or why not?
How can we impact learning for all students?
• Visible Learning: John Hattie • What works in education? Almost everything,
but some things work better and work worse compared to many alternatives.
How do we know what works BEST?
• Meta-Analysis: Compares results of many different research studies and outcomes
• Effect sizes: Converts outcomes of studies to a single scale we can compare outcomes across studies. – .2 small – .5 moderate – .8 large
• Effect size of 1 indicates an increase of one standard deviation on the outcome (achievement). – Advanced achievement by 2-3 years – Student who got “xyz” exceeded 84% of kids who didn’t
get “xyz.”
Formative Evaluation
Variables Impacting Student Achievement: Some of the Biggies!
• Formative Evaluation .90 • Teacher Clarity .75 • Reciprocal Teaching .74 • Feedback .73 • Teacher-Student Relationship .72 • Spaced vs. mass practice .71 • Metacognitive strategies .69
8
Why Monitor Progress? • When teachers USE progress monitoring
– Students learn more! – Teachers design better instructional
programs – Teacher decision
making improves – Students become more
aware of their performance Safer & Fleishman, 2005
Sometimes it is nice to have data… was changing to RR a good idea?
10
What is Progress Monitoring?
• Standardized measures – Reliable – Valid (sensitive, treatment validity)
• Tied to important educational outcomes
• Given frequently (e.g., 1 or 2x weekly) – Simple, brief, efficient, and cheap
Some Common CBM Tasks Area Time Procedure Scoring Early Literacy
- Letter Naming
- Letter Sounds
- Phonemic Segmentation - Nonsense Word
1 minute
Individual
Letter Names Correct Letter Sounds Correct Segments Correct Letter Sounds Correct Words Correct
Reading - Oral Reading Fluency
- Maze
1 minute 3-5 minutes
Individual Group or Individual
Words Read Correct/ Errors Words Correct
Math - Computation
- Concepts
2-5 minutes 8-10 minutes
Group or Individual Group or Individual
Correct Digits Correct Problems Points Correct
Written Expression
3-5 minutes Group or Individual Total Words Written Correct Words Sequence Words Spelled Correctly
What about….
• Using MAP or STAR ? • Using Curriculum tests? • Using Common Core Assessments?
– Skwira, 2011 r= .31 to .62, most <.50
14
Short Term (Mastery) and Long Term Progress Monitoring
BOTH are IMPORTANT!
Short Term Mastery Monitoring
Test subskill mastery and individual lesson effectiveness
Ex: Q&A, worksheets following directions
unit tests, “hot” reads accuracy, skills “checks”
CBE, cba
Long Term General Outcome Measures
Test retention, generalization and progress
toward overall general outcome (reading)
Ex: CBM, DIBELS
15
How Often Do We Monitor Progress?
• For CBM Early Literacy (Letter Sound Fluency, Nonsense Word Fluency) and Oral Reading Fluency – Weekly with 1 “probe” (this is most common) – 2 or 3x per week (1 probe each time) – Every 3 weeks with 3 probes
• For CBM Maze, Math or WE – Weekly (?) or Every two weeks
16
Who does it? When? Where?
• Anyone trained in the procedures can collect progress monitoring data – Be creative but careful
• When and Where? – At a time and place that will provide valid information – Use common sense
17
Using the Data
• Just having progress monitoring data is not enough. You need to USE it.
– Scheduled graph review dates
– Grade level or problem solving meetings
18
Avoid Common Mistakes • Use “Good” data collected with fidelity • Have an organized system in place
– Progress monitoring schedule for students – Preprinted passages/probes in a binder – An easy way to graph and look at the data
• Internet-based data management and reporting system (AIMSweb, EasyCBM, DIBELS, etc)
– Share/look at the data
What Level of Materials? • What materials to use for progress
monitoring? What grade level? – Rule of Thumb – Use Grade Level materials – if you monitor out of grade level, keep in mind
catching up to peers when setting goal and monitor monthly in grade level material
– Switch to grade level monitoring weekly when intervention is successful and level of intervention intensity is decreased (e.g., moving from Tier 3 to Tier 2 or decreasing time in Tier 2)
Progress Monitoring Steps • Establish solid initial data point, set up graph,
set goal, draw aimline
• Measure and plot student performance over time
• Analyze student performance – check data points below the aimline, “go upness”, trendline, etc
• Make Instructional changes if indicated
• Continue to measure and monitor student performance
21
Develop goals and graphs with the student and have them graph their progress
22
Aimline
• Shows general trajectory needed for student to reach his/her goal
– Typically set so student gets back “on target” or “on grade level” within a set amount of time (e.g., by the end of the year) if possible
– Simply draw a straight line from the student’s first data point on the graph to the date and score representing his target or goal
– Use SMART goals: specific, measurable, attainable, realistic, with a clear timeframe
SMART Goals…
• Specific • Measurable • Attainable • Realistic • Timely
23
Not so SMART goals: “sarah will improve in reading…” “Adam will comprehend at a 6th grade level…” “Suzanne will follow classroom rules…” SMART goal: By May 30, 2012 when given a 3rd grade AIMSweb reading passage, Sarah will read aloud 109 wrc per min. with 5 or fewer errors.
24
For setting CBM goals – Can use local norms or benchmark targets set by your district
or based on national datasets and research (e.g., DIBELS targets, AIMSweb targets)
– Can use information on the amount of progress students who were successful have made in the past in this intervention or curriculum (e.g., what was the slope of progress in the research?)
– Good rule of thumb: For Oral Reading Fluency 1.5- 3.0 words
per week growth • You are starting a progress monitoring graph for Finnegan, Gr 2 ORF =
48 in the end of January, what would his end of year goal be?
25
Example Gr 1-5 “Targets” for Aimline Based on the St. Croix River Education District 10-11 Targets linked to success on Minnesota Comprehensive Assessment – II
Grade Measure Target
1 Nonsense Word Fluency January = 52 letter sounds correct/min
1 CBM Grade Level Oral Reading Fluency (ORF)
Spring = 52 words correct/min
2 CBM ORF Spring = 90 words correct/min
3 CBM ORF Spring = 109 words correct/min
4 CBM ORF Spring = 127 words correct/min
5 CBM ORF Spring = 141 words correct/min
6 CBM ORF Spring = 166 words correct/min
Early CBM national “norms” and growth rates in oral reading (words correct per min.)
Grade Percentile Fall ORF Winter ORF Spring ORF Weekly Progress
1 50 27 54 1.80
2 75 50 25
82 53 23
106 78 46
124 94 65
1.66
3 75 50 25
107 79 65
123 93 70
142 114 87
1.18
4 75 50 25
125 99 72
133 112 89
143 118 92
1.01
5 75 50 25
126 105 77
143 118 93
151 128 100
0.58
6+ 50 125-150 125-150 125-150 0.66
Hasbruck & Tindal 1992, Teaching Exc. Children, Deno et al, 2001 School Psych Review
www.studentprogress.org Guidelines for slope of progress and level of skill: Math
Note: These figures may change pending additional RTI research.
Grade Mixed Computation Concepts & Applications
Slope Slope
Grade 1 0.50 0.40
Grade 2 0.40 0.40
Grade 3 0.40 0.70
Grade 4 0.70 0.70
Grade 5 0.70 0.70
Grade 6 0.70 0.70
28
Looking at the Graphs
• Is there “go upness”????
• Is there ENOUGH “go upness”???? Dr. Roland Good
29
Using Progress Monitoring Data: Is this intervention working?
Adam Gr 4
0102030405060708090
100110120130140
1/13/
2009
1/27/
2009
2/10/
2009
2/24/
2009
3/10/
2009
3/24/
2009
4/7/2
009
4/21/
2009
5/5/2
009
5/19/
2009
wor
ds c
orre
ct p
er m
in (
grad
e le
vel p
assa
ge)
w rc
errors
30
And Now? Adam Gr 4
0
10
20
30
40
50
60
70
80
90
100
110
120
130
140
1/13/2
009
1/20/2
009
1/27/2
009
2/3/20
09
2/10/2
009
2/17/2
009
2/24/2
009
3/3/20
09
3/10/2
009
3/17/2
009
3/24/2
009
3/31/2
009
4/7/20
09
4/14/2
009
4/21/2
009
4/28/2
009
5/5/20
09
5/12/2
009
5/19/2
009
5/26/2
009
wor
ds c
orre
ct p
er m
in (g
rade
leve
l pas
sage
)
wrcerr
Repeated Reading 1:1, 20 min day Repeated Reading 1:1, 10 min 2xday
31
Data Decision Guidelines
• If the student has some data points above and some below the aimline (doing the “aimline hug”), keep doing what you are doing!
• If the student has 3 to 5 consecutive data points above the aimline, consider moving the student to less intervention (e.g., decreasing minutes, or moving from Tier 2 to Tier 1 or Tier 3 to Tier 2)
– Also use other pieces of information like Mastery Monitoring Data!
– Continue to progress monitor
32
Data Decision Guidelines (Cont’d) • If the student has 3 to 5 consecutive data points below the
aimline, ASK THE FOLLOWING QUESTIONS (and continue to progress monitor):
– What does the “other” evidence available suggest about the student’s progress?
• Error rates? Behavior during the intervention?
– What is the general “trend” of the data? Is the student likely to get where we want if this continues?
• Use visual analysis and other evidence • Use “trendlines” and “aimlines”
– Is the intervention being implemented with fidelity?
33
Finn Gr 2 CBM-ORF Finnegan Grade 2
0
10
20
30
40
50
60
70
80
90
100
110
120
9/9/
2009
9/23
/200
9
10/7
/200
9
10/2
1/20
09
11/4
/200
9
11/1
8/20
09
12/2
/200
9
12/1
6/20
09
12/3
0/20
09
1/13
/201
0
1/27
/201
0
2/10
/201
0
2/24
/201
0
3/10
/201
0
3/24
/201
0
4/7/
2010
4/21
/201
0
5/5/
2010
5/19
/201
0
week
wor
ds c
orre
ct p
er m
in (g
rade
leve
l pas
sage
)
Aimline Reading Links 1:5 for 15 min.
34
Finn Gr 2 CBM-ORF (Cont’d) Finnegan Grade 2
0
10
20
30
40
50
60
70
80
90
100
110
120
9/9/
2009
9/23
/200
9
10/7
/200
9
10/2
1/20
09
11/4
/200
9
11/1
8/20
09
12/2
/200
9
12/1
6/20
09
12/3
0/20
09
1/13
/201
0
1/27
/201
0
2/10
/201
0
2/24
/201
0
3/10
/201
0
3/24
/201
0
4/7/
2010
4/21
/201
0
5/5/
2010
5/19
/201
0
week
wor
ds c
orre
ct p
er m
in (g
rade
leve
l pas
sage
)
Aimline Added distributed practice and preteaching Reading Links 1:5
for 15 min.
35
Justan Gr 1 NWF Is Justan Making Progress?
36
And Now?
37
And Now????
38
On Track…
39
Abdi Gr 2 What Decision Would You Make?
40
And Now?
41
Is There “Enough” Go Upness?
What about with STAR data?
A math example…. Is this student making progress?
Another math example…
InIn Math, don’t always see growth like this! But it shows what is possible!
Benjamin, gr K • In Fall, was flagged as “at risk” on AIMSweb screening (had 3
corrects on LNF, 6 corrects on LSF but high errors/guessing) – monitored weekly in LNF, received Leveled Literacy
Intervention from GE teacher
• In January, winter benchmark ok in LNF, but very low in sounds (LSF=4) and phoneme segmentation (PSF=12), went to RTI team – Reviewed screening and graph, teacher comments about
immature speech and actions – moved from LLI to small group working on Phon
Awareness (from curriculum materials) with GE teacher and classroom para support, and RTI support 3-4 days per week working on letter recognition and sounds/phonics, progress monitored LNF and LSF
How is Benjamin doing?
Benjamin, cont’d
• In May, team met again, he had shown growth
in LNF, but not LSF. He also has other concerns with social skills, some speech patterns (Ks and Gs) and math. Team recommends summer school and review of data and plan in September.
Jared, Gr 1
• In Fall, Screening data very low on AIMSweb NWF, also low NWEA
• Other: behavior concerns, in group of struggling kids without strong K curriculum (maybe not much exposure to good Curriculum and Instr?)
• RTI team worked with teacher regularly throughout the year – Variety of interventions in phonics – Behavioral support/plan as well – Monitored weekly in NWF and PSF (later in RCBM)
Jared PSF
Was this a concern? Did they need to monitor progress?
Jared NWF
Jared, started RCBM in Spring
End of Year
• Went from among lowest in grade in Winter on RCBM (23) to in “average” range, almost in the middle of his class, on AIMSweb screening (55)- a gain of 32 words in 15 weeks or ? 2 words per week gain, but still
• Will he still need support?
Holly, Gr 1 and “The case of different points of view”
Holly PSF
Holly again NWF
How is this kid doing? I call this one the “kitchen sink kid”
Documentation for Special Education Eligibility…
61
Using a Trendline • Shows the general “trend” or trajectory of the student’s
data so far – Web-based programs typically use an Ordinary Least Squares
regression line • AIMSweb, DIBELS data system, Excel
– Need approx. 10-14 data points, the more the better – Trendlines on few data points or on highly variable data are NOT
reliable!!! • Christ, T. (2006). Short term estimates of growth using CBM ORF:
Estimating Standard Error of Slope to construct confidence intervals. School Psychology Review, 35(1) 128-133.
62
Trendline and Aimline
y = 0.7995x + 52.547 R² = 0.0731
0
10
20
30
40
50
60
70
80
90
100
110
120
1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31
wor
ds c
orre
ct p
er m
in. (
grad
e le
vel p
assa
ges)
Weeks
Moira, Grade 3
AIMLINE
TRENDLINE
63
What Can You Do About “Bounce” in the Data?
64
Dealing With Bounce • Is there a “measurement” problem?
– Fidelity of administration and scoring
– Materials aren’t well designed or are too difficult
– Who, where, and when measurement takes place can matter (esp. for some kids)
– Motivation issues (can’t do vs. won’t do)
65
Dealing with Bounce (Cont’d)
• Other ways to minimize bounce or make decisions despite bounce – Look at data trend over a longer period of time – Do more probes at one time and take median or average
score – Do more frequent measurement (e.g., weekly or 2x week)
• Look at trend over time with many data points
– Look at ALL data together (errors, mastery data, etc.) – Use the least dangerous assumption… it isn’t working.
66
What if There isn’t Adequate Progress?
If you keep doing what you’ve been doing then you will keep
getting what you’ve got.
Insanity is doing the same thing over and over and expecting different results.
67
Back to Problem Solving
RIOT and ICEL Instruction Curriculum Environment Learner
Review
Interview
Observe
Test
Work samples (best if conditions are known)
Dynamic Teaching, Diagnostic Testing
• Vary the ICE! • Can’t do/Won’t do assessment • Brief Experimental Analysis… try stuff! (and
collect good/consistent data) • Collect more data: Quick Phonics Screener,
CTOPP, etc etc • Think about instructional hierarchy and
instructional “match”
Tiered Interventions: Need to “fit” • PALS
• HOSTS
• Read Naturally
• Rewards
• Reading Rockets
• Etc., etc., etc.
Phonemic Awareness
Phonics
Fluency
Vocabulary and Comprehension
Credit: Matt Burns
72
What if There isn’t Adequate Progress?
– Is the intervention being done with fidelity?
• Have fidelity checks been done?
• Is the student in the right level of materials?
• Has the student been in school? Are they getting enough minutes of intervention per week? Do they have good academic engaged time during the intervention?
73
What if There isn’t Adequate Progress? (Cont’d)
– Should the intervention be “tweaked”? Changed? Is there an intervention better “matched” to this student’s needs?
• Changes could include trying a different intervention or just “tweaking” the current intervention such as adding a 5th repeat to a repeated reading or a sticker incentive for accurate reading.
• Grade level or problem solving team members work together to discuss the data, the student, and what intervention changes would have the best chance of success.
74
What Could We Change?
• Focus or skill
• Teaching strategies: More explicit, more modeling, more practice, more previewing, better matched with core
• Materials: Easier, better matched (cultural, interests, etc.)
• Arrangements: Size group, location, who is teaching?
• Time: Amount of time, days per week, time of day
• Motivation: Interests, goals, rewards, home/school
LHS sample intervention inventory “net”: Big Five for Reading
Example Program Phon Aware
Phonics Fluency Vocab Comprehension
Early Reading Intervention X X X X x
PALS (Vanderbilt) X (K,1) X (K,1) X (2-6) X (2-6)
Reading Mastery (or other SBRC)
X X X X x
Great Leaps x x x
Sound Partners x x
Say it Move It X (fluency)
REWARDS X (gr 4+) X X (Sc, SS) X (SC, SS)
Partner Reading (Archer) x
Rep. Rdg, Rep. Phrases x
Incr. Rehearsal (letter sounds) X (fluency) (preview, vocab?)
Vocab Strategies (Archer) x
Keyword Strategy x
Kansas SIM x
Vocab Strategy (Archer) x
76
Web Resources
• Research Institute on Progress Monitoring – http://progressmonitoring.net/ – Includes…
• A Study Group Content Module with 15 sections on CBM including activities – http://progressmonitoring.org/pdf/cbmMOD1.pdf
• Progress Monitoring Leadership Team Content Module with 6 sections (e.g. measureable goals, decision making) including activities
– http://progressmonitoring.org/pdf/cbmMODldrshp.pdf • Handouts, videos, and power point presentations • Technical reports of CBM measures
• Research Institute on Problem Solving • http://www.cehd.umn.edu/EdPsych/RIPS/
77
Web Resources, Cont’d
• www.aimsweb.com, www.edcheckup.com, dibels.uoregon.edu, easyCBM.com – Look for information about progress monitoring as well as
access to materials and graphing for progress monitoring • www.interventioncentral.org
– look for information on CBM, graphing, etc. • www.studentprogress.org
– Growth rates, use in RTI model, etc (not updated)
• http://www.rti4success.org – click on Progress monitoring on right side