large scale studies of dyslexia in florida richard k. wagner and yusra ahmed florida state...
TRANSCRIPT
Large Scale Studies of Dyslexia in Florida
Richard K. Wagner and Yusra Ahmed
Florida State University and FCRR
NIH Multidisciplinary Learning Disabilities Center (P50 HD052120)
Alternative Approaches
• 1. Typical research study.
• 2. Meta-analysis.
• 3. Large-scale study.
Large-Scale Study of Incidence of Specific Reading Comprehension
Disability
Personal Interest in Reading Comprehension Problems
• Comprehension errors run in my family.
Ellis Island
Question on a Form
• If admitted to this country, would you advocate overthrow of the government of the United States by force or violence?
Question on a Form
• If admitted to this country, would you advocate overthrow of the government of the United States by force or violence?
• violence
Reported Incidence of Reading Comprehension Disability
• 10 percent of 7-11 year-olds are poor at reading comprehension despite being accurate and fluent at decoding (Nation, 2004).
Many Possible Causes
• “There is room for lots of things to go wrong when comprehension fails” (Perfetti, 1994, p. 885, cited by Nation, 2005).
Possible Causes (Science of Reading: A Handbook)
• 1. Decoding difficulties.• 2. Difficulties with meaning (vocabulary).• 3. Difficulty with syntax.• 4. Limitations in working memory.• 5. Poor inference making.• 6. Inadequate comprehension monitoring.• 7. Limited prior domain knowledge.• 8. Insensitivity to text structure.
Reading Comprehension Task
• When the next slide appears, read the text as quickly as you can and summarize the passage in a brief sentence.
• Ready?
Answer a Simple Question
• Is necrobiosis a conceivable source of orogeny or of a pomiferous pompelmous?
Simple Question
• Is necrobiosis conceivably related to orogeny or to the development of a pomiferous pompelmous?
• Obviously the normal death of cells (necrobiosis) is not related to mountain building (orogeny), but is part of the process of the development of fruit bearing (pomiferous) trees (pompelmous).
Ordering Causes by Severity of Consequences for Comprehension
• Primary– Decoding difficulties.
• Secondary– Difficulties with meaning (vocabulary).
• Tertiary– Difficulty with syntax; limitations in working memory;
poor inference making; inadequate comprehension monitoring; limited prior domain knowledge; insensitivity to text structure.
Present Studies
• Question Addressed:– What is the incidence of reading comprehension
disability not attributable to primary or secondary causes?
• Design feature:– Attempted to address issue of small samples
sizes of typical studies by using the PMRN (progress monitoring and reporting network) database.
Study Design
• 1. Identify individuals who are poor at reading comprehension.– Score at or below 5th percentile on Stanford
Achievement Test (SAT-10) Reading Comprehension.
Study Design
• 2. Determine how many individuals are poor at comprehension yet adequate at decoding.– SAT 10 at or below 5th percentile;– DIBELS Nonsense Word Fluency (NWF)
greater than or equal to 25th percentile.
Study Design
• 2. Determine how many individuals are poor at comprehension yet adequate at decoding and vocabulary.– SAT 10 at or below 5th percentile;– DIBELS Nonsense Word Fluency (NWF)
greater than or equal to 25th percentile.– Peabody Picture Vocabulary Test (PPVT)
greater than or equal to 25th percentile.
Out of a First Grade-Cohort 1 (N = 35,314)….
How Many Were Poor at…
• Reading comprehension (SAT 10 < 5th%)– 1,669 (4.73%) out of 35,314
How Many Were Poor at…
• Reading comprehension (SAT 10 < 5th %)– 1,669 (4.73%) out of 35,314
• yet adequate at decoding (nonword fluency >= 25th %)?
How Many Were Poor at…
• Reading comprehension (SAT 10 < 5th %)– 1,669 (4.73%) out of 35,314
• yet adequate at decoding (nonword fluency >= 25th %)?– only 85 (0.24%) out of 35,314!
How Many Were Poor at…
• Reading comprehension (SAT 10 < 5th %)– 1,669 (4.73%) out of 35,314
• yet adequate at decoding (nonword fluency >= 25th %)?– only 85 (0.24%) out of 35,314!
• and in vocabulary (PPVT >= 25th %)?
How Many Were Poor at…
• Reading comprehension (SAT 10 < 5th %)– 1,669 (4.73%) out of 35,314
• yet adequate at decoding (nonword fluency >= 25th %)?– only 85 (0.24%) out of 35,314!
• and in vocabulary (PPVT >= 25th %)?– only 23 (0.07%) out of 35,314!
Surprising Result: Virtually All Had Problems in Decoding
• Sample size was 35,314.
• But it was a single study. Results need to be replicated.
So We Did Replicate: Not One, Not Two, but…
Three First-Grade Cohorts
Cohorts 1 (0304) 2 (0405) 3 (0506)
N 35,314 43,712 64,645
Poor RC 1,669
(4.73%)
1,721
(3.94%)
1,896
(2.93%)
But OK decoders
85
(0.24%)
197
(0.45%)
266
(0.41%)
And OK vocabulary
23
(0.07%)
69
(0.16%)
81
(0.13%)
First Grade Results
• Of 150,000 6-year-olds, only .2 to .5 percent were poor at comprehension yet adequate at decoding.
• May be nature of reading comprehension at age 6—decoding explains about everything.
Three Second-Grade Cohorts
Cohorts 1 (0304) 2 (0405) 3 (0506)
N 32,820 41,052 62,071
Poor RC 1,403
(4.27%)
1,428
(3.48%)
1,885
(3.04%)
But OK decoders
735
(2.25%)
834
(2.03%)
1162
(1.87%)
And OK vocabulary
72
(0.22%)
70
(0.17%)
93
(0.15%)
Comparing Second-Grade Results to First-Grade Results
• Second-grade results differ a little:– Percentage of children who are poor at reading
comprehension yet adequate at decoding is about 2 percent, compared to .5 percent in first grade.
• But identical when adequate vocabulary is also imposed:– Less than .2 percent for both.
What about Third Grade?
• We could not do identical study because we don’t have nonsense word fluency (NWF) as decoding measure for third grade.
• But we do have Gates-McGinnite reading vocabulary as a combined measure of decoding and vocabulary.
Three Third-Grade Cohorts
Cohort 1 (0304) 2 (0405) 3 (0506)
N 36,925 42,546 65,344
Poor RC 568
(1.54%)
1216
(2.86%)
2046
(3.13%)
But OK decoder and OK vocab
34
(0.09%)
137
(0.32%)
138
(0.21%)
What About Less Severe Reading Comprehension Problems
• Select if SAT 10 =< 20th %ile.
• Require decoding and vocabulary 1 standard deviation higher (56th %ile) or just somewhat higher (40th %ile).
• Representative second-grade results for most lenient (40th %ile) criterion.
Less Severe 2nd Grade Cohorts
Cohorts 1 (0304) 2 (0405) 3 (0506)
N 32,820 41,052 62,071
Poor RC
=<20th %ile
7,857
(23.12%)
7,603
(18.52%)
10,776
(17.36%)
Decoding
=>40th %ile
1786
(5.44%)
2257
(5.50%)
3479
(5.56%)
Vocabulary
=>40th %ile
242
(0.66%)
1060
(2.50%)
686
(1.05%)
What These Results Say
• Specific reading comprehension not associated with presence of primary (decoding) or secondary (vocabulary) causes is exceedingly rare:– Less than 0.1 % of first-graders.– Less than 0.2 % of second-graders.– Less than 0.3 % of third-graders.
What These Results Don’t Say
• Results don’t imply that individuals poor at reading comprehension don’t also show deficits in various tertiary factors.
What These Results Don’t Say
• Results don’t imply that individuals poor at reading comprehension don’t also show deficits in various tertiary factors.– decoding gains from intervention rarely
translate into equivalent gains in comprehension.
Conclusions
• Individuals with tertiary causes (e.g., metacognitive deficiency) in absence of primary (decoding) and secondary (vocabulary) causes are rare.
Conclusions
• For screening purposes, a combination of decoding and vocabulary should be remarkably effective.
Conclusions
• Its worse not to know the words (primary problem in decoding or secondary problem in decoding vocabulary) than to not know whether you know the words (tertiary problem in metacognition).
Gender Differences in Reading Disability: Reasons to Care
• 1. An active and controversial issue. • 2. Gender bias in identification and
provision of services may be pervasive.• 3. Implications for theories of etiology.• 4. New approaches to identification
being considered potentially could mitigate referral bias if it exists.
• Ex. Universal screening as front end of RTI.
Current Controversy: Two Views
• 1. Male vulnerability is a myth.– observed ratios of 2:1 or 3:1 in clinics and
classrooms reflect referral bias.– true ratio is 1:1 or boys favored only
minimally.
Key Study: Shaywitz et al. (1990)
– Obtained both school-identified ratio and objective ratio for same sample.
• Statistically significant ratio of 2.2:1 found for school identified ratio.
• Non-significant ratio of 1.4:1 found for objective criteria.
– Sample size modest however (18 boys versus 13 girls with RD).
• A 2:1 ratio would not even be significant.
Current Controversy: Two Views
• 2. Male vulnerability is real.– males roughly twice as likely to be
affected.
Key Supporting Studies
• Liederman et al., 2005, review of the literature.– Ratios ranged from 1.2:1 to 6.8:1.– Concluded that true ratio was between
1.7:1 and 2:1.
Key Questions
• 1. What is the magnitude of male vulnerability for reading disability if it exists?– Answered by examining gender ratios for
research-based operational definitions applied universally.
Key Questions
• 2. How accurately does school-identification predict research-based identification?– Answered by classification analyses that
use school-identification to predict research-based identification.
Key Questions
• 3. What is the magnitude of referral bias if it exists? Answered by three empirical analyses:– A. Magnitude of difference in gender ratios for
school-identified versus research identified samples.
– B. Less overlap between school- and research-identification for boys than for girls.
– C. Lower mean performance for girls compared to boys for school-identified samples.
Key Questions
• 4. Do gender ratios vary as a function of:– A. Level of severity of reading problem?
• Studies differ on level used.
– B. Whether operational definition is based on low-achievement or IQ-achievement discrepancy?
• Some suggestion that gender differences occur for IQ-discrepancy but not low-achievement definitions.
– C. The kind of reading measure examined?
Sample
• Five cohorts of beginning second-grade students in Reading First schools in Florida (03/04, 04/05, 05/06, 06/07, 07/08).
• N = 491…
Sample
• Five cohorts of beginning second-grade students in Reading First schools in Florida (03/04, 04/05, 05/06, 06/07, 07/08).
• N = 491 thousand (491,000)!
Measures
• Phonological decoding: Nonword fluency (DIBELS).
• Reading connected test: Oral reading fluency (DIBELS).
• IQ proxy: Peabody Picture Vocabulary Test
Remaining Design Issues
• Compared four levels of severity:– 30th percentile.– 15th percentile.– 5th percentile.– 3rd percentile.
Remaining Design Issues
• Low achievement operational definition determined by identifying the 4 target percentiles in the distributions.
• Discrepancy operational definition determined by regression reading measures of PPVT and identifying same target percentiles in the distributions of residuals.
Key Question 1: Male Vulnerability?
Gender Ratios for Research-Based Definitions
• Ratios greater than 1:1 quantity extent of male vulnerability.
Gender Ratios: Low-Achievement Definition, Nonword Fluency
Level of Severity Gender Ratio
30th %-ile 1.2:1
15th %-ile 1.3:1
5th %-ile 1.5:1
3rd %-ile 1.7:1
Gender Ratios: Discrepancy Definition, Nonword Fluency
Level of Severity Gender Ratio
30th %-ile 1.2:1
15th %-ile 1.4:1
5th %-ile 1.6:1
3rd %-ile 1.7:1
Gender Ratios: Low-Achievement Definition, Oral Reading Fluency
Level of Severity Gender Ratio
30th %-ile 1.4:1
15th %-ile 1.7:1
5th %-ile 2.1:1
3rd %-ile 2.2:1
Gender Ratios: Discrepancy Definition, Oral Reading Fluency
Level of Severity Gender Ratio
30th %-ile 1.6:1
15th %-ile 1.9:1
5th %-ile 2.4:1
3rd %-ile 2.6:1
Comparing Low-Achievement and Discrepancy Definitions: Nonword Fluency
Severity Low-Achievement
Discrepancy
30th %-ile 1.2:1 1.2:1
15th %-ile 1.3:1 1.4:1
5th %-ile 1.5:1 1.6:1
3rd %-ile 1.7:1 1.7:1
Comparing Low-Achievement and Discrepancy Definitions: ORF
Severity Low-Achievement
Discrepancy
30th %-ile 1.4:1 1.6:1
15th %-ile 1.7:1 1.9:1
5th %-ile 2.1:1 2.4:1
3rd %-ile 2.2:1 2.6:1
Key Question 2: Accuracy of School-Based
Identification?
School-Identification as Learning Disabled
• 5.1 percent of all second-grade students.
• Carried out classification study using school-identification to predict membership in 5th %-ile research-based group.
Key Classification Statistics
• 1. Sensitivity.– Proportion of research-based reading
disabled correctly classified by school-based determination.
• 2. Specificity.– Proportion of research-based reading non-
disabled correctly classified by school-based determination.
Key Classification Statistics
• 3. Positive Predictive Value.– Proportion of school-identified reading
disabled who actually were according to research-based criteria.
• 4. Negative Predictive Value.– Proportion of school-identified non-reading
disabled who actually were according to research-based criteria.
Guidelines for Interpretation of Classification Statistics
• Level 3 (highest level of support)– Sensitivity, specificity, and positive predictive
value >= .75.
• Level 2 (moderate support)– Sensitivity, specificity, and positive predictive
value >= .70.
• Level 1 (lowest but still supportive)– Sensitivity and specificity or sensitivity and positive
predictive value >= .70.
Classification Results
Definition Sensitivity Specificity PPV NPV
NWF lo .18 .96 .19 .95
NWF dis .14 .96 .14 .96
ORF lo .26 .96 .26 .96
ORF dis .20 .96 .20 .96
Classification Results
• Accuracy of school-based identification poor for research-based identification of reading disability.
• Accuracy is high for determination of non reading disability, but this reflects base rate of 5%.
Key Question 3: Referral Bias?
Comparing Gender Ratios for School- and Research-Identified Samples
• Extent to which gender ratios for school-identified samples exceed those for research-identified quantifies extent of referral bias.
Comparing Gender Ratios for School- and Research-Identified Samples
• Gender ratio for school-identified sample: 2.25 to 1.– Exceeds gender ratio for nonword fluency
(1.5:1 or 1.6:1 depending on definition, for same level of severity).
– Comparable to gender ratio for oral reading fluency (2.1:1 or 2.4:1).
Is Accuracy of School-Identification Higher for Girls than Boys?
• If boys are referred for behavior and other issues and girls for reading problems, accuracy of school-based identification should be higher for girls than for boys.
Is Accuracy of School-Identification Higher for Girls than Boys?
Definition Sensitivity for
Girls
Sensitivity for Boys
NWF lo .12 .22
NWF dis .08 .18
ORF lo .20 .29
ORF dis .14 .22
Lower Mean Performance for School-Identified Girls than Boys?
• If boys are referred for behavior and other issues and girls for reading problems, girls should be poorer readers than boys in school-identified samples.
Lower Mean Performance for School-Identified Girls than Boys?
Measure Group Mean (SD) t
NWF lo Girls 40.3 (21.6) -0.97
Boys 40.1 (24.4)
NMW dis Girls -13.6 (20.8) -7.09***
Boys -17.0 (23.1)
Lower Mean Performance for School-Identified Girls than Boys?
Measure Group Mean (SD) t
ORF lo Girls 31.7 (23.6) -10.51***
Boys 28.3 (23.7)
ORF dis Girls -17.8 (22.3) -13.24***
Boys -24.1 (22.5)
Conclusions
• 1. Male vulnerability is real and quantifiable.– Increases with level of severity of reading
problem.– Greater for broader-based oral reading fluency
(2.4:1 at 3rd %-ile) than for more narrowly-based nonword fluency (1.7:1 at 3rd %-ile).
Conclusions
• 2. Accuracy of school-identification is abysmal for research-based criteria.– Sensitivities ranged from .14 to .26 (far below
minimally acceptable value of .70).
Conclusions
• 3. Little support for substantial referral bias.– Gender ratios for school-identified samples
greater than research-based for nonword fluency but not for oral reading fluency.
– School identification is not more accurate for girls than for boys, but just the opposite.
– School-identified girls are not poorer readers than boys, but just the opposite.
Conclusions
• 4. Gender ratios not greater for discrepancy than for low achievement definitions for nonword fluency; only marginally higher (from 0.2:1 to 0.4:1) higher for oral reading fluency.
Questions?