large scale studies of dyslexia in florida richard k. wagner and yusra ahmed florida state...

Large Scale Studies of Dyslexia in Florida

Richard K. Wagner and Yusra Ahmed

Florida State University and FCRR

NIH Multidisciplinary Learning Disabilities Center (P50 HD052120)

Alternative Approaches

• 1. Typical research study.

• 2. Meta-analysis.

• 3. Large-scale study.

Large-Scale Study of Incidence of Specific Reading Comprehension

Disability

Personal Interest in Reading Comprehension Problems

• Comprehension errors run in my family.

Ellis Island

Question on a Form

• If admitted to this country, would you advocate overthrow of the government of the United States by force or violence?

Question on a Form

• If admitted to this country, would you advocate overthrow of the government of the United States by force or violence?

• violence

Reported Incidence of Reading Comprehension Disability

• 10 percent of 7-11 year-olds are poor at reading comprehension despite being accurate and fluent at decoding (Nation, 2004).

Many Possible Causes

• “There is room for lots of things to go wrong when comprehension fails” (Perfetti, 1994, p. 885, cited by Nation, 2005).

Possible Causes (Science of Reading: A Handbook)

• 1. Decoding difficulties.• 2. Difficulties with meaning (vocabulary).• 3. Difficulty with syntax.• 4. Limitations in working memory.• 5. Poor inference making.• 6. Inadequate comprehension monitoring.• 7. Limited prior domain knowledge.• 8. Insensitivity to text structure.

Reading Comprehension Task

• When the next slide appears, read the text as quickly as you can and summarize the passage in a brief sentence.

• Ready?

Answer a Simple Question

• Is necrobiosis a conceivable source of orogeny or of a pomiferous pompelmous?

Simple Question

• Is necrobiosis conceivably related to orogeny or to the development of a pomiferous pompelmous?

• Obviously the normal death of cells (necrobiosis) is not related to mountain building (orogeny), but is part of the process of the development of fruit bearing (pomiferous) trees (pompelmous).

Ordering Causes by Severity of Consequences for Comprehension

• Primary– Decoding difficulties.

• Secondary– Difficulties with meaning (vocabulary).

• Tertiary– Difficulty with syntax; limitations in working memory;

poor inference making; inadequate comprehension monitoring; limited prior domain knowledge; insensitivity to text structure.

Present Studies

• Question Addressed:– What is the incidence of reading comprehension

disability not attributable to primary or secondary causes?

• Design feature:– Attempted to address issue of small samples

sizes of typical studies by using the PMRN (progress monitoring and reporting network) database.

Study Design

• 1. Identify individuals who are poor at reading comprehension.– Score at or below 5th percentile on Stanford

Achievement Test (SAT-10) Reading Comprehension.

Study Design

• 2. Determine how many individuals are poor at comprehension yet adequate at decoding.– SAT 10 at or below 5th percentile;– DIBELS Nonsense Word Fluency (NWF)

greater than or equal to 25th percentile.

Study Design

• 2. Determine how many individuals are poor at comprehension yet adequate at decoding and vocabulary.– SAT 10 at or below 5th percentile;– DIBELS Nonsense Word Fluency (NWF)

greater than or equal to 25th percentile.– Peabody Picture Vocabulary Test (PPVT)

greater than or equal to 25th percentile.

Out of a First Grade-Cohort 1 (N = 35,314)….

How Many Were Poor at…

• Reading comprehension (SAT 10 < 5th%)– 1,669 (4.73%) out of 35,314

How Many Were Poor at…

• Reading comprehension (SAT 10 < 5th %)– 1,669 (4.73%) out of 35,314

• yet adequate at decoding (nonword fluency >= 25th %)?

How Many Were Poor at…

• Reading comprehension (SAT 10 < 5th %)– 1,669 (4.73%) out of 35,314

• yet adequate at decoding (nonword fluency >= 25th %)?– only 85 (0.24%) out of 35,314!

How Many Were Poor at…

• Reading comprehension (SAT 10 < 5th %)– 1,669 (4.73%) out of 35,314

• yet adequate at decoding (nonword fluency >= 25th %)?– only 85 (0.24%) out of 35,314!

• and in vocabulary (PPVT >= 25th %)?

How Many Were Poor at…

• Reading comprehension (SAT 10 < 5th %)– 1,669 (4.73%) out of 35,314

• yet adequate at decoding (nonword fluency >= 25th %)?– only 85 (0.24%) out of 35,314!

• and in vocabulary (PPVT >= 25th %)?– only 23 (0.07%) out of 35,314!

Surprising Result: Virtually All Had Problems in Decoding

• Sample size was 35,314.

• But it was a single study. Results need to be replicated.

So We Did Replicate: Not One, Not Two, but…

Three First-Grade Cohorts

Cohorts 1 (0304) 2 (0405) 3 (0506)

N 35,314 43,712 64,645

Poor RC 1,669

(4.73%)

1,721

(3.94%)

1,896

(2.93%)

But OK decoders

85

(0.24%)

197

(0.45%)

266

(0.41%)

And OK vocabulary

23

(0.07%)

69

(0.16%)

81

(0.13%)

First Grade Results

• Of 150,000 6-year-olds, only .2 to .5 percent were poor at comprehension yet adequate at decoding.

• May be nature of reading comprehension at age 6—decoding explains about everything.

Three Second-Grade Cohorts

Cohorts 1 (0304) 2 (0405) 3 (0506)

N 32,820 41,052 62,071

Poor RC 1,403

(4.27%)

1,428

(3.48%)

1,885

(3.04%)

But OK decoders

735

(2.25%)

834

(2.03%)

1162

(1.87%)

And OK vocabulary

72

(0.22%)

70

(0.17%)

93

(0.15%)

Comparing Second-Grade Results to First-Grade Results

• Second-grade results differ a little:– Percentage of children who are poor at reading

comprehension yet adequate at decoding is about 2 percent, compared to .5 percent in first grade.

• But identical when adequate vocabulary is also imposed:– Less than .2 percent for both.

What about Third Grade?

• We could not do identical study because we don’t have nonsense word fluency (NWF) as decoding measure for third grade.

• But we do have Gates-McGinnite reading vocabulary as a combined measure of decoding and vocabulary.

Three Third-Grade Cohorts

Cohort 1 (0304) 2 (0405) 3 (0506)

N 36,925 42,546 65,344

Poor RC 568

(1.54%)

1216

(2.86%)

2046

(3.13%)

But OK decoder and OK vocab

34

(0.09%)

137

(0.32%)

138

(0.21%)

What About Less Severe Reading Comprehension Problems

• Select if SAT 10 =< 20th %ile.

• Require decoding and vocabulary 1 standard deviation higher (56th %ile) or just somewhat higher (40th %ile).

• Representative second-grade results for most lenient (40th %ile) criterion.

Less Severe 2nd Grade Cohorts

Cohorts 1 (0304) 2 (0405) 3 (0506)

N 32,820 41,052 62,071

Poor RC

=<20th %ile

7,857

(23.12%)

7,603

(18.52%)

10,776

(17.36%)

Decoding

=>40th %ile

1786

(5.44%)

2257

(5.50%)

3479

(5.56%)

Vocabulary

=>40th %ile

242

(0.66%)

1060

(2.50%)

686

(1.05%)

What These Results Say

• Specific reading comprehension not associated with presence of primary (decoding) or secondary (vocabulary) causes is exceedingly rare:– Less than 0.1 % of first-graders.– Less than 0.2 % of second-graders.– Less than 0.3 % of third-graders.

What These Results Don’t Say

• Results don’t imply that individuals poor at reading comprehension don’t also show deficits in various tertiary factors.

What These Results Don’t Say

• Results don’t imply that individuals poor at reading comprehension don’t also show deficits in various tertiary factors.– decoding gains from intervention rarely

translate into equivalent gains in comprehension.

Conclusions

• Individuals with tertiary causes (e.g., metacognitive deficiency) in absence of primary (decoding) and secondary (vocabulary) causes are rare.

Conclusions

• For screening purposes, a combination of decoding and vocabulary should be remarkably effective.

Conclusions

• Its worse not to know the words (primary problem in decoding or secondary problem in decoding vocabulary) than to not know whether you know the words (tertiary problem in metacognition).

Gender Differences in Reading Disability: Reasons to Care

• 1. An active and controversial issue. • 2. Gender bias in identification and

provision of services may be pervasive.• 3. Implications for theories of etiology.• 4. New approaches to identification

being considered potentially could mitigate referral bias if it exists.

• Ex. Universal screening as front end of RTI.

Current Controversy: Two Views

• 1. Male vulnerability is a myth.– observed ratios of 2:1 or 3:1 in clinics and

classrooms reflect referral bias.– true ratio is 1:1 or boys favored only

minimally.

Key Study: Shaywitz et al. (1990)

– Obtained both school-identified ratio and objective ratio for same sample.

• Statistically significant ratio of 2.2:1 found for school identified ratio.

• Non-significant ratio of 1.4:1 found for objective criteria.

– Sample size modest however (18 boys versus 13 girls with RD).

• A 2:1 ratio would not even be significant.

Current Controversy: Two Views

• 2. Male vulnerability is real.– males roughly twice as likely to be

affected.

Key Supporting Studies

• Liederman et al., 2005, review of the literature.– Ratios ranged from 1.2:1 to 6.8:1.– Concluded that true ratio was between

1.7:1 and 2:1.

Key Questions

• 1. What is the magnitude of male vulnerability for reading disability if it exists?– Answered by examining gender ratios for

research-based operational definitions applied universally.

Key Questions

• 2. How accurately does school-identification predict research-based identification?– Answered by classification analyses that

use school-identification to predict research-based identification.

Key Questions

• 3. What is the magnitude of referral bias if it exists? Answered by three empirical analyses:– A. Magnitude of difference in gender ratios for

school-identified versus research identified samples.

– B. Less overlap between school- and research-identification for boys than for girls.

– C. Lower mean performance for girls compared to boys for school-identified samples.

Key Questions

• 4. Do gender ratios vary as a function of:– A. Level of severity of reading problem?

• Studies differ on level used.

– B. Whether operational definition is based on low-achievement or IQ-achievement discrepancy?

• Some suggestion that gender differences occur for IQ-discrepancy but not low-achievement definitions.

– C. The kind of reading measure examined?

Sample

• Five cohorts of beginning second-grade students in Reading First schools in Florida (03/04, 04/05, 05/06, 06/07, 07/08).

• N = 491…

Sample

• Five cohorts of beginning second-grade students in Reading First schools in Florida (03/04, 04/05, 05/06, 06/07, 07/08).

• N = 491 thousand (491,000)!

Measures

• Phonological decoding: Nonword fluency (DIBELS).

• Reading connected test: Oral reading fluency (DIBELS).

• IQ proxy: Peabody Picture Vocabulary Test

Remaining Design Issues

• Compared four levels of severity:– 30th percentile.– 15th percentile.– 5th percentile.– 3rd percentile.

Remaining Design Issues

• Low achievement operational definition determined by identifying the 4 target percentiles in the distributions.

• Discrepancy operational definition determined by regression reading measures of PPVT and identifying same target percentiles in the distributions of residuals.

Key Question 1: Male Vulnerability?

Gender Ratios for Research-Based Definitions

• Ratios greater than 1:1 quantity extent of male vulnerability.

Gender Ratios: Low-Achievement Definition, Nonword Fluency

Level of Severity Gender Ratio

30th %-ile 1.2:1

15th %-ile 1.3:1

5th %-ile 1.5:1

3rd %-ile 1.7:1

Gender Ratios: Discrepancy Definition, Nonword Fluency

Level of Severity Gender Ratio

30th %-ile 1.2:1

15th %-ile 1.4:1

5th %-ile 1.6:1

3rd %-ile 1.7:1

Gender Ratios: Low-Achievement Definition, Oral Reading Fluency

Level of Severity Gender Ratio

30th %-ile 1.4:1

15th %-ile 1.7:1

5th %-ile 2.1:1

3rd %-ile 2.2:1

Gender Ratios: Discrepancy Definition, Oral Reading Fluency

Level of Severity Gender Ratio

30th %-ile 1.6:1

15th %-ile 1.9:1

5th %-ile 2.4:1

3rd %-ile 2.6:1

Comparing Low-Achievement and Discrepancy Definitions: Nonword Fluency

Severity Low-Achievement

Discrepancy

30th %-ile 1.2:1 1.2:1

15th %-ile 1.3:1 1.4:1

5th %-ile 1.5:1 1.6:1

3rd %-ile 1.7:1 1.7:1

Comparing Low-Achievement and Discrepancy Definitions: ORF

Severity Low-Achievement

Discrepancy

30th %-ile 1.4:1 1.6:1

15th %-ile 1.7:1 1.9:1

5th %-ile 2.1:1 2.4:1

3rd %-ile 2.2:1 2.6:1

Key Question 2: Accuracy of School-Based

Identification?

School-Identification as Learning Disabled

• 5.1 percent of all second-grade students.

• Carried out classification study using school-identification to predict membership in 5th %-ile research-based group.

Key Classification Statistics

• 1. Sensitivity.– Proportion of research-based reading

disabled correctly classified by school-based determination.

• 2. Specificity.– Proportion of research-based reading non-

disabled correctly classified by school-based determination.

Key Classification Statistics

• 3. Positive Predictive Value.– Proportion of school-identified reading

disabled who actually were according to research-based criteria.

• 4. Negative Predictive Value.– Proportion of school-identified non-reading

disabled who actually were according to research-based criteria.

Guidelines for Interpretation of Classification Statistics

• Level 3 (highest level of support)– Sensitivity, specificity, and positive predictive

value >= .75.

• Level 2 (moderate support)– Sensitivity, specificity, and positive predictive

value >= .70.

• Level 1 (lowest but still supportive)– Sensitivity and specificity or sensitivity and positive

predictive value >= .70.

Classification Results

Definition Sensitivity Specificity PPV NPV

NWF lo .18 .96 .19 .95

NWF dis .14 .96 .14 .96

ORF lo .26 .96 .26 .96

ORF dis .20 .96 .20 .96

Classification Results

• Accuracy of school-based identification poor for research-based identification of reading disability.

• Accuracy is high for determination of non reading disability, but this reflects base rate of 5%.

Key Question 3: Referral Bias?

Comparing Gender Ratios for School- and Research-Identified Samples

• Extent to which gender ratios for school-identified samples exceed those for research-identified quantifies extent of referral bias.

Comparing Gender Ratios for School- and Research-Identified Samples

• Gender ratio for school-identified sample: 2.25 to 1.– Exceeds gender ratio for nonword fluency

(1.5:1 or 1.6:1 depending on definition, for same level of severity).

– Comparable to gender ratio for oral reading fluency (2.1:1 or 2.4:1).

Is Accuracy of School-Identification Higher for Girls than Boys?

• If boys are referred for behavior and other issues and girls for reading problems, accuracy of school-based identification should be higher for girls than for boys.

Is Accuracy of School-Identification Higher for Girls than Boys?

Definition Sensitivity for

Girls

Sensitivity for Boys

NWF lo .12 .22

NWF dis .08 .18

ORF lo .20 .29

ORF dis .14 .22

Lower Mean Performance for School-Identified Girls than Boys?

• If boys are referred for behavior and other issues and girls for reading problems, girls should be poorer readers than boys in school-identified samples.

Lower Mean Performance for School-Identified Girls than Boys?

Measure Group Mean (SD) t

NWF lo Girls 40.3 (21.6) -0.97

Boys 40.1 (24.4)

NMW dis Girls -13.6 (20.8) -7.09***

Boys -17.0 (23.1)

Lower Mean Performance for School-Identified Girls than Boys?

Measure Group Mean (SD) t

ORF lo Girls 31.7 (23.6) -10.51***

Boys 28.3 (23.7)

ORF dis Girls -17.8 (22.3) -13.24***

Boys -24.1 (22.5)

Conclusions

• 1. Male vulnerability is real and quantifiable.– Increases with level of severity of reading

problem.– Greater for broader-based oral reading fluency

(2.4:1 at 3rd %-ile) than for more narrowly-based nonword fluency (1.7:1 at 3rd %-ile).

Conclusions

• 2. Accuracy of school-identification is abysmal for research-based criteria.– Sensitivities ranged from .14 to .26 (far below

minimally acceptable value of .70).

Conclusions

• 3. Little support for substantial referral bias.– Gender ratios for school-identified samples

greater than research-based for nonword fluency but not for oral reading fluency.

– School identification is not more accurate for girls than for boys, but just the opposite.

– School-identified girls are not poorer readers than boys, but just the opposite.

Conclusions

• 4. Gender ratios not greater for discrepancy than for low achievement definitions for nonword fluency; only marginally higher (from 0.2:1 to 0.4:1) higher for oral reading fluency.

Questions?