slide 1 national center on educational outcomes (nceo) snapshots: what we know about state...

National Center on Educational Outcomes (NCEO)

Snapshots:What We Know about State

Assessment Practices

Martha L. Thurlow

National Center on Educational Outcomes

[email protected]

www.nceo.info

Assessment Options

• General Assessments

• General Assessments with Accommodations

• Alternate Assessments based on Grade-Level Achievement Standards

• Alternate Assessments based on Alternate Achievement Standards

• Alternate Assessments based on Modified Achievement Standards


General Assessments

• Participation

• Performance

• Assessment Development

• Narrowing the Achievement Gap


Participation

• Public reporting of participation provides one view – most often NCLB view (students count only if in school for one year)

• Reporting to the Secretary of Education, as required by law gives another view – based on total population of students with IEPs


Reading Assessment Participation Rates in Elementary School: Percent Participation is of IEP Enrollment (Includes Regular and Alternate Assessment)

95% - 105% (n=45 regular states and 7 unique states)

< 95% (n=1 regular states and 2 unique states)

md =missing data (n=3 regular state and 1 unique state)

Key

> 105% (n=1 regular states and 0 unique states)

99

100

99

99

98100

99

100 100

100

99

100

98

98

99

100

100

100

99

100

99

99

98

100

99

md

99

99100 99

100

95

92

99

99

99

100

98

99

md

107

100

100

9999

md

99

98

98

99

AS

CNMI

DC

FSM

PR

GU

Palau

BIE

RMI

VI

9594

100

95

md96

100

99

100

84

2005-2006


From NCEO analysis of APR data – Thurlow et al., 2008


Reading Assessment Participation Rates in High School: Percent Participation is of IEP Enrollment (Includes Regular and Alternate Assessment)



md =missing data (n=2 regular states and 1 unique state)

Key

> 105% (n=0 regular states and 0 unique states)

84

86

97

93

9198

96

97 99

97

98

98

92

94

95

97

100

97

98

98

98

93

87

100

96

md

md

97 98 97

100

91

90

97

96

100

96

98

96

84

89

100

98

95 96

97

98

89

96

99

AS

CNMI

DC

FSM

PR

GU

Palau

BIE

RMI

VI

2005-2006

100

62

75

72

46

84

100

96

80

md


Participation Reporting Practices Changing

Situations Under Which Students are Counted as Nonparticipants (# of States):

2007 2005Did not take assessment47 35

Did not complete enough items to score 17 6

Accommodations produced invalid results 18 8

Tested at a lower grade than enrollment 16 2


Based on data from the 2007 NCEO Survey of States

Performance

• Public reporting of performance provides one view – most often NCLB view (students count only if in school for one year)

• Reporting to the Secretary of Education, as required by law gives another view – based on total population of students with IEPs



Reading Assessment Proficiency Rates in Elementary School: Percent Proficient of IEP Enrollment (Regular and Alternate Assessment)

Key – Percent Proficient

50% – 100% (n=22 regular states and 0 unique states)





23

26

29

65●

4645

31

50 32

58●

51

31

21

65

69

62

61●

70

37

21

51

49

14

21

48●

60

53● 45 58

65

44

54

51●

47

20

55

70●

35

md

33*

59

50

27 27

md

19

38●

50

23●

AS

CNMI

DC

FSM

PR

GU

Palau

BIE

RMI

VI

State had >1% proficient on alternate assessment based on alternate achievement standards

* State had >105% participation on assessments

2005-2006

6

17

30

17

md

2

0

33

16

0

0% –9% (n=0 regular states and 4 unique states)

53●

md = missing data (2 regular states and 1 unique state



Reading Assessment Proficiency Rates in High School: Percent Proficient of IEP Enrollment (Regular and Alternate Assessment)

14

31

25

17

3929

19

33 26*

57●

20

30●

9

50

58

20

44●

45

32

9

42

39

5

19

32

55●

md

8 3872●

81

24

12

56●

26

17

15

68●

20

44

11

21

45

40 24

40

15

16●

24●

32

AS

CNMI

DC

FSM

PR

GU

Palau

BIE

RMI

VI

State had >1% proficient on alternate assessment based on alternate achievement standards

* State had >105% participation on assessments

2005-2006

Key – Percent Proficient

16

13

19

7

16

2

0

19

8

md






0% –9% (n=4 regular states and 4 unique states)

md =missing data (n=1 regular state and 1 unique state


Assessment Development


How States Addressed Universal Design in General Assessment Development Process

(# of States):

Included in RFP for test development 39

Expert review 27

Think aloud methods as part of field test 2

Statistical analyses 10

“UDA” NOT addressed in test development 1


Assessment Development


Bias/Sensitivity Reviews – Representation of Disability Perspective

Disability No Disability

Representation Representation

Content Review 31 3

Bias Review33 2



Narrowing the Achievement Gap


Perceived Factors Related to Positive Achievement


Positive Trends in Performance


Data Availability for Positive Trend Factors



General Assessments with Accommodations

• Numbers Using Accommodations

• Accommodations Policies

• Hints & Tips for States


Accommodations

• States are required to report on the number of students using accommodations during the general assessment

• Not all states yet report these data; some report data for different types of accommodations than others


Reading Assessment Accommodation Rates in Elementary School: Percentage of Students with IEPs Taking the Regular Reading Assessment with Accommodations




Key

≥ 75% (n=3 regular states and 1 unique states)

11

41

52

md

3263

40

49 63

1

71

73

57

39

42

47

54

8

71

59

55

67

65

35

52

22

52

4520 62

66

52

69

47

36

47

33

61

53

md

72

69

80

7761

md

40

64

24

79

AS

CNMI

DC

FSM

PR

GU

Palau

BIE

RMI

VI

2005-2006

94

59

73

71

md

67

md

59

43

md

md =missing data (n=3 regular states and 3 unique states)



Reading Assessment Accommodation Rates in High School: Percentage of Students with IEPs Taking the Regular Reading Assessment with Accommodations

49

56

md

1843

37

23 41

0

59

59

81

35

45

44

53

5

70

59

53

66

28

22

47

md

md

620 61

46

73

65

53

27

50

0

43

42

md

51

59

79

7738

57

36

39

46

76

AS

CNMI

DC

FSM

PR

GU

Palau

BIE

RMI

VI

2005-2006

5

100

22

42

48

md

61

md

46

53

md



< 25% (n=8 regular states and 1 unique state)

Key

≥ 75% (n=4 regular states and 1 unique state)

md =missing data (n=4 regular states and 3 unique states)


Accommodation Policies Change

States with Read Aloud Question Changes in Policy – comparing 2003 to 2005 and 2007


Number of States

No changes since 2003 19One change since 2003 19Two changes since 2003 12

From NCEO Data Viewer at http://data.nceo.info


Alternate Assessments based on Grade-Level Achievement

Standards (AA-GLAS)

• Number of States Providing This Option

• Issue – A Misunderstood Option?


Reading Math

Elementary Middle

High

School Elementary Middle

High

School

Kansas 18% 21% 19% 15% 22% 19%

Louisiana 5% 10% 5% 5% 10% 5%

Massachusettsa 0%a 0%a 0%a 0%a 0%a 1%

Minnesota 8% 7% 6% 7% 7% 8%

Mississippia 0%a 0%a 0%a 0%a 0%a 0%

North Carolina 18% 16% 2% 15% 15% 2%

Texas 19% 20% 13% 22% 21% 14%

Virginia 10% 12% 0%a 8% 11% 0%a

Wisconsina 1% 0%a 0%a 1% 0%a 0%a

Percentage of Students with IEPs Participating in AA-GLAS

a Small percentage (< 0.5) participated n the grade level(s) and content area(s) represented as 0%.


Issue: AA-GLAS Misunderstood2005-06 2006-07*

Kansas Indiana

Louisiana Massachusetts

Massachusetts Montana

Minnesota North Carolina

Mississippi Virginia

North Carolina Washington

Texas West Virginia

VirginiaWisconsin



Alternate Assessments based on Alternate Achievement Standards

• Participation

• Performance


Reading Assessment Based on Alternate Achievement Standards Participation Rates in Elementary School: Percent Participation is of IEP Enrollment (both Alternate and Out-of-level)


>9% (n=10 regular states and 2 unique states)

Key

0% − 9% (n=38 regular states and 7 unique states)

8

6

6

21

206

6

6 9

50

6

4

8

8

5

6

12

3

5

5

7

4

4

5

12

23

6

187 6

8

5

6

10

6

12

4

9

8

md

8

6

6

7 3

md

34

6

5

7

AS

CNMI

DC

FSM

PR

GU

Palau

BIE

RMI

VI

2005-2006

0

4

22

7

md

8

18

4

6

0



Reading Assessment Based on Alternate Achievement Standards Participation Rates in High School: Percent Participation is of IEP Enrollment (both Alternate and Out-of-level)


>9% (n=17 regular states and 0 unique states)

0% − 9% (n=33 regular states and 9 unique states)

DC

FSM

PR

GU

Palau

RMI

VI

4

0

2

0

9

9

md

8

5

9

22

31 8

7

8 10

47

6

24

10

6

9

11

19

5

4

6

9

6

2

7

8

22

8

0 10 15

4

7

0

8

7

5

20

11

6

8

11

5

7 3

23

4

13

11

6

AS

CNMI

BIE

8

2005-2006

0

3

6

KeyFrom NCEO analysis of APR data – Thurlow et al., 2008

States Exceeding the 1% Cap

Delaware* Michigan Oregon**

Florida Mississippi Texas

Georgia North Dakota Virginia

Louisiana* Ohio Wisconsin**

Massachusetts**


* High School Only** Elementary School Only


National Center on Educational Outcomes (NCEO)National Center on Educational Outcomes (NCEO)

Alternate Assessments based on Modified Achievement Standards

• Number of States Providing This Option

• Criteria for Participation

• Nature of Assessments


Number of States with AA-MAS

July 2007 – 5 states indicated that they had an AA-MAS (all were developed prior to the April, 2007 regulation:

• Kansas• Louisiana• North Carolina• North Dakota• Oklahoma

In addition, Maryland provided information on its Web site about the development a AA-MAS.

From NCEO Synthesis Report 67 – Lazarus et al., 2007

States Considering AA-MAS



Criteria for Participation


CriteriaNo. of States

Student has IEP 6

Multiple years behind grade level expectations 4

Not based on categorical label 4

Does not have significant cognitive disabilities 4Not due to excessive absences or to social, cultural, environmental, or economic factors

4

Learning grade level content 3

Not based on placement setting 2

Previously scored at unsatisfactory level 2

Eligibility Criteria in 6 States with AA-MAS*

* Includes Maryland



NCEO Survey Showed States Identifying Many Categories As Most Likely to Participate in AA-MAS


Nature of Assessments


State Portfolio Multiple Choice

Constructed Response

Writing Prompt

KS XLA X X XMD XNC XND XOK X X

Total 1 5 1 2

Type of Assessment/Questions in 6 States with AA-MAS



Change No. of States

Distractor Removed 4

Fewer Items 4

Simplified Language 4

Shorter Passages 3

Segmentation of Passages 1

Nature of Changes to Items in 6 States with AA-MAS



Accommodation No. of States

Fewer items/page 3

Larger font size 3

Calculator 2

Breaks as needed 2

Key text underlined/bolded 1

Accommodations Incorporated into Design in 6 States with AA-MAS



Characteristics of Assessment in 40 Planning to Develop and AA-MAS

Characteristic Number of States

Include only multiple choice questions 11Reduce total # of items or time to complete 25Simplify vocabulary 26Use non-traditional items or formats 16Use shorter or fewer reading passages 29Use fewer items or fewer answer choices 23


Stay Tuned for More Snapshots


Visit NCEO’s Web site for more and continuing information on

assessment options:

www.nceo.info

slide 1 national center on educational outcomes (nceo) snapshots: what we know about state...

Documents