Design of the 2011 NAEP-Design of the 2011 NAEP-TIMSS Linking Study TIMSS Linking Study

Andrew KolstadAndrew KolstadSenior Technical Advisor to the Assessment DivisionSenior Technical Advisor to the Assessment Division

National Center for Education StatisticsNational Center for Education Statistics

For states and school districts, to project TIMSS mathematics and science scores from performance on NAEP (without the cost of participating in TIMSS)

2

Goals of the study in 2011Goals of the study in 2011Goals of the study in 2011Goals of the study in 2011

Previous NAEP / TIMSS linkings - 1Previous NAEP / TIMSS linkings - 1

Johnson’s statistical linkage to 1996 state NAEP

• No students took both NAEP and TIMSS assessments

• TIMSS-equivalent mathematics and science scores predicted from NAEP state scores using “statistical moderation” (matching score distributions)

• Evaluated with TIMSS in Minnesota and Colorado 1995

o The actual grade 8 means for Minnesota fell within the confidence intervals

o The actual grade 4 means for both Colorado and Minnesota were higher than the predicted means

3

Previous NAEP / TIMSS linkings - 2Previous NAEP / TIMSS linkings - 2

Johnson and Cohen’s projection linkage to 2000 grade 8 state NAEP mathematics and science• A linking sample of students was assessed by NAEP in the

winter of 2000 and again by TIMSS in the spring of 2000

• TIMSS-equivalent mathematics and science scores predicted from NAEP state scores using “statistical projection” (estimating the average relationship between a student’s score on NAEP and on TIMSS and applying that relationship to state NAEP scores)

• Evaluated with grade 8 TIMSS in twelve states in 1999o The performance differed: the actual grade 8 means generally fell above

the confidence intervals for the predicted means

• No TIMSS score projections were published4

Previous NAEP / TIMSS linkings - 3Previous NAEP / TIMSS linkings - 3

Phillip’s statistical moderation linkage in 2007

• International grade 8 TIMSS-equivalent scores of NAEP achievement level cut scores projected using “statistical moderation” (no student took both assessments)

• Based on national distributions of NAEP 2000 mathematics and science scores matched to U.S. score distributions on o TIMSS 1999 assessment

o TIMSS 2003 assessment

• Result: Estimated percentage of basic, proficient, and advanced students in each country that participated in the 1999 TIMSS and 2003 TIMSS studies

5

Improving on previous linking designs Improving on previous linking designs

How the current linking study differs

1. Both NAEP and TIMSS in 2011, the same year this time

2. Two linking samples of students; no student assessed in both winter and spring

3. Two booklet designs; one looks like NAEP in the winter and the other looks like TIMSS in the spring

4. Projected TIMSS mathematics and science scores predicted from NAEP state scores using “statistical projection”

o (estimating the average relationship between a student’s score on NAEP and on TIMSS and applying that relationship to state NAEP scores)

5. Evaluated with grade 8 TIMSS in eight states in 20116

Components in linking NAEP & TIMSSComponents in linking NAEP & TIMSS

1. NAEP state-level science and mathematics data needed as a basis for projecting TIMSS mathematics and science scores○ For the first time in NAEP history, all 50 states volunteered to

participate in a voluntary state NAEP

2. TIMSS state-level grade 8 data needed to check the validity of the linking function

3. Linking samples of students will provide responses to items from both assessments; needed to estimate relationships○ No students will take NAEP in the winter and TIMSS in the spring, to

avoid the decline in scores found in the previous study

7

Booklet designs for NAEP & TIMSSBooklet designs for NAEP & TIMSS

8

Main study booklets

NAEP first block second blockBQ1

BQ2

TIMSS first block       second block Brk third block       fourth blockBrk BQ

Assessment time 5 10 15 20 25 30 35 40 45 50 55 60 65 70 75 80 85 90 95

100 … 130

 

Legend          

   

NAEP blocks

   

   

TIMSS blocks

                 

• NAEP booklets contain either science or mathematics, but not both

• TIMSS blocks contain both science and mathematics blocks

Math & Science

Each of the 14 booklets includes two blocks of mathematics and two blocks of science items

1 M01 M02 S01 S02

2 S02 S03 M02 M03

3 M03 M04 S03 S04

4 S04 S05 M04 M05

5 M05 M06 S05 S06

6 S06 S07 M06 M07

7 M07 M08 S07 S08

8 S08 S09 M08 M09

9 M09 M10 S09 S10

10 S10 S11 M10 M11

11 M11 M12 S11 S12

12 S12 S13 M12 M13

13 M13 M14 S13 S14

14 S14 S01 M14 M01

Each block appears in two booklets

TIMSS assessment bookletsTIMSS assessment booklets

9

Linking study approachLinking study approach

1. Administer braided booklets in both testing windowso NAEP in Winter (January to March, 2011) o TIMSS in Spring (April to June, 2011)

2. Select national samples independent of the main NAEP and TIMSS sampleso 10,000 students in NAEP in Winter (half science, half math) o 7,500 students in TIMSS in Spring (half science, half math)

3. Collect the data using NAEP field staff in winter and TIMSS field staff in spring (Westat is the contractor for both)

4. Score the items using the same scoring staff with the same training and QC procedures that NAEP and TIMSS use

10

Booklet Designs for NAEP-TIMSS Linking Studies

NAEP window first block second block  

BQ1

BQ2

first block   second block  

BQ1

BQ2

TIMSS window first block       second block  

Brk third block       fourth block BQ   

first block       second block  Brk third block       fourth block BQ   

Assessment time 5 10 15 20 25 30 35 40 45

50 55 60 65 70 75 80 85 90 95

100…

115

 

Legend          

   

NAEP blocks

   

   

TIMSS blocks

                 

Booklet designs for braided studyBooklet designs for braided study

11

NAEP-window booklets contain either science or mathematics blocks, but not both

• TIMSS-window booklets contain both science and mathematics TIMSS blocks

• TIMSS-window booklets contain three blocks of science or math, plus one block of NAEP

Example booklet configurations in the TIMSS window

TIMSS M01 NAEP M01 TIMSS S13 TIMSS M03

TIMSS S02 NAEP S01 TIMSS S05 TIMSS M03

Mathematics

Science

TIMSS Math blocks

NAEP Math block

TIMSS Science block

TIMSS Science blocksTIMSS Math block

NAEP Science block

Block composition of braided bookletsBlock composition of braided booklets

12

TIMSS at the state levelTIMSS at the state level

• Planned as a validity check on the NAEP projection methodology

• Administered as part of regular, national TIMSS• Funded by NCES, not by states • 29 states volunteered;

○ 8 states will participate○ MA, MN, AL, CA, CT, IN, CO, & NC

13

NAEP at the state levelNAEP at the state level

• A state-level grade 8 mathematics assessment was already planned

• All states have volunteered to participate in the new 2011 NAEP science assessment

• Science will not be a regular NAEP state-level assessment with full reporting (no urban districts)

• The braided linking samples are national only, with a very small sample in any given state

14

Timeline for NAEP-TIMSS linkingTimeline for NAEP-TIMSS linking

• Release of the national and state TIMSS results is scheduled for December 2012

• Linking results will be reported a few months after the main TIMSS release

15

Benefits of the proposed approachBenefits of the proposed approach

• Avoids interference with the conduct of regular operational assessments

• Provides a NAEP-TIMSS link through common items taken by the same student

• Measures the amount of growth during the period from winter to spring

• Provides states with international benchmarks in science and mathematics at grade 8, without the cost of conducting TIMSS at the state level

16