validation of the assessment and comparability to the pisa framework hao ren and joanna tomkowicz...

Validation of the Assessment and Comparability to the PISA Framework

Hao Ren and Joanna TomkowiczMcGraw-Hill Education CTB

Copyright © 2015 CTB/McGraw-Hill LLC 2

Outline

• PISA and OECD Test for Schools (OTS) – similarities and differences

• Linking OTS to PISA • Cluster rotation design for OTS• Scoring model• Selected results• Summary


PISA and OTS (Cognitive Test)

• The same domains covered by PISA and OTS: reading, mathematics and science– PISA focuses on one major domain in each administration (with the

other two domains receiving less coverage)– OTS has a balanced design between the three domains.

• New items were developed for OTS under the same framework as PISA.– PISA 2009 blueprint used in the test development.

• Similar test question type and format• Similar total testing time


PISA and OTS (Student and School Questionnaires)

• Both PISA and OTS use student questionnaires to collect information from students on various aspects of their home and family life, school environment and attitudes towards learning

• Both PISA and OTS use school questionnaires to collect information from schools about various aspects of organisation and educational provision in schools.

• The information collected is the same or very similar to PISA


PISA and OTS Test Scale Comparability

• OTS was designed to yield school-level scored comparable to PISA scores

• International field test and equating study conducted during 2011 and early 2012.

• OTS was equated to PISA scales using a set of common items between PISA 2009 and OTS field test– Linking updated after PISA 2012 administration

• The three domains of OTS were put on the PISA scales:– Reading is on PISA 2000– Mathematics is on PISA 2003– Science is on PISA 2006


Content design for OTS

• The design comprises two item clusters for each domain (reading, mathematics and science) and one additional hybrid cluster including items from all three domains

• The considerations in constructing the test booklets are– There are 7 booklets and 3 clusters in each booklet.– The clusters are organized to appear in 3 separate booklets, in the

1st, 2nd and 3rd block (position) – In order to balance the effect of item position in the booklet, each

pair of clusters appears in one (and only one) booklet and each cluster appears with every other clusters

– The testing time is 120 minutes, which matches the PISA main design and therefore gives the students a “PISA-like” test experience


Cluster Rotation design for OTS

Book Block 1 Block 2 Block 3

1 R1 RSM M1

2 RSM M2 S2

3 M2 M1 R2

4 M1 S2 S1

5 S2 R2 R1

6 R2 S1 RSM

7 S1 R1 M2


Scoring Model

• PISA scoring model was adopted for OTS:– The mixed coefficients multinomial logit model– PISA uses multi-dimensional model– OTS uses uni-dimensional model

• The generalized form of the Rasch model is used to describe the item response part

• The population model describes the relationship between the student’s ability and all the background information– Gender, grade, booklet administered, parents’ occupation, and

all other information from the student’s questionnaire


Scoring Model

• All information from the student questionnaire was included in the scoring model for OTS – PISA uses both student questionnaire and school questionnaire in the scoring

model– OTS only involve student questionnaire in the scoring model but use school

questionnaire for reporting purpose and other analysis– Principal component analysis conducted on all student responses to the

questionnaire items and the generated principal scores are used in estimation of student’s latent ability

• The scores are adjusted by considering the impact of all factors• Estimated scores at the student level are plausible values • Student level scores are not reported

– Main PISA reports country-level scores– OTS reports school-level score

• The detail description about the method can be found in PISA report


Selected OTS Pilot 2012 Results

OTS results are reported at the school level • Performance in three domain (scale scores, achievement levels)• Extensive comparison with similar schools in the United States and

international countries and economies• Economic, Social, and Cultural Status (ESCS) and student performance • Various analyses related to non-cognitive factors


OTS Pilot School Performance*

Domain Mean SD Min Max

Reading 509.0 55.3 343.9 624.9

Mathematics 506.9 67.3 320.6 679.5

Science 515.0 47.9 376.9 625.3

*The schools with too small number (<42) of students , or too low participation rate (<50%) were excluded from the summary in this table.


Mathematics vs. Reading

US MeanReading 500

US MeanMathematics 487

Correlation0.98


Reading vs. Science

US MeanReading 500

US MeanScience 502

Correlation0.98


How the pilot schools compared with schools in other countries and economies in reading in PISA 2009

Schools in The United States

Schools in Shanghai-China

Schools in Mexico

Schools in Pilot


Reading vs. ESCS

US MeanReading 500

ESCS

Economic, social, and cultural status

Correlation0.81


Reading vs. Disciplinary Climate in English Lessons

US MeanReading 500

DISCLIMA

Disciplinary climate in English Lessons

Correlation0.73


Summary (What We Learned from the Pilot)

• Data analysis methodology implemented for OTS allows for comparing a school’s results to the country’s results and international benchmarks

• Similar trends of PISA and OTS results observed– Strong correlations between the performance in three domains– Association between ESCS and performance– Association between the non-cognitive factors and academic

achievement• The OTS Pilot confirmed the administrative conditions and

procedures of the assessment, validated the methodology, and explored how results could be reported before the OTS was made publicly available in 2013 and beyond

validation of the assessment and comparability to the pisa framework hao ren and joanna tomkowicz...

Documents

pisa copyright

pisa scales

learningboth pisa

pisa main design

domains of ots

school questionnairesboth

pisa cluster rotation

ots student