assessing reading literacy in the language of instruction

LANGUAGE TESTING SERVICES John H.A.L. de Jong 20061

Assessing Reading Literacy in the Language of Instruction

John H.A.L. de Jong

Intergovernmental Conference: Languages of Schoolingtowards a Framework for Europe Strasbourg, 16-18 October 2006


Overview

• Background of the PISA studies

• Design of the PISA studies

• PISA approach to Reading Literacy

• Some central findings of PISA 2000

• Extensions for PISA 2009

• Relationships between PISA & CEF

• Applying CEF in Upper Secondary Vocational


Background of PISA study

OECD: Organisation for Economic Co-operation and Development

• 30 member countries committed to democracy and market economy

• Provide comparative data and analyses• For countries

– to compare policy experience; – seek answers to common problems; – identify good practice;– co-ordinate policies.

• To support economic growth, boost employment, raise living standards.


History OECD• 1948 Organisation for European Economic Co-operation

(OEEC), set up with support from the United States and Canada to co-ordinate the Marshall Plan for the reconstruction of Europe after World War II (www.oecd.org/history).

• 1961 Created as an economic counterpart to NATO, the OECD took over from the OEEC

• Early recognition of importance of education of its Citizens

• 1985 Gather indicators on investment in education

• Education at a glance yearly publication

• 2000 first PISA


PISA Goals

After studying resources & investments in education for a number of consecutive years:Wish to know the yield of education:

at the end of compulsory education how well are students prepared to continue further education or to start working life?


2000 2003 2006 2009 2012

Reading Reading Reading Reading Reading

Maths Maths Maths Maths Maths

Science Science Science Science Science

PISA assessment schedule


Participation in OECD/PISA

PISA Year OECD Partner

2000 28 15

2003 30 10

2006 30 28


Definition

Reading literacy is understanding, using and reflecting on written texts, in order to achieve one’s goals, to develop one’s knowledge and potential and to participate in society.


Retrieving information

Interpreting text

Reflection & Evaluation


Let’s look at ReadingIn the activity of reading we can distinguish four main elements:

1 there is a reader

2 there is a text (the author is part of the text, but not present in the activity)

Both of these are observable.

3 the reader has a goal (a self-chosen reason for reading the text)

4 the reading activity will have a result

(a change in the reader: more knowledge, satisfaction, etc )

These are not observable.


Text - Reader - Goal - Result

Assessing Reading:

- Response- Test taker - TaskText

Reading in ‘real’ life

- Scoring rules - Score

Not chosen, but imposed by

testMust be

observable

Conform to the task

Reflects difference between task and observed response

End point of process


These elements should be relevant to the intended traitThese elements

should interfere

the least

possible

• refers to (part of) a text

• sets task pertaining to the text, requiring particular subskill(s)

• specifies format for overt response demonstrating application of subskill

• specifies rules for scoring response

A Reading item

• may require to link information from the text to prior knowledge and experience

Only then are score differences likely to reflect differences in true ability on the intended trait: i.e. reliable and valid


Difficulty of comprehension depends on text characteristics

Reader

Aspects of item difficulty

TextComprehension is the interaction between text &

reader ▪ Familiarity▪ # of elements

▪Ling.Characteristics▪Complexity

Difficulty of task depends on prior

knowledge & experience of

candidate

– P

erso

nal /

Pri

vate

– F

orm

al /

Pub

lic

+ +Evaluate / Hypothesize

Find / Connect /Assess

Comprehension

Difficulty of item depends on

scoring rules

Difficulty of item depends on skills

required to perform task

Scoring Guide

Specificity

Difficulty of item depends on

response format Response

Format


PISA 2000: Relation SES & Reading

SES READING

High 540

Medium 500

Low 460


Mean PISA 2003 scores per educational track in the Netherlands


Mean Reading Literacy score for Dutch-born students and for immigrant students


Language proficiency in upper secondary vocational education

CINOP Study using CEF

• what is the actual language proficiency of students?

• is there a discrepancy between actual and required language proficiency for further education and professional occupation?


Research design

- written questionnaires for language teachers and vocational teachers, N = 210

- self assessment on language proficiency for students, N= 345

- followed by a test for reading, N= 328

- oral interviews with teachers, staff members and management, N= 40


Teachers’ views

• 60 to 70 % estimate language proficiency (reading A2-B1) insufficient for school

• 10 % estimate the language proficiency on a higher level than needed for school

• 80 % estimate the language proficiency insufficient for professional occupation


Self assessment by students

• > 2/3 of students estimate their reading proficiency level at B2

• In reality: 7 % at A1 24 % at A2 52 % at A2-B1 17 % at B1-B2


CEF appropriate for description of language of instruction• Coherent and workable structure for teachers

(contributes to improvement of their professional competences)

• Competence based descriptions more suitable for language of instruction than linguistic descriptions of L1

• Descriptors applicable

• More transparency and motivating for students (specially in combination with a language portfolio)

• Offers opportunity for reflection on language learning by students


Shortcomings of CEF in describing language of instruction in vocational education

• Can do statements are geared at academic learning routes: not well adjusted to describing specific tasks in vocational education and specific tasks in the workplace

• More situations and examples have need to be offered for a range of vocational sectors


“Milestones”

• 2000 1st PISA Cycle: Reading Major Domain

• 2002 Publication Reading for Change

• 2004 CINOP Dutch in Vocational Education

• 2006 ALARM: Language skills at risk

assessing reading literacy in the language of instruction

Documents