aligning levels to the european framework the origó experience budapest, february 2006

Aligning levels to the Aligning levels to the European FrameworkEuropean Framework

The Origó ExperienceThe Origó Experience

Budapest, February 2006Budapest, February 2006

The number of ORIGÓ exams in 2005The number of ORIGÓ exams in 2005(oral and written):(oral and written):

108 401108 401

ITK ORIGÓITK ORIGÓ

• The number of languagesThe number of languages:: 2929

• Arabic, Beás- Romani, Bulgarian, Chinese, Croatian, Czech, English, Danish, • Esperanto, Finnish, French, German, Hebrew, Hungarian, Italian, • Latin, Lovari-Romani, Modern-Greek, Old-Greek, Polish, Portugese, • Romanian, Russian, Serb, Spanish, Slovakian, Slovenian, Swedish, Turkish

• 2 testing systems2 testing systems:: Hungarian as a Foreign LanguageHungarian as a Foreign Language - - ALTE-membershipALTE-membership

The ratio of languages (%) in 2005The ratio of languages (%) in 2005

English 53,0German 25,5Esperanto 6,4French 3,3Lovari 2,5Italian 2,5Spanish 1,7Russian 1,1Latin 1,4Croatian 0,4Romanian 0,3Serb 0,2Slovakian 0,3Other 2,0

The number of exam centresThe number of exam centres::

70 - accredited/registered70 - accredited/registered

Oral Written

69% 49%

Pass ratePass rate

20052005

Population characteristics: age and sexPopulation characteristics: age and sex

female

male

64%

36%20052005

0

1000

2000

3000

4000

5000

6000

7000

8000

14 21 28 35 42 49 56

férfi

nő

Distribution: age and sex, 2005Distribution: age and sex, 2005

0

1000

2000

3000

4000

5000

6000

14 16 18 20 22 24 26 28 30 32 34 36 38 40 42

Candidates

• The oldest: 1935. 06. 23., female - Basic German, C, Pass

• The youngest: 1991.12. 14., female - Basic German, A, Passed

• Maximum performance (oral): 62 candidates 100%: English, Arabic, German, Esperanto, Latin, Hungarian, Russian,

Italian, Rumanian, Spanish, Serb, Slovakian, Modern-Greek, Hebrew - Basic-, Intermediate-, Advanced levels

• Maximum performance (written): 9 candidates 100%: Basic English- (5), Basic Hungarian-, Intermediate Italian-, Basic

Spanish-, Basic Modern-Greek

The Framework levels and The Framework levels and Origó levels ???Origó levels ???

EuropeanCouncil

ITK ORIGÓ Accreditedlanguage

exams

Maturaexams

Cambridge Goethe

C2Mastery CPE

ZOP,KDS

C1Operationalproficiency

Proficiency/Advanced

Proficiency/Advanced

CAE ZMP

B2Vantage Intermediate Intermediate Advanced FCE

B1Threshold Basic Basic PET ZD

A2Waystage

Junior not accredited IntermediateKET

A1Breakthrough

notaccredited

not accredited

The Hungarian context

• the law 1st September, 2006

• the Accreditation Board July, 2006

• the Matura - one-way equivalence

• international consensus?• levels?

Schedule

• familiarization - scales• specification - documentation• standardization - scales, calibrated samples • empirical validation - calibrated samples,

level setting

Origó - Previous experience

• level setting - empirical

alinging Rigó and international exams (ESU framework, Goethe)

• quality control procedures

• accreditation (1999)

Origó - work accomplished

• schedule• familiarisation (exam developers)

• specification - modifications in Origó scales

suggestions for other modifications

• familiarisation - all oral examiners

Origó - work to follow:

• familiarisation - item writers, markers (written exams)

• benchmarking - sample materials (exam developers)

• standardisation - all examiners

• empirical validation - major languages

Origó - benefits

• positive working experience (CEFR)• useful guidelines (Manual)• useful discussions (awareness raising)• useful modifications

Origó - problem areas

• terminology• translation problems • advanced (C1)/proficiency (C2) levels• availability of sample reference materials• mediation

Terminology„claims”

• linking• relating• harmonizing• aligning• level setting

Translation problems

• „claim”• „a degree of spontaneity and fluency”

Levels

Proficiency (C2)

Advanced (C1)

Difficulties with sample materials

videos in English (Eurocentre, Cambridge)

quality (copy of a copy)

accents (international)

levels

Jihun B2 / intermediate

0

5

10

15

20

25

30

35

40

near pass at level good pass

Estelle B2- / intermediate

0

5

10

15

20

25

30

35

good basic weak interm. interm.

Origó 2004 / intermediate

0

10

20

30

B1 12 B2- 13

B1+ 19 B2 28

B2- 12 B2+ 5

Gábor, 30 Lilla, 35

Renate - / B1

0

5

10

15

20

25

30

35

40

B1 B2

Annemarie - / C1

0

5

10

15

20

25

30

B2 C1

Evgeny C1 / Proficiency

0

5

10

15

20

25

good interm. weak prof. prof.

Familiarization - scales

Pragmatic competence item 4 A1:2 A2:135 B1:6

item 3 B1:53 B2:83

Strategic competence item 1 A2:46 B1:78 B2:13

item 5 B2:26 C1:106

Language competence item 1 B2:3 C1:94

item 4 B1:34 B2:59

Interpretation of data

IRT information - Eurocentre

- Dialang, Cambridge, Finnish Matriculation

not enough background info for reproduction

no measuring instruments attached

Reference measures

Eurocentre Dialangreading

Cambridgereading

Finnishreading

C2 3.90 2.01 70-90 (80) 47-50

C1 2.80 1.23 61-81 (71) 40-46

B2 0.72 0.21 57-72 (62) 25-39

B1 -1.23 -.22 50-70 (56) 8-24

A2 -3.23 -.56 30-50 (41) 4-7

A1 -4.29 -.94 3

Origó IRT measures (grammar)

Intermediate Proficiency

English 1.10 – 1.25 1.60 – 1.80

German 1.00 –1.20 1.30 – 1.70

French 1.20 – 1.40

Spanish 1.10 – 1.40

Italian 1.50 – 1.80

Russian 1.20 – 1.50

Esperanto 1.20 – 1.30

Lovari 1.20 – 1.30

Latin 1.60 – 1.90

Origó - Mediation

• major difficulty: no CEFR scales• no Council of Europe sample materials• external validation? • accreditation?