implementing a new multilingual selection model across 27 european countries gilles guillard, markus...
TRANSCRIPT
Implementing a New Multilingual Selection Model Across 27 European Countries Gilles Guillard, Markus NussbaumGilles Guillard, Markus Nussbaum
The European Union
Economic &Social Committee
Committee of the Regions
Ombudsman
Data ProtectionSupervisor
Council
Parliament
Court of Justice
Court of Auditors
Commission
The EU Institutions
Posts in 2012
European Parliament 6655
Council of the European Union 3153
European Commission 25478
Court of Justice 1952
European Court of Auditors 887
European Economic and Social Committee 724
Committee of the Regions 531
European Ombudsman 66
European Data Protection Supervisor 43
European External Action Service 1670
Total 41159
European External
Action Service4% Others
3%
European Parliament
16%
Council of the
European Union
8%
European Commission
62%
European Court
of Auditors2%
Court of Justice
5%
Administrators (AD)e.g. policy officers, lawyers, auditors, scientific officers, translators, interpreters, communication & press officers …
Qualification: University degree (bachelor)
Assistants (AST)e.g. secretaries, HR assistants, financial assistants, conference technicians …
Qualification: Secondary education (at least) and relevant professional experience
Administrators (AD)e.g. policy officers, lawyers, auditors, scientific officers, translators, interpreters, communication & press officers …
Qualification: University degree (bachelor)
Assistants (AST)e.g. secretaries, HR assistants, financial assistants, conference technicians …
Qualification: Secondary education (at least) and relevant professional experience
Entry Streams
Selection in Cycles
Competition launched in March 2012
45000+ candidates for 296 posts
5 fields: Audit, EU Public Administration, Communication, Law, External Relations
Competition launched in March 2012
45000+ candidates for 296 posts
5 fields: Audit, EU Public Administration, Communication, Law, External Relations
The AD Cycle 2012
Verbal reasoning20 items from a database, 35 min, 23 languages
Numerical reasoning10 Items from a database, 20 min, 23 languages
Abstract reasoning10 Items in fixed test forms, 10 min
Situational Judgement20 Items in fixed test forms, 30 min, EN/DE/FR
Verbal reasoning20 items from a database, 35 min, 23 languages
Numerical reasoning10 Items from a database, 20 min, 23 languages
Abstract reasoning10 Items in fixed test forms, 10 min
Situational Judgement20 Items in fixed test forms, 30 min, EN/DE/FR
Computer Based Tests
Verbal Reasoning
Numerical Reasoning
Abstract Reasoning
X
X
Situational Judgement
Results
EU Public Admin Audit Communication Law External Rel
Sum
AD5 AD7 AD5 AD7 AD5 AD7 AD5 AD7 AD5 AD7
validated application 12287 7892 2675 1389 3467 2023 4636 2130 6519 2338 45356
booked CBT 9557 6076 2182 1005 2525 1430 3615 1592 5016 1706 34704
participated CBT 8323 5336 1942 893 2132 1263 3113 1330 4412 1470 30214
reached CBT pass-marks 5502 3767 1427 603 1253 723 2076 830 2935 948 20064
succeeded CBT 190 102 108 34 77 51 144 43 64 21 834
Score Distribution - VR
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
0250500750
10001250150017502000225025002750300032503500
3750
AD177: VR
participated
admitted
reserve list
Test score
Co
un
t
0 1 2 3 4 5 6 7 8 9 10
0
500
1000
1500
2000
2500
3000
3500
4000
4500
5000
5500
6000
AD177: NR
participated
admitted
reserve list
Test score
Co
un
t
Score Distribution - NR
0 1 2 3 4 5 6 7 8 9 10
0500
1000150020002500300035004000450050005500600065007000
7500
AD177: AR
participated
admitted
reserve list
Test score
Co
un
t
Score Distribution - AR
Situational Judgement
AD-2011: SJ-AP
0
1000
2000
3000
4000
5000
6000
7000
8000
9000
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
Score
Co
un
t
AD-2011: SJ-RE
0
1000
2000
3000
4000
5000
6000
7000
8000
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
Score
Co
un
t
AD-2011: SJ-QR
0
1000
2000
3000
4000
5000
6000
7000
8000
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
Score
Co
un
t
AD-2011: SJ-PO
0
1000
2000
3000
4000
5000
6000
7000
8000
9000
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
Score
Co
un
t
AD-2011: SJ-WO
0
1000
2000
3000
4000
5000
6000
7000
8000
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
Score
Co
un
t
Intercorrelations
Verbal R
easoning
Num
erical Reasoning
Abstract R
easoning
SJT
Analysis &
Problem
S
olving
SJT
Planning &
Organising
SJT
Quality &
Results
SJT
Resilience
SJT
Working w
ith others
Verbal Reasoning - .41 .29 .16 .23 .15 .17 .18
Numerical Reasoning .41 - .32 .15 .20 .14 .15 .15
Abstract Reasoning .29 .32 - .14 .17 .12 .14 .14
SJT Analysis & Problem Solving .16 .15 .14 - .40 .33 .31 .35
SJT Planning & Organising .23 .20 .17 .40 - .32 .35 .35
SJT Quality & Results .15 .14 .12 .33 .32 - .35 .33
SJT Resilience .17 .15 .14 .31 .35 .35 - .32
SJT Working with others .18 .15 .14 .35 .35 .33 .32 -
How EPSO monitors the behaviour of items
How EPSO monitors the behaviour of items
Item Analysis
Based on the dicotomous / polytomous Rasch model
Conducted after every majorcompetition
Calibrate items and monitor their behaviour
Making psychometrics as vivid as possible
Based on the dicotomous / polytomous Rasch model
Conducted after every majorcompetition
Calibrate items and monitor their behaviour
Making psychometrics as vivid as possible
Item Analysis
Georg Rasch (1901-1980)
Test Responses Candidates Items
VR 2 295 360 114 768 5 071
NR 1 147 680 114 768 3 490
Total 3 443 040 8 561
Example:Cumulative Analysis VR+NR
Oct 2012
Item Difficulty Adjustment Across Languages (IDAAL)
How EPSO ensures fair opportunities for all candidates although testing in 23 different languages
How EPSO ensures fair opportunities for all candidates although testing in 23 different languages
Aligned Averages Algorithm (AAA)
A
0
1
2
3
4
-4
-3
-2
-1
5
EN
0
1
2
3
4
-4
-3
-2
-1
5
EN
A
-4-3
-2-1
0+
1+
2+
3+
4
Aligned Averages Algorithm (AAA)
citem = common item in en and lang
lang
m
encitemencitem
n
langcitemlangcitem
mn
1)(
)(1)(
)(
Minimum Meansquare Method (MMM)
0
1
2
3
4
-4
-3
-2
-1
5
EN A B C
A
B
C
0
1
2
3
4
-4
-3
-2
-1
5
EN
Minimum Meansquare Method (MMM)
min!
2
1 1
1,
,
n
item
m
lang
m
langlanglangitem
langlangitem m
Person Parameter Contrast Measure (PPCM)
0
1
2
3
4
-4
-3
-2
-1
5
Cdts EN
Cdts A
Person Parameter Contrast Measure (PPCM)
lang
m
encdtencdt
n
langcdtlangcdt
mn
1)(
)(1)(
)(
... is a combination of these three approaches:
Aligned Averages Algorithm (AAA)
Minimum Meansquare Method (MMM)
Person Parameter Contrast Measure (PPCM)
... is a combination of these three approaches:
Aligned Averages Algorithm (AAA)
Minimum Meansquare Method (MMM)
Person Parameter Contrast Measure (PPCM)
Item Difficulty Adjustment Across Languages (IDAAL)
EPSO Rasch Item Combination Algorithm (ERICA)
EPSO Rasch Item Combination Algorithm (ERICA)
How to assemble test forms
Candidates have to be treated equally: Same number of items Same difficulty of test form
➾ Adaptive testing is not possible
EPSO has developed an algorithm involving two steps:
1. Set up a "master test form"
2. Create test forms corresponding to master test form
Candidates have to be treated equally: Same number of items Same difficulty of test form
➾ Adaptive testing is not possible
EPSO has developed an algorithm involving two steps:
1. Set up a "master test form"
2. Create test forms corresponding to master test form
Framework
Actual test forms correspond to master test form
Limited overlap between test forms
No over-exposed items
Further improving gender balance
Actual test forms correspond to master test form
Limited overlap between test forms
No over-exposed items
Further improving gender balance
Target (quality criteria)
Defining the master test form
easy difficult
Master test form difficulty: 8
Master test form difficulty: 2
Master test form difficulty: 5
Items are approximately standard normal distributed
Most items are around the average (0) available
Weighting was implemented to better take account of item availabilities
Items are approximately standard normal distributed
Most items are around the average (0) available
Weighting was implemented to better take account of item availabilities
Weighting
Sample Test FormTarget Item Actual
-1.00 EN1626VEN -1.001
-0.56 EN1555V -0.550
-0.22 EN1170V -0.227
0.06 EN1063V 0.062
0.30 EN2361V 0.295
0.51 EN1003VEN 0.517
0.71 EN1686V 0.703
0.89 EN2089V 0.891
1.06 EN2041VEN 1.064
1.22 EN2133V 1.211
1.38 EN2191V 1.394
1.54 EN2261V 1.532
1.69 EN2260VEN 1.684
1.84 EN2180VEN 1.811
2.00 EN2196VEN 2.032
2.16 EN2264V 2.198
2.33 EN2142V 2.333
2.51 EN1578VEN 2.451
2.73 EN1292VEN 2.809
3.00 EN2371V 2.941
0.9Sum SqDev:
Thank you very much for your attention!
Any questions? Any questions?