feast chetcuti
DESCRIPTION
...TRANSCRIPT
1 FEAST PRESENTATION SUMMER SCHOOL GRAZ 3 JULY 2FEAST PRESENTATION SUMMER SCHOOL GRAZ 3 JULY 2007007 © EUROCONTROL 2007© EUROCONTROL 2007
The FEAST Service - Combining the professional with the practical
across organizational boundaries
Fidel ChetcutiFidel ChetcutiDAP/SSHDAP/SSH
EUROCONTROLEUROCONTROL
FEAST ServiceFEAST Service
2 FEAST PRESENTATION SUMMER SCHOOL GRAZ 3 JULY 2FEAST PRESENTATION SUMMER SCHOOL GRAZ 3 JULY 2007007 © EUROCONTROL 2007© EUROCONTROL 2007
Overview
Background and development
Initial Validation Study
Using FEAST for decision making
Future Challenges
Predictive Validation Study
Current FEAST Package
Making FEAST work in practice
Questions and Discussion
3 FEAST PRESENTATION SUMMER SCHOOL GRAZ 3 JULY 2FEAST PRESENTATION SUMMER SCHOOL GRAZ 3 JULY 2007007 © EUROCONTROL 2007© EUROCONTROL 2007
F
E
S
A
T
IRST
UROPEAN
IR TRAFFIC CONTROLLER
ELECTION
EST
4 FEAST PRESENTATION SUMMER SCHOOL GRAZ 3 JULY 2FEAST PRESENTATION SUMMER SCHOOL GRAZ 3 JULY 2007007 © EUROCONTROL 2007© EUROCONTROL 2007
FEAST ServiceFEAST Service
4
The FEAST Service Team
FEAST Service Expert
[email protected] Service Manager
[email protected] Service Assistant
Marc Damitz Fidel Chetcuti Michaela Heese
5 FEAST PRESENTATION SUMMER SCHOOL GRAZ 3 JULY 2FEAST PRESENTATION SUMMER SCHOOL GRAZ 3 JULY 2007007 © EUROCONTROL 2007© EUROCONTROL 2007
KEY FEATURES
Professional standards crossing national boundariesProfessional standards crossing national boundaries
+
Web platform quick, cost effective and easy to useWeb platform quick, cost effective and easy to use
+
State of the art testing option conforming to EUROCONTROL Selection Guidelines. State of the art testing option conforming to EUROCONTROL Selection Guidelines.
Standardised administrative procedures Europe-wide. Fairly simple to use and manageStandardised administrative procedures Europe-wide. Fairly simple to use and manage
+
FEAST User Group and CommunityFEAST User Group and Community+
+
6 FEAST PRESENTATION SUMMER SCHOOL GRAZ 3 JULY 2FEAST PRESENTATION SUMMER SCHOOL GRAZ 3 JULY 2007007 © EUROCONTROL 2007© EUROCONTROL 2007
KEY I.T. ISSUES
QUALITY AND STANDARDSSetting and maintaining European standards
QUALITY AND STANDARDSSetting and maintaining European standards
SECURITY AND CONFIDENTIALITYSECURITY AND CONFIDENTIALITY
DATA scores and material for research and
validation
DATA scores and material for research and
validation
REAL TIME AND ONLINEREAL TIME AND ONLINE
7 FEAST PRESENTATION SUMMER SCHOOL GRAZ 3 JULY 2FEAST PRESENTATION SUMMER SCHOOL GRAZ 3 JULY 2007007 © EUROCONTROL 2007© EUROCONTROL 2007
Job and Task
analysis
Model of knowledge,
skills, abilities and other
characteristics
Find tests to
measure KASOs
FEAST test
package
THE BASIS OF FEAST
8 FEAST PRESENTATION SUMMER SCHOOL GRAZ 3 JULY 2FEAST PRESENTATION SUMMER SCHOOL GRAZ 3 JULY 2007007 © EUROCONTROL 2007© EUROCONTROL 2007
TASK ANALYSIS
Behaviour
Decision making
competence
Skills and abilities
Knowledge
Situational Situational AwarenessAwareness
9 FEAST PRESENTATION SUMMER SCHOOL GRAZ 3 JULY 2FEAST PRESENTATION SUMMER SCHOOL GRAZ 3 JULY 2007007 © EUROCONTROL 2007© EUROCONTROL 2007
THE FEAST MODEL of 26 KASOs
English asthe working
language
&
Computerskills
PerceptionComprehension
Projection(Planning &
Problem Solving)
Decision-making
Situational Awareness
Personality
Characteristics
(Behaviouralelements)
(Skills, Abilities)
(Knowledge)
10 FEAST PRESENTATION SUMMER SCHOOL GRAZ 3 JULY 2FEAST PRESENTATION SUMMER SCHOOL GRAZ 3 JULY 2007007 © EUROCONTROL 2007© EUROCONTROL 2007
CONTROLLER SKILLS AND ABILITIES
Perception of environmentPerceptual SpeedSelective AttentionMedium-term memoryMultiple-task performanceScanning abilityMonitoring/Vigilance
Comprehension of current
situation3-dimensional abilityVerbal comprehensionConscientiousnessMental Flexibility
Projection of future statusPlanning ability
Problem solving
Decision-makingDecision-makingCommunicationDecisivenessComposure/ResilienceAdaptability
11 FEAST PRESENTATION SUMMER SCHOOL GRAZ 3 JULY 2FEAST PRESENTATION SUMMER SCHOOL GRAZ 3 JULY 2007007 © EUROCONTROL 2007© EUROCONTROL 2007
Personality related and behaviouralcompetences
PERSONALITY AND LEARNED BEHAVIOUR
Composure/resilience
Self-awareness
Achievement motivation Flexibility
Social sensitivity
Emotional stability
Decisiveness
Co-operation
Conscientiousness
12 FEAST PRESENTATION SUMMER SCHOOL GRAZ 3 JULY 2FEAST PRESENTATION SUMMER SCHOOL GRAZ 3 JULY 2007007 © EUROCONTROL 2007© EUROCONTROL 2007
GETTING THE RIGHT TESTS AND GETTING THE TESTS RIGHT!
Available tests
Individual tests evaluated against minimum reliability and validity requirements
Further ratings
according to 13 other criteria
Detailed questions
to shortlisted
test suppliers
FEAST test
battery
13 FEAST PRESENTATION SUMMER SCHOOL GRAZ 3 JULY 2FEAST PRESENTATION SUMMER SCHOOL GRAZ 3 JULY 2007007 © EUROCONTROL 2007© EUROCONTROL 2007
CONCURRENT VALIDATION STUDY
3 populations
(ATC trainees,ATCOs and unselected applicants)
10 volunteer ANSPs
577 test takers
FEAST web•low reliability for 1 test , other tests satisfactory•Low validity between single tests and BOS overall score, but significant correlation between BOS criteria and composite test scores
FEAST SDM•Reliability not rechecked but test developer reported satisfactory•Low validity between overall score and BOS overall but significant correlation between BOS criteria and various raw scores and composites in both groups
Conclusions: despite the issues encountered (sample size, care with BOS completion, motivation of
participants), results sufficiently encouraging to start to use FEAST carefully
14 FEAST PRESENTATION SUMMER SCHOOL GRAZ 3 JULY 2FEAST PRESENTATION SUMMER SCHOOL GRAZ 3 JULY 2007007 © EUROCONTROL 2007© EUROCONTROL 2007
FEAST CURRENT VERSION
FEAST TEST BATTERY
COGNITIVE ABILITY Heading and Range (SAHR)
English Listening and Comprehension (ENA)Planning (SAP)Attention (SORT)- Multi-tasking (PMT) andVisualization (FOLD)
KNOWLEDGE English language listening and
comprehension
WORK SAMPLE
Strip Display Management
INTERVIEWSituational Interview
15 FEAST PRESENTATION SUMMER SCHOOL GRAZ 3 JULY 2FEAST PRESENTATION SUMMER SCHOOL GRAZ 3 JULY 2007007 © EUROCONTROL 2007© EUROCONTROL 2007
What’s in FEAST?
web based run from PC 5 ability tests and 1 English
language listening comprehension test-
1 work sample test run from CD-ROM
2.5hrs 2.25hrs
16 FEAST PRESENTATION SUMMER SCHOOL GRAZ 3 JULY 2FEAST PRESENTATION SUMMER SCHOOL GRAZ 3 JULY 2007007 © EUROCONTROL 2007© EUROCONTROL 2007
Selection criteria and hurdles
Cognitive abilities
Job relevant
Knowledgee.g. English language
More complex
skills/abilities e.g. multi-
tasking
Behavioural/ personality
related factors e.g.
Questionnaire, self
assessment
Personal interaction and mutual “checking”e.g. final interview
Medical exam
PHASE1 PHASE2 PHASE3FINAL
DECISION
FEAST Web FEAST SDM Assessment Centre, Interview
Selection Board
Assessment Centre
17 FEAST PRESENTATION SUMMER SCHOOL GRAZ 3 JULY 2FEAST PRESENTATION SUMMER SCHOOL GRAZ 3 JULY 2007007 © EUROCONTROL 2007© EUROCONTROL 2007
Integration of FEAST Web-based Tests
1. Are proposed as Phase 1 tests to filter candidates for a second selection Phase
2. The Multiple Stage Selection Approach is recommended (‘multiple hurdle’)
FEASTweb
FEAST SDM
Inter-view
Phase 1 Decision Phase 2 Decision Phase 3 Final Decision
18 FEAST PRESENTATION SUMMER SCHOOL GRAZ 3 JULY 2FEAST PRESENTATION SUMMER SCHOOL GRAZ 3 JULY 2007007 © EUROCONTROL 2007© EUROCONTROL 2007
Tests with low and high validity
True positive
False positive
False negative
True negative
Test with low validity Test with high validity
19 FEAST PRESENTATION SUMMER SCHOOL GRAZ 3 JULY 2FEAST PRESENTATION SUMMER SCHOOL GRAZ 3 JULY 2007007 © EUROCONTROL 2007© EUROCONTROL 2007
Influence of the ‘base rate’
Cut-off Score
Cut-off Score
Base rate : how many people could do this job?
Lower base rate Higher base rate
20 FEAST PRESENTATION SUMMER SCHOOL GRAZ 3 JULY 2FEAST PRESENTATION SUMMER SCHOOL GRAZ 3 JULY 2007007 © EUROCONTROL 2007© EUROCONTROL 2007
Influence of the ‘selection ratio’
Cut-off Score
Cut-off Score
Higher selection ratio Lower selection ratio
Selection ratio: how strict we are with pass scores
21 FEAST PRESENTATION SUMMER SCHOOL GRAZ 3 JULY 2FEAST PRESENTATION SUMMER SCHOOL GRAZ 3 JULY 2007007 © EUROCONTROL 2007© EUROCONTROL 2007
FEAST Decision Strategies
MULTI STAGE SELECTION using a combination of both the Composite & Multiple cut-off Models
Stage 1: Selection of the ‘right profile’following the Multiple cut-off Model (cut-off
score for each test, select across all tests, the Mean (Stanine 5) is used as the cut-off point)
Stage 2: Selection ‘from the top’ based on the Composite Model according to number required
22 FEAST PRESENTATION SUMMER SCHOOL GRAZ 3 JULY 2FEAST PRESENTATION SUMMER SCHOOL GRAZ 3 JULY 2007007 © EUROCONTROL 2007© EUROCONTROL 2007
TEST RESULTS AND NORMS
Results expressed as raw scores (e.g. 79 out of 120 items answered
correctly)Raw scores compared to norm group
Norms based on multinational and multicultural set of applicants Result given as stanine scores (1 to 9)Norms updated from time to time. Possibility of developing country norms in future
Individual and Group Selection Reports
23 FEAST PRESENTATION SUMMER SCHOOL GRAZ 3 JULY 2FEAST PRESENTATION SUMMER SCHOOL GRAZ 3 JULY 2007007 © EUROCONTROL 2007© EUROCONTROL 2007
Composition of FEAST Overall Score
VisualisationFOLD (3DC)
Multi Tasking - AlertnessPMT
Sorting AbilitySORT 6
Mean Reaction Time (MRT)SORT 3
Planning ATCSAP
English proficiencyENA
Heading and RangeSAHR
FEAST Overall ScoreTotal number of indexes: 6
Test
OVERALL SCORE•equal weighting for each test index•Sum of all weighted indexes standardised as 0-100•Reflects pure cognitive abilities dimension•English language score is separated
24 FEAST PRESENTATION SUMMER SCHOOL GRAZ 3 JULY 2FEAST PRESENTATION SUMMER SCHOOL GRAZ 3 JULY 2007007 © EUROCONTROL 2007© EUROCONTROL 2007
Decision Strategies - Example
Candidate A:
4ENA
5PMT
6SAHR
6SAP
7FOLD
8SORT
7FEAST Overall
StanineTest Result
9ENA
5PMT
6SAHR
4SAP
3FOLD
6SORT
5FEAST Overall
StanineTest Result
Candidate B:
25 FEAST PRESENTATION SUMMER SCHOOL GRAZ 3 JULY 2FEAST PRESENTATION SUMMER SCHOOL GRAZ 3 JULY 2007007 © EUROCONTROL 2007© EUROCONTROL 2007
Example for combination of multiple cut-off and selection from the top
64254332Cand. F
65555654Cand. D
Selection
6
4
5
ENA
5
6
6
FOLD
5
4
7
PMT
66565Cand. C
84687Cand. B
58795Cand. A
SAHRSAPSORTFEAST overall
candidate
Stage 1: Selection of the ‘right profile’ following the Multiple cut-off Model, i.e. FEAST cognitive tests ≥ 5, ENA ≥ 4.
Stage 2: Selection ‘from the top’ based on the Composite Model, e.g. selection of the best 2 candidates.
26 FEAST PRESENTATION SUMMER SCHOOL GRAZ 3 JULY 2FEAST PRESENTATION SUMMER SCHOOL GRAZ 3 JULY 2007007 © EUROCONTROL 2007© EUROCONTROL 2007
PREDICTIVE VALIDATION
Behaviour Observation Scale BOS
Common criterion for all
countries. Available in
different languages
•Cognitive aspects ATC •Personal/inter-personal aspects of ATC job•Self confidence•English language proficiency and ATC phraseology
Common training for assessors before
completing BOS form
Relates to the same job and task analysis as the FEAST tests
Trainee assessment done
at two points during training
27 FEAST PRESENTATION SUMMER SCHOOL GRAZ 3 JULY 2FEAST PRESENTATION SUMMER SCHOOL GRAZ 3 JULY 2007007 © EUROCONTROL 2007© EUROCONTROL 2007
The BOS Form elements
Behavioural performance measurement
6 point scale coveringInferior/Comparable/Superior
Anchored Examples of inferior / superior behaviours
38 items plus General Assessment
BOS Form
28 FEAST PRESENTATION SUMMER SCHOOL GRAZ 3 JULY 2FEAST PRESENTATION SUMMER SCHOOL GRAZ 3 JULY 2007007 © EUROCONTROL 2007© EUROCONTROL 2007
Validation Process
Selection tests
Final Decision
BOS Assessments
Compare BOS assessments with FEAST results
Quantify predictive power of FEAST tests
Enhance and refine
test battery
Establish and maintain close links
with trainers
29 FEAST PRESENTATION SUMMER SCHOOL GRAZ 3 JULY 2FEAST PRESENTATION SUMMER SCHOOL GRAZ 3 JULY 2007007 © EUROCONTROL 2007© EUROCONTROL 2007
FEAST Business ApplicationFEAST Business Application
Central maintenance and technical problem resolution
Hosted confidentially and securely
Database Management of job applicants
FEAST computer-based tests
Scores computed and stored for useWeb based
tools
30 FEAST PRESENTATION SUMMER SCHOOL GRAZ 3 JULY 2FEAST PRESENTATION SUMMER SCHOOL GRAZ 3 JULY 2007007 © EUROCONTROL 2007© EUROCONTROL 2007
WHAT IS IMPORTANT FOR OUR USERS
high quality
cost
effective
common
platform
perceived
fairness
harmonised process
31 FEAST PRESENTATION SUMMER SCHOOL GRAZ 3 JULY 2FEAST PRESENTATION SUMMER SCHOOL GRAZ 3 JULY 2007007 © EUROCONTROL 2007© EUROCONTROL 2007
KEY BENEFITS FOR USERS
Security
Quality and professional Standards
Validation data
FEAST User
Group
Cost effective, shared
developments
32 FEAST PRESENTATION SUMMER SCHOOL GRAZ 3 JULY 2FEAST PRESENTATION SUMMER SCHOOL GRAZ 3 JULY 2007007 © EUROCONTROL 2007© EUROCONTROL 2007
How does the FEAST Service operate?
TECHNICAL PROVIDERSInfrastructure -MISTest platform-External providerPotential new test providers
FEAST SERVICESelection testsTechnical supportProfessional supportUser Group facilitationValidation and Development
Means of delivery:WebCD_ROMHelpdeskPersonal contactFUG
USER ACTIVITIESTesting candidates
Selection
High quality process
Cost effective process
Networking opportunitiesContract and SLA
33 FEAST PRESENTATION SUMMER SCHOOL GRAZ 3 JULY 2FEAST PRESENTATION SUMMER SCHOOL GRAZ 3 JULY 2007007 © EUROCONTROL 2007© EUROCONTROL 2007
FEAST Service functions
FEAST Service Team
Implementation support and
training
User Group facilitation
Daily Helpdesk
Validation and Studies
Test Maintenance and Development
Management and co-ordination of internal
and external technical partners
Tool and methodology for validation (Behaviour Observation Scale BOS)
34 FEAST PRESENTATION SUMMER SCHOOL GRAZ 3 JULY 2FEAST PRESENTATION SUMMER SCHOOL GRAZ 3 JULY 2007007 © EUROCONTROL 2007© EUROCONTROL 2007
FEAST User Group
Facilitated by EUROCONTROL -
meets at least once a year
Forum for exchange of information and opinions
Knowledge base to enhance recruitment
and selection procedures and
standardsContinuing improvements and evolution of the
FEAST tools
A common goal:
a practical and high quality tool for selection of ab initio trainee
controllers
35 FEAST PRESENTATION SUMMER SCHOOL GRAZ 3 JULY 2FEAST PRESENTATION SUMMER SCHOOL GRAZ 3 JULY 2007007 © EUROCONTROL 2007© EUROCONTROL 2007
1619 FEAST Test Takersin the first half of 2007 (15 Users)
FEAST completed 2004-June 2007
157232 293 313
508
285 300
790
422
651
1003
1209
780 839
0 00
200
400
600
800
1000
1200
1400Ja
n-M
ar
Apr-
Jun
Jul-S
ept
Oct
-Dec
Jan-
Mar
Apr-
Jun
Jul-S
ept
Oct
-Dec
Jan-
Mar
Apr-
Jun
Jul-S
ept
Oct
-Dec
Jan-
Mar
Apr-
Jun
Jul-S
ept
Oct
-Dec
2004 (∑=995) 2005 (∑=1883) 2006 (∑=3285) 2007 (∑=1619)
time (in months)
Num
ber o
f app
lican
ts w
ho
com
plet
ed F
EAST
36 FEAST PRESENTATION SUMMER SCHOOL GRAZ 3 JULY 2FEAST PRESENTATION SUMMER SCHOOL GRAZ 3 JULY 2007007 © EUROCONTROL 2007© EUROCONTROL 2007
FEAST implementation
FEAST IMPLEMENTATION CHALLENGES
Facilities, equipment, Security, IT
Roles and Responsibilities
Selection process
Validation
Training and competencies of staff
Review existing policies
Organizational issues
Identify new issues and questions
37 FEAST PRESENTATION SUMMER SCHOOL GRAZ 3 JULY 2FEAST PRESENTATION SUMMER SCHOOL GRAZ 3 JULY 2007007 © EUROCONTROL 2007© EUROCONTROL 2007
FUTURE CHALLENGES FOR ATC SELECTION
Harmonized and Harmonized and high standard high standard ATC selection ATC selection
processes across processes across EuropeEurope
Providing service to more customers while maintaining
high quality
Staying cost effective
The impact if any of cultural
differences
Spoken English language test for
selection
Behaviouralassessment
Scope for further harmonization
•Age issues•Requirements for educational level
Predictive Validation
Study
38 FEAST PRESENTATION SUMMER SCHOOL GRAZ 3 JULY 2FEAST PRESENTATION SUMMER SCHOOL GRAZ 3 JULY 2007007 © EUROCONTROL 2007© EUROCONTROL 2007
Knowledge skills ability requirements for future ATC job
Controller job largely unchanged
Controller more passive. Skills used more infrequently but when called for,some skills
could be more demanding e.g. problem solving
Future changes to controller task likely from•Continuing automation•Standardisation and harmonisation of activities across national boundaries•More flexible use of airspace (Single European Sky, Functional Airspace Blocks) to increase capacity•Delegation of responsibilities from ground to pilot
39 FEAST PRESENTATION SUMMER SCHOOL GRAZ 3 JULY 2FEAST PRESENTATION SUMMER SCHOOL GRAZ 3 JULY 2007007 © EUROCONTROL 2007© EUROCONTROL 2007FEAST PRESENTATION FEAST PRESENTATION Brussels, 21 March 2007Brussels, 21 March 200739
FEAST ServiceFEAST Service
Don’t hesitate to ask easy questions!