gp and specialty selection - asme and specialty selection ... study 4. mrcgp outcomes after 2-3...
TRANSCRIPT
GP and Specialty SelectionGP and Specialty Selection
Professor Fiona Patterson Professor Fiona Patterson Work Psychology Group & University of CambridgeWork Psychology Group & University of Cambridge
Professor Jane DacreProfessor Jane DacreUniversity College LondonUniversity College London
OverviewOverview
Principles of best practice selectionPrinciples of best practice selection Defining selection criteria & the criterion problemDefining selection criteria & the criterion problem Accuracy of different selection methodsAccuracy of different selection methods
Overview of selection methodsOverview of selection methods Overview of selection methodsOverview of selection methods Situational Judgement TestsSituational Judgement Tests
GP & Specialty SelectionGP & Specialty Selection
Selection centresSelection centres
AoMRCAoMRC Clinical problemClinical problem--solving test pilotsolving test pilot
Designing selection systemsDesigning selection systems “…“…the function of selection is to provide a means of estimating the function of selection is to provide a means of estimating
candidates’ aptitude so that the most suitable applicants are candidates’ aptitude so that the most suitable applicants are appointed…appointed…”.”.
Selection uses a Selection uses a predictivistpredictivist paradigm. In exams, end of paradigm. In exams, end of training capability is tested & in theory, all applicants can passtraining capability is tested & in theory, all applicants can passtraining capability is tested & in theory, all applicants can passtraining capability is tested & in theory, all applicants can pass
Selection aims to Selection aims to predict trainabilitypredict trainability, i.e. identify those who will , i.e. identify those who will successfully complete training, successfully complete training, beforebefore it commences. it commences.
Parameters for Parameters for designing & validating designing & validating selection systems are selection systems are different to other assessment settingsdifferent to other assessment settings
Create competency model
Create person specification
Knowledge, skills, abilities,
characteristics
JobAnalysis
Tasks, roles, responsibilities
Competency analysis
Choose selection methods
Interviews, Simulations,
Selection centre
Attract pool of applicants
RecruitmentSelect out unsuitable applicants
The Selection ProcessThe Selection Process
Identify selection criteria
Make selection decisions
characteristicsRequired level of
competence
Selection centre
Reliability, validity, financial utility,
fairness
Validate selection decisions
SelectionSelect in and
rank
Aptitude?Attainment?
Patterson & Ferguson, 2007. Patterson & Ferguson, 2007.
What attributes are important to be an effective clinician?
How are they best tested at selection?How are they best tested at selection?
Beyond some basic assessment – is a lottery the best option?
Selection methodSelection method Evidence Evidence for Validityfor Validity
Applicant Applicant ReactionsReactions
Extent of Extent of UseUse
Structured interviewsStructured interviews HighHigh Moderate to positiveModerate to positive HighHigh
Cognitive abilityCognitive ability HighHigh Negative to moderateNegative to moderate ModerateModerate
Situational Judgement TestsSituational Judgement Tests HighHigh ModerateModerate ModerateModerate
Personality testsPersonality tests ModerateModerate Negative to moderateNegative to moderate ModerateModerate
BiodataBiodata HighHigh ModerateModerate ModerateModerate
Work sample testsWork sample tests HighHigh PositivePositive LowLow
Assessment centresAssessment centres HighHigh PositivePositive Moderate Moderate
GraphologyGraphology LowLow Negative to moderateNegative to moderate LowLow
ReferencesReferences LowLow PositivePositive HighHigh
LotteryLottery N/AN/A Negative Negative LowLow
Innovations in selectionInnovations in selectionmethodsmethodsmethodsmethods
Situational Judgement TestsSituational Judgement Tests
What’s a Situational Judgement Test?What’s a Situational Judgement Test?
Situational Judgement Tests (SJTs) are a Situational Judgement Tests (SJTs) are a measurement measurement methodmethod designed to assess judgement in workdesigned to assess judgement in work--relevant relevant situations:situations:
Present challenging situations likely to be encountered at workPresent challenging situations likely to be encountered at work
Make judgements about possible responsesMake judgements about possible responses
Scored against preScored against pre--determined keydetermined key
Validity (predictive & incremental) is wellValidity (predictive & incremental) is well--establishedestablished
Recent examples in postgraduate specialty selectionRecent examples in postgraduate specialty selection GP, CMT, Surgery, Acute specialties, Public HealthGP, CMT, Surgery, Acute specialties, Public Health
Example SJT item
You are reviewing a routine drug chart for a patient with rheumatoid arthritis during an overnight shift. You notice that your consultant has inappropriately prescribed methotrexate 7.5mg daily instead of weekly.
Rank in order the following actions in response to this situation (1= Most appropriate; 5= Least appropriate).
A Ask the nurses if the consultant has made any other drug errors recently
B Correct the prescription to 7.5mg weeklyC Leave the prescription unchanged until the consultant ward round
the following morning D Phone the consultant at home to ask about changing the
prescription E Inform the patient of the error
‘Job families’ of SJTs
Job family
1. Interventionist Surgery, Radiology, O&G
2. Family/Community-based
SJT Pilot studies
Surgery SJT pilot (Rowley et al)
Radiology SJT evaluation (Lindsell et al)
GP SJT (Irish, Jones, GPNRO) 2. Family/Community-based GP, Medicine, Psychiatry, Paeds
3. Acute Anaethesia, Emergency
medicine
4. Laboratory-based Histopathology
CMT SJT evaluation (Berkin, Burr, Lazell et al)
Public Health pilot (Pashynan, Williams)
Acute specialties SJT pilots (Gale, Anderson, Carr et al)
GP selectionGP selectionGP National Recruitment OfficeGP National Recruitment OfficeGP National Recruitment OfficeGP National Recruitment Office
GP SelectionGP Selection
6,500 applicants per year
job analysis to develop selection criteria
longitudinal validation studies
analysis of applicant reactions analysis of applicant reactions
Patterson F, Ferguson E, Lane P, Farrell K, Martlew J, Wells A. A competency model for general practice: implications for selection, training, and development. Br J Gen Pract 2000;50(452):188-93.
Patterson F, Ferguson E, Norfolk T, Lane P. A new selection system to recruit general practice registrars: preliminary findings from a validation study. BMJ 2005; 330(7493):711-4
National Deanery
GP Specialty Selection
Longlisting Shortlisting
Stage 2
Selection Centre
Stage 3
ElectronicApplicationProcess
GroupWritten SimulationClinical Problem solving testSituational Judgement test
Ranking
Matching to Deanery
Foundation Competency
National panel
GP Selection SystemGP Selection SystemPredictive & construct validity studies Predictive & construct validity studies Predictive & construct validity studies Predictive & construct validity studies Study 1. Predictive validity of short-listingStudy 2. Comparing the CPS, SJT & cognitive ability testsStudy 3. Supervisor ratings 1 year into trainingStudy 4. MRCGP outcomes after 2-3 years in training
Study 1. Predictive validity
CorrelationsCorrelations N N CorrelationsCorrelations
CPS & SJT 8399 .53**
CPS & Selection Centre 6824 .41**
SJT & Selection Centre 6824 .49**
Prediction Prediction of of SC SC (N=6824)(N=6824) AdjAdj RR22 RR2 2 changechange
CPS & SJT 0.28CPS & SJT 0.28
Incremental prediction of CPS over SJT .04**
Incremental prediction of SJT over CPS .11**
• SJT is the best single predictor for short-listing• Both tests are strong predictors & show incremental validity over the other
** p<.001 CPS – Clinical Problem-Solving test. SJT – Situational Judgement test
Study 2. Comparing the validity of Study 2. Comparing the validity of CPS, SJT & cognitive ability testsCPS, SJT & cognitive ability tests Although the cognitive ability tests were correlated with SC Although the cognitive ability tests were correlated with SC
outcomes, the current outcomes, the current shortlistingshortlisting tests remained the best tests remained the best predictors overall, particularly the SJTpredictors overall, particularly the SJT
Candidate reactions to the cognitive ability tests were negative, Candidate reactions to the cognitive ability tests were negative, Candidate reactions to the cognitive ability tests were negative, Candidate reactions to the cognitive ability tests were negative, especially for relevance & opportunity to demonstrate skills especially for relevance & opportunity to demonstrate skills
CorrelationsCorrelations
N=196 N=196 MeanMean SDSD αα CPSCPS SJTSJT SCSC
Supervisor ratings after 12 months24 items
4.63 .86 .94 .48** .50** .47**
Study 3. Predictive validity in trainingStudy 3. Predictive validity in training
** p<.001
• All methods (CPS, SJT & SC) are highly significant predictors of in-training performance
• Significant incremental validity accounted for by the SC for important domains (e.g. communication skills, empathy)
• Results indicate priorities for improving efficiency
Study 4. LongStudy 4. Long--term Predictive validityterm Predictive validity
Mean CorrelationsMean Correlations
NN MeanMean SDSD CPSCPS SJTSJT SelectionSelection CentreCentre**
MRCGP Applied Knowledge Test 6 diets
2,550 150.6 96.6 .76** .47** .29**Knowledge Test 6 diets
MRCGP Clinical SkillsOSCE 5 diets
1,277 9.3 2.2 .35** .48** .35**
* Uncorrected correlations for restriction of range ** p<.001
• The GP selection methods are highly significant predictors of end-of-training competence
Specialty Specific Definitions of Specialty Specific Definitions of Communication SkillsCommunication Skills
PaediatricsPaediatrics Communication SkillsCommunication Skills
Is able to switch conversational style from adult to child, and to manage Is able to switch conversational style from adult to child, and to manage the doctorthe doctor--patientpatient--parent triad during consultation; engages child in parent triad during consultation; engages child in conversation; positions self at child’s level when speaking to childconversation; positions self at child’s level when speaking to child
AnaestheticsAnaesthetics Communication SkillsCommunication Skills
Is able to Is able to verbaliseverbalise intentions, explain actions to various people in a intentions, explain actions to various people in a
Obstetrics & GynaecologyObstetrics & Gynaecology Communication SkillsCommunication Skills
Is able to discuss psychoIs able to discuss psycho--sexual problems with patient; uses open sexual problems with patient; uses open questions or puts ‘feeler’ questions out to patients who may be shy / questions or puts ‘feeler’ questions out to patients who may be shy / embarrassed to discuss symptoms; uses cues (such as fetal heart embarrassed to discuss symptoms; uses cues (such as fetal heart monitor) to illustrate explanationsmonitor) to illustrate explanations
Is able to Is able to verbaliseverbalise intentions, explain actions to various people in a intentions, explain actions to various people in a team (e.g. surgeons & ODAs); asks open questions at preteam (e.g. surgeons & ODAs); asks open questions at pre--med to med to ascertain the facts ascertain the facts -- rather than just running through a checklist; can rather than just running through a checklist; can offer clear explanation quickly to patients with establishing an onoffer clear explanation quickly to patients with establishing an on--going doctorgoing doctor--patient relationshippatient relationship
Job analysis to define selection criteria Job analysis to define selection criteria for Surgery for Surgery
Technical Knowledge Technical Knowledge & Clinical Expertise & Clinical Expertise
Judgement under Judgement under PressurePressure
Organisation & Organisation & PlanningPlanning
Personal AttributesPersonal Attributes Decision MakingDecision Making Professional Professional IntegrityIntegrityIntegrityIntegrity
Leadership & Team Leadership & Team InvolvementInvolvement
Situation Situation AwarenessAwareness
Learning & Learning & DevelopmentDevelopment
Communication SkillsCommunication Skills Problem SolvingProblem Solving Practical skillsPractical skills
Key messagesKey messages
1.1. Research evidence shows there is more commonality than Research evidence shows there is more commonality than difference between the specialtiesdifference between the specialtiesegeg. communication skills, professional integrity, problem. communication skills, professional integrity, problem--solvingsolving
2.2. Research shows different priorities between specialties at Research shows different priorities between specialties at 2.2. Research shows different priorities between specialties at Research shows different priorities between specialties at selectionselection
Paediatrics:Paediatrics: Communication skills, EmpathyCommunication skills, Empathy O&G: O&G: Professional Integrity, EmpathyProfessional Integrity, Empathy Anaesthesia:Anaesthesia: Vigilance & Situational awarenessVigilance & Situational awareness
AoMRCAoMRC MMT PilotMMT Pilot
Investigating:Investigating:
1.1. If a single test could work for a range of specialtiesIf a single test could work for a range of specialties
2.2. If several specialties & deaneries can collaborateIf several specialties & deaneries can collaborate
3.3. Which test format is best Which test format is best -- a CPS, an SJT or botha CPS, an SJT or both3.3. Which test format is best Which test format is best -- a CPS, an SJT or botha CPS, an SJT or both
4.4. If computer based testing makes sense with respect toIf computer based testing makes sense with respect to costcost logisticslogistics stakeholder acceptabilitystakeholder acceptability
AoMRC MMT Pilot
Multispecialty/multideanery
Computer based
CPS & Acute Specialties SJT
Anaesthetics, ACCS, GP, Anaesthetics, ACCS, GP, Histopathology, Medicine, & Paediatrics
25 locations, over 2 days
N= 682 approx
Origins & evidenceOrigins & evidence When designed appropriately When designed appropriately SCs areSCs are the best predictor of the best predictor of
future work performance for future work performance for nonnon--cognitive skillscognitive skills RReliabilityeliability && validity validity gains gains -- combining different selection tools, combining different selection tools,
standardised scoring systems standardised scoring systems SCsSCs usesuses jobjob--relevant exercisesrelevant exercises,, observation by observation by independent, independent, SCsSCs usesuses jobjob--relevant exercisesrelevant exercises,, observation by observation by independent, independent,
trained assessors trained assessors Recent reviews estimate over 95% of large organisations Recent reviews estimate over 95% of large organisations
employing more than 10,000 individuals, use SCs for graduate employing more than 10,000 individuals, use SCs for graduate recruitment & almost all high stakes selection used SC recruitment & almost all high stakes selection used SC techniquestechniques
Candidate Reactions: Percentage Candidate Reactions: Percentage agreement reported for each methodagreement reported for each method
2008 2009
S Listing N=5866
Selection Centre N=4631
S ListingN=2894
Selection CentreN=3571
CPS SJT Sim Written Group CPS SJT Sim Written Group
Relevant 84% 52% 89% 71% 78% 90% 64% 85% 64% 76%
Fair 78% 50% 89% 75% 81% 86% 53% 86% 75% 81%
Appropriate 78% 57% 91% 80% 83% 86% 69%
Opportunity 56% 38% 75% 47% 49% 73% 35%
Candidate Reactions: Percentage Candidate Reactions: Percentage agreement reported for each methodagreement reported for each method
2008 2009
S Listing N=5866
Selection Centre N=4631
S ListingN=2894
Selection CentreN=3571
CPS SJT Sim Written Group CPS SJT Sim Written Group
Relevant 84% 52% 89% 71% 78% 90% 64% 85% 64% 76%
Fair 78% 50% 89% 75% 81% 86% 53% 86% 75% 81%
Appropriate 78% 57% 91% 80% 83% 86% 69%
Opportunity 56% 38% 75% 47% 49% 73% 35%
Patient simulation
viewed positively
Implications
• High fidelity assessments (valid, but costly) are rated as significantly more relevant & fair by candidates
• Introduced methods beyond academic qualifications to select doctors (eg. interpersonal skills, integrity, empathy)
• Disjoint between operational validity vs. stakeholder acceptance ie. - “Why shouldn’t climbing Everest count”?
• Political validity important concept in this context
SummarySummary
Undergraduate and specialty postgraduate Undergraduate and specialty postgraduate ‘selection’ may have different goals ‘selection’ may have different goals --undergraduate courses selecting primarily on undergraduate courses selecting primarily on academic ability, with a focus on passing the academic ability, with a focus on passing the course and postgraduate selection focuses more course and postgraduate selection focuses more course and postgraduate selection focuses more course and postgraduate selection focuses more on jobon job--fit…it cannot necessarily be assumed that fit…it cannot necessarily be assumed that those with high academic ability alone can be those with high academic ability alone can be turned into potentially good doctors via medical turned into potentially good doctors via medical training training -- other skills & qualities may need to be other skills & qualities may need to be present from the start.”present from the start.”