the effect of interviewer on rank list: an imperfect science becomes more imperfect
DESCRIPTION
The Effect of Interviewer on Rank List: An Imperfect Science Becomes More Imperfect. Daniel Vargo, MD Program Director, General Surgery Associate Professor, Dept. of Surgery University of Utah School of Medicine. Disclosures. None. Background. Applicant Selection: Surgery Job Description - PowerPoint PPT PresentationTRANSCRIPT
The Effect of Interviewer on Rank List: An Imperfect Science Becomes More
Imperfect
Daniel Vargo, MDProgram Director, General Surgery
Associate Professor, Dept. of SurgeryUniversity of Utah School of Medicine
Disclosures
• None
Background
• Applicant Selection: Surgery– Job Description
• Website• “Red Book”
– Applications• 526 last year
– Nebulous scoring system– Interviews
Background
• Interviews– Interviewers not HR trained– Go on “gut feeling”
• “I wanted to like/not like this candidate”
– Spend interview trying to validate feelings or impressions• Trip up questions
– Base opinion on “unusual” things…..
His socks didn’tmatch his pants
Who wears a pants suitto an interview
He had this weirdlook in his eyes
PGY-1 Summative Meeting
• Interview Comments vs. Performance– No correlation– Lowest scored intern last two years highest
performance
“Six Sigma” Evaluation
• Took process apart• Biggest perceived variable
– Interviews
Question
• How variable are the interviews?• What effect does this variability have on
process?
Methods
• 5 years data• Interviewers and scores
– “Easy Scorers”– “Hard Scorers”
• Applicants– Strong– Average– Weak
Methods
• Applicant group ranking– Compared with interview panel composition
Results
• 30 Interviewers• 303 applicants
– 909 interviews
Applicant Distribution
Strong Average Weak0
50
100
150
200
250
70(23%)
200(66%)
33(11%)
Top 10Appl.
>40Or NR
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 3050
55
60
65
70
75
80
85
90
95
100
Interview Score
“ES”
“HS”
Interview Scores:All Candidates
303 Appl Strong(70)
Average(200)
Weak(33)
ES(20) 95 91 82
HS(10) 91 81 71
ES= Easy ScorerHS= Hard Scorer
Scoring: Strong Applicants
HS=Hard ScorerP= NS
70 Appl 0-1 HS(44)
2-3 HS(26)
Ave. Int. Score 95 92
Ave. Rank Position 6 7
Scoring: Average Applicants
200 Appl. 0 HS(34)
1 HS(85)
2 HS(60)
3 HS(21)
Ave Int Score 91 89 82 76
Ave Composite
Score830 817 789 740
Rank ListPosition 22 24 26 31
Scoring: Average Applicants
200 Appl. 0 HS(34)
3 HS(21)
Ave Int Score 91 76*
Ave Composite Score 830 740*
Rank ListPosition 22 31
* p<0.05
200 Appl
134 Discussed
66 Not Discussed
200 Appl
134 Discussed
24 No HS
106 ≥ 1 HS
66 Not Discussed
200 Appl
134 Discussed
24 No HS
106 ≥ 1 HS
66 Not Discussed
6 No HS
60 ≥ 1 HS
200 Appl
134 Discussed
24 No HS
106 ≥ 1 HS
66 Not Discussed
6 No HS
60 ≥ 1 HS
Results
• ≥ 1 HS– Lower interview scores– Lower composite scores– Lower position on rank list– Less likely to be discussed at rank meeting
Assumptions
• Candidate pools are equally distributed• Interviewer “toughness” did not vary• Other variables in score calculation
consistent
Conclusions
• Interviewers do vary in type• Scores effect applicants• Another area of variability to be addressed
in the interview process