can a structured, behavior-based interview predict future resident success?

bmgds

0©d

Research www.AJOG.org

EDUCATION

Can a structured, behavior-based interviewpredict future resident success?Eric A. Strand, MD; Elizabeth Moore, PhD; Douglas W. Laube, MD, MEd

OBJECTIVE: To determine whether a structured, behavior-based appli-cant interview predicts future success in an obstetrics and gynecologyresidency program.

STUDY DESIGN: Using a modified pre-post study design, we comparedehavior-based interview scores of our residency applicants to a post-atch evaluation completed by the applicant’s current residency pro-

ram director. Applicants were evaluated on the following areas: aca-emic record, professionalism, leadership, trainability/suitability for thepecialty, and fit for the program.

RESULTS: Information was obtained for 45 (63%) applicants. The over-
all interview score did not correlate with overall resident performance.
2011;204:446.e1-13.

pcuttmtMhwnf

rfrcology residency trainin

See related editorial, page 369

446.e1 American Journal of Obstetrics & Gynecology MAY 2011

Applicant leadership subscore was predictive of leadership perfor-mance as a resident (P � .042). Academic record was associated withpatient care performance as a resident (P � .014), but only for gradu-ates of US medical schools. Five residents changed programs; theseresidents had significantly lower scores for trainability/suitability for thespecialty (P � .020).

CONCLUSION: Behavioral interviewing can provide predictive informa-tion regarding success in an obstetrics and gynecology trainingprogram.

Key words: internship/residency, personnel selection, professional
competence
Cite this article as: Strand EA, Moore E, Laube DW. Can a structured, behavior-based interview predict future resident success? Am J Obstet Gynecol

t
Each academic year, residency pro-grams and applicants expend signifi-
cant time, energy, and financial resourcesinterviewing applicants for future resi-dency positions. In the 2009 National Res-idency Matching Program (NRMP) main

From the Department of Obstetrics andGynecology (Drs Strand and Moore), St.Vincent Hospital, Indianapolis, IN, and theDepartment of Obstetrics and Gynecology(Dr Laube), University of Wisconsin,Madison, WI.

Presented at the annual joint meeting of theAssociation of Professors of Gynecology andObstetrics and the Council on ResidentEducation in Obstetrics and Gynecology,Orlando, FL, March 3-6, 2010.

Received Aug. 26, 2010; revised Nov. 8, 2010;accepted Feb. 4, 2011.

Reprints: Eric A. Strand, MD, Department ofObstetrics and Gynecology, St. VincentHospital, 8111 Township Line Rd, Indianapolis,IN 46260. [email protected].

002-9378/free2011 Mosby, Inc. All rights reserved.

oi: 10.1016/j.ajog.2011.02.019

For Editors’ Commentary,see Table of Contents

residency match, a total of 51,882 appli-cants pursued the 25,185 training posi-tions available in the United States. Thisincluded 1796 applicants for the 1185 res-idency positions in the specialty of obstet-rics and gynecology.1

Despite these investments, little is knownabout the use of the residency interview inpredicting an applicant’s future perfor-mance as a resident. For instance, Metro etal2 reviewed interview scores for their ap-

licants to determine whether the scoresorrelated with any measures used to eval-ate the residents during their training. In-

erview scores did not correlate with any ofhe measures, including knowledge, judg-

ent, motor skills, or intrapersonal atti-udes.2 Performance on the United States

edical Licensing Examination (USMLE)as been shown to be positively correlatedith intraining service examinations, butot with faculty evaluation of resident per-

ormance.3,4 Other studies have shown in-terview scores did not predict physicians atrisk of later impairment.5 Furthermore,esident attrition is a significant problemor many programs. After a cohort of 1055esidents started their obstetrics and gyne-

g in 2001, McAlis-

er et al6 found that 21.6% of residentseither: switched programs, changed spe-cialties, completed training on an atypicalacademic cycle, or left graduate medicaleducation all together.

In 2006, the St. Vincent Hospital De-partment of Obstetrics/Gynecoloy em-barked on a program of behavior-basedinterviewing for all applicants to its resi-dency program, with the hope of identi-fying applicants who would be successfulboth in the chosen specialty and in the St.Vincent program. This study reports theresults of this interview process to pre-dict future resident performance.

MATERIALS AND METHODSThe St. Vincent Hospital Obstetrics/Gy-necology residency is a community-based program consisting of 4 residentsper academic year (approved for expan-sion to 5 positions in June 2007). Thehospital is 1 of 4 main teaching sites forthe required third year obstetrics/gyne-cology clerkship for medical studentsfrom the Indiana University School ofMedicine. Approximately 200-250 stu-dents apply for the available residencypositions, with the majority of the USgraduates matriculating from programs
in the Midwest region.
mailto:[email protected]

www.AJOG.org Education Research

Beginning in November 2006, theSt. Vincent Hospital Department ofObstetrics/Gynecology embarked on astructured behavior-based interview forapplicants to its residency program. Ap-plicants were scored on the following ar-eas: academic record, professionalism,leadership, trainability/suitability for thespecialty, and fit for the program. Aca-demic record was scored by the programdirector through a review of the infor-mation available in the written applica-tion. The other 4 aspects were measuredthrough a series of behavior-based ques-tions at individual interview stations.Interviews for professionalism, leader-ship, and trainability/suitability wereconducted simultaneously by 2 facultymembers. When scheduling preventedthe presence of both members, 1 facultymember conducted the interview. Fit forthe program was assessed by a panel of 3residents. Through literature review anddiscussions with subject matter experts,behavior-based questions pertinent toeach subject area were developed, aswere scoring sheets with specific exam-

FIGURE 1Overall interview score vs overall pas assessed by program director

P � .784.

Strand. Predicting resident success. Am J Obstet Gynecol 2011

ples of scores for particular responses

(measured aspects of each subject area,sample scoring sheets, and sample ques-tions, Supplementary Figures 1-4). Fac-ulty and residents were educated regard-ing the new interview protocol during aseries of formal and informal meetings.During the interviews, faculty and resi-dents could ask any of the potential ques-tions available for their session, allowingsome flexibility from interview to inter-view. At the end of each interview en-counter, every interviewing faculty orresident completed their score sheet sep-arately and independently from theirpartner(s). Scores for all 5 categorieswere determined by the average of theinterviewer’s scores. With each categoryhaving a potential value of 36, the maxi-mum interview score for each applicantwas 180. These scores provided the initialranking list of applicants on which all fu-ture discussions were based. A databasewas maintained with all interview data,as well as medical school location (US-based program or international-basedprogram).

For all applicants from November

ormance

2006 through January 2008, eventual

MAY 2011 Americ

match sites were identified through acombination of (1) the NRMP MatchResults by Matched Applicant providedto the St. Vincent program, (2) theCouncil on Resident Education in Ob-stetrics and Gynecology (CREOG) resi-dent database, and (3) individual pro-gram web site reviews. The programdirector for each program was identifiedthrough the listing of accredited pro-grams at the Accreditation Council forGraduate Medical Education website(www.acgme.org).

An electronic survey was developedand sent to each director of a programinto which a St. Vincent applicant hadmatched. Program director e-mails wereidentified through the AGCME data-base. E-mails with links to the electronicsurvey were sent to the appropriate pro-gram directors in September 2009, atwhich time the former applicants wouldhave completed either 1 or 2 years of res-idency training. The survey was designedto measure the following resident attri-butes:

● Patient care● Medical knowledge● Surgical skills● Communication● Professionalism● Clinical documentation● Leadership● Teamwork● Overall impressionIn addition, questions regarding awards,

disciplinary actions for academic or pro-fessionalism-related issues, and the resi-dent’s continued status in the programwere included. If the resident was no lon-ger with the training program, program di-rectors were asked to explain the change intraining status. Information on type ofprogram (community or university) wasalso collected.

For residents matching into the St. Vin-cent training program, it was thought theprogram director’s completion of the sur-vey would lead to potential bias, as he is theprincipal investigator of this study (E.S.).Therefore, the same survey for St. Vincentresidents was completed by the consensusopinion of a core group of 3 faculty mem-bers with extensive clinical experience with

erf

.

the residents.

an Journal of Obstetrics & Gynecology 446.e2

http://www.acgme.org

ctbb

Research Education www.AJOG.org

To maintain applicant confidentiality,once data were collected from the elec-tronic survey and matched to the origi-nal interview score, all data were deiden-tified. When means between groups werecompared, Student t tests were per-formed to determine statistical signifi-cance. Linear regression was used to de-termine relationships between pretest(interview) and posttest (electronic sur-vey) scores. A P value of � .05 was con-sidered significant.

The project was submitted to the St.Vincent Hospital Institutional ReviewBoard (IRB) and was granted exemptstatus as an educational project.

RESULTSFrom November 2006 through January2008, a total of 80 applicants interviewedfor residency positions in obstetrics/gynecology at St. Vincent Hospital. Ofthese applicants, 65 (81%) were women,15 (19%) were men, and 74 (92.5%)were from US allopathic training pro-grams. Of these, 72 matched into obstet-rics/gynecology training programs, and8 either did not match or matched intoother specialty programs (4 family med-icine, 1 pediatrics, 1 general surgery, 1anesthesia, and 1 unmatched).

The 72 applicants matched into a total of42 obstetrics/gynecology residency train-ing programs. Of the 42 electronic surveyssent to these program directors, a total of24 surveys (57%) were completed, provid-ing follow-up data for 36 of the applicants.Including the 9 residents matching into theSt. Vincent program, this provided fol-low-up data for 45 (63%) of the applicantsmatching into obstetrics/gynecology train-ing programs.

In evaluating the assessment of overallresident performance by their programdirector, there was not a significant rela-tionship between the overall interviewscore and the overall performance score(P � .784, Figure 1). This remained thease when separating results based byype of training program (universityased, n � 25, P � .337; communityased, n � 20, P � .952).Scores for the applicant’s academic re-

cord did not predict resident scores for pa-
tient care (P � .147) or medical knowledge
446.e3 American Journal of Obstetrics & Gynecolo

(P � .125). However, when analyzing re-sults for only graduates of US medicalschools (43/45 applicants with availabledata), a significant relationship did existbetween academic record and patient care(P � .014, Figure 2). Scores for medicalknowledge, though, remained nonsignifi-cant (P � .075).

Applicant scores for professionalism didnot predict resident scores for profession-alism (P � .685, Figure 3). However, a re-lationship did exist between applicantscores in leadership and leadership scoresas a resident (P � .042, Figure 4).

Of the 45 applicants with availabledata, 5 (11%) changed programs dur-ing the time of the study (3 changed todifferent obstetrics/gynecology pro-grams, 1 to anesthesia, and 1 to emer-gency medicine). Based on univariatelogistic regression of interview scoresto program status (still in program orchanged programs), only trainability/suitability was a significant predictorof remaining in the training program

FIGURE 2Interview score of academic recordas a resident, US medical school g

P � .014.


(P � .020, Table 1). Fit for the program

gy MAY 2011

was not predictive of resident attrition(P � .898, Table 1).

COMMENTDeveloped by industrial psychologists,behavioral interviewing focuses on thepremise that past performance is the bestindicator of future performance. Insteadof using vague questions such as “Whatare your strengths and weaknesses?” be-havioral interviews involve questions re-garding specific behaviors that oftenrepresent decision-making skills, criticalthinking, and interpersonal communi-cation styles. In one of the first studies toevaluate the use of behavioral interviews,Janz7 compared behavior-based inter-views to traditional, unstructured inter-views of prospective teaching assistants.Using student ratings of the teaching as-sistants as the outcome measure, behav-ioral interviews were significantly morepredictive of performance than the tra-ditional interview.7 Given the potentialbenefit, many businesses have changed

s patient care performanceuates only

vrad

.

to behavior-based interviewing, includ-

ctrs


MAY 2011 Americ

ing 65% of Fortune 500 companies.8 Be-ause cognitive factors, such as grades/ranscripts, USMLE scores, and classank have not correlated with clinicaluccess as a resident,3-4,9,10 programs are

turning to behavioral interviewing forcandidate assessment.11

In this study of a standardized behav-ior-based interview process, importantsignificant associations were found. Spe-cifically, applicants with higher scoresfor leadership tended to be rated higherfor this characteristic as a resident. In ad-dition, applicants who changed trainingprograms had lower interview scores intrainability/suitability. This may illus-trate an important risk factor to be con-sidered in selecting an applicant, giventhe significant time, expense, and dis-ruption that occurs when a residentleaves a program and a replacement issought.

Interestingly, one measure of ourapplicant assessment, academic record,was not behavior-based and yet did sig-nificantly correlate with patient care per-formance as a resident. This stands incontrast to the cognitive factors assessedin the previous studies. Our finding wassignificant only when comparing datafor graduates of US medical schools. Theassociation lost significance when datafor applicants from international med-ical schools was included. This may in-

TABLEUnivariate logistic regression ofinterview scores vs continuedstatus in the program

Interview item

Prediction ofremainingin program(P value)

Academic record .930...........................................................................................................

Professionalism .916...........................................................................................................

Leadership .430...........................................................................................................

Suitability/trainability .020...........................................................................................................

Fit for program .898...........................................................................................................

Overall score .575...........................................................................................................

Strand. Predicting resident success. Am J ObstetGynecol 2011.

FIGURE 3Interview professionalism score vs professionalism rating as a resident

P � .685.

Strand. Predicting resident success. Am J Obstet Gynecol 2011.

FIGURE 4Interview leadership score vs leadership rating as a resident

P � .042.


dicate that cognitive factors (USMLE

an Journal of Obstetrics & Gynecology 446.e4

pNysIbdscT(dwv“al

briipaspfs

mepolaRrit

fphvu

wuttsTev

sdkodpltrPdanamgaAocop

ttrTmiivo


scores, transcripts) for international grad-uates may be particularly poor predictorsof future performance, and should beconsidered in the applicant’s overallassessment.

Our global interview score, though,did not predict future performance as aresident. This stands in contrast to astudy by Olawaiye et al,12 where an ap-

licant’s position on the program’sRMP rank list did predict future first

ear clinical performance. There may beeveral reasons for the different findings.n the study by Olawaiye et al,12 similarut different categories were measureduring the interview— communicationkills, insight into specialty, motivation,ompassion, and “fit” into the program.he nature of the interview questions

behavior-based or otherwise) was notescribed. Finally, resident performanceas not compared with the overall inter-iew score (or subscore), but rather torank percentile,” which represented thepplicant’s position on the final rankingist.

Our study is the first attempt to useehavioral interviewing to predict futureesident performance. In addition, ourss the first to go beyond global scores andnvestigate subscores such as leadership,rofessionalism, and trainability/suit-bility as possible predictors of futureuccess. These findings point to areas ofotential future study, as programs look

or predictive characteristics of residentuccess or failure.

One strength of the study is its assess-ent of residents in a number of differ-

nt programs. Program directors com-leting the survey were unaware of theriginal applicant’s evaluation, which al-

owed for an unbiased assessment of thepplicant’s performance as a resident.emoving the St. Vincent program di-

ector from the evaluation process lim-ted this bias as well, although a poten-
ial for bias remained if the St. Vincent f
446.e5 American Journal of Obstetrics & Gynecolo

aculty involved remembered the ap-licant’s interview results. Interviewsad occurred roughly 2 to 3 years pre-iously, which makes significant biasnlikely.An additional strength of the studyas the preparation of residents and fac-lty for the new interview technique, and

he creation of standardized gradingemplates with specific examples of an-wers corresponding to specific marks.his should have limited variation invaluations submitted on different inter-iew dates.There are several weaknesses to the

tudy. Although the survey providedata for 63% of our applicants, it is un-nown if the remaining 37% were morer less successful as residents. Programirectors with residents demonstratingoor performance may have been less

ikely to complete a survey regardingheir performance, despite reassurancesegarding confidentiality of the subjects.rogram expectations of a quality resi-ent may also be substantially differentmong programs. Although our data didot show significant differences whennalyzed comparing results from com-unity-based or university-based pro-

rams, variations in programs may cre-te different definitions of “success.”lthough we attempted to standardizeur survey for the program directors, weertainly must allow that expectationsf performance vary from program torogram.Overall, our study points to the poten-

ial usefulness of behavioral interviewingo identify either predictors of success orisk factors for failure during residency.rainability/suitability, in particular,ay be a subject of further interest in

dentifying applicants at risk for chang-ng programs. Future studies should in-estigate additional noncognitive traitsf our applicants to help better predict
uture residency success. f
gy MAY 2011

REFERENCES1. National Residency Matching Program re-sults and data: 2009 main residency match.National Residency Matching Program, 2009(vol 2009).2. Metro DG, Talarico JF, Patel RM, WetmoreAL.The resident application process and its cor-relation to future performance as a resident.Anesth Analg 2005;100:502-5.3. Bell JG, Kanellitsas I, Shaffer L. Selection ofobstetrics and gynecology residents on the ba-sis of medical school performance. Am J ObstetGynecol 2002;186:1091-4.4. Brothers TE, Wetherholt S. Importance of thefaculty interview during the resident applicationprocess. J Surg Educ 2007;64:378-85.5. Dubovsky SL, Gendel M, Dubovsky AN,Rosse J, Levin R, House R. Do data obtainedfrom admissions interviews and residentevaluations predict later personal and prac-tice problems? Acad Psychiatry 2005;29:443-7.6. McAlister RP, Andriole DA, Brotherton SE,Jeffe DB. Attrition in residents entering US ob-stetrics and gynecology residencies: analysis ofnational GME census data. Am J Obstet Gyne-col 2008;199:574.e1-6.7. Janz T. Initial comparisons of patterned be-havior-based interviews versus unstructured in-terviews. J Appl Psychol 1982;67:577-80.8. Byers M. Interview Rx: a powerful guide formaking your next interview a success, 3rd ed.Conyers, GA: Nearline Publishers; 2007.9. Erlandson EE, Calhoun JG, Barrack FM, et al.Resident selection: applicant selection criteriacompared with performance. Surgery 1982;92:270-5.10. Hojat M, Gonnella JS, Veloski JJ, ErdmannJB. Is the glass half full or half empty? A reex-amination of the associations between assess-ment measures during medical school and clin-ical competence after graduation. Acad Med1993;68:S69-76.11. Lyon D, Wiper D. Finding our colleagues,finding ourselves: behavioral interviewing and acritical assessment of how we identify and in-terview residency candidates. APGO/CREOGAnnual Meeting. San Diego, CA, 2009.12. Olawaiye A, Yeh J, Withiam-Leitch M.Resident selection process and prediction ofclinical performance in an obstetrics and gy-necology program. Teach Learn Med 2006;18:310-5.


SUPPLEMENTARY FIGURE 1Factors measured in St. Vincent’sbehavioral interviews

Strand. Predicting resident success. Am J Obstet Gynecol2011.

MAY 2011 American Journal of Obstetrics & Gynecology 446.e6


SUPPLEMENTARYEvaluation scor

Continued on next pagStrand. Predicting resident s

446.e7 American Journa

FIGURE 2ing sheets

e.uccess. Am J Obstet Gynecol 2011.

l of Obstetrics & Gynecology MAY 2011



Professio

1. Interview Day/Appearan

1

Unkempt; rude; late

2. Ethics

1

Cannot identify ethical challenge

or situation

3. Stress

1

Denies stressful situations or

being effected by stress

4. Initiative

1

Offers no evidence of

personal initiative

Continued from the preStrand. Predicting resident s

FIGURE 2ing sheetsnalism

ce

3 5 7 9

Well-groomed; slow or difficult

to engage

Appropriately dressed, timely, polite; interacts well but must be

led by interviewers

Pleasant, courteous, well-

groomed; enough self-confidence to

have some poise in interaction

Pleasant, courteous, well-groomed; self-assured, well-

spoken; completely

comfortable with the interview

3 5 7 9

Can identify ethical situation or challenge, but

has no insight into response

Shows basic understanding of

complexities within ethical

dilemmas

Demonstrates understanding of ethical principles

(informed consent,

beneficence, etc.)

Provides engaging

discussion of difficult ethical

situations

3 5 7 9

Aware of stressful

situations but not of personal

effects

Describes stressful

situation and some evidence

of personal approach

Has clearly defined ways of

personally handling stress

Describes stressful situation,

well-adjusted response, and

individual growth from experience

3 5 7 9

Limited evidence of initiative

Repeated evidence of initiative in

professional or personal life

Evidence of initiative leading to personal pride

Describes several situations

requiring initiative as well as how

this contributed to personal growth

Total Score:________________

vious page.uccess. Am J Obstet Gynecol 2011.





446.e9 American Journa

FIGURE 2ing sheets


l of Obstetrics & Gynecology MAY 2011




FIGURE 2ing sheets






446.e11 American Journ

FIGURE 2ing sheets


al of Obstetrics & Gynecology MAY 2011


SUPPLEMENTARY FIGURE 3Sample interview questions




SUPPLEMENTARY FIGURE 4Resident performance survey

.
446.e13 American Journal of Obstetrics & Gynecology MAY 2011

can a structured, behavior-based interview predict future resident success?

Documents