Does Medical School Performance Predict Radiology Resident Performance?
Post on 25-Dec-2016
Embed Size (px)
Does Medical School Performance PredictRadiology Resident Performance?1
Tedric D. Boyse, MD, Stephanie K. Patterson, MD, Richard H. Cohan, MD, Melvyn Korobkin, MDJames T. Fitzgerald, PhD, Mary S. Oh, MS, Barry H. Gross, MD, Douglas J. Quint, MD
Rationale and Objectives. The authors performed this study to examine the relationship, if any, of a large number ofmeasures of medical school performance with radiology residency performance.Materials and Methods. Applications of 77 radiology residents enrolled from 1991 to 2000 were reviewed. Medical schoolgrades, deans letter summary statements, letters of recommendation, selection to Alpha Omega Alpha (AOA), and NationalBoard of Medical Examiners (NBME) and U.S. Medical Licensing Examination (USMLE) Step 1 scores were recorded. Stu-dent t tests, analysis of variance, and correlation coefficients were used to examine the relationship between these measures ofmedical school performance and subsequent performance during radiology residency as determined by rotation evaluations, ret-rospective faculty recall scores, and American College of Radiology (ACR) and American Board of Radiology (ABR) examina-tion scores. Resident performance was also correlated with prestige of the medical school attended.Results. Preclinical grades of Honors or A; clinical grades of Honors or A in medicine, surgery, and pediatrics; and highNBME/USMLE scores strongly predicted success on the ABR written clinical examination but did not predict rotationperformance. Most other measures of medical school performance, including outstanding Deans letters and letters of rec-ommendation, AOA selection during the senior year, and high medical school prestige did not predict high examinationscores or superior rotation performance during residency.Conclusion. Success on the ABR examination can be predicted by medical school success in preclinical courses, someclinical courses, and USMLE examination scores. Deans letters, letters of recommendation, AOA selection during thesenior year, and medical school prestige do not appear to predict future resident performance as reliably.Key Words. Radiology residency performance, residency performance, American Board of Radiology ExaminationScores, American College of Radiology Examination Scores, residency rotation performance. AUR, 2002
Radiology resident selection is vitally important both toresidency training programs and to residency candidatesentering the match. Resident selection committees have aresponsibility to their departments to rank the applicantswho are most qualified and best suited for that particular
program. The committees also have an obligation to thecandidates, whose careers, lives, and families are greatlyaffected by the committees decisions. These responsibili-ties demand fairness and due process toward each candi-date.
Given the increasingly competitive pool of applicants,this is no small task. Accordingly, most radiology resi-dency programs devote tremendous time and effort toranking candidates. A variety of measures of medicalschool performance are considered. In one survey (1),more than 90% of program directors considered medicalschool grades and class rank to be very important factors,
Acad Radiol 2002; 9:437445
1 From the Departments of Radiology (T.D.B., S.K.P., R.H.C., M.K., B.H.G.,D.J.Q.) and Medical Education (J.T.F, M.S.O.), University of Michigan Hos-pitals, 1500 East Medical Center Drive, Ann Arbor, MI 48109-0030. Re-ceived October 8, 2001; revision requested November 14; revision receivedand accepted December 19. Address correspondence to R.H.C.
74% regarded selection to Alpha Omega Alpha (AOA) tobe very important, and just more than 50% emphasizedscores on the U.S. Medical Licensing Examination(USMLE).
Despite widespread use of this information in residentselection, few data indicate whether any of these mea-sures of medical school performance predict a given can-didates subsequent success as a resident. The few pub-lished studies on this topic have reported conflicting re-sults (2). For example, three studies (35) failed to findany correlation between USMLE scores and later resi-dency performance, whereas two others (6,7) demon-strated a positive correlation. One study (8) showed thatclass rank based on medical school grades had a mod-est correlation with future residency performance; an-other (9) revealed that applicant rank order did not predictsubsequent residency performance. Most of these pub-lished studies (and several others) have included smallsample sizes and utilized only short-term follow-up, usu-ally assessing resident performance solely during the 1styear of postgraduate training (7,8,10). Furthermore, lim-ited data specifically address the assessment of future ra-diology residents. Only three of the previously cited stud-ies (3,4,9) assessed the performance of medical studentswho subsequently entered radiology residencies, and eachof these studies was limited in scope.
We performed this study to examine the relationship, ifany, between a large number of measures of medicalschool performance and radiology residency performance.Our hope was to determine which elements of a medicalstudents application are most predictive of future successin a radiology residency. It might then be possible to givespecial consideration to these elements during residentselection.
MATERIALS AND METHODS
We reviewed the records of 77 residents who com-pleted 4 years of training in our diagnostic radiology resi-dency program between 1991 and 2000. Information fromthe residency application regarding medical school perfor-mance and data regarding subsequent residency perfor-mance were recorded.
Assessment of Medical School PerformanceWe recorded data from each residents radiology resi-
dency application, including medical school transcripts,National Board of Medical Examiners (NBME) orUSMLE Step 1 results, deans letters, and letters of rec-
ommendation. Selection to AOA was noted, as well. Wealso recorded the perceived prestige of each residentsmedical school (determined according to the U.S. Newsand World Report listing of best graduate schools in2001). Finally, we noted whether the applicant attended amedical school located in the same general region (withinthe same state or neighboring states) as our program orhad any other substantial connection to the region, suchas a regional birthplace or college.
Performance in five preclinical (anatomy, biochemistry,pathology, pharmacology, and physiology) and five clini-cal (medicine, surgery, pediatrics, obstetrics and gynecol-ogy, and radiology) courses was reviewed. Students wereplaced into one of two categories: those who receivedgrades of Honors or A, and those who did not. Nodata were recorded for the five applicants who attended amedical school in which only passing or failing grades(without the Honors or A designation) were assigned.
As mentioned, the NBME and USMLE Step 1 scoreswere recorded. Because these two examinations employdifferent scoring systems, the results for each were re-corded separately. A normalized scoring system was alsodevised so that scores for each examination could be con-sidered together.
Deans letters were sorted into one of two categories:those in which the word outstanding was used in thesummary statement or in which the overall tone indicatedsuperlative medical school performance, and those inwhich the tone reflected less than outstanding perfor-mance. Twenty-nine of the 67 available deans letterswere classified as indicating outstanding performance.
Three letters of recommendation were evaluated foreach resident. These letters were sorted into one of twocategories: those describing the student as the best candi-date in his or her class or as one of the best students withwhom the author had interacted, and those that were posi-tive but did not describe the resident in such glowingterms.
We categorized each residents medical school byplacing it into one of three categories: (a) elite schools(ranked 112 according to the U.S. News and World Re-port survey), (b) competitive but not elite schools (ranked1350 according to the U.S. News and World Report sur-vey), and (c) all other schools.
Assessment of Radiology ResidencyPerformance
Residency performance was assessed in three ways:(a) by a review of rotation evaluation forms, (b) by a sur-
BOYSE ET AL Academic Radiology, Vol 9, No 4, April 2002
vey of three faculty members regarding resident perfor-mance, and (c) by American College of Radiology (ACR)in-training examination scores during the first three yearsof residency and the American Board of Radiology(ABR) written clinical examination score during the 4thyear of residency.
The method utilized for considering the rotation evalu-ation forms has been described previously (9). Briefly,evaluation forms were obtained for each resident through-out his or her residency at the conclusion of each 34-week subspecialty rotation. Usually, the rotation directorsor the faculty members who had the most contact withthe resident completed that residents evaluation form.Although residents were subjectively evaluated in a num-ber of categories, we chose to record data from two cate-gories applicable to every rotation: general knowledgeand overall performance. General knowledge consisted ofan assessment of general medical and radiology knowl-edge as it related to the specific rotation. Overall perfor-mance reflected a broad impression of how the residentperformed during the rotation. No specific criteria wereavailable to guide the faculty members in their evalua-tions. A mean score from all evaluation forms was calcu-lated for each resident in general knowledge and overallperformance for each year of residency; these scores wereassumed to reflect the residents performance during thatyear of residency. Because two different evaluation formswere employed during the study period (a five-point scalefrom 1991 to 1995 and a four-point scale from 1995 to2000), we created a normalized z-score for the evaluationform ratings so that the two types of evaluation formscould be combined.
Three senior faculty members with expertise in threedifferent subspecialty areas performed the retrospectiveevaluation of residents. A four-point grading scale wasused. The faculty members were asked to assess residentsin the same two categories that were used on the rotationevaluation forms (general knowledge and overall perfor-mance). The faculty members were also asked, however,to evaluate the residents in an additional category: inter-personal skills. We added this additional category becausewe believed that an assessment of interpersonal skillsmight reflect more closely measures of performance usedduring some medical school clinical rotations.
At our institution, 1st-, 2nd-, and 3rd-year residentsregularly take the ACR examination. The overall numberof answers correct, percentage scores, and percentilescores for this examination were recorded for each resi-dent. With the exception of 1991 and 1992, the 4th-year
residents did not take this examination. Because the num-ber of 4th-year ACR examination scores was very small(n 11), these results were not included in our analysis.We also recorded the overall scores for the ABR writtenclinical examination.
Statistical AnalysesThe measures of medical school performance were
compared with the measures of radiology residency per-formance. Given the variety of data acquired for thisstudy, a number of different statistical methods were em-ployed.
Medical school performance in the five selected pre-clinical and clinical courses was compared with the radi-ology residency rotation evaluation scores and the retro-spective faculty evaluation scores by using Student t tests(for independent samples). For each of the 10 assessedpreclinical and clinical courses, we compared rotationevaluation scores and retrospective evaluation scores ingeneral knowledge and overall performance and retro-spective evaluation scores in interpersonal skills for thoseresidents who received grades of Honors or A with thoseresidents who did not. Student t tests were also used todetermine whether those who received grades of Honorsor A had higher ACR and ABR scores than those whoreceived lower grades.
Student t tests were employed to determine whetherany difference existed in rotation evaluation scores, retro-spective evaluation scores, ACR examination perfor-mance, and ABR examination scores between residentswho had outstanding deans letters versus residents whodid not.
Student t tests were also used to determine whetherresidents who had received the highest-scored letters ofrecommendation had higher rotation evaluation scores,retrospective evaluation scores, or ACR/ABR scores thanthose who did not.
Additionally, Student t tests were used to determinewhether residents who had been selected to AOA as med-ical students had better radiology rotation or retrospectivefaculty evaluation scores or scored more highly on theACR/ABR examinations than those who had not beenselected.
Finally, Student t tests were used to compare the rota-tion and retrospective evaluation scores and the ACR/ABR examination scores of residents who did and resi-dents who did not have any substantial connection to ourregion.
Academic Radiology, Vol 9, No 4, April 2002 RADIOLOGY RESIDENCY PERFORMANCE
Correlation coefficients were obtained to compare thenumber of medical school Honors or A grades with therotation and retrospective evaluation scores and with theACR/ABR examination scores. They were also used tocompare NBME and USMLE scores with radiology resi-dency rotation and retrospective evaluation scores and tocompare NBME/USMLE and ACR/ABR examinationscores with one another. For this last analysis, NBMEscores, USMLE scores, and combined, normalizedUSMLE/NBME scores were all compared with the num-ber correct, percentage correct, and percentile scores onthe ACR examination and with the overall ABR examina-tion scores. (It should be remembered that, in general,correlation coefficients of 0.80.9 are considered to benotable in laboratory research, but correlation coefficientsof 0.20.4 are considered to be notable in social science[behavioral] research.)
We used an analysis of variance to compare medicalschool prestige with subsequent resident performance asassessed by means of rotation and retrospective evaluationscores and ACR/ABR examination scores.
Given the large number of comparisons in this study,we were concerned about the possibility of fortuitouslysignificant results. Therefore, for each test, a Bonferroni
adjustment was performed. This adjustment lowers the Pvalue that is necessary to reach statistical significancebased on the number of comparisons that are made foreach test. For the tests performed in our study, the Bon-ferroni adjustments lowered the threshold P values fordetermining significance from .05 to .012 or .016.
Preclinical Medical School Grades as Predictorsof Rad...