evaluating the fairness, impact, use and predictive ... the fairness, impact, use and predictive...
TRANSCRIPT
Evaluating the Fairness, Impact, Use and Predictive Validity of the New MCAT Exam
Catherine Lucey, MDVice Dean for Education
University of California, San Francisco School of Medicine
Joshua Hanson, MD, MPHAssistant Dean for Student Affairs, Assistant Professor
The University of Texas School of Medicine at San Antonio
Carol Terregino, MDSenior Associate Dean for Education, Associate Dean for Admissions; Professor of Medicine
Rutgers Robert Wood Johnson Medical School
Daniel Clinchot, MDAssociate Vice President for Health Sciences Education; Chair, Department of Education and Anatomy
The Ohio State University College of Medicine
CONFIDENTIAL – DO NOT DISTRIBUTE2 CONFIDENTIAL – DO NOT DISTRIBUTE2 CONFIDENTIAL – DO NOT DISTRIBUTE2
Diversity is a Core Driver of High Quality Health Care Minority physicians are significantly more likely to
practice primary care and in underserved areas Similar race/ethnicity identification between
physicians and patients results in longer visits and increased patient satisfaction
Communicating in patients’ native language often improves adherence to treatment
Medical students uniformly agree that a diverse student body improves the experience of medical school; enhances classroom discussions
Whitla DK et al. Acad Med. 2003; 78: 460Hung R et al. Acad Med.2007; 82(2):184AAMC Diversity in the Physician Workforce Facts and Figures 2014
Black/African-Americans comprise only 4 percent of the physician workforce
The majority of younger minority physicians in the workforce are women
CONFIDENTIAL – DO NOT DISTRIBUTE3 CONFIDENTIAL – DO NOT DISTRIBUTE3 CONFIDENTIAL – DO NOT DISTRIBUTE3
Slightly More Than Half of 2016 Matriculants Self-Identified as White
0.3%
21.3%
7.1% 6.3%0.1%
51.5%
8.8%1.7% 1.6% 1.3%
0%
10%
20%
30%
40%
50%
60%
70%
AmericanIndian orAlaskaNative
Black orAfrican
American
NativeHawaiian/
OtherPacific
Islander
More thanOne Race
UnknownRace/
Ethnicity
Percentage of U.S. Medical School Matriculants by Race/Ethnicity, 2016
Native Hawaiian/
Other PacificIslander
Asian Hispanic orLatino
White Non-U.S. Citizen/
Non-Perm.Resident
Note: The “Non-U.S. Citizen and Non-Permanent Resident” category may include students with unknown citizenship. Matriculants who declined to report sex are not reflected. Students who reported more than one race/ethnicity are included under “More than One Race”. During the 2016 application cycle, a technical malfunction in the collection of race/ethnicity data necessitated a request that applicants review and re-submit responses to the race/ethnicity question in their AMCAS applications. Applicants in previous years were not asked to review responses to this question.
Other
CONFIDENTIAL – DO NOT DISTRIBUTE4 CONFIDENTIAL – DO NOT DISTRIBUTE4 CONFIDENTIAL – DO NOT DISTRIBUTE4
The Path to Becoming a Physician Starts Early
Pre-College Experiences• Early environment• Family support• K-12 education• Extracurricular
experiences
Premedical Preparation• # colleges attended• Quality of instruction• Coursework• Extracurricular
experiences
Medical Education• Family and financial
support• Academic and other
support• Culture and climate
Residency Program• Academic and other
support• Culture and climate
Medical School Admissions
• Holistic review (Including MCAT)
• Diverse class
Residency Selection
• Holistic review (Including USMLE)
• Diverse cohort
CONFIDENTIAL – DO NOT DISTRIBUTE5 CONFIDENTIAL – DO NOT DISTRIBUTE5 CONFIDENTIAL – DO NOT DISTRIBUTE5
Group Differences in Academic Achievement Are Associated with Societal Inequalities MCAT, LSAT, GRE, GMAT, and other exams show population
group differences Presence of differences does not equate to test bias
(i.e., construct irrelevant content or alterations in administration)
Structural racism and privilege likely contribute to the differences seen across the spectrum of exams
• More likely to experience adverse environmental factors (poverty, food insecurity, low quality day care)
• More likely to have had disrupted or low quality K-12 education
• Less likely to have high quality exam prep experiences or advising experiences in college
Compared with majority
examinees, minority
examinees:
Davis D et al. Acad Med. 2013;88:593Dietrichson, J., Martin B., Filges, T., and Jorgensen, AMK (2017). Academic interventions for elementary and middle school students with low socioeconomic status: A systematic review and meta-analysis. Review of Educational Research. 87, 243-283.
CONFIDENTIAL – DO NOT DISTRIBUTE6 CONFIDENTIAL – DO NOT DISTRIBUTE6 CONFIDENTIAL – DO NOT DISTRIBUTE6
Type of Fairness Definition Influence on New
MCAT ExamSocietal Fairness
Aspiring physicians from different groups have equity in access to preparation materials and opportunities to prepare for the exam.
Blueprints, test preparation resources
Procedural Fairness
Admissions officers and their committees have ample information and resources to make appropriate and balanced use of MCAT scores in admissions.
New score scales, score reports, and resources for admissions officers
Exam Fairness
MCAT scores have the same meaning and predict student performance equally well for examinees from different backgrounds.
Item development/review, test form development
Fairness was Front and Center in Designing and Developing the New Exam
CONFIDENTIAL – DO NOT DISTRIBUTE7 CONFIDENTIAL – DO NOT DISTRIBUTE7 CONFIDENTIAL – DO NOT DISTRIBUTE7
Overview of the MCAT Validity Research Program
CONFIDENTIAL – DO NOT DISTRIBUTE8 CONFIDENTIAL – DO NOT DISTRIBUTE8 CONFIDENTIAL – DO NOT DISTRIBUTE8
21 Medical Schools and 2 Pre-Health Advisors Are Working Together to Evaluate the New Exam
CONFIDENTIAL – DO NOT DISTRIBUTE9 CONFIDENTIAL – DO NOT DISTRIBUTE9 CONFIDENTIAL – DO NOT DISTRIBUTE9
The MCAT Validity Research Program is Complex Multiple research questions in three broad areas Qualitative and quantitative data from
ExamineesApplicantsMedical studentsMedical schools
Multiple methods and data collection designs (longitudinal, cross sectional)
The validity study will last about nine years (2014 to 2023)
CONFIDENTIAL – DO NOT DISTRIBUTE10 CONFIDENTIAL – DO NOT DISTRIBUTE10 CONFIDENTIAL – DO NOT DISTRIBUTE10
The MCAT Validity Research Addresses Multiple Goals Provides evidence about the value of the new
MCAT exam in admissions decisions Answers questions about the fairness and
consequences of introducing the new MCAT exam for examinees, applicants, and medical students
Presents data to admissions officers that they can act on to improve their admissions decisions
Uses findings about the needs of aspiring physicians from underrepresented backgrounds to improve test preparation resources and outreach
CONFIDENTIAL – DO NOT DISTRIBUTE11 CONFIDENTIAL – DO NOT DISTRIBUTE11 CONFIDENTIAL – DO NOT DISTRIBUTE11
The MCAT Validity Research Agenda Includes Three Broad Areas
Predicting Medical Student
Performance
Admissions Decision Making
Academic Preparation,
Diversity, and Fairness
CONFIDENTIAL – DO NOT DISTRIBUTE12 CONFIDENTIAL – DO NOT DISTRIBUTE12 CONFIDENTIAL – DO NOT DISTRIBUTE12
We’re Testing 3 Hypotheses about Predicting Medical Student Performance
Do scores from the new exam correlate with academic performance throughout medical school?
Evaluating the predictive validity of
the new exam
Will scores from the Psychological, Social, and Biological Foundations of Behavior section correlate with performance in medical school courses that call on the behavioral and social sciences better than section scores from the old exam?
Examining the predictive validity of
the newest test section
Do scores from the new exam add value to the academic information applicants already provide about themselves through applications and transcripts?
Comparing the predictive validity of the new MCAT exam to other predictors
CONFIDENTIAL – DO NOT DISTRIBUTE13 CONFIDENTIAL – DO NOT DISTRIBUTE13 CONFIDENTIAL – DO NOT DISTRIBUTE13
We’re Testing 2 Hypotheses about Admissions Decision Making
Will medical schools increase the percentage of applicants with total scores in the middle of the MCAT score scale who are invited to interview and receive acceptance offers?
Acceptance of a wide range scores
Will admissions committees use information about applicants’ strengths and weaknesses from the MCAT score reports to identify applicants who best fit their academic missions and goals?
Use of section scores
CONFIDENTIAL – DO NOT DISTRIBUTE14 CONFIDENTIAL – DO NOT DISTRIBUTE14 CONFIDENTIAL – DO NOT DISTRIBUTE14
We’re Testing 9 Hypotheses about Academic Preparation, Diversity, and Fairness, For Example:
Will more individuals learn about psychology, sociology, and biochemistry in preparation for the MCAT exam?What resources, information, and outreach will provide equity in access for students from sociodemographic groups underrepresented in medicine?
Change in breadth of academic preparation
Will the diversity of examinees, applicants, and medical students who took the new exam change?
Diversity of aspiring physicians
Will scores from the new exam predict academic performance equally well for medical students from different racial, ethnic, or disadvantaged backgrounds?
Fairness in score meaning
CONFIDENTIAL – DO NOT DISTRIBUTE15 CONFIDENTIAL – DO NOT DISTRIBUTE15 CONFIDENTIAL – DO NOT DISTRIBUTE15
How well did scores from the new MCAT
exam predict students’
performance across the first year of medical school?
How did admissions officers work with new MCAT scores in 2017 admissions
decisions?
What can we learn about preparation
resources needed by examinees from
underrepresented backgrounds?
Today’s Presentation Will Focus on Three Questions
1616
How well did scores from the new MCAT exam predict students’ performance
across the first year of medical school?
CONFIDENTIAL – DO NOT DISTRIBUTE17 CONFIDENTIAL – DO NOT DISTRIBUTE17 CONFIDENTIAL – DO NOT DISTRIBUTE17
Validity School
Participants
The Predictive Validity Research Includes National and Validity School Data (2015-2017 Entrants)
National Population with New Scores
National Population
CONFIDENTIAL – DO NOT DISTRIBUTE18 CONFIDENTIAL – DO NOT DISTRIBUTE18 CONFIDENTIAL – DO NOT DISTRIBUTE18
MCAT Scores Will Be Correlated with Multiple Types of Student Performance Outcomes
Holistic Medical Student Academic Performance
Academic Readiness to Graduate
Academic Readiness for Clerkships
Year-End Academic PerformanceCourse-Based
Academic Performance
Holistic Outcomes
Granular Outcomes
CONFIDENTIAL – DO NOT DISTRIBUTE19 CONFIDENTIAL – DO NOT DISTRIBUTE19 CONFIDENTIAL – DO NOT DISTRIBUTE19
Holistic Medical Student Academic Performance
Academic Readiness to Graduate
Academic Readiness for Clerkships
Year-End Academic PerformanceCourse-Based
Academic Performance
Some Analyses Will Include Outcomes from the National Population
Holistic Outcomes
Granular Outcomes
National OutcomesProgression to M2, M3USMLE Step exams
Graduation
CONFIDENTIAL – DO NOT DISTRIBUTE20 CONFIDENTIAL – DO NOT DISTRIBUTE20 CONFIDENTIAL – DO NOT DISTRIBUTE20
Holistic Medical Student Academic Performance
Academic Readiness to Graduate
Academic Readiness for Clerkships
Year-End Academic PerformanceCourse-Based
Academic Performance
Other Analyses Will Include Outcomes from the Validity Schools
Holistic Outcomes
Granular Outcomes
Outcomes from Validity SchoolsCourse grades
NBME Subject examsClerkship gradesGPA/Class Rank
CONFIDENTIAL – DO NOT DISTRIBUTE21 CONFIDENTIAL – DO NOT DISTRIBUTE21 CONFIDENTIAL – DO NOT DISTRIBUTE21
Today’s findings are from the 2016 entrants with new scores
w/ new scoresN=8,521
Participants
N=1,027 w/ new scores
2016 MatriculantsN = 21,030
Matriculants
CONFIDENTIAL – DO NOT DISTRIBUTE22 CONFIDENTIAL – DO NOT DISTRIBUTE22 CONFIDENTIAL – DO NOT DISTRIBUTE22
2016 Matriculants and Validity School Participants with New Scores Are Similar to the 2016 Matriculant Population
Demographics Fee assistance status Parental education/occupation Race/ethnicity
Academic metrics MCAT total and section scores Undergraduate GPAs
w/ new scoresN=8,521
Participants
N=1,027 w/ new scores
2016 MatriculantsN = 21,030
Matriculants
CONFIDENTIAL – DO NOT DISTRIBUTE23 CONFIDENTIAL – DO NOT DISTRIBUTE23 CONFIDENTIAL – DO NOT DISTRIBUTE23
Holistic Medical Student Academic Performance
Academic Readiness to Graduate
Academic Readiness for Clerkships
Year-End Academic PerformanceCourse-Based
Academic Performance
Today’s Results Include Two M1 Year-End Outcomes
Holistic Outcomes
Granular Outcomes
National OutcomeProgression to M2
without delay
Outcome from Validity Schools
Performance Across M1 Courses
CONFIDENTIAL – DO NOT DISTRIBUTE24 CONFIDENTIAL – DO NOT DISTRIBUTE24 CONFIDENTIAL – DO NOT DISTRIBUTE24
Nationally, 2016 Entrants w/ a Wide Range of Scores Progressed to M2 Without Delay
% S
tude
nts
Prog
ress
ing
to M
2
MCAT Total Score Range
50%
80% 81%
93% 94% 96% 97% 98% 98% 99%
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
472-485 486-489 490-493 494-497 498-501 502-505 506-509 510-513 514-517 518-528
% of 2016 Entrants Who Progressed to M2 Without Delay
(N=10) (N=15) (N=106) (N=309) (N=811) (N=1,360) (N=1,727) (N=1,654) (N=1,200) (N=778)
Note. Only students enrolled in regular, 4-year MD programs who took the new exam were included in this analysis.
CONFIDENTIAL – DO NOT DISTRIBUTE25 CONFIDENTIAL – DO NOT DISTRIBUTE25 CONFIDENTIAL – DO NOT DISTRIBUTE25
At the Validity Schools, New Scores Predict Performance across M1 Courses
r Effect Size0.1 Small0.3 Medium0.5 Large
0.43
0.69
0.55
00.
20.
40.
60.
81.
0C
orre
cted
Cor
rela
tion
Overall(NSchool = 15)
Medium Effect
Correlation of MCAT Total Scores with Performance Across M1 Courses: Median and Interquartile Range
Overall, the associations of the new MCAT total scores with performance across M1 courses are medium to large
Cohen, J. (1992). A power primer. Psychological Bulletin, 112(1), 155-159. doi:10.1037/0033-2909.112.1.155
CONFIDENTIAL – DO NOT DISTRIBUTE26 CONFIDENTIAL – DO NOT DISTRIBUTE26 CONFIDENTIAL – DO NOT DISTRIBUTE26
These Validities Compare Well to Those for Other Exams
Author(Year) Exam
Type of ExamScore
Type of Outcome
Median Validity
CoefficientUnit of
AnalysisAAMC (2017) New Exam Total Score First-Year Course Average .55 School
(Nschool=15)AAMC (2017) Old MCAT Total score First-Year Course Average .53 School
(Nschool=17)Kuncel, et al. (2007) GMAT Total score First-Year Graduate GPA .47 Meta-
analysis of multiple studies
Stilwell, et al. (2011) LSAT Total score First-Year Average Course Grade .58 School(Nschool=170)
CONFIDENTIAL – DO NOT DISTRIBUTE27 CONFIDENTIAL – DO NOT DISTRIBUTE27 CONFIDENTIAL – DO NOT DISTRIBUTE27
Holistic Medical Student Academic Performance
Academic Readiness to Graduate
Academic Readiness for Clerkships
Year-End Academic PerformanceCourse-Based
Academic Performance
At the Validity Schools, We Also Analyzed Outcomes from Individual M1 Courses
Holistic Outcomes
Granular Outcomes
CONFIDENTIAL – DO NOT DISTRIBUTE28 CONFIDENTIAL – DO NOT DISTRIBUTE28 CONFIDENTIAL – DO NOT DISTRIBUTE28
At the Validity Schools, New Scores Predict Performance in Individual M1 Courses
Overall, the associations of the new MCAT total scores with performance in M1 courses are medium to large
r Effect Size0.1 Small0.3 Medium0.5 Large
0.40
0.61
0.51
00.
20.
40.
60.
81.
0C
orre
cted
Cor
rela
tion
(NCourse = 109)
Medium Effect
Correlation ofMCAT Total Scores with
Performance in Individual M1 Courses:
Median and Interquartile Range
CONFIDENTIAL – DO NOT DISTRIBUTE29 CONFIDENTIAL – DO NOT DISTRIBUTE29 CONFIDENTIAL – DO NOT DISTRIBUTE29
0.40 0.40 0.40
0.61 0.62 0.64
0.51 0.51 0.50
00.
20.
40.
60.
81.
0C
orre
cted
Cor
rela
tion
Overall(NCourse = 109)
GeneralPrinciples ofFoundational
Science(NCourse = 66)
BodySystems
(NCourse = 46)
Course Content Area
At the Validity Schools, New Scores Predict Performance in Courses about Foundational Sciences and Body Systems
r Effect Size0.1 Small0.3 Medium0.5 Large
Medium Effect
Correlation of MCAT Total Scores with Performance in Individual M1 Courses by Course Content Area:
Median and Interquartile Range
CONFIDENTIAL – DO NOT DISTRIBUTE30 CONFIDENTIAL – DO NOT DISTRIBUTE30 CONFIDENTIAL – DO NOT DISTRIBUTE30
New Scores Predict Equally Well for Students from Different Sociodemographic Backgrounds Research studied these early relationships for students
grouped by: Race/ethnicity Socioeconomic status
So far, MCAT scores neither over- nor under-predict the performance of students from underrepresented backgrounds based on two types of performance outcomes: National outcome: Progression to M2 without delay Validity school outcome: Performance across M1
courses
CONFIDENTIAL – DO NOT DISTRIBUTE31 CONFIDENTIAL – DO NOT DISTRIBUTE31 CONFIDENTIAL – DO NOT DISTRIBUTE31
Predictive Validity Findings Will Be Reported from 2017 through 2022 for Medical Students Who Took the New Exam
2016 Entrants
2017 Entrants 2018 2019 2020 2021 2022
We have a lot more to learn about how students do in their second year, in their clerkships and on their USMLE exams, and their graduation from undergraduate medical school
CONFIDENTIAL – DO NOT DISTRIBUTE33 CONFIDENTIAL – DO NOT DISTRIBUTE33 CONFIDENTIAL – DO NOT DISTRIBUTE33
Admissions Committees Put MCAT Scores in Context There are more qualified applicants to medical school than there
are places in medical school Medical schools weigh information about attributes,
experiences, and academic metrics in deciding whom to admit Applicants from sociodemographic groups underrepresented in
medicine obtain lower mean scores on the MCAT exam and lower undergraduate GPAs than applicants from other groups
Medical schools aim to select students who will succeed and who will help build a diverse physician workforce
CONFIDENTIAL – DO NOT DISTRIBUTE34 CONFIDENTIAL – DO NOT DISTRIBUTE34 CONFIDENTIAL – DO NOT DISTRIBUTE34
The New Score Scales Draw Attention to the Center of the Scale Because on the old exam, students who entered
medical school with scores in the center of the MCAT score scale succeeded
The new scale uses a nice round number at 500 to draw attention to applicants who might otherwise be overlooked
The new test also includes more questions per section, providing better information about examinees’ strengths and weaknesses on the exam
The new score reports use confidence bands to describe measurement precision and score profiles to describe strengths and weaknesses
CONFIDENTIAL – DO NOT DISTRIBUTE35 CONFIDENTIAL – DO NOT DISTRIBUTE35 CONFIDENTIAL – DO NOT DISTRIBUTE35
Admissions Committees Put MCAT Scores in Context in 2017 Selection
Percentage and Number of 2017 Applicants Accepted into at Least One Medical School, by New MCAT Total Score and Undergraduate GPA Range
New MCAT TotalGPA Total 472-485 486-489 490-493 494-497 498-501 502-505 506-509 510-513 514-517 518-528 All3.80-4.00 4% 5% 10% 21% 34% 54% 67% 76% 85% 89% 67%
2/51 5/98 23/241 111/528 353/1,040 920/1,692 1,538/2,310 1,870/2,461 1,798/2,112 1,796/2,014 8,416/12,5473.60-3.79 0% 1% 5% 15% 26% 38% 54% 68% 76% 85% 48%
0/126 3/223 23/442 129/867 366/1,430 764/1,996 1,197/2,225 1,388/2,040 987/1,301 639/755 5,496/11,4053.40-3.59 2% 1% 4% 10% 21% 29% 40% 55% 61% 71% 33%
4/183 3/293 22/552 100/955 265/1,286 479/1,635 661/1,659 694/1,260 405/660 235/329 2,868/8,8123.20-3.39 0% <1% 3% 9% 18% 24% 32% 40% 56% 58% 22%
0/226 1/306 14/527 64/749 158/884 246/,1027 279/873 255/631 176/314 63/109 1,256/5,6463.00-3.19 0% 2% 1% 7% 14% 21% 25% 35% 42% 37% 15%
0/248 4/253 4/363 36/490 74/511 109/507 100/395 98/282 43/103 21/57 489/3,2092.80-2.99 1% 1% 2% 5% 6% 17% 25% 30% 29% 42% 10%
2/247 2/181 5/201 12/238 14/233 34/203 40/160 25/84 14/49 5/12 153/1,6082.60-2.79 0% 1% 0% 2% 6% 18% 18% 17% 40% - 6%
0/157 1/117 0/142 3/132 8/126 14/79 10/57 6/36 6/15 53/8682.40-2.59 0% 2% 2% 5% 3% 13% 25% 24% - - 5%
0/118 1/57 1/48 3/59 1/40 4/30 6/24 4/17 21/3982.20-2.39 0% 0% 0% 0% 12% 33% 8% - - 5%
0/69 0/29 0/24 0/21 2/17 4/12 1/12 9/1922.00-2.19 0% 0% 0% - - - - - 0%
0/41 0/13 0/10 0/86<2.00 0% - - - - 3%
0/15 1/31All 1% 1% 4% 11% 22% 36% 50% 64% 75% 84% 42%
8/1,481 20/1,576 93/2,556 458/4,051 1,241/5,571 2,574/7,188 3,832/7,717 4,341/6,819 3,430/4,558 2,765/3,285 18,762/44,802
CONFIDENTIAL – DO NOT DISTRIBUTE36 CONFIDENTIAL – DO NOT DISTRIBUTE36 CONFIDENTIAL – DO NOT DISTRIBUTE36
Some 2017 Applicants w/ High UGPAs and MCATs Weren’t Accepted
Percentage and Number of 2017 Applicants Accepted into at Least One Medical School, by New MCAT Total Score and Undergraduate GPA Range
New MCAT TotalGPA Total 472-485 486-489 490-493 494-497 498-501 502-505 506-509 510-513 514-517 518-528 All3.80-4.00 4% 5% 10% 21% 34% 54% 67% 76% 85% 89% 67%
2/51 5/98 23/241 111/528 353/1,040 920/1,692 1,538/2,310 1,870/2,461 1,798/2,112 1,796/2,014 8,416/12,5473.60-3.79 0% 1% 5% 15% 26% 38% 54% 68% 76% 85% 48%
0/126 3/223 23/442 129/867 366/1,430 764/1,996 1,197/2,225 1,388/2,040 987/1,301 639/755 5,496/11,4053.40-3.59 2% 1% 4% 10% 21% 29% 40% 55% 61% 71% 33%
4/183 3/293 22/552 100/955 265/1,286 479/1,635 661/1,659 694/1,260 405/660 235/329 2,868/8,8123.20-3.39 0% <1% 3% 9% 18% 24% 32% 40% 56% 58% 22%
0/226 1/306 14/527 64/749 158/884 246/,1027 279/873 255/631 176/314 63/109 1,256/5,6463.00-3.19 0% 2% 1% 7% 14% 21% 25% 35% 42% 37% 15%
0/248 4/253 4/363 36/490 74/511 109/507 100/395 98/282 43/103 21/57 489/3,2092.80-2.99 1% 1% 2% 5% 6% 17% 25% 30% 29% 42% 10%
2/247 2/181 5/201 12/238 14/233 34/203 40/160 25/84 14/49 5/12 153/1,6082.60-2.79 0% 1% 0% 2% 6% 18% 18% 17% 40% - 6%
0/157 1/117 0/142 3/132 8/126 14/79 10/57 6/36 6/15 53/8682.40-2.59 0% 2% 2% 5% 3% 13% 25% 24% - - 5%
0/118 1/57 1/48 3/59 1/40 4/30 6/24 4/17 21/3982.20-2.39 0% 0% 0% 0% 12% 33% 8% - - 5%
0/69 0/29 0/24 0/21 2/17 4/12 1/12 9/1922.00-2.19 0% 0% 0% - - - - - 0%
0/41 0/13 0/10 0/86<2.00 0% - - - - 3%
0/15 1/31All 1% 1% 4% 11% 22% 36% 50% 64% 75% 84% 42%
8/1,481 20/1,576 93/2,556 458/4,051 1,241/5,571 2,574/7,188 3,832/7,717 4,341/6,819 3,430/4,558 2,765/3,285 18,762/44,802
11% of applicants with GPAs at or above 3.8 and
MCAT scores at or above 518 were
not admitted into any medical
schools
CONFIDENTIAL – DO NOT DISTRIBUTE37 CONFIDENTIAL – DO NOT DISTRIBUTE37 CONFIDENTIAL – DO NOT DISTRIBUTE37
Other 2017 Applicants w/ Modest Credentials Were Accepted
Percentage and Number of 2017 Applicants Accepted into at Least One Medical School, by New MCAT Total Score and Undergraduate GPA Range
New MCAT TotalGPA Total 472-485 486-489 490-493 494-497 498-501 502-505 506-509 510-513 514-517 518-528 All3.80-4.00 4% 5% 10% 21% 34% 54% 67% 76% 85% 89% 67%
2/51 5/98 23/241 111/528 353/1,040 920/1,692 1,538/2,310 1,870/2,461 1,798/2,112 1,796/2,014 8,416/12,5473.60-3.79 0% 1% 5% 15% 26% 38% 54% 68% 76% 85% 48%
0/126 3/223 23/442 129/867 366/1,430 764/1,996 1,197/2,225 1,388/2,040 987/1,301 639/755 5,496/11,4053.40-3.59 2% 1% 4% 10% 21% 29% 40% 55% 61% 71% 33%
4/183 3/293 22/552 100/955 265/1,286 479/1,635 661/1,659 694/1,260 405/660 235/329 2,868/8,8123.20-3.39 0% <1% 3% 9% 18% 24% 32% 40% 56% 58% 22%
0/226 1/306 14/527 64/749 158/884 246/,1027 279/873 255/631 176/314 63/109 1,256/5,6463.00-3.19 0% 2% 1% 7% 14% 21% 25% 35% 42% 37% 15%
0/248 4/253 4/363 36/490 74/511 109/507 100/395 98/282 43/103 21/57 489/3,2092.80-2.99 1% 1% 2% 5% 6% 17% 25% 30% 29% 42% 10%
2/247 2/181 5/201 12/238 14/233 34/203 40/160 25/84 14/49 5/12 153/1,6082.60-2.79 0% 1% 0% 2% 6% 18% 18% 17% 40% - 6%
0/157 1/117 0/142 3/132 8/126 14/79 10/57 6/36 6/15 53/8682.40-2.59 0% 2% 2% 5% 3% 13% 25% 24% - - 5%
0/118 1/57 1/48 3/59 1/40 4/30 6/24 4/17 21/3982.20-2.39 0% 0% 0% 0% 12% 33% 8% - - 5%
0/69 0/29 0/24 0/21 2/17 4/12 1/12 9/1922.00-2.19 0% 0% 0% - - - - - 0%
0/41 0/13 0/10 0/86<2.00 0% - - - - 3%
0/15 1/31All 1% 1% 4% 11% 22% 36% 50% 64% 75% 84% 42%
8/1,481 20/1,576 93/2,556 458/4,051 1,241/5,571 2,574/7,188 3,832/7,717 4,341/6,819 3,430/4,558 2,765/3,285 18,762/44,802
14% of applicants with GPAs 3.00 - 3.19
and MCAT scores 498-501 were
admitted into at least one medical
school
CONFIDENTIAL – DO NOT DISTRIBUTE38 CONFIDENTIAL – DO NOT DISTRIBUTE38 CONFIDENTIAL – DO NOT DISTRIBUTE38
Progression of 2016 Entrants to M2 Without Delay, By New MCAT Total Score Range
% S
tude
nts
Prog
ress
ing
to M
2
MCAT Total Score Range
50%
80% 81%
93% 94% 96% 97% 98% 98% 99%
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
472-485 486-489 490-493 494-497 498-501 502-505 506-509 510-513 514-517 518-528
% of 2016 Entrants Who Progressed to M2 Without Delay
(N=10) (N=15) (N=106) (N=309) (N=811) (N=1,360) (N=1,727) (N=1,654) (N=1,200) (N=778)
Note. Only students enrolled in regular, 4-year MD programs who took the new exam were included in this analysis.
Percentage and Number of 2017 Applicants Accepted into at Least One Medical School, by New MCAT Total Score and Undergraduate GPA Range
New MCAT TotalGPA Total 472-485 486-489 490-493 494-497 498-501 502-505 506-509 510-513 514-517 518-528 All3.80-4.00 4% 5% 10% 21% 34% 54% 67% 76% 85% 89% 67%
2/51 5/98 23/241 111/528 353/1,040 920/1,692 1,538/2,310 1,870/2,461 1,798/2,112 1,796/2,014 8,416/12,5473.60-3.79 0% 1% 5% 15% 26% 38% 54% 68% 76% 85% 48%
0/126 3/223 23/442 129/867 366/1,430 764/1,996 1,197/2,225 1,388/2,040 987/1,301 639/755 5,496/11,4053.40-3.59 2% 1% 4% 10% 21% 29% 40% 55% 61% 71% 33%
4/183 3/293 22/552 100/955 265/1,286 479/1,635 661/1,659 694/1,260 405/660 235/329 2,868/8,8123.20-3.39 0% <1% 3% 9% 18% 24% 32% 40% 56% 58% 22%
0/226 1/306 14/527 64/749 158/884 246/,1027 279/873 255/631 176/314 63/109 1,256/5,6463.00-3.19 0% 2% 1% 7% 14% 21% 25% 35% 42% 37% 15%
0/248 4/253 4/363 36/490 74/511 109/507 100/395 98/282 43/103 21/57 489/3,2092.80-2.99 1% 1% 2% 5% 6% 17% 25% 30% 29% 42% 10%
2/247 2/181 5/201 12/238 14/233 34/203 40/160 25/84 14/49 5/12 153/1,6082.60-2.79 0% 1% 0% 2% 6% 18% 18% 17% 40% - 6%
0/157 1/117 0/142 3/132 8/126 14/79 10/57 6/36 6/15 53/8682.40-2.59 0% 2% 2% 5% 3% 13% 25% 24% - - 5%
0/118 1/57 1/48 3/59 1/40 4/30 6/24 4/17 21/3982.20-2.39 0% 0% 0% 0% 12% 33% 8% - - 5%
0/69 0/29 0/24 0/21 2/17 4/12 1/12 9/1922.00-2.19 0% 0% 0% - - - - - 0%
0/41 0/13 0/10 0/86<2.00 0% - - - - 3%
0/15 1/31All 1% 1% 4% 11% 22% 36% 50% 64% 75% 84% 42%
8/1,481 20/1,576 93/2,556 458/4,051 1,241/5,571 2,574/7,188 3,832/7,717 4,341/6,819 3,430/4,558 2,765/3,285 18,762/44,802
CONFIDENTIAL – DO NOT DISTRIBUTE40 CONFIDENTIAL – DO NOT DISTRIBUTE40 CONFIDENTIAL – DO NOT DISTRIBUTE40
Let’s put these two data sources together
CONFIDENTIAL – DO NOT DISTRIBUTE41 CONFIDENTIAL – DO NOT DISTRIBUTE41 CONFIDENTIAL – DO NOT DISTRIBUTE41
Early Data Suggest Students with Scores in the Middle of the New Scale Do Well
50%
80% 81%
93% 94% 96% 97% 98% 98% 99%
1% 1% 4%11%
22%
36%
50%
64%
75%84%
0%10%20%30%40%50%60%70%80%90%
100%
472-485 486-489 490-493 494-497 498-501 502-505 506-509 510-513 514-517 518-528
% of 2016 Entering Medical Students Who Progressed to M2 Without Delay, and % of 2017 Applicants Accepted to Medical School
Progression to M2 (2016 Entrants) Acceptance Rates (2017 Applicants)
CONFIDENTIAL – DO NOT DISTRIBUTE42 CONFIDENTIAL – DO NOT DISTRIBUTE42 CONFIDENTIAL – DO NOT DISTRIBUTE42
As a community, should we ask ourselves if there is more we want to do?
CONFIDENTIAL – DO NOT DISTRIBUTE43 CONFIDENTIAL – DO NOT DISTRIBUTE43 CONFIDENTIAL – DO NOT DISTRIBUTE43
As a Community, Should We Ask Ourselves if There Is More We Want to do? Admissions committees identify the applicants with the
ingredients for success, and they support these students once they’re admitted
Are we missing some opportunities to pay more attention to students with scores in the middle of the MCAT score scale?
There is still a lot to learn about students’ performance in the second year and in their clerkships
On the new exam, findings so far suggest that, just like they did with the old exam, students admitted with scores in the middle of the new MCAT score scale do well
CONFIDENTIAL – DO NOT DISTRIBUTE44 CONFIDENTIAL – DO NOT DISTRIBUTE44 CONFIDENTIAL – DO NOT DISTRIBUTE44
Interested in Learning More about Your Colleagues’ Use of New Exam Scores?
Strategies for Mission-Oriented Admissions in Light of the New MCAT Exam Monday, November 6 at 3:00 PM ET
Center 304-306
Tanisha Price-Johnson, PhD, University of Arizona-Tuscon College of MedicineTheodore Hall, MD, David Geffen School of Medicine at UCLAAaron Saguil, MD, F. Edward Herbert School of Medicine, Uniformed Services University Sunny Nakae, PhD, Loyola University Chicago Stritch School of MedicineCarol Teitz, MD, University of Washington School of Medicine
4545
What can we learn about preparation resources needed by examinees from
sociodemographic groups underrepresented in medicine?
CONFIDENTIAL – DO NOT DISTRIBUTE46 CONFIDENTIAL – DO NOT DISTRIBUTE46 CONFIDENTIAL – DO NOT DISTRIBUTE46
Type of Fairness Definition Influence on New
MCAT ExamSocietal Fairness
Aspiring physicians from different groups have equity in access to preparation materials and opportunities to prepare for the exam.
Blueprints, test preparation resources
Procedural Fairness
Admissions officers and their committees have ample information and resources to make appropriate and balanced use of MCAT scores in admissions.
New score scales, score reports, and resources for admissions officers
Exam Fairness
MCAT scores have the same meaning and predict student performance equally well for examinees from different backgrounds.
Item development/review, test form development
The New MCAT Exam Was Designed to Increase Access
CONFIDENTIAL – DO NOT DISTRIBUTE47 CONFIDENTIAL – DO NOT DISTRIBUTE47 CONFIDENTIAL – DO NOT DISTRIBUTE47
The New Test Blueprints Were Developed to Advance Societal Fairness Test concepts widely taught at undergraduate institutions,
including minority-serving and under-resourced institutions Test psychology and sociology concepts like discrimination,
stereotype threat, and socio-economic inequalities Pay increased attention to population health, studies of
diverse cultures, and ethics Balance the percentage of questions devoted to natural
sciences concepts with the percentage devoted to behavioral and social sciences concepts and information processing
CONFIDENTIAL – DO NOT DISTRIBUTE48 CONFIDENTIAL – DO NOT DISTRIBUTE48 CONFIDENTIAL – DO NOT DISTRIBUTE48
Preparation Resources Were Developed to Advance Societal Fairness The Khan Academy has over 1,100 free tutorials on exam content Practice materials and resources are available on AAMC’s website: What’s on the MCAT Exam? Interactive Content Outline Roadmaps to MCAT Content in Biochemistry, Psychology, and
Sociology Textbooks Guide to Creating a Study Plan How I Prepared for the MCAT Exam Testimonials Practice tests and question banks
Students-residents.aamc.org/mcatprep
CONFIDENTIAL – DO NOT DISTRIBUTE49 CONFIDENTIAL – DO NOT DISTRIBUTE49 CONFIDENTIAL – DO NOT DISTRIBUTE49
New Outreach Strategies Address Societal Fairness Expand outreach directly to students from sociodemographic
groups underrepresented in medicine Expand outreach to students through their advisors, with a
particular focus on faculty at under-resourced institutions Distribute a monthly newsletter Premed Navigator with important
information, resources, and tips Work with prehealth advisors on the MCAT Validity Committee
to share findings and promote resources, such as the “Find an Advisor” resource for students at schools with no access to an advisor ([email protected])
CONFIDENTIAL – DO NOT DISTRIBUTE50 CONFIDENTIAL – DO NOT DISTRIBUTE50 CONFIDENTIAL – DO NOT DISTRIBUTE50
Societal Fairness: Expanding Outreach to Faculty at Under-resourced and Minority-Serving Institutions Developed new contacts at minority-serving and/or under-
resourced colleges and universities—adding 350 contacts from: Almost 90 HBCUs More than 200 Hispanic-serving institutions More than 30 Tribal colleges and universities
Add to the list multiple times/year
Please encourage your advisor colleagues to add their names to our mailing list. They can contact us at [email protected]
CONFIDENTIAL – DO NOT DISTRIBUTE51 CONFIDENTIAL – DO NOT DISTRIBUTE51 CONFIDENTIAL – DO NOT DISTRIBUTE51
Societal Fairness: Expanding Outreach Directly to Students from Sociodemographic Groups Underrepresented in Medicine Reach out to students directly through emails and newsletters Participate in health professions career fairs focused on minority
populations (e.g., UC Davis Pre-Health Conference, Stanford University Minority Medical Alliance Conference)
Participate in virtual events like Health Professions Week Virtual Fair and AAMC Medical School Virtual Fair
Monitor students’ use of MCAT preparation products to determine if better outreach about available resources is needed
CONFIDENTIAL – DO NOT DISTRIBUTE52 CONFIDENTIAL – DO NOT DISTRIBUTE52 CONFIDENTIAL – DO NOT DISTRIBUTE52
Despite This Work, Group Differences Still Exist• The group differences in scores on the new exam are similar to
those on the old exam• They are similar to those on other standardized tests:
o GREo LSATo GMAT
CONFIDENTIAL – DO NOT DISTRIBUTE53 CONFIDENTIAL – DO NOT DISTRIBUTE53 CONFIDENTIAL – DO NOT DISTRIBUTE53
Now We’re Studying How Students Prepare for the MCAT Exam
Pre-College Experiences• Early environment• Family support• K-12 education• Extracurricular
experiences
Premedical Preparation• # colleges attended• Quality of instruction• Coursework• Extracurricular
experiences
Medical Education• Family and financial
support• Academic and other
support• Culture and climate
Residency Program• Academic and other
support• Culture and climate
Medical School Admissions
• Holistic review (Including MCAT)
• Diverse class
Residency Selection
• Holistic review (Including USMLE)
• Diverse cohort
CONFIDENTIAL – DO NOT DISTRIBUTE54 CONFIDENTIAL – DO NOT DISTRIBUTE54 CONFIDENTIAL – DO NOT DISTRIBUTE54
Review Courses Are Used by Proportionately Fewer Examinees from Underrepresented Backgrounds
43% 40%
6% 9%
52% 51%
25% 27%
020
4060
8010
0
CommercialPrep. Courses
University-BasedPrep. Courses
Khan Academy None of the Three
Use of Preparation Courses and Khan to Prepare for the ExamBy URiM Status (Race/Ethnicity)
Non-URiMURiM
Note. Data came from examinees who took the MCAT exam and in 2017.
44%
33%
7% 7%
52% 52%
24%30%
020
4060
8010
0
CommercialPrep. Courses
University-BasedPrep. Courses
Khan Academy None of the Three
Use of Preparation Courses and Khan to Prepare for the ExamBy First-Generation College Status
Non-First Generation College StudentsFirst-Generation College Students
CONFIDENTIAL – DO NOT DISTRIBUTE55 CONFIDENTIAL – DO NOT DISTRIBUTE55 CONFIDENTIAL – DO NOT DISTRIBUTE55
Practice Resources Are Used by Proportionately Fewer Examinees from Underrepresented Backgrounds
76%71%
53% 51%
21% 24%
020
4060
8010
0
Practice Exams Questions Packs &Section Banks
None of the Two
Use of AAMC Resources to Practice for the ExamBy URiM Status (Race/Ethnicity)
Non-URiMURiM
Note. Data came from examinees who took the MCAT exam and responded to the Post-MCAT Questionnaire in 2017.
75% 73%
53% 53%
21% 23%
020
4060
8010
0
Practice Exams Questions Packs &Section Banks
None of the Two
Use of AAMC Resources to Practice for the ExamBy First-Generation College Status
Non-First Generation College StudentsFirst-Generation College Students
CONFIDENTIAL – DO NOT DISTRIBUTE56 CONFIDENTIAL – DO NOT DISTRIBUTE56 CONFIDENTIAL – DO NOT DISTRIBUTE56
Gaps in the Use of Review Courses and Practice Resources Are Larger for Those Underrepresented on Both Race/Ethnicity and Parental Education
Note. Data came from examinees who took the MCAT exam and responded to the Post-MCAT Questionnaire in 2017.
44%
33%
6% 9%
52% 50%
24%31%
020
4060
8010
0
CommercialPrep. Courses
University-BasedPrep. Courses
Khan Academy None of the Three
Use of Preparation Courses and Khan to Prepare for the ExamBy URiM Status (Race/Ethnicity) and First-Generation College Status
Neither URiM Nor First-Generation College StudentsURiM and First-Generation College Students
76%71%
53% 52%
21% 24%
020
4060
8010
0
Practice Exams Questions Packs &Section Banks
None of the Two
Use of AAMC Resources to Practice for the ExamBy URiM Status (Race/Ethnicity) and First-Generation College Status
Neither URiM Nor First-Generation College StudentsURiM and First-Generation College Students
CONFIDENTIAL – DO NOT DISTRIBUTE57 CONFIDENTIAL – DO NOT DISTRIBUTE57 CONFIDENTIAL – DO NOT DISTRIBUTE57
We are Conducting Qualitative and Quantitative Research to Understand Students’ Preparation Strategies and Barriers What is easy and difficult for examinees when they prepare for
the MCAT exam? What is easy and difficult about using the AAMC’s free and low-
cost materials to prepare for the MCAT exam? Are these barriers different or greater for those from
sociodemographic groups underrepresented in medicine? What additional resources and information do examinees and
their advisors need?
CONFIDENTIAL – DO NOT DISTRIBUTE58 CONFIDENTIAL – DO NOT DISTRIBUTE58 CONFIDENTIAL – DO NOT DISTRIBUTE58
Interview Students
andAdvisors
Revise PMQ
Administer PMQ and Compare
Results for Examinees
from Different Groups
Learn what barriers
exist
Develop new
information and
resources
Findings Will Be Used to Improve Access to Resources and Information
CONFIDENTIAL – DO NOT DISTRIBUTE59 CONFIDENTIAL – DO NOT DISTRIBUTE59 CONFIDENTIAL – DO NOT DISTRIBUTE59
Questions?
Stay tuned for new research findings
aamc.org/validitycommittee
CONFIDENTIAL – DO NOT DISTRIBUTE60 CONFIDENTIAL – DO NOT DISTRIBUTE60 CONFIDENTIAL – DO NOT DISTRIBUTE60
Don’t forget to join us tomorrow!
The New MCAT Exam:Where We Are and Where We’re Going
Time: 7:00 AM - 8:30 AM Location: Center Ballroom C
CONFIDENTIAL – DO NOT DISTRIBUTE61 CONFIDENTIAL – DO NOT DISTRIBUTE61 CONFIDENTIAL – DO NOT DISTRIBUTE61
References AAMC. (March, 2017). Predicting medical student performance. Presented at
the MCAT Validity Committee meeting at the Association of American Medical Colleges in Washington, D.C.
Kuncel, N., Credé, M., & Thomas, L. (2007). A meta-analysis of the predictive validity of the graduate management admission test (GMAT) and undergraduate grade point average (UGPA) for graduate student academic performance. Academy of Management Learning & Education, 6(1), 51-68. Retrieved from http://www.jstor.org/stable/40214516
Stilwell, L. A., Reese, L. M., & Diamond-Dalessandro, S. P. (2011). Predictive validity of the LSAT: A national summary of the 2009–2010 LSAT correlation studies. LSAT Technical Report Series, 11(2), 1-19. Retrieved from https://www.lsac.org/docs/default-source/research-(lsac-resources)/tr-11-02.pdf