roadmap to excellence: key concepts for evaluating the

Roadmap to Excellence: Key Concepts for Evaluating the Impact of Medical School Holistic Admissions

Association ofAmerican Medical Colleges

Learn

Serve

Lead

2013

Association ofAmerican Medical Colleges


About this Publication Roadmap to Excellence: Key Concepts for Evaluating the Impact of Medical School Holistic Admissions is the third in a series of publica-tions produced by the AAMC Holistic Review Project intended to help medical schools establish and implement institution-specific, diversity-related policies that will advance their core educational goals with minimal legal risk. To successfully achieve the educational and health care-related benefits that come from a diverse student body requires school-wide, concerted efforts. Therefore, the AAMC encourages medical schools to use this publication as a tool to guide collaborations and discus-sions among their institution’s leadership; faculty; admissions, diversity affairs, financial aid, and recruitment and retention officers; admissions committee members; legal counsel; students; and others engaged in and affected by diversity-related issues.

This publication was edited and coordinated by Amy Addams, lead engagement specialist with the AAMC Holistic Review Project.

Note: The content of this publication should not be construed as legal advice, and readers should not act upon information contained in this publication without consulting professional counsel.

AcknowledgmentsThe AAMC extends its appreciation to all those who participated in producing this publication. In particular, we thank the current and former members of the AAMC Advisory Committee on Holistic Review, whose members guided the development of this publication, and internal AAMC reviewers Carol Aschenbrener, M.D., (AAMC chief medical education officer), Marc Nivet, Ed.D., (AAMC chief diversity officer), Robert Sabalis, Ph.D., (LCME assistant secretary, emeritus), and Frank Trinity, J.D., (AAMC chief legal officer), who offered expert review of the document. We would also like to thank Ruth Beer Bletzinger, the former director of the Holistic Review Project, for her extensive contributions in editing, writing, and shaping this publication.

© 2013 AAMC. All rights reserved. This report may be reproduced and distributed

by and within AAMC-member organizations only.

This publication is also available free of charge on the AAMC Web site at: https://

www.aamc.org/initiatives/holisticreview/.

For more information about this publication, please email [email protected].

Association of American Medical Colleges, 2013iii


Table of Contents

Foreword ...........................................................................................................................................v

Executive Summary ........................................................................................................................vii

Chapter 1: Evaluation 101: Why, What, Where, When, Who, and How? ....................................1Authors: Kate Lipper, J.D.; Arthur L. Coleman, J.D.

Why is Evaluating Holistic Admissions and Related Diversity Programs Critical for My Medical School? ......1Which Data Are Needed and Where Can They Be Found? ..........................................................................4When Should My Medical School Conduct Holistic Admissions-related Evaluation? ....................................6Who is Responsible for Evaluating Holistic Admissions-related Efforts at My Medical School? .....................7How Should Evaluation Be Carried Out?.....................................................................................................8Summary ....................................................................................................................................................9

Chapter 2: Evaluating Enrollment Management Services to Promote Successful Holistic Admissions Outcomes ....................................................................................................................11Authors: Kate Lipper, J.D.; Arthur L. Coleman, J.D.

Strategic Use of Enrollment Management ................................................................................................11Structure-Process-Outcomes Framework ...................................................................................................13Applying the Structure-Process-Outcomes Framework ..............................................................................14Summary .................................................................................................................................................17

Chapter 3: The Matriculated Student: Assessing the Impact of Holistic Review ......................19Authors: Jeffrey F. Milem, Ph.D.; Celia L. O’Brien, Ph.D.; W. Patrick Bryan, M.A.

A Model for Determining the Impact of Holistic Review ...........................................................................19Student Input and Entry Characteristic Measures .....................................................................................21Environmental Measures ...........................................................................................................................23Outcome Measures ..................................................................................................................................25Summary ..................................................................................................................................................29

Chapter 4 :Evaluating Workforce Outcomes ................................................................................31Author: Somnath Saha, M.D., M.P.H

Measuring Domain Outcomes ..................................................................................................................31Potential Data Sources .............................................................................................................................34Summary ..................................................................................................................................................37

Endnotes ..........................................................................................................................................39

Appendix A: AAMC Experiences-Attributes-Academic Metrics Model ....................................A-1

Appendix B: Important Considerations for Evaluating Enrollment Management Functions ....A-3

Appendix C: Committees ............................................................................................................A-13AAMC Advisory Committee on Holistic Review (Current)...................................................................... A-13AAMC Advisory Committee on Holistic Review (2007–2012) ............................................................... A-14

Association of American Medical Colleges, 2013v


Foreword

With all the responsibilities and pressures that you face, why should you take the time to read this document? The answer is simple—the information you will find here will likely change your view of your work. Each of you is committed to enhancing the physician workforce in America. If you were not committed to this goal, you would not be reading this Roadmap to Excellence. However, if the medical schools that have implemented holistic review in their admissions processes, and the medical schools that are about to do so, do not each put in place an evaluation framework for their work, then how will we as a community be able to demonstrate what has changed in medical school admissions, and what the effects of those changes have been, five-, ten-, or twenty years from now?

If you are involved in identifying, recruiting, selecting, and/or educating future physicians who are prepared to provide excellent health care in a complex and diverse society—or oversee those who do—read on to learn how evaluation is key to assuring that your school’s:

• holistic admissions process and related efforts align with institutional mission

• outcomes in the short-, mid-, and long-term support the school’s mission-driven goals

• holistic admissions policies pass legal muster

• decisions about who is and who is not accepted to your school are based on evidence, not anecdote

While there are multiple paths to medical school, there is only one path to becoming a physician. Admissions is the only point of access—the only door to a medical school education and a medical career. What a school selects for determines the types of people who will study and then practice medicine. This is the reason that implementing a holistic admissions process (described in previous AAMC Holistic Review Project publications: Roadmap to Diversity: Integrating Holistic Review Practices into Medical Admissions Processes and Roadmap to Diversity: Key Legal and Educational Policy Foundations for Medical Schools) is so important. It creates a basis for the school to attract, select, and train a diverse student body capable of meeting the aspirations articulated by the institution.

Holistic admissions requires ongoing evaluation. In addition to examining data for improving processes, evalu-ation is necessary to demonstrate the vital connections across holistic admissions, the school’s interest in diversity, and achieving institutional mission and excellence. Through evaluation, your school can address the following mission-critical questions:

• Are the school’s enrollment practices aligned to attract and support the broad range of diversity it seeks?

• If the school is now more balanced in its use of metrics (MCAT® scores, GPA) than previously, how does it gather, measure, and communicate data about the attributes and experiences it uses to assess applicants?

• What is the interplay between individual student attributes and experiences within the institutional environment, and is it yielding the personal and professional outcomes the school desires for its students?

• Has implementing holistic admissions—and, by extension, fully integrating holistic thinking into the institution’s mission-driven diversity efforts—fundamentally changed and improved the institution and the educational experience of students in ways that positively affect the physician workforce and, most important, the health of the public?

Association of American Medical Colleges, 2013vi


Defining—in clear and concrete terms—what “success” and “excellence” mean at your school is essential to evaluating whether integrating mission-driven, holistic admissions and related diversity interests help achieve that success and excellence. The school, then, has the foundation to determine whether goals are met, how to sustain and enhance existing efforts, and which programs, policies, processes, or practices need to be modified or abandoned. By developing an institutionally shared, specific, and measurable definition of success, any school will be better able to put into perspective externally defined and valued—if not validated—metrics (notably, national rankings) with data that are directly applicable to achieving the school’s intended mission.

Fortunately, medical schools are repositories of significant amounts of applicant, student, and graduate data. The challenge lies in making sense of these data and determining which are most relevant and how best to use them. This publication offers key stakeholders—medical educators and administrators—frameworks, methods, and tools to employ these data effectively to identify domains and other information to be collected and/or more closely examined.

Without systematic evaluation, holistic admissions and the benefits associated with it in the near- and long-term can easily be criticized as nebulous and unquantifiable. While the effort described herein requires thought, time, and energy, it also provides the basis for a clear, evidence-based and documentable rationale for using holistic admissions. The results of these analyses also allow insight into the quality and effectiveness of essential related initiatives, as well as the means to enhance and sustain them.

For those schools that have begun—congratulations! For those considering these changes—we wish you all the best! Wherever your school might be on this important journey, we believe the Roadmap to Excellence will help get you there.

Alicia D. H. Monroe, M.D. Jim Scott, M.D.Chair, AAMC Advisory Committee on Former Chair, AAMC Advisory Committee onHolistic Review Holistic ReviewChief Academic Officer Professor, Emergency MedicineVice Dean, Educational Affairs The George Washington University School of USF Health Morsani College of Medicine Medicine and Health SciencesUniversity of South Florida

Association of American Medical Colleges, 2013vii


Executive Summary

The Roadmap to Excellence: Key Concepts for Evaluating the Impact of Medical School Holistic Admissions is designed to help your medical school achieve mission-driven educational excellence through its diversity efforts. The content builds on and can be used in conjunction with two previous publications in the AAMC Holistic Review Project Roadmap series (see sidebar).

The potential impact of conducting holistic admissions is wide-ranging, particularly in relation to the “diversity rationale.” Explored in more depth in Chapter 3, simply put, the diversity rationale emphasizes that student body diversity contributes to attaining core mission-driven goals, including:

• achieving academic excellence

• fostering an enriched educational environment and learning community that benefit all students

• shaping a physician workforce that is better able to meet the needs of a diverse nation within a global society

Additionally, holistic review is a requirement for conducting legally viable race- and ethnicity-conscious admissions policies.

This Roadmap focuses on evaluating whether your institution’s admissions policies, processes, and practices are yielding the diverse student body your school wants to educate and graduate, and the degree to which your school is lever-aging that diversity to achieve intended educa-tional and professional outcomes. Specifically, the document offers guidance for rigorously and reliably assessing the extent to which holistic admissions is achieving or contributing to intended institutional outcomes over the short-, mid-, and long-term. The approaches will also assist your school in gathering evidence to make and review institutional decisions over time. On the following pages, you will find examples of frameworks, methods, research questions, and resources. While each framework is presented in a particular context (e.g., applying a structure-process-outcome framework to enrollment management efforts), these tools can be applied to multiple program and policy areas.

The AAMC Holistic Review Project’s two previous Roadmap publications communicate the importance of evaluation from different perspectives. Both also emphasize the centrality of mission to admissions and fully aligning admissions with other key institutional policies, processes, and practices. In combination with this Roadmap to Excellence, your medical school has a toolbox for developing, implementing, and assessing the effectiveness of its holistic admissions process, diversity policies and programs, and the impacts of holistic admissions on your medical students and graduates.

Roadmap to Diversity: Key Legal and Educational Policy Foundations for Medical Schools introduces evaluation to determine whether policies are attaining desired diversity, as defined by each school. It also discusses evaluation as a means of creating an evidentiary basis for the school-specific benefits of diversity to convince stakeholders, including faculty, staff, students, the public, and the legal system. The document concludes with a tool, “Medical School Diversity Self-Assessment,” designed to facilitate ongoing develop-ment and refinement of access and diversity policies.

Roadmap to Diversity: Integrating Holistic Review Practices into Medical School Admissions Processes focuses narrowly and in detail on implementing and maintaining a fully developed holistic admissions process. The document delves into considerations for membership on the admissions committee; criteria for screening, interviewing, and selecting applicants; developing communications and training strat-egies; and, of course, evaluating the effectiveness of a school’s holistic review admissions policies, processes, and practices. The “Holistic Review Admissions Checklist” at the end of the publication provides each medical school with a means of ascertaining the success of the holistic admissions process itself.

Association of American Medical Colleges, 2013viii


This publication does not require a background in evaluation and assessment, but rather a common sense understanding of why such measures are important. The chapters are written as stand-alone essays with consis-tent themes and tools throughout. With each chapter, the authors have attempted to strike a balance between a high-level evaluation treatise and a detailed evaluation primer. A brief description of each chapter appears below.

• Chapter 1: Evaluation 101 lays out the fundamentals of evaluation using the familiar “Journalism 101” framework: Why, What, Where, When, Who, and How.

• Chapter 2: Evaluating Enrollment Management Services to Promote Successful Holistic Admissions Outcomes proposes a framework for assessing strategic enrollment management activities, such as recruitment, financial aid and debt management, and academic and career advising and support so they can be aligned to create and support a diverse student body.

• Chapter 3: The Matriculated Student: Assessing the Impact of Holistic Review presents means for gathering and classifying data so that medical educators and administrators can:

– determine how successful they have been in implementing holistic review in admissions

– evaluate the impact that holistic admissions has had in advancing the mission of the medical school

• Chapter 4: Evaluating Workforce Outcomes describes outcomes in the domains of attitudes, competen-cies, and career goals, as well as possible data sources and collection methods.

Before delving into the body of Roadmap to Excellence, please review the sidebar, “Points to Keep in Mind About…” This is an at-a-glance chart with descriptions for terms frequently used by the AAMC Holistic Review Project, such as “diversity,” “holistic review,” “holistic admissions,” “benefits of diversity,” and so forth. In addition to providing key background information about holistic admissions, it is also constructed as a ready-to-use handout for presentations and discussions at meetings with admissions committee members, faculty, medical school leaders, administrators, students, and other stakeholders.

Association of American Medical Colleges, 2013ix


Points to Keep in Mind About……what is meant by medical school holistic review.

Holistic review is a flexible, individualized way of assessing an applicant’s capabilities by which balanced consideration is given to experiences, attributes, and academic metrics (E-A-M) and, when considered in combination, how the individual might contribute value as a medical student and future physician.

…what is meant by a holistic admissions process.

An integrated holistic admissions process incorporates four core principles at each stage: screening, interview, and selection.

The Four Core Principles of Holistic Admissions:

1. Selection criteria are broad-based, clearly link to school-specific mission and goals, and promote diversity as essential to achieving institutional excellence.

2. A balance of E-A-M is:

• used to assess applicants with the intent of creating a richly diverse interview and selection pool and student body

• applied equitably across the entire candidate pool

• grounded in data that provide evidence supporting the use of selection criteria beyond grades and test scores

3. Admissions staff and committee members give individualized consideration to how each applicant may contribute to the medical school learning environment and practice of medicine, weighing and balancing the range of criteria needed in a class to achieve the outcomes desired by the school.

4. Race and ethnicity may be considered as factors when making admission-related decisions only when aligned with mission-related educational interests and goals associated with student diversity AND when considered among a broader mix of factors, which may include personal attributes, experiential factors, demographics, and other considerations.

…what is meant by diversity in the context of holistic review.

• Diversity is not an end goal, but a means to achieving core educational goals as defined by the medical school.

– As such, diversity serves as a driver of educational excellence and a mechanism for graduating physicians who contribute to health care consistent with institutional mission.

• Diversity is a multidimensional concept.

– It may include, but does not exclusively refer to race, ethnicity, or gender. Diversity may encompass other dimensions of experiences and attributes, such as distance traveled, educational background, languages spoken, resilience, socioeconomic status, and geography, among others.

• Diversity is not a “one-size-fits-all” concept, but an inherently institution-specific concept.

– While likely sharing common elements, the diversity interests of one medical school may be quite different from those of another school based on differences in institutional mission, educational goals, the kind of students a medical school wants to educate, and the kind of physicians it wants to graduate.

Association of American Medical Colleges, 2013x


Points to Keep in Mind About……implementing a holistic admissions process.

• Medical school admissions is a key point on the medical education continuum, not an isolated event.

• For holistic review admissions policies to be effective, these policies need to be:

– aligned with institutional mission and goals

– integrated across outreach, recruitment, financial aid, retention, and curriculum

– consistent with and directly connected to institutional values

• Establishing holistic review admissions policies may constitute an organizational change effort that requires clear support from institutional leaders and the participation of administrators, faculty, students, and other stakeholders.

…the AAMC Holistic Review Project.

• The goal of the AAMC Holistic Review Project is to support excellence in admissions, while also working across medical education in order to maximize the benefits of holistic admissions across the full spectrum of education and development.

• It does so by assisting medical schools in establishing, implementing, and evaluating mission-driven, student diversity-related policies, processes, and practices spanning the medical education continuum that help build a physician workforce capable of and committed to improving the health of all.

Association of American Medical Colleges, 20131


Chapter 1Evaluation 101: Why, What, Where, When, Who, and How?

Authors: Kate Lipper, J.D.; Arthur L. Coleman, J.D. EducationCounsel LLC

Evidence-based medicine is firmly established as the gold standard in practice; similarly, quality improve-ment principles are integral to health care. Why should our approach to the components of medical education be any different? Throughout this publica-tion, we draw from accepted principles and practices in health care and educational research and apply them to domains within undergraduate medical education. There are multiple evaluation models and frameworks from which to choose; we highlight a few in the following chapters. Regardless of which model you use, there are some fundamental principles to consider.

This chapter introduces a basic evaluation framework adapted from a familiar “Journalism 101” approach (see Table 1.1) by examining five questions:

• Why is evaluating holistic admissions and related programs critical for my medical school?

• Which data are needed and where can they be found?

• When should my medical school conduct evalua-tion of its holistic admissions and related practices?

• Who is responsible for evaluating holistic admis-sions-related efforts at my school?

• How should evaluation be carried out?

There is, of course, overlap across the questions. Nonetheless, they offer a solid schema for under-standing what evaluation entails. Especially for those of you who are not regularly engaged in conducting assessments, our intention is to help you clarify your thinking about evaluating the impact of holistic admissions at your institution and on the graduates you produce.

Why is Evaluating Holistic Admissions and Related Diversity Programs Critical for My Medical School?

In the context of holistic admissions, the question of “What is evaluation?” folds neatly into the more complex question of “Why is evaluation critical?” At its most literal, to evaluate means to judge or determine the significance, value, or worth of something; to assess it.1 However, this deceptively simple definition underlies the substantial and multi-faceted impact that doing evaluation—and learning from and sharing the results—can achieve.

Fundamentally, from both education and legal stand-points,2 evaluation helps medical educators determine how successful they have been in advancing insti-tutional mission and goals—including benefits derived from increased diversity among students and graduates—by implementing holistic admissions and related programs. Specifically, conducting evaluation helps you learn about what is working, what is not working, and why. It also enables you to think, plan, and act based on the evidence and provides insights into what was not known previously, as well as what and how to improve. These analyses can be used to better understand the experiences students have in medical school, how students change over time, and what role their experiences have in influencing these changes. Moreover, evaluation is important to ensuring that, as the student body becomes more diverse, your school is doing everything it can to facil-itate the success of all of its students.



Table 1.1: At-a-Glance: The Why, What, Where, When, Who, and How of Evaluating Holistic Admissions and Related Programs

The Questions The Considerations

Why is evaluating holistic admissions and related programs critical for my medical school?

• Assess whether outcomes align with institutional mission and goals

• Ensure with clear evidence that diversity efforts meet legal muster

• Make decisions and performance improvements based on data and evidence, not on anecdote, memory, or “common knowledge”

• Tell the school’s story using data, not untested assertions

What data are needed? • Student data, both qualitative and quantitative

• Institutional environmental data, both qualitative and quantitative

Where can the data be found? • School-based applicant, student, and graduate sources

• AAMC questionnaires and databases

• State and national organizations

When should my medical school conduct holistic admissions-related evaluation?

• Plan evaluation activities from the beginning, keeping the end in mind

• Conduct ongoing evaluation from recruitment and outreach activities through admissions and at key intervals during students’ time at your medical school and after graduation to produce:

– process/formative evaluation

– outcomes/summative evaluation

Who is responsible for evaluating holistic admissions-related efforts at my school?

• Leadership staff: ensuring institutional commitment, resources, and recognition

• Administrative staff and faculty: ensuring that their specific programs are functioning effectively

• A team of stakeholders: coordinating and conducting comprehensive evaluation across holistic admissions-related efforts

How should evaluation be carried out? • Define success based on institutional mission

• Translate mission into clear goals, including diversity goals

• Set outcome measures to assess progress toward goals

• Identify data and sources needed to measure outcomes

• Collect and analyze data at regular intervals for short-term, intermediate-term, and long-term impacts

• Craft narrative based on findings, identify areas for improvement, make recommendations, dispel incorrect assumptions

• Disseminate findings/recommendations to stakeholders (see Figure 1.5)



Evaluating the impact of diversity both among matriculating students and in the educational environment on the decisions graduates make about where, what, and how they will practice can help your medical school determine whether or not its culture, policies, processes, and practices are contributing to desired long-term, as well as shorter-term, outcomes. Additionally, continuing to study the relationship across the educational environment, characteristics of the physician workforce, and patient outcomes is likely to yield significant insights into how to improve the comprehensive system of medical education and practice. (see Figure 1.1)

At the core, conducting evaluation enables your school to tell its own story by using data and evidence. In the absence of data, people often make

assumptions based on anecdote, memory, or so-called common knowledge. Evidence and data collected over time helps to dispel deeply rooted beliefs and biases, including those about student performance and success. The real power comes in interpreting and communicating the data you collect and in telling the story as it actually is, not how you might wish it were. Through the process of evaluation, you are able to identify and document your successes and develop compelling, data-driven messages that communicate those successes. Similarly, you can use findings from your analyses of these data to make recommendations for ongoing performance improvement based on the evidence, not on well-worn, yet untested assertions.

Figure 1.1: Outcomes Influenced by Holistic Admissions

Jeff Milem, Celia O’Brien, and Patrick Bryan have identified three outcomes influenced by holistic admissions and related policies and processes. The student outcomes they identify, in combination with several other factors, contribute to workforce and population health outcomes. The students an institution selects and graduates and how the school educates them influence not only the composition, but the preparedness, cultural competence, and professionalism of the physician workforce.

• Learning outcomes: These outcomes can measure how well the curriculum functions in teaching students core competencies necessary in future physicians. Learning outcomes can also be assessed in relation to the types of experiences a student is exposed to during medical school (e.g., extra-curricular activities or interactions with diverse others). Typically, these outcomes are related to obtaining and maintaining the knowledge necessary to succeed in the classroom, in clinical exams, and on USMLE® Step exams.

• Developmental outcomes: Medical schools also strive to teach students core values and attitudes related to the humanistic and altruistic practice of medicine, as well as others defined by the institutional mission. These outcomes may be measured as the development of ethical attitudes and behaviors during their time in medical school. This is closely related to the concept of teaching “professionalism” to medical students.

• Professional outcomes: Medical school evaluators should also be concerned with the professional or career devel-opment of their students. Again, this is always linked closely to the mission of any given school. However, these outcomes most likely are related to specialty choice, choice of population served, and the geographic region of future practice. In many cases, a thorough evaluation of professional outcomes requires a medical school to conduct ongoing follow-up surveys of graduates as they leave their residencies and enter professional practice.



Which Data Are Needed and Where Can They Be Found?

Our intent here is not to provide you with an exhaus-tive list of data to collect and analyze. It is, instead, to present some basic types of data that might be useful in evaluating the impact of holistic admissions at your medical school. Rather than gathering data for data’s sake, the point is to think critically about what data you need in order to answer essential evaluation questions. Your institutional mission and goals serve as the cornerstone for identifying both the questions and the requisite data (see Figure 1.2).

Educationally and, in certain instances, legally,3 sustain-able evaluation efforts require collecting and examining evidence. Medical schools must identify and otherwise develop evidence that can assess the issues and effects of holistic admissions and related activities, including diversity-focused policies and practices. That being said, evidence collection can be robust without being onerous. It can range from requiring few resources and little training to more complex and extensive endeavors. For example, results from the AAMC Graduation Questionnaire (GQ) and the National Resident Matching Program (NRMP) offer ready indications

about whether your school’s graduates plan to pursue careers that align with institutional mission. Whether they fulfill their original aspirations, however, would require more intensive, longitudinal data collection.

In evaluating the impact of holistic admissions, neither individual/cohort student data, such as grades, time to completion, and the range of experiences, attributes, and academic metrics (see Appendix A, p. A-1, for the E-A-M model), nor institutional data that reflect the school’s pedagogy, climate, and culture are suffi-cient on their own. These data in combination reveal a fuller picture of the medical school’s performance, success, and areas for improvement that cannot be exposed by concentrating narrowly on student progress and outcomes. In Chapter 3, Milem, O’Brien, and Bryan expand on evaluating the relationship between individual student data and institutional environ-mental factors in student and cohort development and performance, which are vital to a school’s under-standing of how best to leverage the student body diversity attained through holistic admissions.

As with student and institutional data, it is good practice to gather both quantitative and qualita-tive data. Looking at the two types of data together

Figure 1.2: Institutional Mission: The Foundation for Evaluation

Just as your institutional mission guides all of your policies and programs, it also undergirds your evaluation efforts. However, institutional mission statements are often intentionally broad and conceptual. Translating your mission statement into clear, concrete, actionable goals can therefore be a valuable exercise in both developing and evaluating policies and programs.

Your school’s mission and goals also form the foundation for your institution’s commitment to diversity. Think carefully about how your institution defines diversity and consider what experiences, attributes, and academic metrics it seeks in its student body.

As a result, evaluating the impact of your school’s holistic admissions process and other diversity-related policies and programs is about far more than numbers. It is a comprehensive plan that takes into account:

• specific institutional context • particular kinds of diversity sought • synergy between diversity and environmental factors that results in educational excellence

If a medical school seeks to learn the degree to which student body diversity supports and advances the institutional mission, it is critical to evaluate how diversity is integrated into the medical education enterprise and how it drives educational excellence.



provides a more nuanced understanding of what is working or not working and why. While quantita-tive data offer a good sense of what is going on, qualitative data often elucidate the why and how. For example, you might collect quantitative data over time through surveys or questionnaires to document changes in students’ empathy levels throughout their time in medical school. Through qualitative methods, such as interviews or focus groups, you can tease out why and how those changes occur. Your school, then, would have the information it would need to develop specific interventions or make program modifications.

A related question to “What data?” is, “Where to find the data?” Fortunately, medical schools already have a great deal of data at their disposal. In addition, Chapters 2, 3, and 4 contain information about where schools can access data to conduct evaluations. A partial list of data sources appears in Table 1.2. It identifies general data categories and examples of associated data sources that are institutionally developed, available through the AAMC, or produced by other organizations.

Table 1.2: Examples of Data Categories and Sources for Evaluating the Impact of Holistic Admissions at a Medical School

Examples of Data Categories Examples of Data Sources

Applicants • AMCAS®

• School’s secondary application

• Letters of recommendation

• MCAT® examination

• School-developed applicant surveys, focus groups, individual interviews

Students • AAMC Matriculating Student Questionnaire (MSQ)

• Course and clerkship grades and test results

• USMLE®, Step 1 and Step 2 results

• Faculty feedback

• Patient feedback

• School-developed student surveys, focus groups, individual interviews

• Time to completing medical school

Graduates • AAMC GQ

• NRMP results

• USMLE® Step 3 results

• Residency feedback

• Data compiled by state medical boards, professional societies, and the American Medical Association

• School-developed alumni surveys, focus groups, and individual interviews

Institution • AAMC Medical School Mission Management Tool

• AAMC GQ

• Student course and clerkship evaluations

• School-developed student and graduate surveys, focus groups, and individual interviews

• School-developed faculty and staff surveys, focus groups, and individual interviews



When Should My Medical School Conduct Holistic Admissions-related Evaluation?

There are two fundamental timing considerations for evaluating the effectiveness of holistic admissions at your medical school. The first consideration addresses when to develop an evaluation plan; the second addresses when to carry out the evaluation.

When to develop an evaluation planEvaluation is a critical component of holistic admis-sions, as with any initiative or change effort, though it is all too often an afterthought. How else will we know whether complementary enrollment and student support-related policies, programs, and practices support the school’s mission and goals? For evaluation to play a meaningful role in your school’s continuous performance improvement efforts, it must be regular, ongoing, and thoroughly integrated into institutional planning and work. While it is ideal for the evaluation to be a focus from the beginning of the effort, the most important point is that the evaluation is conducted. For example, if your school has already implemented holistic admissions, it can still develop and implement an evaluation plan.

Starting with the end in mind, as opposed to building a process as you go along, often illuminates your work in ways you may not have considered earlier. For example, beginning with what we want to measure challenges us to be clear about what we are trying to

achieve and how we go about doing so. Remember, evaluation is an iterative process. You do not have to wait until everything is decided to begin developing an evaluation plan. Rather, you can start with a seed of an idea—a hypothesis—and nurture and develop it over time.

When to carry out the evaluationAdmissions and matriculation data are obviously important for schools conducting holistic admis-sions, and they are addressed in Roadmap to Diversity: Integrating Holistic Review Practices into Medical School Admissions Processes.4 However, what happens in student selection is only one part of a compre-hensive plan for evaluating the impact of holistic admissions. It is equally important to explore what comes before and after your students come in the door. By conducting evaluation at critical points, your school can use what it learns to guide decision making, planning, and action. Minimally, as shown in Figure 1.3, evaluation should take place:

• during the application and admissions process

• after selecting a class, but immediately prior to or at matriculation

• at key intervals during medical school, such as the end of each course/clerkship, the end of each academic year, and following important events as determined by the school

1. Admissions 2. Orientation 3. During Medical School

4. Graduation 5. Post-Medical School

Figure 1.3: When to conduct evaluation?

* Milem JF and CL O’Brien. “Evaluating the Impact of Holistic Review.” 2012 AAMC Annual Meeting. San Francisco, Calif. November 2, 2012.



• at the time of graduation from medical school

• at key intervals post-medical school, such as after the first year of residency training, at completion of residency, and at points determined by the school post-residency5

Collecting data on an ongoing basis allows your medical school to conduct two types of evaluation:

• formative evaluation, which is typically conducted during the development or improvement of a program

• summative evaluation, which involves making judgments about the efficacy of a program… at its conclusion6

Both assess whether the school’s holistic admissions and related efforts are supporting institutional mission and goals. Formative evaluation, which includes process evaluation, uses data “snapshots” to gain a sense about needs, implementation, and program status in the short-term. Over time, the same data offer longitudinal summative or outcome informa-tion that reveals trends, impact, and effectiveness. Together, the analyses from these evaluations are vital to meaningful performance improvement (see Figure 1.4). As important, formative and summative evaluations provide a powerful tool for leveraging diversity to the benefit of the school, its students, its graduates, and the patients they serve.

Who is Responsible for Evaluating Holistic Admissions-related Efforts at My Medical School?

Achieving the full impact of holistic admissions relies on more than what happens during the student selection process. Your school’s programs across the education continuum, as illustrated in Figure 1.3, are equally critical components. This means that the institution’s holistic admissions stakeholders include administrators, faculty, and students, as well as legal counsel. They all have central roles to play in achieving successful outcomes. Leadership, genuine commitment of resources, and institutional recog-nition for the importance of the work are also key factors in insuring that those tasked with conducting evaluation are able and motivated to do so.

Establishing an evaluation teamStaff members who direct individual programs or processes are, of course, responsible for ensuring the effectiveness of the projects assigned to them. However, given the potential multifaceted impact of holistic admissions, your school should consider establishing an evaluation team. In this way, the responsibilities for collecting, analyzing, and inter-preting data are shared, rather than resting with a single individual or office. Such a team could include representatives from student services, academic

Figure 1.4: Formative and Summative Evaluation

William Trochim’s Web site, Research Methods Knowledge Base, provides a comprehensive introduction to social research methods information. It is a useful resource for those who are interested in conducting evaluation but do not have a background in research methods. Trochim describes formative and summative evaluation as follows:

Formative evaluations [emphasis added] strengthen or improve the object being evaluated—they help form it by examining the delivery of the program or technology, the quality of its implementation, and the assessment of the organizational context, personnel, procedures, inputs, and so on. Summative evaluations [emphasis added], in contrast, examine the effects or outcomes of some object—they summarize it by describing what happens subsequent to delivery of the program or technology; assessing whether the object can be said to have caused the outcome; determining the overall impact of the causal factor beyond only the immediate target outcomes; and, estimating the relative costs associated with the object.*

* Trochim, WM. The Research Methods Knowledge Base. “Introduction to Evaluation.” Retrieved on January 22, 2013 from http://www.socialresearchmethods.net/kb/intreval.php.



advising, diversity affairs, and financial aid; faculty; students; and community members, as well as admis-sions. In addition, it is important to involve legal counsel at all stages so that programs pass legal muster and to ensure that the evidence is convincing from a legal standpoint. The previous Roadmap publications from the AAMC Holistic Review Project provide useful guidance on identifying stakeholders, as well as convening an evaluation team.1,2 Such a team would also benefit from tapping into expertise both within and outside the medical school and the institution. Here are a few considerations for finding and partnering with experts, both within and external to your institution.

Expertise within the medical schoolWhen it comes to developing and implementing a comprehensive evaluation plan, think of key partners who might already exist at your school:

• What evaluation processes are already in place? Who is collecting data and what data are they collecting?

• Does your medical school have an office of institu-tional research/effectiveness?

• Are there faculty members who have background or experience in conducting educational research?

• Might your learning specialists and advisors/counselors help you evaluate different learning or developmental outcomes?

• Are there other programs or units that have recently undertaken a comprehensive evaluation effort?

Expertise within the universityIf you are part of a university campus, consider engaging faculty, researchers, and/or graduate students in departments such as education, sociology, psychology, and public health.

Expertise outside of the institutionThink about and engage your medical education colleagues at other schools who are doing or want to do this work. The staff of the AAMC Holistic Review Project recognize the importance of institutions talking to and learning from each other. One of our commitments is to create forums for you to do just that. There are also publications to support this work, such as this one, along with other resources that can be found in the Appendix.

How Should Evaluation Be Carried Out?

The answer to “how to evaluate” begins by clearly and specifically articulating what success looks like at your medical school. For a holistic admissions process, as well as related policies and programs across the continuum, success must be informed by institu-tional mission and clarity about goals. Evaluating the impact of holistic admissions also necessitates having a shared, mission-driven understanding among stake-holders about the kind of student diversity the school seeks.

To assess whether holistic admissions efforts are successful, the school’s evaluation team should:

• translate mission into clear goals

• set outcome measures to assess progress toward goals

• identify the data and sources needed to measure outcomes, including student-specific data and data about the institutional environment likely to affect holistic admissions outcomes

• collect and analyze the data at regular intervals to determine short-, intermediate-, and long-term impacts

• craft narratives around findings to tell the story of successes, identify areas for improvement, recommend modifications, and dispel incorrect, anecdote-based assumptions



• disseminate findings and recommendations to stakeholders (see Figure 1.5)

In other words, think about what is most important to your school. Think of why, what, when, and who as the ingredients. What story is to be told? To whom? What evidence is needed? Then, you can consider what data you need to collect, when you need to collect them, and who can provide them or help you gather them. “How” is the challenging, but ultimately rewarding process of putting it all together to assess impact and identify areas of improvement and refinement. This is good educational practice, good business practice, and, for schools using diversity admissions policies that consider race and ethnicity, necessary legal practice.1

Summary

Evaluation can be done on a continuum from simple to complex. It is possible to do rigorous evaluation without it being overwhelming or complex. This Roadmap is intended to provide you with infor-mation, frameworks, and resources to spark your thinking and help you develop a robust institutional evaluation plan to assess the full impact of holistic admissions at your medical school. In other words, keep reading…!

Figure 1.5: Disseminating Findings and Recommendations

In discussing how to conduct evaluations, communicating evaluation results deserves emphasis. If not broadly shared with stakeholders, evaluation data, analyses, and findings will languish, making it difficult to tell the story, dispel assumptions, promote recommendations, inform decision making, and take action to improve performance. In preparing to disseminate the information, be sure to think strategically and practically about when, how, with whom, and in what format so the data are optimally useful and influential.

The “Considerations for Developing a Holistic Review Communications Strategy” included in Chapter 5 of Roadmap to Diversity: Integrating Holistic Review Practices into Medical School Admission Processes might be a useful tool as you think about disseminating evaluation findings.



Chapter 2Evaluating Enrollment Management Services to Promote Successful Holistic Admissions Outcomes

Authors: Kate Lipper, J.D.; Arthur L. Coleman, J.D. EducationCounsel LLC

As Chapter 1 emphasizes, fulfilling the promise of holistic admissions depends on far more than what happens during the student selection process. Just as you give careful and genuine consideration to how each applicant might contribute to medical education and practice, you must simultaneously think criti-cally about what your institution does to ensure that all students succeed and how to leverage their diverse contributions to enrich the learning environ-ment. Being intentional about what comes before and after individuals matriculate is vital to your medical school’s mission and the ability of the holistic admis-sions process to advance the school’s mission-driven goals. These efforts encompass a range of what is known at the undergraduate level, and increasingly among medical schools, as enrollment management services (see Table 2.1).

Evaluating the full spectrum of your enrollment management services, both pre-matriculation and post-matriculation, enables your school to determine whether it is recruiting, admitting, and sufficiently supporting a diverse student body that can support its institutional mission and goals. If your school uses race- and ethnicity-conscious policies, it is also a legal necessity. To these ends, we discuss the strategic use of enrollment management and present a straight-forward framework for evaluating, coordinating, and improving functions and activities in the enrollment management sphere.

Strategic Use of Enrollment Management

At its best, enrollment management is a coordinated, evidence-driven approach to activities that “concern student college choice, transition to college, student

Table 2.1: Examples of Enrollment Management Services

This figure presents a partial list of enrollment management services. It categorizes them into two main areas: pre-matriculation and post-matriculation. The functions and activities under each are generally assigned across such medical sc`hool offices as student affairs, academic affairs, diversity affairs, and, in some cases, even the business office. The actual distribution of responsibilities varies from school to school. Differences notwithstanding, the key for your school is to ensure that the programs are aligned with institutional mission and work both discretely and in tandem to support and leverage the student diversity developed through its holistic admissions process.

Pre-Matriculation Functions and Activities Post-Matriculation Functions and Activities

• Outreach and recruitment

• Pipeline efforts

• Diversity programming and support

• Admissions

• Financial aid (including debt management)

• Aacademic advising and support

• Financial aid (including debt management)

• Career advising

• Diversity programming and support

• Mentoring

• Student health services (physical health and mental health)

• Registrar/student records

• Housing



attrition and retention, and student outcomes.”7 In addition, as noted in Roadmap to Diversity: Key Legal and Educational Foundations for Medical Schools, the concept

“…can be useful for medical schools in designing and implementing an array of strategies integral to a medical school’s efforts to enroll and retain a class of students consis-tent with its core values and mission-driven goals. More specifically, the term means a strategic, integrated, and holistic process that influences the size, shape, and characteristics of a school’s student body—from efforts in recruitment, admissions, and financial aid (all targeted toward the matricula-tion of the desired entering class) to efforts focused on academic and career advising, as well as retention and student services. In essence, enrollment management consti-tutes an integrated and holistic approach for analyzing and influencing enrollments, involving a team of individuals on campus working together to achieve enrollment goals, with the ultimate aim of ensuring student and school success.”1

To be effective, enrollment management activities must be coordinated as part of the larger educational enterprise. Two hypothetical case studies show that institutions are not likely to reach their goals when individual enrollment management functions are operating from different playbooks and/or misaligned with institutional mission (see Figures 2.1 and 2.2).

As these case studies illustrate, aligning work across program areas and functions helps maximize effec-tiveness and efficiency within discrete areas. Unless program staff and faculty members are commu-nicating and collaborating across specific tasks/functions/the institution, they might not know about significant policies, processes, or changes in one area that have direct implications for the work in other areas. After all, it is what happens in enrollment-related activities, including recruitment, admissions, and financial aid, that make matriculation possible. Also, it is what happens after matriculation, including academic advising and support, diversity program-ming and support, ongoing financial support and debt management, and career advising that makes it possible for students and the school to thrive. Consider the following at your school:

• Are recruitment policies and practices grounded in admissions criteria, or are some populations of applicants being actively recruited who are unlikely to be admitted to the school?

Figure 2.1: Enrollment Management Case Study–Outreach, Financial Aid, and Leadership

The Case: Medical School A has a mission-driven objective to increase socioeconomic diversity within its student body and the physician workforce. To support the objective, the admissions office and diversity affairs office collaborated to implement a college recruitment program that increased the representation of low socioeconomic status (SES) applicants in the pool. However, the school diverted institutional funds from need-based assistance to merit-based scholarships without reconsidering its definition of “merit-based” aid to reflect applicants who would contribute to mission-driven diversity goals. This made the cost of attendance (i.e., tuition, fees, living expenses, etc.) unaffordable to accepted applicants with limited means, thus, stymieing matriculation rates from this cohort. The admissions office identified this obstacle while examining disparities in attributes of all accepted students and those who matriculated.

Considerations: In this case, Medical School A’s admissions office and diversity affairs office collaborated on developing an outreach program that succeeded in changing the makeup of the applicant pool to include more low SES candidates. The program failed to change the makeup of the entering class, however, because the financial aid office’s funding constraints were not addressed. The program’s ability to achieve a stated mission-driven goal relies on coordinating respec-tive efforts within admissions, diversity affairs, and financial aid in conjunction with a funding commitment from institu-tional leadership.



• If an institution changes its admissions criteria so that it is admitting applicants demonstrating a broader range of academic readiness, is it simul-taneously developing or enhancing academic advising and assistance programs that will help support the success of all students?

• If the school admits students whose backgrounds and experiences differ from those of the students the school has historically admitted, is it also considering whether post-matriculation enroll-ment management services—including diversity affairs, academic assistance, career advising, and debt management—are designed and imple-mented in ways that support these new student populations?

In short, if individual programs or functions are approached piecemeal, it is easy for them to become centered on insular issues and not on coordinated efforts for the benefit of students and the school as a whole.

Structure-Process-Outcomes Framework

Avedis Donabedian, M.D., a leader in health care quality assessment, developed the structure-process-outcome evaluation framework.8 We have adapted this foundational evaluation framework for the medical education context to help you evaluate the extent to which your medical school’s enrollment management strategy is able to support and leverage the diversity created through holistic admissions among its students and graduates. As with any evalu-ation effort, collecting and analyzing data over time allows the school to assess what is working and not working, identify trends, and develop and implement continuous improvement.

The structure-process-outcomes framework is apt because it is already part of the vernacular within medicine, and its organizing principles and catego-ries lend themselves well to evaluating enrollment management as a system. Donabedian’s approach captures the interplay between systemic components, institutional actors (i.e., administrators, counselors,

Figure 2.2: Enrollment Management Case Study–Pipeline Programs and Admissions Decisions

The Case: As part of Medical School B’s mission-based community outreach efforts and interest in improving health care in local underserved urban neighborhoods, it undertook a pipeline program to encourage area middle school and high school students to aspire to be physicians. The program was sponsored jointly by the school’s academic affairs and diversity affairs offices in collaboration with the local school system and with input from experts at the university’s school of education. The program was successful in producing a cohort of students who graduated from college and applied for admission. Through a combination of financial circumstances and available educational opportunities for students from this community, many of them attended two-year colleges before transferring to four-year schools. Because several members of Medical School B’s admissions committee strongly preferred applicants from elite undergraduate universities, qualified applicants from the pipeline program were rarely offered acceptances. The result was that the pipeline program succeeded in helping area students attain higher education degrees, but not in producing physicians who graduated from Medical School B. Though there was informal awareness of this bias among certain committee members, it was a formal evaluation conducted by the pipeline program director that documented the evidence and led to policy and practice changes.

Considerations: Medical School B’s pipeline program was created to carry out its mission to serve the community by providing access to both educational and health care opportunities. While the pipeline program itself was well-designed with input and sign-off from key stakeholders, the medical school’s admission committee included members who had difficulty considering candidates from the program. These members deferred, instead, to metrics—no community college coursework, graduation from an elite undergraduate school—that would deny most of the applicants from the pipeline program, even though qualified, the chance of attending Medical School B. In this case, coordination with the admissions office is central to achieving the pipeline program’s purpose. Such coordination includes aligning admissions criteria with institutional mission, clarifying leadership commitment to the pipeline program, examining admissions policies, training admissions committee members, and, perhaps, even reconsidering the committee’s composition.



faculty), and students (in his article, patients). It is not only the learners’ or the institutional actors’ behaviors that are assessed; rather, responsibility for the process and outcomes is shared across both groups of stakeholders.

Specifically, this framework examines the structures or institutional elements that need to be in place, the processes or actions needed to achieve intended outcomes, and the actual outcomes themselves.

• Structures are the raw materials you need to achieve desired outcomes. They include such elements as the institution’s organizational infra-structure (e.g., mission, committed leadership, areas of responsibilities and program offices, lines of reporting, accountabilities), policies that guide processes, resources (e.g., funding, time, personnel/staffing), and facilities (e.g., space, equipment, location of offices and staff across interlinking programs).

• Processes are the actions taken to implement enrollment management policies and related programs. This means looking at how programs and activities are performed from alignment with institutional mission, to collaborations and coordi-nation across programs, to communicating with key stakeholders and publicizing the programs.

• Outcomes are the actual short-term, mid-term, and/or long-term effects and impact of your school’s enrollment management policies and processes.

Structure, process, and outcomes are interactive, not stand-alone elements for evaluating the effectiveness of your school’s enrollment management strategy. Structures establish the context within which the medical school performs its processes. Structures and processes combined influence the ensuing outcomes.

Applying the Structure-Process-Outcomes Framework

The structure-process-outcomes framework can be applied by your medical school to evaluate the school’s overall enrollment management strategy. Table 2.3 has a series of questions to begin that assess-ment. In reviewing Table 2.3, consider modifying or adding questions to accommodate your school’s mission-related goals and interests.

To assist you and your school with implementing the structure-process-outcome framework, we have prepared an evaluation tool, “Important Considerations for Evaluating Enrollment Management Functions.” It presents examples of enrollment management functions typical at most medical schools: outreach and recruitment, admis-sions, financial aid and debt management, diversity programming and counseling, academic advising and support, and career advising. For each function, the tool poses a separate set of sample questions for structure, process, and outcomes. It also identifies potential data sources and methods for conducting the evaluation. You will find the tool in Appendix B; an excerpt that examines outreach and recruitment appears in Table 2.4.

While not an exhaustive evaluation “manual,” this evaluation tool does serve as a template. To adapt the tool to correspond directly to your school’s range of enrollment management programs and activities, institutional mission, and the diversity that you want to develop though holistic admissions, consider the following steps:

• identify the programs and activities at your school that fall under an enrollment management rubric

• modify, add, and/or omit questions to reflect your school’s needs

• determine the data needed to answer the questions

• choose the method(s) best suited to collect and analyze the data



Table 2.3: Structure-Process-Outcomes: Evaluating the Effectiveness of the Medical School’s Overall Enrollment Management Strategy on Achieving Institutional Mission-driven Diversity Goals

Focus Questions

Central Focus • What role do mission-driven, diversity-related institutional goals play in guiding the medical school’s enrollment management strategy?

Structure • What policies, institutional organization and systems, and resources does the medical school provide in support of its enrollment management efforts?

• Are the school’s diversity policies explicitly aligned with institutional mission and goals? Are they transparent and shared with a wide range of stakeholders?

• Do these structures align with the institutional mission and goals, including the educational and professional benefits associated with educating a diverse student body?

• Does the institution’s organizational infrastructure, including reporting structure and assignment of accountability and responsibility, mirror the goals and values leadership publicly promotes?

• Are financial and personnel resources sufficient to achieve intended outcomes?

• Do facilities, including location of staff and offices, advance collaboration across complementary programs?

• Which data do the dean and other medical school leaders report to stakeholders and the public? For example, are reported student data primarily about academic metrics, or are other dimensions related to the school’s diversity interests also highlighted, such as service orientation and career choice?

Process • In what activities does the school engage to implement and manage its enrollment management strategy?

• Are policies and related programs congruent with institutional mission and goals?

• Are administrators, staff, and faculty members working across program areas to ensure that program implementation is coordinated and aligned?

• Are any programs duplicating activities, or are there efforts that could be streamlined?

• Are policies and processes transparent, clearly articulated, and easily accessible to individuals within and external to the medical school (e.g., faculty, administrative staff, prospective applicants, students, the public)?

• Are data collection and analyses conducted on an on-going basis to make decisions about structures or processes that need to be developed, refined, or removed to improve outcomes?

Outcome • What impact has each enrollment management activity/program made alone or in combination with other activities/programs?

• Do program outcomes align with/contribute to achieving institutional mission and goals?

• Do outcomes of discrete program and function areas follow logically from and build on each other such that they contribute to achieving institutional mission?

• Are key institutional stakeholders knowledgeable and conversant about enrollment management strategies and their respective outcomes?

• Are there any unintended consequences that have arisen from any policies or programs, individual or in combination?



Table 2.4: Excerpt from “Important Considerations for Evaluating Enrollment Management Functions” (see Appendix B for full tool)Outreach and Recruitment

Structure(i.e., organizational infrastructure, policies, resources, facilities)

• Do institutional policies that guide outreach and recruitment support institutional mission? For schools that consider race and ethnicity, do the policies address relevant legal issues?

• What explicit goals, if any, has the school made for reaching out to and recruiting a diverse student body?

• Who is accountable for outreach and recruitment efforts?

• How much time and how many resources are devoted to outreach and recruitment programs?

• What percentage, if any, of those resources is dedicated to diversity-related efforts?

• Are outreach and recruitment policies aligned with admissions criteria?

• Is participation in outreach and recruitment efforts sufficiently important that it is considered in performance evaluation, promotion, and tenure decisions?

Process(i.e., actions)

• Is the purpose of each outreach and recruitment activity clearly articulated and commonly understood?

• Are there specific activities to engage students and communities that are targeted by diversity initiatives?

• Are outreach and recruitment activities coordinated with each other and across people/units with responsibility in this area, such as the admissions and diversity affairs offices?

• Are there opportunities for collaboration to enhance and streamline outreach and recruitment activities?

Outcome(i.e., short-term, mid-term, and/or long-term)

• Which outreach and recruitment activities are most effectively contributing to an applicant pool with the diversity of experience, attributes, and academic metrics the school seeks?

• Are applicants who are recruited from targeted populations likely to be accepted?

• To what extent have the offices and staff accountable for outreach and recruitment established relationships with colleges, communities, and organizations serving as “pipelines” for targeted groups?

• What is the level of knowledge about your medical school among potential applicants from targeted populations?

• What is the level of knowledge among internal and external stakeholders about outreach and recruitment activities and the populations targeted?

• Do internal and external stakeholders recognize the value of outreach and recruitment efforts to enhance diversity with the student body and physician workforce?

Potential Data Sources and Methods

• Institutional budget audits to determine levels and kinds of resources dedicated to outreach and recruitment, both generally and diversity-related

• Inventory charting strategy and design of annual outreach and recruitment activities to assess separate and collective efforts

• Surveys and tracking of participants who attended outreach and recruitment activities to assess satisfaction and identify areas for improvement

• Analysis of applicant yields, both generally and specific to outreach and recruitment activities, to document and analyze effectiveness



• In addition, it might be useful to use this enroll-ment management evaluation tool in conjunction with resources found in other Roadmaps, including the following:

• “Medical School Diversity Self-Assessment,”1 which spans medical school diversity-related policies and processes

• “Holistic Review Admissions Checklist,”2 which focuses primarily on admissions policies and processes

Summary

Achieving the learning and medical practice outcomes associated with student diversity does not just happen by virtue of admitting a diverse student body. It requires intentionality, leadership, and commit-ment on the part of the institution in the formal and informal learning spaces, as well as in the formal and informal enrollment, support, and enrichment services that enable all students to develop, learn, and thrive. Chapter 1 pointed out the importance of collecting and analyzing data over time to evaluate and support the impact of holistic admissions and other diversity-related initiatives. This chapter proposed a framework for evaluating how the medical school’s enrollment management services are working to support and attain the benefits of increased diversity among medical students. In the next two chapters, Milem, et al., and Saha present evaluation approaches that will help your medical school assess whether the institution is fully leveraging student body diversity to achieve intended educational and professional outcomes.



Chapter 3The Matriculated Student: Assessing the Impact of Holistic Review

Authors: Jeffrey F. Milem, Ph.D., University of Arizona Celia L. O’Brien, Ph.D., University of Arizona College of Medicine W. Patrick Bryan, M.A., University of Arizona

Chapter 1 discussed the importance of gathering data at different points of time (admissions, at entry, during medical school, after medical school). This chapter describes a means for gathering and classifying these data so that medical educators can determine how successful they have been in imple-menting holistic review, and evaluate the impact that holistic review has in advancing the mission of the medical school. These data can also be used to better understand the experiences medical students have in medical school, how students change over time, and what role these experiences have in influencing these changes. Moreover, these data are also important in helping to ensure that medical educators are doing all that they can to facilitate the success of all students in the medical school.

When medical educators advance the “diversity rationale,” they assert that diverse learning environ-ments provide opportunities for teaching and learning that homogeneous learning environments do not. A key component of the diversity rationale is the idea that students’ engagement with diversity while in medical school can positively influence a range of important learning, developmental, and professional outcomes. As discussed in Chapter 1, medical schools should gather relevant data from their students at key time points during the time that they are enrolled in medical school.

These data serve two essential functions in evalua-tion efforts. First, they provide information about the types of activities, experiences, etc. in which students are engaged while in medical school. This information is helpful in determining whether or not the diversity that the medical school creates in its admitted class produces the type of learning environment described in its institutional mission. For example, if a medical school argues that having a more racially and

ethnically diverse class provides greater opportunities for students to be exposed to diverse people, informa-tion, and ideas, it is important that medical educators are able to determine whether or not this happens. Two examples of data that help to determine if this happens are measures of students’ interactions with people who are different from them in formal and informal educational settings, and the extent to which students are exposed to diverse information and ideas via curricular and co-curricular experiences while in medical school.

Second, these data serve an important role in longi-tudinal analyses of the impact of diversity in the medical school on students’ learning, developmental, and professional outcomes. In these analyses, medical educators can determine whether certain activi-ties and experiences actually produce the positive changes in outcomes the institution asserts through its mission and goals. Using the example cited above regarding a medical school’s commitment to having a racially and ethnically diverse class, these data would be used to determine the extent to which student engagement with diverse others and diverse informa-tion and ideas in medical school positively influenced important learning and professional outcomes such as empathy, humanism, cultural competence, specialty choice, and/ or interest in serving medically under-served populations.

A Model for Determining the Impact of Holistic Review

In order to determine whether or not holistic review has had its intended impact on key outcomes defined by the medical school in its mission, data must be gathered at different points in time. Only then is there the type of information available that is necessary



to do what evaluators describe as an impact assess-ment.9 Higher education scholar Alexander Astin10 provides a helpful model for assessment and evalua-tion of educational programs that he describes as the I-E-O model, or input-environment-output model. While Astin developed the I-E-O model to study the impact of college on undergraduate students, it is easily adaptable and well-suited for efforts to evaluate holistic review in medical school because of its ability to estimate the impact of holistic review on learning, developmental, and professional outcomes of medical students (see Figure 1.2).

In the I-E-O model, outputs or outcomes (described later in this chapter) refer to the learning, develop-mental, and professional talents that schools seek to develop in medical students. Astin asserts that in order to determine the impact of a particular program or educational activity, schools must account for more than simply the outputs or outcomes of an educational program. Instead, outputs must always be considered in the context of the educational inputs of the institution. In the instance of holistic review, these student inputs or entry characteristics comprise the experiences and attributes students have at the time they enter medical school, before they ever set foot on our campuses. These entry characteristics must be considered in our evaluation activities because of the important role that they play in influencing student outcomes. The third type of measure that must be included in evaluation efforts assesses the medical school environment. These measures include assessments of the medical school environment and students’ experiences while in medical school, including courses, programs, contact with faculty, engagement with peers, volunteer activities, and curricular and co-curricular activities. Environmental measures are most important to medical educators because the medical school environment is the area over which schools have the most control. In essence, the environment represents all of the things medical schools do that are designed to prepare students to be successful as physicians. Astin’s I-E-O model can help educators structure learning environments in ways that maximize opportunities to develop important learning outcomes.

This model of assessment and evaluation allows you to take into account, or control for, the differ-ences in student entry characteristics, or inputs, on student outcomes so you can develop a more accurate estimate of the relationship between environmental experiences on these outcomes. In addition, it can account for the different types of environmental experiences in which different students will engage while they are in medical school, as well as the different effects that participation in these experiences may have on different students. We have adapted an example Astin provided about health care to explain the importance of applying the I-E-O approach to assessment and evaluation as follows:

Perhaps an even better analogy can be found in the field of health care. Efforts to improve the quality of care in hospitals rely on rigorous evaluations of which treatments and work environments lead to the best clinical outcomes. If we were trying to enhance our understanding of how best to treat patients, imagine how difficult it would be if all we did was collect output information (length of hospital stay, mortality, condition at discharge), without taking into account their condition on admission to the hospital. Knowing patients’ condition on admission allows us to interpret their outcomes at discharge. But knowing the input and output data alone is not enough. In order to improve the quality of care, we must also know how patients were managed while in the hospital: which treatments they received, when they received them, and from which types of health care providers. Having such “environ-mental” data about the care patients received while in the hospital is critical to refining the environment to deliver the best care possible. The same is true for medical education. We must understand where students started, as well as the environment and experiences they were offered, exposed to, and engaged in, to be able to interpret their outcomes upon graduation.11



The best way to understand the I-E-O model, and how it applies to evaluation of holistic review processes, is to describe its component parts. The remaining portion of this chapter further defines the “input,” “environment,” and “output” measures, and it provides suggestions and examples for how to use the model to plan and implement a comprehensive evalu-ation of holistic review.

Student Input and Entry Characteristic Measures

As described earlier, student inputs or entry character-istics comprise characteristics, experiences, attitudes, values, and knowledge that students have at the time they enter medical school.10 These measures are fairly easy to assess and should be collected during orienta-tion or prior to entry into the program. The most common method for gathering these data is to have students complete a survey instrument, although some of these variables are contained in AMCAS® and through other data gathered during the recruit-ment and admissions process and are accessible to medical education evaluators.

The following text offers examples of the types of student inputs or entry characteristics that should be gathered from students prior to beginning medical school.

The evaluation of demographic inputsDemographic inputs are the most frequently consid-ered when evaluating the diversity of any group. Although measures of racial and ethnic background often take the spotlight, there are many other demographic characteristics that should be consid-ered when evaluating the extent to which diversity is present within the medical school environment. A critical consideration should be for the inclusion of various measures related to each institution’s priorities. The richness of the data collected lies in the ability to disaggregate the data in meaningful ways, and the school’s mission should drive these considerations.

Depending on the mission and goals of the medical school, institutions may have additional demographic

attributes they wish to include (rural vs. urban origin, citizenship, tribal affiliations, support systems in place, etc.). The categories provided in Figure 3.1 are meant to serve as a guide and should be amended to fit the different missions of different institutions.

The evaluation of traditional admissions metricsGiven that they are key elements of the student selection process, traditional admissions criteria also should be included as entry characteristics. In addition to the examples provided below, each insti-tution may have additional measures it wishes to include in its evaluation.

Suggested Measures for Evaluating Traditional Admissions Metrics

• MCAT® examination Scores

• Undergraduate/graduate cumulative GPA

• Science GPA

• Major(s)

• Degrees earned prior to medical enrollment

• Interview scores

The evaluation of previous environmentsIn addition to the individual student’s demographic characteristics, his or her experiences in different environments, and with others of differing background characteristics, are another set of important measures to include. For these, the context of the experience is the critical element of differentia-tion. (see Figure 3.2)

The evaluation of attitudes, values, and knowledgeOther entry measures to assess are the student’s attitudes, values, and knowledge. These measures will span a number of topics, which should include attitudes surrounding diversity and the practice of medicine, as well as perceptions of different social issues. These measures provide helpful insights into the perceptions and perspectives each student has as he or she enters medical school. In measuring



Figure 3.2: Suggested Questions to Guide the Evaluation of the Context of Previous Environments

• What was the racial/ethnic composition of the student’s neighborhood, school, and dominant peer group prior to his or her enrollment in college?

• How much prior experience in medicine does the student have? Was that experience clinical, nonclinical, research, volunteer, paid, etc.?

• Did the student participate in any medical outreach opportunities such as internships, summer programs, or seminars? Did the student attend a post-baccalaureate program?

• Did the student have family or work responsibilities while in college (e.g., children of their own, expected to contribute to family income, siblings and/or elderly family members to care for, etc.)?

• Did the student have other important undergraduate experiences (e.g., work as a resident assistant, leader in a club/organization/fraternity/sorority, sustained community service commitments, etc.)?

Figure 3.1: Suggested Measures for Evaluating Demographic Inputs

• Race/Ethnicity: White, Black (African American), Asian/Pacific Islander, Native American (American Indian)/Alaska Native, Hispanic/Latino, Other, etc. All students should have the opportunity to self-define this category and to indicate multiple identities (e.g., Latino and Black, etc.)

• Gender: Male, female, and transgender

• Parental Income: This will be broken into income ranges differentiated according to average incomes appropriate for the type of institution (a school with national reach may have different averages than a school with a mission to serve a particular state). In general, this should include between eight and 10 categories for income

• Parental Education: Elementary, middle school, some high school, high school graduate, some college, associate degree, bachelor’s degree, master’s degree, doctoral degree, and professional degree

• Parental Occupation: Information about parental occupation can be very useful in determining the socioeconomic status of students

• Socioeconomic Status (SES): Determined as an index, typically using some combination of parental education, parental occupation, and/or parental income

• Personal Education History: Student has completed a bachelor’s degree(s), post-baccalaureate program, master’s degree(s), doctoral degree(s), and professional degree(s)

• Sexual Orientation: Heterosexual, gay/lesbian, bisexual, questioning/unsure

• Religion: Agnostic, atheist, Roman Catholic, Greek Orthodox, Protestant, Islam/Muslim, Jewish, Mormon, Hindu, Buddhist, non-denominational Christian, Sikh, and other

• Ability: Deaf/hard of hearing, blind/severe visual impairment, learning disabled, medical, physical disability, Attention Deficit Disorder, Attention Deficit/Hyperactivity Disorder, and psychological disability

• Political Orientation: Very liberal, somewhat liberal, moderate, somewhat conservative, very conservative



attitudes, values, and knowledge, it is most common to use a “Likert item” (see Figures 3.3 and 3.4).

When viewed in the aggregate, these student inputs and entry measures data represent the qualities and characteristics that the admissions process yields for the class. As such, medical educators are able to determine how effective they have been in creating a class that represents the goals and mission of the school. In disaggregated form, this information provides baseline data about students that can be used

to evaluate how students’ characteristics change over time as a result of their medical school experiences.

Environmental Measures

Environmental measures include assessments of the medical school environment and students’ experi-ences while in medical school. These include courses, programs, contact with faculty, engagement with peers, volunteer activities, and other curricular and co-curricular activities.10 The environment serves as the “treatment” or “intervention,” which along with

Figure 3.3: Likert Item

The Likert item is a psychometric response scale used to measure a respondent’s level of agreement or disagreement with a statement. The scale asks the respondent to indicate their level of agreement/disagreement using one of usually five ordinal options for agreement.*

Example: It is fair to give preference in medical school admissions to children of alumni.

Strongly agree Somewhat agree

Neutral Somewhat disagree

Strongly Disagree

¡ ¡ ¡ ¡ ¡

* Likert R. A technique for the measurement of attitudes. New York: 1932.

Figure 3.4: Suggested Prompts to Guide the Evaluation of the Context of Attitudes, Values, and Knowledge

• Race, Class, Ethnicity, and Gender“It makes me uncomfortable to talk about racial issues.”“Producing more physicians of color should be a top priority for this college.”“A person’s racial background in this society does not interfere with achieving everything he or she wants to achieve.”“I spend little time thinking about the role of race in this country.”

• The Practice of Medicine“A patient who feels understood can experience a sense of validation that is therapeutic in its own right.”“Patients’ illness can be cured only by medical treatment; physicians’ affectional ties with their patients do not have a significant place in this endeavor.”“Reading nonmedical literature and enjoying the arts can enhance physicians’ abilities to render better care.”

• Politics and Society“Democracy thrives on different views.”“People should respect authority.”“There is a clear line between what is right and what is wrong.”



the entry characteristics, affect student’s educational outcomes.12

In evaluating the impact of the medical school environment on student outcomes, it is helpful to consider the different ways to categorize measures of the environment. In the context of medical education, environmental measures can include the physical surroundings and layout of facilities and personnel; the organization of the curriculum, including co- and extra-curricular activities; interactions and engage-ment with faculty, staff, peers, and patients; racial climate; and even activities that occur outside the college. The examples below are ways a school can operationalize its environment.

The evaluation of curricular and pedagogical influencesThese measures include different aspects of the curriculum and the pedagogy used in teaching the content.

Tools for Assessment:• Content analysis of syllabi from all courses

(e.g., Does the curriculum develop cultural competency?).

• Content of optional curricular elements (electives)

• Extent to which active inquiry, collaborative learning, and cooperative learning are used in classes and clinical rotations

• Grades on key tests and scores on clinical observa-tions (e.g., Who scores well? Who does not?).

The evaluation of interactions and engagementInteraction with faculty members and residents inside and outside of the classroom are environmental measures that can be referred to as student engage-ment. Clinical experiences are also aspects of engage-ment, as is formal and informal mentorship. While these experiences are not part of the “formal” curric-ulum, opportunities that students have to engage faculty, residents, and the clinical setting do not happen randomly. They are key parts of the educa-tional environment.

Interactions with peers can have a critical role in shaping student outcomes. When students feel that they belong or “fit” in the institution, they are more likely to engage the institution in a way that will have a positive effect on educational outcomes. Peer interactions should be assessed from both academic and social perspectives, and medical schools should evaluate the extent to which students engage other students across communities of difference. A key aspect of the diversity rationale is that students in more heterogeneous institutions are more likely to engage diverse others and diverse information and ideas. Moreover, these interactions can have a positive effect on other important learning, developmental, and professional outcomes.

Tools for Assessment:• A variety of survey items that assess the extent to

which students engage with faculty, residents, staff, and peers, as well as the quality of these interactions

• Focus groups that evaluate the nature of inter-actions and relationships between and across communities of difference in the medical school (e.g., faculty and students, Asian and White, lower and middle class, etc.) What is the nature of these interactions (e.g., academic, social)? Where do they occur (e.g., in professional/clinical settings, in class, in study groups, outside of class in social situations)?

• Observations of instructional activities to assess the extent to which students have the opportunity to engage diverse others and diverse content

The evaluation of climate for diversityMeasures of the psychological/perceptual dimension of climate offer information about minority and majority perspectives on relationships and oppor-tunities between and among students from different groups. One aspect of these measures should evaluate the sense of belonging and engagement of all students—with special emphasis on students who are from groups underrepresented in medicine (URiM) or are from historically marginalized groups, including lesbian, gay, bisexual, transgender and questioning (LGBTQ) or disabled students. Diversity



climate assessments should include everything from what the college says it values in its mission statement to understanding events in which overt and covert racism or acts of intolerance occur, along with assessments of the impact of institutional responses that result from these acts. The degree to which all students feel they belong on campus is another important consideration, as is the degree of inter-action across communities of difference for both students and faculty.

Tools for Assessment:• Survey instruments that assess the degree to which

all students—with particular attention paid to URiM and LGBTQ students and women—feel supported in the medical school setting, the extent to which they have access to key learning opportu-nities, and their ability to develop important and meaningful connections with faculty

• Focus groups that examine student and faculty perspectives on diversity and whether the insti-tution is supportive of diverse perspectives and students

• Content analysis of college policies and formal communications to students and faculty to determine the extent to which diversity has become institutionalized at the medical school

• Content analysis of the curriculum to determine if courses and clinical experiences promote cultural competency in students

The evaluation of experiences outside the medical schoolA final consideration for environmental assess-ment activities involves measures of involvement in programs and activities that occur away from the physical campus. These may or may not be sponsored by the medical school. Community-based programs, summer internships, and activities abroad are examples of activities that can influence important learning, developmental, and professional outcomes. Involvement in the community via civic, community, and faith-based groups can also enhance students’ educational outcomes and should not be ignored.

Tools for Assessment:• Survey instruments that assess student involvement

in activities outside of the school (e.g., hours spent in service, internships, professional experience, or other programs; involvement in activities abroad or in the community; faith-based health initiatives)

• Focus groups that assess student and faculty perceptions of student involvement in educational opportunities beyond the campus. Efforts should include the extent to which these opportunities are available to all students and not reserved for just a select few

Outcome Measures

Medical evaluators are familiar with reporting outcomes, as many of these measurements are required to meet standards of national licensing and credentialing boards. While less than four percent of matriculating students fail to graduate from medical school, attrition rates are higher among students from racial and ethnic groups historically underrepresented in medicine (e.g., American Indian/Alaska Native, Black/African American, and Hispanic/Latino),13 and among students from low socioeconomic status backgrounds (SES).14 Moreover, students withdraw from medical school for academic and nonacademic reasons. Medical educators must be concerned with how well they are enhancing and supporting the success of all students, particularly those who are at risk for attrition or who may be marginalized in traditional medical school curricula and cultures.

For many institutions, student success is measured through retention rates in addition to various indica-tors of academic, personal, and professional achieve-ment. However, these indicators may differ widely among medical schools, and they should be deter-mined by local data whenever possible. Once again, outcomes should be interpreted within the context of input and environmental factors. This section provides support in defining and evaluating student success in medical school using three overarching outcome categories: learning outcomes, develop-mental outcomes, and professional outcomes.



Connecting outcomes with institutional mission and goalsThe first step in measuring student success is to clearly define success. This definition may vary widely among medical schools, but it should be closely aligned with institutional mission as discussed throughout this Roadmap. What are the priori-ties of the institution? What are the short-term and long-term institutional goals? What is the purpose of diversity-related policies and programs, and what is their desired effect on students?

Once success is defined, each medical school must identify specific and measurable outcomes. Often, these outcomes are intermediary and relate to experi-ences and performance during medical school. However, students also should be followed and tracked well into their residencies and professional careers to determine how entering characteristics and environmental factors relate to future performance.

The evaluation of learning outcomesA key concern of medical schools is how well they produce competent physicians who possess a body of knowledge essential to a successful career. To this end, a medical school evaluation team must be concerned with how students perform academically, and how this relates to background characteristics and educa-tional environment (see Figure 3.5). Multiple studies have shown that it is possible to identify variables associated with academic outcomes,15,16 suggesting that medical schools may be able to partially predict student success on a local institutional level. A well-executed assessment plan can help educators identify

students who are at risk for poor academic perfor-mance. Since approximately half of the students who leave medical school before degree completion do so for academic reasons,13 an early intervention support system may help decrease an institution’s attrition rate.

Simple reports that focus on average test scores or retention rates do not illustrate the entire picture of student success. In order to truly understand how these numbers fulfill mission-based outcomes, evaluators must examine the relationships between academic performance and input characteristics, as well as the interaction between an individual student and the educational environment. This is an essential step in fully understanding the implications of holistic admissions policy and practices, as well as environmental effects on student performance. As with everything, the results of these evaluations must be considered within the context of the goals, mission, and circumstances of a particular medical school. For example, any changes in the curriculum (however slight) must be taken into account when interpreting and disseminating findings related to academic outcomes.

Tools for assessment:• Institutional records: Academic progress data

should be obtained, including student GPA, USMLE® scores, time to completion, and gradu-ation rates. In addition, it is imperative that these data are disaggregated by background characteris-tics to note any significant trends among students from diverse backgrounds.

Figure 3.5: Suggested Questions to Guide the Evaluation of Learning Outcomes:

• Do all students—especially those from diverse backgrounds—have the support and resources they need to succeed academically?

• Are all students succeeding academically (passing classroom exams, OSCEs and USMLE®)?

• How are students performing on regular evaluations of clinical performance?

• Are all students progressing to graduation in a timely manner?

• Which students are at risk for academic dismissal or withdrawal?

• Which students are at risk of suboptimal performance on USMLE® Steps 1 and 2?



• Data from the first year: Evaluators also should be familiar with first-year retention rates, and how success is typically measured during the first year as a student is introduced to the medical school curriculum.

• Surveys: Student surveys should include items intended to measure student perceptions of the environment as it relates to academic success. Do they feel supported by faculty and staff? Is the format of the curriculum compatible with their learning style? Are they being taught information they believe they should know? Have they struggled academically, and if so, to what do they attribute these struggles? Are they engaged in activities that support learning about diverse issues, or which relate to the diversity mission of the institution? As always, these items will be closely aligned with the institutional mission in order to measure desired outcomes. In analyzing these data, it is important to disaggregate the analyses by dimensions of difference that are important to the medical school (e.g., race/ethnicity, gender, SES).

• Interviews/focus groups: Qualitative data can be collected that provide insight into how students perceive the curriculum, as well as their relation-ship with faculty and mentors. Focus groups made up of students from similar backgrounds can highlight certain issues faced by particular populations.

The evaluation of developmental outcomesAs discussed earlier, medical schools must be concerned with providing an environment that is conducive to student learning and success. This environment can influence the development of ethical attitudes and values or can lead to unhealthy outcomes such as psychological distress, which is negatively related to retention and professional conduct.17,18 While formal parts of the educational experience can shape this development, perhaps the stronger influence on these outcomes is the “hidden curriculum,” in which students learn values, beliefs, and behaviors through informal interactions in addition to organizational and cultural influences.19 As medical students are socialized into the profes-sion, it is important to track developmental outcomes to determine if the institution provides an environ-ment that supports the healthy development of future physicians (see Figure 3.6).

As medical school evaluators assess how the learning environment affects developmental outcomes, they must take into account the entering characteristics of the student. Therefore, assessment techniques must include variables that measure the values and attitudes of individuals at the time of admission to note how these may change as students progress through their medical education. Pre-validated measurement tools exist that may help to determine whether a student is suffering from depression20 or burnout.21 Still, understanding how students are developing during medical school can be a complex process. Evaluators must be sure to take care when interpreting the findings and not attribute causation to what may be a simple association between two variables.

Figure 3.6: Suggested Questions to Guide the Evaluation of Developmental Outcomes

• What are the values and attitudes of medical students during and after medical school?

• How are medical students developing ethical attitudes and behaviors?

• How many students show signs of personal and professional burnout or depression? How is this associated with learning and professional outcomes?

• Are there students who feel marginalized or isolated during medical school? If so, who are they and what effect does this sense of isolation or marginalization have on other important medical education outcomes?



Tools for assessment:• Institutional records: Academic scores, including

exam grades and peer assessment ratings, can be of interest to evaluators who wish to examine the effects of certain curricular experiences on devel-opmental outcomes. However, qualitative analyses of institutional records are also possible. Both faculty and peer evaluations of a student within a small group learning environment can be studied to note any relationships with values, attitudes, and ethical viewpoints. In addition, multiple studies have touted the use of reflective writing assignments to qualitatively assess how values and attitudes are shaped during clerkships.22,23

• Surveys: Items on surveys should focus on the intensity, frequency, and quality of interaction with certain components of the curriculum (e.g., electives taken or interaction with formal mentors). In addition, surveys can also measure satisfaction levels of educational experience and perceptions of the medical school climate. Finally, survey items have already been developed to measure burnout,21 depression,22 and quality of life24 variables.

• Interviews/focus groups: Qualitative data can help tease out the nuances associated with percep-tions of the environment. In particular, questions should focus on how students perceive the educa-tional climate, especially as it supports (or does not support) diversity and acknowledges the experi-ences of individuals from diverse backgrounds. Whenever possible, focus groups should include students, faculty, and staff from similar

backgrounds to allow for a safe and comfortable space for sharing.

The Evaluation of Professional OutcomesProfessional outcomes are often aligned quite closely with institutional mission. Many medical schools explicitly state that they wish to produce primary care physicians or those who will work with underserved populations. Other institutions may wish to measure how well they prepare future physician scientists. Yet another aspect of these outcomes goes beyond specialty or career choice to include professional conduct and ethical behavior. Medical schools should have a vested interest in examining the competency and professionalism of their alumni to determine how well their admission criteria relate to future perfor-mance in the physician workforce (see Figure 3.7).

Medical schools should work closely with their alumni offices to create and maintain a database of former students that will permit longitudinal analyses of professional behavior. In particular, evalu-ators should be concerned with how professional behaviors change after the undergraduate experi-ence, particularly in residency. Efforts also should be made to track any changes in specialty choice or residency placement. Finally, case studies of alumni who have had difficulty with professionalism in their practice may identify entering characteristics and environmental factors that may predict or enable such behavior in a future student.

Figure 3.7: Suggested Questions to Guide the Evaluation of Professional Outcomes

• Are students succeeding professionally and making progress toward successful careers that are consistent with mission-based goals?

• Are students succeeding in choosing a future career path that aligns with their personal values and preferences? How does this differ by race/ethnicity group?

• Are graduating students demonstrating competency working with diverse populations? Are they developing into culturally competent physicians?

• Are they adhering to professional standards and guidelines as students, residents, and physicians?

• Are students and alumni engaged in altruistic activities in which they give back to the community?



Tools for assessment:• Institutional records: Determine USMLE® pass

rates, OSCE scores, residency match performance, and other indicators to determine what predicts success and if these outcomes are consistent with mission-based goals.

• Surveys: Matching pre-matriculation surveys with follow-up data can determine how values and past specialty choices align with current goals. In addition, these longitudinal studies can examine changes in these attitudes over time and the factors that may influence these changes. Follow-up surveys should include measures of humanism, cognitive complexity/critical thinking, engagement with diversity, dogmatism/flexibility, and profes-sionalism to determine if medical school experi-ences enhanced or detracted from these attributes.

• Interviews/focus groups: These tools are especially useful in assessing how students or alumni are engaged in various communities.

• External data: Use the alumni database; keep track of any awards, board appointments, research efforts, community engagement, disciplinary actions, etc.

Summary

The benefits associated with student body diversity are not achieved solely through selecting the “right” students. They require intentional, purposeful effort on the part of the medical school to create and sustain an environment in which all students can be successful. The I-E-O model presented in this chapter provides a useful framework for evaluating the role the medical school environment plays in achieving desired learning, developmental, and professional outcomes. Professional and workforce outcomes are explored more fully in Chapter 4, which presents a basic framework for assessing these long-term outcomes.



Chapter 4 Evaluating Workforce Outcomes

Author: Somnath Saha, M.D., M.P.H. Oregon Health & Science University

The previous chapters have laid out evaluation funda-mentals and provided basic frameworks for evalu-ating the impact of holistic admissions on medical school enrollment management processes and the relationship between entering student characteris-tics and environmental factors. The learning and developmental outcomes sought through admitting and educating a diverse student body contribute to longer-term workforce and professional outcomes. Workforce outcomes are directly affected by many factors over which medical schools have little or no control, particularly environmental elements in residency and practice. Nevertheless, evaluating workforce outcomes provides critical feedback to inform whether medical school structures, processes, and environment related to increasing and supporting diversity are achieving the goal of producing the type of physicians the school hopes to develop.

Evaluating the impact of holistic review on workforce outcomes requires that a medical school view its role in producing future physicians as a purposeful one. That is, a medical school committed to this goal is not simply striving to produce excellent physicians, but rather excellent physicians with specific attitudes, competencies, and career goals and practices that reflect the school’s mission. As with the evalua-tive components discussed in Chapters 1, 2, and 3, workforce outcomes should therefore be tightly linked to a school’s mission.

Evaluation of workforce outcomes generally occurs after students have completed medical school. This realm of evaluation therefore focuses less on the structures and processes a medical school uses to successfully implement holistic review, as described in Chapter 2, and more on the outcomes as a result of that implementation. Similarly, workforce outcomes are a product of the input and environmental factors explored in Chapter 3. Evaluating workforce

outcomes helps us determine whether what we do in admissions and in medical education produces the mission-driven outcomes that schools seek in their graduates.

This chapter describes outcomes in the domains of attitudes, competencies, and career goals and practice, and provides sample questions to guide evaluation. We, then, propose sources and methods for collecting data that may be used to evaluate these outcomes both at the time of graduation and after students enter the physician workforce. It is important that workforce data be gathered in a manner that allows medical educators and administrators to link them to data collected earlier in students’ careers in order to create the longitudinal data necessary for the I-E-O model described in Chapter 3 to work.

Measuring Domain Outcomes

As we emphasize throughout this publication, each medical school should identify key workforce and professional outcomes grounded in its institutional mission and goals. Although we have identified several workforce outcomes in three domains —attitudes, competencies, and career goals and practice—the questions and measures most appro-priate and applicable for your school’s context may differ from the samples included in this chapter. We hope these examples still will provide a solid founda-tion and useful guidance as you evaluate long-term outcomes associated with how, where, and what your school’s graduates practice.

AttitudesAttitudinal outcomes should reflect the professional attitudes that the school strives to instill in producing future physicians. The mission statements of many schools include descriptors such as “humanistic” and



“compassionate.” Others include a commitment to lifelong learning, leadership, advocacy, ethical princi-ples, service, and patient-centered care. Evaluating these attitudes at the time of graduation will tell a school whether it has been successful in recruiting students with strong and sustained commitment to the school’s values, sustaining these values through their medical school experiences, and effectively tapping into the benefits of student body diversity to nurture desired attitudes among students. However, if these attitudes wane over time and are not sustained when graduates become practicing physicians, the school will not have succeeded in its ultimate goal of contributing to a physician workforce that reflects its values and mission.

At GraduationPotential measures: Measures of attitudes should reflect the values implied or explicitly stated in a school’s mission statement. Examples of questions that might guide the evaluation of attitudinal outcomes include the following:

• Do students endorse the notion that all patients deserve compassionate care?

• Do students respond to challenging clinical situa-tions in an ethical manner?

• Do students have respectful attitudes toward colleagues and health care team members?

• Do students prioritize patient autonomy?

• Do students view access to medical care as a right or a privilege?

• Do students espouse a sense of responsibility for others?

After GraduationPotential measures: Once students have graduated, it becomes more difficult to directly measure their attitudes and values. However, there are surrogate measures that can provide evidence that graduates espouse attitudes and exhibit behaviors in line with a school’s mission. Guiding questions in this realm include:

• Are residency programs providing positive feedback about graduates’ attitudes and work ethic?

• How many graduates receive awards for their commitment to patients and professionalism during residency?

• How many graduates receive recognition for their commitment to service as practicing physicians?

• Are graduates engaging in advocacy to promote values in line with the school’s mission?

CompetenciesNearly all medical schools evaluate core competen-cies among their graduating students. The diversity of the student body may or may not contribute substantively to the development of many of these competencies. When evaluating general competen-cies as outcomes of having a diverse student body, a school should select those for which diversity is a key contributor. For example, many schools strive to instill competency in caring for culturally diverse patient populations, an outcome for which having a culturally diverse student body might be consid-ered a key contributor. There also may be specific competencies (e.g., research skills, leadership, second language proficiency) that schools prioritized among applicants for admission, because those students were more likely than others to go on to fulfill the school’s mission-based goals of producing, correspondingly, future researchers, leaders, and providers of care for non-English speaking patients. Evaluating whether those competencies persist at the time of gradua-tion and beyond should be part of a school’s plan for determining whether holistic review processes are achieving their intended goals.



At GraduationPotential measures: Competency measures should be based on specific dimensions of knowledge and skill that the school explicitly strives to teach students and to which student body diversity plausibly contributes. Examples of questions to guide competency assess-ment include:

• Are students prepared to address the needs of diverse patient populations?

• How many students are proficient in non-English languages spoken among patients in the local community?

• Do students communicate effectively with patients and colleagues from diverse backgrounds?

• Are students competent in addressing the psycho-social needs and concerns of patients?

• Do students consistently uphold high professional and ethical standards?

After GraduationPotential measures: Competencies to be evaluated among graduates are similar to those evaluated at the time of graduation. The goal is to determine whether competencies are sustained and evolving.

• Are graduates upholding high professional and ethical standards?

• Are graduates receiving recognition for their competence in caring for diverse patient populations?

• Does feedback from residency programs indicate that graduates are working effectively with team members of different disciplines and backgrounds?

• Is there evidence that graduates are demonstrating leadership skills in residency and beyond?

Career goals and practicesMost medical schools have explicit goals of producing physicians who will contribute to the workforce in specific ways. Common goals include producing primary care physicians, practitioners working in rural or other medically underserved areas, researchers, or medical educators. Holistic review can contribute to fulfilling these goals by prioritizing the selection of applicants who are likely to fill these workforce niches, and by nurturing interest in these areas through exposure to a diverse group of classmates. Evaluating the degree to which the school is succeeding in addressing workforce priorities should be done by assessing students’ career intentions at the time of graduation and their actual practice patterns after entry into the workforce.

At GraduationPotential measures: Career intentions at the time of graduation are only promissory notes, but they are an important measure of whether, during the years students are under the school’s purview, their desire to engage in specific types of activities and careers has been sustained and nurtured.

• How many students intend to practice in:

– disadvantaged communities?

– urban communities?

– rural areas?

– the state where the medical school is located?

• How many students plan to practice high need specialties?

• How many students plan to engage in:

– teaching?

– research?

– health care quality improvement?

– health policy and leadership?

– academic medicine?

– community/volunteer service?



After GraduationPotential measures: Evaluation of practice charac-teristics at the time of graduation can be considered intermediate outcomes leading to the more defini-tive outcomes of practice patterns among graduates who have entered the physician workforce. Guiding questions should mirror those for at graduation:

• How many graduates are practicing in:

– disadvantaged communities?

– urban communities?

– rural areas?

– the state where the medical school is located?

• How many graduates are practicing high need specialties?

• How many graduates are engaged in:

– teaching?

– research?

– health care quality improvement?

– health policy and leadership?

– academic medicine?

– community/volunteer service?

Potential Data Sources

Gathering workforce data after students graduate from your medical school can be challenging. We have identified some methods and sources that might be helpful in collecting workforce outcome data. Your institution likely already has some of these mecha-nisms or similar mechanisms in place.

At graduationData sources to evaluate student outcomes should come both from students themselves and from those who are able to observe students. Attitudes represent personal beliefs, thereby making students’ responses the principal source for measuring them. Similarly, at the time of graduation, students can report only on their intended career goals. However, it can be difficult to measure students’ true attitudes and career goals due to social desirability bias—the tendency for people to respond in a way that they know is valued by others, or “correct.” Observation of student behavior provides a more objective way of determining whether students act in ways that reflect desired attitudes. Competencies generally should be evaluated, to the extent possible, through observation of student behavior.

• Surveys: Surveys are the principal method for evaluating student attitudes. Carefully crafted questions or statements for which students rate their agreement/disagreement can tap into the values students espouse. Career goals are likewise primarily assessed via surveys. The AAMC Graduation Questionnaire includes questions about attitudes, self-rated competencies, and career goals, and can be used as a resource. Due to the possibility of social desirability bias, ensuring students that survey responses will be confidential or anonymous will elicit more reliable data.

• Interviews with graduating students: Interviews are another method of evaluating student outcomes and can provide richer and more finely textured information than surveys. They are, however, more time consuming. Selecting a small sample of students representing a range of backgrounds and interests can be useful in supple-menting survey data from a larger sample.

• Tests and objective structured clinical exams (OSCEs): Students are often asked to respond to case vignettes as part of written tests and OSCEs. Evaluation of responses for desired attitudes and ethical principles can be used as part of attitu-dinal assessments. Written tests or OSCEs are also often used to evaluate competencies. Some of



these evaluations can be structured with the goal of determining whether students are accruing the knowledge and skills that diversity is intended to foster. Tests and OSCEs should be designed with diversity-related attitude and competency assess-ment in mind.

• Clerkship evaluations: Summative data from third and fourth year clerkship evaluations can provide important information to determine whether students are displaying attitudes and competencies that are in line with the school’s stated mission. As such, it is important to ensure that measures of desired attitudes and competencies related to diversity are included in clerkship evaluation forms.

• Residency match data: Data on chosen residency programs can help schools determine if they have accomplished mission-based goals. While the type and location of residency does not neces-sarily determine long-term practice patterns, the proportions of students entering specific types of programs (e.g., academic, community, urban, rural, high need specialties, specialty, in state, out of state) can give valuable information about whether mission-based goals related to workforce are being met.

After graduation Tracking outcomes among graduates should ideally be done using an alumni database. Such a database can help schools keep track of alumni locations, personal data, and career information that can be easily tabulated and displayed for outcomes assessment.

• Feedback from residency programs: Schools may receive unsolicited feedback from residency programs when residents’ attitudes and behaviors are particularly positive or negative. Proactively soliciting feedback from residency program directors for a sample of graduates on a periodic basis can provide a more holistic view of whether the school is producing physicians with desired attitudes and competencies. Many relevant attitu-dinal outcomes and competencies are routinely evaluated by residency programs.

• Surveys: Surveying alumni can be difficult, as most physicians are busy and often do not have time to respond to surveys. Data gathering through alumni association efforts, or periodically at the time of class reunions, can be leveraged to collect information on mission-based outcomes. Desired outcome metrics can be incorporated into such data gathering efforts. Data to be gathered through surveys might include receipt of awards, selection for leadership positions, and advocacy efforts.

• Internet searches and physician databases: Data on physicians are now available through a variety of internet resources and from vendors maintaining physician data. Alumni databases can be kept up to date by obtaining alumni data, which may include documentation of location and other information including practice characteristics, responsibilities, service commitments, publica-tions, awards, and other forms of recognition, as well as negative outcomes reflecting unprofessional behavior (e.g., licensure restrictions). GIS mapping can be used to provide a picture of how many graduates are practicing in different locations relevant to a schools’ mission (e.g., rural, urban, in state, and/or in medically underserved areas). The American Academy of Family Physicians (AAFP) Med School Mapper is a good example. The Medical School Mission Management Tool published by the AAMC every year also can be a good resource for evaluating where and in which specialties/fields your school’s graduates practice.

To assist your school in using this evaluation model, we have created a table (Table 4.1), which highlights sample questions and data sources.



Table 4.1: Assessing Workforce Outcomes Associated with Holistic Admissions

Sample Domains

At Graduation After Graduation

Potential Measures Potential Data Sources Potential Measures Potential Data Sources

Attitudes • Compassion toward all patients

• Responding ethically

• Respectfulness

• Priority of patient autonomy

• Medical care as a right

• Responsibility toward others

• Student surveys

• AAMC Graduation Questionnaire

• Interviews with graduating students

• Tests with case studies

• Objective structured clinical exams (OSCEs)

• Clerkship evaluations

• Residency match data

• Good work ethic

• Commitment to patients and professionalism

• Commitment to service

• Health care advocacy

• Feedback from residency programs

• Alumni surveys and interviews

Competencies • Prepared to address needs of diverse patient populations

• Proficiency in another language

• Effective communication with diverse others

• Addressing psychosocial needs and concerns of patients

• Upholding high professional and ethical standards

• AAMC Graduation Questionnaire

• Tests with case studies

• Objective structured clinical exams (OSCEs)

• Clerkship evaluations

• Upholding high professional and ethical standards

• Recognition/awards for skill in caring for diverse patient populations

• Effective communication and teamwork

• Demonstrating leadership skills

• Feedback from residency programs

• Patient satisfaction surveys

• HRSA National Practitioner Data Bank (NPDB)

• State medical board records

Career Goals and Practice

• Intention to practice in disadvantaged areas (e.g., rural, urban, etc.)

• Intention to practice high need specialty

• Intention to engage in:

– teaching – research – health care quality improvement

– health policy and leadership

– academic medicine

– community/volunteer service

• Student surveys

• Interviews with graduating students

• Residency match data

• Practicing in disadvantaged communities (e.g., rural, urban, etc.)

• Practicing high need specialties

• Engaged in: – teaching – research – health care quality improvement

– health policy and leadership

– academic medicine

– community/volunteer service

• Alumni surveys

• Internet searches

• Physician databases

• GIS mapping/AAFP Med School Mapper

• AAMC Medical School Mission Management Tool



Summary

The benefits of using holistic review to increase student body diversity within medical schools can be viewed generally as twofold. First, when intention-ally and actively fostered, diversity within the student body can lead to a more robust learning environ-ment that helps students develop into more broad-thinking, socially engaged, competent, and caring professionals. Second, increasing and nurturing medical student body diversity contributes to shaping a diverse physician workforce that not only meets discrete institutional missions, but also serves the health care needs of a diverse population. Combined, these benefits contribute to every medical school’s ultimate goal, independent of institutional mission, of improving the health of the public.

Holistic admissions—or any admissions process—alone is insufficient for effecting the learning and practice outcomes associated with a diverse student body and physician workforce. Chapter 2 focuses largely on the institution and proposes using a modified structure-process-outcome framework to evaluate the effectiveness of your school’s compre-hensive enrollment management efforts (i.e., Is your school attracting, admitting, and supporting the diverse student body it seeks?). Chapter 3 uses the inputs-environment-output model to explore the relationship between student inputs and the medical school environment to assess key learning, develop-mental, and professional outcomes. Finally, Chapter 4 presents a simple framework for evaluating workforce outcomes and the extent to which your graduates are fulfilling your institutional mission in their profes-sional practice.

It is difficult to over emphasize the importance of institutional mission both in conducting holistic admissions and evaluating the impact thereof, a theme addressed throughout this publication and the previous Roadmap to Diversity documents. Just as institutional mission serves as the polestar for developing your school’s policies and processes, it also guides the questions a school asks and the data it collects through its evaluation efforts. Evaluating the core processes and environmental factors imple-mented and short- mid- and long-term outcomes sought by your medical school will provide powerful feedback on the extent to which both the school and its graduates are meeting institutional mission and goals. It also provides useful data about promising practices, strategies, and approaches for the medical education community at large. We encourage you to share your school’s evaluation results not only with your medical school’s key stakeholders, but also with other medical educators to help nurture a community of practice focused on the impact of holistic admis-sions and increased diversity in medical education and the physician workforce.



Endnotes

1 “Evaluate.” The American Heritage Dictionary, 5th ed. 2012.

2 Coleman AL, SR Palmer, SY Winnick. Roadmap to Diversity: Key Legal and Educational Policy Foundations for Medical Schools. Washington, DC: AAMC; 2008.

3 To comply with federal law, an institution must assemble and retain relevant evidence that demonstrates the effectiveness of its race- and ethnicity-conscious practices. See generally Grutter v. Bollinger, 539 U.S. 306 (2003); Coleman AL, et al. Admissions and Diversity After Michigan: The Next Generation of Legal and Policy Issues. College Board, 2006.

4 Addams AN, RB Bletzinger, HM Sondheimer, SE White, LM Johnson. Roadmap to Diversity: Integrating Holistic Review Practices into Medical School Admissions Processes. Washington, DC: AAMC; 2010.

5 Milem JF and CL O’Brien. “Evaluating the Impact of Holistic Review.” 2012 AAMC Annual Meeting. San Francisco, Calif. November 2, 2012.

6 “Formative and Summative Evaluation.” University of Michigan Center for Research on Learning and Teaching Web site. Retrieved on January 11, 2013 from www.crlt.umich.edu/tstrategies/tsfse.

7 Vander Schee BA. American Association for Collegiate Registrars and Admissions Officers (AACRAO) Consulting. “Organizational Models for Enrollment Management at Small Colleges.” AACRAO College and University 82, no. 3. Retrieved on January 11, 2013, from the American Association for Collegiate Registrars and Admissions Officers (AACRAO) Consulting Web site: http://consulting.aacrao.org/publications_events/publications/organizational-models-for-enrollment-management-at-small-colleges/.

8 Donabedian A. The quality of care: How can it be assessed? JAMA. 1988; 260(12): 1743-1748.

9 Rossi PH, HE Freeman, MW Lipsey. Evaluation: A Systematic Approach (6th edition). Thousand Oaks, Calif: Sage; 1999.

10 Astin AW. Assessment for Excellence: The Philosophy and Practice of Assessment and Evaluation in Higher Education. New York: American Council on Education; 1991.

11 Adapted by Som Saha from Astin’s Assessment for Excellence: The Philosophy and Practice of Assessment and Evaluation in Higher Education.

12 Several of the post-matriculation services identi-fied in Chapter 2 could also fall under the environ-mental measures presented in this chapter. We are intentionally providing you with multiple frameworks for evaluating key areas of interest to medical schools and encourage you to explore, identify, and modify the frameworks that best suit your goals and capacity.

13 Garrison G, C Mikesell, D Matthew. Medical school graduation and attrition rates. AAMC Analysis in Brief. April 2007; 7.

14 Brewer L, D Grbic. Medical students’ socioeco-nomic background and their completion of the first two years of medical school. AAMC Analysis in Brief. December 2010; 9.

15 Case SM, DB Swanson, DR Ripkey, LT Bowles, DE Melnick. Performance of the class of 1994 in the new era of USMLE. Acad Med. 1996;71(10 suppl):S91–S93.

16 Andriole DA, DB Jeffe. A national cohort study of U.S. medical school students who initially failed Step 1 of the United States Medical Licensing Examination. Acad Med. 2012;87.

17 Dyrbye LN, MR Thomas, DV Power, et al. Burnout and serious thoughts of dropping out of medical school: A multi-institutional study. Acad Med. 2010; 85(1): 94-102.



18 Dyrbye LN, FS Massie, A Eacker, et al. Relationship between burnout and professional conduct and attitudes among US medical students. JAMA. 2010; 304(11): 1173-1180.

19 Hafferty FW. Beyond curriculum reform: Confronting medicine’s hidden curriculum. Acad Med. 1998;73:403-407.

20 Spitzer RL, JB Williams, K Kroenke, et al. Utility of a new procedure for diagnosing mental disorders in primary care. The PRIME-MD 1000 study. JAMA. 1994;272: 1749–1756.

21 Maslach C, SE Jackson, MP Leiter. Maslach Burnout Inventory Manual. 3rd ed. Palo Alto, Calif: Consulting Psychologists Press; 1996.

22 Gaufberg EH, M Batalden, R Sands, SK Bell. The hidden curriculum: What can we learn from third-year medical student narrative reflections? Acad Med. 2010;85:1709-1716.

23 Karnieli-Miller O, TR Vu, RM Frankel, MC Holtman, SG Clyman, SL Hui, TS Inui. Which experiences in the hidden curriculum teach students about professionalism? Acad Med. 2011;86:369-377.

24 Ware J, M Kosinski, J Dewey, B Gandek. How to Score and Interpret Single-Item Health Status Measures: A Manual for Users of the SF-8 Health Survey. Lincoln, RI: QualityMetric Incorporated; 2.

Association of American Medical Colleges, 2013A-1


Appendix AAAMC Experiences-Attributes-Academic Metrics ModelThe Experiences-Attributes-Metrics Model, illustrated below, captures several of the dimensions, both visible and invisible, that might comprise an individual.

Experiences: This category encompasses the path the applicant has taken to get to where he or she is. Examples include being the primary caregiver for an ill family member, distance travelled, employment history, research experience, and experience in a health care setting.

Attributes: This category includes the applicant’s personal characteristics and demographic factors. Examples include empathy, resilience, first generation college student, sexual orientation, race, ethnicity, and intellectual curiosity.

Metrics: This category includes the numeric information about an applicant’s academic performance, most notably GPA, MCAT® scores, and grade trends.

Adapted from Workforce America: Managing Employee Diversity as a Vital Resource, McGraw Hill Publishing, 1990.

AA

MC

Experie

nces-Attributes-Metrics Model

© AAMC

historicalevents

educationbackground

ethnicity

genderidentity

faith

familystatus

nationalorigin

communityservice

healthcareexperience

citizenship

sexleadership

roles

age

distancetravelled

raceindividualinterests

sexualorientation

culturalevents

lifeexperiences

geography

physicalability values &

beliefs

MCATscores

leadership

grade trends

GPA

EXCELLENCE

perspectives

relationshipstatus

languagesspoken

maturity

intellectualcuriosity

fields ofinterest

worldevents

researchexperience

politicalevents

affiliations

ATTRIBUTES

EXPERIENCES

METRICS



Important Considerations for Evaluating Enrollment Management FunctionsOutreach and Recruitment

Structure (i.e., organizational infrastructure, policies, resources, facilities)

• Do institutional policies that guide outreach and recruitment support institutional mission? For schools that consider race and ethnicity, do the policies address relevant legal issues?

• What explicit goals, if any, has the school made for reaching out to and recruiting a diverse student body?

• Who is accountable for outreach and recruitment efforts?

• How much time and how many resources are devoted to outreach and recruitment programs?

• What percentage, if any, of those resources is dedicated to diversity-related efforts?

• Are outreach and recruitment policies aligned with admissions criteria?

• Is participation in outreach and recruitment efforts sufficiently important that it is considered in performance evaluation, promotion, and tenure decisions?

Process (i.e., actions)

• Is the purpose of each outreach and recruitment activity clearly articulated and commonly understood?

• Are there specific activities to engage students and communities that are targeted by diversity initiatives?

• Are outreach and recruitment activities coordinated with each other and across people/units with responsibility in this area, such as the admissions and diversity affairs offices?

• Are there opportunities for collaboration to enhance and streamline outreach and recruitment activities?

Outcome (i.e., short-term, mid-term, and/or long-term)

• Which outreach and recruitment activities are most effectively contributing to an applicant pool with the diversity of experience, attributes, and academic metrics the school seeks?

• Are applicants who are recruited from targeted populations likely to be accepted?

• To what extent have the offices and staff accountable for outreach and recruitment established relationships with colleges, communities, and organizations serving as “pipelines” for targeted groups?

• What is the level of knowledge about your medical school among potential applicants from targeted populations?

• What is the level of knowledge among internal and external stakeholders about outreach and recruitment activities and the populations targeted?

• Do internal and external stakeholders recognize the value of outreach and recruitment efforts to enhance diversity with the student body and physician workforce?


• Institutional budget audits to determine levels and kinds of resources dedicated to outreach and recruitment, both generally and diversity-related

• Inventory charting strategy and design of annual outreach and recruitment activities to assess separate and collective efforts

• Surveys and tracking of participants who attended outreach and recruitment activities to assess satisfaction and identify areas for improvement

• Analysis of applicant yields, both generally and specific to outreach and recruitment activities, to document and analyze effectiveness

Appendix BImportant Considerations for Evaluating Enrollment Management Functions



*Addams AN, et al. Roadmap to Diversity: Integrating Holistic Review Practices into Medical School Admission Processes. Washington, DC: AAMC; 2010.

Important Considerations for Evaluating Enrollment Management FunctionsAdmissions (also see “Holistic Review Admissions Checklist”*)


• Do the institution’s admissions policies and criteria explicitly reflect the school’s mission, values, and culture, including a commitment to diversity? For schools that consider race and ethnicity, do the policies address relevant legal issues?

• Are admissions criteria informed by the student experiences, attributes, and academic metrics found to be predictive of success at your medical school?

• Do the institution’s admissions policies unintentionally screen out applicants who would contribute to the school’s mission?

• Who is accountable for admissions results?

• What resources in terms of funding, staffing, and time are devoted to the admissions process?

• Does the admission committee’s membership reflect the diversity of experiences, attributes, and academic background sought in the entering class?

• Do the admissions committee and staff receive training on using admissions criteria to screen, interview, and select applicants?

• Is the medical school dean committed to a holistic admissions system to increase diversity in alignment with institutional mission?

• Is membership on the admissions committee sufficiently valued that participation is considered in performance evaluation, promotion, and tenure decisions?


• Does the medical school dean issue a charge to the admissions committee that explicitly emphasizes the school’s mission and the range of diversity the school seeks?

• How and when are the institution’s mission and goals communicated to admissions staff, committee members, and interviewers?

• Does the admissions office work closely with outreach and recruitment, financial aid, and other program office staff to limit barriers to matriculation and academic success for any accepted applicant?

• At what point(s) in the admissions process are applicants’ potential contributions to the learning environment and practice of medicine taken into account? Is diversity in terms of experiences, attributes, and academic metrics part of this consideration?

• When in the process do admissions committee members receive school-specific student performance data to help inform their decision making?


• Are the dimensions of diversity embodied in admitted and matriculated students congruent with the medical school’s mission?

• Is the school maintaining that diversity on a longitudinal basis?

• Are common qualities, experiences, or attributes shared by applicants who are not invited to interview? By applicants who are not accepted?

• Are there disparities between the experiences, attributes, and academic metrics of accepted students and those who actually matriculate?

• Which admissions outcomes/incoming student characteristics are reported by institutional leadership?

• Does the school’s faculty acknowledge the qualifications of the diverse students who enroll with the entering class?

table continued on next page



Important Considerations for Evaluating Enrollment Management FunctionsAdmissions (also see “Holistic Review Admissions Checklist”*)


• Internal policy documentation to assess if admissions policies, criteria, and procedures support institutional mission and related diversity goals

• AMCAS® and secondary institutional applications to assess:

– whether data gathered and used are consistent with institutional mission and admissions policies

– snapshot and trend analyses to compare the applicant pool, accepted applicants, and matriculated students with intended mission-driven outcomes

• Documentation of interview protocols to determine consistency with institutional mission and admissions policies

• Interviews or focus groups with admission committee members, interviewers, and staff to evaluate their understanding of the school’s diversity-related goals and whether/how they balance applicant experiences, attributes, and academic metrics to make interview and selection decisions

• Applicant surveys to assess effectiveness of the admissions process from applicant to interview to selection

• Interviews or focus groups with matriculated students to gather their feedback about the admissions process

• Interview ratings/scores to assess inter-rater reliability and the predictive validity of admissions interviews

• Student outcomes data using new or existing survey measures possibly including, but not limited to, a focus on applicants’ levels of humanism, empathy, cognitive complexity/critical thinking, dogmatism/flexibility, attitudes toward serving the underserved, and engagement with diversity

• Analysis of accepted student and matriculating student yields to determine the degree to which they reflect mission-driven goals

*Addams AN, et al. Roadmap to Diversity: Integrating Holistic Review Practices into Medical School Admission Processes. Washington, DC: AAMC; 2010.

table continued from previous page



Important Considerations for Evaluating Enrollment Management FunctionsFinancial Aid and Debt Management


• Are the institution’s financial aid policies for awarding grants, scholarships, and loans congruent with supporting the school’s mission and goals, including diversity? For schools that consider race and ethnicity, do the policies address legal considerations?

• Are the policies congruent with outreach, recruitment, and holistic admissions efforts?

• What are the funding sources/streams for grants, scholarships, and loans?

• What is the breakdown of grants, scholarships, and loans?

• Are grants/scholarships need- or merit-based? If merit-based, is merit defined relative to institutional mission?

• Who is accountable for the medical school’s financial aid program, including debt management counseling?

• Does the medical school allocate sufficient financial aid funds so that award packages attract and retain a student body that encompasses the diversity of experiences, attributes, and academic metrics congruent with institutional mission?

• Does the school provide sufficient resources in terms of staff, funding, and facilities to carry out meaningful and timely debt management counseling for its medical students and graduates?

• Is involvement in financial aid and debt management programs sufficiently important that it is considered in performance evaluation, promotion, and tenure decisions?


• How are the institution’s financial aid award determinations made, and by whom?

• How and when do admitted students apply for financial aid?

• Are awards made on a timely basis so that accepted applicants and current students can make plans without being rushed or stressed?

• Are diversity and other mission-oriented factors incorporated into the decision-making processes for awarding financial aid?

• Does the school consider each applicant for financial support individually/separately or among the larger applicant pool?

• What portion of your school’s financial support for students is dedicated exclusively to diversity-related goals? What percentage includes some diversity-related focus?

• Are financial aid awards sufficient to cover the cost of attendance (e.g., tuition, fees, living expenses, etc.) for students from low socioeconomic backgrounds?

• Do the awards minimize students’ educational debt?

• Does the school counsel applicants and students about budgeting as a means of reducing indebtedness and about the pitfalls of taking out commercial loans (e.g., using credit cards)?

• What training does the financial aid office staff receive about the school’s diversity-related goals and policies and the financial aid office’s role in meeting those goals?

• Does the school encourage alumni and others to make donations to increase institutional financial aid funds for medical students?






• Are the institution’s financial aid policies for awarding grants, scholarships, and loans congruent with supporting the school’s mission and goals, including diversity? For schools that consider race and ethnicity, do the policies address legal considerations?

• Are the policies congruent with outreach, recruitment, and holistic admissions efforts?

• What are the funding sources/streams for grants, scholarships, and loans?

• What is the breakdown of grants, scholarships, and loans?

• Are grants/scholarships need- or merit-based? If merit-based, is merit defined relative to institutional mission?

• Who is accountable for the medical school’s financial aid program, including debt management counseling?

• Does the medical school allocate sufficient financial aid funds so that award packages attract and retain a student body that encompasses the diversity of experiences, attributes, and academic metrics congruent with institutional mission?

• Does the school provide sufficient resources in terms of staff, funding, and facilities to carry out meaningful and timely debt management counseling for its medical students and graduates?

• Is involvement in financial aid and debt management programs sufficiently important that it is considered in performance evaluation, promotion, and tenure decisions?


• How are the institution’s financial aid award determinations made, and by whom?

• How and when do admitted students apply for financial aid?

• Are awards made on a timely basis so that accepted applicants and current students can make plans without being rushed or stressed?

• Are diversity and other mission-oriented factors incorporated into the decision-making processes for awarding financial aid?

• Does the school consider each applicant for financial support individually/separately or among the larger applicant pool?

• What portion of your school’s financial support for students is dedicated exclusively to diversity-related goals? What percentage includes some diversity-related focus?

• Are financial aid awards sufficient to cover the cost of attendance (e.g., tuition, fees, living expenses, etc.) for students from low socioeconomic backgrounds?

• Do the awards minimize students’ educational debt?

• Does the school counsel applicants and students about budgeting as a means of reducing indebtedness and about the pitfalls of taking out commercial loans (e.g., using credit cards)?

• What training does the financial aid office staff receive about the school’s diversity-related goals and policies and the financial aid office’s role in meeting those goals?

• Does the school encourage alumni and others to make donations to increase institutional financial aid funds for medical students?



• What is the relationship among accepted students who seek financial support, those who receive or do not receive aid, and the students who matriculate?

• To what degree do the school’s available financial aid funding and resulting award packages support or hinder the school’s commitment to diversity?

• Has the school been effective in encouraging alumni or community partners to build sufficient financial aid funds to assure that attending the medical school will not be financially out of reach?

• To what extent do anticipated or actual debt loads influence potential applicants’ decisions to apply for admission, accepted applicants’ decisions to matriculate, and current students’ specialty and practices choices?

• Do indebtedness levels have an impact on your school’s students’ and graduates’ academic performance, professionalism, or mental or physical well-being?


• Review of the medical school’s financial aid budgets over time to assess whether funding is keeping up with cost of attendance

• Trends in donations to and size of the medical school’s financial aid funds to ascertain if funding has increased

• Review of accepted students’ award packages to determine if the amount and configuration of financial aid awards (e.g., grants vs. loans) affect decisions to matriculate

• Review of matriculated students’ award packages to determine if the amount or configuration of awards affects performance, persistence, mental and physical well-being, etc.

• Surveys of applicants, accepted applicants, and matriculated and current students to assess effectiveness of the medical school’s financial aid and debt management counseling programs, and identify areas for improvement

• Interviews or focus groups with financial aid officers to determine approaches for supporting holistic admissions and institutional mission-driven diversity goals




Important Considerations for Evaluating Enrollment Management FunctionsDiversity Programming and Support (also see “Medical School Diversity Self-Assessment”**)


• Does institutional leadership, including the medical school’s dean, expressly value and support developing a student body and cadre of graduates who reflect the diversity intended by institutional mission?

• Are diversity policies congruent with institutional mission, including diversity goals? For schools that consider race and ethnicity, do policies address legal concerns

• Who has primary responsibility for diversity programming and support? Where is the position placed in the institution’s/medical school’s organizational chart?

• Is diversity affairs considered a resource specifically for students or does its scope include the entire institution (e.g., faculty, curriculum, etc.)?

• Is diversity programming and support considered part of the school’s mainstream curricular and co-curricular efforts?

• Does your institution have dedicated diversity affairs and/or diversity personnel or office(s)?

• Does the medical school allocate sufficient staff, time, funding, and facilities to enable diversity programming and support to be successful?

• Is participation in diversity programs sufficiently important that it is considered in performance evaluation, promotion, and tenure decisions?


• Are diversity efforts integrated across the medical school? Across the institution?

• Do diversity programs support the medical school’s holistic admissions efforts, and conversely?

• Are diversity efforts seamlessly integrated into the medical school’s formal learning spaces and its formal/informal enrollment, support, and enrichment activities?

• In developing diversity-building programs, both pre- and post-matriculation, is collaboration sought with curriculum staff and faculty members, as well as with enrollment management areas, such as outreach and recruitment, admissions, financial aid, and academic and career advising?

• Does the school share information with internal and external stakeholders to highlight how student body diversity specifically contributes to institutional mission and excellence? How is the information shared/publicized?

• Do faculty and staff receive training on diversity-related issues about health and medical education, as well as the relationship between diversity and institutional mission? Does the training present diversity as a mainstream endeavor, not as an adjunct to the medical education enterprise?

• Do faculty and staff receive training on implementing diversity programs to maximize effectiveness and minimize unintended negative outcomes?

**Coleman AL, et al. Roadmap to Diversity: Key Legal and Educational Policy Foundations for Medical Schools. Washington, DC: AAMC; 2008.




Important Considerations for Evaluating Enrollment Management FunctionsDiversity Programming and Support (also see “Medical School Diversity Self-Assessment”**)


• Do enrolled students from different backgrounds feel that they are part of the medical school community, or do they perceive themselves as outsiders?

• Are students from different backgrounds engaging each other in positive ways in both formal and informal settings?

• Do students view diversity as fundamental to the quality of education that they are receiving at the medical school?

• Do students view diversity as fundamental to their ability to practice medicine in the future?

• Does the medical school curriculum consistently integrate diversity-related considerations across all courses and clerkships?

• Do the faculty and staff recognize that diversity is core to educational, practice, and research excellence?

• Do the medical school graduates feel that they are better prepared to practice medicine because they were educated in an educational environment that values diversity?


• Surveys/focus groups of enrolled students to assess satisfaction with diversity programming, determine immediate effects, and identify areas for improvement

• Review of course and clerkship syllabi to establish the extent to which diversity-related issues are integrated into the curriculum and identify where diversity materials should be modified or added

• Review of course/clerkship grades, test results, and evaluations to determine whether students understand how diversity affects health and health care

• Surveys/focus groups of graduates to determine the mid- and longer-term effects of the school’s diversity programming on how they practice medicine, what they chose to practice, and where they practice

• Surveys of residency directors to follow up on graduated physicians’ ability to work with health providers and treat patients from diverse backgrounds

• Surveys/focus groups of faculty members and staff to assess satisfaction of diversity training efforts, determine immediate- and longer-term effects, and identify areas for improvement

**Coleman AL, et al. Roadmap to Diversity: Key Legal and Educational Policy Foundations for Medical Schools. Washington, DC: AAMC; 2008.




Important Considerations for Evaluating Enrollment Management FunctionsAcademic Advising and Support


• Are policies for conducting academic advising and support congruent with institutional mission, including diversity goals? For schools that consider race and ethnicity, do policies address legal considerations?

• Do academic advising policies consider admissions criteria in relationship to the range of diversity within the student body and the academic expectations of the school?

• Who has primary responsibility for academic advising and support?

• Does your institution have dedicated staff to carry out academic advising and support?

• Does the medical school allocate sufficient staff, time, funding, and facilities to provide appropriate help to any student who needs it?

• Given the breadth of diversity that a holistic admissions system will create, has your medical school developed mechanisms to identify students (both matriculating and enrolled) who are likely to need support and provide them with effective academic advising before they encounter difficulty?

• Are academic advising and support sufficiently important that they are considered in performance evaluation, promotion, and tenure decisions?


• How and when are students in need of academic assistance identified and referred?

• In addition to the teaching faculty, does the academic advising staff rely on personnel in such offices as student affairs, financial aid, registrar/student records, and so forth, because they interact often with students and may be able to detect early signs of trouble?

• Does academic advising staff work closely with curriculum staff and faculty to ensure that the content of the assistance provided to students is on target?

• Are academic advising interventions designed to assess and address underlying causes for students struggling with the curriculum, such as educational background prior to medical school, stress brought on by personal/family situations, and physical or mental health issues?

• How is academic assistance provided (e.g., in person, Web-based)? Who provides the assistance (e.g., learning specialists, faculty, peer tutors)? Are the approaches/methods tailored to the needs of each student?

• How does the school ensure that students do not perceive barriers to requesting help and that confidentiality is honored?

• Does the school train academic advising staff on institutional mission-driven interest in diversity, holistic admissions criteria, and the pivotal role that post-matriculation enrollment management services play in maximizing success for students and the school?


• How many students use academic advising and support services? How many students who may have benefited from these services are not accessing them?

• What are the experiences, attributes, and/or academic metrics that students using the services have in common?

• Of the students who used these services, how many successfully completed the coursework/clerkship in which they were having difficulty? How many did not?

• Does the timing of when assistance is offered make a difference in rates of success and failure?

• Is confidentiality preserved for students receiving assistance?

• Are certain interventions and approaches more effective than others?




Important Considerations for Evaluating Enrollment Management FunctionsAcademic Advising and Support


• Inventory of policies related to academic performance and advising

• Surveys, interviews, and/or focus groups with students to assess perceived availability, usefulness, and accessibility of services

• Surveys, interviews, and/or focus groups with students, faculty members, and administrators to assess attitudes and possible barriers to seeking academic assistance

• Review of medical school academic records to pinpoint when students actually began encountering difficulty, identify when the school recognized they needed help, discern if there are any commonalities among these students, and determine which interventions were most effective and where improvements need to be made




Important Considerations for Evaluating Enrollment Management FunctionsCareer Advising


• Are policies for career advising and support congruent with institutional mission, including diversity goals? For schools that consider race and ethnicity, do policies address legal concerns?

• Has the medical school designed its career advising program to balance the aspirations of the institution with the aspirations of its individual medical students?

• Who has primary responsibility for career advising and support?

• Does your institution have dedicated staff to carry out career advising and support? What are their backgrounds (e.g., psychology, advising, education)?

• Does the medical school allocate sufficient staff, time, funding, and facilities to provide meaningful career guidance to every student?

• Are career advising and support sufficiently important that they are considered in performance evaluation, promotion, and tenure decisions?


• When does the medical school’s career advising program begin? Does career advising take place on a continuum throughout medical school?

• How is career advising delivered (e.g., group information sessions, one-on-one, written material, online, a combination)?

• Does the career advising staff coordinate its efforts with the diversity affairs staff, as well as with student affairs, to integrate considerations and issues that students from different backgrounds may face as they choose a specialty and apply for residency positions?

• Are career advising efforts tailored to help each medical student choose a specialty based on individual interest, not on assumptions that may be grounded in traditional biases regarding medical careers for certain demographic groups, such as women?

• Does the school train career advising staff on the medical school’s mission-driven interest in diversity and how to assist medical students in choosing a satisfying career that also advances health care for all?


• Do medical students feel that they are able to make well-informed decisions about specialty and career path?

• Do the medical school’s graduates who have entered residency and/or practice feel that their career decisions were well-informed?

• What is the degree of alignment between specialty choices at graduation and institutional mission?

• What is the degree of alignment between graduates’ practice decisions upon completing residency and institutional mission?


• Inventory of policies, processes, and offerings

• Surveys, interviews, and/or focus groups with students to assess perceived availability, usefulness, and accessibility of services

• Interviews or focus groups with admissions committee members and interviewers to evaluate their understanding of the school’s diversity-related goals and whether/how they apply diversity-related factors in advising students about career options



Appendix C

AAMC Advisory Committee on Holistic Review (Current)

David Acosta, M.D., FAAFP Associate Dean for Multicultural Affairs, Clinical Professor, Department of Family MedicineUniversity of Washington School of Medicine

Dwight Davis, M.D.Professor of MedicineAssociate Dean for Admissions and Student AffairsPennsylvania State College of Medicine

J. Manuel de la Rosa, M.D.DeanTexas Tech University Health Sciences Center Paul L. Foster School of Medicine

Anne C. Gill, DrPH, M.S.Associate Professor Pediatrics and Medical EthicsDirector of Longitudinal ProgramsBaylor College of Medicine

Peter KaminskiMedical StudentAlpert Medical School at Brown University

O’Rese J. Knight, M.D.Ophthalmology ResidentUniversity Hospitals Case Medical Center

Catherine R. Lucey, M.D.Vice Dean for EducationUniversity of California, San Francisco, School of Medicine

Jeffrey F. Milem, Ph.D.McFarland Distinguished Professor Professor of Higher Education and Professor of Medicine (by courtesy)University of Arizona

Alicia D.H. Monroe, M.D. [Chair]Chief Academic OfficerVice Dean, Educational AffairsUSF Health Morsani College of MedicineUniversity of South Florida

Somnath Saha, M.D., M.P.H.Staff Physician, Portland VA Medical CenterAssociate Professor of MedicineOregon Health & Science University

David A. Verrier, Ph.D.Director, Pre-Professional Programs and AdvisingJohns Hopkins University

Legal ConsultantsArthur L. Coleman, J.D.EducationCounsel LLC

Kate Lipper, J.D.EducationCounsel LLC

Project StaffAmy N. AddamsHope GrayLily May JohnsonHenry Sondheimer, M.D.S. Elizabeth WhiteGeoffrey H. Young, Ph.D.



AAMC Advisory Committee on Holistic Review (2007–2012)

David Acosta, M.D., FAAFP Associate Dean for Multicultural Affairs, Clinical Professor, Department of Family MedicineUniversity of Washington School of Medicine

Cynthia E. Boyd, M.D., M.B.A., FACPAssociate Professor of MedicineRush Medical CollegeChief Compliance Officer, Associate Vice PresidentRush University Medical Center

Dwight Davis, M.D.Professor of MedicineAssociate Dean for Admissions and Student AffairsPennsylvania State College of Medicine

Linda K. Don, M.Ed.Assistant DeanStudent and Educational AffairsUniversity of Arizona College of Medicine–Tucson

O’Rese J. Knight, M.D.Ophthalmology ResidentUniversity Hospitals Case Medical Center

Jeffrey F. Milem, Ph.D.McFarland Distinguished Professor Professor of Higher Education and Professor of Medicine (by courtesy)University of Arizona

Somnath Saha, M.D., M.P.H.Staff Physician, Portland VA Medical CenterAssociate Professor of MedicineOregon Health & Science University

Jim Scott, M.D. [Chair]Professor of Emergency Medicine The George Washington University School of Medicine and Health Sciences

David A. Verrier, Ph.D.Director, Pre-Professional Programs and AdvisingJohns Hopkins University

Robert A. Witzburg, M.D.Associate Dean and Director of AdmissionsBoston University School of Medicine

Geoffrey H. Young, Ph.D.Associate Dean for AdmissionsAssociate Professor of Psychiatry and HealthBehavior Medical College of Georgia at Georgia Regents University

Steve Zweck-Bronner, J.D.Senior Associate University Counsel University of Colorado Denver

Legal ConsultantsArthur L. Coleman, J.D. EducationCounsel LLC

Kate Lipper, J.D.EducationCounsel LLC

Project StaffAmy N. AddamsRuth Beer BletzingerHope GrayLily May JohnsonRobert Sabalis, Ph.D.Henry Sondheimer, M.D.S. Elizabeth White

roadmap to excellence: key concepts for evaluating the

Documents