Medical School Performance of Socioeconomically Disadvantaged and Underrepresented Minority Students Matriculating after a Multiple Mini-Interview
Multiple Mini-Interviews (MMIs) are increasingly used in medical school admissions. We previously reported that while under-represented minority (URM) status was not associated with MMI scores, self-designated disadvantaged applicants had lower MMI scores, possibly affecting their matriculation prospects. No studies have examined how URM status or socioeconomic disadvantage (SED) are associated with academic performance following admission through an MMI. We examined the adjusted associations of MMI scores, SED, and URM status with U.S. Medical Licensing Examination Steps 1 and 2 performance and third-year clerkship Honors, measures affecting residency matching. While URM status was not associated with the measures, students with greater SED had lower Step 1 scores and fewer Honors. Students with higher MMI scores had more Step 1 failures, but more Honors. The findings identify areas to address in medical school admissions, student support, and evaluation processes, which is important given the need for a more representative physician workforce.
Socioeconomic disadvantage, medical school admissions, Multiple Mini-Interview, medical licensing examinations, medical school academic performance, physician workforce diversity, medical school clerkship ratings, racial/ethnic minority groups underrepresented in medicine
[End Page 303]
Many United States (U.S.) medical schools have replaced traditional one-on-one interviews (TIs) with Multiple Mini-Interviews (MMIs), in which applicants work through a series of brief, semi-structured assessment stations, each attended by one or more trained raters.1,2 The shift from TIs to MMIs was spurred by studies from Canada and other non-U.S. countries reporting greater inter-rater reliability for MMIs than for TIs, in turn suggesting MMIs may better distinguish applicants likely to succeed in training.1,3,4 However, little is known about the implications of this change for recruitment of under-represented minority (URM)5 students or students with socioeconomic disadvantage (SED)6 into medical schools and subsequently into the physician workforce.
Two analyses we conducted with data from U.S. medical school applicants indicated possible adverse effects of MMI adoption on the admission prospects of individuals from backgrounds under-represented in the medical profession. The first analysis employed data from 1,420 applicants completing an MMI at the University of California (UC) Davis School of Medicine over three consecutive admission cycles (2011–2013).7 Adjusting for academic metrics and other demographic characteristics, we found that while URM status was not associated with MMI performance, applicants with greater SED (measured by a continuous composite measure using both self-reported and objective information from applications) had significantly lower MMI scores than did their less disadvantaged counterparts. Further, among all student characteristics, MMI score was the strongest predictor of admission offers. We expanded on this line of inquiry in subsequent analyses of data from 4,993 applicants to five public medical schools from 2011–2013: UC Davis, Irvine, Los Angeles, San Diego, and San Francisco (unpublished manuscript, currently in revision at another journal). Two of the five schools employed MMIs, while three used TIs. Adjusting for academic metrics and other demographic characteristics, we found that while URM status was not associated with either MMI or TI performance, self-designated disadvantaged applicants again had lower MMI scores than their non-disadvantaged counterparts, but had higher TI scores.
The findings of our prior studies suggest unfortunate unintended consequences to the ongoing shift from TIs to MMIs, and raise concern given the pressing need to improve the diversity of the physician workforce in the U.S.8,9 The findings also point to a need to examine how interview performance, URM status, and SED are associated with academic performance after matriculation to medical schools employing an MMI. Such analyses would seem particularly prudent for at least two reasons. First, our aforementioned unpublished work suggests that the MMI and TI each reward or "select for" different student characteristics. Thus, it cannot be safely assumed that the findings of studies examining student academic performance following admission through a TI process are applicable to students admitted through an MMI process. Second, recent work (not focused on MMI-based admissions) has revealed troubling inequities in evaluation at U.S. medical schools,10,11 which could adversely affect the career choices and residency matching success of URM students and those with greater SED.
The relatively few prior studies examining students' subsequent performance after admission using the MMI have all been conducted at non-U.S. medical schools.12–15 These studies reported that higher MMI scores predict better performance on some [End Page 304] educational measures, such as national licensing examination scores12 and clinical clerkship ratings.14 However, none reported how URM students or students with high SED fared in the process. Indeed, none of the studies even presented descriptive data on student race/ethnicity or socioeconomic status/disadvantage, likely reflecting the limited diversity of non-U.S. medical school classes compared with those in many U.S. schools.16–18 Other factors limiting the applicability of findings from non-U.S. studies to U.S. schools include substantial differences among countries in medical school prerequisite coursework, pre-interview application screening approaches, and educational performance measures (e.g., licensing exam content, timing).19–21 Taken together, the foregoing observations underscore the need for U.S. medical school studies to examine the educational performance of URM students and students with SED selected through MMI-based admissions processes.
Using data from four consecutive admissions cycles at the University of California Davis (UC Davis) School of Medicine (SOM), we examined the associations of matriculating students' admission MMI scores, URM status, and SED with subsequent failures and scores on U.S. Medical Licensing Examination (USMLE) Steps 1 and 2, and with the number of Honors received in six core third-year clinical clerkships. These objective and subjective performance measures were chosen because each may affect student specialty choice and residency matching success.22–24
We conducted the study activities from February 7, 2017 through March 8, 2017. We obtained ethics approval from the UC Davis institutional review board (protocol #605118).
Participants were individuals who matriculated into UC Davis SOM during four consecutive years (2011–2014), all of whom had completed an admission MMI. The school provided us with linked anonymized data for all participants.
Application, screening, and MMI invitation and scheduling
Following computer-based screening of American Medical College Application Service (AMCAS) applications based on undergraduate grade point average (GPA) and Medical College Admission Test (MCAT) scores, a subset of applicants was invited to submit a secondary application. A member of the admissions committee evaluated all completed secondary applications for invitation to the MMI based on cumulative GPA, MCAT scores, personal statements, extracurricular activities, recommendation letters, and other characteristics aligned with UC Davis SOM educational missions.
MMI process and scoring
Details regarding the UC Davis SOM MMI have been published previously.25 Briefly, the MMI employed 10 individually scored 10-minute stations, most adapted from commercial content.26 Each station was attended by one rater, who was blinded to participants' applications. All raters received one hour of standardized training, including a review of the admissions process, rater roles and duties, and the need to avoid consideration of legally protected categories of information (e.g., race/ethnicity, gender).27 At some stations, raters interacted directly with applicants, while at others raters observed applicant interactions with actors. Collectively, the [End Page 305] MMI raters assessed integrity/ethics, professionalism, interpersonal communication, diversity/cultural awareness, teamwork, ability to handle stress, and problem solving, using standardized forms. The rater pool included physicians, medical students, other clinicians (e.g., nurses), basic science faculty, patients, and various non-clinician leaders (e.g., deans), professionals (e.g., lawyers), and high-level administrative staff (e.g., curriculum manager). The range of backgrounds reflected the conviction that diverse perspectives are important in selecting future physicians able to work effectively with diverse populations.
Raters at each MMI station scored overall applicant performance using an anchored four-point scale: 0, <25th percentile performance (relative to other applicants); 1, 25th–50th percentile; 2, 51st–75th percentile; or 3, >75th percentile. Raters were instructed to consider both the applicant's communication skills and the content (e.g., comprehensiveness) of their statements. The total MMI score was the mean of each applicant's individual station scores. Scale internal consistency (Cronbach's alpha=0.69) was comparable to that reported elsewhere.12,25,28,29
Student SED, URM status, and other characteristics
UC Davis SOM provided anonymized socio-demographic information from students' AMCAS applications including age, sex, race/ethnicity, self-designated disadvantaged status (yes or no),6 cumulative undergraduate GPA, and total MCAT score. We used the race/ethnicity information to determine under-represented racial/ethnic minority (URM) status. Students who self-reported Black, Southeast Asian, Native American, or Pacific Islander race and/or Hispanic ethnicity were classified as URM.5
Using methods we described previously,7,30 we employed the following additional information from the AMCAS application to derive a validated composite SED score (range 1–99, higher scores=greater SED): (1) AMCAS fee waiver received for medical school application; (2) childhood spent in an underserved area; (3) family assistance program recipient; (4) family income level (categorized as less than $25,000, $25,000–$49,999, $50,000–$74,999, or $75,000 or more); (5) any applicant contribution to family income; (6) any financial need-based scholarship for post-secondary education; (7) percentage of post-secondary education costs contributed by family; and (8) parents' highest level of educational attainment (categorized as less than high school graduate, high school graduate, some college (did not graduate), or college graduate with at least some graduate school).
USMLE Step 1 and Step 2 scores and failures
The three-step USMLE examination is sponsored by the Federation of State Medical Boards and the National Board of Medical Examiners (NBME). In the U.S., physicians with a MD degree are required to pass all three steps before being permitted to practice medicine. Because only 38 study students (9%) had USMLE Step 3 scores available, we focused our analyses on Steps 1 and 2. Step 1 assesses understanding and application of basic science concepts relevant to medical practice (score range 1–300).31 Before January 2014, the minimum Step 1 passing score was 188; it was subsequently increased to 192.32
USMLE Step 2 has two parts. Step 2 Clinical Knowledge (CK) assesses the ability to apply the medical knowledge, skills, and understanding of clinical science needed to provide patient care under supervision (score range 1–300).31 Before July 2014 the minimum passing score for Step 2 CK was 203; it was subsequently increased to 209.31 [End Page 306] The other part of Step 2 is the Clinical Skills (CS) exam, which employs standardized patients to assess information gathering, physical examination, and interpersonal communication abilities.33 Step 2 CS is a pass/fail examination.
The USMLE statistically adjusts for differences in difficulty across years using statistical procedures, and considers scores to be comparable across years within a three to four year window.31
Third-year clerkship Honors
We considered final grades (Honors, Pass, or Fail) in the six core required third-year clerkships at UC Davis SOM: family medicine, internal medicine, obstetrics and gynecology, pediatrics, psychiatry, and surgery. While grading formulas varied among the clerkships, key components included supervising residents' and attending physicians' evaluations and NBME subject ("shelf") examination scores.34 In all clerkships, most students received a Pass, with too few Fail grades for meaningful analysis. Thus, we examined the total number of Honors received (0–6).
Analyses were conducted using Stata (version 14.2, StataCorp, College Station, TX). We employed the chi-square test (for categorical variables) and ANOVA test (for continuous variables) to compare characteristics of students across four MMI score categories (<1.5, 1.5–<2, 2–<2.5, or 2.5–3). We employed five separate regression models to examine the adjusted associations of student URM status (no [reference] or yes) and SED category (1–24 [reference], 25–74, or 75–99) (the key independent variables) with the following dependent variables: (1) any USMLE Step 1 failure and (2) any Step 2 failure (inclusive of CK and CS) (logistic regressions);31 (3) Step 1 score and (4) Step 2 CK score (linear regressions); and (5) total number of Honors in the six core third-year clerkships (negative binomial regression). Other independent variables in all models were age (<24 [reference] or ≥24); sex (male [reference] or female); GPA (<3.4 [reference], 3.43.6, >3.6–3.8, or >3.8); total MCAT score percentile (<65 [reference], 66–75, 76–84, 85–90, or >90); total MMI score category (<1.5 [reference], 1.5–<2, 2–<2.5, or 2.5–3); and matriculation year (2011, 2012, 2013, or 2014). We defined statistical significance as p<.05. Our SED measure is not widely available at U.S. medical schools. Thus, to enhance the applicability of our findings, we conducted secondary analyses for all of the study academic outcomes, replacing the SED score with student self-designated disadvantaged status (yes/no) from the AMCAS application, a widely available and widely used marker of disadvantage.
Logistic regression models yield adjusted odds ratios (AORs), while negative binomial regression models yields incidence rate ratios (IRRs). To facilitate interpretation of the logistic regression findings,35 for these models we also report adjusted marginal percentages (the adjusted percentages of Step 1 or Step 2 CK failure associated with each predictor variable category). To facilitate interpretation of the negative binomial regression findings, for these models we also report adjusted marginal effects (the adjusted numbers of core third-year clerkship Honors associated with each predictor variable category).
Table 1 shows the personal characteristics, USMLE Step 1 and 2 performance, and mean number of clerkship Honors for the 402 students [End Page 307]
matriculating during the study years, overall and by MMI score category. Just over half were female, a third URM, and 35% had SED scores at the 25th percentile or higher. This mirrored the 40% who self-designated as DA. Twenty-four students (6%) had any USMLE Step 1 failure; of these, 22 (92%) had passed by the time of study completion. Fourteen students (5%) had any Step 2 failure, with 12 failing the CK exam and two failing the CS exam; of these, 12 (86%) had passed by the time of study completion. [End Page 308] Student personal characteristics did not differ significantly across MMI score categories. The proportion of students with any USMLE Step 1 failure and the number of clerkship Honors were greater among higher MMI score categories.
Neither URM status nor SED was associated with failing Step 1 or 2. Higher MMI scores were associated with an increasing risk of USMLE Step 1 failure (adjusted risk 26% in the highest MMI score category versus 2% in the lowest category), but not with Step 2 failure. Higher GPAs were associated with decreasing risk of failing Step 1 (but not Step 2), while higher MCAT scores were associated with decreasing risk of failing Step 2 (but not Step 1).
USMLE Step 1 and Step 2 CK scores (Table 4)
Under-represented minority status was not associated with Step 1 or Step 2 CK scores. Socioeconomic disadvantage was negatively associated with Step 1 score, but not with Step 2 CK score. There were no consistent associations of MMI score with USMLE Step 1 or Step 2 CK scores. Three student characteristics were associated with both Step 1 and Step 2 CK scores: age ≥24 (lower scores relative to younger students); and GPA and MCAT (for each metric, higher USMLE scores for higher vs. lowest categories). By contrast, female gender was associated only with Step 1 scores (lower scores relative to male students).
Number of third-year clerkship Honors (Table 5)
Under-represented minority status was not associated with the number of clerkship Honors, but students with SED scores at the 25th percentile or higher received fewer Honors. Higher MMI scores and higher GPAs were also associated with more Honors, whereas older age was associated with fewer Honors.
Secondary analyses employing student self-designated disadvantaged status in place of SED score
For all of the study outcomes, the findings from analyses employing student self-designated disadvantaged status in place of the SED score were essentially the same as in the primary analyses (data not shown, available from authors on request).
To our knowledge, this is the first study conducted at a U.S. medical school examining student academic performance after an MMI-based admissions process. Additionally, it is the first study worldwide to report on the performance of URM medical students and those with SED who were selected with an MMI-based process.
Adjusting for MMI score, undergraduate academic metrics, and other student characteristics, and we found that students with greater SED subsequently received fewer third-year clerkship Honors than their less SED counterparts, and had lower USMLE Step 1 (but not Step 2 CK) scores. To our knowledge, no prior studies have reported on either association. The finding for Step 1 is consistent with extensive work suggesting lower standardized test performance among students with SED, possibly reflecting fewer opportunities for test preparation due to socioeconomic and other challenges.36 This finding underscores the need to closely monitor pre-clinical performance, particularly for students with SED, and to consider proactively offering them additional academic [End Page 309]
[End Page 310]
support. That SED was not associated with Step 2 CK failures or scores is encouraging, suggesting students with SED were able to overcome whatever academic disadvantages they faced early in medical school.
Nonetheless, students with SED appeared to remain at some academic disadvantage in the clinical years, receiving fewer clerkship Honors. This has important implications [End Page 311]
for disadvantaged students' career trajectories, since clerkship grades may shape student specialty choices, and residency program directors cite clerkship Honors as a key factor in determining their candidate rank order lists for the match.22–24 Why students with SED earn fewer clerkship Honors is not clear, but we speculate that implicit biases disfavoring disadvantaged individuals may play a role. Prior studies suggest such biases are prevalent in our society, including among physicians,37,38 yet have received little attention within the medical profession.39 For example, subtle information apparent [End Page 312]
to clerkship supervisors, such as use of unfamiliar language, could unwittingly lead to lower ratings of students with SED, reflecting the typically large social distance between such students and their clerkship faculty and residents.38,40 Though preliminary, our findings suggest the need to consider additional training for medical school faculty and residents to minimize any potential influences of socioeconomic status-driven biases.
Under-represented minority status was not associated with any of the performance [End Page 313] measures in our study. Several (but not all) previous studies examining USMLE performance found that URM students fared worse on Steps 1 and 2 exams.41–46 However, the prior studies differ from ours in several key ways. Most employed data collected over 10 years ago (some more than 20 years ago). Several examined large national samples in which the overall proportion of URM students was far lower than in our study. The absence of a relationship between URM status and third-year clerkship Honors in our study is notable. To our knowledge, only one prior U.S. study examined the association of URM status and clerkship grades. While the study reported lower grades for URM students, it was greatly limited in relying on voluntary student participation and student self-reporting of clerkship grades, and employed data from more than 10 years ago. Thus, we believe our findings are more robust and up to date. In contrast to biases toward individuals with SED, biases toward URM individuals have received more attention in recent years,8,11,39,47,48 which may help to explain why we found no association of URM status with clerkship Honors. Implicit biases against URM students among clerkship supervisors might have been offset by greater awareness of and conscious efforts to minimize such bias. On the other hand, a recent study of 123 U.S. allopathic medical schools demonstrated racial disparities in medical student membership in the Alpha Omega Alpha medical honor society,10 suggesting there remains much work to be done to reduce the impact of race/ethnicity-driven implicit biases in medical education.
We also found that the admission MMI score was independently associated with several study performance measures. The greater risk of failing USMLE Step 1 with higher MMI scores is a novel finding, and likely to have practical relevance given the magnitude of the effect (a 26% adjusted increase in failure for the highest versus lowest MMI score category). While nearly all students who fail USMLE examinations pass them on retake, the few who do not eventually pass cannot practice medicine in the U.S. Even initial failures followed by passing are dismaying and disruptive to affected students, consume faculty and staff time and resources, may alter students' specialty aspirations, and may reduce the chances of matching into certain specialties.22–24 In this context, our finding that higher MMI performance was associated with Step 1 failure suggests the need for further studies examining the impact of adopting the MMI process in U.S. medical school admissions. In the meantime, the finding also indicates that schools adopting an admission MMI should carefully monitor Step 1 failure rates, and consider developing and implementing proactive strategies to minimize such failures.
The reasons for more USMLE Step 1 failures among higher MMI performers remain unclear. A plausible explanation is that certain characteristics confer both an advantage in the MMI process and a disadvantage in tests of basic science knowledge such as USMLE Step 1. For example, we and others have reported better MMI performance among students with higher levels of the personality factor extroversion.25,49 Yet a prior study also found more extroverted students had worse performance on pre-clinical written examinations,50 possibly reflecting a tendency to engage in extracurricular activities that could detract from basic science studies.
Our finding that MMI performance did not predict total scores on USMLE Step 1 or Step 2 CK contrasts with the findings of a study from McMaster University in Canada, [End Page 314] in which higher MMI performance was associated with higher scores on Parts 1 and 2 of the Medical Council of Canada Qualifying Examination (MCCQE).12 The differences in findings between the McMaster study and ours may relate to differences in both the content and timing of the respective examinations. The MCCQE Parts 1 and 2 are usually taken immediately after completing medical school and 16 months into residency training, respectively.12 In contrast, USMLE Step 1 and Step 2 CK typically are taken at the end of the second year and during the fourth year of medical school, respectively. Consistent with these differences in timing, the MCCQE examinations have more clinical elements than USMLE Step exams.12 Admissions MMIs may tap student attributes and behaviors that predict performance on more clinically-oriented examinations. Notably, compared with UC Davis, the McMaster MMI invitation screening process weighs GPA more heavily12 and there is less socio-demographic diversity in the applicant pool.18 These and other inter-school differences may explain the differences in findings between the McMaster study and ours.
We also found that students with higher MMI scores received more clerkship Honors, even after adjusting for other factors. High MMI scorers may simply outperform others in the non-test taking aspects of their clerkships. Alternatively, student dispositional factors that confer an advantage in the MMI may also foster certain behaviors that clerkship supervisors weigh heavily in assigning an overall grade, including interpersonal communication. Prior studies indicate that students with higher levels of extroversion, who tend to be "socially ascendant, affectionate, and warm," receive more favorable ratings for interpersonal behavior than their less extroverted counterparts.51,52
Regarding other student characteristics associated with academic performance, we found older students had lower USMLE Step 1 and Step 2 scores and fewer clerkship Honors, adjusting for other factors. The findings might relate to competing (e.g., family) responsibilities which limit time for preparation. Female students had lower Step 1 scores than their male counterparts. In a prior study, we found that female students outperformed men in our MMI, likely due to the tendency for women to more quickly establish interpersonal connections than men.25 The skills that allow women to perform better on the MMI may not align with the skills necessary for the largely non-clinical Step 1 exam. Consistent with this reasoning, we found neither gender nor MMI score was associated with performance on the more clinically oriented Step 2 exam. We also found that higher GPA and MCAT scores were associated with higher USMLE Step 1 and Step 2 CK scores, whereas higher GPA was associated with lower risk of Step 1 failure and more clerkship Honors, and higher MCAT scores were associated with lower risk of USMLE Step 2 CK failure. These findings are broadly consistent with prior work.42,53–58
The chief limitation of our study was that it involved a single U.S. medical school. As such, the findings may reflect institution-specific priorities and missions. Whether the findings may generalize to other U.S. medical schools is uncertain. Multi-school collaborative studies would help to address this question. Additionally, while the analysis included several years of matriculating students, the sample size was too small to examine differences in academic performance measures among individual race/ethnicity groups and potentially important interactions among key variables. While we examined several important academic outcomes, others (e.g., breaches in professionalism) [End Page 315] also merit study. Multi-school studies would offer the resources and statistical power required to examine professionalism and other indicators of physician quality.
In conclusion, in the first U.S. medical school study to examine the academic performance of URM students and those with SED selected through a MMI-based admissions process, we found that students with greater SED had lower USMLE Step 1 (but not Step 2) scores than their less disadvantaged counterparts, and received fewer Honors in core third-year clerkships. By contrast, URM status was not associated with performance on study educational measures. We also found that higher MMI performance was associated with greater risk of USMLE Step 1 failure, but also with receiving more Honors in core third-year clerkships. These findings have implications for the ongoing improvement of admissions, student monitoring and support, curricular, and evaluation processes at our institution. Whether the findings extend to other U.S. medical schools is an open question, best addressed by multi-institutional studies.
ANTHONY JERANT and PETER FRANKS are affiliated with the Department of Family and Community Medicine at the University of California, Davis, School of Medicine in Sacramento, CA. MARK C. HENDERSON is affiliated with the Division of General Medicine, Department of Internal Medicine at the University of California, Davis, School of Medicine. ERIN GRIFFIN is affiliated with the Research and Evaluation Outcomes Unit at the University of California, Davis, School of Medicine. EFRAIN TALAMANTES and TONYA FANCHER are affiliated with the Division of General Medicine, Department of Internal Medicine, at the University of California, Davis, School of Medicine. FRANCIS SOUSA is affiliated with the the Department of Internal Medicine at the University of California, Davis, School of Medicine.