publisher colophon
Abstract

Multiple Mini-Interviews (MMIs) are increasingly used in medical school admissions. We previously reported that while under-represented minority (URM) status was not associated with MMI scores, self-designated disadvantaged applicants had lower MMI scores, possibly affecting their matriculation prospects. No studies have examined how URM status or socioeconomic disadvantage (SED) are associated with academic performance following admission through an MMI. We examined the adjusted associations of MMI scores, SED, and URM status with U.S. Medical Licensing Examination Steps 1 and 2 performance and third-year clerkship Honors, measures affecting residency matching. While URM status was not associated with the measures, students with greater SED had lower Step 1 scores and fewer Honors. Students with higher MMI scores had more Step 1 failures, but more Honors. The findings identify areas to address in medical school admissions, student support, and evaluation processes, which is important given the need for a more representative physician workforce.

Key words

Socioeconomic disadvantage, medical school admissions, Multiple Mini-Interview, medical licensing examinations, medical school academic performance, physician workforce diversity, medical school clerkship ratings, racial/ethnic minority groups underrepresented in medicine

[End Page 303]

Many United States (U.S.) medical schools have replaced traditional one-on-one interviews (TIs) with Multiple Mini-Interviews (MMIs), in which applicants work through a series of brief, semi-structured assessment stations, each attended by one or more trained raters.1,2 The shift from TIs to MMIs was spurred by studies from Canada and other non-U.S. countries reporting greater inter-rater reliability for MMIs than for TIs, in turn suggesting MMIs may better distinguish applicants likely to succeed in training.1,3,4 However, little is known about the implications of this change for recruitment of under-represented minority (URM)5 students or students with socioeconomic disadvantage (SED)6 into medical schools and subsequently into the physician workforce.

Two analyses we conducted with data from U.S. medical school applicants indicated possible adverse effects of MMI adoption on the admission prospects of individuals from backgrounds under-represented in the medical profession. The first analysis employed data from 1,420 applicants completing an MMI at the University of California (UC) Davis School of Medicine over three consecutive admission cycles (2011–2013).7 Adjusting for academic metrics and other demographic characteristics, we found that while URM status was not associated with MMI performance, applicants with greater SED (measured by a continuous composite measure using both self-reported and objective information from applications) had significantly lower MMI scores than did their less disadvantaged counterparts. Further, among all student characteristics, MMI score was the strongest predictor of admission offers. We expanded on this line of inquiry in subsequent analyses of data from 4,993 applicants to five public medical schools from 2011–2013: UC Davis, Irvine, Los Angeles, San Diego, and San Francisco (unpublished manuscript, currently in revision at another journal). Two of the five schools employed MMIs, while three used TIs. Adjusting for academic metrics and other demographic characteristics, we found that while URM status was not associated with either MMI or TI performance, self-designated disadvantaged applicants again had lower MMI scores than their non-disadvantaged counterparts, but had higher TI scores.

The findings of our prior studies suggest unfortunate unintended consequences to the ongoing shift from TIs to MMIs, and raise concern given the pressing need to improve the diversity of the physician workforce in the U.S.8,9 The findings also point to a need to examine how interview performance, URM status, and SED are associated with academic performance after matriculation to medical schools employing an MMI. Such analyses would seem particularly prudent for at least two reasons. First, our aforementioned unpublished work suggests that the MMI and TI each reward or "select for" different student characteristics. Thus, it cannot be safely assumed that the findings of studies examining student academic performance following admission through a TI process are applicable to students admitted through an MMI process. Second, recent work (not focused on MMI-based admissions) has revealed troubling inequities in evaluation at U.S. medical schools,10,11 which could adversely affect the career choices and residency matching success of URM students and those with greater SED.

The relatively few prior studies examining students' subsequent performance after admission using the MMI have all been conducted at non-U.S. medical schools.1215 These studies reported that higher MMI scores predict better performance on some [End Page 304] educational measures, such as national licensing examination scores12 and clinical clerkship ratings.14 However, none reported how URM students or students with high SED fared in the process. Indeed, none of the studies even presented descriptive data on student race/ethnicity or socioeconomic status/disadvantage, likely reflecting the limited diversity of non-U.S. medical school classes compared with those in many U.S. schools.1618 Other factors limiting the applicability of findings from non-U.S. studies to U.S. schools include substantial differences among countries in medical school prerequisite coursework, pre-interview application screening approaches, and educational performance measures (e.g., licensing exam content, timing).1921 Taken together, the foregoing observations underscore the need for U.S. medical school studies to examine the educational performance of URM students and students with SED selected through MMI-based admissions processes.

Using data from four consecutive admissions cycles at the University of California Davis (UC Davis) School of Medicine (SOM), we examined the associations of matriculating students' admission MMI scores, URM status, and SED with subsequent failures and scores on U.S. Medical Licensing Examination (USMLE) Steps 1 and 2, and with the number of Honors received in six core third-year clinical clerkships. These objective and subjective performance measures were chosen because each may affect student specialty choice and residency matching success.2224

Methods

We conducted the study activities from February 7, 2017 through March 8, 2017. We obtained ethics approval from the UC Davis institutional review board (protocol #605118).

Study population

Participants were individuals who matriculated into UC Davis SOM during four consecutive years (2011–2014), all of whom had completed an admission MMI. The school provided us with linked anonymized data for all participants.

Application, screening, and MMI invitation and scheduling

Following computer-based screening of American Medical College Application Service (AMCAS) applications based on undergraduate grade point average (GPA) and Medical College Admission Test (MCAT) scores, a subset of applicants was invited to submit a secondary application. A member of the admissions committee evaluated all completed secondary applications for invitation to the MMI based on cumulative GPA, MCAT scores, personal statements, extracurricular activities, recommendation letters, and other characteristics aligned with UC Davis SOM educational missions.

MMI process and scoring

Details regarding the UC Davis SOM MMI have been published previously.25 Briefly, the MMI employed 10 individually scored 10-minute stations, most adapted from commercial content.26 Each station was attended by one rater, who was blinded to participants' applications. All raters received one hour of standardized training, including a review of the admissions process, rater roles and duties, and the need to avoid consideration of legally protected categories of information (e.g., race/ethnicity, gender).27 At some stations, raters interacted directly with applicants, while at others raters observed applicant interactions with actors. Collectively, the [End Page 305] MMI raters assessed integrity/ethics, professionalism, interpersonal communication, diversity/cultural awareness, teamwork, ability to handle stress, and problem solving, using standardized forms. The rater pool included physicians, medical students, other clinicians (e.g., nurses), basic science faculty, patients, and various non-clinician leaders (e.g., deans), professionals (e.g., lawyers), and high-level administrative staff (e.g., curriculum manager). The range of backgrounds reflected the conviction that diverse perspectives are important in selecting future physicians able to work effectively with diverse populations.

Raters at each MMI station scored overall applicant performance using an anchored four-point scale: 0, <25th percentile performance (relative to other applicants); 1, 25th–50th percentile; 2, 51st–75th percentile; or 3, >75th percentile. Raters were instructed to consider both the applicant's communication skills and the content (e.g., comprehensiveness) of their statements. The total MMI score was the mean of each applicant's individual station scores. Scale internal consistency (Cronbach's alpha=0.69) was comparable to that reported elsewhere.12,25,28,29

Student SED, URM status, and other characteristics

UC Davis SOM provided anonymized socio-demographic information from students' AMCAS applications including age, sex, race/ethnicity, self-designated disadvantaged status (yes or no),6 cumulative undergraduate GPA, and total MCAT score. We used the race/ethnicity information to determine under-represented racial/ethnic minority (URM) status. Students who self-reported Black, Southeast Asian, Native American, or Pacific Islander race and/or Hispanic ethnicity were classified as URM.5

Using methods we described previously,7,30 we employed the following additional information from the AMCAS application to derive a validated composite SED score (range 1–99, higher scores=greater SED): (1) AMCAS fee waiver received for medical school application; (2) childhood spent in an underserved area; (3) family assistance program recipient; (4) family income level (categorized as less than $25,000, $25,000–$49,999, $50,000–$74,999, or $75,000 or more); (5) any applicant contribution to family income; (6) any financial need-based scholarship for post-secondary education; (7) percentage of post-secondary education costs contributed by family; and (8) parents' highest level of educational attainment (categorized as less than high school graduate, high school graduate, some college (did not graduate), or college graduate with at least some graduate school).

USMLE Step 1 and Step 2 scores and failures

The three-step USMLE examination is sponsored by the Federation of State Medical Boards and the National Board of Medical Examiners (NBME). In the U.S., physicians with a MD degree are required to pass all three steps before being permitted to practice medicine. Because only 38 study students (9%) had USMLE Step 3 scores available, we focused our analyses on Steps 1 and 2. Step 1 assesses understanding and application of basic science concepts relevant to medical practice (score range 1–300).31 Before January 2014, the minimum Step 1 passing score was 188; it was subsequently increased to 192.32

USMLE Step 2 has two parts. Step 2 Clinical Knowledge (CK) assesses the ability to apply the medical knowledge, skills, and understanding of clinical science needed to provide patient care under supervision (score range 1–300).31 Before July 2014 the minimum passing score for Step 2 CK was 203; it was subsequently increased to 209.31 [End Page 306] The other part of Step 2 is the Clinical Skills (CS) exam, which employs standardized patients to assess information gathering, physical examination, and interpersonal communication abilities.33 Step 2 CS is a pass/fail examination.

The USMLE statistically adjusts for differences in difficulty across years using statistical procedures, and considers scores to be comparable across years within a three to four year window.31

Third-year clerkship Honors

We considered final grades (Honors, Pass, or Fail) in the six core required third-year clerkships at UC Davis SOM: family medicine, internal medicine, obstetrics and gynecology, pediatrics, psychiatry, and surgery. While grading formulas varied among the clerkships, key components included supervising residents' and attending physicians' evaluations and NBME subject ("shelf") examination scores.34 In all clerkships, most students received a Pass, with too few Fail grades for meaningful analysis. Thus, we examined the total number of Honors received (0–6).

Analyses

Analyses were conducted using Stata (version 14.2, StataCorp, College Station, TX). We employed the chi-square test (for categorical variables) and ANOVA test (for continuous variables) to compare characteristics of students across four MMI score categories (<1.5, 1.5–<2, 2–<2.5, or 2.5–3). We employed five separate regression models to examine the adjusted associations of student URM status (no [reference] or yes) and SED category (1–24 [reference], 25–74, or 75–99) (the key independent variables) with the following dependent variables: (1) any USMLE Step 1 failure and (2) any Step 2 failure (inclusive of CK and CS) (logistic regressions);31 (3) Step 1 score and (4) Step 2 CK score (linear regressions); and (5) total number of Honors in the six core third-year clerkships (negative binomial regression). Other independent variables in all models were age (<24 [reference] or ≥24); sex (male [reference] or female); GPA (<3.4 [reference], 3.43.6, >3.6–3.8, or >3.8); total MCAT score percentile (<65 [reference], 66–75, 76–84, 85–90, or >90); total MMI score category (<1.5 [reference], 1.5–<2, 2–<2.5, or 2.5–3); and matriculation year (2011, 2012, 2013, or 2014). We defined statistical significance as p<.05. Our SED measure is not widely available at U.S. medical schools. Thus, to enhance the applicability of our findings, we conducted secondary analyses for all of the study academic outcomes, replacing the SED score with student self-designated disadvantaged status (yes/no) from the AMCAS application, a widely available and widely used marker of disadvantage.

Logistic regression models yield adjusted odds ratios (AORs), while negative binomial regression models yields incidence rate ratios (IRRs). To facilitate interpretation of the logistic regression findings,35 for these models we also report adjusted marginal percentages (the adjusted percentages of Step 1 or Step 2 CK failure associated with each predictor variable category). To facilitate interpretation of the negative binomial regression findings, for these models we also report adjusted marginal effects (the adjusted numbers of core third-year clerkship Honors associated with each predictor variable category).

Results

Applicant characteristics

Table 1 shows the personal characteristics, USMLE Step 1 and 2 performance, and mean number of clerkship Honors for the 402 students [End Page 307]

Table 1. STUDENT CHARACTERISTICS, MULTIPLE MINI-INTERVIEW PERFORMANCE, U.S. MEDICAL LICENSING EXAMINATION PERFORMANCE, AND MEAN NUMBER OF THIRD YEAR CLERKSHIP HONORS aN=401 (missing data for 1 student) bN=391 (missing data for 11 students) cN=279 (123 students had not yet taken Step 2) Abbreviations: MCAT, Medical College Admissions Test; SD, standard deviation; USMLE, United States Medical Licensing Examination
Click for larger view
View full resolution
Table 1.

STUDENT CHARACTERISTICS, MULTIPLE MINI-INTERVIEW PERFORMANCE, U.S. MEDICAL LICENSING EXAMINATION PERFORMANCE, AND MEAN NUMBER OF THIRD YEAR CLERKSHIP HONORS

aN=401 (missing data for 1 student)

bN=391 (missing data for 11 students)

cN=279 (123 students had not yet taken Step 2)

Abbreviations: MCAT, Medical College Admissions Test; SD, standard deviation; USMLE, United States Medical Licensing Examination

matriculating during the study years, overall and by MMI score category. Just over half were female, a third URM, and 35% had SED scores at the 25th percentile or higher. This mirrored the 40% who self-designated as DA. Twenty-four students (6%) had any USMLE Step 1 failure; of these, 22 (92%) had passed by the time of study completion. Fourteen students (5%) had any Step 2 failure, with 12 failing the CK exam and two failing the CS exam; of these, 12 (86%) had passed by the time of study completion. [End Page 308] Student personal characteristics did not differ significantly across MMI score categories. The proportion of students with any USMLE Step 1 failure and the number of clerkship Honors were greater among higher MMI score categories.

The adjusted associations of MMI scores, URM status, and SED with performance measures are shown in Tables 25.

USMLE Step 1 and Step 2 failures (Tables 2 and 3, respectively)

Neither URM status nor SED was associated with failing Step 1 or 2. Higher MMI scores were associated with an increasing risk of USMLE Step 1 failure (adjusted risk 26% in the highest MMI score category versus 2% in the lowest category), but not with Step 2 failure. Higher GPAs were associated with decreasing risk of failing Step 1 (but not Step 2), while higher MCAT scores were associated with decreasing risk of failing Step 2 (but not Step 1).

USMLE Step 1 and Step 2 CK scores (Table 4)

Under-represented minority status was not associated with Step 1 or Step 2 CK scores. Socioeconomic disadvantage was negatively associated with Step 1 score, but not with Step 2 CK score. There were no consistent associations of MMI score with USMLE Step 1 or Step 2 CK scores. Three student characteristics were associated with both Step 1 and Step 2 CK scores: age ≥24 (lower scores relative to younger students); and GPA and MCAT (for each metric, higher USMLE scores for higher vs. lowest categories). By contrast, female gender was associated only with Step 1 scores (lower scores relative to male students).

Number of third-year clerkship Honors (Table 5)

Under-represented minority status was not associated with the number of clerkship Honors, but students with SED scores at the 25th percentile or higher received fewer Honors. Higher MMI scores and higher GPAs were also associated with more Honors, whereas older age was associated with fewer Honors.

Secondary analyses employing student self-designated disadvantaged status in place of SED score

For all of the study outcomes, the findings from analyses employing student self-designated disadvantaged status in place of the SED score were essentially the same as in the primary analyses (data not shown, available from authors on request).

Discussion

To our knowledge, this is the first study conducted at a U.S. medical school examining student academic performance after an MMI-based admissions process. Additionally, it is the first study worldwide to report on the performance of URM medical students and those with SED who were selected with an MMI-based process.

Adjusting for MMI score, undergraduate academic metrics, and other student characteristics, and we found that students with greater SED subsequently received fewer third-year clerkship Honors than their less SED counterparts, and had lower USMLE Step 1 (but not Step 2 CK) scores. To our knowledge, no prior studies have reported on either association. The finding for Step 1 is consistent with extensive work suggesting lower standardized test performance among students with SED, possibly reflecting fewer opportunities for test preparation due to socioeconomic and other challenges.36 This finding underscores the need to closely monitor pre-clinical performance, particularly for students with SED, and to consider proactively offering them additional academic [End Page 309]

Table 2. ADJUSTED ASSOCIATIONS OF STUDENT CHARACTERISTICS AND MULTIPLE MINI-INTERVIEW SCORES WITH ANY U.S. MEDICAL LICENSING EXAMINATION STEP 1 FAILURE aLogistic regression bAnalyses were also adjusted for matriculation year (2011, 2012, 2013, 2014). Abbreviations: CI, confidence interval; MCAT, Medical College Admissions Test; SED, socio economic disadvantage; URM, under-represented racial/ethnic minority; USMLE, United States Medical Licensing Examination.
Click for larger view
View full resolution
Table 2.

ADJUSTED ASSOCIATIONS OF STUDENT CHARACTERISTICS AND MULTIPLE MINI-INTERVIEW SCORES WITH ANY U.S. MEDICAL LICENSING EXAMINATION STEP 1 FAILURE

aLogistic regression

bAnalyses were also adjusted for matriculation year (2011, 2012, 2013, 2014).

Abbreviations: CI, confidence interval; MCAT, Medical College Admissions Test; SED, socio economic disadvantage; URM, under-represented racial/ethnic minority; USMLE, United States Medical Licensing Examination.

[End Page 310]

Table 3. ADJUSTED ASSOCIATIONS OF STUDENT CHARACTERISTICS AND MULTIPLE MINI-INTERVIEW SCORES WITH ANY U.S. MEDICAL LICENSING EXAMINATION STEP 2 FAILURE aLogistic regression bAnalyses were also adjusted for matriculation year (2011, 2012, 2013, 2014). Abbreviations: CI, confidence interval; MCAT, Medical College Admissions Test; URM, under-represented racial/ethnic minority; SED, socioeconomic disadvantage; USMLE, United States Medical Licensing Examination.
Click for larger view
View full resolution
Table 3.

ADJUSTED ASSOCIATIONS OF STUDENT CHARACTERISTICS AND MULTIPLE MINI-INTERVIEW SCORES WITH ANY U.S. MEDICAL LICENSING EXAMINATION STEP 2 FAILURE

aLogistic regression

bAnalyses were also adjusted for matriculation year (2011, 2012, 2013, 2014).

Abbreviations: CI, confidence interval; MCAT, Medical College Admissions Test; URM, under-represented racial/ethnic minority; SED, socioeconomic disadvantage; USMLE, United States Medical Licensing Examination.

support. That SED was not associated with Step 2 CK failures or scores is encouraging, suggesting students with SED were able to overcome whatever academic disadvantages they faced early in medical school.

Nonetheless, students with SED appeared to remain at some academic disadvantage in the clinical years, receiving fewer clerkship Honors. This has important implications [End Page 311]

Table 4. ADJUSTED ASSOCIATIONS OF STUDENT CHARACTERISTICS AND MULTIPLE MINI-NTERVIEW SCORES WITH U.S. MEDICAL LICENSING EXAMINATION STEP 1 AND STEP 2 SCORES aLinear regression bAnalyses were also adjusted for matriculation year (2011, 2012, 2013, 2014). Abbreviations: CI, confidence interval; CK, Clinical Knowledge; MCAT, Medical College Admissions Test; URM, under-represented racial/ethnic minority; SED, socioeconomic disadvantage; USMLE, United States Medical Licensing Examination.
Click for larger view
View full resolution
Table 4.

ADJUSTED ASSOCIATIONS OF STUDENT CHARACTERISTICS AND MULTIPLE MINI-NTERVIEW SCORES WITH U.S. MEDICAL LICENSING EXAMINATION STEP 1 AND STEP 2 SCORES

aLinear regression

bAnalyses were also adjusted for matriculation year (2011, 2012, 2013, 2014).

Abbreviations: CI, confidence interval; CK, Clinical Knowledge; MCAT, Medical College Admissions Test; URM, under-represented racial/ethnic minority; SED, socioeconomic disadvantage; USMLE, United States Medical Licensing Examination.

for disadvantaged students' career trajectories, since clerkship grades may shape student specialty choices, and residency program directors cite clerkship Honors as a key factor in determining their candidate rank order lists for the match.2224 Why students with SED earn fewer clerkship Honors is not clear, but we speculate that implicit biases disfavoring disadvantaged individuals may play a role. Prior studies suggest such biases are prevalent in our society, including among physicians,37,38 yet have received little attention within the medical profession.39 For example, subtle information apparent [End Page 312]

Table 5. ADJUSTED ASSOCIATIONS OF STUDENT CHARACTERISTICS AND MULTIPLE MINI-INTERVIEW SCORES WITH NUMBER OF THIRD-YEAR CLERKSHIP HONORS aNegative binomial regression bAnalyses were also adjusted for matriculation year (2011, 2012, 2013, 2014). Abbreviation: CI, confidence interval; MCAT, Medical College Admissions Test; SED, socioeconomic disadvantage; URM, under-represented racial/ethnic minority.
Click for larger view
View full resolution
Table 5.

ADJUSTED ASSOCIATIONS OF STUDENT CHARACTERISTICS AND MULTIPLE MINI-INTERVIEW SCORES WITH NUMBER OF THIRD-YEAR CLERKSHIP HONORS

aNegative binomial regression

bAnalyses were also adjusted for matriculation year (2011, 2012, 2013, 2014).

Abbreviation: CI, confidence interval; MCAT, Medical College Admissions Test; SED, socioeconomic disadvantage; URM, under-represented racial/ethnic minority.

to clerkship supervisors, such as use of unfamiliar language, could unwittingly lead to lower ratings of students with SED, reflecting the typically large social distance between such students and their clerkship faculty and residents.38,40 Though preliminary, our findings suggest the need to consider additional training for medical school faculty and residents to minimize any potential influences of socioeconomic status-driven biases.

Under-represented minority status was not associated with any of the performance [End Page 313] measures in our study. Several (but not all) previous studies examining USMLE performance found that URM students fared worse on Steps 1 and 2 exams.4146 However, the prior studies differ from ours in several key ways. Most employed data collected over 10 years ago (some more than 20 years ago). Several examined large national samples in which the overall proportion of URM students was far lower than in our study. The absence of a relationship between URM status and third-year clerkship Honors in our study is notable. To our knowledge, only one prior U.S. study examined the association of URM status and clerkship grades. While the study reported lower grades for URM students, it was greatly limited in relying on voluntary student participation and student self-reporting of clerkship grades, and employed data from more than 10 years ago. Thus, we believe our findings are more robust and up to date. In contrast to biases toward individuals with SED, biases toward URM individuals have received more attention in recent years,8,11,39,47,48 which may help to explain why we found no association of URM status with clerkship Honors. Implicit biases against URM students among clerkship supervisors might have been offset by greater awareness of and conscious efforts to minimize such bias. On the other hand, a recent study of 123 U.S. allopathic medical schools demonstrated racial disparities in medical student membership in the Alpha Omega Alpha medical honor society,10 suggesting there remains much work to be done to reduce the impact of race/ethnicity-driven implicit biases in medical education.

We also found that the admission MMI score was independently associated with several study performance measures. The greater risk of failing USMLE Step 1 with higher MMI scores is a novel finding, and likely to have practical relevance given the magnitude of the effect (a 26% adjusted increase in failure for the highest versus lowest MMI score category). While nearly all students who fail USMLE examinations pass them on retake, the few who do not eventually pass cannot practice medicine in the U.S. Even initial failures followed by passing are dismaying and disruptive to affected students, consume faculty and staff time and resources, may alter students' specialty aspirations, and may reduce the chances of matching into certain specialties.2224 In this context, our finding that higher MMI performance was associated with Step 1 failure suggests the need for further studies examining the impact of adopting the MMI process in U.S. medical school admissions. In the meantime, the finding also indicates that schools adopting an admission MMI should carefully monitor Step 1 failure rates, and consider developing and implementing proactive strategies to minimize such failures.

The reasons for more USMLE Step 1 failures among higher MMI performers remain unclear. A plausible explanation is that certain characteristics confer both an advantage in the MMI process and a disadvantage in tests of basic science knowledge such as USMLE Step 1. For example, we and others have reported better MMI performance among students with higher levels of the personality factor extroversion.25,49 Yet a prior study also found more extroverted students had worse performance on pre-clinical written examinations,50 possibly reflecting a tendency to engage in extracurricular activities that could detract from basic science studies.

Our finding that MMI performance did not predict total scores on USMLE Step 1 or Step 2 CK contrasts with the findings of a study from McMaster University in Canada, [End Page 314] in which higher MMI performance was associated with higher scores on Parts 1 and 2 of the Medical Council of Canada Qualifying Examination (MCCQE).12 The differences in findings between the McMaster study and ours may relate to differences in both the content and timing of the respective examinations. The MCCQE Parts 1 and 2 are usually taken immediately after completing medical school and 16 months into residency training, respectively.12 In contrast, USMLE Step 1 and Step 2 CK typically are taken at the end of the second year and during the fourth year of medical school, respectively. Consistent with these differences in timing, the MCCQE examinations have more clinical elements than USMLE Step exams.12 Admissions MMIs may tap student attributes and behaviors that predict performance on more clinically-oriented examinations. Notably, compared with UC Davis, the McMaster MMI invitation screening process weighs GPA more heavily12 and there is less socio-demographic diversity in the applicant pool.18 These and other inter-school differences may explain the differences in findings between the McMaster study and ours.

We also found that students with higher MMI scores received more clerkship Honors, even after adjusting for other factors. High MMI scorers may simply outperform others in the non-test taking aspects of their clerkships. Alternatively, student dispositional factors that confer an advantage in the MMI may also foster certain behaviors that clerkship supervisors weigh heavily in assigning an overall grade, including interpersonal communication. Prior studies indicate that students with higher levels of extroversion, who tend to be "socially ascendant, affectionate, and warm," receive more favorable ratings for interpersonal behavior than their less extroverted counterparts.51,52

Regarding other student characteristics associated with academic performance, we found older students had lower USMLE Step 1 and Step 2 scores and fewer clerkship Honors, adjusting for other factors. The findings might relate to competing (e.g., family) responsibilities which limit time for preparation. Female students had lower Step 1 scores than their male counterparts. In a prior study, we found that female students outperformed men in our MMI, likely due to the tendency for women to more quickly establish interpersonal connections than men.25 The skills that allow women to perform better on the MMI may not align with the skills necessary for the largely non-clinical Step 1 exam. Consistent with this reasoning, we found neither gender nor MMI score was associated with performance on the more clinically oriented Step 2 exam. We also found that higher GPA and MCAT scores were associated with higher USMLE Step 1 and Step 2 CK scores, whereas higher GPA was associated with lower risk of Step 1 failure and more clerkship Honors, and higher MCAT scores were associated with lower risk of USMLE Step 2 CK failure. These findings are broadly consistent with prior work.42,5358

The chief limitation of our study was that it involved a single U.S. medical school. As such, the findings may reflect institution-specific priorities and missions. Whether the findings may generalize to other U.S. medical schools is uncertain. Multi-school collaborative studies would help to address this question. Additionally, while the analysis included several years of matriculating students, the sample size was too small to examine differences in academic performance measures among individual race/ethnicity groups and potentially important interactions among key variables. While we examined several important academic outcomes, others (e.g., breaches in professionalism) [End Page 315] also merit study. Multi-school studies would offer the resources and statistical power required to examine professionalism and other indicators of physician quality.

In conclusion, in the first U.S. medical school study to examine the academic performance of URM students and those with SED selected through a MMI-based admissions process, we found that students with greater SED had lower USMLE Step 1 (but not Step 2) scores than their less disadvantaged counterparts, and received fewer Honors in core third-year clerkships. By contrast, URM status was not associated with performance on study educational measures. We also found that higher MMI performance was associated with greater risk of USMLE Step 1 failure, but also with receiving more Honors in core third-year clerkships. These findings have implications for the ongoing improvement of admissions, student monitoring and support, curricular, and evaluation processes at our institution. Whether the findings extend to other U.S. medical schools is an open question, best addressed by multi-institutional studies.

Anthony Jerant, Mark C. Henderson, Erin Griffin, Efrain Talamantes, Tonya Fancher, Francis Sousa, and Peter Franks

ANTHONY JERANT and PETER FRANKS are affiliated with the Department of Family and Community Medicine at the University of California, Davis, School of Medicine in Sacramento, CA. MARK C. HENDERSON is affiliated with the Division of General Medicine, Department of Internal Medicine at the University of California, Davis, School of Medicine. ERIN GRIFFIN is affiliated with the Research and Evaluation Outcomes Unit at the University of California, Davis, School of Medicine. EFRAIN TALAMANTES and TONYA FANCHER are affiliated with the Division of General Medicine, Department of Internal Medicine, at the University of California, Davis, School of Medicine. FRANCIS SOUSA is affiliated with the the Department of Internal Medicine at the University of California, Davis, School of Medicine.

References

1. Eva KW, Rosenfeld J, Reiter HI, et al. An admissions OSCE: the multiple mini-interview. Med Educ. 2004 Mar;38(3):314–26. PMid:14996341
2. Glazer G, Startsman LF, Bankston K, et al. How many schools adopt interviews during the student admission process across the health professions in the United States of America? J Educ Eval Health Prof. 2016 Feb 27;13:12. http://dx.doi.org/10.3352/jeehp.2016.13.12 PMid: PMC4789565
3. Rees EL, Hawarden AW, Dent G, et al. Evidence regarding the utility of multiple mini-interview (MMI) for selection to undergraduate health programs: A BEME systematic review: BEME Guide No. 37. Med Teach. 2016 May;38(5):443–55. http://dx.doi.org/10.3109/0142159X.2016.1158799 PMid:27050026
4. Pau A, Jeevaratnam K, Chen YS, Fall AA, Khoo C, Nadarajah VD. The Multiple Mini-Interview (MMI) for student selection in health professions training—a systematic review. Med Teach. 2013 Dec;35(12):1027–41. http://dx.doi.org/0.3109/0142159X.2013.829912 PMid:24050709
5. Underrepresented in medicine definition. Association of American Medical Colleges, 2017. Available at: https://www.aamc.org/initiatives/urm/
6. Childhood information. American Medical College Application Service, 2015. Available at: https://services.aamc.org/AMCAS2_2015/WebApp/Help/WebHelp/Disadvantaged_Status.htm
7. Jerant A, Fancher T, Fenton JJ, et al. How medical school applicant race, ethnicity, and socioeconomic status relate to multiple mini-interview-based admissions outcomes: findings from one medical school. Acad Med. 2015 Dec;90(12):1667–74. http://dx.doi.org/10.1097/ACM.0000000000000766 PMid:26017355
8. Saha S, Guiton G, Wimmers PF, et al. Student body racial and ethnic composition and diversity-related outcomes in US medical schools. JAMA. 2008 Sep 10;300(10): 1135–45. [End Page 316] htpp://dx.doi.org/10.1001/jama.300.10.1135 PMid:18780842
9. Iglehart JK. Diversity dynamics—challenges to a representative U.S. medical workforce. N Engl J Med. 2014 Oct 16;371(16):1471–4. http://dx.doi.org/10.1056/NEJMp1408647
10. Boatright D, Ross D, O'Connor P, et al. Racial disparities in medical student membership in the Alpha Omega Alpha Honor Society. JAMA Int Med. Epub ahead of print Mar 7 2017. http://dx.doi.org/10.1001/jamainternmed.2016.9623 PMid:28264091
11. Capers Qt, Clinchot D, McDougle L, Greenwald AG. Implicit Racial Bias in Medical School Admissions. Acad Med. 2017 Mar;92(3):365–9. http://dx.doi.org/10.1097/ACM.0000000000001388 PMid:27680316
12. Eva KW, Reiter HI, Rosenfeld J, et al. Association between a medical school admission process using the multiple mini-interview and national licensing examination scores. JAMA. 2012 Dec 5;308(21):2233–40. http://dx.doi.org/10.1001/jama.2012.36914 PMid:23212501
13. Husbands A, Dowell J. Predictive validity of the Dundee multiple mini-interview. Med Educ. 2013 Jul;47(7):717–25. http://dx.doi.org/10.1111/medu.12193
14. Reiter HI, Eva KW, Rosenfeld J, et al. Multiple mini-interviews predict clerkship and licensing examination performance. Med Educ. 2007 Apr;41(4):378–84. http://dx.doi.org/10.1111/j.1365-2929.2007.02709.x PMid:17430283
15. Eva KW, Reiter HI, Rosenfeld J, et al. The ability of the multiple mini-interview to predict preclerkship performance in medical school. Acad Med. 2004 Oct;79(10 Suppl):S40–2. PMid:15383385
16. Association of American Medical Colleges Facts & Figures 2016. Diversity in medical education. Washington, D.C.: AAMC, 2016. Available at: http://www.aamcdiversityfactsandfigures2016.org/
17. Tiffin PA, Dowell JS, McLachlan JC. Widening access to UK medical education for under-represented socioeconomic groups: modelling the impact of the UKCAT in the 2009 cohort. BMJ (Clinical research ed). 2012 Apr 17;344:e1805. http://dx/doi.org/10.1136/bmj.e1805 PMid:22511300
18. Dhalla IA, Kwong JC, Streiner DL, et al. Characteristics of first-year students in Canadian medical schools. CMAJ. 2002; Apr 16;166(8):1029–35. PMid:100877
19. Medical school in Canada. Wikipedia, 2016. Available at: https://en.wikipedia.org/wiki/Medical_school_in_Canada
20. Medical school in the United Kingdom. Wikipedia, 2017. Available at: https://en.wikipedia.org/wiki/Medical_school_in_the_United_Kingdom
21. Medical school in the United States. Wikipedia, 2017. Available at: https://en.wikipedia.org/wiki/Medical_school_in_the_United_States
22. Results of the 2014 NRMP Program Director Survey. Washington, D.C.: National Resident [End Page 317] Matching Program, 2014. Available at: http://www.nrmp.org/wp-content/uploads/2014/09/PD-Survey-Report-2014.pdf
23. Ginsburg S, Eva K, Regehr G. Do in-training evaluation reports deserve their bad reputations? A study of the reliability and predictive ability of ITER scores and narrative comments. Acad Med. 2013 Oct;88(10):1539–44. http://dx.doi.org10.1097/ACM.0b013e3182a36c3d PMid:23969371
24. Charting outcomes in the match. Characteristics of applicants who matched to their preferred specialty in the 2014 main residency match. Washington, D.C.: National Resident Matching Program, 2014. Available at: http://www.nrmp.org/wp-content/uploads/2014/09/Charting-Outcomes-2014-Final.pdf
25. Jerant A, Griffin E, Rainwater J, et al. Does applicant personality influence multiple mini-interview performance and medical school acceptance offers? Acad Med. 2012 Sep;87(9):1250–9. http://dx.doi/org/10.1097/ACM.0b013e31826102ad PMid:22836836
26. Welcome to ProFit HR. Hamilton, ON, Canada: Advanced Psychometrics for Transitions Inc, 2016. Available at: http://www.profithr.com/
27. Title VII of the Civil Rights Act of 1964. Washington, D.C.: U.S. Equal Employment Opportunity Commission, 1964. Available at: http://www.eeoc.gov/laws/statutes/titlevii.cfm
28. Dowell J, Lynch B, Till H, Kumwenda B, et al. The multiple mini-interview in the U.K. context: 3 years of experience at Dundee. Med Teach. 2012;34(4):297–304. http://dx.org/10.3109/0142159X.2012.652706 PMid:22455698
29. Harris S, Owen C. Discerning quality: using the multiple mini-interview in student selection for the Australian National University Medical School. Med Educ. 2007 Mar;41(3):234–41. http://dx.doi.org/10.1111/j.1365-2929.2007.02682.x PMid:17316207
30. Fenton JJ, Fiscella K, Jerant AF, et al. Reducing medical school admissions disparities in an era of legal restrictions: adjusting for applicant socioeconomic disadvantage. J Health Care Poor Underserved. 2016 Feb;27(1):22–34. http://dx.doi.org/10.1353/hpu.2016.0013 PMid:27763458
31. U USMLE score interpretation guidelines. Philadelphia, PA: National Board of Medical Examiners, 2016. Available at: http://www.usmle.org/pdfs/transcripts/USMLE_Step_Examination_Score_Interpretation_Guidelines.pdf
32. USMLE Scores & Transcripts. Philadelphia, PA: National Board of Medical Examiners, 2017. Available at: http://www.usmle.org/transcripts/
33. USMLE. Step 2 CS. Philadelphia, PA: National Board of Medical Examiners, 2017. Available at: http://www.usmle.org/step-2-cs/
34. Subject examinations. Philadelphia, PA: National board of Medical Examiners, 2017. Available at: http://www.nbme.org/schools/Subject-Exams/
35. Prasad K, Jaeschke R, Wyer P, et al. Tips for teachers of evidence-based medicine: understanding odds ratios and their relationship to risk ratios. J Gen Intern Med. 2008 May;23(5):635–40. http://dx.doi.org/10.1007/s11606-007-0453-4 PMid:2324134 [End Page 318]
36. Davis D, Dorsey JK, Franks RD, et al. Do racial and ethnic group differences in performance on the MCAT exam reflect test bias? Acad Med. 2013 May;88(5):593–602. http://dx.doi.org/10.1097/ACM.0b013e318286803a PMid:23478636
37. Chapman EN, Kaatz A, Carnes M. Physicians and implicit bias: how doctors may unwittingly perpetuate health care disparities. J Gen Intern Med. 2013 Nov;28(11):1504–10. http://dx.doi.org/10.1007/s11606-013-2441-1 PMid:23576243
38. Loignon C, Boudreault-Fournier A, Truchon K, et al. Medical residents reflect on their prejudices toward poverty: a photovoice training project. BMC Med Educ. 2014 Dec 31;14:1050. http://dx.doi.org/10.1186/s12909-014-0274-1c PMid:25551370
39. Magnus SA, Mick SS. Medical schools, affirmative action, and the neglected role of social class. Am J Pub Health. 2000 Aug;90(8):1197–1201. PMid:10936995
40. Lammers J, Galinsky AD, Gordijn EH, et al. Power increases social distance. Soc Psychol Personal Sci. 2011 Aug 15;3(3):282–90. https://doi.org/10.1177/1948550611418679
41. Andriole DA, Jeffe DB. Prematriculation variables associated with suboptimal outcomes for the 1994–1999 cohort of US medical school matriculants. JAMA.2010 Sep 15;304(11):1212–9. http://dx.doi.org/10.1001/jama.2010.1321 PMid:20841535
42. Andriole DA, Jeffe DB. A national cohort study of U.S. medical school students who initially failed Step 1 of the United States Medical Licensing Examination. Acad Med. 2012 Apr;87(4):529–36. http://dx.doi.org/10.1097/ACM.0b013e318248dd9c PMid:22361789
43. Ogunyemi D, Taylor-Harris D. Factors that correlate with the U.S. Medical Licensure Examination Step-2 scores in a diverse medical student population. J Natl Med Assoc. 2005 Sep;97(9):1258–62. PMid:16296216
44. Kleshinski J, Khuder SA, Shapiro JI, et al. Impact of preadmission variables on USMLE step 1 and step 2 performance. Adv Health Sci Educ Theory Pract. 2009 Mar;14(1): 69–78. http://dx.doi.org10.1007/s10459-007-9087-x PMid:17987399
45. Dawson B, Iwamoto CK, Ross LP, et al. Performance on the National Board of Medical Examiners. Part I Examination by men and women of different race and ethnicity. JAMA. 1994 Sep 7;272(9):674–9. PMid:7710487
46. Veloski JJ, Callahan CA, Xu G, et al. Prediction of students' performances on licensing examinations using age, race, sex, undergraduate GPAs, and MCAT scores. Acad Med. 2000 Oct;75(10 Suppl):S28–30. PMid:11031165
47. Ko M, Heslin KC, Edelstein RA, et al. The role of medical education in reducing health care disparities: the first ten years of the UCLA/Drew Medical Education Program. J Gen Intern Med.2007 May;22(5):625–31. [End Page 319] http://dx.doi.org/10.1007/s11606-007-0154-z PMid:1852922
48. Whitla DK, Orfield G, Silen W, et al. Educational benefits of diversity in medical school: a survey of students. Acad Med. 2003 May;78(5):460–6. PMid:12742780
49. Griffin B, Wilson I. Associations between the big five personality factors and multiple mini-interviews. Adv Health Sci Educ Theory Pract. 2012 Aug;17(3):377–88. http://dx.doi.org/10.1007/s10459-011-9316-1
50. Lievens F, Coetsier P, De Fruyt F, et al. Medical students' personality characteristics and academic performance: a five-factor model perspective. Med Educ. 2002 Nov;36(11):1050–6. PMid:12406265
51. Lievens F, Ones DS, Dilchert S. Personality scale validities increase throughout medical school. J Appl Psychol. 2009 Nov;94(6):1514–35. http://dx.doi.org/10.1037/a0016137 PMid:19916659
52. Chibnall JT, Blaskiewicz RJ. Do clinical evaluations in a psychiatry clerkship favor students with positive personality characteristics? Acad Psychiatry. 2008 May–Jun;32(3):199–205. http://dx.doi.org/10.1176/appi.ap.32.3.199 PMid:18467476
53. Casey PM, Palmer BA, Thompson GB, et al. Predictors of medical school clerkship performance: a multispecialty longitudinal analysis of standardized examination scores and clinical assessments. BMC Med Educ. 2016 Apr 27;16:128. https://dx.doi.org/10.1186/s12909-016-0652-y
54. Saguil A, Dong T, Gingerich RJ, et al. Does the MCAT predict medical school and PGY-1 performance? Mil Med. 2015 Apr;180(4 Suppl):4–11. http://dx.doi.org/10.7205/MILMED-D-14-00550 PMid:25850120
55. Basco WT, Jr., Way DP, Gilbert GE, et al. Undergraduate institutional MCAT scores as predictors of USMLE step 1 performance. Acad Med. 2002 Oct;77(10 Suppl):S13–6. PMid:12377692
56. Cuddy MM, Swanson DB, Clauser BE. A multilevel analysis of examinee gender and USMLE step 1 performance. Acad Med. 2008 Oct;83(10 Suppl):S58–62. http://dx.doi.org/10.1097/ACM.0b013e318183cd65
57. Wang-Cheng RM, Fulkerson PK, Barnas GP, et al. Effect of student and preceptor gender on clinical grades in an ambulatory care clerkship. Acad Med. 1995 Apr;70(4): 324–6. PMid:7718067
58. Lee KB, Vaishnavi SN, Lau SK, et al. "Making the grade:" noncognitive predictors of medical students' clinical clerkship grades. J Natl Med Assoc. 2007 Oct;99(10):1138–50. PMid:17987918 [End Page 320]

Additional Information

ISSN
1548-6869
Print ISSN
1049-2089
Pages
303-320
Launched on MUSE
2018-02-27
Open Access
No
Back To Top

This website uses cookies to ensure you get the best experience on our website. Without cookies your experience may not be seamless.