Multiple Mini Interviews vs Traditional Interviews: Investigating Racial and Socioeconomic Differences in Interview Processes

Background

Established in 1891, The University of Texas Medical Branch John Sealy School of Medicine (JSSOM), an allopathic medical school in Galveston, Texas, USA, annually receives over 5000 applications, interviewing no more than 1100 candidates to select a marticulatingclass of 230 students. With 24 voting faculty members on the admissions committee (AC) and no student representation, the AC encountered in maintaining consistency and fairness across the interview process. In the 2019–2020 admissions cycle, the JSSOM admissions committee identified noteworthy inconsistencies in interview scoring, discussion topics, and interviewer comments stemming from an unstructured interview format. This issue, while specific to JSSOM, resonates with a broader trend within medical school admissions. Discussions and literature within the field suggest inconsistencies in interview practices are not unique to our institution as medical education has grappled with standardizing their interview processes, potentially leading to variations in candidate evaluation.¹ This inconsistency raises concerns about the fairness and reliability of the admissions process on a larger scale. Medical schools are committed to fairness by treating all applicants justly and equitably, striving to eliminate systemic barriers that disadvantage marginalized groups. Addressing these challenges is essential to aligning the selection process with best practices and ensuring a level playing field for all prospective medical students.

Importance of Interviews in the Admissions Process

Medical schools seek applicants with strong interpersonal and intrapersonal characteristics along with academic qualifications. A physician’s ability to communicate and connect effectively with patients is fundamental in providing quality healthcare.² Empathy, active listening, and effective verbal and non-verbal communication are pivotal in establishing trust and rapport, positively influencing patient’s well-being and satisfaction with care.³ Moreover, physicians possessing strong interpersonal skills may be better equipped to collaborate within multidisciplinary healthcare teams, facilitate comprehensive patient care, and improve overall health outcomes. Intrapersonal attributes such as self-awareness, resilience, and ethical discernment are vital for a physician’s personal and professional growth. For example, resilience equips physicians with the capacity to cope with the demands and stresses of the medical profession. Sound ethical discernment aids in navigating complex clinical scenarios and making decisions that align with the patient’s best interests, upholding the medical profession’s integrity. Relationship-building skills, attitude, and empathy significantly influence the quality of patient interviews³ as observed by medical student faculty, suggesting that interpersonal and intrapersonal characteristics are pertinent for ensuring effective communication and patient care. Thus, medical schools employ a multifaceted approach to assess interpersonal and intrapersonal characteristics in prospective medical students. The interview process is one method medical schools may use to evaluate applicants’ personal characteristics.

The two most popular interview formats are traditional and multiple-mini interview (MMI). In a traditional interview, a single interviewer or a panel of interviewers asks applicants a series of questions, typically focusing on their background, experiences, motivations, and ethical reasoning. Interviews can be structured or unstructured and focus on applicants’ qualifications and personal qualities. Structured traditional interviews involve a predetermined set of questions that interviewers consistently ask. Questions assess specific competencies and skills deemed relevant to the programs’ requirements. Unstructured traditional interviews provide a more flexible and open-ended format, frequently yielding less consistency across applicants and less reliability in scoring. Structured interviews are more valid and reliable for assessing candidate competencies than unstructured interviews. MMIs are a series of short, structured, timed stations, each featuring a different scenario or question. Applicants rotate through these stations, encountering scenarios that assess interpersonal and intrapersonal qualities, often more standardized and objective than traditional interviews. While both formats aim to evaluate an applicant’s suitability for a career in medicine, they do so by using distinct approaches, with traditional interviews focusing on comprehensive discussions and MMIs employing a multifaceted, scenario-based assessment.⁴

Importance of Interviews for Diversity

Finding a way to assess the interpersonal and intrapersonal characteristics of applicants accurately is critical given the recent Supreme Court decisions in Students for Fair Admissions, Inc. (SFFA) v. University of North Carolina and SFFA v. President & Fellows of Harvard College, which ended race-conscious admissions, or affirmative action, and upheld holistic admissions practice. While most medical schools began to embrace holistic admissions over a decade ago, implementing holistic admissions without affirmative action presents a substantial challenge due to the deeply entrenched inequalities in access to quality education in the United States. Furthermore, admissions committees may rely heavily on academic metrics such as the Medical College Admission Test (MCAT) and undergraduate Grade Point Average (GPA). While these metrics have proven significant predictors of success through the first year of medical school,⁵ they fail to consider a broader range of qualities and neglect promising candidates who possess unique perspectives underrepresented and economically disadvantaged backgrounds.⁶ Medical schools have established admission criteria to mitigate the effects of traditional inequities by focusing on applicants’ interpersonal and intrapersonal characteristics to account better for these systemic problems.

Potential Bias and Reliability Concerns

Concerned about potential bias and reliability of interview scores, the AC began to consider more structured interviewing approaches and ultimately convened an MMI task force to determine feasibility and create an implementation plan. In the 2020–2021 admission cycle, the admission committee transformed its interview format substantially from an unstructured traditional model to a semi-structured approach, introducing two additional mandatory questions for each faculty interview. This adjustment aimed to deepen the assessment process and make it more uniform but lacked alignment with program-specific or AAMC competencies for entering medical students, resulting in inconsistencies that hampered the unstructured approach. Thus, the committee implemented an MMI based on task force recommendations in the subsequent 2021–2022 and 2022–2023 admission cycles. Implementation challenges included faculty training, faculty buy-in, and virtual platform selection. After the first year of implementation, revisions were made based on applicant and interview feedback.

Traditional vs Multiple-Mini Interviews

MMIs are similar to the Objective Structured Clinical Evaluation (OSCE) model commonly used in medical school programs. Unlike traditional interviews evaluating multiple competencies within a single interview/station, MMIs adopt an OSCE-inspired approach by assessing one competency or task per station. This method allows for a more granular and specific evaluation of candidate abilities, improving the precision of scoring by reducing subjectivity and enhancing reliability in assessing key competencies. Over the past two decades, the MMI has gained popularity in medical school admissions because of its reliability in evaluating noncognitive characteristics with minimal nonsignificant racial and gender differences in scoring.⁷ Using independent evaluators at each station promotes more objectivity, and applicants receive a fresh start for each question. Scholars have found that MMIs are reliable, content-valid, and practical assessment tools; however, they may be less feasible to implement than traditional interview methods due to their resource-intensive nature.¹ However, socioeconomic factors such as educational background and financial resources have been shown to influence MMI performance. Studies have indicated that students from higher-income backgrounds tend to have significantly higher scores.⁵ The findings suggest that socioeconomic status may be associated with perceived differences in communication skills and life experiences. Literature supports structured traditional interviews, as they provided more precise and consistent evaluation criteria.⁴ Structured interviews that blinded the interviewers to candidates’ academic scores also helped reduce preference toward higher scores.³ This study has two primary aims: first, to compare the reliability of traditional interviews and Multiple Mini Interviews (MMIs) in evaluating medical school applicants, and second, to investigate whether applicants from different racial and socioeconomic backgrounds face potential disadvantages in the admissions process. The University of Texas Medical Branch (UTMB) Institutional Review Board and John Sealy School of Medicine Educational Research Committee reviewed this study and determined it exempt due to de-identified historical data, requiring no informed consent from participants.

Casper Test

The Altus Assessments Casper test is an online situational judgment test designed to evaluate an applicant’s non-cognitive skills, including ethical judgment, communication, and professionalism. The Casper test presents candidates with hypothetical scenarios and assesses their responses, providing a standardized measure of ethical judgment, communication skills, and professionalism. However, studies have identified significant minor to moderate differences in Casper scores across ethnic, gender, and socioeconomic groups,⁸ underscoring the need to interpret its results within the context of these disparities. As a standardized screening tool, Casper was intended to be used in conjunction with other academic metrics for the entire applicant pool prior to the interview and generally exhibited more significant mean differences than interviews.

The assumption underlying this study is that Casper serves as a gold standard for assessing these non-cognitive attributes, given its widespread adoption and focus on key competencies essential for medical professionals.

Methods

Preexisting quantitative data from six admissions cycles (2018–2023) were analyzed using SPSS, resulting in a total sample size of 5799 interviewees. To assess the strength of the relationship between the MMI, Casper scores, and the outcomes from both traditional interviews and the MMI, the authors performed Spearman correlation analyses were initially conducted (Table 1). These analyses revealed that the MMI had a stronger correlation with the Casper score compared to traditional interviews. Notably, this correlation increased with semi-structured interviews, suggesting that the more structured the interview format, the stronger the relationship. However, caution is advised when using CASPer, as its effectiveness in predicting outcomes may vary depending on the context and the demographics of applicants.⁸ As outlined in Table 1, these findings helped illuminate the interrelationships among key evaluation metrics, contributing to a more nuanced understanding of their combined predictive power. In this analysis, correlation coefficients of 0.20, 0.40, and 0.60 were used to indicate small, moderate, and large effect sizes, respectively.

Table 1 Correlation Between Casper Scores and Interview Scores

A second analysis focused on exploring demographic differences among four comparison groups via T-tests (Table 2). The groups were: 1) underrepresented in medicine (URM) applicants (including Black or African American, Latinx, American Indian and Alaska Native, and Native Hawaiian and other Pacific Islander) and non-URM applicants; 2) African American and non-URM applicants; 3) Hispanic and non-URM applicants; and 4) disadvantaged and non-disadvantaged applicants. The evaluative criteria for comparison included MMI score, traditional interview score, Casper score, MCAT total score, overall GPA (undergraduate and graduate), science GPA, and science GPA. Cohen’s d was calculated for each criterion and then averaged to compare differences between interview formats. Cohen’s d values of 0.20, 0.50, and 0.80 were used to define small, moderate, and large effect sizes, respectively.⁹

Table 2 Demographic Comparison of Average Effect Size (Cohen’s d) 2018–2023

Results

From 2018 to 2020, traditional unstructured interviews were used, and the relationship between Casper and the interview was weak, which supported data suggesting that unstructured interviews were less reliable. Results did not support claims Due to ongoing concerns about bias and the lack of helpful information, the admissions committee transitioned to a nine-station MMI process in 2022 and, subsequently, a seven-station MMI process in 2023. This transition yielded a stronger relationship, providing evidence that MMIs had a stronger relationship with Casper than traditional interview approaches. Results did not support claims that MMIs were superior to traditional interviews.

Both MMIs and traditional interviews appeared equally in reducing group ethnic and seriocomic group differences (URM versus non-URM, African American versus non-URM, Hispanic versus non-URM, and disadvantaged versus non-disadvantaged). MMIs reduced differences compared with traditional interviews for African American candidates and slightly increased differences for URM, Hispanic, and disadvantaged candidates; however, the effect size was small. Casper score had more group differences than interview score. Casper score had a lower magnitude of difference than the MCAT, overall GPA, and science GPA. Results also revealed differences in MCAT scores between disadvantaged groups, which support the literature related to disadvantage medical school applicants.⁶

Discussion

Early identification of medical school applicants who will be competent physicians with strong interpersonal skills is essential to the admission process. Literature has shown that traditional interviews are lacking. Our study found that MMI reduced bias for some underrepresented and disadvantaged groups. Therefore, the effect and relationships that the MMI has on predicting academic and clinical performance are critical to determining its utility. The findings provide further evidence that systemic barriers disadvantaged groups faced due to a lack of resources (such as test preparation classes or additional barriers, such as being employed during their undergraduate experience) can impact their scoring on standardized assessments. Future research will include assessing the MMI’s utility better, exploring the correlations between MMI scores and clinical evaluations and objective structured clinical examination (OSCE) results. This multifaceted approach will provide a comprehensive view of how MMI performance aligns with real-world clinical assessments and standardized evaluation metrics, offering valuable insights into its effectiveness as a predictor of future medical proficiency. Additionally, this investigation will serve as a foundation for refining the MMI’s role within the admissions process, ensuring it is a reliable and valid tool in identifying candidates who demonstrate the attributes necessary for success in medical school. By adopting longitudinal assessment strategies that are designed to reduce disparities and account for the diverse backgrounds of applicants, medical schools can cultivate a more equitable admissions process. These approaches provide a comprehensive understanding of an individual’s professional development and ethical conduct, ensuring that future regardless of ethnic or socioeconomic background physicians not only meet the highest standards of professionalism and social intelligence but also reflect the diversity and inclusivity necessary to address healthcare disparities effectively.

Lastly, by increasing the weight assigned to the MMI and reducing the emphasis on metrics exhibiting significant demographic disparities (ie, MCAT, Casper, GPA). JSSOM aims to facilitate a more comprehensive evaluation of applicants. This approach acknowledges some candidates’ systemic barriers and underscores the commitment to eliminating potential biases in post-interview review and subsequent admissions decisions. Recognizing the importance of personal characteristics in a race-neutral admissions process, this strategic shift aligns with the broader mission of JSSOM, which includes diversity and clinical excellence. This evolution towards a more holistic assessment of applicants ensures that the admissions process considers academic prowess and the personal qualities integral to becoming a compassionate and effective physician, irrespective of background or identity.

Limitations

This study benefits from several key strengths, including a large sample size spanning multiple admissions cycles, which enhances the reliability and generalizability of the findings. Additionally, the focus on analyzing the magnitude of differences by group rather than individual performance provides valuable insights into systemic patterns and equity within the admissions process. However, some limitations should be noted. The non-URM sample was significantly larger than the URM sample, which may limit the generalizability of findings for underrepresented groups. Furthermore, the reliance on self-reported data to assess socioeconomic status introduces potential bias and inaccuracies, as applicants may misreport or misinterpret their socioeconomic circumstances. These factors should be considered when interpreting the results.

Conclusion

In conclusion, the findings underscore the importance of continuously evaluating and refining medical school admissions processes to foster diversity, fairness, and reliability. The transition from unstructured interviews to semi-structured interviews and eventually MMIs highlighted the potential of structured formats to improve the consistency of candidate evaluations. While MMIs demonstrated a stronger relationship with Casper scores than traditional interviews, results did not support claims of their overall superiority in predictive ability or reducing demographic disparities.

Both MMIs and traditional interviews were found to be equally effective in mitigating group differences based on ethnicity and socioeconomic status, though MMIs slightly reduced differences for African American candidates while marginally increasing them for other groups. Importantly, Casper scores showed more group differences than interview scores but fewer than academic metrics like the MCAT and GPA, reaffirming its utility as part of a holistic admissions framework when used cautiously and in combination with other tools.

By reducing reliance on metrics with significant demographic disparities and emphasizing structured approaches like MMIs, admissions committees can create a more equitable evaluation framework. JSSOM’s strategy reflects a commitment to race-neutral and competency-focused evaluations while promoting diversity, inclusion, and clinical excellence. This forward-thinking approach sets a precedent for medical schools nationwide, ensuring a future workforce that is academically proficient, empathetic, and equipped to meet the diverse needs of patients and communities.