Piloting Virtual Multiple Mini Interviews (MMI) on Undergraduate Medical Students: A Multisource Feedback Analysis

Introduction

Admission to health professional training programmes is a high-stakes decision. It has been always a daunting task for medical schools’ admission committees to select a cadre of candidates from a large pool of applicants to produce competent physicians. Medical schools select suitable candidates mostly based on three domains: cognitive or academic ability, aptitude for medical training, and non-cognitive attributes considered desirable for a doctor.^1,2 The prior academic performance, ie high/secondary examination results, is used alone or in conjunction with the results of the medical aptitude test as the basis of the invitation to interview to assess non-cognitive attributes to study medicine.³ Non-cognitive attributes measured through interviews include ethics, empathy, leadership, teamwork and collaboration, communication, professionalism, ability to cope with stress, adaptability, honesty and integrity.^1,4–6

In recent years, interviews for the selection of students have become more important in medical school acceptance decisions.^7–9 Various formats are employed globally, each with distinct advantages and limitations. The traditional one-on-one interview typically consists of 1–2 interviewers spending time with a single candidate, asking pre-determined questions and engaging in discussion about the candidate’s experiences, opinions, and/or beliefs. However, research suggests the possibility of bias negatively impacting the application process as well as the validity and reliability of interview results.^10,11 Another alternative is the panel interview, where a candidate is assessed by a group of interviewers. However, panel interviews have limited ability to predict future success and that the pressure of facing multiple interviewers may discourage some applicants.¹ Among these approaches, the MMI has emerged as the most robust structured interview in use today, avoiding many shortcomings of the traditional interview.^12,13 Recent literature has further strengthened the evidence for the MMI’s predictive validity in identifying candidates who excel in medical schools, emphasizing its effectiveness in non-cognitive competencies.^14,15

MMI is an Objective Structured Clinical Examination (OSCE) - style exercise consisting of a 6 to 10 short stations with one or two interviewers in each station. Each designed to assess one or more personal attributes, such as critical thinking, decision-making, communication skills, ethical reasoning, interpersonal abilities, compassion, ability to work as a team, etc.^16–18 For example, one station may assess ‘ethical reasoning’, where candidates are given a scenario involving patient confidentiality and must discuss how they would handle the situation while adhering to medical ethics. The candidates move through the stations in which they respond to a set of predefined case-based scenarios or questions while being rated by the trained interviewers using a standard scoring scheme.¹⁹ This structured format enables candidates to engage with multiple interviewers, providing a comprehensive evaluation of their skills and attributes.

Literature suggests that MMI used to select candidates for undergraduate health programmes appear reasonable feasibility, acceptability, reliability, and validity.^4,9,20 The MMI can measure a number of key outcomes deemed necessary in the practical context of the health professions.²¹ According to the Association of American Medical Colleges, MMI assesses communication skills, specifically verbal and nonverbal skills that cannot be measured using standardized written exams or reviewing coursework transcripts.²² MMI also helps offset the effect of interviewer bias by considering a number of assessments by multiple interviewers while selecting a candidate.²³

MMI was predominately conducted face-to-face before the COVID-19 pandemic. The face-to-face mode of MMI was understood to be resource-intensive and costly, and it was influenced by unintended bias to assessment.¹⁶ Social distance policies and cancellation of face-to-face MMI due to the pandemic provided opportunities to many medical schools across the world to consider the online interview process.^24,25 Research by Alam et al (2024) suggests that participants favored virtual MMIs, citing higher effectiveness, adequate timing, and a better understanding of interview stations.²⁶ Similarly, Yolanda et al (2020) highlighted the feasibility and cost-efficiency of virtual MMIs, though they emphasized the need to address technical challenges such as internet connectivity and IT support.²⁷ Thus, the University of the West Indies (UWI), St Augustine embarked upon the virtual pilot MMI to the Year 1 medical students, recently admitted to the faculty observing the COVID protocols.

There is growing literature focused on the feasibility, acceptability, reliability and validity of the MMI process. However, studies seldom explore interviewers’ and interviewees’ feedback on the MMI and their importance in the admission process. The UWI, St. Augustine is one of the leading medical schools in the Caribbean. As part of commitment to maintaining high standards in medical training, the school is exploring innovative ways to enhance admission process, particularly in the light of global shifts towards virtual learning and assessment during the COVID-19 pandemic.

In this context, the University offers an opportunity to assess how this method can be effectively implemented in the Caribbean region, where there is limited research and experience with virtual MMIs. Understanding the challenges and benefits of virtual MMIs in this context is critical for improving the admissions process for medical students. The research questions of the study are as follows:

• What are the experiences and perceptions of interviewees regarding virtual MMI process?
• What are the experiences and perceptions of interviewers about the virtual MMI process, particularly in terms of its effectiveness in conducting interviews and assessing students’ performances?
• What are the experiences and perceptions of technical facilitators in managing and supporting the virtual MMI process?

Therefore, the primary objectives of this study were to:

• Evaluate the feedback from interviewees, interviewers, and technical facilitators regarding the virtual MMI process.
• Assess the effectiveness and feasibility of the online MMI piloted with newly admitted medical students.

Methods

MMI Setting and Process at the UWI

The University of the West Indies (UWI), a regional indigenous institution, operates campuses and medical faculties, each with its own Dean, in Barbados, Jamaica, and Trinidad, along with a clinical site in the Bahamas. UWI’s medical program enrolls approximately 3000 students in its 5-year MBBS degree programme. This programme is structured into two phases: a predominantly preclinical Phase 1 (Years 1–3) and a clinical Phase 2 (Years 4–5). The curriculum follows an integrated modular approach, incorporating continuous in-course evaluations and final summative assessments. Successful candidates are eligible for provisional licensure as interns in most English-speaking Caribbean countries.

The selection of MBBS candidates for medical school in the UWI is based on measures of the student’s performance in high/secondary examinations. In addition, candidates are required to submit a short (250–300 words) autobiographic sketch explaining the reasons for their career choice. Besides, candidates’ voluntary participation in other extracurricular activities such as sports, debate, music, drama or proficiency in a foreign language are taken into consideration. Based on the recent recommendations of The University Medical Curriculum Committee (UMCC), the MMI was piloted in the St. Augustine Campus in virtual mode in September 2021 and August/September 2022.

Based on the blueprint covering domains such as communication, ethical issues, problem-solving ability, integrity, teamwork, motivation and empathy, multiple scenarios along with a rating scale were developed for both pilots. The domains assessed in the MMI were chosen based on established research in medical education, which highlights the importance of non-cognitive attributes in predicting future clinical competence and professionalism.^16–18,22 These attributes align with global standards for medical education and the objectives of the Faculty of Medical Sciences (FMS), UWI, St. Augustine. A total of 4 academic staff and two medical educationists were involved in developing the scenarios, rating scale /checklist and rubric, which ensured face validity. The content validity was assured by this group of expert judgment.

The MMI at The UWI was conducted virtually on Zoom utilizing breakout rooms. There were seven active stations and one rest station. The students responded to a set of predefined scenarios. Each scenario was presented via the ‘share screen’ for one minute, for the student to review the given scenario. At the end of 6 minutes, every student was rotated through virtual rooms using a videoconferencing platform. The transition time was 2 minutes. Each Station was assessed by two interviewers. The average score of two interviewers was used as obtained marks for the students in each station. For both the 2021 and 2022 pilots, training in the areas of MMI and principles of assessment for the interviewers was offered.

Study Design and Sampling

We conducted a descriptive survey at the School of Medicine, The University of the West Indies, St. Augustine Campus, Trinidad and Tobago. Our study involved Year 1 students who participated as interviewees in the pilot MMI, as well as the interviewers and technical facilitators. The design involved collecting separate feedback from each group using distinct questionnaires, allowing us to assess their individual experiences with virtual MMI process. The independent variable in this study is the virtual MMIs, whereas the dependent variables include the experiences and feedback of interviewees, interviewers, and facilitators regarding the virtual MMI process.

We employed convenience sampling to gather data from the listed participants. In 2021, a total of 95 interviewees participated, while in 2022, 79 interviewees took part. The number of interviewers 25 in 2021 and 32 in 2022, technical facilitators 9 in 2021 and 8 in 2022. The sample size is based on participant availability and logistical feasibility rather than a priori calculation. Our study’s exclusion criteria included individuals who did not participate in at least one MMI pilot session and those who failed to complete the questionnaire.

Instrument

The researchers developed three self-directed questionnaires for interviewees (19 items), interviewers (14 items), and technical staff (09 items) by conducting a literature search using web-based search engines such as Google Scholar and PubMed.^23,28,29 Each questionnaire comprised three parts. The first part collected demographic information from the participants. The second part assessed the participants’ perceptions of the virtual MMI. Respondents were required to indicate their agreement level on a 4-point Likert scale, ranging from “strongly agree” to “strongly disagree”. In the third part, we used the open-ended questions to identify challenges, benefits of online MMI, and offer suggestions for improvement. The content validity of the questionnaires was assessed through feedback from five experts, including four medical educationists. Their recommendations were implemented to enhance grammatical accuracy, refine word usage, and improve word order. Furthermore, based on their input, two items were removed from both the interviewee and interviewer questionnaires. The reliability of the questionnaires was confirmed using Cronbach’s Alpha, with acceptable values ranging from α=0.78 to α=0.94 for interviewees, interviewers, and facilitators.

Data Collection

For this study, we administered web-based questionnaires to interviewees, interviewers, and technical facilitators who participated in the MMI in 2021 and 2022. We sent an Email message to all the participants, explaining the aims of the study and requesting respondents to click the link to fill out the Google Form on the day of MMI. One week later, we sent a follow-up Email message to encourage completion of the instrument.

Ethical Approval

This pilot study was part of the University’s internal quality assurance process and contributed to the monitoring of the Multiple Mini Interview (MMI), as mandated by the Office of the Dean, Faculty of Medical Sciences, UWI, St. Augustine Campus, Trinidad and Tobago. According to standard practices, studies involving surveys or educational tests with students and faculty typically do not require IRB approval. It is important to clarify that the study was conducted with students who had already been selected for the MBBS programme, meaning there was no possibility of rejection, and thus, no risk of exclusion due to participation. Nevertheless, approval to conduct the study was obtained from the Centre for Medical Sciences Education at UWI, St. Augustine, Trinidad and Tobago. The study ensured the confidentiality and anonymity of all participants, who were fully informed about the study procedures. Informed consent was obtained from all participants, including approval for the publication of anonymized responses.

Data Analysis

We analyzed the collected data using SPSS version 29 software (IBM Corporation, New York, USA). Descriptive statistics, including percentages, medians, and interquartile ranges, were used to summarize the responses. Additionally, we employed the chi-square test of goodness of fit to assess whether the distribution of frequencies of the responses for each item significantly deviated from a uniform distribution. Data distribution was assessed for normality using the Shapiro–Wilk test. Since the data did not follow a normal distribution, we used the chi-square test. A p-value of <0.05 was set as the threshold for statistical significance for the chi-square test statistics. The information gathered from open-ended questions underwent thematic analysis.

Results

Interviewees’ Perception on Online MMI

A total of 95 interviewees in 2021 and 79 interviewees in 2022 participated in the survey with a response rate of 62.5% and 51.6% respectively. (see Table 1) The Cronbach’s alpha for the interviewees’ questionnaires in 2021 (0.94) and 2022 (0.90) are found to be of acceptable levels.

Table 1 Participants and Response Rate

Table 2 Interviewees Feedback on Virtual MMI 2021 (N = 95), and 2022 (N = 79)

Table 2 shows the piloted online MMI experienced by interviewees in 2021 and 2022. All chi-square values shown in the table were found to be statistically significant at 0.05 level. The majority of interviewees (85.2% in 2021 and 91.1% in 2022 with median = 3, IQR = 1) agreed that MMI helped to assess their non-academic personal attributes, interests, and motivation to study medicine which are vital to the success of health care providers. The chi-square value of the interviewees in 2022 was 66.37 (p < 0.05) compared to 59.15 (p < 0.05) in 2021, which indicates an increased positive response among the participants towards online MMSs in the second year. Furthermore, (92.6% with median = 3, and IQR = 1) in 2021 and 89.9%) with median = 3, and IQR = 1) agreed that MMI is designed to evaluate applicants fairly, minimizing the influence of demographic factors such as age, gender, or socio-economic status. However, a little less than half (49.5% with median = 3, and IQR = 1) in 2021 and more than half (53.1% with median = 2, and IQR = 1) of the interviewees in 2022 believed that MMI did not allow them to demonstrate their abilities to work as a team.

Interviewers’ Perception on Online MMI

In 2021, 25 interviewers responded to the online Google survey, representing a response rate of 39.1%. In 2022, 32 interviewers responded, indicating a response rate of 59.3%. The Cronbach’s alpha for the interviewers’ questionnaires was found to be at acceptable levels in both 2021 (0.82) and 2022 (0.91).

Interviewers’ perceptions of online MMI are given in Table 3. Overall, the majority of the interviewers (80% with median = 3 and IQR = 0 in 2021 and 90.6% with median = 3 and IQR = 1 in 2022) agreed that MMI is a good measure to assess the non-cognitive ability of projection medical students. Further, the majority of interviewers (64–90.6% with median 3 and IQR = 0.75 to 2) in both 2021 and 2022 believed that the MMI process was well organized, and the training given to them was clear enough to successfully conduct the interview. The increased chi-square values in these items in 2022 as compared to 2021 indicate that significant improvement has been made by the University in conducting MMI.

Table 3 Interviewers Feedback on Virtual MMI 2021 (N = 25) and 2022 (N = 32)

Table 4 Technical Facilitators Feedback on Online MMI 2021 (N = 9) and 2022 (N = 8)

Technical Facilitators’ Perception on Online MMI

Table 4 presents the technical facilitators’ perceptions of the piloted online MMI in both 2021 and 2022. The Cronbach’s alpha for the technical facilitators’ questionnaires was deemed acceptable in both 2021 (0.89) and 2022 (0.78). Due to the smaller number of technical facilitators (9 in 2021 and 8 in 2022) and responses in both years, the scores of “strongly agree” and “agree” were combined, as were “disagree” and “strongly disagree” for analysis purposes. The majority of technical facilitators believed that the MMI process was well-organized, and they expressed satisfaction with time management (ranging from 66.7% to 88.9% with a median of 2 and IQR of 0 to 1). Both items were found to be significant (p < 0.01).

Challenges

Regarding challenges of online MMI, 66.3–80% of participants in 2021 and 63.3–75% participants in 2022 responded to the open-ended item.

Technical Issues

Although interviewees, interviewers and technical facilitators were familiar and comfortable with the online MMI, a few of them experienced some unavoidable technical difficulties which included temporary disconnection of the internet and electricity outrage. One candidate stated:

Whilst I do believe that the MMI is an excellent idea and concept, I faced internet connection issues which drew an inaccurate representation of who I am due to my audio breaking up, etc.

Management Issues

The majority of interviewees perceived that the MMI process was well organized (60% −89.9% with median = 3, IQR = 1). However, they identified a few management issues in conducting the MMI. One interviewee noted:

The lack of communication during the wait time was a challenge as I was unaware that I had missed a step in the process. If delays arise, clearer communications to the attendees should be given,’ while another interviewee stated, ‘Not enough time in advance to prepare.

Issues Related to MMI Scenarios

The interviewees experienced challenges in a few MMI scenarios. One candidate argued:

It was difficult for me to comment on tabular COVID-19 data after only seeing it for one minute. The pressure of trying to remember the exact data led to me being unable to answer the question and my anxiety levels were at an ultimate high.

Benefits

Assessment of Non-Academic Aspects

Despite some challenges, interviewees agreed that MMI as a good additional method for entry to the MBBS programme. Of the total, 62.1–64% of participants in 2021 and 64.6–71.9% of participants in 2022 responded to the open-ended item on the benefits of online MMI. Both interviewees and interviewers agreed that MMI assessed non-academic aspects of students such as ethics, empathy, communication skills, problem-solving, and critical thinking skills. One student stated:

It allowed for a more well-rounded and less biased analysis of candidates rather than just academic performance.

Comfortable and Convenient

Online MMI reduced the risk of transmission of COVID-19. It allowed students to express their views and be properly assessed in the comfort of their homes. One interviewee stated:

I was able to appreciate doing the MMI in the comfort of my home, reduced anxiety/nervousness since I was not physically there with the interviewer. It increased my motivation before the beginning of Medical School.

Unbiased and Effective Assessment

Interviewers believed that the online MMI was unbiased and allowed a more holistic assessment of candidates. They stated that online MMI was the fastest way to assess and compile marks. The conversations give a much better assessment than a paper application. One interviewer note:

Having multiple interviewers and many stations creates an environment for unbiased scoring.

Suggestions for Improvement

Management of the MMI

Participants in the survey suggested that communication of MMI organizers to students for the MMI should be improved. Students should be given more time to prepare (a few weeks) and they should be allowed to choose a time slot that works best for them. One student suggested:

For the future, I think it would be best to organize it a lot beforehand and to contact students at least a week before the interview and allow them to pick a time and day best suited for them.

During the Interview

Interviewees suggested to start the MMI on time to reduce their anxiety levels. Students proposed to show the questions on the screen while they answer so they can refer to the questions, especially if it is a table with data. One interviewee stated:

In questions that use graphs/charts/any visual diagrams, the diagram should be visible at all times as the 1-minute time limit could result in interviewees just memorizing the data as opposed to understanding and making evaluations of the trends observed.

Online Vs Face-to-Face MMI

The majority of participants (75.9–100% with median = 2–4 and IQR = 0–1) recommended the use of online MMI for future medical admissions. However, few interviewees advocated for face-to-face MMI for better assessment. One interviewee stated:

I hope that MMI are conducted in person as a better assessment of an individual’s response can be made taking into account their body language and demeanors.

Discussion

The primary aim of this study was to assess the perceptions of interviewees, interviewers, and technical facilitators regarding the pilot implementation of virtual MMI for MBBS admission at the UWI, St. Augustine campus. The findings offered significant insights into the feasibility, acceptability, and reliability of online MMI as an alternative to traditional face-to-face interviews. While virtual MMIs demonstrated logical advantages, such as cost savings and acceibility, face-to-face MMIs are often considered superior in fostering interpersonal interactions and assessing non-verbal communication. These insights are particularly valuable in the context of modernizing medical school admissions while addressing challenges posed by global disruptions, such as the COVID-19 pandemic.

The results of this study align with existing literature that highlights MMIs as an effective tool for evaluating the non-cognitive attributes of medical school applicants.^12,17 A majority of interviewers agreed that MMIs are a good measure to assess the non-cognitive abilities of prospective medical students. These findings echo those of a pilot study conducted in Oman, where interviewers and interviewees alike found MMI method acceptable and effective selection method.³⁰

Additionally, a systematic review by Yusoff (2019) reinforced the flexibility of MMI in assessing various important attributes of candidates such as professionalism, ethics, communication skills, critical thinking and problem-solving.⁹ The findings in this study further validate the MMI’s capacity to assess diverse attributes critical for medical practice. Research by Alam et al²⁶ demonstrated that different MMI stations assessing students’ motivation, communication, self-reflection, qualities and achievement were positively correlated with students’ cumulative grade point average. This suggests that MMI not only evaluates traits essential for clinical practice but also serves as a predictor of academic success.

A notable advantage of MMIs observed in this study is their ability to reduce bias in candidate assessment. The design of MMI allowed interviewees to independently interact with multiple interviewers across various stations, which reduced bias in assessing interviewees. Both raters and interviewees in this study agreed that MMI did not disproportionately favor or disadvantage candidates based on age, gender, or socio-economic status. These findings are consistent with previous studies that similarly reported the fairness of MMI in avoiding selection bias.^1,23,31,32 Both virtual and face-to-face MMIs share this strength; however, while in-person MMIs ensure a controlled environment where all candidates have equal access to infrastructure, online MMIs are influenced by the internet stability of both interviewers and candidates.³³

The shift to virtual MMI, prompted by pandemic restrictions, was a critical adaptation explored in this study. Both interviewers and interviewees expressed positive attitudes toward the virtual format, with one interviewer noting that the “Virtual MMI is COVID-19 protocol friendly and less time consuming because we did not have to travel”. These findings are consistent with the previous studies, where applicants and raters considered online MMI an acceptable means of interviewing students during the pandemic.^25,34,35 Virtual MMI also presented significant benefits, including the elimination of travel costs and saving travel time for candidates, as highlighted in prior research.¹⁷ In contrast, traditional face-to-face MMIs require significant logical arrangements, including institutional infrastructure and candidate travel, which can increase costs and administrative burden.¹³

However, this study also identified challenges inherent to the virtual format. Some candidates experienced technical difficulties, such as internet connectivity issues and power outages, leading to temporary disconnections during interviews. However, the facilitators effectively managed these disruptions by reorganizing interviews, highlighting the importance of responsive technical support. While face-to-face MMIs eliminate such concerns, they introduce other challenges such as the need for physical space and logical arrangements.¹³ Furthermore, both interviewees and interviewers noted the issue with time management and delayed start on the first day of the interview. These findings are consistent with those of Ungtrakul et al²⁴ who suggested that effective time management, trained staff, appropriate scenarios and reliable technology were the essential factors to make the virtual MMI successful.

Despite these strengths, virtual MMIs had inherent drawbacks. For example, Abraham et al³⁶ reported that while virtual MMI addressed immediate logistical needs during the pandemic, they lacked the depth of interaction and interpersonal connection inherent to in-person interviews. This trade-off highlights the need for further exploration of hybrid or alternative formats to preserve the benefits of face-to-face interaction while leveraging the convenience of virtual platforms.

Limitations

This study had several specific limitations. The study was conducted at a single institution (only one campus of the UWI), which may limit the generalizability of the findings to other settings. Additionally, there is likely a non-response bias because many participants who participated in the MMI did not respond to the survey, possibly skewing the results. Moreover, self-selection bias may have influenced the study’s results due to voluntary participation. The use of technology may have been influenced by other factors not covered in this study, such as participants’ home environment and personal circumstances. Furthermore, the cross-sectional design of the study precludes conclusions regarding long-term outcomes regarding student performance or satisfaction with the MMI programme. While Cronbach’s Alpha was calculated to assess internal consistency, further psychometric validation (such as construct validity and factor analysis) was not conducted. Further studies should consider additional validation methods to enhance the robustness of the questionnaire. Another limitation of the study is that it involved students who had already been selected for the MBBS programme, eliminating the possibility of rejection or exclusion. This may reduce the diversity of participant experiences and responses, as all participants were assured admission.

In light of above limitations, the findings should be interpreted with caution. Future research should aim to address these gaps by expanding the scope of study to include multiple institutions, ensuring a higher response rate, and controlling for external factors influencing virtual MMIs. Additionally, longitudinal studies are recommended to evaluate the long-term impact of virtual MMIs on student outcomes and the overall admissions process.

Conclusions

The findings of this study demonstrate that virtual MMIs are a feasible and acceptable alternative to in-person interviews for assessing medical school applicants. Interviewees, interviewers, and technical facilitators showed a predominantly positive experiences, highlighting the convenience and accessibility of the piloted virtual MMI. However, technical and logistical challenges remain key areas for improvement. To systematically address technical issues and ensure smooth interview experience, institutions should invest in stable digital platforms, reliable internet connectivity, and dedicated technical support teams. Additionally, admission staff should focus on refining the selection of interview cases, enhancing the scoring system, improving time management strategies, providing comprehensive training for interviewers and technical facilitators, and strengthening infrastructure and technical support. With appropriate modifications, the virtual MMI could become an important factor in admitting medical schools in the future, ensuring an unbiased and standardized assessment of non-cognitive traits required for the medical profession.