Implementation and Analysis of a Fully Immersive Virtual Reality-Based Emergency Training in a Surgical Curriculum

Introduction

The rapid and significant expansion of medical knowledge¹ poses a particular challenge for young doctors at the start of their careers. Given the vast amount of information, the traditional rote learning methods used in medical education are no longer sufficient, underscoring the need for new, meaningful learning approaches.^2,3 This is especially critical in emergency medicine where often immediate action is necessary, leaving no time to consult reference materials. Despite the increasing emphasis on practical training in this area, junior doctors still show deficits in the management of emergency patients.^4–6 However, due to their need for personnel and resources, current simulation approaches are unlikely to adequately address this gap in the near future.

Fortunately, there has been a recent expansion in the methods of delivering medical education. The use of head-mounted Virtual Reality (VR) displays can deliver fully immersive and interactive environments offering the chance for meaningful learning experiences.⁷ The use of VR technology in medical education can enhance learning effectiveness and efficiency⁸ and learning outcomes may even be superior to traditional teaching methods.⁹ Within the literature, various methods are grouped under the term “VR”, ranging from 3D representations on screens to 360-degree videos and fully immersive 3D environments on head-mounted displays. Research by Gutiérrez et al indicates that fully immersed VR trainings yield notably higher learning gains compared to partially immersed VR, particularly in medical student education.¹⁰ Due to decreasing hardware prices, scenarios using head-mounted displays are becoming more cost-effective and scalable^11,12 in comparison to analogous simulation settings. Consequently, as an extension of the traditional curriculum, VR trainings offers a way to provide meaningful learning experiences to a large number of students in the future.

Thus far, VR has found successful applications primarily in trainings for robotic surgery, interventional procedures, and hygiene practices^13–15 as well as brain death diagnostic and emergency medicine.^16,17 Especially the latter trainings rely on advanced knowledge and skills (eg, clinical reasoning, task prioritization), making their use particularly beneficial later during the medical curriculum. Recently, Mühling et al implemented a VR-based training approach for complex medical emergencies, exploring perceived stress levels, and subjective learning outcomes among senior students.¹⁸

Currently, existing studies on the use of complex VR scenarios in medical education take into account, to some extent, the impact of demographic parameters (such as age, gender, and prior experience with 3D/VR applications) on the learning experience.^17,18 However, important user-specific characteristics, such as technological proficiency, as well as hardware- and software-specific factors such as usability, are often overlooked. Yet, poor usability has been proven to be one of the greatest barriers to the curricular implementation of virtual reality.^19,20 Therefore, this study aims to understand not only learning outcomes but also the influence of technological proficiency and usability. Since it is considered good practice, especially in the field of simulation-based training, to take participants’ prior knowledge into account, the current study will also assess perceived competence before and after the training.²¹ Against that background, the following research questions were formulated:

1. How is the VR-training module perceived by our students with respect to usability (intuitive use, degree of presence and grade of immersion)?
2. Does the VR-training provide a positive learning experience for the students in terms of subjective learning success and perceived competence (the latter measured in a pre-/post design)? Are there any differences between active users and observers?
3. How do technological proficiency, usability (intuitive use, presence) of the VR-training and learning experience correlate with each other?

Materials and Methods

Medical Curriculum

The medical program in Germany is divided into pre-clinical and clinical years. After completing their surgical curriculum consisting of lectures, problem-based-learning and various seminars, students perform a 1-month clinical rotation in surgery. The VR module was implemented as the last teaching unit in our surgical curriculum before students were to take their OSCE (Figure 1) and prior to entering their clinical rotation.

Figure 1 Surgical curriculum before entering the surgical clerkship. The role of the teacher decreases while the role of the student increases. The VR seminar concludes the curriculum.

Study Design

The VR-based simulation was carried out as the last mandatory teaching session of the surgical curriculum at the European Medical School Oldenburg prior to taking the OSCE and advancing into the four-week surgical clerkship (see Figure 1). Thus, students gradually increased their active role while the teacher's role decreased. All participants were fourth-year medical students. The study was conducted as a pre-post study.

Before participating in the VR seminar, students underwent an inverted classroom, during which they prepared for the virtual seminar using online videos as instructions. The videos explained the handling of the hardware and software. Only the students who successfully completed the inverted classroom were able to participate as active users (AU). Students that completed the inverted classroom participated as AU voluntarily. The remaining students participated as observers (OBS) and were able to view the perspective of the AU on a monitor and provide verbal assistance.

A total of eight VR-based teaching sessions were carried out over a timespan of two hours each. A maximum of 10–15 students were in each group. The VR-based teaching sessions were supervised by a surgical consultant. Three cases for interdisciplinary training, covering both medical and surgical learning objectives, were simulated during each seminar with repeated content: 1. acute biliary pancreatitis with persistent cholestasis, 2. abdominal sepsis due to fistulizing Crohn’s disease, 3. oesophageal variceal bleeding due to liver cirrhosis. All medical content of the scenarios was based on established guidelines and reviewed by at least two experienced faculty members with specialization in medical education and/or emergency medicine.^22–24 A questionnaire was provided before and after the seminar. The study was conducted in accordance with the ethics committee of the University of Oldenburg (AZ 2023–130). A flow chart of the study design is shown in Figure 2.

Figure 2 Study protocol displaying the pre-post design including patient-cases and questionnaires.

VR-Based Simulation

The STEP-VR program was designed for medical students to simulate the role of physicians encountering various critically ill patients in an emergency department. The training software (version 0.15 beta) was developed by ThreeDee GmbH (Munich, Germany), a startup specializing in 3D visualization. Two head-mounted displays (HMD) were used (Meta Quest 2, META) as seen in Figure 3. The training software and head-mounted displays were running on a high-performance gaming computer and connected by wireless LAN (WLAN). STEP-VR aims to enhance cognitive abilities such as executive functions, situational awareness, time management, and task prioritization. Diagnostic tools provided include medical history, laboratory tests, various imaging techniques, and treatment options, all based on authentic patient data. Certain procedures, such as blood work and intravenous (IV) cannula placement, are simulated in an abstract manner. Once the user determines the most likely diagnosis, therapeutic interventions can be performed, with real-time effects on the simulated patient’s physiology. Lastly, a checklist based on German treatment guidelines assessed whether the correct actions have been taken.

Figure 3 (A) Head-mounted displays by META (Quest 2). (B) In-App view of the virtual emergency department displaying a patient connected to a monitor measuring vital signs. (C) Active (AU) and passive (OBS) participants during the VR-based simulation.

Questionnaire

The questionnaire was developed with the department of academic affairs, quality management and evaluation. The software used was “QuestPro 5 by Blubbsoft GmbH”. Questionnaires that were adapted or modified are marked in Table 1. It was divided into three parts. Part 1 was answered by all participants and included demographics, experience with VR as well as technological proficiency and perceived clinical competence before the seminar. Part 2 was answered by AU only and included the performed case, motion sickness, presence, intuitive use and perceived stress. Part 3 was answered by all participants and included degree of immersion, subjective learning success, perceived clinical competence after the seminar and a field for comments. A simplified overview can be seen in Table 1.

Table 1 Overview of the Questionnaire Used

Statistical Analysis

The analysis was conducted using descriptive statistics, including mean (M), minimum (Min), maximum (Max) and standard deviation (SD). The Little’s Missing Completely at Random Test (MCAR) was applied to the entire dataset to assess whether values were missing systematically or completely randomly. As most of the parameters were not distributed normally, we chose non-parametric tests: Mann–Whitney Test was used for comparisons between groups (eg AU/OBS). Perceived competence before and after the seminar was analysed using Wilcoxon matched pairs signed rank tests. Spearman’s rank test was used for calculating correlation coefficients (r). For the correlation analysis, we calculated the difference in perceived competence between values collected before and after the VR training, labelled as “perceived competence gain”. Data analysis was performed using IBM SPSS Statistics 29.0.1 (International Business Machines Corporation, Armonk, NY, USA), and table creation was done using Excel or Word (Microsoft 2010, Redmond, WA, USA).

Results

Demographics

Out of all 84 students enrolled in the study, 60 students (71.4%) were OBS and 24 students (28.6%) were AU. A vast majority of the students were female (81.9%). 52 students (61.9%) were between the ages of 18–25, 25 students (29.8%) between the ages of 26–30 and seven students (8.3%) were over 30 years old. No significant differences were found regarding age (p=0.655) and gender (p=0.591) when comparing AU and OBS. 57 students (65.9%) never played any sort of PC or console-gaming and 79 students (94%) denied any previous use of VR or AR. Students’ demographics are displayed in Table 2.

Table 2 Students Demographics

Technological Proficiency Before the VR-Module

The analysis of “technological proficiency” showed no significant statistical differences for all items questioned. While the enjoyment of dealing with technical problems was below average (2.95 ± 1.35), participants felt confident in resolving them independently (3.94 ± 1.34). Results of the comparison between AU and OBS for “technological proficiency” are displayed in Table 3.

Table 3 Comparison Between AU and OBS for the Item “Technological Proficiency” Before the VR-Module

However, gender-specific differences were found in the self-assessment of “technological proficiency” as well as in the “use of PC or console games”. Among all male students, 46.7% stated that they use PC or console games at least once a week. Among female students, 81% denied any use. Supplement Table 1 presents a gender-specific comparison of “technological proficiency”. Male participants especially rated their abilities higher when it came to “independently solving technical problems” (p=0.002) and “enjoying dealing with technical problems and solving them” (p=0.003). Two statements were not statistically significant.

AU Results for Presence, Intuitive Use and Perceived Stress and Motion Sickness

The questions answered for the items presence, intuitive use, perceived stress by AU after the simulation are displayed in Table 4. Regarding the item presence, the students fully agreed with the statement “I had the feeling of being in the room with the patient” (4.42 ± 0.50) and “during the simulation I had the feeling to be fully immersed” (4.29 ± 0.69). Most items of the measure “intuitive use” were answered around the average value, with moderate agreement that the use of the VR simulation was “straightforward” (3.50 ± 0.98). In contrast, there was very strong confirmation of the statement “I enjoyed the VR simulation” (4.79 ± 0.51). Regarding the items of perceived stress, the students had mostly issues with managing several tasks at once (4.00 ± 0.98) and some were concerned not being able to address the emergency appropriately (3.50 ± 1.38). The feeling of being observed by fellow students exerted only little pressure on participants (2.48 ± 1.44). Five out of 24 AU reported symptoms of motion sickness such as nausea, headache and slight dizziness.

Table 4 AU Results for Presence, Intuitive Use and Perceived Stress.

Comparison Between AU and OBS Regarding Grade of Immersion, Subjective Learning Success and Perceived Competence

Grade of Immersion

While the AU were nearly “completely immersed” in the VR simulation (4.58 ± 0.83), the OBS reported only average agreement for this question (3.02 ± 0.93, p<0.001). Interaction with the patient was rated as less immersive compared to other items by the AU, but still above average (3.58 ± 1.06). All results are listed in Table 5. In general, AU rated the degree of immersion significantly higher (AU: 4.05 ± 0.72, OBS: 3.28 ± 0.70, p<0.001).

Table 5 Results of the Items of “Grade of Immersion” and “Subjective Learning Success” After the VR-Module for All Students (AU+OBS) and Separated in AU and OBS

Subjective Learning Success

The outcomes of the subjective learning success after completion of the VR training for both AU and OBS are displayed in Table 5. While learning success across all items was rated high by both groups, the AU’s ratings for some items were significantly higher. For instance, the VR simulation was seen as an effective learning tool (AU: 4.79 ± 0.42, OBS: 4.20 ± 0.78, p<0.001). Items related to individual learning benefit were consistently rated more reserved but still above average (eg, “I now have more confidence in being able to respond appropriately in emergency situations”, AU: 3.50 ± 0.91, OBS: 3.22 ± 0.90, p=0.26). On average, subjective learning success was also rated higher by AU compared to OBS (AU: 4.15 ± 0.55, OBS: 3.69 ± 0.72, p = 0.005).

Perceived Competence Before and After the VR-Module

All students rated their own subjective perceived competence in dealing with emergency situations, identifying physiological relationships, coming to a diagnosis and initiating a subsequent therapy both before and after the VR-module as seen in Table 6. Three of the seven items were graded significantly better after the VR-module as depicted in Figure 4. Students felt well-prepared (before: 2.76 ± 0.80, after: 3.00 ± 0.84, p=0.001), deducing a diagnosis (before: 3.21 ± 0.73, after: 3.41 ± 0.74, p=0.005) and initiating an appropriate treatment (before: 2.51 ± 0.82, after: 2.89 ± 0.94, p<0.001). The items “I am confident to respond appropriately to emergency situations”, “I feel confident in managing medical emergencies” and “I am able to understand the underlying physiology in emergency situations” did not statistically differ before/after the VR-module while the item “I feel confident in understanding connections and deriving consequences” was graded worse (before: 3.80 ± 0.76, after: 3.60 ± 0.73, p=0.023).

Table 6 Results of the Item “Perceived Competence” Before and After the VR-Module

Figure 4 Comparing the results for perceived competence before and after the VR-module.

Correlation Analysis

To identify factors influencing the perceived competence gain (Δpost – pre) among the AU, correlation analyses were conducted. Other measures considered in these analyses were the results of the questionnaires on technological proficiency, presence, intuitive use and subjective learning success.

The perceived competence gain correlated moderately and significantly (r = 0.46, p = 0.03) with the subjective learning success. A slight, yet non-significant correlation was found between the perceived competence gain and the intuitive use (r = 0.33, p = 0.11). The connection between intuitive use and subjective learning success was more pronounced (r = 0.51, p = 0.01).

Regarding the interplay of user-specific factors with hardware-/software specific factors, there were no correlations with the technological proficiency of participants or the experience of presence in the VR-based scenario. However, participants’ technological proficiency correlated moderately to strongly with Intuitive Use (r = 0.62, p = 0.001). Lastly, prior experience with computers or VR technology did not correlate with any of the learning outcomes, intuitive use, or immersive experience. Only between prior experience with computers and technological proficiency, a moderate, significant correlation was found (r = 0.44, p < 0.001).

Discussion

In this study, we integrated a VR-based training program for complex interdisciplinary emergencies into the surgical curriculum and compared the usability and subjective learning experiences between fully immersed AUs and partially immersed OBS. Another focus was on examining the impact of technological proficiency and usability measures on the learning experience.

Regarding our primary objective of assessing various usability aspects, participants rated the intuitiveness of use as slightly above average. Comparable results are reported by other studies: Users of a VR-based training for paramedics reported a Usability Score (SUS) of 65/100.²⁴ Similarly, participants in a VR-based emergency training for medical students (conducted in a non-native language) also reported average values for intuitive use.¹⁸ However, subjective experiences with the intuitive use of VR training are currently being shaped by the novelty of VR applications as most participants are using such technology for the first time. Considering this, it is difficult to determine whether such values reflect the actual usability of the programs or rather the fact that users are generally not familiar with the technology. In our specific population, a particularly low prior experience with 3D or VR applications, compared to other studies,^17,18 may have additionally influenced the results. The relatively high correlation of intuitive use with subjective learning outcomes in our study, however indicates, that VR learning applications should generally undergo rigorous optimization of accessibility and usability to maximize the potential for an optimal learning experience.

Concurrently, our participants assessed the level of immersion and presence as high to very high. Similar to a previous study with a comparable concept, but focusing on an internal medicine perspective, the substantial difference in immersion experience between AU and OBS could be confirmed.¹⁸ A higher degree of realism in the case of fully immersed VR training can enhance satisfaction among learners³⁰ and also objective learning outcomes.¹⁰ In our study, however, a stronger feeling of presence among AU was not correlated with learning outcomes. Given the high degree of reported presence in our study, this could imply that a ceiling effect has been reached, beyond which greater immersion no longer has a subjective impact on learning success.

As a second objective, this study aimed to analyse two subjective learning outcomes: The first one, subjective learning success, was rated by the participants as substantially above average, with AU giving higher ratings than OBS. Retrospective subjective reports of learning success are often used to assess (simulated) learning experiences. However, the items of such reports frequently capture aspects of the individual learning experience together with a general assessment of VR-based learning.^17,18,31 Moreover, the retrospective nature of the assessment also raises doubts about its validity.

Therefore, as a second learning outcome, a measure of perceived competence before and after the VR module was introduced. A similar approach was already successfully taken for analogous simulation training.³² Interestingly, the increases in perceived competence were more heterogeneous than would have been expected from the high subjective learning success. While specific competency facets such as diagnostic reasoning and initiation of a treatment algorithm demonstrated significant improvements, others remained unchanged or even worsened (such as understanding of connections and deriving of consequences). Mühling et al already suggested that the potential students generally see in the method may not necessarily translate into personal competence improvement within relatively short training duration.¹⁸ Furthermore, participants may have been confronted with their own professional deficiencies due to the complexity of the cases, leading them to worse (but more realistic) self-assessments.

Of note, all significant differences between groups were exploratory assessed without controlling for type-1 error caused by multiple testing.³³ Given the moderate sample size of AU in combination with predominantly highly significant results, findings are considered likely to be valid.³⁴ Moreover, it must be emphasized that also the individual competency gain, collected in a pre/post design is still a subjective measure. While some authors suggest a poor correlation between subjective and objective knowledge outcomes,³⁵ others found that high self-reported preparedness translates into good performance³⁶ Although these considerations highlight the need for objective, randomized-controlled data, there is currently no evidence on learning outcomes of VR-based complex emergency medical trainings.^9,37 Admittedly, a randomized-controlled setup for examining objective endpoints (skills or knowledge), eg within the context of an objective structured clinical examination (OSCE) is significantly more elaborate and may initially lead to technical issues. Therefore, as part of the implementation, we opted for a questionnaire-based study design as a first step. However, a corresponding approach should be undertaken to objectively investigate the translatability of the results into the clinical-practical context.

Lastly, this study addressed the relationships between technological proficiency, usability, and learning experience to identify potential barriers to the widespread implementation of VR technology. We found no significant correlation between technological proficiency and learning outcomes, suggesting that all students benefited from VR training regardless of technical skills. The minimization of the operational complexity of the software STEP-VR is likely one of the factors responsible for this. All interactions occur with a single controller button (plus an additional one for occasional teleports) and the interaction elements within the software have been continuously refined to be more intuitive since 2018. However, the homogeneous and low prior experience in PC and VR usage of the sample must also be mentioned as a possible (additional) factor. Perhaps a correlation would emerge more clearly with a more varied distribution of prior experience.

While literature reports some correlations between acceptance, learning success, and computer affinity or prior 3D/VR experience,¹⁸ many studies overlook the link between these parameters and the actual (self-assessed) technical abilities.^15,17 In the correlation between the attribute “technological proficiency” and the degree of intuitive use, we see support for the validity of this construct. In contrast to that, demographic factors on previous use of console and VR gaming exhibited no influence on the degree of intuitive use (group comparisons, data not shown). This might suggest that the measure of technological proficiency is more suitable for examining the usability of a new VR program based on the individual prerequisites of the users.

While subjective learning success correlated significantly with intuitive use, this was only marginally the case for perceived competence gain. We interpreted this to mean that the retrospective assessment of subjective learning success is more strongly influenced by the positive usability experience. Therefore, determining perceived competence gain could be a more independent and hence more meaningful parameter.

Strengths

The study documents the implementation of a VR training program in the surgical curriculum. While most studies in a surgical context analyse VR technology used to train practical skills, our study included the development of clinical competencies in managing interdisciplinary emergencies. The relatively large sample size of this mandatory course also included students with critical attitudes towards new technologies. In addition to a comprehensive set of user-related and technical parameters, learning outcomes were measured retrospectively as well as in a pre/post design. This approach provided further insights into the complex interplay of individual-specific, technical factors and learning experience in immersive settings.

Limitations

The study was conducted at a single medical school in Germany and restricted to the final academic year, limiting generalizability of the results. As only students that successfully completed the inverted classroom were able to participate as AU, a selection bias cannot be ruled out. This discrepancy in interaction may have influenced their perceptions of the effectiveness of VR. Future studies should consider evaluating learning outcomes separately for each group to assess the differential impact of active and passive participation. Moreover, the study relied solely on subjective responses from participants, and the effects of the training were not compared with a control group. Future research ideally should assess objective performance (eg in the framework of clinical-practical examinations), resulting from VR training and control conditions.

Conclusion

The curricular implementation of a VR training program for interdisciplinary emergencies during the later stages of surgical education is organizationally and technically feasible. Student acceptance and enthusiasm are high, and most competency facets can be positively influenced. Further research such as randomised-controlled-trials comparing OSCE results for VR and non-VR sessions are necessary to evaluate the objective learning outcomes.