A Randomized-Controlled Trial Using Point of Care Ultrasound to Evaluate Volunteer Patients in the Emergency Department Versus a Manikin Simulator for Improving Knowledge and Confidence of Hypotension and Shock in Medicine Sub-Internship Students

Introduction

Point of Care Ultrasound (POCUS) has rapidly grown in importance in clinical practice due to its ability to quickly answer diagnostic questions, guide clinical decision-making, and facilitate safe procedures at the bedside.¹ POCUS has spread broadly from its initial usage in Emergency Medicine and Critical Care to pervasive adoption, including by Internal Medicine physicians as exemplified by training guidelines and society statements.^2–4

In undergraduate medical education (UME), several medical schools in the United States have also incorporated ultrasound education in their curricula.⁵ While there has been an interest in creating an integrated competency-based ultrasound curriculum at the UME level, specific methods to accomplish this are not yet available.^6,7

One aspect of POCUS education that is cited as a challenge in curriculum development is resource limitations in both number of instructors and patients or simulation models available.⁵ In an Academic Medicine study from 2022, one hundred fifteen (94%) schools identified barriers to implementing POCUS training in UME, which included lack of trained faculty (63%), lack of time in current curricula (54%), and lack of equipment (44%).⁵ Only seven (6%) schools identified no barriers.⁵

Given the broad evidence for the benefits of simulation over traditional didactics, particularly for acute clinical care education,^8–14 simulation as a method of learning POCUS may offer an excellent way of approaching this practical, resource limitation challenge. It is uncertain from multiple studies whether students gain more skills in POCUS from real patients or manikin simulators.^14–17

One clinical area where POCUS particularly excels is in the diagnostic assessment of patients with hypotension or shock.^18–20 POCUS has multiple ways of aiding diagnosis including at least seven proposed structured POCUS assessments during cardiac arrest.²¹ These assessments largely overlap and direct sonographic assessment for evidence of cardiac tamponade, massive pulmonary embolism, tension pneumothorax, aortic dissection, ruptured aortic aneurysm, and/or hypovolemia.²¹ In order to provide medical students on an internal medicine sub-internship the skills to assess patients with hypotension and shock using POCUS, we designed a randomized-controlled trial to compare two different teaching methods. We randomized students to learn from real, volunteer patients in the emergency department (ED) versus learn from a manikin simulator in a simulation center. The primary end point was assessment of student knowledge and confidence after a workshop and an individual POCUS practice session at a 3-month longitudinal time period. Our objective was to determine whether live volunteer patient practice with POCUS was needed to acquire confidence and knowledge or whether the manikin simulator provided an adequate opportunity for learning the necessary knowledge and developing confidence in POCUS.

Methods

Institutional Review Board Approval

Institutional review board approval for this randomized-controlled trial was obtained from both the Mass General Brigham Institutional Review Board and the Harvard Medical School Program in Medical Education’s Executive Committee in 2021. The study was conducted from January to August 2022. Our study complies with the Declaration of Helsinki. Verbal consent was required of the Volunteer Patients in the Emergency Department by our Ethical Review Board.

Participants

All 47 Harvard Medical School medicine sub-internship students who rotated at Brigham and Women’s Hospital during the study period participated in an educational workshop and individual ultrasound practice sessions as required components of their clinical clerkship. Prior to the workshop, students received information informing them of voluntary participation in this study through completion of optional surveys. This included information informing the students the purpose of the study, the voluntary nature of the study, and the consent process. Students also received information about this study, in person, prior to the start of the educational workshop. Completion of the approved anonymous study surveys was voluntary. Students were told that they would receive a gift card of $10.00 per survey for each of three surveys for a maximum of $30.00 for the three surveys’ completion.

Volunteer patients in the ED were recruited by the study faculty (VD, AG) to receive supervised POCUS examinations by participating medical students. Volunteer patients in the ED were required to be clinically stable without hypotension, shock, or altered mental status. These patients provided verbal consent to participating in supervised POCUS examinations by the sub-internship students.

Patients were included if they met inclusion criteria of age 18–99 while in the ED. Patients were excluded if they had emergent serious medical problems that could have posed a disruption to Emergency Department operations, any vital sign abnormalities deemed to be unstable, altered mental status that prevented the patient from providing verbal consent, or were under the direct care of the study faculty investigators. The study faculty were required to not be involved in the recruited patients’ clinical care in the ED. Images obtained were reviewed at the time of acquisition and patients were not billed for these images. A plan for discovery of incidental findings included ensuring follow-up with the patient’s clinical team or primary care physician. During the study period, no incidental findings were discovered.

Didactic Workshop

All students participated in a 90-minute educational workshop experience consisting of three components: 1. A didactic review of the physiology and assessment of patients with hypotension and shock; 2. A hands-on ultrasound session with a manikin simulator focusing on three standard views (parasternal long axis, subxiphoid four chamber, and subxiphoid longitudinal inferior vena cava views); and 3. A simulation-based case of a manikin patient with cardiac tamponade including relevant X-ray and ultrasound findings.

Student Randomization

Students were block randomized by their sub-internship month using a random number generator to either an experimental group (volunteer patients in the ED) or a control group (simulated patients on a manikin simulator program) with an equal number of students per group. Each student was assigned a unique identifier by month and randomization number to provide anonymity in analysis.

Setting 1: Experimental Group – ED at Brigham and Women’s Hospital

Medical students randomized to the ED performed two separate POCUS examinations on two separate volunteer patients directly supervised by one of two faculty with expertise in POCUS (VD, AG). Students obtained the three standard ultrasound views using a Mindray ME8 Ultrasound Machine (Mindray North America, Mahwah, NJ). Volunteer patients in the ED at Brigham and Women’s Hospital were required to be clinically stable without hypotension, shock, or altered mental status so that they could provide verbal consent to having the sub-internship student perform a supervised ultrasound examination.

Setting 2: Control Group – Neil and Elise Wallace STRATUS Simulation Center at Brigham and Women’s Hospital

Medical students randomized to the simulation center performed two separate supervised POCUS examinations using the manikin simulator program, Vimedix ultrasound simulator (Elevate Healthcare, Sarasota, FL), supervised by study faculty (HS). Students obtained the three standard ultrasound views. The students completed simulated cases of normal anatomy and various cardiac pathologies such as acute anterior myocardial infarction and chronic dilated cardiomyopathy.

Study Surveys

Students were assessed using surveys consisting of knowledge-based questions and Likert-scale-based questions gauging their responses to the workshop. These surveys were completed at baseline prior to the workshop (Pre-Survey, Supplementary Material A), immediately after the workshop (Post-Survey, Supplementary Material B), and three months following their completion of the workshop and ultrasound practice (3-Month Longitudinal Retention Survey, Supplementary Material C). Of the 47 sub-intern students, 94% (n = 44/47) completed the pre-survey, 87% (n = 41/47) completed the post-survey, and 81% (n = 38/47) completed the 3-month longitudinal survey with no differences in completion between the two randomized groups.

Statistical Analysis

Survey data were analyzed using IBM SPSS Statistics v.28 predictive analytics software.

Knowledge-based questions were re-coded into dichotomous (correct/incorrect) variables in which correct answers were coded ‘1’ and incorrect ones ‘0’. Each question was analyzed using Fisher’s Exact Test. In addition, cumulative total scores of the five knowledge-based items were calculated. Total score means, pre-surveys, post-surveys, and longitudinal surveys were analyzed using Fisher’s Exact Test.

Confidence questions were re-coded into Likert-style attitudinal questions, using a 1 to 5 scale (1 = Strongly Disagree and 5 = Strongly Agree) for each item. Mean scores for each item, pre- and post-survey, were analyzed using the Wilcoxon Signed Rank test. The same process was employed to re-code and analyze responses to the 3-month longitudinal survey. The point of comparison for analysis of the 3-month longitudinal data is the immediate post-survey.

Results

A total of 47 students participated in the workshop on hypotension and shock. Twenty-four students were randomized to the experimental group in the ED, while 23 students were randomized to the control group using the manikin simulator in a simulation center. Of these students, 94% (n = 44/47) completed the pre-survey, 87% (n = 41/47) completed the post-survey, and 81% (n = 38/47) completed the 3-month longitudinal survey with no differences in completion between the two randomized groups.

At the 3-month longitudinal retention survey, there was no statistical difference in the primary outcome of questions correctly answered by students in the experimental group compared to those in the control group (88% vs 86.5%, p = 0.713, NS) (Table 1). Additionally, there was no statistical difference in the primary outcome of confidence between students in the experimental group from that of the control group (4.22 vs 4.10, p = 0.846, NS) (Table 1).

Table 1 Differences in Confidence and Knowledge Between Experimental and Control Groups at the 3-month Longitudinal Survey

In terms of secondary outcomes, all students between the pre-survey and post-survey following the workshop reported statistically significant increases in confidence in their ability to systematically examine patients with hypotension (3.0 to 4.1, p < 0.001) and in their understanding of the underlying physiology of differing entities that lead to shock (3.7 to 4.2, p < 0.001) (Table 2). Students also reported significant increases in confidence in their ability to recognize (2.4 to 4.1, p < 0.001) and obtain (2.0 to 4.0, p < 0.001) ultrasound images of the inferior vena cava, and incorporate these images with the clinical examination for volume status assessment (2.4 to 4.1, p < 0.001) (Table 2).

Table 2 Results of Likert Scale-Based Confidence Questions

In the knowledge-based questions, students showed statistically significant increases in correct questions between the pre-surveys and post-surveys with demonstration of knowledge retention at 3 months (Table 3). There were no statistical differences between the experimental group and control groups.

Table 3 Results of Knowledge-Based Questions

Discussion

Our randomized-controlled trial of POCUS training in medicine sub-internship students comparing their knowledge and confidence after performing POCUS on volunteer patients in the ED versus on a manikin simulator in a simulation center showed no statistically significant differences between the experimental (volunteer patients) and control (manikin simulator) groups. We demonstrate equivalence rather than superiority of volunteer patients versus manikin simulators for learners’ knowledge and confidence. These randomized-controlled trial results support the fact that teaching POCUS to early learners can be done equally well with volunteer patients or through utilization of a manikin simulator. These findings align with the previous literature in POCUS which has not shown definitive benefits of one method over the other.^14–17 When educational resources are limited, it is valuable to recognize that a manikin simulator will provide students with sufficient knowledge and confidence for basic POCUS skills.

Our study also had the opportunity to evaluate secondary outcomes in various aspects of POCUS education. All students demonstrated significant increases in confidence in their ability to systematically examine patients with hypotension and in their understanding of the differing physiology of entities that result in hypotension and shock. Students also reported increased confidence in their ability to recognize, obtain, and interpret images of the inferior vena cava for volume status assessment. Although these findings are hypothesis generating because they are secondary outcomes, the findings suggest that a similar workshop can improve students’ confidence by incorporating POCUS into their clinical assessment of hypotensive patients.

The lessons we learned from our randomized-controlled study include the fact that early learners’ knowledge and confidence of POCUS are increased by supervised teaching. This occurs regardless of whether individual practice was on a volunteer patient or a manikin simulator. At a longitudinal time point, students did not show decay of knowledge or confidence in either of the two groups.

There are several limitations to our findings. These include a small sample size and lack of direct assessment of observed POCUS skills. In addition, there was a lack of blinding of the faculty and students to the randomized intervention. Also, given the nature of our design, crossover in exposure to a manikin simulator occurred during the workshop when introducing students to basic POCUS views. The students also had a high baseline level of knowledge prior to the workshop as evidenced in the pre-surveys. These limitations may potentially limit the generalizability of our findings.

The strengths of our study include a high response rate (augmented by gift cards) with evidence of longitudinal knowledge retention at 3 months and the achievement of at least Level 2 (Learning) in the Kirkpatrick Model of Program Evaluation.²² Participants in the workshop showed significant gains in learner confidence before and after exposure to the workshop teaching. All three faculty members (VD, AG – Experimental Group; HS – Control Group) involved in supervising the students were experienced medical educators committed to providing the best educational experience possible for each student regardless of their randomized setting.

Given our results, we encourage other medical schools to incorporate POCUS education into their clinical clerkship rotations through either supervised volunteer patients or supervised manikin simulators. Both proved valuable to our students’ knowledge and confidence. The availability of faculty, willing patients, and simulation equipment will dictate which resource option is preferable for each institution. We believe that our workshop and teaching methods for POCUS could be utilized at points earlier in a medical school’s curriculum, as well as inserted into advanced electives in trauma, emergency medicine, critical care, and cardiology. Future work can include a direct competency assessment of students’ POCUS skills using audio-video recordings analyzed by judges using a blinded checklist. This assessment would be done to detect differences between students who learned from volunteer patients versus from a manikin simulator.

Conclusion

In this randomized-controlled trial using POCUS to assess hypotension and shock, we demonstrate equivalence rather than superiority of volunteer patients versus manikin simulators for learners’ knowledge and confidence with no significant differences between these two randomized groups. All enrolled students demonstrated significant increases in both their knowledge and confidence of hypotension, shock, and basic POCUS skills after completing a workshop and an individual practice session. Both knowledge and confidence were maintained at a 3-month longitudinal time period for each group. In future studies, we recommend that a direct competency assessment of POCUS skills using audio-video recordings be performed to enhance the evaluation of POCUS skills in addition to survey questions assessing knowledge and confidence.