A Clinically Immersive Medical Innovation Program for US MD Students: Curricular Description and Program Outcomes

Introduction

The modern American health care system faces a diversity of challenges, including but not limited to the soaring costs of care, the need for improved patient experience, and particularly in recent years the need for rapid innovation of medical technology and processes capable of addressing such challenges.^1,2 Considering these challenges, the need for physicians and all healthcare workers to bring an innovative mindset into their work and to actively solve problems in the clinical setting has never been more apparent.³ To prepare physicians for these challenges, it is paramount that medical education programs across the country bestow an innovation-oriented mindset in medical students.

While traditional medical education excels at training clinical skills, there is much less focus on the development of physician-innovators, who are trained to actively identify areas for clinical and systems-level improvement throughout their practice.⁴ Medical students will eventually become clinical leaders within their respective specialties and will be among those who most often experience these health care challenges first-hand. Therefore, it is essential for modern physician-trainees to learn the skills necessary to identify, evaluate, and rapidly solve unmet needs in our health care system, all skills traditionally taught in programs known as “clinical immersion programs”. Such programs teach students how to identify unmet clinical needs, design solutions to solve them, and empower students to engage meaningfully with the clinical staff and environment.

While prevalent within the biomedical engineering education field, the rate of adoption of such clinical immersion medical innovation programs has been slow among US MD programs. Indeed, only 26 of 170+ US MD programs offer an innovation or entrepreneurship program, and the rate of growth has averaged only two new programs per year since 2007.⁴ The described slow rate of adoption is despite a strong interest among medical students in learning the associated content.⁵ To our knowledge, there is no published literature reporting on the efficacy of such an innovation program for teaching methods of innovation to medical students, and reports of such innovation courses for other training schools (engineering, nursing, etc.). are scant.^6,7 Furthermore, the literature describing the specific curricular structure of US MD medical innovation programs and their associated educational effectiveness is limited.⁸

To fill the described gap in the literature, our team set out to develop and offer a clinically immersive medical innovation program geared towards the educational needs of US MD students and readily integrated with the US MD curriculum. Clinical immersion programs are defined by their unique incorporation of clinical ethnography and interviewing sessions conducted by the students in order to refine needs identification and problem solving skills. The objective was to offer medical students a formal and hands-on education in medical innovation with course theory rooted in existing medical innovation frameworks^9,10 but tailored for the US MD student. In this report, first we describe the curriculum of a novel clinically immersive medical innovation program that we offered to 16 UCLA MD students over the course of six weeks during an otherwise unscheduled summer period. We then assess program efficacy, assess student and faculty perceptions of the novel clinical immersion program, and determine the reasons for success or failure of the proposed teaching methods. The traditional¹¹ Kirkpatrick model was followed to measure the program’s ability to achieve these objectives. Specifically, learners’ feelings, learnings, transference of knowledge, and real-world impact were assessed using this model. Additionally, the process by which success or failure for each outcome was achieved was determined through a quality improvement conceptual framework examining multiple process measures. Such a framework enabled iterative and real-time improvement of the program.

Methods

This research was conducted with the approval from the University of California, Los Angeles institutional review board (IRB# 21–001608). A waiver of consent was granted for this research.

Curricular Development

The curriculum offered to the UCLA MD students in this study was modeled after existing and reputable medical innovation frameworks, namely approaches developed at Stanford University and Johns Hopkins University. Stanford first proposed the formal instruction of biomedical technology innovation in 2011 and has since been credited with the creation of the “Biodesign” field.⁹ Select topics and teaching examples derived from the Stanford Biodesign methodology include needs statement formulation, needs prioritization, methods for solution concept generation, solution concept prioritization, regulatory pathways for medical technology, and reimbursement of medical goods and services.⁹ In 2013, researchers at the Johns Hopkins Center for Bioengineering Innovation and Design published a 4-year experience describing a “Spiral Model of Innovation” which had at the time already successfully spun out numerous well-funded medical technology companies from student-led projects.¹⁰ The four quadrants of the spiral model implemented in our curriculum include clinical, commercial, technical, and organizational – strategic. The aforementioned concepts and techniques detailed in the Stanford Biodesign methodology were adapted to communicate methods for selecting and de-risking medical innovation projects and solutions specific to each quadrant of the Johns Hopkins spiral model in order to form the curriculum offered to the students at UCLA.

Student and Faculty Recruitment

MD students and faculty at UCLA were made aware of the opportunity to participate in our 6-week summer program through emails, class announcements, and social media. The recruitment phase concluded with 16 MD students registered for the summer course, each paired with at least one clinical faculty mentor in a medical specialty of interest with whom they completed their clinical immersion. Faculty was excluded if they were unable to oversee a minimum of six hours of weekly clinical immersion for each week of the course.

Course Offering

The first didactic session that was taught via video conferencing due to ongoing coronavirus (COVID) restrictions. All subsequent sessions were taught in-person because COVID restrictions were lifted. Students reported once per week for didactics and in parallel attended clinical immersion sessions at a time coordinated with their clinical mentor. Students completed one pre-course and one post-course survey. Student additionally completed two surveys each week: one gauging perceptions of the prior week’s didactic session and one gauging perceptions of the prior week’s clinical immersion. By constantly gauging student feelings throughout the program and simultaneously eliciting qualitative feedback, the curriculum was iteratively improved throughout its offering.

Study Design

This was an experimental study. Participant baseline attitudes, perceived knowledge, and skill in medical innovation were surveyed before the intervention, namely the course offering. Then, each participant completed a seven-week medical innovation course, and they were subsequently reassessed for changes in the originally assessed categories. Medical faculty provided expert opinions on student progression through the course in addition to student perceptions. All surveys were designed to assess the training program using the traditional Kirkpatrick model. Kirkpatrick Level 1 “Reaction” evaluation was achieved by surveying students on a weekly basis about how they felt about that week’s didactics and clinical immersion. Kirkpatrick Level 2 “Learning” evaluation was completed by comparing students’ perceived confidence in their own skills at key tasks during the pre- and post-course surveys, and by having clinical mentors evaluate the change in their mentee’s skill at key tasks and knowledge of relevant topics. Kirkpatrick Level 3 “Behavior” was assessed by tracking the extent to which students attended and engaged in clinical immersion sessions for each week following didactics. Finally, Kirkpatrick Level 4 “Results” was assessed by measuring overall outcomes of the program for each student, including the number of needs discovered and stakeholders interviewed, as well as faculty-perceived importance of needs, and overall value-added to their clinical practice due to their mentee’s clinical immersion.

Data Extraction

All surveys were administered during weekly didactic sessions. A unique ID was assigned to each student, so survey response forms were de-identified. Each week, one survey pertained to the didactics session, and a second surgery pertained specifically to the clinical immersion experience for that week. Surveys were administered using Google Forms during the week one video conferencing session, and on pencil and paper during all subsequent weeks when sessions were in-person. All responses were manually transferred into local Excel files on an encrypted computer.

Data Analysis

All data analysis was conducted using Microsoft Excel (2022). The charts feature within Excel was used to generate all figures. All numerical manipulations including frequency counts were also performed in Excel.

Results

Curricular Structure

The curricular design phase of this study resulted in the conception of a six-week clinically immersive medical innovation curriculum followed by a pitch session during the seventh week. Each week of this curriculum consists of one two-hour didactics session paired with approximately six hours of clinical immersion each week for six weeks. Each student rotated in a different clinical setting, depending on which medical subspecialty their clinical faculty mentor practiced in. Weekly didactic sessions typically consisted of a one-hour student-led lecture followed by a guest lecture given by a clinician or entrepreneur with real-world experience in the topic of that didactic session. We designed each didactic session with a “learn by doing” mindset with the additional goal of promoting a collaboration in innovation. Accordingly, we interspersed frequent small-group student activities throughout our lectures whenever possible.

After learning the theory and seeing applied examples in the weekly didactic session, students then went into the clinical setting and utilized their acquired knowledge to identify, refine, and solve unmet clinical needs on a weekly basis. During clinical immersion sessions, students performed a combination of ethnography (passive observation of ongoing clinical activities) and interview-based insight extraction (when the student interviewed someone in the health care setting to identify and characterize needs or solution concepts). Students observed clinical practice and additionally conducted interviews with health care providers to identify and validate the unmet clinical needs they discovered. A final needs pitch was given by each student to an audience of clinical faculty, course instructors, and peers. A student “pitch” session constituted the seventh and final didactic session, which consisted of each student delivering a 15-minute presentation explaining the top unmet clinical need they had identified and outlining their leading potential solutions. The audience of each pitch session consisted of fellow course participants, course instructors, and medical faculty. A question and answer period followed each presentation, where students were also given feedback from instructors regarding next steps. A curriculum timeline is shown in Figure 1. A detailed description of the curriculum on a week-by-week basis is available in Supplemental Material 1.

Figure 1 Curriculium timeline detailing week-by-week topics and duration of clinical immersion.

Kirkpatrick Model Evaluations

The results for Kirkpatrick level 1 (reaction) evaluations are summarized in Figure 2. The average student rating was 9.5/10 for the main didactics lectures (88% response rate) and 9.5/10 for the guest lectures (83% response rate). The overall rating for the clinical immersion experience was 9.1/10 (96% response rate). Students displayed strong support for the adoption of a similar medical innovation program into the official MD curriculum, felt significantly more empowered to add value to the medical system, and were largely interested in continuing work on their projects, and became more interested in medical innovation by the end of the course.

Figure 2 Kirkpatrick level 1 (reaction) evaluations: (A) Student ratings of didactics main lecture, didactics guest lecture, and clinical immersion experience by week, and student reactions; (B) Student post-course perceptions of the program.

The results for Kirkpatrick level 2 (learning) evaluations are summarized in Figure 3. Student-perceived confidence increased in all 14 key skills, with an average increase in confidence of 43% across all skills. Overall, students were most confident in performing clinical ethnography by the end of the course (2.7/3 or 90% confidence). The majority of faculty (n = 9, 75%) reported observable improvement in their mentee’s clinical ethnography and stakeholder interviewing skills.

Figure 3 Kirkpatrick level 2 (learning) evaluations: (A) Student-perceived level of confidence before and after the course in 14 key skills; (B) Faculty post-course assessment of mentee skill progression.

The results for Kirkpatrick level 3 (behavior) evaluations are summarized in Figure 4. The extent to which students applied the theoretical framework introduced during didactics was assessed by the amount of time spent each week in their voluntary clinical immersion experience and by their faculty-perceived level of engagement during clinical immersion sessions. Overall, each student averaged 6.1 hours of clinical immersion per week. Faculty felt strongly that the students were engaged in their clinical immersion experience.

Figure 4 Kirkpatrick level 3 (behavior) evaluations: (A) Class wide hours of clinical immersion by clinical location; (B) Faculty perception of their mentee’s level of engagement during clinical immersions.

The results for Kirkpatrick level 4 (results) evaluations are summarized in Figure 5. The results focused on clinical value derived from the students’ immersion experiences. Overall, each student discovered on average 2.6 faculty-verified unmet clinical needs per week and interviewed on average 2.4 stakeholders per week. The vast majority of clinical faculty agreed that the program resulted in the identification of important unmet clinical needs within their practice and that value was added to their clinical practice by participating in the program.

Figure 5 4 Kirkpatrick level 4 (results) evaluations: (A) Average number of faculty-verified needs discovered and stakeholders interviewed per student per week; (B) Faculty assessment of if the program resulted in the identification of unmet clinical needs; (C) Faculty assessment of if the program resulted in the identification of important unmet clinical needs; (D) Faculty assessment of if the program added value to their clinical practice.

Discussion

Existing Literature

There remains a paucity of formalized and clinically immersive medical innovation educational opportunities for US MD students. A 2020 scoping review of innovation and entrepreneurship (IE) programs in health science concluded that a significant gap exists in our understanding of IE programs in medical education after finding that no reviews or experimental papers focused on health education methods. The authors called for increased recognition of the importance of IE curricula, increased faculty familiarity and involvement with such curricula, and continued reporting on curricular innovations in IE medical education. We believe our research takes a significant step towards addressing the described gap in understanding by experimentally investigating the efficacy of such an innovation curriculum when trialed with US MD students. Although a small number of other medical schools have IE programs advertised on their websites,¹² none to our knowledge have reported outcomes, let alone outcomes of a clinically immersive curriculum. Sling Health National and the organization out of which it was spun, IDEA Labs, have both developed national medical technology incubators, which provide medical students with experiential learning opportunities in medical innovation.^13,14 These, however, are not formal curricula integrated into US MD programs, and do not emphasize the development of clinical ethnography and needs discovery skills, but rather the advancement of ideas that already exist. Indeed, despite working on multidisciplinary teams, the students themselves are less involved in problem identification. We believe a curriculum geared towards honing the ethnographic skills of MD students in their early stages of training is critical, given they serve as a fresh set of eyes and have an entire career of clinical duties before them during which to identify unmet needs.

Additional review of the literature revealed that other published medical innovation programs were designed for the education of engineering students or were not clinically immersive. One program was clinically immersive and enrolled chemical engineering and nursing students to find unmet clinical needs and subsequently develop solution concepts and prototypes.⁶ Another program involved clinical immersion for needs discovery by undergraduate biomedical engineering students who then completed a capstone project with the input of medical students, residents, and faculty.¹⁵ The bioengineering department at the University of Illinois Chicago has offered its bioengineering students with clinical immersion experiences for years and has reported positive outcomes.⁷ Among programs for medical students, none were clinically immersive: one study conducted a three-day innovation and design thinking workshop without any clinical immersion and assessed medical students’ design thinking mindset before and after the course with a 71 question survey, citing improvement in various design thinking metrics.¹⁶ A final study had students complete “virtual” clinical rotations and create documentary summaries based on their experience.¹⁷

Program Outcomes

Kirkpatrick level 1 (feelings) outcomes were positive for the training program, with ratings of 9.5/10 for all lectures and 9.1/10 for clinical immersion. Specific qualitative feedback from students emphasized a particular preference for hands-on activities during didactic sessions, such as clinical immersion reflection activities, needs formation practice sessions, genealogy mapping, and mutually exclusive, collectively exhaustive (MECE) mapping in-class activities. The overall didactics attendance rate of 97% and average of >6 hours of clinical immersion per week are testament to the students’ positive experience each week.

Kirkpatrick level 2 (learning) outcomes were positive for the training program, with increased student-perceived confidence in all 14 key skills and improvement in ethnography and stakeholder interviewing skills as perceived by the vast majority of clinical faculty. Being a soft skill, knowledge and comfort in ethnography and stakeholder interviewing are not easily assessed in a traditional examination. Therefore, student confidence levels and faculty perceptions of clinical performance were prioritized for Kirkpatrick level 2 evaluations.

Kirkpatrick level 3 (behavior) outcomes were positive for the training program, each student voluntarily completing over six hours of clinical immersion per week to apply theoretical learnings, and the vast majority of clinical faculty judging their students as highly engaged during clinical immersion. The relevant skills of ethnography were used as evidenced by over two faculty-verified needs discovered and stakeholders interviewed per week, per student. The rate of needs discovery and stakeholder interviewing held relatively constant on a week-by-week basis.

Kirkpatrick level 4 (results) outcomes were positive for the program, with steady needs discovery and stakeholder interviewing weekly by students as previously discussed, and clinical faculty endorsing the discovery of important clinical needs and added value to their clinical practice because of student clinical immersion. The beauty of a clinically immersive ethnography program for is that in addition to trainee development, there is genuine clinical value derived from student activities as seen here. Indeed, multiple clinical faculty continued work on discovered needs with their mentees, resulting in multiple published abstracts, poster presentations, university-supported projects, and publications.

Novelty

To our knowledge, we are the first in literature to design and offer a medical innovation curriculum specifically to medical students that is both clinically immersive and readily integrated into a typical US MD curriculum. We designed this course specifically for medical students because them being in the nascence of their clinical career offers the chance to impress upon them the impact that they can have as an individual entering an unfamiliar environment. Didactic sessions accordingly emphasized developing clinical ethnography skills because the students’ fresh perspectives play well into being an unbiased ethnographer. Hands-on ethnographic and stakeholder interviewing activities were held during didactic sessions to empower students to exercise these skills during weekly clinical immersion. A significant amount of didactics time was spent teaching root cause analysis of clinical observations for true need determination, and on subsequent need statement formulation. This allowed students to transfer their raw observations into well-crafted unmet clinical needs. Indeed, students ranked ethnography skills that needed statement formulation were ranked within the top three out of 14 key skills. Guest lectures with success in the medical technology startup space were brought in each week to provide longitudinal and real-world examples of the theoretical concepts emphasized that week in didactics. No other medical innovation courses have described the integration of field experts as guest lectures. Qualitative feedback from students indicated that guest lectures and clinical immersion were important aspects of the course for learning how to apply medical innovation theory to real-world scenarios. The innovation skillset taught in this course may be applicable in emergency scenarios requiring rapid innovation. One such example is most obviously the COVID-19 pandemic, served as a major catalyst for medical innovation.¹⁸ Equipping future generations of physicians with the skillset to identify unmet clinical needs and rapidly develop solutions for such needs could prove instrumental in handling future upheavals in modern health care which demand rapid response to changing circumstances in health care.

This study also describes the first clinically immersive medical innovation program offered to medical students with an ongoing mechanism for iterative program improvement driven by real-time student feedback. Weekly surveying of student feelings helped identify most effective learning methods. It became apparent that hands-on activities were the most engaging for students and therefore facilitated learning, so the portion of didactic sessions dedicated to hands-on activities was steadily increased. Approximately 30 minutes per main didactics lecture were dedicated to hands-on activities in the final weeks. Didactics instructors also held 1-on-1 check-ins with each student halfway through the course to provide personalized mentorship regarding needs discovery and address any educational gaps. We believe investing in students in this way improved their overall engagement in the curriculum and enhanced the educational experience.

Future Direction

Given the positive outcomes of the described training program, this curriculum has been formally integrated into our institution’s medical school curriculum as an optional track within the ‘Year of ‘Discovery, which is a largely unstructured block of approximately nine months when students are afforded the opportunity to explore a specific area of interest within medicine in great depth. By making this course an optional elective, it allows for it to fit into an already busy MD curriculum and attract only those students who are most inclined to learn the fundamentals of medical education. Future studies will explore the efficacy and reception of the program as a part of the official MD curriculum at our institution.

Limitations

This study has several limitations. The sample size of students was small and consisted of medical students from a single institution, therefore limiting external validity. This is why further evaluation of this methodology for teaching medical innovation to medical students is necessary at additional institutions to determine the generalizability of these findings. Additionally, course enrollment was voluntary, introducing the risk for volunteer bias, which could make the group of enrollees inherently different from the average medical student. Finally, this study only assesses initial student and faculty perceptions, and long-term follow-up is necessary to assess the durability of program impact on student learning and activities.

Conclusion

We developed and trialed a novel clinically immersive medical innovation curriculum tailored for medical students. As evaluated by the traditional Kirkpatrick model, this program achieved positive outcomes on all four levels. Overall, the described curriculum requires a relatively low commitment in time of fewer than ten hours per week for six weeks so as not to interfere with existing student activities, but still allows for weekly didactics covering the fundamentals of medical innovation coupled with clinical immersion. Our findings have driven the local adoption of this program into our institutions official medical school curriculum as an available track within the “Year of Discovery” which is a built-in nine-month period of unstructured scholarly exploration. We hope that the program efficacy demonstrated herein catalyzes more institutions to trial similar medical innovation programs with medical students and report their findings.