Delving into the Practical Applications and Pitfalls of Large Language Models in Medical Education: Narrative Review

Background

Large language models (LLMs), known as a cutting-edge artificial intelligence (AI) technology, have garnered significant attention since the release of the Chat Generative Pre-trained Transformer (ChatGPT) in November 2022.¹ In contrast to traditional tools like AI generators, which typically rely on predefined algorithms and datasets, LLMs leverage vast datasets and advanced neural network architectures to perform a wide range of tasks. These models possess a remarkable ability to understand textual subtleties and generate more nuanced and contextually relevant outputs in real-time conversations, blurring the distinction between human and machine generated content.²

LLMs are increasingly recognized as essential instruments in medical education to equip future physicians and healthcare professionals with adequate training.³ With the help of LLMs, teachers may develop multiple-choice questions that more accurately reflect real-world clinical scenarios,⁴ and educational institutions can make data-driven decisions regarding curriculum development, student assessment, and resource allocation. For learners, LLMs offer the capability to generate content tailored to their individual interests, cognitive abilities, and learning styles, thereby enhancing the creation of adaptive learning environments.⁵ Collectively, LLMs have made substantial strides in transforming medical education practices.

Despite the existing literature highlighting the potential of LLMs to transform educational paradigms, it often falls short of providing concrete, real-world examples of implementation within medical curricula.^6–9 It may be attributed to the fact that the audience for this topic is typically broad, encompassing medical educators, researchers, students, and policymakers, among others. Due to the significant differences in the specific needs and focuses, providing concrete examples may not meet the demands of all readers and could potentially compromise the universality and applicability. Furthermore, much of the current discourse relies on grey literature, including websites, blogs, and news outlets, which may propagate misinformation and lack the rigor of peer-reviewed research.¹⁰ The challenges associated with LLMs necessitate careful consideration as well. Given these gaps, there is a pressing need, particularly for most medical educators, to explore and evaluate the implementation of LLMs in real-world contexts and to generate robust, peer-reviewed evidence.

In this study, we endeavor to offer concrete examples of LLMs bridge the gap between theoretical frameworks and practical applications, with a primary focus on insights derived from peer-reviewed literature. We begin by discussing the application scenarios of LLMs before, during, and after class. Following this, we delineate the challenges encountered in employing LLMs and present corresponding possible solutions (Figure 1). This research contributes to advancing the application of artificial intelligence language models within the context of medical education.

Figure 1 The landscape of large language models (LLMs) in medical education domain. Both of teachers and students can benefits this AI tools, however, some challenges need to be solved.

Method

Given the primary objective of this review is integrative rather than summative, we determined that a narrative review would be more appropriate than a systematic review. Although systematic reviews are often preferred for their reproducibility, they typically have a narrower scope compared to narrative reviews.¹¹ Narrative reviews are particularly advantageous when the aim is to provide a comprehensive overview of a topic. This approach permits a more flexible examination of the literature, accommodating diverse methodologies and study designs.¹¹ While systematic reviews adhere to stringent reporting standards, the standards for narrative reviews are less well established. In formulating our methodological approach, we referred to the Scale for the Assessment of Narrative Review Articles to ensure the quality assessment of narrative review articles.¹²

Articles were identified via the Biomed Central, Web of Science, and Medline databases, using the key words “large language model”, “ChatGPT”, “medical education” and “surgical training” separately and in various combinations. In addition, we examined the references cited and citing articles. Studies were included if they: (1) utilized empirical data (eg, randomized controlled trials, cohort studies, case reports), (2) were peer-reviewed, and (3) were published from 2010 onwards. Studies were excluded if they: (1) were non-empirical works (eg, editorials, opinion pieces, theoretical frameworks), (2) were not related to the intersection of medical education and LLMs, (3) reported duplicate records, or (4) were written in a language other than English. This approach allowed us to identify and include studies that provided practical applications of LLMs. Both authors were actively involved in the literature search process.

Transformation of Teaching Strategy

Medical educators are encouraged to creatively incorporate LLMs with routine curriculum (Figure 2). Before class, LLMs can assist in formulating teaching plans tailored to diverse learning styles of students. For visual learners, LLMs can transform radiologic images into 3D models within virtual reality interfaces, which allows students to explore anatomical structures in a virtual environment during anatomy classes.¹³ The integration of LLMs enhances virtual simulations by making them more dynamic and offering real-time feedback and personalized guidance as students navigate through virtual scenarios.¹⁴ Auditory learners can benefit from lectures, discussions, and podcasts generated by LLMs. Research on pharmacology podcasts indicates that audio resources improve comprehension and empower students to revisit lectures, prepare for examinations, and clarify complex topics.¹⁵ Similarly, in the field of anesthesiology and intensive care, podcasts show significant positive impacts on knowledge acquisition and clinical skills development.¹⁶ For kinesthetic learners, LLMs in virtual surgery training offer detailed, context-aware instructions at each procedural step, and even interpret and respond to user inputs.⁵ This interaction allows learners to ask questions and receive immediate, customized feedback, which is essential for reinforcing learning and ensuring a comprehensive understanding of complex surgical techniques.¹⁷ Such ChatGPT-assisted surgical training was implemented in a two-week training program, where significant improvement in practical clinical skills was observed.¹⁸

Figure 2 Application areas for teachers (green) and students (red) are shown. Key participants that are relevant to the development of LLMs powered medical education include medical educator, AI scientists, school regulations, and publishers.

During class, LLMs allows for the rapid generation of interactive educational materials, such as questions, task cards, and clinical scenarios (Figure 3A–C). Case-based learning scenarios designed by LLMs may represent diverse patient demographics, thereby ensuring that students encounter a broad spectrum of clinical situations.¹⁹ The creation of culturally sensitive educational cases is essential for training healthcare professionals who can cater to the needs of diverse patient populations.¹⁹ In our examples, LLMs could generate self-check quizzes and clinical cases with detailed answer explanations. This aligns with the finding that the quality and readability of clinical contents produced by ChatGPT are comparable to those authored by human experts.²⁰ Zeng et al utilized open-ended questions generated by ChatGPT for a cohort of urology interns within problem-based learning sessions. They found that students assisted by ChatGPT outperformed the traditional group in exams, and had significant gains in medical interviewing, clinical judgment, and overall clinical competence.²¹ Moreover, LLMs can effectively create realistic and clinically relevant scenarios that facilitate the application of theoretical knowledge to practical situations.²² A recent study demonstrated that engaging students in the diagnosis and treatment of “virtual patients” significantly enhanced compliance and satisfaction.²³ In diagnostic course, LLMs can serve as powerful tools for generating differential diagnoses and supporting clinical decision-making processes. GPT-4, in particular, can produce differential diagnosis lists with a high degree of accuracy, thereby exposing medical students and trainees to a wider array of potential diagnoses.²⁴

Figure 3 Example prompts to ask LLM questions and its response about (A) teaching questions, (B) writing clinical cases, (C) simulating conversation, and (D) testing.

After class, LLMs function as a valuable supplementary resource to construct medical questions for homework assignments and assessments. For instance, LLMs can be used to generate questions that align with Entrustable Professional Activities, which are crucial for assessing medical students’ readiness for clinical practice.²⁵ Another example is the generation of context-rich short answer questions (CR-SAQs). While traditional multiple-choice questions fail to assess higher cognitive skills effectively, CR-SAQs encourage students to engage more deeply with the material, and foster a better understanding of the subject matter.²⁶ It was found that the overall quality of questions generated by LLMs and human are comparable.²⁷ Overall, the incorporation of LLMs into the design and execution of medical education assessments signifies a substantial advancement in educational technology.

To enhance teachers’ proficiency in working with LLMs, some academic guidelines have been developed.²⁸ A structured digital teaching competency framework has been proposed to ensure that medical teachers acquire the requisite skills for effective digital teaching.²⁹ This framework encompasses general digital competencies, specific digital teaching skills, and expertise in employing innovative digital technologies. Another significant pipeline involves dynamic classrooms, intelligent lesson plans, and personalized learning experiences.³⁰ As a whole, these guidelines focus on integrating LLMs into the teaching process to facilitate better learning experiences for students.

Customization for Student Learning

LLMs offer advantages in facilitating students’ learning process in various dimensions. The language barriers and reading comprehension challenges can be easily addressed by LLMs. One study highlights the importance of linguistic clues in reading comprehension among medical students, suggesting that language interventions could improve their comprehension skills.³¹ Additionally, LLMs can be designed to incorporate metacognitive strategies, so as to guide medical students in self-monitoring and employing effective reading techniques.³² For students with specific language impairments, LLMs provide personalized language trainings, and ensure they can fully engage with medical curricula.³³

In other cases, the integration of LLMs into clinical decision support systems has shown promising outcomes in improving the interpretation of medical guidelines.³⁴ In a double-blind randomized study conducted by Emilia, a control group engaging in conversations with AI patients was compared to an intervention group receiving additional feedback from LLMs. After four sessions, the feedback group demonstrated superior performance in creating context and securing information, indicating that AI-simulated conversations with feedback can aid clinical decision-making.³⁵ Furthermore, LLMs increase engagement in self-regulated learning in student-centered curricula, where students are encouraged to plan, monitor, and evaluate their learning strategies.³⁶ Through a randomized controlled trial involving 129 undergraduate medical students, researchers observed that students utilizing ChatGPT not only performed better in short-term orthopedics tests but also achieved higher scores in the final exams of surgery and obstetrics and gynecology.³⁷ Moreover, in collaborative learning environments, LLMs serve as a convenient platform for students to discuss and share knowledge. The use of social media platforms, such as WeChat, alongside LLMs, has been shown to support interactive and participatory teaching methodologies.³⁸

Existing research has evaluated the performance of ChatGPT in advanced Cardiovascular Life Support Training exams, the United State Medical Licensing Examination, and Japanese national medical exams. These findings demonstrated that ChatGPT reached a high level of proficiency so they could also be used to improve the students’ performance.³⁹ However, in our example the answer is incorrect, partly because of the difficulty in identifying images (Figure 3D). In these cases, students should avoid relying on LLMs for every task, like some work requiring students to reflect and document their personal feelings. Engaging in reflective practices allows students to critically evaluate their experiences, potentially disrupting negative perceptions and enhancing their learning outcomes. It is found that nursing students’ perceptions of working with older adults are significantly influenced by their personal experiences and the social contexts they encounter.⁴⁰

In terms of the student feedback, the integration of LLMs into educational practices is demonstrated to significantly enhance their learning experience. One study reported a high level of satisfaction (7.9/10) among students when incorporating LLMs into medical education.⁴¹ Park found that a significantly higher percentage of students supported the use of ChatGPT in class (75.6% vs 17.1%).⁴² For residency education, ChatGPT received positive perceptions regarding ease of use (4.48/5), and helpfulness (4.61/5). Nevertheless, the reasonability (4.00/5) required further improvement.⁴³ Additionally, research indicated that positive openness to LLMs correlated with increased satisfaction and higher applicability.⁴⁴ Therefore, by providing access to a vast array of information and facilitating interactive learning environments, LLMs are revolutionizing traditional educational paradigms and improving student outcomes across diverse disciplines.

Promotion of Academic Performance

The efficacy of LLMs in processing and synthesizing information is particularly beneficial in academic contexts, where time constraints and the necessity for comprehensive data analysis are prevalent. LLMs are capable of creating drafts based on users’ prompts, allowing students to request abstracts, bibliographies, tables of contents, data analysis, literature reviews, and rapid summaries of key points within minutes – tasks that previously required days or even weeks to complete. As an example, four LLMs, namely FlanT5, OpenHermes-NeuralChat, Mixtral and Platypus 2, were used to screen titles and abstracts in systematic reviews, demonstrating promising results in the automation of publications.⁴⁵ In addition, LLMs can conduct thematic analysis of qualitative data, and achieve substantial similarity to human-generated themes.⁴⁶

The integration of LLMs into academic writing and publishing is gaining traction. PubReCheck guides researchers in preparing text-mining-ready articles, thereby improving the discoverability and impact of their work.⁴⁷ Likewise, BioVisReport creates interactive websites for visualizing published data, thus enhancing the reproducibility and accessibility of research findings.⁴⁸ Given the prevalence of grammatical and scientific errors in existing literature, LLMs offer valuable feedback on revisions, including grammar and spelling corrections, as well as suggest alternative phrases to enhance the overall quality of the text.⁴⁹ The styles of a manuscript can be adjusted based on the user’s needs and receive a score. Furthermore, the role of LLMs in academic writing extends to the citation and reference management. The OpCitance project, which involves identifying citation contexts in PubMed Central articles, illustrates the potential to facilitate reference mapping and ensure that citations are accurately formatted and linked to their corresponding sources.⁵⁰ In summary, LLMs provide a multifaceted approach to improving academic manuscripts by offering feedback on grammar and spelling, suggesting alternative phrases, aiding in citation management, and supporting educational initiatives.

By offering direct access to a wide array of academic resources, LLMs enable the exploration of more thoroughly, nuanced, and well-informed perspectives. This, in turn, leads to improved assignment outcomes and a stronger understanding of researchers. LLMs can also be leveraged to automatically generate machine-readable protocols from scientific publications. ProtoCode can interpret and curate knowledge from complex scientific literature, converting literature-based protocols into operational files suitable for laboratory equipment.⁵¹

Challenges and Solutions About LLMs

Although LLMs show great values in multiple aspects, they still face several challenges in the context of medical education, such as automation bias, hidden hallucination problems, and the model interpretability.

Automation Bias

Automation bias arises when students rely excessively on LLMs. It means students are easy to accept responses, or even wrong answers, without question, and their critical thinking, problem-solving, and innovation abilities are therefore restricted.^52,53 Young medical students, as digital natives, are comfortable with internet information, and may assume that LLMs are always accurate, due to limited expertise, and lower overall confidence. Like the response in Figure 3D, despite LLMs provided explicit reasoning, their answer is incorrect. Research on testing the answers of ChatGPT, ERNIE Bot, and ChatGLM about breast cancer reveals that while LLMs can provide comprehensive answers, their accuracy is notably lower for specialized topics.⁵⁴ Importantly, it remains the responsibility of physicians to verify the accuracy of LLM’s answers when submitting their assignments.⁵⁵ Otherwise, a physician exhibiting automation bias may apply flawed conclusions in patient care, potentially causing significant and irreversible harm.⁵⁶

Solutions: (1) It is crucial to underscore the necessity of not relying solely on LLMs but engaging in meaningful learning exercises. Educators should teach students to critically assess the accuracy and reliability of information from LLMs. These include fostering a student-centric evaluation approach to ensure students can discern and process information effectively, instilling a strong sense of responsibility and ethical awareness in their learning journey. With proper guidance, knowledge-based chatbots may benefit medical students’ academic performance, critical thinking, and learning satisfaction.⁵⁷

(2) Facing LLMs, students are encouraged to question results rather than passively accept them. It is consistent with the Illusion of Explanatory Depth, where students are encouraged to delve deeper into the information provided by LLMs.⁵⁸ The concept of active engagement with LLMs is further supported by research in cognitive enhancement, where students are encouraged to make deliberate choices rather than impulsive decisions.

Hallucination

LLMs prioritize generating the most suitable response based on input, and they cannot verify the accuracy of statements. LLMs may produce references and clinical cases that appear legitimate but are fabricated or incorrect, and this condition is called hallucinations. Hallucinations can present as fabricated bibliographic citations that do not correspond to genuine scholarly works. An investigation into ChatGPT-3.5 and ChatGPT-4 revealed that a considerable proportion of the citations generated by these models were fabricated, with 55% of GPT-3.5 citations and 18% of GPT-4 citations being non-existent.⁵⁹ Another study assessing ChatGPT’s performance in managing distal radius fractures found that a significant portion of the references provided by the model were fabricated.⁶⁰ In certain cases, Bard even failed to retrieve any relevant papers and had a hallucination rate of 91.4%.⁶¹ This potential for misinformation can undermine the academic integrity of institutions and may lead to disciplinary actions. Articles that include hallucinations has happened in journals like Medical Teacher.⁶² In severe conditions, plagiarism may happen, which is a more pressing concern.

Solutions: (1) To prevent the misuse of LLMs, it is critical for educators to cultivate self-discipline and ethical awareness among students. This can be achieved by guiding students to establish realistic goals, manage their time efficiently, and devise strategies to overcome challenges within supportive learning environments.⁶³ Additionally, comprehensive ethics education programs should be developed to teach medical students to apply these principles in practical, real-world situations.⁶⁴

(2) The existence of hallucinations necessitates the establishment of robust guidelines, frameworks and tools to ensure the responsible utilization of LLMs.⁶⁵ Plagiarism detection systems, AI output detectors, and human reviewers are crucial in identifying hallucinations. Turnitin and iThenticate are common plagiarism detection software. These tools work by comparing submitted texts against extensive databases of academic publications, internet sources, and previously submitted documents to identify similarities.⁶⁶ AI output detectors assess the structure and style of the text to determine the likelihood of AI generation, with some tools achieving high accuracy while others performed less effectively.^67,68 However, despite the capabilities of these tools, human judgment remains essential for the accurate interpretation and validation of these findings.

(3) Most journals and publishers have implemented new strategies to detect and avoid inappropriate use of the AI. Submitting authors are required to disclose any relevant AI technologies used in their manuscripts.⁶² Such transparency is vital for sustaining trust within the scientific community and guaranteeing the ethical application of LLM technologies.⁶⁹ The journal Medical Teacher proposed eight key lessons, emphasizing the importance of carefully verifying citations and references, as even page numbers can provide critical clues.⁶²

(4) Institutions can adopt a multifaceted approach to ensure the accuracy, reliability, and ethical standards of LLM-generated content. By reviewing existing frameworks and criteria from other fields, institutions can develop robust evaluation frameworks to assess the external validity, applicability, and transferability.⁷⁰ This enables institutions to better determine whether the content is applicable and useful in various contexts. Furthermore, institutions can enhance the content and structure of educational resources, such as LibGuides, to improve the usability and accessibility, as demonstrated in the field of dentistry.⁷¹

Incorrect Response

A significant risk associated with ChatGPT 3.0 stems from its inability to access the most up-to-date information. ChatGPT was found to provide incorrect or incomplete response concerning pharmacological data, posing a high risk of harm to patients.⁷² It has also led to the dissemination of incorrect details concerning teratogenic drugs, potentially resulting in unwarranted pregnancy terminations.^73,74 Moreover, there have been instances where LLMs inappropriately suggested that a simulated mental health patient should engage in self-harm.⁷⁵ Consequently, the inability of ChatGPT to access the most recent data warrants careful scrutiny.

Inaccurate responses can manifest in geographical data and exhibit discriminatory tendencies. As LLMs are primarily trained on data from resource-rich regions, there is a consequent scarcity of research and applications in underrepresented areas, such as Aotearoa New Zealand, where distinct cultural and social contexts necessitate customized debiasing approaches.⁷⁶ On this condition, LLMs may propagate race-based medicine, perpetuating harmful and inaccurate medical practices based on race.⁷⁷ LLMs may discriminate against certain groups, including women and individuals of Black descent, even when utilizing ostensibly unbiased prompts.⁷⁸ Such biases underscore the imperative for meticulous curation and specialized fine-tuning of training data to mitigate these adverse effects.³⁰

Solutions: (1) Addressing biases effectively requires a multifactorial framework that engages raters from diverse backgrounds and expertise. Recently, Stephen et al conducted a large-scale empirical study with the Med-PaLM, and developed a toolbox for identifying health equity harms and biases.⁷⁹ The study highlighted the importance of employing diverse assessment methodologies and involving raters with varying backgrounds.⁷⁹ It is strongly recommended that LLMs associated ethical implications necessitate a systematic review and should involve diverse stakeholders, including researchers, healthcare professionals, and ethicists, to ensure that the deployment of LLMs is guided by comprehensive ethical frameworks and human oversight.⁸⁰

(2) Embedding systemic equity throughout AI applications is imperative for addressing unfair biases. As bias can be introduced during data collection, algorithm design, and even during the deployment of AI systems, a critical aspect lies in the recognition and mitigation of biases at various stages of AI development and implementation. Researchers can delineate and mitigate unfair biases by employing tailored and comprehensive questionnaires. It helps identify domain-specific socioecological inequities and the selection of appropriate mitigation strategies.⁸¹ Furthermore, there is a need for a regulatory framework to address health inequity in LLM applications. For example, the European Union’s regulatory framework on medical devices has set rules for performance and data quality, with continuously evolving measures to address the full spectrum of biases in AI systems.⁸²

“Black Box” Effect

Since LLMs use advanced techniques to obtain results, and do not rely on predefined rules, the results are sometimes difficult for users to verify. LLMs cannot explain the sources of their advice, the rationale behind it, or the extent to which ethical considerations have been weighed. This is especially relevant in areas like cardiovascular imaging, where AI systems are used for diagnosis but often lack explain ability, posing legal and ethical challenges.⁸³ Decision making in clinical cases testing have numerous confounders, including patient age, history, culture conditions, and variations in diagnostic and treatment procedures. Whether will LLMs adapt to these modifications is essential for teachers and students to assess the results.

Solutions: It is of great significance to develop more readable AI systems that can provide more clear and understandable outputs. The Transparency and Interpretability for Understandability framework has been suggested to ensure that LLMs in mental health are comprehensible, thereby fostering trust and usability.^84,85 The development of explainable AI techniques, such as SHAP (SHapley Additive exPlanations) and LIME (Local Interpretable Model-agnostic Explanations), has been proposed to enhance the interpretability of AI systems in drug discovery, making the decision-making process more transparent and understandable.⁸⁴

Although being difficult, Durán and Jongsma argue for the necessity of design transparency to enhance the trustworthiness of medical AI systems. They propose a framework called computational reliabilism, which aims to provide epistemic warrants for the reliability of AI outputs. However, they acknowledge that justified knowledge is not sufficient for normative justification, emphasizing the need for a more transparent design and implementation of AI systems to address these challenges.⁸⁶ Yang et al developed BlastAssist to measure interpretable features, including size, symmetry, fragmentation etc., to replace time-consuming and subjective manual procedure. Readable LLMs in medical education will be powerful resources to promote the development of this field.

Taken together, developing an effective medical education system that incorporates LLMs necessitates collaborative efforts among medical educators, AI scientists, ethicists, regulatory bodies, and publishers is urgent (Figure 2). Medical educators must understand both the strengths and limitations of LLMs, as well as the diverse needs of students. AI scientists play a vital role by providing technical expertise and addressing potential challenges. Ethicists’ duty is to ensure that practices across various domains adhere to ethical standards. Regulatory bodies are responsible for ensuring the evaluation system adheres to relevant legal and ethical standards, including the formulation and enforcement of rules to prevent data misuse. Additionally, the involvement of publishers is crucial for addressing scientific opinions and concerns related to academic ethics and writing.⁸⁷

Conclusion

As technology advances, medical education will increasingly rely on technological tools. Teaching becomes more efficient and captivating through utilizing LLMs to design engaging lessons, providing instant feedback, and simulating real-life scenarios. On the other hand, students can carry out individualized learning more effectively based on their own pace and learning styles, and conduct in-depth discussions deepen their understanding. Furthermore, LLMs have demonstrated a vast potential and broad performance in the academic field. However, it is important to recognize that LLM is not inherently powerful, and it may also generate bias, hallucination, and opaque response. Looking ahead, it is essential to educate medical students on scholarly integrity, patient and public safety, and the ethical use of LLMs. Such education will equip them with the necessary skills and knowledge to excel in a rapidly changing healthcare environment. Future research needs to focus on the long-term effects of LLM integration. Longitudinal studies spanning multiple academic years should be designed to evaluate how LLMs impacts students’ medical knowledge acquisition, clinical decision-making skills, and professional ethics. Only through such comprehensive efforts can we ensure that future doctors are well-equipped to harness the power of LLMs while safeguarding the quality and ethics of medical education.