Application of Artificial Intelligence Generated Content in Medical Examinations

Background

Artificial intelligence generated content (AIGC) refers to the utilization of artificial intelligence to produce text, image, audio, and video. It is greatly promoted by the release of Chat Generative Pre-trained Transformer (ChatGPT), and causes an immediate change in medical education.¹ AIGC offers educators opportunities to innovate teaching methodologies, enhance learning experiences, and improve educational effectiveness.¹ As an example, Fuller et al analyzed the course evaluation comments through instructors and ChatGPT, and found a high agreement between these two groups, so that ChatGPT could ease the burden of instructors during teaching feedback.² Additionally, integrating AI tools like ChatGPT into learning practices has been shown to assist learners in conducting literature reviews, addressing personalized inquiries, and promoting autonomous learning. As a result, students become more adept at navigating resources and synthesizing information.³ This self-directed learning model is essential in today’s fast-paced educational environment, where learners are encouraged to develop critical thinking and problem-solving skills. However, despite these advancements, the potential of AIGC in the development of medical questions remains underrecognized.

Examinations serve as an essential tool for assessing and promoting learning outcomes, as well as provide students with a structured means of revision and self-assessment. Repeated testing through quizzes significantly enhances material retention in subjects such as anatomy, where students have demonstrated improved scores after multiple assessments.⁴ Most medical schools employ question banks to generate streamlined examination papers. It has become a prevalent practice, primarily due to the ability of question banks to enhance the efficiency and effectiveness of examination preparation and assessment. Question banks can serve as a repository of multiple-choice questions (MCQs) that can be utilized to create tailored assessments, while also ensure that the content aligns with the curriculum.^5,6 Despite their advantages, challenges such as inconsistent question quality and inadequate real-world scenarios persist. In this context, AIGC offers several benefits, including low cost, high volume, and novelty. For instance, Klang et al leveraged GPT-4 to generate a substantial set of 210 medical MCQs, demonstrating the model’s potential in educational contexts. Remarkably, the study reported only one error in the entire set of questions, underscoring the impressive accuracy and reliability of GPT-4. This finding suggests that AI is highly efficient in generating high-quality medical educational materials, where precise knowledge assessment is crucial for training future healthcare professionals.⁷

This study explores the application of AIGC in the development of medical examination questions (Figure 1). It begins by discussing the limitations of traditional paper-based examination methods, highlighting the need for innovation in assessment techniques. We then propose practical strategies for designing medical exams using AIGC, supported by illustrative examples. Additionally, the study examines the future potential of AIGC in conjunction with other emerging technologies, anticipating a transformative impact on medical examinations. As AIGC evolves, it is expected to significantly enhance the quality of examination preparation and contribute to the effective training of medical students.

Figure 1 Flowchart of artificial intelligence generated content in medical examinations.

Issues in Traditional Medical Examination Questions

Inconsistent Quality of Questions

Constructing high-quality examination questions is a highly specialized process that requires specific skills, which even experienced medical educators may lack. The National Board of Medical Examiners’ Item-Writing Guide provides essential frameworks to assist educators in this endeavor.⁸ When designing exams, it is crucial for educators to focus on the key knowledge points outlined in the teaching syllabus, adhere to specific principles in question formulation, and ensure that answer options reflect appropriate levels of difficulty. Meticulously constructed questions exhibit psychometric properties that enhance learners’ performance.⁹ Conversely, poorly designed MCQs reduce pass rates, impede clinical reasoning skills, and misinterprets test scores.¹⁰ As a whole, although question banks can cover a wide range of topics, they often fall short in addressing the nuances of complex subjects that require critical thinking and the application of knowledge.

Difficulties in Developing and Maintaining Question Banks

As revealed by Bharat et al that developing a question bank for rheumatology took four years compromising of a rigorous 12-step process.¹¹ This process is not only time-consuming but also labor-intensive and requires substantial resources. Most importantly, given the rapid evolution of clinical medicine, the questions in exam banks may quickly become outdated, necessitating annual revisions to reflect the latest advancements. Failure to update these questions results in students receiving inaccurate information, potentially leading to incorrect clinical management decisions. Consequently, it is imperative that new questions are continually developed and rigorously assessed to ensure the accuracy.¹² This iterative process also requires collaborations between experienced educators and professional software engineers.¹¹ Although commercial question banks are available, they often lack alignment with specific teaching content and their quality remains uncertain. These commercial resources are typically designed to prepare students for board certification examinations, making them less suitable for college-level graduation assessments.

Declined Learning Ability

The increasing reliance on question banks has led to significant challenges in maintaining the integrity and effectiveness of the assessment process. Students may become overly familiar with question types and content, and even cause leakage and omission of questions. When students have access to previous exam questions or practice materials, it can undermine the integrity of the assessment process. This leakage not only diminishes the value of the evaluation but also fosters an environment where students may prioritize rote memorization over genuine understanding and critical thinking.^13,14 In addition, the prevalent use of multiple-choice questions (MCQs) or fill-in-the-blank formats encourages students to focus on short-term recall rather than long-term comprehension. This concern is echoed in discussions about the effectiveness of different question types, where non-compound questions tend to yield more reliable assessments than compound questions, as the latter can confuse respondents.¹⁰ In the long term, such declines in educational rigor are likely to exacerbate disparities and uneven competences in educational quality among different universities.

Taken together, ensuring a fair and effective evaluation of medical students’ competence necessitates the quality of examination questions, the construction of question banks. By addressing these challenges, we can not only enhance the assessment process but also contribute to the overall improvement of healthcare standards and uphold the professionalism of medical students. It is essential to emphasize the importance of flexibility and innovation in medical examinations.

Construction of Medical AIGC Questions

Step 1 Select an Appropriate AIGC Model

The quality of AIGC questions is significantly influenced by the type of large language models (LLMs) employed in their creation. Recent studies have highlighted the varying capabilities of different LLMs.^15,16 Rohaid et al conducted a study on the accuracy of three LLMs in neurosurgery examinations, revealing that GPT-4 achieved a high score (82.6%), which surpassed that of GPT-3.5 (62.4%) and Google Bard (44.2%).¹⁷ These discrepancies may be attributed to different data quality, model architecture, and technical version. Notably, GPT-4 relies on pre-existing training data and lacks web crawling capabilities. In contrast, Google Bard can access and incorporate real-time information from the internet. GPT-4 demonstrated a 20% improvement in three United States Medical Licensing examinations.¹⁸ These variabilities underscore the necessity for careful selection and fine-tuning of LLMs to enhance their effectiveness in generating AIGC questions.

Step 2 Specific Prompts to Construct Questions

While AI can streamline the question creation process, the validity of these AI-generated questions varies significantly based on the specificity and clarity of the prompts used by educators.¹⁹ This underscores the necessity for medical educators to engage in effective prompt engineering to maximize the potential of AI tools. By refining and enhancing the instructional prompts, it becomes feasible to achieve more accurate and scientific output. Fundamental prompts include delineating the knowledge areas and assessment criteria explicitly, thereby enabling LLMs to generate medical examination questions, answers and explanations. This approach minimizes the risk of question leakage, and facilitates students’ foundational understanding of specific concepts. Examination questions can further be derived from existing question banks. This method optimizes the use of existing resources, and keeps consistency in question styles. It also saves time and efforts in selecting relevant knowledge points.

Multiple-Choice Questions

MCQs are the predominant questions in medical examinations, and encompass a wide array of learning objectives. Poorly constructed questions can lead to misleading assessments of student knowledge and skills. High-quality MCQs are particularly effective in assessing students’ abilities to apply knowledge, interpret information, and synthesize concepts.²⁰ AIGC further enhances the efficiency and clarity of MCQ creation. Medical educators can leverage AIGC to formulate more intricate prompts, such as generating plausible yet incorrect distractors, or crafting questions that mirror real-life clinical scenarios, or increasing the difficulty. Table 1 presents the original options as simple and clear, allowing students to directly compare them to find answers. In contrast, the new test questions include options with extensive modifications and descriptions, making them appear more familiar and authentic while also imposing greater demands on students. These innovative strategies can improve the accuracy and complexity of MCQs, as well as differentiate students across different competency levels.²¹ Moreover, the use of AIGC in MCQ creation can lead to a more standardized approach, ensuring that questions are not only relevant but also aligned with learning objectives. This standardization can help mitigate biases that may arise from human-generated questions, as AIGC can be programmed to adhere to specific guidelines and criteria. For instance, studies have indicated that negatively-marked MCQ assessments, which reward partial knowledge, do not cause gender bias, and can enhance the performance and satisfaction of students.²²

Table 1 Examples of AIGC in Multiple-Choice Questions

Case Study Questions

The incorporation of AIGC into case study questions enhances the richness of situational descriptions, making a more realistic learning environment and better reflect the actual clinical practice. By this way, AIGC can facilitate deeper learning experiences and improve the assessment of higher-order cognitive skills among students. It aligns with the findings of various studies that emphasize the importance of context and complexity in educational assessments.²³ Medical students need to decipher clues and formulate innovative responses. Moreover, the use of AIGC in crafting case-based questions can draw from a wide array of data sources, providing a more comprehensive view of patient scenarios. This is particularly relevant in medical education, where understanding the interplay of various factors, such as patient history, social context, ethical dimensions, and clinical guidelines, is crucial for effective decision-making.²⁴ Consequently, such an alteration encourages medical students to engage in critical thinking. It aligns with the growing recognition of the importance of holistic approaches in healthcare education, and understanding the patient’s narrative is as vital as the clinical facts.

Video Question

By utilizing text driven image generation technology, students have access to highly simulated medical imaging questions without contact with real patients. This not only enriches students’ learning experience but also protects patients’ personal privacy. Moreover, it alleviates the shortage and inequality of medical teaching resources and multimedia resources. The development of denoising diffusion probabilistic models has enabled the stable generation of high-quality medical images. Image generation tools, such as Imagen and latent diffusion models, utilize text prompts to provide fine-grained guidance during the image generation process.^25,26 This capability holds significant promise for the formulation of questions related to medical imaging. Recently, Xu et al introduced a model named MedSyn, which generates high-resolution, anatomy-aware CT images based on user-input text prompts.²⁷ These images effectively retain intricate details of the lung’s airway, vessels, and lobular structure while addressing privacy concerns. Furthermore, MedSyn demonstrated superior performance compared to the most advanced models based on GAN and diffusion techniques. Therefore, it can be used to help students grasp a deeper understanding of the anatomical structure and pathological manifestations of the lungs.

Traditional skill examinations often face limitations related to venues, equipment, and personnel. AIGC can generate simulated examination questions, students are required to complete tasks that involve moving objects and drawing diagrams.²⁸ This innovation significantly broadens the range of examination formats, and raises the standards for candidates’ foundational knowledge and judgment abilities.²⁹

Step 3 Assess and Review the Outcomes

Although AIGC has shown great potential in generating high-quality medical questions, ensuring the accuracy and reliability of its output remains a key challenge. A study that examined the performance of ChatGPT on medical examinations found that the accuracy and applicability of AIGC varied based on the complexity of the questions posed.³⁰ Most open-access LLMs lack specific medical training. Consequently, they may lead to misdiagnoses or misinterpretations, particularly in contexts involving age and gender. For example, a 35-year-old female presenting with irregular menstruation might be incorrectly diagnosed with menopausal syndrome. A male patient with abdominal pain could receive an option for ectopic pregnancy among the choices. A study compared the responses of GPT-3.0 to those of medical consultants on 41 case-based questions, and found the medical inadequacy and unconciseness of GPT-3.0.³¹ This deficiency may be partly attributed to the model’s inability to recognize non-verbal cues. Therefore, routine assessment and review of AIGC questions before formal examinations is essential to ensure the scientific validity, clarity, and feasibility. Teachers must evaluate the alignment of these questions with course objectives, and exclude any inaccurate or misleading contents. Furthermore, it is crucial to analyze the difficulty, discrimination, and reliability of the questions based on the performance of students, facilitating the optimization of teaching research and practice.

Some other AI tools can also be utilized for error recognition and correction. One method is to compare the generated information with a structured medical knowledge base, such as knowledge graphs, to find errors or inconsistencies.³² Another approach is to encourage the medical expert systems to review and edit the AIGC content. Medical expert systems combine the judgement from human experts and AI systems, which can significantly improve the accuracy and reliability of AIGC applications in medical education.³³ As a result, the application of AIGC provides a more comprehensive and authentic assessment of students’ clinical abilities and literacy. This approach contributes to their holistic development and better prepares them for professional practice.

Future Prospects

Retrieval Augmented Generation

Retrieval augmented generation (RAG) technology, which integrates retrieval and generation functions by introducing an external knowledge base, significantly improves the quality and accuracy of AIGC. This benefit is particular useful in the medical field, where accurate information is critical for patient care and decision-making. One notable application of RAG technology is in the development of advanced chatbots for healthcare, such as the GastroBot. The system demonstrated a remarkable improvement in response accuracy and contextual relevance, achieving a context recall rate of 95% and a faithfulness to the source of 93.73%.³⁴ Similarly in emergency medical triage, RAG significantly outperformed traditional methods, achieving a correct triage rate of 70% compared to lower rates from human practitioners.³⁵ When designing questions, educators can leverage RAG technology to accurately retrieve relevant case information, treatment methods, and the latest research progress from a wide range of medical literature and databases, thereby deepening the comprehensive understanding of students concerning the knowledge point.

Multi-Agent System

Multi-agent systems (MAS) have emerged as a powerful framework for simulating collaboration and communication in complex environments. By modeling various agents, such as doctors, nurses, and patients, these systems can effectively replicate the dynamics of real-world interactions, thereby enhancing our understanding of cooperative behaviors and improving healthcare delivery. In a microworld experiment, it was found that participants who interacted with a highly cooperative agent were more likely to engage in effective interactions and resource sharing compared to those who worked with a less cooperative agent. This finding underscores the potential of MAS to enhance resilience in healthcare systems by fostering better collaboration among human and automated agents.³⁶ MASs have demonstrated unique advantages in the representation and inference of medical knowledge. Students can better understand the interrelationships between medical concepts and enhance their abilities in comprehensive analysis and problem-solving. When used in medical exams, students may be required to effectively respond to the condition of simulated patients, communicate and collaborate effectively with other intelligent agents (nurse, attending physician, and others), and develop reasonable treatment plans. This teaching method not only enhances students’ clinical practice ability, but also strengthens their teamwork and communication skills.

Video Generation Technology

Video generation technology (VGT) represented by Sora provides more intuitive and vivid teaching resources for medical education by generating high-quality medical teaching videos. This technology presents unintelligible medical concepts and complex physiological processes in a dynamic and visual form, thereby reducing the difficulty of understanding and enhancing the learning interest. In anatomy teaching, VGT can generate 3D animated videos that showcase various organs and structures of the human body in detail. Instructional videos can serve as effective teaching tools, especially in remote education settings, where traditional methods may fall short.³⁷ By watching videos, students can gain a clearer understanding of the internal structure and physiological functions of the human body. The application of cognitive load theory (CTL) in designing these educational technologies can further optimize their effectiveness. By structuring video content to align with CLT principles, educators can minimize extraneous cognitive load and enhance intrinsic cognitive load, leading to better learning outcomes. A scoping review highlighted that CLT-based interventions in medical education often yield positive results, suggesting that thoughtful integration of technology can foster a more productive learning environment.³⁸ When used in designing medical exams, Sora can be used to generate error example videos with incorrect operation sequences or violated medical regulations. These negative materials add value to deepen the judgement of the potential harm of erroneous behavior and the importance of correct practices through comparing correct operating procedures, thereby avoiding similar errors in practice, and promote clinical decision-making abilities.

Conclusions

The integration of AI technology into medical education represents a significant trend for its future development. Against the limitations brought about by the use of question banks, AIGC in medical examinations is an innovative approach that enhances medical students’ foundational knowledge, clinical understanding, and practical skills. This study offers a practical solution in designing MCQs, case-based questions, and Video question. It is anticipated that future development will incorporate with technologies like RAG, multi-agent system, and VGT. By leveraging these tools, educators can create more intuitive and vivid questions that not only enhance student learning but also improve patient education and engagement. The ongoing exploration of innovative educational technologies will undoubtedly continue to shape the future of medical training and practice.