Assessing the Difficulty and Long-Term Retention of Factual and Conceptual Knowledge Through Multiple-Choice Questions: A Longitudinal Study

Introduction

Multiple choice questions (MCQs) are ubiquitous in the medical profession. In the United States, the Medical College Admissions Test (MCAT), the United States Medical Licensing Examination (USMLE) examinations, specialty-specific board certification examinations, and other essential and clinical sciences examinations are built using MCQs. We may take them for granted today, but in previous decades, the usefulness of MCQs in clinical medical education was disputed.^1–3 Subsequently, data emerged that MCQs could be reliable, valid, and efficient,^4,5 but questions persisted about the ability of MCQs to evaluate cognitive levels above rote recall.⁶ Several studies have attempted to classify MCQs and link them to the six cognitive domains described in Bloom’s taxonomy with varying levels of success.⁷ This stratification of question content predates contemporary neuroscientific understanding of memory formation and retrieval with even the most basic mapping of brain network functionality to learning (for review, see.⁸ Bloom’s taxonomy does not adequately capture the complexities of learning in the brain, even in revision,⁹ making it difficult to demonstrate this framework’s effectiveness in assessing learning.^6,10,11

Recent research in medical education highlights the impact of various educational strategies on knowledge retention, emphasizing the value of theoretical and practical approaches. Long-term knowledge retention is generally higher with active, blended-learning approaches than passive learning methods, with some variations in retention rates between student groups.¹² Spaced and repeated testing and retrieval-based formats have consistently outperformed mass practice or cramming, improving long-term retention.^13,14 Feedback further solidifies learning, as knowledge retention can decline rapidly within days to weeks without reinforcement.¹⁵ Problem-based learning (PBL) and small group learning promise to enhance problem-solving skills and retention,^16,17 while spaced education and distributed practice effectively improve clinical performance.¹⁸ Active learning strategies, though challenging to implement, have been recognized for their benefits,¹⁹ with collaborative reflection gaining traction as a method to engage students.²⁰ Podcasts, smartphones, and clinical applications have emerged as valuable tools for contemporary medical education, offering alternative and flexible learning modalities.^14,21,22 Incorporating intentional difficulties, or “desirable difficulties”,²³ or knowledge decrement²⁴ has also proven effective in enhancing long-term retention, while strategies focusing on rural workforce retention have demonstrated considerable success.²⁵ For long-term retention of medical knowledge, multiple-choice (MCQs) and short-answer (SAQs) improve delayed retention compared to no testing.^26,27 While MCQs offer high reliability and easy marking, SAQs may require more effortful retrieval and promote better retention.^28,29 Repeated testing enhances long-term retention compared to repeated study.³⁰ To construct high-quality MCQs, educators should focus on assessing higher-order thinking skills and use test blueprints.³¹ Ultimately, varied teaching approaches and well-prepared lectures are crucial for engaging students and fostering durable knowledge retention.^32,33

A challenge related to the widespread use of MCQs in medical education is the issue of assessing different levels of cognitive function: that is, testing critical thinking and understanding as opposed to rote memorization. To test critical thinking, a MCQ often needs considerable revision, which can be successful if the MCQ is crafted appropriately.³⁴ This can be coupled to the systematic design of instruction that typically begins with objectives and ends with an assessment to determine if the objectives (ie learning outcomes) have been achieved. Promoting higher-order thinking is desirable in contemporary pre-clinical curricula given the complexity of clinical medicine. In medical education, a shift from memorized facts to applied concepts occurs in the clinical years where diagnosis and treatment skills are honed. The cognitive scientific principles at play during this transition are complex and ultimately give rise to melding the art and science of medicine.³⁵ Therefore, an ideal MCQ-based exam should use questions that provide a meaningful assessment of higher cognitive functions, testing understanding and applying knowledge beyond rote memorization. Studies suggest that examinations should prioritize comprehension-level (higher-order thinking) questions over knowledge-level questions to better predict successful learning, according to Bloom’s taxonomy.³⁶

Assessing higher cognitive function also has possible implications for education research. For example, the posttest-only control group design is a gold-standard intervention study design commonly found in such research.³⁷ Briefly, an intervention is administered to an experimental group but not a control group. A comparative assessment of the two groups, which may take the form of MCQs, then tests the effect of the intervention. Since MCQs can be written to different difficulty levels and potentially assess various levels of cognitive function, they introduce an additional variable to the experimental design and a potential source of measurement bias. The selection of MCQs for comparative assessments may influence whether an effect is observed, how large it is, and how it evolves.

Karpicke and Blunt report such a finding,³⁸ where in certain experimental groups, the posttest questions that they classified as “inference” showed superior recall than those classified as “verbatim”. While not the focus of that particular study, the observation is nonetheless intriguing. Zaidi et al showed a related observation using a dichotomized Bloom’s taxonomy to classify MCQs as “lower order” and “higher order” based on cognitive level.³⁹ They found a modest but statistically significant difference in performance on the two question types, with lower-order questions answered correctly more often than higher-order questions. Both studies suggest it is possible to differentiate MCQs based on function: those designed to test memorization and others designed to test deeper conceptual understanding.

Our study used Karpicke and Blunt’s³⁸ work as a starting point to develop a classification scheme for medically relevant MCQs. We created three MCQ categories, including (1) recall/verbatim, (2) concept/inference, and (3) mixed/ambiguous. The hallmark of recall/verbatim questions is that they are based primarily on facts and assess whether a learner has retained relevant information. In contrast, concept/inference questions are based primarily on relationships and, therefore, consider whether a learner knows how facts are connected, which provides greater context and enables the ability to make predictions. In the present study, we added a third category, mixed/ambiguous, for MCQs that could be answered with the recall of facts or inference and interpretation. This study aimed to assess whether conceptual/inferential knowledge is retained longer than factual/verbatim knowledge among medical students over two years.

We formed and tested a two-part hypothesis using this classification system. First, after preparing for an exam, we posited that 1) scores for recall/verbatim questions would be significantly higher than for concept/inference questions, and 2) memory loss over time would be more significant for factual knowledge than conceptual understanding compared to a loss in the ability to reason about concepts critically. This hypothesis predicts that concept/inference questions will generally be more challenging based on the relatively more sophisticated cognitive processes needed to answer them. Further, the erosion over a defined time interval of conceptual knowledge, as assessed with concept/inference questions, will be slower than the erosion of straight factual knowledge, as assessed with recall/verbatim questions. This part of the hypothesis predicts that conceptual knowledge is relatively complex to acquire but, once encoded, is more durable, especially in a longitudinal manner.

Materials and Methods

Participants and Procedure

Question Categorization

The Kirk Kerkorian School of Medicine curriculum includes a comprehensive coverage of basic and clinical sciences, and questions were selected from summative examinations covering critical pulmonary and renal physiology topics. Forty-five questions were extracted from a faculty-authored summative examination. One of the authors (NH) developed a system to categorize each question as (1) recall/verbatim or (2) concept/inference. A third category, mixed/ambiguous, was for questions that could be answered using recall or inference. Seven faculty members, representing a mixture of MDs and PhDs with content expertise in assessment, pulmonary and renal physiology, and curriculum development, reviewed the questions independently and as a group, then reached a consensus on the best category for each question.

Recall/verbatim questions were categorized as having relatively simple cognitive processes to answer. These include straightforward recall of simple facts, solving arithmetic equations, recognizing basic patterns, or deductive reasoning that requires little or no content mastery. Verbatim/inquiry questions involving more esoteric and less familiar information are expected to have relatively high difficulty indices (the percentage of students who answer a question correctly). A question may also be classified as recall/verbatim if answering it requires recall of a specific, non-intuitive piece of information, regardless of what other cognitive processes are involved. Concept/inference questions require fairly complex cognitive processes to answer them, which commonly involve deductive reasoning to discern or predict relationships, recognizing subtle patterns, and solving problems that require substantial practice and content mastery. Mixed/ambiguous questions are answerable through more than one pathway. For example, diagnosing an acid-base disorder based on laboratory values can be done through logical inference or straight pattern recognition.

Assessment Implementation

The categorization system described above was first applied retrospectively. Performance data on each question were collected from medical student results in a summative pre-clinical basic science exam in April 2020. The same questions were then re-administered to volunteers from the same cohort of medical students in January-February 2022, after they had progressed through most of the clerkship year. To reduce the testing burden, the questions were broken into four quizzes (acid-base physiology, renal physiology, pulmonary physiology, and pulmonary/renal anatomy). To be included in the analysis, a medical student must have taken the 2020 exam and completed at least one 2022 quiz.

Ethical Considerations

Ethical approval was obtained from the University at Las Vegas Biomedical IRB on April 3, 2017 (Protocol 1030906–1). The approval permits the use of aggregated, de-identified data. The data was primarily collected to assess and inform curriculum, specifically to inform curricular use of faculty-authored examinations. The de-identified data was then used secondarily for research, as per the protocol, and is not considered human subjects’ research. The UNLV Biomedical Institutional Review Board designated that no individual informed consent is required as per an excluded protocol since this is not human subjects’ research. Further, all procedures performed in studies involving human participants were following the ethical standards of the institutional and/or national research committee and with the 1964 helsinki Declaration and its later amendments or comparable ethical standards (Goodyear et al, 2007).

Statistical Analysis

All scores from student assessments were individually de-identified after the 2020 and 2022 data were linked to avoid bias and preserve anonymity. The included students (n = 56), from a total cohort of 120 pre-clinical students, were organized into subgroups (top/bottom half or quartiles) based on their overall basic science exam scores relative to the class median of 84.1%. Statistical analyses were performed using paired Student’s t-tests with a two-tailed distribution and two-sample equal variance. The paired t-tests were used as the same group of students was assessed at two different time points (2020 and 2022). Such analysis is warranted, given we are comparing the means of two groups.^40,41 Using paired t-tests has established statistical significance in various medical education assessment studies.^42–45 A homoscedastic linear regression model⁴⁶ was applied to analyze the relationship between student performance on different MCQ types and the associated variables over time. This model assumes constant variance (homoscedasticity) across all levels of the independent variables, making it appropriate for examining the stable patterns in student performance without introducing bias due to unequal variances.^47,48 The model helped evaluate how performance on recall/verbatim and concept/inference questions changed between 2020 and 2022, and whether these changes were influenced by factors such as student quartile and overall test difficulty. All calculations were performed using Microsoft Excel’s regression analysis functions, which allowed for precise computation of p-values, confidence intervals, and other relevant statistical measures. A P-value threshold of 0.050 was used to determine statistical significance, ensuring a rigorous interpretation of the results.

Results

Question Categorization and Initial Performance

Using a basic framework (Table 1 and Table 2 for examples), the consensus of seven faculty members categorized 10 questions (22%) as concept/inference, 33 questions (73%) as recall/verbatim, and 2 (4%) as mixed/ambiguous. The mixed/ambiguous questions were excluded from further analysis.

Table 1 Basic Framework for Categorizing Multiple Choice Questions

Table 2 Example Recall/Verbatim and Concept/Inference Questions

Performance on recall/verbatim questions was significantly higher than on concept/inference questions, but concept/inference questions were more effective in differentiating student performance overall, and this differentiation persisted over time (Table 3). A retrospective analysis of question performance on the 2020 exam showed the recall/verbatim questions were answered correctly more often than the concept/inference questions (82.0% vs 60.9%, P = 0.002). When the top and bottom halves of the class were compared, they answered recall/verbatim questions correctly at approximately the same frequency (84.0% vs 80.2%, P = 0.433). The top half answered concept/inference questions correctly more often than the bottom half (68.6% vs 53.2%, P = 0.027). Further delineation showed a progressive decline in performance on concept/inference questions by quartile, while performance on recall/verbatim questions was relatively stable (Figure 1).

Table 3 Comparative Performance by Question Type and Year

Figure 1 Baseline Performance on Multiple Choice Questions by Question type and Quartile. A summative exam with multiple choice questions was administered to 56 medical students in 2020. A panel of medical school faculty retrospectively categorized the questions as recall/verbatim or concept/inference using a metric described in Materials and Methods. The percent correct was then computed and expressed by quartile. Question performance on the 2020 exam showed the recall/verbatim questions was answered correctly more often than the concept/inference questions (this is a graphical depiction of data in Table 3).

Performance on Recall/Verbatim and Concept/Inference Questions

Analysis of question performance on the 2022 quizzes showed no statistically significant difference between performance on recall/verbatim questions versus concept/inference questions (59.8% vs 42.9%, P = 0.079). Likewise, no observed significant difference occurred between the top and bottom halves of the class on recall/verbatim questions (62.5% vs 57.1%, P = 0.432) or concept/inference questions (44.2% vs 41.7%, P = 0.827).

Comparison of Knowledge Retention Over Time

A comparison of question performance between 2020 and 2022 showed a statistically significant decrease in performance on recall/verbatim questions (82.0% vs 59.8%, P < 0.001) and a significant decrease in performance on concept/inference questions (P = 0.050) (Figure 2a). The decrease on recall/verbatim questions was similar for both the top half (84.0% vs 62.5%, P < 0.001) and the bottom half (80.1% vs 57.1%, P < 0.001) of participants. A statistically significant decrease was observed on concept/inference questions for only the top half (68.6% vs 44.2%, P = 0.024). No significant difference was found on concept/inference questions for the bottom half (53.2% vs 41.7%, P = 0.204) (Figure 2b).

Figure 2 Comparison of Exam Question Performance over Two Years. The same questions used to measure baseline performance were administered a second time two years later to the same cohort of students (see Figure 1). (a) Depicts the percent correct for all students over two years using recall/verbatim or concept/inference questions. (b) Depicts the percent correct for the top and bottom half of the students over two years using recall/verbatim or concept/inference questions. The decrease in performance on recall/verbatim questions was similar for both the top and bottom halves. A statistically significant decrease in performance occurred on concept/inference questions for only the top half and no significant difference was found on concept/inference questions for the bottom half (this is a graphical depiction of data in Table 3).

Subgroup Analysis of Performance Decline

The decline in performance was different for all analyzed subgroups. The decline in recall/verbatim questions was similar between the top and bottom halves of the class. However, this was not observed for concept/inference questions. A statistically significant decrease of 24.4 points (P = 0.024) was observed for the top half of students on concept/inference questions and the bottom half averaged a statistically insignificant 11.5-point decrease (P = 0.204). Table 4 shows that the decline in performance on concept/inference questions was significant for the top half of the class. In contrast, performance on recall/verbatim questions declined equally for all groups. The gap between these two groups averaged 15.4 points in 2020, and this difference narrowed to a statistically insignificant 2.5 points in 2022 (P = 0.827). In fact, by 2022, the performance of both groups on both question types had become statistically indistinguishable in all but one comparison (top half, recall/verbatim questions vs bottom half, concept/inference questions (P = 0.036).

Table 4 Statistical Comparisons (P-values) of Top/Bottom Half Subgroups by Question Type and Year

Discussion

This study investigated the performance and retention of medical students on multiple-choice questions (MCQs) that assessed either recall/verbatim knowledge or conceptual/inferential reasoning. The results demonstrate that while students performed better on recall/verbatim questions initially, their performance on these questions declined significantly over two years. In contrast, conceptual/inferential questions, although more challenging initially, showed a smaller decline in performance over time. These findings suggest that conceptual understanding, though harder to acquire, is more resilient to memory decay and is potentially reinforced by clinical experience. This has important implications for the design of medical education assessments, emphasizing the value of testing higher-order cognitive skills to promote long-term knowledge retention.

Question Performance

Our first hypothesis was that recall/verbatim and concept/inference questions would have significantly different performance characteristics. The data from the 2020 exam support this hypothesis, with concept/inference questions having a substantially higher difficulty index than recall/verbatim questions. Moreover, all students had similar performance on the recall/verbatim questions. In contrast, it varied considerably for the concept/inference questions, which were the primary determinant of the final exam score. All participants, therefore, demonstrated a similar ability to memorize and perform simple cognitive tasks and were differentiated based on a subset of more challenging questions.

Our second hypothesis was that retention of more fact-oriented knowledge, as evidenced by performance on recall/verbatim questions, would degrade more rapidly than the knowledge tested by concept/inference questions. Both question types showed similar proportional declines in performance from 2020 to 2022, but the decline in recall/verbatim performance was statistically significant, while the decline in concept/inference was nearly statistically significant. These results concerning our second hypothesis are, therefore, inconclusive.

We observed that student performance on both question types tended to converge over time, suggesting a regression toward core clinical knowledge but the reasons for this are unclear. One possibility is that the participants in this study were near the end of their clerkship year and, therefore, had spent significant time participating in patient care and studying for clinical subject examinations. The “floor” performance, the lowest observed performance in each group for each question type in 2022 (~60% for recall verbatim, ~43% for concept/inference), may, in part, represent knowledge that was relevant to their clinical learning and reinforced during the clerkships. Conversely, the questions that showed a substantial decline in performance may reflect knowledge that is obscure or of low clinical importance and was consequently not revisited. In other words, while the 2020 exam could stratify students mainly based on their performance on concept/inference questions, performance tends to converge toward a baseline of core clinical knowledge essential for practice.

If borne out by additional studies, this work has implications for exam design. The high performance seen on recall/verbatim questions and their lack of ability to discriminate between different examinees calls into question the utility of even including such questions on summative examinations. It is well recognized that assessment drives learning,⁴⁹ and testing relatively basic information with recall/verbatim questions could appropriately incentivize students to learn foundational knowledge. However, overreliance on them could disincentivize more significant mastery of the content by allowing those with superficial knowledge to pass. Rather than designing a summative exam with a high proportion of relatively easy recall/verbatim questions, it may be better to compose a shorter exam with a higher proportion of more difficult concept/inference questions and a commensurately lower passing threshold. In theory, such an approach could incentivize students to focus on higher-level cognitive processes that would presumably be of greater value than transient memorization.

Defining Question Types

The notion that MCQs can be categorized based on the level of cognitive function they assess has been introduced previously. In the 1970s and 1980s, several studies attempted to classify MCQs according to Bloom’s taxonomy. This approach is both tempting and problematic. Learning and memory are complicated phenomena, and their complexity is not necessarily reflected in the six clean layers that comprise Bloom’s hierarchy. Zaidi et al proposed that Bloom’s taxonomy may have no meaningful utility in categorizing MCQs. Instead, they developed a dichotomized scheme of “higher order” and “lower order” questions.¹¹

The classification scheme developed for this study started with the very basic idea that answering a given MCQ involves (1) information and (2) cognitive processing. Information was divided into familiar or esoteric forms. Familiar information is integral to the learning of a given subject. An example from pulmonary physiology is compliance, a core concept underpinning an accurate understanding of how the lungs function. By contrast, examples of esoteric information (also from the lung) are the locations and nomenclature for the twenty individual lung segments. While this information is important to a subset of practitioners, it would be a time-consuming and low-yield endeavor for the average medical student to commit the segments to memory.

Cognitive processing was likewise subdivided into simple or complex forms. Simple processing encompasses functions that all medical students should be able to carry out with minimal cognitive load. Complex processing involves all cognitive functions that go beyond the simple. The classification of a given MCQ depends on the interplay between information and processing. Most will be classified based on processing type, with simple processing indicating a verbatim/recall question, and complex processing indicating a concept/inference question. An exception to this rule exists where recall of specific esoteric information is necessary to answer a question, regardless of the processing involved correctly. For example, consider a question where one must determine if a patient with an acid-base disorder has the expected compensatory response. Interpreting this question involves (among other things) understanding the relationship between PaCO₂, and HCO₃⁻, but to answer it confidently one must know the precise quantitative aspect of this relationship. A “cognitive bottleneck” such as this one, which requires recall of information that is not intuitive and cannot be deduced, classifies such a question as recall/verbatim. It should be recognized that the lines dividing simple from complex and familiar from esoteric are sometimes not easily drawn or static. Complex processing may become simple with practice and acquisition of expertise, and esoteric information may become familiar to the point of being implicit.

These findings also underscore the importance of MCQ selection for educational research experiments that utilize them. For example, if the MCQs from this study were used to form a question bank for a prospective experiment, their known performance characteristics would show a variation in performance between 2020 and 2022, ranging from a 64-point decrease to a 17-point increase, representing significant differences in how students retained factual versus conceptual knowledge. An alternative approach to assessing critical reasoning in medical education could be relying on short-answer, free-response questions. This could also avoid question writing flaws that often appear in MCQs. However, MCQs have an established ability to address higher-order reasoning used in critical thought when their stems and answer choices have been carefully constructed.⁵⁰ The results of our present study should also aid in the design and writing of MCQ-based examinations that are aimed at assessing critical thinking, which will reflect higher-order cognitive function.

In designing assessments for educational research, it would be ideal to mitigate such potential distortions. MCQs should be chosen based on pre-defined learning objectives, categorized by cognitive level, and their respective performance characteristics determined in various testing environments. Such deliberation would ensure that the assessments used in educational research experiments are optimal regarding validity and difficulty. The dichotomous categorization of questions in this and other studies⁵¹ may be a more direct method to assess critical thinking vs factual recall. Even with carefully written or selected MCQs, perception of question difficulty varies widely for both students and faculty when based on the six levels of Bloom’s taxonomy, blurring lower-order vs higher-order cognition performance.⁵²

Limitations

This study used questions from a limited-scope exam administered to a single cohort of just 56 medical student participants. This is a relatively small sample size. Further, the number of included concept/inference questions was modest at just 43 total, limiting the analysis’s power. Participants had completed a substantial portion of the clerkship year by the time of the 2022 assessments, which is a confounding variable in our interpretation of the findings. Whereas the 2020 assessment was a summative examination, participation in the 2022 assessments was voluntary and had no stakes, which may have impacted the amount of effort expended on the questions. Finally, the mindset of participants is impossible to control and will always be an uncontrolled variable in assessment studies for medical education or research. For example, just being sleepy can affect cognition and clinical reasoning.⁵³

Conclusion

The findings from our study provide critical insights into how different types of cognitive assessments impact learning outcomes in medical education. Our results show that conceptual knowledge, though initially more challenging to master, demonstrates greater durability over time than factual recall, particularly when reinforced through clinical practice. This observation raises important questions about the efficacy of traditional recall-based assessments, which may promote short-term performance but fail to support long-term knowledge retention. These results align with emerging research suggesting that recall-based methods are less effective than those that foster deeper conceptual understanding.^54,55 Prior work demonstrates that implementing retrieval practice questions and spaced learning supports this approach and helps prevent knowledge decay.⁵⁶

Our findings advocate for reevaluating current medical assessment strategies, emphasizing the need to shift the focus from recall-driven evaluations to ones that promote critical thinking and conceptual integration. This aligns with the broader competency-based medical education (CBME) trend, prioritizing practical skills and attitudes alongside clinical knowledge.⁵⁷ However, defining and assessing competencies remains challenging.⁵⁸ Our findings suggest innovative approaches, such as cognitive diagnostic assessments, may offer valuable ways to measure factual recall and retention of more significant, clinically relevant knowledge.⁵⁹

Our results further highlight the need for frequent, longitudinal assessments, which provide formative feedback and support lifelong learning among physicians.⁶⁰ Ultimately, our research points toward integrating conceptual/inferential questions in medical education to better prepare students for long-term success in clinical practice and to encourage the retention of knowledge critical to patient care. Moving forward, medical educators should explore the mechanisms behind knowledge retention and the role of applied clinical learning to refine these assessment practices.