Comprehensive Machine Learning-Based Prediction Model for Delirium Risk in Older Patients with Dementia: Risk Factors Identification

Introduction

China’s rapidly aging population is projected to reach 487 million individuals aged ≥60 years by 2039.¹ This demographic shift will result in a substantial increase in the proportion of those aged 80 and above, highlighting the rising significance of older adults in the population. Delirium, a common neuropsychiatric syndrome characterized by acute cognitive decline, inattention, and altered consciousness, poses significant risks to older adults, particularly those with pre-existing dementia.^2–4 While delirium superimposed on dementia (DSD) represents a distinct clinical phenotype, the broader issue of delirium in older patients with dementia remains understudied, with limited predictive tools for early intervention.

Delirium is a major contributor to adverse outcomes, including prolonged hospitalization, functional decline, and mortality.⁵ However, its diagnosis remains challenging due to overlapping symptoms with dementia and underrecognition in clinical settings.^6–9 The ICD-code for the diagnosis of delirium is ICD-10 F05, which is commonly used in clinical practice. However, a study by Chuen et al⁵ reported that the ICD-10 codes for delirium (eg, F05) exhibit a sensitivity of 46.3% and a specificity of 99.6%. This finding underscores the limitation of ICD-10 coding in fully capturing delirium, especially in populations with pre-existing cognitive impairment, where symptoms may be overlooked or misattributed to dementia. This diagnostic gap highlights the urgency to develop accurate prediction models that can identify high-risk patients before delirium onset.

Previous studies have identified risk factors for DSD, such as cerebrovascular disease, sedative use, and inflammation markers.¹⁰ However, these investigations were limited by small sample sizes, retrospective designs, and narrow variable selection (eg, excluding socioeconomic factors). Additionally, most studies focused exclusively on DSD rather than general delirium in dementia patients, restricting their clinical applicability. To address these gaps, this study aimed to: 1) develop a machine learning-based prediction model for delirium risk in older dementia patients; 2) identify key risk factors using comprehensive clinical data; and 3) validate the model’s performance across multiple algorithms.

We leveraged a large, prospectively collected dataset of 636 hospitalized dementia patients to construct and compare five machine learning models (Extreme Gradient Boosting [XGB], Random Forest [RF], Multilayer Perceptron [MLP], Categorical Boosting [CB], and Logistic Regression [LR]). By incorporating demographic, biochemical, and clinical variables, we sought to improve predictive accuracy and identify modifiable risk factors. This study advances previous work by using a robust prospective design, diverse algorithm comparison, and interpretability tools (SHapley Additive exPlanations [SHAP]) to enhance clinical translation.

Methods

Study Population and Data Sources

This prospective study aimed to develop a machine learning-based risk prediction model for delirium in older patients with dementia. During the study period, a total of 671 patients were initially screened for eligibility. Among them, 35 patients were excluded according to the criteria: 15 cases transferred to other hospitals (primary reasons: need for specialized neurological care, n=8; family request due to personal reasons, n=7), 9 cases died during hospitalization (primary causes: advanced dementia-related complications, n=5; acute cardiovascular events, n=4), and 11 cases had ≥90% missing data in key variables. Finally, a total of 636 patients aged 65 years or older, diagnosed with dementia (The Tenth Revision of International Classification of Diseases, ICD-10 code G30), were recruited from the International Department of China-Japan Friendship Hospital between January 2020 and December 2023. All participants were hospitalized for the first time due to dementia. Comprehensive clinical data were collected for each patient, including demographic information (age, gender, body mass index [BMI], body temperature), biochemical parameters (C-reactive protein [CRP], SOD), comorbidities (hypertension, diabetes, hyperlipidemia, cardiovascular disease, cerebrovascular disease, renal insufficiency, anemia, and hypoalbuminemia), smoking and alcohol consumption history, presence of accompanying family members, surgical history during hospitalization, Visual Analogue Scale (VAS) scores, use of sedative medications, and placement of drainage tubes. For laboratory data, we used the results of the first tests conducted after the dementia diagnosis. To construct an accurate risk prediction model, we specifically focused on delirium onset during hospitalization, recording whether delirium occurred. The study was approved by the ethics committee of China-Japan Friendship Hospital and conducted in accordance with the principles of the Declaration of Helsinki. As the study involves elderly dementia patients, informed consent was obtained from their legal guardians or family members before participation.

Data Preparation

To mitigate potential bias, all variables were blinded. Categorical variables were encoded as 0 and 1, while continuous variables were standardized using the Z-score method. Laboratory results were categorized as 1, 2, or 3, corresponding to values below the normal range, within the normal range, and above the normal range, respectively. Variables with missing data exceeding 90%, those with a single dominant value accounting for over 90%, or those with a coefficient of variation below 0.1 were excluded. Missing values were then imputed using a random forest (RF) algorithm, and Lasso regression was applied for variable selection.

Algorithms for Machine Learning

To develop the optimal prediction model, we employed five prominent machine learning algorithms: XGB, RF, Multilayer Perceptron (MLP), Categorical Boosting (CB), and Logistic Regression (LR). These algorithms were selected for their predictive capabilities in model development.

XGB

XGB is an ensemble classification method that leverages regression trees, designed to offer both fast training and superior predictive accuracy.¹¹ It enhances model performance by utilizing a gradient boosting technique, where weak learners are combined iteratively to improve predictions. XGB adjusts residuals from previous iterations to minimize errors, optimizing the model’s overall precision. Additionally, it incorporates an internal regularization mechanism, which helps prevent overfitting, ensuring the model’s generalizability. This combination of regularization and iterative optimization contributes to XGB’s robustness and effectiveness in handling complex datasets.

RF

RF is a powerful ensemble learning technique that constructs a large collection of decision trees, often ranging from hundreds to thousands, to solve classification and regression problems.¹² Each tree in the forest is built using a random subset of the data and features, which promotes diversity among the trees and enhances the model’s predictive power. The final prediction is made by taking the majority vote (in classification tasks) or averaging the outputs (in regression tasks) of all individual trees. One of the key strengths of the RF algorithm is its robustness and stability, which makes it highly effective in handling small amounts of noise and outliers in the data. Due to its ensemble nature, RF is less prone to overfitting compared to single decision trees, especially when the number of trees is sufficiently large.

MLP

The MLP is a type of machine learning algorithm that utilizes feedforward neural networks, which function similarly to the way human neurons process information.¹³ In its architecture, the MLP is composed of several layers, including an input layer that receives the data, one or more hidden layers that process the information, and an output layer that generates the final result.¹⁴ Each layer contains multiple neurons that work in unison to learn complex patterns from the input data. The layers are connected through weighted links, with the network adjusting these weights during the training process to improve accuracy.

CB

CB is a machine learning classification technique that utilizes oblivious trees, a specialized tree structure designed to handle classification tasks in an efficient and interpretable way.¹⁵ Unlike traditional decision trees, oblivious trees ensure that the same feature is used for all branches at each level of the tree, thereby reducing the model’s complexity. This approach effectively mitigates common challenges in machine learning models, such as gradient bias and prediction shift, which can contribute to overfitting. By addressing these issues, CB enhances the model’s overall performance, leading to improved accuracy and stronger generalization capabilities.

LR

An LR model is a statistical tool used to examine the relationship between one or more independent variables and a binary dependent variable, often referred to as the outcome or response.¹⁶ It is particularly useful for predicting the probability of a binary event occurring, such as the presence or absence of a disease, success or failure, or other dichotomous outcomes. LR models offer a straightforward interpretation, where the estimated coefficients represent the impact of each predictor on the odds of the outcome occurring.¹⁷ Due to its simplicity and robustness, LR has been extensively applied across various medical research domains, including epidemiology, clinical studies, and public health, to model and understand complex relationships between risk factors and health outcomes.

Algorithm Comparison

LR offers the highest interpretability due to its linear coefficients, making it particularly well-suited for scenarios that require transparent risk factor analysis. RF and XGB provide a balance of interpretability through feature importance metrics (such as SHAP values) while maintaining strong generalization capabilities. CB exhibits robustness against gradient bias, though it may require more tuning for optimal performance. MLP, a deep learning model, has the lowest interpretability but excels in capturing complex non-linear relationships. However, it demands larger datasets and higher computational resources to prevent overfitting.

In terms of computational efficiency, LR and CB are the most cost-effective, followed by RF. XGB and MLP are more computationally intensive, particularly for large-scale datasets. Regarding sample size requirements, MLP and XGB perform best with datasets larger than 500 samples, while LR and RF remain stable even with smaller cohorts. These characteristics make LR ideal for initial hypothesis testing, XGB/RF preferable for complex clinical prediction tasks, and MLP a viable option when rich data and sufficient computational resources are available.

For readers interested in learning more about machine learning tools, I recommend visiting the website https://scikit-learn.org/stable/, which provides valuable resources and documentation on various algorithms and their applications.

Model Establishment

The prediction models were constructed using XGB, MLP, RF, LR, and CB algorithms. The data were first split into training and test sets using an 8:2 ratio. All preprocessing steps, including data cleaning, normalization, and missing data estimation, were performed separately on the training and test sets to prevent data leakage and ensure an independent evaluation of model performance. The training set was used to build the classification models, and the test set was used for performance evaluation. To address the class imbalance in the training set, the Borderline Synthetic Minority Oversampling Technique (SMOTE) was employed.¹⁸ This improved version of SMOTE generates synthetic samples from borderline instances, enhancing the class distribution. The models were trained on the original data, and their predictive accuracy was evaluated on the test set.

Model Evaluation

To assess the predictive capability of the machine learning models, we computed several key performance metrics, including the area under the receiver operating characteristic curve (AUC), accuracy, precision, recall, and F1 score. Given the importance of precision and recall in clinical contexts, we also evaluated the area under the precision-recall curve (AUPRC). The calibration of the model was assessed using the Brier score and a calibration plot, with favorable calibration defined by a Brier score ≤0.25.¹⁹ To understand the influence of each variable on the model, SHapley Additive exPlanations (SHAP) values were utilized.

Statistical Analysis

Categorical variables, presented as frequencies and percentages, were analyzed using chi-square tests. For continuous data, descriptive statistics included mean ± standard deviation or median with interquartile ranges (Q1–Q3). The t-test or Mann–Whitney U-test was employed to evaluate statistical differences between groups. A p-value below 0.05 indicated statistical significance. Data processing and statistical evaluations were carried out using SPSS software (Version 25, IBM SPSS Statistics, IBM Corp., Armonk, NY, USA). Additionally, statistical modeling was conducted with the stats and sklearn libraries in Python (Version 3.8).

Results

Study Population

A total of 636 cognitively impaired patients aged over 65 years, admitted to our hospital’s comprehensive ward, were included in the study. Among them, 357 (56.1%) were men and 279 (43.9%) were women, with a mean age of 78.4 ± 6.5 years. Patients were categorized into two groups: 187 in the delirium group and 449 in the non-delirium group, yielding a delirium incidence rate of 29.4%. Baseline characteristics are summarized in Table 1.

Table 1 Baseline Characteristics of Delirium vs Non-Delirium Patients

Data Processing and Variable Identification

A total of 115 variables were initially collected, as detailed in Supplementary Table 1. These variables included three general information parameters (X1–X3), eight comorbidities (X4–X11), five medications (X12–X16), and 99 laboratory test results (X17–X115). Variables with missing data exceeding 90%, those with a single value comprising over 90% of the observations, and those with a coefficient of variation below 0.1 were excluded. This left 84 variables for further analysis (Supplementary Table 2). Lasso regression was then applied for variable selection, resulting in the final inclusion of 52 variables in the model (Supplementary Table 2).

Model Establishment and Validation

To predict delirium in older dementia patients, machine learning models, including XGB, MLP, RF, LR, and CB, were built using 52 features from the training set. Their predictive performance was assessed on a separate test set, with metrics such as AUC, accuracy, precision, recall, and F1 score reported in Table 2. Among the models, XGB outperformed the others, achieving the highest AUC (0.967), accuracy (0.907), and F1 score (0.918). To improve clinical feasibility, a streamlined version of the XGB model was developed, using the top 10 most important variables based on their mean absolute SHAP values. The performance of this compact XGB model yielded an AUC of 0.930, accuracy of 0.870, precision of 0.908, recall of 0.872, and F1 score of 0.892 (Table 2). The corresponding ROC curves for these models are shown in Figure 1A.

Table 2 Performance of Machine Learning Models on the Test Set

Figure 1 Model establishment. (A) Receiver operating characteristic (ROC) curves and (B) precision-recall curves for the six machine learning models on the test set. Abbreviations: AUC, area under the receiver operating characteristic curve; MLP, multilayer perceptron; LR, logistic regression; RF, random forest; CB, categorical boosting; SHAP, SHapley Additive exPlanations; XGB, extreme gradient boosting.

For the original XGB model, trained on the test set, the AUC, accuracy, precision, recall, and F1 score were 0.905, 0.862, 0.873, 0.927, and 0.608, respectively (see Supplementary Table 3). Additional ROC curves for these models can be found in Supplementary Figure 1A.

Recall represents the proportion of delirium cases correctly identified, whereas precision reflects the accuracy of predicting delirium among the cases identified. Both metrics are critical in clinical practice for predicting delirium in patients and assisting healthcare providers in making informed decisions. To evaluate the performance of the machine learning models, we utilized precision-recall curves (Figure 1B). The XGB model achieved the highest AUPRC of 0.989, followed by the compact XGB model, which yielded an AUPRC of 0.966. The precision-recall curves for models trained on the original dataset are shown in Supplementary Figure 1B. In addition, calibration curves were employed to assess the models’ calibration, with both the XGB and compact XGB models demonstrating strong calibration performance (Figure 2). The Brier score, which quantifies both calibration and discrimination, was 0.07 for the XGB model and 0.11 for the compact XGB model (Table 2).

Figure 2 Calibration plot for both the XGB and compact XGB models. XGB: extreme gradient boosting.

Interpretation of Model Variables

Based on SHAP value analysis, the 10 most influential variables in the XGB model were identified, including cerebrovascular disease, use of sedative drugs, hemoglobin, VAS score ≥4 points, SOD, diabetes, hsCRP, hypertension, whether accompanied by family members, and hyperlipidemia. These factors were subsequently employed to develop a streamlined XGB model.

Discussion

This study aimed to develop a robust machine learning-based prediction model to assess the risk of delirium in older patients with dementia, a group highly susceptible to this severe and often underrecognized condition. Our analysis demonstrated that machine learning algorithms, particularly XGB, can effectively predict the onset of delirium in this population, with the model showing excellent performance in terms of both accuracy and predictive power. We utilized XGB, RF, MLP, CB, and LR algorithms, combined with a comprehensive set of clinical data, to predict delirium risk in older dementia patients. Among these, XGB emerged as the most effective model, achieving the highest AUC, accuracy, and F1 score. While MLP exhibited a slightly higher recall than XGB, its lower precision and accuracy limited its clinical applicability.^2,20 Moreover, the XGB model excelled in calibration, achieving the highest AUPRC and the best Brier score. To enhance clinical applicability, we refined the XGB model by focusing on the top 10 most significant risk factors, resulting in a more efficient model that maintained strong predictive power.

Our findings are consistent with existing research highlighting the increased risk of delirium in hospitalized older patients, particularly those with cognitive impairment.²¹ With a delirium incidence rate of 29.4% in our cohort, our study emphasizes the clinical relevance of this issue. The identification of key risk factors, including cerebrovascular disease, sedative drug use, and elevated VAS scores, aligns with prior studies that have associated these variables with a higher likelihood of delirium.^2,22 These factors provide critical insights into the pathophysiology and risk stratification of delirium, which are essential for timely intervention.

The XGB model demonstrated superior performance compared to other machine learning algorithms in this study, achieving the highest AUC of 0.967, accuracy of 0.907, and F1 score of 0.918. These results underscore the efficacy of XGB in clinical prediction tasks. By reducing the model to the top 10 most important features based on SHAP values, we developed a streamlined version that maintained high predictive performance, with an AUC of 0.930 and an F1 score of 0.892. These findings emphasize the potential of machine learning as a robust and clinically feasible tool for predicting delirium, which could be implemented in high-paced hospital environments to enhance patient outcomes.

Our study also provided valuable insights into the key risk factors for delirium in patients with dementia. Factors such as cerebrovascular disease, sedative use, and elevated hsCRP levels were identified as the most significant predictors. These findings align with existing literature, which highlights the critical roles of cerebrovascular insults and inflammation in the onset of delirium.^8,23 The identification of cerebrovascular disease, sedative use, and elevated hsCRP levels as significant predictors is biologically plausible. Cerebrovascular disease may disrupt brain perfusion and increase vulnerability to delirium by impairing neurovascular coupling and promoting neuroinflammation.²⁴ Sedative medications, such as benzodiazepines, enhance GABAergic neurotransmission, which can exacerbate cognitive dysfunction and increase delirium risk through sedation and altered arousal regulation.²⁵ Elevated hsCRP levels reflect systemic inflammation, which contributes to blood-brain barrier permeability and microglial activation, driving neuroinflammation and cognitive impairment.²⁶ These mechanisms underscore the multifactorial nature of delirium in dementia patients and highlight potential targets for intervention. Additionally, the use of sedative medications remains a well-established risk factor, underscoring the importance of careful medication management in this vulnerable population.^9,27 Notably, family support emerged as a significant variable, suggesting that the presence of family members may reduce the risk of delirium, potentially by fostering cognitive stimulation and providing emotional support.

One of the strengths of this study lies in the use of a large, well-characterized cohort of older dementia patients, with comprehensive clinical data collected prospectively. This methodology facilitated the identification of a broad range of potential risk factors, providing a solid foundation for the development of our prediction model. However, several limitations should be acknowledged. First, while the model demonstrated strong performance on the validation set, its generalizability to other populations—such as those with different dementia subtypes or individuals in community settings—requires further validation. Additionally, despite the robustness of the dataset, some potential risk factors, such as education level or socioeconomic status, were not included, and their omission may influence the model’s performance and broader applicability. Future research should examine these variables and assess their impact on delirium risk.

In conclusion, our study highlights the feasibility and effectiveness of machine learning in predicting delirium risk in older patients with dementia. By identifying key risk factors and developing a streamlined, clinically applicable model, we provide a valuable tool for clinicians in managing and preventing delirium. Future research should focus on validating the model across diverse clinical settings, exploring additional risk factors, and further refining the prediction algorithms to enhance their clinical utility.