Machine Learning-Based Prediction of Postoperative Pneumonia Among Super-Aged Patients With Hip Fracture

Introduction

With the global aging population on the rise, hip fractures have become a significant health concern,¹ particularly among the older adults. Super-aged patients, defined as those aged 80 years and older,^2,3 were at a heightened risk of complications following surgery due to their frailty and the presence of multiple comorbidities. One of the most severe and common complications in this demographic is postoperative pneumonia,⁴ which can lead to increased morbidity, extended hospital stays, and higher mortality rates. Given the aging population and the increasing prevalence of hip fractures, addressing postoperative pneumonia was a critical component of improving surgical outcomes and overall patient care among super-aged patients with hip fracture.

The proposal of predictive models relied on risk factors, and numerous studies have already reported risk factors associated with postoperative pneumonia following hip fractures.^5–8 The main factors included age,^5–8 ASA score,^5,7 smoking,⁵ diabetes,⁵ coronary heart disease,^5,6,8 chronic obstructive pulmonary disease,⁵ dementia,^5,6 hemoglobin levels,⁷ albumin levels,⁷ and blood urea nitrogen.⁷ These factors provided significant guidance for clinical practice in preventing postoperative lung infections among patients with hip fracture. Notably, using well-established risk factors to build a model could increase its robustness and enhance its predictive performance.

Thus, several prediction models have been specifically developed for patients with hip fractures to assess postoperative pneumonia,^9,10 acute respiratory failure,¹¹ and early infections.¹² These models can be beneficial for clinical guidance in preventing postoperative pneumonia in patients with hip fractures. However, the AUC values of these models were mostly below 0.85, indicating that their accuracy needs further improvement. Traditional clinical assessment tools might not be sufficiently predictive for this high-risk group, thus necessitating the development of more advanced and precise predictive models. Recent advancements in machine learning offer a promising avenue for enhancing predictive accuracy in medical outcomes.^13,14 Machine learning algorithms could analyze vast amounts of data and identify complex patterns that might be overlooked by conventional methods.^15,16 By incorporating various patient-specific factors, these algorithms could provide personalized risk assessments, thereby aiding clinicians in identifying patients at high risk for postoperative pneumonia and tailoring preventive strategies accordingly. For instance, there were two prediction models for postoperative pneumonia in hip fracture patients,^17,18 and the two studies employed different statistical and machine learning approaches. Nonetheless, previous studies primarily included populations aged 60 or 65 years and above, rather than super-aged patients.

Therefore, the objective of this study was to establish and validate a robust machine learning-based model to predict the occurrence of postoperative pneumonia in super-aged patients undergoing surgery for hip fracture. By accurately identifying those at higher risk, healthcare providers could implement targeted interventions and preventive measures, ultimately reducing the incidence of this serious complication, enhancing patient recovery, and optimizing the use of medical resources.

Patients and Methods

Patients and Study Design

The study reanalyzed data of 555 patients from the Chinese PLA General Hospital (PLAGH) Hip Fracture Cohort Study. The study focused on hip fracture patients diagnosed with femoral neck fractures or femoral intertrochanteric fractures. All patients meeting the following criteria were included: (1) diagnosed with a femoral neck or intertrochanteric fracture, and (2) received surgical treatment. Exclusion criteria were: (1) patients under 80 years old, (2) radiograph-confirmed pneumonia within 48 hours of admission, (3) patients receiving conservative treatment, and (4) history of pneumonia within 3 months prior to hospitalization. Figure 1 illustrates the patient enrollment flowchart. The Medical Research Ethics Board of the Chinese PLA General Hospital approved the study protocol and waived the need for patient consent for the review of medical records and images due to the retrospective nature of the review. In addition, the data was anonymized or maintained with confidentiality. The study was conducted in accordance with the Declaration of Helsinki. In addition, the study adhered to the STROCSS criteria¹⁹ and the TRIPOD Checklist.²⁰

Figure 1 Patient’s flowchart.

Collection of Variables

This study gathered a comprehensive range of variables, encompassing baseline characteristics like age, gender, body mass index (BMI), and smoking status. It also included data on comorbidities such as chronic obstructive pulmonary disease (COPD), diabetes, hypertension, coronary heart disease, previous stroke, dementia, digestive system disorders, renal disease, and the total number of comorbidities. Additionally, information on fractures, including fracture type, and surgical details such as the American Society of Anesthesiologists (ASA) classification, type of surgery, and use of mechanical ventilation, were collected. Laboratory test results, including hemoglobin (HB), albumin levels, blood creatinine, and red cell distribution width (RDW), were also documented. All these variables were recorded post-hospitalization but prior to surgery.

Identification of Model Features

Feature importance analysis was utilized to assess the relative significance of each feature in influencing the outcome. By pinpointing the most impactful features, we gained insights into the key factors driving the results, thereby enhancing the interpretability and validation of the model’s predictions. In this investigation, Shapley Additive Explanation (SHAP) values were employed to evaluate the importance of each input parameter.^21,22 SHAP values specifically measure the contribution of each feature to the model’s output. For this study, variables with SHAP values equal to or greater than 0.25 were deemed significant model features.

Definition of Outcome

Postoperative pneumonia was diagnosed based on the presence of new infiltrates on chest X-ray, along with one or more of the following symptoms observed during hospitalization after surgery: (1) new or worsening respiratory symptoms such as coughing and expectoration; (2) fever or hypothermia; (3) physical examination revealing signs of lung consolidation or moist rales; (4) a white blood cell count exceeding 10×10⁹/L or below 4×10⁹/L; (5) isolation of pathogens from blood culture or sputum.

Modeling

This study employed six algorithms to train and optimize models: Decision Tree (DT), eXtreme Gradient Boosting Machine (eXGBM), Support Vector Machine (SVM), Random Forest (RF), Neural Network (NN), and Logistic Regression (LR). Each model was provided with the same input features to maintain consistency. Prior to modeling, the SMOTETomek resampling strategy^23–25 was applied to address data imbalance and enhance model robustness. A comprehensive data preprocessing pipeline ensured consistent and reproducible data transformations. Additionally, a stratified sampling approach was used to preserve the proportion of outcome classes within sub-datasets. During modeling, both grid and random hyperparameter searches were conducted alongside cross-validation to determine the optimal hyperparameters for each model. The Area Under the Curve (AUC) was used as the primary optimization metric. To accommodate variability in model performance, a wide range of hyperparameter values was explored, resulting in a combination of underfitted and overfitted models.

Model Validation

The models were validated using several evaluation metrics, such as AUC, accuracy, precision, recall, F1 score, Matthews’s correlation coefficient (MCC), Brier score, and log loss. To evaluate the models’ discriminative ability, calibration, and clinical net benefits, probability density curves, calibration curves, and decision curves were employed, respectively. Additionally, a confusion matrix was utilized to assess accuracy, precision, and recall metrics.

The Brier Score is a metric used to evaluate the accuracy of probabilistic predictions. It quantifies the mean squared difference between the predicted probabilities and the actual binary outcomes, providing a clear measure of how closely the predictions align with the observed results. It is defined mathematically as:

Where, N represents the total sample, is the predicted risk of postoperative in-hospital mortality, and is the actual probability of postoperative in-hospital mortality.

Log loss, commonly referred to as logistic loss or cross-entropy loss, is a widely used performance metric in classification tasks, especially in binary classification scenarios. This metric evaluates the accuracy of a classifier by comparing the predicted probabilities with the actual class labels, thereby providing insight into the model’s performance. The log loss formula is:

Where, N is the number of samples, M is the number of classes, represents the true label of sample i for class j (0 or 1), and is the predicted probability of sample i belonging to class j.

Statistical Analysis

Categorical variables were represented as percentages, while continuous data were summarized using medians and interquartile ranges. Machine learning algorithms and hyperparameter tuning were conducted using Python (version 3.9.7) with the scikit-learn library (version 1.2.2). All other statistical analyses were performed using R (version 4.1.2). A p-value of less than 0.05 was deemed statistically significant.

Results

Baseline Characteristics

The baseline clinical characteristics of 555 patients who underwent surgery for hip fractures were summarized in Table 1. The median age was 84 years (IQR 82–88), with 32.1% male and 67.9% female patients. The median BMI was 22.49 kg/m² (IQR 19.89–23.41). The fracture types included 37.3% intertrochanteric and 62.7% femoral neck fractures. The prevalence of smoking was low (3.4%), while other comorbid conditions such as COPD (5.6%), diabetes (23.4%), hypertension (58.0%), coronary heart disease (25.8%), and previous stroke (16.4%) were noted. Dementia was present in 5.8% of patients, and 4.3% had digestive system disorders. Renal disease was observed in 3.2% of patients. According to the ASA classification, 40.9% were ASA 2, 51.4% were ASA 3, and 5.8% were ASA 4. Regarding the type of surgery, 41.6% underwent intramedullary fixation, 50.3% had hip replacements, and 8.1% had other procedures. Mechanical ventilation was required in 31.0% of cases. Median hemoglobin level was 118 g/L (IQR 103–130), median albumin was 36.3 g/L (IQR 33.0–39.35), and median blood creatinine was 65.4 µmol/L (IQR 60.05–76.55). The median RDW was 13.3% (IQR 12.7–14.0). Comorbidities were present in varying numbers: 26.7% had no comorbidities, 28.3% had one, 20.7% had two, and 24.3% had three or more. Postoperative pneumonia occurred in 7.2% of patients.

Table 1 Baseline Clinical Characteristics Among Patients With Hip Fracture Treated With Surgery

Identification of Model Features

Based on SHAP, the variable importance ranking was as follows: RDW, blood creatinine, surgery type, HB, albumin, age, diabetes, BMI, the number of comorbidities, coronary heart disease, ASA, gender, dementia, previous stroke, fracture type, mechanical ventilation, hypertension, renal disease, COPD, digestive system disorders, and smoking (Figure 2A). There were 10 variables with SHAP values greater than or equal to 0.25: RDW, blood creatinine, surgery type, HB, albumin, age, diabetes, BMI, the number of comorbidities, and coronary heart disease (Figure 2B). The top three variables in terms of importance were RDW, blood creatinine, and surgery type, with SHAP values of 0.69, 0.42, and 0.41, respectively. Hence, these 10 variables were used as model features to train models.

Figure 2 Feature importance analysis. (A) Feature importance ranking; (B) SHAP value of all features.

Model’s Prediction Performance

Among the six models, the eXGBM model demonstrated the optimal model, with the area under the curve (AUC) value of 0.929 (95% CI: 0.900–0.959), followed by the RF model (AUC: 0.916, 95% CI: 0.885–0.948). The LR model only had an AUC value of 0.720 (95% CI: 0.662–0.778) (Figure 3). In addition, the eXGBM model demonstrated the best overall prediction performance across multiple metrics. It achieved the highest values for accuracy (0.858), precision (0.870), and F1 score (0.855) (Table 2). Additionally, the eXGBM model had the best Brier score (0.104) and the lowest log loss (0.349), indicating superior calibration and overall model reliability. Regarding MCC, the eXGBM model also had the highest MCC value of 0.717, suggesting it performs best among the models evaluated. The RF model followed closely with an MCC of 0.695, while the NN model and the SVM model showed moderate performance with scores of 0.683 and 0.459, respectively. The DT model and the LR model had lower MCC values of 0.439 and 0.356, indicating relatively poorer performance compared to the other models.

Table 2 Prediction Performance of All Models in the Study

Figure 3 Area under the curve analysis for all machine learning models. Abbreviations: DT, decision tree; eXGBM, extreme gradient boosting machine; SVM, support vector machine; RF, random forest; NN, neural network; LR, logistic regression.

The probability density curve analysis of the DT model showed that patients with postoperative pneumonia could be distinguished from those without it, but there was considerable overlap between the two groups (Figure 4A). In contrast, the eXGBM model (Figure 4B), SVM model (Figure 4C), RF model (Figure 4D), and NN model (Figure 4E) demonstrated more favorable predictive performance, as the peak risk probability for the postoperative pneumonia group shifted to the right, while that for the non-postoperative pneumonia group shifted to the left. However, the LR model exhibited the poorest discrimination (Figure 4F). Calibration curve analysis demonstrated that most models had favorable calibration ability, as their curves closely aligned with the ideal calibration curve (Figure 5). Additionally, decision curve analysis indicated that the eXGBM model, the RF model, and the NN model provided favorable clinical net benefits across various threshold risks (Figure 6).

Figure 4 Probability density curve analysis for all machine learning models. (A) Decision tree; (B) Extreme gradient boosting machine; (C) Support vector machine; (D) Random forest; (E) Neural network; (F) Logistic regression.

Figure 5 Calibration curve analysis for all machine learning models.

Figure 6 Decision curve analysis for all machine learning models.

Individualized Prediction

Individualized predictions were achieved in this study using the optimal model, the eXGBM model. For instance, in a true positive case (Figure 7), the patient was 89 years old and underwent hip replacement surgery. The patient’s hemoglobin (HB) level was 97 g/L, albumin was 26.8 g/L, blood creatinine was 145 µmol/L, red cell distribution width (RDW) was 17%, and body mass index (BMI) was 22.2 kg/m². The patient had three comorbidities but did not have coronary heart disease or diabetes. In this scenario, HB, albumin, and surgery type were the most significant features contributing to the risk of postoperative pneumonia.

Figure 7 Individualized prediction: A true positive case.

In another case, which was a true negative (Figure 8), the patient was 84 years old with an RDW of 12.8%, albumin of 40.9 g/L, HB of 124 g/L, blood creatinine of 78.9 µmol/L, and a BMI of 24.0 kg/m². This patient had one comorbidity, was treated with hip replacement, and did not have coronary heart disease or diabetes. The total score for this case was −6.07, which was lower than the base score of −0.02, indicating a low risk of postoperative pneumonia.

Figure 8 Individualized prediction: A true negative case.

Discussion

Main Findings

The main finding of this study is the development and validation of a machine learning-based model to predict postoperative pneumonia in super-aged patients with hip fractures. The eXGBM model, in particular, demonstrates superior predictive performance and calibration compared to other algorithms tested. This model identifies key variables such as RDW, blood creatinine, surgery type, and hemoglobin, providing a robust basis for accurate risk stratification.

The findings from our study indicate a promising application of machine learning techniques in the clinical setting, particularly for predicting postoperative pneumonia in super-aged patients with hip fractures. By identifying patients at elevated risk for postoperative pneumonia, healthcare providers can implement targeted preventive measures and tailored postoperative care protocols. Furthermore, the integration of this predictive model into routine clinical workflows can facilitate early interventions that may reduce the incidence of postoperative complications, ultimately improving patient outcomes and hospital resource utilization.

Incidence of Postoperative Pneumonia in Hip Fracture Patients

The incidence of postoperative pneumonia in hip fracture patients varied across different studies. Previous research has reported rates ranging from 1.8% to 9.26%,^5–7,26 depending on the population studied and the definitions used for pneumonia. A meta-analysis included 12,084 patients for analysis, and found 809 patients had postoperative pneumonia, and thus the incidence of postoperative pneumonia was found to be 6.69%.⁵ Another meta-analysis revealed that the incidence of postoperative pneumonia was 7.8% after analyzing ten studies consisting of 37,130 patients.⁷ In our study, the incidence of postoperative pneumonia was found to be 7.2%, which falls within the previously reported range. This comparison highlights the consistency of our findings with existing literature and underscores the clinical relevance of our predictive model.

Risk Factors for Postoperative Pneumonia in Hip Fracture Patients

Several risk factors for postoperative pneumonia in hip fracture patients have been identified in the previous literature. For example, in a meta-analysis of ten studies, Gao et al⁵ demonstrated that older age, male gender, ASA score of three or above, dependent functional status, smoking, chronic obstructive pulmonary disease, diabetes mellitus, coronary heart disease, arrhythmia, cerebrovascular disease, dementia, chronic renal failure, hip arthroplasty, delayed surgery, preoperative creatinine, and preoperative serum albumin were significantly associated with postoperative pneumonia. A large retrospective cohort study found that the predictive factors of pneumonia among older patients treated with hip fracture surgery included advanced age, male sex, lean body, cerebrovascular disease, dementia, and dependency for activities of daily living.⁶ A study of collecting 1208 patients aged more than 65 years and treated with hip fracture surgery also revealed that an ASA score of 3 or above, higher Charlson Comorbidity Index, and postoperative delirium were significantly risk factors for postoperative pneumonia.²⁶ More recently, Yao et al²⁷ neutrophil-to-lymphocyte ratio was an important predictor for postoperative pneumonia among older adults with hip fracture. Han et al⁷ also conducted a meta-analysis of ten studies and found that advanced age, male sex, ASA score of ≧3, chronic obstructive pulmonary disease, coronary heart disease, arrhythmia, congestive heart failure, chronic kidney disease, and cerebrovascular accident. Additionally, the following factors are associated with postoperative pneumonia: Hemoglobin levels, albumin levels, blood urea nitrogen, alanine aminotransferase, arterial oxygen pressure, time from injury to surgery, and surgery within 48 hours. A retrospective analysis also revealed that age, sex, respiratory disease, heart disease, cerebrovascular disease, liver disease, preoperative stay, general anesthesia was found to be risk factors for postoperative pneumonia.⁸ Our findings were consistent with previous studies.^28,29 In our studies, we also found that surgery type, albumin, age, diabetes, BMI, the number of comorbidities, and coronary heart disease were the most important ten contributors to postoperative pneumonia. Additionally, our model highlights the importance of specific laboratory values like RDW and blood creatinine, which are less frequently reported in previous studies. Using well-established risk factors to build a model can increase its robustness and enhance its predictive performance. In addition, this broader inclusion of variables may enhance the predictive power of our model and provide a more comprehensive understanding of the risk factors involved.

Comparison With Previous Predictive Models

After comprehensively reviewing literature, we found that there were two prediction models for postoperative pneumonia in hip fracture patients, and the two studies employed different statistical and machine learning approaches. For example, Guo et al¹⁷ developed machine learning models to identify older adults at high risk of postoperative pneumonia after hip fracture surgery. Conducted at a central hospital, the study included 805 patients (≧60 years), with 75 (9.3%) developing postoperative pneumonia. Data collected within 24 hours of admission were used to build models with seven machine learning algorithms: CART, GBM, KNN, LR, NNet, RF, and XGBoost. Key predictive features included age, cerebral infarction, COPD, WBC, HB, GLU, STB, GLOB, and Ka⁺. Among these, the XGBoost model demonstrated the best performance, making it highly effective for early detection of the risk of postoperative pneumonia among older adults with fracture. Another study conducted by Dai et al¹⁸ also developed and validated a machine learning model, specifically the Catboost model, to predict postoperative pneumonia in older adults with hip fractures. Using clinical data from two hospitals, the model demonstrated high predictive accuracy, with AUC values of 0.895 and 0.835 in training and testing sets, respectively, and 0.894 during external validation. Key predictors identified included CRP levels, the modified five-item frailty index, and ASA body status. Notably, in comparison to these models, our eXGBM-based approach exhibited superior predictive performance, achieving an AUC value of 0.929. Additionally, the MCC for the eXGBM model was the highest among all, with a value of 0.717. Our model was developed based on a cohort of super-aged patients, specifically including those aged 80 years and older. In contrast, previous studies primarily included populations aged 60 or 65 years and above. Incorporating patients aged 80 years and older into our study allowed for a more targeted and relevant analysis for this unique demographic, who often presented different clinical characteristics and healthcare needs compared to relatively younger populations. Furthermore, it enhanced the generalizability and applicability of our model to the oldest-old population, who were typically underrepresented in clinical research. In addition, the inclusion of a wide range of variables and the application of a robust machine learning framework distinguished our model from previous efforts. However, like earlier models, our study underscored the need for further external validation to confirm its applicability across diverse clinical settings. Additionally, integrating machine learning-based models with clinical practice is the future trend, although there are still numerous challenges in the eHealth field that need to be addressed to improve patient outcomes.³⁰

Limitations

Despite the promising results of our study, several limitations should be noted. Firstly, the retrospective nature of the analysis may introduce inherent biases related to data collection and patient selection. Secondly, the study was conducted in a single institution, which may limit the generalizability of the findings to other settings or populations. Thirdly, although we employed a variety of machine learning algorithms, the study’s performance metrics may still be affected by the relatively small sample size (555 patients), especially when dealing with a rare outcome like postoperative pneumonia. Fourthly, our dataset lacked certain potentially influential variables, such as detailed nutritional status, preoperative functional status, and intraoperative factors, which could further refine the predictive model. Finally, while the eXGBM model showed the best performance, external validation in different cohorts is necessary to confirm its robustness and applicability in broader clinical practice.

Conclusions

This study develops and validates a machine learning-based model to predict postoperative pneumonia in super-aged patients with hip fractures. Utilizing a range of advanced algorithms, the eXGBM model emerges as the most effective, demonstrating superior predictive performance and calibration. The identified key variables, including RDW, blood creatinine, surgery type, and hemoglobin, among others, provide a strong basis for accurate risk stratification. Despite the study’s limitations, the model holds promise as a valuable clinical tool to guide preoperative planning and postoperative management, ultimately improving patient outcomes in this vulnerable population. Further external validation is recommended to ensure the model’s generalizability and robustness across diverse clinical settings.