A Predictive Model for Pulmonary Aspergillosis in ICU Patients: A Multicenter Retrospective Cohort Study

Background

Aspergillus is one of the most common causes of fungal infections in humans.¹ According to statistics, in 2017, there were over 1.8 million cases of invasive fungal infections globally, with approximately 250,000 cases of invasive pulmonary aspergillosis (IPA).² The mortality rate for invasive pulmonary aspergillosis (IPA) is extremely high, ranging from 45% to 90%.^3,4 Lack of understanding of host factors in critically ill patients, non-specific pulmonary imaging, and the ambiguous clinical significance of positive Aspergillus cultures from airway secretions lead to delayed diagnosis and treatment of IPA in the intensive care unit (ICU), thereby contributing to elevated mortality rates.^5–7 At present, clinical diagnostic algorithms and prediction models for IPA have been developed.^8–11 However, these models have traditionally been based solely on clinical features. Meanwhile, for patients with IPA, the detection of other concurrently present microorganisms, such as Pseudomonas aeruginosa, Klebsiella pneumoniae, respiratory syncytial virus, Epstein-Barr virus (EBV), and cytomegalovirus, is as important.¹² Nevertheless, there is currently a lack of IPA prediction models that also incorporate microbiological features.

Recently, clinical metagenomics has been widely used in diagnosing infectious diseases, particularly for the non-targeted diagnosis of specific, unknown, or mixed pathogens.^13,14 Its distinct advantage lies in unbiased sampling, which enables the broad identification of known and unexpected pathogens and even the discovery of new organisms, thus providing a comprehensive overview of pathogens in a given sample.^15,16 Therefore, this study aimed to use clinical metagenomics methods to identify microorganisms associated with IPA. Additionally, it sought to construct an IPA risk prediction model that combines clinical and microbiological characteristics and to evaluate its predictive performance.

Methods

Patient Enrollment

This retrospective cohort study was conducted in the adult ICUs of 17 medical centers in China. We collected clinical data of all patients admitted to the ICU between January 1, 2019, and June 30, 2023. Inclusion criteria were: 1. age ≥ 18 years; 2. diagnosed with community-acquired pneumonia (CAP); 3. undergoing invasive mechanical ventilation; and 4. receiving commercial metagenomic next-generation sequencing (mNGS) of bronchoalveolar lavage fluid. Exclusion criteria were: 1. loss to follow-up within 28 days of ICU admission; 2. diagnosis or clinical suspicion of IPA before ICU admission. This study received approval from the ethics committees of all participating hospitals. This study was conducted in accordance with the declaration of Helsinki. As a retrospective study, informed consent was waived.

Definitions and Data Collection

The diagnosis of CAP was based on the official clinical practice guidelines of the American Thoracic Society and the Infectious Diseases Society of America (IDSA).¹⁷ Specifically, it was defined as community onset with chest imaging showing new patchy infiltrates, lobar or segmental consolidation, ground-glass opacities or interstitial changes, with or without pleural effusion, and any pneumonia-related clinical manifestation. The diagnostic criteria for IPA, we use the 2021 EORTC/MSG criteria from IDSA, which are more suitable for ICU patients. This standard classifies IPA into proven and probable.¹⁸ Based on the IDAS criteria, our study considers clinical metagenomic testing showing positive for Aspergillus to be consistent with mycological evidence. Since biopsy was not feasible in the ICU, all patients were classified as probable cases. The 100 probable IPA cases in this study were diagnosed based on meeting at least one mycological evidence, one clinical feature, and one host factor. Immunosuppression was defined as (1) neutropenia (a neutrophil count < 0.5 × 10^9/L within 10 days of admission); (2) use of immunosuppressive drugs, including tacrolimus, cyclosporine, mycophenolate mofetil, or monoclonal antibodies (eg, rituximab) within 30 days before mNGS testing; and (3) history of acquired immunodeficiency syndrome, hematologic malignancies, or transplantation.¹⁹

Relevant data were independently collected from patients’ electronic medical records by an experienced team of clinicians. For the included patients, data obtained included gender, age, comorbidities, immunosuppressive status, laboratory test results, galactomannan test results, sputum fungal culture results, and clinical metagenomics results. Disease severity was assessed using the sequential organ failure assessment (SOFA) score at ICU admission, with organ dysfunction defined as a score of ≥ 2 points. All ICU centers collected cases according to unified standards. The clinical metagenomics laboratories were accredited by either the College of American Pathologists or the external quality assessment programs of the National Health Commission of China.^20,21

Model Construction

Factors believed to be associated with IPA were included in a multivariate logistic regression model based on previous literature and clinical expertise.¹¹ These variables were then screened using the forward selection method, and those with a p-value < 0.05 were considered independent risk factors for IPA and included in the analysis. Two risk prediction models were constructed based on the results of univariate analysis and multivariate logistic regression: one without microbiological characteristics (Model 1) and one with microbiological characteristics (Model 2). The models constructed by multivariate logistic regression were visualized using nomograms. Model performance was evaluated using data from the training cohort. Discriminatory ability of the models was assessed by calculating the area under the receiver operating characteristic curve (AUC). Model calibration was performed using the Hosmer–Lemeshow (HL) goodness-of-fit test, with a p-value > 0.05 indicating an acceptable fit. Calibration curves were plotted, and decision curves were used to evaluate the clinical benefit of the models. Predictive abilities of the two models were compared using net reclassification improvement (NRI). An NRI > 0 indicated improved predictive ability of the new model compared to the old model, NRI < 0 indicated a decline, and NRI = 0 indicated no significant difference. The models were internally validated using the validation cohort data through AUC, HL test, and decision curve analysis (DCA).

Statistical Analysis

Continuous data were first tested for normality. Normally distributed data were expressed as mean ± standard deviation () and compared using independent sample t-tests. Non-normally distributed data were expressed as median (interquartile range) [M (QL, QU)] and compared using the Wilcoxon rank-sum test. Categorical data were expressed as frequencies (percentages) and compared using the χ²-test, with the continuity correction χ²-test employed when expected values were < 5. All tests were two-sided, and a p value less than 0.05 was considered statistically significant. There were no missing data in this dataset. During the construction of the IPA risk prediction models, the “pROC” and “ggplot2” packages in R were used to plot receiver operating characteristic (ROC) curves and evaluate model discrimination. The “rmst” package and Bootstrap method, with 1000 repeated samples, were employed to plot calibration curves and test model fit. Decision curves were plotted using the “dcurves” and “rmda” packages to evaluate the clinical benefit of the models. The “nricens” package was applied to calculate NRI to compare the predictive abilities of the two models. Statistical analysis was performed using SPSS 23.0 software (SPSS Inc.) and R Statistics software 4.4.0.

Results

Basic Characteristics of Enrolled Patients

Out of 1897 patients screened, 898 met the inclusion criteria and were included in this study (Figure 1). According to the revised IDSA diagnostic criteria, 100 patients were diagnosed with probable IPA (100/898, 11%), of which 70 cases (70/100, 70%) exhibited Aspergillus positivity from clinical metagenomics, and 38 cases (38/100, 38%) showed Aspergillus positivity from sputum fungal culture. Among the 100 patients, 74 died within 28 days, resulting in a 28-day ICU mortality rate of 74% for the IPA group.

Figure 1 Flow chart.

Compared to the non-IPA group, patients in the IPA group had higher incidences of concurrent myocardial infarction (12.0% vs 4.9%, p = 0.004), CKD (26.0% vs 12.4%, p < 0.001), hematologic malignancies (13.0% vs 3.0%, p < 0.001), CTD (15.0% vs 3.8%, p < 0.001), and history of transplantation (15.0% vs 4.6%, p < 0.001). Additionally, the IPA group exhibited lower lymphocyte counts [0.46 (0.21–0.77) vs 0.54 (0.31–0.90), p = 0.014] and CRP levels [69.82 (31.54–152.21) vs 98.19 (43.02–171.35), p = 0.031]. Clinical metagenomics indicated that patients in the IPA group were more likely to develop co-infections with Pneumocystis spp. (23.0% vs 9.4%, p < 0.001), EBV (31.0% vs 15.7%, p < 0.001), and cytomegalovirus (33.0% vs 16.3%, p< 0.001), whereas patients in the control group were more likely to experience co-infections with Klebsiella spp. (16.0% vs 29.4%, p = 0.005). The IPA group also had significantly shorter hospital stays [15 (7–23) vs 18 (10–31), p < 0.001], shorter ICU stays [9 (6–15) vs 13 (8–22.25), p < 0.001], and a higher 28-day mortality rate (74.0% vs 47.0%, p < 0.001) compared to the non-IPA group (Table 1). In this study, the 898 included patients were randomly divided into a training cohort (n = 637) and a validation cohort (n = 261) in a 7:3 ratio. There were no notable differences between the two cohorts in terms of gender, age, comorbidities, immunosuppressive status, laboratory test results, SOFA scores, degree of organ dysfunction, duration of hospital stay, clinical metagenomics results, duration of ICU stay, and 28-day mortality rate (p > 0.05) (Table S1).

Table 1 Baseline Characteristics and Outcomes in Full Population

Comparison Between IPA Group and Control Group Based on Univariate Analysis

In the training cohort, patients diagnosed with probable IPA were designated as the IPA group (n = 67), while the remaining patients were designated as the control group (n = 570). Compared to the control group, the IPA group had higher incidences of CKD (28.4% vs 12.5%, p < 0.001), hematologic malignancies (13.4% vs 3.7%, p = 0.001), CTD (16.4% vs 3.9%, p < 0.001), and a history of transplantation (16.4% vs 5.1%, p = 0.001). Additionally, a higher proportion of patients in the IPA group were in an immunosuppressive state (64.2% vs 21.8%, p < 0.001), with lower CRP levels [68.04 (27.05–130.00) vs 105.00 (42.94–180.83), p = 0.011] (Table 2).

Table 2 Comparisons Between the IPA and Control Groups for Univariate Analysis

Model Construction

In this study, variables considered significant in previous literature were included in the multivariate logistic regression model for analysis. Five variables were found to be independent risk factors for IPA: age, immunosuppression, CKD, CTD, and liver failure (Table 3). Subsequently, based on the multivariate logistic regression analysis, an IPA risk prediction model was constructed based on clinical characteristics (Model 1). The regression system of each self-variable was used as the weight system, the logistic regression equation was established, and the final fitting risk prediction model was:

Table 3 Multivariate Logistic Regression Model-1

The nomogram based on this model is shown in Figure 2A.

Figure 2 (A). Nomogram prediction model-1; (B). Nomogram prediction model-2.

Clinical metagenomics analysis indicated that, compared to the control group, the IPA group was more likely to develop co-infections with Klebsiella spp., Pneumocystis spp., EBV, and cytomegalovirus (Table 4). A second risk prediction model incorporating microbial characteristics (Model 2) was then established. In Model 2, six factors were identified as independent risk factors for IPA, namely age, immunosuppression, CKD, CTD, liver failure, and cytomegalovirus positivity (Table 5). The regression system of each self-variable was used as the weight system, the logistic regression equation was established, and the final fitting risk prediction model was:

Table 4 Microbial Analysis of IPA and Control Groups in the Training Cohort

Table 5 Multivariate Logistic Regression Model-2

The nomogram based on this model is shown in Figure 2B.

Model Evaluation and Validation

ROC curves were plotted using training cohort data. Model 1 was observed to have an AUC of 0.782 (0.725–0.840), and Model 2 had an AUC of 0.791 (0.735–0.847). Both models demonstrated good discriminative ability, with overall AUC values higher than those of any individual parameter within the models (Figure 3A). In the validation cohort, the AUCs of Model 1 and Model 2 were 0.787 (0.703–0.871) and 0.792 (0.710–0.874), respectively, again exceeding the AUC values of any individual parameter (Figure 3B). Calibration curves based on the training cohort revealed that the mean absolute error (MAE) for Models 1 and 2 were 0.012 and 0.011, respectively, and closely matched the ideal curve. The HL test indicated p-values of 0.263 for Model 1 and 0.323 for Model 2, suggesting good model fit (Figure 4A and B). In the validation cohort, the MAEs for Models 1 and 2 were 0.017 and 0.02, respectively, with HL test p-values of 0.252 and 0.083 (Figure 4C and D).

Figure 3 (A). The ROC curves of Model 1 and Model 2 in the training cohort; (B). The ROC curves of Model 1 and Model 2 in the validation cohort.

Figure 4 (A). Calibration curve of model 1 in the training cohort; (B). Calibration curve of model 2 in the training cohort; (C). Calibration curve of model 1 in the validation cohort; (D). Calibration curve of model 2 in the validation cohort.

Sensitivity analysis of the models was conducted based on the Youden index. In the training cohort, Model 1 exhibited a sensitivity of 0.806 and a specificity of 0.698 at an optimal cut-off value of 0.083. In contrast, Model 2 had a sensitivity of 0.731 and a specificity of 0.749 at an optimal cut-off value of 0.103 (Table 6). The positive predictive values for Models 1 and 2 were 0.239 (0.183–0.295) and 0.255 (0.194–0.317), respectively, while their negative predictive values were 0.968 (0.951–0.985) and 0.960 (0.941–0.978), respectively. Comparison of the two models yielded an NRI = 0.035 > 0 (Table 6), indicating improved predictive capability of Model 2 over Model 1.

Table 6 Sensitivity Analysis of the Two Models

DCA suggested that in the validation cohort, Model 1 had a higher net benefit than the “All” and “None” lines between thresholds of 8% and 38%, whereas Model 2 showed a higher net benefit between thresholds of 8% and 50%. The net benefit area for Model 2 was larger than that for Model 1, indicating the superiority of the former. In the training cohort, Model 1 exhibited a higher net benefit than the “All” and “None” lines between thresholds of 5% and 37%, while Model 2 showed a higher net benefit than the “All” and “None” lines between thresholds of 8% and 35% (Figure 5).

Figure 5 (A). Decision curves for model 1 and model 2 in the training cohort; (B). Decision curves for model 1 and model 2 in the validation cohort.

Discussion

There is an increasing incidence of IPA, especially among critically ill hospitalized patients.²² Early identification and treatment of IPA are closely linked to reduced mortality rates. Although traditional culture methods are considered the gold standard for diagnosing IPA due to their accuracy in identifying strains, they are time-consuming and yield low positive rates, making them unsuitable for early clinical diagnosis.²³ Non-culture methods such as histopathological examination, while significant, cannot distinguish species and involve invasive sampling processes often limited by the patient’s condition, thus restricting their clinical application.²⁴ Therefore, it is necessary to develop clinical prediction models that can help predict the likelihood of IPA at an earlier stage. In 2020, Huang developed a predictive scoring system for influenza-associated aspergillosis (IAA) called Asper-PreSS.¹¹ In 2023, Massart developed a prediction model for IPA in patients with ventilator-associated pneumonia.⁸ However, to our knowledge, previous predictive models for IPA have only focused on the clinical characteristics of IPA. This study is the first to combine clinical characteristics and microbiota to construct a predictive model for IPA.

While numerous IPA prediction models based on clinical characteristics have been proposed, this study is the first to incorporate microbial characteristics alongside clinical features.

This study evaluated the incidence of IPA among 898 patients with CAP who underwent invasive mechanical ventilation. Using readily available variables from early ICU admission or pre-admission stages, 12 influencing factors were identified through univariate analysis of clinical characteristics, clinical practice, and previous literature. Subsequently, a risk prediction model based on clinical characteristics (Model 1) was constructed and validated using logistic regression analysis. The model comprised five predictive factors, namely age, immunosuppression, CKD, CTD, and liver failure, aligning with previous research findings.^8,11,25–27 An observational study revealed that patients with invasive Aspergillus infections typically presented with more underlying diseases and were often immunosuppressed.⁷

After establishing Model 1, it was evaluated and validated. The AUCs of the model in the training and validation cohorts were 0.782 (0.725–0.840) and 0.787 (0.703–0.871), respectively, indicating a high discriminative ability. Additionally, the p-values of the HL test for both cohorts were > 0.05, suggesting a good model fit. Sensitivity analysis of the model revealed a sensitivity of 80.6% and specificity of 69.8% at an optimal cut-off value of 0.083. Furthermore, DCA for the training and validation cohorts demonstrated that the model exhibited significant clinical benefits within a certain range.

Based on clinical characteristics, microbial features were further analyzed and compared between the IPA and non-IPA groups. It was observed that the IPA group was more likely to develop co-infections with Pneumocystis spp., EBV, and cytomegalovirus, while the control group was more likely to experience co-infection with Klebsiella spp. Subsequently, the microbial characteristics were combined with clinical features to construct a risk prediction model (Model 2). This model consisted of six predictive factors: age, immunosuppression, CKD, CTD, liver failure, and cytomegalovirus positivity, consistent with the findings of previous research.^8,11,25–28 IPA has been reported to be closely associated with cytomegalovirus, as a retrospective study has demonstrated that cytomegalovirus is an independent risk factor for IPA.²⁸ Following its establishment, the model was evaluated, validated, and compared against Model 1 to explore its clinical utility. The AUCs of Model 2 in the training and validation cohorts were 0.791 (0.735–0.847) and 0.792 (0.710–0.874), respectively, indicating a high discriminative ability in both cohorts. Although Model 2 had a higher AUC compared to Model 1, the p-value for the AUCs between the two models was 0.35, indicating no statistically significant difference. This suggests that the clinical benefit of Model 2 may be limited, possibly due to the insufficient sample size. Additionally, the p-values of the HL test for both the training and validation cohorts were > 0.05, indicating superior model fit and robustness. Compared to Model 1, Model 2 produced a better HL test result in the training cohort but performed worse in the validation cohort, likely due to the smaller sample size. Sensitivity analysis suggested that at an optimal cut-off value of 0.083, the sensitivity was 73.1% and the specificity was 74.9%. Compared to Model 1, the sensitivity of Model 2 decreased while the specificity increased. Furthermore, the DCA results in the training cohort showed that Model 2 exhibited better clinical utility than Model 1. The NRI was calculated to compare the predictive capabilities of the two models. In both the training and validation cohorts, the NRI was > 0, indicating that Model 2 had better predictive ability for events than Model 1.

This study has some distinct advantages. Firstly, to the authors’ knowledge, this is the first study to propose an IPA risk prediction model in the ICU that integrates microbial characteristics with clinical features, and the proposed model demonstrated high discriminative ability and appropriate calibration. Secondly, through clinical metagenomics, we have identified the specific microorganisms involved in the occurrence of IPA. In subsequent clinical practice, we can replace clinical metagenomics with PCR detection of these microorganisms. Thirdly, our statistical methods are quite comprehensive. Using univariate analysis and logistic multivariable regression methods to construct predictive models in the training cohort. This method is suitable for binary outcome variables. Subsequently, use a nomogram to visually present the constructed model. Using ROC, calibration curves, and decision curves to evaluate the model. In addition, we also validated the constructed model in the validation cohort using internal validation methods. Finally, a quantitative comparison of Model 1 and Model 2 was conducted. Making our research more reliable.

However, the study also has some limitations. First, this study is a retrospective study with some missing data. Second, clinical metagenomics only reported coexisting species, without providing absolute quantification. Third, this study did not undergo external validation, which may result in overfitting of the model. A well-designed prospective cohort can serve as prospective validation of this model²⁹。

Conclusion

This was the first study that combined microbiological features with clinical characteristics to construct a clinically applicable IPA risk prediction model. The model exhibited superior discriminative ability and calibration, as well as high sensitivity and specificity. These findings suggest that incorporating microbial characteristics can significantly improve the early identification and management of IPA in critically ill patients, potentially leading to timely and targeted therapeutic interventions and ultimately reducing mortality rates. Future studies should focus on external validation of this model and explore its applicability in different clinical settings to enhance its generalizability and reliability.