An Updated Systematic Review on Asthma Exacerbation Risk Prediction Models Between 2017 and 2023: Risk of Bias and Applicability

Introduction

In 2019, asthma was estimated to affect 262 million people worldwide by the Global Burden of Disease collaboration,¹ contributing to substantial economic burdens.^2–4 The milestones of asthma management are to optimize asthma symptom control and prevent the risk of adverse outcomes in particular asthma exacerbations.⁵ Asthma exacerbation is defined as acute or subacute episodes of progressively worsening symptoms, with severe exacerbations requiring the consecutive use of oral corticosteroids (OCS) and/or leading to emergency department (ED) visits and hospitalizations.⁶ The ongoing risk of exacerbation is a major source of asthma burden, especially in severe asthma patients. In 2016, 46.9% of asthmatics in the United States had at least one asthma exacerbation in the prior year.⁷ Asthma exacerbations are associated with accelerated lung function decline, contributing to worse long-term outcomes and quality of life.⁸ Accurate prediction of exacerbations in asthma patients could improve the efficiency of preventive intervention and/or treatment escalation, promote shared decision-making, and ultimately improve patient outcomes.⁹

To date, commonly used risk prediction tools for asthma exacerbations are limited in number. In 2018, Loymans et al reviewed and validated 12 asthma exacerbation risk prediction models that were published globally up to April 2017.¹⁰ Due to unsatisfactory prediction accuracy, none of these models were applicable for clinical use. In recent years, several new biomarkers such as blood eosinophil count (BEC) and nitric oxide were found to be predictive of risk of asthma exacerbations, and hence were included only in a few new asthma risk prediction models.^11,12 Meanwhile, advancements in big data analytics such as machine learning (ML) algorithms may further improve the accuracy of asthma risk prediction.¹³ In 2023, Xiong et al¹⁴ extended the review to ML-based asthma risk prediction models that were published up to 2021. However, their strict inclusion criteria which required externally validated models with clear description of ML methodology had led to the omission of several commonly used models, such as the Oxford Asthma Attack Risk Scale (ORACLE).¹⁵ To bridge this knowledge gap, this study aimed to update the systematic review of new prediction models for exacerbations in adult asthma patients between May 2017 and August 2023, and critically appraised the risk of prediction bias and model applicability.

Materials and Methods

Systematic Literature Review

We performed a systematic search in the Embase and PubMed databases for studies published between May 1^st 2017 and Aug 31^st 2023. The search criteria were summarized in Appendix file Table A1, and the search strategy was listed in Table A2.

Studies were independently selected by two reviewers based on their titles and abstracts. We included studies focused on adult patients (age ≥ 18 years) with asthma, aimed to develop one or more prognostic models to predict the risk of asthma exacerbations within a certain time frame, with a goal to identify patients at differential risk of asthma exacerbations in the future. Of note, we only included peer-reviewed studies that were published and/or translated into English. On the other hand, we excluded studies which were only focused on identifying and/or evaluating important predictors of asthma exacerbation but did not develop or update any risk prediction equations or composite risk scores. We also excluded studies which were focused on predicting the overall asthma impairment but not exact exacerbations, such as the studies that combined asthma control and exacerbation into a composite outcome.¹⁶ In addition, we also excluded studies that did not build any risk prediction tools.

Data Extraction

Two independent reviewers performed data extraction for study information (author’s name, publication year, and setting), characteristics of study population (number, age, sex, and inclusion criteria), study purpose and outcome, and model specifications (statistical analysis, model presentation, predictors, and performance).

Risk of Bias and Model Applicability

The potential risk of prediction bias of the included models was assessed using a well-established and commonly used checklist, namely the Prediction Risk of Bias Assessment Tool (PROBAST).¹⁷ This tool has been widely applied in systematic reviews of prediction model studies and has demonstrated reliability in assessing the development, validation, and updating of multivariable prediction models.¹⁸,¹⁹ Two parts constituted the tool. Part 1 consisted of 4 domains (Participants, Predictors, Outcome, and Analysis), with 20 signaling questions to assess model risk of bias (ROBs). Each component in Part 1 was graded as “yes”, “probably yes”, “no”, “probably no”, or “unclear” for the evaluation of model ROB. Part 2 evaluated model applicability in the domains of Participants, Predictors and Outcomes with 4 signaling questions. Each component in Part 2 was classified as “low”, “high” or “unclear” concerns for model applicability. And for both ROB and applicability, if there was at least one high domain, the overall evaluation result would be high. If there was at least one unclear domain and no high domain, the overall result would be unclear. Otherwise, the overall result would be low.

Results

Figure 1 shows the flowchart of study selection. From 415 searched records, we reviewed 27 relevant studies with full text, of which 17 were excluded. The most common reason for exclusion was that the outcome was not direct exacerbation measures, but rather a derived composite of asthma exacerbation and symptom control. A total of 10 studies were included in this review, which reported 41 prediction models.^20–29

Figure 1 Overview of systematic literation search.

Study Characteristics

Table 1 presents the study information and model specifications of the included studies, and a detailed summary of the included studies is presented in Appendix Table A3. Among the 10 studies, 8 (80%) studies^{20–22,24–26,28,29} developed the prediction models for use in clinical settings, while 2 (20%) studies^23,27 developed prediction models for home monitoring. For the derivation cohort, 3 (30%)^21,23,25 used derivation data from multiple countries, and 7 (70%) used single-country data (the USA,^20,24,27,28 Canada,²⁶ Sweden,²² and Korea²⁹). 6 (60%) studies^{20–22,26,28,29} built the model using electronic health records, 3(30%)^23,25,27 used clinical trial data, and 1 (10%) study²⁴ collected data through a longitudinal study design. Regarding derivation cohort size, great variation was observed, ranging from 298 to over 90,000 patients. Outcome definition was rather consistent, all of which included 1 or more of the following components: courses of systemic corticosteroids, outpatient visits, ED visits, and/or hospitalizations due to asthma exacerbation. In particular, 4 (40%) studies^20,23,25,27 adopted the American Thoracic Society/ European Respiratory Society (ATS/ERS) Task Force criteria,³⁰ to define severe exacerbation as a lung attack which required the use of systemic corticosteroids for at least 3 days, and/or an ED visit and/or hospitalization because of asthma. The time frame of the prediction ranged from 3 days to 12 months. Regarding predictor selection, 2 (20%) studies^25,29 included biomarkers in the prediction model, such as BEC and fractional exhaled nitric oxide (FeNO). Meanwhile, 6 (60%) studies^{21–23,27–29} did not report the final list of selected predictors, and all these studies applied ML algorithms that often utilized implicit knowledge representation and reasoning, such as decision trees, random forests, and extreme gradient boosting (XGBoost). For the remaining 4 studies, they used statistical approaches to derive the prediction equation.^20,24–26 Notably, all of these studies had described included predictors, with the number of predictors ranging between 4 and 24. Among these latter 4 studies, the most common predictors were indicators of health resource utilization, use of asthma medications, historical exacerbations, and healthcare visits. For the model performance, 8 (80%) studies^{20,21,23,24,26–29} reported area under the curve (AUC) as the indicator of discrimination, and only 1 of 10 (10%)²⁶ adopted the measurement of calibration.

Table 1 Overview of Identified Prediction Reports (n=10) and Models (n=41)

Risk of Bias and Applicability

For ROBs, Figure 2 shows the rating (high, low, unclear) in four domains (Participants, Predictors, Outcomes, and Analysis) of each study and Figure 3 presents the summary results. Detailed ratings for all 19 questions are presented in Table 2 and Figure A1. For model applicability, Figures 4 and 5 respectively present the individual study ratings and summarized results in three domains (Participants, Predictors, and Outcomes). Overall, 8 out of 10 studies were rated as high ROBs, while 5 out of 10 studies were classified as high concerns in applicability.

Table 2 PROBAST Signaling Questions for Model Development and Validation Analysis

Figure 2 Traffic light plot for risk of bias assessment.

Figure 3 Summary plot for risk of bias assessment.

Figure 4 Traffic plot for applicability assessment.

Figure 5 Summary plot for applicability assessment. Figures 2–5 were generated using the robvis tool developed by McGuinness and Higgins (2021).31

For the Participants domain, 6 studies (60%) were rated as having high ROBs. Using RWD sources such as retrospective cohorts and health administrative data without a predefined protocol as the derivation dataset was the main source of bias in 5 studies (50%). Concern regarding applicability in the Participants domain was rated as high level in 4 studies (40%), mainly because those studies derived their models from controlled trials which were conducted in restricted, experimental clinical environments.

9 out of 10 studies (90%) were considered to have low ROBs in the Predictors domain, because their definitions and measurements of the predictors were assessed as appropriate by the PROBAST tool. There was only 1 study (10%) rated as having unclear risk because it did not sufficiently describe the measurements of predictors. Meanwhile, 1 study (10%) was associated with high concern of model applicability in the Predictors domain mainly because of model complexity (e.g., derived from a large electronic medical records database with an extensive list of predictors); 2 studies (20%) were associated with unclear concern of model applicability because relevant information about predictors was not reported.

In the Outcomes domain, all studies reported low ROBs, which indicated suitable outcome determinations as most studies defined their outcomes using ATS/ERS criteria.³⁰ Concerns related to “outcomes” applicability were low given the consistent definition of asthma exacerbation using valid records of OCS use, ED, and/or inpatient encounters.

In the Analysis domain, 8 studies (80%) were classified as having high ROBs. All these high ROBs studies lacked calibration evaluation and appropriate internal validation. Meanwhile, 7 studies (70%) did not have detailed descriptions about how the missing values were handled or simply excluded cases with missing data, resulting high ROBs.

Subgroup Analysis

Based on the PROBAST result, we categorized the 10 included studies into four distinct subgroups.

Studies with both high ROBs and high concerns regarding applicability (n = 9 models from 4 studies)^22,25,27,29 reported the highest AUC performance among the four groups, with a median AUC of 0.84 (range: 0.83–0.85), predicting impending exacerbations within a time frame ranging from 5 days to 1 year. Their high ROBs were largely due to retrospective study designs and inadequate handling of missing data. Meanwhile, the high concern for applicability was primarily associated with the use of randomized control trials (RCTs) with small sample sizes or real-world single-center, local datasets, which restricted model generalizability.

The subgroup with high ROBs but low concerns for applicability (n = 6 models from 3 studies)^20,24,28 had generally lower predictive performance, with a median AUC of 0.72 (range: 0.67–0.74). Similarly, high ROBs were mainly due to retrospective study designs. However, overfitting was a distinct issue with this subgroup, which was mainly owing to the absence of internal validation or the use of simple holdout validation. Concerns regarding applicability were low, as these models were derived from data of mixed care settings and multiple medical centers.

Studies with unclear ROBs or applicability concerns (n = 14 models from 2 studies)^21,23 were associated with a lack of transparency. The predictive AUCs of models in this subgroup ranged from 0.60 to 0.85. Both studies employed neural networks, but did not disclose details on data preprocessing, feature selection criteria, selected predictors, and evaluation strategies.

On the other hand, Jiao 2022 (n = 12 models)²⁶ was the only study with both low ROBs and applicability concerns. These models achieved moderate AUCs, ranging between 0.69 and 0.72. The study retrieved health administrative data with low missingness from one Canadian province to predict asthma exacerbation risk in the upcoming year. Elastic-net regularized logistic regressions were performed. Essential predictors were mostly demographics and asthma medication use. Although external validity had yet to be confirmed, the large population size (N=109,536) and cross-validation method enhanced validity and generalizability for the targeted population – local patients in the province.

Discussion

Loymans reviewed 12 asthma risk prediction studies up to 2017 and concluded that none were suitable for immediate use in general practice.¹⁰ In 2023, Xiong et al reviewed 11 additional studies by 2022, focusing on ML-based models.¹⁴ Extending from the reviews by Loymans 2017¹⁰ and Xiong 2023,¹⁴ we conducted a systematic review and ROB assessment on asthma risk prediction models published between May 2017 and August 2023. A total of 10 new studies were identified from an initial search of 415 studies, encompassing 41 risk prediction models for asthma exacerbations. All studies focused on adult asthma patients without specifying disease severity. Our findings confirmed that the recent mainstream approach in asthma risk prediction was the integration of RWD and ML algorithms into modeling. Seven out of ten studies were based on RWD, with six studies relying on routinely collected electronic health records (EHRs)^{20–22,26,28,29} and one based on a longitudinal cohort study,²⁴ while the remaining studies used data from clinical trials.^23,25,27 Meanwhile, seven studies applied ML algorithms.^{21–23,26–29} However, these models faced significant challenges in clinical implementation due to generally high ROBs and concerns of low applicability.

In Loymans’ review,¹⁰ 4 out of the 12 studies used basic Classification and Regression Trees (CART) models. Although Xiong’s review¹⁴ focused exclusively on ML-based prediction models, the modeling methods were mainly limited to traditional tree-based ML algorithms like decision trees, random forests, and boosting. In our review of the 10 newest studies, more sophisticated and time-sensitive neural network models — such as long short-term memory (LSTM) and recurrent neural networks (RNNs) — have been employed, even to incorporate temporal dynamics into risk prediction.³² However, advancement in prediction algorithms did not guarantee improvement in predictive performance. For instance, Xiang and colleagues applied LSTM to predict asthma exacerbations, but model discrimination was moderate (maximum AUC = 0.70).²¹ In addition, these newer models included T2 inflammatory biomarkers in asthma risk prediction, such as BEC and FeNO, which were demonstrated to be strong predictors of asthma exacerbations.^12,33–35 In one study conducted by Lee et al, BEC had the highest feature importance in Shapley Additive exPlanations (SHAP), and the model achieved a maximum AUC of 0.85, the highest among 10 reviewed studies.²⁹ In Xiong’s review, two studies included new biomarkers such as volatile organic compounds and single nucleotide polymorphisms.^36,37 In our reviewed studies, however, no such new biomarkers were included. While new biomarkers, such as serum soluble ST2 levels, periostin, and dipeptidyl peptidase-4, can potentially enhance asthma endotyping and phenotyping,³⁸ their measurement requires advanced medical equipment, limiting their predictive use in routine clinical practice.

The feasibility and results of the assessment of ROBs and applicability varied significantly across the three reviews. Loymans 2017’s review did not sufficiently address ROB and applicability issues, rather, its conclusion on the limited clinical usefulness of those prior models was based on the general unsatisfactory predictive performance.¹⁰ Xiong 2023’s review applied the PROBAST tool, which showed that all included studies had high ROBs, with 11 out of 12 rated as having high concerns regarding applicability.¹⁴ Similar to Xiong’s findings, our review also revealed substantial ROB (80%) but a decrease in concerns of low applicability (from 92% to 50% of reviewed models).

Given the highest proportion of studies with high ROBs and high concerns for applicability in Xiong’s review (11/12)¹⁴ and our review (4/10), this subgroup warrants closer examination. Both reviews exhibited consistent issues in the Participants domain. Models derived from EHR data were susceptible to missing, incomplete, and poorly logged information, as well as human-induced biases in referral, admission, diagnosis, and prognosis.³⁹ Also, all studies included in this subgroup showed high ROBs in the Analysis domain. Multiple ML models employed a large number of variables from extensive databases, further limiting their practical use in typical clinical environments. The complexity inherent in more sophisticated ML models can obscure the understanding of their function to guide decision-making processes, thus diminishing their practical utility.⁴⁰ Furthermore, model validation was often insufficient, with many studies either overlooking validation or relying solely on hold-out validation. Additionally, the validation analysis mainly focused on discriminative performance, while neglecting calibration — the latter being more relevant to clinical practice.⁴¹ In the Outcome domain, Xiong’s review reported inconsistent outcome definitions, contributing to high ROBs and applicability concerns, while the newer studies of our review consistently applied standardized definitions of asthma exacerbation. Notably, Loymans et al reported that over 80% of included studies did not use ATS/ERS-related asthma exacerbation criteria, whereas Xiong et al found this proportion reduced to 34%, and our review identified no studies with such a concern, suggesting improvement in consistency and reliability in outcome reporting in asthma risk prediction over time.³⁰

A noteworthy trend towards improved practical utility emerged in our review. On one hand, simpler tools such as ORACLE²⁵ and AIRQ²⁴ showed promise for enhancing clinical applicability. The ORACLE scale relied on a minimal list of predictors encompassing symptoms, exacerbation history, BEC and FeNO, which were commonly collected and allowed hands-on calculation at a clinic.²⁵ Recent analyses have demonstrated that the ORACLE scale not only quantifies exacerbation risk but also captures biomarker-dependent treatment responses, highlighting its potential as a theranostic tool.¹⁵ However, the ORACLE scale is derived from RCTs, which requires external validation and refinement using individual patient data from large, well-characterized populations. Meanwhile, the AIRQ questionnaire, a validated 10-item asthma control questionnaire, has demonstrated comparable discriminative capacity compared to ML-based models in assessing the relative risk of exacerbations across scores in patients aged above 12 years.²⁴ Nevertheless, predicting score-specific relative risk may limit the clinical utility of the AIRQ tool. On the other hand, user-friendly home-monitoring tools show potential for empowering patient self-management and facilitating integration into clinical workflows. Two home-monitoring prediction tools were developed to enable patients to track exacerbation risk in real-time.^23,27 However, their reliance on high-cost sensors restricts broader clinical use. Further research should explore cost-effective solutions to increase their access.

Our study has several limitations. First, we were unable to perform external validation because it was difficult to access a suitable dataset that contained all relevant predictors across the studies. Instead, we applied the PROBAST tool to assess the generalizability of reviewed models. Second, the non-disclosure of a final list of predictors and detailed model specifications in several ML-based models further hindered our attempts at external validation. Third, the heterogeneity in modeling methodology and reporting across different reviewed studies precluded a direct quantitative synthesis. Additionally, the small sample sizes within certain subgroups may limit the generalizability of our findings, as subgroup-specific findings could be unstable. Lastly, our study selection was restricted to articles published in or translated into English, which may have narrowed the scope of our review and excluded valuable research published in other languages.

The current review conveys important clinical implications. First, we found that ML-based, data-driven models have limited potential for further improving the accuracy of asthma risk prediction without either incorporating new robust biomarkers or increasing model complexity, both of which may compromise their generalizability. Therefore, in the near future, the development of asthma risk prediction tools should prioritize enhancing their clinical applicability, such as developing user-friendly prediction tools or integrating the ML algorithms into EHR-embedded, automated clinical decision support systems. Second, improving model generalizability and standardizing risk prediction can facilitate the development of clinical guidelines for asthma precision medicine. To fulfill these, future studies should consider real-world datasets across diverse healthcare settings and regions, and address country-specific variability in clinical risk due to system-level factors, such as local healthcare practices and patient health-seeking behaviors. Last but not least, causal prediction analysis was rarely found, which should be considered in future studies to enhance knowledge translation, better support clinical decision-making and gain physician trust.⁴² Shifting the focus toward these areas will strengthen the clinical relevance and utility of asthma risk prediction models, thus extending their impact on improved patient care.

Conclusion

Despite the increased use of big health data, biomarkers and advancement in ML algorithms, the prediction performance and clinical applicability of asthma risk prediction models developed between 2017 and 2023 remain inadequate. Most of these models exhibited high ROB, largely due to the reliance on RWD and the lack of comprehensive performance evaluations, particularly model calibration. Furthermore, model applicability was generally low, hindered by restricted study population, excessive complexity, and insufficient transparency regarding model specifications. To improve future asthma risk prediction models, efforts should focus on enhancing generalizability, practicality, and interpretability.