Effect of Machine Learning on Risk Stratification for Antiretroviral Treatment Failure in People Living with HIV

Introduction

To date, acquired immune deficiency syndrome (AIDS) is still a global public health problem, representing a substantial peril with no available cure.^1,2 According to the latest data released by the Joint United Nations Programme on HIV/AIDS (UNAIDS) in late 2022, there were 1.3 million new HIV infections and 0.63 million deaths attributed to AIDS-related illnesses worldwide.³ Fortunately, the introduction of antiretroviral therapy (ART) achieved virologic suppression, restored immune function, and played a significant role in reducing HIV-related morbidity and mortality among people living with HIV (PLWH).^4,5 By the end of 2022, an estimated 29.8 million of the 39 million (33.1 million–45.7 million) PLWH globally are receiving life-saving treatment, with almost three-quarters (71%) achieving viral load (VL) suppression.³ However, The ambitious strategic objective aims to have 95% of People Living With HIV (PLWH) diagnosed, 95% of those diagnosed receiving life-saving ART, and 95% of those on ART achieving viral load suppression by 2030, with the ultimate goal of improving human health and curbing the transmission of HIV.⁶ Therefore, it is necessary to focus on patients who have experienced virologic failure or are at high risk of developing it.

Failure of viral load control accelerates disease progression and fosters the occurrence of HIV-related co-infections among PLWH. Beyond its impact on patients, virologic failure exacerbates viral drug resistance, elevating the incidence of acquired drug resistance and complicating HIV transmission control efforts.⁷ Notably, virologic failure can be influenced by low-level viremia (LLV).^8,9 Chun et al⁹ revealed that patients with LLV had an increased likelihood of experiencing virological non-suppression and failure. According to a cohort research, individuals with persistent VL > 200 copies/mL were twice as likely to experience virologic failure in the future as those with undetectable VL.^8,10

In order to salvage the consequences of poor viral control in a timely manner, some countries use a lower VL threshold to define treatment failure. The US Department of Health and Human Services (DHHS) characterizes treatment failure as HIV VL ≥ 200 copies/mL. In China, after at least 24 weeks antiretroviral therapy, viral load > 200 copies/mL persistently, or virologic rebound occurs after the virus reached suppression regarded as virological failure.¹¹

Virologic failure has emerged as a prevalent public health concern among PLWH undergoing treatment. The main challenge of antiretroviral therapy today is to reduce virologic failure to achieve epidemic control and decrease the prevalence of drug resistance.¹² Considering that virologic failure would jeopardize long-term treatment success and contribute to the emergence of drug-resistant transmission, early identification of populations which have a high risk of virologic failure in PLWH is important. In addition, early identification of virologic failure in patients on first-line antiretroviral therapy plays a crucial role in selection and maintenance another efficacy second-line antiretroviral regimens. However, there is limited evidence on the incidence of virologic failure and its predictors. Up to date, several studies have discovered that many factors like, higher baseline VL,¹³ poor adherence,^14–17 lower baseline CD4⁺ T-cell counts,^18,19 lower CD4/CD8 ratio,¹³ Nevirapine (NVP)-based regimens,^20,21 lower BMI,²² and advanced WHO stage (III/IV)^23,24 would all impact virologic failure. Li et al²⁵ discovered that low-abundance pretreatment drug resistance mutations, ranging from 2% to 9%, were associated with an increased risk of virologic failure. Recent research has demonstrated a consistent link between LLV and virologic failure in individuals with baseline HIV RNA levels exceeding 10⁵ copies/mL, even when treated with integrase strand transfer inhibitors (INSTIs).¹³ A study by Avendaño-Ortiz et al identified a potential biomarker of HIV infection, sPD-L1, exhibiting a notable positive correlation with VL, particularly among PLWH experiencing virologic failure.²⁶ While these individual markers possess limited predictive accuracy and can be costly and challenging to obtain in primary healthcare settings, there remains a necessity to investigate more accessible prediction tools to aid clinicians in making informed clinical decisions.

In the follow-up of PLWH, assessments of liver enzymes, creatinine, blood glucose, lipids, and blood cell counts are routinely conducted as they are indicative of the organism’s functional status and potential adverse reactions to medications. These parameters are crucial in evaluating virological treatment efficacy. Following the previous approach of modeling multiple variables of the same type to form a new variable,²⁷ we integrated these indicators to create a new variable to assess the physical status of patients at the time of consultation and during treatment. This new variable will be incorporated into regression analyses in conjunction with diverse variables to formulate an innovative predictive model.

Therefore, the aim of this study was to develop a predictive model for early identification of patients at high risk of primary antiretroviral failure by analyzing the demographic characteristics and laboratory indicators of PLWH at baseline.

Methods

Study Design and Population

The retrospective study utilized data from the China Centers for Disease Prevention and Control’s information system and included PLWH receiving antiretroviral therapy (ART) from 2010 to 2022 at Wuhan Jinyintan Hospital. The participants were divided into training and internal validation sets, and an external validation set was obtained from Xiangyang and Jingzhou for further validation, which covered the years 2017 through 2022.

We defined the inclusion criteria as:¹ an exhaustive laboratory examination confirming HIV infection;² treatment with an ART regimen consisting of at least three medications;³ follow-up findings following six months of ART; and⁴ age more than fifteen. Patients with prior ART and incomplete follow-up information were excluded.

Data Collection and Definition

Data from demographic characteristics, clinical and laboratory indexes have been collected including age at ART start, body mass index (BMI, calculated as weight/height²), duration of ART delayed, adherence, side effects, sex, infection route, WHO clinical stage, opportunistic infections, specific clinical symptoms, ART regimens, CD4⁺ T cell counts, VL, along with some biochemical and blood markers.

Virologic failure defined as after at least 24 weeks antiretroviral therapy, viral load > 200 copies/mL persistently, or virologic rebound occurs after the virus reached suppression.²⁸ Treatment adherence was assessed by the proportion of days covered, which is the sum of days during follow-up. Poor adherence was defined as the proportion of days covered less than 80%.^29,30

Data Processing

The sample size for this study was determined based on the widely accepted guideline of including at least 10 valid observations per predictor in multivariate regression analysis. Multiple imputation by chained equations (MICE) was employed to handle missing data. A total of 5 imputation models were generated to account for the uncertainty in the missing values, and the imputed datasets were combined using standard methods to produce a final estimate. Missing values in the training set were analyzed using descriptive statistics, and the missingness mechanism was assessed using Little’s MCAR test. Sensitivity analyses were conducted to evaluate the impact of missing data on the robustness of our results (Supplementary Figure 1).

In this study, we developed the Follow-up Safety Index (FSI), an observational metric derived from multiple follow-up indicators to assess patient safety during routine visits. The FSI was constructed based on key clinical parameters, including white blood cell count (WBC), platelet count (PLT), hemoglobin (HB), serum creatinine (Scr), triglycerides (TG), total cholesterol (TC), fasting blood glucose (FBG), alanine aminotransferase (ALT), aspartate aminotransferase (AST), and total bilirubin (TBIL). Independent predictors were identified through stepwise logistic regression, ensuring that the FSI effectively integrates variables with significant influence on patient outcomes.

Statistical Analysis

The variables in both the training and validation sets were presented as either frequencies (percentages) or means ± standard deviations. For comparing continuous variables between groups, the Mann–Whitney U-test was used. Categorical variables were compared using the Wilcoxon rank-sum test, Fisher’s exact test, or the chi-square test as appropriate. The Least Absolute Shrinkage and Selection Operator (LASSO) method was employed to identify the most relevant predictors.³¹ The six key predictors identified were then included in subsequent analyses, and a nomogram was developed based on multivariate logistic regression. To enhance clinical applicability, the nomogram was transformed into a scoring system with integer scores assigned to each variable. The optimal cutoff values for each variable and the final total score were determined using the Youden Index.

After developing the clinical prediction model based on the six significant factors, its validity was evaluated by assessing both discrimination and calibration. Discriminatory performance was measured using receiver operating characteristic (ROC) curve analysis across three distinct datasets. Calibration was assessed using a calibration curve. Additionally, decision curve analysis (DCA) was conducted to evaluate the clinical utility of the model.

Data analysis was performed using R-Studio (version 4.2.0). A two-sided P-value < 0.05 was considered statistically significant.

Results

According to the above inclusion and exclusion criteria, 7417 hIV/AIDS patients were diagnosed and followed up in Wuhan Jinyintan Hospital (5189 participants in training set and 2228 in internal validation set), and 2462 hIV/AIDS patients were diagnosed and followed up in Xiangyang and Jingzhou cities in Hubei (external validation set). The characteristics of the training and validation sets are listed in Table 1. Detailed comparison of the baseline characteristic between the three groups were showed in the Supplementary Table 1. Significant differences were observed between the training and external validation cohorts (p < 0.05), including older age, higher proportion of females, predominance of heterosexual transmission, and lower WHO clinical stage in the external validation cohort. Notably, such differences were not observed between the training and internal validation cohorts. In addition, sensitivity analyses comparing baseline characteristics between included and excluded individuals revealed no significant differences (p > 0.05) (Supplementary Table 2). The proportion of missing data was low (5%-10% across variables) in the train set, and Little’s MCAR test supported the assumption of completely random missingness (p = 0.687) (Supplementary Figure 1). Due to its high missing rate (69.6%), CD8 was excluded from further analysis. Further analyses showed no significant differences between pre- and post-imputation distributions, confirming the robustness of the imputation process (Supplementary Table 3).

Table 1 Comparison of the Baseline Characteristic Between the Three Groups

In this study, virologic failures occurred in 6.92% (684/9879) of all PLWH, including 5.18% (296/5189) in the training set, 5.92% (132/2228) in the internal validation set, and 11.49% (283/2462) in the external validation set. Based on the safety indicator variables, FSI was established. After stepwise regression analysis, TBIL was found to be statistically significant (p < 0.05). However, its clinical relevance to virological treatment failure remains uncertain, as existing literature does not support a robust association. Additionally, TBIL had a high proportion of missing data (over 20%). Although sensitivity analyses showed no significant differences between pre- and post-imputation distributions (p > 0.05), the substantial missingness may still introduce potential bias into the results. Given these limitations, TBIL was excluded from the final model to ensure that all predictors were both statistically and clinically meaningful. Finally, six laboratory indicators were derived to form the FSI. The AUC of FSI was 0.711 (95% confidence interval (CI) 0.679–0.743), and the multivariate analysis of these six variables was described in Table 2. Calculating by the β-value, the FSI index was represented as 0.139*WBC + 0.002*PLT + 0.019*HB + 0.478*TC - 0.120*FBG - 0.005*ALT. Subsequently, an optimal threshold value of 5.10 was extrapolated from the ROC analysis, thereby bifurcating FSI into binary variables. This dichotomization process stratified the FSI into high-risk (lower FSI) and low-risk (higher FSI) groups.

Table 2 Multivariate Regression Analysis for Constructing FSI in Training Set

To derive model variables that are easy to use and calculate clinically, all variables were transformed into dichotomous variables based on the best cutoff values from the ROC analysis (Supplementary Table 4). In order to have selected the most efficient parameters for a highly accurate prediction model, we added all variables besides FSI and other safety indicators in LASSO regression analysis (Figure 1). Then, six predictors (delayed ART, adherence, discontinue ART, side effects, CD4⁺ T cell counts and FSI) were selected as independent risk factors associated with virologic failure (Figure 2). These six predictors were utilized to create a nomogram model (Figure 3). The combined performance of these six factors was subsequently assessed comprehensively through ROC analysis, AUC was 0.807 (95% CI 0.800–0.834). Furthermore, the nomogram model also exhibited good calibration and clinical utility.

Figure 1 Variables selection using the LASSO regression analysis. (A) A coefficient profile plot was created against the log (lambda) sequence; (B) Tuning parameter (lambda) selection of deviance in the LASSO regression based on the minimum criteria (left dotted line) and the 1-SE criteria (right dotted line).

Figure 2 Forest plots of significant risk factor for virologic failure in multivariate regression analyses.

Figure 3 Nomogram model for predicting virologic failure in people with HIV/AIDS and its evaluation in training set. (A) Nomogram model for identifying virologic failure. (B) ROC curve of the predictive nomogram. (C) Calibration curve of the predictive nomogram. (D) DCA of the predictive nomogram.

To enhance its applicability in clinical practice, we transformed the predictive model into a user-friendly scoring system by assigning integer points to each predictor based on their respective β coefficients. The scoring system is as follows: delayed ART initiation (6 points), adherence (7 points), discontinuation of ART (6 points), side effects (9 points), CD4+ T cell counts (10 points), and FSI (9 points). The detailed scoring criteria are presented in Table 3. With this scoring system, the VF risk score for HIV individuals = 6*delayed ART (No=0, Yes=1) + 7*adherence (No=0, Yes=1) + 6*discontinue ART (No=0, Yes=1) + 9*side effects (No=0, Yes=1) −10*CD4⁺ T cell counts (<324=0, ≥324=1) - 9*FSI (<5.1=0, ≥5.1=1). The optimal threshold for the scoring system is 15.5 points, as calculated by the Youden Index. When the total score exceeded 15.5 points, PLWH were more likely to develop virologic failure, whereas when the total score was lower than 15.5 points, PLWH were less likely to develop virologic failure.

Table 3 A Scoring System Developed from a Nomogram in the Train Set

Subsequently, we comprehensively evaluated the performance of the scoring model through ROC analysis. The AUCs were 0.807 (95% CI 0.780–0.834), 0.784 (95% CI 0.744–0.823), and 0.745 (95% CI 0.713–0.777) for the training, internal validation, and external validation sets, respectively (Figure 4A–C). There were no statistically significant discrepancies in the precise matches between the anticipated and actual values, as indicated by the calibration curves for the training and validation sets (Figure 4D–F). DCA also shown the value and use of nomograms in clinical decision-making. In both the training and validation sets, the clinical applications utilizing the created nomograms produced good clinical advantages spanning the threshold probability range of 0.1 to 0.8, as seen in Figure 4G and H. Figure 5 presents the confusion matrix of prediction model in each set. Across the three sets, the model demonstrated consistent performance, with recall (sensitivity) values of 77.3%, 75.0%, and 61.8%, and accuracy values of 70.5%, 70.7%, and 73.8%, respectively. Additionally, the specificity achieved was 70.1%, 70.5%, and 75.4% in the three sets, indicating the model’s ability to correctly identify both positive and negative cases. These results highlight the model’s robustness and generalizability across different datasets.

Figure 4 Discriminations, calibration and evaluation of the scoring system for the prediction of virologic failure in training set, internal and external validation sets. ROC curves of the scoring system in the training set (A), internal validation set (B) and external validation set (C). Calibration curves of the scoring system in the training set (D), internal validation set (E) and external validation set (F). DCA of the scoring system in training set (G), internal validation set (H) and external validation set (I).

Figure 5 Confusion matrix of the score risk group in training set (A), internal validation set (B) and external validation set (C).

Discussion

This study constructed a clinical predictive model and created a unique scoring system to facilitate practical application. The nomogram graphically demonstrates how the model can be applied. In Figure 5 we can see a patient has not discontinue ART, has no side effects, with a poor adherence, delayed ART<2.2m, FSI≥5.1 and CD4⁺ T-cell counts<324 cells/μL, holding the risk of developing virologic failure of 8.6%.

Our research has revealed that delaying the initiation of ART can have a detrimental impact on the effectiveness of antiviral treatment, leading to virologic failure, particularly among individuals who commence ART more than 2.2 months after diagnosis. A study conducted in sub-Saharan Africa highlighted a significant incidence of virologic failure attributed to late detection and diagnosis of HIV, as well as postponed commencement of ART.³² Rapid-initiation ART has been shown to shorten the interval from HIV diagnosis to achieving virologic suppression, enhance ART adherence, and improve treatment retention in PLWH. In addition, accelerated ART initiation could reduce overall mortality in the medium for a long period of time, recognized as a key strategy for controlling the HIV epidemic.^33,34 Therefore, WHO advocates for the immediate initiation of ART upon confirmation of HIV infection.³⁵

A study conducted in Siberia revealed that inadequate adherence emerged as the primary contributing factor to the failure of antiretroviral therapy in PLWH receiving first-line ART.¹⁶ It has been established that patients face risks of developing drug resistance and compromised immunity when their adherence rates fall below 95%.²¹ In cases of poor adherence, a significant decline in CD4⁺ T-cell counts occurs, resulting in immune system breakdown.^36,37 Once the number of CD4⁺ T-cells decreases, it creates the right conditions for viral replication and leads to virologic failure. In addition, Palladino et al³⁸ found that patients who interrupt treatment experience viral replication rebound leading to immune failure and to some extent also inducing mutations in drug resistance genes. This is in line with the conclusions we reached.

Several previous studies have demonstrated an inverse correlation between viral replication and CD4⁺ T-cell counts. Lower CD4⁺ T-cell counts are linked to heightened susceptibility to virologic failure, thereby escalating the likelihood and frequency of opportunistic infections, consequently fostering disease advancement and elevated mortality rates.^39,40 Santos et al⁴¹ highlighted that among treatment-naïve individuals, those with CD4⁺ T-cell counts below 100 cells/μL were more prone to experiencing virologic failure when utilizing second-generation INSTIs.

A study exploring factors associated with virologic failure in western Kenya found that patients who experienced drug side effects during treatment were at risk of virologic failure.⁴² Similarly, our investigation detected a correlation between adverse reactions and virologic failure, potentially attributable to the notion that intolerable and enduring side effects indirectly impede patients’ adherence to prescribed medications.

Moreover, we assessed the safety of the drug by establishing a new variable, FSI. Consisting of six variables, WBC, PLT, HB, FBG, TC, and ALT, FSI covers the effects of the drug on blood cells and biochemical indicators. The magnitude of the FSI score, determined by a specific formula, directly correlates with the level of safety conferred by the drug.

Our findings have significant implications for clinical practice. By enabling the early identification of high-risk patients, our model supports personalized interventions, such as tailored adherence support and regimen adjustments, which can reduce the likelihood of virological failure. Additionally, the model’s simplicity and reliance on routine clinical data make it a practical tool for resource allocation and patient management. The robust performance of the model in both internal and external validation sets underscores its generalizability across diverse HIV populations. However, we acknowledge that excluding individuals with incomplete follow-up data may introduce attrition bias, particularly if those lost to follow-up were more likely to experience treatment failure. Nevertheless, to enhance the applicability and generalizability of our model and scoring system, further validation across multiple institutions and larger datasets is imperative. Future studies should prioritize minimizing loss to follow-up and consider advanced statistical methods, such as inverse probability weighting, to address potential bias.

In summary, the new scoring system has demonstrated strong calibration and outstanding diagnostic performance in identifying the likelihood of virologic failure in PLWH. It is based on six easily available clinical characteristics. We advise HIV sentinel hospitals to make extensive use of this innovative scoring approach in order to promptly and efficiently identify patients who are at high risk of virologic failure.