Development and Validation of an Interpretable Machine Learning Model for Prediction of the Risk of Clinically Ineffective Reperfusion in Patients Following Thrombectomy for Ischemic Stroke

Introduction

Acute ischemic stroke is caused by the obstruction of cerebral blood vessels, leading to interrupted blood supply, hypoxia, and necrosis of brain tissue.^1–3 It manifests as acute focal neurological deficits and is associated with high rates of disability and mortality.⁴ Reperfusion therapy is considered the first-line treatment for acute ischemic stroke caused by large-vessel occlusion.⁵ Timely restoration of blood flow to the brain is associated with better clinical outcomes.^6,7 However, several studies^8,9 have shown that patients with acute stroke undergoing thrombectomy often achieve recanalization (modified treatment in cerebral infarction (mTICI) grade 2b-3) but fail to attain functional independence (modified Rankin scale (mRS) score 0–2). Clinically ineffective reperfusion (CIR) is regarded as a treatment failure despite successful vascular recanalization.¹⁰ Therefore, there is an urgent need to enhance understanding of CIR-associated risk factors and develop effective predictive methods to facilitate personalized diagnostic, treatment, and follow-up strategies. Prognostic models based on machine learning methods often outperform traditional statistical models in terms of predictive accuracy.¹¹ In this study, we retrospectively analyzed the risk factors for CIR in patients undergoing thrombectomy at our cerebrovascular center. We developed the first machine learning model for CIR prediction, which demonstrated high accuracy. The SHAP algorithm was used to interpret the model and a web-based calculator was developed to facilitate its clinical application.

Methods

Study Design and Population

We collected data from patients with acute ischemic stroke who underwent thrombectomy at Shanghai Fourth People’s Hospital between December 2021 and June 2024. The inclusion criteria were: (1) underwent thrombectomy; (2) had complete imaging and clinical data; and (3) had complete clinical follow-up information. The exclusion criteria were: (1) pre-stroke mRS > 2 due to physical disability, prior history of cancer, or surgery; (2) lack of complete imaging data; and (3) loss to follow-up after surgery. The study was conducted with Institutional Review Board approval from the Shanghai Fourth Hospital affiliated with Tongji University in China (No: 2024218–001). As this study was retrospective, informed consent was waived. The guidelines outlined in the Declaration of Helsinki were followed in this study.

Assessment of Study Outcomes

CIR was defined as a postoperative mTICI grade ≥ 2b and an mRS score > 2 at three months post-discharge. Imaging results were independently assessed by two senior neurosurgeons, both blinded to the predictor information. In cases of disagreement, a third blinded neurosurgeon was consulted, and the final decision was reached by consensus.

Variables Selection

Initially, univariate analysis was conducted to identify statistically significant variables. 22 statistically significant parameters were identified. Using the Lasso algorithm, we further reduced these to 16 parameters, followed by mRMR selection to obtain the top 10 statistically significant variables, which were subsequently included in the machine learning models.

Model Development, Validation and Performance Evaluation

We optimized the hyperparameters of random forest (RF), support vector machine (SVM), k-nearest neighbor (KNN), and decision tree (DT) machine learning models using a grid search with five-fold cross-validation. The hyperparameters were selected based on the highest area under the curve (AUC) score obtained on the validation set. The ranges of hyperparameters tested for each model are as follows: RF: n_estimators (50, 100, 200), max_depth (5, 10, 20), min_samples_split (2, 5, 10), min_samples_leaf (1, 2, 4), max_features (“sqrt”, “log2”); SVM: C (0.1, 1, 10, 100), kernel (“linear”, “rbf”, “poly”, “sigmoid”), gamma (“scale”, “auto”, 0.01, 0.1, 1), degree (for “poly”, 2, 3, 4); KNN: n_neighbors (3, 5, 7, 10), weights (“uniform”, “distance”), metric (“euclidean”, “manhattan”, “minkowski”); DT: criterion (“gini”, “entropy”), max_depth (5, 10, 20), min_samples_split (2, 5, 10), min_samples_leaf (1, 2, 4). Th four machine learning models—RF, SVM, DT, and KNN—were applied to predict CIR risk in patients with ischemic stroke undergoing thrombectomy, and all data were randomly split into training (70%) and testing (30%) sets. The training set was used to construct the models and the test set was used to evaluate them. Model performance was assessed using multiple metrics, including the receiver operating characteristic (ROC) curve, negative predictive value (NPV), positive predictive value (PPV), sensitivity, specificity, accuracy, and F1 score, and the best predictive model was selected based on its performance in the testing dataset. A heat map was used to analyze the performance of the optimal model.

Model Explanation

The SHAP algorithm addresses interpretability challenges posed by the black-box nature of machine learning models. It ranks the importance of input parameters and explains the model’s final outputs. By quantifying the contribution of each parameter across individual samples, the SHAP algorithm enhances model interpretability and transparency.

Developing an Online CIR Risk-Prediction Platform

To facilitate clinical applications, we integrated the optimal model into a Shiny-based web platform to develop an online calculator. The calculator provides the predicted probability of CIR by inputting the relevant parameters of the patients undergoing thrombectomy.

Results

Baseline Clinical Information

A total of 177 patients who underwent thrombectomy were included in this study, 60 of whom were classified as having CIR 3 months postoperatively. The clinical characteristics of the patients are summarized in Table 1. Significant differences were observed in several clinical features between the clinically effective reperfusion and clinically ineffective reperfusion groups. Patients in the CIR group, compared to the non-CIR group, tended to be older (78 vs 71 y) and were more likely to have a history of previous ischemic stroke, diabetes mellitus, atrial fibrillation, and hyperlipidemia. They also had higher NIHSS scores (15 vs 10), systolic blood pressure (145 vs 135 mm Hg), diastolic blood pressure (DBP: 87 vs 78 mm Hg), serum glucose levels (7.95 vs 6.4 mmol/L), endovascular thrombectomy (EVT) attempts (5 vs 1), infarct core volumes (27.5 vs 11 mL), hypoperfusion intensity ratios (0.5 vs 0.4), door-to-magnetic resonance imaging (MRI) times (48 vs 42 min), and door-to-puncture times (48.5 vs 36 min). Additionally, collateral grades and clot burden scores (CBS) were significantly lower in the CIR group than in the clinically effective reperfusion group, and patients with cardioembolic stroke or those not receiving postoperative anticoagulant or antiplatelet therapy were more likely to experience CIR (p < 0.05).

Table 1 Baseline Characteristics of Patients with Effective and Ineffective Reperfusion

Selection of Predictor Variables

We employed the mRMR and LASSO algorithms for feature selection (Figure 1). Based on the univariate analysis, 22 statistically significant parameters were identified. Using the LASSO algorithm, we further reduced these to 16 parameters, followed by mRMR selection to obtain the top 10 statistically significant variables, which were subsequently included in the machine learning models (RF, DT, SVM, and KNN). The importance score for each variable was calculated and visualized using a bar plot.

Figure 1 Feature selection. (a) CV,10-fold cross-validation. (b) Lasso, Least absolute shrinkage and selection operator. (c) mRMR, max-relevance and min-redundancy.

Figure 2 Performance of machine learning model. (a)RF-model (auc=0.96,95% CI=0.91–1.0). (b)DT-model (auc=0.93,95% CI=0.86–1.0). (c)SVM-model (auc=0.89,95% CI=0.80–0.98). (d)KNN-model (auc=0.92,95% CI=0.83–1.0).

Model Construction and Performance Evaluation

We utilized 10-fold internal cross-validation to construct four machine learning models. The RF model demonstrated the best performance, achieving an AUC of 0.96 (95% CI: 0.91–1.0), accuracy of 0.93, and specificity of 0.97 in the test dataset (Figure 2 and Table 2).

Table 2 AUC, Sensitivity, Specificity, Accuracy, F1 Score, PPV and NPV of mL Model

Figure 3 Variable heatmap of the random forest (RF)model predicting clinically ineffective reperfusion (CIR) in patients with thrombectomy. Each row represents a variable and each column reflects a sample, with colors indicating the variables’ value. The legend represents the different values ranges of each variable, including CIR(1), EVT times, Diabetes mellitus(1), PIS (Previous ischemic stroke,1), PIIBGA (Preoperative infarction involving the basal ganglia area,1), CBS/BATMAN (Clot burden score/Basilar Artery on Computed Tomography Angiography score,1:0–4;2:5–7,3:8–10), Cause of stroke(Toast) (1:Large-Artery Atheorothrombosis, LAA;2:Cardioembolism;3:Undetermined Etiology), Collateral grade(ASITN)(1:0–1, 2:2, 3:3–4), MLS (Maximal midline shift), RF predicted value. Color gradients indicate the value ranges of variables, and categorical variables are represented by different colors for categories.

Heatmap Analysis of RF Model Variables

A heatmap was generated to illustrate the performance of the RF model in predicting CIR risk after thrombectomy (Figure 3). Different colors represented the actual values of clinical characteristics across patients. The heatmap analysis showed that the RF model accurately distinguished between the CIR and clinically effective reperfusion groups, demonstrating its excellent classification ability and potential utility in managing high-risk thrombectomy patients.

Model Explanation

The SHAP algorithm was used to evaluate the importance of each variable in the RF model and its contribution to the final prediction (Figure 4). The bar plot emphasized that EVT attempts, with the highest SHAP value (Figure 4b), were the most significant risk factors for CIR. Collateral grade (Figure 4b) was identified as the most critical imaging parameter for predicting CIR.

Figure 4 RF model explanation using the Shapley Additive exPlanation (SHAP) method. (a) SHAP dot plot: Points farther from the centerline indicate higher SHAP values, suggesting an increased probability of CIR. Each point represents the SHAP value of a parameter, with red indicating higher values and blue indicating lower values. The clustering of points reflects parameter density. (b) SHAP bar plot: This bar plot illustrates the contribution of each parameter to the model and ranks them in descending order of importance.

An Online CIR Risk-Prediction Platform

As shown in Figure 5, an online CIR risk-prediction platform was developed based on the optimal model to predict the probability of CIR three months after discharge in patients following thrombectomy. By inputting the ten required parameters, the web calculator (at the following link: https://ineffectivereperfusion.shinyapps.io/calculate/.automatically) outputs the predicted probability of CIR.

Figure 5 An online CIR risk-prediction platform was developed to predict the probability of CIR three months after discharge in patients following thrombectomy.

Discussion

Timely thrombectomy and complete vascular recanalization are often associated with improved prognosis in patients with acute ischemic stroke; however, complete vascular recanalization does not always guarantee favorable clinical outcomes, as numerous factors influence prognosis.^12–16 This study retrospectively analyzed the clinical parameters of patients with acute ischemic stroke treated at our cerebrovascular center, identified the risk factors for clinical CIR, and developed the first machine learning model for CIR prediction, aiding in personalized treatment planning for future patients.

This study identified 22 risk factors for CIR, with the number of EVT attempts being the most critical. Following acute ischemic stroke, a higher number of thrombectomy attempts during standard procedures increases the likelihood of CIR, a finding corroborated by other studies.¹⁷ Procedure durations exceeding 60 min and more than three thrombectomy attempts are significant contributors to CIR, emphasizing the importance of achieving first-pass recanalization.^18,19 Our study demonstrated that CIR is associated with infarction location, with preoperative infarction involving the basal ganglia identified as a significant risk factor. Patients with ischemic stroke who did not have preoperative infarction in the basal ganglia area were more likely to achieve favorable postoperative outcomes. Previous studies^20,21 have shown that core infarct volume is associated with prognosis, a finding supported by our results. Larger core infarcts were associated with worse prognosis; however, the presence of a large infarct alone was not predictive of outcomes, possibly due to the study’s threshold of 80 mL for defining large infarcts. Our findings do not support the conclusion that larger core infarcts are unrelated to prognosis. This study incorporated four temporal parameters, of which three (door-to-MRI time, door-to-puncture time, and puncture-to-recanalization time) demonstrated statistically significant associations with patient outcomes. Our findings indicate that shorter time to diagnostic imaging, earlier surgical intervention, and more rapid vascular recanalization are consistently associated with a more favorable prognosis. Indeed, when patients experience acute cerebral infarction, neurons immediately begin to undergo cell death due to ischemia and hypoxia.²² However, timely vascular recanalization effectively salvages endangered brain tissue within the ischemic penumbra and significantly mitigates the degree of neurological functional impairment, thereby improving long-term patient outcomes.²³

Previous studies have reported that even with successful recanalization, a subset of patients with acute ischemic stroke caused by distal medium vessel occlusion (DMVO) may experience poor outcomes or mortality.^24,25 In this study, among 106 patients with middle cerebral artery (MCA) occlusion, 32 (30.2%) exhibited clinically ineffective reperfusion (CIR), a proportion higher than that reported in the literature.

Consistent with prior findings,^26,27 our results demonstrated that endovascular thrombectomy yielded better outcomes in patients with low baseline NIHSS scores compared to those with higher scores. Notably, the CIR group had significantly higher NIHSS scores (mean ± SD: XX ± XX) than the clinically effective reperfusion group (XX ± XX, P < 0.05), suggesting that the severity of initial neurological deficits may influence reperfusion efficacy.

Previous studies have successfully developed and validated early mortality prediction models for patients with acute ischemic stroke undergoing EVT, demonstrating the significance of machine learning algorithms in predicting outcomes.²⁸ These findings align with our hypothesis that precise predictive tools can effectively optimize stroke treatment strategies and significantly improve patient survival rates.

Based on 10 CIR-related parameters, we developed four machine learning models, with the RF model achieving the highest AUC, accuracy, and specificity. Random forest^29,30 is an ensemble learning algorithm that constructs multiple decision trees and aggregates their votes to complete classification tasks. Each tree is trained on different subsets of data (via bootstrap sampling) and random feature subsets, reducing overfitting and improving generalization. The RF algorithm captures nonlinear relationships within data, making it well-suited for handling complex datasets. The DT model³¹ also performed well (AUC: 0.93, accuracy: 93%), but its lower specificity (91%) suggests a slightly higher false positive rate, likely due to its tendency to overfit. The SVM model³² had the lowest AUC (0.89) and accuracy (85%), potentially owing to its sensitivity to hyperparameter tuning and small sample size. The KNN model³³ achieved moderate performance (AUC: 0.92), but its lower specificity (86%) suggests that it may be more affected by high-dimensional data and noise. RF emerged as the most suitable model for CIR prediction, balancing sensitivity (0.84) and specificity (0.97) while minimizing false positives and false negatives. The ensemble nature of RF makes it more robust than DT, whereas its automatic feature selection enhances generalization, addressing the limitations observed in SVM and KNN.

To enhance the model interpretability, we employed the SHAP algorithm.^34,35 SHAP, based on game theory, calculates the marginal contributions of features to prediction outcomes, ensuring consistent and locally accurate explanations. It quantifies feature importance and interactions, revealing the decision-making logic of the predictive model. Finally, we developed an online CIR risk prediction platform using the RF model. By entering ten key parameters, clinicians can obtain the predicted probability of the 3-month CIR for patients undergoing EVT. This can assist in risk stratification, prognosis estimation, and treatment discussions with patients and their families. We recognize that there may be potential barriers to implementation, including clinical workflow integration, for which the calculator needs to be seamlessly embedded into electronic health records for real-time access, and data privacy and security, to ensure compliance with healthcare data protection regulations.

Our study yielded valuable findings, but it has several limitations. First, our study may be subject to bias from unmeasured confounders because not all relevant clinical variables were included in the analysis. Factors such as comorbidities, prior stroke history, and detailed neuroimaging biomarkers might have influenced the predictive model, but were not fully accounted for. Future studies should incorporate a more comprehensive set of variables to enhance the model accuracy. Second, owing to the relatively small sample size, our model may be prone to overfitting, meaning it might perform well on the training data but generalize poorly to independent datasets. Cross-validation techniques have been applied to mitigate this problem. Finally, the limited sample size may have affected the statistical power, making some subgroup comparisons less reliable. Future studies with larger multicenter datasets are required to validate the model’s performance across different patient populations.

Conclusion

This study identified the risk factors associated with CIR, highlighting the number of EVT attempts as the most critical determinant. We developed the first machine-learning model to accurately predict CIR. Furthermore, a CIR risk-prediction platform was built based on the RF model, enhancing the clinical utility and value.