Prediction of Distant Metastasis of Renal Cell Carcinoma Based on Interpretable Machine Learning: A Multicenter Retrospective Study

Introduction

Renal cell carcinoma (RCC) is the most prevalent pathological subtype of kidney neoplasm and the 6^th most common genitourinary malignancy in males, trailing only prostate and bladder cancer.¹ Simultaneously, it is the deadliest malignant urological tumour, accounting for around 2% of all cancer fatalities globally.² RCC consists of heterogeneous tumor cells, with the most common histological subtypes being clear cell renal cell carcinoma (ccRCC), papillary renal cell carcinoma (pRCC), and chromophobe renal cell carcinoma (crRCC).^3,4 The prognosis of RCC patients with various histological subtypes varies, and it has been demonstrated that patients with ccRCC have a poorer prognosis than patients with other histological subtypes (pRCC and crRCC) even after radical nephrectomy.^5,6

RCC with distant metastasis is prevalent, especially in individuals with high-grade, microvascular invasion, and pathological sarcomatoid, papillary, or chromophobe renal cell carcinoma. Regional lymph nodes and the lungs are the most prevalent probable sites of metastasis for RCC, followed by the brain, bones, ovaries, and soft tissues.^7–13 The distant metastasis was highly associated with the primary tumor’s histological grade, T staging, gene expression, and functional features.¹⁴ According to prior studies, individuals with distant metastasis had a worse prognosis than those who did not.¹⁵ The 5-year survival rates of RCC in TNM stages I and II were as high as 81% and 74%, respectively, while the 5-year survival rates of RCC in stages III and IV were only 53% and 10%. The emergence of distant metastasis is the primary cause of the poor survival rate.^16–18Additionally, it has been demonstrated that individuals with metastatic RCC may benefit from cancer-targeted surgical surgery for a better prognosis.^19,20 As a result, it is vital to identify the clinicopathological risk factors that promote RCC distant metastasis.

In recent years, there have been some clinical diagnostic tools built using sophisticated machine learning algorithms. Compared to traditional statistical regression, machine learning can explore larger and multi-dimensional data and better describe complicated correlations between risk factors and outcomes.^21,22 N. P. Singh et al created a machine learning method to discover biomarkers and built classifiers to differentiate between early and late stages of PRCC based on gene expression profiles.²³ S. T. Chen et al created a machine learning-based pathomics signature (MLPS) for patients with ccRCC, which can be used as a novel prognostic marker.²⁴ A machine learning method was created by H. Kim et al to forecast the likelihood of RCC recrudescence between 5 and 10 years after surgery.²⁵ However, a promising machine learning-based model has not been introduced for prediction of distant metastases in RCC patients. Hence, considering the poor prognosis of patients with metastatic RCC (mRCC), we sought to develop a machine learning model to predict the risk of distant metastases in RCC patients, and visualize the clinical features and their interactions that affect distant metastases in RCC patients using Shapley Additive exPlanations framework, which has potential clinical utility.

This study was reported according to the recommendations of the Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD) statement.^26,27

Methods

Study Population and Data Source

The study cohort was extracted from the Surveillance, Epidemiology, and End Results (SEER) database from the National Cancer Institute. Basic demographic information, clinical characteristics, and survival status of patients are all publicly available through the SEER database (https://seer.cancer.gov/, ID: 15899-Nov2020). The inclusion criteria are as follows: patients pathologically diagnosed with histologically confirmed RCC (site code 64, 65, morphology code 8310, 8260, 8312, 8317, 8255, 8318) between Jan 1, 2004, and Dec 31, 2015; the exclusion criteria are as follows: (1) unknown surgery type; (2) unknown tumor size; (3) unknown T stage; (4) survival time <1 month; (5) unknown N stage; (6) bilateral renal masses. Data of RCC patients meeting the same criteria from 2016 to 2018 from the SEER database were extracted for temporal external validation. Due to the tumor staging of patients in 2016–2017 was mainly based on the 7th edition of the American Joint Committee on Cancer (AJCC) TNM Staging Manual, while the tumor staging of patients in 2018 was based on the 8th edition of AJCC TNM Staging Manual, the temporal validation set was divided into two sets: set 1 (2016–2017) and set 2 (2018). Since the data is public and anonymous and open access policy of the SEER database, ethical review and patients’ informed consent were waived.

The primary outcome of this study was distant metastasis in RCC patients. Distant metastasis of RCC refers to the presence of metastatic lesions in other sites besides primary renal lesions.

Data Extraction

Demographic and clinical information including age at diagnosis, gender, race, laterality, T stage, N stage, histological type, Fuhrman grade, surgery, chemotherapy, radiotherapy, tumor size, Lymph Node Surgery, and marital status were extracted from the SEER database. The patient’s race includes black, white, and other races (American Indian/AK Native, Asian/Pacific Islander). Fuhrman grades I, II, III, and IV represent well-differentiated, moderately differentiated, poorly differentiated, and undifferentiated, respectively. The surgical methods are divided into four groups according to the SEER Kidney Surgery Codes 2018: non-surgical group (code 0), local excision (code 10–27), partial nephrectomy (PN, code 30), and radical nephrectomy (RN, code 40–80).

Model Development and Assessment

The study cohort was split into a training cohort (70%) and an internal validation cohort (30%) using stratified random sampling. First, this study compared the performance of 15 machine learning algorithms with default hyperparameters to screen the best algorithms for further prediction models. The screening was carried out with the PyCaret package (version 2.3.3), an open-source and low-code ML library in Python. Due to the low proportion of positive case (8.84%), we applied several adaptive sampling methods, including Synthetic Minority Oversampling Technique (SMOTE), Random Oversampling, Tomek links, and Random Undersampling, to handle the class imbalance problem and improve the model performance. Bayes optimization method with ten-fold cross-validation was applied to choose the best hyperparameters setting. The hyperparameter tuning process was implemented using the bayes_opt python package (version 1.2.0). We mainly used the area under the receiver operating characteristic curve (AUC) to assess the discrimination of prediction models. The accuracy, sensitivity, specificity, and F1_score were also calculated to comprehensively assess models. Meanwhile, we also used the area under the precision–recall curve (AUPRC), which can provide a robust metric for unbalanced datasets. Prediction performance was also measured by calibration curve, which evaluates how well-predicted probability matches the observed probability. Performance was assessed in different sub-groups stratified by age (20–40, 40–60, >60), gender (male, female), and ethnicity (Black, White, other). Bootstrapping was used to calculate 95% confidence intervals (CIs) of performance metrics.

To alleviate the black-box nature of ML and help clinicians grasp the results offered by models, Shapley Additive exPlanations (SHAP) was applied to interpret the prediction results of the model using python shap package (version 0.39.0).

Statistical Analysis

In this study, the baseline characteristics of RCC patients were compared between the groups with distant metastasis and those without it. The chi-square test was used to compare categorical variables that were presented as frequency and percentages. Continuous variables were reported as median with the first quartile (Q1) and the third quartile (Q3) and compared using Mann–Whitney U-test. A two-sided P <0.05 was judged statistically significant. Statistical analyses were conducted with an open-source Scipy python package (version 1.7.1).

Results

Baseline Characteristics of the Participants

The flowchart for the cohort study was shown in Figure 1. A total of 121,433 patients were involved with histologically confirmed RCC from SEER database between Jan 1, 2004, and Dec 31, 2015. These patients were randomly divided into a training set (n = 85003) and a validation set (n = 36430). The detail demographic clinicopathological information of the patients was shown in Table 1. Of these patients, the mean age was 63 years, most patients were White (82.09% 99684 of 121433), fewer patients were Black and other race (11.49% and 6.42%). A total of 77,213 (63.58%) were male, 74498 (61.35%) were married, 70748 (58.26%) with the histopathological type of renal clear cell carcinoma, 116146 (95.65%) were N0, 84,389 (69.49%) were T1 stage, 109364 (90.06%) were LN. Sur and the average tumor diameter were 41.0mm. Most patients underwent surgery, 67132 (55.28%) patients underwent radical nephrectomy, 114341 (94.16%) patients did not receive chemotherapy, and 117812 (97.02%) patients did not receive radiotherapy. There is no significant difference was found in the characteristics of patients between the training set and the validation set. Characteristics of patients in the external validation sets were shown in Tables S1 and S2.

Table 1 Demographics and Clinical Characteristics for the Training Set and Validation Set

Figure 1 Study design flowchart of specific patient screening process.

Model Performance

These patients were randomly divided into a training set (n = 85003) and an internal validation set (n = 36430). The results of 15 algorithms were shown in Table S3. The Light Gradient Boosting Machine (LightGBM) was selected to further develop the prediction models according to accuracy and AUC. Owing to the extreme class imbalance that exists in our dataset, we selected random undersampling to address the class imbalance problem (Table S4). After the process of hyperparameter tuning, the LightGBM model showed great performance (AUC: 0.955, 95% CI: 0.951–0.959) based on the independent internal validation set. The ROC curve of the LightGBM model was shown in Figure 2A. The accuracy, sensitivity, specificity, and F1_score were listed in Table 2. The areas under the precision-recall curve (AUPRCs) are more informative and was 0.768 (Figure 2B). As shown in Figure 2C, the calibration plot showed that the lightGBM presented excellent goodness of fit.

Table 2 Internal Validation and External Validation Performance for the Prediction of Distant Metastasis in Patients with RCC

Figure 2 The performance of LightGBM model. The ROC curves of the model in three validation sets (A). The precision–recall curve of the model (B). Calibration curve of the model (C).

Furthermore, we validated model performance in different age, gender, and ethnic sub-groups. Patients younger than 60 had a marginally higher AUC than those older than 60 (Figure 3A), with AUCs of 0.961, 0.963, and 0.949, respectively. The AUCs of the distant metastasis were 0.954 and 0.956 for men and women group, respectively (Figure 3B). The AUCs of the distant metastasis were 0.956, 0.946 and 0.956 for white, black and other race, respectively (Figure 3C).

Figure 3 The ROC curve for different sub-groups. The predictive performance of the LightGBM model for different age sub-cohorts (A), genders (B), and races (C) in the independent validation set.

Temporal External Validation

In the temporal validation sets 1 and 2, the incidence of distant metastasis is 10.1% and 8.7%, respectively. The ROC curves for the external validation sets 1 and 2 were showed in Figure 2A. The discrimination for predicting distant metastasis was consistently high in external validation set 1 (AUC: 0.963, 95% CI: 0.960–0.967) and external validation set 2 (AUC: 0.961, 0.954–0.966). The accuracy, sensitivity, specificity, and F1_score analyses were shown in Table 2.

Model Interpretation

We provided insights into the model’s prediction outcome at the global and individual level using the Tree-Explainer class imported from the SHAP package based on the independent internal validation set. As shown in Figure 4, chemotherapy, surgery, tumor size, T stage, and radiation were the top 5 ranked variables that were correlated with distant metastasis among patients with RCC, followed by grade, histologic type, N stage, age, LN. Sur, race, sex, laterality, and marital, positive and negative shapley values indicate the positive and negative impact on the prediction outcome.

Figure 4 SHAP summary plot for the 14 clinical features contributing to our ML model’s prediction for distant metastasis (A). The distribution of the impacts of each feature on the model output (B). The colors represent the feature values for numeric features: red for larger values and blue for smaller ones. The line is made of individual dots representing each RCC patient, and the thickness of the line is determined by the number of examples at a given value. A negative SHAP value (extending to the left) indicates a reduced probability, while a positive one (extending to the right) indicates an increased probability. LN sur, Lymph Node Surgery.

Four samples were demonstrated as the SHAP force plot in Figure 5. For example, in Figure 5A, this is a 63-year-old man with a chemotherapy history and 87mm of tumor size. Factors identified by the model that increased the risk of distant metastasis were chemotherapy = yes, surgery = no, N = N1. The elevated risk is offset by the radiation = no. The ML model’s predicted outcome was the occurrence of distant metastasis, and the actual outcome was also the occurrence of distant metastasis (True Positive).

Figure 5 SHAP force plot for four samples from the internal validation set. The bars in red and blue represent risk factors and protective factors, respectively; longer bars represent greater feature importance. (A) The model correctly predicted the presence of distant metastasis, indicating a True Positive. (B) The model failed to predict the presence of distant metastasis, indicating a False Negative. (C) The model correctly predicted the absence of distant metastasis, indicating a True Negative. (D) The model incorrectly predicted the presence of distant metastasis, indicating a False Positive.

In Figure 5B, this is an 83-year-old man with a tumor size of 46mm and no history of chemotherapy and radiation therapy. The ML model counteracts the increased risk of the patient because the patient had not received chemotherapy and radiotherapy, the tumor size was 46mm, and the surgical procedure was a radical nephrectomy. The ML model incorrectly predicted the outcome as free from distant metastasis for this patient, whereas the actual outcome was distant metastasis (False Negative).

In Figure 5C, this is a 74-year-old woman with a tumor size of 31mm and no history of chemotherapy and radiation therapy. Factors decreasing the risk included tumor size = 31.0, chemotherapy = no, T = T1a, radiation = no, and N = N0. The predicted outcome of the ML model was that the patient had non-distant metastases, while the actual outcome was also non-distant metastases (True Negative).

In Figure 5D, this is a 41-year-old unmarried man with a tumor size of 60mm and no history of chemotherapy and radiation therapy. The patient’s risk for distant metastasis increased because of larger tumor size, higher tumor grade, and histological subtypes (ccRCC). Factors offsetting the risk were no chemotherapy, no radiation, and undergoing a radical nephrectomy. The outcome predicted by the ML model was the patient’s distant metastasis, while the actual outcome was non-distant metastasis (False Positive).

Variable Interactions

The interactions between variables can be analyzed by SHAP dependence plot. The top five variables were shown in Figure S1. For example, the size of the tumor was shown to affect the probability of distant metastasis in RCC patients differently for patients who underwent a type of surgery. With radical nephrectomy and partial nephrectomy, there is a more pronounced trend that smaller tumor size at discharge contributed to a lower risk of distant metastasis, while larger tumor size contributed to an increased risk of distant metastasis (SHAP values from −1.5 to +0.5–1.5). With local excision and no surgery, larger tumor size overall is observed, and the positive trend is more remarkable. This may highlight the importance of considering a variable such as a tumor size in a more complete clinical setting, such as one that includes tumor size, chemotherapy, type of surgery, grade, histologic type, and clinical reasoning (e.g, Patients with smaller tumors who have undergone radical nephrectomy and partial nephrectomy are not likely to be discharged with a significant risk of distant metastasis (Figure S1C). In addition, a similar finding is observed in Figure S1B for distant metastasis prediction, although clinical reasoning is unlikely to work compared to more purely physiologic phenomena: larger tumor overall are observed in patients who underwent local excision, and the relationship between tumor size and distant metastasis is not observed to be as linear for local excision (For certain types of surgery, high and low SHAP values were observed relatively consistently concerning tumor size.).

Discussion

RCC is a harmful malignant tumor of urinary system. The occurrence of distant metastatic disease is a routine marker of its advanced stage, which usually suggests that the prognosis of RCC patients is unfavorable.^19,28 A prior study discovered that up to 18%–30% of RCC patients had systemic metastases at the time of their first diagnosis.²⁹ It is well established that RCC patients with distant metastasis highly influence the prognosis of patients.³⁰ Previous studies have demonstrated that radionic features for kidney cancer could be utilised as prognostic or predictive markers.

Meanwhile, it has a good ability to predict histological subtype, pathological grade, and treatment effects.^31,32 U. Capitanio et al²⁸ found that computed tomography is still the current the first-line approach for detecting probable metastatic disease due to its high staging accuracy.

However, CT examination still has limitations, especially in evaluating the status of lymph nodes.²¹ Consequently, it is crucial for timely as well as accurate prediction of the risk of distant metastasis to potentially guide an optimal treatment decision.

Although conventional statistical analyses such as multivariate logistic regression analyses can predict the probability of RCC distant metastasis, their ability to handle large-scale and multi-dimensional data is limited. In this study, we established a machine learning prediction model to predict the risk of RCC distant metastasis by selecting 14 variables that significantly affect distant metastasis based on data collected from the SEER database. The LightGBM model showed good performance, indicating that this model may have a beneficial impact on decision-making in clinical practice. In addition, the AUC of the model varied from 0.949 to 0.963 for models with distant metastasis at different ages and varied from 0.955 to 0.956 at different race group. The model showed comparable performance between female and male group, indicating the robustness of the present model. We found that chemotherapy, surgery, tumor size, T stage, radiation, grade, histologic type, and N stage were significant features for predicting distant metastasis in RCC patients through SHapley Additive exPlanations framework.

The immune score system, functioning as an assessment approach, has demonstrated significant prognostic value in the treatment of renal cell carcinoma, particularly in forecasting survival and potential adjuvant treatment responses.³³ A high immune score is conspicuously associated with a poor prognosis, which might facilitate clinicians in more effectively evaluating treatment options and prognosis. Based on these discoveries, further research and validation are requisite for the effective integration of immune scoring into the treatment and management of RCC patients. Additionally, by incorporating immune score classification into the classic TNM staging system, we are capable of developing more intensive follow-up protocols for high-risk patients. If we can conduct a more in-depth analysis of the most relevant clinical parameters for each patient to transform it into an effective clinical tool for personalized assessment, it can be employed to predict the incidence of distant metastasis, potentially enhancing the survival outcomes of RCC patients.

Compared with previously nomogram model for distant metastasis prediction in RCC patients, the lightGBM model in our study showed better performance in discrimination and calibration.³⁴ In addition, the good generalization of our model has been proved by the temporal external validation in our study.

With the rapid development of machine learning, predictive tools that can be used to analyze and infer relationships between clinical characteristics and patient outcomes have spawned a wide range of clinical applications.^35,36 Given the weaknesses of traditional statistical methods in handling large data containing highly correlated covariates, we employed an advanced machine learning algorithm called light gradient boosting machine (lightGBM) to develop a prediction model and identify the important clinical features for predicting the occurrence of distant metastasis in RCC patients. LightGBM is an ensemble algorithm developed by Microsoft, which provides an efficient operation for the gradient boosting algorithm. The LightGBM algorithm is characterized by lower memory usage, higher speed and efficiency, and more compatibility with large datasets than other boosting algorithms.³⁷ In addition, compared to other ML methods, LightGBM achieved state-of-The-art performance, especially in structured data.³⁸ ROC and AUC are popular and powerful measure for evaluating the performance of machine learning models with binary classifiers. However, special caution is needed when working with class unbalanced data. Precision–recall curve has been suggested as substitutes for ROC but are used infrequently. Previous analysis of the literature found that most studies of algorithms using unbalanced datasets used ROC as their primary method of performance evaluation.³⁹ This study showed that PR curve represented the sensitivity to unbalanced datasets with clear visual signals and allowed for accurate and intuitive explanations of actual model performance.

Recently, machine learning has emerged as a promising tool for identifying adverse outcomes that may be missed by clinicians. However, the nature of machine learning, black box, has become a major obstacle to its widespread application in clinical practice. Interpretability can be defined as the degree to which clinicians can understand the mechanism for the machine learning model’s prediction.^3,25 In order to illustrate how these clinical features influence the distant metastasis and made the model “explainable”, we applied SHAP framework to show whether each feature contributed positively or negatively to the patient’s outcome. In this study, our analysis provides more direct conclusions and a way to visualize the relationships that produce the predicted results. We found that chemotherapy, surgery, tumor size, T stage, radiation, grade, histologic type, and N stage, were significant features for predicting distant metastasis. Especially, we found that poorly differentiated and undifferentiated patients had a greater risk of distant metastasis than well-differentiated patients, which supports previously reported findings that RCC differentiation was an independent risk factor for distant metastasis in RCC survivors.⁴⁰

The SHAP plot demonstrated that surgery is the second important feature of distant metastasis for tumor metastasis. Patients who undergo surgery have a lower risk of distant metastases than those who do not. This may be the outcome of efficient surgical removal of the tumor, which lowers the possibility of tumour spread.³⁴ For distant metastasis of RCC, we discovered that tumour size is the third crucial factor. The chance of distant metastasis increases with tumour size. Our findings are consistent with earlier research.⁴¹

This study has various limitations that should be mentioned. First, our analysis was based on the SEER database and was not independently evaluated on a separate dataset or at our hospital. Second, because we used retrospective data from the SEER database, our study may have had some inherent bias. To validate our prediction model in the general population, prospective clinical studies with a larger sample size are necessary. Third, while our approach is a clear explanation of the full prediction process, it does not explain the effect of the variables throughout the entire range of potential values, instead highlighting which variables are critical. Fourth, we were unable to obtain some biomarkers from the SEER database, such as plasma osteopontin (OPN), miR-646, and epigenetic markers. To this end, a prospective study is planned. To summarise, as with all MLs that explore cause-and-effect correlations, this is a hypothesis-generating activity that requires strict validation, independent study on promising elements, and finally application at the discretion of patients and clinicians. We expect that putting greater focus on intelligence augmentation, decision support, and interpretability will result in a more sophisticated and effective adoption of ML as a supplementary tool in a comprehensive approach to patient care and research.

Conclusion

In this study, we applied a novel ML algorithm called LightGBM to predict the risk of distant metastasis and the model achieved great performance. SHapley Additive exPlanations framework was used to show the quantitative attributes linked to the probability of distant metastasis in RCC patients and visualize how these attributes continuously affect the distant metastasis. The presented prediction model has the potential to be used to support clinical decision-making and treatment strategies.