CECT-Based Radiomic Nomogram of Different Machine Learning Models for Differentiating Malignant and Benign Solid-Containing Renal Masses

Introduction

Renal masses (RMs) are biologically heterogenous and can be categorized as benign or malignant; based on their likelihood of aggressive malignancy, active surveillance and surgery are optional management methods.¹ Renal cell carcinoma (RCC) is common, and a surgical approach is often necessary; however, benign tumours or those with low malignant potential may not require treatment.^2–4 Preoperatively misclassified renal masses can lead to unnecessary surgery, and data suggest that there is a rapidly increasing trend of overtreatment.⁵ Refraining from unnecessary surgical interventions transcends mere considerations of cost-effectiveness from the perspective of patients.⁶

Accurately differentiating malignant from benign RMs before surgery is important for follow-up management. Imaging tools such as ultrasound, computed tomography (CT), and magnetic resonance imaging (MRI) are widely used in the diagnosis of RMs.^7–9 Although each modality has advantages and characteristics, distinguishing malignant from benign RMs via conventional imaging methods⁴ is still challenging. Nuclear medicine, such as single photon emission computed tomography/computed tomography (SPECT/CT) and positron emission tomography computed tomography (PET-CT) molecular imaging, showed great value in evaluating renal lesions, but there are still some drawbacks to a generalized use.¹⁰ Transmembrane protein mucin-1 (MUC1) might be utilized in clinical settings as a diagnostic marker for renal malignancy,¹¹ however, the guidance of MUC1 for clinical is still deficient. In particular, solid-containing RMs such as oncocytomas and lipid-poor angiomyolipomas (AMLs) can be differentiated from RCC.^4,6 Among the medical imaging modalities, contrast-enhanced CT (CECT) is currently the most commonly used method for evaluating RMs.⁴

Artificial intelligence (AI) has become a key technology for analysing multi-parametric data, with applications in medical imaging, including image acquisition, post-processing, and data mining and modelling.¹² Radiomics is a promising method that involves the development of algorithms and AI and provides high-throughput extraction of quantitative features.¹³ AI holds great potential to enhance cancer imaging interpretation, inference of tumor genotype and biology from radiographic phenotypes, outcome prediction, and evaluation of disease and treatment effects on adjacent organs. AI may also automate initial image interpretation and transform clinical workflows in radiographic detection, management decisions, and post-intervention monitoring.^14,15 Subsequent analysis of these features can facilitate clinical decisions, such as improving diagnostic, prognostic and predictive accuracy.¹³ Machine learning represents a subset of artificial intelligence that employs algorithms to iteratively learn from and make predictions on data, thereby identifying complex patterns that may elude human cognition.¹⁶ Researchers have employed deep learning or radiomics approaches to differentiate clear cell renal cell carcinoma (ccRCC) from other benign renal tumors.^17–19 Nevertheless, few studies have examined the use of different machine learning models to analyse CECT images and differentiate benign and malignant solids containing RMs.

Herein, different machine learning models were tested, and the best model was combined with a clinical model to explore its value in distinguishing benign and malignant solids containing RMs.

Methods

Patients

A total of 122 patients with renal neoplasms diagnosed between December 2019 and December 2022 were retrospectively enrolled in our study. The inclusion criteria for patients were as follows: (1) underwent abdominal three-phase CECT before surgery, including the arterial phase (20–25 s), venous phase (70–90 s) and delayed phase (180 s); (2) patients had no other treatment history, such as chemotherapy or radiotherapy, before CECT; (3) the postoperative pathology diagnosis and clinical data were complete; (4) solid components were found in the renal tumours. The exclusion criteria were as follows: (1) incomplete three-phase CECT; (2) patients without any solid components in renal tumours; (3) patients whose pathology and clinical data were incomplete. A total of 75 patients with malignant solid-containing RMs were included in the study, including: Renal clear cell carcinoma (n=58, 77.33%), urothelial carcinoma (n=3, 4.00%), chromophobe renal cell carcinoma (n=6, 8.00%), leiomyosarcoma (n=2, 2.67%), papillary renal cell carcinoma (n=3, 4.00%), multilocular cystic renal cell carcinoma (n=1, 1.33%), oncocytic papillary renal cell carcinoma (n=1, 1.33%), sarcomatous renal cell carcinoma (n=1, 1.33%). A total of 47 patients with benign solid-containing RMs were included in the study, including renal angiomyolipoma (n=40, 74.47%), renal oncocytoma (n=7, 14.89%), perivascular epithelioid cell tumor (n=3, 6.38%), complex renal cyst (n=2, 4.26%). The patients were randomly allocated to a training cohort and a test cohort at a 7:3 ratio, and the flowchart of patient selection is shown in Figure 1.

Figure 1 Flowchart for patient selection in our study.

This study was conducted in accordance with the declaration of Helsinki and approved by the Ethics Committee of the First People’s Hospital of Yunnan Province (No. KHLL2024-KY008). Due to the study’s methodology was retrospective analysis, which did not entail direct patient involvement, the Ethics Committee of the First People’s Hospital of Yunnan Province, China, sanctioned the study’s protocol and exempted the requirement for acquiring informed consent from participants. All patients’ data was anonymized or maintained with confidentiality.

Image Data Acquisition and Clinical Data Collection

Siemens drive CT scanners were used, and the scanning parameters were as follows: tube voltage, 120 kV; tube current, 150–200 mAs; matrix, 512 × 512; section thickness, 1.5 mm; and section interval, 3 mm. The scan area included at least the entire kidneys. After the nonenhanced CT scan, three-phase CECT was performed by injecting nonionic iodinated contrast medium (iohexol, 350 mg/I) at a dose of 1.5 mL/kg and a speed of 3.5 mL/s. CECT images were acquired in the arterial phase (20–25 s), venous phase (70–90 s) and delayed phase (180 s). All DICOM format images were transferred to the Neusoft picture archiving and communication systems (PACSs). Clinical data, including sex, age, smoking history, drinking history, history of hypertension, history of diabetes and tumour shape, were collected simultaneously. Two radiologists with 8 and 6 years of experience in abdominal CECT were assessed, and the images were agreed upon. The largest lesion with confirmed pathology was selected for subsequent analysis if multiple lesions were present.

Radiomics Analysis of CECT Images

Tumour Segmentation and Feature Extraction

A typical workflow of our study is shown in Figure 2. All DICOM-formatted CT images were saved as a window of 400 hU and a window of 40 hU. Two radiologists segmented the volume of interest (VOI) manually in the arterial phase, venous phase and delayed phase; they were all blinded to the patients’ histopathological results before segmentation. 3D Slicer software (version 5.3.0, https://www.slicer.org/) was used, and the tumours were delineated along margins layer-by-layer on cross-sectional CT images. VOIs were saved in NIFIT format for further analysis. Features were extracted by using PyRadiomics in Python (https://pyradiomics.readthedocs.io/en/2.1.2/).

Figure 2 Workflow of the key steps in our study.

In our study, we classified handcrafted features into three main categories: geometry, intensity, and texture, each targeting distinct aspects of tumor characteristics. Geometric features provide insights into the tumor’s shape and structure, revealing its morphological properties. Intensity features examine the brightness levels of voxels within the tumor, offering information on the distribution of intensities. Texture features, derived using methods such as the Gray Level Co-occurrence Matrix (GLCM), grey-level dependence matrix (GLDM), Gray Level Run Length Matrix (GLRLM), Gray Level Size Zone Matrix (GLSZM), and Neighboring Gray Tone Difference Matrix (NGTDM), capture the spatial patterns within the tumor. Together, these feature categories enable a comprehensive analysis of tumor characteristics, enhancing our understanding of its underlying properties. Finally, the following radiomic features were extracted: shape (n = 42), first order (n = 54), GLCM (n = 72), GLDM (n = 42), GLRLM (n = 48), GLSZM (n = 48), and NGTDM (n = 15). All radiomic features and their corresponding significance were derived in accordance with the guidelines of the Image Biomarkers Standardization Initiative (IBSI).²⁰

In order to evaluate the reproducibility of radiomic feature extraction, both intra- and interobserver consistency were examined using intra- and interclass correlation coefficients (ICC). A set of 40 images was randomly selected for region of interest (ROI) segmentation, performed by two radiologists, each with more than six years of experience in abdominal image interpretation. The interobserver reliability was evaluated by comparing the results of radiologists A and B, while the interobserver consistency was assessed by having radiologist A repeat the segmentation process. Intra- and interclass correlation coefficient (ICC) values > 0.8 was regarded as demonstrating strong agreement in feature extraction.

Radiomics Feature Selection Signature Model Development

Z-score normalization was used to reduce the bias of radiomic features. Features were selected by the following steps: (1) features with an ICC > 0.8 were identified; (2) Student’s t test or U-test was used to compare the extracted features of two groups, and features with P values < 0.05 were maintained for further analysis; and (3) Pearson’s correction coefficients were calculated to identify redundant and collinear features, and features with mutual correlation coefficients > 0.9 were excluded; and (4) the least absolute shrinkage and selection operator (LASSO) regression model with 10-fold cross-validation was used to select optimal features. LASSO regression is a powerful technique for variable selection that effectively reduces the dimensionality of the model by eliminating irrelevant predictors, while preserving model accuracy. This not only improves the interpretability of the model but also enhances its predictive performance.²¹

Radiomic models were built from the final selected features. Four mainstream machine learning algorithm training models were used to construct the best classifier model: support vector machine (SVM), k-nearest neighbour (kNN), light gradient boosting (LightGBM) and logistic regression (LR). SVM is a supervised machine learning model grounded in the Vapnik–Chervonenkis (VC) dimension, and it is proficient in making robust predictions.²² kNN is a classification method that relies on the proximity to nearby data points. The fundamental concept is rooted in statistical theory, where a point is classified by evaluating the weights of its neighbors and assigning it to the category with the highest weight.²³ LightGBM is a fast, scalable, and high-performance gradient boosting framework designed for distributed environments. It offers quick training, high accuracy, and excellent scalability, making it ideal for handling large-scale, high-dimensional datasets in tasks like regression, classification, and ranking.²⁴ LR model predicts the likelihood of a binary dependent variable based on other variables.²⁵ The area under the receiver operating characteristic (ROC) curve (AUC) was calculated to compare the diagnostic performance of the two methods. The sensitivity, specificity, accuracy, positive predictive value (PPV), and negative predictive value (NPV) were also calculated. The best radiomic model was then demonstrated.

Clinical and Combined Model Construction and Validation

Univariate analysis was used to determine the optimal clinical characteristics for differentiating between benign and malignant lesions in the training cohort. Parameters with a P value < 0.05 were chosen for the construction of clinical models. Odds ratios (ORs) with 95% confidence intervals (CIs) were calculated simultaneously for each characteristic. The best radiomic models provided by machine learning were used to construct a combined model with a clinical model. The diagnostic performance of the models was assessed with ROCs, including the area under the curve (AUC), accuracy, sensitivity, specificity, PPV, and NPV. Decision curve analysis (DCA) was used in the training and test cohorts to evaluate the models’ ability to predict tumour differentiation.

Statistical Analysis

All the statistical analyses were performed with Python software (version 3.4; http://www.python.org). Continuous variables are expressed as ± s and were compared by Student’s t test. Categorical variables are expressed as numbers (n%) and were compared by the chi-square test or Fisher’s exact test. A two-sided P value < 0.05 was considered to indicate statistical significance.

Results

Clinical Data

A total of 122 patients were enrolled in our study. The ratio of malignant renal masses in the training cohort and test cohort was not significantly different (50/85 vs 25/37, P=0.362). Patients were randomly divided into a training cohort (n=85) and a test cohort (n=37). A comparison of the clinical data between the two groups is shown in Table 1. All the clinical data were significantly different except for shape in the training cohort (P<0.05). Only age was significantly different in the test cohort (P<0.05). Univariate and multivariate analyses showed that sex, age and alcohol consumption were independent predictors of malignant masses (Table 2). These three features were further used to construct the clinical model.

Table 1 Clinical Characteristics of the Patients in the Training and Test Cohorts

Table 2 Univariate and Multivariate Logistic Regression Analyses for Selecting Clinical Features for Model Construction

Radiomic and Signature Models and Performance

A total of 322 features were extracted from the VOI in three phases. Thirteen features with nonzero coefficients were retained after feature downscaling and selection (Figure 3). The ROC curves of the four mainstream machine learning algorithm training models (LR, SVM, KNN and LightGBM) in the training cohort and test cohort are shown in Table 3 and Figure 4. The best radiomic model in the training cohort was SVM (AUC, 0.953 [95% CI, 0.8981–1.0000]; accuracy, 0.906; sensitivity, 0.900; specificity, 0.914; PPV, 0.937; NPV, 0.865).

Table 3 Diagnostic Performance of Different Models for Differentiating Benign and Malignant Solids Containing Renal Masses in the Training and Test Cohorts

Figure 3 Feature selection with the least absolute shrinkage and selection operator (LASSO) regression model. (A) The penalization parameter λ was selected with 10-fold cross-validation as the minimum criterion in the LASSO model. The vertical lines indicate the optimal value of the LASSO tuning parameter (λ=0.0339). (B) LASSO coefficient profile of the radiomic features. The vertical dashed lines indicate 13 radiomic features with nonzero coefficients. (C) The 13 features contributing to radiomic features based on CECT images weighted by standardized regression according to the LASSO model were used to differentiate malignant from benign RMs.

Figure 4 The receiver operating characteristic (ROC) curves of the LR (A), SVM (B), KNN (C), and LightGBM (D) models in the training and test cohorts. The SVM model (B) achieved the highest AUC in the training cohort (0.953, 95% CI: 0.898–1.000), while the LR model (A) demonstrated the highest AUC in the test cohort (0.887, 95% CI: 0.778–0.995).

Among the various models evaluated, the logistic regression (LR) model exhibited superior performance across all cohorts, with an AUC of 0.952 (95% CI, 0.9072–0.9968) for the training cohort and 0.887 (95% CI, 0.7782–0.9951) for the test cohort (accuracy, 0.784; sensitivity, 0.720; specificity, 0.917; PPV, 0.947; NPV, 0.611). This consistent high performance across different data subsets underscores the robustness and efficacy of the LR model in differentiating between the two classes in our study. Furthermore, the LR model’s ability to maintain a relatively high AUC in the test cohort, compared to other models, suggests its strong potential for generalizing to unseen data. This makes it a promising tool for similar binary classification tasks in future research. Based on these considerations, we selected the LR model for further analysis to ensure model stability.

Combined Model Construction and Performance

In the analysis of fused models combining clinical and radiomic modalities, we observe that the radiomics nomogram model demonstrates better performance in the test cohort compared to other models. The diagnostic performance and ROC curves of the three models are shown in Table 3 and Figure 5. The AUC improved to 0.973 (95% CI, 0.9388–1.0000) in the training cohort and 0.900 (95% CI, 0.7874–1.0000) in the test cohort (accuracy, 0.865; sensitivity, 0.880; specificity, 0.833; PPV, 0.917; NPV, 0.769), which was better than that of the clinical or radiomic signature models. The DCA and calibration curves for both the training and test cohorts of the three models are shown in Figure 6. The analysis of these curves indicates that our Combined model offers a notable advantage in terms of net benefit derived from predicted probabilities.

Figure 5 Receiver operating characteristic (ROC) curves for the clinical signature model, radiomic signature model, and radiomic nomogram model for predicting malignancy in the training cohort (A) and test cohort (B). The radiomics nomogram incorporating the clinical signature and radiomics signature (C).

Figure 6 Decision curve analysis (DCA) and calibration curves for three models for differentiating benign and malignant solids containing renal masses in the training (A and B) and test cohorts (C and D). The graphs show that the combined nomogram has the greatest net benefit for both datasets.

Discussion

Solid-containing RMs, including partial fat-containing and completely solid RMs, were ultimately selected for this study. RMs with complete fat composition were not included in our study because they are easy to detect and distinguish via conventional imaging methods.²⁶ However, some malignant tumours, such as liposarcoma, Wilms tumours and renal cell carcinoma (RCC), may contain fat, which leads to challenges in preoperative diagnosis.²⁶ Additionally, AMLs with poor fat weight, epithelial cysts or haemorrhagic complications have different appearances, and their imaging manifestations may overlap with those of malignant renal tumours.²⁷

Xu Wang⁹ developed a prediction model using thin-section multidetector CT to distinguish hypervascular ultrasmall renal cell carcinoma (≤2 cm) from renal angiomyolipoma with minimal fat in the early stage. However, their study did not encompass renal cell carcinoma (RCC) larger than 2 cm or other hypervascular renal tumours, such as renal oncocytoma (RO). Due to the shared origin of some benign and malignant renal tumours, they exhibit overlapping histologic and imaging characteristics.²⁸ Consequently, distinguishing benign renal tumours from malignant renal tumours using conventional enhanced imaging techniques is still challenging.

Radiomics emphasizes that medical images are not only visual representations but also valuable sources of data that can be analysed and utilized for improving patient care and outcomes through a deeper understanding of the underlying data present.¹³ Xue-Ying Sun²⁹ compared the performance of machine learning models with that of expert radiologists, and their results revealed that radiologic-radiomic machine learning could improve interreader concordance and performance, with a high sensitivity of 86.3% and 73.4% and a specificity of 83.3% and 91.7%, respectively, for differentiating clear cell RCC and chromophobe RCC from fat-poor AML and RO. C. Erdim³⁰ conducted an analysis of texture patterns in CECT images using machine learning algorithms to differentiate malignant RMs from benign RMs. Their results showed a high accuracy of 90.5% in distinguishing them via the random forest (RF) algorithm. However, they also noted that two sets of patients (one for training and one for testing) are needed for further analysis.³⁰ A recent study also showed that enhanced diagnostic accuracy and efficiency are achieved through the utilization of CT texture-based machine learning algorithms to differentiate between benign and malignant cystic RMs.³¹ A study by Cassandre Garnier³² also demonstrated the significant value of radiomics models based on CECT in predicting malignant and benign renal tumours. All these studies support the potential of machine learning algorithms to help radiologists to achieve more accurate diagnostic outcomes.

Our study indicated that CECT-based machine learning models demonstrate potential value in differentiating benign and malignant solids containing RMs. In our study, we developed a radiomic signature comprising 13 radiomic features that exhibited satisfactory predictive ability for distinguishing malignant from benign RM. Among these factors, the original shape flatness (OSF) achieves high predictive performance and holds high variable importance in the radiomic model for distinguishing malignant from benign RM. This result is similar to the findings reported by Yong Huang,³³ where OSF constituted the primary factor for the construction of machine learning models to develop personalized treatment decision models for patients with inoperable oesophageal cancer. The flatness of an object was quantified by OSF, where higher values indicate a greater degree of flatness.³³ The grey-level size-zone matrix (GLSZM) and small area low grey-level emphasis (SALGLE) exhibited the most negative correlations (Figure 3) in the predictive model. Yusufaly³⁴ also reported that the SALGLE score was strongly negatively correlated with the outcome of treatment in patients with previously untreated locoregionally advanced cervical cancer according to a radiomic model. Other features, including the GLDM, NGTDM, GLRLM and first-order features, have also been used to construct radiomic models. Overall, the LR model achieved the best performance, with an AUC of 0.952 in the training cohort and 0.887 in the test cohort (Table 3, Figure 4).

Univariate and multivariate analyses revealed that sex, age, and alcohol consumption were significant predictors for distinguishing between malignant and benign RMs (Table 2). These predictors have the potential to establish both a clinical model and a combined model for identification purposes. The combined model of the radiomic signature and clinical features achieved better discrimination. This finding is consistent with the findings of N. Nassiri,³⁵ who constructed a CT-based radiomic model and achieved good results in distinguishing between malignant and benign RMs. However, they used only the random forest (RF) and REAL AdaBoost platforms for model construction, and the REAL AdaBoost model achieved better diagnostic performance (AUC of 0.83),³⁵ which is different from our research findings where the LR model was optimal. The incorporation of clinical factors into radiomic analysis led to improved performance, with an AUC of 0.973 (95% CI, 0.9388–1.0000) in the training cohort and 0.900 (95% CI, 0.7874–1.0000) in the test cohort for the combined features (Table 3, Figure 5). The above results suggest that the combination of clinical models and machine learning models can effectively predict malignant and benign solids containing RMs.

Our study demonstrates that the nomogram constructed based on clinical and radiomic features has good applicability in predicting solid-containing RMs, aiding clinicians in making more accurate decisions in practice and providing personalized risk assessments for patients, thus avoiding unnecessary overtreatment. However, it is important to acknowledge that different studies may employ varying enhancement scanning protocols and image segmentation methods, which could potentially limit the clinical applicability of the study’s findings.³⁶ Furthermore, there remains an opportunity for further integration of radiomic data with clinical, genetic, and metabolic data to gain a more comprehensive understanding of renal masses, and to leverage the complementary value of each modality in the diagnosis, prognosis, prediction of treatment response, and monitoring of renal masses.³⁷

This study has several limitations that should be acknowledged. First, this was a retrospective, single-centre study, the relatively small sample size may introduce potential selection bias. Second, the random forest (RF) model was not used in our study because RF has poor generalizability when processing small sample cohorts and may overfit.^38,39 Third, this study did not utilize non-CECT images for analysis, as many RMs cannot be distinguished from normal renal parenchyma on noncontrast scans; however, this does not disregard the value of noncontrast scans in the diagnosis of renal tumours. Additionally, we did not utilize a separate validation set. Instead, we employed ten-fold cross-validation to assess the model’s performance, ensuring that the model was tested on different subsets of the data, thereby reducing the risk of overfitting. The independent test set was used solely for the final performance evaluation to simulate the model’s ability to generalize to unseen data. Thus, an external validation is essential in future research to improve the reliability of the research findings. These factors may all serve to limit the generalizability of the results. It is necessary for future studies to be conducted with larger sample sizes and across multiple centers to enhance the reliability and determine the efficacy of machine learning models in distinguishing between malignant and benign solid-containing RMs.

Conclusion

In summary, this study introduces machine learning models that integrate both radiomic signatures and clinical factors, thereby offering a noninvasive and promising approach for the preoperative differentiation of malignant tissue from benign solid tissue containing RM. These enhancements may refine the accuracy of clinical decision-making and potentially avert the risks associated with unwarranted aggressive surgical treatments. However, it is crucial to highlight the imperative need for external validation and multicenter studies to substantially enhance the reliability of the research findings.