Investigating the Use of Generative Adversarial Networks-Based Deep Learning for Reducing Motion Artifacts in Cardiac Magnetic Resonance

Introduction

The increasing use of cardiac magnetic resonance (CMR) is closely tied to the advancements in magnetic resonance imaging (MRI) technology.^1,2 CMR procedures encompass various sections, parameters, and sequences, enabling the comprehensive acquisition of cardiac structural, functional, histological, and hemodynamic information in a single examination, often referred to as a “one-stop” assessment.³ However, CMR faces challenges from respiratory-induced diaphragmatic motion and motion artifacts resulting from cardiac pulsations. These artifacts hinder accurate delineation of the endocardial and epicardial boundaries, leading to inaccuracies in the assessment of cardiac functional and ventricular wall thickness. Additionally, motion artifacts may cause uncertainty among CMR practitioners regarding intramyocardial enhancement observed in gadolinium contrast-delayed enhancement sequences, as such enhancements could be mistakenly interpreted as artifacts.^4,5 Therefore, reducing motion artifacts in CMR images, particularly those caused by motion, is of paramount clinical significance for the precise diagnosis and evaluation of cardiac diseases.

Current research on reducing motion artifacts in CMR includes several approaches. One method focuses on refining the breath-holding technique to extend the duration patient breath-holding, thereby minimizing motion artifacts.^6–8 Additionally, navigation-based free-breathing cardiovascular magnetic resonance imaging offers an alternative for patients unable to comply with breath-holding requirements, effectively reducing respiratory artifacts.^9–11 Rapid imaging sequences in MRI can also alleviate motion artifacts by shortening scan times and enhancing image robustness against motion-induced distortions.^12–15 However, these methods often require pre-scan preparations and have inherent limitations. On the other hand, post-processing techniques for artifact reduction, particularly those applied to k-space data acquired during MRI or to images generated post-scan, align more seamlessly with routine clinical workflows. Recently, the integration of deep learning methodologies has emerged as a key approach in medical image reconstruction and artifact reduction. While numerous studies have addressed MRI image reconstruction, research specifically focused on motion artifact reduction in CMR remains limited, particularly regarding the optimization of CMR image quality post-scanning.^16–21

In clinical practice, CMR cine sequences are vital tools for assessing cardiac structure and function. However, they are highly susceptible to interference from respiratory motion artifacts, which pose significant challenges. As a result, the control and reduction of motion artifacts in CMR cine sequences have become primary areas of interest. This study aims to investigate the potential application of deep learning technology, based on generative adversarial networks (GANs), in reducing motion artifacts in CMR cine sequences, thereby enhancing their diagnostic reliability and accuracy.

Data and Method

This study is a retrospective study and has been approved by the Ethics Review Committee of this institution. Written informed consent was obtained from the participant for the publication.

Magnetic Resonance Imaging Acquisition

All patients were scanned using the Discovery 3.0 T superconducting MR scanner from GE Healthcare, USA, equipped with a gradient field of 50 mT/m and a gradient switching rate of 200 T· (m·s)⁻¹. The patient was positioned in the supine position, with the 8-channel array coil situated in the anterior chest region, and the scan was conducted using electrocardiogram gating and respiratory gating techniques. The MRI procedure captured images along the long axis of the four heart chambers and the long axis of the left ventricle and two heart chambers, using the fast imaging employing steady-state acquisition (FIESTA) sequence, encompassing 1 scanning layer. Additionally, 8–10 layers of short-axis images ranging from the left ventricular fundus to the apical region were obtained. The scanning parameters included a repetition time (TR) of 3.4 milliseconds, echo time (TE) of 1.5 milliseconds, flip angle of 45 degrees, excitation times of 1, scanning field of view of 350 cm × 315 cm, layer thickness of 8 mm, and spacing of 2 mm. Each scan plane encompassed 25 dynamic phases.

Establishment of Data Set

Data from 60 patients were retrospectively included in the study, comprising of 20 normal participants, 20 patients with myocardial hypertrophy, and 20 patients with cardiomegaly. All cases underwent screening by physicians with more than 5 years of experience in CMR diagnosis, and data exhibiting artifacts were excluded. The short-axis bright-blood cine sequence images of the heart were selected for network training.

The GAN-based deep learning algorithm necessitates clear-blurry image pairs with identical positioning and morphology; however, variations exist among patient images from each scan. Consequently, real artifact image data cannot serve as a comparison for clear images. To obtain the required clear-blurry image pairs for the training set, varying degrees of motion artifacts were artificially induced onto clear images to simulate genuine motion artifact conditions. In this study, the approach proposed by Zhang et al was adopted to simulate motion artifacts in MRI.²² This involved randomly selecting multiple frames of images on average along the direction of diaphragmatic motion and generating trajectories using the Markov process. The position of the next point in the trajectory was randomly determined based on Gaussian perturbations, pulse perturbations, and deterministic inertia components, thereby enabling the simulation of short-axis bright-blood cine sequence images of the heart with diverse degrees of motion artifacts. Following the aforementioned processing, the final deep learning network consisted of 2000 sets of clear-blurry image pairs in the training set and 200 sets in the test set. Additionally, 100 artificially generated images with artifacts were selected to assess the established deep learning network. Meanwhile, 37 CMR cine sequence images featuring real motion artifacts from 10 patients in clinical practice were selected to evaluate the clinical applicability of the deep learning network.

In this study, CMR images in DICOM format were converted into three-channel color images, with adjustments in rotation, translation, and scaling to augment the dataset for model training purposes. The resulting original image size was standardized to 512×512×3.

Deep Learning Network Structure

The Selected GAN Principle

In this study, the deep learning network employed the recently proposed GAN architecture, which comprises two modules: a generator and a discriminator, operating in an adversarial manner. The process resembles a zero-sum game, where the objective of the GAN framework is to enhance the capabilities of both the generator and the discriminator. Achieving equilibrium between the generator and discriminator is attained through continuous iteration, by playing a game. This adversarial training fosters concurrent enhancement in the abilities of both components, ultimately leading to the fulfillment of the desired objective.

Generator Structure

The backbone network of the generator serves as the pivotal architectural foundation of GAN, crucial for learning image features in the computer vision. In pursuit of robust de-artifact performance, this study selects Inception-ResNet-v2 as the backbone network due to its proven efficacy.²³ The primary model used is the Feature Pyramid Networks (FPN), which constitutes the backbone of the generator in the de-artifact model.

The integration of an attention mechanism enables the network to focus on essential information, facilitating expedited acquisition of the most pertinent content. To concentrate the network’s attention on the heart and its immediate surroundings, an attention mechanism was incorporated into the generator in this study, with the goal of enhancing image recovery in this region. By introducing attention mechanisms at both the channel and spatial level of the image, there is comprehensive attention coverage across all dimensions of the image.^24,25

At the channel level of the image, the feature map entered into the attention mechanism module underwent global pooling, preserving spatial information extracted from the preceding convolutional and pooling layers. This compression process transformed the two-dimensional features (hw) of the feature map from (h, w, c) to (1, 1, c), retaining only the channel information. Subsequently, a multilayer perceptron was used to assign weight values to each feature channel, giving higher weight to key channels requiring attention and lower weights to others. This assignment process was facilitated through fully connected layers. Following this, the feature map with uncompressed two-dimensional features (hw) was combined with the weighted feature channels, enabling each channel of the original feature map to be endowed with corresponding weights. During subsequent learning stages, the network prioritized features with higher weights while relatively disregarding those with lower weights, thereby incorporating the attention mechanism, as illustrated in Figure 1.

Figure 1 Flowchart of the attention mechanism at the channel level.

The attention mechanism at the spatial level builds upon the foundation established by the attention mechanism at the channel level, aiming to identify the location with the most pertinent information based on channel direction. As illustrated in Figure 2, both average pooling and max pooling were conducted along the channel axis, generating a valid feature descriptor. At each pooling stage, comparison were made between values of different channels rather than values from distinct areas within the same channel. These feature descriptors were then fed into a convolutional network, followed by normalization using the sigmoid activation function to assign varying values between 0 and 1. Subsequently, the attention feature distribution map at the spatial level was obtained.

Figure 2 Flowchart of attention mechanism at the spatial level.

The integration process of the attention mechanism modules at both the spatial and channel levels is depicted in Figure 3. Initially, the original image features were combined with post-processed image features via the channel attention mechanism module on one hand, and through skip connections on the other hand. Subsequently, feature maps containing diverse attention parameters ranging from 0 to 1 were ultimately generated via the spatial attention mechanism module using a similar methodology. Image processing continued accordingly following this integration.

Figure 3 Attention mechanism module after integrating the spatial level and the channel level.

In addition, deformable convolution was incorporated into the generator in this study.²⁶ It constitutes an operation within convolutional neural networks (CNNs) to enhance network performance by introducing a deformable offset into the convolutional kernel. This enables dynamic adjustment of the sampling position of the convolutional kernel in response to input data, thereby enhancing the perception of object deformation of the network. Deformable convolution was implemented by substituting traditional fixed-size filters of the first two layers in the FPN with variable-size filters. Initially, the feature map of the input CMR image was extracted using a traditional convolutional kernel. Subsequently, convolution was applied to this feature map to obtain the deformation offset required for deformable convolution. This offset layer comprised 2N values in both the x and y directions to facilitate translation on a plane. During training, both the convolutional kernel used for generating output features and the kernel used for generating the offset were learned synchronously. The learning process of the offset was accomplished through the interpolation algorithm and backpropagation. Following this, the original feature map and the deformable convolutional kernel underwent convolution, and the weighted mean value was computed to yield the final output result of deformable convolution.

It is important to note that deformable convolution does not change the shape of the convolutional kernel, which remains a regular rectangular shape of size x*x. Rather, it modifies the region on the feature map where the convolutional kernel operates. The introduction of the offset changes the position of the feature map that the kernel acts upon, thereby facilitating feature learning for different shapes.

Finally, the overall structure of the generator, enhanced with the inclusion of attention mechanism and deformable convolution, was illustrated in Figure 4. The initial feature layer was obtained by learning the correlation of features from each part of the original image using the deformable convolution module. This feature layer was then propagated in two directions: Upward, passing through the max pooling layer for further feature extraction and subsampling to reduce resolution size. Subsequently, it traversed through the second deformable convolution module, repeatedly extracting relevant features between different parts of the image and generating feature layers with varying resolutions. Horizontally, dimensionality was reduced based on a 1×1 convolution, followed by fusion with the image of the same resolution size obtained through interpolation upsampling on the right side. This combined detailed low-level features with semantic high-level features, enriching feature learning. In the top-down part on the right, dimensionality reduction of the highest-level feature map was achieved via a 1×1 convolution. Subsequently, through the upsampling layer, a feature map with a higher resolution was generated using interpolation. This feature map was then fused with the feature map of the same resolution from the bottom-up part through an additional layer, facilitating fusion of low-level detail features with high-level semantic features in the same feature layer to bolster feature learning. Ultimately, five final feature maps with different resolutions were obtained as output. These output features were then upsampled to 1/4 the size of the original image and combined into a tensor containing varying levels of semantic information. Subsequently, preliminary de-artifacting of the network output was performed using upsampling and convolutional layers. Following this, the output passed through the attention mechanism module to enhance the selection of CMR image de-artifact positions. Finally, through upsampling and convolutional layer modules, the original image size was restored and artifacts were eliminated. Additionally, the skip connection method proposed by Liu D et al was used, directly connecting the input image with the output feature map to facilitate residual learning focus.²⁷

Figure 4 Network structure of the generator after optimization.

Discriminator Structure

Due to the necessity of discerning and distinguishing the structural features of the heart within CMR images, this study used the PatchGAN discriminator from DeblurGAN.²⁸ PatchGAN operates by classifying localized areas of an image rather than assessing the entire image at once. This localized discrimination approach enables PatchGAN to effectively capture image details and structural information, thereby enhancing its ability for accurate discrimination. The discriminator structure, as depicted in Figure 5, used input images comprising the de-artifact image generated by the generator alongside real images, each with a size of 512*512*3. These images were divided into 16 patches, with each patch sized at 128*128*3. Subsequently, these patches were individually entered into a fully convolutional network to yield a feature map of size 1*1*512. This feature map was then fed into a fully connected layer to determine the probability of each patch being classified as a real sample. The probabilities of each patch were averaged to ascertain the overall authenticity of the image. This assessment was then fed back to the generator for continuous iterative refinement in a round of adversarial training. The final structure of the deep learning GAN established in this study is depicted in Figure 6.

Figure 5 Schematic diagram of the discriminator PatchGAN.

Figure 6 The overall structure of GAN established in this study.

Image Quality Evaluation

Objective Evaluation of Image Quality

The objective assessment indices used for motion artifact correction in CMR images included the peak signal-to-noise ratio (PSNR), structural similarity index (SSIM), and Tenengrad focus measure. The PSNR, SSIM and Tenengrad focus measure of 100 artifacts were compared with the corresponding artifacts removed images. Since there is no original clear image corresponding to the real artifact image, PSNR and SSIM cannot be calculated, and only Tenengrad focus measure were compared.

PSNR serves as a metric to quantify the quality of images or videos, where higher values indicate superior image or video quality. PSNR was calculated by assessing the mean squared error (MSE) between the original image and the post-processed image, followed by conversion into a logarithmic scale.

SSIM is used to assess the similarity between two images, with higher values denoting greater similarity. SSIM computation involves assessing the structural similarity between the original image and the post-processed image.

The Tenengrad function was used for image focus measurement. It is a commonly adopted image clarity assessment function based on gradients. In image processing, it is widely acknowledged that a well-focused image exhibits sharper edges, thus resulting in a larger gradient function value.

Subjective Evaluation

The effect of removing artifacts from 100 artificial artifacts and 37 real artifacts was evaluated by subjective score. The subjective assessment of CMR image quality involved independent scoring by two experienced CMR physicians, each possessing over 5 years of expertise in cardiovascular imaging, using a blinded methodology. In cases of disagreement, the two physicians engaged in discussion to reconcile their assessments; failure to reach a consensus resulted in the invalidation of the image. A 5-point Likert scale was used to score the clarity of myocardial boundaries and the degree of artifacts present in CMR images. The scoring criteria were as follows: 1-Poor myocardial boundary sharpness, accompanied by extremely severe artifacts, rendering diagnosis unattainable.2-Unsatisfactory myocardial boundary sharpness, marked by severe artifacts, posing challenges to diagnosis. 3-Moderate myocardial boundary sharpness, with discernible artifacts, allowing for diagnosis. 4-Relatively good myocardial boundary sharpness, presenting mild artifacts, generally conducive to diagnosis. 5-Good myocardial boundary sharpness, devoid of artifacts, facilitating satisfactory diagnosis.

Statistical Analysis

All data were subjected to statistical analysis using SPSS 26.0 software. Measurement data were presented as mean ± standard deviation ( ±s). Since the samples did not conform to the normal distribution or the variance of the two groups of samples was inconsistent, the paired samples Wilcoxon signed-rank test was used in the comparison of PSNR, SSIM, Tenengrad Focus Measure, and subjective scores. The consistency of subjective image quality scores between the two CMR physicians was assessed using the Kappa test. A Kappa value >0.75 indicated good consistency, while a value falling between 0.40≦ and <0.75 indicated general consistency. A Kappa <0.40 indicated poor consistency. Statistical significance was set at P<0.05.

Results

Ablation Study

The findings from the ablation study are summarized in Table 1. Upon examining the objective outcomes derived from the ablation study, it becomes evident that the inclusion of the channel attention mechanism, spatial attention mechanism, and deformable convolution enhances the efficacy of artifact reduction in images. Notably, the spatial attention mechanism exhibits a more pronounced impact compared to the channel attention mechanism. Furthermore, the effectiveness of using deformable convolution in isolation is comparatively less discernible than when combined with attention mechanisms.

Table 1 Results of Ablation Study

Test results of This Network on CMR Images with Artificial and Real Artifacts

The deep learning network model developed in this study effectively addresses motion artifacts present in CMR cine images. Prior to optimization using the deep learning network model, the PSNR, SSIM, and Tenengrad Focus Measure of the 100 images afflicted with artificial artifacts were recorded as 23.85±2.85, 0.71±0.08, and 4.56±0.67, respectively. Post-optimization, these metrics improved to 27.91±1.74, 0.83±0.05, and 7.74±0.39, respectively. Correspondingly, the Tenengrad Focus Measure of the original CMR images without artifacts stood at 7.91±0.30. The enhancement achieved in the PSNR, SSIM, and Tenengrad Focus Measure metrics after optimization amounted to 17.0%, 16.9%, and 69.7%, respectively. The results of paired samples Wilcoxon signed-rank test showed P<0.001, the difference was statistically significant. Furthermore, the subjective assessments of images in the artificially generated artifacts group yielded an average score of 2.44±1.08, whereas de-artifacted images received a score of 4.44±0.66, marking an average increase of 2 points. The disparity between the two groups was statistically significant (P<0.001). Comprehensive objective and subjective assessment results are presented in Table 2 and depicted in Figures 7 and 8. Notably, the subjective evaluations conducted by the two CMR physicians demonstrated a high level of agreement (kappa test results showed: kappa=0.880, P<0.001 for images with artifacts; kappa=0.912, P<0.001 for de-artifact images).

Table 2 Subjective and Objective Evaluation

Figure 7 Statistical results of objective evaluation. ***P<0.001.

Figure 8 Subjective scores of CMR images. X-axis represents the number of images, and Y-axis represents the subjective scores.

To assess the practical clinical efficacy of this deep learning network, 37 CMR cine sequence images containing authentic artifacts from 10 patients were selected to validate the established model. Results revealed a significant increase in the Tenengrad Focus Measure following artifact removal, with values of 6.06±0.91 pre-treatment compared to 10.13±0.48 post-treatment The result of paired samples Wilcoxon signed-rank test showed P<0.001. The subjective scores exhibited a statistically significant enhancement, rising from 3.03±0.73 pre-treatment to 3.73±0.87 post-de-artifact intervention (The result of paired samples Wilcoxon signed-rank test showed P<0.001. Visual representations of images before and after artifact removal are presented in Figure 9.

Figure 9 A1, A2, A3, B1, B2, and B3 are artificially generated images with artifacts, de-artifact images, and raw clear images, respectively. Figures C1 and D1 are CMR images with real artifacts (arrows), and C2 and D2 are de-artifact images (arrows). As can be seen from the figure, the GAN-based deep learning model has significant effects on the removal of artificial artifacts. The removal of real artifacts is also evident, as indicated by the red arrow, where the edges of the ventricular septum are more clearly displayed in the optimized CMR image.

Discussion

CMR imaging offers comprehensive insights into various aspects of cardiac health, encompassing cardiac structure, function, myocardial histological characteristics (including hemorrhage, edema, fibrosis, and extracellular volume), and heart blood flow dynamics.^29–32 Consequently, CMR holds considerable significance in clinical diagnostics and therapeutic interventions. Notably, the balanced steady-state free precession cine sequence of CMR stands out for its ability to vividly depict cardiac structure and facilitate precise calculations of cardiac function, earning it recognition as the “gold standard” for assessing cardiac structure and function.^33,34 However, respiratory motion artifacts pose a significant challenge in CMR cine sequences, impeding the clear visualization of the endocardium and epicardium, which in turn hinders accurate measurements of myocardial thickness and cardiac function. As such, effective management and optimization of respiratory motion artifacts in cine sequences are critical in clinical practice.

In recent years, the growing importance of deep learning in image processing has sparked considerable research into its application in MR image reconstruction and artifact reduction. Specifically, leveraging deep learning networks for k-space data reconstruction has proven effective in reducing motion artifacts. For instance, Cui et al developed a CNN model capable of identifying and filtering k-space phase encoding lines affected by motion, thereby facilitating the reconstruction of images with diminished artifacts.³⁵ Similarly, Kaniewska et al demonstrated the potential of deep learning in MR image reconstruction of the shoulder joint, highlighting its ability to expedite scanning procedures and attenuate artifacts.¹⁸ Furthermore, established deep learning network models hold promise for direct artifact reduction in MR images post-reconstruction.^36–38 However, research on artifact reduction in CMR images remains relatively limited. Ghodrati et al used an adversarial autoencoder in conjunction with unsupervised learning to construct a deep neural network tailored for respiratory motion compensation in CMR cine sequences, successfully rectifying image artifacts associated with respiratory motion.²⁰ Additionally, Lyu et al developed a recursive generative adversarial network (GAN) model aimed at mitigating motion artifacts in CMR images, enhancing CMR image quality and bolstering the temporal resolution of cardiac cine sequences through the generation of intermediate frames.²¹ Clearly, GAN-based deep learning methodologies hold significant promise for artifact reduction in CMR imaging.

In this study, an optimized GAN was developed to enhance the efficacy of mitigating motion artifacts within reconstructed CMR cine sequence images. GAN, originally proposed by Goodfellow Ian in 2014, serves the primary goal of enabling computers to autonomously generate desired content such as text, images, and sounds.³⁹ Since its inception, GAN has attracted considerable attention among researchers due to its innovative approach and promising outcomes. In this study, the FPN was adopted as the model framework. This choice facilitated the accurate and efficient learning of image features by the generator, leveraging the multi-scale feature aggregation capabilities of the FPN to accommodate both low-level details and high-level semantics. Given that the primary focus of CMR images is on the heart, with minimal attention to other organs, an attention mechanism was incorporated into the GAN-based CMR de-artifact network. This augmentation allowed the network to concentrate its attention on the heart and its immediate surroundings, thereby improving image recovery in these region and avoiding extraneous areas. Furthermore, due to inter-patient variations in body size and cardiac conditions, CMR images exhibit differing overall shapes. To address this variability, deformable convolution was introduced, enabling the network to effectively learn image features across diverse CMR images and enhance its capacity to perceive object deformation without augmenting network parameters or computational costs, thus enhancing overall network performance. Results from this study underscored the efficacy of the enhanced GAN in reducing motion artifacts within CMR cine images, effectively enhancing image quality. Also, the GAN-based deep learning network yielded promising outcomes in clinical practice. Following optimization by the deep learning network model, the Tenengrad Focus Measure of 37 CMR cine sequence images with authentic artifacts exhibited a remarkable increase of 67.2%, accompanied by enhancements in subjective scores. However, the optimization effect on more pronounced authentic artifacts proved to be less favorable. The main reasons for our analysis are as follows: first, the artifact data of the network training set is simulated by clear images. Since the real artifact is generated from K-space images, the simulated motion artifact mode in this study is relatively simple. Second, the simulated artifact is only the movement of a single layer, and does not take into account the movement changes between layers, which will lead to poor effects in the real world. In addition, real-world motion artifacts are more complex, and the simulated artifact data does not fully encompass all situations. Therefore, we believe that in the future, using K-space data to simulate images, or establishing real artifacts and artifacts free image pairs for training, can greatly improve the actual clinical application effect of the network, and is the future direction of efforts.

This study is subject to several limitations that warrant acknowledgment. First, the absence of k-space data for CMR imaging necessitated the simulation of MR motion artifacts solely on clear images, resulting in simulated motion blur rather than the comprehensive array of motion artifacts typically encountered in practice. Second, this is a retrospective study. Due to quality control, fewer cases with more significant artifacts have completed the CMR examination reporting process, resulting in less data of included true artifacts. In addition, due to technical reasons, it is not possible to convert the generated De-artifact images into DICOM files for the calculation of cardiac function and other parameters, so this study did not integrate clinical parameters, such as cardiac function and wall thickness, to assess the network’s efficacy. Finally, the training set comprised of simulated motion artifacts, which may not fully replicate the complexity of MR motion artifacts encountered in clinical settings. Moving forward, future research endeavors should encompass the registration of images featuring authentic artifacts alongside artifact-free counterparts. This approach would facilitate the establishment of genuine artifact-no artifact image pairs, enabling more effective training of the deep learning network model. Such efforts are vital for advancing towards the ultimate objective of clinical application.

Conclusion

The deep learning model devised in this investigation demonstrates notable efficacy in reducing motion artifacts within CMR cine images, thereby enhancing overall image quality. These findings underscore the substantial clinical use of the GAN-based deep learning model for optimizing motion artifacts in CMR images.