MT_GAN: A SAR-to-optical image translation method for cloud removal

Abstract

<p>Synthetic Aperture Radar (SAR) is an active microwave imaging and earth observation device capable of penetrating through clouds, rain, and fog, enabling it to operate effectively regardless of the weather conditions and throughout the day. However, speckle noise in SAR images can make them difficult to interpret, and optical images are often difficult to observe when they are covered by clouds. Therefore, after preprocessing, SAR images can be directly converted to optical images through end-to-end translation learning without optical images as auxiliary information, improving the interpretability of SAR images and realizing cloud removal. Due to the relatively simple structure design of the existing generator based on residual network, it is not perfect to capture and extract the feature information of the image, and the relationship between the features is not well connected, resulting in the existing SAR-optical translation results are not accurate enough. To mitigate this issue, we propose an image translation method utilizing a multilayer translation generative adversarial network (MT_GAN) for cloud removal. First, we design a despeckling module (DSM) to preprocess the speckle noise in SAR. Furthermore, a multilayer translation generator (MTG) is designed for SAR-to-optical (S-O) image translation. It can perform multi-scale image translation on different layers and combine them to enrich the semantic information of features and optimize the translation results. In addition, MTG combined with PatchGAN discriminator is used to compose the optical image generation sub-network (OGS) and SAR image regression sub-network (SRS). Finally, the SRS and OGS are used to establish the connection of cycle consistency loss and optimize the generated optical image. We prepare four datasets for experiments, two of which are used for image translation experiments and the other two for cloud removal experiments. The findings demonstrate that our proposed approach outperforms existing methods across all evaluation metrics and reaches 28.6140 and 0.7069 in PSNR and SSIM indicators, which surpass MS-GAN (28.3348, 0.6403) and DSen2-CR (28.3472, 0.6857), and effectively removes the cloud. The datasets and codes are available at https://github.com/NUAA-RS/MT_GAN.</p>