Deep learning for pansharpening and classification of hyperspectral images

Abstract

Hyperspectral images (HSIs) are well known for containing abundant spectral information and have found numerous applications in various areas, such as environmental monitoring and remote sensing. Therefore, developing effective HSI processing and analysis techniques is crucial for these applications. Recently, deep learning has achieved great success in image processing and computer vision areas. Compared to traditional models based on hand-crafted features, deep networks rely on more relaxed priors and have much stronger learning capability for different levels of vision tasks, ranging from regression to segmentation, and to classification. The tremendous development of deep learning offers a promising direction to advance the field of hyperspectral vision. This thesis is in pursuit of exploring and proposing deep learning methods for HSI processing and analysis. The research areas range from low-level hyperspectral pansharpening task to high-level HSI classification task. In general, hyperspectral pansharpening involves the enhancement of spatial resolution of HSIs where local patterns matter, while HSI classification aims to predict the category of each pixel where high-level semantics are important.

In the first part of this thesis, we focus on the problem of hyperspectral pansharpening. How to achieve complete and accurate fusion of high-resolution panchromatic images and low-resolution hyperspectral images remains a critical challenge for models. To address this issue, we propose a multistage dual-attention guided fusion network (MDA-Net), which employs three streams to explore the intra-image characteristics and inter-image correlation simultaneously and merges them in multiple stages, where dual-attention is used to guide the fusion by identifying the useful spatial and spectral contents. Experimental results on both real and simulated datasets demonstrate the superiority of our network.

Second, we consider the problem of lacking labels in HSI classification. Deep networks require a large amount of labeled data for optimization. However, collecting sufficient labeled HSI samples is very time-consuming and expensive. To overcome this, we propose a self-supervised learning method, termed cross-domain contrastive learning (XDCL), which performs contrast between the spectral and spatial domains to learn semantic representations. We construct effective signals that represent the two domains, respectively, and demonstrate that semantics are shared across them while other contents tend not to be. Experimental results show that our XDCL learns powerful representations that boost the downstream classification task.

Third, we further enhance self-supervised learning in HSI classification. Despite great success in feature extraction without labels, self-supervised pre-training needs to be performed from scratch for each HSI, which is laborious and time-consuming. Moreover, its performance is sensitive to HSI quality. We propose to address these issues through transfer learning. We build a progressive self-supervised pre-training framework that performs sequential training on datasets that are increasingly similar to the target HSI. Specifically, it starts with a large general vision dataset and then moves to a related HSI dataset, acquiring a strong initialization for the final pre-training. Self-supervised elastic weight consolidation is also developed to mitigate catastrophic forgetting in continual pre-training. Apart from improving the convergence speed and representation quality significantly, our framework is applicable to various self-supervised learning algorithms.