Domain adaptation for image synthesis and semantic segmentation

Abstract

Deep-learning based computer vision has witnessed tremendous development in the past few decades. However, the tremendous development comes from the access to big data, big models, and high-performance computers. Researchers in the research area of computer vision move from hand-crafted features with strong human intuitive to CNN-based deep features with only image locality prior, and recently to transformer-based features with no prior at all. The relaxation of prior knowledge improves the performance of the model but introduces the extra cost for the training. One promising direction to alleviate the data-hungry attribute of the model is to use the domain adaptation techniques.

In this thesis, we explore and propose the domain adaptation methods for different applications. The research areas range from low-level image synthesis task to semantic segmentation task. The image synthesis includes the image super-resolution where image local patterns are the most important features during practice, and image translation where both image local structure and image semantic prior play a pivotal role. The high-level image semantic segmentation is the task in which only semantic context matters, and model should overcome with the irrelevant domain gaps such as domain specific local texture. Based on our comprehensive experiment, different modules and pipelines are proposed to either extract or remove the domain specific information to optimize the performance of the corresponding sub-tasks.

In summary, this thesis bridges the gaps from the domain adaptation in the low-level image synthesis to the domain adaptation in the high-level image semantic segmentation, constructing effective ways to extract knowledge across different domains. First, we introduce the Meta-Fusion module to generate super-resolved images with arbitrary level of stochasticity; then we construct a unified approach to reduce the domain gaps for semantic segmentation; at last, a knowledge distillation module is proposed to provide the missing semantic information for image translation task.