Face sketch synthesis and face super resolution in the wild with deep learning

Abstract

Face image processing has been one of the most important problems in computer vision since it is closely related to our daily lives. However, there still exist many challenges in handling natural face images in the wild. In this thesis, we will explore deep learning approaches on two face image processing tasks, namely face sketch synthesis and face super resolution. With the powerful representation ability of deep neural networks, we are able to narrow the gap between research and real world applications in these two tasks.

Face sketch synthesis targets at generating a sketch from an input face photo. We first introduce a cascaded framework to synthesize face sketches which imitates the process of how artists draw sketches. A content image is first generated through a fully convolutional neural network (FCNN) that outlines the shape of the face, and then textures and shadings are added to enrich the details of the sketch with a style transfer approach. Such cascaded framework not only helps to preserve more sketch details than the common style transfer method but also surpasses traditional patch based methods. To process natural face images, we then presents a semi-supervised deep learning architecture which extends face sketch synthesis to handle face photos in the wild by exploiting additional face photos in training. Instead of supervising the network with ground truth sketches, we first perform patch matching in feature space between the input photo and photos in a small reference set of photo-sketch pairs. We then compose a pseudo sketch feature representation using the corresponding sketch feature patches to supervise our photo-to-sketch network. With the proposed approach, we can train our networks using a small reference set of photo-sketch pairs together with a large face photo dataset without ground truth sketches. Besides, we also jointly learn a sketch-to-photo network based on self-supervision of input photos.

Then, we focus on face super resolution which refers to generating high resolution face images from the corresponding low resolution inputs. We introduce a novel SPatial Attention Residual Network (SPARNet) built on the proposed Face Attention Units (FAUs) for face super resolution. FAUs improve the vanilla residual blocks with spatial attention mechanism, and enable the convolutional layers to adaptively bootstrap features related to the key face structures and pay less attention to those less feature-rich regions. We further extend SPARNet with multi-scale discriminators, named as SPARNetHD, to produce high resolution results (i.e., 512x512). We show that SPARNetHD trained with synthetic data can not only produce high quality and high resolution outputs for synthetically degraded face images, but also show good generalization ability to real world low quality face images.