Face alignment and face mask reasoning for the images in the wild

Abstract

Over the past decades, face alignment, a process of localising semantic facial landmarks such as eyebrows and nose tip, has been intensively studied as it is an essential prerequisite for many face analysis tasks, e.g., face animation, 3D face modelling and face recognition. In recent years, remarkable progress has been made and some of them even reported close-to-human accuracy on academic benchmack '300W'. However, these methods are prone to break down when confronting partial facial occlusions or large head pose, which occur frequently in realistic scenarios. In this thesis, we aim to advance face alignment performance under above mentioned challenges. We also conduct an empirical study of recent face alignment methods.

First, we will focus on exploring solution for face occlusion handling. We address the face mask reasoning and facial landmarks localization in an unified Structured Decision Forests framework. We first assign a portion of the face dataset with face masks, i.e., for each face image we give each pixel a label to indicate whether it belongs to the face or not. Then we incorporate such additional information of dense pixel labelling into training the Structured Classification-Regression Decision Forest. The classification nodes aim at decreasing the variance of the pixel labels of the patches by using our proposed structured criterion while the regression nodes aim at decreasing the variance of the displacements between the patches and the facial landmarks. The proposed framework allows us to predict the face mask and facial landmarks locations jointly.

In the second part of the thesis, we present a novel reflective method to estimate 2D-3D face shape across large pose. We include the knowledge that a face is a 3D object into the learning pipeline, and formulate face alignment as a 3D Morphable Model (3DMM) fitting problem, where the camera projection matrix and 3D shape parameters are learned by an extended cascaded pose regression framework. In order to improve algorithm robustness in difficult poses, we introduce a reflective invariant metric for failure alert. We investigate the relation between reflective variance and face misalignment error, and find there is strong correlation between them. This finding is exploited to provide feedback to our algorithm. For samples predicted as failure, we restart the algorithm with better initialisations based on explicit head pose estimation, which enhances the possibility of convergence.

Finally, we carry out a rigorous evaluation of recent face alignment methods: 1) we proposes a new evaluation metric for face alignment on a set of images, i.e., area under error distribution curve within a threshold, $AUC_{a}$, as the traditional evaluation measure (mean error) is very sensitive to big alignment error. 2) we extend the 300W database with more practical face detections to make fair comparison possible. 3) we carry out face alignment sensitivity analysis w.r.t. face detection on both synthetic and real data, using both off-the-shelf and re-retrained models. 4) we study factors that are particularly important to achieve good performance and provide suggestions for practical applications. Most of the conclusions drawn from our comparative analysis cannot be inferred from the original publications. (410 words)