A new probabilistic representation of color image pixels and its applications in image registration, segmentation and person re-identification

Abstract

Representations of color image pixels play a fundamental role in many image processing algorithm as it can be used to construct different similarity measures between image pixels/regions. A distinctive and invariant color representation which can explore certain image prior information will benefit various aspects in image matching and recognition applications. This thesis proposes a novel probabilistic representation of color image pixels (PRCI) and investigate its applications to similarity construction and data augmentation. It describes a given image pixel in an image/image sequences in terms of its membership in a finite multivariate Gaussian or Laplace mixture representation of the image content. Based on this membership representation, a new pixel-wise similarity measure based on the approximation of the continuous domain Bhattacharyya coefficient is proposed, which yields a convenient expression in terms of the memberships of the pixels to be compared. Furthermore, this pixel-wise similarity is extended to measure the similarity between two image regions. The usefulness of the proposed pixel-/region-wise similarities is demonstrated by incorporating them respectively in a Free-form deformation (FFD)-based image registration problem, a dense image descriptor-based motion estimation problem and an unsupervised image segmentation problem. Experimental results show that 1) the proposed pixel-wise similarity improves the registration accuracy and robustness of the FFD algorithm comparing with traditional RGB color space and Euclidean distance measure; 2) the dense Scale-Invariant descriptor (SID) with soft segmentation mask constructed by PRCI pixel-wise similarity yields improved peak signal to noise ratio performance and higher tracking accuracy in terms of Dice coefficient over the state-of-the-art dense SIDs and 3) both the proposed pixel- and region-wise similarities give the best performance in terms of almost all quantitative measurements including Global Consistency Error, Boundary Displacement Error, Variation of Information and Probabilistic Rand Index among all algorithms tested. The application of the proposed representation to data augmentation in pattern recognition problems for addressing the small sample size (SSS) and color variation problem frequently encountered in computer vision tasks is further investigated. In particular, we utilize the proposed mixture representation to generate additional samples to improve the performance of state-of-the-art classifiers for the person re-identification problem. A novel Physically Motivated Data Augmentation (PMDA) scheme which estimates the color/illuminance distribution from the training data to generate new samples under different color/illuminance perturbations is proposed to better capture the objects’ appearance so as to mitigate the SSS and color variation problem. To cope with the generated data, a Color/Illuminance Aware Feature Augmentation (CIAFA) approach, which is applicable to state-of-the-art features and metric learning algorithms, is proposed to integrate the features generated by augmented samples for subspace/metric learning. A new local CIAFA (L-CIAFA) metric learning scheme, which allows the subspace/metric learning to be performed independently on each pair of augmented data sets and fuse a set of “local” distance functions to form an overall distance for recognition, is also proposed. This reduces the memory requirement and complexity over the original CIAFA. Experimental results on common datasets show that the proposed methodologies substantially improve the performance of state-of-the-art subspace/metric learning algorithms.