A model-based approach to the recovery of non-rigid shape from an image sequence

Abstract

The Non-Rigid Structure from Motion (NRSFM) problem is a challenging problem in computer vision aiming at recovering the deforming shape of a flexible 3D object from a sequence of 2D image measurements, such as an expressive face, a human body under motion, or a moving robot arm. It is however an ill-posed problem with more unknown variables than inputs whose solution requires further regulation by imposing additional constraints. Two major constraints that have been utilized by most recent works devoted to this problem are the low-rank condition and the articulated model. Similar to all existing works in this area, the camera model is assumed to be orthographic in this thesis.

This thesis introduces the Small Deformation from Average Shape (SDFAS) condition to remove the ill-posedness of the NRSFM problem. This condition is fundamental to the definition and estimation of the camera motion and the average shape. It is shown to be a sufficient but not necessary condition of the low-rank condition. Our analysis indicates that many existing methods that claim to be based on the low-rank condition in fact implicitly rely on the SDFAS condition, and the commonly assumed low-rank condition alone is not sufficient to guarantee these existing low-rank methods to work. We then developed two new approaches to the NRSFM problem, namely the blend shape method and the ellipsoid fitting method, for non-rigid shapes that may or may not satisfy the SDFAS condition.

The blend shape method is proposed to recover non-rigid structures satisfying the SDFAS condition by modeling them as a linear combination of blend shapes. In our blend shape method, a pseudo view is introduced to suppress distortion of the estimated blend shapes in the direction of the camera axis, such that the blend shapes are guaranteed to be 3D shapes with clear physical meaning. This gives the algorithm an advantage of being robust against overfitting compared with other existing low-rank methods.

For non-rigid structures not satisfying the SDFAS condition, the ellipsoid fitting method is proposed for datasets that can be described by an articulated model. We first revealed that points belonging to a rigid subset must satisfy the ellipsoid property, and design an efficient algorithm to apply this property to segment the feature points into different rigid subsets. The recovered rigid subsets are then linked as a kinematic chain to reconstruct the 3D articulated structure. The blend shape method and the ellipsoid fitting method are combined into a hybrid method to achieve refined results for articulated structures whose individual links are not perfectly rigid but may undergo small deformations such as motion of the human body. The hybrid method is applicable to datasets satisfying the SDFAS condition and those that fit to the articulated model. The effectiveness of all the proposed methods is demonstrated by experiments on both synthetic data and real data, with comparisons with existing methods.