Single view reconstruction of transparent, mirror and diffuse surfaces

Abstract

3D reconstruction has been a fundamental problem in computer vision and has many applications. However, existing methods are mostly designed for diffuse surfaces under multiple viewpoints. This thesis tackles three reconstruction problems under a single view, namely, transparent object reconstruction, mirror surface reconstruction, and diffuse surface reconstruction. Besides, semantic correspondence, which is essential for not only 3D reconstruction but also image understanding, is also investigated in this thesis.

In the first part of this thesis, a novel and practical approach is presented for transparent object reconstruction under a fixed viewpoint. A simple and handy setup is introduced to alter the incident light paths before light rays enter the object, followed by a surface recovery method based on reconstructing and triangulating such incident light paths. Our approach does not need to explicitly model the complex interactions of light as it travels through the object, assuming neither any parametric form for shape of the object nor exact number of refractions and reflections occur when light travels through the object. It can handle a transparent object with a complex structure, with an unknown and even inhomogeneous refractive index.

This thesis then considers the problem of mirror surface reconstruction under a fixed viewpoint. We first derive an analytical solution to recover the camera projection matrix, and then optimize the camera projection matrix by minimizing reprojection errors with a cross-ratio formulation. The mirror surface is finally reconstructed based on the optimized cross-ratio constraint. The proposed method only needs reflection correspondences as input and removes the restrictive assumptions of known motions, $C^n$ continuity of the surface, and calibrated camera(s) that are being used by other existing methods. This greatly simplifies the challenging problem of mirror surface recovery.

In the third part of this thesis, a novel self-calibration method is introduced for single view diffuse surface reconstruction using an unknown mirror sphere. We first derive an analytical solution to recover the focal length of the camera given its principal point, and then introduce a robust algorithm to estimate accurate principal point and the focal length of the camera. Besides, we also present a novel approach for estimating both principal point and focal length of the camera when only a single image of the sphere is available. With the estimated camera intrinsics, the sphere position and a scaled 3D scene object can be obtained.

This thesis finally considers the problem of semantic correspondence estimation, which is crucial for 3D reconstruction as well as scene understanding. Most previous approaches to semantic correspondence focus on combining an effective spatial regularizer with hand-crafted features, or learning a correspondence model for appearance only. We proposed a convolutional neural network architecture, called SCNet, for learning a geometrically plausible model for semantic correspondence. SCNet uses region proposals as matching primitives, and explicitly incorporates geometric consistency.