Fast and high-quality 3D reconstruction and structural understanding

Abstract

Surface reconstruction from multi-view images is a long-standing problem in computer graphics and computer vision, which is demanded in many downstream tasks, such as digital documentation of cultural heritage, robotics, and AR/VR. Although intensive work has been done on this task, fast and accurate 3D surface reconstruction remains an outstanding problem. Additionally, with the development of techniques of surface reconstruction, it becomes easier to reconstruct 3D surfaces of shapes. However, generalized structural understanding of 3D shapes is still under-explored. 3D shapes, especially, man-made objects, are naturally organized in parts and hierarchies; such structures provide important constraints for shape reconstruction and generation, and significant clues for shape recognition, analysis, and modeling. Thus, in this thesis, we focus on fast and accurate 3D surface reconstruction from multi-view images and structural understanding of 3D shapes.

In the first part of this thesis, we address the challenge of fast and high-quality reconstruction of large-scale archaeological fragments for digital documentation. We develop an automatic and efficient reconstruction pipeline. Given multi-view images of the front and back sides of fragments, we first leverage the traditional multi-view stereo (MVS) techniques to reconstruct the geometry of the two sides respectively. Then, a new batch-based matching algorithm and a novel Bilateral Boundary ICP algorithm to pair and register the two sides of the fragments are developed for fast and efficient reconstruction of complete 3D models. With a carefully designed hardware system, the developed system has a high throughput of imaging over 700 fragments per day (8 working hours) with 3D reconstruction accuracy of 0.16mm.

Although traditional MVS techniques can reconstruct high-quality fragments, for large-scale indoor scenes, the large texture-less regions pose challenges for these methods due to the shape ambiguity in 3D reasoning. We propose a new method for high-quality reconstruction of indoor scenes building upon the coordinate-based neural scene representations representation, by integrating estimated normal vectors of indoor scenes as a prior in a neural rendering framework via an adaptive manner. However, neural scene representation usually suffer from slow training/inference speed.

In the third part, we address the challenge of fast and accurate surface reconstruction from a set of unstructured 3D Gaussians via Gaussian Splatting. We present a novel approach for high-quality and fast surface reconstruction, including: 1) a global thickness control strategy to encourage irregular 3D Gaussians to evolve into flat shapes aligned with surface, and 2) guidance from multi-view consistency by leveraging traditional patch-match techniques to constrain the rendered depth from Gaussians. This design not only enables high-quality surface reconstruction but also accelerates the training process.

In the fourth part, to enable generalized structural understanding of 3D shapes, we present a unified structure rewriting system for both structured shape reconstruction and generation. We employ a local and probabilistic approach for structured shape modeling to achieve probabilistic modeling of ambiguous structures and robust generalization across object categories, which can also be generalized to structured shape completion and process multiple objects.