Multi-view reconstruction via global 3D representation

Abstract

Humans naturally engage with three-dimensional content in various applications such as virtual reality, movie visual effects, and gaming. The creation of high-quality 3D assets remains challenging and costly, demanding specialized software and skilled labor. The rise of RGB cameras has democratized access to 3D modeling, focusing on reconstruction solely from images, thereby making it more accessible to the general public. This thesis focuses on the task of multi-view 3D reconstruction from RGB images. The traditional multi-view reconstruction pipeline involves long and complex processing components based on local depth map representations. It operates in an open-loop fashion and is error-prone.

In contrast, in this thesis, our focus lies on addressing the multi-view reconstruction problem through global 3D reconstruction. A proper global 3D representation should directly reflect the contents of the input RGB data and can be optimized or estimated based on the reconstruction error observed in the input images. This inherent simplicity allows for more robust and higher-quality reconstructed results compared to traditional 3D reconstruction pipelines. However, challenges persist in designing the form of 3D representations and reconstruction algorithms. In this thesis, we explore various forms of 3D representations and reconstruction algorithms that are suited for different image capture settings.

In the first part of the thesis, we address the challenge of reconstructing complex thin structures from RGB video frames without the need for camera pose input. We develop a global 3D curve graph representation and simultaneously optimize both the camera poses and curve representations to align with the input images. This optimization process incorporates tailored measures for geometry, topology, and self-occlusion handling, thereby facilitating the reconstruction of 3D thin structures.

While this global curve representation performs well in reconstructing thin structures, it lacks the capability to reconstruct general surfaces. In the second part, we address this issue by employing a neural signed distance function (SDF) to represent surface geometry. Building upon this neural surface representation, we introduce a specialized differentiable volume rendering technique tailored for this surface representation. This enhancement aims to make the reconstruction process more robust compared to previous methods of neural surface reconstruction.

In the third part, we propose a solution to address the challenge of reconstructing and synthesizing novel views for large-scale, unbounded scenes. Our approach involves representing the scene using a grid-based neural radiance field (NeRF), which enables the accommodation of arbitrary input camera trajectories while requiring only a few minutes for training.

In the fourth part, we introduce PF-LRM, a large reconstruction model for generalized sparse-view object reconstruction. PF-LRM utilizes tri-plane tokens to encode object neural radiance fields. This highly scalable method leverages self-attention blocks to enable effective information exchange between 3D object tokens and 2D image tokens, consistently outperforming baseline methods in accurately predicting pose and enhancing 3D reconstruction quality across diverse evaluation datasets.