Geometric analysis for high-level shape understanding

Abstract

We live in a 3D world. Humans can effortlessly perceive the high-level structure of a 3D shape, which enables us to design, create and interact with various 3D objects in the real world. The last decade has witnessed tremendous developments in various 3D applications such as games, VR/AR and robotics, which drives the demand for structural understanding and content creation technologies. As these tasks rely on human perception, introducing supervision from humans becomes a direct solution. However, this requires expensive data annotation and is not enough to capture the complexity of countless shapes. Hence, exploiting the intrinsic properties of 3D shapes based on geometric analysis exhibits unique advantages. That being said, given that the geometric features of the high-level structures for arbitrary shapes are difficult to characterize, this problem remains challenging. In this thesis, we study three important aspects of high-level shape understanding based on geometric analysis: representation, segmentation and modeling.

We first introduce a method to generate structural representations, i.e., skeletons, from point clouds. Existing skeletonization methods are limited to tubular shapes and the stringent requirement of watertight input, while our method aims to produce more generalized skeletal representations to reflect the underlying structures for complex shapes and handle point clouds. Our key idea is to use the insights of the medial axis transform (MAT) to learn a geometric transformation to capture the intrinsic topological natures of the original input points. The generated skeletal representations become a suitable vehicle to exploit the high-level properties of 3D shapes.

Given such a skeletal representation, then, we propose a method to automatically decompose a 3D shape into structurally meaningful parts. Different from the existing methods that use low-level features on the surface meshes, we express the boundaries where different parts meet as high-level junctions on the skeletons. With the rich geometric and structural information encoded in the skeletons, we develop a simple and principled approach to efficiently segment an arbitrary 3D shape by detecting these junctions. Shape segmentation provides a structural understanding and part-based differentiation, of which insights can benefit many tasks for shape analysis.

Based on the structural shape understanding, finally, we explore how to enable machines to model 3D shapes like human modelers. To create a mesh model in 3D modeling software, a modeler needs to first understand the shape structure and identify its functional parts, and then edit the detailed geometry of each part to produce the target shape. Inspired by such artist-based modeling, we propose a two-step neural framework based on reinforcement learning (RL) to learn 3D modeling policies. The trained modeling agents have hierarchical understanding and can preserve high-level regularity within 3D shapes, which shows the potential for assisting human modelers and reducing content creation cost.

We conduct extensive experiments to evaluate the proposed methods, of which results demonstrate the superior effectiveness of using geometric analysis for high-level shape understanding and modeling.