Neural LiDAR fields for resimulation and scene reconstruction

Abstract

LiDAR (Light Detection and Ranging) is a remote sensing method that uses laser pulses to capture precise distance measurements. By recording the time it takes for light to bounce off surfaces and return to the sensor, LiDAR generates dense 3D point clouds of the surrounding environment. Its high accuracy and efficiency make it a key technology in robotics applications such as navigation, obstacle avoidance, and localization. The rapid development of autonomous driving and related robotics technologies has led to an increasing demand for large-scale annotated datasets. However, the distribution of real-world data often follows a long-tail pattern, making it challenging to collect sufficient samples of rare corner cases. Simulators offer a promising solution by enabling the reconstruction and resimulation of such cases, thereby enriching the available data. Compared to traditional simulators that rely on manually created 3D assets, data-driven simulators are gaining attention for their ability to bridge the sim-to-real gap and eliminate the need for labor-intensive, handcrafted design. In this thesis, we leverage the realistic rendering capabilities of Neural Radiance Fields (NeRFs) to focus on LiDAR resimulation and scene reconstruction. We first present a novel LiDAR Fields framework for space-time view synthesis of dynamic urban scenes. We observe that existing methods relying on U-Net architectures to simulate ray-drop characteristics struggle to effectively capture this pattern. By leveraging LiDAR positional embeddings, we are able to render realistic LiDAR patterns in range images. However, due to the inherent sparsity and occlusions in Li- DAR data—particularly around dynamic objects—accurate and complete reconstruction remains a significant challenge. To address this, we estimate dense and consistent scene flow through a self-supervised loss, enabling improved modeling of dynamic motion and structure. Next, we develop a NeRF training pipeline that ensures consistent reconstruction across different sensing modalities while preserving their unique characteristics. Our design leverages the complementary nature of camera and LiDAR inputs to achieve more robust reconstruction, while also mitigating cross-modal noise that degrades the performance of each individual modality.