Toward robust, real-time, radiance map reconstruction with multi-sensor fusion

Abstract

Simultaneous Localization and Mapping (SLAM) plays a crucial role in the fields of robotics and automation, providing solutions for localization, feedback control, and environment mapping. It is widely utilized in Autonomous Ground Vehicles (AGVs), drones, self-driving cars, and has applications in Augmented Reality (AR), Virtual Reality (VR), Building Information Modeling (BIM), and etc.

SLAM systems generate maps representing the robot's environment, and the type of produced map depends on sensor inputs, computational resources, and specific system requirements. These maps can be sparse feature maps, semi-dense maps, dense point cloud maps, triangle meshes, or 3D radiance maps. The choice of map type depends on the application and scenario. For example, sparse visual feature maps are suitable for camera-based localization, where sparse image features are used to calculate the camera's pose. Dense point cloud maps are valuable for robot navigation and obstacle avoidance as they capture detailed geometric structures. Radiance maps, which contain both geometry and radiance information, are employed in various domains such as mobile mapping, AR/VR, video gaming, 3D simulation, and surveying. These applications require both geometric structures and textures to create realistic virtual environments. This thesis focuses on the robust and real-time reconstruction of radiance maps, encompassing both geometric structure and radiance information.

In the initial chapters, this thesis focuses on developing LiDAR-based SLAM systems and improving their robustness and accuracy in reconstructing the geometric structure of the environment. Firstly, the thesis introduces Loam_livox, which is the first LiDAR odometry and mapping system in academics specifically designed for solid-state LiDARs. Building upon this contribution, the thesis presents a decentralized framework for simultaneous calibration, localization, and mapping with multiple LiDARs. This proposed framework is based on an extended Kalman filter, but is specifically formulated for decentralized implementation.

In the subsequent chapters of this thesis, the focus shifts towards achieving the goal of radiance map reconstruction by working on the fusion of LiDAR and camera. Firstly, a novel open-source multi-sensor fusion framework, R2LIVE, is proposed. This framework combines LiDAR, inertial, and visual data, and exhibits robustness in various challenging scenarios. Based on the foundation laid by R2LIVE, this thesis presents R3LIVE++, a LiDAR-inertial-visual fusion framework designed to achieve robust and accurate state estimation while simultaneously reconstructing radiance maps in real-time. R3LIVE++ comprises two subsystems: a LiDAR-inertial odometry (LIO) subsystem that leverages LiDAR measurements to reconstruct the geometric structure, and a visual-inertial odometry (VIO) subsystem that recovers radiance information from input images. The R3LIVE++ framework stores radiance information at map points, which may lead to the loss of data due to the finite density of the point cloud. To overcome this limitation and recover radiance information without loss, this thesis introduces ImMesh, an online mesh reconstruction framework. ImMesh enables the real-time reconstruction of a surface triangle mesh. By applying texture from camera-captured images onto the mesh facets, the radiance information captured by the camera can be preserved in the map without any loss of fidelity.