EVI-SAM: Robust, Real-Time, Tightly-Coupled Event–Visual–Inertial State Estimation and 3D Dense Mapping

Abstract

Event cameras demonstrate substantial potential in handling challenging situations, such as motion blur and high dynamic range. Herein, event–visual–inertial state estimation and 3D dense mapping (EVI-SAM) are introduced to tackle the problem of pose tracking and 3D dense reconstruction using the monocular event camera. A novel event-based hybrid tracking framework is designed to estimate the pose, leveraging the robustness of feature matching and the precision of direct alignment. Specifically, an event-based 2D–2D alignment is developed to construct the photometric constraint and tightly integrated with the event-based reprojection constraint. The mapping module recovers the dense and colorful depth of the scene through the image-guided event-based mapping method. Subsequently, the appearance, texture, and surface mesh of the 3D scene can be reconstructed by fusing the dense depth map from multiple viewpoints using truncated signed distance function fusion. To the best of knowledge, this is the first nonlearning work to realize event-based dense mapping. Numerical evaluations are performed on both publicly available datasets, which qualitatively and quantitatively demonstrate the superior performance of our method. EVI-SAM effectively balances accuracy and robustness while maintaining computational efficiency, showcasing superior pose tracking and dense mapping performance in challenging scenarios.