Mutual information-based depth estimation and 3D reconstruction for image-based rendering systems

Abstract

 Image-based rendering (IBR) is an emerging technology for rendering photo-realistic views of scenes from a collection of densely sampled images or videos. It provides a framework for developing revolutionary virtual reality and immersive viewing systems. There has been considerable progress recently in the capturing, storage and transmission of image-based representations. This thesis proposes two image-based rendering (IBR) systems for improving the viewing freedom and environmental modeling capability of conventional static IBR systems. The first system consists of a circular array with 13 still cameras (Canon 550D) for capturing ancient Chinese artifacts at high resolution. The second one is constructed by mounting a linear array of 8 video cameras (Sony HDR-TGIE) on an electrically controllable wheel chair with its motion being controllable manually or remotely through wireless local area network (LAN) by means of additional hardware circuitry.

Both systems support object-based rendering and 3D reconstruction capability and consist of two main components. 1) A novel view synthesis algorithm using a new segmentation and mutual information (MI)-based algorithm for dense depth map estimation, which relies on segmentation, local polynomial regression (LPR)-based depth map smoothing and MI-based matching algorithm to iteratively estimate the depth map. The method is very flexible and both semi-automatic and automatic segmentation methods can be employed. They rank fourth and sixth, respectively, in the Middlebury comparison of existing depth estimation methods. This allows high quality renderings of outdoor and indoor scenes with improved mobility/freedom to be obtained. This algorithm can also be extended to object tracking. Experimental results also show that the proposed MI-based algorithms are applicable to robust registration in noisy dynamic ultrasound images. 2) A new 3D reconstruction algorithm which utilizes sequential-structure-from-motion (S-SFM) technique and the dense depth maps estimated previously. It relies on a new iterative point cloud refinement algorithm based on Kalman filter (KF) for outlier removal and the segmentation-MI-based algorithm to further refine the correspondences and the projection matrices. The mobility of our system allows us to recover more conveniently 3D model of static objects from the improved point cloud using a new robust radial basis function (RBF)-based modeling algorithm to further suppress possible outliers and generate smooth 3D meshes of objects. Moreover, a new rendering technique named view dependent texture mapping is used to enhance the final rendering effect. Experimental results show that the proposed 3D reconstruction algorithm significantly reduces the adverse effect of the outliers and produces high quality renderings using view dependent texture mapping and the model reconstructed.

Overall, this study provides a framework for designing IBR systems with improved viewing freedom and ability to cope with moving and static objects in indoor and outdoor environment.