Object tracking for a class of dynamic image-based representations

Abstract

Image-based rendering (IBR) is an emerging technology for photo-realistic rendering of scenes from a collection of densely sampled images and videos. Recently, an object-based approach for rendering and the compression of a class of dynamic image-based representations called plenoptic videos was proposed. The plenoptic video is a simplified dynamic light field, which is obtained by capturing videos at regularly locations along a series of line segments. In the object-based approach, objects at large depth differences are segmented into layers for rendering and compression. The rendering quality in large environment can be significantly improved, as demonstrated by the pop-up lightfields. In addition, by coding the plenoptic video at the object level, desirable functionalities such as scalability of contents, error resilience, and interactivity with individual IBR objects, can be achieved. An important step in the object-based approach is to segment the objects in the video streams into layers or image-based objects, which is largely done by semi-automatic technique. To reduce the segmentation time for segmenting plenoptic videos, efficient tracking techniques are highly desirable. This paper proposes a new automatic object tracking method based on the level-set method. Our method, which utilizes both local and global features of the image sequences instead of global features exploited in previous approach, can achieve better tracking results for objects, especially with non-uniform energy distribution. Due to possible segmentation errors around object boundaries, natural matting with Bayesian approach is also incorporated into our system. Using the alpha map and texture so estimated, it is very convenient to composite the image-based objects onto the background of the original or other plenoptic videos. Furthermore, a MPEG-4 like object-based algorithm is developed for compressing the plenoptic videos, which consist of the alpha maps, depth maps and textures of the segmented image-based objects from different video plenoptic streams. Experimental results show that satisfactory renderings can be obtained by the proposed approaches.