Object-based coding and watermarking for image-based rendering

Abstract

Image-based rendering (IBR) has emerged as an important technique in virtual reality, digital museums, interactive visualization in multi-view TVs and many other rapidly developing areas in the information and communication technology industry. IBR utilizes densely sampled two-dimensional images to generate novel views needed at different viewpoints to describe the three-dimensional scene. IBR representations (image-based representations) usually involve large data sizes; thus their efficient compression is vital for IBR’s practical use. As IBR becomes more widely applied in academia and industry, protecting various image-based representations becomes increasingly important to ensure its proper use and the author’s intellectual property. Digital watermarking is a promising way to solve this issue. Though previous research has studied compression of IBR, efficient system has not yet been fully investigated. Meanwhile, watermarking in IBR is still a new demanding area which needs effective schemes to be developed. This work focuses on object-based coding and feature-based watermarking for image-based representations.

Firstly, a multi-view object coding framework is proposed for image-based representations based on the Audio Video Coding Standard of China (AVS). Object-based coding compresses the IBR data (usually multi-view images/videos) at the object level. Image-based representations are first processed using object-based approach to segment and extract different objects within the data, each with their corresponding texture, depth map and shape information. The segmented objects are then compressed with state-of-the-art AVS coding techniques and tools. AVS-based object coding has the advantage of less complexity compared with H.264/AVC, while being more efficient than standardized object coding available in MPEG-4. The proposed framework supports multi-view coding to explore the redundancy between different views of the IBR data with efficient inter-frame and inter-view coding mode. Object-based temporal scalability is also achieved based on the proposed multi-view object coding framework.

Secondly, a novel two-pass rate control framework is proposed based on a non-linear exponential rate-distortion model. Convex optimization is utilized to allocate the optimal bits among different coding units at different levels. Region-of-interest is readily achieved through assigning different important factors to different objects. Rate control with object-based temporal scalability is also addressed for object-based adaptive transmission. At the same time, an analytical model-based bit-allocation approach is further proposed as a complement of convex optimization-based approach towards real-time applications.

Lastly, a feature-based watermarking system for copyright protection of image-based representations is developed. The proposed scheme uses scale invariant feature transform to extract robust feature points and formulate corresponding feature patches centered at the reference points in all the IBR views for watermark embedding. Discrete Fourier Transform coefficients of each patch are modified to embed a circular symmetric 2-D watermark pattern generated with a secret key. The watermark is synchronized by a hierarchical non-rigid image registration method to resist the effect of IBR and various geometrical attacks. Correlation-based detection is applied on each synchronized patch to determine the existence of the original inserted watermark pattern. The key advantage of the proposed watermarking method is that the watermark embedded into the original view can be detected from virtual views even after the rendering process.