Image interpolation and image-based content summary

Abstract

With the prevalence of smartphones and modern digital cameras, people have access to digital photo collections and videos more often than before. Organizing and browsing a considerable amount of image data becomes a challenging problem. This thesis mainly focus on gradient guided image interpolation for single images and compact summarization for videos and photo collections using geometric-based method.

This thesis first investigates the problem of image enlargement for a single image, which aims to improve image details when zooming. A novel edge prior is proposed for image interpolation which assumes that the variation of image intensity away from an edge is locally similar along the edge. This new approach performs significantly better than several representative edge-directed image interpolation methods in terms of PSNR and SSIM, as well as subjective visual quality in most of the test cases.

The second part of this thesis studies the comic-style presentation for videos. A new approach that conveniently converts conversational videos into comics with manga-style layout in a content-driven manner is introduced. The main components, including panels and word balloons, that constitute a visually pleasing comic page are intelligently organized. The information contained in a comic page is qualitatively measured and an efficient Markov chain Monte Carlo sampling algorithm is designed for the proposed optimization. A user study demonstrates that users much prefer the produced manga-style comics to purely Western style comics. Extensive experiments and comparisons against previous work also verify the effectiveness of the proposed approach.

The third part of this thesis presents a new framework for producing a visually appealing photo collage from a photo collection. The core of this framework is a novel divide-and-conquer photo collage approach. While conventional photo collage methods are often built upon complex non-linear optimization systems, the proposed approach divides the canvas from the geometric point of view and assigns each image an independent area to display in a purely saliency-driven manner. To seek an optimal partition of the canvas, a new region partition algorithm which accepts the irregular salient image regions as input has been developed. Each of the subregions resulting from the partition fits its corresponding salient region well. This leads to better utilization of the limited canvas space, thus yielding a more compact yet informative collage. Moreover, a seam carving based scheme is devised to complete the collage result. Extensive experiments and comparisons against state-of-the-art methods are conducted. A user study shows that the generated collage outperforms competitive methods.