Deep saliency detection and color sketch generation

Abstract

In recent years, with a wide spread of mobile devices with cameras, image has become an important medium for people to record and share their life, and has thus been witnessed a massive increase. Intelligent technique of image analysis and understanding, which focuses on extracting meaningful information from images, is becoming increasingly important. To keep up with its rapid development, the research and industry community has endeavored to develop advanced image analysis algorithms and their accompanying applications. This thesis demonstrates both novel algorithms in image analysis and a practical application system. It consists of two novel deep learning based salient object detection algorithms and a color sketch generation system.

For salient object detection, we present two different approaches. The first one formulates saliency detection as a segment-wise regression problem and introduces a neural network architecture to map each segment to a saliency score. The proposed neural network architecture consists of fully connected layers on top of CNNs responsible for feature extraction at three different scales. The second approach is a deep network which consists of two complementary components, a pixel-level fully convolutional stream and a segment-wise spatial pooling stream. The first stream directly produces a saliency map with pixel-level accuracy from an input image while the second stream extracts segment-wise features very efficiently, and better models saliency discontinuities along object boundaries. Finally, a fully connected CRF model can be optionally incorporated to improve spatial coherence and contour localization in saliency maps generated from both of the two proposed methods. Experimental results demonstrate that our two deep learning based saliency detection models significantly improve the state of the art.

For color sketch generation, we introduce an interactive drawing system, called ColorSketch, for helping novice users generate color sketches from photos. Our system is motivated by the fact that novice users are often capable of tracing object boundaries using pencil strokes, but have difficulties to choose proper colors and brush over an image region in a visually pleasing way. To preserve artistic freedom and expressiveness, our system lets users have full control over pencil strokes for depicting object shapes and geometric details at an appropriate level of abstraction, and automatically augment pencil sketches using color brushes, such as color mapping, brush stroke rendering as well as blank area creation. Experimental and user study results demonstrate that users, especially novice ones, can generate much better color sketches more efficiently with our system than using traditional manual tools.