Context and adversarial learning in low-level vision

Abstract

In recent years, with increasing network bandwidth and the popularity of portable cameras, massive video blogs and photographs have been produced and uploaded to webcam sites and social media. It is important to develop automated, intelligent and robust tools for digital image processing. How to harvest and exploit spatio-temporal contexts in visual data to achieve high-quality image processing is also an essential problem. On one hand, adversarial learning based attacks could threaten deep learning based models. On the other hand, generative adversarial networks can synthesize realistic image patches via adversarial learning. It is necessary to improve existing image processing tools with generative adversarial networks and against adversarial attacks. Thus, we propose three solutions based on context and adversarial learning for three low-level vision tasks: video salient object detection, robust salient object detection against adversarial attacks and image inpainting.

For detecting salient objects in a video, we introduce a motion guided attention neural network that is integrated with a family of motion guided attention modules. The proposed motion guided attention modules model how motions computed from temporal contexts determine object saliency. The proposed neural network contains a motion branch and an appearance branch, which work by computing intermediate representations and a final saliency map for a pair of optical flow and color images. Our proposed algorithm significantly surpasses the state-of-the-art video salient object detection methods.

For robust salient object detection resistant to adversarial samples, we develop a novel generic framework that boosts the robustness of arbitrary salient object detection models based on fully convolutional networks. The proposed salient object detection framework adopts a segment-wise shielding component to destroy adversarial perturbations in an input image. The framework includes a context-ware restoration component that uses spatial contextual correlations to refine saliency maps. Experimental results demonstrate that our proposed salient object detection framework performs better than other defenses against adversarial attacks.

For semantic image completion, we propose a context-aware semantic inpainting algorithm and two novel evaluation metrics. The proposed algorithm consists of a fully convolutional generative network and a context-aware joint loss function. The generative network resorts to fully convolutional architecture without fully connected bottlenecks to maintain structural features. The joint loss function forces the proposed network to synthesize contents with the same semantic as the surrounding context. Both quantitative and qualitative comparisons suggest that our proposed algorithm obtains state-of-the-art performance. The proposed metrics can better rate an over-smoothed result and the overall semantic of an image.