Text and content based image retrieval via locality sensitive hashing

Abstract

We present a scalable image retrieval system based jointly on text annotations and visual content. Previous approaches in content based image retrieval often suffer from the semantic gap problem and long retrieving time. The solution that we propose aims at resolving these two issues by indexing and retrieving images using both their text descriptions and visual content, such as features in colour, texture and shape. A query in this system consists of keywords, a sample image and relevant parameters. The retrieving algorithm first selects a subset of images from the whole collection according to a comparison between the keywords and the text descriptions. Visual features extracted from the sample image are then compared with the extracted features of the images in the subset to select the closest. Because the features are represented by high-dimensional vectors, locality sensitive hashing is applied to the visual comparison to speedup the process. Experiments were performed on a collection of 1514 images. The timing results showed the potential of this solution to be scaled up to handle large image collections.