Harvesting visual concepts for image search with complex queries

Abstract

The use of image reranking to boost retrieval performance has been found to be successful for simple queries. It is, however, less effective for complex queries due to the widened semantic gap. This paper presents a scheme to enhance web image reranking for complex queries by fully exploring the information from simple visual concepts. Given a complex query, our scheme first detects the noun-phrase based visual concepts and crawls their top ranked images from popular image search engines. Next, it constructs a heterogeneous probabilistic network to model the relatedness between the complex query and each of its crawled images. The network seamlessly integrates three layers of relationships, i.e., the semantic-level, cross-modality level as well as visual-level. These mutually reinforced layers are established among the complex query and its involved visual concepts, by harnessing the contents of images and their associated textual cues. Based on the derived relevance scores, a new ranking list is generated. Extensive evaluations on a real-world dataset demonstrate that our model is able to characterize the complex queries well and achieve promising performance as compared to the state-of-the-art methods. Based on the proposed scheme, we introduce two applications: photo-based question answering and textual news visualization. Comprehensive experiments well validate the proposed scheme. © 2012 ACM.