Cutout-search: Putting a name to the picture

Abstract

We often come across photographs with content whose identity we can no longer recall. For instance, we may have a picture from a football game we went to, but do not remember the name of the team in the photograph. A natural instinct may be to query an image search engine with related general terms, such as 'football' or football teams' in this case. This would lead to many irrelevant retrievals, and the user would have to manually examine several pages of retrieval results before he can hope to find other images containing the same team players and look at the text associated with these images to identify the team. With the growing popularity of global image matching techniques, one may consider matching the query image to other images on the web. However, this does not allow for ways to focus on the object-of-interest while matching, and may cause the background to overwhelm the matching results, especially when the object-of-interest is small and can occur in varying backgrounds, again, leading to irrelevant retrievals. We propose Cutout-Search, where a user employs an interactive segmentation tool to cut out the object-of-interest from the image, and use this Cutout-Query to retrieve images. As our experiments show, this leads to retrieval of more relevant images when compared to global image matching leading to more specific identification of the object-of-interest in the query image.