Hierarchical semantics of objects (hSOs)

Abstract

We introduce hSOs: Hierarchical Semantics of Objects. An hSO is learnt from a collection of images taken from a particular scene category. The hSO captures the interactions between the objects that tend to co-occur in the scene, and hence are potentially semantically related. Such relationships are typically hierarchical. For example, in a collection of images taken in a living room scene, the TV, DVD player and coffee-table co-occur frequently. The TV and the DVD player are more closely related to each other than the coffee table, and this can be learnt from the fact that the two are located at similar relative locations across images, while the coffee table is somewhat arbitrarily placed. The goal of this paper is to learn this hierarchy that characterizes the scene. The proposed approach, being entirely unsupervised, can detect the parts of the images that belong to the foreground objects, cluster these parts to represent objects, and provide an understanding of the scene by hierarchically clustering these objects in a semantically meaningful way - all from a collection of unlabeled images of a particular scene category. In addition to providing the semantic layout of the scene, learnt hSOs can have several useful applications such as compact scene representation for scene category classification and providing context for enhanced object detection.