VISUAL FEATURE BASED IMAGE/VIDEO INDEXING AND RETRIEVAL

Abstract

Developing a visual database supporting content-based retrieval requires the resolution of two key issues: defining suitable feature sets (along with their similarity metrics) and sorting (indexing) the elements of each feature set to support efficient retrieval. To date the first issue has received far more attention than the second. However, as these two issues are tightly related, suitable feature sets should be developed and evaluated under efficient indexing schemes. Without such a scheme, any visual content based image retrieval approach will lose its effectiveness. In the first part of the thesis, a clustering based indexing and retrieval scheme is proposed. Mainly one kind of well known unsupervised neural networks, Self-Organization Map (SOM), is adapted to realize the indexing of numerical feature data. Texture and color features are incorporated into the indexing scheme for evaluation. It is shown that both the efficient retrieval and hierarchical browsing can be supported in the scheme. At the same time, some new compound features, extraction methods, matching metrics as well as multiple feature query are developed and studied under the indexing scheme. Although many visual feature representation schemes have been developed for still images, there are much less representation schemes concerning activities occurring in videos. Based on optical flow, two types of motion features, directional distributions and speed distributions, are proposed in our study. From the tests, it is demonstrated that these features can be used to retrieval similar activities. It is also shown that the retrieval of perceptual similar video shots can be further facilitated by using motion features and image features together.