Bayesian learning of concept ontology for automatic image annotation

Abstract

Automatic image annotation (AIA) has been a hot research topic in recent years since it can be used to support concept-based image retrieval. In the field of AIA, characterizing image concepts by mixture models is one of the most effective techniques. However, mixture models also pose some potential problems arising from the limited size of (even a small size of) labeled training images, when large-scale models are needed to cover the wide variations in image samples. These potential problems could be the mismatches between training and testing sets, and inaccurate estimations of model parameters. In this dissertation, we adopted multinomial mixture model as our baseline and proposed a Bayesian learning framework to alleviate these potential problems for effective training from three different perspectives. (a) We proposed a Bayesian hierarchical multinomial mixture model (BHMMM) to enhance the maximum-likelihood estimations of model parameters in our baseline by incorporating prior knowledge of concept ontology. (b) We extended conventional AIA by three modes which are based on visual features, text features, and the combination of visual and text features, to effectively expand the original image annotations and acquire more training samples for each concept class. By utilizing the text and visual features from the training set and ontology information from prior knowledge, we proposed a text-based Bayesian model (TBM) by extending BHMMM to text modality, and a text-visual Bayesian hierarchical multinomial mixture model (TVBM) to perform the annotation expansions. (c) We extended our proposed TVBM to annotate web images, and filter out low-quality annotations by applying the likelihood measure (LM) as a confidence measure to check the b goodnessb of additional web images for a concept class. From the experimental results based on the 263 concepts of Corel dataset, we could draw the following conclusions. (a) Our proposed BHMMM can achieve a maximum F1 measure of 0.169, which outperforms our baseline model and the other state-of-the-art AIA models under the same experimental settings. (b) Our proposed extended AIA models can effectively expand the original annotations. In particular, by combining the additional training samples obtained from TVBM and re-estimating the parameters of our proposed BHMMM, the performance of F1 measure can be significantly improved from 0.169 to 0.230 on the 263 concepts of Corel dataset. (c) The inclusion of web images as additional training samples obtained with LM gives a significant improvement over the results obtained with the fixed top percentage strategy and without using additional web images. In particular, by incorporating the newly acquired image samples from the internal dataset and the external dataset from the web into the existing training set, we achieved the best per-concept precision of 0.248 and per-concept recall of 0.458. This result is far superior to those of state-of-the-arts AIA models.