Probabilistic temporal multimedia datamining

Abstract

Advances in data acquisition and storage technology have led to the growth of very large multimedia databases. Analyzing this huge amount of multimedia data to discover useful knowledge is a challenging problem. This challenge has opened the opportunity for research in Multimedia Datamining (MDM), ?the process of finding interesting patterns from media data such as audio, video, image and text that are not ordinarily accessible by basic queries and associated results.? The motivation of doing MDM is to use the discovered patterns to improve decision making. MDM has therefore attracted significant research efforts in developing methods and tools to organize, manage, search and perform specific tasks for data from domains such as surveillance, meetings, broadcast television, sports, archives, movies, medical data, as well as personal and online media collections.

Existing MDM methods consider either low-level content features (e.g., color, texture etc.) or high-level text meta-data features (e.g., object, action etc.) for mining purposes. While the low-level features describe the actual content of the signal data they are unable to provide high level semantics of the mined data. Such, high level semantics are essential for applications like behavior analysis, semantic similarity etc. On the other hand, high-level text meta-data (e.g., tags, comments etc.) are capable of providing semantic interpretation for mining but they are noisy and require manual effort. However, existing MDM techniques assume that the automatically obtained labels (e.g., concepts, events etc.) from detectors are accurate. However, in reality detectors label the events/concepts from different modalities with a certain confidence measure over a time-interval. Therefore, it is important to consider the uncertainties associated with the detected concepts over time in the process of multimedia datamining.

This thesis proposes a framework for multimedia datamining which leverages on the probabilistic, temporal and multimodal characteristics of multimedia data. The proposed Probabilistic Temporal Multimodal (PTM) datamining framework for multimedia applications effectively handles issues like incorporating semantic knowledge, data sparsity in semantic representation of multimedia data, inaccuracy of binary concept detectors, dynamic temporal correlation etc. The utility of the proposed framework is demonstrated in the following three multimedia applications,

? Frequent event patterns for group meeting behavior analysis. ? Concept-based near-duplicate video clip clustering for novelty re-ranking of web video search results. ? Adaptive ontology rule based classification for composite concept detection.

Towards the end of the thesis, we present our conclusions and future research directions.