Multimedia Decision Fusion

Abstract

The amount of multimedia data available on the Internet has increased exponentially in the past few decades and is likely to keep on increasing. Given multimedia's nature of having multiple information sources, fusion methods are critical for its data analysis and understanding. Multimedia fusion is a way to integrate multiple media, their associated features, or the intermediate decisions in order to perform an analysis task. It is useful for several objectives such as detection, recognition, identification, tracking, and decision making in many application domains. Multimedia fusion has been attracting increasing attention. However, some important issues in the multimedia fusion still need to be properly studied, such as how to utilize the correlation among different multimedia information sources, how to cope with the uncertainty and diversification of multimedia information, and how to adapt the fusion models to the conditions of changing and increasing amount of data.

This thesis proposes fusion methods that address the research challenges of proper utilization of the correlation among multimedia information sources. The thesis also addresses how to evolve the multimedia fusion model and improve the performance with new data. In MultiFusion, we make more use of the correlation among multimedia information sources by combining and utilizing the correlation in each iteration of an Adaboost-like structure. In portfolio fusion method, we maximize the return and minimize the risk (uncertainty) to achieve a high dependable performance by introducing the widely used and effective portfolio theory from finance. A more sophisticated model to utilize correlations among different information sources is also presented. For the situation that the multimedia data keep increasing with time and the nature of the data collection can change, we develop the Up-Fusion method. With the utilization of multimedia correlation and refinement, the method evolves the fusion model along with the newly added multimedia data to improve the performance. Moreover, the situations that the labels of newly added data are not available and that the context or nature of data changes, are also handled by using pseudo labels and sliding window. How to fuse the information sources most appropriately is also considered in this thesis. Based on the common practice of seeking opinions from specialists before making a decision, a specialist fusion method that adaptively predicts the expertise of different information sources on different data instances and effectively combines the expertise with decision is proposed in this thesis. The proposed fusion methods are mainly intended for classification and retrieval problems which are the main problems of multimedia applications.

To show the advantages and utility of our methods, simulation and real application experimental results are provided for each fusion method. Moreover, the fusion methods in the thesis aim to solve different objectives. The appropriate situations for different fusion methods are argued in the conclusion chapter. In the end, some limitations and broad vision for multimedia fusion methods are discussed.