Portfolio theory of multimedia fusion

Abstract

The number of multimedia applications has been increasing over the past two decades. Multimedia information fusion has therefore attracted significant attention with many techniques having been proposed. However, the uncertainty and correlation among different modalities have not been fully considered in the existing fusion methods. In general, the predictions of individual modality have uncertainty, furthermore, many modalities are correlated with each other. In this paper, we propose a novel multimedia fusion method based on the Portfolio theory. Portfolio theory is a widely used financial investment theory dealing with how to allocate funds across assets. The key idea is to maximize the performance of the allocated portfolio while minimize the risk in returns. We adapt this approach to multimodal fusion to derive optimal weights that can achieve good fusion results. The optimization is formulated as a quadratic programming problem. Experimental results with both simulated data and real data confirm the theoretical insights and show promising results. © 2010 ACM.