Multimodal alignment of scholarly documents and their presentations

Abstract

We present a multimodal system for aligning scholarly docu- ments to corresponding presentations in a fine-grained manner (i.e., per presentation slide and per paper section). Our method improves upon a state-of-the-art baseline that employs only textual similarity. Based on an analysis of base- line errors, we propose a three-pronged alignment system that combines textual, image, and ordering information to establish alignment. Our results show a statistically significant improvement of 25%, confirming the importance of visual content in improving alignment accuracy. Copyright © 2013 by the Association for Computing Machinery, Inc. (ACM).