Please use this identifier to cite or link to this item:
https://doi.org/10.3233/IDA-2012-0544
Title: | Interactive state-transition diagrams for visualization of multimodal annotation | Authors: | Podlasov, A. Tan, S. O'Halloran, K. |
Keywords: | applications multimodal analysis multimodal annotation State-transition diagrams visual analytics visualization |
Issue Date: | 2012 | Citation: | Podlasov, A., Tan, S., O'Halloran, K. (2012). Interactive state-transition diagrams for visualization of multimodal annotation. Intelligent Data Analysis 16 (4) : 683-702. ScholarBank@NUS Repository. https://doi.org/10.3233/IDA-2012-0544 | Abstract: | Multimodal Analysis is a branch of Social Semiotics that studies how different resources, such as language, gesture, music, imagery, cinematography, etc. are used in modern media, and by humans in general, to construct and communicate meaning. Though theoretical aspects of multimodal analysis are well developed in the literature, and there is progress in the development of annotation software, the interpretation of annotated data still rarely uses algorithmic and data visualization approaches, limiting practical research in multimodal analysis. In the current work we propose how time stamped, tier-based multimodal annotations can be seen as finite state automatons and visualized as state-transition diagrams. We propose an algorithm converting time stamped annotation into a state-transition diagram and discuss how the resulting state-transition diagram can be visualized using freely available graph visualization tools. Then, we discuss common drawbacks of the existing tools, and propose an interactive visualization software tool allowing the analyst to synchronize the annotation state-transition diagram with the original media file. Finally, a case study on how the proposed visualization tools can assist media researchers in detecting patterns in multimodal analyses of news videos is presented. © 2012 - IOS Press and the authors. All rights reserved. | Source Title: | Intelligent Data Analysis | URI: | http://scholarbank.nus.edu.sg/handle/10635/115156 | ISSN: | 1088467X | DOI: | 10.3233/IDA-2012-0544 |
Appears in Collections: | Staff Publications |
Show full item record
Files in This Item:
There are no files associated with this item.
Items in DSpace are protected by copyright, with all rights reserved, unless otherwise indicated.