CONTEXTUALLY GROUNDED AFFECTIVE ANALYSIS OF MEDIA

Abstract

Endowing machines with the capability to sense and emote affect is a long-standing goal of AI. Over the past three decades, the field of Affective Computing has made significant progress in building both affect detectors and generators. However, most works have primarily explored affective learning from data in isolated forms. In contrast, human communication is multimodal and majorly occurs via interactions. Thus, training computational affective models in a contextual environment is an important problem.

This thesis explores the role of context in the affective analysis of user-generated media. We focus on conversational videos containing two major forms of context, conversation histories, and multimodal information. Affective understanding of such resources presents multiple challenges, including modeling interpersonal emotional influences in conversations, leveraging heterogeneous multimodal signals, and using sample-efficient models. We study these challenges in great detail and propose novel solutions on various affective tasks, including emotion, sentiment, and sarcasm analysis.