Establishing Rapport with Conversational Agents: Comparing the Effect of Envelope and Emotional Feedback

Abstract

Conversational agents have become increasingly sophisticated in interacting with humans since their early days as embodied interfaces. One way to improve the design of conversational agents is to study rapport ? that feeling of being able to `click? with someone ? and how agents can be designed to build rapport with humans. So far, most research on rapport using agents has focused on envelope feedback ? nonverbal behaviours that facilitate the process of communication without reference to the content of the conversation. This thesis will examine the effect of emotional feedback ? nonverbal behaviours that indicate an emotional response to the content of a conversation ? in an agent, and how this can affect rapport with humans in a storytelling scenario.

Thirty-six people took part in an experiment in which they re-told a sequence of events they had witnessed to an agent that was capable of producing both appropriate and inappropriate facial expressions. The rapport between the human and the agent was then measured through the duration of the story being told, the fluency of the speaker, and the self-reported feelings of rapport by the speaker. Results showed that inappropriate emotional feedback (in the form of facial expressions) caused the duration of the interaction to increase, which was the opposite of earlier studies on envelope feedback on rapport. Possible explanations for this effect could be attributed to emotional feedback?s greater impact on the conversational grounding process and on the speakers? social anxieties, or by language barriers. This study shows that emotional feedback does have an impact on rapport, and in a way that is different from envelope feedback, and thus makes it an important factor to consider in the design of conversational agents.