Facial emotion recognition using multi-modal information

Abstract

A hybrid method of facial emotion recognition is proposed using multi-modal information. By several intensive subjective evaluation studies, human beings recognize anger, happiness, surprise and dislike by their visual appearance as compared to voice only detection. When the audio track of each emotion clip is dubbed with a different type of auditory emotional expression, still anger, happiness and surprise were video dominant. However, dislike emotion gave mixed responses to different speakers. In both studies, sadness and fear emotions were audio dominant. The results of these studies would help in machine recognition of facial expressions by correctly integrating audio and video data.