Beyong lexical meaning : probabilistic models for sign language recognition

Abstract

This thesis presents a probabilistic framework for recognizing multiple simultaneously expressed concepts in sign language gestures. These gestures communicate not just the lexical meaning but also grammatical information, i.e. inflections that are expressed through systematic spatial and temporal variations in sign appearance. In this thesis we present a new approach to analyse these inflections by modelling the systematic variations as parallel information streams with independent feature sets.We learn from data, the probabilistic relationship between lexical meaning and inflections, and the information streams; and then use the trained model to infer the sign meaning conveyed through observing features in multiple data streams. We propose a novel dynamic Bayesian network structure -- the Multichannel Hierarchical Hidden Markov Model which models the hierarchical, sequential and parallel organization in signing while requiring synchronization between parallel data streams at sign boundaries.