Please use this identifier to cite or link to this item: https://scholarbank.nus.edu.sg/handle/10635/23821
Title: RECOGNIZING LINGUISTIC NON-MANUAL SIGNS IN SIGN LANGUAGE
Authors: NGUYEN TAN DAT
Keywords: facial expression recognition, sign language recognition, tracking, conditional random field, KLT, PPCA
Issue Date: 13-Aug-2010
Citation: NGUYEN TAN DAT (2010-08-13). RECOGNIZING LINGUISTIC NON-MANUAL SIGNS IN SIGN LANGUAGE. ScholarBank@NUS Repository.
Abstract: Besides manual (hand) signs, non-manual signs (facial, head, and body behaviors) play an important role in sign language communication. In this thesis, we focus on recognizing an important class of non-manual signals in American Sign Language (ASL): grammatical markers which are facial expressions composed of facial feature movements and head motions and are used to convey the structure of a signed sentence. Six common grammatical markers are considered: Assertion, Negation, Rhetorical question, Topic, Wh question, and Yes/no question. While there have been attempts in the literature to recognize head movements alone or facial expressions alone, there are few works which consider recognizing facial expressions with concurrent head motion. However, in facial expressions used in sign language, meaning is jointly conveyed through both channels, facial expression (through facial feature movements), and head motion. We propose to track facial features through video, and extract suitable features from them for recognition. We first developed a novel tracker which uses spatio-temporal face shape constraints, learned through probabilistic principal component analysis, within a recursive framework. The tracker has been developed to yield robust performance in the challenging sign language domain where facial occlusions (by hand), blur due to fast head motion, rapid head pose changes and eye blinks are common. We developed a database of facial videos using volunteers from the Deaf and Hard-of-Hearing Federation of Singapore. The videos were acquired while the subjects were signing sentences in ASL. The performance of the tracker has been evaluated on these videos, as well as on videos randomly picked from the Internet, and compared with the Kanade-Lucas-Tomasi tracker and some variants of our proposed tracker with excellent results. Next, we considered isolated grammatical marker recognition using an Hidden Markov Model (HMM)-Support Vector Machine (SVM) framework. Several HMMs were used to provide the likelihoods of different types of head motions and facial feature movements. These likelihoods were then input to an SVM classifier to recognize the isolated grammatical markers. This yielded an accuracy of 91.76%. We also used our tracker and recognition scheme to recognize the six universal expressions using the CMU databse, and obtained 80.9% accuracy. While this is a significant milestone in recognizing grammatical markers, the ultimate goal is to recognize grammatical markers in continuously signed sentences. In the latter problem, simultaneous segmentation and recognition is necessary. The problem is made more difficult due to the presence of coarticulation effects and movement epenthesis. Here, we propose to use the discriminative framework provided by Condition Random Field (CRF) models. Experiments yielded precision and recall rates of 94.19% and 81.36%, respectively. In comparison, the scheme using sing-layer CRF model yielded precision and recall rates of 84.39% and 52.33%, and the scheme using layered HMM model yielded precision and recall rates of 32.72% and 84.06% respectively. In summary, we have advanced the state of the art in facial expression recognition by considering this problem with concurrent head motion. Besides its utility in sign language analysis, the proposed methods will also be useful for recognizing facial expressions in unstructured environments.
URI: http://scholarbank.nus.edu.sg/handle/10635/23821
Appears in Collections:Ph.D Theses (Open)

Show full item record
Files in This Item:
File Description SizeFormatAccess SettingsVersion 
BU_Alg1_xvid.avi89.32 kBAVI

OPEN

None Preview online
Talk_Alg2_xvid.avi570.25 kBAVI

OPEN

None Preview online
If_Alg1_xvid.avi463.92 kBAVI

OPEN

None Preview online
Talk_Alg1_xvid.avi564.2 kBAVI

OPEN

None Preview online
Talk_KLT_xvid.avi547.07 kBAVI

OPEN

None Preview online
NguyenTD.pdf3.03 MBAdobe PDF

OPEN

NoneView/Download

Google ScholarTM

Check


Items in DSpace are protected by copyright, with all rights reserved, unless otherwise indicated.