Please use this identifier to cite or link to this item:
Title: Towards Subject Independent Sign Language Recognition : A Segment-Based Probabilistic Approach
Keywords: sign language recognition, signer independence, Bayesian network, conditional random field (CRF), support vector machine (SVM), semi-Markov CRF
Issue Date: 25-Jul-2011
Source: KONG WEI WEON (2011-07-25). Towards Subject Independent Sign Language Recognition : A Segment-Based Probabilistic Approach. ScholarBank@NUS Repository.
Abstract: This thesis presents a segment-based probabilistic approach to recognize continuous sign language sentences which are signed naturally and freely. We aim to devise a recognition system that can robustly handle the inter-signer variations exhibited in the sentences. In preliminary work, we considered isolated signs which provided insight into inter-signer variations. Based on this experience, we tackled the more difficult problem of recognizing continuously signed sentences as outlined above. Our proposed scheme has kept in view the major issues in continuous sign recognition including signer independence, dealing with movement epenthesis, segmentation of continuous data, as well as scalability to large vocabulary. We use a discriminative approach rather than a generative one to better handle signer variations and achieve better generalization. For this, we propose a new scheme based on a two-layer conditional random field (CRF) model, where the lower layer processes the four parallel channels (handshape, movement, orientation and location) and its outputs are used by the higher level for sign recognition. We use a phoneme-based scheme to model the signs, and propose a new PCA-based representation phoneme transcription procedure for the movement component. k-means clustering together with affinity propagation (AP) is used to transcribe phonemes for the other three components. The basic idea of the proposed recognition framework is to first over-segment the continuously signed sentences with a segmentation algorithm based on minimum velocity and maximum change of directional angle. The sub-segments are then classified as sign or movement epenthesis. The classifier for labeling the sub-segments of an input sentence as sign or movement epenthesis is obtained by fusing the outputs of independent CRF and SVM classifiers through a Bayesian network. The movement epenthesis sub-segments are discarded and the recognition is done by merging the sign sub-segments. For this purpose, we propose a new decoding algorithm for the two-layer CRF-based framework, which is based on the semi-Markov CRF decoding algorithm and can deal with segment-based data, compute features for recognition on the fly, discriminate between possibly valid and invalid segments that can be obtained during the decoding procedure, and merge sub-segments that are not contiguous. We also take advantage of the information given by the location of movement epenthesis sub-segments to reduce the complexity of the decoding search. A glove-based approach was used for the work and raw data was obtained from electronic gloves and magnetic trackers. The data used for the experiments was contributed by seven deaf native signers and one expert signer and consisted of 74 distinct sentences made up from a 107-sign vocabulary. Our proposed scheme achieved a recall rate of 95.7% and precision accuracy of 96.6% for unseen samples from seen signers, and a recall rate of 86.6% and precision accuracy of 89.9% for unseen signers.
Appears in Collections:Ph.D Theses (Open)

Show full item record
Files in This Item:
File Description SizeFormatAccess SettingsVersion 
KongWW.pdf5.55 MBAdobe PDF



Page view(s)

checked on Dec 11, 2017


checked on Dec 11, 2017

Google ScholarTM


Items in DSpace are protected by copyright, with all rights reserved, unless otherwise indicated.