Sign language phoneme transcription with PCA-based representation

Abstract

A common approach to extract "phonemes" of sign language is to use an unsupervised clustering algorithm to group the sign segments. However, simple clustering algorithms based on distance measures usually do not work well on temporal data and require complex algorithms. This paper presents a simple and effective approach to extract phonemes from American sign language (ASL) sentences. We first apply a semi-automatic segmentation algorithm which detects minimal velocity and maximal change of directional angle to segment the hand motion trajectory of signed sentences. We then extract, feature descriptors based on principal component analysis (PCA) to represent the segments efficiently. These high level features are used with k-means to cluster the segments to form phonemes. 25 continuously signed sentences from a native signer are used to perform the analysis. After phoneme transcription, we train Hidden Markov Models (HMMs) to recognize the sequence of phonemes in the sentences. We compare the recognition results from HMMs when the phonemes are labeled by our algorithm, and when they are labeled manually. On the 25 test sentences containing 173 segments, the average number of errors obtained with our approach and the manual approach to labeling phonemes was 24.0 and 33.8, respectively. © 2007 IEEE.