Design and implementation of the note-taking style haptic voice recognition for mobile devices

Abstract

This research proposes the \note-taking style" Haptic Voice Recognition (HVR) technology which incorporates speech and touch sensory inputs in a note-like form to enhance the performance of speech recognition. A note is taken from a user via two different haptic input methods - handwriting and a keyboard. A note consists of some of the key- words in the given utterance, either partially spelled or fully spelled. In order to facilitate fast input, the interface al- lows a shorthand writing system such as Gregg Shorthand. Using this haptic note sequence as an additional knowledge source, the algorithm re-ranks the n-best list generated by a speech engine. The simulation and experimental results show that the proposed HVR method improves the Word Error Rate (WER) and Keyword Error Rate (KER) performance in comparison to an Automatic Speech Recognition (ASR) system. Although it generates an inevitable increase in speech duration due to disfluency and occasional mistakes in haptic input, the compensation is shown to be less than conventional HVR methods. As such, this new note-taking style HVR interaction has the potential to be both natural and effective in increasing the recognition performance by choosing the most likely utterance among multiple hypotheses. This paper discusses the algorithm for the proposed sys- tem, the results from the simulation and the experiments, and the possible applications of this new technology such as aiding spoken document retrieval with haptic notes. Copyright 2012 ACM.