Building class-based language models with contextual statistics

Abstract

In this paper, novel clustering algorithms are proposed by using the contextual statistics of words for class-based language models. The Minimum Discriminative Information (MDI) is used as a distance measure. Three algorithms are implemented to build bigram language models for a vocabulary of 50,000 words over a corpus of over 200 million words. The computational cost of algorithms and resulting LM perplexity are studied. The comparisons between the MDI algorithm and the Maximum Mutual Information algorithm are also given to demonstrate the effectiveness and the efficiency of the new algorithms. It is shown that the MDI approaches make the tree-building clustering[2] possible with large vocabulary.