Improved world-based language model

Abstract

In order to construct a good language model used in the postprocessing phase of a recognition system, a smoothing technique must be used to solve the data sparseness problem. In the past, many smoothing techniques have been proposed. Among them, Katz's smoothing technique is well known. However, we found a weakness with the Katz's smoothing technique. We improved this approach by incorporating one kind of special Chinese language information and Chinese word class information into the language model. We tested the new smoothing technique with a Chinese character recognition system. The experimental result showed that a better performance could be achieved.