By Topic

Language Modeling with the Maximum Likelihood Set: Complexity Issues and the Back-off Formula

Sign In

Cookies must be enabled to login.After enabling cookies , please use refresh or reload or ctrl+f5 on the browser for the login options.

Formats Non-Member Member
$33 $13
Learn how you can qualify for the best price for this item!
Become an IEEE Member or Subscribe to
IEEE Xplore for exclusive pricing!
close button

puzzle piece

IEEE membership options for an individual and IEEE Xplore subscriptions for an organization offer the most affordable access to essential journal articles, conference papers, standards, eBooks, and eLearning courses.

Learn more about:

IEEE membership

IEEE Xplore subscriptions

2 Author(s)
Damianos Karakos ; Center for Language and Speech Processing, Johns Hopkins University, Baltimore, MD, USA. ; Sanjeev Khudanpur

The maximum likelihood set (MLS) was recently introduced in B. Jedynak and S. Khudanpur (2005) as an effective, parameter-free technique for estimating a probability mass function (pmf) from sparse data. The MLS contains all pmfs that assign merely a higher likelihood to the observed counts than to any other set of counts, for the same sample size. In this paper, the MLS is extended to the case of conditional pmf estimation. First, it is shown that, when the criterion for selecting a pmf from the MLS is the KL-divergence, the selected conditional pmf naturally has a back-off form, except for a ceiling on the probability of high frequency symbols that are not seen in particular contexts. Second, the pmf has a sparse parameterization, leading to efficient algorithms for KL-divergence minimization. Experimental results from bigram and trigram language modeling indicate that pmfs selected from the MLS are competitive with state-of-the-art estimates

Published in:

2006 IEEE International Symposium on Information Theory

Date of Conference:

9-14 July 2006