The Likelihood Gain of a Language Model as a Metric for Text Summarization

The gain in the log-likelihood (LLG) of a text under a language model (LM) when the text's summary is provided as a context to the LM, compared to no summary in the context, has been proposed as a reference-free index for the relevance of the summary to the text. We provide an information-theoretic interpretation of the LLG and an empirical analysis of the parts of speech affecting it most. We first show that the LLG describes the reduction in the binary codelength when the summary text is provided as side information to a lossless text compression system involving the LM and an entropy encoder. Consequently, under proper normalization, LLG is a form of the Normalized Compression Distance (NCD) and thus adheres to a universal information distance that is motivated by algorithmic information theory. Empirical results show that an NCD based on LLG is better correlated with human annotators than a gzip-based NCD. Additionally, we empirically show that LLG is affected almost exclusively by tokens associated with the text's content rather than tokens associated with its structure. Our findings support LLG as a natural and useful metric for evaluating text summarization methods.