Multimodal representation: Kneser-ney smoothing/skip-gram based neural language model

For image retrieval and caption generation, this paper considers a multimodal representation that associates image with its text description (caption) by defining a neural language model as the conditional probability of the next word given both n past words in a caption and the image that the caption describes. To address the data sparsity problem, the use of the Kneser-Ney smoothing and skip-gram models is examined by integrating each into the multimodal neural language model. A language model (LM) known as Kneser-Ney smoothing is based on absolute-discounting interpolation while skip-gram LM is based on n-grams organized by allowing intermediate tokens to be “skipped”. The multimodal representation is evaluated on the IAPR TC-12 dataset. Using perplexity and BLEU-n measures, both Kneser-Ney smoothing and skip-gram models are demonstrated to be more effective as approaches to addressing the data sparsity problem than the generic n-gram model used in previous multimodal representations. The modality-biased log-bilinear (MLBL-B) model is set as the base model in the experiment.