Sparsing and Smoothing for the seq2seq Models

Current neural language models are trained to minimize cross-entropy and use softmax to compute the locally normalized probabilities over the target. While this setup provides solid results in several natural language processing (NLP) tasks, one unsatisfying aspect is its dense output. This density is wasteful, making models hard to interpret and assigning probability mass to many implausible outputs. To overcome this problem, we propose T-softmax, a simple but effective method to draw considerably sparse probability out of neural language models than softmax. Our method avoids dense output by truncating the unreliable tail of the probability distribution to improve the model's performance. In addition, we generalize logits with temperature, a critical regularization technique, from the softmax to T-softmax. To show our approach as a drop-in replacement for softmax, we evaluate them on three NLP tasks: summary generation, question answer, and math word problem. Experimental results show that our proposed model significantly improves performance without sacrificing speed; notably, in all experiments, our method outperforms the softmax.