Self-Attention and Retrieval Enhanced Neural Networks for Essay Generation

In this paper, we focus on essay generation, which aims at generating an essay (a paragraph) according to a set of topic words. Automatic essay generation can be applied to many scenarios to reduce human workload. Recently the recurrent neural networks (RNN) based methods are proposed to solve this task. However, the RNN-based methods suffer from incoherence problem and duplication problem. To overcome these shortcomings, we propose a self-attention and retrieval enhanced neural network for essay generation. We retrieve sentences relevant to topic words from corpus as material to assist in generation to alleviate the duplication problem. To improve the coherence of essays, the self-attention based encoders are applied to encode topic and material, and the self-attention based decoder are used to generate essay respectively. The final essay is generated under the guidance of topic and material. Experimental results on a real essay dataset show that our model outperforms state-of-the-art baselines according to automatic evaluation and human evaluation.