A Hierarchical Structured Multi-Head Attention Network for Multi-Turn Response Generation

As a crucial task in conversation systems, response generation for multi-turn conversation aims to generate a coherent, informative and diverse response according to the conversation context. Existing models of this task are limited in their ability to capture long-term dependencies within and between utterances and to identify pertinent or important information in the context. Inspired by the Transformer neural network based solely on attention mechanisms recently proposed in machine translation, we propose a novel hierarchical structured multi-head attention network (HMAN) model to address both problems. Specifically, the context sequences are encoded in a hierarchical structure in which multi-head self-attention is first employed to compute the representation matrix of each utterance, and then, these utterance matrices are integrated to form a context representation with the complicated dependencies learned using multi-head attention. The experimental results on two public conversation datasets demonstrate that our proposed model significantly outperforms several state-of-the-art baselines with respect to both automatic evaluation and human evaluation and can generate more diverse and informative responses.