MJR: Multi-Head Joint Reasoning on Language Models for Question Answering

Language Models (LMs) have achieved impressive success in various question answering (QA) tasks but have shown limited performance on structured reasoning. Recent research suggests that Knowledge Graph (KG) can augment text data by providing a structured background to enhance reasoning capabilities of LMs. Therefore, how to integrate and reason over KG representations and language context remains an open question. In this work, we propose MJR, a novel model to integrate encoded representations of LMs and graph neural network through multiple layers of feature interaction operations. Subsequently, the fused feature representations in two modalities are fed into a multi-head representation fusion module to comprehensively capture semantic and graph structure information, thereby enhancing language understanding and reasoning capabilities. In addition, we investigate the performance and applicability of different types of large language models as text encoder in the question-answering task. We evaluate our model on three common dataset: CommonsenseQA, OpenBookQA, and MedQA-USMLE datasets. The results demonstrate the advancements of MJR over existing LMs, LM+KG and LLMs models in reasoning for question answering.