Reinforcement Learning for Constraint-Aware Knowledge Base Question Answering

Language models (LMs) have demonstrated promising results as sophisticated artificial intelligence assistants. Recently, researchers have explored integrating LMs using external tools; however, grounding LMs to knowledge bases (KBs), which are complex environments, remains unexplored. Existing KB question answering (KBQA) models heavily rely on handcrafted agents that generate programs or refine generated programs following predefined rules to inject knowledge related to faithfulness and grammaticality of programs. However, these handcrafted agents can hinder the potential use of KBQA systems in real-world scenarios. Inspired by the recent success of aligning LMs with reinforcement learning (RL), a constraint-aware KBQA that injected knowledge about the faithfulness and grammaticality of programs in the training phase, thereby eliminating handcrafted agents in the decoding phase, was proposed in this study. The proposed method improved the supervised fine-tuning baseline by 0.4 exact match points and 0.4 F1 score points and generated 2.0% more valid queries in GrailQA. Our findings demonstrated that the proposed method enhanced the executability and validity of the generated query; these improvements can significantly aid the challenging compositional and zero-shot generalization settings in GrailQA.