Neural Symbol Grounding with Multi-Layer Attention for Robot Task Planning

The high-level symbolic representation has proven to be an effective expression for planning problems and is widely used in robot task planning. However, the grounding of symbol based on multimodal raw data in complex environment still remains a significant challenge. In this paper, we put forward a Multi-layer Attention Network for Multimodal Symbol Grounding (maMSG Net) where we combine the high level symbolic representation and multimodal perception effectively, improving the capability and accuracy of understanding complex environment and increasing the diversity of the symbol definition. Meanwhile, we introduce both the cross-modality attention and intra-modality attention in our neural network, which is demonstrated to improve the accuracy of symbol grounding. The maMSG Net takes multimodal raw data as input and estimates values of state symbols defined in given planning domain. We designed computer simulated experiments to evaluate the effectiveness of our method and verify its robustness against external interference.