Scale-Aware Task Message Transferring for Multi-Task Learning

Exploring cross-task interaction has been the mainstream in recent multi-task learning for dense predictions. However, existing works that focus on excavating cross-task contextual information are briefly based on hierarchical all-scale features, ignoring the complementary information from task relationships in multi-scale features. Meanwhile, recent research advances rarely pay attention to efficient information transformation between tasks. In this paper, we propose a novel end-to-end multi-task learning network termed as Scale-Aware Task Message Transferring Network (SATMTN) to explore the task relationships in multi-scale features to enrich the contextual cross-task information in the all-scale features. Specifically, we design Multi-Scale Message Passing Decoder (MSMP) to model interaction information between tasks in the multi-scale features, which is characterized by stacked bi-directional fully connected graph neural networks. Further, we devise another All-Scale Message Passing decoder (ASMP) to extract rich contextual information between tasks in the all-scale features, which is based on adaptive message transformation in the graph. Extensive experiments are implemented to show that our method surpasses the current state-of-the-art works on public multi-task benchmarks.