Channel Attention-Based Method for Searching Task-Specific Multi-Task Network Structures

Multi-task learning (MTL) simultaneously learns multiple related tasks, aiming to improve overall outcomes by leveraging task correlations. The MTL network structure choice greatly impacts individual task results. In contrast, Evolutionary Neural Architecture Search (ENAS) has shown remarkable proficiency in automatically identifying optimal neural network structures for MTL. However, existing methods present certain limitations. Manually designed multi-task learning networks are constrained by the designer’s skill, and they often fail to fully capture multi-task feature interactions within the backbone network. Moreover, these networks do not incorporate the residual concept to preserve crucial backbone network output features. In contrast, automatically searched structures embrace residuals but still need improvement in facilitating multi-task feature interactions. This paper introduces a novel framework for evolutionary MTL network search. It includes a backbone with residuals and a Multi-Task Feature Fusion and Extraction Module (FFEM) with channel attention. The FFEM combines task features and weighs channel(feature) importance. Adopting the residual concept reintegrates them into the backbone network and fuses with the backbone network’s features. Experimental results validate our approach’s efficacy, achieving competitive performance on the NYUD-v2 dataset.