Adaptive Weight Generator for Multi-Task Image Recognition by Task Grouping Prompt

Comparing to adapting the pre-trained backbone to a single image recognition task, multi-task image recognition enables the backbone to perform better when the tasks are related. An interesting research field in multi-task learning (MTL) is to learn the parameter sharing pattern among the involved tasks. Most existing works obtain the sharing pattern, ignoring the task grouping information among the involved tasks. In this work, we aim to build the task parameter sharing pattern based on automatically acquiring the task grouping information. The task grouping information together with the task specific information is then utilized to yield the task adaptive weights. Our method, called Task Grouping prompt-based Adaptive Weight generator (TGAW), consists of Prompt-based Task Representation (PTR) and Prompt-based Weight Generator (PWG). The PTR is modeled as task prompts consisting of task grouping prompt and task specific prompt. The task grouping prompt is automatically chosen from a candidate pool for each task, and tasks selecting the same grouping prompts are divided into the same group. Then, PWG generates task adaptive weights based on the task prompts. The experimental results show that TGAW achieves comparable performance with less than 30% amount of trainable parameters of the pre-trained backbone.