A Tree-Structured Multitask Model Architectures Recommendation System

Neural networks with branched architectures, namely, tree-structured models, have been employed to jointly tackle multiple vision tasks in the context of multitask learning (MTL). Such tree-structured networks typically start with a number of shared layers, after which different tasks branch out into their own sequence of layers. Hence, the major challenge is to determine where to branch out for each task given a backbone model to optimize for both task accuracy and computation efficiency. To address the challenge, this article proposes a recommendation system that, given a set of tasks and a convolutional neural network-based backbone model, automatically suggests tree-structured multitask architectures that could achieve a high task performance while meeting a user-specified computation budget without performing model training. Extensive evaluations on popular MTL benchmarks show that the recommended architectures could achieve competitive task accuracy and computation efficiency compared with state-of-the-art MTL methods. Our tree-structured multitask model recommender is open-sourced and available at https://github.com/zhanglijun95/TreeMTL.