An Alternative Hard-Parameter Sharing Paradigm for Multi-Domain Learning

Hard-parameter sharing in multi-domain learning (MDL) allows domains to share some model parameters in order to reduce storage cost while improving prediction accuracy. One traditional paradigm of the sharing practice borrows an idea from multi-task learning (MTL), which is to share bottom layers of a deep neural network among domains while using separate top layers for each domain. However, it is unclear whether the effectiveness of sharing bottom parameters in MTL can transfer to MDL or not. Therefore in this work, we revisit this common practice via an empirical study on image classification tasks on a diverse set of visual domains and make two surprising observations. (1) Using separate bottom-layer parameters could achieve significantly better performance than the common practice and this phenomenon holds for the different number of domains jointly trained on different backbone architectures with different quantities of domain-specific parameters. (2) A multi-domain model with a small proportion of domain-specific parameters from bottom layers can achieve competitive performance with independent models trained on each domain separately. Our observations suggest that people adopt the new paradigm of using separate bottom-layer parameters as a stronger baseline for model design in MDL.