Learning Linear and Nonlinear Low-Rank Structure in Multi-Task Learning

As the trace norm can discover low-rank structures in a matrix, it has been widely used in multi-task learning to recover the low-rank structure contained in the parameter matrix. Recently, with the emerging of big complex datasets and the popularity of deep learning techniques, tensor trace norms have been used for deep multi-task models. However, existing tensor trace norms exhibit some limitations. For example, they cannot discover all the low-rank structures in a tensor, they require users to manually specify the importance of each component in the corresponding tensor trace norm, and they only capture the linear low-rank structure. To solve the first issue, in this paper, we propose a Generalized Tensor Trace Norm (GTTN). The GTTN is defined as a convex combination of matrix trace norms of all possible tensor flattenings and hence it can discover all the possible low-rank structures. For the second issue, in the induced objective function with the GTTN, we propose four strategies to learn combination coefficients in the GTTN. Furthermore, we propose the Nonlinear GTTN (NGTTN) to capture nonlinear low-rank structure among all the tasks. Experiments on benchmark datasets demonstrate the effectiveness of the proposed GTTN and NGTTN.