A tensor-based approach for training flexible neural networks

Despite excellent prediction performance, state-of-the-art neural network (NN) architectures are very large, up to several millions of weights. In particular, running them on systems with limited computational capacity (embedded systems) becomes a difficult task. In this work, we focus on the compression of NNs, considering a tensor-based approach. In particular, we are interested in the estimation of flexible activation functions using tensor-based solutions. The proposed method is based on a coupled matrix-tensor factorization (CMTF), fusing the zeroth and first order information of the NN. The proposed learning algorithm is based on a constrained alternating least squares (ALS) approach. The activation functions (AFs) are expressed as a weighted sum of pre-defined basis elements. Our method allows to strongly compress large NN layers, with a very slight degradation of the classification accuracy. This also provides insights into the functioning of NNs, which is a problem of utmost interest in NN domain.