AReLU: Agile Rectified Linear Unit for Improving Lightweight Convolutional Neural Networks

Dynamic activation functions usually gain remarkable improvements for neural networks. Dynamic activation functions depending on input features show better performance than the input-independents. But the improvements are achieved with extra memory and computational cost, which is non-negligible for lightweight convolutional networks. To address this issue, a lightweight input-dependent dynamic activation function is proposed, namely, Agile Rectified Linear Unit (AReLU). And Parallel Local Cross-Feature Interaction (PLCFI) is proposed as the lightweight feature extractor of AReLU. Considering the multi-channel characteristic of input features, Channel-Exclusive AReLU (CE-AReLU) and Channel-Shared AReLU (CS-AReLU) are designed based on PLCFI. CE-AReLU learns exclusive structure of activation function for each channel, while the functional structure of CS-AReLU is shared by channels. Furthermore, a specific initialization method for neurons in PLCFI is proposed to facilitate model convergence, called imitative initialization. Extensive experiments on different datasets and lightweight convolutional neural network architectures demonstrate the superiority and generality of AReLU. It shows that CE-AReLU gains remarkable performance in common visual classification benchmarks. Moreover, CS-AReLU shows its capacity of capturing dependencies among different samples in a batch, achieving higher recognition accuracy on fine-grained classification task.