Nonoverlapping Feature Projection Convolutional Neural Network With Differentiable Loss Function

We propose NullSpaceNet, a novel network that maps from the pixel-level image to a joint-nullspace, as opposed to the traditional feature space. The features in the proposed learned joint-nullspace have clearer interpretation and are more separable. NullSpaceNet ensures that all input images that belong to the same class are collapsed into one point in this new joint-nullspace, and the input images of different classes are collapsed into different points with high separation margins. Moreover, a novel differentiable loss function is proposed that has a closed-form solution with no free parameters. NullSpaceNet architecture consists of two components; 1) a feature extractor backbone (i.e., the convolution and pooling layers), which is used to extract features from the input, and 2) a nullspace layer, which maps from the pixel-level image to the joint-nullspace. This novel architecture and formulation results in a significant reduction in the number of learnable parameters in the network. NullSpaceNet is architecture-agnostic, which means it can use any feature extractor as a backbone in its first component. NullSpaceNet exhibits superior performance when tested over four different datasets against VGG16, MobileNet-224, and MNASNET1-0. In general, NullSpaceNet needs only
$\text{1}\hbox{--}\hbox{30}\%$
of the time it takes a traditional CNN to classify a batch of images, and with better accuracy of up to
$+2.57\%$
.
Impact Statement–
Convolutional neural networks (CNNs) have achieved excellent performance in most computer vision tasks. However, the current formulation of CNN lacks a clear interpretation of the learned features in the feature space. Moreover, most learnable parameters are located in the classifier component (i.e., the fully connected layers), which require extensive computations during training and inference. We propose a novel feature space called NullSpaceNet. We provide theoretical and experimental evidence in NullSpaceNet that the learned nullspace features are more discriminative than the feature space. Moreover, the new formulation significantly decreases the number of parameters up to 86% and with better accuracy by up to
$+2.57\%$
. NullspaceNet sets a new line of architecture formulation research by improving the performance while decreasing the number of learnable parameters. Computer vision and deep learning communities will benefit from NullSpaceNet formulation, as it is architecture-agnostic.