Transformation of Non-Euclidean Space to Euclidean Space for Efficient Learning of Singular Vectors

Singular value decomposition (SVD) is a popular technique to extract essential information by reducing the dimension of a feature set. SVD is able to analyze a vast matrix in spite of a relatively low computational cost. However, singular vectors produced by SVD have been seldom used in convolutional neural networks (CNNs). This is because the inherent properties of singular vectors such as sign ambiguity and manifold features make CNNs difficult to learn singular vectors. In order to overcome the limitations, this paper analyzes the undesirable properties of singular vectors and presents the transformation of singular vectors into Euclidean space as a smart solution. If the singular vectors are transformed to follow Euclidean geometry, SVD can be used for pooling to maintain the feature information well, which is called singular vector pooling (SVP). Since SVP can extract essential information from a feature map, it is robust against adversarial attacks in comparison to global average pooling. Thus, SVP shows a quantitative performance improvement of about 36% for the CIFAR10 dataset. In addition, we applied SVP to a knowledge distillation scheme that uses singular vectors in a restricted manner. As a result, SVP improved the performance by up to 1.7% for the CIFAR100 dataset.