Learn Topological Representation with Flexible Manifold Layer

Deep neural networks for classification are one of the most fundamental topics in machine learning. However, the widely used softmax decision layer, originating from the conventional softmax regression for multi-class classification, fails to guide the powerful feature extractor to explore the topological structure hidden in data, which limits the quality of produced representations. Therefore, we propose a flexible manifold layer for better representation learning in this paper, rather than adding some regularized losses to introduce extra mechanisms. The flexible manifold layer is inspired by SVM which maximizes the geometric margin and usually achieves better results than softmax regression. Since the potential structure is captured due to the explicit guidance of the proposed flexible manifold layer, the noise can be easily detected and filtered out via a designed automatic mechanism. In addition to passing the topological knowledge by latent representations, a knowledge distillation from the manifold layer and decision layer is developed. Extensive experiments show that the proposed model has achieved excellent performance. The code is available at https://github.com/jzh9830/DeepSVM.