Hybrid autoencoder and classifier for label-deficient semi-supervised learning: Case study

Label-deficient semi-supervised learning is a challenging setting in which there is an abundance of unlabeled data but a dearth of labeled data. A hybrid network that mixes an autoencoder, capable of extracting information from unlabeled data, and a neural network classifier, which incorporates information from labeled data, can be useful in a label-deficient setting. In this case study, we examine hybrid networks and their behaviors by imposing into them two-dimensional bottlenecks for the sake of visualization. We show that a straight-forward combination of the two objective functions fails in a label-deficient setting, and we propose a novel objective that emphasizes the autoencoding of labeled data in order to remedy the issue. We show the operation and the behaviors of hybrid networks trained with the proposed objective function when using five different unit nonlinearities, one of which is novel, and varying network depths in this case study. The proposed objective function trains networks that can achieve 79% accuracy on unseen testing data compared to a standard support vector classifier trained on the same labeled points which achieves 70% accuracy.