Class Embedding: A BERT-Whitening and Attention-Based Approach to Text Classification

Text classification tasks often encounter setbacks when the amount of data is insufficient. At this moment, the model will enter a state of overfitting, so that it will get a disappointing result because it does not have a generalization ability in the performance of the testset. We propose a concept of class embedding method, which needn’t to train the model but a little attention network with only one-fiftieth of the total number in training sets. By using Bert-Whitening to reduce dimension and improved formula Attention-Euclidean to measure the distance between sentence and each class embedding, we have achieved satisfactory results on Chinese data sets. We propose a method with simple concepts and frameworks to be applied to text classification, and effective results are obtained with only a small number of samples. A comparative experiment on the Chinese text dataset shows that the model has a higher accuracy rate, which verifies the effectiveness of the method.