Leveraging Transfer Learning: Fine-Tuning methodology for Enhanced Text Classification using BERT

Text Classification is the most widely studied problem area in Natural Language Processing (NLP). BERT is the most popular NLP model based on Transfer Learning with its pre-trained model being widely used for such tasks. It generates contextualized word/ sentence embeddings by applying "Self-Attention and Feed Forward" mechanism through multiple layers. The [CLS] token is the only token in BERT which captures semantically-relevant context for the complete input sentence. BERT-based language models output [CLS] token embedding which represents the input sentence effectively, however, the need for more efficient and accurate sentence embedding still remains a research area in NLP. This paper is to explore the effectiveness of [CLS] token pooled from intermediate encoder layers of the BERT model using Mean and Max pooling methods on Amazon product review dataset. Results obtained during the experiment clearly shows the superiority of CLS pooling strategies over default [CLS] tokens.