TYMBERT: Tiny Yet Mighty– Fine-Tuned and Compressed TinyBERT for High-Accuracy, Low-Resource NLP

Deploying large-scale NLP models on edge devices remains a challenge due to their high computational and storage requirements. This study aims to optimize TinyBERT by using knowledge distillation and quantization techniques to enhance its efficiency while maintaining high accuracy. We distill knowledge from BERT-base-uncased into TinyBERT and apply quantization (fp16) to reduce model size. Our proposed model, TYMBERT, achieves a 15x size reduction from the base BERT model while preserving high accuracy, with an AUC score of 0.999535 - outperforming MobileBERT by one-third its size and matching the performance of the original tinyBERT with half its size. Additionally, TYMBERT enables real-time NLP applications on low-power devices, particularly for SMS spam detection in resource-constrained environments. This research contributes to the development of efficient NLP models, suggesting future explorations in more aggressive quantization and pruning techniques to further optimize performance.