Comparative Study of Bert Models and Roberta in Transformer based Question Answering

Using deep learning technique, transformer-based self-supervised pre-trained models have revolutionised the idea of transfer learning in natural language processing (NLP). The self-attention mechanism makes transformers more prevalent in transfer learning across broad range of NLP tasks. In this study, seven prominent models are compared., including the Bertbase- uncased, Distilbert-base-cased, Distilbert-base-uncased, and Roberta models, in terms of their effectiveness (using 3 epochs). Bert base cased, Bert-medium-squad2-distilled, and Electra-basesquad2 on the Stanford Question Answering Dataset were utilised models with two epochs (SQuAD). The analysis shows that in three epochs, Roberta models provide the most accuracy, while Distilbert-base cased models provide the maximum accuracy when compared to Bert-base uncased, Distilbert-base uncased, and Distilbert base cased models. While Electra-base-squad2 performs better than Bert base cased and Bert-medium-squad2- distilled in cases with two epoches. Although all Bert and Roberta models take a long time to run, increasing accuracy requires more time to train the dataset.