Improvement of Semantic-BERT Model with Bidirectional LSTM for Arabic Language

Large, pretrained transformer-based approaches like BERT have transformed the Natural Language Processing (NLP) field. BERT is a multilayer bidirectional Transformer trained on plaintext for masked word prediction and last sentence prediction. The pretrained BERT model can then be finetuned with task-specific training data for a subsequent task, The Improvement of BERT results are one of the main challenges that many studies focus on recently. In this work we proposed a Semantic-Bert model for Arabic Question/ Answer (Q/A) applications that incorporates explicit contextual semantics from pretrained semantic role labeling. It also introduces an enhanced language representation model, which is called Semantics aware BERT. The model is utilized bidirectional Long short-term memory LSTM (Bi-LSTM) to improve the correction in retrieving the answer when it selects the correct paragraph. The model is trained for 200 epoch and the results show that the proposed model achieves 0.75% and 80% in sensitivity. In addition, it achieves 61.5% and 90.4% in the sentence match. The selection of answering for most questions is highly accurate that most answers near from what need. Based on these results, it can be indicated that the proposed model is able to retrieve the answer correctly as it chooses the correct paragraph.