Answering Poetic Verses' Thematic Similarity Multiple-Choice Questions with BERT

We define the task of answering poetic verses' thematic similarity multiple-choice questions (MCQs), which can prove useful for estimating the difficulty of this type of question in the Iranian university entrance exam. Using the pre-trained, multilingual Google BERT language representation on 100 such questions, we have attempted to establish a baseline for the task by attaining an accuracy of 32%. The model's performance was shown to depend heavily on semantic similarity at the word level, and the model struggled with more abstract verses. These particular verses required an interpretation of the verse and contained little or no semantic hints that the model could exploit. As a result, it appears that the model has answered these questions randomly. While the 32% accuracy of the established baseline has much room for improvement, the current results may be used to find a correlation between the performance of the model and that of the test-takers. This correlation may later be used for the task of MCQ difficulty estimation. Furthermore, training the model on a dataset of Persian poetry as well as fine-tuning BERT could also yield better results.