Answer Relevance Score (ARS): Novel Evaluation Metric for Question Answering System

The evaluation of question answering systems plays a crucial role in assessing their performance and effectiveness. Existing evaluation metrics often focus on aspects such as recall, precision and F1 score, but may not fully capture the relevance and informativeness of the generated answers. In this paper, a novel evaluation metric called Answer Relevance Score (ARS) is proposed that takes into aspects i.e. correctness and completeness. Correctness considers the comprehensiveness of answer which consider the semantic and contextual analysis of sentences selected for answer. The answer's completeness analyse the rate of information overlap in reference and generated answer. ARS is compared with existing metrics and to this end, comparative study is also given for evaluating the efficiency of existing metrics across two datasets; Natural Questions and WikiQA. It is find that ARS metric correlates much better with human judgement than other benchmarks.