A Benchmarking Survey: Evaluating the Accuracy and Effectiveness of Benchmark Models in Measuring the Performance of Large Language Models

Large language models have revolutionized artificial intelligence, exhibiting remarkable linguistic abilities across various tasks. However, evaluating the true performance of these models remains a significant challenge, necessitating reliable benchmark models. This study conducts a comprehensive survey to evaluate the accuracy and effectiveness of benchmark models in measuring the performance of LLMs. Through a systematic literature review of 42 papers, six influential benchmarks were selected: AGIEval, MMLU, TruthfulQA, GSM8K, HellaSwag, and Chatbot Arena. These benchmarks were assessed across three key criteria: data contamination, sensitivity to prompting techniques, and the ability to capture the evolving nature of language. Using a 5-point scoring system, each benchmark was evaluated on these criteria, with an overall score calculated as the mean of the three individual scores. The comparative analysis revealed considerable variability among the benchmarks. Data contamination, where LLM training data overlaps with benchmark datasets, emerged as a pervasive issue, rendering many benchmark scores unreliable. Sensitivity to prompting techniques also varied, with some benchmarks being highly susceptible to score manipulation. Additionally, capturing linguistic evolution proved challenging for static dataset-based benchmarks. Notably, Chatbot Arena excelled across all criteria due to its human-centric, dynamic approach. This study highlights the need for more robust, adaptive benchmarking methodologies to accurately assess the rapidly evolving capabilities of large language models.