Scoring Indonesian Research Proposals via LLM-Based Pairwise Comparison and Summarization

In Indonesia, researchers are expected to conduct studies aligned with national priority research areas defined by strategic policy documents. Researchers submit proposals to obtain government funding, and these propos-als are evaluated based on their relevance to these national priorities. Traditionally, this evaluation involves resource-intensive manual scoring, prompting a need for automation. Recently, large language models (LLMs) have enabled automated scoring for various document-assessment tasks. However, standard fine-tuning-based approaches are impractical because of frequently changing policies, making it challenging to collect sufficient training data. Furthermore, in-context learning approaches typically assign high relevancy scores to nearly all proposals, given their general alignment with national priorities, thereby hindering effective differentiation. A potential solution is pairwise comparison using LLMs, but proposals are typically submitted in PDF format containing extensive text, figures, and tables, often exceeding current LLM context window limits. To overcome these issues, we propose summarizing each proposal into a concise format suitable for relevancy scoring, then employing LLM - based pairwise comparisons combined with the Bradley-Terry model to estimate relevancy scores. Applying this zero-shot method to an Indonesian proposal dataset demonstrates substantial improvements over direct zero-shot scoring and notably surpasses a standard supervised fine-tuning baseline.