Foundation Models for Course Equivalency Evaluation

This study investigates the potential of Large Language Models (LLMs) for evaluating course equivalency in higher education. We introduce an innovative approach that utilizes publicly available course descriptions and unmodified LLMs for pairwise course comparison. We selected Google PaLM2 and its successor, Gemini Pro v1.0, due to their accessible free-tier API and their ability to reliably generate structured data.The most challenging aspect of our methodology was extracting data from course descriptions. Nonetheless, Gemini Pro v1.0 demonstrated a serviceable ability to comprehend the context of unprocessed descriptions and effectively categorize their components. Notably, classification t asks u sing r aw text yielded better results compared to those based on extracted topics, indicating potential improvements in topic extraction.Our findings reveal that the model tends to exhibit a conservative bias, often leaning towards non-equivalence judgments. We introduced additional categories such as "unsure" and "inadequate data," which enhanced the statistical performance of the model in the equivalent/nonequivalent classes and simulated the possible decision-making processes of human advisors in ambiguous cases.This study underscores both the challenges and opportunities presented by LLMs in course equivalency evaluation. Key considerations include prompt sensitivity, computational costs, and API limitations. Future research will focus on comparing results across different models and prompt designs, exploring alternative techniques such as embeddings and instruction fine-tuning, and striving to develop a more precise and reliable course equivalency assessment system.