Assessing GPT's Legal Knowledge in Japanese Real Estate Transactions Exam

In recent years, large language models (LLMs) like GPT-3.5 and GPT-4, developed by OpenAI, have demonstrated impressive capabilities in tasks such as natural language understanding, code generation, and even passing professional exams. Trained on vast datasets, these models have shown high accuracy across various fields. One challenging area for LLMs is professional certification exams, which require specialized knowledge and understanding of complex regulations. The Real Estate Transaction Specialist Examination (RETSE) in Japan is a key certification for professionals managing real estate transactions. Passing the RETSE qualifies individuals to handle legal responsibilities like ensuring the legality of property deals. The exam covers civil law, tax law, and real estate regulations, making it a comprehensive test of legal knowledge. With a pass rate of around 15%, it is considered one of Japan's more difficult professional exams, making it a strong benchmark for evaluating LLM performance. This study evaluates the ability of GPT-3.5 and GPT-4 to answer RETSE questions, comparing their performance to official passing criteria and analyzing the effect of prompt engineering. While GPT-4 outperformed GPT-3.5, neither model met the passing standard. Prompt strategies such as “Consider customary laws” improved response accuracy, especially for complex legal questions. These findings suggest that while GPT models cannot fully replace human expertise in legal exams like RETSE, they offer potential as supplementary learning tools for verifying basic knowledge. To mitigate the risk of job displacement by AI, it is crucial for professionals to enhance their expertise in areas where LLMs struggle, such as recent legal changes and ethical judgment. This study highlights both the limitations and potential of GPT in legal contexts, providing insights into how AI can support learners and practitioners in real estate law.