Fine-Tuning Large Language Models for Solving Math Word Problems: A Case Study with LLaMA-3

The rapid development of Large Language Models (LLMs) has led to advances in natural language processing, but handling mathematical tasks remains a challenge due to the demands of text understanding, quantitative reasoning, and numerical computation. Although LLMs show progress in reasoning for simpler problems, they often struggle with more complex tasks requiring multi-step reasoning. To address these challenges, this study focuses on fine-tuning a pre-trained Llama-3-8B-Chinese-Chat model to enhance its ability to solve mathematical word problems (MWPs). A systematic approach was followed, involving dataset difficulty classification, prompt engineering, and LoRA fine-tuning techniques. The model's performance was evaluated to show significant improvements in equation matching and numerical accuracy after fine-tuning, particularly for complex problems. On simple, medium, and difficult test sets, End-to-End accuracy of fine-tuned model improved by 47.1%, 30.2%, and 65.7% correspondingly. However, the model still fails to reach a satisfactory accuracy while handling purely mathematical computations, which may be accounted for its poor capability of handling symbol processing and fraction conversion. Future research directions include investigating diverse mathematical reasoning tasks and integrating external symbolic computation tools to enhance computational precision. This study contributes to evaluating LLMs' mathematical reasoning capabilities and revealing the promising potential of LoRA fine-tuning.