Guiding Grokking: How Curriculum Learning Shapes Algorithmic Understanding in Neural Networks

Understanding how neural networks suddenly achieve strong generalization after extended training (grokking) remains a fundamental challenge in deep learning. While this phenomenon has been observed in algorithmic tasks, the factors influencing its emergence remain poorly understood. We investigate whether curriculum learning—gradually increasing task complexity—can facilitate and accelerate grokking. Through systematic experiments on modular arithmetic tasks with varying prime moduli, we compare four curriculum strategies (linear, exponential, step-wise, and gradual progression) against a baseline. Our results reveal that only modular addition benefits significantly from curriculum learning, achieving 81.1% validation accuracy with gradual progression compared to 65.7% and 57.2% for linear and exponential curricula respectively. In contrast, operations like modular division and permutation show minimal improvement, with validation accuracies below 20% across all curricula. The gradual curriculum maintains smoother training loss transitions and higher training stability, particularly for addition, while step-wise progression shows the worst performance with validation accuracy drop- ping to 34.4% for addition. These findings suggest curriculum learning is most effective for operations with simpler underlying structures, while complex operations remain challenging regardless of curriculum design. This work provides practical insights into curriculum strategies for algorithmic learning tasks and highlights fundamental limitations of current approaches.