Improving Sharpness-Aware Minimization Using Label Smoothing and Adaptive Adversarial Cross-Entropy Loss

Recent advances in learning algorithms have identified loss surface sharpness as an effective metric for reducing the generalization gap. Building on this principle, Sharpness-Aware Minimization (SAM) was introduced to improve model generalization and has achieved state-of-the-art performance through its weight perturbation and updating steps. However, SAM’s perturbation is based solely on the gradient of the standard cross-entropy loss. As the model approaches convergence, this gradient diminishes and oscillates, leading to inconsistent perturbation directions. To address these limitations, we introduce SAM-CE-AACE and SAM-SCE-AACE. SAM-CE-AACE replaces the conventional cross-entropy loss with the Adaptive Adversarial Cross-Entropy (AACE) loss in the perturbation step. Unlike the standard loss, AACE increases in both value and gradient magnitude as the model converges, ensuring stable perturbation directions. Additionally, SAM-CE-AACE’s perturbation utilizes AACE loss without normalization, enhancing the model’s exploratory capabilities in near-optimum stages. Moreover, SAM-SCE-AACE further improves SAM-CE-AACE by integrating label smoothing into the weight-updating step, reducing overconfidence in predictions. Together, this combination of the AACE loss function and label smoothing technique leads to further improvement in model performance. Intensive empirical experiments have been conducted to confirm the effectiveness of our methods, with experiments demonstrating improved performance in image classification tasks using different network architectures across various datasets. The reproduction code is available online at https://www.vip.sc.eng.isct.ac.jp/proj/AACE