Smooth Categorical Cross-Entropy: An Improved Approach to Overcome the Limitations of CCE

Improving the loss function can significantly enhance learning outcomes by more effectively addressing the differences between the model’s predicted values and the actual values. Categorical Cross Entropy (CCE) calculates the loss using the negative logarithm of probability predictions, yielding smaller loss values for predictions closer to the correct answer and larger loss values for incorrect predictions. In deep learning, the model uses the gradient of the loss function to determine how much to update its weights. Once the model’s predictions are sufficiently close to the correct answer, further weight updates are unnecessary. However, CCE has a limitation: its gradient is always less than −1, causing continuous weight updates. This results in overtraining for high-confidence predictions and undertraining for low-confidence predictions. When training on the Caltech-UCSD Birds 200 dataset, we observed that the original CCE loss function led to predicted values clustering near 1, with few small prediction values appearing as outliers. To address this, we propose a new loss function, Smooth CCE, which adjusts the gradient to approach zero as the predicted values approach 1. This prevents unnecessary updates and leads to a more balanced distribution of predictions, increasing the proportion of lower predicted values. Using Smooth CCE with $\lambda=0.7$ and $\alpha=0.5$, we observed performance improvements of $0.6 \%$ p on EfficientNetV2B0, $2.7 \%$ p on MobileNetV3Small, and $2.0 \%$ p on InceptionV3 compared to the original CCE.