Efficient Training of EfficientNetV2-S Using AdaBelief Optimizer

The EfficientNetV2 architectures have classification accuracies exceeding 80% on the Imagenet-1K/21K datasets. Here, the networks used to be trained using stochastic gradient descent with momentum ($SGD+M$) as the optimizer. Such a method is known to be effective but takes too much time, making it impractical to train the networks from scratch especially when computational power is limited. To address this, we provide a guide on how to use the AdaBelief Optimizer for training EfficientNetV2-S. Results show that, even without complex training configurations, using AdaBelief can lead the EfficientNetV2-S network to achieve a top-1 accuracy as high as 80% on the Imagenet-1K validation set within 80 epochs of training.