A Categorical Particle Swarm Optimization for Hyperparameter Optimization in Low-Resource Transformer-based Machine Translation

This paper proposes a categorical particle swarm optimization (PSO) for hyperparameter optimization of the Transformer-based neural machine translation for low-resource training. The proposed PSO is a set-based PSO representing particles as probability distributions rather than solution values. The experiments on two datasets under various low-resource conditions on the standard dataset of German-English and Thai-English translations validate the proposed PSO. Empirical results demonstrate that our proposed PSO algorithm converges to near optimum solutions and improves the translation quality over the non-optimized Transformer models and manually optimized models under low-resource training.