Assessment of Hyperparameter Optimization Techniques for Cross-Stitched Multi-Task Learning

Hyperparameter optimization is critical to successful machine learning model development. Conducting experiments to evaluate various hyperparameter optimization techniques across different models and datasets is essential for advancing knowledge and providing practical guidance. This study aims to compare state-of-the-art hyperparameter optimization algorithms in the developing cross-stitched multitask deep learning models, focusing on a dataset encompassing three citation meanings in text: function, role source, and sentiment. The hyperparameters examined include embedding output dimensions, convolutional filters and kernel size, dense layer configurations, dropout rates, learning rates, and batch sizes. The techniques evaluated are (1) Grid Search, (2) Random Search, (3) Tree-structured Parzen Estimator (TPE), (4) Gaussian Process (GP)-based algorithm, (5) Covariance Matrix Adaptation - Evolution Strategy (CMA-ES) based algorithm, (6) Nondominated Sorting Genetic Algorithm II, (7) a method enabling partially fixed parameters (PF), and (8) Quasi-Monte Carlo (QMC) sampling algorithm. The findings indicate that the TPE and CMA-ES techniques are particularly effective in selecting hyperparameters that obtained the optimal performance (the highest average improvements: 2.48 and 2.42, respectively), with the learning rate emerging as the most important hyperparameter. These results contribute to the literature on hyperparameter optimization, specifically within the case of cross-stitched multi-task learning.