Analysis of Sampling Strategies for Multi-Task Learning in Transformer Models

Multitask learning has emerged as a powerful paradigm for enhancing natural language understanding capabilities in large language models. However, the effectiveness of multitask learning heavily depends on the sampling strategy used to balance exposure to tasks of varying sizes and complexities during training. In this work, we conduct a comprehensive empirical analysis of three sampling strategies using T5-small on a subset of the GLUE benchmark: examples-proportional sampling, which samples based on raw dataset sizes; temperature-scaled sampling with T=10.0, which moderates size-based differences; and equal sampling, which gives uniform probability to all tasks. Our analysis examines the impact of these strategies on model performance, convergence patterns, task representation, and internal model dynamics through spectral analysis. Our results demonstrate that temperature-scaled sampling provides a strong balance between task representation and overall performance, while equal sampling achieves the highest average score across all tasks. Spectral analysis reveals that sampling strategies produce fundamentally different internal representations despite similar performance metrics, with higher temperatures promoting more uniform and stable layer behavior. We provide practical guidelines for selecting appropriate sampling strategies based on dataset characteristics, computational constraints, and specific training goals, contributing to more effective multitask learning approaches for language models.