Dynamic Computing Resources Allocation for Multiple Deep Learning Tasks

To improve the speed of deep learning tasks, the models of Deep Neural Network (DNN) could be trained in a computational node where the parameters for the models are refined by multiple GPUs in the node parallelly. A number of previous works have studied parallel learning focus on the improvement of one single task. In such a situation, the whole hardware resources can be fully utilized for that task. However, in a real system, it is usual to have several learning tasks running in the same node. So in this paper, we propose an adaptive allocation of computing resources for multiple learning tasks, with the knowledge of current learning phase for each task. In experiments we train VGG-16 models using CIFAR-10 dataset on a node with multiple GPUs, and the results show that with adaptive adjustment for computing resources, the training time can achieve a decrease of 4.70% with unchanged per task batch size. Besides, the experiment that mixing up ResNet-50 and VGG-16 models shows that this computing resources adjustment is effective for different models.