CNN Training Latency Prediction Using Hardware Metrics on Cloud GPUs

Convolutional neural network (CNN) models are becoming larger and more sophisticated over time, and training requires a significant amount of time and computing resources. To meet computing demand, graphics processing units (GPUs) are widely used for training. Due to the excessive cost and overhead of maintaining a GPU cluster, users may prefer to use GPUs provided by a public cloud vendor rather than creating their own servers. The initial cloud service is offered in the Infrastructure-as-a-Service (IaaS) model. As the cloud evolves, the abstraction level of the public cloud service becomes higher, and serverless computing is considered the next-generation cloud service. In the new way of offering cloud services, vendors are required to provide an efficient environment for diverse workloads. To meet the new requirement of the evolving cloud service, this paper proposes heuristics to predict the training latency on various GPU devices without using model information to help cloud-service vendors prepare an efficient training environment with minimal exposure to users’ model architectures. Unlike previous work that relies on internal model details for latency prediction, the proposed system uses only the hardware metrics that are extracted during training. Using the information, we first propose an algorithm to detect an epoch period that we aim to predict. The detected epoch period becomes the target latency to predict, for which we apply a stacked regressor to achieve superior prediction accuracy. Detailed experiments revealed that the average prediction accuracy of the proposed training latency prediction model is 11.17%, which is similar to the state-of-the-art approach that references the internal architecture of the model. Unlike previous work, the proposed work does not reference the model’s internal architecture, which proves the applicability of this proposed work in the next-generation cloud service.