A Service Mesh-Based Load Balancing and Task Scheduling System for Deep Learning Applications

In recent years, there are increasing technologies that benefit from cloud computing and edge computing, especially the application in the Deep Learning and Internet of Things topics. Functionalities such as load balancing, traffic routing, and task scheduling, which used to be part of the software, are now enabled and provided as API or microservices on the underlying infrastructure. Therefore, when we want to deliver a container application to the users either on the cloud cluster or edge cluster, developers do not need to worry about these issues and only focus on the development of the functionality. Nevertheless, for the deployment stage, the system architect should design the fundamental architecture to build an ecosystem for the application. In this paper, a practical approach is proposed to develop a flexible and on-demand system for deep learning applications based on container and service mesh technologies. Our experiment chooses Kubernetes as the container orchestration platform and introduces Istio to implement the service mesh. We packaged one Flask based Deep learning model into a Docker container, and successfully deploy and provision this image classifier application to the public users with Functionalities like load balancing and task scheduling. Furthermore, we empower the system with an evaluation of the resource utilization services, and traffic flows inside the Kubernetes cluster in a visualized manner.