EdgeInsight: Characterizing and Modeling the Performance of Machine Learning Inference on the Edge and Cloud

The Internet-of-Things (IoT) is growing in importance enabling an increasing number of scientific, industrial, and societal applications. At the same time, the computational capabilities of IoT and edge devices are rapidly improving making them viable for machine learning. Thus, the deployment of machine learning models on the edge is becoming a critical capability. However, such deployments are challenging as edge devices are more resource-constrained than clouds and cannot elastically scale on demand. Moving the application to the cloud can provide more computational power, but raises other challenges, such as security, reliability, and bandwidth. The integration of edge and cloud computing resources is often essential for many applications allowing them to choose the best configuration with respect to their requirements and characteristics, such as data rates and computational complexity. To understand the trade-offs between edge and cloud computing the assessment of different cloud and edge configurations is required. In this paper, we present EdgeInsight, a framework for characterizing and modeling of the inference performance of edge and cloud infrastructures. EdgeInsight enables system builders to size and fine-tune infrastructure parameters for different workloads and applications. We use EdgeInsight to qualitatively and quantitatively study edge and cloud deployment configuration for deep learning inference. Our evaluation shows, that edge inference can outperform cloud inference when model architecture and accuracy, inference framework and pre-processing parameters are carefully selected.