Contents I. Introduction
With the explosion of data and the increase in hardware computing power, deep neural networks have been successful in supervised learning, unsupervised learning, reinforcement learning, and blended learning [1]. At the same time, with the development of edge computing, deep learning tasks have been widely deployed on edge devices. Nowadays, edge devices can not only support inference but also training and saving models [2]. To support the development of deep learning tasks, a variety of deep learning frameworks (framework will be used in the rest of this paper to represent deep learning framework.), such as TensorFlow [3], PyTorch [4], MXNet [5], Paddle Paddle [6], Caffe [7] etc. have emerged, and new frameworks such as MindSpore [8] and OneFlow [9] are being developed. However, there are many differences between different frameworks, including the implementation principles of themselves, optimization of different deep learning models (The model will be used in the rest of this paper to represent the deep learning model.) and hardware, which lead that different combinations of models, frameworks, and hardware producing different performance. Therefore, the evaluation of frameworks on the edge is especially important for finding the best combinations. Although there are some research on frameworks on the edge, the following three problems remain unsolved. First of all, the existing researches mainly focused on runtime performance and are unable to give a comprehensive evaluation of frameworks. Secondly, these researches were camed out by comparing the performance of hardware or programs under different conditions, without analyzing the constraints of the program, software, and hardware to get optimization suggestions. Finally, the evaluation methods used in these researches were highly complex and inefficient, which make it difficult for non-professional to reproduce.
