Transformer Inference Acceleration in Edge Computing Environment

The rapid development of deep neural networks (DNNs) has provided a strong foundation for the popularization of intelligent applications. However, the limited computing power of IoT devices cannot support the massive computing load of DNNs. Traditional cloud computing solutions suffer from distance and bandwidth constraints, and cannot meet latency-sensitive requirements. Prior research on DNN acceleration has predominantly focused on convolutional neural networks (CNNs), Transformer has recently gained significant popularity due to its outstanding performance in natural language processing, image processing, and other domains. In this context, we have explored the acceleration of Transformer in edge environments. To model Vision Transformer, we have employed the design concept of a multi-branch network and proposed an optimization strategy to accelerate Transformer inference in the edge environment. We have evaluated our approach on three publicly available datasets and demonstrated its superior performance in challenging network conditions when compared to existing mainstream DNN collaborative acceleration inference techniques.