Joint Model Compression and Knowledge Distillation for On-Demand DNN Inference Based on End-Edge Collaboration

End-edge collaborative inference refers to the fact that edge servers (ESs) and end devices (EDs) jointly participate in inference tasks, which can not only reduce communication latency and bandwidth consumption with the cloud but also protect user data privacy. However, existing collaborative inference methods do not fully consider the limited resources of EDs and ignore the latency and accuracy requirements of different inference tasks. In this paper, we design a DNN inference acceleration framework to balance inference latency and accuracy. Specifically, we first use a compression method based on deep reinforcement learning to determine the compression ratio and deeply compress the original model to reduce the complexity of the model. To reduce the cumulative error caused by compression, a knowledge distillation-based scheme is used to fine-tune the compressed model. Finally, the DNN model is partitioned and deployed on the ED and ES, respectively. Extensive experiments demonstrate the effectiveness of the framework in achieving lowlatency DNN inference on demand.