EECS: Efficient End-Cloud Collaborative System With On-Demand Offloading Mechanism for Convolutional Neural Network

The end-cloud synergy with neural networks offers significant promise for enhancing AI-IoT systems. Numerous studies have been conducted to effectively address the dynamic and resource-constrained nature of IoT; however, challenges persist in adaptability to constraints, horizontal integration between technologies, and system-level optimization. In this article, we propose an end-cloud collaborative system (EECS) with an on-demand offloading mechanism for AI-IoT. We enhance three critical areas previously narrowly explored: 1) model pruning to fit into the end device’s constraints; 2) dynamic data transmission between end and cloud; and 3) an effective offloading policy. We pay particular attention to the constraints imposed by multiply-and-add (MAC) operations and bandwidth (BW) at the end, with a focus on image classification tasks. Our contributions are threefold. First, we introduce a two-stage trainable pruning method that can automatically adjust the end model to end constraints and optimize from a system-wide perspective. Second, we propose an adaptive mechanism that accommodates fluctuating BW conditions based on our trainable pruning, integrating static pruning with dynamic inference. Third, we develop a smart offloading policy that enhances decision-making, thereby elevating system-level efficiency. Finally, our EECS shows substantial improvements, achieving $2.6\times $