FrameFeedback: A Closed-Loop Control System for Dynamic Offloading Real-Time Edge Inference

Despite the demand for realtime deep learning applications such as video analytics at the edge, resource-constrained edge devices can largely not process video streams at their source frame rate. However, deep learning execution can be accelerated by offloading tasks to a nearby edge server equipped with a GPU. For a realistic edge system with variable network conditions and server load, we consider optimally partitioning the frames from a video stream between local processing and offloading to maximize the throughput of an edge device under a realtime deadline. To do this, we show that we can simplify the influences on processing latency into a single relevant metric and dynamically determine an appropriate offloading rate using a latency-based feedback control mechanism. Our controller settles on the optimal offloading rate without knowing network conditions, resource availability, or application computation cost. Our measurements show that our feedback controller balances sensitivity and overcorrection given a variety of network and load conditions set. We also show that our controller's Quality of Service outperforms state-of-the-art baselines and approaches.