Infer-EDGE: Dynamic DNN Inference Optimization in Just-in-Time Edge-AI Implementations

In recent times, ‘Just-in-time’ edge environments have gained popularity due to on-demand edge resource requirements for deep neural network (DNN) based video processing applications in mission-critical use cases, such as public safety and tactical situations. However, striking a balance among mutually diverging performance metrics, such as end-to-end latency, accuracy, and device energy consumption in such inherently resource-constrained and loosely coupled environment is non-trivial. In this paper, we design and develop the Infer-EDGE framework that seeks to strike such trade-off. First, using comprehensive benchmarking experiments, we develop intuitions about the trade-off characteristics, which are then used by the framework to develop an Advantage Actor-Critic (A2C) Reinforcement Learning (RL) approach that can choose optimal run-time DNN inference parameters, aligning the performance metrics with the application requirements. Using real-world DNNs and a hardware testbed, we evaluate the benefits of Infer-EDGE framework in terms of energy savings, inference accuracy improvement. and end-to-end inference latency reduction.