Optimal Distribution of ML Models Over Edge for Applications with High Input Frequency

The rise of complex and sizeable Machine Learning (ML) models challenges traditional cloud computing models with respect to the high volume of incoming data which results in increased bandwidth usage and network congestion, as well as delays in inference. Such ML models are being rapidly developed thanks to advances in computing platforms and the real-time computing demands of ML-driven applications such as autonomous vehicles and video processing. To mitigate these challenges, ML model distribution and inference offloading to computing devices close to data sources have been explored, especially through partitioning the models across the IoT-Edge-Cloud continuum. Existing efforts in this area have not successfully mastered the fully automatic and efficient determination of optimal partition points. Additionally, they have not effectively integrated Early-Exit layers that allow for early termination of model inference at earlier stages when feasible. In this paper, we introduce a novel partitioning algorithm designed to distribute ML models across edge devices with the goal of reducing response time when facing high-rate input data streams. Our proposed approach leverages the principles of Dynamic Programming to determine optimal partition points and establish appropriate exit thresholds for Early-Exit layers. Our evaluation results reveal that, in the context of continuous, high-rate input data, our method consistently lowers the maximum round-trip time for processing inference requests compared to state-of-the-art methods such as NeuroSurgeon and Genetics.