MLIoT: Transparent and Secure ML Offloading in the Cloud-Edge-IoT Continuum

As IoT devices and robotic systems increasingly rely on ML inference, resource constraints pose a challenge for real-time execution. Efficient offloading mechanisms that leverage edge and cloud computing are essential to enable scalable, low-latency inference. We build a novel framework that addresses three key challenges in the Cloud-Edge-IoT space: (a) secure device onboarding and OTA updates for microcontrollers, (b) cloud-native integration of microcontrollers, (c) flexible compute-intensive application offloading for ESP32-based devices to neighboring nodes with increased compute capacity. In this paper, we present the motivation for this framework, evaluating common ML inference workloads on ESP32-based devices, low-end edge boards (Raspberry Pi), high-end edge boards (Jetson AGX) and generic cloud servers (x86). We show that offloading ML workloads to heterogeneous hardware across the Cloud-edge-IoT continuum is potentially beneficial, depending on user-defined parameters such as real-time response requirements or power budgets. Additionally, our approach enables pure cloud-native IoT device repurposing, leveraging DICE and EATs for secure device onboarding and secure OTA updates.