Demo: Real-Time Inference on GPU-Based Heterogeneous SoCs with GPU Cache Locking

Embedded real-time systems are increasingly turning to GPU-based SoCs to efficiently handle machine learning tasks at the edge. Modern GPU SoCs often feature specialized AI accelerators to enable concurrent CNN inference while maintaining energy efficiency. While this trend typically leads to a general improvement in performance, the integration of multiple AI accelerators presents challenges for building real-time systems, where ensuring timing predictability is a key design goal. On one hand, contention among various computation units over shared memory can introduce non-deterministic timing behaviors. On the other hand, the built-in GPU scheduling mechanism cannot assure that emergent tasks to be executed first, potentially violating real-time constraints. To tackle these challenges, this study introduces a timing predictability-aware cache locking policy to reduce main memory access volume and mitigate main memory contention with AI accelerators. Additionally, a real-time scheduling framework is proposed to bypass the inherent GPU scheduling algorithm.