ML-Enhanced Performance and Power Estimation for DNNs on Heterogenous SoCs

Efficient deployment of Deep Neural Networks (DNNs) on heterogeneous Systems-on-Chip (SoCs) requires navigating a large and complex design space defined by combinations of choices in network architectures, operator-to-processor mappings, and power/performance operating points for each processor. Exploring this vast space requires estimators that are faster than even real-time execution on the target hardware. Machine learning (ML)-based estimators have been proposed to address this challenge. However, prior ML-based methods fail to accurately capture all the effects of heterogeneous parallel execution such as parallelism across processors, inter-processor communication, and control-flow dependencies from offloading, resulting in significant estimation errors.We introduce CoCO-ML, a Concurrency, Communication, and Offload-aware framework that integrates ML models with classical graph-based algorithms for fast and accurate performance and power estimation. CoCO-ML constructs a Computation-Communication Graph (CCG) that explicitly models operator mappings, data transfers between processors, and offload overheads. ML models are used to predict the execution time and power of individual operators, while symbolic graph traversal produces execution traces and end-to-end execution time and power estimates.We evaluate CoCO-ML on four single-DNN and two multi-DNN workloads executed on the NVIDIA Jetson AGX SoC. CoCO-ML achieves mean estimation errors of 6.54% (performance) and 12.51% (power), outperforming prior methods while being over 1000× faster than native execution. Furthermore, CoCO-ML generates symbolic execution traces that expose application bottlenecks, providing actionable insights to application developers and facilitating efficient deployment of DNNs on heterogeneous SoCs.