Power, Performance and Reliability Evaluation of Multi-thread Machine Learning Inference Models Executing in Multicore Edge Devices

Incorporating Machine Learning (ML) inference models into edge computing devices has presented some performance and reliability enhancement challenges. Multi-threaded ML models demonstrated power efficiency and performance gains when deployed on high-performance multicore platforms, including powerful general-purpose processors and graphics processing units (GPUs). However, there is a relative lack of investigation on the potential impact of parallel pre-trained ML models when executed in resource-constrained edge devices, which rely on low energy and reduced memory footprint processors. With that in mind, this work presents the impact of multi-threaded parallelism on the performance, power consumption, and soft error reliability of different ML inference models. Results show that parallel models enhanced by up to 2.6$\times$ the performance per watt while reducing their susceptibility to the occurrence of radiation-induced soft errors in up to 6.6$\times$ w.r.t. the original sequential versions.