Auto-tuning of AI/ML Graphs for Optimal Performance in a Heterogenous Processor System

High-performance parallel computing is very important to ensure the efficient execution of deep learning algorithms. However, current research focuses more on parallel computing of homogeneous processors. Due to the differences in memory, communication units and computing capabilities of heterogeneous architectures, existing deep learning parallel computing architectures cannot achieve good performance improvement in heterogeneous environment. Current solutions use serial methods to execute algorithms on heterogeneous architectures, rely on developer’s development experience or use simple search strategies to perform tuning to achieve parallel computation. The existing solutions either cannot fully utilize the hardware resources of heterogeneous architectures to achieve performance optimization or cannot maximize performance due to the design of limited search space and low efficient search strategies.In our solution, we designed a new neural network automatic horizontal partitioning framework, which automatically splits the neural network horizontally into multiple sub-graphs based on a given target device or target device cluster, so that the neural network can be running sub-graphs in a pipelined or parallel manner on a target device or target device cluster. Our solution creates a larger search space compared to existing solutions and uses efficient exploration and cost evaluation methods.