SkePU-DNN: Algorithmic Skeleton Programming for Deep Learning on Heterogeneous Systems

Machine learning is rapidly becoming a common class of application workload for high-performance computer systems. In particular, deep neural networks (DNNs) are a popular approach for machine learning that comes with large computational demands and relies on systems with high computational throughput. Such neural networks are therefore usually implemented in domain-specific programming models and frameworks, where individual components have been hand-tuned for one or several target hardware architectures. In this work, we investigate to which extent the algorithmic skeleton model as implemented in the SkePU framework can accommodate deep neural networks, in particular, convolutional neural networks (CNNs). SkePU is an open-source framework for C++-based, single-source high-level programming of heterogeneous parallel systems. It provides general-purpose algorithmic skeletons such as map, reduce, and stencil, which are customizable by problem-specific code, have multiple backends for multithreaded CPU, GPU and multi-GPU execution, and operate on multidimensional array data-container objects. We extend SkePU with new functionality aimed at improving the interface for computational and data-access patterns seen in CNNs and DNNs, and demonstrate the design and implementation with examples and evaluation. Our experimental evaluation of SkePU-DNN with multiple machines, backends, models and benchmark data sets shows consistent performance behavior and that model accuracy is on-par with that of established domain-specific DNN frameworks.