Auto-CNNp: a component-based framework for automating CNN parallelism

Effectively training of Convolutional Neural Networks (CNNs) is a computationally intensive and time-consuming task. Therefore, scaling up the training of CNNs has become a key approach to decrease the training duration and train CNN models in a reasonable time. Nevertheless, introducing parallelism to CNNs is a laborious task in practice. It is a manual, repetitive and error-prone process. In this paper, we present Auto-CNNp, a novel framework that aims to address this challenge by automating CNNs training parallelization task. To achieve this goal, the Auto-CNNp introduces a key component which is called CNN-Parallelism-Generator. The latter component aims to streamline routine tasks throughout (1) capturing cumbersome CNNs parallelization tasks within a backbone structure while (2) keeping the framework flexible enough and extensible for user-specific personalization. Our proposed reference implementation provides a high level of abstraction over MPI-based CNNs parallelization process, despite the CNN-based imaging task and its related architecture and training dataset. We introduce the design and the core building blocks of Auto-CNNp. We further conduct an extensive assessment of our proposal that not only shows its effectiveness in accelerating the process of scaling up CNNs training, but also its generalization for a wider variety of use cases.