Accelerating Deep Learning Training Through Transparent Storage Tiering

We present Monarch,a framework-agnostic storage middleware that transparently employs storage tiering to accelerate Deep Learning (DL) training. It leverages existing storage tiers of modern supercomputers (i.e., compute node's local storage and shared parallel file system (PFS)), while considering the I/O patterns of DL frameworks to improve data placement across tiers. Monarchaims at accelerating DL training and decreasing the I/O pressure imposed over the PFS. We apply Monarchto TensorFlow and PyTorch, while validating its performance and applicability under different models and dataset sizes. Results show that, even when the training dataset can only be partially stored at local storage, Monarchreduces TensorFlow's and PyTorch's training time by up to 28% and 37% for I/O-intensive models, respectively. Furthermore, Monarchdecreases the number of I/O operations submitted to the PFS by up to 56%.