High-Performance Spectral Tensor Layer for Deep Neural Network

Training on large-scale image datasets enables better performance of deep neural networks (DNNs). To have a rea-sonable training time, DNNs are usually trained on large-scale datasets using distributed training. However, the communication overhead of distributed training makes the training more costly and inefficient. And training on large-scale dataset with large batch size has a large G PU memory footprint. In this paper, we present the high-performance spectral tensor layer for distributed training. By tensorizing image data and transforming it to the spectral domain, we split the training dataset into multiple independent sub-datasets. Building multiple sub-networks and utilizing ensemble learning, we implement a distributed training method without communication overhead. Compared to training on a single GPU using CUDA cores, our method achieves a speedup of 5.0 x at four training nodes and a speedup of 18.2 x at sixteen training nodes for training on the ImageN et-IK dataset. We also have an experiment for training on the ImageNet-21K dataset and our method achieves a speedup of 16.4x at 16 training nodes.