A GPU Implementation Method of Deep Neural Networks Based on Data Swapping

Deep neural networks have attracted a great deal of research attention due to the potential of building efficient classifiers for a variety of recognition problems. The major drawbacks of deep neural networks are significant training time and memory usage. In this paper, we propose an implementation method of deep neural networks using a single GPU based on the idea of data swapping. The proposed method introduces the concept of virtual layer on a GPU and employs multiple virtual layers to reduce the overhead of data swapping between a host PC and the GPU. Experimental results showed that the proposed method reduced the memory usage by about 73% compared with the conventional method and the training time by about 33% by the triplication of virtual layers.