Experimental Characterizations and Analysis of Deep Learning Frameworks

Big Data has fueled the wide deployment of Deep Learning (DL) in many fields, such as image classification, voice recognition and NLP. The growing number of open source DL software frameworks has put forward high demands on comparative study of their efficiency with respect to both runtime performance and accuracy. This paper presents a brief overview of our empirical evaluation of four representative DL frameworks: TensorFlow, Caffe, Torch and Theano through a comparative analysis and characterization. First, we show that the complex interactions among neural networks (NN), hyper-parameters, their specific runtime implementations and datasets are latent factors for the uncertainty of runtime performance and accuracy. Second, we characterized the CPU/GPU resource usage patterns under different configurations for different frameworks to obtain an in-depth understanding of the impact of different batch sizes. Third, we describe the data loading process of ImageNet for TensorFlow and present an experimental characterization of TensorFlow with respect to its data loading process when the dataset is too large to fit into the main memory of the CPU server. We conjecture that our experimental characterization and analysis can offer empirical guidance for users and application developers to select the right DL frameworks and configurations for their domain-specific learning tasks and datasets.