A Transistor Operations Model for Deep Learning Energy Consumption Scaling Law

Deep neural networks (DNNs) have transformed the automation of a wide range of industries and found increasing ubiquity in society. The high complexity of DNN models and its widespread adoption has led to global energy consumption doubling every 3–4 months. Current energy consumption measures largely monitor system wide consumption or make linear assumptions of DNN models. The former approach captures other unrelated energy consumption anomalies, whilst the latter does not accurately reflect nonlinear computations. In this article, we are the first to develop a bottom-up transistor operations (TOs) approach to expose the role of nonlinear activation functions and neural network structure. As there will be inevitable energy measurement errors at the core level, we statistically model the energy scaling laws as opposed to absolute consumption values. We offer models for both feedforward DNNs and convolution neural networks on a variety of datasets and hardware configurations—achieving a 93.6%–99.5% precision. This outperforms existing floating-point operations (FLOPs)-based methods and our TOs method can be further extended to other DNN models.