ACCO: Automated Causal CNN Scheduling Optimizer for Real-Time Edge Accelerators

Spatio-Temporal Convolutional Neural Networks (ST-CNN) allow extending CNN capabilities from image processing to consecutive temporal-pattern recognition. Generally, state-of-the-art (SotA) ST-CNNs inflate the feature maps and weights from well-known CNN backbones to represent the additional time dimension. However, edge computing applications would suffer tremendously from such large computation/memory overhead. Fortunately, the overlapping nature of ST-CNN enables various optimizations, such as the dilated causal convolution structure and Depth-First (DF) layer fusion to reuse the computation between time steps and CNN sliding windows, respectively. Yet, no hardware-aware approach has been proposed that jointly explores the optimal strategy from a scheduling as well as a hardware point of view.To this end, we present ACCO, an automated optimizer that explores efficient Causal CNN transformation and DF scheduling for ST-CNNs on edge hardware accelerators. By cost-modeling the computation and data movement on the accelerator architecture, ACCO automatically selects the best scheduling strategy for the given hardware-algorithm target. Compared to the fixed dilated causal structure, ST-CNNs with ACCO reach an ~8.4× better Energy-Delay-Product. Meanwhile, ACCO improves ~20% in layer-fusion optimals compared to the SotA DF exploration toolchain. When jointly optimizing ST-CNN on the temporal and spatial dimension, ACCO’s scheduling outcomes are on average 19× faster and 37× more energy-efficient than spatial DF schemes.

I. Introduction

Over the past decade, Convolutional Neural Networks (CNNs) have become an essential workhorse for computer vision tasks. Carrying forward its success, CNNs have been extended from the image processing field to spatio-temporal reasoning tasks. Recently, many successful models have been developed in different application domains, such as visual tracking [1], [2], acoustic perception [3], [4], biomedical information extraction [5], [6], etc. As shown in Fig. 1a, these designs typically include the time axis as an additional dimension of the input feature map which then allows to leverage conventional SotA CNN structures with a reshaped spatio-temporal input. However, adding the required time dimension to these CNNs inflates the CNN models with massive but unnecessary hardware overhead that hinders real-time edge processing.