CNN Inference Workload Distribution Considering Accumulative Effect of Receptive Fields

Operating an artificial intelligence and machine learning (AI/ML) model requires significant computing power, posing a substantial challenge for edge devices with limited resources. A potential way to accelerate the operations is to distribute the workload to multiple collaborative devices. For the inference using a convolutional neural network (CNN), the workload distributed is complicated by the overlapping input tensors among workers due to the accumulative effect of receptive fields (AERF). This factor incurs extra communication time, which was overlooked by prior studies. In this paper, we propose an approach to CNN inference time minimization by aggregating the inference workload distributed to multiple heterogeneous computing devices. Our result shows a 10% speed-up against the benchmark.