A Memory-aware Performance Optimization of Tensor Programs for Embedded Devices

We present a performance optimization method of tensor programs for embedded devices with memory constraints. Based on the Tensor Virtual Machine, which automatically optimizes tensor programs, we introduce four techniques to reduce memory consumption. First, we improve the search templates for convolution tensor programs to reduce memory consumption. Second, we improve the planner by focusing on a specific target processor, a central processing unit. Third, to avoid unnecessary memory allocations and copies, we improve the runtime. Last, we introduce how to choose the best configuration. Experimental results show that the proposed method gives better performance with respect to both the memory footprint and the inference time when compared to conventional ones.