Integer ConvNets on Embedded CPUs: Tools and Performance Assessment on the Cortex-A Cores

Quantization via fixed-point representation is commonly used to reduce the complexity of Convolutional Neural Networks (ConvNets). It is particularly suited for accelerating edge-inference on embedded devices as it enables to reduce resource requirements with no loss of prediction quality. However, porting integer ConvNets on low-end CPUs is not straightforward: it calls for proper software design and organization with a high degree of hardware awareness. Today there are plenty of fixed-point libraries integrated into different inference engines which provide design support. The aim of this work is to review the most stable tools and analyze their performance on different use-cases processed on embedded boards powered by Arm Cortex-A cores. The collected results provide an interesting analysis with useful guidelines for developers and hardware designers.