Hardware-aware Workload Distribution for AI-based Online Handwriting Recognition in a Sensor Pen

Time series-based applications such as recognition of handwriting benefit from using Deep Neural Networks (DNNs) in terms of accuracy and efficiency. Due to strict power and memory limitations of embedded platforms in the Internet-of-Things (IoT), the inference of such DNNs is usually performed on more powerful and less constrained devices. However, inference on mobile devices such as smartphones or tablets leads to high system requirements. In this paper, we present our approach for distributing computational workload between sensor pen and a mobile device for handwriting recognition. This not only results in lower system requirements for mobile devices, but also opens up the possibility of storing compressed sensor data on the pen when remote devices are not available or connected. Therefore, we first quantize weights of the DNN layers to significantly reduce the memory footprint and optimize a Connectionist Temporal Classification (CTC) algorithm to minimize the link bandwidth. The latter allows transmitting only a small matrix from the pen to the mobile device, on which the decoding is performed. To further improve the accuracy of handwriting recognition, we add a Language Model (LM) to the decoding algorithm executed on the mobile device. As a result, we are able to reduce the memory footprint of the DNN weights by almost a quarter to 736.64 KB. Additionally, applying CTC-based beam search and an LM, we can reduce the link bandwidth by almost 80%. Based on the RISC-V Instruction Set Architecture (ISA), we finally define memory and performance requirements to enable in-pen DNN inference.