On Optimizing Deep Neural Networks Inference on CPUs for Brain-Computer Interfaces using Inference Engines

Achieving real time Brain-Computer Interface (BCI) decoding requires insignificant signal processing latencies and accurate predictive models. Recently, deep neural networks demonstrated state of the art performance on brain activity decoding, yet they come with the caveat of long training and execution times especially in CPUs compared to classical machine learning algorithms. A solution for optimizing the neural networks models execution is to use tailored software runtimes for the computation hardware. In this paper, we investigate the performances of such inference engines on electroencephalography (EEG) based BCI data decoding with neural networks. We compare the performances of three different inference engines against the PyTorch deep learning framework: Open Neural Network Exchange (ONNX) runtime, Intel OpenVINO and Apache TVM. For medium (38527 parameters) and small (3945 parameters) sizes networks, ONNX runtime achieves lesser time in model inference with 5-6x average speedup. These findings guides the selection of the suitable acceleration runtime for online inference on CPUs.