In-Network Inference with Neuralp4: Auto-Generating Full-Fledged NN Models for PDPs

The emergence of data plane programmability motivated the offloading of inference based on Neural Networks (NNs) into network devices. Although performing inference in the data plane can provide faster results by doing computation at the line rate, the constraints of these devices make the offloading challenging. Existing solutions predominantly involve NN binarization or other simplifications to circumvent these constraints, and little emphasis is given to offloading standard NNs. In this paper, we present NeuralP4, an automated approach for generating full-fledged NN models to perform inference on packet forwarding devices. Because manually coding and configuring the NN would be laborious and error-prone, NeuralP4 automatically generates all necessary switch code and configuration files, supporting the offloading for diverse use cases. To execute all NN operations within the device's constraints, we distribute the NN layers across multiple network switches (labor division) and devise customized algorithms employing numerical conversion techniques and algebraic transformations. We implemented a proof-of-concept and conducted experiments considering five use cases. Our results show that NeuralP4 is as accurate as a traditional, server-only NN and that its incurred memory footprint is compatible with existing data plane devices. Furthermore, the system minimizes the manual effort required to write the NN code.