IN3: A Framework for In-Network Computation of Neural Networks in the Programmable Data Plane

Neural networks have been widely used in networking applications due to their high accuracy and generalization. However, the traditional approach of collecting network features from switches and transmitting them to the controller introduces high traffic overhead and extra communication latency. In-network computing (INC) mitigates this issue by running computing tasks directly in the networks on the data paths using programmable data planes (PDP). However, it is challenging to embed more sophisticated computing tasks, such as neural networks, in the networks due to the limitations in the computation and storage resources of PDP. To address this challenge, we propose IN3, a framework that enables complete neural network inference in PDP. IN3 uses model compression techniques to reduce the memory and computational requirements of given neural networks. Additionally, a purposely designed data plane pipeline for per-flow features computation and inference is proposed. We implemented a testbed prototype (based on Intel Tofino ASIC), and experimental results demonstrate that IN3 effectively reduces memory usage, while significantly decreasing the inference time. IN3 demonstrates the feasibility of implementing neural networks in PDP, and we identify potential future research directions for this issue.