Enabling Inference Inside Software Switches

Software Defined Networking (SDN) has been emerged to solve the problem of traditional network architectures. The ability of programmable switches renders us an opportunity to have computational tasks done in the switches. With this nice property, in this work, we investigate the potential of enabling machine learning inside a network. To this end, we propose a new architecture, Intra-Network Inference (INI), which equips each switch with a recently released component, called neural compute stick (NCS), to enable intra-switch neural network inference. Unlike conventional SDN architectures, which relay backend servers to enable inference, our INI performs inference locally at switches and, thereby, reduces the data forwarding overhead and inference latency.