APEnetX: A Custom NIC for Cluster Interconnects

The APEnet project is an established initiative aimed at designing interconnection boards based on FPGAs and tailored for use in HPC clusters whose nodes are arranged in a 3D toroidal network topology. Based on a custom communication protocol and network IPs, deployments of an APEnet Network Interface Card (NIC) can be configured for different operating environments. In this work, we describe APEnetX, the latest version of network architecture in the APEnet family, developed on Alveo U200 boards which leverage the 16 nm technology devices in the Ultrascale+ line by Xilinx. APEnetX features a PCIe Gen3×16 interface driven by the Queue Direct Memory Access (QDMA) Xilinx IP and adheres to the RDMA semantics, ensuring efficient high bandwidth data transfer without the involvement of the host operating system. The communication between adjacent nodes is handled through a proprietary lightweight protocol and enabled by the high-speed serial embedded transceivers of the FPGA, while a custom full crossbar switch and an optimized router allow for the implementation of a direct torus topology. Together with a proprietary software driver, a low level communication library and a dedicated MPI library, APEnetX lines up with current application standards in HPC systems. In addition, the release of an APEnetX network simulator allows for testing network functionalities at large scale. The achievement of this work is twofold: we enhance the performance of the previous APE net cards reaching 2 µs user-level latency over the PCIe for small packet transmission between topologically adjacent host nodes; we define the requirements of the communication generated by a reference spiking neural network simulator (NEST) to drive the co-design of future APE net generation.