Offloading Collective Operations to Programmable Logic on a Zynq Cluster

This paper describes our architecture and implementation for offloading collective operations to programmable logic in the communication substrate. Collective operations - operations that involve communication between groups of co-operating processes - are widely used in parallel processing. The design and implementation strategies of collective operations plays a significant role in their performance and thus affects the performance of many high performance computing applications that utilize them. Collectives are central to the widely used Message Passing Interface (MPI) programming model. The programmable logic provided by FPGAs is a powerful option for creating task-specific logic to aid applications. While our work is evaluated on the Xilinx Zynq SoC, it is generally applicable in scenarios where there is programmable logic in the communication pipeline, including FPGAs on network interface cards like the NetFPGA or new systems like Intel's Xeon with on-die Altera FPGA resources. In this paper we have adapted and generalized our previous work in offloading collective operations to the NetFPGA. Here we present a general collective offloading framework for use in applications using the Message Passing Interface (MPI). The implementation is realized on the Xilinx Zynq reference platform, the Zedboard, using an Ethernet daughter card called EthernetFMC. Results from microbenchmarks are presented as well as from some scientific applications using MPI.