On-the-fly sending: a low latency high bandwidth message transfer mechanism

Low latency high bandwidth network interface architectures are described. This paper proposes two types of architectures, atomic on-the-fly (OTF) sending with a header TLB, and block on-the-fly sending with protection stampable window memory. These techniques work very effectively with MEMOnet which is a class of network interface card (NIC) plugged into a memory slot. We are developing a network interface controller LSI called Martini. The Martini chip is used in two prototype network interface cards, DIMMnet-I based on MEMOnet, and RHiNET-2/NI based on PCI. On a DIMMnet-I, the software overhead needed to generate a message is only 1 CPU cycle and the estimated hardware delay is less than 100 ns using atomic OTF sending. The estimated achievable sending bandwidth of DIMMnet-I using block OTF sending is 984 MB/s which was observed in our experiments. This bandwidth is 7.4 times higher than the maximum bandwidth of PCI. This excellent performance is available for cheap personal computers with DIMM slots. This paper also describes the effects of block OTF sending for a PCI-based NIC.