A Case Study in I/O Disaggregation using PCI Express Advanced Switching Interconnect (ASI)

Decoupling the processor and I/O subsystem provides immense benefits that include high availability, efficient allocation and cost-effective upgrade of system resources. Such a disaggregation model calls for a high-performance interconnect to isolate the processor and I/O subsystem domains, yet provide the veneer of a single system. PCI express (PCIe) is one such interconnect and is becoming the de-facto I/O fabric. However, PCIe, as specified currently, provides limited support for I/O disaggregation and does not yet natively support dynamic sharing of I/O resources amongst processor subsystems - this is the next major step in I/O disaggregation. PCI express advanced switching interconnect (ASI) is well-suited for enhancing the capabilities of PCIe in a non-disruptive manner. ASI is built upon PCIe and has the innate ability to co-exist with PCIe devices due to its commonality of the link/physical layer with PCIe as well as its native support for encapsulating PCIe packets. Towards a simple yet illustrative demonstration of ASI-based disaggregation of PCIe devices, we employed StarGen's ASI products for creating a basic ASI fabric and disaggregated a PCIe based GigE NIC from a host system. The initial set of results showed a marginal effect on the application's latency, but contrary to expectations, the throughput was significantly impacted. Further analysis revealed that this unexpected drop in throughput could be rectified easily and indeed, the final results confirm that the use of ASI for supporting I/O disaggregation does not result in sub-optimal utilization of the GigE NIC