A high-performance cluster storage server

An essential building block for any data grid infrastructure is the storage server. In this paper we describe a high-performance cluster storage server built around the SDSC Storage Resource Broker (SRB) and commodity workstations. A number of performance critical design issues and our solutions to them are described. We incorporate pipeline optimizations into SRB to enable the full overlapping of communication and disk I/O. With these optimizations we were able to deliver to the application more than 95% of the disk throughput achievable through a remote connection. Then we show how our approach to network-striped transport is effective in achieving aggregate cluster-to-cluster throughput which scales with the number of connections. Finally, we present a federated SRB service over MPI that allows fast TCP connections to stripe data across multiple server disks reaching 97% of the combined write capacity of multiple nodes.