Techniques to provide run-time support for solving irregular problems

We present a runtime library design based on the two-phase collective I/O technique for irregular applications. The design is motivated by the requirements of a large number of ASCI (Accelerated Strategic Computing Initiative) applications, although the design and interface is general enough to be used from any irregular applications. We present two designs, namely, Collective I/O and Pipelined Collective I/O. In the first scheme, all processors participate in the I/O at the same time, making scheduling of I/O requests simpler, but creating a possibility of contention at the I/O nodes. In the second approach, processors are grouped into several groups, so that only one group performs I/O simultaneously, while the next group performs communication to rearrange data, and this entire process is pipelined. This reduces the contention at the I/O nodes but requires more complicated scheduling and a possible degradation in communication performance. We obtained up to 40 MBytes/sec application level performance on the Caltech's Intel Paragon (with 16 IO nodes, each containing one disk) which includes on-the-fly reordering costs. We observed up to 60 MBytes/sec on the ASCI/Red machine with only three I/O nodes (with RAIDS).