A flow-guided file layout for out-of-core streamline computation

We present a file reordering method to improve runtime I/O efficiency for out-of-core streamline computation. Because of the increasing discrepancy between the speed of computation and that of I/O on multi-core machines, the cost of I/O becomes a major bottleneck for out-of-core computation. Among techniques that reduce runtime I/O cost, reordering file layout to increase data locality has become popular in recent years. Better layout optimization relies on the knowledge of the data access pattern, which can be acquired from benchmarking. For streamline computation, we observe that the data access pattern is highly dependent on the flow directions. As a disk I/O request can generally be done more efficiently with shorter seek distances, we propose a novel flow-guided file layout method to improve the I/O performance. With a weighted directed graph to model the data access pattern, the file layout problem can be formulated as a linear graph arrangement problem. The goal is to minimized the sum of the disk seek time based on the linear distances between all pairs of adjacent graph nodes. We use a divide-and-conquer algorithm to approximate the optimal layout. The experimental results show that our flow-guided layout outperforms layouts that use space filling curves and some of the more recent cache-oblivious mesh layout methods.