Advanced Algorithms for Distributed File Systems: Copying and Comparison

Operations on large file trees in a DFS (Distributed File System)-server are a bottleneck in large-scale cloud computing, such as distributed build systems for large software projects. Such operations take much longer than local operations, although the remote file system looks like a local directory to the user after mounting. Typical examples are (1) copying a file tree from a local file system to a remote file system and (2) computing the differences between the local and remote file trees. Speeding up these operations is possible if the specified DFS capabilities are utilized and the DFS driver and metadata layer are understood. In this paper, solutions for BeeGFS are proposed that allow speeding up the copying operation by a factor of 5-6, and the comparison operation by a factor of ten. The proposed approaches can be generalized and applied to other DFS types with some effort.