IFLustre: Towards Interference-Free and Efficient Storage Allocation in Distributed File System

Distributed file systems (DFSs) are widely used in large scale computing environments such as cloud computing and high performance computing systems (HPC) where thousands of applications are executed simultaneously. To support applications that produce and process a large amount of data, DFSs manage limited storage resources in the system and allocate the resources to the applications. To efficiently utilize limited storage resources, it is important to consider the I/O characteristics of applications as well as the effect of resource sharing among multiple applications in the system. In this paper, we first perform empirical performance analysis and investigate the effect of I/O interference caused by storage resource allocation. Based on our empirical performance analysis, we propose IFLustre towards interference-free and efficient storage allocation in Lustre file system. IFLustre first utilizes previous execution logs to determine the necessary storage resources for the application and predict the throughput and runtime of the application. Then, it allocates storage resources via file system configurations based on the prediction results and the DFS allocation status. This allows IFLustre to allocate sufficient resources to the application while mitigating interference from resource sharing. The experimental results show that IFLustre can improve the performance by up to 47% compared with Lustre.