Improving the Performance of Lustre File System in HPC Environments

As more and more data is being processed in High Performance Computing (HPC) systems scaling up in terms of the processing power and the amount of data to process, slow I/O can become a major performance bottleneck. Therefore, HPC systems usually use distributed parallel file systems which can lead to high I/O performance. However, it is still possible to enhance the performance of distributed parallel file systems by tuning the parameters or choosing the best configurations. To improve the performance of distributed parallel file systems, we first analyzed and compared the major ones being widely used in HPC environments. Then we focused on the Lustre file system for a further analysis because Lustre is one of the major ones due to its high performance. We evaluated Lustre in the direct I/O mode with various configurations and different parameter settings. Our experimental study suggests that we can obtain up to 114% performance improvements by modifying the numbers of Portal RPC threads in the OSS (Object Storage Server) and client.