Distributed Execution of Dask on HPC: A Case Study

Performing computations on a large dataset using a system with limited memory can present difficulties, as the entire dataset may not be accommodated in memory. Dask is a framework that facilitates parallel and out-of-core computation, thereby demonstrating its efficacy in enhancing the utility of modern hardware for processing larger datasets. High Performance Computing (HPC) is a distributed computing architecture that aims to enhance computational performance by distributing the workload across a network of interconnected computing nodes. Each node in an HPC system is typically equipped with its own set of processors, memory, and storage resources, and is responsible for executing a subset of the overall workload. By utilizing this parallel processing approach, HPC systems can handle large-scale computational tasks that are beyond the capabilities of individual computing nodes. This work presents an approach for managing and overseeing the distributed execution of Dask on HPC using a workload manager that is frequently utilized in HPC environments. We conduct a performance evaluation of Dask’s distributed execution capabilities on HPC. Our evaluation focuses on two critical performance metrics, namely, execution time and memory management. Specifically, we aim to investigate the efficiency of Dask’s distributed execution approach in the context of HPC, and identify potential challenges and opportunities associated with its deployment in this environment.