Towards a Scalable and Cost Efficient Serverless Scheduler for Dask

HPC workflows run either on supercomputers, shared through queues, or Cloud virtual machines, allocated for the experiment. Both options are ill-suited for exploration scenarios, where the user explores the dataset by applying small computations to parts of a much bigger dataset and analyzing the result, either because of the queue time present on supercomputers or costs generated even when idle on traditional IaaS cloud offerings. We propose FaaS-Dask, a runtime for Dask that is capable of running on FaaS offerings, providing the same tailored experience as cloud virtual machines but billing only for executed computation. We will explore current FaaS challenges and optimizations available, and compare execution on our environment with traditional cloud Dask clusters, showing that FaaS-Dask is cheaper in some scenarios, but there is still work to be done to make it as fast as current options.