FinD: Fine-grained Dynamic Task Scheduling with Lightweight Threads on Many-core Processors

The emergence of many-core processors presents significant opportunities for large-scale multithreading. Exploiting these intensive computing resources poses an urgent challenge for data processing systems. Although current big data frameworks offer automatic parallel computing capabilities, their coarse-grained parallelism hinders multi-threaded scalability on many-core processors. The NUMA architecture exacerbates load imbalance and resource wastage during the processing of skewed datasets. This paper introduces FinD, a fine-grained data processing framework that maximizes the utilization of many-core resources with lightweight threads and effectively addresses data skew through dynamic task scheduling. FinD incorporates bilateral NUMA-aware work stealing to achieve load balancing and reduce remote memory access across NUMA nodes. Experimental results demonstrate that FinD achieves a speedup of up to $9.7 \times$ compared to popular frameworks on big data benchmarks and HPC workloads while reducing overall execution time by 40% on skewed datasets.