Concurrency and Parallelism in Speeding Up I/O and CPU-Bound Tasks in Python 3.10

Modern data storage systems often fetch data from a large number of sources. Although synchronous data transfer is simple as it can use simple scheduling of data delivery, it scales poorly. This paper reports our investigations on asynchronous data transfer and processing aiming to substantially improve the system efficiency. The implementation assumes the most recent version of Python programming language, and exploits concurrency and parallelism provided by the relevant Python modules. In particular, asyncio module can be used to parallelize massive I/O requests, multiprocess module is useful to increase the throughput of CPU-bound tasks, and concurrent futures module offers concurrency and parallelism of jobs. The observed speed up compared to a baseline synchronous programming is significant, and it can yield between 77 to 96% reduction in the required overall runtime depending on the specific modules and functions used on a single multi-core CPU.