ParaMET: A Parallel Framework for Efficient Medical Data Extraction on Tianhe-NG Supercomputer

In the burgeoning realm of data-driven medical research, the escalating scale and intricacy of contemporary medical datasets frequently surpass the processing capabilities of traditional computational environments. Specifically, I/O bottlenecks have emerged as pivotal constraints in several research areas. In this paper, a data service framework is introduced that harnesses supercomputers to parallelly access multi-modal datasets and supports multi-node processing, called ParaMET. To enhance user accessibility, a web-based user interface has been integrated, allowing permitted researchers to effortlessly interact via their laptops, complemented by an API suite available through SDK for those adept with supercomputing for deeper data manipulation. Through extensive empirical validation, our framework manifests a remarkable performance elevation in multi-node supercomputing settings, achieving acceleration of up to approximately 1000x compared to existing methods.