Research and Implementation of Efficient Parallel Processing of Big Data at TELBE User Facility

In recent years, improvements in high-speed Analog-to-Digital Converters (ADC) and sensor technology has encouraged researchers to improve the performance of Data Acquisition (DAQ) systems for scientific experiments which require high speed and continuous data measurements - in particular, measuring the electronic and magnetic properties of materials using pump-probe experiments at high repetition rates. Experiments at TELBE are capable of acquiring almost 100 Gigabytes of raw data every ten minutes. The DAQ system used at TELBE partitions the raw data into various subdirectories for further parallel processing utilizing the multicore structure of modern CPUs. Furthermore, several other types of processors that accelerate data processing like the GPU and FPGA have emerged to solve the challenges of processing the massive amount of raw data. However, the memory and network bottlenecks become a significant challenge in big data processing, and new scalable programming techniques are needed to solve these challenges. In this contribution, we will outline the design and implementation of our practical software approach for efficient parallel processing of our large data sets at the TELBE user facility.