Bringing the HPC reconstruction algorithms to Big Data platforms

The HPC and Big Data technologies are two important modern means of the scientific discovery process, but until recently, their development was evolved along two independent paths. Born and matured prior to the Big Data era, HPC scientific-oriented applications were implemented with the Message Passing Interface (MPI) protocol. MPI, however, was not designed for data-intensive applications and therefore requires project-specific extensions. An efficient processing framework offered by the Spark computational platform resolved many issues and built a strong foundation for their subsequent integration. At this time, the Spark ecosystem encompasses a rich collection of data analysis algorithms covering everything from SQL queries, to machine learning, to graph processing. Therefore, the paper proposes to extend this scope with the HPC applications of light source experimental facilities. Specifically, it focuses on the Spark-based integration of the SHARP distributed ptychographic solver developed by the team of the Center for Advanced Mathematics for Research Applications (CAMERA) at Berkeley. Aside from its practical value, this application represents an excellent reference use case that captures the major technical aspects of other beamline applications. The resulting approach is derived from the analysis of alternative communication models implemented in Spark-based distributed deep learning frameworks.