An automated Cryo-EM computational environment on the HPC system using Pegasus WMS

Cryogenic electron microscopy (Cryo-EM) is a method applied to samples embedded in vitreous water and cooled to cryogenic temperatures and has been in development for the past fifty years. Recently, thanks to technological advancements, we can reach a near-atomic resolution of biological molecules. Recent progress in methodology has created an entirely new set of challenges to overcome - among them, the specific environment of the HPC system and coordination and automation of the initial stages. Our solution is an automated Cryo-EM image pre-processing service tailored to an HPC environment with close to real-time feedback allowing the researchers to interact with the data acquisition session located in a facility remote to the HPC cluster. We automated the data transfer, created a service around the Pegasus Workflow Management System, kept the user interaction minimum, and offered the researcher an option to start the pre-processing right after initiating the microscope session. The users receive real-time feedback enabling them to interact with the data acquisition, adjust it and collect a better dataset. Our solution allows the researchers to focus on the results, not the platform, and has the potential to become increasingly practical over time.