Building a Data System for LCLS-II

The volume of data generated by the upcoming LCLS-II upgrade will present a considerable challenge for data acquisition, data processing, and data management. According to current estimates, one instrument could generate instantaneous data rates of hundreds of GB/s. In this high-throughput regime, it will be necessary to reduce the data on-the-fly prior to writing it to persistent storage. Even with data compression, it is expected that operating all four new instruments will produce an estimated 100 PB of data in the first year alone. We present a description of the envisioned LCLS-II Data System, which provides a scalable data acquisition system to acquire shot-by-shot data at repetition rates up to 1 MHz, an inline data reduction pipeline that provides a configurable set of tools including feature extraction, data compression, and event veto to reduce the amount of data written to disk, an analysis software framework for the timely processing of this data set, and an expanded data management system for accessing experiment data and meta-data.