The Real-Time System for Distribution of Clock, Control and Monitoring Commands With Fixed Latency of the LHCb Experiment at CERN

The Large Hadron Collider beauty (LHCb) experiment has gone through a major upgrade for LHC Run 3 at European Organization for Nuclear Research (CERN), scheduled to start in the middle of 2022. The entire readout system of the experiment has been replaced by front-end and back-end electronics that are able to record LHC events at the full LHC bunch crossing rate of 40 MHz. To maintain synchronicity across the full system, clock and control commands are originated from a single readout supervisor (SODIN) and distributed downstream through a passive optical network (PON) infrastructure, reaching all the back-end cards of the readout system, via a two-level hierarchy and using PON splitters. A field programmable gate arrays (FPGA) firmware core, called LHCb-PON and based on the CERN timing, trigger, and control (TTC)-PON, is instantiated on SODIN and all back-end cards and provides the communication protocol while ensuring a stable distribution of control commands within the readout period of 25 ns and a clock signal with fixed and deterministic latency with a precision of a few hundreds of picoseconds. This is achieved by the firmware backbone that provides clock recovery and ensures the transmission of a control word downstream with fixed latency. In addition, the LHCb-PON also provides monitoring functionalities over the upstream link, with less strict timing requirements, connecting multiple back-end cards to the central SODIN. This is achieved by having each back-end card send data upstream through a time-division multiplexing (TDM) scheme. This article describes the implementation, measurements, and results from the usage of the system during the startup of the LHC Run 3.