MAD - Monitoring ALICE Dataflow

ALICE (A Large Ion Collider Experiment) is the heavy-ion detector designed to study the physics of strongly interacting matter and the quark-gluon plasma at the CERN Large Hadron Collider (LHC). Following a successful Run 1 which ended in February 2013, the ALICE data acquisition (DAQ) entered a consolidation phase to prepare for Run 2 which will start in the beginning of 2015. One of the identified points for improvement was the monitoring of the experiment dataflow - from the data arrival on the DAQ farm via the readout links up to its shipment to CERN's main computer centre. To address this requirement, the ALICE MAD (Monitoring ALICE Dataflow) system was developed. MAD uses the MonALISA framework as core module to gather, process, aggregate and distribute monitoring values from the different processes running in the distributed DAQ farm. It allows for data not only to be pulled from the data sources but also to be pushed by dedicated data collectors or the data source processes themselves. To provide a complete view of the data acquisition status, the set of monitored metrics vary from the backpressure status on the readout links to event counters in each of the DAQ nodes and aggregated data rates for the whole data acquisition. MAD also injects alarms in the Orthos alarming facility whenever abnormal conditions are detected. To support the ALICE shift crew, MAD interfaces with a dedicated web-based GUI that uses WebSockets to provide dynamic and on-time status displays. Designed as a widget-based system, it allows not only an easy integration of new visualization blocks but also customization of the information displayed to the shift crew based on the ALICE activities.