Achieving Observability at Scale through Federated Learning

We describe a novel approach to achieving observability at scale based on federated learning and our experiences and lessons learned with implementing it in a real-life commercial production environment. With the system described in this paper, the combination of Edge technologies with Artificial Intelligence allows one to predict up to six hours in advance that a component of a distributed system is about to fail. This permits the proactive execution of actions to either prevent the failure altogether or mitigate its impact on other services. Moving from incidents (unplanned work) to Predictive Maintenance, and thus planned work removes the risk of outages for service providers and administrators and increases the operations of large-scale distributed systems. By defining a set of key performance indicators, we are able to quantify the benefits of our approach.