Intent-Driven Multi-Engine Observability Dataflows for Heterogeneous Geo-Distributed Clouds

With the growth of multi-cloud computing across a heterogeneous substrate of public cloud, edge, and on-premise sites, observability has been gaining importance in compre-hending the state of availability and performance of large-scale geo-distributed systems. Collecting, processing and analyzing observability data from multiple geo-distributed clouds can be naturally modelled as dataflows comprising chained functions. These observability flows pose a unique set of challenges in-cluding (
$a$
) keeping cost budgets, resource overheads, network bandwidth consumed and latency low, (b) scaling to a large number of clusters, (c) adapting the volume of observability data to satisfy resource constraints and service level objectives (d) supporting diverse engines per dataflow depending on each processing function of the flow and (e) automating and optimizing placement of observability processing functions including closed-loop orchestration. Towards this end, we propose Octopus, a multi-cloud multi-engine observability processing framework. In Octopus, declarative observability dataflows (DODs) serve as an intent-driven abstraction for site reliability engineers (SREs) to specify self-driven observability dataflows. A dataflow engine in Octopus, then orchestrates these DODs to automatically deploy and self-manage observability dataflows over large fabrics spanning multiple clouds and clusters. Octopus supports a mix of streaming and batch functions and supports pluggable run-time engines, thereby enabling flexible composition of multi-engine observability flows. Our early deployment experience with Octopus is promising. We have successfully deployed production-grade metrics analysis and log processing data flows in an objective-optimized fashion across 1 cloud and 10 edge clusters spanning continents. Our results indicate data volume and WAN bandwidth savings of
$2.3\mathrm{x}$
and 56 %, respectively, the ability to support auto-scaling of DODs as input load varies, and the ability to flexibly relocate functions across clusters without hurting latency targets.