Overcoming the data-scale management challenge in large, dynamic networks

A large military network may deliver hundreds of applications to thousands of users across many tens of thousands of devices, under conditions of constant and unpredictable change. This poses insuperable problems for traditional network management systems, which attempt to assemble a coherent view of infrastructure operations and their impact on application performance and business processes by aggregating data from thousands of infrastructure elements, agents, and probes into a monolithic master database. Such an approach cannot address the fundamental data-scale management challenge posed by large, dynamic networks: making sense of the sheer volume of data that must be collected, aggregated, and stored to manage such networks; and there are significant additional obstacles to success. These problems can be overcome with a distributed, flow-based approach using techniques adapted from high-energy physics research to address the data-scale management challenge.