Transport-Support Workflow Composition and Optimization for Big Data Movement in High-Performance Networks

High-performance networks (HPNs) are being increasingly developed and deployed to support the transfer of big data. However, such HPN-based technologies and services have not been fully utilized as their use often requires considerable networking and system domain knowledge and many application users are even not aware of their existence. This work develops an integrated solution to discover system and network resources and compose end-to-end paths for big data movement. We first develop profiling and modeling approaches to characterize various types of resources distributed in end systems, edge segments, and backbone networks. A comprehensive set of performance metrics and network parameters are considered in different phases including device deployment, circuit setup, and data transfer. Based on these profiles and models, we then formulate a class of transport-support workflow optimization problems to compose the best end-to-end path that meets various performance requirements. We prove this problem to be NP-complete and design pseudo-polynomial optimal algorithms. We conduct extensive simulations to evaluate the proposed algorithms in comparison with a greedy approach, and also carry out real-life experiments across different network segments in production HPNs to evaluate the validity of the constructed cost models and illustrate the efficacy of the proposed transport solution.