Practical issues of statistical path monitoring in overlay networks with large, rank-deficient routing matrices

Overlay networks can be used to find working paths when direct underlay paths are anomalously slow, e.g. because of a network fault. Overlay paths should not use links that are involved in a fault, so choosing which overlay path to use often requires path monitoring, which introduces an overhead. By using a routing matrix 'M' to define which links are used in each path, and sorting the matrix according to the degree of independence of paths, we can choose a subset of paths to monitor, and so reduce overheads. The performance metrics of the unmonitored paths are then predicted based on information inferred from the monitored. Previous work has shown how statistical prediction errors can occur, even in small networks (11 nodes, 110 paths), when monitoring fewer paths than the rank of the matrix. This paper extends previous work by showing that collinear relationships between variables in paths of larger routing matrices in networks with tens of nodes and 1-2 000 paths are a large source of errors in larger networks. We show that mitigation of such errors leads to improved path metric prediction and anomaly detection.