Selected algorithms in graph theory and linear algebra adapted for Big Data

This extended abstract summarizes the research done [1][2][3] in an effort to develop scalable approaches to select few computationally hard problems from the domains of linear algebra and graph theory in order to make them viable for large datasets.The open source [4][5] solutions herein are the first of their kind both in handling large datasets in a distributed manner and in the map reduce paradigm, which also motivated the choice of the problems.Apache Spark [6] has been chosen as the common infrastructure. This allows for a shared basis and makes the different problems quantitatively comparable in at least one dimension, but also abstracts away various distributed systems basics like communication and reliability, which, while important, don’t contribute to the results. The first solution addresses the power flow problem using the Newton-Raphson method as a well-known approach for solving systems of non-linear equations. In its core lie the necessary matrix operations that are applicable to a large array of other problems. The second problem is the island detection problem in power networks. Here, two classical approaches are implemented. First, the connectedness analysis, which can be reduced to the connected components problem, a common challenge in graph theory. Second, the full matrix analysis, which relies on multiplying the connectedness matrix with itself multiple times, until the islands become apparent. In the core of this approach lie matrix multiplication and the problem of connected components in graphs, which are both hard problems in their respective fields. The third and final solution is a software package that offers tools for spectral graph analysis, ultimately provide the building blocks for the spectral graph sparsification algorithm [7], while also being useful by themselves, e.g. generating a Laplacian matrix. Spectral graph sparsification is a complex procedure for reducing the number of edges, while retaining the spectral properties of the graph. Given that every graph can be sparsified, it becomes clear how this is particularly useful for large graphs.These complex algorithms, which are extensively used in industrial applications, are adapted to the requirements dictated by the Big Data needs. This means they are made suitable for a distributed environment and large amounts of data in such a way, that the increasing problem size can be compensated with scaling of the computing environment. Given that all of them have only made marginal improvements in the performance of algorithms, if any, the distributed approach is the next logical venue for improvement. In a way, this means compensating computational complexity with computing power to overcome large amounts of data, which wouldn’t be possible by using just the classical algorithms. The solutions were verified with experiments conducted using the Amazon Elastic MapReduce (EMR) framework. It is a natural choice, as the integrated support for Apache Spark reduces the margin for configuration error and facilitates rapid deployment. Special care has been taken to select illustrative and publicly available datasets for the verification to show that the solution can handle problems relevant in the real world and that their size supports the common understanding of the Big Data definition. The results indicate a reasonable execution time achieved by open-source software that is easily scalable and serve to direct further research in overcoming the current limitations. In a broader sense, although the problems herein seem very specific in their domain of application, the underlying core mechanics contain common mathematical challenges, applicable to a broad spectrum of problems.