Tessellated Distributed Computing

The work considers the N-server distributed computing scenario with K users requesting functions that are linearly-decomposable over an arbitrary basis of L real (potentially non-linear) subfunctions. In our problem, the aim is for each user to receive their function outputs, allowing for reduced reconstruction error (distortion) $\epsilon $ , reduced computing cost ( $\gamma $ ; the fraction of subfunctions each server must compute), and reduced communication cost ( $\delta $ ; the fraction of users each server must connect to). For any given set of K requested functions — which is here represented by a coefficient matrix $\mathbf {F} \in \mathbb {R}^{K \times L}$ — our problem is made equivalent to the open problem of sparse matrix factorization that seeks — for a given parameter T, representing the number of shots for each server — to minimize the reconstruction distortion $\frac {1}{KL}\|\mathbf {F} - \mathbf {D}\mathbf {E}\|^{2}_{F}$ over all $\delta $ -sparse and $\gamma $ -sparse matrices $\mathbf {D}\in \mathbb {R}^{K \times NT}$ and $\mathbf {E} \in \mathbb {R}^{NT \times L}$ . With these matrices respectively defining which servers compute each subfunction, and which users connect to each server, we here design our $\mathbf {D},\mathbf {E}$ by designing tessellated-based and SVD-based fixed support matrix factorization methods that first split F into properly sized and carefully positioned submatrices, which we then approximate and then decompose into properly designed submatrices of D and E. For the zero-error case and under basic dimensionality assumptions, the work reveals achievable computation-vs-communication corner points $(\gamma ,\delta)$ which, for various cases, are proven optimal over a large class of $\mathbf {D},\mathbf {E}$ by means of a novel tessellations-based converse. Subsequently, for large N, and under basic statistical assumptions on F, the average achievable error $\epsilon $ is concisely expressed using the incomplete first moment of the standard Marchenko-Pastur distribution, where this performance is shown to be optimal over a large class of D and E. In the end, the work also reveals that the overall achieved gains over baseline methods are unbounded.

I. Introduction

We are currently witnessing a growing need for novel parallel computing techniques that efficiently offload computations across multiple distributed computing servers [1], [2]. To address this urgent need, a plethora of works has proposed novel methods that address various elements of distributed computing, such as scalability [3], [4], [5], [6], [7], [8], [9], [10], [11], privacy and security [12], [13], [14], [15], [16], [17], [18], [19], [20], [21], as well as latency and straggler mitigation [22], [23], [24], [25], [26], [27], [28], to mention just a few. For a detailed survey of such related works, the reader is referred to [29] and [30]. In addition to the above elements, the celebrated computation-vs-communication relationship stands at the very core of distributed computing as a fundamental principle with profound ramifications. This principle appears as a limiting factor in a variety of distributed computing scenarios [24], [27], [32], [33], [34], [35], [36], [37], [38], [39], [40], [41], [42], [43] where indeed communication and computation are often the two intertwined bottlenecks that most heavily define the overall performance.