The Best of Both Worlds: Distributed PCA That is Both Exact and Communication Efficient

The effectiveness of machine learning algorithms largely depends on the goodness of the representation of data. While the massiveness in dimension and amount of modern day data requires dimension reduction and feature extraction for efficient use of available computational resources, the use of un-correlated features is known to enhance the performance of such machine learning algorithms. Thus, an efficient representation learning approach should focus on dimension reduction as well as uncorrelated feature extraction. Even though Principal Component Analysis (PCA) and linear autoencoders are fundamental data processing tools largely used for dimension reduction, they can also be used to extract uncorrelated features when engineered properly. At the same time, factors like ever-increasing volume of data or inherently distributed data generation impede the use of existing centralized solutions for representation learning that require availability of data at a single location. This paper proposes two variants of an algorithm called FAST-PCA (Fast and exAct diSTributed PCA) based on a feedforward neural network-based system that learn data representations in a distributed setting such that they are reduced in dimension as well as have uncorrelated features. The proposed variants are meant to curb the communication overheads prevalent in the existing solutions and are shown to converge to the exact solutions at a linear rate. These claims are further supported by extensive numerical experiments.