Big Data Mining And Clustering Using Distributed Bayesian Matrix Decomposition

Technological advancement has enabled us to store and process massive amounts of data in a much shorter period of period. The phrase "Big Data" merely describes the enormous amounts of data that are now regularly utilized in business and academic settings. The emphasis here is not only on data collection, but also on careful data analysis in order to obtain meaningful results. There are several approaches to dealing with massive data streams. Data clustering into compact units is one such method. This not only reduces data size but also allows it to be used more efficiently. Because all of these devices are from different manufacturers, the data generated is heterogeneous in nature and contains noise. A fundamental problem is gaining knowledge from heterogeneous, noisy data. Several strategies for mining knowledge from big data have already been proposed. Because big data is frequently acquired from dispersed locations and stored on multiple machines, handling these extremely large data sets in a single machine is either inefficient or impossible. We provide a distributed Bayesian matrix decomposition model (DBMD) for big data mining and clustering. Statistical inference, alternate direction method of multipliers (ADMM), and accelerated gradient descent (AGD) are the three techniques utilized to implement distributed computing. Investigated are the theoretical convergence tendencies of various approaches. We offer a perfect plug-in weighted average that minimizes estimate variance to get over noise heterogeneity. In real-world experiments, our algorithms outperform or are on par with two popular distributed approaches, Scalable- nonnegative matrix factorization (NMF) and Scalable- k-means++, which are confirmed by our theoretical results from synthetic studies.