Detection of Faulty Nodes in Distributed Environment using Machine Learning

Faulty nodes are expected to have a significant effect on the performance of Big Data systems. The cause of discharges, on the other hand, is a complex issue. The previous work has mostly focused on identifying Stragglers, optimizing the quality of programming, and root cause investigation. These methods can't offer useful information unless they can assist users automate their services. We are developing a framework that determines when there will be stragglers and must identify gaps to prevent those circumstances. This research proposed a dynamic log collection of heterogeneous virtual machine data and detect the faulty node based on execution performance using various parameters. We introduce multi-task learning mechanisms in this paper, which exchange knowledge between the different versions of nodes. The system first collects log data from available data nodes and applies an unsupervised machine learning technique to detect the straggler. Moreover, once a straggler has been detected by the learning algorithm, it tries mitigating such nodes instead of eliminating those nodes. The major benefit of this research is that the system can handle massive data with minimum data nodes or even system efficiently work with already heated nodes in a distributed environment. Our compositions absorb the shared context in our data, enhancing sweeping statement efficiency on minimal data, unlike naive inter learning frameworks.