An Optimal Method for Testing Jobs' Execution in MapReduce Based Systems

Over the past few years, parallel and distributed processing using the MapReduce programming paradigm has gained considerable attention. The purpose of such a model is to ease parallel computing and concurrent processing of large data stored on the Hadoop platform. However, MapReduce may experience performance and timing problems while using more complex and time-consuming data processing jobs. This paper focuses on improving MapReduce's performance. As a first step, we propose optimizing MapReduce jobs. The system conformance is then tested using a distributed test application. Finally, the test process is optimized using the Markov Decision Process.