Machine Learning-Based Framework for Resource Management and Modelling for Video Analytic in Cloud-Based Hadoop Environment

Hadoop framework has recently been adapted for use by the video analytics community for intensive, distributed video processing, storage. However, the challenge is to estimate the required amount of resources to be used in such an environment to fulfil the requirements of a user with requirements constraints. Therefore, it is important to understand how to model the performance of a Hadoop based implementation of video analytic applications in terms of meeting their performance goals. In this paper we propose the use of machine learning approachs in modelling the execution time based on the given resources. The prediction is based on parameters related to typical video analytic applications such as video file characteristics (e, g, resolution, file size, frame rate. etc.), cluster resource consumption,, Hadoop configuration values (reducer slots, tasks). The investigation carried out compares the use of different machine learning classifiers with regard to their best obtainable performance accuracies, show that a decision based model (M5P) outperforms a Linear Regression model, while the Ensemble Classifier, Bagging, out-performs these standard single classifiers. The research conducted bridges an existing research gap in video analytic-related performance predictions, whereby current research focuses on different application types, is largely limited to using standard learning algorithms such as SVM, Linear Regression, Multilayer Perceptron (MLP).