An approach for MapReduce based log analysis using Hadoop

Log is the main source of system operation status, user behavior, system's actions etc. Log analysis system needs not only the massive and stable data processing ability but also the adaptation to a variety of scenarios under the requirement of efficiency and performance, which can't be achieved from available standalone analysis tools or even single computing framework. Hence we propose a log analyzer with the combination of Hadoop and Map-Reduce paradigm. The joint of Hadoop and MapReduce programming tools makes it possible to provide batch analysis in minimum response time and in-memory computing capacity in order to process log in a high available, efficient and stable way.