A distributed monitoring system for jobs processing

In our earlier article "Framework for a Scalable Distributed Job Processing System" (paper submitted in to 'International Conference on Software and Data Engineering' 2010 held at Penang, Malayasia) we suggested a simpler implementation of a job processing system using the core technologies currently available. The system had the capability to scale up and achieve the required performance, so that the average waiting time and overall turnaround time is minimized, thus effectively minimizing the total cost of ownership (TCO). In this work, we are proposing a robust Job Monitoring system that will enhance the earlier proposed Job Processing system and help manage the distribution of jobs easier, detect failures and initiate corrective action. By using the heartbeat messages, an intelligent decision can be made to dispatch jobs to the appropriate processor, thus reducing wait time considerably. The data collected by the monitor can also be used for effective measurement of the performance of the system.