Grid Computing is a form of distributed Computing that has emerged as a viable solution to meet the ever increasing needs for computational power and data management capability. The load scenario in a grid environment is dynamic in nature and thus requires appropriate load balancing mechanism to efficiently handle computational load. In this paper we have presented a hierarchical load balancing algorithm which addresses two important aspects, namely Multiple Parameters based Load Balancing (MPLB) and job scheduling at Logical Hierarchical Levels (LHL). First we proposed a dynamic load balancing technique over a tree based grid model and demonstrate the efficacy of Hierarchical Job Scheduling (HJS) approach over Flat Structure Job Scheduling (FJS). Experiments have been carried out using a grid test bed with gridgain 2.0 as middleware and results show that HJS performs better than FJS.