Video-based information collection has become an important research direction, and moving object tracking technique plays a key role nowadays. The classic corner tracking algorithm doesnpsilat meet the real-time requirement, and loses the object mostly due to occlusions, the change of geometrical scale or/and some similar objects approaching to the object. To solve the problems, a new algorithm based on Kalman filter and point matching estimation is proposed in the paper. Combined with predicting the targetpsilas location based on Kalman filter, the extracted multi-scale corner points which are geometrically invariant are given different weights for the responsible function, and then divided the image into blocks, the location is tracked by its average vector. The experiments results show that the proposed method can perform well in on-line and robust tracking systems.