This paper proposes a novel tracking strategy that can robustly track an object within a fixed environment. We define a robust model-based tracker using Kalman filtering combined with recursive least squares. The tracking is done by fitting successively more elaborate models on the tracked region and the segmentation is done by extracting the regions of the image that are consistent with the computed model of the target. We adopt a competitive and efficient dynamic Kalman filtering to adaptively update the object model by adding new stable features as well as deleting inactive features. The approach is implemented on FIRA Mirosot and tested in the context of ball tracking in the FIRA domain. The implementation of our approach has been proven to be efficient and robust.