In this paper, we present the development of a robust visual servo system for object tracking applications of a nonholonomic mobile robot. The system mainly consists of an adaptive shape tracking algorithm and a robust visual servo controller. The adaptive shape tracking algorithm is designed to automatically detect the shape contours of moving objects, extract the shape parameters, and continuously track the object in shape parameter space. Based on direct measurements of the shape parameters, the visual servo controller is designed using the sliding mode control technique. Through a Lyapunov-based stability analysis, a sufficient condition on the selection of control gains to achieve the tracking goal in finite time is provided, and simulation and experimental tests of the proposed approach are illustrated.