We developed a new active vision system using an artificial retina chip and the shape memory alloy actuator. A foveated CMOS retina chip for edge detection designed and fabricated for an image sensor of the developed system and the shape memory alloy actuator was used for mimicking the roles of the ocular muscles to track a desired target. Also, we proposed a new computational model that mimics the functional roles of our brain organs for generating the smooth pursuit eye movement. In our model, a neuromorphic model for the medial temporal cell generates motion energy, and the medial superior temporal cell is considered to generate an actuating signal so that the developed active vision system smoothly pursues the target with similar dynamics to the motion of our eyeball during the smooth pursuit. Experimental results show that the developed system successfully operates to follow the edge information of a moving object.