Intelligent surveillance systems have become an important research issue recently. They can provide an early warning or help us to retrieve interested video frames. The motion detection techniques play an important role in these two applications. In this paper, we propose an effective surveillance video retrieval method based upon the motion detection technology. We can use this method to search frames with moving objects. We also use the feature, motion vectors, to retrieve the frames with objects in pre-defined moving directions. We use video samples to show the feasibility of our method.