3DTV is becoming more and more popular but the 3D sequences are still very few. In this paper, we proposed an efficient 2D to 3D video conversion method based on skeleton line tracking. In our method, for a key frame, the foreground object and the corresponding depth image can be obtained by interactive method. For a non-key frame, we first generate the skeleton lines of the object in the previous frame and predict them in the current frame, then recover the object by the Lazy Snapping method. A robust and fast optical flow method is introduced to make the prediction better. At last, the depth image is generated to composite the stereo image. Because only the skeleton lines of the object rather than the whole object are tracked, the computational complexity is much lower than other tracking methods. The experimental results show that our method is feasible.