In this paper, we present an efficient algorithm to transcode MPEG-4 to H.264. The algorithm exploits the information decoded from the MPEG-4 stream to reduce H.264 encoding complexity. This information includes the MPEG-4 block modes, motion vectors, and residuals. The algorithm proceeds in two steps. First, a small set of most probable H.264 block mode candidates are obtained from an MPEG-4 to H.264 block mode conversion table. Then, motion estimation is performed for the candidate modes where, based on the residual information, the MPEG-4 motion vectors are either reused or refined. Experimental results show that the algorithm can speed-up the transcoding of QCIF and CIF sequences, from MPEG-4 visual simple to H.264 baseline profiles, by a factor of 2 to 3, with an acceptable loss in quality compared to the cascade spatial domain transcoding approach. It also provides significantly improved quality relative to current state-of-the-art methods.