Prediction, including intra prediction and inter prediction, is the most critical issue in H.264 decoding in terms of processing cycles and computation complexity. These two predictions demand a huge number of memory accesses and the total decoding cycles. In this paper, an efficient hardware architecture for real-time implementation of intra and inter predictions algorithm used in H.264 video coding standard is adopted. Compared with conventional architecture, the predict efficiency is enhanced. Under different prediction modes, our design is able to decode each macroblock (MB) within 500 cycles. The Verilog RTL of intra prediction is verified to work at 103 MHz and the inter prediction is verified to work at 81 MHz in a Xilinx II FPGA.