This paper presents a MP3 / AAC decoder with hardware and software co-design method. In this method, both the flexibility of the software design and the lower power consumption and higher speed of the hardware design are considered. Based on the analysis of algorithm complexity and the compatibility, the Huffman and the IMDCT hardware accelerator module are developed to accelerate the decoding process. At the same time, the fast algorithms adopted in the hardware accelerators have good expandability. The MP3 / AAC decoder is implemented and verified on Xilinx XUP VIIP board. The test results show that MP3 / AAC decoder could reach realtime processing when the Power PC runs at the frequency of 300MHz.