In this paper, we propose a novel high speed VLSI architecture of full search-variable block size motion estimator(VBSME) supporting multiple reference frames. In the proposed architecture, four current macrobocks are concurrently compared with a single search window (SW) of the reference frame to find the best matched block. By reusing the SW of the reference frame and scheduling the architecture in pipeline, the proposed architecture can reduce the memory bandwidth and the execution time. Then, 71.73% of local memory size and 69.34% of system memory bandwidth were saved compared with non pipelined-MRFME (NP-MRFME). Under a operating frequency of 162.43 MHz, the architecture can support the real-time processing of 1280times720 picture size at 30 fps. The architecture has been prototyped in Verilog HDL and synthesized by Synopsys design compiler with Samsung 0.18 um standard cell library.