Image-based spam is becoming a new threat to the Internet and its users. In our early work, we proposed an image filtering system which detects the spam image by matching with user-specified image content using SIFT algorithm. In order to further improve efficiency, we develop a quick image matching algorithm instead of SIFT. After using difference-of-Gaussian to extract image feature points, we adopt geometry transform to judge whether two images are matched. Experimental results show that the proposed method can identify image spam without the need of OCR and it can achieve a good performance. In addition, we adopt mean shift algorithm to locate the highest density area of feature points, which improves the performance of the system.