Image spam poses a great threat to email communications due to high volumes, bigger bandwidth requirements, and higher processing requirements for filtering. We present a feature extraction and classification framework that operates on features that can be extracted from image files in a very fast fashion. The features considered are thoroughly analyzed regarding their information gain. We present classification performance results for C4.5 decision tree and support vector machine classifiers. Lastly, we compare the performance that can be achieved using these fast features to a more complex image classifier operating on morphological features extracted from fully decoded images. The proposed classifier is able to detect a large amount of malicious images while being computationally inexpensive.