A method for building detection in very high spatial resolution multispectral images is presented. Buildings are detected using spectral and contextual information. First, potential building locations are enhanced on the basis of the spectral similarity between their roofs. To do this, the eigenvalue-based spectral similarity ratio is proposed. Next, the hit-or-miss transform (HMT) from mathematical morphology is used to assign pixels to buildings. To compute the HMT, fuzzy erosion and dilation are used. Additional processing based on size criteria is needed in some cases to separate buildings from roads. The method is tested on GeoEye and pan-sharpened Ikonos images. The preliminary results are promising.