Spectral mixture analysis (SMA) has been widely utilized to address the mixed-pixel problem in the quantitative analysis of hyperspectral remote sensing images, in which endmember extraction (EE) plays an extremely important role. In this paper, a novel algorithm is proposed to integrate both spectral similarity and spatial context for EE. The spatial context is exploited from two aspects. At first, initial endmember candidates are identified by determining the spatial purity (SP) of pixels in their spatial neighborhoods (SNs). Several SP measurements are investigated at both intensity level and feature level. In order to alleviate local spectra variability, the average of the pixels in pure SNs are voted as endmember candidates. Then, the spatial connectivity is utilized to merge spatially related endmember candidates by finding connection paths in a graph so that the number of endmember candidates is further reduced, which results in computational efficiency and better performance in SMA by alleviating global spectral variability. Experimental results on both synthetic and real hyperspectral images demonstrate that the proposed SP based EE (SPEE) algorithm outperforms the other popular EE algorithms. It is also observed that feature-level SP measurements are more distinguishable than intensity-level SP measurements to discriminate pure SNs from mixed SNs.