In this paper, a modified conditional random fields (CRFs) classifier, namely the support vector conditional random fields classifier with a Mahalanobis distance boundary constraint (SVRFMC), is proposed to perform the task of classification for high spatial resolution (HSR) remote sensing imagery. In SVRFMC, the CRFs model has the intrinsic ability of incorporating the contextual information in both the observation and labeling fields. Support vector machine (SVM) is set as the spectral term to get a more precise estimation of each pixel's probability of belonging to each possible class. To preserve the spatial details in the classification result, a Mahalanobis distance boundary constraint is considered as the spatial term to undertake appropriate spatial smoothing. By integrating SVM and a Mahalanobis distance boundary constraint, SVRFMC can not only avoid the explicit modeling of observed data, but can also undertake appropriate smoothing with the consideration of contextual information, thereby exhibiting more universality and validity in the application of HSR image classification, especially when the image has a complex land-cover class distribution and the training samples are limited. Three HSR images comprising QuickBird, IKONOS, and HYDICE imagery were utilized to evaluate the performance of the proposed algorithm in comparison to other image classification approaches: noncontextual multiclass SVM, a traditional object-oriented classifier (OOC), an object-oriented classification based on fractal net evolution approach (FNEA) segmentation (OO-FNEA), a simplified CRF model with boundary constraint (BC-CRF), and a recently proposed contextual classifier combining SVM and Markov random fields (Markovian support vector classifier). The experimental results demonstrate that the SVRFMC algorithm is superior to the other methods, providing a satisfactory classification result for HSR imagery, including both multispectral HSR imagery and hyperspectral HSR imager- , even with limited training samples, from both the visualization and quantitative evaluations.