In this paper, we propose a novel ensemble-based classification system for improving the classification accuracy of hyperspectral images. To generate the ensemble, we run the mean-shift (MS) algorithm several times on different bands randomly selected from the hyperspectral cube and with distinct kernel width parameters. The resulting set of MS maps are then successively labeled via a pair wise labeling procedure with respect to a spectral-based classification map generated by the support vector machine (SVM) classifier. To this end, for each region in the MS maps, the weighted-majority-voting (WMV) rule is applied to the corresponding pixels in the SVM map. The output of this step is a set of spectral-spatial classification maps termed as SVM-MS maps. In order to generate the final classification result, we propose to aggregate this set of SVM-MS maps using the ordered weighted averaging (OWA) operator. The determination of the associated weights is made using the idea of a stress function. The performance of the proposed classification system is assessed on three different hyperspectral datasets acquired by the Reflective Optics System Imaging Spectrometer (ROSIS-03), the Digital Imagery Collection Experiment (HYDICE) and the Airborne Visible/Infrared Imaging Spectrometer (AVIRIS) sensors.