In this paper, we present a detection algorithm for MIMO system with nonbinary modulation scheme. The proposed method is based on rectangular searching. It restricts the searching area to a certain rectangle when searching the candidate symbols for each symbol. It has considerably reduced computational complexity compared to ML (maximum likelihood) algorithm because only a small part of candidate constellation points are located in the searching rectangle. However, to our surprise, Simulation results show that the BER (bit error rate) performance of the proposed method outperforms the ML (maximum likelihood) algorithm when the SNR is between -10 dB and 15 dB. The reasons responsible for this unusual phenomena are also given.