Studies have shown that expressions for the error performance analysis of M-ary QAM, in the simultaneous presence of AWGN and cochannel interference (CCI) are in the form of Bessel integrals. These integrals are usually difficult to solve, especially in modulation scheme cases with large alphabet sizes (M>or=64). In this paper the Fourier-Bessel series expansion method is used to solve these integrals numerically and to facilitate the system bit-error-rate (BER) performance evaluation. To illustrate this approach a detailed theoretical study is carried out for the error performance of coherent 64, 256 and 1024 QAM systems, operating in a realistic environment of AWGN and CCI. The reported results are in good agreement with those of other studies and demonstrate the value of the proposed method.