Performance evaluation for widely linear demodulation of PAM/QAM signals in the presence of rayleigh fading and co-channel interference

In this paper, widely linear filtering (WLF) techniques are applied to the demodulation of both pulse amplitude modulation (PAM) and quadrature amplitude modulation (QAM) systems with multiple receiver antennas and multiple cochannel interferers. In the proposed implementation, the existing correlations between the real (I) and imaginary (Q) parts of the complex-valued baseband received signal are exploited using a widely linear (WL) maximum likelihood (ML) receiver. Several bounds and approximations are developed to analyze the average symbol error rate (SER) performance, and to analyze the tradeoff between the diversity advantage and interference cancellation (IC) gain in a flat Rayleigh fading channel. Two new results are shown. First, we show that the IC ability of WL receivers is independent of the modulation type used by the desired signal but the gain depends mainly on the modulation type employed by the individual interferers and the total number of antennas. Secondly, assuming that the system uses a mixture of PAM and QAM signals, we show that a WL receiver with N antennas can reject any combination of M
₁
PAM, M
₂
QAM interferers satisfying the constraint: M
₁
+ 2M
₂
< 2N with an asymptotic diversity order N - M
₁
/2 - M
₂
. In contrast, a conventional receiver whose performance is independent of the modulation characteristics of the individual interferers can reject only up to M
₁
+ M
₂
interferers, where M
₁
+ M
₂
< N with an asymptotic diversity order N - M
₁
- M
₂
. When the system contains either PAM, or a mixture of PAM and QAM type CCI, it is shown that WL processing offers a significant performance advantage with a moderate increase in the receiver complexity.