On the Average of the Product of Two Gaussian Q-Functions over Nakagami-q (Hoyt) Fading and Its Application

In this letter, the average of the product of two Gaussian Q-functions over the Nakagami-q (Hoyt) distribution with arbitrary parameters is presented in terms of the Lauricella hypergeometric function, which can be evaluated numerically using its integral or converging series representation. The result obtained is then applied to evaluate the symbol error probability (SEP) of general rectangular quadrature amplitude modulation (QAM) constellations over Nakagami-q fading channels. The impact of fading severity (parameter q) on the error performance of the general rectangular QAM system is also investigated.