Error Analysis and Optimization of $M$-ary Frequency-Shift-Keying Modulation for Coherent PLC Systems in Nakagami-$m$ Environment

A non-multipath and a receive diversity power line communication (PLC) system subject to Rayleigh fading is considered in this paper with the transmitter employing non-orthogonal
$M$
-ary frequency-shift keying (
$M$
FSK) modulation scheme for data transmission. With the PLC systems corrupted by additive complex Nakagami-
$m$
background noise, an optimal receiver for non-multipath PLC system and a Gauss-optimal receiver structure for the multipath PLC system are proposed using which, closed-form expressions for the symbol error probability (SEP) are derived using a characteristic function approach. The optimal correlation coefficients between the symbols of a binary FSK modulation scheme minimizing the SEP of the systems are obtained, which is further generalized by proposing a search-based algorithm to derive the optimal symbol correlation matrix for an
$M$
FSK modulation leading to the minimal SEP values of the PLC systems. Numerical results show that the index value of the optimal correlation matrix depends on the system parameters namely the noise shape parameter and the average signal-to-noise ratio only for the case of the non-multipath PLC system, whereas there is no such dependency in the case of the multipath PLC system.