Error performance of innovations-based MLSE for Rayleigh fading channels

Error performance of maximum likelihood sequence estimation (MLSE) of digital signals transmitted over Rayleigh fading channels is studied in this paper. The application of the innovations approach provides us not only with a general MLSE receiver structure, but also with a tool for analyzing the performance of the receiver. We show that the sequence pairwise error probability of the MLSE receiver is determined by the eigenvalues of a matrix generated from the autocorrelation function of the received signal. For any practical applications, the MLSE for Rayleigh fading channels exhibits an irreducible error floor that depends on the channel characteristics such as the Doppler frequency bandwidth and frequency selectivity. An upper bound on bit error probability can be calculated by using the sequence pairwise error probability. Also, it is shown that diversity reception can significantly improve the MLSE error performance.