Analysis of maximum-likelihood sequence estimation performance for quadrature amplitude modulation

This paper considers maximum-likelihood sequence estimation (MLSE) for quadrature amplitude modulation (QAM) signaling at rates approaching several baud/Hz. In this regime, intersymbol interference and possibly cross channel coupling are the dominant transmission impairments. We derive the structure of a detector that optimally accommodates both impairments. A bit error rate performance bound is found, and the concept of an error state transition matrix is introduced to facilitate the analysis. We explore a modulation scheme wherein cross-channel coupling is intentionally introduced, and find that it improves detection efficiency. The use of MLSE may be an important consideration for power and spectrally efficient digital radio systems, either terrestrial or satellite, since rates approaching the Shannon limit may be attainable without channel coding, and frequency selective fading is handled in an optimum manner.