Maximum-likelihood sequence estimation of four dimensional signaling schemes subject to ISI

A maximum-likelihood sequence estimation (MLSE) algorithm is derived for decision making in a receiver used for detection of four-dimensional signaling schemes. This algorithm operates directly on the output of two complex matched filters and can be considered as an extension of the modified version of the Viterbi algorithm. As a measure of performance, a bound on the probability of error is derived, and the effect of intersymbol interferences (ISIs) present at the matched filters' output on the error performance of the receiver is discussed and illustrated