Reduced-complexity sequence detection approaches for PR-shaped, coded linear modulations

This paper investigates the combination of partial-response (PR) shaping and trellis coded modulation (TCM), for high-bit-rate wireline data transmission. Combining PR and TCM, we attempt to reduce the normalized bandwidth of the transmitted signal, without imposing a penalty on its minimum distance. The optimum detection method for such signals is the maximum likelihood sequence estimator (MLSE), using a Viterbi algorithm which operates on the combined PR/TCM trellis. Since the large number of states makes the MLSE computationally prohibitive, we investigate the performance of suboptimum reduced-complexity detectors, namely the M-, T- and RSSE/DFSE algorithms, and show that we can obtain near-MLSE performance at a much lower computational cost. Considerations about the PR pulse shape and the trade-off between bandwidth and performance are also given.