Optimal Channel Shortener Design for Reduced- State Soft-Output Viterbi Equalizer in Single-Carrier Systems

We consider optimal channel shortener design for a reduced-state soft-output Viterbi equalizer (RS-SOVE) in single-carrier systems. To use RS-SOVE, three receiver filters need to be designed: a prefilter, a target response, and a feedback filter. The collection of these three filters are commonly referred to as the “channel shortener.” Conventionally, the channel shortener is designed to transform an intersymbol interference (ISI) channel into an equivalent minimum-phase equivalent form. In this paper, we design the channel shortener to maximize a mutual information lower bound based on a mismatched detection model. By taking the decision-feedback quality in the RS-SOVE into consideration, the prefilter and feedback filter are found in closed forms, while the target response is optimized via a gradient-ascending approach with the gradient explicitly derived. The information theoretical properties of the proposed channel shortener are analyzed. Moreover, we show through numerical results that the proposed channel shortener design achieves superior detection performance compared with previous channel shortener designs at medium and high code rates.