Optimum mean-square decision feedback equalization

In this work we report new results relating to decision feedback equalization. The equalizer and the transmitting filter are optimized in a PAM data communication system operating over a linear noisy channel. We use a mean-square error criterion and impose an average power constraint at the transmitter. Assuming correct past decisions, an explicit formula for the minimum attainable mean-square error is given. The possible advantages of signaling faster than the Nyquist rate while decreasing the number of levels to maintain the same information rate are investigated. It is shown that, in all cases of practical interest, signaling faster than the Nyquist rate, while keeping fixed the information rate, increases the mean-square error. Finally, to illustrate the use of the results, application is made to a cable channel where the loss in dB varies as the square root of frequency. Various asymptotic formulas and curves are provided to exhibit the relationships between the quantities of interest.