Design of optimal soft decoding for combined trellis coded quantization/modulation in Rayleigh fading channel

The well-known Bahl-Cocke-Jelinek-Raviv (BCJR) algorithm is used as a minimum mean-squared error (MMSE) decoder for combined trellis-coded quantization and modulation. When the BCJR algorithm is used as decoder, the optimal trellis-coded encoder is derived. Both trellis-coded quantizer and the soft-decoder can be optimized iteratively for minimum overall distortion from transmitter input to receiver output. The optimal encoder/decoder is simulated for Rayleigh fading channel. Significant performance improvement is achieved for both memoryless Gaussian and uniform source, especially for very noisy channel.