Optimum transmitter/receiver design in an overlay system

This paper examines joint optimization of the transmitter/receiver pulse-shaping filter in the context of an overlay communication system. It is assumed that co-channel interference exists between the overlay and legacy systems since both occupy the same spectral band. A composite MSE of convex form consisting of a weighted sum of the MSE of the overlay system and the excess MSE in the legacy system caused by the introduction of the overlay system is used as a figure of merit. Necessary conditions on the jointly optimum transmitter and receiver are derived. The effect of varying the weights of the metric is investigated computationally.