Backhaul-Constrained Joint Beamforming for Non-Orthogonal Multicast and Unicast Transmission

The demand for providing multicast services in emerging cellular networks is increasing. This paper proposes a non-orthogonal transmission framework based on layered-division multiplexing (LDM) to incorporate multicast and unicast services into cellular networks with limited- capacity backhaul. We adopt a two-layer LDM structure where the first layer is intended for multicast services, the second layer is for unicast services, and the two layers are superposed with different beamformers and decoded by each user receiver using successive interference cancellation. To optimize the non- orthogonal transmission, we formulate a joint multicast and unicast beamforming design problem with adaptive base station (BS) clustering that aims to maximize the weighted sum of multicast rate and unicast rate under peak power constraints and peak backhaul constraints on each BS. The problem is a sparse optimization problem and NP- hard. By means of smoothed l0-norm approximation and novel algebraic operations, we transform it into a difference of convex (DC) programming, which is solved using convex-concave procedure (CCP) with guaranteed convergence. Simulation results demonstrate the superiority of the proposed algorithm, especially when the backhaul constraint is not too stringent. Results also show that the proposed LDM-based non-orthogonal scheme can achieve a much larger multicast-unicast rate region than orthogonal schemes.