Joint Broadcast and Unicast Transmission Based on RSMA and Spectrum Sharing for Integrated Satellite–Terrestrial Network

The integrated satellite-terrestrial network (ISTN) is gaining attention for seamless communication services, which can provide diverse services to terminals, i.e., broadcast and unicast services. However, it is challenging to address massive terminal access and meet diverse information services under limited spectrum resources and strong multiple access interference in ISTN. Dynamic spectrum sharing and rate-splitting multiple access (RSMA) have emerged as promising technologies, where RSMA offers non-orthogonal transmission and robust interference management. Motivated by this, we establish a downlink non-orthogonal broadcast and unicast (NOBU) model using the 1-layer rate-splitting strategy for ISTN, which encodes broadcast data and unicast data into common and private streams. Then, we propose four NOBU transmission schemes based on different spectrum sharing modes to maximize the max-min rate (MMR), where schemes based on hybrid spectrum sharing consider the unevenly distributed and time-varying spectrum resources and the number of terminals. Furthermore, we formulate joint MMR optimization problems while satisfying the broadcast information rate requirement in ISTN. To tackle these non-convex problems, we introduce an improved alternating optimization algorithm based on weighted minimum mean square error. Simulation results verify that the RSMA-based NOBU schemes have significant performance gains compared with various baseline schemes.