Securing Two-Way Cooperative Systems With an Untrusted Relay: A Constellation-Rotation Aided Approach

This letter studies the secure transmission protocol design for two-way cooperative systems, where two terminal users communicate via an untrusted relay. The relay acts as both a helper and a potential eavesdropper, from which the users' messages need to be kept secret. A constellation-rotation aided approach is proposed to prevent the relay from decoding these messages. By rotating the signal constellation with a proper angle, the complex-valued symbol can be fully represented by its real component. Then, either user utilizes only one dimension of the signal to deliver its information, with the other dimension carrying the artificial noise (AN). In this manner, the received signal-to-interference-plus-noise ratio (SINR) at the relay is degraded due to the injection of AN, thereby increasing the difficulty of eavesdropping. Moreover, the desired signal and the AN lie in different directions at the terminal users, which makes the signal detection free-of-interference. The approximate expression for the upper bound of the users' symbol error rate (SER) is derived based on which a criterion is developed to choose the rotation angle that can optimize the system SER performance. Simulation results demonstrate the superiority of our scheme compared to the competing candidates.