Separate or Integrated: Comprehensive Analysis of Combining Optical OTFS and OAM in RIS-assisted FSO Communication Systems

Paving the way toward next-generation communications, reconfigurable intelligent surface (RIS) based free-space optical (FSO) links have emerged as a strong candidate in the unlicensed spectrum for enhanced capacity and signal quality. However, existing modulation schemes constrain the FSO links’ performance and are unsuitable for RIS-induced multipath fading. To address the issue, we firstly introduce the optical orthogonal time frequency space (O-OTFS) modulation scheme into such links, with orbital angular momentum (OAM) optical beams combined to enhance the performance. Moreover, we propose two combination systems with separate and integrated OAM-OTFS respectively, and further propose a novel quad-mode (QM)-OAM-OTFS scheme within the integrated system, demonstrating significant performance enhancement over counterparts. Numerical results indicate that the separate OAM-OTFS system enhances performance in terms of power and spectral efficiencies but reduces system robustness to turbulence with increasing OAM modes, the proposed QM-OAM-OTFS scheme effectively mitigates atmospheric turbulence effects and achieves efficient power and spectral utilization.