Hybrid Constellation Shaping for Enhanced 3D-OFDM-DM Performance of VLC Systems

Orthogonal frequency division multiplexing with dual-mode index modulation (OFDM-DM) has been introduced in visible light communication (VLC) systems to enhance spectral efficiency. However, OFDM-DM faces challenges in achieving satisfactory bit error rate (BER) performance under low signal-to-noise ratio (SNR) conditions. To address this limitation, we propose a novel three-dimensional OFDM-DM (3D-OFDM-DM) system that integrates advanced constellation shaping techniques. Specifically, we design a new 3D constellation using geometric shaping (GS) and apply Huffman coding for probability shaping (PS) to optimize the constellation figure of merit (CFM), thereby improving its resistance to noise. The proposed scheme increases the minimum Euclidean distance (MED) among 3D constellation points, enhancing BER performance, while PS dynamically adjusts symbol probabilities to reduce average power and further strengthen system robustness. Simulation results demonstrate that the proposed 3D-OFDM-DM scheme significantly improves both BER performance and spectral efficiency compared to conventional schemes.