Generalized Equalization-Enhanced Phase Noise in Coherent Optical Transceivers Using Arbitrary Raised Cosine Filters

Equalization-enhanced phase noise (EEPN) can degrade the performance of high baud-rate and long-distance coherent optical transmission systems that use electronic dispersion compensation (EDC). Existing theoretical calculations of EEPN are based on the ideal Nyquist filter, which has an unrealizable brickwall response due to its infinite time-domain extent. In this paper, we generalize the EEPN calculation for coherent optical transceivers using arbitrary raised cosine (RC) filters. We demonstrate that when the roll-off factor changes from zero to non-zero, the unitary transmission channel assumption no longer holds in the presence of EEPN. We calculate the resulting energy loss for transceivers using RC filters with arbitrary roll-off factors. The results indicate that while the intra-symbol EEPN remains almost unchanged, the inter-symbol EEPN and the total EEPN decrease with increasing bandwidth. Furthermore, we extend our analysis to high-order modulation formats, where the EEPN distribution varies depending on the moduli of the constellation clusters.