The performance of a wavelength-division multiplexing (WDM) optical network can be severely degraded due to fiber nonlinear effects. In the case where nonzero dispersion (NZD) fibers are employed, the four-wave mixing (FWM) effect sets an upper limit on the input power, especially in the case of narrow channel spacing. In order to reduce FWM-induced distortion two new techniques, the hybrid amplitude-/frequency-shift keying (ASK/FSK) modulation and the use of prechirped pulses are investigated. It is shown that both techniques can greatly improve the Q-factor in a 10 Gb/s WDM system. This happens even for very high input powers (∼10 dBm), where the degradation of the conventional WDM system is prohibitively high. The proposed methods are also applied and tested in higher bit rates (40 Gb/s). It is deduced that although the hybrid ASK/FSK modulation technique marginally improves the system performance, the optical prechirp technique can still be used to greatly increase the maximum allowable input power of the system.