In this paper, we present the generation, the detection and the performance evaluation of a novel return-to-zero frequency shift keying (RZ-FSK) format for 40 Gb/s transmission. Non-return-to-zero (NRZ) FSK signal is generated by using two continues-wave (CW) lasers, one Mach-Zehnder modulator (MZM) and one Mach-Zehnder delay interferometer (MZDI). A RZ-FSK signal is successfully generated by cascading a dual-arm MZM, which is driven by a sinusoidal voltage at half of the bit rate. The demodulation can be simply achieved on 1 bit rate through one MZDI or an array waveguide grating (AWG) demultiplexer with balanced detection. By numerical simulation, two types of frequency modulation schemes using MZM or PM, and impact of the frequency tone spacing (FTS) of the generated FSK signal are discussed. The proposed scheme shows that the novel frequency modulation format offers a few transmission advantages comparing with than that of the other traditional modulation formats, such as RZ and differential phase-shift keying (DPSK), under varying dispersion management. The performance analysis of RZ-FSK signal in a 4 × 40 Gb/s WDM transmission system is introduced. We experimentally demonstrate, transparent wavelength conversion based on four-wave mixing (FWM) in a semiconductor optical amplifier (SOA) and in a highly nonlinear dispersion shifted fiber (HNDSF) for a 40 Gb/s RZ-FSK signal, clearly validating the feasibility of all optical signal processing of high-speed RZ-FSK signal. Moreover, we investigate the receiver power penalty for the RZ-FSK signal after a 100 km standard single-mode fiber (SMF) transmission link with matching dispersion compensating fiber (DCF), under the post-compensation management scheme. Since the frequency modulation format is orthogonal to intensity modulation and vector modulation (polarization shift keying), it can be employed in the context of the combined modulation format to decrease the data rate or enhance the symbol rate. It can - - also be utilized in the orthogonal label-switching as the modulation format for the payload or the label.