This paper describes a coherent optical receiver for demodulating optical quadrature phase-shift keying (QPSK) signals. At the receiver, a phase-diversity homodyne detection scheme is employed without locking the phase of the local oscillator (LO). To handle the carrier phase drift, the carrier phase is estimated with digital signal processing (DSP) on the homodyne-detected signal. Such a scheme presents the following major advantages over the conventional optical differential detection. First, its bit error rate (BER) performance is better than that of differential detection. This higher sensitivity can extend the reach of unrepeated transmission systems and reduce crosstalk between multiwavelength channels. Second, the optoelectronic conversion process is linear, so that the whole optical signal information can be postprocessed in the electrical domain. Third, this scheme is applicable to multilevel modulation formats such as M-array PSK and quadrature amplitude modulation (QAM). The performance of the receiver is evaluated through various simulations and experiments. As a result, an unrepeated transmission over 210 km with a 20-Gb/s optical QPSK signal is achieved. Moreover, in wavelength-division multiplexing (WDM) environment, coherent detection allows the filtering of a desired wavelength channel to reside entirely in the electrical domain, taking advantage of the sharp cutoff characteristics of electrical filters. The experiments show the feasibility to transmit polarization-multiplexed 40-Gb/s QPSK signals over 200 km with channel spacing of 16 GHz, leading to a spectral efficiency as high as 2.5 b/s/Hz.