We theoretically and experimentally investigate the performance of two self-heterodyne detected radio-over-fiber (RoF) links employing phase modulation (PM) and quadrature biased intensity modulation (IM), in terms of bit-error rate (BER) and optical signal-to-noise ratio (OSNR). In both links, self-heterodyne receivers perform down-conversion of the RF subcarrier signal. A theoretical model including noise analysis is constructed to calculate the Q factor and estimate the BER performance. Furthermore, we experimentally validate our prediction from the theoretical modeling. Both the experimental and theoretical results show that the PM link offers superior OSNR receiver sensitivity performance (more than 6 dB) over the quadrature biased IM counterpart.