A wireless video communication system can be designed based on the rate-distortion (R-D) criterion, i.e., minimizing the end-to-end distortion (which includes quantization distortion and transmission distortion) subject to the transmission bit-rate constraint. The minimization can be achieved by adjusting the source encoding parameters and channel encoding parameters. This rate-distortion optimization (RDO) is usually done for each video frame individually in a real-time video communication system, e.g., video calls or videoconferencing. To achieve this, an accurate bit-rate model and distortion model for each frame can be used to reduce the RDO complexity. In this paper, we derive a source bit-rate model and quantization distortion model; we also improve the performance bound for channel coding under a convolutional code and a Viterbi decoder, and derive its performance bound under a Rayleigh block fading channel. Given the instantaneous channel condition, e.g., signal-to-noise ratio and transmission bit-rate constraint, we design an R-D optimized cross-layer rate control (CLRC) algorithm by jointly choosing quantization step size in source coding and code rate in channel coding. Experimental results show that our proposed R-D models are more accurate than the existing R-D models. Experimental results also showed that the rate control under our models has more stable R-D performance than the existing rate control algorithms; using the channel estimation, CLRC can further achieve remarkable R-D performance gain over that without channel estimation. Another important result is that the subjective quality of our CLRC algorithm is much better than the existing algorithms due to its intelligent reference frame selection.