We develop a vector theory of cross-phase modulation (XPM) capable of describing nonlinear coupling between two pulses of different wavelengths and arbitrary states of polarization. We focus for simplicity on the pump-probe configuration and use it to investigate the temporal and spectral polarization effects occurring inside an optical fiber. Using the Stokes-vector formalism we show that the probe polarization changes in general through XPM-induced nonlinear polarization rotation. In the absence of dispersion-induced probe broadening, such nonlinear changes in the probe polarization do not affect the temporal shape of the probe pulse but produce a multipeak spectrum whose different spectral peaks have different states of polarization. When dispersive effects are included, even the shape of the probe pulse becomes polarization dependent, and different parts of the pulse develop different states of polarization. Such nonlinear polarization effects lead to novel phenomena such as polarization-dependent compression and splitting of the probe pulse.