In-phase/quadrature (I/Q) imbalance is one of the main sources of distortion in RF modulators. In this paper, a dual-input nonlinear model based on a real-valued Volterra series is proposed for compensation of the nonlinear frequency-dependent I/Q imbalance. First, different sources of distortion are identified from experimental measurements, then a dual-input nonlinear I/Q imbalance model is developed. Further, the inverse model is used for I/Q imbalance compensation. Finally, the performance of the I/Q imbalance compensator is evaluated with both simulations and experiments. In comparison with previously published results, the proposed I/Q imbalance compensator shows significantly improved performance. Thus, we prove that a complete nonlinear I/Q imbalance compensation can minimize the effects of the RF modulator in high-performance digital communication systems.