We use a nonlinear dynamics approach to study the deterministic behavior of ultrapure microwave generators referred to as optoelectronic oscillators. In conventional studies, the standard nonlinear effects are very strongly rejected because they generate harmonics of the microwave frequency that are definitely out of the selective oscillator bandwidth. However, we show that the nonlinearity still affects the slowly varying dynamics of microwave envelope, thereby inducing dynamical instabilities within the oscillator bandwidth. Starting from a full integro-differential model, we use the multiple timescales method to build a delay-differential equation for the slowly varying complex envelope of the microwave. Then, the corresponding stationary solutions are derived, and the stability of their amplitude and phase is investigated in detail as a function of the feedback gain. We evidence essential bifurcation phenomena, and, in particular, we demonstrate that the generated microwave may turn unstable if the gain is increased beyond a precise critical value. This nonlinear dynamics approach, therefore, demonstrates that the amplitude of the ultrapure microwave's amplitude does not monotonously increase with the gain. The theoretical study is confirmed by numerical simulations and experimental measurements.