We numerically investigate the gain and phase recovery dynamics of quantum-dot (QD) semiconductor optical amplifiers (SOAs) by simultaneous consideration of the temporal and spectral variations. We solve 1088 coupled rate equations to simulate the carrier recovery dynamics of the inhomogeneously broadened QD ensemble, where QD electron and hole states are considered separately. The gain and phase recovery responses induced by an ultrafast pump pulse are calculated by considering different carrier relaxation time constants and spectral line shape functions involved in the successive carrier recovery process. The interband transition is assumed to have a Lorentzian line shape function and the intraband free-carrier absorption is described by the Drude model. We demonstrate that the actual behavior of the gain and phase recovery in QD SOAs can be clearly understood by simultaneously considering the temporal and spectral behavior of the gain and phase recovery responses, which are visualized by means of plots in the time and wavelength domains. We identify how the respective QD state-the ground state, the excited state, and an upper state-and quantum-well carrier reservoirs contribute to the gain and phase recovery of QD SOAs both temporally and spectrally.