Estimation of the Root Mean Square of the Residuals of Sine Fitting in the Presence of Phase Noise or Jitter

Fitting a sinusoidal model to sampled data is a fundamental operation in many engineering disciplines. Typical parameters estimated include amplitude, initial phase, and offset (three-parameter sine fitting), or additionally frequency (four-parameter sine fitting). The uncertainty in these estimates depends on the number of samples and the influence of non-ideal effects such as additive noise, oscillator phase noise, sampling jitter, and frequency errors. This work focuses on proposing an heuristically derived analytical expression for the standard deviation of the root mean square (RMS) of the residuals from three-parameter sine fitting in the presence of phase noise or sampling jitter. The expression is a function of the phase noise or jitter standard deviation, the number of samples, and the signal amplitude. This result aids engineers in selecting appropriate sample sizes and in reporting RMS values with confidence intervals. The formulation also applies directly to RMS estimation using the Discrete Fourier Transform. The analytical result is validated using Monte Carlo simulations.