In recent studies it has been demonstrated that the scanning tunneling microscope (STM) after sophisticated modification is a powerful technique to investigate surface acoustic waves (SAWs) with high spatial resolution. Here we apply our ultrahigh vacuum (UHV) SAW–STM to investigate the surface motion induced by a Rayleigh wave on LiNbO3. After establishing the theoretical basis for evaluating SAW–STM images recorded as usual in the constant–current scanning mode we apply the method to quantitatively determine the complete geometry of the atomic oscillation ellipse given by its eccentricity and the amplitude of the transverse displacement. The obtained eccentricity parameter is 36° being in good agreement with the theoretical value of 35.1°. Our investigations show that the dynamic range of our UHV–SAW–STM comprises at least 3 orders of magnitude down to SAW amplitudes as small as 0.001 Å and, furthermore, that the spatial resolution for acoustic measurements lies in the lower nanometer range. © 2002 American Institute of Physics.