The potential for terahertz (THz) spectroscopy to detect explosives and other materials of interest is complicated by rough surface scattering. Our previous work has demonstrated that by averaging over diffuse observation angles and surfaces, spectral features could be recovered from laboratory measurements and numerical computer simulations. In addition to averaging, a low-pass cepstrum filter was used to reduce noise due to the random rough surface. This paper expands on these concepts by using the cepstrum of both the random rough surface and the material properties of the target material to choose an optimal cutoff frequency for the filter. The utility of these techniques is evaluated using laboratory measurements and Monte Carlo simulations for many sets of random surface realizations. The Kirchhoff Approximation is used to quickly model diffuse scattering from dielectric materials with gradually undulating rough surfaces when the incident and diffuse scattering angles are near the surface normal. The ability to recover the spectral features of rough dielectric materials from diffuse THz scattering may prove useful for the design of future security screening systems.