We predict the scattering loss in rectangular high index-contrast waveguides, using a new variation of the classical approach of coupled-mode theory. The loss predicted by this three-dimensional (3-D) model is considerably larger than that calculated using previous treatments that approximate the true 3-D radiation modes with their two-dimensional counterparts. The 3-D radiation modes of the ideal waveguide are expanded in a series of cylindrical harmonics, and the coupling between the guided and radiation modes due to the sidewall perturbation is computed. The waveguide attenuation can then be calculated semianalytically. It is found that the dominant loss mechanism is radiation rather than reflection, and that the transverse electric polarization exhibits much larger attenuation than transverse magnetic polarization. The method also gives simple rules that can be used in the design of low-loss optical waveguides. The structural properties of sidewall roughness of an InGaAs/InP pedestal waveguide are measured using atomic force microscopy, and the measured attenuation is found to compare well with that predicted by the model.