We studied the atomic-scale growth control and second-harmonic generation (SHG) properties of GdxY1-xCa4O(BO3)3 thin films. Combinatorial laser molecular-beam epitaxy was employed to optimize the growth conditions of the film. Ultrasmooth GdxY1-xCa4O(BO3)3 thin films with step-and-terrace structure were obtained by using atomically controlled GdCa4O(BO3)3 substrates. The dependence of SHG intensity on the fundamental wave polarization angle was studied for the epitaxial film on (010) GdCa4O(BO3)3 substrate in comparison for the randomly oriented film on (0001) sapphire substrate. It was found that only the epitaxially grown film clearly exhibited a fourfold SHG intensity showing type-II phase-matching direction. © 2001 American Institute of Physics.