Nonlinearity is a prominent limitation to the calibration performance of vector magnetometers. A new calibration model is proposed to suppress the nonlinearity of magnetometers and improve calibration performance, in which the nonlinearity coefficients of scale factors are considered. The experimental system mainly consists of a three-axis fluxgate magnetometer (MAG3300), a 2D nonmagnetic rotation equipment, and a proton magnetometer (CZM-3), in which the nonmagnetic rotation equipment is used to change the position of three-axis fluxgate magnetometer, and the scalar value of magnetic field is obtained with the proton magnetometer and considered to be the true value. The principle of this new calibration method is analyzed, and the calibration procedures are introduced. Experimental results show that after nonlinearity suppression, the root-mean-square error of calibration can be reduced by three times. It suggests an effective way to improve the calibration performance of three-axis magnetometers.