This paper presents a convenient self-calibration method for an inertial measurement unit (IMU) using matrix factorization. Using limited information about applied loads (accelerations or angular rates) available from natural references, the proposed method can linearly solve all the parameters of an IMU in any configuration of its inertial components. Our factorization-based calibration method exploits the bilinear form of an IMU measurement, which is the product of intrinsic calibration parameters and exerted loads. For a redundant IMU, we prove that partial knowledge of the loads, such as magnitude, can produce a linear solution space for a proper decomposition of the measurement. Theoretical analysis on this linear space reveals that a 1-D null space should be considered when load magnitudes are all equal (e.g., gravity loads). Degenerate load distributions are also geometrically identified to avoid singular measurement collection. Since a triad IMU has a lower number of sensor components than a 4-D parameter space, we propose an iterative factorization in which only initial bias is required. A wide convergence region of the bias can provide an automatic setting of the initial bias as the mean of the measurements. Performance of the proposed method is evaluated with respect to various noise levels and constraint types. Self-calibration capability is demonstrated using natural references, which are gravity for accelerometers and image stream from an attached camera for gyroscopes. Calibration results are globally optimal and identical to those of nonlinear optimization.