The atomic force microscope (AFM) has been widely used as a nano-effector with a function of force sensing to detect interaction forces between an AFM tip and a sample, thereby controlling the process of the nanomanipulation. However, both the extent and accuracy of force application are significantly limited by the nonlinearity of the commonly used optical lever with a nonlinear position-sensitive detector (PSD). In order to compensate the nonlinearity of the optical lever, a nonlinear calibration method is presented. This method applies the nonlinear curve fit to a full-range position-voltage response of the photodiode, obtaining a continuous function of its voltage-related sensitivity. Thus, interaction forces can be defined as integrals of this sensitivity function between any two responses of photodiode voltage outputs, instead of rough transformation with a single conversion factor. The lateral position-voltage response of the photodiode, a universally acknowledged puzzle, was directly characterized by an accurately calibrated force sensor composed of a tippless piezoresistive microcantilever and corresponding electronics, regardless of any knowledge of the cantilevers and laser measuring system. Experiments using a rectangular cantilever (normal spring constant 0.24 N/m) demonstrated that the proposed nonlinear calibration method restrained the sensitivity error of normal position-voltage responses to 3.6% and extended the force application range.