On Calibration of Pretrained Code Models

Pretrained code models have achieved notable success in the field of Software Engineering (SE). However, existing studies have pre-dominantly focused on improving model performance, with limited attention given to other critical aspects such as model calibration. Model calibration, which refers to the accurate estimation of predictive uncertainty, is a vital consideration in practical applications. Therefore, in order to advance the understanding of model cali-bration in SE, we conduct a comprehensive investigation into the calibration of pretrained code models in this paper. Our inves-tigation focuses on five pretrained code models and four code understanding tasks, including analyses of calibration in both in-distribution and out-of-distribution settings. Several key insights are uncovered: (1) pretrained code models may suffer from the issue of over-confidence; (2) temperature scaling and label smoothing are effective in calibrating code models in in-distribution data; (3) the issue of over-confidence in pretrained code models worsens in different out-of-distribution settings, and the effectiveness of temperature scaling and label smoothing diminishes. All materi-als used in our experiments are available at https://github.com/queserasera22/Calibration-of-Pretrained-Code-Models.