CheckSelect: Online Checkpoint Selection for Flexible, Accurate, Robust, and Efficient Data Valuation

In this paper, we argue that data valuation techniques should be flexible, accurate, robust, and efficient (FARE). Here, accuracy and efficiency refer to the notion of identification of most important data points in less time compared to a full training. Flexibility refers to the ability of the method to be used with various value functions, while robustness refers to the ability to be used with different data distributions from a related domain. We propose a two-phase approach towards achieving these objectives, where the first phase, checkpoint selection, extracts important model checkpoints while training on a related dataset, and the second data valuation and subset selection phase extracts the high-value subsets. A key challenge in this process is to efficiently determine the most important checkpoints during the training, since the total value function is unknown. We pose this as an online sparse approximation problem and propose a novel online orthogonal matching pursuit algorithm for solving it. Extensive experiments on standard datasets show that CheckSelect provides the best accuracy among the baselines while maintaining efficiency comparable to state of the art. We also demonstrate the flexibility and robustness of CheckSelect on a standard domain adaptation task, where it outperforms existing methods in data selection accuracy without the need to re-train on the full target-domain dataset.