Twice-Universal Prediction in Over-Parameterized Linear Regression

This paper addresses the computational and theoretical challenges of implementing twice-universal prediction in over-parameterized, possibly high-dimensional, linear models with additive white Gaussian noise (AWGN). Building on the theoretical foundations of the Predictive Normalized Maximum Likelihood (pNML) and twice-universal predictors, we propose an efficient local search approximation algorithm to handle the exponential complexity of combining all subsets of a high-dimensional parameter space. We validate our approach through a series of experiments: comparing the twice-universal predictor with pNML and regularized least squares, evaluating the proposed approximation algorithm in terms of accuracy and scalability, and showcasing its application in high-dimensional settings. Despite its advantages, the computational cost remains significant, highlighting the need for further refinements such as restricting reference learners to under-parameterized subfamilies. This work advances the practical applicability of twice-universal prediction and lays the groundwork for future research in scalable universal prediction frameworks¹¹This work has been supported by a grant from the Israeli Science Foundation, number 819/20.