Near-optimal parameter selection methods for l2 regularization

This paper focuses on the problem of selecting the regularization parameter for linear least-squares estimation. Usually, the problem is formulated as a minimization problem with a cost function consisting of the square sum of the l
₂
norm of the residual error, plus a penalty term of the squared norm of the solution multiplied by a constant. The penalty term has the effect of shrinking the solution towards the origin with magnitude that depends on the value of the penalty constant. By considering both squared and non-squared norms of the residual error and the solution, four different cost functions can be formed to achieve the same goal. In this paper, we show that all the four cost functions lead to the same closed-form solution involving a regularization parameter, which is related to the penalty constant through a different constraint equation for each cost function. We show that for three of the cost functions, a specific procedure can be applied to combine the constraint equation with the mean squared error (MSE) criterion to develop approximately optimal regularization parameter selection algorithms. Performance of the developed algorithms is compared to existing methods to show that the proposed algorithms stay closest to the optimal MSE.