Should Penalized Least Squares Regression be Interpreted as Maximum A Posteriori Estimation?

Penalized least squares regression is often used for signal denoising and inverse problems, and is commonly interpreted in a Bayesian framework as a Maximum a posteriori (MAP) estimator, the penalty function being the negative logarithm of the prior. For example, the widely used quadratic program (with an
l
¹
penalty) associated to the LASSO/basis pursuit denoising is very often considered as MAP estimation under a Laplacian prior in the context of additive white Gaussian noise (AWGN) reduction. This paper highlights the fact that, while this is one possible Bayesian interpretation, there can be other equally acceptable Bayesian interpretations. Therefore, solving a penalized least squares regression problem with penalty φ(
x
) need not be interpreted as assuming a prior
C
·exp(-φ(
x
)) and using the MAP estimator. In particular, it is shown that for any prior
PX
, the minimum mean-square error (MMSE) estimator is the solution of a penalized least square problem with some penalty φ(
x
) , which can be interpreted as the MAP estimator with the prior
C
·exp(-φ(
x
)). Vice versa, for certain penalties φ(
x
), the solution of the penalized least squares problem is indeed the MMSE estimator, with a certain prior
PX
. In general
dPX
(
x
) ≠
C
·exp(-φ(
x
))
dx
.