Kernel Shape Renormalization In Bayesian Shallow Networks: a Gaussian Process Perspective

The Bayesian approach has proven to be a valuable tool for analytical inspection of neural networks. Recent theoretical advances have led to the development of an effective statistical mechanics description of Bayesian one-hidden-layer networks within the thermodynamic proportional regime, that accurately predicts the outcomes of experiments with finite networks. In this work, we present a general perspective on these results through the lens of Gaussian Processes, underscoring an equivalence between the equilibrium description in statistical mechanics and Gaussian Process regression. Furthermore, we offer a physical interpretation of the order parameters (or kernel hyperparameters) in terms of the posterior weights distribution.