Deep Ridge Regression with Anchor Graph

The complexity of tuning and intensive computation required by deep models often leads to overfitting when applied to small data sets. When dealing with small data sets, traditional machine learning models are still a valid choice. The ridge regression, as a biased estimation method for analyzing collinear data, can be considered as the fundamental basis of all machine learning algorithms. This paper presents a novel deep ridge regression model, which is designed by removing the activation function and threshold of each layer in the neural network structure. The model can undergo layer-by-layer feature extraction from the inside out to capture intricate patterns and deep structures in the data. A multi-layer architecture enhances the expressiveness of the model, enabling it to learn and express more complex patterns by stacking multiple layers. Furthermore, an anchor graph regularization term is introduced to enhance the capturing of consensus features and inherent structural information among the samples. In order to achieve the purpose of feature extraction layer by layer, an adaptive gradient descent optimization strategy is designed to incorporate iterative weights (projection matrix) from inside to outside. Furthermore, the theoretical analysis ensures its convergence. Numerical experiments on three public datasets show that the anchor graph regularized deep ridge regression outperforms the traditional ridge regression and even surpasses the deep learning method on certain datasets under multiple evaluation criteria.