Introduction to Regression Methods

Summary

Regression is one of the fundamental pillars of supervised machine learning (ML). This chapter uncovers the essential concepts in regression analysis and methods, by providing practical examples, designed for graduate students and researchers seeking to gain a solid understanding. It delves into linear regression methods, exploring both closed‐form solutions and the general optimization framework provided by the gradient descent. The chapter introduces the necessary pre‐ and post‐processing steps for building and testing models: feature scaling, hold‐out, and cross‐validation. It focuses on nonlinear regression methods, in particular, polynomial regression. Through extensive examples, the chapter demonstrates the benefits associated with regularization techniques, such as Least Absolute Shrinkage and SelectionOperator Regression (LASSO), Ridge and Elastic Net, and the possibility of constructing interpretable and parsimonious models. It introduces Bayesian approaches with a focus on linear and Gaussian process regression.