Linear Methods for Regression

This paper, developed by AiQR Academy, provides a rigorous and structured introduction to linear regression methods, which form a cornerstone of statistical modeling, machine learning, and quantitative finance.

It begins with a formal presentation of linear regression, establishing the mathematical framework linking a continuous target variable to a set of explanatory features. The paper then introduces the Least Mean Squares (LMS) algorithm and thoroughly derives its optimization via gradient descent, including batch, stochastic, and mini-batch variants.

An analytical solution is presented through the normal equations, offering a closed-form alternative to iterative optimization methods. The paper also develops a probabilistic interpretation of linear regression, showing that parameter estimation can be viewed as a maximum likelihood problem under the assumption of Gaussian errors.

A dedicated section discusses the statistical assumptions underlying linear regression—linearity, independence, homoscedasticity, normality of errors, and absence of multicollinearity—which are critical for ensuring reliable estimation and inference.

The paper then introduces regularization techniques, namely Ridge, Lasso, and Elastic Net, explaining how shrinkage helps control overfitting, improve numerical stability, and perform feature selection in high-dimensional settings. It concludes with a concise overview of essential matrix derivatives, which are fundamental tools for deriving and understanding linear and regularized models.

Overall, this chapter aims to provide strong theoretical foundations, serving as a necessary stepping stone toward more advanced and nonlinear machine learning models used in modern quantitative finance.