⬤

Regression

Tags :: Statistics

Simple regression - Ordinary Least Squares

In the case where we only have a single independent variable, a simple regression can be expressed as \[ \hat{y}_i=\beta_0+\beta_1x_i + e_i \] where

  • \(y_i\) denotes the observed response @ i
  • \(x_i\) denotes the predictor value @ i
  • \(y^i\) is the predicted response (or fitted value) @ i
  • \(\beta_0\) is the intercept (regression constant)
  • \(\beta_1\) is the slope (regression coefficent)
  • \(e_i\) is the error we make from predicting \(y\) from \(x\)

and the goal is to find combination of \(\beta_0\) and \(\beta_1\) which minimizes the sum of squared error across all cases.

The error for a given data point \(i\) is calculated as \[ e_i = y_i - \hat{y}_i \] and the sum of squared errors for a single line is \[ Q = \text{SSE} = \sum_{i=1}^n e_i^2 \] With this formulation, finding the combination of beta values which minimizes the SSE would require computing \(Q\) on an infinite number combinations of \(\beta_0\) and \(\beta_1\). Instead we reformuate the \(Q\) as \[ Q = \text{SSE} = \sum_{i=1}^n (y_i - (\beta_0 + \beta_1 x_i))^2 \] and use the derived formulas for finding the intercept ($β_0) \[ \beta_0 = \bar{y} - \beta_1 \bar{x} \] and slope (\(\beta_1\)) \[ \beta_1 = \frac{\sum_{i=1}^{n}(x_i-\bar{x})(y_i-\bar{y})}{\sum_{i=1}^{n}(x_i-\bar{x})^2} \] of the line which minimizes \(Q\).

Matrix formulation

Given the basic regression equation \[ Y = X\beta + e \]

We can solve for the unknown \(\beta\) with \[ \beta=\begin{bmatrix} \beta_0 \\ \beta_1 \\ \vdots \\ \beta_{k} \end{bmatrix}= (X^{’}X)^{-1}X^{’}Y \] where \((X^{’}X)^{-1}\) is the inverse of the \(X^{’}X\) matrix, and \(X^{’}\) is the transpose of the \(X\) matrix.

In the case of the simple linear regression with only one predictor \(X^{’}X\) can be calculated as \[ X^{’}X=\begin{bmatrix} 1 & 1 & \cdots & 1\\ x_1 & x_2 & \cdots & x_n \end{bmatrix}\begin{bmatrix} 1 & x_1\\ 1 & x_2\\ \vdots & x_n\\ 1& \end{bmatrix}=\begin{bmatrix} n & \sum_{i=1}^{n}x_i \\ \sum_{i=1}^{n}x_i & \sum_{i=1}^{n}x_{i}^{2} \end{bmatrix} \]

References

For the full derivation of the OLS estimator please see

Links to this note