Probstat/notes/regression

This is part of probstat.

In this section, we shall discuss linear regression. We shall focus on one-variable linear regression.

Model

We consider two variables $X$ and $Y$ where $Y$ is a function of $X$. We refer to $X$ as independent or input variable, and $Y$ as a dependent variable. We consider linear relationship between independent variable and dependent variable. We assume that there exist hidden variables $\alpha$ and $\beta$ such that

$Y=\alpha +\beta \cdot X+e,$

where $e$ is a random error. We further assume that the error is unbiased, i.e., $E[e]=0$ and is independent of $X$.

Input: As an input to the regression process, we are given a set of $n$ data points: $(x_{1},y_{1}),(x_{2},y_{2}),\ldots ,(x_{n},y_{n})$ generated from the previous equation.

Goal: We want to estimate $\alpha$ and $\beta$.

The least squares estimators

Denote our estimate for $\alpha$ as $A$ and for $\beta$ as $B$. Using both variables as estimator, the error at data point $(x_{i},y_{i})$, the error is

$y_{i}-(A+Bx_{i})=y_{i}-A-Bx_{i}$.

We focus more on the sum of squared errors, i.e.,

$SS=\sum _{i=1}^{n}(y_{i}-A-Bx_{i})^{2}$.

The method of least squares use the parameters that minimize the squared errors as an estimator. Therefore, we want to find $A$ and $B$ that minimize $SS$. To do so, we partially differentiate $SS$ with respect to $A$ and $B$:

${\frac {\partial }{\partial A}}SS=-2\sum _{i=1}^{n}(y_{i}-A-Bx_{i})$

${\frac {\partial }{\partial B}}SS=-2\sum _{i=1}^{n}x_{i}(y_{i}-A-Bx_{i})$

We set these two equations to zero to find the maximum and obtain these two equations we have to solve.

$\sum _{i=1}^{n}y_{i}=nA+B\sum _{i=1}^{n}x_{i}$

$\sum _{i=1}^{n}x_{i}y_{i}=A\sum _{i=1}^{n}x_{i}+B\sum _{i=1}^{n}x_{i}^{2}$

Before solving these two equations, let's define

${\bar {y}}=\sum _{i=1}^{n}y_{i}/n,\ \ \ \ {\bar {x}}=\sum _{i=1}^{n}x_{i}/n.$