Probstat/notes/sample means and sample variances

This is part of probstat

Consider a certain distribution. The mean $\mu$ of the distribution is the expected value of a random variable $X$ sample from the distribution. I.e.,

$\mu =E[X]$.

Also recall that the variance of the distribution is

$\sigma ^{2}=Var(X)=E[(X-\mu )^{2}]=E[X^{2}]=E[X]^{2}.$.

And finally, the standard deviation is $\sigma ={\sqrt {Var(X)}}$.

Sample Statistics

Suppose that you take $n$ samples $X_{1},X_{2},\ldots ,X_{n}$ independently from this distribution. (Note that $X_{1},X_{2},\ldots ,X_{n}$ are random variables.)

Sample means

The statistic

${\bar {X}}={\frac {X_{1}+X_{2}+\cdots +X_{n}}{n}}={\frac {1}{n}}\sum _{i=1}^{n}X_{i}$

is called a sample mean. Since $X_{1},X_{2},\dots ,X_{n}$ are random variables, the mean ${\bar {X}}$ is also a random variable.

We hope that ${\bar {X}}$ approximates $\mu$ well. We can compute:

$E[{\bar {X}}]=E\left[{\frac {1}{n}}\sum _{i=1}^{n}X_{i}\right]={\frac {1}{n}}E\left[\sum _{i=1}^{n}X_{i}\right]={\frac {1}{n}}\sum _{i=1}^{n}E[X_{i}]={\frac {1}{n}}n\mu =\mu$

and

$Var({\bar {X}})={\frac {\sigma ^{2}}{n}}.$

Sample variances and sample standard deviations

We can also use the sample to estimate $\sigma ^{2}$.

The statistic

$S^{2}={\frac {\sum _{i=1}^{n}(X_{i}-{\bar {X}})^{2}}{n-1}}$

is called a sample variance. The sample standard deviation is $S={\sqrt {S^{2}}}={\sqrt {\frac {\sum _{i=1}^{n}(X_{i}-{\bar {X}})^{2}}{n-1}}}$.

Note that the denominator is $n-1$ instead of $n$.

We can show that $E[S^{2}]=\sigma ^{2}$.

${\begin{array}{rcl}\mathrm {E} [S^{2}]&=&\mathrm {E} \left[{\frac {\sum _{i=1}^{n}(X_{i}-{\bar {X}})^{2}}{n-1}}\right]\\&=&\mathrm {E} \left[{\frac {\sum _{i=1}^{n}(X_{i}^{2}-2X_{i}{\bar {X}}+{\bar {X}}^{2}}{n-1}}\right]\\&=&{\frac {1}{n-1}}\left(\sum _{i=1}^{n}E[X_{i}^{2}]-2\cdot \sum _{i=1}^{n}E[X_{i}{\bar {X}}]+\sum _{i=1}^{n}E[{\bar {X}}^{2}]\right)\\&=&{\frac {1}{n-1}}\left(\sum _{i=1}^{n}E[X_{i}^{2}]-2\cdot \sum _{i=1}^{n}E\left[X_{i}\left((1/n)\sum _{j=1}^{n}X_{j}\right)\right]+\sum _{i=1}^{n}E\left[\left((1/n)\sum _{j=1}^{n}X_{j}\right)^{2}\right]\right)\\&=&{\frac {1}{n-1}}\left(\sum _{i=1}^{n}E[X_{i}^{2}]-(2/n)\cdot \sum _{i=1}^{n}\sum _{j=1}^{n}E\left[X_{i}\cdot X_{j}\right]+n\cdot E\left[\left((1/n)\sum _{j=1}^{n}X_{j}\right)^{2}\right]\right)\\&=&{\frac {1}{n-1}}\left(\sum _{i=1}^{n}E[X_{i}^{2}]-(2/n)\cdot \sum _{i=1}^{n}\sum _{j=1}^{n}E\left[X_{i}\cdot X_{j}\right]+(1/n)\cdot E\left[\left(\sum _{j=1}^{n}X_{j}\right)^{2}\right]\right)\\\end{array}}$

We note that since $X_{i}$ and $X_{j}$ are independent, we have that

$E[X_{i}X_{j}]=E[X_{i}]E[X_{j}]=\mu \cdot \mu =\mu ^{2}$.

Let's deal with the middle term here.

${\begin{array}{rcl}\sum _{i=1}^{n}\sum _{j=1}^{n}E\left[X_{i}\cdot X_{j}\right]&=&\sum _{i=1}^{n}E[X_{i}X_{i}]+\sum _{i=1}^{n}\sum _{j\neq i}E[X_{i}X_{j}]\\&=&\sum _{i=1}^{n}E[X_{i}^{2}]+\sum _{i=1}^{n}\sum _{j\neq i}E[X_{i}]E[X_{j}]\\&=&nE[X^{2}]+n(n-1)E[X]E[X]\\&=&nE[X^{2}]+n(n-1)\mu ^{2}\\\end{array}}$

Let's work on the third term which ends up being the same as the middle term.

$E\left[\left(\sum _{j=1}^{n}X_{j}\right)^{2}\right]=E\left[\sum _{j=1}^{n}\sum _{k=1}^{n}X_{j}X_{k}\right]=\sum _{j=1}^{n}\sum _{k=1}^{n}E[X_{j}X_{k}]=nE[X^{2}]+n(n-1)\mu ^{2}.$

Let's put everything together:

\begin{array}{rcl} \mathrm{E}[S^2] &=& \frac{1}{n-1}\left( \sum_{i=1}^n E[X_i^2] - (2/n)\cdot\sum_{i=1}^n \sum_{j=1}^n E\left[X_i\cdot X_j\right] + (1/n)\cdot E\left[\left(\sum_{j=1}^n X_j\right)^2\right] \right) \\ &=& \frac{1}{n-1}\left( n E[X^2] - (2/n)(n E[X^2] + n(n-1)\mu^2) + (1/n)(n E[X^2] + n(n-1)\mu^2) \right) \\ &=& \frac{1}{n-1}\left( n E[X^2] - 2E[X^2] - 2(n-1)\mu^2 + E[X^2] + (n-1)\mu^2 \right) \\ &=& \frac{1}{n-1}\left((n-1) E[X^2] - (n-1)\mu^2 \right) \\ &=& E[X^2] - \mu^2 = \sigma^2\\ \end{array} </math>