Probstat/notes/parameter estimation

This is part of probstat.

Previously we tried to estimate the population means and variances using the sample means and variances. In this section, we shall see the justification why what we did makes sense.

There are many ways to estimate parameters.

Method of moments estimators

See also wikipedia article.

This is probably the simplest estimators. However, they are often biased (as we shall show in the example).

Definition: For a random variable $X$, $E[X^{k}]$ is called the k-th moment of $X$. Note that the first moment is the mean $E[X]$. The variance of a random variable depends on the first and the second moments.

If we want to estimate a parameter $\theta$, using the method of moments, we start by writing the parameter as a function of the moments, i.e.,

$\theta =g(E[X],E[X^{2}],E[X^{3}],\ldots ,E[X^{r}])$

We then estimate the sample moments

$M_{k}={\frac {\sum _{i=1}^{n}X^{k}}{n}},$

for $k=1,\ldots ,r$. Our estimate ${\hat {\theta }}$ is thus

${\hat {\theta }}=g(M_{1},M_{2},\ldots ,M_{r})$

EX1: We show how to estimate the variance with the method of moments. Recall that the variance

$Var(X)=g(E[X],E[X^{2}])=E[X^{2}]-E[X]^{2}.$

We first estimate the first moment $M_{1}={\frac {\sum _{i=1}^{n}X}{n}}$ and the second moment $M_{2}={\frac {\sum _{i=1}^{n}X^{2}}{n}}$. The estimator is

${\hat {\theta }}=M_{2}-M_{1}^{2}=\left({\frac {\sum _{i=1}^{n}X^{2}}{n}}\right)-\left({\frac {\sum _{i=1}^{n}X}{n}}\right)^{2}={\frac {\sum _{i=1}^{n}(X_{i}-{\bar {X}})^{2}}{n}}.$

Note that the estimate ${\hat {\theta }}$ is biased, because $E[{\hat {\theta }}]={\frac {n-1}{n}}Var(X)$.

As this example shows, other estimation techniques are usually preferred over the method of moments.

Maximum likelihood estimators

Suppose that we want to estimate parameter $\theta$ based on observations (or sample) $X_{1},X_{2},\ldots ,X_{n}$. If we can find a joint-distribution function

$f(x_{1},x_{2},x_{3},\ldots ,x_{n}|\theta )$,

that gives the probability that we observe $(x_{1},x_{2},\ldots ,x_{n})$ given a particular value $\theta$. The maximum likelihood estimator is ${\hat {\theta }}$ such that $f(x_{1},x_{2},\ldots ,x_{n}|{\hat {\theta }})$ is maximum, i.e.,

${\hat {\theta }}=\arg \max _{\theta }f(x_{1},x_{2},x_{3},\ldots ,x_{n}|\theta )$.

EX1: maximum likelihood estimator for a Bernoulli parameter

EX2: maximum likelihood estimator for the Poisson mean