Week4 Machine Learning

Let's first review some linear algebra concepts.

Hilbert spaces

Recall some basic definitions:

Def: An inner product is$<\cdot ,\cdot >$ is a bilinear form on a pair of vectors satisfying

• $<v,v>\geq 0$ and <v,v> $=0\iff v=0$
• $<u,v>={\bar {<v,u>}}$
• $<ru+sv,w>=r<u,w>+s<v,w>$

Note that every inner product space is a normed linear space with the norm

$||v||={\sqrt {<v,v>}}$

And with this norm, the inner product space forms a metric.

Def: A metric space is complete if every cauchy sequence converges to an element in the space

Def: A Hilbert space is a complete inner product space

Reproducing kernel Hilbert spaces

Let ${\mathcal {H}}$ be a Hilbert space consisting of functions on ${\mathcal {X}}$. A function $K:{\mathcal {X}}\times {\mathcal {X}}\rightarrow \mathbb {R}$ is called a reproducing kernel for${\mathcal {H}}$ if

• For all y, $K_{y}=K(\cdot ,y)$ belongs to ${\mathcal {H}}$
• (Reproducing property): For all y, for all $f\in {\mathcal {H}}$,

$f(y)=<f,K_{y}>$

Note that, in this case, dot-product is defined in a natural way, i.e.

$<f,g>=\int _{X}f(x){\bar {g}}(x)$

Properties of r.k. Hilbert spaces

• It is known that if a r.k. exists for a given Hilbert space, then it is unique. The proof is two-line

• (Existence of K):

Example 1: finite domain

We show a motivating example when ${\mathcal {X}}$ is finite.