LEC ALG1 21

Inner Product Spaces

The motivation behind this is to define a notion of length and perpendicularity (angle) for vectors.

Definition 87.1(Definition).

A vector space $V$ over $\mathbb{F}=\mathbb{R},\mathbb{C}$ is an inner product space if for any two $\mathbf{v},\mathbf{w}\in V$, there is defined an element $⟨\mathbf{v},\mathbf{w}⟩\in \mathbb{F}$ such that it satisfies the following properties:

• $⟨\mathbf{v},\mathbf{w}⟩$ = $\overline{⟨\mathbf{w},\mathbf{v}⟩}$;
• $⟨\mathbf{v},\mathbf{v}⟩\ge 0$;
• $⟨\mathbf{v},\mathbf{v}⟩=0⟺\mathbf{v}=0$;
• $⟨\alpha \mathbf{u}+\beta \mathbf{v},\mathbf{w}⟩$ = $\alpha ⟨\mathbf{u},\mathbf{w}⟩+\beta ⟨\mathbf{v},\mathbf{w}⟩$.

dd5bbe

Proposition 87.2(Proposition).

The inner product is anti-linear in the second slot.

Proof.

$$\begin{array}{rl}⟨\mathbf{u},\alpha \mathbf{v}+\beta \mathbf{w}⟩& =\overline{⟨\alpha \mathbf{v}+\beta \mathbf{w},\mathbf{u}⟩}\\ & =\overline{\alpha ⟨\mathbf{v},\mathbf{u}⟩+\beta ⟨\mathbf{w},\mathbf{u}⟩}\\ & =\overline{\alpha }\overline{⟨\mathbf{v},\mathbf{u}⟩}+\overline{\beta }\overline{⟨\mathbf{w},\mathbf{u}⟩}\\ & =\overline{\alpha }⟨\mathbf{u},\mathbf{v}⟩+\overline{\beta }⟨\mathbf{u},\mathbf{w}⟩.\end{array}$$ □

Proposition 87.3(Proposition).

$⟨\alpha \mathbf{u}+\beta \mathbf{v},\alpha \mathbf{u}+\beta \mathbf{v}⟩=\alpha \bar{\alpha }⟨\mathbf{u},\mathbf{u}⟩+\alpha \bar{\beta }⟨\mathbf{u},\mathbf{v}⟩+\bar{\alpha }\beta ⟨\mathbf{v},\mathbf{u}⟩+\beta \bar{\beta }⟨\mathbf{v},\mathbf{v}⟩$.

ef218e

Proposition 87.4(Proposition).

If $\mathbf{u}=0$, then $⟨\mathbf{u},\mathbf{v}⟩=0$.

Proof.

Method 1:

$$⟨0,\mathbf{v}⟩=⟨0+0,\mathbf{v}⟩=⟨0,\mathbf{v}⟩+⟨0,\mathbf{v}⟩⟹⟨0,\mathbf{v}⟩=0$$

Method 2: (used only by the utterly deranged)
Consider the linear map $T:V\to \mathbb{F}$ where $\mathbf{u}\mapsto ⟨\mathbf{u},\mathbf{v}⟩$ for a fixed $\mathbf{v}\in V$. We know that $T(0)=0$. Hence $⟨0,\mathbf{u}⟩=0$. As $\mathbf{v}$ was arbitrary, the proposition holds for any $\mathbf{v}\in V$. ❏□

Example 87.5(Hermitian Dot Product / Standard Hermitian form).

For vectors $\mathbf{v},\mathbf{w}\in {\mathbb{C}}^n$, let $\mathbf{v}=(v_1,v_2,\dots ,v_n)$ and $\mathbf{w}=(w_1,w_2,\dots ,w_n)$. The inner product of $\mathbf{v}$ and $\mathbf{w}$ is defined as

$$⟨\mathbf{v},\mathbf{w}⟩=v_1{\overline{w}}_1+v_2{\overline{w}}_2+\cdots +v_n{\overline{w}}_n,$$

where $\overline{z}$ denotes the complex conjugate.

Motivation

If we used the naïve definition of the inner product, that is

$$⟨\mathbf{v},\mathbf{w}⟩=v_1{w}_1+v_2{w}_2+\cdots +v_n{w}_n$$

Then for the example of $(1,i)\in {\mathbb{C}}^2$, we have $⟨(1,i),(1,i)⟩=1\cdot 1+i\cdot i=0$. This is somewhat of a problem as we expect this to be positive. If we use the Hermitian dot product, we get a nicer answer: $⟨(1,i),(1,i)⟩=1\cdot \bar{1}+i\cdot \bar{i}=2$. The Hermitian dot product in fact guarantees $⟨\mathbf{v},\mathbf{v}⟩\ge 0$ and $⟨\mathbf{v},\mathbf{v}⟩=0⟺\mathbf{v}=0$ along with the other two properties, as you can easily verify.

Example 87.6(Inner Product of Functions (Hilbert Space)).

Let $V$ be the set of all continuous real/complex functions on $[0,1]$.
For $f(t),g(t)\in V$ we define their inner product as

$$⟨f(t),g(t)⟩={\int }_0^{1}f(t)\overline{g(t)}dt$$

It is easy to see that this satisfies all the requirements of the inner product.

Norm

Definition 87.7(Definition).

Given an inner product space, one defines a norm on it by

$$\Vert \mathbf{v}\Vert =\sqrt{⟨\mathbf{v},\mathbf{v}⟩}.$$

2ffc13

If we plug $\beta =0$ in Proposition 3, we get $\Vert \lambda \mathbf{v}\Vert =|\lambda |\Vert \mathbf{v}\Vert$.

The Cauchy-Schwarz Inequality

We will need this to show that the norm defined above satisfies the triangle inequality.

Theorem 87.8(Cauchy-Schwarz).

Let $V$ be an inner product space. For any two $\mathbf{u},\mathbf{v}\in V$, we have

$$|⟨\mathbf{u},\mathbf{v}⟩|\le \Vert \mathbf{u}\Vert \Vert \mathbf{v}\Vert .$$

Proof.

Case 1: $⟨\mathbf{u},\mathbf{v}⟩\in \mathbb{R}$

For any $\lambda \in \mathbb{R}$, we have

$$\begin{array}{rl}⟨\lambda \mathbf{u}+\mathbf{v},\lambda \mathbf{u}+\mathbf{v}⟩& \ge 0\\ {\lambda }^2⟨\mathbf{u},\mathbf{u}⟩+2\lambda ⟨\mathbf{u},\mathbf{v}⟩+⟨\mathbf{v},\mathbf{v}⟩& \ge 0.\end{array}$$

As this holds for all $\lambda$, we have

$$\begin{array}{rl}4⟨u,v{⟩}^2& \le 4⟨u,u⟩⟨v,v⟩\\ |⟨u,v⟩|& \le \Vert u\Vert \Vert v\Vert .\end{array}$$

Case 2: $⟨\mathbf{u},\mathbf{v}⟩\notin \mathbb{R}$

Note that $⟨\mathbf{u},\mathbf{v}⟩\ne 0$ as $⟨\mathbf{u},\mathbf{v}⟩\notin \mathbb{R}$.
Let $\alpha =⟨\mathbf{u},\mathbf{v}⟩$. Observe that

$$⟨\frac{\mathbf{u}}{\alpha },\mathbf{v}⟩=\frac{1}{\alpha }⟨\mathbf{u},\mathbf{v}⟩=1\in \mathbb{R}$$

We can apply Case 1.

$$\begin{array}{rl}\left|⟨\frac{\mathbf{u}}{\alpha },\mathbf{v}⟩\right|& \le \Vert \frac{\mathbf{u}}{\alpha }\Vert \Vert \mathbf{v}\Vert \\ \cancel{\frac{1}{|\alpha |}}|⟨\mathbf{u},\mathbf{v}⟩|& \le \cancel{\frac{1}{|\alpha |}}\Vert \mathbf{u}\Vert \Vert \mathbf{v}\Vert \\ |⟨\mathbf{u},\mathbf{v}⟩|& \le \Vert \mathbf{u}\Vert \Vert \mathbf{v}\Vert \end{array}$$ □

c0606c

Corollary 87.9(Corollary).

$${\left|{\int }_0^{1}f(t)g(t)dt\right|}^2\le \left({\int }_0^1|f^2(t)|dt\right)\left({\int }_0^1|g^2(t)|dt\right)$$

Average velocity over time has to be less than or equal to average velocity over distance, with equality only for constant velocity.

The Triangle Inequality

Proposition 87.10(Proposition).

For any vectors $\mathbf{x}$ and $\mathbf{y}$ in an inner product space, we have

$$\Vert \mathbf{x}+\mathbf{y}\Vert \le \Vert \mathbf{x}\Vert +\Vert \mathbf{y}\Vert .$$

Proof.

Using Theorem 8,

$$\begin{array}{rl}\Vert \mathbf{x}+\mathbf{y}{\Vert }^2& =⟨\mathbf{x}+\mathbf{y},\mathbf{x}+\mathbf{y}⟩\\ & =\Vert \mathbf{x}{\Vert }^2+\Vert \mathbf{y}{\Vert }^2+⟨\mathbf{x},\mathbf{y}⟩+⟨\mathbf{y},\mathbf{x}⟩\\ & =\Vert \mathbf{x}{\Vert }^2+\Vert \mathbf{y}{\Vert }^2+2\mathrm{Re}⟨\mathbf{x},\mathbf{y}⟩\\ & \le \Vert \mathbf{x}{\Vert }^2+\Vert \mathbf{y}{\Vert }^2+2|⟨\mathbf{x},\mathbf{y}⟩|\\ & \le \Vert \mathbf{x}{\Vert }^2+\Vert \mathbf{y}{\Vert }^2+2\Vert \mathbf{x}\Vert \Vert \mathbf{y}\Vert \\ & =(\Vert \mathbf{x}\Vert +\Vert \mathbf{y}\Vert {)}^2.\end{array}$$ □

---

Normed spaces

We have shown that the norm $\Vert \mathbf{v}\Vert$ derived from the inner product satisfies the following properties:

1. Homogeneity: $\Vert \alpha \mathbf{v}\Vert =|\alpha |\Vert \mathbf{v}\Vert$ for all $\mathbf{v}\in V$ and $\alpha \in \mathbb{F}$.
2. Triangle inequality: $\Vert \mathbf{x}+\mathbf{y}\Vert \le \Vert \mathbf{x}\Vert +\Vert \mathbf{y}\Vert$.
3. Non-negativity: $\Vert \mathbf{v}\Vert \ge 0$ for all $\mathbf{v}\in V$.
4. Non-degeneracy: $\Vert \mathbf{v}\Vert =0⟺\mathbf{v}=0$.

Now, suppose in a vector space $V$ we assigned to each vector $\mathbf{v}$ a number $\Vert \mathbf{v}\Vert$ such that the above four properties are satisfied. Then, we say that the function $\mathbf{v}\mapsto \Vert \mathbf{v}\Vert$ is a norm. A vector space equipped with a norm is called a normed space.

Any inner product space is a normed space, as $\Vert \mathbf{v}\Vert =\sqrt{⟨\mathbf{v},\mathbf{v}⟩}$ satisfies the above properties. However, not all normed spaces are inner product spaces.

Orthogonality

If $\mathbf{u},\mathbf{v}\in V$ then is said to be orthogonal to if $⟨\mathbf{u},\mathbf{v}⟩=0$. If $\mathbf{u}$ is orthogonal to then $\mathbf{v}$ is orthogonal to $\mathbf{u}$ as $⟨\mathbf{v},\mathbf{u}⟩=\overline{⟨\mathbf{u},\mathbf{v}⟩}=0$.

Definition 87.11(Definition).

If $W\subset V$ is a subspace then the orthogonal complement of $W$ is defined as

$$W^{\perp }\equiv \{\mathbf{x}\in V\mid ⟨\mathbf{x},\mathbf{w}⟩=0\forall \mathbf{w}\in W\}.$$

Proposition 87.12(Proposition).

If $W$ is a subspace of $V$, then $W^{\perp }$ is a subspace of $V$.

Proof.

1. $0\in W^{\perp }$ as $⟨0,\mathbf{w}⟩=0\forall \mathbf{w}\in W$.
2. For any $\mathbf{u},\mathbf{v}\in W^{\perp }$ and $\mathbf{w}\in W$, we have

$$⟨\alpha \mathbf{u}+\beta \mathbf{v},\mathbf{w}⟩=\alpha ⟨\mathbf{u},\mathbf{w}⟩+\beta ⟨\mathbf{v},\mathbf{w}⟩=0.$$

So, $W^{\perp }$ is closed under addition and scalar multiplication.□

Clearly, $W\cap W^{\perp }=\{0\}$, since if $\mathbf{w}\in W\cap W^{\perp }$ then $⟨\mathbf{w},\mathbf{w}⟩=0⟹\mathbf{w}=0$.

Definition 87.13(Definition).

A set of vectors $\{{\mathbf{v}}_i\}\subseteq V$ is an orthonormal set if

$$⟨{\mathbf{v}}_i,{\mathbf{v}}_j⟩=\{\begin{array}{ll}1& i=j\\ 0& i\ne j.\end{array}$$

Proposition 87.14(Proposition).

If $\{{\mathbf{v}}_i\}\subset V$ is an orthonormal set then $\{{\mathbf{v}}_i\}$ are linearly independent.

Proof.

Assume that for some${\beta }_i$, ${\sum }_{i=1}^n{\beta }_i{\mathbf{v}}_i=0$. Then,

$$\begin{array}{r}0=⟨0,{\mathbf{v}}_k⟩=⟨\sum _{i=1}^n{\beta }_i{\mathbf{v}}_i,{\mathbf{v}}_k⟩=\sum _{i=1}^n{\beta }_i⟨{\mathbf{v}}_i,{\mathbf{v}}_k⟩={\beta }_k.\end{array}$$

As $1\le k\le n$, we have all ${\beta }_k=0$. Hence $\{{\mathbf{v}}_i\}$ is linearly independent.□

For an orthonormal set $\{{\mathbf{v}}_i\}$, if there is some $\mathbf{w}$ such that $\mathbf{w}={\alpha }_1{\mathbf{v}}_1+{\alpha }_2{\mathbf{v}}_2+\cdots +{\alpha }_n{\mathbf{v}}_n$, then ${\alpha }_k=⟨\mathbf{w},{\mathbf{v}}_k⟩$.

For an orthonormal set $\{{\mathbf{v}}_i\}$ in $V$ and any $\mathbf{w}\in V$,

$$\mathbf{u}\equiv \mathbf{w}-⟨\mathbf{w},{\mathbf{v}}_1⟩{\mathbf{v}}_1-⟨\mathbf{w},{\mathbf{v}}_2⟩{\mathbf{v}}_2-\cdots -⟨\mathbf{w},{\mathbf{v}}_n⟩{\mathbf{v}}_n$$

is orthogonal to each of ${\mathbf{v}}_1,{\mathbf{v}}_2,\dots ,{\mathbf{v}}_n$.

Definition 87.15(Definition).

A real $n\times n$ matrix $A$ is orthogonal if $A^{T}A=I$, which is to say $A$ is invertible and $A^{-1}=A^T$.

d11820

Proposition 87.16(Proposition).

An $n\times n$ matrix $A$ is orthogonal if and only if its columns form an orthonormal basis of ${\mathbb{R}}^n$.

Proof.

Let${\mathbf{v}}_1,{\mathbf{v}}_2,\dots ,{\mathbf{v}}_n$ be the column vectors of $A$. Then, the $i,j$-entry of $A^{T}A$ is given by ${\mathbf{v}}_i^T{\mathbf{v}}_j$, which is $⟨{\mathbf{v}}_i,{\mathbf{v}}_j⟩$. Thus, if $A^{T}A=I$, the column vectors must be orthonormal, and if the column vectors are orthonormal, $A^{T}A=I$.□

Theorem 87.17(Gram-Schmidt).

Every finite dimensional inner product space has an orthonormal basis.

Proof.

Let$V$ be any finite dimensional inner product space. Take a basis of $V$, say $\{{\mathbf{v}}_1,{\mathbf{v}}_2,\dots ,{\mathbf{v}}_n\}$. From this, we will construct an orthonormal set of $n$ vectors.

Let ${\mathbf{u}}_1\equiv {\mathbf{v}}_1$. Define

$${\mathbf{w}}_1\equiv \frac{{\mathbf{u}}_1}{\Vert {\mathbf{u}}_1\Vert },$$

which gives us $⟨{\mathbf{w}}_1,{\mathbf{w}}_1⟩=1$. Now, ${\mathbf{u}}_2$ defined as

$${\mathbf{u}}_2\equiv {\mathbf{v}}_2-⟨{\mathbf{v}}_2,{\mathbf{w}}_1⟩{\mathbf{w}}_1$$

is orthogonal to ${\mathbf{w}}_1$. Note that ${\mathbf{u}}_2\ne 0$ as ${\mathbf{v}}_2$ and ${\mathbf{w}}_1$ are linearly independent. Now we can define ${\mathbf{w}}_2$ as

$${\mathbf{w}}_2=\frac{{\mathbf{u}}_2}{\Vert {\mathbf{u}}_2\Vert }.$$

Continuing this process, we get

$$\begin{array}{rlr}{\mathbf{u}}_1& ={\mathbf{v}}_1& {\mathbf{w}}_1={\hat{\mathbf{u}}}_1\\ {\mathbf{u}}_2& ={\mathbf{v}}_2-⟨{\mathbf{v}}_2,{\mathbf{w}}_1⟩{\mathbf{w}}_1& {\mathbf{w}}_2={\hat{\mathbf{u}}}_2\\ {\mathbf{u}}_3& ={\mathbf{v}}_3-⟨{\mathbf{v}}_3,{\mathbf{w}}_1⟩{\mathbf{w}}_1-⟨{\mathbf{v}}_3,{\mathbf{w}}_2⟩{\mathbf{w}}_2& {\mathbf{w}}_3={\hat{\mathbf{u}}}_3\\ & ⋮& ⋮\\ {\mathbf{u}}_k& ={\mathbf{v}}_k-⟨{\mathbf{v}}_k,{\mathbf{w}}_1⟩{\mathbf{w}}_1-⟨{\mathbf{v}}_k,{\mathbf{w}}_2⟩{\mathbf{w}}_2-\cdots -⟨{\mathbf{v}}_k,{\mathbf{w}}_{k-1}⟩{\mathbf{w}}_{k-1}& {\mathbf{w}}_k={\hat{\mathbf{u}}}_k\end{array}$$

where $\hat{\mathbf{u}}$ represents the normalized vector $\mathbf{u}/\Vert \mathbf{u}\Vert$. Thus, $\{{\mathbf{w}}_i\}$ is an orthonormal set, and hence a linearly independent set. Since a linearly independent set of size $n=\dim V$ is a basis, $\{{\mathbf{w}}_i\}$ is a basis of $V$.□

f789a3