ALG1_L21

Inner Product Spaces

The motivation behind this is to define a notion of length and perpendicularity (angle) for vectors.

Definition 1(Definition).

A vector space $V$ over $\mathbb{F}=\mathbb{R},\mathbb{C}$ is an inner product space if for any two $\mathbf{v},\mathbf{w}\in V$, there is defined an element $⟨\mathbf{v},\mathbf{w}⟩\in \mathbb{F}$ such that it satisfies the following properties:

• $⟨\mathbf{v},\mathbf{w}⟩$ = $\overline{⟨\mathbf{w},\mathbf{v}⟩}$;
• $⟨\mathbf{v},\mathbf{v}⟩\ge 0$;
• $⟨\mathbf{v},\mathbf{v}⟩=0⟺\mathbf{v}=0$;
• $⟨\alpha \mathbf{u}+\beta \mathbf{v},\mathbf{w}⟩$ = $\alpha ⟨\mathbf{u},\mathbf{w}⟩+\beta ⟨\mathbf{v},\mathbf{w}⟩$.

Theorem 2(Theorem 1).

The inner product is anti-linear in the second slot.

Proof

$$\begin{array}{rl}⟨\mathbf{u},\alpha \mathbf{v}+\beta \mathbf{w}⟩& =\overline{⟨\alpha \mathbf{v}+\beta \mathbf{w},\mathbf{u}⟩}\\ & =\overline{\alpha ⟨\mathbf{v},\mathbf{u}⟩+\beta ⟨\mathbf{w},\mathbf{u}⟩}\\ & =\overline{\alpha }\overline{⟨\mathbf{v},\mathbf{u}⟩}+\overline{\beta }\overline{⟨\mathbf{w},\mathbf{u}⟩}\\ & =\overline{\alpha }⟨\mathbf{u},\mathbf{v}⟩+\overline{\beta }⟨\mathbf{u},\mathbf{w}⟩.\end{array}$$

❏

Theorem 3(Corollary 1).

$⟨\alpha \mathbf{u}+\beta \mathbf{v},\alpha \mathbf{u}+\beta \mathbf{v}⟩=\alpha \bar{\alpha }⟨\mathbf{u},\mathbf{u}⟩+\alpha \bar{\beta }⟨\mathbf{u},\mathbf{v}⟩+\bar{\alpha }\beta ⟨\mathbf{v},\mathbf{u}⟩+\beta \bar{\beta }⟨\mathbf{v},\mathbf{v}⟩$.

Theorem 4(Theorem 2).

If $\mathbf{u}=0$, then $⟨\mathbf{u},\mathbf{v}⟩=0$.

Proof
Method 1:

$$⟨0,\mathbf{v}⟩=⟨0+0,\mathbf{v}⟩=⟨0,\mathbf{v}⟩+⟨0,\mathbf{v}⟩⟹⟨0,\mathbf{v}⟩=0$$

Method 2: (used only by the utterly deranged)
Consider the linear map $T:V\to \mathbb{F}$ where $\mathbf{u}\mapsto ⟨\mathbf{u},\mathbf{v}⟩$ for a fixed $\mathbf{v}\in V$. We know that $T(0)=0$. Hence $⟨0,\mathbf{u}⟩=0$. As $\mathbf{v}$ was arbitrary, the proposition holds for any $\mathbf{v}\in V$. ❏

Example 5(Hermitian Dot Product / Standard Hermitian form).

For vectors $\mathbf{v},\mathbf{w}\in {\mathbb{C}}^n$, let $\mathbf{v}=(v_1,v_2,\dots ,v_n)$ and $\mathbf{w}=(w_1,w_2,\dots ,w_n)$. The inner product of $\mathbf{v}$ and $\mathbf{w}$ is defined as

$$⟨\mathbf{v},\mathbf{w}⟩=v_1{\overline{w}}_1+v_2{\overline{w}}_2+\cdots +v_n{\overline{w}}_n,$$

where $\overline{z}$ denotes the complex conjugate.

Motivation

If we used the naïve definition of the inner product, that is

$$⟨\mathbf{v},\mathbf{w}⟩=v_1{w}_1+v_2{w}_2+\cdots +v_n{w}_n$$

Then for the example of $(1,i)\in {\mathbb{C}}^2$, we have $⟨(1,i),(1,i)⟩=1\cdot 1+i\cdot i=0$. This is somewhat of a problem as we expect this to be positive. If we use the Hermitian dot product, we get a nicer answer: $⟨(1,i),(1,i)⟩=1\cdot \bar{1}+i\cdot \bar{i}=2$. The Hermitian dot product in fact guarantees $⟨\mathbf{v},\mathbf{v}⟩\ge 0$ and $⟨\mathbf{v},\mathbf{v}⟩=0⟺\mathbf{v}=0$ along with the other two properties, as you can easily verify.

Example 6(Inner Product of Functions (Hilbert Space)).

Let $V$ be the set of all continuous real/complex functions on $[0,1]$.
For $f(t),g(t)\in V$ we define their inner product as

$$⟨f(t),g(t)⟩={\int }_0^{1}f(t)\overline{g(t)}dt$$

It is easy to see that this satisfies all the requirements of the inner product.

Norm

Definition 7(Definition).

Given an inner product space, one defines a norm on it by

$$\Vert \mathbf{v}\Vert =\sqrt{⟨\mathbf{v},\mathbf{v}⟩}.$$

If we plug $\beta =0$ in theorem 2 above, we get $\Vert \lambda \mathbf{v}\Vert =|\lambda |\Vert \mathbf{v}\Vert$.

The Cauchy-Schwarz Inequality

Theorem 8(Theorem).

Let $V$ be an inner product space. For any two $\mathbf{u},\mathbf{v}\in V$, we have

$$|⟨\mathbf{u},\mathbf{v}⟩|\le \Vert \mathbf{u}\Vert \Vert \mathbf{v}\Vert .$$

Proof
Case 1: $⟨\mathbf{u},\mathbf{v}⟩\in \mathbb{R}$

For any $\lambda \in \mathbb{R}$, we have

$$\begin{array}{rl}⟨\lambda \mathbf{u}+\mathbf{v},\lambda \mathbf{u}+\mathbf{v}⟩& \ge 0\\ {\lambda }^2⟨\mathbf{u},\mathbf{u}⟩+2\lambda ⟨\mathbf{u},\mathbf{v}⟩+⟨\mathbf{v},\mathbf{v}⟩& \ge 0.\end{array}$$

As this holds for all $\lambda$, we have

$$\begin{array}{rl}4⟨u,v{⟩}^2& \le 4⟨u,u⟩⟨v,v⟩\\ |⟨u,v⟩|& \le \Vert u\Vert \Vert v\Vert .\end{array}$$

Case 2: $⟨\mathbf{u},\mathbf{v}⟩\notin \mathbb{R}$

Note that $⟨\mathbf{u},\mathbf{v}⟩\ne 0$ as $⟨\mathbf{u},\mathbf{v}⟩\notin \mathbb{R}$.
Let $\alpha =⟨\mathbf{u},\mathbf{v}⟩$. Observe that

$$⟨\frac{\mathbf{u}}{\alpha },\mathbf{v}⟩=\frac{1}{\alpha }⟨\mathbf{u},\mathbf{v}⟩=1\in \mathbb{R}$$

We can apply Case 1.

$$\begin{array}{rl}\left|⟨\frac{\mathbf{u}}{\alpha },\mathbf{v}⟩\right|& \le \Vert \frac{\mathbf{u}}{\alpha }\Vert \Vert \mathbf{v}\Vert \\ \cancel{\frac{1}{|\alpha |}}|⟨\mathbf{u},\mathbf{v}⟩|& \le \cancel{\frac{1}{|\alpha |}}\Vert \mathbf{u}\Vert \Vert \mathbf{v}\Vert \\ |⟨\mathbf{u},\mathbf{v}⟩|& \le \Vert \mathbf{u}\Vert \Vert \mathbf{v}\Vert \end{array}$$

❏

Theorem 9(Corollary).

$${\left|{\int }_0^{1}f(t)g(t)dt\right|}^2\le \left({\int }_0^1|f^2(t)|dt\right)\left({\int }_0^1|g^2(t)|dt\right)$$

Average velocity over time has to be less than or equal to average velocity over distance, with equality only for constant velocity.

Now, bow before the almighty…

The Triangle Inequality

Theorem 10(Theorem).

For any vectors $\mathbf{x}$ and $\mathbf{y}$ in an inner product space, we have

$$\Vert \mathbf{x}+\mathbf{y}\Vert \le \Vert \mathbf{x}\Vert +\Vert \mathbf{y}\Vert .$$

Proof

$$\begin{array}{rl}\Vert \mathbf{x}+\mathbf{y}{\Vert }^2& =⟨\mathbf{x}+\mathbf{y},\mathbf{x}+\mathbf{y}⟩\\ & =\Vert \mathbf{x}{\Vert }^2+\Vert \mathbf{y}{\Vert }^2+⟨\mathbf{x},\mathbf{y}⟩+⟨\mathbf{y},\mathbf{x}⟩\\ & =\Vert \mathbf{x}{\Vert }^2+\Vert \mathbf{y}{\Vert }^2+2\mathrm{Re}⟨\mathbf{x},\mathbf{y}⟩\\ & \le \Vert \mathbf{x}{\Vert }^2+\Vert \mathbf{y}{\Vert }^2+2|⟨\mathbf{x},\mathbf{y}⟩|\\ & \le \Vert \mathbf{x}{\Vert }^2+\Vert \mathbf{y}{\Vert }^2+2\Vert \mathbf{x}\Vert \Vert \mathbf{y}\Vert \\ & =(\Vert \mathbf{x}\Vert +\Vert \mathbf{y}\Vert {)}^2.\end{array}$$

❏

---

Normed spaces

We have shown that the norm $\Vert \mathbf{v}\Vert$ derived from the inner product satisfies the following properties:

1. Homogeneity: $\Vert \alpha \mathbf{v}\Vert =|\alpha |\Vert \mathbf{v}\Vert$ for all $\mathbf{v}\in V$ and $\alpha \in \mathbb{F}$.
2. Triangle inequality: $\Vert \mathbf{x}+\mathbf{y}\Vert \le \Vert \mathbf{x}\Vert +\Vert \mathbf{y}\Vert$.
3. Non-negativity: $\Vert \mathbf{v}\Vert \ge 0$ for all $\mathbf{v}\in V$.
4. Non-degeneracy: $\Vert \mathbf{v}\Vert =0⟺\mathbf{v}=0$.

Now, suppose in a vector space $V$ we assigned to each vector $\mathbf{v}$ a number $\Vert \mathbf{v}\Vert$ such that the above four properties are satisfied. Then, we say that the function $\mathbf{v}\mapsto \Vert \mathbf{v}\Vert$ is a norm.

Definition 11(Definition).

A vector space equipped with a norm is called a normed space.

Any inner product space is a normed space, as $\Vert \mathbf{v}\Vert =\sqrt{⟨\mathbf{v},\mathbf{v}⟩}$ satisfies the above properties. However, not all normed spaces are inner product spaces.

---

Orthogonal Vectors

Definition 12(Definition).

If $\mathbf{u},\mathbf{v}\in V$ then $\mathbf{u}$ is orthogonal to $\mathbf{v}$ if $⟨\mathbf{u},\mathbf{v}⟩=0$.

If $\mathbf{u}$ is orthogonal to $\mathbf{v}$ then $\mathbf{v}$ is orthogonal to $\mathbf{u}$ as $⟨\mathbf{v},\mathbf{u}⟩=\overline{⟨\mathbf{u},\mathbf{v}⟩}=0$

Definition 13(Definition).

If $W\subset V$ is a subspace then the orthogonal complement of $W$ is defined as

$$W^{\perp }\equiv \{\mathbf{x}\in V\mid ⟨\mathbf{x},\mathbf{w}⟩=0\forall \mathbf{w}\in W\}.$$

Theorem 14(Theorem).

$W^{\perp }$ is a subspace of $V$.

Proof

1. $0\in W^{\perp }$ as $⟨0,\mathbf{w}⟩=0\forall \mathbf{w}\in W$.
2. For any $\mathbf{u},\mathbf{v}\in W^{\perp }$, we have

$$⟨\alpha \mathbf{u}+\beta \mathbf{v},\mathbf{w}⟩=\alpha ⟨\mathbf{u},\mathbf{w}⟩+\beta ⟨\mathbf{v},\mathbf{w}⟩=0.$$

So, $W^{\perp }$ is closed under addition and scalar multiplication. ❏

Theorem 15(Theorem).

$W\cap W^{\perp }=\{0\}.$

Proof
If $\mathbf{w}\in W\cap W^{\perp }$ then $⟨\mathbf{w},\mathbf{w}⟩=0⟹\mathbf{w}=0$. ❏

Orthonormal vectors

Definition 16(Definition).

A set of vectors $\{{\mathbf{v}}_i\}$ in $V$ is an orthonormal set if

$$⟨{\mathbf{v}}_i,{\mathbf{v}}_j⟩=\{\begin{array}{ll}1& i=j\\ 0& i\ne j.\end{array}$$

Theorem 17(Theorem).

If $\{{\mathbf{v}}_i\}\subset V$ is an orthonormal set then $\{{\mathbf{v}}_i\}$ are linearly independent.

Proof
Assume that for some ${\beta }_i$, ${\sum }_{i=1}^n{\beta }_i{\mathbf{v}}_i=0$. Then,

$$\begin{array}{r}0=⟨0,{\mathbf{v}}_k⟩=⟨\sum _{i=1}^n{\beta }_i{\mathbf{v}}_i,{\mathbf{v}}_k⟩=\sum _{i=1}^n{\beta }_i⟨{\mathbf{v}}_i,{\mathbf{v}}_k⟩={\beta }_k.\end{array}$$

As $1\le k\le n$, we have all ${\beta }_k=0$. Hence $\{{\mathbf{v}}_i\}$ is linearly independent. ❏

Important

For an orthonormal set $\{{\mathbf{v}}_i\}$, if there is some $\mathbf{w}$ such that $\mathbf{w}={\alpha }_1{\mathbf{v}}_1+{\alpha }_2{\mathbf{v}}_2+\cdots +{\alpha }_n{\mathbf{v}}_n$, then ${\alpha }_k=⟨\mathbf{w},{\mathbf{v}}_k⟩$.

Important

For an orthonormal set $\{{\mathbf{v}}_i\}$ in $V$ and any $\mathbf{w}\in V$,

$$\mathbf{u}\equiv \mathbf{w}-⟨\mathbf{w},{\mathbf{v}}_1⟩{\mathbf{v}}_1-⟨\mathbf{w},{\mathbf{v}}_2⟩{\mathbf{v}}_2-\cdots -⟨\mathbf{w},{\mathbf{v}}_n⟩{\mathbf{v}}_n$$

is orthogonal to each of ${\mathbf{v}}_1,{\mathbf{v}}_2,\dots ,{\mathbf{v}}_n$.

Orthogonal matrices

Definition 18(Definition).

A real $n\times n$ matrix $A$ is orthogonal if $A^{T}A=I$, which is to say $A$ is invertible and $A^{-1}=A^T$.

Theorem 19(Theorem).

An $n\times n$ matrix $A$ is orthogonal if and only if its columns form an orthonormal basis of ${\mathbb{R}}^n$.

Proof
Let ${\mathbf{v}}_1,{\mathbf{v}}_2,\dots ,{\mathbf{v}}_n$ be the column vectors of $A$. Then, the $i,j$-entry of $A^{T}A$ is given by ${\mathbf{v}}_i^T{\mathbf{v}}_j$, which is $⟨{\mathbf{v}}_i,{\mathbf{v}}_j⟩$. Thus, if $A^{T}A=I$, the column vectors must be orthonormal, and if the column vectors are orthonormal, $A^{T}A=I$.

Gram-Schmidt Orthogonalization Process

Theorem 20(Theorem ).

Every finite dimensional inner product space has an orthonormal basis.

Proof
Let $V$ be any finite dimensional inner product space. Take a basis of $V$, say $\{{\mathbf{v}}_1,{\mathbf{v}}_2,\dots ,{\mathbf{v}}_n\}$. From this, we will construct an orthonormal set of $n$ vectors.

Let ${\mathbf{u}}_1\equiv {\mathbf{v}}_1$. Define

$${\mathbf{w}}_1\equiv \frac{{\mathbf{u}}_1}{\Vert {\mathbf{u}}_1\Vert },$$

which gives us $⟨{\mathbf{w}}_1,{\mathbf{w}}_1⟩=1$. Now, ${\mathbf{u}}_2$ defined as

$${\mathbf{u}}_2\equiv {\mathbf{v}}_2-⟨{\mathbf{v}}_2,{\mathbf{w}}_1⟩{\mathbf{w}}_1$$

is orthogonal to ${\mathbf{w}}_1$. Note that ${\mathbf{u}}_2\ne 0$ as ${\mathbf{v}}_2$ and ${\mathbf{w}}_1$ are linearly independent. Now we can define ${\mathbf{w}}_2$ as

$${\mathbf{w}}_2=\frac{{\mathbf{u}}_2}{\Vert {\mathbf{u}}_2\Vert }.$$

Continuing this process, we get

$$\begin{array}{rlr}{\mathbf{u}}_1& ={\mathbf{v}}_1& {\mathbf{w}}_1={\hat{\mathbf{u}}}_1\\ {\mathbf{u}}_2& ={\mathbf{v}}_2-⟨{\mathbf{v}}_2,{\mathbf{w}}_1⟩{\mathbf{w}}_1& {\mathbf{w}}_2={\hat{\mathbf{u}}}_2\\ {\mathbf{u}}_3& ={\mathbf{v}}_3-⟨{\mathbf{v}}_3,{\mathbf{w}}_1⟩{\mathbf{w}}_1-⟨{\mathbf{v}}_3,{\mathbf{w}}_2⟩{\mathbf{w}}_2& {\mathbf{w}}_3={\hat{\mathbf{u}}}_3\\ & ⋮& ⋮\\ {\mathbf{u}}_k& ={\mathbf{v}}_k-⟨{\mathbf{v}}_k,{\mathbf{w}}_1⟩{\mathbf{w}}_1-⟨{\mathbf{v}}_k,{\mathbf{w}}_2⟩{\mathbf{w}}_2-\cdots -⟨{\mathbf{v}}_k,{\mathbf{w}}_{k-1}⟩{\mathbf{w}}_{k-1}& {\mathbf{w}}_k={\hat{\mathbf{u}}}_k\end{array}$$

where $\hat{\mathbf{u}}$ represents the normalized vector $\mathbf{u}/\Vert \mathbf{u}\Vert$. Thus, $\{{\mathbf{w}}_i\}$ is an orthonormal set, and hence a linearly independent set. Since a linearly independent set of size $n=\dim V$ is a basis, $\{{\mathbf{w}}_i\}$ is a basis of $V$. ❏