Inner Product Spaces

Math 303: Section 29

Dr. Janssen

Intro

\[ \def\R{{\mathbb R}} \def\P{{\mathbb P}} \def\B{{\mathcal B}} \def\C{{\mathcal C}} \def\S{{\mathcal S}} \def\b{{\mathbf{b}}} \def\c{{\mathbf{c}}} \def\x{{\mathbf{x}}} \def\y{{\mathbf{y}}} \def\u{{\mathbf{u}}} \def\v{{\mathbf{v}}} \def\w{{\mathbf{w}}} \def\z{{\mathbf{z}}} \def\e{{\mathbf{e}}} \def\r{{\mathbf{r}}} \def\M{{\mathcal{M}}} \DeclareMathOperator{\null}{Nul} \DeclareMathOperator{\span}{Span} \DeclareMathOperator{\dim}{dim} \DeclareMathOperator{\proj}{proj} \DeclareMathOperator{\row}{Row} \DeclareMathOperator{\col}{Col} \DeclareMathOperator{\trace}{trace} \newcommand{\set}[1]{\left\{ {#1} \right\}} \newcommand{\setof}[2]{{\left\{#1\,\colon\,#2\right\}}} \newcommand{\norm}[1]{{\left|\! \left| #1 \right| \! \right|}} \newcommand{\ip}[1]{{\left\langle #1 \right\rangle}} \]

Definition

An inner product \(\ip{\ , \ }\) on a vector space \(V\) is a mapping \(V\times V\to \R\) satisfying:

• \(\ip{\u,\v} = \ip{\v,\u}\) for all \(\u,v\in V\)
• \(\ip{\u + \v, \w} = \ip{\u,\w} + \ip{\v,\w}\) for all \(\u,\v,\w\in V\)
• \(\ip{c\u, \v} = c\ip{\u,\v}\) for all \(\u,\v\in V\) and scalars \(c\)
• \(\ip{\u, \u} \ge 0\) for all \(\u\) in \(V\) and \(\ip{\u,\u} = 0\) if and only if \(\u = \mathbf{0}\)

A vector space \(V\) with an inner product \(\ip{\ , \ }\) defined on \(V\) is called an inner product space.

goal is to extend the dot product

Preview Activity 29.1

Inner Product Spaces

Examples

• If \(a_1, \ldots, a_n\) are positive scalars, then \(\ip{[u_1 \ u_2 \cdots \ u_n]^\textsf{T}, [v_1 \ v_2 \cdots \ v_n]^\textsf{T}} = a_1 u_1 v_1 + a_2 u_2 v_2 + \cdots + a_n u_n v_n\) defines an inner product on \(\R^n\)
• Every invertible \(n\times n\) matrix \(A\) defines an inner product on \(\R^n\) by \(\ip{\u, \v} = (A\u) \cdot (A\v)\).
• The definite integral defines an inner product: \(\ip{f,g} = \displaystyle\int_a^b f(x) g(x)\, dx\), for \(f,g\in C[a,b]\)
• The trace of an \(n\times n\) matrix defines an inner product on \(\M_{n\times n}\): \(\ip{A,B} = \trace\left(AB^\textsf{T}\right)\). This is called the Frobenius inner product.

Activity 29.1

Let \(\u\) be a vector in an inner product space \(V\).

• Why is \(\ip{\u, \mathbf{0}} = \ip{\u,\mathbf{0}} + \ip{\u, \mathbf{0}}\)?
• How does the equation in the previous part show that \(\ip{\u,\mathbf{0}} = 0\)?

Properties of inner products: Theorem 29.2

Let \(\ip{\ , \ }\) be an inner product on a vector space \(V\) and let \(\u,\v, \w\) be vectors in \(V\) and \(c\) a scalar. Then

• \(\ip{\mathbf{0},\v} = \ip{\v,\mathbf{0}} = 0\)
• \(\ip{\u,c\v} = c\ip{\u,\v}\)
• \(\ip{\u,\v+\w} = \ip{\u,\v} + \ip{\u,\w}\)
• \(\ip{\u-\v,\w} = \ip{\u,\w} - \ip{\v,\w}\)

Inner products allow us to extend ideas of orthogonality, length, and angles to any vector space.

Length of a Vector

Definitions 29.3–29.5

Let \(\v\) be a vector in an inner product space \(V\). The length¹ of \(\v\) is the real number

\[ \norm{\v} = \sqrt{\ip{\v,\v}}. \]

A vector \(\v\) in an inner product space is a unit vector if \(\norm{\v} = 1\).

If \(\u,\v\) are vectors in an inner product space \(V\), the distance between \(\u\) and \(\v\) is the length of the difference \(\u - \v\), \(\norm{\u - \v}\).

1. Also called magnitude or norm.

Activity 29.2

Find the length of the vectors \(\u = [1 \ 3]^\textsf{T}\) and \(\v = [3 \ 1]^\textsf{T}\) using the inner product

\[ \ip{[u_1 \ u_2]^\textsf{T}, [v_1 \ v_2]^\textsf{T}} = 2u_1 v_1 + 3 u_2 v_2 \]

in \(\R^2\).

Then, find the distance between the polynomials \(p(t) = t+1\) and \(q(t) = t^2 - 1\) in \(C[0,1]\) using the inner product \(\ip{f,g} = \int_0^1 f(x) g(x)\, dx\).

Orthogonality in inner product spaces

Generalizing orthogonality

Let \(\u,\v\) be nonzero vectors in an inner product space \(V\). The angle \(\theta\) between \(\u\) and \(\v\) is such that

\[ \cos(\theta) = \frac{\ip{\u,\v}}{\norm{\u}\ \norm{\v}} \]

and \(0\le \theta \le \pi\).

We say \(\u\) and \(\v\) are orthogonal if \(\ip{\u,\v} = 0\).

Activity 29.3

• Find a nonzero vector in \(\R^2\) orthogonal to the vector \(\u = [3 \ 1]^\textsf{T}\) using the inner product \(\ip{[u_1 \ u_2]^\textsf{T}, [v_1\ v_2]^\textsf{T}} = 2 u_1 v_1 + 3 u_2 v_2\).
• Determine if the vector \(\v = \left[\begin{matrix} 0 \\ 3 \\ -2 \end{matrix}\right]\) is orthogonal to the vector \(\w = \left[\begin{matrix} -1 \\ 0 \\ 1 \end{matrix}\right]\) using the inner product \(\ip{\u, \v} = (A\u)\cdot (A\v)\) on \(\R^3\), where \(A = \left[\begin{matrix} 0 & 1 & 1 \\ 1 & 1 & 0 \\ 0 & 1 & 0 \end{matrix}\right]\).
• Find the angle between the two polynomials \(p(t) = 1\) and \(q(t) =t\) in \(\P_1\), with inner product \(\ip{r(t), s(t)} = \int_0^1 r(t) s(t)\, dt\).

Orthogonal Sets

A subset \(S\) of an inner product space \(V\) for which \(\ip{\u,\v} = 0\) for all \(\u\ne \v\) in \(S\) is called an orthogonal set.

Theorem 1 Let \(\set{\v_1, \v_2, \ldots, \v_m}\) be a set of nonzero orthogonal vectors in an inner product space \(V\). Then the vectors \(\v_1, \v_2, \ldots, \v_m\) are linearly independent.

An orthogonal basis \(\mathcal{B}\) for a subspace \(W\) of an inner product space \(V\) is a basis of \(W\) that is also an orthogonal set.

Next slide: we can follow the example from \(\R^n\) to write the coordinates with respect to an orthogonal basis in an inner product space.

Coordinates in orthogonal bases

Theorem 2 Let \(\mathcal{B} = \set{\v_1, \v_2, \ldots, \v_m}\) be an orthogonal basis for a subspace of an inner product space \(V\). Let \(\x\) be a vector in \(W\). Then

\[ \x = \frac{\ip{\x, \v_1}}{\ip{\v_1, \v_1}} \v_1 + \frac{\ip{\x, \v_2}}{\ip{\v_2, \v_2}} \v_2 + \cdots + \frac{\ip{\x, \v_m}}{\ip{\v_m, \v_m}} \v_m. \]

Activity 29.4

Let \(p_1(t) = 1- t\), \(p_2(t) = -2 + 4t + 4t^2\), and \(p_3(t) = 7 - 41t +40t^2\) be vectors in the inner product space \(\P_2\) with inner product defined by \(\ip{p(t), q(t)} = \int_0^1 p(t) q(t)\, dt\). Let \(\B = \set{p_1(t), p_2(t), p_3(t)}\). You may assume that \(\B\) is an orthogonal basis for \(\P_2\). Let \(z(t) = 4 - 2t^2\). Find the weight \(x_3\) so that \(z(t) = x_1 p_1(t) + x_2 p_2(t) + x_3 p_3(t)\). Use technology as appropriate to evaluate any integrals.

If \(\ip{\v_k, \v_k} = 1\) for all \(k\), then things get simpler.

Looks like \(x_3 = - 5/54\)

Orthonormality

An orthonormal basis \(\B = \set{\v_1, \v_2, \ldots, \v_m}\) for a subspace \(W\) of an inner product space \(V\) is an orthogonal basis such that \(\norm{\v_k} = 1\) for \(1\le k\le m\).

Activity 29.5. Construct an orthonormal basis for the subspace

\[ W = \span \set{\left[\begin{matrix} 1 \\ 1 \\ 1 \\ 0 \end{matrix}\right], \left[\begin{matrix} -1 \\ 1 \\ -1 \\ 2 \end{matrix}\right], \left[\begin{matrix} 8 \\ 5 \\ -31 \\ -3 \end{matrix}\right]} \]

of the inner product space \(\R^4\) with inner product

\[ \ip{[u_1 \ u_2 \ u_3 \ u_4]^\textsf{T}, [v_1 \ v_2 \ v_3 \ v_4]^\textsf{T}} = 2 u_1 v_1 + 3 u_2 v_2 + u_3 v_3 + 5 u_4 v_4. \]

(Note that you need to check for orthogonality!)

Orthogonal Projections onto Subspaces

Preview Activity 29.2

Preview Activity 29.2 gives an indication of how we can project a vector \(\v\) in \(\R^n\) onto a subspace \(W\) of \(\R^n\). If we have an orthogonal basis for \(W\), we can just add the orthogonal projections of \(\v\) onto each basis vector.

Definition

Let \(W\) be a subspace of an inner product space \(V\) and let \(\B = \set{\w_1, \w_2, \ldots, \w_m}\) be an orthogonal basis for \(W\). For a vector \(\v\) in \(V\), the orthogonal projection of \(\v\) onto \(W\) is the vector

\[ \proj_W \v = \frac{\ip{\v,\w_1}}{\ip{\w_1,\w_1}}\w_1 + \frac{\ip{\v,\w_2}}{\ip{\w_2,\w_2}}\w_2 + \cdots + \frac{\ip{\v,\w_m}}{\ip{\w_m,\w_m}}\w_m. \]

The projection of \(\v\) orthogonal to \(W\) is the vector

\[ \proj_{W^\perp} \v = \v - \proj_W \v. \]

The notation \(\proj_{W^\perp} \v\) indicates that we expect this vector to be orthogonal to every vector in \(W\).

Activity 29.6

Let \(W = \span \set{\left[\begin{matrix} 1 \\ 1 \\ 1 \\ 0 \end{matrix}\right], \left[\begin{matrix} -1 \\ 1 \\ -1 \\ 2 \end{matrix}\right], \left[\begin{matrix} 8 \\ 5 \\ -31 \\ -3 \end{matrix}\right]}\) in \(\R^4\). Find the projection of the vector \(\v = [2 \ 0 \ 0 \ 1]^\textsf{T}\) onto \(W\) using the inner product

\[ \ip{[u_1 \ u_2 \ u_3 \ u_4]^\textsf{T}, [v_1 \ v_2 \ v_3 \ v_4]^\textsf{T}} = 2 u_1 v_1 + 3 u_2 v_2 + u_3 v_3 + 5 u_4 v_4. \]

Show directly that \(\proj_{W^\perp} \v\) is orthogonal to the basis vectors for \(W\).

Best approximations

In many situations we are interested in approximating a vector that is not in a subspace with a vector in the subspace (e.g., linear regression to fit a line to a set of data). In these cases we usually want to find the vector in the subspace that “best” approximates the given vector using a specified inner product. As we will soon see, the projection of a vector onto a subspace has the property that the projection is the “best” approximation over all vectors in the subspace in terms of the length.

In other words \(\proj_W \v\) is the vector in \(W\) that is closest to \(\v\) and therefore the best approximation of \(\v\) by a vector in \(W\).

To see that this is true in any inner product space, we first need a generalization of the Pythagorean Theorem that holds in inner product spaces.

GPT

Theorem 3 (Generalized Pythagorean Theorem) Let \(\u\) and \(\v\) be orthogonal vectors in an inner product space \(V\). Then

\[ \norm{\u - \v}^2 = \norm{\u}^2 + \norm{\v}^2. \]

\(\norm{u-v}^2 = \ip{\u - \v, \u-\v}\) and \(\ip{\u,\v} = 0\)

Theorem 29.14

Let \(W\) be a subspace of an inner product space \(V\) and let \(\u\) be a vector in \(V\). Then

\[ \norm{\u - \proj_W \u} < \norm{\u - \x} \]

for every vector \(\x\) in \(W\) different from \(\proj_W \u\).

Proof. Note that

\[ \u - \x = (\u - \proj_W \u) + (\proj_W \u - \x). \]

Since both \(\proj_W \u\) and \(\x\) are in \(W\), we know that \(\proj_W \u - \x\) is in \(W\). Since \(\proj_{W^\perp} \u = \u - \proj_W \u\) is orthogonal to everything in \(W\), we have that \(\u - \proj_W \u\) is orthogonal to \(\proj_W \u - \x\). By GPT,

\[ \norm{\u - \x}^2 = \norm{\u - \proj_W \u}^2 + \norm{\proj_W \u - \x}^2. \]

Since \(\x \ne \proj_W \u\), it follows that \(\norm{\proj_W \u - \x}^2 > 0\) and

\[ \norm{\u - \x}^2 > \norm{\u - \proj_W \u}^2. \]

Since norms are nonnegative, we can take the square root of both sides to reach out desired conclusion.

Least squares

In \(\R^n\) with the dot product, if \(\v = [v_1 \ v_2 \ \cdots \ v_n]^\textsf{T}\) and \(\proj_W \v = [w_1 \ w_2 \ \cdots \ w_n]^\textsf{T}\), then the square of the error in approximating \(\v\) by \(\proj_W \v\) is given by

\[ \norm{\v - \proj_W \v}^2 = \sum\limits_{i=1}^n (v_i - w_i)^2. \]

Thus \(\proj_W \v\) minimizes this sum of squares over all vectors in \(W\), and so we call it the least squares approximation to \(\v\).

Activity 29.7

The set \(\B = \set{1, t - \frac{1}{2}, t^3 - \frac{9}{10}t + \frac{1}{5}}\) is an orthogonal basis for a subspace \(W\) of the inner product space \(\P_3\) using the inner product \(\ip{p(t), q(t)} = \int_0^1 p(t) q(t)\, dt\).

Find the polynomial in \(W\) that is closest to the polynomial \(r(t) = t^2\) and give a numeric estimate of how good this approximation is.

Should get \(\frac{35}{54}t^3 + \frac{5}{12} t - \frac{1}{27}\).

Then \(\sqrt{\ip{t^2 - \left(\frac{35}{54}t^3 + \frac{5}{12} t - \frac{1}{27})}} = \frac{1}{36\sqrt{5}} \approx 0.01\).

Orthogonal Complements

Definition and a theorem

The orthogonal complement of a subset \(S\) of an inner product space \(V\) is the set

\[ S^\perp = \setof{\v\in V}{\ip{\v,\u} = 0 \text{ for all } \u\in S}. \]

As in \(\R^n\):

Theorem 4 Let \(\B = \set{\w_1, \ldots, \w_m}\) be a basis for a subspace \(W\) of an inner product space \(V\). A vector \(\v\) in \(V\) is orthogonal to every vector in \(W\) if and only if \(\v\) is orthogonal to every vector in \(\B\).

Activity 29.8

Consider \(\P_2\) with the inner product \(\ip{p(t), q(t)} = \int_0^1 p(t) q(t)\, dt\).

• Find \(\ip{p(t), 1-t}\) where \(p(t) = a + bt + ct^2\) is in \(\P_2\).
• Describe as best you can the orthogonal complement of \(\span \set{1-t}\) in \(\P_2\). Is \(p(t) = 1 - 2t - 2t^2\) in this orthogonal complement? Is \(p(t) = 1 + t - t^2\)?

Activity 29.9

Let \(V\) be an inner product space of dimension \(n\) and let \(W\) be a subspace of \(V\). Let \(\x\) be any vector in \(V\). We will demonstrate that \(\x\) can be written uniquely as a sum of a vector in \(W\) and a vector in \(W^\perp\).

• Explain why \(\proj_W \x\) is in \(W\) and \(\proj_{W^\perp} \x\) is in \(W^\perp\).
• Explain why \(\x\) can be written as a sum of vectors, one in \(W\) and one in \(W^\perp\).
• Suppose \(\x = \w + \w_1\) and \(\x = \u + \u_1\), where \(\w,\u\in W\) and \(\w_1, \u_1 \in W^\perp\). Show that \(\w = \u\) and \(\w_1 = \u_1\), so that the representation of \(\x\) as a sum of a vector in \(W\) and a vector in \(W^\perp\) is unique. [Hint: what is \(W\cap W^\perp\)?]

Summary: Theorem 29.17

Let \(V\) be a finite-dimensional inner product space and let \(W\) be a subspace of \(V\). Any vector in \(V\) can be written in a unique way as a sum of a vector in \(W\) and a vector in \(W^\perp\).

Theorem 29.17 is useful in many applications. For example, to compress an image using wavelets, we store the image as a collection of data, then rewrite the data using a succession of subspaces and their orthogonal complements. This new representation allows us to visualize the data in a way that compression is possible.