Orthogonal Diagonalization

Math 303: Section 31

Dr. Janssen

Intro

\[ \def\R{{\mathbb R}} \def\P{{\mathbb P}} \def\B{{\mathcal B}} \def\C{{\mathcal C}} \def\S{{\mathcal S}} \def\b{{\mathbf{b}}} \def\a{{\mathbf{a}}} \def\c{{\mathbf{c}}} \def\x{{\mathbf{x}}} \def\y{{\mathbf{y}}} \def\u{{\mathbf{u}}} \def\v{{\mathbf{v}}} \def\w{{\mathbf{w}}} \def\z{{\mathbf{z}}} \def\e{{\mathbf{e}}} \def\r{{\mathbf{r}}} \def\M{{\mathcal{M}}} \DeclareMathOperator{\null}{Nul} \DeclareMathOperator{\span}{Span} \DeclareMathOperator{\dim}{dim} \DeclareMathOperator{\proj}{proj} \DeclareMathOperator{\row}{Row} \DeclareMathOperator{\col}{Col} \DeclareMathOperator{\trace}{trace} \newcommand{\set}[1]{\left\{ {#1} \right\}} \newcommand{\setof}[2]{{\left\{#1\,\colon\,#2\right\}}} \newcommand{\norm}[1]{{\left|\! \left| #1 \right| \! \right|}} \newcommand{\ip}[1]{{\left\langle #1 \right\rangle}} \newcommand{\tr}[1]{{{#1}^\textsf{T}}} \]

An \(n\times n\) matrix \(A\) is orthogonally diagonalizable if there is an orthogonal matrix \(P\) such that \(\tr{P} A P\) is a diagonal matrix.

Preview Activity 31.1

An \(n\times n\) matrix \(A\) is orthogonally diagonalizable if there is an orthogonal matrix \(P\) such that \(\tr{P} A P\) is a diagonal matrix.

Observation. If \(A\) is orthogonally diagonalizable, \(A = \tr{A}\).

\(A = PD\tr{P} = P \tr{D} \tr{P} = \tr{(\tr{P})} \tr{D} \tr{P} = \tr{\left(PD\tr{P}\right)} = \tr{A}\).

Activity 31.1

Let \(A\) be a symmetric \(n\times n\) matrix and let \(\x,\y \in \R^n\).

• Show that \(\tr{\x} A \y = \tr{(A\x)} \y\).
• Show that \((A\x)\cdot\y = \x\cdot(A\y)\).
• Show that the eigenvalues of a \(2\times 2\) symmetric matrix \(A = \left[\begin{matrix} a & b \\ c & d \end{matrix}\right]\) are real.

Theorem 1 Let \(A\) be an \(n\times n\) symmetric matrix with real entries. Then the eigenvalues of \(A\) are real.

To orthogonally diagonalize a matrix, it must be the case that eigenvectors corresponding to different eigenvalues are orthogonal. This is an important property and it would be useful to know when it happens.

Activity 31.2

Let \(A\) be a real symmetric matrix with eigenvalues \(\lambda_1\) and \(\lambda_2\) and corresponding eigenvectors \(\v_1\) and \(\v_2\), respectively.

• Use Activity 31.1 to show that \(\lambda_1 \v_1 \cdot \v_2 = \lambda_2 \v_1 \cdot \v_2\).
• Explain why the result of the previous part shows that \(\v_1\) and \(\v_2\) are orthogonal if \(\lambda_1 \ne \lambda_2\).

Theorem 2 If \(A\) is a real symmetric matrix, then eigenvectors corresponding to distinct eigenvalues are orthogonal.

Recall that the only matrices than can be orthogonally diagonalized are the symmetric matrices. Now we show that every real symmetric matrix can be orthogonally diagonalized, which completely characterizes the matrices that are orthogonally diagonalizable

Theorem 31.6

Let \(A\) be a real symmetric matrix. Then \(A\) is orthogonally diagonalizable.

Proof on pp. 552-554 of the book. Lengthy and ommitted.

The upshot is that a matrix is orthogonally diagonalizable if and only if it’s real and symmetric.

Spectral Theorem

Definition 1 The set of eigenvalues of a matrix A is called the spectrum of A.

Theorem 3 (The Spectral Theorem for Real Symmetric Matrices) Let \(A\) be an \(n\times n\) symmetric matrix with real entries. Then:

• \(A\) has \(n\) real eigenvalues (counting multiplicities).
• The dimension of each eigenspace of \(a\) is the multiplicity of the corresponding eigenvalue as a root of the characteristic polynomial.
• Eigenvectors corresponding to different eigenvalues are orthogonal.
• \(A\) is orthogonally diagonalizable.

So any real symmetric matrix is orthogonally diagonalizable. We have seen examples of the orthogonal diagonalization of \(n\times n\) real symmetric matrices with \(n\) distinct eigenvalues, but how do we orthogonally diagonalize a symmetric matrix having eigenvalues of multiplicity greater than 1? The next activity shows us the process.

Activity 31.3

Let \(A = \left[\begin{matrix} 4 & 2 & 2 \\ 2 & 4 & 2 \\ 2 & 2 & 4 \end{matrix}\right]\). The eigenvalues of \(A\) are 2 and 8, with eigenspaces of dimensions 2 and 1, respectively.

• Explain why \(A\) can be orthogonally diagonalized.
• Two linearly independent eigenvectors for \(A\) corresponding to the eigenvalue 2 are \(\v_1 = \left[\begin{matrix} -1 \\ 0 \\ 1 \end{matrix}\right]\) and \(\v_2 = \left[\begin{matrix} -1 \\ 1 \\ 0 \end{matrix}\right]\). Note that \(\v_1,\v_2\) are not orthogonal, so cannot be in an orthogonal basis of \(\R^3\) consisting of eigenvectors of \(A\). So find a set \(\set{\w_1, \w_2}\) of orthogonal eigenvectors of \(A\) so that \(\span \set{\w_1, \w_2} = \span \set{\v_1, \v_2}\).
• The vector \(\v_3 = \left[\begin{matrix} 1 \\ 1 \\ 1 \end{matrix}\right]\) is an eigenvector for \(A\) corresponding to the eigenvalue 8. What can you say about the relationship between the \(\w_i\)’s and \(\v_3\)?
• Find a matrix \(P\) that orthogonally diagonalizes \(A\). Verify your work.

The Spectral Decomposition of a Symmetric Matrix

The spectral theorem tells us that we can find an orthonormal basis \(\set{\u_1, \ldots, \u_n}\) of eigenvectors of \(A\).

Let \(A\u_i = \lambda_i \u_i\) for each \(1\le i\le n\), and put \(P = [\u_1 \ \u_2 \ \cdots \ \u_n]\). Then:

\[ \tr{P} = A P = P^{-1} A P = D, \]

where \(D\) is the diagonal matrix with \(\d_{ii} = \lambda_i\).

Since \(A = P D \tr{P}\), we see that

\[ A = [\u_1 \ \u_2 \ \cdots \ \u_n] D \left[\begin{matrix} \tr{\u_1} \\ \tr{\u_2} \\ \vdots \\ \tr{\u_n} \end{matrix}\right] = \sum \lambda_i \u_i \tr{\u_i}. \]

This is called a spectral decomposition of \(A\).

Theorem 31.8: Properties of the spectral decomposition

Let \(A\) be an \(n\times n\) symmetric matrix with real entries, and let \(\set{\u_1, \ldots, \u_n}\) be an orthonormal basis of eigenvectors of \(A\) with \(A\u_i = \lambda_i \u_i\) for each \(i\). Further, let \(P_i = \u_i \tr{\u_i}\). Then:

• \(A = \lambda_1 P_1 + \cdots + \lambda_n P_n\)
• \(P_i\) is symmetric
• \(P_i\) is rank 1
• \(P_i^2 = P_i\)
• \(P_i P_j = 0\) if \(i\ne j\)
• \(P_i \u_i = \u_i\)
• \(P_i \u_j = \mathbf{0}\) if \(i\ne j\)
• For any vector \(\v\) in \(\R^n\), \(P_i \v = \proj_{\span\set{\u_i}} \v\).

Consequence: any symmetric matrix can be written as the sum of symmetric rank 1 matrices. This kind of decomposition turns o ut to contain a lot of information about \(\tr{A} A\) for any matrix \(A\).

Activity 31.4

Let \(A = \left[\begin{matrix} 4 & 2 & 2 \\ 2 & 4 & 2 \\ 2 & 2 & 4 \end{matrix}\right]\). The eigenvalues of \(A\) are \(\lambda_1 = \lambda_2 = 2\) and \(\lambda_3 = 8\), with eigenspaces of dimensions 2 and 1, respectively.

A basis for \(E_8\) is given by \(\set{\tr{[ 1 \ 1 \ 1]}}\) and a basis for \(E_2\) is \(\set{\tr{[1 \ -1 \ 0]}, \tr{[1 \ 0 \ -1]}}\).

• Find orthonormal vectors \(\u_1, \u_2, \u_3\) corresponding to \(\lambda_1, \lambda_2, \lambda_3\), respectively.
• Compute \(\lambda_1 \u_1 \tr{\u_1}\).
• Compute \(\lambda_2 \u_2 \tr{\u_2}\).
• Compute \(\lambda_3 \u_3 \tr{\u_3}\).
• Verify that \(A = \lambda_1 \u_1 \tr{\u_1} + \lambda_2 \u_2 \tr{\u_2} + \lambda_3 \u_3 \tr{\u_3}\).