An orthogonal linear operator is one which preserves not only sums and scalar multiples, but dot products and other related metrical properties such as distances, lengths and angles. Since metrical properties can all be described in terms of dot products, we use the following definition.

A linear operator T on 2-space or on 3-space is called orthogonal if it preserves dot products, i.e. if T(u)•T(v) = u•v for all vectors u and v.

Let's look at the matrix of an orthogonal linear operator T on 2-space. Since i and j are orthonormal (i.e. they are orthogonal and have length 1),

i•j = 0,   i•i= 1   and   j•j = 1 ,

so

T(i)•T(j) = i•j = 0,   T(i)•T(i) = i•i = 1   and   T(j)•T(j) = j•j = 1.

Since T(i) and T(j) are the columns of the standard matrix of T, those columns must also be orthonormal, i.e. orthogonal and with length 1. The same holds in 3-space.

A 2x2 or 3x3 matrix whose columns are orthonormal vectors is called an orthogonal matrix.

Any orthogonal matrix is the standard matrix of an orthogonal transformation as well. Suppose we start with an orthogonal 2x2 matrix M and look at the corresponding linear operator T. The columns of M are T(i) and T(j), so T(i) and T(j) are orthonormal, i.e.

T(i)•T(j) = 0,   T(i)•T(i) = 1   and   T(j)•T(j) = 1.

Then for any vectors u = u1i + u2j and v = v2i + v2j,

T(u) = u1T(i) + u2T(j) and T(v) = v1T(i) + v2T(j),

so

T(u)•T(v) = [u1T(i) + u2T(j)]•[v1T(i) + v2T(j)]
               = u1v1T(i)•T(i) + u1v2T(i)•T(j) + u2v1T(j)•T(i) + u2v2T(j)•T(j)
               = u1v1 + u2v2
               = u•v.

So in general, a linear operator is orthogonal if and only if its standard matrix is orthogonal.

Suppose M is a 2x2 orthogonal matrix, say

.

Since the columns of M are orthonormal, a2 + b2 = 1, c2 + d2 = 1 and ac + bd = 0. Then

,

i.e. MTM = I. You can easily go backward from the equation MTM = I to show that any matrix that satisfies this equation must be orthogonal, so the condition MTM = I is equivalent to the condition that M is orthogonal. Another way of saying it: M is orthogonal if its transpose equals its inverse. But in that case, MMT = I as well, which says that the columns of MT are also orthonormal. Those columns are just the rows of M, so the rows of an orthogonal matrix are orthonormal too.

Again, all of this works just as well in 3-space, so the following are equivalent ways of saying a 2x2 or 3x3 matrix M is orthogonal:

  • M has orthonormal columns
  • M has orthonormal rows
  • MTM = I
  • MMT = I
  • MT = M-1.