The Moore-Penrose Pseudoinverse

(Math 33A(2): Laub)

In these notes we give a brief introduction to the Moore-Penrose pseudoinverse, a generalization of the inverse of a matrix. The Moore-Penrose pseudoinverse is dened for any matrix and, as is shown in the sequel, brings great notational and conceptual clarity to the study of solutions to arbitrary systems of linear equations and linear least squares problems.

Denition and Characterizations

We consider the case of A IRmn . It turns out that every A IRmn has a pseudoinverse r r nm and, moreover, the pseudoinverse, denoted A+ IRr , is unique. A purely algebraic characterization of A+ is given in the next theorem proved by Penrose in 1956. Theorem: Let A IRmn . Then G = A+ if and only if r (P1) AGA = A (P2) GAG = G (P3) (AG)T = AG (P4) (GA)T = GA Furthermore, A+ always exists and is unique. Note that the above theorem is not constructive. But it does provide a checkable criterion, i.e., given a matrix G that purports to be the pseudoinverse of A, one need simply verify the four Penrose conditions (P1)(P4) above. This verication is often relatively straightforward. Example: Consider A = inverses (for example, AL 1 . Verify directly that A+ = [ 1 , 2 ]. Note that other left 5 5 2 = [3 , 1]) satisfy properties (P1), (P2), and (P4) but not (P3).

Still another characterization of A+ is given in the following theorem whose proof can be found on p. 19 in Albert, A., Regression and the Moore-Penrose Pseudoinverse, Academic Press, New York, 1972. We refer to this as the limit denition of the pseudoinverse. Theorem: Let A IRmn . Then r A+ = lim (AT A + 2 I)
0 0 1

AT
1

(1) (2)

= lim AT (AAT + 2 I)

Examples

Each of the following can be derived or veried by using the above theorems or characterizations. Example 1: A+ = AT (AAT ) invertible) Example 2: A+ = (AT A) invertible)
1 1

if A is onto, i.e., has linearly independent rows (A is right

AT if A is 1-1, i.e., has linearly independent columns (A is left

Example 3: For any scalar , + = Example 4: For any vector v IRn , v + = (v T v) v T =

+ +

1 if = 0 0 if = 0

vT vT v

if v = 0 if v = 0

1 0 1 0 = 0 0 0 0 This example was computed via the limit denition of the pseudoinverse. + 1 1 1 1 Example 6: = 4 4 1 1 1 1 4 4 This example was computed via the limit denition of the pseudoinverse. Example 5:

Properties and Applications

Theorem: Let A IRmn and suppose U IRmm , V IRnn are orthogonal (M is orthogonal if M T = M 1 ). Then (U AV )+ = V T A+ U T . Proof: Simply verify that the expression above does indeed satisfy each of the four Penrose conditions. Theorem: Let S IRnn be symmetric with U T SU = D, where U is orthogonal and D is diagonal. Then S + = U D+ U T where D+ is again a diagonal matrix whose diagonal elements are determined according to Example 3. Theorem: For all A IRmn , 1. A+ = (AT A) AT = AT (AAT ) 2. (AT ) = (A+ )
+ T + +

Both of the above two results can be proved using the limit denition of the pseudoinverse. The proof of the rst result is not particularly easy nor does it have the virtue of being especially illuminating. The interested reader can consult the proof in Albert, p. 27. The proof of the second result is as follows: (AT )
+

= lim (AAT + 2 I)
0 0

A
1 T

= lim [AT (AAT + 2 I) = [lim AT (AAT + 2 I)

0

] ]

1 T

= (A+ )

Note now that by combining the last two theorems we can, in theory at least, compute the Moore-Penrose pseudoinverse of any matrix (since AAT and AT A are symmetric). Alternatively, we could compute the pseudoinverse (and this is the way its done in Matlab) by rst computing the SVD of A as A = U V T and then by the rst theorem of this S 1 0 subsection A+ = V + U T where + = . 0 0 The rst theorem is also suggestive of a reverse-order property for pseudoinverses as for inverses. Unfortunately, in general, (AB)+ = B + A+ . As an example consider A = [0 , 1] and B = 1 . Then 1

(AB)+ = 1+ = 1 while
1 B + A+ = [ 2 , 1 2]

0 1

1 2

However, necessary and sucient conditions under which the reverse-order property does hold are known and you can consult the literature for useful results. One particular result is worth mentioning here. Theorem: If A IRnr , B IRrm , then (AB)+ = B + A+ . r r Additional useful properties of pseudoinverses: 1. (A+ ) = A 2. (AT A) = A+ (AT ) , (AAT ) = (AT ) A+ 3. R(A+ ) = R(AT ) = R(A+ A) = R(AT A) 4. N (A+ ) = N (AA+ ) = N ((AAT ) ) = N (AAT ) = N (AT ) 3
+ + + + + +

5. If A is normal then Ak A+ = A+ Ak for all k > 0, and (Ak ) = (A+ ) for all k > 0. Note: Recall that A IRnn is normal if AAT = AT A. Thus if A is symmetric, skewsymmetric, or orthogonal, for example, it is normal. However, a matrix can be none of the preceding but still be normal such as A= 1 1 1 1 .

The next theorem is fundamental to using pseudoinverses eectively for studying the solution of systems of linear equations. Theorem: Suppose A IRnp , B IRnm . Then R(B) R(A) if and only if AA+ B = B. Proof: Suppose R(B) R(A) and take arbitrary x IRm . Then Bx R(B) R(A) so there exists a vector y IRp such that Ay = Bx. Then we have Bx = Ay = AA+ Ay = AA+ Bx where one of the Penrose properties is used above. Since x was arbitrary, we have shown that B = AA+ B. To prove the converse, assume now that AA+ B = B and take arbitrary y R(B). Then there exists a vector x IRm such that Bx = y, whereupon y = Bx = AA+ Bx R(A) .

EXERCISES: 1. Use the limit denition of the pseudoinverse to compute the pseudoinverse of 2. If x, y IRn , show that (xy T ) = (xT x) (y T y) yxT . 3. For A IRmn , prove that R(A) = R(AAT ) using only denitions and elementary properties of the Moore-Penrose pseudoinverse. 4. For A IRmn , prove that R(A+ ) = R(AT ). 5. For A IRpn and B IRmn show that N (A) N (B) if and only if BA+ A = B. 6. Let A IRnn , B IRnm , and D IRmm and suppose further that D is nonsingular. (a) Prove or disprove that A AB 0 D (b) Prove or disprove that A B 0 D
+ + + + +

1 1 . 2 2

A+ A+ ABD1 0 D1

A+ A+ BD1 0 D1