Chapter 2

Linear Algebra and matrix analysis

Contents

2.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.2 Range Space, Null Space and Matrix Rank . . . . . . . . . . . . . . . . . . . . . . . 2.3 Eigenvalue decompostion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
2.3.1 General matrices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

1 1 3
3

2.4 2.5 2.6 2.7 2.8

2.3.2

Singular Value Decomposition and Projection Operator Positive (Semi)Denite matrices . . . . . . . . . . . . . . Matrices with special structure . . . . . . . . . . . . . . . Matrix inversion Lemmas . . . . . . . . . . . . . . . . . . Systems of linear equations . . . . . . . . . . . . . . . . .
Inconsistent systems

Hermitian matrices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

6 10 11 12 12
12 13

2.8.1 2.8.2

Consistent systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

2.9 The multivariate normal distribution . . . . . . . . . . . . . . . . . . . . . . . . . . 14

2.9.1 2.9.2 2.9.3 2.9.4 Linear transformations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Diagonalising transformations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Quadratic forms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 15 15 15

Complex Normal Distribution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

2.1 Introduction
In many digital signal processing, problems are modelised under linear algebra equations. As a consequence, it is classical to manipulate expressions such as:

Y = AX + N
where

is the input signal matrix,

a matrix resulting from from the modelisation process,

the matrix

or noise vector, and as

the measured output of the system.

It is then important to introduce some classical matrix analysis tools to solve

AX = B.

Complementary,

is the matrix or noise vector, the multivariate normal distribution has to be introduced.

This chapter was redacted using mainly the references [1] and [2].

2.2 Range Space, Null Space and Matrix Rank

A matrix is signaled by a bold capitale symbol, for example

or

or

the

transpose

v.

Let

be a

and

conjugate

mn

matrix with complex-valued elements: operators and let

(.)

= (.)

, a vector is A Cmn ,

signaled by a bold minuscule: let

(.)T , (.) denote

respectively

. The determinant of a matrix is denoted by

|.|

or

det(.).

Denition 1.

The range space of

A,

also called the column space, is the subspace spanned by (i.e. all linear

combinaisons of ) the colums of

A: R(A) = Cm1 | = A
for

Cn1

(2.2.1)

The range of the space

is usually called the row space of

A.

CHAPTER 2.

LINEAR ALGEBRA AND MATRIX ANALYSIS

Denition 2.

The null space of

A,

also called kernel, is the following subspace :

N ( A ) = Cn 1 | A = 0

(2.2.2)

Denition 3.

Equivalent denitions of the rank of

follow

r
(i)

rank (A) A.
The latter number is by

is equal to the maximum number of linearly independent columns of

denition the dimension of the

R (A)

r = dim R (A)
(ii)

(2.2.3)

is equal to the maximum number of linearly independent rows of

A,
(2.2.4)

r = dim R AT = dim R AH
(iii)

is the dimension of the nonzero determinant of maximum size that can be built from the element of

A.

Denition 4.

is said to be:

Rank decient whenever Full column rank if Full row rank if Nonsingular if

r < min(m, n).

r = n m.

r = m n.

r = m = n.

Proposition 5. Premultiplication or postmultiplication of A by a nonsingular matrix does not change the rank of A. This directly follows from the denition of rank(A) because the aforementionned multiplications does not change the number of linearly independent columns (or rows) of A.
Then:

Proposition 6. Let

A Cmn and B Cnp be two conformable matrices of rank rA and rB , respectively.

rank (AB) min (rA , rB )

(2.2.5)

Proof.

Using the denition of the rank, the premultiplication of

independent columns of

B,

hence

rank (AB) rB .

Similarly, the post-multiplication of

increase the number of linearly independant columns of

AT ,

B by A can not increase the number of linearly A by B can not wich means that rank (AB) rA .

Proposition 7. Let A Cmn be given by

N

A=
k=1

H xk yk

where xk , yk Cm1 . Then,

rank (A) min(m, n)

Proof.

Since

can be written as

A= x1 xN

y1
. . .

yN

the results follows from 6

2.3.

EIGENVALUE DECOMPOSTION

Proposition 8. Let A Cmn with n m, let B Cnp , and let

rank (A) = n
(2.2.6)

then
rank (AB) = rank (B)
(2.2.7) submatrix, the postmultiplication of

Proof.

The asumption (2.2.6) implies that

which by

gives a block of rank equal to

A contains a nonsingular n n rank (B) (from R1). Hence, rank (AB) rank (B)

However, by 6,

rank (AB) rank (B)

and hence (2.2.7) follows.

2.3 Eigenvalue decompostion

Denition 9.
to be A matrix

symetric.

A Cm m

is said

Hermitian

if

AH = A.

In the real-valued case, such an

is said

Denition 10.

A matrix

U Cm m

is said to be unitary (Orthogonal if

is real valued) whenever

UH U = UUH = I
If

U Cmn ,

with

m > n,

is such that

UH U = I

then we say that

is semi-unitary.

2.3.1

General matrices
A scalar

Denition 11.
of a matrix

AC

mm

C and a (nonzero) vector x Cm1 Ax = x

are an eigenvalue and its associated eigenvetor (2.3.1)

if

In paticular, an eigenvalue

is a solution of the so-called characteristic equation of

A:
(2.3.2)

|A I| = 0
and

is a vector in

N (A I).
p

The pair

(, x)

is called an eigenpair. then we can write the dening equation

{(i , xi )}i=1 are p eigenpairs of A (with p m) Axi = xi (i = 1, , p) in the following compact form:
Observe that if

AX = X
where

(2.3.3)

X = [x1 xp ]
and

1
. . .

0 p

Proposition 12. Let (, x) be an eigenpair of A Cmm . If B = A + I, with C, then ( + , x) is an

eigenpair of B. Proof.
Obvious

eigenvalues. B is said to be related to A by a similarity transformation.

Proposition 13. The matrices

A and B

Q1 AQ, where Q is any nonsingular matrix, share the same

Proof. |B I| = Q1 (A I) Q = Q1 |A I| |Q| = 0 is equivalent to |A I| = 0.

Denition 14.

The trace of a square matrix

A Cmm tr (A) =

is dened as

Aii
i=1

(2.3.4)

CHAPTER 2.

LINEAR ALGEBRA AND MATRIX ANALYSIS

m m Proposition 15. If {i }m then i=1 are the eigenvalues of A C m

tr (A) =
i=1

(2.3.5)

Interestingly, while the matrix product is not commutative, the trace is invariant to commuting the factors in a matrix product.

Proposition 16. Let A Cmn and B Cnm , then

tr (AB) = tr (BA)
(2.3.6)

Proof.

A straightforward calculation based on the trace denition and diagonal elements of a matrices product.

Lemma 17. Let A, B Cmm and let C. Then

|AB| = |A| |B| |A| = m |A|

Proposition 18. Let A Cmn and B Cnm . Then:

|I AB| = |I BA| .
(2.3.7)

Proof.

we can write the matrices:

I A 0 I I B I A B I 0 I

I A B I I A B I

I 0 B I I A 0 I

I AB 0 I 0

0 I

(2.3.8)

0 I BA

(2.3.9)

As matrices of left-hand side of (2.3.8) and (2.3.9) have the same determinant (see lemma 17), equal to , then the right-hand side matrices also have the same determinant.

Proposition 19. Let A Cmn and B Cnm , the non-zero eigenvalues of AB and BA are identical.
Proof.
Let

=0

be an eigenvalue of

AB,

then

0 = |AB I| = m |AB/ I| = m |BA/ I| = mn |BA I|

This result obtained thanks to proposition (18), then we can conclude that

is also an eigenvalue of BA.

2.3.2

Hermitian matrices

An important property of the class of Hermitian matrices, which does not necessarily hold for general matrices, is the following.

(ii) The m eigenvectors of A = A Cmm form an orthogonal set. In other words, the matrices whose columns are the eigenvectors of A is unitary.
It follows from (i) and (ii) and from equation (2.3.3) that for an Hermitian matrix we can write:

Proposition 20. (i) All eigenvalues of A = A Cmm are real valued.

AU = U
where

UU = U U = I

and the diagonal elements of

are

real numbers.

Equivalently:

A = UU
which is so-called Eigen-Value Decomposition (EVD) of

(2.3.10)

A = A .

The EVD of a Hermitian matrix is a

special case of the Singular Value Decomposition (SVD) of a general matrix discussed in next section. Another usefull result, associated with Hermitian matrices is the following:

2.3.

EIGENVALUE DECOMPOSTION

Proposition 21. Let A = A Cmm and let v Cm1

in a decreasing order: then:
m

(v = 0). Also, let the eigenvalues of A be arranged

1 2 m

v Av 1 v v

(2.3.11)

This ratio (2.3.11) is called the Rayleigh quotient. As this ratio (equation 2.3.11) is invariant to the multiplication of v by any non-null complex number, we can rewrite equation (2.3.11) in the form:
m v Av 1 for any v Cm1 with v v = 1
(2.3.12)

The equalities in equation (2.3.12) are evidently achieved when v is equal to the eigenvectors of A associated with m and 1 respectively. Proof.
Let the EVD of

be given by (2.3.10), and let

w = U v =
then we need to prove that

w1
. . .

wm
m

m w w =
k=1
for any

k |wk | 1

w Cm1

satisfying

w w =
k=1
However, this is readily verifyed as follows:

|wk | = 1

1
k=1
and

k |wk | =
k=1 m

(1 k ) |wk | 0

k |wk | m =
k=1
that concludes the proof. The following result extands the previous proposition.

(k m ) |wk | 0
k=1

Proposition 22. Let

C
mm

V Cmn , with m > n, be a semi-unitary matrix (i.e. V V = I), and let A = A have its eigenvalues ordered (1 2 m ) then:
m n

k tr VH AV
k=mn+1 k=1

(2.3.13)

where the equalities are achieved, for instance, when the columns of V are the eigenvectors of A corresponding to (mn+1 , . . . , m ) and, respectively, to (1 , . . . , n ). The rato
tr (V AV) tr (V AV) = tr (V V) n

is somtimes called the extended Rayleigh quotient. Proof.

Let, with (2.3.10)

A = UUH S = UH V sH 1
. . .

and let

(m n)

sH m

CHAPTER 2.

LINEAR ALGEBRA AND MATRIX ANALYSIS

(hence

sH k

is the

k th

row of

S).

By making use of the above notation, we can write:

tr V AV = tr V UU V = tr S S = tr SS
where

=
k=1

k ck

(2.3.14)

ck
obvioulsy

sH k sk , k = 1, . . . , m ck 0 k = 1, . . . , m

(2.3.15)

(2.3.16)

and

ck = tr SSH = tr SH S = tr VH UUH V = tr VH V = tr (I) = n

k=1
Futhermore,

(2.3.17)

ck 1 k = 1, . . . , m
To see this, let row of

(2.3.18)

G Cm(mn)

such that the matrix

[SG]

is a unitary matrix, and let

H gk

denote the

k th

G.

Then, by construction,

sH k
which is (2.3.18).

H gk

sk gk

H H = ck + gk gk = 1 = ck = 1 gk gk 1

Finally, by combining (2.3.14) with (2.3.16)-(2.3.18) we can readily verify that where the equality was achieved for

tr VH AV

satises (2.3.13),

c1 = = cmn = 0; cmn+1 = = cm = 1
and respectively

c1 = = cn = 1; cn+1 = = cm = 0
These conditions on

{ck }

are satised if, for example,

is equal to

and

, respectively.

With this observation, the proof is concluded. Proposition (21) is clearly a special case of Proposition (22).

2.4 Singular Value Decomposition and Projection Operator

Applications: applications which employ the SVD include computing the pseudoinverse, least squares tting of data, matrix approximation, and determining the rank, range and null space of a matrix.
For any matrix

mn

A Cm n

there exist unitary matrices

U Cmm

and

V Cnn

and a diagonal matrix

with non-negative diagonal elements, such that

A = UVH
By appropriate permutation, the diagonal element of

(2.4.1)

can be arranged in a decreasing order:

1 2 min(m,n)
The factorisation (2.4.1) is called the (2.4.1) satisfy:

Singular Value Decomposition

SVD) of A and its existence is

a signicant result for both a theoretical and practical standpoint. We reiterate that the matrices in equation

UH U = UUH = I (m m) VH V = VVH = I (n n) ij = i 0 0
for for

i=j i=j

The following terminology is most commonly associated with the SVD:

the left singular vectors of the matrix

are the columns of

U.

These singular vectors are also the eigenvectors of

AAH .

2.4.

SINGULAR V ALUE DECOMPOSITION AND PROJECTION OPERATOR

The right singular vectors of the matrix

Aare

the columns of

V.

These singular vectors are also the eigenvectors of

AH A. A
are the diagonal elements

The singular values of largest

{i }

of

Note that

{i }

are the square roots of the

min(m, n)

eigenvalues of

AAH

or

AH A.
where

The singular triple of columns of

is the triple

(k , uk , vk )

uk

is the

k th

columns of

and

vk

is the

k th

V.

If

rank (A) = r min (m, n)

then it can be shown that:

k > 0, k = 0,
Hence, for a matrix of rank

k = 1, . . . , r k = r, . . . , min (m, n)

the SVD can be written as:

A=
where

U1
r

U2
mr

1 0

0 0

H V1 H V2

}r H = U1 1 V1 }n r

(2.4.2)

1 Rrr

is non-singular. The factorisation of

A in (2.4.11) has a number of important consequences.

Proposition 23. Consider the SVD of A Cmn in (2.4.11) where r min(m, n). Then:
(i) U1 is an orthogonal basis of R(A) (ii) U2 is an orthogonal basis of N (AH ) (iii) V1 is an orthogonal basis of R(AH ) (iv) V2 is an orthogonal basis of N (AH ).

Proof.

We can see that (iii) and (iv) follow from the properties (i) and (ii) applied to

AH .

To prove (i) and

(ii), we need to show that:

R(A) = R (U1 )
and respectively,

(2.4.3)

N (AH ) = N (U2 )
To show (2.4.3), note that

(2.4.4)

R (A) there
Then

exists

such that

= A

H = U1 1 V1 = U1 and so R (U1 ). Then we show that R (A) R (U1 ) . 1 Now, for R (U1 ), it exists a such that = U1 . From (2.4.11), it follows that U1 = AV1 and 1 = A V1 = A and then R (A), that is to say R (U1 ) R (A). But we had also R (A) R (U1 ), then R (A) = R (U1 ).
Same philosophy for the demonstration of (2.4.4):

1 H H H N A H A H = 0 V 1 1 U H 1 = 0 1 V1 V1 1 U1 = 0 U1 = 0
Now, any vector

can be written as

=
since

U1

U2

H H 0 = UH 1 = U1 U1 + U1 U2 = , so = 0, and thus = U2 . We can then conclude that N A N (U2 ). H H H Now, N (U2 ) there exists such that = U2 , then A = V1 1 U1 U2 = 0 N A

U1

U2

is non singular.

However,

which leads to (2.4.4)

The previous results, readily derived by using the SVD, has a number of intersting corollaries which complement the discussion on the range and nullspace in section 2.2.

CHAPTER 2.

LINEAR ALGEBRA AND MATRIX ANALYSIS

A Cmn the subspaces R(A) and N (AH ) are orthogonal to each other and they together span C . Consequently, we say that N (AH ) is the orthogonal complement of R(A) in Cm , and vice
m

Proposition 24. For any

versa. In particular we have:

dim N (AH ) = m r

(2.4.5)

dim R(A) = r

(2.4.6)

(recall that dim R(A) = dim R(AH ) = r. Proof.

direct corallary of 23.

The SVD of a matrix also provides a conveniant representation for the projectors onto the range and null space of

and

AH .
Let

Denition 25.
matrix (i)

y Cm1
that:

be an arbitrary vector. By denition the orthogonal projector onto

R(A)

is the

, which is such R(A) = R(),

(ii) the Euclidean distance between

and

y R(A)
2

is minimum :

y y
Hereafter,

= min

over

R(A)

= xH x

denotes the

Euclidean vector norm.

Proposition 26. Let A Cmn . The orthogonal projector onto R(A) is given by:
= U1 UH 1
(2.4.7)

whereas the orthogonal projector on N (AH ) is

H = I U1 UH 1 = U 2 U2
(2.4.8)

Proof.

Let

y Cm 1

be an arbitrary vector. As

R(A) = R(U1 ),

according to 23, we can nd the vector

R(A)

that is of minimal distance from

by solving the problem:

min y U1

Because

(2.4.9)

y U1

= =

H y H U1 UH 1 y
2

H UH I U 1 UH 1 y +y 1 y

+ UH 2 y

If readily follows that the solution to the minimization problem (2.4.9) is given by

H vector U1 U1 y is the orthognal projection of

This prove (2.4.7). Then (2.4.8) follows

= UH 1 y. Hence the y onto R (A) and the minimum distance from y to R (A) is UH 2 y . H immediately from (2.4.7) and the fact that N A = R (U2 ). y
onto

Note, for instance, that for projection of

R (A)

the error vector is

H y U1 UH 1 y = U2 U2 y

which is in

R ( U2 )

and is therefore orthogonal to

R(A)

by 23. For this reason,

is given the name orthogonal projector

in 25 and in 26. As an aside, we remark that the orthogonal projector in (2.4.7) and (2.4.8) are next denition.

idempotent matrices ;

see

Denition 27.

The matrix

A Cmm

is

idempotent if
A2 = A
(2.4.10)

Furthermore, observe by making use of 19 that the idempotent matrix in (2.4.7), for example, has values equal to

eigen-

and

(m 1)

eigenvalues equal to zero.

This is a general property of idempotent matrices:

their eigenvalues are either zero or one. Finally, we present a result that even alone would be enough to make the SVD an essential matrix analysis tool.

2.4.

SINGULAR V ALUE DECOMPOSITION AND PROJECTION OPERATOR

decreasing order) be given by:

A=

Proposition 28. Let

A Cmn , with elements Aij . Let the SVD of A(with eigenvalues arranged in a U1
p

U2
mp

1 0

0 2

H V1 H V2

}p H = U 1 1 V 1 }n p

(2.4.11)

where p min (m, n) is an integer. Let

m n min(m,n)

= tr AH A =
i=1 j =1

|Aij | =
k=1

2 k

(2.4.12)

denote the square of the so-called Frobenius norm. Then the best rank-p approximant of A in the Frobenius norm metric, that is the solution of
min A B
B 2

subject to rank (B) = p

(2.4.13)

is given by

B0 = U1 1 V1

(2.4.14)

Futhermore, B0 above is the unique solution to the approximation problem (2.4.13) if and only if p > p+1 . Proof.
It follows from 8 and (2.8.5) that we can parametrise

in (2.4.13) as: (2.4.15)

B = CDH
where

C Cmp

and

D Cnp

are full column rank matrices. The previous parametrisation of

is of

course non-unique but, as we will see, this fact does not introduce any problem. By making use of (2.4.15) we can rewrite the problem (2.4.13) in the following form:

min A CDH
C,D

rank (C) = rank (D) = p

(2.4.16)

The reparametrised problem is essentially constraint free. Indeed, the full column rank condition that must be satised by

and

can be easilly handled, see bellow.

First we minimise (2.4.16) with respect to

D,

for a given

C.

To that end, observe that:

A CDH

= tr

D A H C CH C

CH C

DH CH C

CH A + AH I C CH C
H 1

CH A
(2.4.17)

By result (iii) in denition 29 in the next section, the matrix is positive semidenite for any

DA C C C

C C

D.

This observation implies that (2.4.17) is minimised with

D CH C respect to D for:

CH A

D0 = AH C CH C

(2.4.18)

and the corresponding minimum value of (2.4.17) is given by

tr AH I C CH C
Next we minimise (2.4.19) with respect to

CH A
denote an orthogonal basis of

(2.4.19)

C.

Let

S Cm p

R (C);

that is,

S S=I

and

S = C
for some non singuliar

pp

matrix

It is then straightforward to verify that

I C CH C

CH = I SSH

(2.4.20)

By combining (2.4.19) and (2.4.20) we can restate the problem of minimising (2.4.19) with respect to

as:

S; SH S=I

max tr SH AAH S S
is given by

(2.4.21)

The solution to (2.4.21) follows from 22: the maximising

S0 = U1
which yields

10

CHAPTER 2.

LINEAR ALGEBRA AND MATRIX ANALYSIS

C0 = U1 1
It follows that:

(2.4.22)

B0

= = =

H H C0 DH CH 0 = C0 C0 C0 0 A = S0 S0 A H H U1 UH U V + U V 1 1 1 2 2 2 1 H U 1 1 V 1

Futhermore, we observe that the minimum value of the Frobenius distance in (2.4.13) is given by

min(m,n)

A B0

H 2 U2 2 V2

=
k=p+1

2 k

If

p > p+1

Indeed, whenever corresponding to

B0 derived above is unique. Otherwise it is not unique. p = p+1 we can obtain B0 by using either the singular vectors associated with p or those p+1 , which will generally lead to dierent solutions.

then the best rank-p approximant

2.5 Positive (Semi)Denite matrices

Let

A = AH Cmm

be a Hermitian matrix, and let

{k }k=1

denotes its eigenvalues.

Denition 29.
(i) (ii)

We say that

is positive sem-denite (psd) or positive denite (pd) if any of the following

equivalent conditions hold true.

k 0 (k > 0 for pd) for k = 1, . . . , m. H A 0 (H A 0 for pd) for any (iii) There exists a matrix C such that

non-zero vector

Cm 1 .

A = CCH
with

(2.5.1)

rank (C) = m for pd) |A (i1 , . . . , ik )| 0 (> 0 for pd) for all k = 1, . . . , m and all indices i1 , . . . , ik [1, m], where A (i1 , . . . , ik ) is the submatrix formed from A by eliminating the i1 , . . . , ik rows and columns of A. (A (i1 , . . . , ik ) is called the principal submatrix of A). The condition for A to be positive denite can be simplied to requiring that |A (k + 1, . . . , m)| > 0 (for k = 1, . . . , m 1) and |A| > 0. (A (k + 1, . . . , m) is called a leading submatrix of A).
(iv) Of the previous dening conditions, (iv) is apparently more involved, and named the

Sylvester Criterion.

Denition 30.

Let

A = AH

be a positive semidenite matrix. The any matrix

that satises (2.5.2)

A = CCH
is called the square root of If

A.

Sometimes such a

is denoted

C = A1/2 . B.
There are then an innite number of

is a square root of

A,

then so is

CB

for any unitary matrix

square roots. Two often-used particular choices for square roots are: (i) Hermitian square root:

C = CH .

In this case we can simply write (2.5.2) as

A = C2 ,

and : (2.5.3)

C = U1/2 UH
because as

is Hermitian positive semi-denite,

A = U1/2 1/2 UH

= U1/2 UH

U1/2 UH

CC

The Hermitian square root is unique. (ii)

Cholesky factor.
A.
If

If

is lower triangular with nonnegative diagonal elements, then

is called the

Cholesky factor of

is positive denite, the Cholesky factor is unique. This decomposition is also called

the LU decomposition or Cholesky decomposition.

2.6.

MATRICES WITH SPECIAL STRUCTURE

11

2.6 Matrices with special structure

Denition 31.
A matrix

A Cmn

is called Vandermonde, if it has the following structure:

A=
where

1 z1
. . .

1 zn
. . .

(2.6.1)

m 1 z1

m1 zn

zk C

are usually assumed to be distinct. A matrix

Denition 32.

A Cmn

is called:

Toeplitz when Ai,j = Aij

These matrices are also called persymmetric.Symmetric Toeplitz matrices are both centrosymmetric and bisymmetric.

Hankel when Ai,j = Ai+j

A= a0 a1 a2
. . . . . .

Examples:
a1 a0 a1
.. .

a2 a1
.. .. .. . . .

.. .. .. . . .

..

a1n
. . . . . .

a1 a0 a1

a2 a1 a0

am1
is a Toeplitz matrix.

a1 a2

The discrete convolution operation can be constructed as a matrix multiplication, where one of the inputs is converted into a Toeplitz matrix. For example, the convolution of

and

can be formulated as:

m1

y (n) =
k=0
with

h(k n)x(k ) = H[mn] x[n1] 0 h0 h1

.. .

Hmn

h2 . . . hm2 = hm1 0 . . . . . . 0

h0 h1

0 h0
.. .

0 0
. . .

xn1 xn2 . . . . . . x 0

hm2 hm1 0
. . .

h1 hm2 hm1 0 0

0 h0

0 a A= b c

b c c d d e

d e f

hm1

is a Hankel matrix.

symmetric. More precisely, if J denotes the exchange (or reversal) matrix

J= 0

Proposition 33. The eigenvectors of a symmetric Toeplitz matrix A Rmm are either symmetric or skew. . .
0 1 0 0 1 0 1

0 1

0 0

and if x is an eigenvector of A, then either x = Jx or x = Jx .

12

CHAPTER 2.

LINEAR ALGEBRA AND MATRIX ANALYSIS

2.7 Matrix inversion Lemmas

appearing below exist,
A C D B

Proposition 34. Let A Cmm , B Cnn , C Cmn , D Cnm . Then, providing that the matrix inverses
= I 0 0 I A 1 I 0 + A 1 C I I B1 D B DA1 C
1

DA1

B1

A CB1 D

I CB1

By identication of terms involved in 34, we directly obtain:

Let A Cmm , B Cnn , C Cmn , D Cnm and providing that the matrix inverses appearing below exist,
A CB1 D
1

Lemma 35.

(Matrix Inversion Lemma)

= A1 + A1 C B DA1 C

DA1

2.8 Systems of linear equations

Let

A Cmn , B Cmp ,

and

X Cn p .

A general system of linear equations in

can be written as: (2.8.1)

AX = B
where matrices We say that:

and

are given and where

is the unknown matrix.

(2.8.1) exactly determined whenever (2.8.1) is overdetermined when

m = n,

m > n, m < n,

and (2.8.1) is underdetermined when

In the following, we rst examin the case where (2.8.1) has an exact solution and then the case where (2.8.1) can not be exactly satised.

2.8.1

Consistent systems

R (B) R (A) or equivalently

Proposition 36. The linear system (2.8.1) is consistent, that is it admits an exact solution X, if and only if
rank A B = rank (A)
(2.8.2)

Proposition 37. Let X0 be a particular solution to (2.8.1). Then the set of all solutions to (2.8.1) is given by:
X = X0 +
(2.8.3)

where C

np

is any matrix whose columns are in N (A).

Proposition 38. The system of linear equation (2.8.1) has a unique solution if and only if (2.8.2) holds and
A has a full column rank: rank (A) = n m
(2.8.4)

Proposition 39. Consider a linear system that satises the consistency condition in (2.8.2) . Let A have a rank r min (m, n), and let
A= U1
r

U2
mr

1 0

0 0

H V1 H V2

}r H = U 1 1 V 1 }n r

(2.8.5)

denote the SVD of A (1 is nonsingular), then:

1 H X0 = U1 1 V1 B
(2.8.6)

is the minimum Frobenius norm solution of (2.8.1) in the sense that

X0
2

(2.8.7)

for any solution X = X0 .

2.8.

SYSTEMS OF LINEAR EQUATIONS

13

Denition 40.

The matrix

1 A = U 1 1 V1

(2.8.8)

in (2.8.6) is the so-called It can be shown that

Moore-Penrose pseudo-inverse (or generalised inverse) of A.

is the unique solution to the following set of equations:

Denition 41.

AA A = A A AA = A A A and AA A
.

are Hermitian

Evidently, when inverse given to

is square and nonsingular we have

A 1 = A

which motivates the name of pseudo-

The computation of a solution to (2.8.1) is an important problem. This solution must be numerically stable. That is to say, the solution must be as unsensible as possible to perturbations (for example limited resolution of computer). In eect, what we can hope is to compute an exact solution to a slightly perturbated system of linear equations:

(A + A ) (X + X ) = (B + B )
One can show that the perturbations factor given by:

(2.8.9)

A B

in (2.8.9) are retrieved in

multiplied by a proportionality

cond (A) =
where

1 n A
and where

(2.8.10)

and

are respectively the largest and the smallest singular values of

cond(.)

means

condition number . If the corresponding ratio the problem is said  well-conditioned . If this
2.8.2 Inconsistent systems

introduced in (2.9.2) is small (not to much gretter than one), ratio is large, the problem is said  ill-conditioned .

The systems of linear equation that appear in applications are quite often perturbated and usually they do not admit any exact solution. Such systems are said incosistent and frequently they are overdetermined and have a matrix

that has full columns rank:

rank (A) = n m
linear equations:

(2.8.11)

In what follows, we present two approaches to obtain an approxiamte solution to an inconsistent system of

AX
under the condition (2.8.11).

(2.8.12)

Denition 42.

The least square (LS) approximate solution to (2.8.12) is given by the minimiser

XLS

of the

following criterion

AX B

Proposition 43. The LS solution to (2.8.12) is given by:

XLS = AH A
1

AH B

(2.8.13)

The inverse matrix in the above equation exists in view of (2.8.11). Remark.
Note that

XLS

should not be computed by directly evaluating (2.8.13) as it stand.

always the

condition number : cond (A) =

The reason is 1 and its eect on the perturbated systems. That is why the QR n

decomposition is prefered. For any matrix triangular matrix

A satisfying (2.8.11) R Cnn such that A=Q

there exist a unitary matrix

Q Cmm

and nonsingular upper-

R 0

Q1
n

Q2
mn

R 0

(2.8.14)

The previous factorisation of

is called the

QR decomposition .

Inserting (2.8.14) into (2.8.13) we obtain

XLS = R1 QH 1 B

14

CHAPTER 2.

LINEAR ALGEBRA AND MATRIX ANALYSIS

Hence, once the QR decomposition of

has been performed,

XLS

can be conveniently obtained as the

solution of the triangular system of linear equations:

RXLS = QH 1 B

(2.8.15)

2.9 The multivariate normal distribution

The multivariate normal distribution is the most important distribution in science and engineering. Let vector. The mean value of

denotes a random vector and

its realisation. Let

X = (X1 X2 Xn )

denote an

n1

random

is

m mi
So the mean vector

= E [X] = E (X1 X2 Xn ) = (m1 m2 mn ) = E [Xi ]

T

T
(2.9.1)

is vector of the means.

The covariance matrix for

is

R = E (X m) (X m) = {rij } rij = E [(Xi mi ) (Xj mj )]

A random vector

(2.9.2)

is said to be

multivariate normal

if its density function is

p (x) = (2 )
The characteristic function of
T

n/2

(det (R))

1/2

1 T exp (x m) R1 (x m) 2

(2.9.3)

is the multidimensional Fourier transform of the density function:

( )

= E e j X 1 n/2 1/2 T = dx (2 ) (det (R)) exp j T X (x m) R1 (x m) 2

(2.9.4)

and, after some calculus:

( )

= =

exp j T m j T R
1 exp 2 mT R1 m exp

1 + j R 1 m 2

R + j R1 m .

(2.9.5)

The characteristic function itself is a multivariate normal function of the frequency variable

2.9.1
Let

Linear transformations
be a linear transformation of a multivariate normal random variable:

Y = AT X AT : n m (m n)
thus,

(2.9.6)

is multivariate normal distributed as:

Y : N AT m, AT RA
a

(2.9.7)

m n, rank m. If m > n,
If

the requirement that then

AT RA

be nonsigular is the requirement that

be nonsingular and

have

AT RA

is singular, indicating that at least

mn

components of

are linearly dependent.

Remark.

Linear transformations allow to synthetise random vector for normaly distributed multivariate vector.

2.9.

THE MULTIVARIATE NORMAL DISTRIBUTION

15

2.9.2

Diagonalising transformations

The correlation matrix is symetric and nonnegative denite. Therefor there exists an orthogonal matrix that:

U such
(2.9.8)

2 UT RU = diag 2 1 n
It follows that the vector

Y=U X

is distributed as follows: (2.9.9)

2 Y = UT X : N UT m, UT RU = diag 2 1 n
We say that the random variables

Y1 , Y2 , . . . , Yn
T

are uncorrelated because:

E
Writting the density for

Y UT m Y,

Y UT m Yk

2 = UT RU = diag 2 1 n

(2.9.10)

we see that the

are independents normal random variables with means

UT m

and variances

2 k:
n

p (y) =
k=1

22 k

1/2

exp

1 y k UT m 22 k

2 k

(2.9.11)

This approach provides a possibility to generate independents random variables. The transformation

Y=

UT X

is called a

Karhunen-Loeve

transform.

2.9.3

Quadratic forms

Linear functions of multivariate normal vectors remain multivariate normal. But what about quadratic functions? In some very important cases the quadratic function have a Let

2 distribution

(read Chi-Squared).

denote a

N (m, R)

random variable. The distribution of the quadratic form

Q = (X m) R1 (X m)
is a chi-squared distribution with

(2.9.12) The density function for

degrees of freedom, denoted

2 n.

is

p(q ) =
2.9.4
Let

1 q (n/2)1 eq/2 ; q 0 (n/2) 2n/2

(2.9.13)

Complex Normal Distribution

and

be random vectors in

Rn

such that

vec [x y]

is a

2n-dimensional

normal random vector.

The

complex random vector

z = x + jy
has a

complex normal distribution .

This distribution can be described with 3 parameters:

= E [ z] R = E (z ) (z )
H T

C = E (z ) (z )
where

the location parameter (mean) vector the covariance matrix the relation matrix

can be an arbitrary

n-

dimensionnal complex random vector,

must be Hermitian,

should be symetric.

Moreover, the matrix

P = R CH R1 C
is also non-negative denite. The density function for a complex normal distribution can be computed as:

f (z) =

1 n det (R) det (P)

exp

1 H T [z ] [z ] 2

R CH

C R

[z ] [z ]

16

CHAPTER 2.

LINEAR ALGEBRA AND MATRIX ANALYSIS

A circular symetric complex normal distribution coresponds to the case of a null relation matrix If

C = 0.

z = x + jy

is circular complex normal, then the vector

vec [x y]

is multivariate normal with the following

structure:

x y

N
where .

() ()

1 2 2 ,

(R) (R) (R) ( R) E [(z ) (z )] = 0)

complex if

Remark.

In the case where

normal, centered (

i.e.

z = x + jy ,

z C. z

is said circular (i.e.

zero mean) with a variance equal to

and

are two random independant variable

with the same normal distribution

N 0, 2 /2

f (z ) =

zz 1 exp 2 2 2 2

1 |z | exp 2 2 2

Bibliography
[1] P. Stoica and R. L. Moses,

Introduction to spectral analysisPe. The Matrix Cookbook.

Prentice Hall, 1997.

[2] K. B. Petersen and M. S. Pedersen,

http://matrixcookbook.com, 2008.

17