Positive Definite Matrix

Chia-Ping Chen

Professor
Department of Computer Science and Engineering
National Sun Yat-sen University

Linear Algebra

1/33
Positive Definite, Negative Definite, Indefinite

2/33
Prof. Chen Positive Definite Matrix
Multi-Variate Pure Quadratic Function

A multi-variate pure quadratic function is

n X
X n
f (x1 , . . . , xn ) = cij xi xj
i=1 j=1

It has only quadratic terms in multiple variables.

3/33
Prof. Chen Positive Definite Matrix
Minimization of Pure Quadratic Function
The function
ax 2 + 2bxy + cy 2
is minimized at x = y = 0 if

a ≥ 0, ac − b 2 ≥ 0

b2
a ≥ 0, ac − b 2 ≥ 0 ⇒ a ≥ 0, c − ≥0
!a
2b
⇒ ax 2 + 2bxy + cy 2 = a x 2 + xy + cy 2
a
!2
b2
!
b
=a x+ y + c− y2 ≥ 0
a a
4/33
Prof. Chen Positive Definite Matrix
Maximization
A function
ax 2 + 2bxy + cy 2
is maximized at x = y = 0 if

a ≤ 0, ac − b 2 ≥ 0

(−b)2
a ≤ 0, ac − b 2 ≥ 0 ⇒ − a ≥ 0, (−c) − ≥0
(−a)

⇒ (−a)x 2 + (−2b)xy + (−c)y 2 is minimized at x = y = 0

⇒ ax 2 + 2bxy + cy 2 is maximized at x = y = 0
5/33
Prof. Chen Positive Definite Matrix
Pure Quadratic Function and Matrix

A multi-variate pure quadratic function can be represented by

a symmetric matrix.

n X
X n
f (x) = cij xi xj
i=1 j=1

Let
1
aij = (cij + cji ), A = {aij }
2
Then
A = AT
and
f (x) = xT Ax
6/33
Prof. Chen Positive Definite Matrix
Positive Definite Matrix

A symmetric matrix A is positive definite if

xT Ax > 0, ∀ x 6= 0

A is positive semi-definite if

xT Ax ≥ 0, ∀ x

7/33
Prof. Chen Positive Definite Matrix
Conditions for Positive Definite Matrix

A is positive definite if

A: (definition) xT Ax > 0, ∀ x 6= 0
B: it has only positive eigenvalues
C: it has only principal sub-matrices of positive determinant
D: it has only positive pivots

8/33
Prof. Chen Positive Definite Matrix
A⇒B

A positive definite matrix has only positive eigenvalues.

Let x be an eigenvector of A with eigenvalue λ

Ax = λx

⇒ xT Ax = λ(xT x)
xT Ax
⇒ λ=
xT x
⇒ λ>0

9/33
Prof. Chen Positive Definite Matrix
B⇒A

A matrix with only positive eigenvalues is positive definite.

A = QΛQT

yT y
z }| { z }| {
⇒ xT Ax = xT Q Λ QT x
= yT Λy
λi yi2
X
=
i
> 0, ∀ x 6= 0

10/33
Prof. Chen Positive Definite Matrix
A⇒C

A positive definite matrix has only principal sub-matrices of

positive determinant.
h i
Consider xT = xkT 0T .
" #" #
T
h i Ak B xk
⇒ x Ax = xkT 0 T
BT C 0
= xkT Ak xk > 0, ∀ xk 6= 0

⇒ Ak is positive definite
⇒ all eigenvalues of Ak are positive
⇒ |Ak | = λk,1 × · · · × λk,k > 0

11/33
Prof. Chen Positive Definite Matrix
C⇒D
A matrix with only principal sub-matrices of positive
determinant has only positive pivots.

A = LDU

" # " #" #" # " #

Ak B L 0 Dk 0 Uk ∗ L D U ∗
⇒ = k = k k k
BT C ∗ ∗ 0 ∗ 0 ∗ ∗ ∗

⇒ Ak = Lk Dk Uk
⇒ |Ak | = |Dk | = d1 . . . dk
|Ak |
⇒ dk = > 0, k = 1, . . . , n
|Ak−1 |
12/33
Prof. Chen Positive Definite Matrix
D⇒A

A matrix with only positive pivots is positive definite.

A = LDLT

dii > 0 ⇒ LDLT = LD1/2 D1/2 LT = RT R, R = D1/2 LT

⇒ xT Ax = xT RT Rx = (Rx)T (Rx) = kRxk2 > 0

13/33
Prof. Chen Positive Definite Matrix
Example
 
2 −1 0
A = −1 2 −1
 

0 −1 2

xT Ax = 2x12 + 2x22 + 2x32 − 2x1 x2 − 2x2 x3

2 2
1 3 2 4
  
= 2 x1 − x2 + x2 − x3 + x32 > 0
√2 2 3 3
λ = 2, 2 ± 2
|A1 | = 2, |A2 | = 3, |A3 | = 4
1 − 12 0
   
1 0 0 2
 1 3
A = − 2 1 0 
 2  0

1 − 23 

0 − 23 1 4
3
0 0 1
14/33
Prof. Chen Positive Definite Matrix
Negative Definite Matrix

A symmetric matrix A is negative definite if

xT Ax < 0, ∀ x 6= 0

A is negative semi-definite if

xT Ax ≤ 0, ∀ x

15/33
Prof. Chen Positive Definite Matrix
Conditions for Negative Definite Matrix

A is negative definite if

NA: (definition) xT Ax < 0, ∀ x 6= 0

NB: it has only negative eigenvalues
NC: the determinants of its principal sub-matrices alternates
ND: it has only negative pivots

16/33
Prof. Chen Positive Definite Matrix
Local Approximation and Optimization

17/33
Prof. Chen Positive Definite Matrix
First-order Approximation

A multi-variate function can be approximated locally by

∂f
δxi = f (x) + ∇f T δx
X
f (x + δx) ≈ f (x) +
i ∂xi

where  ∂f 
∂x1
 .. 
∇f =

 . 

 
 
∂f
∂xn

is the gradient vector.

18/33
Prof. Chen Positive Definite Matrix
Second-order Approximation

The second-order local approximation to a multi-variate

function is
X ∂f 1 X X ∂ 2f
f (x + δx) ≈ f (x) + δxi + δxi δxj
i ∂xi 2 i j ∂xi ∂xj
1
= f (x) + ∇f T δx + δxT H δx
2
where
∂ 2f
( )
H=
∂xi ∂xj
is the Hessian matrix.

19/33
Prof. Chen Positive Definite Matrix
Approximation around a Stationary Point

x0 is a stationary point if

∇f (x0 ) = 0

Near a stationary point x0 , the first-order approximation is

f (x) ≈ f (x0 )

and the second-order approximation is

1
f (x) ≈ f (x0 ) + (x − x0 )T H(x0 )(x − x0 )
2

20/33
Prof. Chen Positive Definite Matrix
Minimum, Maximum, and Saddle Point

Let x0 be a stationary point of f (x).

If H(x0 ) is positive definite, x0 is a local minimum
If H(x0 ) is negative definite, x0 is a local maximum
If H(x0 ) is indefinite, x0 is a saddle point

21/33
Prof. Chen Positive Definite Matrix
Example

Approximate
f (x) = 2x 2 + 4xy + y 2
near x = 0.

" #
4 4
∇f (0) = 0, H(0) =
4 2
1
⇒ f (x) ≈ f (0) + xT H(0)x = 2x 2 + 4xy + y 2
2

22/33
Prof. Chen Positive Definite Matrix
Example

Approximate

F (x) = 7 + 2(x + y )2 − y sin y − x 3

near x = 0.

" #
4 4
∇f (0) = 0, H(0) =
4 2

1
⇒ F (x) ≈ F (0) + xT H(0)x = 7 + 2x 2 + 4xy + y 2
2

23/33
Prof. Chen Positive Definite Matrix
Singular Value Decomposition

24/33
Prof. Chen Positive Definite Matrix
Singular Values

The singular values of matrixA are the square roots of the

non-zero eigenvalues of AT A .

25/33
Prof. Chen Positive Definite Matrix
Positivity
Singular values are always positive.
 
AT A is positive semi-definite, so its eigenvalues are
non-negative.
By definition, the square root of a positive number is
positive, so A has only positive singular values

σi > 0

26/33
Prof. Chen Positive Definite Matrix
Number of Singular Values
A matrix of rank r has r singular values.
 
AT A and A have the same nullspace

AT Ax = 0 ⇔ xT AT Ax = 0 ⇔ Ax = 0
 
Let A be of order m × n. Then AT A has n non-negative
eigenvalues, n − r of which are 0s, since
 
dim N (A) = n − r = dim N AT A

So the number of positive eigenvalues is

n − (n − r ) = r

27/33
Prof. Chen Positive Definite Matrix
Singular Value Decomposition

The SVD of a matrix is

A = UΣVT
 
U is an orthogonal eigenvector matrix of AAT
 
V is an orthogonal eigenvector matrix of AT A
Σ is a rectangular matrix with singular values on the
leading diagonal positions, and 0s elsewhere.

28/33
Prof. Chen Positive Definite Matrix
Proof
 
Find orthonormal eigenvectors of AT A . Construct
"  #

V = v1 . . . vr 
 vr +1 . . . vn


Let u1 . . . ur be

Avj   AAT Avj Aλj vj

uj = ⇒ AAT uj = = = λj uj
σj σj σj
 
and let ur +1 . . . um be eigenvectors of AAT of eigenvalue 0.
Construct
"  #

U = u1 . . . ur 
 ur +1 . . . um


29/33
Prof. Chen Positive Definite Matrix
Consider UT AV with

(UT AV)ij = uTi Avj

For j = 1 . . . r

uTi Avj = uTi (σj uj ) = σj δij

For j = r + 1 . . . n
 
AT A vj ⇒ Avj = 0 ⇒ uTi Avj = 0

Thus
UT AV = Σ
That is
A = UΣVT
30/33
Prof. Chen Positive Definite Matrix
Example

Find SVD for " #

−1 1 0
A=
0 −1 1

The eigenvalues of
 
1 −1 0
T
A A = −1 2 −1
 

0 −1 1

are λ = 3, 1, 0, so the singular values of A are

√
σ1 = 3, σ2 = 1

31/33
Prof. Chen Positive Definite Matrix
  
Orthonormal eigenvectors of AT A are
     
1 −1 1
1   1   1  
v1 = √ −2 , v2 = √  0  , v3 = √ 1
6 1 2 1 3 1
 
Orthonormal eigenvectors of AAT are
" # " #
Av1 1 −1 Av2 1 1
u1 = =√ , u2 = =√
σ1 2 1 σ2 2 1

 1
− √26 √1

# "√ √
− √12 √1
" #
3 0 0  61 6
⇒ A = UΣVT = 2 − √ 0 √1 
√1 √1 0 1 0  2 2
2 2 √1 √1 √1
3 3 3

32/33
Prof. Chen Positive Definite Matrix
Application to Image Processing

The SVD of a matrix can be written as

A = UΣVT
= σ1 u1 v1T + · · · + σr ur vrT
= A1 + · · · + Ar

which can be used for data compression.

For a 1000 × 1000 image, SVD achieves 90% compression

when using 50 matrices of rank 1.

Data Compression via SVD

33/33
Prof. Chen Positive Definite Matrix