Lecture 3

Theory of Convexity

L03 - 1 Optimisation

Scope

• Convex (concave) functions

• Hessian matrix
• Necessary and sufficient conditions for convexity
• Convex regions

L03 - 2 Optimisation
 Let us examine a one-dimensional convex function

f(x)
b
a

xa xb x

Observation:
If we draw a line arbitrarily between two points on the convex
curve, all the values of this line lie above the curve.
How to express this observation mathematically?

L03 - 3 Optimisation

Definitions

• A function is called “convex” over the region R, if

f[2xa + (1-2)xb] # 2f(xa) + (1-2)f(xb) ∀xa , xb,R

Points on the curve Points on the straight line

Where 0 < 2 < 1 xa # x # xb

• If the “#” sign changes to “<”, then the function is strictly

convex

L03 - 4 Optimisation
 Similarly, we can define a concave function as follows

• A function is concave if it satisfies the following condition

f[2xa + (1-2)xb] $ 2f(xa) + (1-2)f(xb) ∀xa , xb,R

Points on the curve Points on the straight line

f(x)
b

xa xb x

• If the “$” changes to “>”, then the function is strictly

concave
L03 - 5 Optimisation

Questions

A: Can you draw two curves, one is concave and the

other is convex (not strict)?

B: What about strictly concave and convex?

C: What is the function which is both concave and

convex simultaneously?

D: Can you draw a non-convex curve?

L03 - 6 Optimisation
 Answers
A:
f(x) f(x)
Concave Convex
b b
a a

xa xb x xa xb x
B:
f(x) Strictly concave f(x) Strictly convex

xa xb x xa xb x

C: a straight line D: f(x)

xa xb x

L03 - 7 Optimisation

Why is it important to know whether a

function is convex?

L03 - 8 Optimisation
 Significance of Convexity

• A strictly convex or concave function provides a single

optimum

• A non-convex function may have multiple local optimum

• No method can guarantee a global optimum for a

general non-convex function

L03 - 9 Optimisation

L3-9

How to determine the convexity

for one-dimensional functions?

L03 - 10 Optimisation
 • Use the previous inequality Not convenient!
Let us try to use derivatives
e.g. f(x) = x2 x0R
f(x) fN(x) = 2x fO(x) = 2 > 0

0 X 0 X 0 X

fN(x) can be positive and ˆ f(x) is convex

negative

• fO(x) > 0 ] A strictly convex function

• If fO(x) $ 0 ] A convex function but not strict.

L03 - 11 Optimisation

Necessary and sufficient conditions

In general, use second derivative to determine convexity.

• fO(x) $ 0 x0R ] f(x) is convex

• fO(x) > 0 x0R ] f(x) is strictly convex

• fO(x) # 0 x0R ] f(x) is concave

• fO(x) < 0 x0R ] f(x) is strictly concave

• fO(x) are both negative and positive for x0R ] f(x) is indefinite

L03 - 12 Optimisation
 Examples:

For the following functions, determine if f(x) is convex,

concave, strictly concave, strictly convex, or indefinite in the
range of -4 # x # +4.

a) f(x) = 3x2
b) f(x) = 2x
c) f(x) = -5x2
d) f(x) = 2x2-x3

L03 - 13 Optimisation

Solution:

a) fO(x) = 6, always positive, hence f(x) is strictly convex

b) fO(x) = 0, for all values of x, hence f(x) is both convex and

concave (f(x) is a straight line)

c) fO(x) = -10, always negative, hence f(x) is strictly concave

d) fO(x) = 4 - 6x; can be positive or negative depending on the

value of x, hence f(x) is indefinite

L03 - 14 Optimisation
 So far, we have discussed one-dimensional functions.
How to determine the convexity (concavity) of a multi-
variable function, f(x) where x is a vector of variables?

L03 - 15 Optimisation

Hessian Matrix
• Use Hessian matrix to represent second derivative for a multi-variable
function
M ƒ(x) M ƒ(x) M ƒ(x) T
grad (x) = L ƒ(x) = M x1 M x2 M xn

M 2ƒ(x) M 2ƒ(x) M 2ƒ(x)

M x12 M x1 M x2 M x1 M xn

M 2ƒ(x) M 2ƒ(x) M 2ƒ(x)

H(x) = L ƒ(x) =
2
M x2 M x1 M x22 M x2 M x n

!
2
M ƒ(x) M 2ƒ(x) M 2ƒ(x)
M xn M x1 M xn M x 2 M xn2

• Note that H(x) is always symetric since M 2 f(x)/M xi M xj = M 2 f(x)/M xj M xi

L03 - 16 Optimisation
 Definitions

• H(x) is positive semidifinite if xTHx $ 0 for all x

• H(x) is negative semidifinite if xTHx # 0 for all x
• H(x) is positive definite if and only if xTHx > 0 for all x … 0
• H(x) is negative definite if and only if xTHx < 0 for all x … 0
• H(x) is indefinite if xTHx is negative for some x and positive
for other x

Note: xTHx is a scalar.

L03 - 17 Optimisation

Necessary and sufficient conditions for convexity of

multi-dimensional functions

• f(x) is concave if and only if H(x) is negative semidifinite.

• f(x) is strictly concave if H(x) is negative definite.

• f(x) is convex if and only if H(x) is positive semidifinite.

• f(x) is strictly convex if H(x) is positive definite.

L03 - 18 Optimisation
 Two convenient tests can be used to establish the
status of H(x) for strict convexity:

1) All diagonal elements of H(x) must be positive and the

determinant of all leading principal minors, det {Mi(H)},
are positive.

2) All the eigenvalues of H(x) are positive.

L03 - 19 Optimisation

Example 1: Determine the convexity of the following

function

f(x) = 2x12 - 3x1x2 + 2x22

Solution:
M f(x) M 2f(x) M 2f(x)
M x1 = 4x1 - 3x2 M x 12 = 4 M x 22 = 4
M f(x) M 2f(x) M 2f(x)
M x2 = -3x1 +4x2 M x1 M x2 = M x2 M x1 = -3

H(x) = 4 -3
-3 4

L03 - 20 Optimisation
 Solution (cont.)

The leading principal minors are:

M1 (order 1) = 4; det M1 = 4
M2 (order 2) = H; det M2 = 16 - 9 = 7

ˆ H(x) is positive definite Y f(x) is strictly convex

L03 - 21 Optimisation

Example 2: Determine the convexity of f(x)

f(x) = 2x12 + 2x1x2 + 1.5x22 + 7x1 + 8x2 + 24

Solution: Let’s use eigenvalues

M 2f(x) M 2f(x) M 2f(x) M 2f(x)

M x 12 = 4 M x22 = 3 M x1 M x2 = M x2 M x1 = 2

Hence: H(x) = 4 2
2 3

L03 - 22 Optimisation
 Solution (cont.)

To determine eigenvalues of H(x)

4-8 2
H - 83 =
2 3 -8
4-8 2
det = (4 - 8)(3 - 8) - 2 × 2 = 0
2 3 -8
82 - 7 8 + 8 = 0
81 = 5.56
ö Both positive ö f(x) is strictly convex
82 = 1.44

L03 - 23 Optimisation

Similarly, for strict concavity, two convenient tests can

be employed:

1) All diagonal elements of H(x) must be negative; and

• det (H) and det {Mi(H)}, are positive, if i is even;

• det (H) and det {Mi(H)}, are negative, if i is odd.

2) All the eigenvalues of H(x) are negative.

L03 - 24 Optimisation
 Example 3
Classify the following function as concave or convex.
f (x) = - 2 x12 - 4 x22 + 3 x1 x2 + 5 x2 - 6
Solution:
-4 3
H (x) =
3 -8

-4 -8 3
H - 8I =
3 -8 -8

-4 -8 3
det 3 -8 -8 =0 Y (-4 -8) (-8 -8) - 9 = 0

-2.4
8 + 12 8 + 23 = 0
2
Y 8=
Y function is strictly concave.
-9.6

L03 - 25 Optimisation

Summary

All the eigenvalues Determinants of the leading

f(x) is H(x) is principal minors
of H(x) are
of H; (∆i)

Strictly convex Positive definite >0 ∆1 > 0, ∆2 > 0, þ ∆n > 0

Convex Positive semidifinite $0 ∆1 $ 0, ∆2 $ 0, þ ∆n $ 0

Concave Negative semidifinite #0 ∆1 # 0, ∆2 $ 0, ∆3 # 0 þ

Strictly concave Negative definite <0 ∆1 < 0, ∆2 > 0, ∆3 < 0 þ

L03 - 26 Optimisation
 If a function is separable into

f(x) = f1(x) + f2(x)

• Then if f1(x) is convex and f2(x) is convex ö f(x) is convex

• If f1(x) is concave and f2(x) is concave ö f(x) is concave

• If f1(x) is concave and f2(x) is convex ö f(x) can not be refered

to as convex or concave

L03 - 27 Optimisation

Example

f(x) = a1(x1 - c1)2 + a2(x2 - c2)2 a1, a2 $ 0

where a1(x1 - c1)2 and a2(x2 - c2)2 are all convex.

Hence f(x) is convex

L03 - 28 Optimisation

26
 Convex Regions
• Examine two following regions:

X1 X1
R1 R2

X2 X2
Convex region Non convex region

• A convex set of points (convex region) exists if for any two

points in that region (e.g. xa and xb) all points on the line joining
these two points are in the set.

L03 - 29 Optimisation

• Mathematically,

R is a convex region if

"x1 + (1-")x2 ,R
where x1, x2 ,R and 0 # " # 1

• Theorem:

A region is convex if it is completely bounded by

either convex constraints in the form of gi(x) # 0
or concave constraints in the form of gi(x) $ 0

L03 - 30 Optimisation
 Example: A region is bounded by following constraints
g1(x): -x12 + x2 -1 $ 0
g2(x): x1 - x2 + 2 # 0
Is this region convex?

Solution:

First rearrange g1 (x): x12 - x2 + 1 # 0

g2(x)
x2
Now let’s plot the region:
4

2
g1(x)
1

-2 -1 0 1 2 x1

L03 - 31 Optimisation

g1(x) = x12 - x2 + 1 # 0

g2(x) = x1 - x2 + 2 # 0
2 0
H[g1] = 0 0 , all eigenvalues are non negative Y g1 is convex

0 0
H[g2] = , all eigenvalues are non negative Y g2 is convex
0 0

Y The region is convex.

L03 - 32 Optimisation
 Corollary: If a region is bounded by linear functions,
then it is a convex region

For example, X2

convex region

X1

L03 - 33 Optimisation

Convex Programming Problems

Minimise f(x) f(x) is convex

s.t. hj(x) = 0 j = 1,þ ,m hj(x) are linear
gj(x) # 0 j = m + 1, þ , p gj(x) are convex

• The local minimum is also the global minimum

Question: What is a general form for concave programming

problems?

L03 - 34 Optimisation