CH 03

Chi-Kong Ng, SEEM3440/ESTR3500, Dept.
of SEEM, CUHK 3:1
Chapter 3.
Mathematical Background for Nonlinear

Programming
Chi-Kong Ng, SEEM3440/ESTR3500, Dept. of SEEM, CUHK 3:2
3.1. Sets
Sets of Real Numbers
Let a and b be two real numbers such that a < b; and let n be a finite
positive integer.
• {a, b} = the set that contains two elements “a” and “b”
• (a, b) = {x : a < x < b} • [a, b] = {x : a ≤ x ≤ b}
• (a, b] = {x : a < x ≤ b} • [a, b) = {x : a ≤ x < b}
• (a, +∞) = {x : x > a} • [a, +∞) = {x : x ≥ a}
• (−∞, a) = {x : x < a} • (−∞, a] = {x : x ≤ a}
• R = (−∞, +∞) = {x : −∞ < x < +∞}

Rn = [x1, . . . , xn]T : −∞ < xj < +∞, j = 1, . . . , n

•
Neighborhood
• The ε-neighborhood of a point x̄ ∈ Rn is the set of points inside
the ball with center x̄ and radius ε > 0:
Nε (x̄) = {x ∈ Rn : kx − x̄k < ε} ,
v
u n
uX
where kx − x̄k = t (xj − x̄j )2.
j=1
Open Sets
• A set S is called open if around every point in S there is a
neighborhood that is contained in S.
• Example 3.1. Rn, ∅, and the open interval (a, b) are open sets.
While (a, b] is not an open set.
Closed Sets
• A set S is called closed if its complement S ′ is an open set.
• Example 3.2. Rn, ∅, and the closed interval [a, b] are closed sets.
Interior of a Set
• The interior of any set S is the set of points x ∈ S which are the
center of some ball contained in S.
• Example 3.3. The open interval (a, b) is the interior of the sets
(a, b), [a, b], (a, b], [a, b), etc.
Bounded Sets
• A set is bounded if it can be contained in a ball of a sufficiently
large radius.
• Example 3.4. The open interval (a, b), and the closed interval [a, b]
are bounded sets.
Compact Sets
• A set is compact if it is both closed and bounded.
• Example 3.5. The closed interval [a, b] is a compact set.
Convex Sets
• A set S is convex if, for any elements x̄ and x̂ of S,
λx̄ + (1 − λ)x̂ ∈ S
for all values of λ such that 0 ≤ λ ≤ 1.
• Example 3.6. The set
S = {x ∈ Rn : Ax ≤ b, A ∈ Rm×n , b ∈ Rm }
is convex.
3.2. Differential Calculus

3.2.1. Differential Calculus of a Single Variable
Limits
• The equation
lim f (x) = d
x→c
means that as x gets closer to c (but not equal to c), the value of
f (x) gets arbitrarily close to d.
• It is also possible that lim f (x) may not exist.
x→c
1
• Example 3.7. lim does not exist.
x→0 x
Continuity
• A function f (x) is continuous at a point c if
lim f (x) = f (c).
x→c
• If f (x) is not continuous at c, we say that f (x) is discontinuous (or

has a discontinuity) at c.
Differentiation
• The derivative of f (x) at a point c is defined by

′ df (c) df d df
f (c) ≡ ≡ (c) ≡ f (c) ≡
dx dx dx dx x=c
f (c + ∆x) − f (c) f (x) − f (c)
= lim = lim
∆x→0 ∆x x→c x−c
provided the limit exists.
• When the limit exists, we say that f (x) is differentiable at c.
• Note also that if f (x) is differentiable at c, then f (x) is continuous
at c.
Derivatives of Polynomials
Let c be a constant.
d
• c=0
dx
d
• cx = c
dx
d c
• x = c xc−1 , if both xc and c xc−1 are defined
dx
Derivatives of Some Exponential and Logarithmic Functions
d x
• e = ex
dx
d x
• c = cx ln(c), where c > 0 is a constant
dx
d 1
• ln(x) = , if x > 0
dx x
d 1
• logc(x) = , where c is a constant with c > 0 and c 6= 1
dx x ln(c)
d x
• x = xx [1 + ln(x)], if x > 0
dx
Derivatives of Some Trigonometric Functions
d
• sin(x) = cos(x)
dx
d
• cos(x) = − sin(x)
dx
d 1
• tan(x) = sec2(x) =
dx cos2(x)
d d 1
• sec(x) = = sec(x) tan(x)
dx dx cos(x)
d d 1
• csc(x) = = − csc(x) cot(x)
dx dx sin(x)
d d 1 −1
• cot(x) = = − csc2(x) = 2
dx dx tan(x) sin (x)
Some Elementary Rules of Differentiation

d
• [af (x)+bg(x)] = af ′ (x)+bg ′(x), where a and b are constants
dx
d
• [f (x) g(x)] = f ′ (x) g(x) + f (x) g ′ (x)
dx
f ′ (x) g(x) − f (x) g ′ (x)

d f (x)
• = , if g(x) 6= 0
dx g(x) g(x)2
d df (g(x)) dg(x)
• f (g(x)) =
dx dg(x) dx
dy 1
• If y = f −1 (x), then = ′ , for f ′ (y) 6= 0
dx f (y)
′
d h i f (x)
• f (x)g(x) = f (x)g(x) g(x) + ln(f (x)) g ′ (x)
dx f (x)
Higher Derivatives
• Second Order Derivative:
We define (if it exists)
2 2 2 2

d f (c) d f d d f
f (2)(c) ≡ f ′′ (c) ≡ 2
≡ 2 (c) ≡ 2 f (c) ≡ 2

dx dx dx dx x=c
to be the derivative of the function f ′ (x) at the point c.
• kth-Order Derivative:
Similarly, we define (if it exists) f (k) (c) to be the derivative of
f (k−1) (x) at c.
Continuously Differentiable Functions

• f ∈ C 0(S)
= f is a real-valued continuous function in its domain S.
• f ∈ C 1(S)
= f is a real-valued continuously differentiable function in S
= f ′ exists and is continuous everywhere in S.
• f ∈ C k (S), where k = 1, 2, 3, . . .
= f is a real-valued kth-order continuously differentiable
function in S
= f (k) exists and is continuous everywhere in its domain S
Example 3.8.
(a) f (x) = |x| ∈ C 0(R).

0, if x < 0
(b) f (x) = ∈ C 0(R).
x, if x ≥ 0

0, if x < 0
(c) f (x) = 2 ∈ C 1(R).
x , if x ≥ 0

x, if x < 0
(d) f (x) = ∈ C 2(R).
sin(x), if x ≥ 0
(e) Polynomials, sin(x), cos(x) and exp(x) are examples of C ∞(R)

functions.
Taylor’s Theorem
• Given that f (k+1) (x) exists for every point on the interval [a, b]. Let
c ∈ [a, b]. For every x ∈ [a, b], there exists h(x) between c and x
with
f (x) = Pk (x) + Rk (x),
where
′ f ′′ (c)
Pk (x) = f (c) + f (c)(x − c) + (x − c)2
2!
(k)
f (c)
+ ···+ (x − c)k
k!
(k+1)
f (h(x))
Rk (x) = (x − c)k+1.
(k + 1)!
• Here Pk (x) is called the kth-order Taylor series expansion of f (x)

about c and Rk (x) is called the remainder term (or truncation error )
associated with Pk (x).
• The infinite series obtained by taking the limit of Pk (x) as k → +∞
is called the Taylor series of f (x) about c.
Example 3.9. The zero-, first-, second- and third-order Taylor series
expansion of f (x) = ex about c = 0 are:
20 20
15 f(x)=ex 15 f(x)=ex
10 10
5 5
P (x)=1
0
0 0
P1(x)=1+x
−5 −5
−3 −2 −1 0 1 2 3 −3 −2 −1 0 1 2 3
x x
20 20
15 f(x)=ex 15 f(x)=ex
10 10
5 5
P (x)=1+x+x2/2!
2
0 0
P (x)=1+x+x2/2!+x3/3!
3
−5 −5
−3 −2 −1 0 1 2 3 −3 −2 −1 0 1 2 3
x x
Example 3.10. Let f (x) = cos(x). For k = 3 and c = 0, we have

1 1
f (x) = 1 − x2 + x4 cos(h(x)),
2 24
where 0 < h(x) < x.
Therefore, the approximation f (0.1) = 0.995 is accurate to 5 decimal
places.
3.2.2. Differential Calculus of Several Variables

Partial Derivatives and the Gradient Vector
• The partial derivative of f (x) with respect to the variable xj is
∂f (x) ∂f ∂
≡ (x) ≡ f (x)
∂xj ∂xj ∂xj
f (x1, . . . , xj + ∆xj , . . . , xn) − f (x1, . . . , xj , . . . , xn)
= lim .
∆xj →0 ∆xj
• The gradient of f (x) is defined by:

 
∂f (x)
 ∂x1 

∇f (x) =  ... 

 
 ∂f (x) 
∂xn
Example 3.11. Consider the function

f (x) = x31 + 5x21x2 + 7x1x22 + 2x32
at the point
−2
c= .
3
The gradient of this function is
2
3x1 + 10x1x2 + 7x22

∇f (x) =
5x21 + 14x1x2 + 6x22
At the point c, this becomes

15
∇f (c) = .
−10
Second-Order Partial Derivatives and the Hessian Matrix

• We use the notation
∂ 2f (x) ∂ 2f ∂2
≡ (x) ≡ f (x)
∂xi ∂xj ∂xi ∂xj ∂xi ∂xj
to denote a second-order partial derivative of f (x).
• To find
∂ 2f (x)
,
∂xi ∂xj
we first find
∂f (x)
∂xj
and then take its partial derivative with respect to xi, i.e.,
∂ 2f (x)

∂ ∂f (x)
= .
∂xi ∂xj ∂xi ∂xj
• If the second-order partial derivatives exist and are everywhere

continuous, then
∂ 2f (x) ∂ 2f (x)
= .
∂xi ∂xj ∂xj ∂xi
• Given f ∈ C 2(Rn). The Hessian of f (x) is defined by
 2
∂ 2f (x)

∂ f (x)
 ∂x 2 · · · ∂x1 ∂xn 
1
2

∇ f (x) = 
 .
.
. ... 

2 2

 ∂ f (x) ∂ f (x) 
···
∂xn ∂x1 ∂xn 2
Note that the Hessian matrix of f (x) is symmetric.

f (x) = x31 + 5x21x2 + 7x1x22 + 2x32
at the point c = [−2, 3]T again. Its gradient is
2
3x1 + 10x1x2 + 7x22

∇f (x) = .
5x21 + 14x1x2 + 6x22
The Hessian matrix of this function is

6x1 + 10x2, 10x1 + 14x2
∇2f (x) = .
10x1 + 14x2, 14x1 + 12x2
At the point c, this becomes

18 22
∇2f (c) = .
22 8
Taylor’s Theorem
• Let
S = {x : aj ≤ xj ≤ bj , j = 1, 2, . . . , n}.
Let c ∈ S.
• If f ∈ C 2(S), then for every x ∈ S, the first-order Taylor series
expansion of f (x) about c is
P1(x) = f (c) + ∇f (c)T (x − c).
• If f ∈ C 3(S), then for every x ∈ S, the second-order Taylor series

expansion of f (x) about c is
1
P2(x) = f (c) + ∇f (c)T (x − c) + (x − c)T ∇2f (c)(x − c).
2

f (x) = x31 + 5x21x2 + 7x1x22 + 2x32
at the point c = [−2, 3]T again.
At the point c, we have

15 18 22
f (c) = −20, ∇f (c) = and ∇2f (c) = .
−10 22 8
The second-order Taylor series expansion of f (x) about c is

x1 + 2
P2(x) = −20 + 15, −10
x2 − 3

1 18 22 x1 + 2
+ x1 + 2, x2 − 3 .
2 22 8 x2 − 3
Since
f (−1.9, 3.2) = −19.755
P2(−1.9, 3.2) = −19.81
so the approximation is accurate to 3 digits.
3.3. Convex or Concave Functions

3.3.1. Convex or Concave Functions of a Single
Variable
Definitions
• A function of a single variable f (x) is a convex function if, for each
pair of distinct values of x, say x̄ and x̂, there holds
f (λx̄ + (1 − λ)x̂) ≤ λf (x̄) + (1 − λ)f (x̂)
for all values of λ such that 0 < λ < 1.
• It is a strictly convex function if ≤ can be replaced by <.
• f (x) is a concave function (or a strictly concave function) if −f (x)
is a convex function (or a strictly convex function).
• Or equivalently, it is a concave function (or a strictly concave

function) if the above condition holds when ≤ is replaced by ≥
(or by >).
Convexity Test for a Function of a Single Variable

Consider any function of a single variable f (x) that possesses a second
derivative at all possible values of x. Then f (x) is
• Convex if and only if f ′′ (x) ≥ 0 for all possible values of x;
• Strictly convex if f ′′ (x) > 0 for all possible values of x
(note: this is sufficient, but not necessary);
• Concave if and only if f ′′ (x) ≤ 0 for all possible values of x;
• Strictly concave if f ′′ (x) < 0 for all possible values of x
(note: this is sufficient, but not necessary).
Example 3.14.
(a) f (x) = x2 is strictly convex in R.
(b) f (x) = x4 is strictly convex in R.

(c) f (x) = x is both convex and concave in R, but not strictly convex
or strictly concave in R.
(d) f (x) = |x| is convex but not strictly convex in R.

Exercise.
Is the function f (x) = x3 convex or concave?
3.3.2. Convex or Concave Functions of Several

Variables
Definitions
• A function of several variables f (x) is a convex function if, for each
pair of distinct points, say
   
x̄1 x̂1
x̄ =  ..  and x̂ =  ... 
.
x̄n x̂n
there holds
f (λx̄ + (1 − λ)x̂) ≤ λf (x̄) + (1 − λ)f (x̂)
for all values of λ such that 0 < λ < 1.
• It is a strictly convex function if ≤ can be replaced by <.

• f (x) is a concave function (or a strictly concave function) if −f (x)
is a convex function (or a strictly convex function).
• Or equivalently, it is a concave function (or a strictly concave
function) if the above condition holds when ≤ is replaced by ≥
(or by >).
Convexity Test for a Function of Several Variables

Consider any function of several variables f (x) that possesses all
second-order partial derivatives at all possible values of x. Then f (x) is
• Convex if and only if ∇2f (x) is positive semi-definite for all possible
values of x;
• Strictly convex if ∇2f (x) is positive definite for all possible values
of x (note: this is sufficient, but not necessary);
• Concave if and only if ∇2f (x) is negative semi-definite for all
possible values of x;
• Strictly concave if ∇2f (x) is negative definite for all possible values
of x (note: this is sufficient, but not necessary).
3.4. Positive (Semi-)Definite, Negative

(Semi-)Definite, and Indefinite Real
Symmetric Matrices
3.4.1. Positive (Semi-)Definite Real Symmetric
Matrices
Definition
• An n × n real symmetric matrix A is called positive semi-definite if
v T Av ≥ 0, for all non-zero vectors v ∈ Rn.
• A is positive definite if ≥ can be replaced by >.

Eigenvalue Test
• A real symmetric matrix A is positive semi-definite if and only if
all its eigenvalues are non-negative, i.e., all the solutions, λ, to the
characteristic equation |A − λI| = 0 are real and non-negative.
• Similarly, A is positive definite if and only if all its eigenvalues are
real and positive.
• Remark: In particular, if A is a diagonal matrix, then its diagonal
elements are its eigenvalues.
Principal Minor Test

• A principal submatrix of an n×n matrix A is any m×m submatrix
obtained by deleting n − m rows of A and the corresponding
columns.
• A real symmetric matrix A is positive semi-definite if and only
if all its principal minors (i.e., determinants of all its principal
submatices) are non-negative.
a11 ≥ 0, · · · , ann ≥ 0;

a11 a12 a11 a1n an−1,n−1 an−1,n
≥ 0, · · · ≥ 0, · · · ≥ 0;
a21 a22 an1 ann an,n−1 an,n

a11 a12 a13 a11 a12 a1n an−2,n−2 an−2,n−1 an−2,n

a21
a22 a23 ≥ 0, · · · a21 a22 a2n ≥ 0, · · · an−1,n−2 an−1,n−1 an−1,n ≥ 0;
a31 a32 a33 an1 an2 ann an,n−2 an,n−1 an,n
..
.
a11 · · · a1n

. .. ≥ 0.
.. .

n1 · · · ann
a
• Similarly, A is positive definite if and only if all its principal minors

are positive.
Leading Principal Minor Test for Positive Definite Real

Symmetric Matrices
• A real symmetric matrix A is positive definite if and only if all its
leading principal minors are positive, that is

a11 · · · a1n
a a
a11 > 0, 11 12 > 0, · · · , ... ... > 0.

a21 a22 an1 · · · ann

Example 3.15. f (x) = (x1 − x2)2 is a convex function for all x ∈ R2.
3.4.2. Negative (Semi-)Definite Real Symmetric

Matrices
• An n×n real symmetric matrix A is said to be negative semi-definite
if the matrix −A is positive semi-definite.
• In other words, A is negative semi-definite if
v T Av ≤ 0, for all non-zero vectors v ∈ Rn.
• Similarly, A is negative definite if the matrix −A is positive definite.

• In other words, A is negative definite if ≤ can be replaced by < in
the above condition.
Example 3.16. f (x) = x1x2 − x21 − 2x22 is a strictly concave function for
all x ∈ R2.
3.4.3. Indefinite Real Symmetric Matrices

• An n × n real symmetric matrix A is said to be indefinite if the
matrix A is neither positive semi-definite nor negative semi-definite.
Example 3.17. f (x) = 3x21 +4x1x2 +x22 is a non-convex and non-concave
function for all x ∈ R2.
3.5. Optimization
General Optimization Problems
Min f (x)
s.t. x ∈ S ⊂ Rn.
Weierstrass Theorem
• A continuous function f (x) defined on a compact set S has a
minimum point in S.
Local Minimizers
• A point x∗ ∈ S is said to be a local minimizer of f over S if there
is an ε > 0 such that f (x) ≥ f (x∗) for all x ∈ S ∩ Nε (x∗), where
Nε (x∗) is the ε-neighborhood of x∗.
• If f (x) > f (x∗) for all x ∈ S ∩ Nε (x∗) and x 6= x∗, then x∗ is said
to be a strict local minimizer of f over S.
Global Minimizers
• A point x∗ ∈ S is said to be a global minimizer of f over S if
f (x) ≥ f (x∗) for all x ∈ S.
• If x∗ is a global minimizer and is a strict local minimizer of f over S,
then x∗ is said to be a strict global minimizer of f over S.
Remarks
• We always intend to seek a global minimizer when formulating an
optimization problem.
• In most situations, however, optimization theory and methodologies
only enable us to locate local minimizers.
• Global optimality can be achieved when
◦ the feasible region S is a convex set, and
◦ the objective function f (x) of a minimization problem is convex.
• When analytical solution is not achievable, numerical methods are
essential to the success of finding an optimum solution.
A General Scheme of an Iterative Solution Procedure

Step 1. Start from a feasible solution x ∈ S.
Step 2. Check if the stopping criteria (such as the optimality condi-
tions) are met. If the answer is YES, then stop. If the answer
is NO, then continue.
Step 3. Move to a better feasible solution and return to Step 2.

CH 03

Caricato da

Informazioni sul documento

Titolo originale

Copyright

Formati disponibili

Condividi questo documento

Condividi o incorpora il documento

Opzioni di condivisione

Hai trovato utile questo documento?

Questo contenuto è inappropriato?

Copyright:

Formati disponibili

CH 03

Caricato da

Copyright:

Formati disponibili

Chi-Kong Ng, SEEM3440/ESTR3500, Dept.

of SEEM, CUHK 3:1

Mathematical Background for Nonlinear

• R = (−∞, +∞) = {x : −∞ < x < +∞}

3.2. Differential Calculus

• If f (x) is not continuous at c, we say that f (x) is discontinuous (or

Some Elementary Rules of Differentiation

Continuously Differentiable Functions

(e) Polynomials, sin(x), cos(x) and exp(x) are examples of C ∞(R)

• Here Pk (x) is called the kth-order Taylor series expansion of f (x)

Example 3.10. Let f (x) = cos(x). For k = 3 and c = 0, we have

3.2.2. Differential Calculus of Several Variables

• The gradient of f (x) is defined by:

Example 3.11. Consider the function

Second-Order Partial Derivatives and the Hessian Matrix

• If the second-order partial derivatives exist and are everywhere

Example 3.12. Consider the function

• If f ∈ C 3(S), then for every x ∈ S, the second-order Taylor series

Example 3.13. Consider the function

3.3. Convex or Concave Functions

• Or equivalently, it is a concave function (or a strictly concave

Convexity Test for a Function of a Single Variable

(b) f (x) = x4 is strictly convex in R.

(d) f (x) = |x| is convex but not strictly convex in R.

3.3.2. Convex or Concave Functions of Several

• It is a strictly convex function if ≤ can be replaced by <.

Convexity Test for a Function of Several Variables

3.4. Positive (Semi-)Definite, Negative

• A is positive definite if ≥ can be replaced by >.

Principal Minor Test

• Similarly, A is positive definite if and only if all its principal minors

Leading Principal Minor Test for Positive Definite Real

3.4.2. Negative (Semi-)Definite Real Symmetric

• Similarly, A is negative definite if the matrix −A is positive definite.

3.4.3. Indefinite Real Symmetric Matrices

A General Scheme of an Iterative Solution Procedure

Potrebbero piacerti anche