Optimization types & classes

CP -1A
M S Prasad
Optimization

1. Introduction to Optimization
2. Classical Optimization Techniques
3. Linear programming and the Simplex method
4. Nonlinear programming-One Dimensional Minimization Methods
5. Nonlinear programming-Unconstrained Optimization Techniques
6. Nonlinear programming-Constrained Optimization Techniques
7. Global Optimization Methods-Genetic algorithms
8. Global Optimization Methods-Simulated Annealing
9. Global Optimization Methods- Coupled Local Minimizers

2
Classification of optimization problems

Classification based on:

• Constraints
• Constrained optimization problem
• Unconstrained optimization problem

• Nature of the design variables

• Static optimization problems
• Dynamic optimization problems

3
Classification of optimization problems

Classification based on:

• Physical structure of the problem

• Optimal control problems
• Non-optimal control problems

• Nature of the equations involved

• Nonlinear programming problem
• Geometric programming problem
• Quadratic programming problem
• Linear programming problem

4
Classification of optimization problems

Classification based on:

• Permissable values of the design variables

• Integer programming problems
• Real valued programming problems

• Deterministic nature of the variables

• Stochastic programming problem
• Deterministic programming problem

5
Classification of optimization problems

Classification based on:

• Separability of the functions

• Separable programming problems
• Non-separable programming problems

• Number of the objective functions

• Single objective programming problem
• Multiobjective programming problem

6
Geometric Programming
• A geometric programming problem (GMP) is one in which the
objective function and constraints are expressed as posynomials in X.

7
8
Quadratic Programming Problem
• A quadratic programming problem is a nonlinear programming problem with a
quadratic objective function and linear constraints. It is usually formulated as follows:

n n n
F (X)  c   qi xi   Q ij xi x j
subject to i 1 i 1 j 1

a
i 1
ij xi  b j , j  1,2, , m

xi  0, i  1,2, , n
Min f(x) = ½ X” AX +X” b +c
Subject to constraints
where c, qi,Qij, aij, and bj are constants. A1X=C and A2X < d

9
Integer Programming Problem
• If some or all of the design variables x1,x2,..,xn of an
optimization problem are restricted to take on only
integer (or discrete) values, the problem is called an
integer programming problem.

• If all the design variables are permitted to take any

real value, the optimization problem is called a real-
valued programming problem.

10
Stochastic Programming Problem
• A stochastic programming problem is an optimization
problem in which some or all of the parameters
(design variables and/or preassigned parameters) are
probabilistic (nondeterministic or stochastic).
• In other words, stochastic programming deals with
the solution of the optimization problems in which
some of the variables are described by probability
distributions.

11
Separable Programming Problem
• A function f (x) is said to be separable if it can be expressed as the sum
of n single variable functions, f1(x1), f2(x2),….,fn(xn), that is,
n
f (X)   f i xi
i 1

• A separable programming problem is one in which the objective

function and the constraints are separable and can be expressed in
standard form as: n

Find X which minimizes f ( X)   f i ( xi )

i 1

subject to n
g j ( X)   g ij ( xi )  b j , j  1,2,  , m
i 1

where bj is constant.

12
Multiobjective Programming Problem
• A multiobjective programming problem can be stated as follows:

Find X which minimizes f1 (X), f2 (X),…., fk (X)

subject to
g j ( X)  0, j  1,2,..., m

where f1 , f2,…., fk denote the objective functions to be minimized

simultaneously.

13
 Review of mathematics
Concepts from linear algebra:
Positive definiteness
• Test 1: A matrix A will be positive definite if all its
eigenvalues are positive; that is, all the values of  that satisfy
the determinental equation
A  I  0
should be positive. Similarly, the matrix A will be negative
definite if its eigenvalues are negative.

14
Optimization

1. Introduction to Optimization
2. Classical Optimization Techniques
3. Linear programming and the Simplex method
4. Nonlinear programming-One Dimensional Minimization Methods
5. Nonlinear programming-Unconstrained Optimization Techniques
6. Nonlinear programming-Constrained Optimization Techniques
7. Global Optimization Methods-Genetic algorithms
8. Global Optimization Methods-Simulated Annealing
9. Global Optimization Methods- Coupled Local Minimizers

15
Classification of optimization problems

Classification based on:

• Constraints
• Constrained optimization problem
• Unconstrained optimization problem

• Nature of the design variables

• Static optimization problems
• Dynamic optimization problems

16
Classification of optimization problems

Classification based on:

• Physical structure of the problem

• Optimal control problems
• Non-optimal control problems

• Nature of the equations involved

• Nonlinear programming problem
• Geometric programming problem
• Quadratic programming problem
• Linear programming problem

17
Classification of optimization problems

Classification based on:

• Permissable values of the design variables

• Integer programming problems
• Real valued programming problems

• Deterministic nature of the variables

• Stochastic programming problem
• Deterministic programming problem

18
Classification of optimization problems

Classification based on:

• Separability of the functions

• Separable programming problems
• Non-separable programming problems

• Number of the objective functions

• Single objective programming problem
• Multiobjective programming problem

19
Geometric Programming
• A geometric programming problem (GMP) is one in which the
objective function and constraints are expressed as posynomials in X.

20
21
Quadratic Programming Problem
• A quadratic programming problem is a nonlinear programming problem with a
quadratic objective function and linear constraints. It is usually formulated as follows:

n n n
F (X)  c   qi xi   Q ij xi x j
subject to i 1 i 1 j 1

a
i 1
ij xi  b j , j  1,2, , m

xi  0, i  1,2, , n
Min f(x) = ½ X” AX +X” b +c
Subject to constraints
where c, qi,Qij, aij, and bj are constants. A1X=C and A2X < d

22
Integer Programming Problem
• If some or all of the design variables x1,x2,..,xn of an
optimization problem are restricted to take on only
integer (or discrete) values, the problem is called an
integer programming problem.

• If all the design variables are permitted to take any

real value, the optimization problem is called a real-
valued programming problem.

23
Stochastic Programming Problem
• A stochastic programming problem is an optimization
problem in which some or all of the parameters
(design variables and/or preassigned parameters) are
probabilistic (nondeterministic or stochastic).
• In other words, stochastic programming deals with
the solution of the optimization problems in which
some of the variables are described by probability
distributions.

24
Separable Programming Problem
• A function f (x) is said to be separable if it can be expressed as the sum
of n single variable functions, f1(x1), f2(x2),….,fn(xn), that is,
n
f (X)   f i xi
i 1

• A separable programming problem is one in which the objective

function and the constraints are separable and can be expressed in
standard form as: n

Find X which minimizes f ( X)   f i ( xi )

i 1

subject to n
g j ( X)   g ij ( xi )  b j , j  1,2,  , m
i 1

where bj is constant.

25
Multiobjective Programming Problem
• A multiobjective programming problem can be stated as follows:

Find X which minimizes f1 (X), f2 (X),…., fk (X)

subject to
g j ( X)  0, j  1,2,..., m

where f1 , f2,…., fk denote the objective functions to be minimized

simultaneously.

26
 Review of mathematics
Concepts from linear algebra:
Positive definiteness
• Test 1: A matrix A will be positive definite if all its
eigenvalues are positive; that is, all the values of  that satisfy
the determinental equation
A  I  0
should be positive. Similarly, the matrix A will be negative
definite if its eigenvalues are negative.

27
 Review of mathematics
Positive definiteness
• Test 2: Another test that can be used to find the positive definiteness
of a matrix A of order n involves evaluation of the determinants
A  a11
a11 a12 a13  a1n
a11 a12 a21 a22 a23  a2 n
A2 
a21 a22 A3  a31 a32 a33  a3n
a11 a12 a13 
A3  a21 a22 a23 an1 an 2 an 3  ann
a31 a32 a33

• The matrix A will be positive definite if and only if all the values A1,
A2, A3,An are positive
• The matrix A will be negative definite if and only if the sign of Aj is
(-1)j for j=1,2,,n
• If some of the Aj are positive and the remaining Aj are zero, the matrix
A will be positive semidefinite
28
 Review of mathematics

Negative definiteness

• Equivalently, a matrix is negative-definite if all its

eigenvalues are negative

• It is positive-semidefinite if all its eigenvalues are all

greater than or equal to zero

• It is negative-semidefinite if all its eigenvalues are all

less than or equal to zero

29
Review of mathematics

Concepts from linear algebra:

Nonsingular matrix: The determinant of the matrix is not

zero.

Rank: The rank of a matrix A is the order of the largest

nonsingular square submatrix of A, that is, the largest
submatrix with a determinant other than zero.

30
Review of mathematics

Solutions of a linear problem

Minimize f(x)=cTx
Subject to g(x): Ax=b
Side constraints: x ≥0
• The existence of a solution to this problem depends on the rows of A.

• If the rows of A are linearly independent, then there is a unique solution to the system of
equations.

• If det(A) is zero, that is, matrix A is singular, there are either no solutions or infinite
solutions.
31
Review of mathematics

Suppose

1 1 1 3  1 1 1 3 
A  b  A*   
 1 1 0.5 1.5 - 1 1 0.5 1.5

The new matrix A* is called the augmented matrix- the columns of b are added to A.
According to the theorems of linear algebra:

• If the augmented matrix A* and the matrix of coefficients A have the same rank r which is
less than the number of design variables n: (r < n), then there are many solutions.

• If the augmented matrix A* and the matrix of coefficients A do not have the same rank, a
solution does not exist.

• If the augmented matrix A* and the matrix of coefficients A have the same rank r=n,
where the number of constraints is equal to the number of design variables, then there is a
unique solution.
32
Review of mathematics

In the example

1 1 1 3  1 1 1 3 
A  b  A*   
  1 1 0 .5 1.5  - 1 1 0.5 1.5 

The largest square submatrix is a 2 x 2 matrix (since m = 2

and m < n). Taking the submatrix which includes the first two
columns of A, the determinant has a value of 2 and therefore
is nonsingular. Thus the rank of A is 2 (r = 2). The same
columns appear in A* making its rank also 2. Since
r < n, infinitely many solutions exist.

33
Review of mathematics

In the example

1 1 1 3  1 1 1 3 
A  b  A*   
  1 1 0. 5  1.5  - 1 1 0.5 1.5 

One way to determine the solutions is to assign ( n-r) variables arbitrary

values and use them to determine values for the remaining r variables.
The value n-r is often identified as the degree of freedom for the system of
equations.

In this example, the degree of freedom is 1 (i.e., 3-2). For instance x3 can
be assigned a value of 1 in which case x1=0.5 and x2=1.5

34
Example
Determine the maximum and minimum values of the function:

f ( x)  12 x 5  45 x 4  40 x 3  5

Solution: Since f’(x)=60(x4-3x3+2x2)=60x2(x-1)(x-2),

f’(x)=0 at x=0,x=1, and x=2.

The second derivative is:

f ( x)  60(4 x 3  9 x 2  4 x)
At x=1, f’’(x)=-60 and hence x=1 is a relative maximum. Therefore,
fmax= f (x=1) = 12

At x=2, f’’(x)=240 and hence x=2 is a relative minimum. Therefore,

fmin= f (x=2) = -11

35
Example

Solution cont’d:
At x=0, f’’(x)=0 and hence we must investigate the next derivative.
f ( x)  60(12 x 2  18 x  4)  240 at x  0
f ( x)  0
Since at x=0, x=0 is neither a maximum nor a minimum, and it is an
inflection point.

36
Multivariable optimization with no constraints

• Definition: rth Differential of f

If all partial derivatives of the function f through order r ≥ 1
exist and are continuous at a point X*, the polynomial
n n n
 r f ( X*)
d f ( X*)  
r
   hi h j  hk
i 1 j 1 k 1 xi x j  xk

r summations

is called the rth differential of f at X*.

37
Multivariable optimization with no constraints

• Example: rth Differential of f

n n n
 r f ( X*)
d f ( X*)  
r
   hi h j  hk
i 1 j 1 k 1 xi x j  xk

r summations

when r = 2 and n = 3, we have

3 3
 2 f ( X*)
d f ( X*)  d f ( x1 *, x 2 *, x3 *)  
2 2
 hi h j
i 1 j 1 xi x j
2 f 2  f
2
2  f
2
h 1
2
( X*)  h2 ( X*)  h3 ( X*)
x12
x 2
2
x32

2 f 2 f 2 f
 2h1 h2 ( X*)  2h2 h3 ( X*)  2h1 h3 ( X*)
x1x 2 x 2 x3 x1x3

38
Multivariable optimization with no constraints

• Definition: rth Differential of f

The Taylor series expansion of a function f (X*)
about a point X* is given by:
1 2 1
f ( X)  f ( X*)  df ( X*)  d f ( X*)  d 3 f ( X*)
2! 3!
1 N
 d f ( X*)  R N ( X*, h )
N!

where the last term, called the remainder is given by:

1
RN ( X*, h)  d N 1 f ( X*, h)
( N  1)!
where 0    1 and h  X - X *
39
Example
Find the second order Taylor’s series approximation of the function
f ( x1 , x 2 , x3 )  x 22 x3  x1e x3

 1
about the point  
X*   0
- 2 
 

Solution: The second order Taylor’s series approximation of the function f about point
X* is given by  1  1  1
    1  
f ( X)  f  0   df  0   d 2 f  0 
  2   2  2!   2
     

40
 Linear Programming I:
Simplex method
• Linear programming is an optimization method applicable
for the solution of problems in which the objective function
and the constraints appear as linear functions of the decision
variables.

• Simplex method is the most efficient and popular method

for solving general linear programming problems.

• At least four Nobel prizes were awarded for contributions

related to linear programming (e.g. In 1975, Kantorovich of
the former Soviet Union and T.C. Koopmans of USA were
awarded for application of LP to the economic problem of
allocating resources).

41
 Linear Programming I:
Simplex method-Applications
• Petroleum refineries
– choice of buying crude oil from several different sources with
differing compositions and at differing prices
– manufacturing different products such as aviation fuel, diesel fuel,
and gasoline, in varying quantities
– Constraints due to the restrictions on the quantity of the crude oil
from a particular source, the capacity of the refinery to produce a
particular product
– A mix of the purchased crude oil and the manufactured products is
sought that gives the maximum profit
• Optimal production plan in a manufacturing firm
– Pay overtime rates to achieve higher production during periods of
higher demand
• The routing of aircraft and ships can also be decided using
LP

42
 Standard Form of a Linear
Programming Problem
• Scalar form
Minimize f ( x1 , x2 , , xn )  c1 x1  c2 x2    cn xn
subject to the constraint s
a11 x1  a12 x2    a1n xn  b1
a21 x1  a22 x2    a2 n xn  b2

am1 x1  am 2 x2    amn xn  bm
x1  0
x2  0

xn  0
where c j , b j and aij (i  1,2, , m; j  1,2, , n) are known constants,
and x j are the decision v ariables
43
 Standard Form of a Linear
Programming Problem
• Matrix form
Minimize f ( X)  c T X
subject to the constraint s
aX  b
X0
where
 x1  b1  c1 
x  b  c 
 2  2  
X   , b   , c   2
   
 xn  bm  cn 
a11 a12 a13  a1n 
a a a  a 
a 21 22 23 2 n 
 
 
a
 m1 a m2 a m3  a mn 
44
 Characteristic of a Linear
Programming Problem

• The objective function is of the minimization type

• All the constraints are of the equality type

• All the decision variables are nonnegative

• The number of the variables in the problem is n. This

includes the slack and surplus variables.

• The number of constraints is m (m < n).

45
 Characteristic of a Linear
Programming Problem
• The number of basic variables is m (same as the number of
constraints).

• The number of nonbasic variables is n-m.

• The column of the right hand side b is positive and greater

than or equal to zero.

• The calculations are organized in a table.

• Only the values of the coefficients are necessary for the

calculations. The table therefore contains only coefficient
values, the matrix A in previous discussions. These are the
coefficients in the constraint equations.
46
 Characteristic of a Linear
Programming Problem
• The objective function is the last row in the table. The
constraint coefficients are written first.

• Row operations consist of adding (subtracting)a definite

multiple of the pivot row from other rows of the table.

47
 Transformation of LP Problems into
Standard Form
• The maximization of a function f(x1,x2,…,xn ) is equivalent
to the minimization of the negative of the same function.
For example, the objective function
minimize f  c1 x1  c2 x2    cn xn
is equivalent to
maximize f    f  c1 x1  c2 x2    cn xn

Consequently, the objective function can be stated in the

minimization form in any linear programming problem.

48
 Transformation of LP Problems into
Standard Form
• A variable may be unrestricted in sign in some problems. In
such cases, an unrestricted variable (which can take a
positive, negative or zero value) can be written as the
difference of two nonnegative variables.

• Thus if xj is unrestricted in sign, it can be written as xj=xj'-

xj", where
x j '  0 and x j "  0

• It can be seen that xj will be negative, zero or positive,

depending on whether xj" is greater than, equal to, or less
than xj’ .
49
Transformation of LP Problems into
Standard Form
If a constraint appears in the form of a “less than or equal
to” type of inequality as:
ak1 x1  ak 2 x2    akn xn  bk

it can be converted into the equality form by adding a

nonnegative slack variable xn+1 as follows:
ak 1 x1  ak 2 x2    akn xn  xn 1  bk

50