Nonlinear Programming

Elimination Method Fibonacci Method, Golden Section Method Chapters: 5.7 & 5.8 (Engineering Optimization by S S Rao)

Satpathi DK, BITS-Hyderabad Campus

Unimodal function:

Satpathi DK, BITS-Hyderabad Campus

Satpathi DK, BITS-Hyderabad Campus

Satpathi DK, BITS-Hyderabad Campus

Interval of uncertainty: This is understood by saying that initially the interval of uncertainty is [a,b] i.e. the optimum is somewhere in [a,b]. Then after two experiments [finding ], the interval of uncertainty reduces to .

Satpathi DK, BITS-Hyderabad Campus

Measure of effectiveness: Let be the initial interval of uncertainty. Let be the interval of uncertainty after N experiments. The measure of effectiveness is defined as

Satpathi DK, BITS-Hyderabad Campus

Fibonacci Method This method is used to find the minimum of a function of one variable. Method: The method uses the Fibonacci sequence {Fn} i.e. F0=1=F1, Fn=Fn-1+Fn-2 , n>1 which yields the sequence 1,1,2,3,5,8,13,21..

Satpathi DK, BITS-Hyderabad Campus

Procedure: Let L0 be the initial interval of uncertainty defined by a x b and n be the number of experiments to be conducted. Define F (1) L L
* 2 n 2

Fn

and place the first two experiments at points x1 and x2, which are located at distance of L* from each end of L0. 2 This gives
Satpathi DK, BITS-Hyderabad Campus

x1 x2

a b b

L* 2 L* 2 Fn

a b Fn Fn

Fn 2 L0 Fn Fn 2 L0 Fn
1

L0

b (b a ) a Fn 1 L0 Fn

Fn 1 L0 Fn ....(2)

(This shows that b-x2=x1-a)

L0
a x1 x2 b
Satpathi DK, BITS-Hyderabad Campus

Discard part of the interval by using unimodality assumption. Interval of uncertainty depending on f(x1) < or > f(x2) after two experiment is
L2 Length of a, x2 or x1 , b x2 a or b - x1 (b x2 x2 a L2 L0 L
* 2

x1 a

b x1

x2 a )

Fn 1 L0 Fn Fn 2 L0 1 Fn Fn 1 L0 Fn

and with one experiment left in it.

Satpathi DK, BITS-Hyderabad Campus

This experiment will be at a distance of

L
* 2

Fn 2 L0 Fn

Fn 2 Fn 1 L0 Fn 1 Fn

Fn 2 L2 Fn 1

from one end and

L2 L
* 2

Fn 2 L2 1 Fn 1

Fn 1 Fn 2 Fn 3 L2 L2 from the other end. Fn 1 Fn 1

Now place the third experiment in L2 so that the current two experiments are located at a distance of Fn 3 Fn 3 Fn 1 Fn 3 * L3 L0 L0 L2 from each end of the Fn Fn 1 Fn Fn 1 interval L2.
Satpathi DK, BITS-Hyderabad Campus

Again the unimodality property will allow us to reduce the interval of uncertainty to L3 given by
L3 L2 L
* 3

L2

Fn 3 L2 Fn 1

Fn 1 Fn 3 L2 Fn 1

Fn 2 L2 Fn 1

Fn 2 L0 Fn

This process of discarding a certain interval and placing a new experiment in the interval can be continued, so that the location of the jth experiment and the interval of uncertainty at the end of jth experiments are, given by
Satpathi DK, BITS-Hyderabad Campus

* j

Fn Fn Fn

Lj L0

( j 2) ( j 1)

Lj

Fn

and for j=n, we have The ratio, Ln/L0 will permit us to determine n, the required number of experiments, to achieve any desired accuracy in locating the optimum point.

Ln L0

F1 Fn

1 Fn

Satpathi DK, BITS-Hyderabad Campus

Position of the final experiment:

L*n Ln 1 F0 F2 1 n 2 L
* j

Fn Fn

Lj 1

( j 2)

Thus after concluding n-1 experiments and discarding the appropriate interval in each step, the remaining interval will contain one experiment namely the nth experiment, is also to be placed at the centre of the present interval of uncertainty. That is, the position of the nth experiment will be same as that of (n-1)th one, and this is true for whatever value we choose for n.

Satpathi DK, BITS-Hyderabad Campus

Since no new information can be gained by placing the nth experiment exactly at the same location as that of (n-1)th experiment, we place the nth experiment very close to the remaining valid experiment. This enables us to obtain the final interval of uncertainty to 1 within 2 L .
n 1

Satpathi DK, BITS-Hyderabad Campus

Limitations of the method: 1. The initial interval of uncertainty, in which the optimum lies, has to known. 2. The function being optimized has to be unimodal (Unimodal function is one that has only one peak (maximum) or valley (minimum) in a given interval) in the initial interval of uncertainty. 3. The exact optimum cannot be located in this method. Only an interval known as the final interval of uncertainty will be known. The final interval of uncertainty can be made as small as desired by using more computations. 4. The number of function evaluations to be used in the search or the resolution required has to be specified beforehand.
Satpathi DK, BITS-Hyderabad Campus

Q. Find the minimum of f ( x) x 2 2 x, 0 x 1.5 with n 4

L*2 Fn 2 L0 Fn F2 L0 F4 2 1.5 0.6 5

L0 is the original interval of uncertainty.

Thus the positions of the first two experiments are

x1 a L*2 0.6 x2 b L f x1
* 2

x1 x2 0
0.99

0.9

1.5 -.84 -.99

0.84, f x2

Since f(x1) > f(x2) we reject [0,x1] The third experiment is placed at
x3 b ( x2 x1 ) 1.5 (.9 .6) 1.2 with f ( x3 ) 0.96

x1 x2 x3 b .6 .9 1.2 1.5 -.99 -.96

Satpathi DK, BITS-Hyderabad Campus

Since f(x3) > f(x2) we reject [1.2,1.5] Now the interval of uncertainty is [0.6,1.2] Note that the point 0.9 is the middle point i.e. it is equal distance from both end points. Take x4 =0.95. So f(x4)=-0.9975 Since f(.9)>f(.95) the new interval of uncertainty L4 is [.9,1.2].
L4 .3 1 .2 .25 L0 1.5 5
Satpathi DK, BITS-Hyderabad Campus

Golden Section Method

Satpathi DK, BITS-Hyderabad Campus

The golden section method is same as the Fibonacci method except that in the Fibonacci method the total number of experiments to be conducted has to be specified before beginning of the calculation, whereas this is not required in golden section method. In the Fibonacci method, the location of the first two experiments is determined by the total number of experiments, n. In the golden section method we start with the assumption that we are going to conduct a large number of experiments. Of course, the total number of experiments can be decided during the computation.
Satpathi DK, BITS-Hyderabad Campus

Satpathi DK, BITS-Hyderabad Campus

Satpathi DK, BITS-Hyderabad Campus

Satpathi DK, BITS-Hyderabad Campus

Procedure: The procedure is same as Fibonacci method, except that the location of the first two experiments is defined by
L
* 2

Fn 2 L0 Fn

Fn 2 Fn 1 L0 Fn 1 Fn

L0
2

0.382 L0

The desired accuracy can be specified to stop the procedure.

Satpathi DK, BITS-Hyderabad Campus

Satpathi DK, BITS-Hyderabad Campus

Definition: A function f(X)=f(x1,x2,xn) of n variables is said to be convex if for each pair of points X,Y on the graph, the line segment joining these 2 points lies entirely above or on the graph.
f((1-)X + Y) (1-)f(X)+ f(Y)

f is called concave( strictly concave) if f is convex( strictly convex) Convexity test for function of one variable
d2 f dx 2 d2 f dx 2

Convex if
concave if

0 0

Convexity test for functions of 2 variables

quantity
fxx-(fxy)2 fxx fyy

convex
0 0 0

Strictly convex >0

>0 >0

concave
0 0 0

Strictly concave >0

<0 <0

When is a locally optimal solution also globally optimal?

For minimization problems
The objective function is convex. The feasible region is convex.

Local Maximum Property

A local max of a concave function on a convex feasible region is also a global max. Strict concavity implies that the global optimum is unique. Given this, the following NLPs can be solved Maximization Problems with a concave objective function and linear constraints

Nonlinear Programming
Steepest ascent Method, Steepest descent Method, Conjugate Gradient Method Chapters: 19.1.1, 19.1.2 (H. A. Taha) & 6.11 (Engineering Optimization by S S Rao)

Gradient of a function The gradient of a function is an n-component f vector is given by x

1

f
n 1

..... f xn

The gradient has a very important property. If we move along the gradient direction from any point in n-dimensional space, the function value increases at the fastest rate. Hence the gradient direction is called the direction of steepest ascent. Unfortunately the direction of steepest ascent is a local property and not a global one.

Since the gradient vector represents the direction of steepest ascent, the negative of the gradient vector denotes the direction of steepest descent. Thus any method that makes use of the gradient vector can be expected to give the minimum point faster than one that does not make use of the gradient vector. Theorem: The gradient vector represents the direction steepest ascent.

Theorem: The maximum rate of change of f at any point X is equal to the magnitude of the gradient vector at the same point. Termination of gradient method occurs at the point where the gradient vector becomes null. This is the only necessary condition for optimality. Optimality cannot be verified unless it is known a priori that f(X) is concave or convex.

Steepest Ascent method: Suppose f(x) is maximized. Let X(0) be the initial point from which the procedure starts f X k as the gradient of f at the and define point Xk. The idea is to determine a particular f path p along which p is maximized at a given point. This result is achieved if successive points Xk and Xk+1 are selected such that
Xk
1

Xk

rk f X k

where

is the optimal step size at Xk.

The step size r k is determined such that the next point, Xk+1 lead to the largest improvement in f. This is equivalent to determining r=rk that maximizes the function h r f X r f X The proposed procedure terminates when two successive trial points Xk and Xk+1 are approximately equal. This is equivalent to having r f X 0 . k f X k 0 is Because r , the necessary condition 0 satisfied at Xk.
k k
k k

Q. Max f X
f x1
2

2x1x2 2x2 x
2 x1 f x2
2

2 1

2x
2
2

2 2

2 x2 2
2

2 x1 4 x2 2
2

f x1 x2
2

f
2 1

x
2

0 4 0

f
2 2

f
2 1

f
2 2

f x1 x2

f(x1,x2) is strictly concave. Suppose that X0=(0,0)

f 0,0

0,2

X
h r

r f 0,0
1

0,0
0

r 0,2
4r
0

0,2r

f X

f 0, 2r
0 4 16r
0

8r
1 4

02

d 0 02 4r 8r 0 dr
X
1

1 0, 2
2

f X
1

1,0
1 0, 2 r 1, 0
1

X
1

f X
2

1 r , 2
1

h r

f X

12

1 2

and
X
2

d 1 12 r r 1 dr

1 2

1 2

1 1 , 2 2

By continuing in this process the subsequent trial solutions would be

1 3 3 3 3 7 7 7 , , , , , , , ,........ 2 4 4 4 4 8 8 8

Because these points are converging to this solution is the optimal as f 1,1 0

X*

1,1

Steepest Descent Method

The use of the negative of the gradient vector as a direction of minimization problem was first made by Cauchy in 1847. In this method we start from an initial trial point X0 and iteratively move along the steepest descent directions until the optimum point is found. The steepest descent method can be summarized by the following steps:

1. Start with arbitrary initial point x0 2. Find the search direction Sk as S k

f X

3. Determine the optimal step length rk in the direction Sk and set X k 1 X k r k f X k 4. Test the new point, Xk+1 , for optimality
f X
k 1

Q. Min
f x1
2

f X
1 4 x1 2
2

x1 x2 2x
2 x2
2

2 1

2x1x2 x

2 2

f x2 f
2 1

1 2 x1
2

2 x2 2

f x1 x2
2

x
2

4
2

f
2 2

x 4 0

f
2 1

f
2 2

f x1 x2

f(x1,x2) is strictly convex. Suppose that X0=(0,0)

f 0,0

1, 1

r f 0,0
1

0,0
r ,r
1 1

r 1, 1
r
12

r ,r

h r

f X
dh 0 1 dr

2r1

2r 1 2 0

r1 1
1

1,1
1

f X
1

1, 1 1,1 r
2

X
h r

r f X
f X
2

1, 1

1 r ,1 r

5r

22

2r 2 1

and
X
2

dh 2 dr

10r

2 0

1 5

4 6 , 5 5

By continuing in this process the subsequent trial solutions would be

1,1.4 ,.....X * 1,1.5