Optimization using Calculus

Optimization of Functions of Multiple Variables: Unconstrained Optimization

D Nagesh Kumar, IISc

Optimization Methods: M2L3

Objectives
To study functions of multiple variables, which are more difficult to analyze owing to the difficulty in graphical representation and tedious calculations involved in mathematical analysis for unconstrained optimization. To study the above with the aid of the gradient vector and the Hessian matrix. To discuss the implementation of the technique through examples

D Nagesh Kumar, IISc

Optimization Methods: M2L3

Unconstrained optimization
If a convex function is to be minimized, the stationary point is the global minimum and analysis is relatively straightforward as discussed earlier. A similar situation exists for maximizing a concave variable function. The necessary and sufficient conditions for the optimization of unconstrained function of several variables are discussed.

D Nagesh Kumar, IISc

Optimization Methods: M2L3

Necessary condition
In case of multivariable functions a necessary condition for a stationary point of the function f(X) is that each partial derivative is equal to zero. In other words, each element of the gradient vector x f defined below must be equal to zero. i.e. the gradient vector of f(X), at X=X*, defined as follows, must be equal to zero:
f ( * ) x 1 f ( * ) x 2 =0 x f = M M f ( * ) dxn

D Nagesh Kumar, IISc

Optimization Methods: M2L3

Sufficient condition
For a stationary point X* to be an extreme point, the matrix of second partial derivatives (Hessian matrix) of f(X) evaluated at X* must be: positive definite when X* is a point of relative minimum, and negative definite when X* is a relative maximum point. When all eigen values are negative for all possible values of X, then X* is a global maximum, and when all eigen values are positive for all possible values of X, then X* is a global minimum. If some of the eigen values of the Hessian at X* are positive and some negative, or if some are zero, the stationary point, X*, is neither a local maximum nor a local minimum.

D Nagesh Kumar, IISc

Optimization Methods: M2L3

Example
2 Analyze the function f ( x) = x12 x2 x32 + 2 x1 x2 + 2 x1 x3 + 4 x1 5 x3 + 2 and classify the stationary points as maxima, minima and points of inflection Solution

f ( * ) x1 2 x1 + 2 x2 + 2 x3 + 4 0 f = 0 x f = 2 x2 + 2 x1 ( * ) = x2 2 x + 2 x 5 0 3 1 f * ( ) x3

D Nagesh Kumar, IISc

Optimization Methods: M2L3

Example contd.

D Nagesh Kumar, IISc

Optimization Methods: M2L3

Example contd.

D Nagesh Kumar, IISc

Optimization Methods: M2L3

Example contd.

D Nagesh Kumar, IISc

Optimization Methods: M2L3

Thank you

10

D Nagesh Kumar, IISc

Optimization Methods: M2L3