Documenti di Didattica
Documenti di Professioni
Documenti di Cultura
Course Outline
Instructor
Dr. Wu-Sheng Lu
Phone: 721-8692
E-mail: wslu@ece.uvic.ca
URL: http://www.ece.uvic.ca/~wslu
Office Hours
Days: Wednesdays
Time: 14:00 16:00
Location: EOW 427
Lectures
Days: Tuesday, Wednesday, Friday
Time: 12:30 1:20 p.m.
Location: CLE A206
Labs
Location: ELW B326
Section Days Time
B01 (30274) Wednesday 14:30 17:30
B04 (30277) Tuesday 14:30 17:30
ELEC 403 Lab starts on May 22, Wednesday.
Required Text
Title: Practical Optimization: Algorithms and Engineering Applications
http://www.ece.uvic.ca/~optimization/
Author: A. Antoniou and W.-S. Lu
Publisher: Springer
Year: 2007
Assessment
Assignments: 10%
Labs (ELEC403) 15%
Labs & Project (ELEC503) 15%
Mid-term exam 20% Date: July 9 (Tuesday)
Final exam 55%
2
Other Things to Note:
Course web site:
http://www.ece.uvic.ca/~wslu/403/index.html
You need the user name and password to access Course Material.
About assignments:
Only paper copies will be accepted.
Submit your assignment on the due day by 4:00 pm.
You may submit your work to the instructor at class; or leave
it in his mailbox in EOW 448; or slide it into his office EOW 427.
ELEC403/503 does not use drop box for submissions.
Please clearly print your name and the course number on the
cover page.
Due day for each assignment will be announced in class and
posted on the course website.
3
ELEC 403/503 classes on May 21 (Tuesday) and May 22 (Wednesday)
are cancelled. A make-up class of 100 minutes will be scheduled
in the second week.
4
ELEC 403/503
Engineering Design by Optimization
People Optimize
Investors seek to create portfolios that avoid excessive risk while
achieving high rate of return;
Manufacturers aim for maximum efficiency in the design and
operation of their production processes;
Engineers adjust parameters to optimize the performance of their
designs.
Nature Optimizes
Physical systems tend to a state of minimum energy;
Molecules in an isolated chemical system react with each other
until the total potential energy of their electrons is minimized;
Rays of light follow paths that minimize their travel time.
a
Optimization is a tool for decision science and analysis of physical
systems.
c
1.2 The Basic Optimization Problem
Before optimization is attempted, the problem at hand must be
properly formulated. A performance criterion F must be derived
in terms of n parameters x1 , x2 , . . . , xn as
F = f (x1 , x2 , . . . , xn )
Chap. 1
To simplify the notation, matrix notation is usually employed. If
x is a column vector with elements x1 , x2 , . . . , xn , the transpose
of x, namely, xT , can be expressed as the row vector
xT = [x1 x2 xn ]
Chap. 1
On many occasions, the optimization problem consists of
finding the maximum of the objective function. Since
Chap. 1
1.3 General Structure of Optimization Algorithms
Most of the available optimization algorithms entail a series of
steps which are executed sequentially. A typical pattern is as
follows:
Algorithm 1.1 General optimization algorithm
Step 1
(a) Set k = 0 and initialize x0 .
(b) Compute F0 = f (x0 ).
Step 2
(a) Set k = k + 1.
(b) Compute the changes in xk given by column vector x k where
xTk = [x1 x2 xn ]
Chap. 1
Step 3
Check if convergence has been achieved by using an appropriate
criterion, e.g., by checking Fk and/or xk . If this is the case,
continue to Step 4; otherwise, go to Step 2.
Step 4
(a) Output x = xk and F = f (x ).
(b) Stop.
Chap. 1
Convergence can be checked in several ways, depending on
the optimization problem and the optimization technique used.
For example, one might decide to stop the algorithm when the
reduction in Fk between any two iterations has become
insignificant, that is,
Chap. 1
1.4 Constraints
In many optimization problems, the variables are interrelated by
physical laws like the conservation of mass or energy,
Kirchhoffs voltage and current laws, and other system
equalities that must be satisfied. In effect, in these problems
certain equality constraints of the form
ai (x) = 0 for x E n
Chap. 1
In other optimization problems a collection of inequality
constraints might be imposed on the variables or parameters to
ensure physical realizability, reliability, compatibility, or even to
simplify the modeling of the problem.
In these problems, a collection of inequality constraints of the
form
cj (x) 0 for x E n
where j = 1, 2, . . . , q must be satisfied before the optimization
problem can be considered solved.
Chap. 1
An optimization problem may entail a set of equality constraints
and possibly a set of inequality constraints. If this is the case,
the problem is said to be a constrained optimization problem.
The most general constrained optimization problem can be
expressed mathematically as
Chap. 1
Example 1.2 Consider a control system that comprises a
double inverted pendulum as depicted in Fig. 1.3. The objective
of the system is to maintain the pendulum in the upright position
using the minimum amount of energy. This is achieved by
applying an appropriate control force to the car to damp out any
displacements 1 (t ) and 2 (t ). Formulate the problem as an
optimization problem.
u(t)
2 (t ) 0 0 0
Chap. 1
with > 0, > 0, and 6= . In the above equations, x(t ),
1 (t ), and 2 (t ) represent the first derivatives of x(t ), 1 (t ), and
2 (t ), respectively, with respect to time, 1 (t ) and 2 (t ) would be
the second derivatives of 1 (t ) and 2 (t ), and parameters and
depend on system parameters such as the length and weight
of each pendulum, the mass of the car, etc. Suppose that at
instant t = 0 small nonzero displacements 1 (t ) and 2 (t )
occur, which would call for immediate control action in order to
steer the system back to the equilibrium state x(t ) = 0 at time
t = T0 . In order to develop a digital controller, the system model
in (1.5) is discretized to become
Chap. 1
Let x(0) 6= 0 be given and assume that T 0 is a multiple of t ,
i.e., T0 = K t where K is an integer. We seek to find a
sequence of control actions u(k ) for k = 0, 1, . . . , K 1 such
that the zero equilibrium state is achieved at t = T 0 , i.e.,
x(T0 ) = 0.
Let us assume that the energy consumed by these control
actions, namely,
K 1
u 2 (k )
X
J=
k =0
needs to be minimized.
Chap. 1
This optimal control problem can be formulated analytically as
K 1
u 2 (k )
X
minimize J = (1.7a)
k =0
subject to: x(K ) = 0 (1.7b)
Chap. 1
Hence constraint (1.7b) is equivalent to
K
X 1
gk u(k ) = h (1.8)
k =0
minimize uT u (1.9a)
where a(u) = Gu h.
Chap. 1
In practice, the control actions cannot be made arbitrarily large
in magnitude. Consequently, additional constraints are often
imposed on |u(i )|, for instance,
|u(i )| m for i = 0, 1, . . . , K 1
m + u(i ) 0
m u(i ) 0
Hence if we define
m + u(0)
m u(0)
..
c(u) = .
m + u(K 1)
m u(K 1)
Chap. 1
then the magnitude constraints can be expressed as
c(u) 0 (1.9c)
Obviously, the problem in Eq. (1.9) fits nicely into the standard
form of optimization problems given by Eq. (1.4).
where R E n .
Chap. 1
One More Example: The Portfolio Optimization Problem
22a
What is a portfolio?
Todays investors can choose from a wide range of possible
securities bills and notes, bonds, individual stocks from
exchanges around the world, mutual funds, and so on each
has its own potential risks and rewards. A portfolio is a list of
investments held by a person.
Some Observations
22b
The returns on different securities are often correlated,
positively or negatively. For example, the stock prices of
gold mining companies and of gold itself tend to increase or
decrease more or less synchronously, i.e., they are positively
correlated. On the other hand, returns on oil investments may
be negatively correlated with returns on the stocks of fuel-
cell manufacturers.
E[( xi i )( x j j )]
i , j = i, j = 1,, n
i j
22d
We try to construct a portfolio by allocating a fraction wi of
the available resources to security i, for each i. So we have
the simple constraints on wi as
n
0 wi 1, w =1
i =1
i
i =1 j =1
n n
= E[ wi w j ( xi i )( x j j )]
i =1 j =1
n n
= wi w j E[( xi i )( x j j )]
i =1 j =1
n n
= wi w j i j i , j
i =1 j =1
Qij
n n
= Qij wi w j
i =1 j =1
22f
Portfolio Optimization Problem 1: Minimize risk for a
given level of return:
n n
minimize
{ wi }
Q w w
i =1 j =1
ij i j
n
subject to: w
i =1
i i
wi 0 for 1 i n
n
w =1
i =1
i
22g
Portfolio Optimization Problem 2: Maximize return for a
given level of risk:
n
maximize
{ wi }
w
i =1
i i
n n
subject to: Q w w
i =1 j =1
ij i j 2
wi 0 for 1 i n
n
w =1
i =1
i
22h
Portfolio Optimization Problem 3: Keep the expected
return large and the variance small:
n n n
minimize Qij wi w j i wi
{ wi }
i =1 j =1 i =1
w =1
i =1
i
22i
Example Consider a portfolio optimization problem with
only two stocks on the investors list. The statistical data for
these stocks on a per $100 scale are as follows:
Stock # 1: 1 = 5, 12 = 9
Stock # 2: 2 = 12, 22 = 196
12 1 2 12 9 21
Q= =
1 2 21 2
2
21 196
22j
and the portfolio optimization problem that minimizes the
risk while having an acceptable minimum expected return
is explicitly given by
2 2
minimize Q w w
i =1 j =1
ij i j = 9 w 42 w1 w2 + 196 w
2
1
2
2
22k
Solution Below we consider two cases.
Case 1: The minimum expected return is set to = 7.
The feasible region in this case is a portion of the segment of
the straight line w1 + w2 = 1 in the first quadrant of the w1w2
plane with endpoints P1 and P2, where P2 = [0 1]T and P1 is
determined by the equations
w1 + w2 = 1
5w1 + 12 w2 = = 7
whose solution is given by w1 = 5/7 = 0.7143 and w2 = 2/7 =
0.2857, i.e.,
P1 = [0.7143 0.2857]T
22l
The solution in this case can readily be identified as point P1
because as soon as the candidate solution point starts to leave
P1 towards P2 (see Figure A below), it suggests a buying
policy that invests more money on the risky stock # 2
therefore increases the objective function.
22m
Feasible Region
2
1.5
P2
1
w2
w1 + w2 = 1
0.5
P
1
5*w1 + 12*w2 = 7
0.5
0.5 0 0.5 1 1.5 2
w1
22o
Polynomial 247*w12 434*w1 + 196 on [0 0.7143]
250
200
150
100
50
0
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7
w1
22p
The minimized risk at the solution point P1 is found to be
2 2
Q w w
i =1 j =1
ij i j = 12.0204
22q
same endpoint P2 = [0 1]T, but endpoint P1 is now
determined by the equations
w1 + w2 = 1
5w1 + 12 w2 = = 9
P1 = [0.4286 0.5714]T
Q w w
i =1 j =1
ij i j = 55.3673
22s
1.5 The Feasible Region
where R E n .
22t
The optimum point x must be located in the feasible region,
and so the general constrained optimization problem can be
stated as
minimize f (x) for x R
Any point x not in R is said to be a nonfeasible point .
If the constraints in an optimization problem are all inequalities,
the constraints divide the points in the E n space into three
types of points, as follows:
1. Interior points
2. Boundary points
3. Exterior points
Aninterior point is a point for which c j (x) > 0 for all j . A
boundary point is a point for which at least one c j (x) = 0, and
an exterior point is a point for which at least one c j (x) < 0.
Interior points are feasible points, boundary points may or may
not be feasible points, whereas exterior points are nonfeasible
points.
Chap. 1
Example 1.5 By using a graphical method, solve the following
optimization problem
Chap. 1
Solution The objective function can be expressed as
Hence the contours pof f (x) in the (x1 , x2 ) plane are concentric
circles with radius f (x) centered at x1 = 2, x2 = 0.
Constraints c1 (x) and c2 (x) dictate that
x2 12 x1 + 3
and
x2 x12 + 1
respectively, while constraints c 3 (x) and c4 (x) dictate that x1
and x2 be positive. The contours of f (x) and the boundaries of
the constraints can be constructed as shown in Fig. 1.5.
Chap. 1
5
c 2 (x) = - x12 + x 2 -1
x2 c 3 (x)
4
c 1 (x) = x1 -2 x 2 +6
3
Feasible 9
region
2
4
A
1 1
c 4 (x)
B
-2 -1 0 1 2 3 4 5
x1
-1
-2
Chap. 1
The feasible region for this problem is the shaded region in
Fig. 1.5. The solution is located at point A on the boundary of
constraint c2 (x). In effect, the solution is a constrained optimum
point. Consequently, if this problem is solved by means of
mathematical programming, constraint c 2 (x) will be active when
the solution is reached.
In the absence of constraints, the minimization of f (x) would
yield point B as the solution.
Chap. 1
From Fig. 1.5, we see that the minimum point is point A at
which the parabola intersects with one of the contour circles
with one intersection point.
x12 + x2 1 = 0 (a)
(
1x 2) 2
+ x2
2
= r 2
(b)
0 r.r. 2 r.r.
0 r.r.
0 r.r.
The search for r was implemented using MATLAB (see the code
below). It was found that r* = 1.9491 gives two equal real roots.
With this value of r, Eq. (c) yields x1 = 0.5536 and Eq. (a) then
gives x2 = 1.3064. Therefore, the minimizer of the constrained
optimization problem is given by
x = [ 0.5536 1.3064]
T
27c
% ex1_5.m
% This function uses a bisection method to identify radius r
% with which the contour circle tangents to the parabola,
% see the analysis in Example 1.5 of the class notes.
% Using this r, the function also computes the minimizer of
% the constrained optimization problem.
% Input: epsi -- tolerance for the uncertainty range of radius r.
% Outputs: x -- the minimizer
% r -- value of the radius
% k -- number of iterations used.
rl = 1;
ru = 2;
d = ru - rl;
k = 0;
while d > epsi,
27d
r = 0.5*(rl+ru);
rts = roots([1 0 3 -4 5-r^2]);
im1 = imag(rts(1));
im2 = imag(rts(3));
if im1*im2 ~= 0,
rl = r;
else
ru = r;
end
d=d/2;
k = k+1;
end
27f