1

Optimisation and Search

Optimisation and Search is something to optimally
achieve a goal with a minimum of cost in terms of
time, money, energy, power, etc.
HLW, Optimisationand Search, Lecture01, Slide 1 of 20
In this module, we shall be discussing
TWO topics:
1 Why Do We Use Optimisation? 1. Why Do We Use Optimisation?
Where do optimisation problems come from? Where do optimisation problems come from?
2. How to Optimise?
Given an optimisation problem, how to solve it? Given an optimisation problem, how to solve it?
HLW, Optimisationand Search, Lecture01, Slide 2 of 20
2
Motivations of Motivations of optimsation optimsation
LECTURE 01: INTRODUCTION LECTURE 01: INTRODUCTION
Motivations of Motivations of optimsation optimsation
Some basic concepts of optimisation and search Some basic concepts of optimisation and search
HLW, Optimisationand Search, Lecture01, Slide 3 of 20
(I) Why Use Optimization? (I) Why Use Optimization?
There exist a huge range of scenarios in the real world
where people need to make important decisions to try and
hi h b t ibl t ( l ti ) achieve the best possible outcomes (or solutions).
However, the best solutions ARE NOT always straight-
forward and can easily be obtained. This is because:
Many problems have different ways of operating;
HLW, Optimisationand Search, Lecture01, Slide 4 of 20
Many problems have different ways of operating;
It is often important to choose the most beneficial way.
3
Prototype 1: Tank Design Problem Prototype 1: Tank Design Problem
** The tank design problem here is to find the dimensions of
a rectangular open-topped tank in order to obtain the
smallest surface area that encloses a given volume V
*
.
** Denote the height by z and the
lengths of the two edges of the base
by x and y. The volume and surface
area are then
V = xyz and S = xy + 2xz + 2yz
x
y
z
HLW, Optimisationand Search, Lecture01, Slide 5 of 20
** The tank design problem can then be formulated as
Minimize S = xy + 2xz + 2yz
Subject to xyz = V
*
(fixed volume)
Prototype 2: Travelling Salesman Problem Prototype 2: Travelling Salesman Problem
** A salesman has to visit a certain number of cities (places),
where every city has to be visited once,
and only once, and the whole trip
should be as short as possible should be as short as possible.
Then, how should the
salesman arrange his trip?
A B
4
A B
4 4
A B A B
4
Example 1 Example 1
S
HLW, Optimisationand Search, Lecture01, Slide 6 of 20
A B
D C
5
2
2
3
2
A B
D C
5
2
2
3
2
A B
D C
5
2
2
3
2
A B
D C
5
2
2
3
2
A-B-C-D-A: 13 A-B-D-C-A: 11 A-D-B-C-A: 12
4
Prototype 2: Travelling Salesman Problem (Cont.) Prototype 2: Travelling Salesman Problem (Cont.)
Example 2 Example 2
B C
2
3
2 1
3
4
4
B C
D
2
3
2
1
3
4
4
A
D
E
3
5
2
A-E-B-C-D-A: 11
A
D
E
3
5
2
** The total number of candidate solutions would become
enormous when the number of cities (places) is large
HLW, Optimisationand Search, Lecture01, Slide 7 of 20
enormous when the number of cities (places) is large.
For example, if there are 10 cities to travel, the total
number of candidate solutions is 181,440. If there are 20
cities to visit, then the total number of candidate solutions
will become N = 610
16
.
Prototype 3: Optimisation for Control Engineering Prototype 3: Optimisation for Control Engineering
In control engineering, we often need to solve
different types of optimisation problems, e.g.
Parameter Optimisation for systems modelling,
i l ti d d i h i t ffi i t ( simulation and design, e.g., choose appropriate coefficients (or
gains) of a PID controller
System Control where we need to design an input signal
that takes a system from one state (position) to another
using minimum power (fuel or energy).
Path Planning for a robot to
HLW, Optimisationand Search, Lecture01, Slide 8 of 20
minimise the distance to be
travelled to a specified location.
Product Quality Optimizing within a fixed cost budget.
5
Prototype Prototype 4: Optimal Control 4: Optimal Control
** Consider a linear system
x
t+1
=Ax
t
+ Bu
t
, t0.
x
t
R
n
: the system state variable.
u R
m
: the system input variable u
t
R : the system input variable.
A
nn
and B
nm
are known matrices.
The initial value x
0
is known.
** Let c
t
=c(x
t
, u
t
) be a cost measure. Atypical optimal
control problem is to control the system using a
designed control sequence (time series) {u
0
,, u
T
}, such
HLW, Optimisationand Search, Lecture01, Slide 9 of 20

= =
= =
T
t
t t
T
t
t
u x c c J
0 0
) , ( ) u ( Minimize
g q ( ) {
0
, ,
T
},
that the total cost over the time instants {0, 1, , T} is
minimized, i.e.,
Prototype Prototype 5: Two Trivial Examples 5: Two Trivial Examples
** Consider a nonlinear 2D function
9 14 2 2 5 ) , (
2 1 2 1
2
2
2
1 2 1
+ + + = x x x x x x x x f
Do the function have a minimum point? p
If the minimum point exist, how to find it?
** Let u
1
,u
2
and y be the system inputs and output (response
or dependent variable). It is empirically known that the
relationship between u
1
,u
2
and y is given by:
u
1
u
2
y
HLW, Optimisationand Search, Lecture01, Slide 10 of 20
1 1 2 2 3 1 2
2
4 1
a u a u a u u
y
a u
+ +
=
+
u
1
u
2
y
u
1
(1) u
2
(1) y(1)
u
1
(2) u
2
(2) y(2)

u
1
(N) u
2
(N) y(N)
How to estimate the
parameters a
k
(k=1,2,3,4)
6
(II) What is Optimisation? (II) What is Optimisation?
** As far back as the 18th Century, the famous
Swiss mathematician and physicist
Leonhard Euler proclaimed that: p
... nothing nothing at all takes place in the Universe
in which some rule of maximum or minimum
does not appear.
** Optimisation is a process of seeking and selecting the best
l ti f t f candidate sol tions
Leonhard Euler
(1707-1783)
http://en.wikipedia.org/wiki/Leonhard_Euler
HLW, Optimisationand Search, Lecture01, Slide 11 of 20
** From mathematical point of view, optimisation is about
the study of maximising or minimising mathematical
functions.
solution from a set of candidate solutions.
Optimization and Mathematical Programming Optimization and Mathematical Programming
** Optimisation often goes by the name mathematical
programming.
** It should be stressed that the word programming here,
which is used in the sense of planning, should not be
confused with computer programming; it was merely a
coincidence that optimisation was given such a name.
HLW, Optimisationand Search, Lecture01, Slide 12 of 20
7
(III) How to Optimize? (III) How to Optimize?
** So far we know that optimisation is important for all
engineering areas and applied sciences. In fact, modelling
and optimisation are what engineering is all about, in the
th t sense that:
Once we have obtained a mathematical model of a system,
we can then select the system parameters in such a way
that the system executes its role optimally with respect a
particular performance criterion.
B h i i h fi d h b l i
HLW, Optimisationand Search, Lecture01, Slide 13 of 20
But the question is: how to find the best solution to a
given problem, so that we can optimally achieve our goal
(in terms of accuracy, time, money, energy, etc.) .
What Do We Learn? What Do We Learn?
** Formulate an optimization problem using models:
How to represent real-world optimization problems using
mathematical models? We will learn and discuss:
Linear Programming Model
Nonlinear Programming Model
HLW, Optimisationand Search, Lecture01, Slide 14 of 20
8
What Do We Learn and Discuss? (cont.) What Do We Learn and Discuss? (cont.)
** Understand Optimization & Search Methods and
Algorithms.
Necessary conditions
Sufficient conditions
Unconstrained optimization
Constrained optimization
HLW, Optimisationand Search, Lecture01, Slide 15 of 20
What Do We Learn and Discuss? ( What Do We Learn and Discuss? (Cont Cont.) .)
** Learn Methods, Algorithms, Toolkits:
Computational methods and algorithms for typical
optimization problems.
PART I: Classical Methods PART I: Classical Methods
Gradient methods
Newton methods
Quasi-Newton methods
Lagrange methods
Penalty methods
HLW, Optimisation and Search, Lecture01, Slide 16 of 26
PART II: Megaheuristic Search
Megaheuristic search methods
Evolutionary methods
Isaac Newton
(1642-1727)
J oseph-Louis Lagrange
(1736-1813)
9
References References
Introduction to Operations Research
F. S. Hillier, and G. J. Lieberman, McGraw Hill, 2005.
HLW, Optimisationand Search, Lecture01, Slide 17 of 20
Seventh Edition
Eighth Edition
References ( References (Cont Cont.) .)
Practical Optimization
Philip E. Gill, Walter Murray,
and Margaret H. Wright, g g
Academic Press, 1981.
Linear and Nonlinear Programming
D. G. Luenberger, and Y. Ye,
Springer, 2008 (3rd ed.).
HLW, Optimisation and Search, Lecture01, Slide 18 of 26
10
References ( References (Cont Cont.) .)
Nonlinear Optimization with
Engineering Application
Michael Bartholomew-Biggs Michael Bartholomew Biggs,
Springer-Verlag, 2008.
Linear and Nonlinear
Programming
S.G. Nash and A. Sofer,
HLW, Optimisationand Search, Lecture01, Slide 19 of 20
S.G. Nash and A. Sofer,
McGraw Hill, 1996.
References ( References (Cont Cont.) .)
Applied Optimization with
MATLAB Programming
P Venkataraman P. Venkataraman,
Wiely-Interscience, 2009.
Genetic Algorithms in Search,
HLW, Optimisationand Search, Lecture01, Slide 20 of 20
Genetic Algorithms in Search,
Optimzation & Machine
Learning
D. E. Goldberg,
Addison-Wesley, 1989.