UUM 526 Optimization Techniques in Engineering Lecture 1: Introduction

UUM 526 Optimization Techniques in Engineering
Lecture 1: Introduction
Asst. Prof. N. Kemal Ure
Istanbul Technical University

ure@itu.edu.tr
February 5, 2019
Asst. Prof. N. Kemal Ure (ITU) Lecture 1 February 5, 2019 1 / 29

Overview
1 Introduction
2 Problem Formulation
3 Examples
4 Types of Optimization Problems
5 A Tour of the Course
6 Summary

Introduction
Introduction

Introduction
Introduction
▶ The main focus of this lecture is to give semi-formal answers to

following questions:
∎ What is optimization? Why do we need it?
∎ Is it useful for engineering?
∎ What kind of different optimization problems are out there?
▶ The objective is to give you a ”big picture” view of optimization and

motivate you for the rest of the class.

Problem Formulation
Problem Formulation

Problem Formulation
What is Optimization?
▶ Wikipedia: selection of a best element (with regard to some criteria)

from some set of available alternatives.
∎ Nice, but this is not useful. We need a formal (mathematical) description
▶ Mathematical Optimization (Programming)
∎ Set of elements (will be also called decision variables): x ∈ A
∎ Evaluate the cost of the element: f ∶ A → R
∎ Optimization Problem:
minimize f (x)
x∈A

Problem Formulation
Constrained and Unconstrained Optimization
▶ Generally A = Rn and we specify the constraints via functional

equalities and inequalities
▶ Unconstrained Problem: minimize f (x), f ∶ Rn → R
▶ Constrained Problem:
minimize f0 (x)
subject to fi (x) ≤ 0, i = 1, . . . , m
∎ Functions fi ∶ Rn → R represents constraints of the problem.

∎ We can embed the equality constraints by imposing fi (x) ≤ 0 and
−fi (x) ≤ 0 at the same time.
▶ Cool. But how is this connected to engineering?

Examples
Examples

Examples
Data Fitting
xi Experiment yi
▶ An experiment is conducted with inputs xi ∈ Rn . Outputs y i ∈ R are

observed, i = 1, . . . , ndata
▶ Your job is to find a linear function that explains the relationship

between x and y as good as possible
▶ Can we formulate this as an optimization problem? What is the set

of decisions? What is the cost function? What are the constraints?

Examples
Data Fitting
xi Experiment yi
▶ Let w ∈ Rn , so that y = wT x is a linear function. Here w is the

decision variable.
▶ The unconstrained formulation:
ndata
2
∑ [y − w x ]
i T i
minimize
i=1
▶ If we do not want the weights too large, we can add:

¿
Án 2
subject to ∥w∥2 = Á À∑ w ≤ R
i
i=1
∎ R is a positive number that constrains weights from growing too large

Examples
Spacecraft Control
▶ A satellite is orbiting the earth with

ability to control its orientation with
thrusters. The pitch angle of the
spacecraft is represented by θ ∈ [−π, π].
Let θ = [θ, θ̇].
▶ Spacecraft dynamics can be modeled as
a discrete time linear system
θk+1 = Aθk + Buk , k = 0, 1, . . . , tf
▶ Your job is to find a sequence of thruster commands uk such that

spacecraft is steered from θinit to θf inal in t = tf
▶ The fuel usage should be as low as possible. Thrusters can also exert
a limited amount of moment at per step.

Examples
Spaceraft Control
▶ Main idea: Objective function represents fuel usage, constraint

functions represent spacecraft dynamics and thruster limits
k=t −1
∑k=0 ∣uk ∣
f
minimize
subject to ∣uk ∣ ≤ umax , k = 0, . . . , tf − 1
θ0 − θinit = 0,
θtf − θf inal = 0,
θk+1 − Aθk − Buk = 0, k = 0, . . . , tf − 1
▶ Note that problem gets interesting when the dynamics are

time-variant A = Ak , B = Bk . We can handle that case too.
▶ Any optimal open loop linear control problem looks similar.

Examples
Design of a Wall Bracket
▶ Your job is to design a wall bracket that

supports W amount of load. The
material for the bars are fixed with
failure stress σa .
▶ You need to determine the cross
sectional areas A1 and A2 , such that the
total weight is minimized.
▶ Length of bars l1 and l2 can be
computed from geometry. Forces acting
bars F1 and F2 can be computed by
solving the statics equations

Examples
Design of a Wall Bracket
▶ Main idea: Cost function represents the weight, constraint functions

represent stress failure conditions and positivity of decision variables
minimize l1 A1 + l2 A2
subject to −A1 ≤ 0
−A2 ≤ 0,
A1 − σa ≤ 0,
F1
A2 − σa ≤ 0
F2
▶ Many other structural design problems can be formulated similarly

Examples
More Examples
▶ We will be looking at many other examples throughout the class

∎ Airline Operations: Origin-Destination Pairs, Crew Planning,
∎ Space Robotics: Shortest Path Problems, Descent Trajectory Generation
∎ Propulsion: Fuel Burning Rate, Nozzle Design
∎ Aerodynamics: Airfoil Design, CFD
∎ Avionics : Circuit Design, Sensor Design
▶ Good News: Almost any engineering problem can be cast as an

optimization problem!
▶ Bad News: We do not know how to efficiently solve the general

optimization problem. We need to understand the special cases we
can efficiently solve.

Types of Optimization Problems

Taxonomy of Optimization Problems
▶ The general problem is difficult for arbitrary f and fi
minimize f0 (x)
subject to fi (x) ≤ 0, i = 1, . . . , m
▶ There are certain exceptions which can be solved efficiently

∎ Least-squares problems
∎ Linear Programming
∎ Convex Optimization
▶ If you can formulate your problem as one of the above, you have
basically solved the problem
▶ Otherwise, you might need to develop new algorithms

Least-Squares Problems
minimize ∥Ax − b∥22
▶ Analytical solution exists! x∗ = (AT A)

−1
AT b
▶ There are efficient algorithms for computing the solution
▶ If A ∈ Rk×n , k ≥ n, runtime complexity is around n2 k
▶ It is easy to recognize least-squares problems
∎ Data Fitting is a least-squares problem!

∎ Just set A = [x1⊺ ; . . . ; xndata ⊺ ], b = [y 1 , . . . , y ndata ]⊺

Linear Programming
minimize c⊺ x
subject to a⊺i x ≤ bi , i = 1, . . . , m
▶ Cost and constraints are linear functions of the decision variable x

▶ No analytical solution
▶ There are efficient algorithms for computing the solution
▶ If x ∈ Rn , runtime complexity is around n2 m
▶ It is not as easy to recognize linear programming problems but there
are a few standard tricks
∎ Spacecraft Control is a linear programming problem! Cost function and
constraints are linear functions of uk
∎ Many other practical examples, such as airline scheduling.

Convex Optimization
▶ The general problem where f and fi are convex functions with

convex domains.
minimize f0 (x)
subject to fi (x) ≤ 0, i = 1, . . . , m
▶ Let θ1 , θ2 ≥ 0, θ1 + θ2 = 1
∎ Convex Function :f (θ1 x + θ2 y) ≤ θ1 f (x) + θ2 f (y)
∎ Convex Set: x, y ∈ A Ô⇒ θ1 x + θ2 y ∈ A
▶ Includes least-squares and linear programming as special cases
▶ Efficient polynomial time algorithms exist for finding global optimum,
even for nonlinear convex problems
▶ Can be difficult to recognize
∎ Wall Bracket design is a convex optimization problem!
∎ Many other important applications exist. Convex Optimization is at the
heart of modern applications.
Nonlinear Programming
▶ Back to the general form. f and fi are arbitrary (i.e. not convex)
minimize f0 (x)
subject to fi (x) ≤ 0, i = 1, . . . , m
▶ No general polynomial time algorithm exists for global optimum.

∎ Existing algorithms are either slow or get stuck in local optima
▶ Many challenging engineering problems are nonlinear and non-convex

∎ Nozzle design, almost anything that involve PDEs
▶ Convex approximations are available! (Major research trust)

▶ Focus is more on developing efficient algorithms rather than
formulating problems

A Tour of the Course

▶ Review of Mathematical Preliminaries

∎ Linear Algebra
∎ Geometry
∎ Multivariable Calculus
▶ Basics of Optimization
∎ Local Minimizers
∎ First and Second Order Conditions

▶ One Dimensional Methods

∎ Golden Search, Bisection Method
∎ Newton’s and Secant Method
▶ Gradient Methods
∎ Steepest Descent
∎ Convergence Analysis
▶ Newtons’ Method
▶ Conjugate Gradient Methods
▶ Quasi-Newton Methods
∎ DPF and BFGS algorithms

▶ Least Squares Problems

∎ Exact Solutions
∎ Recursive Least Squares
∎ Minimum Norm Problems
▶ Global Search Methods

∎ Nelder-Mead Algorithm
∎ Simulated Annealing
∎ Particle Swarm Optimization
∎ Genetic Algorithms

▶ Linear Programming
∎ Basic Solutions
∎ Geometry of LPs
▶ Simplex Method
∎ Canonical Augmented Matrix
∎ Two-Phase Simplex Algorithm
▶ Duality
▶ Integer Programming
∎ Branch and Bound
∎ Cutting Plane Methods

▶ Constrained Nonlinear Optimization

∎ Equality and Inequality Constraints
∎ KKT Conditions
▶ Convex Optimization
∎ Convex Sets and Functions
∎ Semidefinite Programming
▶ Algorithms for Constrained Optimization
∎ Projection Methods
∎ Lagrangian Methods
▶ Multiobjective Optimization
∎ Pareto Front
∎ Uncertain Linear Programs
Summary
Summary

Summary
Summary
▶ Now we know:
∎ The formal definition of an optimization problem
∎ How most (if not all) engineering problems can be converted into
optimization problems
∎ The most important types of optimization problems we can attack (least

squares, linear programming, convex programming)
▶ What is next?
∎ Reviewing the mathematics we are going to use for the rest of the class.

UUM 526 Optimization Techniques in Engineering Lecture 1: Introduction

Caricato da

Informazioni sul documento

Titolo originale

Copyright

Formati disponibili

Condividi questo documento

Condividi o incorpora il documento

Opzioni di condivisione

Hai trovato utile questo documento?

Questo contenuto è inappropriato?

Copyright:

Formati disponibili

UUM 526 Optimization Techniques in Engineering Lecture 1: Introduction

Caricato da

Copyright:

Formati disponibili

UUM 526 Optimization Techniques in Engineering

Asst. Prof. N. Kemal Ure

Istanbul Technical University

Asst. Prof. N. Kemal Ure (ITU) Lecture 1 February 5, 2019 1 / 29

4 Types of Optimization Problems

5 A Tour of the Course

Asst. Prof. N. Kemal Ure (ITU) Lecture 1 February 5, 2019 2 / 29

Asst. Prof. N. Kemal Ure (ITU) Lecture 1 February 5, 2019 3 / 29

▶ The main focus of this lecture is to give semi-formal answers to

∎ What is optimization? Why do we need it?

∎ Is it useful for engineering?

∎ What kind of different optimization problems are out there?

▶ The objective is to give you a ”big picture” view of optimization and

Asst. Prof. N. Kemal Ure (ITU) Lecture 1 February 5, 2019 4 / 29

Asst. Prof. N. Kemal Ure (ITU) Lecture 1 February 5, 2019 5 / 29

▶ Wikipedia: selection of a best element (with regard to some criteria)

∎ Nice, but this is not useful. We need a formal (mathematical) description

▶ Mathematical Optimization (Programming)

∎ Set of elements (will be also called decision variables): x ∈ A

∎ Evaluate the cost of the element: f ∶ A → R

Asst. Prof. N. Kemal Ure (ITU) Lecture 1 February 5, 2019 6 / 29

Constrained and Unconstrained Optimization

▶ Generally A = Rn and we specify the constraints via functional

∎ Functions fi ∶ Rn → R represents constraints of the problem.

▶ Cool. But how is this connected to engineering?

Asst. Prof. N. Kemal Ure (ITU) Lecture 1 February 5, 2019 7 / 29

Asst. Prof. N. Kemal Ure (ITU) Lecture 1 February 5, 2019 8 / 29

▶ An experiment is conducted with inputs xi ∈ Rn . Outputs y i ∈ R are

▶ Your job is to find a linear function that explains the relationship

▶ Can we formulate this as an optimization problem? What is the set

Asst. Prof. N. Kemal Ure (ITU) Lecture 1 February 5, 2019 9 / 29

▶ Let w ∈ Rn , so that y = wT x is a linear function. Here w is the

▶ If we do not want the weights too large, we can add:

∎ R is a positive number that constrains weights from growing too large

▶ A satellite is orbiting the earth with

θk+1 = Aθk + Buk , k = 0, 1, . . . , tf

▶ Your job is to find a sequence of thruster commands uk such that

Asst. Prof. N. Kemal Ure (ITU) Lecture 1 February 5, 2019 11 / 29

▶ Main idea: Objective function represents fuel usage, constraint

▶ Note that problem gets interesting when the dynamics are

Asst. Prof. N. Kemal Ure (ITU) Lecture 1 February 5, 2019 12 / 29

Design of a Wall Bracket

▶ Your job is to design a wall bracket that

Asst. Prof. N. Kemal Ure (ITU) Lecture 1 February 5, 2019 13 / 29

Design of a Wall Bracket

▶ Main idea: Cost function represents the weight, constraint functions

▶ Many other structural design problems can be formulated similarly

Asst. Prof. N. Kemal Ure (ITU) Lecture 1 February 5, 2019 14 / 29

▶ We will be looking at many other examples throughout the class

▶ Good News: Almost any engineering problem can be cast as an

▶ Bad News: We do not know how to efficiently solve the general

Asst. Prof. N. Kemal Ure (ITU) Lecture 1 February 5, 2019 15 / 29

Types of Optimization Problems

Asst. Prof. N. Kemal Ure (ITU) Lecture 1 February 5, 2019 16 / 29

Taxonomy of Optimization Problems

▶ The general problem is difficult for arbitrary f and fi

▶ There are certain exceptions which can be solved efficiently

Asst. Prof. N. Kemal Ure (ITU) Lecture 1 February 5, 2019 17 / 29

minimize ∥Ax − b∥22

▶ Analytical solution exists! x∗ = (AT A)

▶ There are efficient algorithms for computing the solution

▶ If A ∈ Rk×n , k ≥ n, runtime complexity is around n2 k

▶ It is easy to recognize least-squares problems