Convex Optimization Theory

and
Applications
Lecture 1
Introduction
Prof. Chun-Hung Liu
Dept. of Electrical and Computer Engineering
National Chiao Tung University
Fall 2016

2016/9/21 Lecture 1: Introduction 1

 Course Information
Instructor: Prof. Chun-Hung Liu ()
Email: chungliu@nctu.edu.tw
Phone: 03-571-2121 ext. 54512
Office: ED 703

Class Website: e3 campus http://web.it.nctu.edu.tw/~chungliu/

Textbooks: Convex Optimization, S. Boyd and and L. Vandenberghe,
Cambridge University, 2004. (This book is downloadable)
All course materials are on the above website.
Prerequisites: Undergraduate linear algebra and/or probability

Office Hours: 9:00am~12pm on Wednesday (But, you are always

welcome to talk to me when you see me in my office)
Teaching Assistant: (ED 717)

(* Part of the lecture slides is based on the slides of Prof. S. Boyd L. Vandenberghe)
2016/9/21 Lecture 1: Introduction 2
 Course Grading and Coverage
Grading
Homework (20%)
Including written and computer simulation problems
Midterm Exams (45%)
Two short exams in class
Final Exam/Project (35~40%)
Which one would you like?

Categories of Optimization Theory

Linear programming
Nonlinear programming
Convex/Non-Convex optimization
Linear and Nonlinear
More advanced optimization
Dynamic programming
Stochastic dynamic programming, Markov decision processes

2016/9/21 Lecture 1: Introduction 3

 Introduction
Mathematical optimization

Least-squares and linear programming

Convex optimization

Example

Linear programming

Nonlinear optimization

Brief history of convex optimization

2016/9/21 Lecture 1: Introduction 4

 Mathematical Optimization

2016/9/21 Lecture 1: Introduction 5

 Examples
Portfolio optimization
variables: amounts invested in different assets
constraints: budget, max./min. investment per asset,
minimum Return
objective: overall risk or return variance
Device sizing in electronic circuits
variables: device widths and lengths
constraints: manufacturing limits, timing requirements,
maximum area
objective: power consumption
Data fitting
variables: model parameters
constraints: prior information, parameter limits
objective: measure of misfit or prediction error
2016/9/21 Lecture 1: Introduction 6
 Solving Optimization Problems
General optimization problem
very difficult to solve (Does this truth frustrate you?)
methods involve some compromise, e.g., very long
computation time, or not always finding the solution

Exceptions: Certain problem classes can be

solved efficiently and reliably
least-squares problems
linear programming problems
convex optimization problems

2016/9/21 Lecture 1: Introduction 7

 Least-Squares

2016/9/21 Lecture 1: Introduction 8

 Linear Programming

2016/9/21 Lecture 1: Introduction 9

 Convex Optimization Problem

2016/9/21 Lecture 1: Introduction 10

 Solving Optimization Problems

2016/9/21 Lecture 1: Introduction 11

 Solving Optimization Problems

2016/9/21 Lecture 1: Introduction 12

 Example

2016/9/21 Lecture 1: Introduction 13

 How to Solve?

2016/9/21 Lecture 1: Introduction 14

 How to Solve?

2016/9/21 Lecture 1: Introduction 15

 Nonlinear Optimization
Traditional techniques for general nonconvex problems
involve compromises
local optimization methods (nonlinear programming)
find a point that minimizes f0 among feasible points near it
fast, can handle large problems
require initial guess
provide no information about distance to (global) optimum
global optimization methods
find the (global) solution
worst-case complexity grows exponentially with problem
size
these algorithms are often based on solving convex
subproblems

2016/9/21 Lecture 1: Introduction 16

 Brief History of Convex Optimization
Theory (convex analysis): ca19001970
Algorithms
1947: simplex algorithm for linear programming (Dantzig)
1960s: early interior-point methods (Fiacco & McCormick,
Dikin, . . . )
1970s: ellipsoid method and other subgradient methods
1980s: polynomial-time interior-point methods for linear
programming (Karmarkar 1984)
late 1980snow: polynomial-time interior-point methods for
nonlinear
convex optimization (Nesterov & Nemirovski 1994)
Applications
before 1990: mostly in operations research; few in engineering
since 1990: many new applications in engineering (control, signal
processing, communications, circuit design, . . . ); new problem
classes
(semidefinite and second-order cone programming, robust
optimization)
2016/9/21 Lecture 1: Introduction 17
 Some Useful Reviews

2016/9/21 Lecture 1: Introduction 18

 Vector Operations

2016/9/21 Lecture 1: Introduction 19

 Inner Products

2016/9/21 Lecture 1: Introduction 20

 Euclidean Norm

2016/9/21 Lecture 1: Introduction 21

 Inner Products and Angles

2016/9/21 Lecture 1: Introduction 22

 Cauchy-Schwarz Inequality

2016/9/21 Lecture 1: Introduction 23

 Hyperplanes

2016/9/21 Lecture 1: Introduction 24

 Halfspaces

2016/9/21 Lecture 1: Introduction 25

 Affine Sets

2016/9/21 Lecture 1: Introduction 26

 Polyhedra

2016/9/21 Lecture 1: Introduction 27

 Polyhedra

2016/9/21 Lecture 1: Introduction 28

 Examples of Polyhedra

2016/9/21 Lecture 1: Introduction 29

 Sets of Linear Equations

2016/9/21 Lecture 1: Introduction 30

 Polyhedron (inequality form)

2016/9/21 Lecture 1: Introduction 31

 Extreme Points and Vertices

2016/9/21 Lecture 1: Introduction 32

 Basic Feasible Solutions

2016/9/21 Lecture 1: Introduction 33

 Example

2016/9/21 Lecture 1: Introduction 34

 Convex Functions

2016/9/21 Lecture 1: Introduction 35

 Convex Sets

2016/9/21 Lecture 1: Introduction 36

 Closed and Open Sets
We say that x is a closure point of a subset X of Rn if there exists a
sequence {xk } X that converges to x. The closure of X, denoted cl(X),
is the set of all closure points of X.

Definition of Closed Set: A subset X of Rn is called closed if it is equal to

its closure. It is called open if its complement, {x|x 2
/ X} , is closed. It is
called bounded if there exists a scalar c such that kxk c for all x 2 X .
It is called compact if it is closed and bounded.

For any > 0 and x 2 Rn , consider the sets

{x|kx x k < }, {x|kx x k }.
The first set is open and is called an open ball centered at x, while the
second set is closed and is called a closed ball centered at x.

A subset X of Rn is open if and only if for every x 2 X there is an open

ball that is centered at x and is contained in X. A neighborhood of a
vector x is an open set containing x.
2016/9/21 Lecture 1: Introduction 37
 Interior Point and Boundary Point
We say that x is an interior point of a subset X of Rn if there exists a
neighborhood of x that is contained in X. The set of all interior points
of X is called the interior of X, and is denoted by int(X).

A vector x 2 cl(X) which is not an interior point of X is said to be a

boundary point of X. The set of all boundary points of X is called the
boundary of X.

A set is open if and only if all of its elements are interior points.

2016/9/21 Lecture 1: Introduction 38

 Supremum and Infimum of Sets
The supremum of a nonempty set X of scalars, denoted by sup X , is
defined as the smallest scalar y such that y x for all x 2 X. If no
such scalar exists, we say that the supremum of X is 1 .
If sup X is equal to a scalar x that belongs to the set, we say that
x is the maximum point of X and we write x = max X.

Similarly, the infmum of X, denoted by inf X , is defined as the largest

scalar y such that y x for all x 2 X , and is equal to 1 if no such
scalar exists.
Similarly, if inf X is equal to a scalar x
that belongs to the set X,
we say that x is the minimum point of X and we write x = min X.

Thus, when we write max X (or min X) in place of sup X (or inf X ,
respectively), we do so just for emphasis: we indicate that it is either
evident, or it is known through earlier analysis, or it is about to be shown
that the maximum (or miminum, respectively) of the set X is attained at
one of its points.
2016/9/21 Lecture 1: Introduction 39
 Continuity

Let X be a subset of Rn

A function f :! Rm is called continuous at a vector x 2 X if

lim f (z) = f (x)
z!x

A function f :! Rm is called right-continuous (respectively, left-

continuous) at a vector x 2 X if lim f (z) = f (x) (respectively,
z#x
lim f (z) = f (x) )
z"x

A real-valued function f : X ! R is called upper semicontinuous

(respectively, lower semicontinuous) at a vector x 2 Xif
f (x) lim sup f (xk ) (respectively, f (x) lim inf f (xk ) for every
k!1
k!1
sequence {xk } X that converges to x. )

2016/9/21 Lecture 1: Introduction 40