Sei sulla pagina 1di 48

EE127: Optimization Models

Vu Pham

Electrical Engineering and Computer Science


University of California, Berkeley
vu@eecs.berkeley.edu
Outline

Linear Programming Midterm Review


Quiz
Quadratic Programming Robust Quadratic Program
Quiz
Semidefinite Programming
Second Order Cone Programming Quiz
Quiz Math background
SOCP Review Duality. Lagrangian
Quiz Quiz

V. Pham EE127: Optimization Models Linear Programming 2


Linear Programming

General form:
minimize
n
cT x + d
x∈R
subject to Gx  h
Ax = b

Standard form:
minimize
n
cT x
x∈R
subject to Ax = b
x0

V. Pham EE127: Optimization Models Linear Programming 3


Quiz. Convert LP from general form to standard form.

V. Pham EE127: Optimization Models Linear Programming 4


Outline

Linear Programming Midterm Review


Quiz
Quadratic Programming Robust Quadratic Program
Quiz
Semidefinite Programming
Second Order Cone Programming Quiz
Quiz Math background
SOCP Review Duality. Lagrangian
Quiz Quiz

V. Pham EE127: Optimization Models Quadratic Programming 5


Quadratic Programming

With affine constraints:


minimize
n
(1/2)xT P x + q T x + r
x∈R
subject to Gx  h
Ax = b

where P  0.

With quadratic constraints:

minimize
n
(1/2)xT P0 x + q0T x + r0
x∈R
subject to (1/2)xT Pi x + qiT x + ri ≤ 0 i = 1, . . . , m
Ax = b

where Pi  0 for i = 0 . . . m.

V. Pham EE127: Optimization Models Quadratic Programming 6


Quiz. Give explicit solution to the optimization problem

minimize
n
xT Bx
x∈R
subject to xT Ax ≤ 1

with A  0 and B symmetric.

Application. Canonical Correlation Analysis (CCA), Linear Discriminant


Analysis (LDA). Further reading:
http://machinelearning.org/archive/icml2009/papers/315.pdf

V. Pham EE127: Optimization Models Quadratic Programming::Quiz 7


Outline

Linear Programming Midterm Review


Quiz
Quadratic Programming Robust Quadratic Program
Quiz
Semidefinite Programming
Second Order Cone Programming Quiz
Quiz Math background
SOCP Review Duality. Lagrangian
Quiz Quiz

V. Pham EE127: Optimization Models Second Order Cone Programming 8


Second Order Cone Programming

minimize
n
fT x
x∈R
subject to kAi x + bi k2 ≤ cTi x + di , i = 1, . . . , m

Second order cone program subsumes:


• QCQP: when ci = 0 and squaring
the constraints
• LP: when Ai = 0

V. Pham EE127: Optimization Models Second Order Cone Programming 9


Quiz. Formulate the following problem as an SOCP (λ, ε > 0)

minimize
n
kAx − bk2 + λkxk1 + εkxk2
x∈R

V. Pham EE127: Optimization Models Second Order Cone Programming::Quiz 10


Quiz. A second order cone constraint on optimization variable x has the
form:
kAffine(x)k2 ≤ Scalar-valued-affine(x)
Given optimization variable x ∈ Rn and nonnegative scalars α, β ≥ 0,
transform the following hyperbolic constraint into a second order cone con-
straint:
xT x ≤ αβ

V. Pham EE127: Optimization Models Second Order Cone Programming::Quiz 11


Quiz. Given an optimization problem
kAx−bk1
minimize
n cT x+d
x∈R
subject to kxk∞ ≤ 1

with d > kck1 so that cT x + d > 0 for all kxk∞ ≤ 1. Formulate it as a


convex optimization problem.

V. Pham EE127: Optimization Models Second Order Cone Programming::Quiz 12


Outline

Linear Programming Midterm Review


Quiz
Quadratic Programming Robust Quadratic Program
Quiz
Semidefinite Programming
Second Order Cone Programming Quiz
Quiz Math background
SOCP Review Duality. Lagrangian
Quiz Quiz

V. Pham EE127: Optimization Models SOCP Review 13


Quiz. Tranform the l4 -norm optimization problem as QCQP:

kAx − bk4 , m T 4 1/4


P
minimize
n
x∈R i=1 ((ai x − bi ) )

V. Pham EE127: Optimization Models SOCP Review::Quiz 14


Quiz. Transform the convex quadratic constraint into a second order cone
constraint:
xT AT Ax + bT x + c ≤ 0
(note that AT A  0)

V. Pham EE127: Optimization Models SOCP Review::Quiz 15


Quiz. Our goal is to find a linear seperation of two sets of ellipsoids. Given
K + L ellipsoids:

Ei := Pi u + qi | kuk2 ≤ 1 , i = 1, . . . , K + L

where Pi ’s are symmetric. Our problem is to find a hyperplane that strictly


seperates the first K ellipsoids and the last L ellipsoids:

aT x + b > 0 for x ∈ E1 , . . . EK ,
aT x + b < 0 for x ∈ EK+1 , . . . EK+L

Formulate this as a feasibility problem.

V. Pham EE127: Optimization Models SOCP Review::Quiz 16


Outline

Linear Programming Midterm Review


Quiz
Quadratic Programming Robust Quadratic Program
Quiz
Semidefinite Programming
Second Order Cone Programming Quiz
Quiz Math background
SOCP Review Duality. Lagrangian
Quiz Quiz

V. Pham EE127: Optimization Models Midterm Review 17


Quiz (Geometry problem). Given m + 1 points in Rn : p0 , p1 , . . . , pm .
Formulate the set of points closer (in Euclidean norm) to p0 than to the
remaining points.

P := {x ∈ Rn : ∀i = 1, . . . , m, kx − p0 k2 ≤ kx − pi k2 }

V. Pham EE127: Optimization Models Midterm Review::Quiz 18


Quiz (Linear Algebra). Let A, B ∈ Rm×n and R(A) ⊥ R(B). Prove
that: kA + Bk2F = kAk2F + kBk2F .
Hint: show that
1. AT B = 0
2. kA + Bk2F = kAk2F + kBk2F

V. Pham EE127: Optimization Models Midterm Review::Quiz 19


Quiz. Given A square and nonsingular:

minimize
n
cT x
x∈R
subject to Ax  b

Find the closed form expression for the optimal value.

V. Pham EE127: Optimization Models Midterm Review::Quiz 20


Quiz. Given A  0 and b, are two constraints Ax  b and x  A−1 b
equivalent (all dimensions match)? Prove or give a counter-example.

V. Pham EE127: Optimization Models Midterm Review::Quiz 21


Quiz (Maximize harmonic mean). Given domain {x | Ax  b} and aTi is
the i-th row of A, formulate the following problem as a convex optimization
problem:
( m T −1
P
maximize
n
x∈R i=1 1/(ai x − bi ))

V. Pham EE127: Optimization Models Midterm Review::Quiz 22


Quiz (Maximize geometric mean). Given domain {x | Ax  b} and aTi is
the i-th row of A, formulate the following problem as a convex optimization
problem:
( m T −1/m
Q
maximize
n
x∈R i=1 (ai x − bi ))

V. Pham EE127: Optimization Models Midterm Review::Quiz 23


Outline

Linear Programming Midterm Review


Quiz
Quadratic Programming Robust Quadratic Program
Quiz
Semidefinite Programming
Second Order Cone Programming Quiz
Quiz Math background
SOCP Review Duality. Lagrangian
Quiz Quiz

V. Pham EE127: Optimization Models Robust Quadratic Program 24


Robust QP

Consider a convex quadratic program:

minimize
n
(1/2)xT P x + q T x + r
x∈R
subject to Gx  h

where (q, r, G, h) are exactly known while the matrix P is subject to error.

V. Pham EE127: Optimization Models Robust Quadratic Program 25


Robust QP

Consider a convex quadratic program:

minimize
n
(1/2)xT P x + q T x + r
x∈R
subject to Gx  h

where (q, r, G, h) are exactly known while the matrix P is subject to error.

We then make a robust quadratic program:

supP ∈E (1/2)xT P x + q T x + r

minimize
n
x∈R
subject to Gx  h

where E is the set of possible matrices P .

V. Pham EE127: Optimization Models Robust Quadratic Program 25


Quiz. Consider a robust QP

supP ∈E (1/2)xT P x + q T x + r

minimize
n
x∈R
subject to Gx  h

where E is a finite set of PSD matrices: E = {P1 , . . . , PK }, Pi  0.


Formulate this robust QP as a convex problem.

V. Pham EE127: Optimization Models Robust Quadratic Program 26


Quiz. Consider a robust QP

supP ∈E (1/2)xT P x + q T x + r

minimize
n
x∈R
subject to Gx  h

where E is specified by a nominal PSD P0 with a bound on the eigenvalues


of the deviation P − P0 by a real value γ:

E = {P symmetric | − γI  P − P0  γI}

Formulate this robust QP as a convex problem.

V. Pham EE127: Optimization Models Robust Quadratic Program 27


Outline

Linear Programming Midterm Review


Quiz
Quadratic Programming Robust Quadratic Program
Quiz
Semidefinite Programming
Second Order Cone Programming Quiz
Quiz Math background
SOCP Review Duality. Lagrangian
Quiz Quiz

V. Pham EE127: Optimization Models Semidefinite Programming 28


Semidefinite Programming

LMI form. Given symmetric matrices F1 , . . . , Fn of the same size Rk×k :

minimize
n
cT x
x∈R
subject to x1 F1 + . . . + xn Fn + G  0
Ax = b

The inequality above is the Linear Matrix Inequality (LMI).

Conic form. Given symmetric matrices C, A1 , . . . , Ap of the same size as


optimization matrix variable X, we minimize over a positive semidefinite cone:

minimize hC, Xi
X
subject to hAi , Xi = bi
X0

V. Pham EE127: Optimization Models Semidefinite Programming 29


Quiz. Convert an LP problem into an SDP:

minimize
n
cT x + d
x∈R
subject to Gx  h
Ax = b

V. Pham EE127: Optimization Models Semidefinite Programming::Quiz 30


Quiz (Multiple LMIs and affine inequality) Given a program with multiple
LMIS and one affine inequality:

minimize
n
cT x
x∈R
(i) (i)
subject to L(i) (x) := x1 F1 + . . . + xn Fn + G(i)  0, i = 1, . . . , K
Ax = b
Gx  0

Convert this program into the standard LMI form of SDP.

V. Pham EE127: Optimization Models Semidefinite Programming::Quiz 31


Quiz. Convert a QP problem into an SDP:

minimize
n
xT P x + q T x + r
x∈R
subject to Gx  h
Ax = b

V. Pham EE127: Optimization Models Semidefinite Programming::Quiz 32


Quiz. Convert an SOCP problem into an SDP:

minimize
n
fT x
x∈R
subject to kAi x + bi k2 ≤ cTi x + di , i = 1, . . . , m

V. Pham EE127: Optimization Models Semidefinite Programming::Quiz 33


Quiz. Given an affine function A : Rn → Sm with

A(x) := A0 + x1 A1 + + . . . + xn An .

Convert the following problem as an SDP:


1. Minimize the maximum eigenvalue λmax (A(x)) over x.
2. Minimize the spread of the eigenvalue λmax (A(x)) − λmin (A(x))
over x.

V. Pham EE127: Optimization Models Semidefinite Programming::Quiz 34


Quiz (Matrix norm minimization) Given A0 , A1 , . . . An ∈ Rp×q , consider
the program:

minimize
n
kA0 + x1 A1 + + . . . + xn An k2
x∈R

Convert this program into an SDP.

V. Pham EE127: Optimization Models Semidefinite Programming::Quiz 35


Outline

Linear Programming Midterm Review


Quiz
Quadratic Programming Robust Quadratic Program
Quiz
Semidefinite Programming
Second Order Cone Programming Quiz
Quiz Math background
SOCP Review Duality. Lagrangian
Quiz Quiz

V. Pham EE127: Optimization Models Math background 36


Schur complement: Convex unconstrained QP

Quiz. Give closed form expression for the optimal variable x∗ and optimal
value p∗ :
1 T
minimize
n 2
x P x + qT x + r
x∈R

given P  0.

V. Pham EE127: Optimization Models Math background 37


Schur complement

Quiz. Given A  0 and v is fixed, give a closed form expression for the
optimal variable and optimal value:
 T   
u A B u
minimize
u∈Rn v BT C v

Facts.
• X  0 ⇐⇒ A  0, S(A)  0
• If already know A  0, then X  0 ⇐⇒ S(A)  0

V. Pham EE127: Optimization Models Math background 38


Schur complement: block elimination

 
A11 A12
Quiz. Given X = and A11 nonsingular. Prove that
A21 A22

det(X) = det(A11 ) det(S(A11 ))

V. Pham EE127: Optimization Models Math background 39


 
A11 A12
Quiz. Given X = and A11 nonsingular. Prove that
A21 A22

det(X) = det(A11 ) det(S(A11 ))

V. Pham EE127: Optimization Models Math background 40


Outline

Linear Programming Midterm Review


Quiz
Quadratic Programming Robust Quadratic Program
Quiz
Semidefinite Programming
Second Order Cone Programming Quiz
Quiz Math background
SOCP Review Duality. Lagrangian
Quiz Quiz

V. Pham EE127: Optimization Models Duality. Lagrangian 41


Duality. Lagrangian.

The optimization problem (no assumption on convexity) with domain D:

minimize
n
f0 (x)
x∈R
subject to fi (x) ≤ 0, i = 1, . . . , m
hj (x) = 0, j = 1, . . . p
Lagrangian (most of the case we want λ  0):
m p
X X
L(x, λ, µ) = f0 (x) + λi fi (x) + µj hj (x)
i=1 j=1

Lagrange dual function (always concave, can be −∞ for some λ, µ):

g(λ, µ) = min L(x, λ, µ)


x∈D

Lower bound on the primal optimal value:

λ  0 =⇒ g(λ, µ) ≤ p∗

V. Pham EE127: Optimization Models Duality. Lagrangian 42


Quiz. Find the dual function of the least-norm solution problem:

minimize
n
xT x
x∈R
subject to Ax = b

V. Pham EE127: Optimization Models Duality. Lagrangian::Quiz 43


Quiz. Find the dual function of the LP
minimize
n
cT x
x∈R
subject to Ax = b
x0

Lagrange Dual problem.

maximize g(λ, µ)
λ∈Rn ,µ∈Rm
subject to λ0

Often simplified by making implicit constraints in g(λ, µ) explicit. Example:

maximize
m
−bT µ
µ∈R
subject to AT µ + c  0

V. Pham EE127: Optimization Models Duality. Lagrangian::Quiz 44


Quiz. Given Fi , G symmetric (Fi , G ∈ Sk ), derive a dual problem for:

minimize
n
cT x
x∈R
subject to x1 F1 + . . . xn Fn − G  0

V. Pham EE127: Optimization Models Duality. Lagrangian::Quiz 45


Quiz. Derive a dual problem for
Pm
minimize
n
− i=1 log(bi − aTi x)
x∈R

given domain {x : aTi x < bi , i = 1, . . . , m}.

The solution to the above problem is the analytic center, used in barrier
methods.

V. Pham EE127: Optimization Models Duality. Lagrangian::Quiz 46


Quiz. Derive a dual problem for
PN
minimize
n i=1 kAi x + bi k2 + 12 kx − x0 k22
x∈R

given Ai ∈ Rmi ×n , bi ∈ Rmi , and x0 ∈ Rn .

V. Pham EE127: Optimization Models Duality. Lagrangian::Quiz 47

Potrebbero piacerti anche