Notes

Slides adopted from Prof. Gesualdo Scutari (SP 2016)

Lesson 5: Classification of Optimization Problems

IE 335: Operations Research - Optimization

Instructor: Professor Chris Quinn

August 31, 2016

Notes

Today
Classification of optimization problems

Based on characteristics of:

I

optimization variables

objective function

constraints

Notes

Decision variables: continuous vs integral

A decision variable is continuous if it takes real values

Example: a nonnegative variable is continuous, since it takes
values in [0, +)

A decision variable is integral (a.k.a. discrete) if it is restricted to a

specified interval of integers
Example: binary (0 or 1) variables are integral

Notes

Decision variables: continuous vs integral

Continuous
(1

(2)

x2

)
(3

Top view

(4)
(5)

Side view
1

x1

Notes

Decision variables: continuous vs integral

Integer
(1

(2)

x2

(4)

(3

Top view

(5)
3

Side view

x1

Notes

Decision variables: continuous vs integral

Mixed integer
(1

(2)

x2

(4)

(3

Top view

(5)
3

Side view
1

x1

Notes

A First Classification

optimization variables

optimization problem

real-valued

non-integer programming

integer
real-valued &
integer

integer programming
mixed-integer
programming

Notes

Linear/Affine and nonlinear functions

Linear/Affine

f (x)
3

3
8

Notes

Linear/Affine and nonlinear functions

Nonlinear

f (x)
3

3
9

Notes

Linear/Affine and nonlinear functions

Linear functions are functions that preserve scaling and addition of
the input argument
A function f : Rn R is linear if and only if
f (x)
= f (x),
x Rn , R
f (x1 + x2 ) = f (x1 ) + f (x2 ), x1 , x2 Rn
Equivalently, f is linear if and only if it can be written in the form
f (x1 , . . . , xn ) = c1 x1 + c2 x2 + + cn xn
= cT x
where c1 , . . . , cn are real constants and cT = [c1 , . . . , cn ]
(Recall notation: x = [x1 , , xn ]T is the (column) n-length vector of
the optimization (a.k.a. decision) variables)
10

Notes

A function f : Rn R is affine if and only if it can be expressed as

f (x1 , . . . , xn ) = cT x + b

(1)

for some b R
f in (1) is linear if and only if b = 0.
A function is nonlinear if it is not affine
f (x)

f (x)

Linear

Affine

11

Notes

Linear, Affine, or Nonlinear Function?

f (x1 , x2 , x3 ) = 9x1 17x3
f (x1 , x2 , x3 ) =

f (x1 , x2 , x3 ) =

5
+ 3x2 6x3
x1
3
X

xj + 3

linear
nonlinear

affine

j=1

f (x1 , x2 , x3 ) =

x1 x2
x2 + x3

f (x1 , x2 , x3 ) = x1 x2 + 3x3

nonlinear
nonlinear

f (x1 , x2 , x3 ) = x1 + (sin 4)x2 + 3x3 + 4

affine

f (x1 , x2 , x3 ) = x1 + sin(4 x2 ) + 3x3 + 4

nonlinear
12

Notes

Constraints: linear vs nonlinear

Given g : Rn R, a constraint in the form
g(x1 , . . . , xn ) 0

()

is
I

linear if g(x1 , . . . , xn ) is affine

nonlinear if g(x1 , . . . , xn ) is nonlinear

Same definition applies to equality constraints

Therefore, a set of m linear inequalities can always be written as
Ax b 0
where A Rmn and b Rm

13

Notes

Classification of (Some) Optimization Problems

Objective function f

Feasible set

arbitrary

variables in Rn

arbitrary
affine
f = cT x + b
arbitrary
affine
f = cT x + b

variables in Rn & finite

number of nonlinear
equalities/inequalities
variables in Rn & finite
number of linear
equalities/inequalities
integral variables & finite
number of nonlinear
equalities/inequalities
integral variables & finite
number of linear
equalities/inequalities

name of the
optimization problem
unconstrained
optimization
nonlinear
programming
linear
programming
nonlinear
(mixed-)integer
programming
linear
integer
programming
14

Notes

Optimization problems
Unconstrained

15

Notes

Optimization problems
Nonlinear program (nonlinear obj fun and nonlinear constraints)

16

Notes

Optimization problems
Nonlinear program (linear obj fun and nonlinear constraints)

17

Notes

Optimization problems
Mixed integer linear program

18

Notes

Optimization problems
Nonlinear integer program

19

Notes

Optimization problems
Linear program

20

Notes

Who am I?
min

12z1 + 4z3

s.t.

z1 z2 z3 = 1

nonlinear

z1 , z2 0
max
s.t.

3z1 + 14z2 + 7z3

linear

10z1 + 5z2 25 18z3

zj 0 for j = 1, . . . , 3

min
s.t.

7z1 z2 + 4z1 z3 + 2z1 z3

3
X

nonlinear integer

zj = 2

j=1

zj {0, 1, 2}

for j = 1, 2, 3

21

Notes

min

3w1 + 14w2 w3

nonlinear mixed-integer

s.t. w1 w2 1
w1 + w2 + w3 = 10
wj 0

for j = 1, . . . , 3

w1 integer
max
s.t.

19w1

linear

w1 w2
w2 + w3 = 10 + w1
w2 1, w3 1
w1 0

22

Notes

Quadratic Programming (QP)

QP is an important class of nonlinear programming. It can be view as
an extension of LP wherein the objective function is quadratic (and
thus nonlinear)
A quadratic function in a vector of variables x = [x1 , . . . , xn ]T is a
polynomial in x where the maximum degree of the monomials is equal
to two
I x R f (x) = ax2 + bx + c
I

x [x1 , x2 ]T R2
f (x) = a11 x21 + a22 x22 + a21 x1 x2 + a12 x1 x2 + b1 x1 + b2 x2 + c






 a11 a12

 x1
x1
+ b1 b2
+c
= x1 x2
x2
| {z } x2
| {z } a21 a22
|
{z
}| {z }
| {z }
xT
bT
x

xT A x

bT x

+c
23

Notes

Quadratic functions 1D
f (x)

24

Notes

Quadratic programs: 2D

25

Notes

Quadratic Programming (QP)

In a QP
I the optimization variables are real-valued
I the objective function is quadratic
I the constraints are linear
More formally
minimize
x
subject to

1 T
x A x + bT x + c
2
Hx d 0
Gx q = 0
x Rn

QP arise naturally from many applications in different fields such as

finance, signal processing, machine learning, etc. (more later in the
course)
26

Notes

Relation among Problems

nonlinear
programming
quadratic
programming
linear
programming

There are other important classes (such as convex programs) we will

discuss later in the semester if there is enough time.
27

Notes

Relative Tractability
Not all optimization problems are created equal. Some problem
classes can be numerically solved very efficiently; for some other
classes, no reliable efficient algorithm is known
In general:
I

linear constraints and objective functions are preferred to

nonlinear ones

continuous variables are preferred to integral variables

However,
I

Some nonlinear programs can be solved as quickly as a linear

program

Some integer linear programs can be solved as quickly as a

linear program
28

Notes

Relative Tractability
Tractability is often not a property of the problem itself, but of our
formulation and modeling
A problem that may seem hard under a certain formulation may well
become tractable if we put some more effort and intelligence in the
modeling phase.
Sometimes, tricks of the trade can be used to convert certain
nonlinear hard programs into simpler ones (e.g., linear programs)
Even for intrinsically hard problems, for which exact solutions maybe
unaffordable, approximate, or relaxed, solutions may be readily
computable

29

Notes

Summary
Classification of optimization models via characteristics of variables,
constraints, and objective functions
I

linear programs

nonlinear programs

quadratic programs

(pure and mixed) integer linear programs

(pure and mixed) integer nonlinear programs

Relative tractability
I

linear better than nonlinear

continuous better than integral

30