Deterministic

Optimization
Discrete Optimization:
Introduction
Shabbir Ahmed
Anderson-Interface Chair and Professor
School of Industrial and Systems Engineering

Why integer variables?

Why integer variables?

Learning Objectives
• Recall discrete optimization models
• Recognize the need for discrete
variables
Discrete Optimization Models
• A discrete optimization or an integer programming model is an
optimization problem where some of the decisions variables are
required to take integer values

min f (x)
s.t. gi (x)  bi i = 1, . . . , m
x 2 Rn p ⇥ Z p
Linear Discrete Optimization
Models
• If the objective and constraints of a discrete optimization problem
consist of linear functions then it is known as a linear discrete
optimization problem or a mixed integer linear program

min c> x
s.t. Ax b
x 2 Rn p ⇥ Z p
Binary Optimization Models
• A special and important class of discrete optimization models are
those where the discrete variables are required to be binary, that is,
they are required to take values of 0 or 1.

min f (x)
s.t. gi (x)  bi i = 1, . . . , m
x 2 Rn p ⇥ {0, 1}p
Why discrete variables?

• Indivisible decisions
• Yes/No choices
• Logical conditions
• Nonconvex functions and sets
Etc.
Example: Indivisible Decisions
• Given a set of supply ports and a set of demand ports with given
supplies and demands, how many cargo ships to lease to for each
route (supply-demand pair) at minimum cost?

P P
min c y
Pi2I j2J ij ij
s.t. Pi2I xij dj 8j2J
j2J xij  si 8i2I
xij  U yij 8i 2 I, j 2 J
xij 0, yij 2 Z+ 8i 2 I, j 2 J
Example: Yes/No Decisions
• Given a set of projects with their returns and costs, and a budget,
decide whether to investment in a project or not to maximize returns

n
X
max rj x j
j=1
Xn
s.t. c j xj  B
j=1
xj 2 {0, 1} 8 j = 1, . . . , n
Example: Logical Conditions
• If I invest in project 1 then I must invest in projects 2 and 3

n
X
max rj x j
j=1
Xn
s.t. c j xj  B
j=1
x1  x2
x1  x3
xj 2 {0, 1} j = 1, . . . , n
 Example: Nonconvex Function
• Decide how much to buy from a set of different vendors (offering
quantity discounts) to satisfy a total need
n $0.5/unit
X
min fj (xj )
j=1
$1/unit
s.t. Ax b
0  xj  Uj j = 1, . . . , n

⇢ 0
pj x j 0  xj  Lj 100 150
fj (xj ) =
pj Lj + qj (xj Lj ) Lj  xj  U j
Example: Nonconvex Function
• Decide how much to buy from a set of different vendors (offering
quantity discounts) to satisfy a total need
n
X $0.5/unit
min [pj uj + pj Lj yj + qj (vj Lj )]
j=1
s.t. Ax b $1/unit
0  xj  Uj j = 1, . . . , n
xj = uj + vj j = 1, . . . , n
0  uj  Lj (1 yj ) j = 1, . . . , n
Lj yj  vj  Uj yj j = 1, . . . , n 0 100 150
yj 2 {0, 1} j = 1, . . . , n
Summary
• Discrete variables arise from the
need to model indivisible
decisions, yes/no choice, logical
relations, nonconvex
functions/sets
• Optimization models with binary
variables are very important class
of discrete optimization problems
Deterministic
Optimization
Discrete Optimization:
Introduction
Shabbir Ahmed
Anderson-Interface Chair and Professor
School of Industrial and Systems Engineering

Discrete Optimization Challenges

Discrete Optimization:
Challenges

Learning Objectives
• Recognize difficulties of solving
discrete optimization problems
Nonconvexity
• Discrete optimization involves optimization over a nonconvex set of
feasible solutions
Local vs Global Optima
• Local optimal solutions may be very far from a global optimal solution
• Example:
min x + y
s.t. 100x + y 99
x, y 2 Z+

• The solution (1,0) is a global optimal solution with objective value 1

• The solution (0,99) is a “local” optimal solution with objective 99. It
has equal or better objective than all “neighboring” feasible integer
solutions.
Enumeration
• Small discrete optimization problems can be solved by enumerating
all possibilities
X n
• Example:
max rj x j
j=1
Xn
s.t. c j xj  B
j=1
xj 2 {0, 1} 8 j = 1, . . . , n
• There are 2n possible values of the binary variables
• If n is small, we can check each to see if it is feasible, and then
choose a feasible solution with maximum objective value
Enumeration in Python
n Time (secs)

5 0.0012

10 0.0049

15 0.1943

20 7.4745

25 311.9747

With n=50 estimated time > 330 years!!

Summary
• Discrete optimization problems
are nonconvex
• Local solutions may be very far
from global solutions
• Full enumeration can only work
for very small problems
Deterministic
Optimization
Discrete Optimization:
Introduction
Shabbir Ahmed
Anderson-Interface Chair and Professor
School of Industrial and Systems Engineering

Computational Complexity
Discrete Optimization:
Challenges

Learning Objectives
• Discover computational complexity
Running time of Algorithms
• Given an instance I of a problem P and an algorithm A, we would
like to understand how much effort the algorithm requires to solve I

• The effort is measured as the number of steps or elementary

operations required by Algorithm A on the instance

• The effort required by Algorithm A of course depends on the size of

the instance I
Example 1
• Problem P: Compute the scalar product of two vectors of size n

• Algorithm A:

• Effort: Need to do n multiplications and then sum up the n products

• The effort of algorithm A scales linearly with size n of the problem

instance
Example 2: Enumeration
n Time (secs)

5 0.0012

10 0.0049

15 0.1943

20 7.4745

25 311.9747

With n=50 estimated time > 330 years!!

The effort increases exponentially

(in the order of 2^n) as n
increases!
Big-O Notation
• Let T(n) be the effort or running time of algorithm A on an instance of
size n

• We want to understand how T(n) grows with n

• Let f(n) be a function of n

• We say that T(n) grows in “order” of f(n), i.e. T(n) = O(f(n)) if there
exists constants C and N such that
T(n) ≤ Cf(n) for all n ≥ N
More on Big-O

Example:
The function T (n) = 3n2 + 2n + 10 is O(n2 )
since T (n)  15n2 8 n 1

The leading terms determining

the big O order
Remark on Instance Size
• Note that the effort in the previous examples not only depend on n but
also on the size of the numbers (since multiplying two large numbers
requires more work than multiplying two small numbers)

• Computers work with finite precision, so we only consider rational

problem data ( we can assume integers only for simplicity).
Computers work with binary (or bit) representation of numbers

• Size of an instance is the total space required by the problem data

• Example: Product problem Size ≤ 2*n*log U where U = max{a_j,b_j}

Polynomial Time Algorithms
• An algorithm is polynomial time if its effort/running time T(n) as a
function of the instance size n is O(nk) for some k
• Polynomial time algorithms are “efficient”, and problems for which
such algorithms exists are deemed “easy”

Source:
https://i.stack.imgur.com/ia6VB.png
Problem Class P and NP

• Problem class P: The set of all problems for which can be solved in
polynomial time (“easy” problems)

• Problem class NP: The set of all problems for which a given solution
can be “verified” or “checked” in polynomial time

• Of course if a problem can be solved in polynomial time then we can

verify a given solution in polynomial time so
P ⊆ NP
Example
• Partition problem:
Given a set of n integers a1 , . . . , an , find a subset S ⇢ {1, . . . , n} such that
X X
aj = aj
j2S j62S

• We do not know a polynomial time algorithm to find a solution

• However, given a set S we can easily check if it is a solution!
• So this problem is not know to be in P but is known to be in NP
Problem Class NP-Complete
• A million dollar question: Is P = NP?

• Widely believed to be false, but a formal proof is not known

• Within the set of problems in NP there is a set of problems such that

every problem in NP can be “easily converted” to a problem in this
set. These problems are called NP-complete.

• If we can show one of the problems in the NP-complete set is in P

then P = NP
P, NP and NP-Complete

NP

P NP-complete
Problems Not in NP
• For some problems it is not easy to check whether a given solution is
correct
• Example: Knapsack problem
n
X
max rj x j
j=1
Xn
s.t. c j xj  B
j=1
xj 2 {0, 1} 8 j = 1, . . . , n

• Checking if a given solution is optimal is as hard as solving the

problem. This problem is not in NP
• This is the case with most difficult optimization problems
Problem Class NP-Hard
• A problem is NP-hard if an algorithm for it can be converted to one for
solving any problem in NP

• A NP-hard problem may not be in NP

• All NP-complete problems are NP-hard but not vice versa

• NP-hard means at least as hard as any problem in NP

Complexity Classes

Source:
https://en.wikipedia.org/wiki/NP-
hardness#/media/File:P_np_np-
complete_np-hard.svg
Optimization Problems
• Linear programming and many important classes of convex
optimization problems (e.g. conic programming and semidefinite
programming) are known to be in P

• Most discrete optimization problems are NP-hard

• It is widely believed that there is little hope of finding a polynomial

time algorithm for discrete optimization problems
Summary
• An algorithm is polynomial time if
its running time grows at a rate
no more than a polynomial
function of the instance size
• Problems for which polynomial
time algorithms are known are
easy
• Classes: P, NP, NP-complete,
NP-hard
• Discrete optimization problems
are NP-hard
Deterministic
Optimization
Discrete Optimization:
Modeling w/ Binary
Variables
Shabbir Ahmed
Anderson-Interface Chair and Professor
School of Industrial and Systems Engineering

Nonconvex functions
Modeling Nonconvex
Functions

Learning Objectives
• Figure out how to model nonconvex
functions using mixed integer linear
constraints
Problem Class

• Bounded variables n
X
• Separable objective function, min fj (xj )
i.e. sum of univariate functions
j=1
• Piecewise linear s.t. Ax b
• Nonconvex and nonlinear 0  xj  Uj j = 1, . . . , n
• Continuous
Piecewise Linear Function

f (x) = f (`k ) + ck (x `k ), `k  x  `k+1 , k = 0, . . . , K 1

f (`0 )
f (`K 1)
cK 1
c0
c1

f (`1 )
`0 `1 `K 1 `K
Piecewise Linear Function
function value
at breakpoint slope
breakpoint

f (x) = f (`k ) + ck (x `k ), `k  x  `k+1 , k = 0, . . . , K 1

f (`0 )
f (`K 1)
cK 1
c0
c1

f (`1 )
`0 `1 `K 1 `K
Modeling Approach 1
Introduce new variables u0 , . . . , uK 1 and y0 , . . . , yK 1.
Introduce the following constraints:
K
X1
x= uk
k=0
`k yk  uk  `k+1 yk k = 0, . . . , K 1
K
X1
yk = 1
k=0
yk 2 {0, 1} k = 0, . . . , K 1

K
X1
Replace f (x) by [f (`k ) + ck (uk `k )]
k=0
A Wrong Approach
Introduce new variables y0 , . . . , yK 1 .
Introduce the following constraints:

`k yk  x  `k+1 yk k = 0, . . . , K 1
K
X1
yk = 1
k=0
yk 2 {0, 1} k = 0, . . . , K 1

K
X1
Replace f (x) by [f (`k ) + ck (x `k )] yk
k=0

Correct formulation but is nonlinear !!!

 Example
• Decide how much to buy from a set of different vendors (offering
quantity discounts) to satisfy a total need
n $0.5/unit
X
min fj (xj )
j=1
$1/unit
s.t. Ax b
0  xj  Uj j = 1, . . . , n

⇢ 0
pj x j 0  xj  Lj 100 150
fj (xj ) =
pj Lj + qj (xj Lj ) Lj  xj  U j
Example
• Decide how much to buy from a set of different vendors (offering
quantity discounts) to satisfy a total need
n
X $0.5/unit
min [pj uj + pj Lj yj + qj (vj Lj )]
j=1
s.t. Ax b $1/unit
0  xj  Uj j = 1, . . . , n
xj = uj + vj j = 1, . . . , n
0  uj  Lj (1 yj ) j = 1, . . . , n
Lj yj  vj  Uj yj j = 1, . . . , n 0 100 150
yj 2 {0, 1} j = 1, . . . , n
Modeling Approach 2
Introduce new variables 0 , . . . , K and y0 , . . . , yK 1
Introduce the following constraints:
K
X K
X K
X1
x= k `k , k = 1, yk = 1
k=0 k=0 k=0
yk 2 {0, 1}, k = 0, . . . , K 1, k 0, k = 0, . . . , K
0  y0 , k  yk 1 + yk , k = 1, . . . , K 1, K  yK 1

K
X
Replace f (x) by f (`k ) k
k=0
 Example
• Decide how much to buy from a set of different vendors (offering
quantity discounts) to satisfy a total need

n
X
min [(0) j,1 + (pj Lj ) j,2 + (pj Lj + qj (Uj Lj )) j,3 ]
j=1
s.t. Ax b, 0  xj  Uj j = 1, . . . , n
xj = (0) j,1 + (Lj ) j,2 + (Uj ) j,2 , j = 1, . . . , n
j,1 + j,2 + j,2 = 1, j = 1, . . . , n, 0
yj,1 + yj,2 = 1, y 2 {0, 1}
j,1  yj,1 , j,2  yj,1 + yj,2 , j,3  yj,2 , j = 1, . . . , n
Summary
• We learned two ways to model
piece-wise linear nonconvex
functions by introducing new
continuous and binary variables
• It is very important to make sure
the resulting formulations are
linear
Deterministic
Optimization
Discrete Optimization:
Modeling w/ binary
variables
Shabbir Ahmed
Anderson-Interface Chair and Professor
School of Industrial and Systems Engineering

Nonconvex Sets
Modeling Nonconvex
Sets

Learning Objectives
• Observe how to model some
nonconvex sets using mixed integer
linear constraints
Discrete Variables
• A discrete variable has a domain of the form

x 2 {a1 , a2 , . . . , aK }

• If a1 , a2 , . . . , aK are contiguous integers, we can model as

a1  x  aK , x 2 Z

• Otherwise we can model as

K
X K
X
x= a k yk , yk = 1, yk 2 {0, 1} 8 k = 1, . . . , K
k=1 k=1
Semicontinuous Variables
• A semicontinuous variable has a domain of the form

x 2 [`1 , u1 ] [ [`2 , u2 ] [ · · · [ [`K , uK ]

where `1  u1  `2  u2  · · ·  `K  uK .

• One modeling approach for this system is

K
X
x= zk , ` k y k  zk  u k y k 8 k = 1, . . . , K
k=1
K
X
yk = 1, yk 2 {0, 1} 8 k = 1, . . . , K
k=1
Union of Polytopes

• Suppose we require that x satisfies one of the following K systems of

inequalities: Ak x  bk for k = 1, . . . , K
• Suppose that each system defines a polytope, i.e. a bounded polyhedron
• Then a formulation for this system is
K
X K
X
A k u k  b k yk , x = uk , yk = 1, yk 2 {0, 1} 8 k = 1, . . . , K
k=1 k=1
Example 1

✓ ◆ ✓ ◆
x1 + x2  2 1  x1  3
[
x1 0, x2 0 0  x2  2
Example 1

✓ ◆ ✓ ◆
x1 + x2  2 x1  1, x1  3
[
x1  0, x2  0 x2  0, x2  2
Example 1

✓ ◆ ✓ ◆
x1 + x2  2 x1  1, x1  3
[
x1  0, x2  0 x2  0, x2  2

x1 = u11 + u21 , x2 = u21 + u22

u11 + u12  2y1 , u11  0, u12  0
u21  1y2 , u21  3y2 , u22  0, u22  2y2
y1 + y2 = 1, y1 2 {0, 1}, y2 2 {0, 1}
Bounds
• Note that we have been making use of the bounds on the variables
to develop these formulations

• In general, it is impossible to develop correct mixed integer linear

formulations without bounds on variables

• If there are no bounds, we will impose large artificial numbers (big-

M) as variable bounds
Example 2
Model the nonconvex set X = {x 2 Rn : kxk0  k} where k is an integer
between 0 and n. Here kxk0 is the number of nonzero components of x.

This set is not bounded. We will impose bounds, M  xj  M for all

j = 1, . . . , n where M is a large number.

Then a mixed integer linear formulation is

n
X
M yj  x j  M y j , yj  k, yj 2 {0, 1} 8 j = 1, . . . , n
j=1
Summary
• We saw some examples on how
to model nonconvex sets by
mixed integer linear programming
• It is important that the sets to be
modeled are bounded
• It is very important to make sure
the resulting formulations are
linear
Deterministic
Optimization
Discrete Optimization:
Modeling w/ binary
variables
Shabbir Ahmed
Anderson-Interface Chair and Professor
School of Industrial and Systems Engineering

Logical Relations
Modeling Logical
Relations

Learning Objectives
• Observe some examples of
modeling logical relations using
mixed integer linear constraints
An investment example
• We would like to invest in 5 assets, labelled {1, 2, 3, 4, 5}
• We can only invest nonnegative amounts, and cannot invest more than a
total of $1000 dollars (budget)
• Let xj denote the amount invested in asset j, then the constraints on our
decisions are
5
X
0  xj , 8 j = 1, . . . , 5, xj  10000
j=1
Modeling Logical Relations using
Binary variables
• We will model various logical constraints on our investment decisions
• We will make use of binary variables that indicate whether an investment
as been made of not
• Introduce binary variable yj which will take a value 1 if xj > 0

• The constraints to force this are

0  xj  1000yj , yj 2 {0, 1} 8 j = 1, . . . , 5,
“OR” constraint

• We have to invest at least $100 in asset 1 or asset 5

• Logically this constraint is: (x1 100) _ (x5 100)
• Algebraically the constraint is:

x1 100y1 , x5 100y2 , y1 + y2 1
“XOR” Constraint

• We have to invest at least $100 in asset 1 or asset 5 but not both

• Logically this constraint is: (x1 100) Y (x5 100)
• Algebraically the constraint is:

x1 100y1 , x5 100y2 , y1 + y2 = 1
Implication
• If we invest $100 or more in asset 1 then we must invest $100 or more in
asset 5
• Logically this constraint is: (x1 100) ) (x5 100)
• Introduce a new binary variable z that gets set to 1 if (x1 100)

• Algebraically the constraint can be written as:

(x1 (100 ✏))

z 2 {0, 1}, z , 100z  x5
(900 + ✏)

where ✏ is a small positive number

Additional Examples

• If x1 > 0 then x2 = 0 or x3 = 0. Constraint:

y2 + y3  2 y1

• If x1 > 0 or x2 > 0 then x5 = 0. Constraint:

y5  1 y1 , y5  1 y2
Summary
• We saw some examples on how
to model logical relations by
mixed integer linear programming
• It is very important to make sure
the resulting formulations are
linear
Deterministic
Optimization
Discrete Optimization:
Modeling w/ binary
variables
Shabbir Ahmed
Anderson-Interface Chair and Professor
School of Industrial and Systems Engineering

Set Packing, Covering,

Partitioning
Set Packing, Covering
and Partitioning

Learning Objectives
• Discover integer programming
formulations of set based models
• Observe a python example of
modeling and solving such a model
A Store Location Example
• Budget Stores is considering to locate superstores in 4 cities
• Its customer base is spread across 6 neighboring cities
• Customers from a city that is within driving distance from a store will
visit that store

• Budget does not want to locate two or more stores that are with driv-
ing distance of the same customer city (to avoid cannibalization of the
customer base)
• Budget wants to maximize the number of customers it can attract
Example: Data
City Location A Location B Location C Location D Customers
A 1 1 0 1 100
B 1 1 1 0 50
C 0 1 1 0 70
D 1 0 0 1 150
E 1 1 1 0 20
F 0 1 0 1 80
Cell (i, j) has an entry of 1 if city i is within driving distance of location j.
Example: Data
City Location A Location B Location C Location D Customers
A 1 1 0 1 100
B 1 1 1 0 50
C 0 1 1 0 70
D 1 0 0 1 150
E 1 1 1 0 20
F 0 1 0 1 80
Total 320 320 140 330
Model and Code
max 320xA + 320xB + 140xC + 330xD
s.t. xA + xB + xD  1
xA + xB + xC  1
xB + xC  1
xA + xD  1
xA + xB + xC  1
xB + xD  1
xA , xB , xC , xD 2 {0, 1}
Model and Code
max 320xA + 320xB + 140xC + 330xD
s.t. xA + xB + xD  1
xA + xB + xC  1
xB + xC  1
xA + xD  1
xA + xB + xC  1
xB + xD  1
xA , xB , xC , xD 2 {0, 1}
Model and Code
max 320xA + 320xB + 140xC + 330xD
s.t. xA + xB + xD  1
xA + xB + xC  1
xB + xC  1
xA + xD  1
xA + xB + xC  1
xB + xD  1
xA , xB , xC , xD 2 {0, 1}
Set Packing: General Formulation
Given m elements and a collection of subsets S1 , . . . , Sn ✓ {1, . . . , m} with
associated nonnegative weights w1 , . . . , wn , pick sets from this collection such
that they are disjoint and the sum of the weights is maximized.
n
X
max w j xj
j=1
X
s.t. xj  1 8 i = 1, . . . , m
j: i2Sj
xj 2 {0, 1} 8 j = 1, . . . , n.
Set Covering: General Formulation
Given m elements and a collection of subsets S1 , . . . , Sn ✓ {1, . . . , m} with
associated nonnegative weights w1 , . . . , wn , pick sets from this collection such
that they cover each item and the sum of the weights is minimized.
n
X
min w j xj
j=1
X
s.t. xj 1 8 i = 1, . . . , m
j: i2Sj
xj 2 {0, 1} 8 j = 1, . . . , n.
Set Partitioning: General
Formulation
Given m elements and a collection of subsets S1 , . . . , Sn ✓ {1, . . . , m} with asso-
ciated nonnegative weights w1 , . . . , wn , pick sets from this collection such that
they cover exactly one item and the sum of the weights is maximized/minimized.
n
X
max / min w j xj
j=1
X
s.t. xj = 1 8 i = 1, . . . , m
j: i2Sj
xj 2 {0, 1} 8 j = 1, . . . , n.
Summary
• Set packing/covering/partitioning
models arise in many applications
where decisions involve choosing
subsets of a given set of items
• We also observed an example of
using PuLP and glpk to solve an
integer programming formulation
of such a problem
Deterministic
Optimization
Discrete Optimization:
Modeling w/ binary
variables
Shabbir Ahmed
Anderson-Interface Chair and Professor
School of Industrial and Systems Engineering

Network Problems
Network Problems

Learning Objectives
• Discover minimum cost network flow
problems
• Observe an example using PuLP
Example
300 1200
• A company has two manufacturing
plants and 3 distribution centers (nodes) ($12, 1000) ($15, 300)
1 2 3 800
• The plants and DCs are connected by

($10, 200)
transportation channels (arcs)

($5, 100)
• The production amount and demand at
each node is shown
• The capacity and cost per unit for each
4 5
channel (arc) is shown ($15, 1000)
• Find the cheapest way to move units
from plants to DCs 500 200
 Model
300 1200
min 12x12 + 10x14 + 15x23 + 8x24
+5x25 + 15x45 + 12x53 ($12, 1000) ($15, 300)
s.t. x12 + x14 = 300 1 2 3 800
x23 + x24 + x25 x12 = 1200

($10, 200)

($5, 100)
x23 x53 = 800
x45 x14 x24 = 500
x53 x25 x45 = 200
0  x12  1000, 0  x14  200,
0  x23  300, 0  x24  5000 4 5
0  x25  100, 0  x45  1000 ($15, 1000)
0  x53  1000
500 200
PuLP Code
300 1200

($12, 1000) ($15, 300)

1 2 3 800

($10, 200)

($5, 100)
4 5
($15, 1000)

500 200
Solution
300 1200

($12, 1000) ($15, 300)

1 2 3 800
100 300

($10, 200)
200

($5, 100)
100
4 600 5
($15, 1000)

500 200
Minimum Cost Network Flow
Problem
• We are given a directed network G = (N, A) with a set of nodes N and a
set of arcs A.
• Each node i 2 N has an associated “supply” bi . If bi > 0 the node is
a supply node, if bi < 0 Pit is a demand node. We will assume that the
network is balanced, i.e. i2N bi = 0.

• Each arc (i, j) 2 A has an associated cost cij and capacity uij .
• A flow on this network is a set of values on the arcs that obey capacities
and satisfy flow conservation at the nodes.
• The goal is to find a flow of minimum cost.
Minimum Cost Network Flow
Problem
X
min cij xij
(ij)2A

X X
s.t. xij xji = bi 8i2N
(ij)2A (ji)2A

0  xij  uij 8 (ij) 2 A

Characteristics
• The constraint matrix has +1 and 1 in every column and all other entries
are 0. Such a matrix is known as a network matrix.
• If the supply data bi and the capacities uij are integers, then every extreme
point (or basic feasible solution) will have integer entries.

• Thus a network flow problem with integrality requirements can be solved

as a linear program.
• This is an important class of integer programs that are “easy”
• Next, we will see some integer programs that can be converted in to net-
work flow problems, and hence can be solved easily.
Shortest Path Problem
Given a graph G = (N, A) with a start node s and destination node t, where
each arc (ij) is of length cij 0, find the shortest path from s to t.
X
min cij xij
(ij)2A

8
X X < +1 i=s
s.t. xij xji = 0 i 2 N \ {s, t}
:
(ij)2A (ji)2A 1 i=t

xij 2 {0, 1} 8 (ij) 2 A

Shortest Path Problem
Given a graph G = (N, A) with a start node s and destination node t, where
each arc (ij) is of length cij 0, find the shortest path from s to t.
X
min cij xij
(ij)2A

8
X X < +1 i=s
s.t. xij xji = 0 i 2 N \ {s, t}
:
(ij)2A (ji)2A 1 i=t

xij 2 {0, 1} 8 (ij) 2 A 0  xij  1 8(ij) 2 A

Assignment Problem
p X
X q

We are given a set of p workers and q tasks. min cij xij

i=1 j=1
Assume p > q. Worker i 2 {1, . . . , p} charges q
X
$ cij to complete task j 2 {1, . . . , q}. Each s.t. xij  1 8i
worker can be assigned to at most one task. j=1
Each task has to be assigned to a worker. Xp

Find the minimum cost assignment of worker xij = 1 8j

to tasks. i=1
xij 2 {0, 1} 8 i, j
Assignment Problem
The assignment problem can be converted to a min 1 1 1 1
cost network flow problem as follows:
• Each worker is a supply node with supply of 1 1 2 2 1
• Each task is a demand node with demand of 1

…
…
• Introduce a “slack” node with demand of p q
q 1
• There is a arc from worker node i to task node j 1 p
with capacity of 1 and cost of cij
• There is a arc from each worker node to the slack 0 p-q
node with capacity of 1 and cost of 0
Summary
• Network problems are an easy
class of integer programs
• Some important classes of
integer programs can be
converted to network problems
Deterministic
Optimization
Discrete Optimization:
Modeling w/ binary
variables
Shabbir Ahmed
Anderson-Interface Chair and Professor
School of Industrial and Systems Engineering

Modeling Exercises 1
Modeling Exercises 1

Learning Objectives
• Observe modeling a facility location
problem
• Observe modeling a generation
scheduling problem
A Facility Location Problem
• We need to pick sites to locate distribution centers (DC) from a set of n
potential sites
• Let fj and vj denote the fixed and variable (per-unit capacity) cost of
locating at site j, respectively

• The DCs will serve m retail locations and the demand of retail i is di
• The per-unit-cost of transporting from DC location j to retail location i
is cij
• Decide where to locate the DCs, how much capacity to install, and how
to match DC capacity to retail demand at minimum total cost
A Facility Location Problem
Variables:
yj : Binary variable indicating whether location j is chosen
xj : Continuous variable indicating the capacity of DC located at j
zij : Units shipped from DC j to retail location i
n
X m X
X n
min (fj yj + vj xj ) + cij zij
j=1 i=1 j=1
n
X
s.t. zij = di 8 i = 1, . . . , m
j=1
Xm
zij  xj 8 j = 1, . . . , n
i=1 Pm
0  xj  ( i=1 di )yj 8 j = 1, . . . , n
0  zij  di yj 8 i = 1, . . . , m, j = 1, . . . , n
yj 2 {0, 1} 8 j = 1, . . . , n
A Generator Scheduling Problem

Schedule generator
operation to meet
load over a time Dt
horizon

t=1 t=T
A Generator Scheduling Problem

Generation schedule
must satisfy some
operational
constraints
A Generator Scheduling Problem

Min/Max Production
Limits: If generator is
on it must produce
Pmax
within given min/max
limits

Pmin
A Generator Scheduling Problem

Ramp up/down limits:

Production cannot Rup Rdown
increase/decrease
faster than prescribed
limits
A Generator Scheduling Problem

Min up/down time

limits: If a generator is
turned off, it must be
off for a certain
minimum time.
Similarly, once it is
turned, it must stay
one for a certain
amount of time

Tdown Tup
A Generator Scheduling Problem

Total cost =
generation cost + cost
of unmet load (spot
price)

C = Per-unit generation cost St = Spot cost in period t

A Generator Scheduling Problem
Variables:
xt : Generation in period t (t = 1, . . . , T )
zt : Power from spot market in period t
yt : Status (on/o↵) of generator in period t
T
X
min (Cxt + St zt )
t=1
s.t. x t + zt = D t t = 1, . . . , T
Pmin yt  xt  Pmax yt t = 1, . . . , T
xt xt 1  Rup t = 2, . . . , T
xt 1 xt  Rdown t = 2, . . . , T
xt , zt 0, yt 2 {0, 1} t = 1, . . . , T
+ min up/down constraints
Min Up Constraints

y⌧ yt yt 1 8 ⌧ = t + 1, . . . , min{t + Tup , T }, 8 t = 1, . . . , T 1

Example: Suppose T = 6, Tup = 3 and y0 = 0

t=1: y2 y1 , y3 y1
t=2: y3 y2 y1 , y4 y2 y1
t=3: y4 y3 y2 , y5 y3 y2
t=4: y5 y4 y3 , y6 y4 y3
t=5: y6 y5 y4
Min Down Constraints

y⌧  yt yt 1 + 1 8 ⌧ = t + 1, . . . , min{t + Tdown , T }, 8 t = 1, . . . , T 1

Example: Suppose T = 6, Tdown = 3 and y0 = 0

t=1: y2  y1 + 1, y3  y1 + 1
t=2: y3  y2 y1 + 1, y4  y2 y1 + 1
t=3: y4  y3 y2 + 1, y5  y3 y2 + 1
t=4: y5  y4 y3 + 1, y6  y4 y3 + 1
t=5: y6  y5 y 4 + 1
Summary
• We observed some examples of
modeling with integer variables
Deterministic
Optimization
Discrete Optimization:
Modeling w/ binary
variables
Shabbir Ahmed
Anderson-Interface Chair and Professor
School of Industrial and Systems Engineering

Modeling Exercises 2
Modeling Exercises 2

Learning Objectives
• Observe modeling the traveling
salesman problem (TSP)
• Solve a TSP example with PuLP and
Python
Traveling Salesman Problem (TSP)

Given a collection of n cities,

along with pair-wise distances
between them, find the shortest
way to visit all cities are return
to the starting point.

A tour through 42 US Cities

Source: http://www.math.uwaterloo.ca/tsp
Formulation
Variables (for i 6= j):
⇢
1 if city i is visited immediately before city j
xij =
0 otherwise

n X
X n
min cij xij
i=1 j=1
Xn
s.t. xij = 1 8 i = 1, . . . , n
j=1
Xn
xji = 1 8 i = 1, . . . , n
j=1 X
X
xij  |S| 1 8 S ⇢ {1, . . . , n}, |S| 2
i2S j2S
Formulation: Subtour Elimination

i ji i
i
i
i
i
i

S = {i, j}
xij + xji  1
Formulation: Subtour Elimination

i ji i
i
ki
i
i
i

S = {i, j, k}
xij + xik + xji + xjk + xki + xkj  2
Formulation: Subtour Elimination
• There are many (exponential in n) subtour elimination constraints
• Solve without any subtour elimination constraints
• Inspect the solution to check if there are subtours
• Add the subtour elimination constraint corresponding to the found
subtour
• Repeat
PuLP Code
PuLP Code
PuLP Code
PuLP Code
PuLP Code Output
Summary
• We studied a formulation of the
TSP and solved it using PuLP
• There are alternative formulations
Deterministic
Optimization
Discrete Optimization:
Modeling w/ binary
variables
Shabbir Ahmed
Anderson-Interface Chair and Professor
School of Industrial and Systems Engineering

Modeling Exercises 3
Modeling Exercises 3

Learning Objectives
• Observe modeling a mixed integer
nonlinear programming problems
Portfolio Optimization
• We are given a set of n stocks
• The expected return (in $/share) of stock j is estimated from historical
data and is given by rj
• The covariance between the return of asset i and j is ij

• We want to determine how many shares of each stock to buy so as to

minimize risk (variance) subject to some investment constraints
Portfolio Optimization: Constraints

• We can invest in at most k stocks (here k < n)

• The expected return of the portfolio should be at least R
• If we invest in stock j we must buy at least lj shares
• We cannot buy more than uj shares of stock j
Formulation
Variables:
xj : Number of shares of stock j to buy (nonnegative integer)
yj : Whether to invest in stock j (binary)

n X
X n
min ij xi xj
i=1 j=1
Xn
s.t. rj x j R
j=1
l j y j  x j  u j yj 8 j = 1, . . . , n
X n
yj  k
j=1
xj 2 Z+ , yj 2 {0, 1} 8 j = 1, . . . , n
Remarks

• The formulation is an example of a mixed integer (convex) quadratic pro-

gram (MIQP)
• Dropping integrality we get a relaxation that is a (convex) quadratic pro-
gram

• Similar formulation arises in sparse regression applications

• Such formulation can be solved by solvers such as cplex or gurobi
Summary
• We observe an example of a
mixed integer nonlinear program
Deterministic
Optimization
Discrete Optimization:
Linear Programming
Relaxations
Shabbir Ahmed
Anderson-Interface Chair and Professor
School of Industrial and Systems Engineering

Linear Programming Relaxation

Linear Programming
Relaxation

Learning Objectives
• Examine the information provided by
the LP relaxation of an integer
program
LP Relaxation

Consider a mixed-integer linear program (MILP):

vIP = min c> x

s.t. Ax b
x 2 Rn p ⇥ Z p
It’s linear programming relaxation is:

vLP = min c> x

s.t. Ax b
x 2 Rn
Example vIP = min x 1 + y1
s.t. 3x + 2y 6
x + 4y 4
0  x  3, 0  y  3
x, y integer

vLP = min x 1 + y1
s.t. 3x + 2y 6
x + 4y 4
0  x  3, 0  y  3
LP Relaxation Properties
• If the LP relaxation is infeasible then so is the IP.
• (As long as all data is rational) If the LP relaxation is unbounded then
the IP can be either infeasible or unbounded.
• If the LP relaxation has an optimal solution, then the IP could be infeasible
or have an optimal solution.
• It always holds that vLP  vIP (for minimization problems)
• If an optimal solution to the LP relaxation is integral, then it is an optimal
solution to the IP.
Optimality Gap
• The optimal value of the LP relaxation allows us to estimate the optimal
gap of a given feasible solution to the IP.
• Example: Consider the solution (2, 1) which is feasible and has an objec-
tive value of 3. The LP relaxation has an objective value of 2.2. Thus
2.2  vIP  3. The optimality gap of the solution (2, 1) is at most
(3 2.2) = 0.8.

vIP = min x 1 + y1 vLP = min x 1 + y1

s.t. 3x + 2y 6 s.t. 3x + 2y 6
x + 4y 4 x + 4y 4
0  x  3, 0  y  3 0  x  3, 0  y  3
x, y integer
Rounding LP Solution
• Sometimes by rounding the LP relaxation solution we may be able to find
a good solution to the IP.
Summary
• The LP relaxation provides
valuable information about an IP
• It’s optimal value provides
bounds and optimality gap
estimates
• It’s solutions can sometimes be
rounded to get a good solution to
the IP
Deterministic
Optimization
Discrete Optimization:
Linear Programming
Relaxations
Shabbir Ahmed
Anderson-Interface Chair and Professor
School of Industrial and Systems Engineering

Ideal Formulations
Ideal Formulation

Learning Objectives
• Identify strong and ideal formulations
of a mixed integer linear program
 Formulations of a MILP
• The feasible region of a mixed integer linear program (MILP) is a set of
the form
X = {x : x 2 P \ (Rn p ⇥ Zp )}
where P is a polyhedral set.
• The same MILP can be formulated in di↵erent ways, e.g.

X = {x : x 2 Q \ (Rn p
⇥ Zp )}

where Q is a di↵erent polyhedral set than P .

• We say the formulation using P is at least as strong as that using Q when
P ✓Q
Example
Formulation
8 1: 9
>
> 3x + 2y 6 >
>
< =
x + 4y 4
X = (x, y) :
>
> 0  x  3, 0  y  3 >
>
: ;
x, y integer
Formulation
8 2: 9
< x+y 3 =
X = (x, y) : 0  x  3, 1  y  3
: ;
x, y integer
Strong Formulations
• Stronger formulations (of an MILP) lead to stronger LP relaxations, and
so better LP relaxation better bounds, and sometimes LP relaxation so-
lutions that are feasible to the MILP
• The formulation of an MILP can be strengthened

– by adding constraints (valid inequalities)

– by tightening constraint coefficients
– by introducing new variables and constraints (not covered)
 Convex Hull / Ideal
Formulation
• The convex hull of a set X, denoted by conv(X) is the smallest convex set
that contains X
• Fact: If X is the feasible region of a MILP, then conv(X) is a polyhedral
set.

• Fact: If X is the feasible region of a MILP, then all extreme points of

conv(X) are feasible solutions to the MILP
• Then
min{c> x : x 2 X} ⌘ min{c> x : x 2 conv(X)}

• Thus MILP is equivalent to an LP with an ideal formulation!!

 Strong Formulations

• If we had the convex hull or ideal formulation then we can solve an MILP
by just solving an LP
• Getting the ideal formulation is as hard as (if not harder) to solve the
MILP

• We try to get strong formulation that are closer to the ideal formulation
so that the LP relaxation gives a good approximation of the MILP
Summary
• An ideal formulation of a MILP is
one whose LP relaxation solves
the MILP
• Ideal formulations are hard to
obtain so we strive to obtain
strong formulation that
approximate the ideal formulation
Deterministic
Optimization
Discrete Optimization:
Branch-and-Bound

Shabbir Ahmed
Anderson-Interface Chair and Professor
School of Industrial and Systems Engineering

The Branch-and-Bound Algorithm

The Branch-and-
Bound Algorithm

Learning Objectives
• Inspect the branch-and-bound
algorithm for solving integer
programs
LP Relaxation
• We would like to solve the integer program:

vIP = min c> x

s.t. Ax b
x 2 Rn p ⇥ Z p
• First solve the LP relaxation.
• If the LP relaxation is infeasible then the IP is infeasible. We can stop.

• If the LP relaxation is unbounded then, Assuming rational data, the IP

is either infeasible or unbounded. We can stop.
LP Relaxation Solution
• Suppose the LP relaxation has an optimal solution.
Let vLP denote its optimal value and xLP be an optimal solution.
• If it happens that the LP relaxation solution is integral, that is
xLP 2 Rn p ⇥ Zp , then we know that xLP is an optimal solution to the
IP and v IP = v LP . So we can stop.

• Suppose that the LP relaxation solution is not integral, i.e. there is a

component j of the solution vector that is supposed to integral but now
has a fractional value, i.e. xLP
j 62 Z. We also know that v LP  v IP .
Branching
• Create two new LPs:
v0LP = min c> x v1LP = min c> x
(P 0) : s.t. Ax b (P 1) : s.t. Ax b
xj  bxLP
j c xj dxLP
j e

• The optimal solution to the IP (if it exists) must be contained in the

feasible region of the one of the two LPs.
• Note that v IP min{v0LP , v1LP }
• Solve these LPs and proceed recursively.
Bounding LP

LP0 LP1

• If any of the LPs return an integer solution then we have feasible solution
to the IP. If the objective value of this solution is better (smaller) then
any we have found so far, then we keep it as an “incumbent” solution and
maintain the objective as the best upper bound U B, i.e. v IP  U .
• If an LP is infeasible, then we need not explore it any further.

• If an LP has an objective value that is worse that U B then we need not

explore this any further.
 Branch-and-Bound Tree
• Any LP that has an integer optimal solution
need not be explored further.
• The LPs that need not be explored further are
deemed to be “fathomed” or “pruned”

• An LP that has not been fathomed need to be branched creating two new
LPs
• This process is arranged into a (inverted) tree where each LP is called a
node, and the initial LP relaxation is the “root” node.
Lower Bound
• At any point in time this tree has some nodes (LPs) that are pruned and
has some nodes that need to further explored.
• At any point in the recursive process, the minimum of the objective values
of the LPs that still need to be explored provide a lower bound LB.

• We can stop the search if the LB and U B are close together and return
the incumbent solution as a feasible solution with an optimality gap of at
most U B LB.
Branch and Bound

• The branch-and-bound scheme simply relies on the solutions of LPs ob-

tained by adding constraints to the root LP relaxation
• The same idea can be extended to nonlinear IPs as long as we can solve
the relaxations exactly.

• For linear integer programs, the basic algorithm is enhanced by many

heuristics to obtain a feasible integer solution from the node LPs and by
adding valid inequalities to improve the formulation of the node LPs.
Summary
• We discussed the branch-and-
bound algorithm which is the
backbone of IP solvers.
• We will observe a small example
of branch-and-bound in the next
lesson
Deterministic
Optimization
Discrete Optimization:
Branch-and-Bound

Shabbir Ahmed
Anderson-Interface Chair and Professor
School of Industrial and Systems Engineering

An Example
Example of the
Branch-and-Bound
Algorithm

Learning Objectives
• Observe an example of the branch-
and-bound algorithm
Example
We would like to solve:
v IP = min 3x1 + 2x2 + x3
s.t. 3x1 + 4x2 + 5x3 6
x1 , x2 , x3 0 and integer

The LP relaxation:
v LP = min 3x1 + 2x2 + x3
s.t. 3x1 + 4x2 + 5x3 6
x 1 , x2 , x3 0
Branch-and-Bound LB = 1.2  v IP  U B = +1

v LP = min 3x1 + 2x2 + x3

s.t. 3x1 + 4x2 + 5x3 6 xLP = (0, 0, 1.2), v LP = 1.2
x 1 , x2 , x3 0
Branch-and-Bound LB = 1.2  v IP  U B = +1

v LP = min 3x1 + 2x2 + x3

xLP = (0, 0, 1.2), v LP = 1.2
s.t. 3x1 + 4x2 + 5x3 6
x3  1
x 1 , x2 , x3 0 x3  1

xLP = (0, 0.25, 1), v LP = 1.5

 Branch-and-Bound LB = 1.5  v IP  U B = 2

v LP = min 3x1 + 2x2 + x3

s.t. 3x1 + 4x2 + 5x3 6 xLP = (0, 0, 1.2), v LP = 1.2
x3 2
x 1 , x2 , x3 0 x3  1 x3 2

xLP = (0, 0.25, 1), v LP = 1.5 xLP = (0, 0, 2), v LP = 2

 Branch-and-Bound LB = 1.5  v IP  U B = 2

v LP = min 3x1 + 2x2 + x3

s.t. 3x1 + 4x2 + 5x3 6 xLP = (0, 0, 1.2), v LP = 1.2
x3 2
x 1 , x2 , x3 0 x3  1 x3 2

xLP = (0, 0.25, 1), v LP = 1.5 xLP = (0, 0, 2), v LP = 2

Branch-and-Bound LB = 1.5  v IP  U B = 2

xLP = (0, 0, 1.2), v LP = 1.2

x3  1 x3 2

xLP = (0, 0.25, 1), v LP = 1.5 xLP = (0, 0, 2), v LP = 2

x2  0

xLP = (0.5, 0, 1), v LP = 2.5

Branch-and-Bound LB = 2  v IP  U B = 2

xLP = (0, 0, 1.2), v LP = 1.2

x3  1 x3 2

xLP = (0, 0.25, 1), v LP = 1.5 xLP = (0, 0, 2), v LP = 2

x2  0 x2 1

xLP = (0.5, 0, 1), v LP = 2.5 xLP = (0, 1, 0.4), v LP = 2.4

Summary
• We observed a simple example
of branch and bound.
• Notice we have ties in selecting
which node to process first and
which variable we need to branch
on.
• Sophisticated rules are used to
make these choices.