Interior Point Methods

ME575 – Optimization Methods

John Hedengren

40
=0
30 =0.1
Barrier Function, - ln(x)

=1.0
20 =2.0
=5.0
10
=10
0

-10

-20
0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5
Variable (x)
Overview
• Nonlinear constrained minimization
• Slack variables
• Barrier function
• Newton’s method for KKT conditions
• Interior Point Method (IPM)
• Tutorial Examples
Nonlinear Constrained Optimization
min𝑛 𝑓(𝑥)
𝑥∈𝑅
𝑠. 𝑡. 𝑐 𝑥 =0
𝑥≥0

• Linear, quadratic, or general nonlinear objective and constraints

• Convex optimization, local solution possible for non-convex problems
• Convert maximization by minimizing negative of the objective
• Convert general inequalities to simple inequalities with slack variables
Slack Variables
min𝑛 𝑓(𝑥) min𝑛 𝑓(𝑥)
𝑥∈𝑅 𝑥∈𝑅
𝑠. 𝑡. 𝑔 𝑥 ≥𝑏 𝑠. 𝑡. 𝑐 𝑥 =0
ℎ 𝑥 =0 𝑥≥0
• Complete worksheet on slack variables

𝑚𝑖𝑛 𝑥1 𝑥4 𝑥1 + 𝑥2 + 𝑥3 + 𝑥3
𝑥
𝑠. 𝑡. 𝑥1 𝑥2 𝑥3 𝑥4 ≥ 25
𝑥12 + 𝑥22 + 𝑥32 + 𝑥42 = 40

• Additional slack variable tutorial:

• http://apmonitor.com/me575/index.php/Main/SlackVariables
Barrier Function
𝑛
min𝑛 𝑓(𝑥) min 𝑓 𝑥 − 𝜇 ෍ ln 𝑥𝑖
𝑥∈𝑅 𝑥∈𝑅𝑛
𝑠. 𝑡. 𝑐 𝑥 =0 𝑖=1

𝑥≥0 𝑠. 𝑡. 𝑐 𝑥 =0
40
• Natural log barrier term for =0
30 =0.1
inequality constraints

Barrier Function, - ln(x)

=1.0
20 =2.0
• The term ln 𝑥𝑖 is undefined 10
=5.0
=10
for 𝑥𝑖 < 0 0

• Search points only in interior -10

feasible space -20

0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5
Variable (x)
Barrier Function Example 1
2 2
min 𝑥−3 min 𝑥−3 − 𝜇 ln 𝑥
𝑥∈𝑅 𝑥∈𝑅
𝑠. 𝑡. 𝑥≥0 40
Unconstrained
=1
30 =2
=5

Augmented Objective Function

=10
20

10

-10

-20
0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5
Dependent Variable (x)
Barrier Function Example 2
2 2
min 𝑥+1 min 𝑥+1 − 𝜇 ln 𝑥
𝑥∈𝑅 𝑥∈𝑅
𝑠. 𝑡. 𝑥≥0 10
Unconstrained
9
=1
8 =2
=5

Augmented Objective Function

7 =10

0
-2 -1.5 -1 -0.5 0 0.5 1 1.5 2 2.5 3
Dependent Variable (x)
How to Solve a Barrier Problem: Step 1
• KKT Conditions for Barrier Problem
𝑛 𝑛
1
min𝑛 𝑓 𝑥 − 𝜇 ෍ ln 𝑥𝑖 𝛻𝑓 𝑥 + 𝛻𝑐 𝑥 𝜆 − 𝜇 ෍ = 0
𝑥∈𝑅 𝑥𝑖
𝑖=1 𝑖=1
𝑠. 𝑡. 𝑐 𝑥 =0 𝑐 𝑥 =0
1
• Define 𝑧𝑖 = and solve modified version of KKT conditions
𝑥𝑖

𝛻𝑓 𝑥 + 𝛻𝑐 𝑥 𝜆 − 𝑧 = 0
𝑐 𝑥 =0
𝑋𝑍𝑒 − 𝜇𝑒 = 0
How to Solve a Barrier Problem: Step 2
• Find KKT solution with Newton-Raphson method
𝛻𝑓 𝑥 + 𝛻𝑐 𝑥 𝜆 − 𝑧 = 0 𝑊𝑘 𝛻𝑐 𝑥𝑘 −𝐼 𝑑𝑘𝑥 𝛻𝑓 𝑥𝑘 + 𝛻𝑐 𝑥𝑘 𝜆𝑘 − 𝑧𝑘
𝑐 𝑥 =0 𝛻𝑐 𝑥𝑘 𝑇
0 0 𝑑𝑘𝜆 =− 𝑐 𝑥𝑘
𝑋𝑍𝑒 − 𝜇𝑒 = 0 𝑍𝑘 0 𝑋𝑘 𝑑𝑘𝑧 𝑋𝑘 𝑍𝑘 𝑒 − 𝜇𝑗 𝑒
• Where:
𝑧1 0 0 𝑥1 0 0
2 2 𝑇
𝑊𝑘 = 𝛻𝑥𝑥 𝐿 𝑥𝑘 , 𝜆𝑘 , 𝑧𝑘 = 𝛻𝑥𝑥 𝑓 𝑥𝑘 + 𝑐 𝑥𝑘 𝜆𝑘 − 𝑧𝑘 𝑍𝑘 = 0 ⋱ 0 𝑋𝑘 = 0 ⋱ 0
0 0 𝑧𝑛 0 0 𝑥𝑛
Find a Search Direction and Step Size: Example 3
2 𝑊𝑘 𝛻𝑐 𝑥𝑘 −𝐼 𝑑𝑘𝑥 𝛻𝑓 𝑥𝑘 + 𝛻𝑐 𝑥𝑘 𝜆𝑘 − 𝑧𝑘
𝑚𝑖𝑛1 𝑓=𝑦
𝑥∈𝑅 𝛻𝑐 𝑥𝑘 𝑇
0 0 𝑑𝑘𝜆 =− 𝑐 𝑥𝑘
𝑠. 𝑡. 2𝑦 ≤ 9 𝑍𝑘 0 𝑋𝑘 𝑑𝑘𝑧 𝑋𝑘 𝑍𝑘 𝑒 − 𝜇𝑗 𝑒
𝑦≥0 2
𝑊𝑘 = 𝛻𝑥𝑥 2
𝐿 𝑥𝑘 , 𝜆𝑘 , 𝑧𝑘 = 𝛻𝑥𝑥 𝑓 𝑥𝑘 + 𝑐 𝑥𝑘 𝑇 𝜆𝑘 − 𝑧𝑘
𝑠𝑡𝑎𝑟𝑡𝑖𝑛𝑔 𝑎𝑡 𝑦 = 3,
𝜇 = 10, 𝜆 = 1, 𝛼 = 0.5
How to Solve a Barrier Problem: Step 2
• Find KKT solution with Newton-Raphson method
𝛻𝑓 𝑥 + 𝛻𝑐 𝑥 𝜆 − 𝑧 = 0 𝑊𝑘 𝛻𝑐 𝑥𝑘 −𝐼 𝑑𝑘𝑥 𝛻𝑓 𝑥𝑘 + 𝛻𝑐 𝑥𝑘 𝜆𝑘 − 𝑧𝑘
𝑐 𝑥 =0 𝛻𝑐 𝑥𝑘 𝑇
0 0 𝑑𝑘𝜆 =− 𝑐 𝑥𝑘
𝑋𝑍𝑒 − 𝜇𝑒 = 0 𝑍𝑘 0 𝑋𝑘 𝑑𝑘𝑧 𝑋𝑘 𝑍𝑘 𝑒 − 𝜇𝑗 𝑒
• Where:
𝑧1 0 0 𝑥1 0 0
2 2 𝑇
𝑊𝑘 = 𝛻𝑥𝑥 𝐿 𝑥𝑘 , 𝜆𝑘 , 𝑧𝑘 = 𝛻𝑥𝑥 𝑓 𝑥𝑘 + 𝑐 𝑥𝑘 𝜆𝑘 − 𝑧𝑘 𝑍𝑘 = 0 ⋱ 0 𝑋𝑘 = 0 ⋱ 0
0 0 𝑧𝑛 0 0 𝑥𝑛
• Rearrange into symmetric linear system
𝑊𝑘 + Σ𝑘 𝛻𝑐 𝑥𝑘 𝑑𝑘𝑥 𝛻𝑓 𝑥𝑘 + 𝛻𝑐 𝑥𝑘 𝜆𝑘
=− Σ𝑘 = 𝑋𝑘−1 𝑍𝑘
𝛻𝑐 𝑥𝑘 𝑇 0 𝑑𝑘𝜆 𝑐 𝑥𝑘

• Solve for 𝑑𝑘𝑧 after the linear solution to 𝑑𝑘𝑥 and 𝑑𝑘𝜆 with explicit solution
𝑑𝑘𝑧 = 𝜇𝑘 𝑋𝑘−1 𝑒 − 𝑧𝑘 − Σ𝑘 𝑑𝑘𝑥
Step Size (𝛼)
• Two objectives in evaluating progress
• Minimize objective
• Minimize constraint violations
• Two popular approaches
• Decrease in merit function, 𝑚𝑒𝑟𝑖𝑡 = 𝑓 𝑥 + 𝜈 σ 𝑐(𝑥)
• Filter methods
• Cut back step size until improvement
𝑥𝑘+1 = 𝑥𝑘 + 𝛼𝑘 𝑑𝑘𝑥
𝜆𝑘+1 = 𝜆𝑘 + 𝛼𝑘 𝑑𝑘𝜆
𝑧𝑘+1 = 𝑧𝑘 + 𝛼𝑘 𝑑𝑘𝑧
Convergence Criteria
• Convergence when KKT conditions are satisfied with a tolerance
max 𝛻𝑓 𝑥 + 𝛻𝑐 𝑥 𝜆 − 𝑧 ≤ 𝜖𝑡𝑜𝑙
max 𝑐(𝑥) ≤ 𝜖𝑡𝑜𝑙
max 𝑋𝑍𝑒 − 𝜇𝑒 ≤ 𝜖𝑡𝑜𝑙
• Tolerance for constraint violation may be more restrictive
 Interior Point Method Overview
Initialize 𝜆
Initialize x0 , l0 , z0 𝑥0 = 𝑓𝑒𝑎𝑠𝑖𝑏𝑙𝑒
𝜇 𝐼 𝛻𝑐 𝑥0 𝑤
𝑧0 = 𝛻𝑓 𝑥0 − 𝑧𝐿,0 − 𝑧𝑈,0
𝑥0 𝑇 𝜆0 = −
𝛻𝑐 𝑥0 0 0

Yes
Check for Convergence Optimal Solution
No 𝐸 𝑥, 𝜆, 𝑧 ≤ 𝜖𝑡𝑜𝑙

Compute the Search Direction 𝑊𝑘 + Σ𝑘 𝛻𝑐 𝑥𝑘 𝑑𝑘𝑥 𝛻𝑓 𝑥𝑘 + 𝛻𝑐 𝑥𝑘 𝜆𝑘 − 𝑍𝑘

Σ𝑘 = 𝑋𝑘−1 𝑍𝑘 = −
with the linearized Barrier Problem 𝛻𝑐 𝑥𝑘 𝑇 0 𝑑𝑘𝜆 𝑐(𝑥𝑘 )

𝑑𝑘𝑧 = 𝜇𝑋𝑘−1 𝑒 − 𝑧𝑘 − Σ𝑘 𝑑𝑘𝑥

Backtracking Line Search 𝑥𝑘+1 = 𝑥𝑘 + 𝛼𝑘 𝑑𝑘𝑥

Determine 𝛼 by decrease in Merit Function or
𝜆𝑘+1 = 𝜆𝑘 + 𝛼𝑘 𝑑𝑘𝜆
with Filter Methods
𝑧𝑘+1 = 𝑧𝑘 + 𝛼𝑘 𝑑𝑘𝑧
Barrier Function Example 4
𝑚𝑖𝑛2 𝑥2 5 + 𝑥1
𝑥∈𝑅
𝑥1 𝑥2 ≥ 5
𝑠. 𝑡.
𝑥12 + 𝑥22 ≤ 20
Barrier Function: Example 4
min2 𝑥2 5 + 𝑥1
𝑥∈𝑅
𝑥1 𝑥2 ≥ 5
𝑠. 𝑡.
𝑥12 + 𝑥22 ≤ 20
Comparing Interior Point and Active Set Methods
100

90

80
APOPT+BPOPT
APOPT1.0
70
Percentage (%)

BPOPT1.0
60 IPOPT3.10
IPOPT2.3
50
SNOPT6.1
40 MINOS5.5

30

20 NLP Benchmark – Summary (494 Problems)

10
0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5
Not worse than 2 times slower than the best solver ()
APOPT Solver
• AMPL / MATLAB / Python Interfaces from http://apopt.com
IPOPT Solver
• Download Source Code from https://projects.coin-or.org/Ipopt
Interior Point Homework Problem 1
min2 𝑥12 − 2𝑥1 𝑥2 + 4𝑥22
𝑥∈𝑅
𝑠. 𝑡. 0.1𝑥1 − 𝑥2 > 1
Interior Point Homework Problem 2
min2 𝑥12 + 2𝑥22
𝑥∈𝑅
𝑠. 𝑡. 2𝑥1 + 𝑥2 ≤ 9
𝑥1 + 2𝑥2
𝑥1 > 0, 𝑥2 > 0
Homework Help: IPOPT Output
• Visit: http://apmonitor.com/online/view_pass.php?f=ipm.apm
Homework Help: IPOPT Output
IPOPT Output Headings
• iter: The current iteration count.
• objective: The unscaled objective value at the current point.
• inf_pr: The unscaled max constraint violation at the current point.
• inf_du: The max scaled dual infeasibility at the current point.
• lg(mu): log10 of the value of the barrier parameter m
• ||d||: The infinity norm (max) of the primal step
• lg(rg): log10 of the value of the regularization term for the Hessian of the
Lagrangian. Dash (-) indicates that no regularization was done.
• alpha_du: The stepsize for the dual variables
• alpha_pr: The stepsize for the primal variables
• ls: The number of backtracking line search steps
Homework Help: IPOPT Output
Homework Help
• Download BPOPT Solver (MATLAB version) from Interior Point Page
• http://apmonitor.com/me575/index.php/Main/InteriorPointMethod
• Homework Problem 1 = BPOPT problem 10
• Homework Problem 2 = BPOPT problem 6

% load bpopt libraries

addpath('bpopt')

% Solve problem 1
prob = bp_create(10); % create problem
sol = bpopt(prob); % solve problem