T he E llipsoid m ethod

R.M. Freund/ C. Roos (MIT/TUD) e-mail: C.Roos@tudelft.nl URL: http://www.isa.ewi.tudelft.nl/roos

WI 4218
March 21, A.D. 2007

Optimization Group

1/26

Outline
Sherman-Morrison formula Ellipsoids Ellipsoid method Basic construction Prototypical iteration Iteration bound Two theorems Two more theorems Optimization with the ellipsoid method Convexity of the homogeneous problem The Key Proposition

Optimization Group

2/26

In memory of George Danzig and Leonid Khachyan (1)

Leonid Khachiyan (1952-2005) passed away Friday April 29 at the age of 52. He died in his sleep, apparently of a heart attack, colleagues said. Khachiyan was best known for his 1979 use of the ellipsoid algorithm, originally developed for convex programming, to give the rst polynomial-time algorithm to solve linear programming problems. While the simplex algorithm solved linear programs well in practice, Khachiyan gave the rst formal proof of an ecient algorithm in the worst case. He was among the worlds most famous computer scientists, said Haym Hirsh, chairman of the computer science department at Rutgers. George B. Dantzig, the father of Linear Programming and the creator of the Simplex Method, died at the age of 90 on May 13. In a statement INFORMS President Richard C. Larson mourned his death (see http://www.informs.org/Press/dantzigobit.htm). The tributes included obituaries in the Washington Post, the Francisco Chronicle, Mercury News and New York Times. National Public Radio commentator Keith Devlin remembered George Dantzig in a broadcast on Saturday, May 21.
Optimization Group 3/26

In memory of George Danzig and Leonid Khachyan (2)

Optimization Group

4/26

Leonid Khachyan in Shanghai (with Bai, Peng and Terlaky, 2002)

Optimization Group

5/26

Sherman-Morrison formula
Let Q, R, S, T be matrices such that Q and Q + RS T are nonsingular. R and S are n k matrices of rank k n. Then
(Q + RS T )1 = Q1 Q1 R(I + S T Q1 R)1 S T Q1

Lemma 1 Let U be such that QU = R. Then I + S T U is invertible. Proof: Suppose w Rk satises (I + S T U )w = 0. Then,
(Q + RS T )U w = (QU )w + R(S T U w) = Rw Rw = 0.

Q + RS T being nonsingular, this gives U w = 0. Since rank (U ) = k this implies w = 0.

Theorem 1 If Qx0 = q and (I + S T U )y = S T x0 then x = x0 U y satises (Q + RS T )x = q . Proof:

(Q + RS T )(x0 U y) = Qx0 + RS T x0 QU y RS T U y = q + R(I + S T U )y Ry RS T U y = q. 2

The solution x of (Q + RS T )x = q is given by x = x0 U y = Q1 q Q1 R(I + S T U )1S T x0 Since Theorem 1 holds for all q Rn the Sherman-Morrison formula follows.
= (Q1 Q1 R(I + S T Q1 R)1 S T Q1 )q.

Reference: W. W. Hager. Updating the inverse of a matrix. SIAM Rev., 31(2):221239, June 1989.
Optimization Group 6/26

Ellipsoids
Let M be a (symmetric) positive denite n n matrix and z Rn . Then
EM,z := x : (x z)T M (x z) 1

denotes an ellipsoid centered at z . Note that

x EM,z
1

M 2 (x z) 1.
1 1

Hence, putting u = M 2 (x z), we have x = z + M 2 u and u 1. Therefore,

EM,z = x = z + M 2 u : uT u 1 .

In other words,
EM,z = z + M 2 B(0, 1), where B(0, 1) denotes the unit sphere, centered at the origin. The volume of B(0, 1) is given by 2 (n) = , n +1 2
n 1

and the volume of EM,z by

vol EM,z = (n) det(M ) .

Hence
ln vol EM,z = ln (n) 1 ln det(M ). 2
7/26

Optimization Group

Ellipsoid method Given is a set convex S . We want to nd s S . We assume that an ellipsoid EM,z is given such that S EM,z . Step 1 : k = 0; M k = M ; z k = z ; Step 2 : if z k S : STOP; Step 3 : Find nonzero vector a such that
aT x aT z k , x S;

Step 4 : Construct smallest volume ellipsoid that contains

EM,z x Rn : aT (x z k ) 0 ;

Let this ellipsoid have matrix M k+1 and center z k+1. Step 5 : k = k + 1; Step 6 : Goto Step 2.
Optimization Group 8/26

Basic construction (1)

Find m = (z, 0, . . . , 0), M = diag (a1, a2, . . . , an), such that the ellipsoid
EM,m = x Rn : (x m)T M (x m) 1 contains B(0, 1) {x : x1 0} and has minimal volume. Note that
n

(x m) M (x m) = a1(x1 z) +

aix2 . i
i=2

The unit vectors e1, e2, . . . , en are on the boundary of B(0, 1) {x : x1 0}. We require them to lie on the boundary of the ellipsoid. This gives:
a1(1 z)2 = 1

a1z 2 + ai = 1,

From this we obtain

1 a1 = , (1 z)2
2

2 i n. i 2.

Recall that vol EM,m is minimal if det M is maximal. Since

z2 1 2z ai = 1 a1 z = 1 = , (1 z)2 (1 z)2 (1 2z)n1 det M = , (1 z)2n

1 . n+1 2

one may easily verify that this occurs if z =

a1 =

Substituting this value we obtain

ai = 1 1 , 2 n i 2.
9/26

1 1+ n

Optimization Group

Illustration for n = 2

-1 -1 0 1

Optimization Group

10/26

Basic construction (2)

(n + 1)2 a1 = , n2 B M E = = n2 1 ai = , i 2. n2 H = {y : y1 0} , z=
1 e n+1 1

n2 1 n2

y : yT y 1 , I+

2 e eT n1 1 1

= EM , z

Theorem 2 (B H) E. Proof: One has y E if and only if

e1 y n+1
T

n2 1 n2
n

This is equivalent to

2e1 eT 1 I+ n1

e1 n+1

1.

n2 1 1 2n + 2 2 yi + 2 + y1 (y1 1) 1. n2 i=1 n n2

Now let y B H . Then 0 y1 1 implies y1 (y1 1) 0. Also

n2 1 1 + 2 = 1, n2 n

n 2 i=1 yi

1. Since

we obtain y E .
Optimization Group

11/26

Basic construction (3)

B = y : yT y 1

2 = n 1 I + 2 e1eT M 1 n1 n2 E = EM , z

Theorem 3 vol (E ) < Proof: One has

1 vol (B ) e 2(n+1) .

vol(E) = det I = 1 . vol(B) det M det M n2 1 n2 n 2 1 + n1

Moreover,
=
n2 1 n2 n1 n+1 2 . n

det M =

Hence, using 1 + x ex we get

1 det M

n1 2 n2 1 e n+1 e

n2 n2 1

n1

n1 2 n 2 = 1+ 1 1 1 n+1 n+1 n21

1 n+1 . e

This implies the theorem.

Optimization Group

12/26

Prototypical iteration
EM,z := x : (x z ) M (x z ) 1 = z + M
T 1 2

B(0, 1)

We know M , z , and a nonzero vector a. Dene

2 T = n 1 M + 2 T aa 1 , M n1 a M a n2

1 M 1a z = z + n+1 T 1 a M a

By the Sherman-Morrison formula, one has

M 1 n2 2 M 1aaT M 1 1 = 2 M n 1 n + 1 aT M 1a

Theorem 4
EM,z x : aT x aT z EM ,. z

Theorem 5
vol EM , < vol EM,z z
1 e 2(n+1) .

Optimization Group

13/26

Proof of Theorem 5
u := M
1 2

a,

b := u e1, RT = R,

R :=

2(u + b)(u + b)T u+b R2 = I.

2

One has Hence

M = = = = =

Ru = b,
2 aaT T M 1 a n1 a

n2 1 n2

M+

1 n2 1 M2 n2

I+

2 M 2 aaT M 2 n1 aT M 1 a
1

M2
1 1

1 n2 1 M 2R n2 1 n2 1 M 2R n2 1 n2 1 M 2R n2

I+ I+ I+

2 RM 2 aaT M 2 R n1 aT M 1a 2 RM 2 aaT M 2 R n1 aT M 1 RRM 1 a 2 2 2 e1 eT 1 n1

1

RM 2 RM 2
1

RM 2 .

Therefore,
det M =
n2 1 n det M n2

(det R)2 det I + n2 1 n2

n

2 e eT n1 1 1

n2 1 n n2

1+

2 n1

det M,

and
det M = det M

1+

Hence
vol EM , z vol EM,z

2 n1

> e n+1 .

1 det M 2(n+1) = <e det M

This proves Theorem 5.

Optimization Group

14/26

Proof of Theorem 4
EM,z x : aT x aT z EM ,. z

Proof: Suppose (x z)T M (x z) 1 and aT x aT z . Put

u = M 2 a,
1

Ru = b = u e1 ,
1 1

Then Moreover,

y := RM 2 (x z)

y T y = (x z)T M 2 R2M 2 (x z) = (x z)T M (x z) 1. u y1 = u eT y = bT y = uT Ru 1

1 1

Now
(x z

= aT M 2 RRM 2 (x z) = aT (x z) 0. (x z )T M (x z ) =

1 1 M 1 a 1 )T M (x z n+1 M a ) = T 1 T n+1 a M a a M 1 a 1 1 1 2 1 1 2 (x z n+1 M a )T M 2 R n n2 I + n1 e1eT RM 2 1 aT M 1 a 1 2 1 y n+1 uu e1 I + n1 e1 eT y n+1 uu e1 1. 1

xz

M a aT M 1 a

The inequality follows just as in the proof of Theorem 1, since y T y 1 and y1 0.

Optimization Group

15/26

Two theorems
Theorem 6 Suppose we want to nd a point in the set S , with S EM 0,z0 and vol (S) > 0. Then the algorithm will nd a point in S after at most
2(n + 1) ln vol EM 0,z0 vol (S)

iterations. Proof: After k iterations, we have

vol EM k ,zk vol EM 0,z0 e 2(n+1) . vol (S) vol EM 0,z0 e 2(n+1) . ln vol (S) ln vol EM 0,z0 k , 2(n + 1)
k k

Since S EM k ,zk for each k we obtain Hence

which implies
k 2(n + 1) ln vol EM 0,z0 vol (S) .

This proves the theorem.

Optimization Group

16/26

Two theorems (cont.)

Let
B(c, ) := {x : x c } .

Theorem 7 Suppose we know R such that S B(0, R) and that S contains a ball B (, r) for some x and x r > 0. Then the algorithm will nd a point in S after at most
2n(n + 1) ln R r

iterations. Proof: After k iterations, we have

vol (B (, r)) vol (S) vol (B(0, R)) e 2(n+1) . x k , 2(n + 1)
k

Taking logarithms again we get

ln vol (B (, r)) ln vol (B(0, R)) x

and hence
k 2(n + 1) ln
vol(B(0,R)) vol(B(,r)) x

= 2(n + 1) ln

(n)Rn (n)r n

= 2(n + 1)n ln

R r

This proves the theorem.

Optimization Group

17/26

Two more theorems Actually, we can improve the previous two results a lot in terms of what we need to know. Theorem 8 Suppose vol S EM 0,z 0 > 0. Then the algorithm will nd a point in S after at most
2(n + 1) ln

vol EM 0,z 0 vol S EM 0,z 0

iterations. Proof: Obvious.

Theorem 9 Suppose we know R and that S B(0, R) contains a ball B (x, r) for some x and r > 0. Then the algorithm will nd a point in S after at most
R 2n(n + 1) ln r

iterations. Proof: Obvious.

Optimization Group

18/26

Optimization with the ellipsoid method

Consider the minimization problem Let
z = min cT x : x S . P := x : x S, cT x z + .

Theorem 10 Suppose we know EM 0,z0 such that EM 0,z0 P = . Then the algorithm will nd a point in P after at most
2(n + 1) ln vol EM 0,z0 vol EM 0,z0 P

iterations. Proof: Obvious.

Theorem 11 Let us know R and B(0, R) P contains a ball B (, r) for some x and r > 0. Then the x algorithm will nd a point in P after at most
2(n + 1)n ln R r

iterations. Proof: Obvious.

Optimization Group

19/26

Optimization (cont.) Unfortunately we must know R, and this is most unfortunate. Let us x this.
min c x : x S
T

(H)

y cT y min : S, > 0 .

Note that the problem (H) at the right is homogeneous: if (y, ) is feasible (or optimal) then so is (y, ) for any > 0. Taking = 1/ we get a feasible (or optimal) solution of (P ). The objective function in (H) is not convex. However, the level sets are convex and this is enough for the ellipsoid method. As far as y concerns, the feasible region is the cone generated by the convex set S , without the origin (since > 0). Hence the (convex!) feasible region H is not closed, but also this does not hurt the ellipsoid method. Finally, by adding the constraint
(y, ) 1

we lose nothing, and the whole feasible region lies in B((0, 0), 1)!
Optimization Group 20/26

Convexity of the homogeneous problem Proposition 1 The set

(y, ) : y S, > 0

is convex. Proof: Let (y1, 1), (y2, 2) belong to the set. For 0 1, let (y, ) = (y1, 1) + (1 )(y2, 2).
y y1 + (1 )y2 = = 1 + (1 )2 1 y1 + (1 )2 y2
1 2

where =

1 . 1 +(1)2

1 + (1 )2

y y = 1 + (1 ) 2 S, 1 2

Proposition 2 The level sets of Proof: For any z R one has

cT y

are convex.

cT y z cT y z 0. Here we used > 0. This implies the result.

Optimization Group

21/26

The Key Proposition We have seen that we can apply the ellipsoid method to the homogeneous problem (H), with R = 1.
Bn denotes the unit sphere in Rn.

Proposition 3 Suppose the level set P contains a ball B(, r). Let x
HP := (y, ) : y P, > 0 .
2
n+1 2

Then
vol Bn+1 vol HP Bn+1 (n + 1)(n + 1) 1 + (r + x ) (n)rn

Theorem 12 Solving (P ) via (H), starting at the unit ball Bn+1, the algorithm will nd an -solution of (H) (and hence of (P )) after at most
(n + 1)(n + 2) ln 1 + r2 x 1+ r
2

+ 2(n + 2) ln r n

iterations.
Optimization Group 22/26

Graphical illustration

Bn+1
x
Optimization Group 23/26

Proof of Proposition 3
HB(, r) := (y, ) : x y B(, r), > 0 . x

The longest vector in the set

T := (y, ) : y B(, r), 1 HB(, r) x x

is the vector and has length

:= x 1

+r

x x

=
2

1+ 1

r x

1+

r 1+ x

1 + (r + x )2.

Hence Therefore,
1T Bn+1 HB(, r) Bn+1 HP. x 1 T 1 n+1 1 n+1
0
n+1 2

vol (Bn+1 HP )

vol

vol (T ) =

(n)n rnd .

(n)rn (n)rn = (n + 1) n+1 (n + 1) 1 + (r + x )2

Hence Proposition 3 follows.

Optimization Group

24/26

Second proof of Proposition 3

HB(, r) := (y, ) : x

Let dS(x) denote an innitesimal surface element around x T , where y T := (y, ) : B(, r), = 1 HB(, r). x x The surface on Bn+1 cut o by the cone generated by dS(x) is equal to
sin (x) dS(x) , (x)n where (x) = (x, 1) , and sin (x) = 1/(x). Thus the surface on Bn+1 cut o by the cone generated by T equals dS(x) 1 (n) rn n+1 dS(x) = n+1 , n+1 xB(,r) (x) x xB(,r) x because we have for each x T that (x) , with as in the previous proof. Hence vol Bn+1 HP is at least equal to (n) rn vol Bn+1 = . vol Bn+1 = n+1 (n + 1) (n + 1) (n + 1) surf Bn+1
(n) rn n+1 (n) rn n+1

y B(, r), > 0 . x

Hence, in the same way as in the previous proof, Proposition 3 follows.

Optimization Group

25/26

Proof of Theorem 12
By Theorem 10 the algorithm needs at most
2(n + 2) ln vol (Bn+1 ) vol (HP Bn+1 )

iterations. Due to Proposition 3 this is (n + 1)(n + 1) 1 + (r + x )2 2(n + 2) ln (n)rn Since

n+1

n+1 2

(n + 1) 2 n + 1 (n + 1)(n + 1) 2 = n, n (n) 2 n+1 + 1 2

we obtain (omitting the brackets) the bound

2(n + 2) ln
n+1 2 n(1+(r+ x ) ) 2 rn

= 2(n + 2) ln r n = (n + 1)(n + 2) ln
1 r2

1 r2

+ 1+
x r

x r 2

n+1 2

+ 1+

+ 2(n + 2) ln r n .

This completes the proof.

Optimization Group

26/26