Notes 3: From Geometry to Simplex Method

IND E 513

IND E 513

Notes 3

Slide 1

Polyhedra in Standard Form

Definition
A polyhedron in standard form is given by {x R n |Ax = b, x 0},
where A is an m n matrix and b is a vector in R m .
I
I

There are m equality constraints in n non-negative variables.

No loss of generality (LOG) in assuming rows of A are linearly
independent for a non-empty standard form polyhedron (page
57). We will make this assumption all through. (m n).

How can we tell whether a vector is a basic solution?

Theorem
A vector y R n is a basic solution if and only if Ay = b and there
exist indices B(1), . . . , B(m) such that
1. AB(1) , AB(2) , . . . , AB(m) are linearly independent;
2. If i 6= B(1), B(2), . . . , B(m), then yi = 0.
This also hints at a procedure for constructing basic solutions.
IND E 513

Notes 3

Slide 2

Constructing a Basic Solution for Standard Form Polyhedra

1. Choose m linearly independent columns
AB(1) , AB(2) , . . . , AB(m) .
2. Let xi = 0 for all i 6= B(1), B(2), . . . , B(m).
3. Solve the system of m equations
AB(1) xB(1) + AB(2) xB(2) + . . . + AB(m) xB(m) = b
for m unknowns xB(1) , . . . , xB(m) .

Example

1
0

1
0

1
1
0
1

2
6
0
0

1
0
0
0

0
1
0
0

IND E 513

0
0
1
0

0
8

0
12
x =
4
0
1
6
Notes 3

Slide 3

Terminology
If x is a basic solution
I

AB(1) , AB(2) , . . . , AB(m) are called basic columns. They are

linearly independent and form a basis for R m .

xB(1) , xB(2) , . . . , xB(m) are called basic variables; the remaining

variables nonbasic.

By arranging the m basic columns next to each other, we

obtain an m m matrix B called a basis matrix. B is
invertible.

xB(1)
|
|
|

B = AB(1) AB(2) AB(m) , xB = ...

|
|
|
xB(m)

xB

= B 1 b

IND E 513

Notes 3

Slide 4

Visualizing Standard Form Polyhedra

Can we visualize and draw standard form polyhedra for n 3?
Yes, if n m = 2, the feasible region can be drawn as a
two-dimensional region defined by n linear inequality constraints.

Example
x1 + x2 + x3 = 1
x1 , x2 , x3 0

IND E 513

Notes 3

Slide 5

Existence of Extreme Points

Back to general polyhedra {x R n |Ax b}. Does a polyhedron
always have an extreme point, i.e., a basic feasible solution, i.e, a
vertex?
Of course not! Example: a halfspace in R n for n > 1.

Definition
A polyhedron P R n contains a line if there exist a vector x P
and a nonzero vector d R n such that x + d P for all scalars .
IND E 513

Notes 3

Slide 6

Theorem
Suppose that the polyhedron
P = {x R n |ai0 x bi , i = 1, 2, . . . , m} is nonempty. Then, the
following are equivalent
1. P has at least one extreme point.
2. P does not contain a line.
3. There exist n vectors out of a1 , a2 , . . . , am , which are linearly
independent.

Corollary
Every nonempty bounded polyhedron and every nonempty
polyhedron in standard form has at least one extreme point.

IND E 513

Notes 3

Slide 7

Partial Proof of the Above Theorem

Proof that (1) (3): If P has an extreme point x, then x is also a
basic feasible solution. Hence there exist n constraints that are
active at x and the corresponding vectors ai are linearly
.
independent
Proof that (3) (2): by contradiction. Suppose (WLOG) that
vectors a1 , a2 , . . . , an are linearly independent and yet P contains
some line x + d for some vector d 6= 0. Hence ai0 (x + d) bi
for all i and all scalars . This implies that ai0 d = 0 for all i.
Specifically,
n
X
di ai = 0.
i=1

But this implies that d = 0 because the vectors a1 , . . . , an are

linearly independent
.
Proof that (2) (1): (page 63).
IND E 513

Notes 3

Slide 8

Optimality of Extreme Points

Theorem
Consider the linear programming problem of minimizing c 0 x over a
polyhedron P. Suppose that P has at least one extreme point and
that there exists an optimal solution. Then, there exists an optimal
solution that is an extreme point of P.
Proof: Let Q be the set of optimal solutions (assumed to be
non-empty). Suppose P = {x R n |Ax b} and let v be the
optimal value of the cost c 0 x. Then
Q = {x R n |Ax b, c 0 x = v }, which is also a polyhedron. Since
Q P and P contains no line, Q contains no line and hence has
at least one extreme point. Let x be an extreme point of Q.
Claim: x is also an extreme point of P.

IND E 513

Notes 3

Slide 9

Optimality of Extreme Points contd.

Proof of Claim: by contradiction. Suppose x is not an extreme

point of P. Then there exist y , z P such that y 6= x , z 6= x ,
and some [0, 1] such that x = y + (1 )z. Then
v = c 0 x = c 0 y + (1 )c 0 z. Furthermore, since v is the optimal
cost, c 0 y v and c 0 z v . This implies that c 0 y = v and c 0 z = v
and therefore y Q, z Q. However, this contradicts the fact
that x is an extreme point of Q.
To conclude, x is optimal to
min c 0 x x P
and is an extreme point of P

IND E 513

Notes 3

Slide 10

Optimality of Extreme Points contd.

Theorem
Consider the linear programming problem of minimizing c 0 x over a
polyhedron P. Suppose that P has at least one extreme point.
Then, either the optimal cost is equal to , or there exists an
extreme point which is optimal.
This implies that if P has an extreme point, and the optimal cost
is finite, then the problem has an extreme point optimal solution.
Since a non-empty standard form polyhedron always has an
extreme point, it has an extreme point optimal solution if the
optimal cost is finite.

Corollary
Consider the linear programming problem of minimizing c 0 x over a
nonempty polyhedron P. Then, either the optimal cost is or
there exists an optimal solution.
IND E 513

Notes 3

Slide 11

Degeneracy
Definition
A basic solution x R n is said to be degenerate if more than n of
the constraints are active at x. x is said to be non-degenerate
otherwise.

Example
Suppose P R 3 is given by
x1 + x2 + 2x3 8
x2 + 6x3 12
x1 4
x2 6
x1 , x2 , x3 0
(4, 0, 2) is a degenerate basic feasible solution.
(2, 6, 0) is a non-degenerate basic feasible solution.
IND E 513

Notes 3

Slide 12

C is a degenerate basic feasible solution, D is a degenerate basic

solution (infeasible), E is a nondegenerate basic feasible solution

IND E 513

Notes 3

Slide 13

Degeneracy in Standard Form Polyhedra

Definition
A basic solution x to a standard form polyhedron is degenerate if
more than n m of the components of x are zero.

Example

1
0

1
0

1
1
0
1

2
6
0
0

1
0
0
0

0
1
0
0

0
0
1
0

0
8
12
0
x =
4
0
1
6

n = 7, m = 4. A1 , A2 , A3 , A7 are linearly independent. So set

x4 , x5 , x6 to zero, and solve the system Ax = b to get
x = (4, 0, 2, 0, 0, 0, 6), a degenerate basic feasible solution.

IND E 513

Notes 3

Slide 14

Optimality Conditions
Feasible Directions:
Many optimization algorithms move from one feasible point to
another until they find an optimal solution. At a point x P, we
are contemplating moving away from it in the direction of vector
d R n . We should consider those choices of d that do not take us
outside the feasible region.

Definition
Let x be an element of polyhedron P. A vector d R n is said to
be a feasible direction at x, if there exists a positive scalar for
which x + d P.

IND E 513

Notes 3

Slide 15

Basic Directions for Standard Form Problems

Throughout this section, we will focus on standard form problems.
min c 0 x Ax = b, x 0.
Let x be a basic feasible solution, and B(1), . . . , B(m) be the
indices of the basic variables. Let B = [AB(1) , . . . , AB(m) ] be the
corresponding basis matrix. Thus, xi = 0 for i 6= B(1), . . . , B(m),
while xB = (xB(1) , . . . , xB(m) ) is given by
xB = B 1 b.
We consider the possibility of moving away from x, to x + d, by
selecting a nonbasic variable xj (which is currently at zero level),
and increasing it to a positive value , while keeping the remaining
nonbasic variables at zero. Thus, dj = 1, and di = 0 for every
nonbasic index i other than j.
The vector of basic variables xB changes to xB + dB , where
dB = (dB(1) , . . . , dB(m) ).
IND E 513

Notes 3

Slide 16

We have Ax = b and we also want A(x + d) = b for > 0

(why?). Thus, we must have Ad = 0.

Ad =

n
X

Ai di =

i=1

m
X

AB(i) dB(i) + Aj = BdB + Aj

i=1

dB = B 1 Aj .
The direction d we just constructed is called the jth basic direction.
We ensured that the equality constraints hold. How about
non-negativity? Need to worry only about basic variables (why?)
1. If x is nondegenerate, xB > 0, and xB + dB 0 for
sufficiently small .
2. some complications if x is degenerate (see picture below).

IND E 513

Notes 3

Slide 17

x3=0
F

x5=0
x4=0
x1=0

x2=0
G

IND E 513

Notes 3

Slide 18

Change in Cost Along Basic Directions

Rate of cost change along basic direction d: c 0 (x + d) c 0 x = c 0 d.

cd=

n
X

c i di =

i=1

m
X

cB(i) dB(i) + cj = cB0 dB + cj = cj cB0 B 1 Aj ,

i=1

where cB0 = (cB(1) , . . . , cB(m) ).

Definition
Let x be a basic solution, let B be an associated basis matrix, and
let cB be the vector of costs of the basic variables. For each j, we
define reduced cost cj of the variable xj according to the formula
cj = cj cB0 B 1 Aj .

IND E 513

Notes 3

Slide 19

Reduced Costs of Basic Variables Are Zero

Let xB(i) be a basic variable. Since B = [AB(1) , . . . , AB(m) ], we

have B 1 [AB(1) , . . . , AB(m) ] = I . I is the m m identity matrix.
In particular, B 1 AB(i) is the ith column of I , denoted ei . Then
we have
cB(i) = cB(i) cB0 B 1 AB(i) = cB(i) cB0 ei = cB(i) cB(i) = 0.

IND E 513

Notes 3

Slide 20

Optimality Conditions

Theorem
Consider a basic feasible solution x associated with a basis matrix
B, and let c be the corresponding vector of reduced costs.
1. If c 0, then x is optimal.
2. If x is optimal and nondegenerate, then c 0.

IND E 513

Notes 3

Slide 21

Proof of Optimality Conditions

Proof of the first part.
y be an arbitrary feasible solution. We show that c 0 y c 0 x, i.e.,
0 d 0.
c 0 (y x) 0. Let d = y x. We need to show c P
We must have Ad = 0 (why?). Therefore, BdB +
Ai di = 0,
iN

where N is the set of indices of the nonbasic

P variables
corresponding to B. This yields dB =
B 1 Ai di .
iN

c 0 d = cB0 dB +

X
iN

X
X
c i di =
(ci cB0 B 1 Ai )di =
ci di .
iN

iN

Now, for any i N, xi = 0. Moreover, yi 0 for all i and in

particular, for i N. Thus, di 0 for i N. In addition,
ci 0
P
for all i, and in particular for i N. Thus, c 0 d =
ci di 0
.
iN

IND E 513

Notes 3

Slide 22

Proof of Optimality Conditions contd.

Proof of the second part (by contrapositive).

Suppose x is a nondegenerate basic feasible solution and that
cj < 0 for some j. Then xj must be a nonbasic variable (why?)
and cj is the rate of change of cost along the jth basic direction.
By moving in this direction, we can obtain feasible solutions with
.
less cost because x is nondegenerate. Thus x is not optimal

Definition
A basis matrix B is said to be optimal if
1. B 1 b 0 (feasibility).
2. c0 = c 0 cB0 B 1 A 0 (optimality).

IND E 513

Notes 3

Slide 23