ECE750 Lecture

10:
NP-completeness
and Approximation
Algorithms
Todd Veldhuizen
tveldhui@acm.org
ECE750 Lecture 10: NP-completeness and
Approximation Algorithms
Todd Veldhuizen
tveldhui@acm.org
Electrical & Computer Engineering
University of Waterloo
Canada
Nov 20, 2007
1 / 79
ECE750 Lecture
10:
NP-completeness
and Approximation
Algorithms
Todd Veldhuizen
tveldhui@acm.org
Tractable problems I

Recall that P is the class of decision problems (i.e.,

those with yes/no answers) that can be answered in
time O(n
c
) on a Turing machine, c a constant.

In the 1960s, proposals were made to identify the class

P with feasible problems i.e., problems for which
computers can obtain an answer in a reasonable amount
of time. (One early proponent of this idea was Jack
Edmonds, a professor at Waterloo and a pioneer of
combinatorial optimization.)

This idea of P = feasible has gained widespread

acceptance, of course modulo the common-sense
caveats that must accompany such a blanket
generalization: algorithms requiring 10
100
n
2
or
O(n
1000
) time are unlikely to be practical. However,
most natural problems in P (those that arise from
2 / 79
ECE750 Lecture
10:
NP-completeness
and Approximation
Algorithms
Todd Veldhuizen
tveldhui@acm.org
Tractable problems II
questions of interest, as opposed to those articially
constructed by e.g., diagonalization) have low
exponents such as O(n), O(n
2
), O(n
3
), etc. Very few
interesting problems in P have best-known exponents of
n
10
or more. So, equating P with practical algorithms is
a quite reasonable generalization.

However, many very important problems are not known

to be in P. Some of these problems are conceptually
very close to problems that are known to be in P:

We have seen a polynomial-time algorithm to nd the

shortest path between two vertices in a graph. What
about nding the longest path between two vertices?
Nobody knows if this problem can be solved in
polynomial time.
3 / 79
ECE750 Lecture
10:
NP-completeness
and Approximation
Algorithms
Todd Veldhuizen
tveldhui@acm.org
Tractable problems III

An Euler tour of a connected graph is a path that starts

and ends at the same vertex and traverses each edge
exactly once. Testing whether a graph has an Euler tour
is trivial: a graph has an Euler tour if and only if every
vertex has an even number of edges incident on it. This
can clearly be tested in polynomial time! On the other
hand, a Hamiltonian circuit is a path that starts and
ends at the same vertex and visits each vertex exactly
once. It is not known whether Hamiltonian-circuit has a
polynomial time decision procedure.

Many of the interesting problems not known to be in P

are easily seen to belong to the class NP, which
4 / 79
ECE750 Lecture
10:
NP-completeness
and Approximation
Algorithms
Todd Veldhuizen
tveldhui@acm.org
Tractable problems IV
informally is the class of problems whose an-
swers we can check in polynomial time. More technically:
NP is the class of decision problems where, if
the answer is YES, there exists a certicate (a
string in some alphabet) that can be checked
in polynomial time by a Turing machine.

The terms proof and witness are commonly used in

place of certicate.

NP stands for nondeterministic polynomial time.

A deterministic machine is one whose state transition

relation allows only one possible trace: every state is
succeeded by a uniquely dened state, so that a trace
looks like
// // //

5 / 79
ECE750 Lecture
10:
NP-completeness
and Approximation
Algorithms
Todd Veldhuizen
tveldhui@acm.org
Tractable problems V

A nondeterministic machine can branch simultaneously

into several successor states, and the semantics are
typically dened so that the machine accepts an input if
any of its branches accept. A trace is a tree branching
into the future:
//


BB

// //


BB

`
`
`
`
`
//


AA

`
`
`
`
`
//

6 / 79
ECE750 Lecture
10:
NP-completeness
and Approximation
Algorithms
Todd Veldhuizen
tveldhui@acm.org
Tractable problems VI

Do not confuse nondeterminism with randomness! If

you know what threads are, a good conceptual model is
a multithreaded program where each thread can create
another new thread whenever it wants, and all threads
run simultaneously with no slowdown, as if there were
an innite number of processors and no resource
contention.

A nondeterministic machine can simultaneously explore

all possible certicates, and halt with a YES answer if it
nds one that certies the answer is YES.

The following diagram illustrates the containment

relationships between the classes we will visit in this
lecture:
7 / 79
ECE750 Lecture
10:
NP-completeness
and Approximation
Algorithms
Todd Veldhuizen
tveldhui@acm.org
Tractable problems VII
NPComplete
P
NP
coNP
NP hard

Note that NP contains the class P: every problem in P

is automatically an NP problem also.

Whether NP is a strict superset of P is a central

problem in complexity theory, and one of the Clay
Institute Millenium Problems. It could be that:
1. P ,= NP; or
2. P = NP; or
8 / 79
ECE750 Lecture
10:
NP-completeness
and Approximation
Algorithms
Todd Veldhuizen
tveldhui@acm.org
Tractable problems VIII
3. It could be that the truth of P = NP is independent
of the usual (ZF) set theory axioms, as the continuum
hypothesis and axiom of choice were found to be.

It is widely believed that P ,= NP, i.e., that hard

problems in NP require superpolynomial time.

Note that superpolynomial includes, but is not

equivalent to, exponential time. The class of
superpolynomial subexponential functions includes
functions of the form n
f (n)
where 1 f
n
ln n
. For
example, n
1+log log n
is superpolynomial and
subexponential.

A didactic ditty on the theme of superpolylogarithmic

subexponential functions may be found in [5].
9 / 79
ECE750 Lecture
10:
NP-completeness
and Approximation
Algorithms
Todd Veldhuizen
tveldhui@acm.org
Circuit-Satisability I

Example: Circuit-satisability. Given a single-output

boolean circuit composed of AND, OR, and NOT gates,
is there a setting of the input signals causing the output
to be true? (If so, the circuit is said to be satisable.
a
c
b
d
10 / 79
ECE750 Lecture
10:
NP-completeness
and Approximation
Algorithms
Todd Veldhuizen
tveldhui@acm.org
Circuit-Satisability II

Clearly we can answer this problem by enumerating a

truth table of all possible inputs:
a b c d
0 0 0 0
0 1 0 0
1 0 0 0
1 1 0 0
0 0 1 0
0 1 1 1
This requires 2
n
time, where n is the number of input
signals.

A certicate that the circuit is satisable is a setting

of the input signals a = 0, b = 1, c = 1 causing
the output to be true. We can say that
a = 0, b = 1, c = 1 is a witness to the satisability of
the circuit.
11 / 79
ECE750 Lecture
10:
NP-completeness
and Approximation
Algorithms
Todd Veldhuizen
tveldhui@acm.org
Circuit-Satisability III

A verier can check very eciently (on a RAM model

in, say, O(m) time where m is the number of gates)
that this combination of inputs does, in fact, produce
d = 1. Hence the problem is in NP.
12 / 79
ECE750 Lecture
10:
NP-completeness
and Approximation
Algorithms
Todd Veldhuizen
tveldhui@acm.org
The class co-NP I

The class co-NP consists of decision problems having a

NO certicate (proof,witness) that can be checked in
polynomial time.

Example: circuit equivalence. Decision problem: given

two single-output boolean circuits, do they compute the
same function?
a
c
b
d
d
Are d and d

always the same?

13 / 79
ECE750 Lecture
10:
NP-completeness
and Approximation
Algorithms
Todd Veldhuizen
tveldhui@acm.org
The class co-NP II

Again, we can decide whether the two circuits are

equivalent by enumerating the truth table:
a b c d d

0 0 0 0 0
0 1 0 0 0
1 0 0 0 0
1 1 0 0 0
0 0 1 0 0
0 1 1 1 1
1 0 0 0 0
1 0 1 1 0
They dier on the input a = 1, b = 0, c = 1. The
answer to the decision problem is NO.

The input signal setting a = 1, b = 0, c = 1 is a

witness to the fact that the two circuits are not
equivalent.
14 / 79
ECE750 Lecture
10:
NP-completeness
and Approximation
Algorithms
Todd Veldhuizen
tveldhui@acm.org
The class co-NP III

Note that the complement of an NP set is a co-NP

set, and vice versa.

Example: the set of circuits that are not satisable is a

co-NP set. (The complement of satisable circuits.)

Example: the set of pairs of circuits that are

inequivalent is an NP set. (The complement of
equivalent pairs of circuits.)

It is believed that P ,= NP co-NP, but this is not

known.
15 / 79
ECE750 Lecture
10:
NP-completeness
and Approximation
Algorithms
Todd Veldhuizen
tveldhui@acm.org
Part I
Some examples of NP problems
16 / 79
ECE750 Lecture
10:
NP-completeness
and Approximation
Algorithms
Todd Veldhuizen
tveldhui@acm.org
Decision vs. optimization problems I

The classes P and NP are dened in terms of decision

problems with YES/NO answers.

Many of the interesting problems in NP are

optimization problems, where one aims to minimize or
maximize an objective function. Such problems can be
expressed in two versions:
1. The decision version: Does there exist a ... of cost less
than/more than ...? (This has a YES/NO answer, and
can be said to be in the class NP if the objective
function can be evaluated in polynomial time, etc.)
2. The optimization version: Find a ... of
minimal/maximal cost/prot. (There is a class NPO of
NP optimization problems we shall see next lecture, in
which problems are expressed in this form.)

The class of NP-hard optimization problems includes

many of high industrial relevance:

VLSI layout
17 / 79
ECE750 Lecture
10:
NP-completeness
and Approximation
Algorithms
Todd Veldhuizen
tveldhui@acm.org
Decision vs. optimization problems II

optimizing compilers (register allocation, alias analysis)

choosing warehouse locations

vehicle routing (e.g., designing delivery routes)

designing networks (bre, highways, subways, public

transit)

spectrum allocation on wireless networks

airline schedule planning

packing shipping containers

choosing an investment portfolio

verication of circuits

mixed-integer programming

(simplied models of) protein folding

and thousands more!

18 / 79
ECE750 Lecture
10:
NP-completeness
and Approximation
Algorithms
Todd Veldhuizen
tveldhui@acm.org
Example: Graph colouring I

Graph colouring: given a graph G, an r -colouring of G

is a function : V 1, 2, . . . , r from vertices to
colours 1, 2, . . . , r such that no two adjacent vertices
have the same colour: if (v
1
, v
2
) E, then
(v
1
) ,= (v
2
).

The least r such that an r -colouring is possible is called

the chromatic number of G.

Graph colouring is an NP problem: a witness to the

decision problem Does there exist a colouring with r
colours? is an r -colouring of the graph.

Graph colouring has numerous practical applications. It

is an excellent tool for reasoning about contention for
resources.
19 / 79
ECE750 Lecture
10:
NP-completeness
and Approximation
Algorithms
Todd Veldhuizen
tveldhui@acm.org
Example: Graph colouring II

For example, one of the last steps of compiling a

program is to emit assembly code. The program will run
faster if variables are placed in registers, rather than in
stack- or heap-locations. In a typical register allocation
scenario, one has a sequence of operations such as this:
y := 3
x := y + 1
z := x + 2
w := x + 1
u := z w
Each variable has a liveness range: from its initial
assignment to its last use. We want to place these
variables in registers in such a way that a register is
never simultaneously being used to store two variables.
(Here, the registers are the resource under contention.)
20 / 79
ECE750 Lecture
10:
NP-completeness
and Approximation
Algorithms
Todd Veldhuizen
tveldhui@acm.org
Example: Graph colouring III

This problem can be solved by graph coloring [1]:

Construct a graph whose vertices are variables, and

there is an edge (x, y) if the liveness ranges of x and y
overlap. Liveness ranges:
u
y = 3
x = y + 1
z = x + 2
w = x + 1
u = z * w
return u
x
y
z
w

Registers are colours: a valid colouring ensures that if

the liveness ranges of two variables overlap, they will be
in dierent registers.
21 / 79
ECE750 Lecture
10:
NP-completeness
and Approximation
Algorithms
Todd Veldhuizen
tveldhui@acm.org
Example: Graph colouring IV
y
x
z
w
u

In this example, three registers suce: put the red

variables in r
1
, the green variables in r
2
, and the blue
variable in r
3
.
r
1
3
r
2
+(r
1
, 1)
r
1
+(r
2
, 2)
r
3
+(r
2
, 1)
r
2
+(r
1
, r
3
)
22 / 79
ECE750 Lecture
10:
NP-completeness
and Approximation
Algorithms
Todd Veldhuizen
tveldhui@acm.org
Example: Graph colouring V

Some other examples of colouring:

Spectrum allocation: given a set of transmitter

locations, construct a graph where the vertices are
transmitters and edges indicate pairs of transmitters
close enough to interfere with each other if assigned the
same frequency. Then, the chromatic number of this
graph tells you how many dierent frequencies are
required for none of the towers to interfere with each
other. A colouring gives you a frequency assignment
that achieves this.

Circuit layout: given a planar layout of a circuit, build a

graph where vertices represent wires and there is an
edge between two wires if they cross. Then, the
chromatic number tells you how many layers suce to
route all the signals with none of them crossing. (This
also works for subways!)
23 / 79
ECE750 Lecture
10:
NP-completeness
and Approximation
Algorithms
Todd Veldhuizen
tveldhui@acm.org
Example: Steiner networks I

Suppose a company has a number of downtown

locations it wants to connect with a highspeed ber
network. Suppose bre costs $100/m to lay in a
downtown area, and can only travel along streets.
location
street

Problem: nd a minimal-cost network connecting the

locations.
location
street
24 / 79
ECE750 Lecture
10:
NP-completeness
and Approximation
Algorithms
Todd Veldhuizen
tveldhui@acm.org
Example: Steiner networks II

This is the rectilinear Steiner tree problem, which has

been used in VLSI layout to route signals. (Rectilinear
here refers to the constraint that edges travel only
up-down or left-right.)

Finding a minimal Steiner tree is an optimization

problem, the decision version of which (Does there exist
a network of cost < x?) is in NP.
25 / 79
ECE750 Lecture
10:
NP-completeness
and Approximation
Algorithms
Todd Veldhuizen
tveldhui@acm.org
Example: Graph Partitioning

Problem: Given a weighted graph G = (V, E, w), where

w : E Q
+
, partition V into V
1
, V
2
where
[[V
1
[ [V
2
[[ 1 and so that the total weight of cut
edges is minimized:
minimize

(v
1
,v
2
)E : v
1
V
1
,v
2
V
2
w(v
1
, v
2
)

This has numerous applications in parallel load

balancing, circuit layout, improving locality of reference,
etc.
26 / 79
ECE750 Lecture
10:
NP-completeness
and Approximation
Algorithms
Todd Veldhuizen
tveldhui@acm.org
Example: Travelling Salesperson Problem (TSP)

Problem: Given a weighted graph G = (V, e, w), where

w : E Q
+
, nd a circuit that starts and ends at the
same vertex, visiting each vertex exactly once, and
minimizes the sum of the costs of the edges in the
circuit.
27 / 79
ECE750 Lecture
10:
NP-completeness
and Approximation
Algorithms
Todd Veldhuizen
tveldhui@acm.org
Part II
NP-Complete problems
28 / 79
ECE750 Lecture
10:
NP-completeness
and Approximation
Algorithms
Todd Veldhuizen
tveldhui@acm.org
Propositional Satisability I

Recall that propositional logic consists of

Boolean variables V = p
1
, p
2
, . . .
Literals p
i
, p
i
Formulas ::= p
i
[ [ [

A formula in conjunctive normal form (CNF) is a

conjunction of disjunctions, e.g.,
(p
1
p
2
) (p
2
p
3
p
4
) (p
1
p
3
)

is true, is false.
29 / 79
ECE750 Lecture
10:
NP-completeness
and Approximation
Algorithms
Todd Veldhuizen
tveldhui@acm.org
Propositional Satisability II

A truth assignment is a substitution : V , .

We extend to a function on formulas:
(p
i
) = (p
i
)
() =

if () =
if () =
( ) =

if () = or () =
otherwise
( ) =

if () = and () =
otherwise

The propositional satisability problem, or SAT: given a

propositional formula with free variables p
1
, . . . , p
k
,
decide whether there exists a truth assignment such
that () = .
30 / 79
ECE750 Lecture
10:
NP-completeness
and Approximation
Algorithms
Todd Veldhuizen
tveldhui@acm.org
Propositional Satisability III

Obviously SAT and Circuit-satisability are closely

related. We can formalize this by means of a Karp
reduction (or many-one reduction):

Let us make the simplifying assertion that all decision

problem instances can be expressed as binary strings.
For example, a boolean circuit can be described by a
binary string that enumerates the gates and their
interconnections in some prex-free code. Letting
Z

be the set of all strings representing circuits

that are satisable, the decision problem Is this circuit
satisable? becomes Is the string x (representing the
circuit) in the set Z?

A many-one, or Karp, reduction from a problem Y to a

problem Z is a function r :

computable in
polynomial time such that y Y if and only if
r (y) Z.
31 / 79
ECE750 Lecture
10:
NP-completeness
and Approximation
Algorithms
Todd Veldhuizen
tveldhui@acm.org
Propositional Satisability IV

Intuitively, we take a problem of one sort and turn it

into a problem of a dierent sort, while preserving the
YES/NO outcome of the decision.

For example, let Y be the set of binary strings

representing satisable boolean formulas, and Z be
binary strings representing satisable circuits. To reduce
Y to Z, we just convert the boolean formulas into a
circuit: r ( ) becomes an AND gate whose inputs
are r () and r (), and so forth. This translation from a
boolean formula to a circuit can be done in polynomial
time: SAT is many-one reducible to Circuit-SAT.

Note:
If Y is Karp-reducible to Z, then a
polynomial-time solution for Z would imply a
polynomial time solution for Y.
(We can turn instances of Y into instances of Z in
polynomial time.)
32 / 79
ECE750 Lecture
10:
NP-completeness
and Approximation
Algorithms
Todd Veldhuizen
tveldhui@acm.org
Propositional Satisability V

Karp-reducibility is a preorder: transitive and reexive.

We can write
Y
P
m
Z
to mean Y is Karp-reducible to Z. (The P stands for
polynomial time, and the m stands for many-one.)

Recall that whenever we have a preorder, we can turn it

into an equivalence relation and a partial order (a
hierarchy):

Dene X Y to mean (X
P
m
Y) (Y
P
m
X). (This
is an equivalence relation, whose equivalence classes
consist of problems that are interreducible. These
equivalence classes are sometimes called degrees.)

Then dene [X]

[Z]

on equivalence classes by
[X]

[Z]

i X
P
m
Z.

The class P is the least element of the partial order.

33 / 79
ECE750 Lecture
10:
NP-completeness
and Approximation
Algorithms
Todd Veldhuizen
tveldhui@acm.org
Propositional Satisability VI

If we restrict ourselves to problems in NP, this poset

has a maximal element: the class NPC of NP-complete
problems.

It is known that if P ,= NP, then there are equivalence

classes strictly between P and NP, called
NP-intermediate problems. Proving the existence of
such a class would immediately imply P ,= NP.

Defn: X is NP-hard if, for any Y NP, Y

P
m
X.
(Note: X is not required to be in NP. For example, the
halting problem is NP-hard.)

Defn: X is NP-complete if:

1. X NP; and
2. X is NP-hard.
34 / 79
ECE750 Lecture
10:
NP-completeness
and Approximation
Algorithms
Todd Veldhuizen
tveldhui@acm.org
Propositional Satisability VII

Roughly speaking, NPC (the set of NP-complete

problems) consists of the very hardest problems known
to be in NP; if we could solve just one of those
problems in polynomial time, then every problem in NP
could be solved in polynomial time, which would imply
P = NP.

First NP-completeness proof was the Cook-Levin

theorem (1971): propositional satisability (SAT) is an
NP-complete problem. (Stephen Cook is a professor at
the University of Toronto.)

Proof sketch: encode valid traces of a nondeterministic

Turing machine running in s steps and s tape locations
as a propositional formula. Set boundary conditions so
that if the formula is satisable the machine halts with
a YES output.
35 / 79
ECE750 Lecture
10:
NP-completeness
and Approximation
Algorithms
Todd Veldhuizen
tveldhui@acm.org
Propositional Satisability VIII

Import: any problem in NP is Karp-reducible to SAT; a

polynomial time algorithm for SAT would imply
P = NP.

To prove a problem X is NP-complete, it is necessary to

prove that (i) the problem is NP (this is usually easy);
and that (ii) some problem already known to be in NPC
can be reduced to X.

Since the original NPC problems were discovered, many

useful problems have been proven to be in this class;
there are some 1000+ known NPC problems!

The classic text on the subject: Garey and Johnson [2].

Why is knowing whether a problem is in NPC

important?

Know not to waste your time looking for a

polynomial-time algorithm (unless have a fondness for
tilting at windmills).
36 / 79
ECE750 Lecture
10:
NP-completeness
and Approximation
Algorithms
Todd Veldhuizen
tveldhui@acm.org
Propositional Satisability IX

There is a rich understanding of how NPC optimization

problems can be approximately solved in polynomial
time, e.g., to within a small dierence from optimality.
Identifying a problem as NPC is a rst step to guring
out how to best approach its approximation.
37 / 79
ECE750 Lecture
10:
NP-completeness
and Approximation
Algorithms
Todd Veldhuizen
tveldhui@acm.org
Example NPC proof: Independent set I

To determine if a problem is NP-complete, your rst

step should be to check a catalogue of known
NP-complete problems, such as the online survey A
compendium of NP optimization problems.

We will look at a simple example of an NP-completeness

proof as an example of how such results are obtained.

In a graph G = (V, E), a subset of vertices V

V are
an independent set if for all v
1
, v
2
V

, there is no
edge (v
1
, v
2
) E.
The green vertices form an independent set of size 4.
38 / 79
ECE750 Lecture
10:
NP-completeness
and Approximation
Algorithms
Todd Veldhuizen
tveldhui@acm.org
Example NPC proof: Independent set II

An independent set V

is the opposite of a clique, in the sense

that the vertices V

form a clique in the graph (V, V

2
\ E).

The Independent Set problem: given a graph

G = (V, E) and a positive integer m, decide if G has an
independent set on m vertices.

Proving NP-completeness requires proving (i) the

problem is in NP; (ii) a known NP-complete problem
can be reduced to it.

Clearly, independent set is in NP: a certicate is the set

of m vertices forming the independent set, which can be
checked in polynomial time.

We will reduce 3-SAT to Independent Set. (This proof

and example are from [3].)
39 / 79
ECE750 Lecture
10:
NP-completeness
and Approximation
Algorithms
Todd Veldhuizen
tveldhui@acm.org
Example NPC proof: Independent set III

3-SAT is a restricted form of propositional satisability,

in which formulas are in CNF form, with each
disjunction containing at most three literals, for
example:
(x
1
x
2
x
3
) (x
1
x
2
x
3
) (x
1
x
2
x
3
) (x
1
x
2
x
3
)
(We write x
i
as a shortform for x
i
.)

3-SAT is known to be an NP-complete problem: every

SAT problem can be rewritten into the above form.

We demonstrate a translation from a 3-SAT problem to

an independent set problem, such that an independent
set of m vertices exists if and only if the 3-SAT formula
is satisable.

For a 3-SAT formula of m clauses (disjunctions),

construct a graph where:

There is a vertex for each occurrence of a literal in the

formula;
40 / 79
ECE750 Lecture
10:
NP-completeness
and Approximation
Algorithms
Todd Veldhuizen
tveldhui@acm.org
Example NPC proof: Independent set IV

Within each disjunction, there is an edge between each

vertex corresponding to a literal in that disjunction;

There is an edge between every occurrence of a literal

and its negation, i.e., we connect each x
1
to each x
1
,
etc.

This graph can be constructed in polynomial time.

1
x
1
x
3
x
3
x
3
x
1
x
1
x
2
x
2
x
2
x
3
x
2
x

We then ask: Is there an independent set of size m?

In the above graph, the green vertices show an

independent set of size 4.
41 / 79
ECE750 Lecture
10:
NP-completeness
and Approximation
Algorithms
Todd Veldhuizen
tveldhui@acm.org
Example NPC proof: Independent set V

Because each clause (disjunction) corresponds to a

clique (e.g. a triangle) in the graph, any independent
set on m vertices must have at most one vertex from
each clique.

Choose a truth assignment so that the literal

corresponding to each vertex of the independent set is
(true). E.g., if x
2
is in the independent set, we
choose x
2
= so that x
1
= .

Since there is an edge between all literals and their

negations, we cannot simultaneously choose x
i
= and
x
i
= ; so a consistent truth assignment exists.

We will have one vertex from each clique = one literal

from each clause made true. Therefore each disjunction
will evaluate to , and their conjunction will be also.

E.g., in the above example we choose x

1
= , x
2
= ,
and x
3
can be either or .
42 / 79
ECE750 Lecture
10:
NP-completeness
and Approximation
Algorithms
Todd Veldhuizen
tveldhui@acm.org
Example NPC proof: Independent set VI

The graph has an independent set of size m if and only

if the 3-SAT formula is satisable. Since the reduction
can be done in polynomial time, this is a Karp
reduction.

This reduction demonstrates that if we could solve

Independent Set in polynomial time, we could solve
3-SAT in polynomial time, which in turn implies we
could solve any problem in NP in polynomial time.

Hence Independent Set is NP-Complete.

43 / 79
ECE750 Lecture
10:
NP-completeness
and Approximation
Algorithms
Todd Veldhuizen
tveldhui@acm.org
Part III
Approximation algorithms
44 / 79
ECE750 Lecture
10:
NP-completeness
and Approximation
Algorithms
Todd Veldhuizen
tveldhui@acm.org
Approximation Algorithms I

Recall that

P = decision problems, decidable in polynomial time.

NP = decision problems, YES certicates checkable in

polynomial time.

NPC = the hardest problems in NP; if any of these

can be solved in polynomial time then P = NP.

It is believed that P ,= NP: NPC problems are thought

to require superpolynomial time.

However, for many useful optimization problems we can

obtain approximate solutions in polynomial time.

In an optimization problem, we are trying to minimize

or maximize some objective function; the corresponding
NP decision version is a question Is there a solution
with cost k? or Is there a solution with prot
k?
45 / 79
ECE750 Lecture
10:
NP-completeness
and Approximation
Algorithms
Todd Veldhuizen
tveldhui@acm.org
Approximation Algorithms II

The class NPO consists of optimization problems

whose decision version is in NP. Technically, an NPO
problem consists of:
1. A set of valid problem instances D

, recognizable
in polynomial time. (Here we take

to be binary
strings, for simplicity.)
2. A set of feasible solutions S

, and a polynomial
time algorithm that decides S given a problem instance.
3. An objective function J(I , s) that maps an instance
I D and a solution s S to a cost or prot in Q
+
.
4. An indication of whether we aim to maximize or
minimize J.
46 / 79
ECE750 Lecture
10:
NP-completeness
and Approximation
Algorithms
Todd Veldhuizen
tveldhui@acm.org
Approximation Algorithms III

For example, the problem Maximum Independent Set

asks for the largest set of vertices in a graph such that
no two vertices have an edge between them. This is an
NPO problem: the problem instances are graphs
(suitably encoded as binary strings), feasible solutions
are independent sets, and the objective function
measures the size of an independent set.

Note that a polynomial time algorithm for an NPO

optimization problem implies a polynomial time
algorithm for the decision version e.g., to answer the
question Does this graph has an independent set of
4 vertices? we can nd the largest independent set
and see how large it is.

If the decision version of an optimization problem is

NP-complete, then a polynomial time algorithm for the
optimization problem would imply P = NP.
47 / 79
ECE750 Lecture
10:
NP-completeness
and Approximation
Algorithms
Todd Veldhuizen
tveldhui@acm.org
Approximation Algorithms IV

Between P and NP-completeness there are several

grades of optimization problems where we can obtain
approximate solutions in polynomial time. Let OPT
represent the optimal value of an objective function,
and suppose for simplicity we seek to minimize the
objective function.
1. The class APX (approximable) contains NPO problems
where it is possible to obtain OPT in polynomial
time, for some xed constant .
2. The class PTAS (Polynomial Time Approximation
Scheme) contains NPO problems where for any xed
constant > 0, we can obtain (1 + ) OPT in
polynomial time. (However, the time required might be
n
f ()
where f () as 0. The algorithm is
polynomial only if is xed.)
48 / 79
ECE750 Lecture
10:
NP-completeness
and Approximation
Algorithms
Todd Veldhuizen
tveldhui@acm.org
Approximation Algorithms V
3. The class FPTAS (Fully Polynomial Time
Approximation Scheme) contains NPO problems where
for any xed constant > 0, we can obtain
(1 + ) OPT in time O(n
c

d
) where c, d are
constants.
4. For maximization problems, replace (1 + ) OPT with
(1 ) OPT.

The containment relations between these classes are

illustrated by this diagram:
NPO
APX
PTAS
FPTAS
P
49 / 79
ECE750 Lecture
10:
NP-completeness
and Approximation
Algorithms
Todd Veldhuizen
tveldhui@acm.org
Approximation Algorithms VI
50 / 79
ECE750 Lecture
10:
NP-completeness
and Approximation
Algorithms
Todd Veldhuizen
tveldhui@acm.org
Part IV
Minimal Vertex Cover (APX)
51 / 79
ECE750 Lecture
10:
NP-completeness
and Approximation
Algorithms
Todd Veldhuizen
tveldhui@acm.org
Example: Vertex cover I

Suppose that to improve the performance of a computer

network, you want to collect statistics on packets being
transmitted. Draw a graph where the vertices are
computers/routers, and edges are communication links:
Collecting statistics can slow down the network, and
requires installing custom software etc. So, we want to
monitor the trac on as few nodes as possible. We
choose as small a set of nodes as possible on which to
install the monitoring software, so that each
52 / 79
ECE750 Lecture
10:
NP-completeness
and Approximation
Algorithms
Todd Veldhuizen
tveldhui@acm.org
Example: Vertex cover II
communication link has monitoring software on at least
one end, e.g.:
If we install the monitoring software on the green nodes,
every communication link has at least one end green.

This is a vertex cover.

Minimum Vertex Cover: Given a graph G = (V, E) nd

a set V

V of minimal size such that every edge has

at least one vertex in V

This problem is in NPO:

1. Instances: graphs.
53 / 79
ECE750 Lecture
10:
NP-completeness
and Approximation
Algorithms
Todd Veldhuizen
tveldhui@acm.org
Example: Vertex cover III
2. Feasible solutions: subsets of vertices V

such that
every edge has at least one end in V

.
3. Objective function: [V

[.
4. It is a minimization problem.

The decision version (Does G have a vertex cover of

m vertices?) is in NP: a vertex cover V

with
[V

[ m is a YES certicate.

Minimum Vertex Cover is known to be NP-complete (in

its decision version).
54 / 79
ECE750 Lecture
10:
NP-completeness
and Approximation
Algorithms
Todd Veldhuizen
tveldhui@acm.org
An approximation algorithm for vertex cover I

Now lets see an approximation algorithm for vertex

cover that lets us get a nearly optimal answer in
polynomial time.

The approximation well see uses a maximal matching.

A matching of a graph G = (V, E) is a subset of edges

E

E such that no vertex is the endpoint of more

than one edge.

A matching is maximal if it cannot be enlarged.

A graph (left) and a maximal matching (right). Picking any more
edges would result in a vertex being the endpoint of two edges.
55 / 79
ECE750 Lecture
10:
NP-completeness
and Approximation
Algorithms
Todd Veldhuizen
tveldhui@acm.org
An approximation algorithm for vertex cover II

The vertices that are endpoints of the edges in the

matching form a vertex cover:
The endpoints of the edges in the matching (green) form a vertex
cover of the graph.

Why? Suppose the endpoints of the maximal matching didnt

form a vertex cover. From the denition of a vertex cover, this
would imply there was an edge (v
1
, v
2
) such that neither v
1
nor v
2
are in the vertex cover. This would imply there was an edge
(v
1
, v
2
) such that neither v
1
nor v
2
are endpoints of an edge in the
maximal matching. So, (v
1
, v
2
) could be added to the matching.
But, this contradicts the premiss that the matching is maximal.
56 / 79
ECE750 Lecture
10:
NP-completeness
and Approximation
Algorithms
Todd Veldhuizen
tveldhui@acm.org
An approximation algorithm for vertex cover III

This gives us an approximation algorithm for Minimum

Vertex Cover:
1. Find a maximal matching E

E. (This can be done

by simply considering the edges in arbitrary order, and
adding them to E

if they do not have an endpoint in

common with an edge already in E

.)
2. Output the list of vertices that are endpoints of E

.
Theorem
This approximation algorithm yields a vertex cover of size
2 OPT.
Proof.
57 / 79
ECE750 Lecture
10:
NP-completeness
and Approximation
Algorithms
Todd Veldhuizen
tveldhui@acm.org
An approximation algorithm for vertex cover IV
Let E

E be a maximal matching. We prove that

[E

[ OPT; since the vertex cover output is of size 2[E

[,
this establishes that the size of the vertex cover is 2 OPT.
Proof: Suppose to the contrary that [E

[ > OPT. Let

V

V be a vertex cover with [V

[ = OPT. By pigeonhole
there must be an edge (v
1
, v
2
) E

such that neither v

1
nor
v
2
are in V

; but this contradicts the premiss that V

is a
vertex cover.
Illustration of a contradiction: if the maximal matching contains 4
edges, it is impossible that there could be a vertex cover of only 3
vertices (green), since this would leave an edge uncovered.
58 / 79
ECE750 Lecture
10:
NP-completeness
and Approximation
Algorithms
Todd Veldhuizen
tveldhui@acm.org
An approximation algorithm for vertex cover V

The maximal matching approximation nds a vertex

cover that is at most twice optimal. This establishes
that Minimum Vertex Cover is in the class APX.

Whether it is possible to better than 2 OPT is an open

problem.
59 / 79
ECE750 Lecture
10:
NP-completeness
and Approximation
Algorithms
Todd Veldhuizen
tveldhui@acm.org
Part V
Steiner networks (APX)
60 / 79
ECE750 Lecture
10:
NP-completeness
and Approximation
Algorithms
Todd Veldhuizen
tveldhui@acm.org
Steiner trees I

A network design problem is one of the following form:

given a set of points/vertices, nd a minimal network
spanning them. For example, we might consider how to
build roads between four cities so that the total road
length is minimized:
61 / 79
ECE750 Lecture
10:
NP-completeness
and Approximation
Algorithms
Todd Veldhuizen
tveldhui@acm.org
Steiner trees II

A more general setting of the

problem is the Minimum Steiner Tree problem on graphs:
Given a graph G = (V, E), a positive weight
w(v
1
, v
2
) on edges, and a subset V

V of
vertices to be connected, nd a minimum cost
subtree of G that includes all the vertices in
V

The cost is measured as the sum of the weights of the

edges in the tree.

The tree may contain vertices not in V

; these are
called Steiner nodes.
62 / 79
ECE750 Lecture
10:
NP-completeness
and Approximation
Algorithms
Todd Veldhuizen
tveldhui@acm.org
Steiner trees III

Example:
a
3
2
4
1
1
b
c
d
4
2
2
4
1
1
1
4
The black vertices are the set V

to be connected.

The decision version of this problem is known to be

NP-complete.

However, there is a fast approximation algorithm:

1. For each pair v
1
, v
2
V

, compute the shortest path

between them, and call this d(v
1
, v
2
).
63 / 79
ECE750 Lecture
10:
NP-completeness
and Approximation
Algorithms
Todd Veldhuizen
tveldhui@acm.org
Steiner trees IV
2. Construct a complete graph G

on V

, having edges for

every pair v
1
, v
2
V

, and weights d(v

1
, v
2
).
2
3
3
3
2
b
c
d
a
3
3. Find a minimum spanning tree on the graph G

.
2
3
3
2
3
b
c
d
a
3
64 / 79
ECE750 Lecture
10:
NP-completeness
and Approximation
Algorithms
Todd Veldhuizen
tveldhui@acm.org
Steiner trees V
4. Construct a tree on the original graph G by taking the
union of each path represented by an edge of the
minimum spanning tree on the graph G

. When
necessary, prune edges to remove cycles.
1
4
1
1
b
c
d
a
1
4
2
2
4
1
4
3
2
Theorem
This approximation algorithm produces a network of cost
2 OPT.
The proof involves Euler tours and Hamiltonian cycles; see
e.g. [6].
65 / 79
ECE750 Lecture
10:
NP-completeness
and Approximation
Algorithms
Todd Veldhuizen
tveldhui@acm.org
Steiner trees VI

This approximation algorithm establishes that Minimum

Steiner Tree is in APX. However, it is known that this
problem is in APX-Complete, which is the hard
problems of APX; this implies that Minimum Steiner
Tree is not in PTAS unless P = NP.

There is an approximation algorithm for this problem

that achieves (1 + (ln 3)/2) OPT 1.55 OPT.
66 / 79
ECE750 Lecture
10:
NP-completeness
and Approximation
Algorithms
Todd Veldhuizen
tveldhui@acm.org
Bibliography
Part VI
Knapsack (FPTAS)
67 / 79
ECE750 Lecture
10:
NP-completeness
and Approximation
Algorithms
Todd Veldhuizen
tveldhui@acm.org
Bibliography
Knapsack I

So far we have seen two examples of problems in APX.

Now lets see one in FPTAS, where we can get to
within (1 + ) OPT in O(n
c

d
) time, i.e., time
polynomial in n and
1

In a knapsack problem, one has

1. A set S = a
1
, . . . , a
n
of objects;
2. A prot function prot : S Z
+
;
3. A size function size : S Z
+
;
4. A knapsack capacity B Z
+
.

A feasible solution is a subset S

S of objects with

sS

size(s) B.

The objective is to maximize the prot

sS

prot(s).
68 / 79
ECE750 Lecture
10:
NP-completeness
and Approximation
Algorithms
Todd Veldhuizen
tveldhui@acm.org
Bibliography
Knapsack II

Example: here is a knapsack problem with 3 objects

a
1
, a
2
, a
3
.
Object size(a
i
) prot(a
i
)
a
1
3 2
a
2
2 1
a
3
1 3
If the capacity of the knapsack is B = 5, the best
solution is to pick objects a
1
and a
3
, which achieves a
prot of 5 and a size of 4.

Knapsack has numerous real, practical applications:

Shipping (deciding how to pack shipping containers to

maximize prot);

Multirate networks (given a total available bandwidth

and users who bid for various bandwidths at various
prices, what subset of users should be chosen to
maximize prot?)
69 / 79
ECE750 Lecture
10:
NP-completeness
and Approximation
Algorithms
Todd Veldhuizen
tveldhui@acm.org
Bibliography
Knapsack III

Capital budgeting (given a xed amount of money

available for capital purchases, a set of objects that
could be bought, and an estimate of how much their
purchase would benet the company, what subset of
objects should be purchased?)

Web caching (given a xed amount of memory/disk

space on a web cache, web pages of various sizes, and
estimates of how much latency/bandwidth could be
saved by caching those pages, what subset of pages
should be cached?)

The naive approach of greedily choosing objects with

the best prot/size ratio can be made to perform
arbitrarily badly. (Consider for example objects a, b with
sizes 1, 100, prots 10, 900, and capacity 100.)
70 / 79
ECE750 Lecture
10:
NP-completeness
and Approximation
Algorithms
Todd Veldhuizen
tveldhui@acm.org
Bibliography
Knapsack IV

Knapsack (in its decision version) is an NP-complete

problem. However, it has an FPTAS: we can obtain a
solution that is (1 ) OPT in time O(n
3

1
): with
each doubling of the amount of time we are willing to
spend, we can halve the distance between the
approximate answer and the optimal answer.

The approximation algorithm we will see is based on

dynamic programming. Let P = max
sS
prot(s) be
the maximum prot of any item. We can nd an exact
solution to the knapsack problem in time O(n
2
P):
1. Clearly the optimal solution has to have prot n P:
number of objects times the maximum prot of any
object.
2. We will consider the objects in order, and gure out
how much prot we can make with some subset of the
rst i objects.
71 / 79
ECE750 Lecture
10:
NP-completeness
and Approximation
Algorithms
Todd Veldhuizen
tveldhui@acm.org
Bibliography
Knapsack V
3. Let S(i , p) be a subset of s
1
, . . . , s
i
with prot p and
minimum size, or if no such set exists.
4. Let A(i , p) be the size of S(i , p), or + if it doesnt
exist:
A(i , p) =

+ if S(i , p) =

sS(i ,p)
size(s) otherwise
5. Given the solutions to A(i , p) for the rst i objects, we
can obtain the solutions for A(i + 1, p) by considering
either taking or not taking object a
i +1
, for each value of
p:

If prot(a
i +1
) < p, then
A(i + 1, p) = min( A(i , p),
size(a
i +1
) + A(i , p prot(a
i +1
)))

Otherwise, A(i + 1, p) = A(i , p).

72 / 79
ECE750 Lecture
10:
NP-completeness
and Approximation
Algorithms
Todd Veldhuizen
tveldhui@acm.org
Bibliography
Knapsack VI
6. Once we have solved all the A(n, p), we choose the
maximum p such that A(n, p) < B. This is the optimal
answer. (To recover the objects in the optimal set, we
need to keep track of them as we build the table; but
this is not dicult.)

Example: Here is the table of A(i , p) for the example

shown earlier: we have the maximum prot of any one
object is P = 3, so we neednt look farther than a prot
of 9.
i 0 1 2 3 4 5 6 7 8 9
1 0 3
2 0 2 3 5
3 0 2 3 1 3 4 6

The running time is determined by the size of this table,

which is controlled by n and P. If we could make either
n or P smaller, the algorithm would run faster.
73 / 79
ECE750 Lecture
10:
NP-completeness
and Approximation
Algorithms
Todd Veldhuizen
tveldhui@acm.org
Bibliography
Knapsack VII

We choose to make P smaller by scaling and rounding

the prot values so the table A(i , p) becomes smaller;
this yields an approximate answer.

Approximation algorithm:
1. Let K =
P
n
, where P is the maximum prot of any
item. We will be eectively rounding prots down to
the nearest multiple of K; as 0, the rounding has
less and less eect.
2. For each object a
i
, let prot

(a
i
) =
prot(a
i
)
K
|.
3. Use dynamic programming to solve this new problem
and nd the most protable set S

.
4. Output S

.
74 / 79
ECE750 Lecture
10:
NP-completeness
and Approximation
Algorithms
Todd Veldhuizen
tveldhui@acm.org
Bibliography
Knapsack VIII
Theorem
This approximation algorithm yields a prot of
(1 ) OPT.
Proof. Let O be a set of objects achieving the optimal
prot. Write OPT = prot(O) for the sum of the prots of
the objects in O. Since for each object our maximum
rounding error of the prot is K, we have
prot(O)

OPT
K prot

(O)

prot with rounding
nK (1)
Rearranging,
K prot

(O) prot(O) nK (2)

75 / 79
ECE750 Lecture
10:
NP-completeness
and Approximation
Algorithms
Todd Veldhuizen
tveldhui@acm.org
Bibliography
Knapsack IX
Since the dynamic programming algorithm yields an optimal
solution for the rounded problem, the answer we get (S

)
must be at least as good as O under the rounded prots:
prot(S

) K prot

(O) (3)
prot(O) nK from Eqn.(2) (4)
= OPT P using K =
P
n
(5)
Since OPT P (assuming no objects are bigger than the
knapsack!),
prot(S

) (1 )OPT

This establishes that Knapsack is in FPTAS: there is an

approximation algorithm yielding (1 ) OPT in
time O(n
3

1
).
76 / 79
ECE750 Lecture
10:
NP-completeness
and Approximation
Algorithms
Todd Veldhuizen
tveldhui@acm.org
Bibliography
Where to go from here?

Vaziranis text Approximation Algorithms [6] is an

excellent introduction to this area.

There is another text I have not yet had the opportunity

to read by Ausiello et al, Complexity and
Approximation, that includes a reference guide to
known results on approximation. The reference guide is
available online and is well worth browsing:

A Compendium of NP-Hard Optimization Problems

http://www.nada.kth.se/
~
viggo/wwwcompendium/
The compendium is organized hierarchically by problem
class (graph theory, network design, etc.) and for each
problem lists the best known approximation algorithms.
Includes almost 500 references to the literature.
77 / 79
ECE750 Lecture
10:
NP-completeness
and Approximation
Algorithms
Todd Veldhuizen
tveldhui@acm.org
Bibliography
Bibliography I
[1] Gregory J. Chaitin, Marc A. Auslander, Ashok K.
Chandra, John Cocke, Martin E. Hopkins, and Peter W.
Markstein.
Register allocation via coloring.
Computer Languages, 6(1):4757, 1981.
[2] M. R. Garey and D. S. Johnson.
Computers and intractability; a guide to the theory of
NP-completeness.
W.H. Freeman, 1979.
[3] H. R. Lewis and C. H. Papadimitriou.
Elements of the theory of computation.
Prentice-Hall, Englewood Clis, New Jersey, 1981.
[4] A compendium of NP optimization problems, 2006.
78 / 79
ECE750 Lecture
10:
NP-completeness
and Approximation
Algorithms
Todd Veldhuizen
tveldhui@acm.org
Bibliography
Bibliography II
[5] Alan T. Sherman.
On superpolylogarithmic subexponential functions (Part
I).
SIGACT News, 22(1):65, 1991.
[6] Vijay V. Vazirani.
Approximation Algorithms.
Springer-Verlag, 2001.
79 / 79