Lecture Outline

2.1 Computational Tractability

2.2 Asymptotic Order of Growth
2.4 A Survey of Common Running Times
Extra slides on Analysis of Algorithms

Polynomial-Time

Brute force. For many non-trivial problems, there is a natural brute

force search algorithm that checks every possible solution.
„ Typically takes 2Ntime or worse for inputs of size N.
„ Unacceptable in practice.
n ! for stable matching
„ But they are intuitive. with n men and n women

Desirable scaling property. When the input size doubles, the algorithm
should only slow down by some constant factor C.

There exists constants c > 0 and d > 0 such that on every

input of size N, its running time is bounded by c Nd steps.

Def. An algorithm is poly-time if the above scaling property holds.

choose C = 2d
observe the difference between c and C

2
 Worst-Case Analysis

Worst case running time. Obtain bound on largest possible running time
of algorithm on input of a given size N.
„ Generally captures efficiency in practice.
„ Draconian view, but hard to find effective alternative.
Too severe

Average case running time. Obtain bound on running time of algorithm

on random input as a function of input size N.
„ Hard (or impossible) to accurately model real instances by random
distributions.
„ Algorithm tuned for a certain distribution may perform poorly on
other inputs.

Worst-Case Polynomial-Time

Def. An algorithm is theoretically efficient if its running time is

polynomial.
Justification: It really works in practice!
„ Although 6.02 × 1023 × N20 is technically poly-time, it would be
useless in practice.
„ In practice, the poly-time algorithms that people develop almost
always have low constants and low exponents.
„ Breaking through the exponential barrier of brute force typically
exposes some crucial structure of the problem.

Exceptions.
„ Some poly-time algorithms do have high constants and/or
exponents, and are useless in practice.
„ Some exponential-time (or worse) algorithms are widely used
because the worst-case instances seem to be rare.
simplex method
Unix grep

4
 Why It Matters

Asymptotic Order of Growth

Upper bounds. T(n) is O(f(n)) if there exist constants c > 0 and n0 ≥ 0

such that for all n ≥ n0 we have T(n) ≤ c · f(n).

Lower bounds. T(n) is Ω(f(n)) if there exist constants c > 0 and n0 ≥ 0

such that for all n ≥ n0 we have T(n) ≥ c · f(n).

Tight bounds. T(n) is Θ(f(n)) if T(n) is both O(f(n)) and Ω(f(n)).

Ex: T(n) = 32n2 + 17n + 32.

„ T(n) is O(n2), O(n3), Ω(n2), Ω(n), and Θ(n2) .
„ T(n) is not O(n), Ω(n3), Θ(n), or Θ(n3).

6
 Asymptotic Order of Growth in Plain English

Big family.
O(g(n)): class of functions f(n) that grow no faster than g(n).
Θ (g(n)): class of functions f(n) that grow at same rate as g(n).
Ω(g(n)): class of functions f(n) that grow at least as fast as g(n).

Small family.
o(g(n)): class of functions f(n) that grow slower than g(n).
ω(g(n)): class of functions f(n) that grow faster than g(n).

Notation

Slight abuse of notation. T(n) = O(f(n)).

„ Not transitive:
– f(n) = 5n3; g(n) = 3n2
– f(n) = O(n3) = g(n)
– but f(n) ≠ g(n).
„ Better notation: T(n) ∈ O(f(n)).

Meaningless statement. Any comparison-based sorting algorithm

requires at least O(n log n) comparisons.
„ Statement doesn't "type-check."
„ Use Ω for lower bounds. O notation is for upper bounds.

8
 Properties

Transitivity.
„ If f = O(g) and g = O(h) then f = O(h).
„ If f = Ω(g) and g = Ω(h) then f = Ω(h).
„ If f = Θ(g) and g = Θ(h) then f = Θ(h).

Additivity.
„If f = O(h) and g = O(h) then f + g = O(h).
„If f = Ω(h) and g = Ω(h) then f + g = Ω(h).
„If f = Θ(h) and g = O(h) then f + g = Θ(h).

Multiplicity
„If f = O(h) and g = O(i) then f × g = O(h × i)
„If f = Ω(h) and g = Ω(i) then f × g = Ω(h × i)
„If f = Θ(h) and g = Θ(i) then f × g = Θ(h × i)

Asymptotic Bounds for Some Common Functions

Polynomials. a0 + a1n + … + adnd is Θ(nd) if ad > 0.

Polynomial time. Running time is O(nd) for some constant d independent

of the input size n.

Logarithms. O(log a n) = O(log b n) for any constants a, b > 0.

can avoid specifying the

base

Logarithms. For every x > 0, log n = O(nx).

log grows slower than every polynomial

Exponentials. For every r > 1 and every d > 0, nd = O(rn).

every exponential grows faster than every polynomial

10
 Linear Time: O(n)

Linear time. Running time is at most a constant factor times the size
of the input.

Computing the maximum. Compute maximum of n numbers a1, …, an.

max ← a1
for i = 2 to n {
if (ai > max)
max ← ai
}

11

Linear Time: O(n)

Merge. Combine two sorted lists A = a1,a2,…,an with B = b1,b2,…,bn

into sorted whole.

i = 1, j = 1
while (both lists are nonempty) {
if (ai ≤ bj) append ai to output list and increment i
else(ai ≤ bj)append bj to output list and increment j
}
append remainder of nonempty list to output list

Claim. Merging two lists of size n takes O(n) time.

Pf. After each comparison, the length of output list increases by 1.

12
 O(n log n) Time

O(n log n) time. Arises in divide-and-conquer algorithms.

also referred to as linearithmic time

Sorting. Mergesort and heapsort are sorting algorithms that perform

O(n log n) comparisons.

Largest empty interval. Given n time-stamps x1, …, xn on which copies

of a file arrive at a server, what is largest interval of time when no
copies of the file arrive?

O(n log n) solution. Sort the time-stamps. Scan the sorted list in
order, identifying the maximum gap between successive time-stamps.

13

Quadratic Time: O(n2)

Quadratic time. Enumerate all pairs of elements.

Closest pair of points. Given a list of n points in the plane (x1, y1), …,
(xn, yn), find the pair that is closest.

O(n2) solution. Try all pairs of points.

min ← (x1 - x2)2 + (y1 - y2)2

for i = 1 to n {
for j = i+1 to n {
d ← (xi - xj)2 + (yi - yj)2 don't need to
if (d < min) take square roots

min ← d
}
}

Remark. Ω(n2) seems inevitable, but this is just an illusion. see chapter 5

14
 Cubic Time: O(n3)

Cubic time. Enumerate all triples of elements.

Set disjointness. Given n sets S1, …, Sn each of which is a subset of

1, 2, …, n, is there some pair of these which are disjoint?

O(n3) solution. For each pairs of sets, determine if they are disjoint.

foreach set Si {
foreach other set Sj {
foreach element p of Si {
determine whether p also belongs to Sj
}
if (no element of Si belongs to Sj)
report that Si and Sj are disjoint
}
}

15

Polynomial Time: O(nk) Time

Independent set of size k. Given a graph, are there k nodes such that
no two are joined by an edge?
k is a constant

O(nk) solution. Enumerate all subsets of k nodes.

foreach subset S of k nodes {

check whether S in an independent set
if (S is an independent set)
report S is an independent set
}
}

„ Check whether S is an independent set = O(k2).

„ Number of k element subsets = ⎛ n ⎞ n(n − 1)(n − 2)L(n − k + 1) n k
O(k2 nk / k!) = O(nk).
⎜⎜ ⎟⎟= ≤
„
⎝k⎠ k(k − 1)(k − 2)L(2)(1) k!
poly-time for k=17,
but not practical

16
 Exponential Time

Independent set. Given a graph, what is maximum size of an

independent set?

O(n2 2n) solution. Enumerate all subsets.

S* ← φ
foreach subset S of nodes {
check whether S in an independent set
if (S is largest independent set seen so far)
update S* ← S
}
}

17

Analysis of Algorithms

Issues
„ Correctness. Most important, algorithm must be correct otherwise
it is useless.
„ Time efficiency. How much time requires to run the algorithm.
„ Space efficiency. The extra space the algorithm requires.
– Both time and space were premium resources in the early days.
At the present, the space issue is not of as much concern. The
memory is cheap.
„ Optimality. (In theory) no algorithm can be more efficient.

Approaches
„Theoretical analysis. Mathematical analysis.
„Empirical analysis. Through experiments.

18
 Empirical Analysis of Time Efficiency

Consider a program as a black block

„ Select a specific (typical) sample of inputs
„ Use physical unit of time (e.g., milliseconds)
– Depend on other factors such as computer architecture, compiler
optimization.

OR

Instrument a program with some counting mechanism

„ Profiling
„ Count actual number of basic operations (or sample number of basic
ops)
„ Analyze the empirical data

19

Best-case, Average-case, Worst-case

For some algorithms, the efficiency depends on type of input.

Worst case. Worst(n) – maximum over inputs of size n

Best case. Best(n) – minimum over inputs of size n

Average case. Avg(n) – average over inputs of size n

„ NOT the average of worst and best case, even they can be similar
at times. Avg(n) ≠ (Worst(n)+Best(n))/2
„ Considered over the distribution of all possible inputs of size n.

Amortized efficiency. Average over a sequence of operations

performed on the same data structure.
„ Same operations along the sequence require a different amount of
time.

20
 Example: Sequential Search

Problem. Given a list of n elements and a search key K, find an element

equal to K, if any.

Algorithm. Scan the list and compare its successive elements with K
until either a matching element is found (successful search) or the list
is exhausted (unsuccessful search).

Worst case?
Search all elements, no match found.

Best case?
Match at the first comparison.

Average case?
Depends on how often the search key matches the list.

21

Recurrence Relations

Running time can be expressed as a function of n.

x(n) = 2n for n ≥ 0

The generic terms (n) can be expressed as one or more of other terms
(with lower index number) of the function, with one or more explicit
values of the first terms.

x(n) = x(n-1) + n for n > 0

x(0) = 0
initial condition

This type of definition is called recurrence equation, recurrence

relation.

22
 Methods for Solving Recurrences

Method of forward substitutions. Start with the initial term(s), use

recurrence equation to generate few more terms and analyze for the
pattern, then prove it using mathematical induction.

Method of backward substitutions. Start with the generic term,

substitute x(n-1) on the right side of recurrence equation with
equivalent x(n-2) term. Repeat the substitution and observe for the
pattern. Rewrite the recurrence so that x(n) is a function of x(n-i) for
any i, then select i to make n-i reach the initial condition.

Guess and substitute. Guess the solution of the recurrence, substitute

the guessing term. Then show that it is correct.

23

Example1: Recursive Evaluation of n !

Def. n ! = 1*2*…*(n-1)*n
recursive definition of n!

Factorial(n) {
if (n=0)
fac = 1;
else
fac = Factorial(n-1) * n
return fac;
}

Recurrence for number of multiplications, M(), to compute n!. Using

method of backward substitutions: (for n > 0)

M(n) = M(n-1) + 1 substitute M(n-1) with M(n-2)+1

= [M(n-2) + 1] +1 = M(n-2) + 2 substitute M(n-2) with M(n-3)+1
= [M(n-3) + 1] +1 +1 = M(n-3) + 3
= M(n-i) + i = … = M(n-n) + n = n.

24
 Example 2: Tower of Hanoi

Tower of Hanoi

Def. There are n disks of different sizes and 3 pegs

Initially, all disks are on the first peg in order of size, the largest on
the bottom, the smallest on top.

Goal. Move all disks to the third peg using second one as an auxiliary, if
needed.

Constraints.
„ One disk can be move at a time
„ A larger disk cannot be on top of a smaller one at any time.

25

Recurrence of Tower of Hanoi

Let M(n) denote number of moves

M(n) = M(n-1) + 1 + M(n-1) for n > 1.

M(1) = 1.

Using method of backward substitutions

M(n) = 2M(n-1) + 1 substitute M(n-1) with 2M(n - 2) + 1

= 2[2M(n-2) + 1] + 1
= 22M(n-2) + 2 + 1 substitute M(n-2) with 2M(n - 3) + 1
= 22[2M(n-3) + 1] + 2 + 1
= 23M(n-3) + 22 + 2 + 1
= 2iM(n-i) + 2i-1 + … + 22 + 2 + 1
= 2iM(n-i) + 2i – 1
= 2n-1M(1) + 2n-1 – 1 = 2n-1 + 2n-1 – 1 = 2n -1.

26
 Master Theorem

Let T(n) be an eventually nondecreasing function.

T(n) = aT(n/b) + f(n) for n= bk, k = 1, 2, …

T(1) = c,
where a ≥ 1, b ≥ 2, c ≥ 0. If f(n) ∈ Θ(nd), d ≥ 0 then

Case 1. a < bd T(n) ∈ Θ(nd)

Case 2. a = bd T(n) ∈ Θ(nd lg n )
Case 3. a > bd T(n) ∈ Θ(nlog b a)

The same results hold for O, Ω too.

27

Idea of Master Theorem

Recursion tree Work for a level

f(n) f(n)
a

f(n/b) f(n/b) ... f(n/b) a f(n/b)

a
h =logb(n)

... a2 f(n/b2)
f(n/b2) f(n/b2) f(n/b2)

T(1) # leaves = ah = alog_b n = nlog_b a # leaves * T(1)

28
 Idea of Master Theorem

Case 1. The weight decreases geometrically from the root to leaves.

The root holds a constant fraction of the total weight.

Θ(nd )

Case 2. The weight is approximately the same on each of the logbn

levels.

Θ(nd lg n)

Case 3. The weight increases geometrically from the root to leaves.

The leaves hold a constant fraction of the total weight.

Θ(nlogb a )

29

Master Theorem Examples

Example 1.
T(n) = T(n/2) + n
T(1) = 1 Î a = 1, b = 2, c = 1, f(n) = n, d = 1.
Match case 1, therefore T(n) = Θ(n1) = Θ(n) = O(n) = Ω(n).

Example 2.
T(n) = 2T(n/2) + n
T(1) = 1 Î a = 2, b = 2, c = 1, f(n) = n, d = 1.
Match case 2, therefore T(n) = Θ(n lg(n)).

Example 3.
T(n) = 7T(n/4) + n
T(1) = 1 Î a = 7, b = 4, c = 1, f(n) = n, d = 1.
Match case 3, therefore T(n) = Θ(nlog4 7)

30
 Recursion as Mathematical Induction

Recursion is mathematical induction

Example.
n
n(n + 1)
∑i =
i =1 2
for all n ≥ 1

We need to prove.

1
1(1+ 1) 2
2(2 + 1) 3
3(3 + 1) n
n(n + 1)
∑i =
i =1 2
; ∑i =
i =1 2
; ∑i =
i =1 2
; L;∑i =
i =1 2

31

Mathematical Induction

1
1(1+ 1)
Basis step, prove. ∑i =
i=1 2
⇒1= 1

Inductive step (weak induction)

„ Assume the nth term is true and show the (n+1)th term is true.
n
n(n + 1)
Assume.
∑i =
i =1 2
n+1
(n + 1)(n + 2)
Show. ∑i =
i=1 2
n+1
⎛ n ⎞
Rewrite (n+1)th term in form of nth term ∑
i =1
i = ⎜ ∑i ⎟ + (n + 1)
⎝ i =1 ⎠
Use the inductive assumption

n+1
n(n + 1) (n + 1)(n + 2) (n + 1)((n + 1) + 1)
∑i =
i =1 2
+ (n + 1) =
2
=
2
32
 Recursion Tree

Visualize a recurrence
T(n) = 2T(n/2) + n2

T(n) n2

T(n/2) T(n/2)

33

Recursion Tree (cont.)

Recursion tree Work for a level

n2 n2

(1/2)n2
(n/2)2 (n/2)2

T(n/4) T(n/4) T(n/4) T(n/4) (1/4)n2

Values decrease geometrically Total work Θ(n2)

The top level dominates the run time.
Total run time is a constant factor of the top term.
34
 Recursion Tree (cont.)

Recursion tree Work for a level

n2 n2

(5/16)n2
(n/4)2 (n/2)2

T(n/16) T(n/8) T(n/8) T(n/4) (25/256)n2

Total work = n2(1+5/16+(5/16)2+…) = Θ(n2)

T(n) = T(n/4) + T(n/2) + n2. Different recurrence but the same complexity.

35

Recursion Tree (cont.)

T(n) = T(n/3) + T(2n/3) + n

Recursion tree Work for a level

n n

n/3 2n/3 n

36
 Recursion Tree (cont.)

Recursion tree Work for a level

n n

n/3 2n/3 n
log3/2(n)

T(n/9) T(2n/9) T(2n/9) T(4n/9) n

Total work O(n lg n)

37