Algorithms and Data Structures

SE240 - Spring 2009

Lecture 3
Chapter 3, 4.1, 4.2, 4.3
Ladan Tahvildari, PEng
Dept. of Electrical and Computer Engineering
University of Waterloo
Asymptotic - Overview
™ A way to describe behaviour of functions in the limit.

™ Describe growth of functions.

™ Focus on what’s important by abstracting away low-

order terms and constant factors.

™ Indicate running times of algorithms.

™ A way to compare “sizes” of functions.

May 12, 2009 SE240 2
Asymptotic Notation

™ Ο − notation ⎯
⎯→ Ο ≈≤ (big-oh, upper-bound)

™ Ω − notation ⎯
⎯→ Ω ≈≥ (big-omega, lower bound)

™ Θ − notation ⎯
⎯→ Θ ≈= (theta, tight bound)

™ ο − notation ⎯
⎯→ ο ≈p (little-oh, upper-bound not tight)

™ ω − notation ⎯
⎯→ ω ≈f (little-omega, lower-bound but not tight)

May 12, 2009 SE240 3

 Ο − Notation
Ο(g(n)) = { f (n) : there exist positive constants c and n0
such that 0 ≤ f (n) ≤ cg(n) for all n ≥ n0 }

Example:
2 n = Ο( n )
2 3

with c = 1, n0 = 2

g(n) is an asymptotic upper bound for f(n)

May 12, 2009 SE240 4

 Ω − Notation
Ω (g(n)) = { f (n) : there exist positive constants c and n0
such that 0 ≤ cg(n) ≤ f (n) for all n ≥ n0 }

Example:

n = Ω(log n)
with c = 1, n0 = 16

May 12, 2009 SE240 5

 Θ − Notation
Θ(g(n)) = { f (n) : there exist positive constants c1, c2, and
n0 such that 0 ≤ c1 g(n) ≤ f (n) ≤ c2 g(n)
for all n ≥ n0 }
Example:
1 2
n − 3n = Θ(n 2 )
2
1 1
with c1 = , c2 = , n0 = 7
14 2
g(n) is an asymptotic tight bound for f(n)

May 12, 2009 SE240 6

Recurrences

™A recurrence is an equation or inequality

that describes a function in terms of its
value on smaller inputs.

Merge Sort
Θ(1) if n = 1;
T(n) =
2T(n/2) + Θ(n) if n > 1.

May 12, 2009 SE240 7

Methods for Solving Recurrences

™ Recursion-Tree Method

™ Master Method

™ Substitution Method

May 12, 2009 SE240 8

 Example of Recursion Tree
Solve T(n) = T(n/4) + T(n/2) + n2:

May 12, 2009 SE240 9

 Example of Recursion Tree
Solve T(n) = T(n/4) + T(n/2) + n2:
T(n)

May 12, 2009 SE240 10

 Example of Recursion Tree
Solve T(n) = T(n/4) + T(n/2) + n2:
n2
T(n/4) T(n/2)

May 12, 2009 SE240 11

 Example of Recursion Tree
Solve T(n) = T(n/4) + T(n/2) + n2:
n2
(n/4)2 (n/2)2

T(n/16) T(n/8) T(n/8) T(n/4)

May 12, 2009 SE240 12

 Example of Recursion Tree
Solve T(n) = T(n/4) + T(n/2) + n2:
n2
(n/4)2 (n/2)2

(n/16)2 (n/8)2 (n/8)2 (n/4)2

…

Θ(1)

May 12, 2009 SE240 13

 Example of Recursion Tree
Solve T(n) = T(n/4) + T(n/2) + n2:
n2 n2
(n/4)2 (n/2)2

(n/16)2 (n/8)2 (n/8)2 (n/4)2

…

Θ(1)

May 12, 2009 SE240 14

 Example of Recursion Tree
Solve T(n) = T(n/4) + T(n/2) + n2:
n2 n2
5 n2
(n/4)2 (n/2)2
16
(n/16)2 (n/8)2 (n/8)2 (n/4)2
…

Θ(1)

May 12, 2009 SE240 15

 Example of Recursion Tree
Solve T(n) = T(n/4) + T(n/2) + n2:
n2 n2
5 n2
(n/4)2 (n/2)2
16
25 n 2
(n/16)2 (n/8)2 (n/8)2 (n/4)2
256

…
…

Θ(1)

May 12, 2009 SE240 16

 Example of Recursion Tree
Solve T(n) = T(n/4) + T(n/2) + n2:
n2 n2
5 n2
(n/4)2 (n/2)2
16
25 n 2
(n/16)2 (n/8)2 (n/8)2 (n/4)2
256

…
…

Θ(1) Total = n
2
( ( ) + ( ) +L
1 + 16 + 16
5 5 2 5 3
16
)
= Θ(n2) Geometric Series
May 12, 2009 SE240 17
Appendix A: Geometric Series

n +1
1 − x
1 + x + x2 + L + xn = for x ≠ 1
1− x

1
1+ x + x +L =
2
for |x| < 1
1− x

May 12, 2009 SE240 18

Master Method

The master method applies to recurrences of

the form
T(n) = a T(n/b) + f (n) ,
where a ≥ 1, b > 1, and f is asymptotically
positive.

May 12, 2009 SE240 19

 Idea of Master Theorem
Recursion tree:
f (n) f (n)
a
f (n/b) f (n/b) … f (n/b) a f (n/b)
h = logbn a
f (n/b2) f (n/b2) … f (n/b2) a2 f (n/b2)

…
…

#leaves = ah
= alogbn nlogbaΤ (1)
Τ (1)
= nlogba
May 12, 2009 SE240 20
Master Method Intuition

™Three common cases:

¾ Running time dominated by cost at leaves
¾ Running time evenly distributed throughout the tree
¾ Running time dominated by cost at the root

™Consequently, to solve the recurrence, we need

only to characterize the dominant term

™In each case compare f (n) with O(n logb a

)

May 12, 2009 SE240 21

Three Common Cases

Compare f (n) with nlogba:

1. f (n) = O(nlogba – ε) for some constant ε > 0.

• f (n) grows polynomially slower than nlogba
(by an nε factor).
Solution: T(n) = Θ(nlogba) .
Running time dominated by cost at leaves

May 12, 2009 SE240 22

 Idea of Master Theorem
Recursion tree:
f (n) f (n)
a
f (n/b) f (n/b) … f (n/b) a f (n/b)
h = logbn a
f (n/b2) f (n/b2) … f (n/b2) a2 f (n/b2)

…
…

C ASE 1:
CASE 1: The
The weight
weight increases
increases
geometrically
geometrically from
from the
the root
root to the nlogbaΤ (1)
to the
Τ (1) leaves.
leaves. The
The leaves
leaves hold
hold aa constant
constant
fraction
fraction of
of the
the total
total weight.
weight. Θ(nlogba)
May 12, 2009 SE240 23
Three Common Cases (cont.)

Compare f (n) with nlogba:

2. f (n) = Θ(nlogba ).
• f (n) and nlogba grow at similar rates.
Solution: T(n) = Θ(nlogba lgn) .

Running time evenly distributed throughout the tree

May 12, 2009 SE240 24

 Idea of Master Theorem
Recursion tree:
f (n) f (n)
a
f (n/b) f (n/b) … f (n/b) a f (n/b)
h = logbn a
f (n/b2) f (n/b2) … f (n/b2) a2 f (n/b2)

…
…

C ASE 2:
CASE 2: The
The weight isis
weight nlogbaΤ (1)
Τ (1) approximately
approximately the the same
same on
on
each
each of
of the
the log
logbbnn levels.
levels.
Θ(nlogbalg n)
May 12, 2009 SE240 25
Three Common Cases (cont.)
Compare f (n) with nlogba:
3. f (n) = Ω(nlogba + ε) for some constant ε > 0.
• f (n) grows polynomially faster than nlogba (by
an nε factor),
and f (n) satisfies the regularity condition that
a f (n/b) ≤ c f (n) for some constant c < 1.
Solution: T(n) = Θ( f (n)) .
Running time dominated by cost at the root
May 12, 2009 SE240 26
 Idea of Master Theorem
Recursion tree:
f (n) f (n)
a
f (n/b) f (n/b) … f (n/b) a f (n/b)
h = logbn a
f (n/b2) f (n/b2) … f (n/b2) a2 f (n/b2)

…
…

C ASE 3:
CASE 3: The
The weight
weight decreases
decreases
geometrically
geometrically from
from the
the root
root to the nlogbaΤ (1)
to the
Τ (1) leaves.
leaves. The
The root
root holds
holds aa constant
constant
fraction
fraction of
of the
the total
total weight.
weight. Θ( f (n))
May 12, 2009 SE240 27
Master Method Strategy

1. Extract a, b, and f(n) from a given recurrence

logb a
2. Determine n
logb a
3. Compare f(n) and n asymptotically

4. Determine appropriate Master Theorem Case,

and apply it

May 12, 2009 SE240 28

Examples

™ T(n) = 4T(n/2) + n
a = 4, b = 2 ⇒ nlogba = n2; f (n) = n.
CASE 1: f (n) = O(n2 – ε) for ε = 1.
∴ T(n) = Θ(n2).

™ T(n) = 4T(n/2) + n2
a = 4, b = 2 ⇒ nlogba = n2; f (n) = n2.
CASE 2: f (n) = Θ(n2).
∴ T(n) = Θ(n2lg n).
May 12, 2009 SE240 29
Examples

™ T(n) = 4T(n/2) + n3
a = 4, b = 2 ⇒ nlogba = n2; f (n) = n3.
CASE 3: f (n) = Ω(n2 + ε) for ε = 1
and 4(n/2)3 ≤ cn3 (reg. cond.) for c = 1/2.
∴ T(n) = Θ(n3).

May 12, 2009 SE240 30

Substitution Method

1. Guess the form of the solution.

2. Verify by induction.
3. Solve for constants.
Example: T(n) = 4T(n/2) + n
™ [Assume that T(1) = Θ(1).]
™ Guess O(n3) . (Prove O and Ω separately.)
™ Assume that T(k) ≤ ck3 for k < n .
™ Prove T(n) ≤ cn3 by induction.
May 12, 2009 SE240 31
 Example of Substitution
T (n) = 4T ( n / 2) + n
≤ 4c ( n / 2) 3 + n
= (c / 2) n3 + n
= cn3 − ((c / 2) n3 −n ) desired – residual
≤ cn 3 desired
whenever (c/2)n3 – n ≥ 0, for example,
if c ≥ 2 and n ≥ 1.
residual

May 12, 2009 SE240 32

Example (cont.)
™ We must also handle the initial
conditions, that is, ground the induction
with base cases.
™ Base: T(n) = Θ(1) for all n < n0, where n0
is a suitable constant.
™ For 1 ≤ n < n0, we have “Θ(1)” ≤ cn3, if
we pick c big enough.

This bound is not tight!

May 12, 2009 SE240 33
A Tighter Upper Bound?
We shall prove that T(n) = O(n2).
Assume that T(k) ≤ ck2 for k < n:
T ( n) = 4T ( n / 2) + n
≤ cn 2 + n
≤ cn2

for no choice of c > 0. Lose!

May 12, 2009 SE240 34

A Tighter Upper Bound
IDEA: Strengthen the inductive hypothesis.
™ Subtract a low-order term.
Inductive hypothesis: T(k) ≤ c1k2 – c2k for k < n.
T (n) = 4T ( n / 2) + n
≤ 4(c1 (n / 2) 2 − c2 (n / 2))+ n
= c1n 2 − 2c2 n + n
= c1n 2 − c2 n − (c2 n − n)
≤ c1n 2 − c2 n if c > 1.
2
Pick c1 big enough to handle the initial conditions.
May 12, 2009 SE240 35