Analysis of Algorithms: Algorithm Input

Analysis of Algorithms
Input Algorithm Output
An algorithm is a step-by-step procedure for

solving a problem in a finite amount of time.
Analyze an algorithm = determine its efficiency

CSI2110A (2013), Prof. Lucia Moura
1

Input Algorithm Output
An algorithm is a step-by-step procedure for

solving a problem in a finite amount of time.
Analyze an algorithm = determine its efficiency

2

Efficiency ?
Running time
Memory
Quality of the result .
Simplicity .

3

Running Time
The running time depends on

the input size
It also depends on the input data:

Different inputs can have different
running times

4

Running time
Best case

Average case

The running time depends on Worst case

the input size
120
100

Running Time
80

60

40

It also depends on the input data:
Different inputs can have different 20

running times 0

1000

2000 3000

4000
Input Size


5

Running Time of an algorithm
Average case time is often

difficult to determine.
We focus on the worst
case running time.
Easier to analyze
Crucial to applications such
as games, finance and
robotics

6

Example .
If x is odd
return x
15
If x is even 15

compute the sum S
of the first x integers
4
return S 10


7

Measuring the Running Time
How should we measure the running time of

an algorithm?
Approach 1: Experimental Study

8

Beyond Experimental Studies
Experimental studies have several

limitations:
need to implement
limited set of inputs
hardware and software environments.

9

Theoretical Analysis
We need a general methodology that: -
Uses a high-level description of the algorithm

(independent of implementation).
Characterizes running time as a function of the

input size.
Takes into account all possible inputs.
Is independent of the hardware and software environment.

10

Analysis of Algorithms
Primitive Operations: Low-level computations independent

from the programming language can be identified in
pseudocode.
Examples:
calling a method and returning from a method
arithmetic operations (e.g. addition)
comparing two numbers, etc.
By inspecting the pseudo-code, we can count the number

of primitive operations executed by an algorithm.

11

Example:
Algorithm arrayMax(A, n):

Input: An array A storing n integers.
Output: The maximum element in A.
currentMax A[0]
for i 1 to n -1 do
if currentMax < A[i] then
currentMax A[i]
return currentMax

12

A

5
13
4
7
6
2
3
8
1
2

currentMax
currentMax A[0]
for i 1 to n -1 do
currentMax A[i]
return currentMax

13

A

5
13
4
7
6
2
3
8
1
2

currentMax 5

currentMax A[0]
for i 1 to n -1 do
currentMax A[i]
return currentMax

14

A

5
13
4
7
6
2
3
8
1
2

currentMax 5

currentMax A[0]
for i 1 to n -1 do
currentMax A[i]
return currentMax

15

A

5
13
4
7
6
2
3
8
1
2

currentMax 13

currentMax A[0]
for i 1 to n -1 do
currentMax A[i]
return currentMax

16

A

5
13
4
7
6
2
3
8
1
2

currentMax 13

currentMax A[0]
for i 1 to n -1 do
currentMax A[i]
return currentMax

17

A

5
13
4
7
6
2
3
8
1
2

currentMax 13

currentMax A[0]
for i 1 to n -1 do
currentMax A[i]
return currentMax

18

What are the primitive operations to count ?
A

5
13
4
7
6
2
3
8
1
2

-- Comparisons
.
-- Assignments
currentMax 13

currentMax A[0]
for i 1 to n -1 do
currentMax A[i]
return currentMax

19

WORST CASE
currentMax A[0] 1 assignment

for i 1 to n -1 do
n-1 comparisons
currentMax A[i]
n-1 assignments
return currentMax
5
7
8
10
11
12
14
16
17
20


20

BEST CASE
15
1
12
3
9
7
6
4
2
2

currentMax A[0] 1 assignment

for i 1 to n -1 do
n-1 comparaisons
currentMax A[i]
0 assignments
return currentMax
21

Complexity - Summary
Worst Case: Best Case:

n-1 comparisons n-1 comparisons
n assignments 1 assignment

22

Another Example
Determining the rank of an element in array A

of size sizeA
i0 1 assignment
while (A[i] element)
sizeA comparisons
i i+1
& assignment
return i (if we are not lucky)
Worst Case


23

Big-Oh
(upper bound)
given two functions f(n) and g(n), we say that
f(n) is O(g(n))
if and only if there are positive constants
c and n0 such that
f(n) c g(n) for n n0
c g(n)
f(n)
n0
24n

What does it mean
c g(n) ?
Example:
g(n) = n


25

What does it mean
c g(n) ?
2 g(n) = 2 n


26

What does it mean
c g(n) ?
3 g(n) = 3 n


27

g(n) = n2

Graphical example
f(n) = 2n+1
f(n) is O(n2)
f(n) g(n) for n 1+ 2

n
1+ 2

28

But also
f(n) = 2n+1
g(n) = n

n
f(n) c g(n) for n n0 29


But also
f(n) = 2n+1
2 g(n) = 2 n

n
f(n) c g(n) for n n0 30


But also
f(n) = 2n+1
3 g(n) = 3 n

f(n) is O(n) because

f(n) 3 g(n) for n 1

31

On the other hand
n2 is not O(n) because there is no c and n0 such that:

n2 cn for n n0
( no matter how large a c is chosen there is an n big

enough that n2 > c n ) .
n2

32

n0 n
An Example
Prove that f(n) = 60n2 + 5n + 1 is O(n2)

We must find a constant c and a constant n0 such that:
60n2 + 5n + 1 c n2 for all nn0
5n 5n2 for all n1 f(n) 60n2 +5n2 + n2 for all

1 n2 for all n1 n1
f(n) 66n2 for all n1 c= 66 and n0=1 => f(n) = O(n2)

33

O(1) < O(log n) < O(n) < O(n log n) < O(n2) <
O(n3) < O(2n)
n2
log(n)
n0 n
5
n0 n
34

n =
2
16
256
1024

log log n
0
2
3
3.32

log n
1
4
8
10

n
2
16
256
1024

n log n
2
64
448
10 200

n2
4
256
65 500
1.05 * 106

n3
8
4 100
16 800 800
1.07 * 109

n

2 4
35 500
11.7 * 106
1.80 * 10308


35

Asymptotic Notation (cont.)
Note: Even though it is correct to say

7n - 3 is O(n3), a better statement is
7n - 3 is O(n), that is,
one should make the approximation as tight as possible

36

Theorem:
If g(n) is O(f(n)) , then for any constant
c >0
g(n) is also O(c f(n))
Theorem:
O(f(n) + g(n)) = O(max(f(n), g(n)))
Ex 1:
2n3 + 3n2 = O (max(2n3, 3n2))
= O(2n3) = O(n3)
Ex 2:
n2 + 3 log n 7 = O(max(n2, 3 log n 7))
= O(n2)
37

Simple Big Oh Rule:
Drop lower order terms and constant factors
7n-3 is O(n)
8n2log n + 5n2 + n is O(n2log n)
12n3 + 5000n2 + 2n4 is O(n4)

38

Other Big Oh Rules:
Use the smallest possible class of

functions
Say 2n is O(n) instead of 2n is O(n2)
Use the simplest expression of the class

Say 3n + 5 is O(n) instead of
3n + 5 is O(3n)

39

Asymptotic Notation
(terminology)
Special classes of algorithms:
constant: O(1)
logarithmic: O(log n)
linear: O(n)
quadratic: O(n2)
cubic: O(n3)
polynomial: O(nk), for some fixed k >0
exponential: O(an), for some fixed a > 1

40

Example of Asymptotic Analysis
An algorithm for computing prefix averages
The i-th prefix average of an array X is average of

the first (i + 1) elements of X
A[i] = X[0] + X[1] + + X[i]
(i + 1)
5
13
4
8
6
2
3
8
1
2

5
9
7.3
7.5

41

Algorithm prefixAverages1(X, n)
Input array X of n integers

Output array A of prefix averages of X #operations
A new array of n integers

for i 0 to n - 1 do
s X[0]
for j 1 to i do
s s + X[j]
A[i] s / (i + 1)
return A

42

for i 0 to n - 1 do
s X[0]
i
for j 1 to i do
s s + X[j]
A[i] s / (i + 1)

5
13
4
7
6
2
3
8
1
2

The inner for-cycle is never executed

43

for i 0 to n - 1 do
s X[0]
i
for j 1 to i do
s s + X[j]
A[i] s / (i + 1)

5
13
4
7
6
2
3
8
1
2

j=1
5
9


44

for i 0 to n - 1 do
s X[0]
i
for j 1 to i do
s s + X[j]
A[i] s / (i + 1)

5
13
4
7
6
2
3
8
1
2

j = 1,2
5
9
7.3


45

for i 0 to n - 1 do
s X[0]
i
for j 1 to i do
s s + X[j]
A[i] s / (i + 1)

5
13
4
7
6
2
3
8
1
2

j = 1,2,3
5
9
7.3
7.5


46

i

5
13
4
7
6
2
3
8
1
2

Basic Operations: Additions, assignments
for i 0 to n - 1 do
s X[0]
for j 1 to i do
s s + X[j]
A[i] s / (i + 1)


47

Lets count the number of for i 0 to n - 1 do
ADDITIONS s X[0]
i
for j 1 to i do
s s + X[j]
A[i] s / (i + 1)

5
13
4
7
6
2
3
8
1
2

While i grows, the number of additions inside

the inner for-cycle grows as well
i=0 0
i=1 1
How many additions in total ?

i=2 2
i=3 3
i=4 4
..
i=n-1 n-1 CSI2110A (2013), Prof. Lucia Moura
48

Counting as basic operations: additions and divisions

Algorithm prefixAverages1(X, n)
Input array X of n integers

Output array A of prefix averages of X
A new array of n integers

for i 0 to n - 1 do
s X[0] 1+2+3+4 +n-1 additions
for j 1 to i do
s s + X[j]
A[i] s / (i + 1) n division
return A
49

--- The running time of
prefixAverages1 is
then O (1 + 2 + + n)
--- The sum of the first n

integers is n(n + 1) / 2
There is a simple visual

proof of this fact

50

Thus, algorithm prefixAverages1 runs in time
O(n(n + 1) / 2)
which is
O(n2)
TO REMEMBER
1 + 2 + + n = n(n+1)
2

51

Another Example:
A better algorithm for computing prefix averages
Algorithm prefixAverages2(X):

Input: An n-element array X of numbers.

Output: An n -element array A of numbers such that A[i] is the
average of elements X[0], ... , X[i].

Let X be an array of n numbers.

s 0

for i 0 to n-1 do

s s + X[i]

A[i] s/(i+ 1)

return array A


52

s 0

for i 0 to n-1 do

s s + X[i]

i

A[i] s/(i+ 1)

5
13
4
7
6
2
3
8
1
2

s=0

53

s 0

for i 0 to n-1 do

s s + X[i]

i

A[i] s/(i+ 1)

5
13
4
7
6
2
3
8
1
2

s=5
5
9


54

s 0

for i 0 to n-1 do

s s + X[i]

i

A[i] s/(i+ 1)

5
13
4
7
6
2
3
8
1
2

s=18
5
9
7.3


55

s 0

for i 0 to n-1 do

s s + X[i]

i

A[i] s/(i+ 1)

5
13
4
7
6
2
3
8
1
2

s=22
5
9
7.3
7.5


56

Complexity of Algorithm prefixAverages2

Counting as basic operations: additions and divisions

Let X be an array of n numbers.

s 0

for i 0 to n-1 do

s s + X[i]
O(n) additions

A[i] s/(i+ 1)
O(n)
divisions

return array A

O(n) time

57

Comparison

prefixAverages1
prefixAverages2

O(n2) O(n)

58

big-Omega
(lower bound)
f(n) is (g(n))
if there exist c > 0 and n0 > 0 such that
f(n) c g(n) for all n n0
(thus, f(n) is (g(n)) iff g(n) is O(f(n)) )

f(n)
c g(n)
n0
n
59

big-Theta
is big theta
g(n) is (f(n))
<===>
if g(n) O(f(n))
AND
f(n) O(g(n))

60

An Example
We have seen that
f(n) = 60n2 + 5n + 1 is O(n2)
but 60n2 + 5n + 1 60n2 for n 1

So: with c = 60 and n0 = 1
f(n) is (n2)
f(n) c n2 for all n 1
f(n) is O(n2)
AND
f(n) is (n2)
f(n) is (n2)
61

Intuition for Asymptotic
Notation
Big-Oh
f(n) is O(g(n)) if f(n) is asymptotically
less than or equal to g(n)
big-Omega
f(n) is (g(n)) if f(n) is asymptotically
greater than or equal to g(n)
big-Theta
f(n) is (g(n)) if f(n) is asymptotically
equal to g(n)

62

Math You Need to Review
Logarithms and Exponents
properties of logarithms:
logb(xy) = logbx + logby

logb (x/y) = logbx logby

logb xa = alogbx

logba=
logxa/logxb

properties of exponentials:
a(b+c) = aba c

abc = (ab)c

ab /ac = a(b-c)

b = a logab

bc = a c*logab

63

More Math to Review
Floor: x = the largest integer x

Ceiling: x = the smallest integer x
Summations:
Geometric progression:

64

More Math to Review
Arithmetic Progression
n
S=id= 0+ d + 2d + + nd
i=0
S = nd+(n-1)d+(n-2)d + + 0
2S = nd + nd + nd + + nd
= (n+1) nd

S = d/2 n(n+1)

for d=1 S = 1/2 n(n+1)

65

More Math to Review
Geometric Progression
n
S = ri = 1 + r + r2 + + rn
i=0
rS = r + r2 + + rn + rn+1
rS - S = r + r2 + + rn + rn+1
- r - r2 - - r n - 1
= (r-1)S = rn+1 - 1
S = (rn+1-1)/(r-1)

For r=2, S = (2n+1-1)

66

Analysis of Algorithms: Algorithm Input

Caricato da

Informazioni sul documento

Titolo originale

Copyright

Formati disponibili

Condividi questo documento

Condividi o incorpora il documento

Opzioni di condivisione

Hai trovato utile questo documento?

Questo contenuto è inappropriato?

Copyright:

Formati disponibili

Analysis of Algorithms: Algorithm Input

Caricato da

Copyright:

Formati disponibili

Analysis of Algorithms

Input Algorithm Output

An algorithm is a step-by-step procedure for

Analyze an algorithm = determine its efficiency

An algorithm is a step-by-step procedure for

Analyze an algorithm = determine its efficiency

Quality of the result .

CSI2110A (2013), Prof. Lucia Moura

The running time depends on

It also depends on the input data:

CSI2110A (2013), Prof. Lucia Moura

CSI2110A (2013), Prof. Lucia Moura

Average case time is often

CSI2110A (2013), Prof. Lucia Moura

CSI2110A (2013), Prof. Lucia Moura

How should we measure the running time of

CSI2110A (2013), Prof. Lucia Moura

Experimental studies have several

CSI2110A (2013), Prof. Lucia Moura

We need a general methodology that: -

Uses a high-level description of the algorithm

Characterizes running time as a function of the

Takes into account all possible inputs.

Is independent of the hardware and software environment.

Primitive Operations: Low-level computations independent

By inspecting the pseudo-code, we can count the number

CSI2110A (2013), Prof. Lucia Moura

Algorithm arrayMax(A, n):

CSI2110A (2013), Prof. Lucia Moura

CSI2110A (2013), Prof. Lucia Moura

CSI2110A (2013), Prof. Lucia Moura

CSI2110A (2013), Prof. Lucia Moura

CSI2110A (2013), Prof. Lucia Moura

CSI2110A (2013), Prof. Lucia Moura

CSI2110A (2013), Prof. Lucia Moura

CSI2110A (2013), Prof. Lucia Moura

currentMax A[0] 1 assignment

CSI2110A (2013), Prof. Lucia Moura

currentMax A[0] 1 assignment

Worst Case: Best Case:

CSI2110A (2013), Prof. Lucia Moura

Determining the rank of an element in array A

CSI2110A (2013), Prof. Lucia Moura

CSI2110A (2013), Prof. Lucia Moura

CSI2110A (2013), Prof. Lucia Moura

CSI2110A (2013), Prof. Lucia Moura

f(n) is O(n) because

f(n) c g(n) for n n0

CSI2110A (2013), Prof. Lucia Moura

n2 is not O(n) because there is no c and n0 such that:

( no matter how large a c is chosen there is an n big

CSI2110A (2013), Prof. Lucia Moura

Prove that f(n) = 60n2 + 5n + 1 is O(n2)

5n 5n2 for all n1 f(n) 60n2 +5n2 + n2 for all

f(n) 66n2 for all n1 c= 66 and n0=1 => f(n) = O(n2)

CSI2110A (2013), Prof. Lucia Moura

CSI2110A (2013), Prof. Lucia Moura

Note: Even though it is correct to say

CSI2110A (2013), Prof. Lucia Moura

Drop lower order terms and constant factors

8n2log n + 5n2 + n is O(n2log n)

12n3 + 5000n2 + 2n4 is O(n4)

CSI2110A (2013), Prof. Lucia Moura