Complexity Analysis

Text is mainly from Chapter 2 by Drozdek

Page:1
Dr. Mehreen Saeed
CSC 201:Data Structures and Algorithms
SPRING 2015

Computational complexity

Developed by Juris Hartmanis & Richard E. Stearns

Compares efficiency of algorithms
Assesses the degree of difficulty of an algorithm
How much effort is needed to APPLY an algorithm
Estimate the resources required by an algorithm
How costly it is

Page:2
Dr. Mehreen Saeed
CSC 201:Data Structures and Algorithms
SPRING 2015

What is the COST of an algorithm???

Time
Space

Page:3
Dr. Mehreen Saeed
CSC 201:Data Structures and Algorithms
SPRING 2015

How can you practically compare 2

algorithms???
Run them on the same machine
Implement them using the same language
Only time units such as nano-seconds or micro
seconds should not be usedWHY???
Logical units expressing the relation between
the size of file n and the amount of time t
required to process it should be used
Same input data should be used
Page:4
Dr. Mehreen Saeed
CSC 201:Data Structures and Algorithms
SPRING 2015

Problems with this approach

Effort
One program may be better written than the
other
Choice of tests may favor one of the
algorithms

We want a method in which we compare two algorithms WITHOUT executing them

The method should be independent of hardware/compiler/operating system

Page:5
Dr. Mehreen Saeed
CSC 201:Data Structures and Algorithms
SPRING 2015

What do we need?????
We require algorithm analysis!!!
Estimate the performance of an algorithm through
The number of operations required to process an input
Process an input of certain size

Require a function expressing relation between n & t called

time complexity function T(n)
For calculating T(n) we need to compute the total number of
program steps (can be the number of executable statements
or meaningful program segment)

Page:6
Dr. Mehreen Saeed
CSC 201:Data Structures and Algorithms
SPRING 2015

Calculating T(n)
A program step is the syntactically / semantically
meaningful segments of a program
A step DOESNOT correspond to a definite time unit
A step count is telling us how run time for a program
changes with change in data size
Calculate the total number of steps/executable
statements in a program
Find the frequency of each statement and sum them
up
Dont count comments and declarations
Page:7
Dr. Mehreen Saeed
CSC 201:Data Structures and Algorithms
SPRING 2015

Analysis of for Loop

1:

for (i=0;i<n;++i)

2:
sum++;
3: cout << sum
You can see.
That statement 1 executes n+1 times
Statement 2 executes n times
Statement 3 executes 1 time
T(n) = ???

The condition always executes one more time than the loop
itself
Page:8
Dr. Mehreen Saeed
CSC 201:Data Structures and Algorithms
SPRING 2015

Nested for loops

for (i=0;i<n;++i)
{ for (j=0;j<m;++j)
sum++
}
The statement sum++ executes n*m times
So in a nested for loop if the loop variables are independent then:
The total number of times a statement executes =
outer loop times * inner loop times

Page:9
Dr. Mehreen Saeed
CSC 201:Data Structures and Algorithms
SPRING 2015

Problems with T(n)

T(n) is difficult to calculate
T(n) is also not very meaningful as step size is not exactly
defined
T(n) is usually very complicated so we need an approximation
of T(n).close to T(n).
This measure of efficiency or approximation of T(n) is called
ASYMPTOTIC COMPLEXITY or ASYMPTOTIC
ALGORITHM ANALYSIS

Asymptotic complexity studies the efficiency of an algorithm as

the input size becomes large
Page:10
Dr. Mehreen Saeed
CSC 201:Data Structures and Algorithms
SPRING 2015

Example
If T(n) = 7n+100
What is T(n) for different values of n???
n

T(n)

Comment

107

Contributing factor is 100

135

Contributing factor is 7n and 100

10

170

Contributing factor is 7n and 100

100

800

Contribution of 100 is small

1000

7100

Contributing factor is 7n

10000

70100

Contributing factor is 7n

106

7000100

What is the contributing factor????

When approximating T(n) we can IGNORE the 100 term for very large value of n
and say that T(n) can be approximated by 7(n)

Page:11
Dr. Mehreen Saeed
CSC 201:Data Structures and Algorithms
SPRING 2015

Example 2
T(n) = n2 + 100n + log10n +1000
n

T(n)

n2

100n

log10n

1000

Val

Val

Val

Val

1101

0.1%

100

9.1%

0%

1000

90.8%

10

2101

100

5.8%

1000

47.6%

0.05%

1000

47.6%

100

21002

10000

47.6%

10000

47.6%

0.99%

1000

4.76%

105

10,010,001,005

1010

99.9%

107

.099%

0.0%

1000

0.00%

When approximating T(n) we can IGNORE the last 3 terms and say say that T(n) can
be approximated by n2
Page:12
Dr. Mehreen Saeed
CSC 201:Data Structures and Algorithms
SPRING 2015

Big-Oh or Big-O

Definition:

f(n) is O(g(n)) if there exist positive numbers c & N such that f(n)<=cg(n)
for all n>=N

g(n) is called the upper bound on f(n) OR

f(n) grows at the most as large as g(n)

Example:
T(n) = n2 + 3n + 4
n2 + 3n + 4 <= 2 n2 for all n>10
(What is f(n) and what is g(n)????)
(What is c ???? & What is N???)

so we can say that T(n) is O(n2) OR

T(n) is in the order of n2.
T(n) is bounded above by a + real multiple of n2
Page:13
Dr. Mehreen Saeed
CSC 201:Data Structures and Algorithms
SPRING 2015

What does it mean???? An example

For three constants c1, c2, c3 (doesnt
matter what the values are) there will be an n
beyond which the program with complexity
c3n will be faster than the one with
complexity c1n2+c2n
This value of n is called the break even
point
If n=0 then c3n will always be faster
Page:14
Dr. Mehreen Saeed
CSC 201:Data Structures and Algorithms
SPRING 2015

COMPLEXITY
Space complexity is the amount of memory a
program needs to run to completion
Time complexity is the amount of computer
time a program needs to run to completion
If
f(n) = amnm+am-1nm-1+.+a1n+a0
then f(n) = O(nm)
Page:15
Dr. Mehreen Saeed
CSC 201:Data Structures and Algorithms
SPRING 2015

FACTS ABOUT O(n)

Gives us means for comparing algorithms
It tells us about the growth rate of an algorithm
as input size becomes large
Its also called the asymptotic growth rate or
asymptotic order or simply order of the
function. ASYMPTOTIC NOTATION

Page:16
Dr. Mehreen Saeed
CSC 201:Data Structures and Algorithms
SPRING 2015

Properties of Big-Oh

If f(n) is O(g(n)) and g(n) is O(h(n)) then f(n) is O(h(n))

If f(n) is O(h(n)) and g(n) is O(h(n)) then f(n) + g(n) is O(h(n))
ank is O(nk) where a is a constant
nk is O(nk+j ) for any positive j
If f(n)=cg(n) then f(n) is O(g(n))
The function logan is O(logbn) for all positive a & b 1
so we can just write O(log n) (without the base)

Page:17
Dr. Mehreen Saeed
CSC 201:Data Structures and Algorithms
SPRING 2015

Growth rates
This is O(1)

sum++;
for (i=0;i<n;++i)
sum++;

This is O(n)

This is O(n2)

for (i=0;i<n;++i)
for (j=0;j<n;++j)
sum++;
Page:18
Dr. Mehreen Saeed
CSC 201:Data Structures and Algorithms
SPRING 2015

Growth rates
Which growth rate is better???
Add n!, nn , 2n to the
graph and see where
they fit

Here we have:
Constant
Linear
Logarithmic
Quadratic
cubic
This is certainly what I
would like my program
to be!!! (but this is just
wishful thinking)

Graph from Adam Drozdeks book

Page:19
Dr. Mehreen Saeed
CSC 201:Data Structures and Algorithms
SPRING 2015

Comparison of Growth Rates

N

log2N

N log2N

N2

N3

2N

24

64

512

256

64

384

4096

262,144

About 5 years

128

896

16,384

2,097,152 Approx 6 billion years,

600,000 times more than
age of univ.

(If one operation takes 10-11 seconds)

REF: Nell Dale

Page:20
Dr. Mehreen Saeed
CSC 201:Data Structures and Algorithms
SPRING 2015

EXAMPLE OF O(log n)
sum = 0;
for (i=1;i<n;i=i*2)
sum = sum+1;

i
1
2
4
8

n/2

sum
1
1+1=2
2+1=3
3+1=4

exit loop

loop will not execute for i=n

The value of sum is log2n at the end of the loop

log2n
sum not incremented
further

Page:21
Dr. Mehreen Saeed
CSC 201:Data Structures and Algorithms
SPRING 2015

EXAMPLE OF O(log n)suppose you are coloring the table

take half

discarded

take half

discarded

take half

take half

take half
Page:22
Dr. Mehreen Saeed
CSC 201:Data Structures and Algorithms
SPRING 2015

You didnt color all cells..everytime you discarded half the cells

EXAMPLE OF O(n)suppose you are coloring the table

You will color all the cells one by one

take half
Page:23
Dr. Mehreen Saeed
CSC 201:Data Structures and Algorithms
SPRING 2015

EXAMPLE OF O(n2)suppose you are coloring the table

You will color each cell n times

take half
Page:24
Dr. Mehreen Saeed
CSC 201:Data Structures and Algorithms
SPRING 2015

EXAMPLE OF O(2n)suppose you are coloring the table

The number of times you color a cell is written there

1

16

32

64

128 256 512 1024 2048

Page:25
Dr. Mehreen Saeed
CSC 201:Data Structures and Algorithms
SPRING 2015

SOME MATH
Normally these formulas are very handy :

x y z then

If
n

ai
i 1

i
i 1

n
( a1 a n )
2

n
(n 1)
2

i2

i 1

y log x z

x (n 1) x ( n 1) nx ( n 2 )
kx

2
(
x

1
)
k 0
n

n(n 1)( 2n 1)
( for n 1)
6

m 1
1

r
k
r

1 r
k 0
m

Page:26
Dr. Mehreen Saeed
CSC 201:Data Structures and Algorithms
SPRING 2015

SOME RULES
Rule 1
for (i=0;i<n;i=i+k) Anything inside the loop will run approximately n/k times
Rule 2
for (i=n;i>0;i=i-k) Anything inside the loop will run approximately n/k times
Rule 3
for (i=1;i<n;i=i*k) Anything inside the loop will run approximately logkn times
Rule 4
If the loop variables are independent then the total times a statement inside a
nested loop is executed is equal to the product of the times the individual loops
run
e.g. for (i=0;i<n;++i)
for (j=0;j<m;++j)
A statement inside the above nested loop will run n*m times

Page:27
Dr. Mehreen Saeed
CSC 201:Data Structures and Algorithms
SPRING 2015

Some rules(contd.)
Rule 5
for(i=1;i<=n;++i)
for (j=1;j<=i;++j)
The above nested loop approximately runs (1/2)n(n+1) times.
The variable j depends upon the value of i
Rule 6
for(i=1;i<=n;i=i*2)
for (j=1;j<=i;++j)
The statements in the above nested loop approximately run 2n-1 times.
Rule 7
for(i=0;i<n;i=i+1)
for (j=i;j<n;++j)
The statements in the above nested loop approximately run n(n+1)/2
times.
Page:28
Dr. Mehreen Saeed
CSC 201:Data Structures and Algorithms
SPRING 2015

SELECTION SORT
for (i=0;i<n;++i)
{
maxIndex = FindMaxIndex(arr,i,n-1);
swap(arr[i],arr[maxIndex]);
}
//FindMaxIndex(int arr[],int startIndex,int endIndex)
//finds the maximum item in the partial array
//from start index to end index
what is the complexity of the above???
Page:29
Dr. Mehreen Saeed
CSC 201:Data Structures and Algorithms
SPRING 2015

BUBBLE SORT
bool done = false;
for(i = 1; (i < n) && !done; i++) //repeat a pass of bubble sort
{
done = true;
for (j=0; j <n-i; j++)
//inner loop swaps consecutive items
{
if (arr[j+1] < arr[j])
{
swap(arr[j+1],arr[j])
done = false;
//a swap is made and so sorting continues
}//end of if
}//end of inner for
}//end of outer for

what is the complexity of the above???

Page:30
Dr. Mehreen Saeed
CSC 201:Data Structures and Algorithms
SPRING 2015