MIT203.pdf For MSC IT

Analysis and Design of Algorithms Unit 1
Sikkim Manipal University Page No. 1

Unit 1 Introduction to Algorithms
Structure:
1.1 Introduction
Objectives
1.2 Concept of Algorithm
Etymology
Definitions of Algorithm
1.3 Role of Algorithm in Computing
Analysis and design of algorithms
Properties of algorithms
Algorithm, program and psuedocode
Use of algorithms in computing
1.4 Fundamentals of Algorithm
Asymptotic notations
Time and space complexity and efficiency
1.5 Important Types of Algorithm
Simple recursive
Backtracking
Divide and conquer
Dynamic programming
Greedy algorithms
Branch and bound algorithms
Brute force algorithms
Randomized algorithms
1.6 Fundamental Data Structures
Data structure definition
Use of data structure
Data structure classification
Data structure characteristics
1.7 Summary
1.8 Glossary
1.9 Terminal Questions
1.10 Answers

1.1 Introduction
We use algorithms to solve problems in a systematic way. It is difficult to
give a single specific definition for an algorithm. Different fields of
development and study use algorithms to solve their own problems.
Levitin defines algorithm as given below:
An algorithm is a sequence of unambiguous instructions for solving a
problem, i.e., for obtaining a required output for any legitimate input in a
finite amount of time.
This unit covers the various definitions of algorithm, its types, properties and
the steps for designing it. This unit gives a brief idea on solving a problem
using different algorithmic problem solving techniques. It also introduces
various data structures, their classification and characteristics.
Objectives:
After studying this unit, you should be able to:
define Algorithm
explain the role of algorithm in computing
describe the fundamentals of algorithms
list the important types of algorithms
describe the fundamental data structures

1.2 Concept of Algorithm
We use computers to solve lots of problems in a fast and accurate way.
Therefore we need efficient algorithms for every process in a computer. The
concept of algorithms is in use from the early years of computation and
study. Let us start with the etymology of algorithms. We will also study the
different definitions of algorithms used to solve problems in different areas.
1.2.1 Etymology
The word algorithm is derived from the name of Abu Abdullah Muhammad
Ibn Musa al-Khwarizmi, a 9th-century Persian mathematician, astronomer,
geographer and scholar. This scholar wrote the first book on systematic
solution of linear and quadratic equations namely Kitab al-Jabr wa-l-
Muqabala. The rules for performing arithmetic using Arabic numerals were
originally known as algorism but later in the 18
th
century it was changed to
algorithm.
1.2.2 Definitions of algorithm
An algorithm is defined as a set of well defined instructions used to
accomplish a particular task. It is considered as the cornerstone of good
programming. The efficiency of algorithms depends upon speed, size and
resource consumption. Different algorithms may finish the same process
with a different set of instructions in more or less time, space, or effort than
others. There is no fixed definition for an algorithm. It varies according to the
area of use.
Various definitions of algorithms are given below:
It is the exact set of instructions describing the order of actions to
achieve the result of the decision problem in finite time (Old
interpretation).
"Algorithm is a finite set of rules that defines the sequence of operations
to solve a specific set of goals and has five important features:
finiteness, definiteness, input, output, efficiency" (D. Knuth).
"Algorithms are all systems of calculations performed on strictly defined
rules, which, after a number of steps obviously leads to the solution of
the problem" (A. Kolmogorov).
"Algorithm is the exact prescription, defining the computing process,
going from the variable input data to the desired result" (A. Markov).
"Algorithm is the exact requirement of the performance in a specific
order of certain operations, leading to the solution of all problems of this
type" (Philosophical dictionary, ed. M. Rosenthal).
Self Assessment Questions
1. The rules for performing arithmetic using Arabic numerals were originally
known as _____________.
2. The efficiency of algorithms depends upon ____________, ________
and __________ consumption.
3. An algorithm is considered as the cornerstone of ____________.

1.3 Role of Algorithm in Computing
In the previous section we discussed the definitions and basic use of
algorithms. Now, let us see how algorithms are used for computing.
We believe that a computer can do anything and everything that we
imagine. But the truth is that the computers work on algorithms written by
humans themselves! We need to write more software algorithms due to the
increasing complexity of computers. Algorithms are available for many
different applications and highly advanced systems such as artificial
intelligence. These algorithms may become common in the future.
1.3.1 Analysis and design of algorithms
Analysis of algorithm means to study the specification of the algorithm and
come to conclusions about how the implementation of that algorithm will
perform in general. To analyse an algorithm means to determine the amount
of resources necessary to execute it. Analysis of algorithm gives the
following output:
Total or maximum memory space needed for the data
Total size of the code
Correctness of the result
Complexity of the process
Robustness of the process
Most algorithms can work with inputs of arbitrary length. We usually state
the efficiency or running time of an algorithm as a function relating the input
length to the number of steps (time complexity) or storage locations (space
complexity).
Algorithm analysis provides theoretical estimates for the resources needed
by any algorithm which solves a given computational problem. An important
result of analysing is that it allows you to make a quantitative judgment
about the value of one algorithm over another.
Designing an algorithm is the process of creating an algorithm that meets all
the output of analysis phase. Algorithm design is a specific method to create
a mathematical process in solving problems.
We need to consider the following points while designing an algorithm:
Optimize the algorithm
Choose proper data structures
Recognize the problems
Resolve the problems using well-researched standard problems
Once an algorithm is devised, it is necessary to show that it computes the
correct answer for all legitimate inputs. This process is known as algorithm
validation.
1.3.2 Properties of algorithms
An algorithm may have zero or more inputs externally and it should produce
one or more output. Also, an algorithm must terminate after a finite number
of steps. Properties of algorithm include:
Correctness It should provide correct and accurate output for all
legitimate input.
Definiteness Each instruction should be clear, precise and
unambiguous. Input and output must be well defined.
Finiteness When we trace out the instructions of an algorithm, it has
to terminate after a finite number of steps, for all cases.
Effectiveness Every instruction must be very basic so that it can be
carried out in principle, by a person using only pencil and paper.
Generality Algorithms need to have general instructions which can be
applied in any case.
1.3.3 Algorithm, program and psuedocode
An algorithm is a finite set of unambiguous statement to solve a problem in
finite amount of time. It can be natural language expressions designed for
any common man.
A program can be called as an algorithm implemented using the required
data structures. It is the expression of an algorithm in a programming
language with all the language specific codes. Procedure, function and
subroutine are synonyms for a program.
A pseudocode is a compact, informal environment-independent description
of a computer programming algorithm. It uses the structural conventions of a
programming language, but only humans can read it and not machines.
Pseudocode omits details that are not essential for human understanding of
the algorithm, such as variable declarations, system-specific code and
subroutines. There is no specific standard for pseudocode syntax.
Consider the example of bubble sort, which is a straightforward and
simplistic method of sorting an array of numbers.
Algorithm for bubble sort
1) Traverse the array from the beginning to the end
2) For each item, compare it to the next item.
3) If the item is larger than the next item, swap them.
4) Do the same again for all items except the last item.
5) Do it again for all items except the last two items, and so on.
6) Sort the array when there are only two items left and they have been
swapped (if necessary).
The C program for bubble sort is given below:
C Program for bubble sort
#include<stdio.h>
#include<conio.h>
#define SIZE 5
void main()
{
int a[SIZE],n,i,j,temp;
clrscr();
printf("enter the elements ");
for(i=0;i<SIZE;i++)
scanf("%d",&a[i]);
printf("the sorted list is :->\n");
for(i=0;i<SIZE;i++)
for(j=i;j<SIZE-i;j++)
if(a[j]>a[j+1])
{ temp=a[j];
a[j]=a[j+1];
a[j+1]=temp;
}
for(i=0;i<SIZE;i++)
printf("%d",a[i]);
getch();
}

1.3.4 Use of algorithms in computing
We use algorithms to perform various logical and mathematical operations
such as sorting, searching, merging, logic programming, matrix processing,
parsing, predicting statistics, solving basic geometric problems, displaying
graphics and performing common mathematical calculations.
Sorting algorithms arrange the data elements in a specific order. It can be in
ascending or descending order. Internal sorting is a type of sorting which
arranges data internally in the memory of the computer. Algorithms become
more sophisticated when the number of elements to be sorted is more.
Searching is the process where a particular element is looked up in a
collection of elements. The two basic searching algorithms are linear search
and binary search. Linear search looks up a sequence of data objects one
by one. Binary search is a sophisticated search and is faster than linear
search.
We use merging algorithms to combine sets of elements to form a single set
according to some criteria. Programming uses merge sort which is a basic
algorithm.
Algorithms that manipulate strings include those that arrange text data into
lines and paragraphs or search for occurrences of a given pattern in a
document.
Science and engineering field have a wide range of applications that use
algorithms. Basic algorithms for mathematical computation include those for
generating random numbers, performing operations on matrices, solving
simultaneous equations, and numerical integration.
Activity 1
Write a sorting algorithm, its psuedocode and the program in any
language.
4. To analyze an algorithm is to determine the amount of ________
necessary to execute it.
5. __________ algorithms are used to combine to sets of elements to form
a single set according to some criteria.
6. ___________ is the expression of an algorithm in a programming
language with all the language specific codes.

1.4 Fundamentals of Algorithm
Now that we have studied the role of algorithm in computing, let us discuss
the some fundamentals of algorithms.
Algorithms are widely used in various areas of study. We can solve different
problems using the same algorithm. Therefore, all the algorithms must
follow a common standard. In this section we discuss these standards.
1.4.1 Asymptotic notations
A problem may have various algorithmic solutions. In order to choose the
best algorithm for a particular process, you need to be able to judge how
much time a particular solution will take to run. More accurately, you need to
be able to judge how much time two solutions will take to run, and choose
the better of the two.
Asymptotic complexity expresses the main component of cost of the
algorithm. Consider the algorithm for sorting a deck of cards, which
proceeds by repeatedly searching through the deck for the lowest card. The
actual time to perform search and sort each card depends upon the
processor speed. Therefore, we use the below given asymptotic notations to
make the task easier.
Big-O notation Big-O is the method of expressing the upper bound of the
running time of an algorithm. It is a measure of the longest amount of time it
could possibly take for the algorithm to complete.
For two non-negative functions, f(n) and g(n), if there exists an integer n
0

and a constant c > 0 such that for all integers n > n
0
, f(n) cg(n), then f(n) is
Big O of g(n). This is denoted as "f(n) = O(g(n))". If graphed, g(n) serves as
an upper bound to the curve you are analyzing, f(n).
Big-Omega notation It describes the best that can happen for a given
data size. This has the same definition as Big-O, except that "f(n) cg(n)",
this makes g(n) a lower bound function, instead of an upper bound function.
For non-negative functions, f(n) and g(n), if there exists an integer n
0
and a
constant c > 0 such that for all integers n > n
0
, f(n) cg(n), then f(n) is
omega of g(n). We denote this as "f(n) = (g(n))".
Theta notation For non-negative functions, f(n) and g(n), f(n) is theta of
g(n) if and only if f(n) = O(g(n)) and f(n) = (g(n)). We denote this as
"f(n) = (g(n))". This implies that the function, f(n) is bounded both from the
top and bottom by the same function, g(n).
Little-O notation For non-negative functions, f(n) and g(n), f(n) is little o of
g(n) if and only if f(n) = O(g(n)), but f(n) (g(n)). We denote this as
"f(n) = o(g(n))". This represents a loose bounding version of Big O. g(n)
bounds from the top, but it does not bound the bottom.
Little Omega notation For non-negative functions, f(n) and g(n), f(n) is
little omega of g(n) if and only if f(n) = (g(n)), but f(n) (g(n)). We denote
this "f(n) = (g(n))".
1.4.2 Time and space complexity and efficiency
The efficiency of algorithms strongly depends on the time and space
complexity. Let us see how it affects the algorithm.
Time complexity It is a function that describes the time of execution of an
algorithm based on its input parameters. In complex algorithms it is difficult
to find the exact time taken. Therefore we find the order of the functions
expressed in Big-O notation using the asymptotic notations. Complexity
classes are equivalence classes of time complexities which are equal in
Big-O notation. There are meta-complexity classes of time complexities
which have Big-O notations that differ only by some specific parameter. For
instance, O(n2) and O(n3) are both polynomial time complexity classes,
similarly O(2n) and O(3n) are exponential time complexity classes.
The time T(p) taken by a program P is the sum of the compile time and the
runtime. The compile time does not depend on the instance characteristics.
Also we may assume that a compiled program will run several times without
recompilation. So we concern ourselves with just the runtime of a program.
This runtime is denoted by t
p
. Therefore time complexity can also be known
as the number of instructions which a program executes during its run time.
Space complexity Space complexity is defined as the amount of memory
that an algorithm needs. A good algorithm minimizes the memory needed.
The space needed by algorithm is the sum of the following components:
A fixed part that is independent of the characteristics of the input and
outputs. This part typically includes the instruction space, space for
simple variables and fixed size component variables, space for constant
and so on.
A variable part that consist of the space needed by components
variables whose size is dependent on the particular problems instance
being solved, the space needed by referenced variables and the
recursion stack space.
The space requirement S(p) of any algorithm P may therefore be written as
S(p) = C+ Sp(instance characteristics), where C is a Constant.
7. An algorithm that invokes itself within the process is called _______.
8. ______ is the method of expressing the upper bound of an algorithm's
running time.
9. _____________ is defined as the number of memory cells which an
algorithm needs.

1.5 Important Types of Algorithm
Now that we have an idea about the different notations and complexity of
algorithms, let us study some commonly known classification of algorithms.
We group algorithms that use similar problem-solving techniques together.
Let us now discuss the different types of algorithms.
1.5.1 Simple recursive
A simple recursive algorithm solves the basic problems using recursion. It
converts the solution to a simpler sub problem to arrive at the correct
solution. It is called simple because several of the other algorithm types are
inherently recursive.
Example of a recursive algorithm:
To count the number of elements in a list:
1) If the list is empty, return zero; otherwise go to step 2.
2) Leave the first element and go to step 3.
3) Count the element remaining and add one to the result.
4) Start again from step 1.
1.5.2 Backtracking
Backtracking algorithms are based on a depth-first recursive search. It can
be applied only for problems which have the concept of a partial candidate
solution and a quick test of whether it can possibly be completed to a valid
solution.
It first tests to see if a solution has been found, and if so, returns it
otherwise;
1) For each choice that can be made at this point, make that choice
2) Recur
If the recursion returns a solution, return it
If no choices remain, return failure
Example of backtracking algorithm:
Backtracking is used for the eight queens puzzle, that gives all
arrangements of eight queens on a standard chessboard so that no queen
attacks any other. In the common backtracking problems, the partial
candidates are arrangements of k queens in the first k rows of the board, all
in different rows and columns. We can abandon any partial solution that
contains two mutually attacking queens since it cannot possibly be
completed to a valid solution.
1.5.3 Divide and conquer
We use divide and conquer algorithms when there are two or more
recursive calls. A divide and conquer algorithm consists of two parts:
Divide the problem into smaller sub parts of the same type, and solve
these sub parts recursively.
Combine the solutions to the sub parts into a solution to the original
problem.
Examples of divide and conquer algorithm:
Quick sort:
1) Split the array into two parts, and quick sort each of the parts
2) No additional work is required to combine the two sorted parts
Merge sort:
1) Cut the array in half, and merge sort each half
2) Combine the two sorted arrays into a single sorted array by merging
them
1.5.4 Dynamic programming
A dynamic programming algorithm stores the previous results and uses
them to find new ones. We generally use dynamic programming for
optimization problems. It finds the best solution from the possible ones.
Dynamic programming algorithms require optimal substructure and
overlapping sub problems:
Optimal substructure: Optimal solution contains optimal solutions to
sub problems.
Overlapping sub problems: Overlapping sub problems are solutions to
sub problems that can be stored and reused in a bottom-up manner.
This differs from divide and conquer algorithm, where sub problems
generally need not overlap.
Example of dynamic programming algorithm:
The steps required to find the n
th
Fibonacci number is given as follows:
1) If n is zero or one, return one; otherwise,
2) Compute or look up in a table, Fibonacci (n-1) and Fibonacci (n-2)
3) Find the sum of these two numbers
4) Store the result in a table and return it
Since finding the n
th
Fibonacci number involves finding all smaller Fibonacci
numbers, the second recursive call has little work to do and the solutions
thus formed may be preserved and used again later.
1.5.5 Greedy algorithms
An optimization problem is one in which you want to find, not just a solution,
but the best solution. It works by making the decision that seems most
promising at any moment. It never reconsiders this decision, irrespective of
the situation that arises later. It works to achieve a local optimum at each
step and finally finds the global optimum.
Example of greedy algorithm:
Counting money: Suppose you want to count out a certain amount of
money, using the fewest possible notes and coins. A greedy algorithm that
would do this would be:
At each step, take the largest possible note or coin.
Example: To make Rs.6.39, you can choose:
1) a Rs. 5 note
2) a Rs.1 note , to make Rs. 6
3) a 25p coin, to make Rs. 6.25
4) A 10p coin, to make Rs. 6.35
5) four 1p coins, to make Rs. 6.39
Here, the greedy algorithm always gives the optimum solution.
1.5.6 Branch and bound algorithms
We use branch and bound algorithms generally for optimization problems.
We form a tree of sub problems in branch and bound algorithm. We
consider the original problem to be the root problem. We use a method to
construct an upper and lower bound for a given problem.
At each node, apply the bounding methods:
If the bounds match, it is taken as a feasible solution to that particular sub
problem.
If bounds do not match, split the problem represented by that node, and
make the two sub problems into children nodes.
Continue, using the best known feasible solution to solve every other node
of the tree.
Example of branch and bound algorithm:
Travelling salesman problem: A salesman has to visit each of n cities once
and wants to minimize the total distance traveled:
Consider the root problem to be the problem of finding the shortest route
through a set of cities visiting each city once
Split the node into two child problems:
1) Shortest route visiting city A first
2) Shortest route not visiting city A first
Continue subdividing similarly as the tree grows
1.5.7 Brute force algorithms
A brute force algorithm simply tries all possibilities until a satisfactory
solution is found. Brute force algorithm is:
Optimizing: It finds the best solution. This may require finding all
solutions. But, if a value for the best solution is known, it may stop when
any best solution is found. Finding the best path for a travelling
salesman is an example.
Satisfying: It stops as soon as a solution is found that is good enough.
Finding a travelling salesman path that is within 20% of optimal is an
example.
Often, brute force algorithms take exponential time to find a solution. We
can use various heuristics and optimizations to improve brute force
algorithm.
Heuristic: A thumb rule that helps you decide which possibilities to look
at first.
Optimization: A way to eliminate certain possibilities without fully
exploring them.
1.5.8 Randomized algorithms
A randomized algorithm or a probabilistic algorithm uses a random number
at least once during the computation to make a decision. It employs a
degree of randomness as part of its logic.
Examples of randomized algorithm are:
Quick sort uses a random number to choose a pivot
Trying to factor a large prime by choosing random numbers as possible
divisors
10. A ____________ algorithm converts the solution to a simpler sub
problem to arrive at the correct solution.
11. A ______________ algorithm simply tries all possibilities until a
satisfactory solution is found.
12. ___________ algorithms are based on a depth-first recursive search.

1.6 Fundamental Data Structures
Now that we have seen the classification of algorithms, let us study the
fundamentals of data structures. We require various data structures while
programming. Every data structure that we choose for a particular program
depends on the basic objective of the program.
1.6.1 Data structure definition
A data structure is a set of data elements grouped together under one
name. It means a set of rules that hold the data together. The data elements
are also known as members and can have different types and different
lengths. Data structures can have other data structures. This is known as
nesting of data structures. Modern programming languages allow the
programmer to create user-defined data structures.
1.6.2 Use of data structure
Almost all computer programs use data structures. Data structures are an
essential part of algorithms. We can use it to manage huge amount of data
in large databases. Some modern programming language emphasis more
on data structures than algorithms. We use data structures such as priority
queue, binary heaps, and binomial heaps to build complex algorithms.
1.6.3 Data structure classification
There are many data structures that help us to manipulate the data stored in
the memory. Let us study the different data structure classifications in this
section.
Primitive and Non-primitive Data Structures Primitive data structures
are the basic data structures that can be used directly in machine
instructions. Examples of primitive data structures are integer, character,
Boolean, string, double, float. Non primitive data structures are the data
structures that are derived from primitive data structures. Examples of non
primitive data structures are array, stack, queue, list etc.
Array It is a set of same data elements grouped together. Arrays can be
one dimensional or multi dimensional. We store elements in an array in
continuous locations and identify the array using the location of the first
element of the array.
Stack It is a last in first out (LIFO) data structure. The fundamental
operations performed on a stack are push and pop. Push is used to insert
an element into the stack and pop is used to retrieve an element from the
stack. The stack operations are shown in figure 1.1.

Figure 1.1: Stack Operations
Queue It is a first in first out (FIFO) data structure. The basic operations
on a queue are enqueue and dequeue. Enqueue is used for insertion and
dequeue is used for retrieval of elements
List It is a sequence of data elements connected to each other where
every element has a link field which refers to the location of the next
element. Different types of lists are linked list, doubly linked lists, skip list,
jump list etc.
Tree It is a hierarchical representation of elements as nodes linked with
each other. The fist element of the tree structure is known as the root. Each
node has a sub node known as child node. The node without a child node is
known as leaf node. Binary tree is a type tree structure.
Homogeneous and heterogeneous data structures Non primitive data
structures can be homogenous or heterogeneous. Homogeneous data
structure has all the elements of same data type in it. Example of a
homogeneous data structure is an array. Heterogeneous data structure
contains different types of data elements in it. Example of a heterogeneous
data structure is a record.
Static and dynamic data structures We classify data structures as static
and dynamic considering the time of memory allocation. The system
allocates memory for static data structure at the compile time itself (design
time allocation). For a dynamic data structure memory is allocated during
run time (through functions such as calloc(), malloc()).
Linear and non-linear data structures We classify data structures as
linear and non linear based on the type of relationship between its elements.
Linear data structure such as arrays and lists maintains a constant and
direct relationship between its elements. Non-linear data structures such as
a tree do not maintain a constant and direct relationship between their
elements.
1.6.4 Data structure characteristics
Choosing the best data structure for a program is a challenging task. Same
task may have suitable different data structures for it. We derive new data
structures for complex tasks using the already existing ones. We need to
compare the characteristics before choosing the right data structure. The
table 1.1 shows the characteristics of the commonly used data structures.
Table 1.1: Characteristics of Data Structures

Activity 2
Write a program to sort two arrays and identify the data structures used
in it.
13. A ______ is a set of data elements grouped together under one name.
14. _________ data structure has all the elements of same data type in it.
15. For ____________ data structure, the memory is allocated at the
compile time itself.

1.7 Summary
Let us summarize what we have discussed in this unit.
An algorithm is defined as a set of instructions describing the sequence of
actions to achieve the result of the problem with limited resources. The word
algorithm is derived from the name of Abu Abdullah Muhammad ibn Musa
al-Khwarizmi, a 9th-century Persian mathematician.
Data structure Advantages Disadvantages
Array Easy to insert elements
Fast access of elements
if the index is known
Fixed size
Slow search
Slow deletion of elements
Stack Last-in, first-out access
Good for simple
programming
Only small number of operations
can be performed
No random access of elements
possible
Queue First-in, first-out access
Easy to insert
East to delete
Memory consumption is more
than stack
List Easy to insert
East to delete
Slow search
Tree Quick search
Quick inserts
Quick deletes
Deletion algorithm is complex

Algorithm has a major role in computing. Analysis of algorithm means a
detailed study of the specification of the algorithm and arriving at
conclusions about how the implementation of that algorithm will perform in
general. Designing an algorithm is a process that actually implements the
algorithm in order to get the desired results. Properties of algorithm are
correctness, definiteness, finiteness and effectiveness. An algorithm differs
from a program and a psuedocode. Different types of algorithms are simple
recursive, backtracking, divide and conquer, dynamic programming, greedy
algorithm, branch and bound algorithm, brute force algorithm, randomized
algorithm.
Data structures are a set of data elements grouped together under one
name. It is classified as primitive and non primitive, homogeneous and
heterogeneous, static and dynamic, linear and non linear. We study the
characteristics of data structures in order to choose the best from them.

1.8 Glossary
Terms Description
Subroutine A program unit which does not return any value through
its name and has a number of arguments.
Optimization Problem Optimization problem is a computational problem which
has the objective to find the best of all possible solutions.

1. What are the properties of an algorithm?
2. What are asymptotic notations?
3. Explain greedy algorithm with a suitable example.
4. How are data structures classified?
5. Compare the characteristics of different types of data structures.

1.10 Answers
1. Algorism
2. Speed, size, resources
3. Good programming
4. Resources
5. Merging
6. Programs
7. Direct recursive
8. Big O
9. Space complexity
10. Simple recursive
11. Brute force
12. Backtracking
13. Data structure
14. Homogeneous
15. Static
Terminal Questions
1. Refer section 1.3.2 Properties of algorithms
2. Refer section 1.4.1 Asymptotic notations
3. Refer section 1.5.5 Greedy algorithm
4. Refer section 1.6.3 Data structure classification
5. Refer section 1.6.4 Data structure characteristics

References
Cormann, Thomas (1990). Introduction to algorithms. The MIT press,
Cambridge
Levitin, A. (2003). Introduction to the analysis and design of algorithms.
Dorling Kindersley, India
Wirth, N. (1991). Algorithms + Data Structures = Programs. Jay Print
Pack publications, New Delhi

E-References
http://www.brpreiss.com
http://www.esirav.info/definitions of algorithms, By Reality Software,
copyright 2000
http://www.stackoverflow.com/stack overflow internet services.
Unit 2 Fundamentals of the Analysis of
Algorithm Efficiency
Structure:
2.1 Introduction
Objectives
2.2 Algorithm Analysis Framework
Complexity of algorithm
Measuring an input size
Measuring running time
Best case, worst case and average case analysis
2.3 Methodologies for Analyzing Algorithms
Pseudocode
Random access machine (RAM) model
Counting primitive operations
Analyzing recursive algorithms
2.4 Amortization
Amortization techniques
Analyzing an extendable array implementation
2.5 Case Studies in Algorithm Analysis
A quadratic time prefix averages algorithm
A linear time prefix averages algorithm
2.6 Summary
2.7 Glossary
2.9 Answers
2.1 Introduction
In the previous unit we studied about the basics of algorithm and data
structures. In this unit we will discuss analysis of algorithm efficiency.
Algorithm analysis is an important part of computational complexity theory. It
provides theoretical estimates for the resources needed for any algorithm to
solve a given problem. These estimates help to provide an insight regarding
the measures that determine algorithm efficiency.
This unit covers algorithm analysis framework with respect to complexity of
an algorithm. It gives an idea about various methods used for analyzing
algorithms and explains the techniques involved in amortization. It also
includes some case studies in algorithm analysis.
Objectives:
After studying this unit you should be able to:
define algorithm analysis framework
explain different methods for analyzing the algorithms
describe amortization analysis in algorithm
2.2 Algorithm Analysis Framework
Algorithm analysis framework involves finding out the time taken by a
program and the memory space it requires to execute. It also determines
how the input size of a program influences the running time of the program.
2.2.1 Complexity of algorithm
We can determine the efficiency of an algorithm by calculating its
performance. The two factors that help us to determine the efficiency of an
algorithm are:
Amount of time required by an algorithm to execute
Amount of space required by an algorithm to execute
These are generally known as time complexity and space complexity of an
algorithm.
Time complexity
Time complexity of an algorithm is the amount of time required for it to
execute.
The time taken by an algorithm is given as the sum of the compile time and
the execution time. The compile time does not depend on the instance
characteristics as a program once complied can be run many times without
recompiling. So only the run time of the program matters while calculating
time complexity and it is denoted by t
p
(instance characteristics).
It is difficult to calculate the time complexity in terms of physically clocked
time.
For example in multi-user operating system, it depends on various factors
such as:
System load
Number of programs running on the system
Instruction set used
Speed of the hardware
The time complexity of an algorithm is given in terms of frequency counts.
Frequency count is the count that indicates the number of times the
statement is executed.
Space complexity
Space complexity of an algorithm is the amount of storage required for it to
execute.
The space required by an algorithm to execute is given as the sum of the
following components:
1) A fixed part that is independent of the characteristics of inputs and
output. This part includes instruction space (for example space for the
code), space for simple variables and fixed size component variables
and space for constants.
2) A variable part that consists of the space needed by the component
variables whose size is dependent on a particular problem instance
being solved and the space needed by the referenced variable.
Therefore to calculate the space complexity of an algorithm we have to
consider two factors:
Constant characteristic
Instance characteristic
Equation 2.1 depicts the space requirement S(p) of an algorithm.
S(p) = C + S
p
Eq: 2.1
Where C is the constant part and indicates the space required for inputs and
outputs which includes instructions, variables and identifiers.
S
p
defines the space required for an instance characteristic. This is a
variable part whose space requirement depends on a particular problem.
2.2.2 Measuring input size
The time required to execute an algorithm depends on the input size of the
algorithm. If the input size is longer, then the time taken to execute it is
more. Therefore we can calculate the efficiency of an algorithm as a function
to which the input size is passed as a parameter.
Sometimes to implement an algorithm we need prior information of the input
size.
For example when performing multiplication of two matrices we should know
the order of these matrices, only then we can enter the elements of the
matrices.
2.2.3 Measuring running time
The time measured for analyzing an algorithm is generally called as running
time.
For measuring the running time of an algorithm we consider the following:
1. First recognise the basic operation (the operation contributing the most
to total run time) of an algorithm.
2. Identifying the basic operation of an algorithm is not that difficult. It is
generally the most time consuming operation in the algorithm. Normally
such operations are located in the inner loop of an algorithm.
For example, the basic operation of sorting algorithms is to compare the
elements and place them in appropriate position.
The table 2.1 gives an example of the concept of basic operation.
Table 2.1: Basic Operations for Input Size
Problem statement Input size Basic operation
Computing GCD (greatest
common divisor) of two
numbers.
Two numbers Division
Searching a key element
from the list of n elements.
List of n elements. Comparison of key
element with every
element of the list.
Performing matrix
multiplication.
The two matrices with
order n n
Multiplication of the
elements in the matrices.
Then calculate the total time taken by the basic operation using the following
formula:
T(n) = C
op
C(n)
Where T(n) is the running time of basic operation
C
op
is the time taken by the basic operation to execute
C(n) is the number of time the operation needs to be executed
Using this formula we can obtain the approximate computing time.
2.2.4 Best case, worst case and average case analysis
Best Case: If an algorithm takes the least amount of time to execute a
specific set of input, then it is called best case time complexity.
Worst Case: If an algorithm takes maximum amount of time to execute a
specific set of input, then it is called the worst case time complexity.
Average Case: If the time complexity of an algorithm for certain sets of
inputs is on an average same then such a time complexity is called average
case time complexity. The average case time complexity is not the just the
average of best case and worst case time complexities.
Let us now consider the algorithm for sequential search and find its best,
worst and average case time complexities.
Algorithm for sequential search
Algorithm Seq_search{H [0n-1], key}
// Problem description: This algorithm searches the key elements for an
//array H [0n-1] sequentially.
// Input: An array H [0n-1] and search the key element
// Output: Returns the index of H where the key element is present
for p = 0 to n -1 do
if {H [p] = key} then
return p
Let us now trace the sequential search algorithm.
Algorithm tracing for sequential search algorithm
//Let us consider n=4, H[ ] = {10, 14, 18, 20}, key = 14
for p = 0 to 4 -1 do // this loop iterates from p = 0 to 4-1
if {H [0] = key} then// the loop continues to iterate as H[0] is not the
search
//element
return 1// finally the array retutns1 which is the position of the key
element.

Best case time complexity
The above searching algorithm searches the element key from the list of n
elements of the array H [0n-1]. If the element key is present at the first
location of the list (H [0n-1]) then the time taken to execute the algorithm
is the least .The time complexity depends on the number of times the basic
operation is executed. Thus we get the best case time complexity when the
number of basic operations is minimum. If the element to be searched is
found at the first position, then the basic operation is only one and the best
case complexity is achieved. Equation 2.2 denotes the best case time
complexity as:
C
best
= 1 Eq: 2.2
Worst case time complexity
In the above searching algorithm if the search element key is present at the
nth position (the last position) of the list then the basic operations and time
required to execute the algorithm is more and thus it gives the worst case
time complexity. Equation 2.3 gives the worst case time complexity as:
C
worst
= n Eq: 2.3
Average case time complexity
This complexity gives information about an algorithm on specific input. For
instance in the sequential search algorithm:
Let the probability of getting successful search be P.
The total number of elements in the list is n.
If we find the first search element at the i
th
position then the probability of
occurrence of the first search element is P/n for every ith element.
(1 - P) is the probability of getting unsuccessful search.
Therefore the average case time complexity C
avg
(n) is given as:
C
avg
(n) = Probability of successful search + Probability of unsuccessful
search
C
avg
(n) =
2
) n 1 ( P
+ n (1 - P) Eq: 2.4
Equation 2.4 gives the general formula for computing average case time
complexity.
If the search is complete and the search element is not found, then
P = 0 which means that there is no successful search in such
circumstances.
C
avg
(n) = 0(1 + n)/2 + n (1 + 0)
C
avg
(n) = n Eq: 2.5
From the equation 2.5 we see that the average case time complexity C
avg
(n)
will be equal to n.
Thus calculating average case time complexity depends on the probability of
getting a successful search and the number of operations required to
perform the search. Therefore calculating average case time complexity is
difficult compared to calculating worst case and best case complexities.
Activity 1
Write a searching algorithm, and find its best, worst and average case
time complexity.
1. The ________ of an algorithm can be determined by calculating its
performance.
2. ___________ of an algorithm is the amount of time required by an
algorithm to execute.
3. If an algorithm takes least amount of time to execute a specific set of
input then it is called __________ time complexity.

2.3 Methodologies for Analyzing Algorithms
In the previous section you have studied the analysis framework, time and
space complexities, and tradeoff in algorithms. We also discussed
measuring input size, running time of an algorithm, and best, worst and
average case analysis. In this section we will study the methodologies used
for analyzing the algorithms.
There are different methodologies for analyzing the algorithms. Following
are some of the methodologies used for analyzing the algorithm:
Pseudocode
Random access machine model
Counting primitive operations
Analyzing recursive algorithms
Testing and measuring over a range of instances
2.3.1 Pseudocode
It is a compact and informal high level explanation of computer algorithms
that uses the structural principles of a programming language. It is meant for
human reading rather than machine reading.
It excludes some details from the algorithms which are not needed for
human understanding such as variable declaration, system specific code
and subroutines. The objective of using pseudocode is to make the
programming language code easier for the human to understand.
Example for pseudocode
Pseudocode for selection sort
Selection sort (B)
for p 1 to n-1 do
min p q;
min x B[p]
for j m + 1 to n do
If B[q] < min x then
min q q
min x B[q]
B[min q] B [p]
B[p] min x
2.3.2 Random access machine (RAM) model
We can use the analytical approach directly on the high level code or
pseudocode in order to analyze an algorithm without experimenting its
running time.
The method of calculating primitive operations produces a computational
model called Random Access Machine. This model views the computer as a
CPU connected to a depository of memory cells. The memory cells stores
words, which may be a character string, an address, that is, the value of a
basic data type or a number. The word random access indicates the CPUs
capability to access an arbitrary memory cell with a primitive operation.
There is no limit on the size of data that can be stored in the memory. The
CPU in the RAM model performs a primitive operation in a number of steps
independent of the input size. Therefore the number of primitive operations
an algorithm performs corresponds directly to the running time of the
algorithm.
2.3.3 Counting primitive operations
While analyzing the algorithm we have to count the number of primitive
operations that the algorithm executes. Examples of some of the primitive
operations are as follows:
Calling a method
Returning a value from a function
Performing some arithmetic operations like addition, subtraction etc.
Assigning a value to a variable
Comparing two variables
Reference to the pointer
Indexing an array
Counting primitive operations describes how to count the maximum number
of primitive operations an algorithm executes.
Let us now consider the arrayMax algorithm which helps in finding the
maximum number of primitive operations.
Algorithm Factorial
Algorithm factorial(x)
//Input x
//F is the factorial of the given number x
Fact(x)
{
If x = 1
Return 1
Else
F= x*fact(x-1)// recursive function which stops when value of x =1
}

Let us now trace the factorial algorithm
Algorithm tracing for factorial algorithm
Let x = 5
Fact(5)
{
If x = 1// x is not equal to 1 so the else part gets executed
Return 1
Else
F= 5*fact(5-1)// recursive function which stops when value of x =1
// finally the algorithm returns 120 as the factorial of the number 5
}
Analysis of factorial algorithm
For analyzing this algorithm and computing the primitive operation we have
to consider some operations that are repeated as they are enclosed in the
body of the loop.
First getting the value for x corresponds to one primitive operation.
This step executes only once at the beginning of the algorithm and
contributes one unit to the total count.
In the function Fact(x), there are two primitive operations.
To check if the value of x is 1
To multiply x and Fact(x-1)
The function recurs these two operations x number of times. Therefore it
contributes a 2x value to the total count.
Hence the total number of primitive operations T(n) that the factorial
algorithm executes is given in equation 2.6.
T(n) = 1+2x Eq. 2.6
2.3.4 Analyzing recursive algorithms
Problem solving not only uses the method of iteration but also the method of
recursion. In this technique, a procedure P is defined, which makes calls to
itself, provided, as a function the calls to P are for solving sub problems.
Recursive calls are the function calls to P in smaller instances. Recursive
procedure should define a base case which is small enough to solve without
using recursion.
For example let us consider the recursiveMax algorithm:
Algorithm recursiveMax
Algorithm recursiveMax (B, n)
//Input: An array B sorting n 1 integer
//Output: The maximum element in B
//max() returns the maximum value
If n = 1 then
return B[0]
return max (recrusiveMax(B, n-1), B[n-1])// max function returns the
//maximum value
Let us now trace the recursiveMax algorithm
Algorithm tracing for recursiveMax algorithm
Let us assume n=2, B[ ] = {1, 4}
B is an array for sorting 2 1 integers
//max() returns the maximum value
If n= 1 then
return B[0]
return max (recrusiveMax(B, 2-1), B[2-1])// max (recrusiveMax(B, 1), 4)
//max(1, 4)
return 4
Analysis of recursiveMax algorithm
Here the algorithm checks if the array contains the maximum item, therefore
the simple base case can immediately solve the problem. Otherwise, the
algorithm first recursively calculates the maximum of n-1 elements in the
array and then returns the maximum value and the last element of the array.
Analyzing the running time of a recursive algorithm is a bit difficult and
requires the use of recurrence equation. The function T(n) denotes the
running time of an algorithm based on the input size n. Equation 2.7
describes the running time of the recursiveMax algorithm.
7 ) 1 (
3
) (
n T
n T
Eq: 2.7
We can count each comparison, array reference, recursive call and max
calculation as one primitive operation. The closed form of the recurrence
equation is where there is no reference to the function T on the right hand
side. Equation 2.8 gives the closed form for the recursiveMax algorithm
T
(n)
= 7(n-1) + 3 = 7n-2. Eq: 2.8
4. The method of calculating primitive operations produces a
computational model called _______________.
5. _________ describes how to count the maximum number of primitive
operations an algorithm executes.
6. Recursive procedure should define a ________ which is small enough
to solve without using recursion.

2.4 Amortization
In the previous section we have studied the different methodologies that
analyze algorithms. In this section we will study about amortization and its
techniques.
Amortization is an important tool used to find the average running time of an
algorithm. Amortization analysis assures the performance of each operation
in the worst case. In this section we will see some of the techniques of
amortization analysis and an extendable array implementation.
2.4.1 Amortization techniques
The following are the two fundamental techniques to perform an
amortization analysis:
Accounting method
Potential function method

Accounting Method
The accounting method explains the amortized cost for any particular
operation. This technique performs an amortized analysis based on a
financial model. According to this method if the amortized cost of any given
operation is greater than its actual cost, then this difference in cost is
considered to be as credit and if the amortized cost is lesser than the actual
cost, the difference in cost is considered to be debit. The credits that are
gathered are used for those operations whose amortized cost is less than its
actual cost. The sum of amortized cost for any given series of operations
gives the upper bound of the sum of the actual cost of that series.
For example of accounting method let us consider the table expansion:
We usually create a table without knowing the actual space required for it.
We may use a method where the size of the table is doubled when it is filled
up. With this example we are trying to show that the amortized cost of an
operation to insert an element into the table is O(1).
Consider the following: TB = Table, P = Element to be inserted, no(TB)=
number of elements in table TB, size(TB) = Size allocated for TB.
Assume the following operations: create_table(x) to create a table of size x,
insert_element(TB, P) to insert element P into table TB with cost =1.
Let us see the pseudocode for the insertion of an element into the table.
Pseudocode for Element Insertion
function insert_element(TB,P)
if no(TB) = size(TB)
U = create_table(2 size(TB))
for each F in T
insert_element(U,F)
T = U
insert_element(TB,P)
The for loop in the pseudocode performs no(TB) insertions. Therefore, the
best bound is given as O(n
2
) without using amortized analysis. For the
accounting method, let us assign a cost of 3 to each insertion.
The first time insertion of element will cost (3 - 1)*x =2x, where x is the size
of the empty table TB. To insert an element x+1 we need reallocation. The
for loop requires x insertions with cost = x each. The last step insertion
requires cost = x+1.
Therefore the total cost of all operations = 2x + 3 (x + 1) = x + 2.
Now let us insert x 1elements into the table TB. Here we can compute the
cost as x + 2 + 2 * (x 1) = 3x.
Next let us try to insert an element 2x + 1 into the table TB. The cost will be
3x + 3 (2x + 1) = x + 2. This cost is same as that after inserting x + 1.
We choose 3 as the cost for the following three operations:
1. To insert element into the table TB for the first time
2. To move it when we expand the table
3. To move the element already present in the table while the table is
expanded.
Therefore, the cost of creating a table of size x is given as O(x). When we
create a new table the size will be 2x.The cost of inserting elements in the
table is calculated from the last
2
m

entries to pay the cost 2m. We must
consider
4
2 / m
m 2
for each entry which adds up to 3+4=7.

Potential Function Method
This technique performs an amortized analysis based on an energy model.
In this technique, a value , which represents the current energy state of
the system, is associated with the structure. Every operation performed
contributes to some additional amount, known as amortized time, to , and
also extracts value from proportionate to the actual time spent.
Formally, n 0 denotes the initial value of before an operation is
performed and i denotes the value of the potential function, , after the ith
operation is performed.
The reason for using the potential function argument is to use the change in
potential algorithm for the ith operation, i i-1, to characterise the
amortized time required for that operation.
The amortized running time of the i
th
operation is defined as
ti = ti + i i-1 where ti denotes the actual running time.
The amortized cost of the i
th
operation is the actual running time plus the net
change in potential that operation causes (which can be positive or
negative) or the actual time spent is the amortized cost plus the net drop in
potential, that is, ti = ti + i-1 i.
T denotes the total amortized time for performing n operations on the
structure as given in equation Eq: 2.9.
Eq: 2.9
The total actual time, T, taken by the operations is given by the equation
2.10.

= T +

= T + Eq: 2.10
Thus, the total actual time spent is equal to the total amortized time plus the
net drop in potential over the total sequence of operations. Therefore, if
n o, then T T, the actual time spent is not more than the amortized
time.
To make this concept clear, consider the stack operations PUSH, POP and
MULTIPOP. Let us define the potential function to the number of elements
in the stack.
When the stack is empty E
0
, the potential function (E
0
) = 0. The stack
element count is not negative, therefore the i
th
operation of stack always has
a non negative potential. Therefore we form the equation Eq: 2.11.
(Ei) 0 Eq: 2.11
= (E0).
We an compute the total cost of n operations with which represents an
upper bound on actual cost.
Let us now compute the amortized costs of the various stack operations. If
the ith operation on a stack containing p objects is a PUSH operation, then
the potential difference is given in equation Eq: 2.12.
(Ei) - (Ei-1) = (p + 1) p Eq: 2.12
= 1
We can calculate the amortized cost of this PUSH operation as in equation
Eq: 2.13.
ci= ci + (Di) - (Di-1) Eq: 2.13
= 1 + 1
= 2
Similarly we can calculate the amortized cost of POP and MULTI POP to be
0. The amortized cost of each operation is O(1), and the total amortized
cost of a sequence of n operations is O(n). From equation Eq: 2.11, we
know that (E
i
) (E
0
) therefore, the total amortized cost of n operations is
an upper bound on the total cost. The worst-case cost of n operations is
O(n).
2.4.2 Analyzing an extendable array implementation
The limitation of the single array implementation is that it needs advanced
specification of a fixed capacity N for the total number of elements that are
stored in the table. If the actual number of elements in the table is n, is
smaller than N, then this implementation will waste space. If n increases
beyond N, then the implementation crashes. So let us provide a means to
increase the size of the array A to store the elements of table S.
In programming languages such as C, C++ and Java it is not possible to
increase the size of the array as the capacity of it is fixed for some number
N. If there is an overflow in the elements of an array then we can follow the
below steps:
Allocate a new array B of capacity 2N.
Copy A[i] to B[i]
Let A = B, that is, we use the array B supporting S.
This kind of array replacement method is known as an extendable array
implementation. After the array replacement, the new array allows to add n
new elements to the table before the array must be replaced again.
Therefore the running time of a series of operation performed on an empty
extendable array is quite efficient.
7. __________ technique is used to perform an amortized analysis
method based on a financial model.
8. If you can setup such a scheme called amortization scheme then each
operation in the series has an ______________.
9. _____________ technique is used to perform an amortized analysis
method based on an energy model.

2.5 Case Studies in Algorithm Analysis
In the previous section we have studied amortization and its techniques and
analyzing an extendable array implementation. In this section we will study
some cases of algorithm analysis which help us to have a better
understanding of the process.
2.5.1 A quadratic time prefix averages algorithm
First let us consider the prefix averages problem called prefixAvg1. It
calculates all elements of an array separately and gives the average of the
elements in the array M.

Algorithm prefixAvg1 for quadratic time prefixAverages algorithm
Algorithm prefixAvg1(M)
// Input: a n element array M of numbers
// Output: A n element array N of numbers such that N[p] is the average
of //elements M[0],.,M[i].
//Let N be an array of n elements
for p = 0 to n-1 do
b = 0
for q = 0 to p do
b = b + M[p]
N[p] = a / (p + 1)
return array N
Let us now trace the prefixAvg1 algorithm
Algorithm tracing for prefixAvg1 algorithm
Let us consider n=4, M[ ] = {2, 4, 6, 8}
//M is an array of 4 numbers, N is an output array
for p = 0 to 4 -1 do // this loop iterates for from p = 0 to 4 - 1
b = 0
for q = 0 to p do // inner for loop it iterates for every value of i
b = 0 + X[0]
N[0] = b / (0 + 1)
return array N
Analysis of prefixAvg1 algorithm
Analysis of the algorithm prefixAvg1 helps us to make the following
deductions:
Initiating the array N at the beginning of the algorithm and returning it at
the end is done with a constant number of primitive operations and takes
O(n) time.
In the algorithm, p and q are the counter variables that control the two
nested for loops. The outer for loop which has the counter p executes
for n times, (for p = 0 to n -1). Therefore the statements b = 0 and N[p] =
b/(p-1) also execute n times. This means that the two statements and
the testing and incrementing of counter p corresponds to the number of
primitive operations and takes O(n) time.
The inner for loop which has the counter q executes for p + 1 times
depending on the present value of the counter p of outer for loop.
Therefore the statement b = b + M[p] executes n (n + 1) / 2 times which
means that it corresponds to two primitive operations, incrementing and
testing, and takes O(n
2
) times.
The sum of three terms where the first and the second term are O(n)
and the third term is O(n
2
) gives the running time of the algorithm
prefixAvg1. Therefore the running time of the algorithm prefixAvg1 is
O(n
2
).
2.5.2 A linear time prefix averages algorithm
To calculate prefix averages more efficiently, let us consider two
consecutive averages M[p-1] and M[p] that are similar.
M[p-1] = (N[0] + N[1] + . + N[p-1]) /p
M[p] = (N[0] + N[1] +. + N[p-1] + N[p]) / (p + 1)
If G
p
is the prefix sum of N[0] + N[1] +.+ N[p], we can calculate the
prefix averages as M[p] = G
p
/ (p+1). It is easy to keep track of the current
prefix sum while scanning the array N.
Algorithm prefixAvg2 for linear time prefixAverages algorithm
Algorithm prefixAvg2(N)
// Input: a n element array N of numbers
// Output: A n element array M of numbers such that M[p] is the
average of //elements N[0],.,N[p].
//Let M be an array of n numbers
g = 0
for p = 0 to n-1 do
g = g + N[p]
N[p] = g/(p + 1)
return array M

Let us now trace the prefixAvg2 algorithm
Algorithm tracing for prefixAvg2 algorithm
//Let us consider n=4, g = 0, N[ ] = {2, 5, 4, 7}
//N is an array of 4 numbers, M is an output array
for p = 0 to 4-1 do // this loop iterates for from p = 0 to 4 - 1
g = g + N[0]
M[0] = g/(0 + 1)
return array M
Analysis of prefixAvg2 algorithm
Analysis of the algorithm prefixAvg2 helps us to make the following
deductions:
Initiating the array M at the beginning of the algorithm and returning it at
the end is done with a constant number of primitive operations and takes
O(n) time.
Initiating the variable g at the beginning of the algorithm takes O(1)
times.
The algorithm has one for loop where the counter p executes for n
times, hence the body of the loop executes n times (for p = 0 to n-1).
Therefore the statements g = g + N[p] and N[p] = g/(p + 1)executes n
times each. This means that the two statements and the testing and
incrementing of counter i corresponds to the number of primitive
operations and takes O(n) time.
The sum of three terms where the first and the third term are O(n) and
the second term is O(1) gives the running time of the algorithm
prefixAvg2. Therefore the running time of the algorithm prefixAvg2 is
O(n), which is much better than the running time of quadratic time
algorithm prefixAvg1.
10. The running time of the algorithm prefixAverages1 is _______.
11. The running time of the algorithm prefixAverages2 is _______.
12. In prefixAverages2 algorithm ________ is the time taken for initiating
the variable s.
2.6 Summary
Analysis framework of an algorithm includes the complexities involved in the
executing an algorithm, measuring the input size and running time of an
algorithm and calculating best, worse and average case analysis.
We discussed different methodologies of algorithm analysis like
pseudocode, random access machine model, counting primitive operations,
analyzing recursive algorithms and testing and measuring over a range of
instances.
We analyzed amortization techniques and an extendable array
implementation.
We discussed case studies like a quadratic time prefix averages algorithm
and a linear time prefix averages algorithm which helped us to analyze
algorithms better.
2.7 Glossary
Term Description
Subroutines A program unit which does not return any value through its name
and has a number of arguments
Recursive
algorithm
It is an algorithm which calls itself with smaller inputs and
obtains the inputs for the current input by applying simple
operations to the returned value of the smaller input.
Array An array is a sequence of elements of the similar type placed in
continuous memory locations that can be individually referenced
by adding an index to a single identifier.

1. Explain space complexity.
2. Define best case, worst case and average case complexity.
3. What are the methodologies used for analyzing algorithms?
4. Explain amortization with the help of an example.
5. Explain a quadratic time prefixAverages1 algorithm.

2.9 Answers
1. Efficiency
2. Time complexity
3. Best case
4. Random assess machine
5. Counting primitive operations
6. Base case
7. Accounting method
8. Amortised running time O(a)
9. Potential function method
10. O(n
2
)
11. O(n)
12. O(1)
Terminal Questions
1. Refer section 2.2.1 Complexity of algorithm
2. Refer section 2.2.4 Best case, worst case and average case analysis
3. Refer section 2.3 Methodologies for Analyzing Algorithms
4. Refer section 2.4 Amortization
5. Refer section 2.5.1 A quadratic time prefix averages algorithm
References
Puntambekar, A. A. (2008). Design and Analysis of Algorithms.
Technical Publication, Pune
Cormen, H. Thomas (2003). Introduction to Algorithms. Mc Graw-Hill
Book Company, North America
Michael T. Goodrich., &Roberto Tamassia (2008). Algorithm Design
Foundation, Analysis & Internet Examples. Sanat Printers, Kundli,
Haryana.
E-References
http://www.personal.kent.edu/~rmuhamma/Algorithms/MyAlgorithms/
Sorting/selectionSort.htm

Unit 3 Mathematical Aspects and
Analysis of Algorithms
Structure:
3.1 Introduction
Objectives
3.2 Asymptotic Notations and Basic Efficiency Classes
Asymptotic notations
Basic asymptotic efficiency classes
3.3 Mathematical Analysis of Non-Recursive Algorithms
Analyzing efficiency of non-recursive algorithms
Matrix multiplication
Time efficiency of non-recursive algorithms
Tower of Hanoi puzzle
Conversion of recursion algorithm to non-recursion algorithm
3.4 Summary
3.5 Glossary
3.7 Answers
3.1 Introduction
In the earlier unit you were introduced to the concepts of analysis
framework. In this unit you will be learning about the basic concepts of
mathematical analysis of algorithms.
It is essential to check the efficiency of each algorithm in order to select the
best algorithm. The efficiency is generally measured by calculating the time
complexity of each algorithm. The shorthand way to represent time
complexity is asymptotic notation.
For simplicity, we can classify the algorithms into two categories as:
- Non recursive algorithms
- Recursive algorithms
We compute non-recursive algorithm only once to solve the problem.
In this unit, we will mathematically analyze non-recursive algorithms.

Objectives:
- explain the types of asymptotic notations
- list the basic asymptotic efficiency classes
- describe the efficient analysis of non-recursive algorithms with
illustrations
3.2 Asymptotic Notations and Basic Efficiency Classes
To choose the best algorithm we need to check the efficiency of each
algorithm. Asymptotic notations describe different rate-of-growth relations
between the defining function and the defined set of functions. The order of
growth is not restricted by the asymptotic notations, and can also be
expressed by basic efficiency classes having certain characteristics.
Let us now discuss asymptotic notations of algorithms
3.2.1 Asymptotic notations
Asymptotic notation within the limit deals with the behavior of a function,
i.e. for sufficiently large values of its parameter. While analyzing the run time
of an algorithm, it is simpler for us to get an approximate formula for the run-
time.
The main characteristic of this approach is that we can neglect constant
factors and give importance to the terms that are present in the expression
(for T(n)) dominating the functions behavior whenever n becomes large.
This allows dividing of un-time functions into broad efficiency classes.
To give time complexity as fastest possible, slowest possible or average
time, asymptotic notations are used in algorithms. Various notations such
as (omega), (theta), O (big o) are known as asymptotic notations.
Big Oh notation (O)
O is the representation for big oh notation. It is the method of denoting the
upper bound of the running time of an algorithm. Big Oh notation helps in
calculating the longest amount of time taken for the completion of algorithm.
A function T(n) is said to be in O(h(n)), denoted as T(n)eO(h(n)), if T(n) is
bounded above by some constant multiple of h(n) for all large n, i.e., if there
exist some positive constant C and some non negative integer n
0
such that
T(n s C h(n) for all n > n
0
.
The graph of C h(n) and T(n) can be seen in the figure 3.1. As n becomes
larger, the running time increases considerably. For example, consider
T(n)=13n
3
+42n
2
+2nlogn+4n. Here as the value on n increases n
3
is much
larger than n
2
, nlogn and n. Hence it dominates the function T(n) and we
can consider the running time to grow by the order of n
3
. Therefore it can be
written as T(n)=O(n
3
). The value of n for T(n) and C h(n) will not be less than
n
0.
Therefore values less than n
0
are considered as not relevant.

Figure 3.1: Big Oh Notation T(n) O(h(n))
Example:
Consider function T(n)=1n+2 and h(n)=n
2
. Determine some constant C so
that t(n)C*h(n).
T(n)=n+2 and h(n)=n
2
, find C for n=1, Now
T(n)=n+2
=(1) + 2
T(n)=3 and h(n)=n
2
=1
2
=1
T(n)>h(n)
If n=3
Then T(n)=(3)+2 = 5 and h(n)=n
2
=9
T(n)<h(n)
Hence for n>2, T(n)<h(n), Therefore Big Oh notation always gives the
existing upper bound.
Let us next discuss another asymptotic notation, the omega notation.
Omega notation ()
is the representation for omega notation. Omega notation represents the
lower bound of the running time of an algorithm. This notation denotes the
shortest amount of time that an algorithm takes.
A function T(n) is said to be in O(h(n)), denoted as T(n)
e
O(h(n)), if T(n) is
bounded below by some constant multiple of h(n) for all large n, i.e., if there
exist some positive constant C and some non-negative integer n0 such that
T(n) > C h(n) for all n > n0.
The graph of C h(n) and T(n) can be seen in the figure 3.2.

Figure 3.2: Omega Notation T(n) (h(n)
Example:
Consider T(n)=3n
2
+2 and h(n)=5n
Then assume n = 0
If n=0, T(n)=2 h(n)=0 i.e. T(n)>h(n)
Assume n=1
Then T(n)=5 and h(n)=5 i.e. T(n)=h(n)
Assume n=3
Then T(n)=29 and h(n) = 15 i.e. T(n)>h(n)
Therefore for n>3, T(n)>C h(n)

Theta notation ()
The depiction for theta notation is . This notation depicts the running time
between the upper bound and lower bound.
A function T(n) is said to be in O(h(n)), denoted as T(n)eO(h(n)), if T(n) is
bounded both above and below by some positive constant multiples of h(n)
for all large n, i.e., if there exist some positive constant C
1
and C
2
and some
non-negative integer n
0
such that C
2
h(n)s T(n)s C
1
h(n) for all n>n
0
The graph of C
1
h(n), C
2
h(n) and T(n) can be seen in the figure 3.3.
Figure 3.3: Theta Notation T(n) (h(n)
Example:
Assume T(n)=2n+5 and h(n)=4n similarly T(n)=2n+5 and h(n)=6n
Where n2 so now 4n<2n+5<6n
Here C
1
=4, C
2
=6 and n
0
=2
Theta notation is more accurate than Big Oh notation and Omega notation.
We will next discuss some rules that help us to analyze algorithms.
The maximum rule
Maximum rule is a useful tool that proves that one function is in the order of
another function.
Let F, h: N RZ
0
be two arbitrary functions. The maximum rule says that
O(F(n)+h(n))=O(max (F(n), h(n))).
Let p, q: N RZ
0
be labeled for each natural number n by p(n)=F(n) +h(n)
and q(n)=max (F(n), h(n)), and let an arbitrary function T: N RZ
0
be
defined then the maximum rule states that
T(n)eO(p(n)) if and only if T(n)eO(q(n)).
Maximum rule is applied for even Theta notation.
Example:
Consider an algorithm that progress in three steps: initialization, processing
and finalization.
These steps take time in (n
2
), (n
3
) and (nlogn) respectively. It is
therefore clear that to complete the algorithm, time taken is (n
2
+n
3
+nlog n).
From the maximum rule
(n
2
+ n
3
+ n log n) = (max (n
2
, n
3
+ nlog n))
= (max (n
2
, max (n
3
, nlog n)))
= (max (n
2
, n
3
))
= (n
3
)
From the above example, it is clear that the maximum rule simplifies to any
finite constant number of functions. We do not use this rule if some of the
functions often tend to be negative. When the rule is used then the risk
reasoning will be as below:
(n) = (n + n
2
- n
2
) = (max (n, n
2
- n
2
) = (n
2
)
Example:
Consider the function
(T (n)) = (max (12 n
3
logn, 6 n
2
, log
2
n, 36)
= (12 n
3
log n)
= (n
3
log n)
The above reasoning is not correct as the function 6n
2
is always negative.
The following reasoning is correct:
(T (n)) = (max (11n
3
logn + n
3
log n 6n
2
+ log
2
n + 36)
= (max (11n
3
log n, n
3
log n 6n
2
, log
2
n, 36)
= (11n
3
log n)
= (n
3
log n)
Even if n
3
log n6n
2
is negative for small values of n, for large values of n
(i.e. n4), it is always non-negative.
The limit rule
This powerful rule states that, given arbitrary functions F and h: N RZ
0

1. If
n
lim
) (
) (
n h
n F

e
R+ then F(n)
e
(h(n))
2. If
n
lim
) (
) (
n h
n F
= 0 then F(n)eO(h(n)) but F(n)e(h(n)
3. If
n
lim
) (
) (
n h
n F
= then F(n)e(h(n)) but F(n)e(h(n))
The LHpitals rule
L'Hpital's rule uses derivatives to help compute limits with indeterminate
forms. Application of the rule converts an indeterminate form to a
determinate form, allowing easy computation of the limit.
In simple cases, L'Hpital's rule states that for functions F(n) and h(n), if:
0 ) n ( h lim ) n ( F lim
x n x n
= =

or = =

) n ( h lim ) n ( F lim
x n x n

then:
) n ( h
) n ( F
lim
) n ( h
) n ( F
lim
x n x n
'
'
=

where F(n) and h(n) are the derivatives of F(n) and h(n) respectively.
Example:
Let F(n)=log n and h(n)= n be two functions. Since both F(n) and h(n) tend
to infinity as n tends to infinity, de lHpitals rule is used to compute

n
lim
) (
) (
n h
n F
=
n
lim
n
n log
=
1
lim
n
n
n
2 / 1
/ 1
=
n
lim
n
2

Therefore it is clear that F(n)eO(h(n)) but F(n)e(h(n)).
Conditional asymptotic notation
When we start the analysis of algorithms it becomes easy for us if we
restrict our initial attentions to instances whose size is satisfied in certain
conditions (like being a power of 2). Consider n as the size of integers to be
multiplied. The algorithm moves forward if n=1, that needs microseconds for
a suitable constant a. If n>1, the algorithm proceeds by multiplying four
pairs of integers of size n/2 (or three if we use the better algorithm). To
accomplish additional tasks it takes some linear amount of time.
This algorithm takes a worst case time which is given by the function
T: N R
0
recursively defined where R is a set of non negative integers.
T(1) = a
T(n) = 4T([n /2]) + bn for n >1
Conditional asymptotic notation is a simple notational convenience. Its main
interest is that we can eliminate it after using it for analyzing the algorithm. A
function F: N R
0
is non decreasing function if there is an integer
threshold n
0
such that F(n)F(n+1) for all nn
0
. This implies that
mathematical induction is F(n)F(m) whenever m nn
0
.
Consider b2 as an integer. Function F is b-smooth if it is non-decreasing
and satisfies the condition F(bn)eO(F(n)). Other wise, there should be a
constant C (depending on b) such that F(bn)C F(n) for all nn
0
. A function
is said to be smooth if it is b-smooth for every integer b2.
Most expected functions in the analysis of algorithms will be smooth, such
as log n, nlogn, n
2
, or any polynomial whose leading coefficient will be
positive. However, functions those grow too fast, such as nlogn, 2n or n! will
not be smooth because the ratio F(2n)/F(n) is unbounded.
Which shows that (2n)
log (2n)
is not approximate of O(n
log n
) because a
constant never bounds 2n
2
. Functions that are bounded above by a
polynomial are usually smooth only if they are eventually non-decreasing. If
they are not eventually non-decreasing then there is a probability for the
function to be in the exact order of some other function which is smooth. For
instance, let b(n) represent the number of bits equal to 1 in the binary
expansion of n, for instance b(13) = 3 because 13 is written as 1101 in
binary. Consider F(n)=b(n)+log n. It is easy to see that F(n) is not eventually
non-decreasing and therefore it is not smooth because b (2
k
1)=k whereas
b(2
k
)=1 for all k. However F(n)e(log n) is a smooth function.
A constructive property of smoothness is that if we assume f is b-smooth for
any specific integer b2, then it is actually smooth. To prove this, consider
any two integers a and b (not smaller than 2). Assume that f is b-smooth. It
is important to show that f is a-smooth as well. Consider C and n
0
as
constants such that F(bn)C F(n) and F(n)F(n+1) for all nn
0
. Let i= [log
b
a].
By definition of the logarithm
i a log a log
b b b a
b b
= s = .
Consider nn
0
. It is obvious to show by mathematical induction from
b-smoothness of F that F(b
i
n)C
i
F(n). But F(an)F(b
i
n) because F is
eventually non-decreasing and approximate to b
i
n an n
0
. It implies that
F (an) CF(n) for C=C
i
, and therefore F is a-smooth.
Smoothness rule
Smooth functions seem to be interesting because of the smoothness rule.
Consider F: N R
0
(where R
0
is non negative integers) as a smooth
function and T: N R
0
as an eventually non-decreasing function. Consider
an integer where b2. The smoothness rule states that T(n)e(F(n))
whenever T(n)e(F(n) | n is a power of b). We apply this rule equally to
O and notation. The smoothness rule assumes directly that T(n)e(n
2
) if
n
2
is a smooth function and T(n) is eventually non-decreasing. The first
condition is immediate since the function is approximate to n
2
(which is
obviously nondecreasing) and (2n)
2
=4n
2
. We can demonstrate the second
function from the recurrence relation using mathematical induction.
Therefore conditional asymptotic notation is a stepping stone that generates
the final result unconditionally. i.e. T(n)=(n
2
).
Some properties of asymptotic order of growth
- If F1(n)eO(h1(n)) and F2(n)eO(h2(n)) , then
F1(n)+F2(n)eO(max{h1(n),h2(n)})
- Implication: We determine the overall efficiency of the algorithm with a
larger order of growth.
o For example,
-6n
2
+2nlogneO(n
2
)
Asymptotic growth rate
- O(h(n)): Class of function F(n) that grows no faster than h(n)
- (h(n)): Class of function F(n) that grows at least as fast as h(n)
- (h(n)): Class of function F(n) that grows at same rate as h(n)

Comparing growth rate using limits
The limit-based approach is convenient for comparing the growth rate of
asymptotic functions.
n
lim
) n ( h
) n ( T
= 0 (order of growth of T(n)<order of growth of h(n))
n
lim
) (
) (
n h
n T
= C>0 (order of growth of T(n)=order of growth of h(n))
n
lim
) (
) (
n h
n T
= (order of growth of T(n)>order of growth of h(n))
3.2.2 Basic asymptotic efficiency classes
We have previously analyzed that we can obtain a different order of growth
by means of constant multiple (C in C*h(n)). We used different types of
notations for these orders of growth. But we do not restrict the classification
of order of growth to , and O. There are various efficiency classes and
each class possesses certain characteristic which is shown in table 3.1.
Table 3.1: Basic Asymptotic Efficiency Classes
Growth
order
Name of
the
efficiency
class
Explanation

Example

1 Constant Specifies that algorithms
running time is not changed
with increase in size of the
input.
Scanning the
elements of array
log n Logarithmic For each iteration of algorithm
a constant factor shortens the
problems size.
Performing the
operations of binary
search
N Linear Algorithms that examine the
list of size n.
Performing the
operations of
sequential search
n logn n-log-n Divide and conquer algorithms. Using merge sort or
quick sort elements
are sorted
n
2
Quadratic Algorithms with two embedded
loops.
Scanning the
elements of matrix

n
3
Cubic Algorithms with three
embedded loops.
Executing matrix
multiplication
2
n
Exponential Algorithms that generate all the
subsets which are present in n
element sets
Generating all the
subsets of n different
elements
n! Factorial Algorithms that generate all the
permutations of an n-element
set
All the permutations
are generated

Activity 1
Determine a constant p for a given function F(n)p*h(n) where
F(n)=2n+3 and h(n)=n
2
.
1. ___________ is more accurate than Big Oh notation and Omega
notation.
2. ____________ asymptotic notation is a simple notational convenience.
3. ___________ depicts the running time between the upper bound and
lower bound.
3.3 Mathematical Analysis of Non-Recursive Algorithms
In the previous section we studied and analyzed the types of asymptotic
notations involved in algorithms and examined the rules and limitations of
the algorithms with examples.
In this section we are going to deal with the mathematical analysis of non-
recursive algorithms.
We execute non-recursive algorithms only once to solve the problem.
3.3.1 Analyzing efficiency of non recursive algorithms
The steps involved in analyzing the efficiency of non-recursive algorithms
are as follows:
- Decide the input size based on the constraint n
- Identify the basic operations of algorithm
- Check the number of times the basic operation is executed. Find whether
the execution of basic operation is dependent on input size n or not. If
the basic operation is depending on worst case, best case and average
case then analysis of algorithm needs more attention.
- Set up summation formula for the number of times the basic operation is
implemented.
- Simplify the sum using standard formula and rules.
The commonly used summation rules are listed next.
Summation formula and rules:
1)
=
n
i 1
1= 1+1+1-----------1 = ne(n)
2)
=
n
i
i
1
= 1+2+3-----------n =
2
) 1 ( + n n
e(n
2
)
3)
=
n
i
i
1
K
= 1+2
k
+3
k
-------n
k
=
1
) 1 (
+
+
k
n n
k k
e(n
k+1
)
4)
=
n
i
i
a
1
= 1+a+ ---------a
n
=
1
1
1
+
a
a
n
e(a
n
)
5)
=
n
i
i
ca
1
= c
=
n
i
i
a
1

6)
i
n
i
i
b a
=1
=
=
n
i
i
a
1

i
n
i
b
=1

7)
=
n
k i
1 = n-k+1 where n and k are upper and lower limits
Example 1
We will now discuss an example for identifying the element which has the
minimum value in a given array.
We can find the element with the help of the general plan. Let us now see
the algorithm for finding the minimum element in an array.
Algorithm for finding the minimum element in an array
Algorithm Min_Element (A [0------n-1])
// Input: Array (A [0------n-1])
// Output: Returns the smallest element in the given array
Min_value A [0]
For i1 to n-1 do
{
If (A[i]) <Min_value) then
Min_value A [i]
}
return Min_value
Mathematical analysis
Step 1: The total number of elements in the given array is n, i.e. the size of
the input is n.
Step 2: The operation done here is comparison in loop in order to find the
smaller value.
Step 3: We implement the comparison recurrence of the loop. In this case
as comparison is done for each loop there is no need for analysis of best
case, worst case and average case.
Step 4: Consider h(n) as the iteration for the comparison to execute. We
compare the algorithm is every time the loop gets executed which implies
that for each value of i comparison is done. Therefore comparison is made
for i=1 to n-1 times.
h(n) = Single comparison made for each i
Step 5: Simplifying the sum i.e.
h(n)=
=
1
1
1
n
i

h(n) = n-1e(n) {Using the rule
=
n
i 1
1=n}
Therefore the efficiency of above algorithm is (n)
Let us now trace the algorithm for finding the minimum element in the array.
Algorithm tracing for finding the minimum element in an array
Let us consider n=4
(A [0,1,3,10])// A is an array of elements in which we are going to find the
//smallest element
Min_value A [0]//Min_value=0
For i1 to 4-1 do// this loop iterates from i=1 to i=4-1
{
If (A[1]) <Min_value) then// if 1<0
Min_value A[1]
}
return Min_value // Min_value = 0
Example 2
Let us next discuss an algorithm for counting the number of bits in an
integer.
Algorithm for counting the number of bits in an integer
Algorithm Binary (p)
// Input: p is the decimal integer
// Output: Number of digits
countA 1
While (p>1)
{
countA countA+1
p [p/2]
}
Return countA
Mathematical analysis:
Step 1: let the size of the input be p.
Step 2: The while loop indicates the basic operation and checks whether
p>1. Execution of while loop is done whenever p>1 is true. It gets executed
once more when p>1 is false. When p>1 is false the statements inside the
while loop is not executed.
Step 3: The value of n gets halved whenever the loop gets repeated. Hence
the efficiency of the loop is log
2
p.
Step 4: The total number of times the while loop gets executed is given by
[log
2
p] + 1.
Let us now trace the algorithm for counting the number of elements in an
integer.
Algorithm tracing for counting the number of bits in an integer
Let us consider p=16// p is a decimal integer
// Input: p is the decimal integer
// Output: Number of bits
countA 1
While (p>1)
{
countA countA+1// countA=2 in first iteration of this loop
p [p/2]// p=8 in first iteration of this loop
}
Return countA//the value returned will be 5
3.3.2 Matrix multiplication
In general, to multiply 2 matrices given, the number of rows in the first matrix
should be the same as the number of columns in the second matrix. In other
words two matrices can be multiplied only if one is of dimension mn and
the other is of dimension np where m, n, and p are natural numbers
{m,n,peN}. The resulting matrix will be of dimension mp.
A square matrix is one in which the number of rows and columns are the
same. We will now discuss the algorithm for matrix multiplication of two
square matrices with n elements each.
Algorithm for matrix multiplication:
Algorithm MatrixMultiplication(A[0.. n - 1, 0.. n - 1], B[0.. n - 1, 0.. n - 1])
//Multiplies two square matrices of order n by the definition-based
algorithm
//Input: Two n-by-n matrices A and B
//Output: Matrix C = AB
- 1 do
for j = - 1 do
C[i, j] =
for k = - 1 do

end
end
end
return C
The tracing for the matrix multiplication algorithm is given below.
Algorithm tracing for matrix multiplication of two 4X4 matrices:
Algorithm MatrixMultiplication(A[0.. 3, 0..3 ], B[0..3, 0..3])
//Multiplies two square matrices of order n by the definition-based
algorithm
//Input: Two 4-by-4 matrices A and B
//Output: Matrix C = AB
for i = - 1 do
for j = - 1 do
C[i, j] =
- 1 do
C[0, 0] = 0,0j] + A[0, 0] * B[0, 0] // this executes for k = 0 to 3
end
end
end
return C
Let us now analyze the steps involved in matrix multiplication:
- Let the input size be n
- Addition and multiplication are the two operations that take place in the
innermost loop according to the following rules:
o Both get executed exactly once for each repetition of the innermost
loop
o Choice is of no importance between two operations for the algorithms
basic operation
o The total number of basic operation (multiplication) executions are
dependent on n

- The basic operation count formula includes:
o Multiplication is performed once on each repetition of the innermost
loop (k) and finding the total number of multiplications are done for all
pairs of i and j
o The algorithm computes n
2
elements of the product matrix. Each
element is calculated as the scalar (dot) product of an n-element row
of A and an n-element column of B, which accepts n multiplications.
Therefore the number of basic operations performed to multiply two
matrices of n elements can be given by C(n) which is of the order n
3

i.e. C(n) =
=
1
0
n
i

=
1
0
n
j

=
1
0
1
n
k
=
=
1
0
n
i

=
1
0
n
j
n =
=
1
0
2
n
i
n =
3
n
Let us next discuss the time efficiency of recursive algorithms.
3.3.3 Time efficiency of non-recursive algorithms
Time efficiency approximation depends on the type of definition that is
needed to describe the steps involved in an algorithm. The time required to
perform a step should always bound above by a constant. In some
instances, count of addition of two numbers might be as one step. In such
cases approximation of time efficiency becomes critical. This consideration
might not justify certain situations. If the numbers involved in a computation
are randomly large, then the time required for performing single addition is
no longer considered as constant.
The steps involved in mathematical analysis of nonrecursive algorithms are:
- Decide the parameter n based on the input size
- Identify the basic execution of algorithm
- Determine the worst, average, and best case for the size of input (n)
- Set up a sum for C(n) which represents the loop structure of the
algorithm
- Simplify the sum using standard formulas
3.3.4 Tower of Hanoi puzzle
Let us now discuss a non-recursive algorithm for the Towers of Hanoi
problem.
Tower of Hanoi or Towers of Hanoi is a mathematical game or puzzle that
has three pegs A, B, and C (refer figure 3.4) and a number of disks of
different sizes which can slide onto any peg. Initially, we arrange all the
disks in a neat stack in ascending order of size on one peg putting the
smallest disc at the top. This makes a conical shape as can be seen in the
figure 3.4.

Figure 3.4: The Tower of Hanoi
A French mathematician Eduardo Lucas invented the puzzle in 1883. There
is a tale about a Vietnamese temple which comprises of a large room with
three time-worn posts in it surrounded by sixty four golden disks. The priests
of Hanoi used to move these disks according to the rules of the puzzle
during ancient times. The puzzle is therefore known as the Tower of Brahma
puzzle. According to the tale, the world will end when the last move of the
puzzle is accomplished.
This puzzle is played with any number of disks. The main aim of the puzzle
is to move all the disks to another peg according to the following rules:
- We can move only one disk at a time.
- In each move we can lift the upper disk from one of the pegs and slide it
onto another peg on top of the other disks which are already present on
that peg.
- We need to ensure that no disk is placed on the top of a smaller sized
disk.
The list of moves for a tower has much regularity. While counting, the
moves that start from 1 and the ordinal of the disk that moves during the
move m is divided by 2. Therefore every odd move will have the smallest
disk. The non-recursive algorithm solution is simpler than the recursive
algorithm.
In varying moves:
- Move the smallest disk to the peg from which it has not recently come
from.
- Move the other disk legally (there will be only one option)
With this knowledge, we can recover a set of disks in the middle of an
optimal solution having no state information other than the positions of each
disk.
- Examining the smallest top disk (that is not disk 0), and noting what will
be its only (legal) move.
- If that move is the disk's 'natural' move, then the disc is not moved since
the (last disc 0) move, and that move should be handled.
- If that move is not the disk's 'natural' move, then move disk 0.
Proven statement 1: In minimal length solution of towers of Hanoi puzzle
the first move is always with the smallest disk.
Proof: A single move is always combination of two consecutive moves with
a smallest ring.
Algorithm for moving the rings in clock wise direction in one post:
If n is odd then d: = clockwise else d: = counterclockwise
Repeat
Move the smallest ring in one post in the direction d till all rings are on
same post.
Make the legal move that will not include the smallest ring until all the
rings come to the same post.
3.3.5 Conversion of recursive algorithm to non-recursive algorithm
Let us now discuss how to convert recursive algorithms to non-recursive
algorithms.
We declare and initialize a stack (recursion stack) on insertion of the code at
the beginning of the function. Generally, the stack holds the values of
parameters, local variables, and a return address for each recursive call. We
use separate stacks for each value. A label of 1 is attached to the first
executable statement. If it is a value returning function, then we need to
change all appearances of the function name on the left hand side of
assignment statements by a new variable (say z) of the similar type as the
function.

A set of instructions replaces every recursive call which performs the
following:
1. Store the values of all pass by value parameters and local variables in
the stack. Declare the pointer to the top of the stack as global.
2. Create i-th new label, i, and store i in the stack. The value i is used as
return address in this label.
3. Compute the arguments of this call that relate to pass by value
parameters and assign these values to the right formal parameters.
4. Insert an absolute branch to the beginning of the function.
5. If it is a void function then add the label created in step 2 to the
statement followed by the unconditional branch. If this statement has
already labels then replace it and change all references to it.
6. If it is a value returning function then go according to the absolute
branch by code for using the value of the variable z in the same way as
the function value was used earlier. The label created in step 2 gives
the first statement of the code.
7. Following the above steps removes all recursive calls from the function.
Finally, it is the time to head the end statement of the function by code
to do the following:
8. Assign the value of z to the function name if the recursion stack is
empty. If it is a value returning function, then it computes the value till
the end of the function. Void functions are executed till the end of
function.
9. Restore all pass by value parameters and local variables if the stack is
not empty, then all pass by value parameters and local variables are
restored. They will be at the top of the stack by using the return label
from the top of the stack and executing a branch to this label. This is
done using a case statement.
10. Additionally, any label (if any) attached to the end of the function
statement is moved to the codes first statement for step 8 and 9.
Activity 2
Write an algorithm for counting even number of bits in an integer

4. Tower of Hanoi is a ____________ puzzle.
5. The time required to perform a step should always bound above by a
________________.
6. ____________ is of no importance between two operations for the
algorithms basic operation.
3.4 Summary
It is very important to obtain the best algorithm for analysis. For selecting the
best algorithm, checking the efficiency of each algorithm is essential. The
shorthand way for representing time complexity is asymptotic notation.
Asymptotic notation within the limit deals with the character of a function that
is a parameter with large values. The main characteristic of this approach is
that constant factors are neglected and importance is given to the terms that
are present in the expression (for T(n)) dominating the functions behavior
whenever n becomes large. This helps in classification of run-time functions
into broad efficiency classes. The different types of asymptotic notations are
Big Oh notation, Omega notation and Theta notation.
We classify algorithms broadly into recursive and non-recursive algorithms.
In this unit we have analyzed non-recursive algorithms mathematically with
suitable examples. Non-recursive algorithm is an algorithm which is
performed only once to solve the problem.
3.5 Glossary
Term Description
Recursive algorithm It is an algorithm which calls itself with smaller inputs
and obtains the inputs for the current input by applying
simple operations to the returned value of the smaller
input.
Runtime

The time when a program or process is being executed
is called as runtime.
Notation It is the activity of representing something by a special
system of characters.

1. Explain Big Oh notation with suitable example.
2. Define and explain Theta notation.
3. Explain conditional asymptotic notation with an example.
4. What are the various types of basic efficiency classes?
5. Explain Towers of Hanoi puzzle for non recursive algorithm with an
example.
6. How recursive algorithm is converted in to non recursive algorithm?

3.7 Answers
1. Theta notation
2. Conditional
3. Theta notation
4. Mathematical
5. Constant
6. Choice
Terminal Questions
4. Refer section 3.2.2 Basic efficiency classes
5. Refer section 3.3.5 Towers of Hanoi
6. Refer section 3.3.6 Conversion of recursive algorithm in to non
recursive algorithm
References
- A. A. Puntambekar (2008). Design and Analysis of Algorithms, First
edition, Technical publications, Pune.
- James Andrew Storer. An Introduction to Data Structures and
Algorithms, Brandies university Waltham, U.S.A.
E-Reference
- http://www.cmpe.boun.edu.tr/~akin/cmpe160/recursion.html
- www.cs.utsa.edu/~bylander/cs3343/chapter2handout.pdf
- www.scuec.edu.cn/jsj/jpkc/algorithm/res/ppt/Lecture03.ppt
Unit 4 Mathematical Aspects and
Analysis of Algorithms 2
Structure:
4.1 Introduction
Objectives
4.2 Mathematical Analysis of Recursive Algorithms
Recursion
Recursive algorithms
4.3 Empirical Analysis of Algorithms
Plan for Empirical Analysis of Algorithms
Pros and Cons of Empirical Analysis
4.4 Algorithm Visualization
Need for algorithm visualization
4.5 Summary
4.6 Glossary
4.8 Answers

4.1 Introduction
From the previous units, we know that algorithm analysis is very important
for the efficient working of an algorithm. We analyze an algorithm for its time
and space complexities to resolve a problem easily.
In the previous unit, we analyzed non recursive algorithms. Recursive
algorithms form a better way of solving problems. Therefore, in this unit we
mathematically analyze recursive algorithms.
In this unit we will define recursion and recursive algorithm with examples.
We will also discuss the empirical analysis of algorithms and algorithm
visualization with examples.
Objectives:
define Recursion and Recursive algorithms
analyze recursion with an example of Fibonacci numbers
explain empirical analysis of algorithms
describe algorithm visualization
4.2 Mathematical Analysis of Recursive Algorithms
We can classify the algorithm analysis into three types. They are:
Mathematical analysis
Empirical analysis
Algorithm visualization
In mathematical analysis of recursive algorithms, we analyze the algorithms
using mathematical and logical operations. Usually we use mathematical
induction to prove the correctness of the algorithms.
Let us first define recursion.
4.2.1 Recursion
Definition Recursion is defined as the process that refers itself to simplify
a problem.
More precisely, recursion is the process of simplifying large and complex
problems into small and easily solvable problems. In recursion, we divide
large problems into its sub parts and again recur the sub parts to solve the
problem.
Let us use a simple mathematics problem as given in equation Eq 4.1 to
study the concept of recursion.
Solve (((1 + 2) * (8 - 3)) + ((4 * 5) (9 / 3))). Eq: 4.1
We can divide equation Eq 4.1 into sub parts and solve it separately.
(1 + 2) = 3
(8 3) = 5
(4 * 5) = 20
(9 / 3) = 3
Now let us combine the resolved sub parts and form the second level
answers.
(3 * 5) = 15
(20 - 3) = 17
Finally, we combine the second level answers to get the final answer.
(15 + 17) = 32
We can use the recursion method for problems that satisfy the following two
conditions:
Each step of recursion has to break down the problem into smaller
problems of same type.
Each step of recursion needs to reduce the problem significantly.
The two types of recursion are:
Direct recursion This is a form of recursion in which a procedure or a
function refers to itself.
Indirect recursion This is a form of recursion in which a function P
calls another function Q, and Q in turn calls P in its body.
Use of Recursion We use recursion to make a complex code, simple.
Initially, the study of recursion and recursive algorithms seems to be
complicated but, once we identify the base process it becomes easy to
comprehend. Often, programming algorithms are written using recursion to
study it easily. In some cases recursion takes lots of programming time and
space. We use recursion only if the problem is recursively defined. Usually,
we use direct recursion more than indirect recursion.
Some practical uses of recursion are:
Disk directory trees navigation
Binary tree parsing
Searching
Sorting
Sorting is a real life example of recursion. We use lots of sorting algorithms
while programming. To enhance the efficiency of sorting algorithms we use
recursion. A sorting algorithm that includes recursion can have small
amount of coding in it and will decrease the man-hours required to write the
code.
Next, let us define recursive algorithms.
4.2.2 Recursive algorithms
Definition An algorithm defined at least partially in terms of the same
algorithm is known as a recursive algorithm.
Recursive algorithms play a huge role in all the programming we do. It is not
necessary to use recursion for simple computational problems, but if we
define algorithms recursively, it becomes very easy to write, study and
check for errors. Recursive algorithms need very few lines of code as it
performs the same process again and again on different data.
Before explaining recursive algorithms using examples, let us study what
recursive functions mean.
Recursive functions are functions defined recursively. We can write
functions directly or recursively. The example shown below explains the
difference.
Direct implementation:
f(x) = 40x+7 Eq: 4.2
Recursive implementation:
f(0) = 7(Base Case); Eq: 4.3
f(x) = f(x-1)+40 Eq: 4.4
Equation Eq 4.2 gives the equation to find the value of f(x). Here the value is
found by substituting suitable values for x in the equation. In equation
Eq: 4.3 the function f(x) is solved recursively. The base case is a terminal
condition of the function f(x). In equation Eq: 4.4, f(x) calls itself with a lesser
value of x. this continues until the base case is met.
We can define recursive functions in terms of the following factors:
Base case The base case of a function is the non-recursive part of the
problem that we already know. This base case is also known as an escape
clause, as we can use it to terminate the algorithm.
A recursive step Recursive step or steps are used to compute the current
value of a function by calling the function repeatedly. Eventually this will lead
the function to a stopping condition.
Let us take the above example. Suppose you have a complex function f(x)
which cannot be defined directly. Compute the function for some input
values, and find a pattern. If you find that f(0)=7 and for each f(x) value there
is a difference of 40 between them, then you can write the function
recursively as f(x)=f(x-1)+40. Here you can see that f(0) is the base case
and f(x)=f(x-1)+40 is the recursive step.
Now we will discuss some examples of recursive algorithms.
Examples of recursive algorithms
Fibonacci numbers The Fibonacci numbers are defined by F
0
=0, F
1
=1,
and for n
2, F
n
= F
n-1
+ F
n-2
. The first few numbers in the Fibonacci series
is given 0, 1, 1, 2, 3, 5 ,8, 13, 21, 34
By this definition we get
F
2
= F
1
+ F
0
= 1 + 0 = 1.
F
3
= F
2
+ F
1
= 1 + 1 = 2.
F
4
= F
3
+F
2
= 2 + 1 = 3.
Few values for N and F
n
are given below:
N 0 1 2 3 4 5 6 7 8 9
F
n
0 1 1 2 3 5 8 13 21 34
The recursive algorithm for Fibonacci numbers is given below:
Algorithm for Fibonacci Numbers
fib(interger;n)
{
if(n==0 or n==1)
{
return 1;
}
else
{
return fib(n-1)+fib(n-2);
}
}
We will next trace the algorithm for Fibonacci numbers.
Algorithm Tracing For Fibonacci numbers
n=2
fib(2)
{
if(n==0 or n==1)
{
return 1;
}
else
{
return fib(1)+fib(0) ;// return 1+1=2
}// this is a recursive function
}
We can see that the algorithm calls itself (fib()) to find the Fibonacci
numbers. It terminates when it reaches fib(0) and fib(1). The running time of
this recursive Fibonacci algorithm is given as T(n) =
(F
n+1
).
We will now discuss binary search, another recursive algorithm.
Binary search Binary search is a recursive searching algorithm used to
look for an item in a sorted list of items.
Binary means two, and at each step, we are dividing the remaining list into
two parts. The basic idea of binary search is that we use two variables to
keep track of the endpoints of the range in the list where the item could be.
Initially, the target could be anywhere in the list, so we start with variables
low and high set to the first and last positions of the list, respectively.
We can use recursion to look for the middle value and then recursively
search either the lower half or the upper half of the list. The recursive loop
stops when either of the following conditions is met:
When the searched value is found
When there are no more values in the list to search
Examples for binary search are given below:
Example 1: Find 6 in [1, 2, 6, 17, 19, 25, 45, 76, 100, 112].
Step 1 (middle element is 19 > 6): 1 2 6 17 19 25 45 76 100 112
Step 2 (middle element is 2 < 6): 1 2 6 17
Step 3 (middle element is 6 == 6): 6 17

Let us now discuss the recursive algorithm for binary search.
Algorithm for Recursive Binary Search
def recBinSearch(x, nums, low, high)
if low > high( No place left to look, return -1)
return -1
mid = (low + high) / 2
item = nums[mid]
if item == x(Found the item. Return the index)
return mid
else if x < item (Search lower half)
return recBinSearch(x, nums, low, mid-1)
else(Search upper half)
return recBinSearch(x, nums, mid+1, high)
We can then recursively call the first function to start the search between
0 and len(nums)-1. Let us see the algorithm tracing for recursive binary
search.
Algorithm Tracing for Recursive Binary Search
low=1,high=3, X=3;
nums[ ]=[1,2,3,4];
def recBinSearch(x, nums, low, high)// this is a recursive function
if 1> 4//No place left to look, return -1
return -1
mid = (1 + 3) / 2 // the value will be 2/2=2
item = nums[2]
if item == 3//Found the item. Return the index
return mid
else if x < 3//Search lower half
return recBinSearch(3, nums, 1, 1)
else(Search upper half)
return recBinSearch(3, nums, 3, 3)
def search(x, nums):
return recBinSearch(x, nums, 0, len(nums)-1)
This recursive binary search is easy to write and execute, but the binary
search using loops is a bit faster than this algorithm. The recursive versions
of binary search use a divide and conquer structure. Here, the efficiency of
algorithm lies in the way of searching an item. Instead of searching each
item individually, the binary search algorithm cuts the list into half every time
it calls itself again.
The Towers of Hanoi puzzle also uses a recursive algorithm. In the previous
unit we had discussed the non-recursive algorithm for the Towers of Hanoi
problem. Let us now discuss the recursive algorithm for this problem.
Towers of Hanoi Towers of Hanoi is a mathematical puzzle, which has
three towers or rods and a set of disks with different diameters. Initially the
disks are arranged one above the other, so that a smaller disc is placed
above a larger one. All the discs are to be transferred from the first tower
(Tower A) to the third tower (Tower C) with the help of the second tower
(Tower B) as a temporary storage.
The basic rules to be followed while transferring the discs are:
We can move only one disk at a time.
We cannot set aside the disk; it should be kept in another tower.
We cannot keep a larger disk on a smaller one.
The task is to move the disks from the first tower to the third tower.
Consider a simple example of this puzzle with three disks as shown in
figure 4.1. Let us label the towers as A, B, C. So the problem here is to
move the disks from A to C.

Figure 4.1: Towers of Hanoi Puzzle with Three Disks

To solve this puzzle, we have to seven moves as shown in the figure 4.2.

Figure 4.2: Solution of Towers of Hanoi Puzzle with Three Disks

The steps are:
Step 1 Move the smallest disc from A to C
Step 2 Move the second largest disc from A to B
Step 3 Move the smallest disc from C to B
Step 4 Move the largest disc from A to C
Step 5 Move the smallest disc from B to A
Step 6 Move the second largest disc from B to C
Step 7 Move the smallest disc from A to C
The steps to solve the towers of Hanoi puzzle with 5 disks arranged
randomly on the three towers is given in the table 4.1.
Table 4.1: Towers of Hanoi Puzzle with Five Disks
Step No: Tower A Tower B Tower C
Initial case Disc 3 Disc 4 Discs 1,2,5
1 3 1, 4 2, 5
2 2, 3 1, 4 5
3 1, 2, 3 4 5
4 1, 2, 3 4, 5
5 2, 3 1, 4, 5
6 3 2 1, 4, 5
7 3 1, 2 4, 5
8 1, 2 3, 4, 5
9 1 2 3, 4, 5
10 1 2, 3, 4, 5
11 1, 2, 3, 4, 5
The recursive algorithm for solving the towers of Hanoi puzzle with n
number of disks is given below:
Step 1 Move n-disk tower from source to destination via resting place
Step 2 Move n-1 disk tower from source to resting place
Step 3 Move 1 disk tower from source to destination
Step 4 Move n-1 disk tower from resting place to destination
The base case for this algorithm is when there is only one disk. Here we use
two recursive moves of n-1 disks and an additional move in step 3. As the
algorithm proceeds, the solvable number of disks in the puzzle is reduced.
Let us now define the recurrence relation for the towers of Hanoi puzzle.
The total number of moves required to solve the towers of Hanoi puzzle T[N]
is given in the equation 4.5.
T[N] T[N-1]+1+T[N-1] = 2T[N-1]+1 Eq: 4.5

Now let us solve this recurrence relation in Eq: 4.5.
T[N] = 2T[N-1]+1
= 2 [2 T[N-2]+1]+1
= 2 [2 [2 T[N-3]+1]+1]+1
= 2[2 [2 [2 T[N-4]+1]+1]+1]+1
T[N]= 2
4
T[N-4] +15
For N number of moves, we can derive T[N] as in equation Eq 4.6.
T[N]= 2
x
T[N-x] + 2
x
-1 Eq: 4.6
The number of moves, N is considered as finite. Therefore we can derive
T[N] as given in equation 4.7.
T[N]
N x
= 2
N
-1 Eq: 4.7
The complexity of this algorithm is given as O(2
N
). This efficiency of the
puzzle depends upon the number of disks and the initial case of the puzzle.
If there are many disks, then it will take exponential time to solve the puzzle.
If the initial case is close to the final case then the complexity of the puzzle
is reduced and it becomes more efficient. The recursive tree for the towers
of Hanoi puzzle with 4 disks is given in figure 4.3.

Figure 4.3: Recursive Tree for Towers of Hanoi Puzzle with Four Disks
The number of steps for solving the puzzle increases with the number of
disks. The table 4.2 given below shows the number of steps required for
disks up to a number of 5. This is calculated using equation 4.2.

Table 4.2: Number of Steps for Different Number of Disks
Disks 1 2 3 4 5
Steps 1 3 7 15 31
Even though we can write a simple algorithm to solve the towers of Hanoi
puzzle, it is considered as an intractable problem. This puzzle requires too
much of computing resources such as time and memory.
Analyzing efficiency of recursive algorithms
Let us see the general plan for analyzing the efficiency of recursive
algorithms. The steps are as follows:
1) Decide the size of the input based on a parameter n.
2) Identify and analyze the basic operations of the recursive algorithm.
3) Determine the number of times the basic operations are used. Check
whether the basic operations require more size than the decided input
size n.
4) Determine the best, worst and average cases for the input size n.
We have to analyze the cases separately if the basic operations depend
on it.
5) To solve the basic operation, set a recurrence relation.
6) Solve the recurrence relation using the forward and backward
substitution method. We can prove the solution using mathematical
induction.
These steps will be clearer if we discuss it using some examples.
Example 1: Computing factorial of a number n.
We can find the factorial of a number n by performing repeated
multiplication.
Consider n=5, then n factorial (n!) is computed by following steps:
1) n!= 5!
2) 4! * 5
3) 3! * 4 * 5
4) 2! * 3 * 4 * 5
5) 1! * 2 * 3 * 4 * 5
6) 0! * 1 * 2 * 3 * 4 * 5 * 6
7) 1 * 1 * 2 * 3 * 4 * 5 * 6
The recursive algorithm for finding factorial of a number is given as:
Algorithm to Find the Factorial of a Number
factorial(n)
{
//Input: A non negative integer, n
//Output: Factorial value of n
If(n=0)
return 1
else return factorial(n-1) * n
Let us see the algorithm tracing to find the factorial of a number.
Algorithm Tracing to Find the Factorial of a Number
n=2
factorial(2)// this is a recursive function which traces to itself
{
If(2=0)//this condition is not met
{
return 1
}
else return factorial(2-1) * 2 // recursively calls factorial (1)
Now let us see the mathematical analysis of factorial algorithm.
Step 1: The algorithm works for input n.
Step 2: The basic operation of computing the factorial is multiplication.
Step 3: The recursive function call can be formulated as:
F(n) = F(n-1) *n where n >0
The basic operation, multiplication is given as M(n).
M(n) = M(n-1) + 1
Where the term M(n-1) is the multiplication which is used to calculate the
factorial (n-1).
The last term 1 represents the product of (n-1) and n.
Step 4: The obtained recurrence relation M(n) = M(n-1) +1, can be solved
using forward and backward substitutions.
Forward substitution
M(1) = M(0) + 1 Eq: 4.8
M(2) = M(1) + 1= 1 + 1= 2 Eq: 4.9
M(3) = M(2) + 1 = 2 + 1=3 Eq: 4.10
In equations Eq: 4.8, Eq: 4.9, and Eq: 4.10, we are substituting value of n
and directly finding the value for M(n).
Backward substitution
M(n) = M(n-1) + 1
= [M(n-2)+1]+1 = M(n-2) + 2
= [M(n-3)+1]+1+1 = M(n-3) +2 Eq: 4.11
From the substitution method shown in equation Eq: 4.11, we can establish
a general formula as given in equation Eq: 4.12.
M(n) = M(n-i) + i Eq: 4.12
Let us now prove the correctness of the formula using mathematical
induction.
Prove: M(n)= n By mathematical induction
Let n = 0 then
M(n) = 0
i.e. M(0) = 0 = n
Induction: if we assume M(n-1) = n-1 then
M(n) = M(n-1) + 1
= n -1+1
= n
i.e. M(n) = n
Thus the time complexity of factorial function is
(n).
Even if we know the concepts of recursion, we need to know when and
where to use it. We will next discuss the precautions to be kept in mind
while using recursion.
Precautions on recursion
In all cases recursion is not the only best way to solve a problem. We can
use recursion only when the problem is recursively defined. Any function
that is defined using recursion can also be defined using iteration. The time
taken to define a recursive function is more and some recursive method
calls can be difficult. Sometimes, recursion uses lots of space to solve the
problem. In such cases, a direct implementation of the problem works
better. Recursion should be used only if this direct implementation is very
complex to study.
The rules to be kept in mind while deciding on using recursion are:
We can use recursion if the problem definition requires a recursive
solution, provided it follows the next point.
We have to analyze the time and memory space of the recursive
solution. If we can solve it using a non recursive way, with lesser time
and space then go for it.
Do not use lots of recursion to solve a single problem. It becomes
complex to track every sub solution.
If we get a small and elegant recursive solution, then go for it.
A recursive function with its last line as a recursive call, does not waste
lots of memory. We can use this optimized way to write a recursive
function
Activity 1
Write an algorithm to calculate x
n
for different values and analyze its
efficiency.
1. _________ is defined as the process that refers itself to simplify a
problem.
2. _____________ need very few lines of code as it performs the same
process again and again on different data.
3. In the towers of Hanoi problem, if the numbers of disks is n, the number
of steps will be ___________.

4.3 Empirical Analysis of Algorithms
The analyses discussed in the previous units are mathematical analysis of
algorithms. We can also analyze algorithms empirically.
What is empirical analysis?
Empirical analysis of algorithm means analyzing the behavior of the
algorithm with a specific set of inputs. We apply a set of input values on the
algorithm and analyze the corresponding outputs.
4.3.1 Plan for Empirical Analysis of Algorithms
The general plan for empirical analysis of algorithms is given below:
1) Comprehend the purpose of the given operation.
2) Create an efficiency metric M and decide the unit of measurement.
3) Decide the type and range of inputs.
4) Write a program for the algorithm.
5) Generate a sample set of input values.
6) Run the program for this sample set of inputs and record the resulting
outputs.
7) Analyze the outputs.
Let us discuss the steps in detail.
1. Comprehend the purpose of the given operation
We perform the empirical analysis of algorithm for the following reasons:
To check the accuracy of the algorithm
To compare the efficiencies of different algorithms working to solve
the same problem
To analyze the efficiency of a particular algorithm on a specific
machine.
To develop the algorithms efficiency class
2. Create an efficiency metric M and decide the unit of measurement
We can measure the efficiency of algorithms using the following
methods:
We have to insert a counter in the algorithm to count the number of
times the operation is computed.
Consider that we are writing a function to calculate the sum of n
numbers in an array, the efficiency of the algorithm can be found by
inserting a frequency count. We can write a count to know the number of
times a statement is executed.
int sum(int b[20], int n)
{
int i,sum=0
for(i=0;i<n;i++)
{
sum=sum+b[i];
}
return sum;
}
We can leave the declaration part while measuring and analyze the
loop.
Case 1: if i = 0; frequency count is 1
Case 2: if i<n, frequency count= (n+1); Here the statement executes n
times for the condition i<n and executes once for the condition
i>n
Case 3: i++; frequency count is n
Case 4: sum= sum+b[i]; frequency count is n
Case 5: Return sum; frequency count is 1
Therefore, the total frequency count = (3n+3).After neglecting the
constant part, we can see the efficiency of the algorithm is O(n).
We can measure the time taken by the execution time of an algorithm
using the system clock.
But it is very difficult to calculate the system time for the following
reasons:
o It might not be accurate.
o It depends upon the type of computer. We can solve a problem in
limited time on a modern computer.
o If it is a time sharing system, it may include CPU execution time for
execution of the problem.
3. Decide the type and range of inputs
It is necessary to decide a set of inputs, even if we measure the
efficiency by frequency count or by time clocking methods. Observe the
behavior of the algorithm for various set of inputs. For example, we may
measure the ratio R(2*n)/R(n), where 2*n is the doubled set of inputs.
Thus we can find the efficiency of the algorithm.
4. Write a program for the algorithm.
Write a suitable program to solve the problem. We should write the
program in a simple way to comprehend each step easily.
5. Generate a sample set of input values.
We have to generate a sample set of input values for the program.
6. Run the program for this sample set of inputs and record the resulting
outputs.
Execute the program written in any suitable language. Collect the output
obtained from it and record it accordingly.
7. Analyze the outputs
Analyze the output values using a graph or any other suitable pictorial
representation.
4.3.2 Pros and Cons of Empirical Analysis
Empirical analysis is used for algorithms for which the mathematical
analysis is difficult. Let us study the advantages and disadvantages of
empirical analysis.
Advantages of empirical analysis
We can solve the problem without using any complicated mathematics
in it.
We can always apply empirical analysis to any algorithm easily.
The pictorial analysis of the algorithm makes it easy to study.
Disadvantage of empirical analysis
It depends upon the sample set of inputs.
It depends upon the computer on which it is executed.

4. _________________________ of algorithm means analyzing the
behavior of the algorithm with a specific set of inputs.
5. We can measure efficiency of algorithms using ________________ and
_______________ methods.
6. The __________ analysis of the algorithm makes it easy to study.

4.4 Algorithm Visualization
So far we have analyzed algorithms mathematically and empirically. Now let
us study how to analyze algorithms visually. Certain algorithms are so
complex that it becomes difficult for us to track the steps and find a solution
to the problem. For this reason, algorithm visualization is used.
Algorithm visualization is defined as a technique which uses images to
convey the information about algorithms.
4.4.1 Need for algorithm visualization
In algorithm visualization, we use some images or animation to illustrate the
operations of the algorithm. We can highlight the different behavior of
algorithms on different set of inputs on these images. We can compare
different algorithms using this method.
The two types of algorithm visualization are as follows:
Static algorithm visualization: This type of visualization uses still images
to illustrate the algorithm.
Dynamic algorithm visualization: This type of visualization uses
animations to illustrate the algorithm. This is also known as algorithm
animation.
Dynamic visualization is difficult to represent, but is used for better study of
algorithms. This is inspired from the color sound film created by University of
Toronto about sorting algorithm.
We need to implement the following measures for algorithm visualization:
Create a consistent visualization.
Create an interactive visualization, so that any ordinary user can study it.
Create clear and crisp visualizations.
Adopt friendliness while developing the system.
Have a clear idea about the system, before presenting it to the users.
Reduce the use of text and include more graphics.
Incorporate all the symbolic and ionic representations.
Include the detail algorithm analysis and comparisons with other
algorithms.
Include the execution history in such systems.
The two fields for which algorithm visualization is applied are:
Education algorithm visualization is widely used in the field of education.
Students can easily understand animated algorithms. For example, we can
study sorting and searching algorithms using bar charts and line graphs.
Research Researches use algorithm visualization to study the uncovered
features of different algorithms. This leads to further development and
progress in the particular field.
The concept of algorithm visualization will be clear with the following
examples.
Examples of Algorithm Visualization
Example 1: Bubble sort

Figure 4.4: Algorithm Visualization of Bubble Sort (n)
Figure 4.4 shows the algorithm visualization of bubble sort. The guidelines
for performing this visualization are as follows:
We can see the sorted portion of the list as a triangular block in the
bottom-right part of the image. We can use this to measure the
percentage of time taken.
We can track each element in the sort as the start and end of the
element is visible.
We can measure the percentage of list sorted at 20% and 80% of the
process.
Here we also see that the curve of the sorted elements is not linear, but
it is close to n
2
.
Example 2: Quick sort
Figure 4.5 shows the algorithm visualization of Quick Sort. The basic idea of
this algorithm visualization is to choose one element that we call pivot
(which is shown as the lines crossing in the centre) and to place all the
elements lower than the pivot on the left side and all the elements higher
than the pivot on the right side. This way the pivot is placed on the right
place and we repeat the same procedure for the two remaining sub lists and
sort the list recursively.

Figure 4.5: Algorithm Visualization of Quick Sort (n)

Activity 2
Write an algorithm to search an element from a list and illustrate using
the algorithm visualization features.

7. ______________________ is defined as a technique which uses
images to convey the information about algorithms.
8. ___________ visualization is the type of visualization which uses still
images to illustrate the algorithm.
9. ____________ visualization is the type of visualization which uses
animations to illustrate the algorithm. This is also known as algorithm
animation.
4.5 Summary
In this unit, we have defined recursion and recursive algorithms. We defined
recursion as something that calls itself to solve the problem easily.
Recursive function is any function that uses recursion in it. Next we
discussed the examples for recursive algorithms, namely. Fibonacci
numbers, binary search and the towers of Hanoi puzzle.
We also studied the steps for analyzing efficiency of recursive algorithm
using the example of finding the factorial of a number. Recursion should not
be used beyond a limit. We have also discussed when to avoid recursion.
We have discussed empirical analysis of algorithms which uses a set of
inputs to solve the problem. We explained the steps for empirical analysis
and its pros and cons. We have analyzed the process of algorithm
visualization which is the method of representing algorithms using images
and animations. We have also studied some illustrated examples of sorting
algorithms.

4.6 Glossary
Terms Description
Recurrence
relation
Recurrence relation is an equation that recursively defines a list
where each term of the list is defined as a function of the
preceding terms.
Induction Induction is a method of mathematical proof used to establish
that a given statement is true for all natural numbers.

1. What are the basic rules of the towers of Hanoi puzzle?
2. What are the rules for analyzing the efficiency of recursive algorithms?
3. What are the basic conditions to avoid recursion?
4. What are the steps for analyzing an algorithm empirically?
5. What are the guidelines followed while performing algorithm
visualization?

4.8 Answers
1. Recursion
2. Recursive algorithms
3. 2n-1
4. Empirical analysis
5. Counters, system clocking
6. Pictorial
7. Algorithm visualization
8. Static algorithm
9. Dynamic algorithm
Terminal Questions
1. Refer section 4.2.2 Towers of Hanoi
2. Refer section 4.2 Analyzing efficiency of recursive algorithms
3. Refer section 4.2 Precautions on recursion
4. Refer section 4.4.1 Plan for empirical analysis of algorithms
5. Refer section 4.5 Need for algorithm visualization
References
Corman. Thomas (1990). Introduction to algorithms. McGraw-Hill Book
Company
Liang, Daniel (2009). Introduction to Java Programming. Pearson
Eduction Ltd
Putembekar, A. A (2009). Design & Analysis of Algorithms. Technical
Publications
E-References
http://www.algolist.net
http://www.cargalmathbooks.com/5%20Recursive%20Algorithms.pdf
http://cse.unl.edu/~dsadofs/RecursionTutorial/index.php?s=intro#recfunc
http://www.devshed.com/c/a/Practices/Solving-Problems-with-Recursion
http://www.corte.si/posts/visualizingsorting/index.html
http://mila.cs.technion.ac.il/~yona/c/lecture12/slide19.html
Unit 5 Brute Force Method
Structure:
5.1 Introduction
Objectives
5.2 Brute Force
Brute force algorithm
5.3 Selection Sort and Bubble Sort
Selection sort
Bubble sort
5.4 Sequential Search and Brute Force String Matching
Sequential search
Brute Force string matching
5.5 Exhaustive Search
Definition
Implementation
Reordering the exhaustive search
Speeding up exhaustive search
Alternatives to the exhaustive search
5.6 Summary
5.7 Glossary
5.9 Answers
5.1 Introduction
In the earlier unit you studied about the mathematical analysis of algorithms.
In this unit you will study about brute force method in detail with algorithms.
Brute force is a problem solving technique wherein we compute a series of
possible answers and test each possible answer for accuracy. It simply tries
all possibilities until a satisfactory solution is found. Once it finds the value
for the best solution it stops. In this unit we will discuss various algorithms
which make use of the brute force method.
Objectives:
define brute force method
describe and analyze selection sort and bubble sort
analyze and compute sequential search and brute force string matching
explain and discuss the exhaustive search
5.2 Brute Force
Let us first define brute force.
Definition Brute force is defined as a primitive programming technique
where the programmer relies on the processing strength of the computing
system rather than his own intelligence to simplify the problem. Here the
programmer applies appropriate methods to find a series of possible
answers and tests each possible answer for accuracy.
Let us now discuss the brute force algorithm.
5.2.1 Brute force algorithm
Brute force algorithm is a basic algorithm which works through every
possible answer until the correct one is found. We can solve a particular
problem easily using brute force algorithm rather than wasting time on
creating a more elegant solution, especially when the size of the problem is
small.
A good brute force algorithm should have the following factors in it.
Small number of sub solutions
Specific order in the sub solutions
Sub solutions must be evaluated quickly
Let us take an example of counting change.
Consider a cashier who counts some amount of currency with a collection of
notes and coins of different denominations. The cashier must count a
specified sum using the smallest possible number of notes or coins.
Let us now analyze the problem mathematically.
Consider n as number of notes or coins and the set of different
denominations of currency . ,... 2 , 1 n p p p P .
Let d
i
= denomination of p
i

In the Indian system p
i
= { Rs 1, Rs 2, Rs 5, Rs 10,..} the d
i
= { 1, 2,5, 10..}
If we have to count a given sum of money A we need to find the smallest
subset of P such that
S pi
i d
= A.
Let us represent the subset S with n variables as . ,... 2 , 1 n x x x X
Such that

For {d1,d2..dn} we have to minimize as
n
i
i x
1

Such that
n
i
i ix d
1
= A
Since each element of X, . ,... 2 , 1 n x x x X
is either equal to zero or one,
there will be 2
n
possible values for any X in an algorithm. The best solution
for brute force algorithm to solve a problem is to compute all the possible
values, given any variable X.
For each possible value of X, we check whether the constraint
n
i
i ix d
1
= A
is satisfied for it or not. A value that satisfies the constraint is called as a
feasible solution. An objective function
n
1 i
i x
is associated with an
optimization problem determining how good a solution is.
Since there are 2
n
possible values of X, we assume that the running time of
brute-force solution is (2
n
). The running time needed to determine whether
a possible value of a feasible solution is O(n) and the time required to
compute the objective function is also O(n) is O(n2
n
).
Activity 1
Write a brute force algorithm to compare 25 text characters in an array
and match with one character.

1. A value that satisfies the constraint is called a ___________.
2. ____________ is a function that is associated with an optimization
problem determining how good a solution is.
3. The running time needed to determine whether a possible value of a
feasible solution is O(n) and the time required to compute the objective
function is also O(n) is ________.
5.3 Selection Sort and Bubble Sort
In the previous section we analyzed how brute force algorithm works. Now
in this section let us analyze and implement brute force algorithms for
selection sort and bubble sort.
5.3.1 Selection sort
First, let us define selection sort.
Selection sort is one of the simplest and performance oriented sorting
techniques that work well for small files. It has time complexity as O(n
2
)
which is unproductive on large lists.
Let us see an example for selection sort. In figure 5.1, we use selection sort
to sort five names in alphabetical order.

Figure 5.1: Example of Selection Sort
Here we compare consecutive names and exchange it if not in order.
Implementation of selection sort
The Selection sort spends most of its time in finding out the minimum
element in the "unsorted" part of the array. Selection sort is quadratic in both
worst and average case and needs no extra memory.
For each i from 1 to n - 1, there exists one exchange and n - i comparisons.
So there will be total of n - 1 exchanges and (n - 1) + (n - 2) + . . . + 2 + 1 =
n (n - 1)/2 comparisons. These observations will not bother about what input
is given as data. In the worst case, we assume that time complexity of
selection sort is quadratic, but in the average case, we consider that time
complexity is O(n log n). This implies that the running time of selection sort
is quite insensitive to the input.
Pseudocode implementation
Let us find the smallest element in the array and exchange it with the
element which is in first position. Similarly we shall find the second smallest
element and exchange it with the element that is present in the second
position. We continue this process until the entire array is sorted. Let us see
the pseudocode implementation for selection sort.
Pseudocode for selection sort (ascending order)
Selection_Sort (A)
for i 1 to n-1 do
min j i;
min y A[i]
for j i + 1 to n do
If A[j] < min y then
min j j
min y A[j]
A[min j] A [i]
A[i] min y
If the array is already sorted in descending order then the worst case
occurs. "If A[j] < min y" is computed exactly the same number of times in
every case then the variation in time is only due to the number of times the
"then" part (i.e., min j j; min y A[j] of this test is executed.

Analysis
We can analyze selection sort very easily compared to other sorting
algorithms since none of the loops depend on the data that is present in the
array. For selecting the lowest element in the array we scan all n elements
(this takes n 1 comparisons) and then swap it with the first position. For
finding the next lowest element we scan the remaining n 1 element and so
on for (n 1) + (n 2) + ... + 2 + 1 = n (n 1) / 2 (n
2
) comparisons. Each
of these scans require one swap for n 1 elements (because the final
element is already in place).
Comparison with other sorting algorithms
Amongst simple average-case (n
2
) algorithms, selection sort always
outperforms bubble sort, insertion sort and gnome sort. Insertion sort's main
advantage is that it can only scan as many elements as it needs in order to
place the k + 1st element, while selection sort scans all remaining elements
to find the k + 1st element.
Another key difference which we can observe is that selection sort always
performs (n) swaps while insertion sort performs (n
2
) swaps in the
average and worst cases because generally swaps require writing to the
array. In such case, selection sort is more preferable.
5.3.2 Bubble sort
Definition A bubble sort is a sorting algorithm that continuously moves
through a list swapping the items till they appear in a correct order. Bubble
sort is the simplest sorting algorithm.
We can execute bubble sort by iterating it down an array (that has to be
sorted) from the first element to the last and compare with each pair of
elements and switch their positions if required. We should repeat this
process till the array is sorted.
Performance of algorithm
Let us now analyze the performance of bubble sort algorithm. For analyzing
the algorithm, we assume an array containing elements to be sorted. Now
we will look through the array and pick the smallest element to put it in
position 1. This is the first pass. For second pass, we need to consider the
remaining list from the second element to the last in order to put the next
smallest element in position 2 and so on till we sort all the elements.
For instance, let us consider the array of numbers as A[4] = {4, 5, 3, 2}.
(A[0]=4, A[1]=5, A[2]=3, A[3]=2) Pictorial representation of how sorting (first
pass and second pass) is performed is shown in figure 5.2.
In the first pass of figure 5.2, we check for the smallest element in the array.
The smallest element is located at .A[3] i.e. 2. Now we swap 2 with 3
(that is located at A[2]). Let us compare A[1] and A[2]. We find 2 as the
smallest element. So we swap 2 with 5 (that is present in A[1]). Let us now
compare A[0] with A[1]. After comparison we swap 2 with the element that is
present in A[0]=4. Hence the order of elements in first pass is 2, 4, 3, 5.
In second pass of figure 5.2, we take the order of elements obtained from
the first pass i.e. 2 4 3 5. Let us compare A[3] with A[2] and swap the
smallest element. So 3 is swapped with 5. In the next step we compare
A[2] with A[1] and swap 2 with 5.In the last step, we compare A[1] with
A[0] and swap 2 with 4. Hence after sorting the elements, the order of the
elements is 2 3 4 5.

Figure 5.2: Bubble Sort
Example code for sorting the array elements
for i = n down to 2
for j = 1 to i-1
if A[j] < A[j+1]
swap(A,i,j)
Algorithm analysis
In the above example, the outer loop runs n times. The complexity in this
analysis lies in the inner loop. If we run the inner loop for a single time, we
get a simple bound by noting that it can never loop more than n times. Since
the outer loop makes the inner loop to complete n times, we cannot
compare more than O(n
2
) times. This seems to be a very high bound
because when we run the inner loop for last time, it will only make one
comparison which is less than n. When we run the inner loop for first time, it
will make n-1 comparisons, then next time it will make n-2 comparisons; and
so on. Therefore, the actual number of comparisons is
1 n
1 k
k
that has a value
of (n-1)n/2 which is also O(n
2
). Thus bubble sort has worst, best and
average case run-time of O(n
2
).
Let us now discuss the pseudocode of bubble sort for sorting an array of
integers.
Pseudocode for bubble sort to sort an integer array
procedure bubble sort( A : list of sortable items ) defined as:
do
swapped := false
for each i in 0 to length(A) - 2 inclusive do:
if A[i] > A[i+1] then
swap ( A[i], A[i+1] )
swapped: = true
end if
end for
while swapped
end procedure
Optimizing bubble sort
We can optimize bubble sort easily after each pass by observing the
smallest element that will always move up the array. Let us assume a list of
size n. The nth element will be in its final place. Hence we can sort the
remaining n - 1 elements. Now after this pass, the n - 1
st
element will be in
its final place. This allows us to skip over a lot of the elements and helps in
tracing of the "swapped" variable. This will ultimately lead to a worst case of
50% improvement in iteration count but will have no improvement in swap
counts.
We optimize bubble sort for sorting an integer array using the below
pseudocode implementation.
Pseudocode for optimized bubble sort for sorting an integer array
procedure bubble Sort( A : list of sortable items ) defined as:
n := length( A )
do
swapped := false
n := n 1
for each i in 0 to n do:
if A[ i ] > A[ i + 1 ] then
swap( A[ i ], A[ i + 1 ] )
swapped := true
end if
end for
while swapped
end procedure
We then do bubbling passes on smaller parts in an increasing order in the
list. To be more precise, instead of doing n
2
comparisons and swaps, we
can use (n-1) + (n-2) + ... + 1 comparison which will sum up to n(n - 1) / 2
comparisons.
Activity 2
Write an algorithm to sort four elements in an array list.
4. Selection sort is one of the simplest and ________ sorting techniques.
5. Bubble sort has __________, best and average case run-time of O(n2).
6. ______________________ is the simplest sorting algorithm.
5.4 Sequential Search and Brute Force String Matching
The previous section helped us to analyze the implementation of selection
sort and bubble sort algorithms. In this section we will deal with the
implementation of sequential search and brute-force string matching
algorithms.
5.4.1 Sequential search
Let us first define sequential search.
Definition Sequential search is a process for finding a particular value in a
list that checks every element (one at a time) in sequence till the desired
element is found.
Sequential search is the simplest brute force search algorithm. It is also
known as linear search. This search is a special case of brute-force search.
The algorithms worst case cost is proportional to the number of elements in
the list and its expected cost. Therefore, if the list contains more than a few
elements then other methods like binary search or hashing becomes more
efficient.
Consider an example as given in figure 5.3.

Figure 5.3: Example for Sequential Search
Here we are searching for the element 15 in a sorted list sequentially, by
comparing every element with the given value.
Implementation of the algorithm
We can implement sequential search very easily. This search is very
practical when we perform a single search in an unordered list or when list
contains only few elements.
Let us analyze how pseudocode implementation is performed in various
ways.
Onward iteration of the algorithm
The pseudocode portrays a typical variant of sequential search where the
result of the search is assumed to be either the location of the list item
where the desired value is present or an invalid location () to show that the
desired element does not occur in the list.
Pseudocode for onward iteration of the sequential search algorithm
For each item in the list:
if particular item has desired value,
stop the search and return to the location where the item is present.
Return .//Invalid location
In this pseudocode implementation, we execute the last line only after all list
items are examined with none matching.
If we store the list as an array data structure then the location of the item at
the index of the list will be usually between 1 and n, or 0 and n1. In such
case the invalid location () will be any index before the first element (such
as 0 or 1, respectively) or after the last element (n+1 or n, respectively).
Recursive version
We will next describe the recursive algorithm of sequential search.
Pseudocode for recursive version of sequential search algorithm
If the list is empty, return ;// Invalid location
else
if the first item of the list has the desired value, return its location;
else search the value in the remainder of the list and return the
result.
Analysis
A list with n items has best case when the value of n is equal to the first
element of the list and we do not need to do any comparisons in this case.
The worst case happens when the value is not in the list or appears only
once at the end of the list and in this case we need n comparisons.
When the value which we require occurs k times in the list then the
estimated number of comparisons are asymptotic. Hence O(n) is the worst-
case cost and also the expected cost of sequential search.
Searching the array elements
We program sequential search in an array by stepping up an index variable
until it reaches the last index. We normally require two comparison
instructions for each list item. One, we use for checking whether the index
has reached the end of the array and another for checking whether the item
contains the desired value.
Let us consider an array A with elements indexed from1 to n. We have to
search for a value x in the array. We can perform a forward search using
pseudocode and this code returns n + 1 if the value is not found. Let us now
see the pseudocode for forward search.
Pseudocode implementation for forward search
Set i to 1.
Repeat this loop:
If i > n, then exit the loop.
If A[i] = x, then exit the loop.
Set i to i + 1.
Return i.
Let us now search array using pseudocode in the reverse order and return 0
when the element is not found.
Pseudocode implementation for reverse order
Set i to n.
Repeat this loop:
If i 0, then exit the loop.
If A[i] = x, then exit the loop.
Set i to i 1.
Return i.
We will next discuss the brute force string matching algorithm.

5.4.2 Brute Force string matching
If we are given a string of n characters known as text and a string of m
characters (m <= n) known as pattern then we can find a substring of the
text that matches the pattern using string matching algorithm. This implies
that we find i the index of the leftmost character of the first matching
substring.
Let us now discuss an example of brute-force string matching algorithm.
Brute-force string matching algorithm
Algorithm BruteForceStringMatch (A [0n-1], B [0m-1])
// executes brute-force string matching
//Input: An array A [0n-1] of n characters for a text, an array B [0m-1]
of m characters for a pattern
//Output: The index of first character in the text starts a matching string or
-1 if the search is not successful\
For i 0 to n-m do
j 0
while j<m and B[j] = A[i+j] do
j j+1
if j=m return i
return -1
Let us trace brute-force string matching algorithm
Algorithm tracing for brute-force string matching
A[ ]= {S H I N E B I K E}
B[ ]= {I N E}
n=9
m=3
For i 0 to 9-3 do
j 0
while 0<3 and B[0] = A[0] do
0 0+1
if 0=3 return 0 //the final value returned is 2.
return -1

Let us consider a string of characters and analyze the string matching
algorithm.
SHINE BIKE is the character string here. Let us see how the algorithm
computes. INE is our pattern here with which we are going to match our
string.
S H I N E _B I K E
I N E
I N E
I N E
In the above example, first the pattern I N E points to the first letter of the
text which is S. As I which is first letter of pattern I N E is not matched with
S , I N E checks for the next letter of the text i.e. H which is not a matching
string again. Now the pattern I N E checks for the third letter in the text
which is I. Text letter I matches with patterns first letter i.e. I. It then
checks if the next letter of the pattern N matches with the next character of
the text which is also N. And thus the text I N E (last three letters of
S H I N E) matches with the pattern I N E.
Let us discuss the Nave string matching algorithm which is a type of brute
force string matching algorithm.
Nave String Matching algorithm
The nave approach tests all the possible placements of Pattern A [1 . . . p]
relative to text T [1 . . . q]. , we try to shift s = 0, 1. . . q - p, successively and
for each shift compare T[s +1 . . . s + p] to A [1 . . p]. Let us now discuss the
nave string matching algorithm.
We can interpret nave string-matching procedure graphically as a sliding
pattern A [1 . . . p] over the text T [1 . . . q] and note down for which shift all
of the characters in the pattern are matching the corresponding characters
in the text.
In this execution, we use notation A [1 . . . j] to represent the substring of A
from index i to index j. i.e., A [1 . . . j] = A[i] A [i +1] . . . A[j]. Let us analyze
the algorithm when there is no substring of T matching A.

Nave String Matching algorithm
Naive_String_Macher (T, A)
q length [T]
p length [A]
for s 0 to q-p do
j 1
while j p and T[s + j] = A[j] do
j j +1
If j > p then
return valid shift s
return no valid shift exist // i.e., there is no substring of T matching A.
Let us trace the above algorithm.
Algorithm tracing for Nave String Matching algorithm
T[ ] = [S H I N E B I K E]
A[ ] = [I N E]
Naive_String_Matcher (T, A)
q 9
p 3
for s 0 to 1 do
j = 1
while j 3 and T[1] = A[1] do
j = 2
If j > 3 then
return valid shift s
return no valid shift exist // i.e., there is no substring of T matching A.
When we refer to implementation of nave matcher, we see that the for-loop
in line 3 is executed at most q - p + 1 times and the while-loop in line 5 is
executed at most m times. Hence we can say that the running time of the
algorithm is O((q - p +1) p) which is clearly O(pq). We say this algorithm is in
the quadratic time when the length of the pattern m is roughly equal in worst
case. We assume one worst case as that the text T has n number of A's and
the pattern A has (p - 1) number of A's which is followed by a single B.

7. ________________ is also known as linear search.
8. We program sequential search in an array by _________an index
variable until it reaches the last index.
9. In this pseudocode implementation, we execute the __________ only
after all list items are examined with none matching.
5.5 Exhaustive Search
In the above section you studied how to analyze and implement sequential
search and string matching algorithms. In this section we will profoundly
analyze the implementations of exhaustive search.
5.5.1 Definition
Exhaustive search or Brute force search (also called as generate and test)
is a trivial but general problem-solving technique that systematically
specifies all possible candidates for the solution and checks whether each
candidate satisfies the problem's statement.
However, we know that its cost is proportional to the number of candidate
solutions which in many practical problems tend to grow very quickly as
problem size increases. Therefore, we use exhaustive search typically when
the problem size is limited and when implementation is more important than
speed.
5.5.2 Implementation of the algorithm
For implementing exhaustive search to a specific class of problems, we
need to follow four procedures. They are first, next, valid and output. To
solve the problem, these procedures will take a parameter (data p) for
particular instance.
Algorithm procedure for exhaustive search
We should follow the given below algorithm procedure to implement
exhaustive search.
Algorithm procedure for exhaustive search
First (P): Generating a first candidate solution for P.
Next (P, c): Generating the next candidate for P after the current one c.
Valid (P, c): Verifying whether candidate c is a solution for P.
Output (P, c): Using the solution c of P as an appropriate to the
application.
We implement Next procedure when there are no candidates for the
instance p after the current one. A convenient way for us is that we can
return a null candidate (some conventional data value ) that is distinct
from real candidate. In the same way we implement First procedure when
there are no candidates for instance p and we return some conventional
data value, .
5.5.3 Reordering the exhaustive search
Some applications require only one solution rather than preferring all the
solutions. In such cases, expected running time of exhaustive search often
depends on the order in which the candidates are tested. As a general rule,
we should test the most promising candidates first. For example, when we
search for a proper divisor of a random number n we should enumerate the
candidate divisors in increasing order from 2 to n - 1 because the probability
we obtain when we divide n by c is 1/c.
5.5.4 Speeding up exhaustive search
One way by which we can speed up the exhaustive search algorithm is to
reduce the search space i.e. we can give the set of candidate solutions to
specific problem class by using experienced based techniques which help in
problem solving.
For example let us consider the problem of finding all integers between 1
and 1000 that are evenly divisible by 20. A naive brute-force search solution
will generate all integers in the range (from 1 to 1000) testing each of them
for divisibility. However, this problem can be solved much more efficiently by
starting with 20 and repeatedly adding 20 until the number exceeds 1000
which takes only 50 (i.e.1000/20) steps and no tests. This way we can
speed up the search in finding the integers that are divisible by 20.
5.5.5 Alternatives to the exhaustive search
We can use experience based techniques (heuristics) to make an early
cutoff of parts of the search. One example for exhaustive search is the
minimax principle for searching game trees that eliminates many sub trees
in the search at an early stage.
We can reduce the search space for problems by replacing the full problem
with a simplified version. For example in a computer chess game we
compute a more limited tree of minimax possibilities rather than computing
the full minimax tree of all possible moves for the remainder of the game.
Hope you are clear about the brute force method and the different
algorithms that use this method.
10. Exhaustive search implementation is more important than _________.
11. Exhaustive search algorithm gives the ______________ for every
candidate that is a solution to the given instance P.
12. Exhaustive search is typically used when the problem size is
___________.
5.6 Summary
It is very essential for us to obtain a best algorithm for any analysis. Brute
force method is a mathematical proof which helps in simplifying the finite
number of classes of each case and proves each case separately for
analysis.
We analyzed the performance of selection sort and bubble sort algorithms
and implemented the algorithms in pseudocode with suitable examples and
figures.
Sequential search and bruteforce string matching algorithms are the
simplest algorithms to implement. In this unit we examined the performance
of sequential search algorithm in a systematic way. We implemented
sequential search algorithm in the following ways - forward iteration,
recursive version, searched in an ordered list and reverse order search.
Exhaustive search is a method which helps us in determining all the
possible candidates for the solutions and helps in verifying whether the
candidates satisfy the problems solution.
5.7 Glossary
Term Description
Pseudocode Pseudocode is an artificial and informal language that helps
programmers to develop algorithms.
Heuristics

Heuristics is an adjective for experience-based techniques
that help in problem solving, learning and discovery.
Gnome sort Gnome sort is a sorting algorithm which is similar to insertion
sort but moves an element to its proper place by a series of
swaps as in bubble sort.
Substring Substring of a string is a subset of the symbols in a string
where order of the elements is preserved.
1. Explain brute-force algorithm?
2. Explain selection sort algorithm implementation with suitable example.
3. Give the pseudocode implementation of bubble sort algorithm.
4. Explain the analysis of sequential search algorithm with examples.
5. What is exhaustive search and how it is implemented?

5.9 Answers
1. Feasible solution
2. Objective function
3. O(n2
n
).
4. Performance oriented
5. Worst
6. Bubble sort
7. Sequential search
8. Stepping up
9. Last line
10. Speed
11. Output
12. Limited
Terminal Questions
1. Refer section 5.2.1 Brute-force algorithm
2. Refer section 5.3.1 Selection sort
3. Refer section 5.3.2 Bubble sort
4. Refer section 5.4.1 Sequential search
5. Refer section 5.5.1 Definition of exhaustive search
References
Rashid Bin Muhammad. Design and Analysis of Computer Algorithms.

E-Reference
http://caveshadow.com/CS566/Sabin%20M.%20Thomas%20-%20String
%20 Matching%20Algorithms.ppt
http://www.cse.unl.edu/~ylu/csce310/notes/BruteForce.ppt.
http://www.cs.miami.edu/~burt/learning/Csc517.051/notes/selection.html
http://cprogramminglanguage.net/c-bubble-sort-source-code.aspx
http://knol.google.com/k/bubble-sort#
cs.unco.edu/course/CS101/F06/Chapter09.ppt
http://www.personal.kent.edu/~rmuhamma/Algorithms/MyAlgorithms/
Sorting/selectionSort.htm
http://webspace.ship.edu/cawell/Sorting/bubanal.htm

Unit 6 Divide and Conquer
Structure:
6.1 Introduction
Objectives
6.2 Mergesort
Procedure for mergesort
Sorting and merging two arrays
Analysis of mergesort algorithm
6.3 Quicksort
Procedure of quicksort
Algorithm
Analysis of Quicksort
6.4 Binary Search
Algorithm
Analysis of binary search
Advantages and disadvantages
6.5 Binary Tree Traversals and Related Properties
Concepts of binary tree
Binary tree traversal
Applications
6.6 Strassens Matrix Multiplication
6.7 Summary
6.8 Glossary
6.10 Answers
6.1 Introduction
In the previous unit we discussed the concepts of Brute Force Method. In
this unit we will discuss the Divide and Conquer methodology which is a
well-known general design methodology. We have to implement the
following steps to solve a given problem using the Divide and Conquer
methodology.
Step 1: Breakdown the problem into several sub problems of the same type
Step 2: Solve each problem recursively
Step 3: Combine these solutions to generate a solution for the main
problem
This methodology is the base of efficient algorithms for different kinds of
problems, like Mergesort, Quicksort, the Fast Fourier Transfer, and large
number multiplications.
Objectives:
- explain the Divide and Conquer methodology
- perform different sorting techniques
- build and traverse a binary tree
- apply Strassens matrix method
6.2 Mergesort
Sorting is a regular and important problem in computing. Mergesort is a
perfect example of a successful application of the Divide and Conquer
methodology. It was invented by John von Neumann in 1945. It is a
comparison-based sorting algorithm. In most implementations it preserves
the input order of equal elements in the sorted output and hence it is stable.
If N is the number of items to be sorted, mergesort takes O(N*log N) time to
sort the algorithm. For example, if N is 10,000(10
4
), then N
2
is
100,000,000(10
8
), while N*log N is only 40,000. Mergesort takes about 40
seconds to sort these items whereas it would take almost 28 hours for
insertion sort.
First, let us study the procedure of mergesort.
6.2.1 Procedure of mergesort
We can perform mergesort by implementing the following procedure:
1) Divide the array A[0..n-1] into almost equal halves and make copies of
each half in arrays B and C
2) Sort arrays B and C recursively
3) Merge sorted arrays B and C into array A by repeating the following
steps until no elements remain in any of the arrays
a. Compare the first elements in the remaining unprocessed portions of
the arrays
b. Copy the smaller of the two into A, while incrementing the index
indicating the unprocessed portion of that array
4) Once all elements in one of the arrays are processed, copy the
remaining unprocessed elements from the other array into A
The merge sort procedure will be clear with the help of the figure 6.1 given
below.

Figure 6.1: Example for Mergesort
Let us see the merging procedure in detail
Merging two sorted arrays B and C creates a third array A that contains all
the elements of B and C, also arranged in sorted order. We will examine the
merging process first, and later we will see how its used in sorting. Lets say
array B has 4 elements and array C has 6. They will be merged into an
array A that starts with 10 empty cells. Figure 6.2 shows these arrays.

Figure 6.2: Merging Two Arrays
The steps given below explain in detail the figure 6.2, shown above,
Step 1: Compare 23 and 7, since 7 is lesser than 23 copy 7 from C to A
Step 3: Compare 23 and 39, since 23 is lesser than 39 copy 23 from B to A
Step 4: Compare 39 and 47, since 39 is lesser than 47 copy 47 from B to A
Step 9: Copy 81 from B to A
Step 10: Copy 95 from B to A
We can observe that, because B is empty after step 8, no more
comparisons are required. Therefore all the remaining elements are simply
copied from A into C.
6.2.2 Sorting and merging two arrays
Now, let us see an algorithm to sort and merge two arrays.
Sorting two arrays
To sort an array, we divide it into two parts and sort it recursively. Let us
now discuss the algorithm for sorting the array using mergesort.
Mergesort (Sorting) Algorithm
Algorithm Mergesort(A, i, j)
//sorts array A[i..j] by recursive mergesort
If (i<j)
{
mid =

/2 j) (i +

Mergesort(A, i, mid)
Mergesort(A, mid+1,j)
Merge(A, i, j, mid)
}
Here the Mergesort() function accepts the array A ,its upper and lower
bounds i and j as arguments. It splits the array into two parts using a mid
value. We sort the partial arrays i to mid and mid+1 to j using the same
Mergesort() function and repeat the process recursively.
Let us see the tracing for the merge sort (sorting) algorithm.
Tracing of the Mergesort (Sorting) algorithm
Let us assume the number of elements n=6 in an array.
Algorithm Mergesort(A, 0, 5)
If (0<6)
{
mid =

/2 5) (0+
// mid =2
Mergesort(A, 0, 2) // recursively sorts the array elements from 0-2
Mergesort(A, 2+1,5) // recursively sorts the array elements from 3-5
Merge(A, 0, 5, 2) // Combines the sub-solutions
}

Merging two sorted arrays
We will next discuss the algorithm for merging the two sorted arrays.
Algorithm for merging the two sorted arrays
Merge(A, i, j, mid)
//Merges two sorted arrays into one sorted array
//input: Arrays A[i..mid] and A[mid+1..j] both sorted
//Output: Sorted array A[i..j]
//Creates an array B and C
n1 = mid i + 1
n2 = j mid
create array B[1, n1+1] and C[1, n2+1]
for m = 1 to n1 do
{
B[m] = A[i+m 1]
}
for k = 1 to n2 do
{
C[k] = A[mid +k]
}
m = 1, k =1
for x = i to j do
{
if B[m] C[k]
{
A[x] = B[m]
m =m + 1
}
else
{
A[x] = C[k]
k = k + 1
}
}
Now let us trace the merge sort algorithm and the algorithm for merging the
two sorted arrays.
Tracing of algorithm for merging two sorted arrays
Merge(A, 0, 5, 2)
//Merges two sorted arrays into one sorted array
//input: Arrays B[0..2] and C[3..5] both sorted
//Output: Sorted array A[0..5]
n1 = 2 0 + 1//n1 = 3
n2 = 5 2// n2 = 3
create array B[1, 3+1] and C[1, 3+1]//B[1,4] and C[1,4]
for m = 1 to 3 do
{
B[1] = A[0+1 1]// B[1] = A[0]
}
for k = 1 to 3 do
{
C[1] = A[2 +1]// C[1] = A[3]
}
m = 1, k =1
for x = 0 to 5 do
{
if B[1] C[1]
{
A[0] = B[1]
m =1 + 1//m = 2
}
else
{
A[0] = C[1]
k = 1 + 1//k =2
}
}
Thus by merging two sorted arrays, mergesort creates a third array that
contains all the elements from both arrays in sorted order.
6.2.3 Analysis of mergesort algorithm
Let us analyze the mergesort algorithm.
We understand that, mergesort on an input sequence S with n elements
consists of three steps:
1) Divide Partition S into two sequences S1 and S2 of about n/2
elements each
2) Recur Recursively sort S1 and S2
3) Conquer Merge S1 and S2 into a unique sorted sequence
The height h of the mergesort tree is O(log n). At each recursive call we split
the sequence into half. The overall amount or work done at the nodes of
depth i is O(n). Divide and merge 2
i
sequences of size n/2
i
and make 2
i+1

recursive calls. Thus, the total running time of merge-sort is O(n log n).
To analyze the efficiency of mergesort let us consider the recurrence
relation and deduce the conclusion for best case, average case and worst
case.
In merge sort algorithm two recursive calls are made, each recursive call
focuses on n/2 elements of the list. One call is made to merge two sub-lists
after two recursive calls that is to merge all n elements.
Therefore,
T(n)=T(n/2)+T(n/2)+cn Eq: 6.1
Where T(n/2) is time taken by left or right sub-list to get sorted, cn is the
time taken to combine two sub-list and n>1 therefore T(1)=0
With the help of substitution method let us derive the complexity of
algorithm.
From equation 6.1, T(n)=2T(n/2)+cn Eq: 6.2
Substitute n = 2
k
k = log
2
n Eq: 6.3
T(2
k
) = 2T(
2
2
k
)+c 2
k

T(2
k
) = 2T(2
k-1
)+c2
k

Put k = k-1, therefore T(2
k
) = 2T(2
k-1
)+c 2
k

T(2
k
) = 2
2
T(2
k-2
)+2c 2
k-1
+c 2
k

Hence, T(2
k
)= 2
2
T(2
k-2
)+2c 2
k

Similarly we can write,
T(2
k
)=2
k
T(1)+kc 2
k
Eq: 6.4
T(2
k
) = 2
k
.0+kc 2
k

T(2
k
) = kc 2
k

T(n) = log
2
n.cn
T(n) = O(log
2
(n))

Therefore O(log
2
(n)) is the best, average, and worst case time complexity of
merge sort. Some issues regarding merge sort are large storage,
complication in implementation .Therefore it is practically not very useful.
Hope you are clear with the concepts of mergesort. Now, let us next discuss
quick sort.

Activity 1
Consider an array A[ ] having values [17, 4, 3, 5, 20, 10, 13, 15]. Explain
the steps to perform mergesort on this array.
1. Mergesort is a perfect example of a successful application of the
____________ and ____________ methodology.
2. ____________ is a comparison-based sorting.
3. What are the three steps involved in mergesort?
6.3 Quicksort
C.A.R. Hoare in 1960 invented the quicksort algorithm and formally
introduced quick sort in 1962. It is another important sorting algorithm based
on the Divide and Conquer approach. Unlike the mergesort, which breaks
up its input elements according to their position in the array, quicksort
breaks them according to their value.

6.3.1 Procedure of Quicksort
Following are the recursion step to quicksort algorithm:
1) Choose a pivot value Middle element is considered as pivot value,
but it can be any value, which is in range of sorted values, even if it is
not present in the array.
2) Partition Rearrange elements in such a way that all elements which
are lesser than the pivot go to the left part of the array, and all elements
greater than the pivot goes to the right part of the array. Values
equivalent to the pivot can stay in any part of the array. The array may
be divided into non-equal parts.
3) Sort both parts Apply quicksort algorithm recursively to the right and
the left parts.
Figure 6.3 has a detailed explanation in steps to depict the procedure of
quick sort technique for sorting an array having values {0, 11, 4, 25, 6, 13,
2, 6, 1}. We randomly choose 6 as the pivot in this example.

Figure 6.3: Quicksort Technique
In figure 6.3, the underlined value is the value ready for comparison and
swapping.
Choosing a Pivot Value:
The following ideas will explain how to choose a pivot value to partition the
array:
1) The pivot value is the key value of an actual data item.
2) Randomly pick a data item as a pivot value. For better performance of
the algorithm, choose the median element as the pivot.
3) After the partition, pivot will be in its final sorted position if the pivot is
inserted at the boundary between the left and right sub-arrays.
The last point may sound distinct. The pivots key value is used to partition
the array. Following the partition the right sub-array holds items larger than
the pivot, and the left sub-array holds items smaller. The pivot starts out on
the right, but if it could somehow be placed between these two sub-arrays, it
would be in the correct place that is, in its final sorted position.
6.3.2 Algorithm
Let us next discuss the algorithm for quicksort.
Quicksort Algorithm
Partition(A[l.r])
//Partitions a sub-array by using its first element as a pivot
//Input: A sub-array A[lr] of A[0..n-1], defined by its left and right
//Indices l and r(l<r)
//output: A partition of a[l..r] with the split position returned as
//this functions value
p A[l] // pivot element
i l;
j r+1
repeat
repeat ii+1 until A[i]p
repeat jj-1 until A[j]p
swap(A[i], A[j])//undo last swap when i j
swap(A[l], A[j])
return j
Let us now trace the quicksort algorithm.
Tracing of Quicksort Algorithm
Let the number of whole numbers n in the array be 10.
Input:[5, 15, 16, 14, 13, 21, 24, 18, 10, 30]=A[0, 1, 2, 3, 4, 5,
69]=A[0,,n-1]
Let us assume l=2 and r=5, where l is the left end of the sub-array and
r be the right end of the sub-array.
Therefore sub-array A[l..r]=A[2, 3, 4, 5]=[16, 14, 13, 21]
Partition(A[25])
pA[2]=16 // the first element of array A[lr] is taken as the pivot
ii=2;
j5+1;
Repeat
Repeat i2+1=3 (A[3]=14)16
I 3+1=4 (A[4]=13)16//since 13 is smaller than 16, the step
//repeats
i4+1=5 (A[5]=21)16
Repeat j6-1=5 (A[5]=21)16
j5-1=4 (A[4]=13)16
until 54
swap(A[5]=21, A[4]=13)
swap(A[2]=16, A[4]=13)
return 4
Let us now discuss the recurrence quicksort algorithm.
Recurrence Quicksort Algorithm
recQuickSort(int left, int right)
{
If(right-left<=0)//if size is1,
return ; //it is already sorted
else //size is 2 or larger
{
//partition range
int partition=partitionIt(left, right);
recQuickSort(left,partition-1);//sort left side
recQuickSort(partition+1, right);//sort right side
}
}
The tracing of the recurrence quicksort algorithm is given below.
Tracing of Recurrence Quicksort algorithm
left=1
right=4
recQuickSort(1,4)
{
If(4-1<= 0)//if size is1,
return;//it is already sorted
else//size is 2 or larger
{
//partition range
int partition=partitionIt(1,4);
recQuickSort(1, partition-1);//sort left side
recQuickSort(partition+1, 4);//sort right side
}
}
After a partition, all the items in the right sub-array are greater than all those
on the left. Now sort the left sub-array and sort the right sub-array, the entire
array will be sorted.
The arguments to the recQuickSort() method determine the left and right
ends of the array or sub-array it is supposed to sort. The method first checks
if this array consists of only one element. If so, then the array is by definition
already sorted, and the method returns immediately.
If the array has two or more cells, the algorithm calls the partitionIt() method.
This method returns the index number of the partition, the left element in the
right sub-array. The partition marks the boundary between the sub-arrays.
Figure 6.4 illustrates the recurrence quicksort.


Figure 6.4: Recursive Calls Sorts Sub-arrays
After the array is partitioned, recQuickSort() calls itself recursively, once for
the right sub-array (from partition+1to right), and once for the left sub-array
(from left to partition-1). The data item at the index partition is not included in
either of the recursive calls because it needs to be sorted. It is important to
choose the correct pivot value. Let us now analyze quicksort.
6.3.3 Analysis of Quicksort
We know that the quicksort procedure consist of three steps namely,
choosing the pivot, partitioning the array and sorting the parts recursively.
The complexity of the algorithm depends upon the choosing the correct
pivot element. If we choose the pivot as the middle element the sorting
steps of the algorithm are reduced. Let us consider that there are n
elements in the array. The pivot P chosen divides the array into k and n-k
elements. The time taken to sort the array is given as in equation Eq: 6.5.
T(n) = T(k) + T(k-n) + n Eq: 6.5
Where T(n) is the time taken to sort an array of elements n and n or O(n) is
the time taken to rearrange the array.
Now consider that the pivot is the first element of the array.
Therefore k = 1 and n k = n 1. We can modify equation Eq: 6.5 as in
Eq: 6.6.
T(n) = T(1) + T(n 1) + n Eq: 6.6
Now let us analyze the time complexity in detail by solving the recurrence
T(n) = T(n 1) + T(1) + n
= [T(n 2) + T(1) + (n 1)] + T(1) + n
Let us assume that the pivot element chosen divided the original of size n1
into two parts: one of size n 2 and the other of size 1.
= T(n 2) + 2T(1) + (n 1 + n)
= [T(n 3) + T(1) + (n 2)] + 2T(1) + (n 1 + n)
= T(n 3) + 3T(1) + (n 2 + n 1 + n)
= [T(n 4) + T(1) + (n 3)] + 3T(1) + (n 2 + n 1 + n)
= T(n 4) + 4T(1) + (n 3 + n 2 + n 1 + n)

= T(n i) + iT (1) + (n i + 1 + ..... + n 2 + n 1 + n)
= T(n i) + iT (1) + (
=

1
0
) (
i
j
j n
) Eq: 6.7
The recurrence shown in equation Eq: 6.7 will be valid until i = n1.
Otherwise n-1 would be less than 1. If we substitute i= n-1 in equation
Eq: 6.7 then
= T(1) + (n - 1)T (1) + (
=

2
0
) (
n
j
j n
)
Worst case:
Since
=
2
0
n
j
j
=
=
2
1
n
j
j
= (n 2)(n 1)/2
= nT(1) + (n(n 2) (n 2)(n 1)/2)
which is O(n
2
).
Therefore the worst case complexity of quicksort is given as O(n
2
).
Best case:
The best case of quick sort is when we pick the pivot which divides the array
into two equal parts at every step. In that case k= n/2 and nk = n/2.
The recurrence will be as in equation Eq: 6.8
T(n) = 2T(n/2) + n Eq: 6.8
= 2(2T(n/4) + n/2) + n
= 2T(n/2) + n)
= 2
2
T(n/4) + 2n
= 2
2
(2T(n/8) + n/4) + 2n
= 2
3
T(n/8) + 3n

= 2
k
T(n/2k) + kn Eq: 6.9
This recurrence given in equation Eq: 6.9 will continue until n = 2
k
i.e. until
k=log n. Thus the equation Eq: 6.9 can be rewritten as in equation Eq: 6.10.
T(n) = nT(1) + n log n Eq: 6.10
which is O(n log n).
Therefore the best case complexity of quicksort is given as O(n log n).
Average case:
The average case complexity of quicksort is also the same as that of the
best case i.e. O(n log n).
This section analyzed the concepts of quick sort. The next section will
explain the technique of binary search.
4. If the array has two or more cells, the algorithm calls the ___________
method.
5. Unlike the merge sort, which breaks up its input elements according to
their position in the array, quick sort breaks them according to their
____________.
6. After the partition, if the pivot is inserted at the boundary between the
___________ sub-arrays, it will be in its final sorted position.
6.4 Binary Search
Binary search is an algorithm for identifying the position of an element in a
sorted array. To find the position of an element, it scrutinizes the middle
element of the sorted list. If the middle element is equal to the sought value,
then that is the position, else the same process continues to lower half or
upper half to finally arrive at the position. The method reduces the number
of elements to exactly half for all iterations to confirm the presence of the
sought value.
Let us study the algorithm for binary search.
6.4.1 Algorithm
The algorithm for binary search is given below.
Binary Search Algorithm
binary_search(A, target, lo, hi)
.......{
if(hi<lo)
return-1//target not found
mid=lo+((hi-lo)/2)
if(A[mid]>target)
return binary_search(A, target, lo, mid-1)//repeat from the start
else if(A[mid]<target)
return binary_search(A, target, mid+1, hi)//repeat from the start
else
return mid // target found
}
The tracing of the binary search algorithm is given below.
Tracing of binary search algorithm
Let the array A[1, 2,...,11]=[0, 5, 13, 19, 22, 41, 55, 68, 72, 81, 98]
Here n =11, lo=1, hi=11, soughtvalue target=55
binary_search(A, target, lo, hi)
........{
if(11<1)
return-1//target not found
mid=1+((11-1)/2)=6
if((A[6]=41)>55)
return binary_search(A, 55, 1, 5)//repeat from the start
else if(A[mid]<target)
return binary_search(A, 55, 7, 11)//repeat from the start
else
return 6//target found
}
From this algorithm we can understand that initially the search value is
compared with the centre element of an array. If condition
"value<arr[middle]" is true, it means, that value may be present only in the
first part of an array. In this case, the second part of an array is discarded
and the search continues in the first part, so as to establish the result.
Examples
The following examples illustrate the binary search methodology.
Example 1: To find 6 in {-1, 5, 6, 18, 19, 25, 46, 78, 102, 114}.
Step 1. (middle element is 19>6): -1, 5, 6, 18, 19, 25, 46, 78, 102, 114
Since 6 is lesser than 19, only the first half is considered.
Step 2. (middle element is 5<6): -1, 5, 6, 18, 19, 25, 46, 78, 102, 114
Since5 is lesser than 6, we are sure 6 is not there in first half.
Step 3. (middle element is 6==6): -1, 5, 6, 18, 19, 25, 46, 78, 102, 114
Here we found the number 6, hence the search is completed.
Example 2: To find 103 in {-1, 5, 6, 18, 19, 25, 46, 78, 102, 114}.
Step 1: (middle element is 19<103): -1, 5, 6, 18, 19, 25, 46, 78, 102, 114
Since19 is lesser than 103, only the second half is considered.
Since 78 is lesser than 103, we are sure 103 is not there in first half.
Since 102 is lesser than 103, we are sure 103 is not there in first half.
Step 4: (middle element is 114>103): -1, 5, 6, 18, 19, 25, 46, 78, 102, 114
Since 103 is lesser than 114, we are sure 103 is not there in the list
given.
Step 5: (searched value is absent): -1, 5, 6, 18, 19, 25, 46, 78, 102, 114
Let us next analyze binary search.
6.4.2 Analysis of binary search
Generally, to find a value in an unsorted array, we must scan through
elements of the array one by one. We need to scan all elements if the
searched value is absent from array. In general, the complexity of such an
algorithm is relative to the length of the array.
Condition changes notably, when array is sorted. If we know it, we can
utilize random access capability very efficiently to quickly find the searched
value. Cost of searching algorithm trims down to binary logarithm of the
array length.
The benefit of this algorithm is that it's complexity depends on the array size
logarithmically in the worst scenario. In general, it means, that algorithm will
do log
2
(n) iterations at most, which is very small in number even for big
arrays. It can be proved very easily, solving following inequality in whole
numbers.
Consider n whole numbers in an array, Hence we know that 1
2
n
iterations
> ,
therefore ) n ( log iterations
2
s . It indicates that binary search algorithm time
complexity is )) n ( (log O
2
.
Let us next discuss the advantages and disadvantages of binary search.
6.4.3 Advantages and disadvantages
Advantages of binary search
Binary search is an optimal searching algorithm using which we can look for
the desired element very efficiently.
Disadvantage of binary search
The algorithm is useful only for the sorted array hence it is important to sort
the unsorted array.
In this section we discussed the concepts of binary search. Let us next
discuss binary tree traversals and its properties.
Activity 2
Consider an array A[] with values [3, 5, 7, 11, 13, 17, 19] and explain the
steps to perform binary search if the search key=13.
7. Binary search is an algorithm for identifying the position of an element
in a ____________ array.
8. Say if the following statement is true or false. To find a value in
unsorted array, we need to scan through only half the elements of an
array.
9. Say if the following statement is true or false. The benefit of binary
search is that its complexity depends on the array size logarithmically.
6.5 Binary Tree Traversals and Related Properties
A binary tree traversal is a finite set of nodes that is either empty or consists
of a root and two disjoint binary trees T
L
and T
R
, which are called the left
and the right sub-tree of the root.
6.5.1 Concepts of binary tree
We already know that the binary search splits the binary tree into the, left
sub-tree and right sub-tree, of the same type. Divide and Conquer
methodology can solve most of the problems regarding binary trees.
Now let us consider a recursive algorithm for calculating the height of a
binary tree. We know that, the height is the longest path from the root to
leaf. Thus we have the following recursive algorithm.
The three typical traversals: pre-order, in-order, and post-order are the most
important Divide and Conquer algorithms of binary trees. All the three
traversals visit the trees root and its left and right sub-tree, hence visiting
the nodes of a binary tree recursively. They only differ in the visit timings:
In the preorder traversal, it visits the root before visiting the left and right
sub-tree.
In the in-order traversal, it visits the root after visiting the left sub-tree but
before visiting the right sub-tree.
In the post traversal, it visits the root after visiting the left and right sub-tree.
6.5.2 Binary tree traversal
Tree structures can be traversed in many different ways, compared to linear
data structures like linked list and one dimensional array, which have only
one logical means of traversal. The following are three main steps that can
be performed starting at the root of a binary tree and the order in which they
are performed defines the traversal type:
1) Performing an action on the current node (referred to as "visiting" the
node)
2) Traversing to the left child node
3) Traversing to the right child node
The names for individual style of traversal came from the location of root
element with respect to the left and right nodes. The two kinds of traversal
are depth-first traversal and breadth-first traversal.
Depth-first traversal Here the traversal starts from the root node. The
traversal always visits a node first and then recursively the left and right sub-
trees of each node.
Breadth-first traversal Here the traversal is done breadth-wise. The nodes
at one level are visited before going to a node at the below level. The
breadth-first traversal starts with the root node and then the nodes at level 1
and so on.
We traverse a binary tree to examine each of its nodes. Many different
binary tree algorithms involve traversals. For example:
o If we wish to count the number of nodes in a tree, we must visit each
node.
o If we wish to find the largest value in each node, we must examine the
value contained in each node.
Example: Consider the binary search tree given in figure 6.5.

Figure 6.5: Binary search tree
The following are the traversal sequences of the binary search tree given in
figure 6.5:
Pre-order traversal:
1) Start at the root node
2) Traverse the left sub-tree
3) Traverse the right sub-tree
The nodes of the tree shown in the figure 6.5 would be visited in the order
D, B, A, C, F, E, G.

In-order traversal:
2) Visit the root node
The nodes of the tree shown in the figure 6.5 would be visited in the order A,
B, C, D, E, F, G.
Post-order traversal:
3) Visit the root node
The nodes of the tree shown in the figure 6.5 would be visited in the order A,
C, B, E, G, F, D.
6.5.3 Applications
Preorder traversal: While slotting in the values into a new tree, traversing a
tree in preorder is a regular way of making an entire copy of a binary search
tree.
In-order traversal: It is normal to use an in-order traversal on a binary search
tree because this will put back values from the underlying set in order,
according to the comparator.
10. ________________________ methodology can solve most of the
problems regarding binary trees.
11. The three typical traversals: _____________, _____________, and
_____________ are the most important Divide and Conquer algorithms
of binary trees.
12. Two kinds of traversal are _____________ and _____________.
6.6 Strassens Matrix Multiplication
In this section, we will discuss the Strassens matrix multiplication algorithm
for asymptotically straight forward tasks like multiplying two numbers and
multiplying two square matrices. At the cost of a slight increase in the
number of additions, this algorithm will decrease the total number of
multiplications performed.
Strassens matrix multiplication algorithm is a Divide and Conquer algorithm
that is asymptotically quicker. The normal method of matrix multiplication of
two n X n matrices takes O( ) operators.
The standard multiplication of two 2 2 matrices takes 8 multiplicators and 4
adders but Stassen demonstrates that same can be achieved by only 7
multiplicators and 18 adders.
Concepts of Strassens matrix multiplication
Let us discuss in detail, the multiplication of 2 2 matrices, which contends
that we need only seven multipliers as opposed to the eight required by the
Brute-Force algorithm. This is achieved by the following formulas.

Assume A and B as two n x n matrix, where n is a power of two; matrices
are padded with zeroes for both rows and columns, if n is not the power of
two. We can separate A, B and their product C into four n/2 x n/2 each as
follows,

It is not difficult to verify that one can treat these sub-matrices as numbers to
get the correct product. For example, can be computed either as *
+ * or as where and .are
found by Strassens formulas, with the numbers replaced by the
corresponding sub-matrices.
13. At the expense of a slight increase in the number of additions, the
strassens matrix multiplication algorithm will _____________ the total
number of multiplications performed.
14. The normal method of matrix multiplication of two n X n matrices takes
___________ operators.
15. By the strassens matrix multiplication algorithm, two 2 X 2 matrices
takes 2 only 7 multiplicators and _______ adders.
6.7 Summary
Divide and Conquer is a general design methodology that solves a
problems instance by splitting it into more than a few instances. These
instances are ideally of identical size, solving each of them recursively, and
then merging their solutions to get a solution to the original instance of the
problem. A lot of efficient algorithms are built on this methodology.
Mergesort is a sorting technique that works by splitting an input array into
two, sorting them recursively, and then combining the two sorted parts to get
the original array sorted. Mergesort requires O(n log n) time in all cases.
Quicksort breaks a given array into two non-empty sub arrays such that
every key in one of the two array is less than or equal to every key in
another array. The best case, average case complexity of quick sort is
O(n log n). The worst case complexity is O(n
2
).
Binary search can be performed only on sorted arrays. It is a perfect
example of the Divide and Conquer methodology since it needs to solve just
one problem of half the size of an array for all iterations.
Strassens algorithm needs only seven multiplications to multiply 2X2
matrices but requires more additions than the definition-based algorithm.
6.8 Glossary
Term Description
Binary The binary numeral system, or base-2 number system, represents
numeric values using two symbols, 0 and 1.
Node A point of intersection of the branches of the tree
1. How is Divide and Conquer a better method for sorting?
2. What is the procedure to perform mergesort?
3. Discuss the recursion step to compute quicksort using Divide and
Conquer strategy.
4. Explain the application of binary tree traversal.
5. How can the Strassens matrix multiplication algorithm make the
multiplication of large numbers easier?
6.10 Answers
1. Divide, Conquer
2. Mergesort
3. Divide, recur, conquer
4. Partition
5. Value
6. Left and right
7. Sorted
8. False
9. True
10. Divide and Conquer
11. Pre-order, in-order, post-order
12. Breadth-first traversal, depth-first traversal
13. Decrease
14. O( )
15. 18
Terminal Questions
1. Refer section 6.1 Introduction
2. Refer subsection 6.2.3 Analysis of mergesort algorithm
3. Refer subsection 6.3.2 Algorithm
4. Refer subsection 6.5.3 Applications
5. Refer subsection 6.6 Strassens Matrix Multiplication
References
- Puntambekar (2008). Design and Analysis of Algorithms. Technical
Publication, Pune
- Anany Levitin (2009). Introduction to Design and Analysis of Algorithms.
- Mohan Chandra (2008). Design and analysis of Algorithms. Prentice
Hall Publishers, New Delhi
- O. Tonev, Dimov (1994). Advances in Parallel Algorithms. IOS Press.
Netherlands
E-References
- www.cise.ufl.edu/class/cot3100fa07/quicksort_analysis.pdf
- www.wally.cs.iupui.edu/csci240/files/t23AMergesortAlgorithm.ppt
- http://webcache.googleusercontent.com
Unit 7 Decrease and Conquer
Structure:
7.1 Introduction
Objectives
7.2 Concepts of Decrease and Conquer
7.3 Insertion Sort
Algorithm
Best, worst and average cases
Advantages of insertion sort
7.4 Depth-First search
Efficiency of Depth-first search
Application of Depth-first search
7.5 Breadth-First Search
Application of Breadth-first search
7.6 Topological Sorting
Algorithm
Uniqueness
7.7 Algorithm for Generating Combinatorial Objects
Generating permutations
7.8 Summary
7.9 Glossary
7.11 Answers
7.1 Introduction
By now, you should be familiar with the divide and conquer algorithm. This
unit explains the concepts of decrease and conquer and the methodology it
uses in various algorithms.
Decrease and conquer is a method by which we find a solution to a given
problem based upon the solution of a number of problems. The principle
idea of decrease and conquer algorithm is to solve a problem by reducing its
instance to a smaller one and then extending the obtained solution to get a
solution to the original instance. Here, the problem instance is reduced by
decreasing its size to a constant. So, the difference between decrease-and-
conquer and divide-and-conquer is that it makes the problem instance
smaller by decreasing its size by a constant, while divide-and-conquer
usually divides the problem into a number of smaller but similar sub-
problems.
Objectives:
define decrease and conquer methodologies
explain the algorithm used for insertion sort, Depth-first search, Breadth-
first search and topological sorting
analyze the algorithm for generating combinatorial objects
7.2 Concepts of Decrease and Conquer
Let us study the basics of decrease and conquer.
Decrease and conquer is a concept wherein larger solutions for problems
are broken down based upon the solution to a number of smaller problems.
The solution to the original instance is found out by extending the solution of
the smaller instance. Decrease and conquer can be implemented by a top-
down or a bottom-up approach. It is also known as incremental approach.
The three main distinctions in decrease and conquer are:
1) Decrease by a constant In this kind of method, the size of the
instance is made smaller by the same constant in all iterations. The
constant is generally equal to one. For example, x
10
can be computed
as:
x
10
=x
9
.x
The general formula for this is:
x
p
=x
p-1
.x
If the function a(p) = b
p
, then we can use a top-down recursive approach
and express this as:
a(p) = a(p-1). b if p>1
and a(p) = b if p=1
Using bottom up approach we multiply b by p-1 times of b. We can say
that:
a(p) = b(p-1) times b

A few applications of decrease by constant are:
Graph traversal algorithms (DFS and BFS)
Topological sorting
Algorithms for generating permutations, subsets
2) Decrease by a constant factor The size of a problem instance is
reduced by some constant factor in all iterations of the algorithm.
For example: we can compute x
8
=x
.4.
x
4
If we consider this formula as an example:
x(n) =(x
p/2
)*(x
p/2
) if p is even eg: 2
8
= 2
8/2
* 2
8/2

x(n) = (x
(p+1/2)
)* (x
(p-1/2)
) if p is odd eg: 2
9
= 2
10/2
* 2
8/2
= 2
5
* 2
4

x(n) = x if p =1
The application of decrease by a constant factor is found in binary
search where an element is divided into two sub-arrays and only one
sub-array would be considered to sort the numbers. Hence, the problem
instance is reduced to half the size here.
3) Variable size decrease method - In variable size decrease method, the
outline of size reduction will vary from one algorithm to another. Euclids
algorithm is one of the examples for this method.
1. Decrease and conquer can be implemented by a ________________
or ____________ approach.
2. Decrease and conquer is also known as ______________ approach.
3. Decrease and conquer is a method by which we find a solution to a
given problem based upon the __________ of a number of problems.
7.3 Insertion Sort
In the previous section, we studied about the concepts of decrease and
conquer. In this section we will study about insertion sort, its working and
also the advantages of insertion sort.
Insertion sort is a simple algorithm that implements the decrease and
conquer methodology. Insertion sort executes in O(n
2
) time, but its about
twice as fast as the bubble sort and somewhat faster than the selection sort
in normal situations. Its also not too complex, although its slightly more
complex than the bubble and selection sorts. Its often used as the final
stage of more sophisticated sorts, such as quick sort.
There are numerous approaches to sort an unsorted array in insertion sort
like, start sorting the array from:
The left
The right
Any desired place where the marker is placed.
7.3.1 Algorithm
The basic operation of the insertion sort algorithm is the comparison A[j]>v
because the working of the algorithm will be much faster than the computed
results.
Algorithm for Insertion Sort
ALGORITHM Insertion sort(A[0..n-1])
//sorts a given array by insertion sort
//Input: An array A[0.n-1] of n orderable elements
//Output: Array A[0n-1] sorted in non decreasing order
//Let the array contain 6 elements
For i 1 to n-1 do
v A[i]
j i-1
While j 0 and A[j] > v do
A[j+1] A[j]
j j-1
A[j+1] v
Let us now trace the algorithm.
Algorithm Tracing for Insertion Sort
//Let us consider n=6 and A[n] = {12, 16, 8, 4, 2, 9}
For i 1 to 6-1 do
v 16 // A[1]
j 1-1//j=0
While 0 0 and A[0] > 0 do // condition is true as j = 0 and A[0] = 12
A[0+1] A[0] //A[1]=12
j j-1// j = -1
A[0] 16
The number of key comparisons in this algorithm depends on the type of
input given. In the worst case, A[j]>v is executed the most number of times.
For every j = i-1 to 0.Since v=A[i], it happens if and only if A[j] > A[i] for j = i-1
to 0 For the worst case input, we get A[0]>A[1] (for i=1), A[1]>A[2] (for
i=2)..A[n-2]> A[n-1] (for - =n-1). Hence, we can conclude that worst case
input array will consist of decreasing values.
Example
Let us consider an example of cricket players lined up in random order to
understand the significance of marker. Each one has a jersey with the
numbering from 1 to 11, and that they are required to stand according to the
numbers. Its easier to think about the insertion sort if we begin in the middle
of the process, when the team is half sorted.
We also call this marker technique as partial sorting in stages. At this point
theres an imaginary marker somewhere in the middle of the line. Let us
consider the players to the left of this marker are partially sorted. This
means that they are sorted among themselves; each one has a jersey with
smaller number than the person to his or her left. However, the players
arent necessarily in their final positions because they may still need to be
moved when previously unsorted players are inserted between them. The
player where the marker is whom we will call the marked player, and all the
players on his or her right, is as yet unsorted. To have a clear picture let us
consider the figure 7.1 which represents the jersey numbers of the players,
that is, {1, 10, 2, 4, 6, 8, 5, 11, 3, 7, 9}.

Figure 7.1: Insertion Sort
Similarly if we can choose to perform insertion sort either from the right or
left end of the array, we will have to position the marker to the last
player/last number.
7.3.2 Best, worst and average cases
The best case input is an array that is already sorted. In this case insertion
sort has a linear running time (i.e., O(n)). Each cycle, the first remaining
element of input is compared with the right-most element of the sorted sub-
section of the array.
The worst case input is an array sorted in reverse order.
In this case every cycle of the inner loop examines the entire sorted sub-
section of the array before inserting the next element. For this case insertion
sort has a quadratic running time (i.e., O(n
2
)).
The average case is also quadratic, which makes insertion sort unrealistic
for sorting large arrays. However, insertion sort is the fastest algorithm for
sorting arrays containing fewer than ten elements.
7.3.3 Advantages of insertion sort
The following are the advantages of insertion sort:
It is simple to implement.
The method is useful when sorting smaller number of elements.
It is more efficient than other algorithms.
It is a very stable algorithm.
It is easy to understand.
Hope you are clear about insertion sort and its working. Let us move on
to the next section.
4. There are ________ major categories in insertion sort.
5. In insertion sort, the best case input is an array that is already
__________.
6. To carry out an insertion sort, begin at the ______ most element of the
array.

7.4 Depth-First Search
In this section we will study another decrease and conquer algorithm -
Depth-first search. We will study the Depth-first search algorithm and its
efficiency.
Depth-first search (DFS) is an algorithm for traversing graphs, and trees.
One begins at the root (selecting some node as the root in the graph case)
and then explore along each branch. Depth-first search works by
considering vertices, checking its neighbors, extending the first vertex it
finds across its neighbors, checking if that extended vertex is our
destination, and if not, continue exploring more vertices.
Depth-first search starts exploring vertices of a graph at a random vertex
and marks the vertices explored as visited. In each step, the algorithm
moves to an unvisited vertex that is adjacent to the one recently visited. If
there are many adjacent unvisited vertices, a tie would be solved randomly.
The data structure which depicts the graph indicates which unvisited
vertices are chosen. In our example we will always cut off ties in
alphabetical order. This procedure will continue until a vertex which has no
adjacent vertices is found. After this, the algorithm comes back to one edge
of vertex that it came from and tries to go to other unvisited vertices from
there. The algorithm finally stops after going back to the starting vertex. By
now, all vertices in the same connected section as the starting vertex have
been visited. If any unvisited vertices are found, Depth-first search must be
begun again at any one of these vertices.
DFS takes out a vertex from the stack when it reaches the end. Figure 7.4
shows a simple Depth-first search traversal with vertices and their
respective edges.

Example: Let us consider a sub-tree for performing Depth-first search as
given in the table 7.1.
Table 7.1: Depth-First Search

Step 1: Here we can see that vertex P will be visited
first and marked. Now our list contains only P.

Step 2: The tree will be searched in depth that is
the sub-trees corresponding to vertex P are
searched. Hence, vertex Q is visited and marked.
Now our list contains P, Q

Step 3: Here we visit vertex R and mark it. After
this, sub-tree for vertex R is checked. Since there is
no sub-tree present for vertex R, the algorithm
comes back to vertex Q. Here, our list would remain
as P, Q, and R

Step 4: Here the vertex S is visited, since it is a
sub-tree of Q. Now, our list contains P, Q, R, S.

Step 5: As there is no sub-tree present for vertex S,
algorithm traces back to vertex Q, and then back to
vertex P. The algorithm then visits vertex T and
marks it as visited. Now, our list contains P, Q, R,
S, T.

Step 6: Here we can see that vertex U is visited
since it is a sub-tree of T and the final list contains
P, Q, R, S, T, and U. Then, the algorithm traces
back to vertex T, and then to P again.

The following pseudo code explains the Depth-first search algorithm.
Pseudo code for Depth-first Search
DFS (G)
// Implements a Depth-first search traversal of a given graph
// Input: Graph(G) = V(E)
// Output: Graph G with vertices marked with consecutive integers
// In the order they have been found by the DFS traversal
Mark each vertex in V as 0 as an indication of being unvisited
Count 0
For each vertex v in V do
If v is marked with 0
dfs(v)
// visits recursively all the unvisited vertices connected to vertex v by a
path
// and numbers them in the order they are met
// via global variable count
Count count+1; mark v with count
For each vertex w in v adjacent to v do
If w is marked with 0
dfs(w)
7.4.1 Efficiency of Depth-first search
The Depth-first search algorithm is quite efficient because it just takes time
proportional to the size of the data structure which is used for depicting the
data structure for the question here. The traversal time is O(v
2
) for a tree; a
depth-first search may take an excessively long time to find even a very
nearby goal node. There are two loops in the algorithm; the first one loops
through all vertices and the second loops through all neighbors of a
particular vertex. All other operations performed within these loops, such as
changing times, are considered O(1). The time spent on assignment
statements in the outer loop is O(|V|). The loop through all neighbors of
vertices takes a check at other sub-trees A useful aspect of the DFS
algorithm is that it traverses through the connected components visiting one
at a time, and thus it can be used to identify the connected components in a
given graph.

Therefore the total complexity is O(|V| + |E|), which is optimal, as mentioned
in the introduction. V and E are the number of graph vertices, and edges
respectively.
7.4.2 Applications of Depth-first search
We can use the Depth-first search algorithm for the path finding application.
In the path finding application we can find the path between vertices a and
b. Here we use a stack S to keep track of the visited nodes. While we
traverse and reach b, the stack returns all its elements to form a path from a
to b.
Depth-first search algorithm is commonly used for the following:
To find the path from one vertex to another
To solve connected or unconnected graphs
To compute a spanning tree for a connected graph by using a
backtracking technique
To check connectivity and acyclicity of graphs
To find the articulation point
7. DFS uses the ______________________ technique.
8. It is easier to use a ________________ to trace the working of a depth-
first search.
9. Depth-first search starts exploring vertices of a graph at a
________________ vertex.
7.5 Breadth-First Search
In the previous section we studied about the Depth-First Search algorithm.
In this unit we will study about the Breadth-first search algorithm and its
working.
Breadth-first search (BFS) is an algorithm which travels in such a way that
all vertices are visited along every sub-tree and which are adjacent to the
starting vertex, then all the unvisited vertices will be visited which are a part
of it. This process continues until all vertices in the same section as the
starting vertex are visited. If unvisited vertices remain, the algorithm will
begin considering any random vertex which is connected to the graph.
The data structure which is used to track the working of Breadth-first search
is a queue. The table 7.2 depicts the working of Breadth-first search. Here,
Breadth-first search is performed level by level.
Table 7.2: Breadth-First Search

Step 1: Here the vertex P would be marked
first, here vertex P is considered to be at the
first level. Our list now has only P.

Step 2: The vertex Q would be visited and
marked. Here, vertex Q is at second level. Now
our list contains P, Q.

Step 3: Here vertex S would be visited and
marked. Here, vertex S is at second level of the
tree. After marking vertex S, the algorithm
returns back to the initial vertex

Step 4: The vertex R is visited which is at the
third level. Our final list would consist of P, Q,
R, and S.

The total running time of Breadth-first search algorithm is given as O(V+E).
Where V is the set vertices and E is the set of edges.
Let us now discuss the applications of Breadth-first search algorithm.

7.5.1 Application of Breadth-first search
Breadth-first search is commonly used for the following:
To test the connectivity of a graph
To compute the spanning forest of a graph
To check for and compute cycles of a graph
To check for a path with minimum number of edges between the start
and the end vertex
To check connectivity and acyclicity of graphs

Activity 1
Draw a Breadth-first search graph traversal for the Depth-first search figure
given.
10. The data structure which is used to track the working of Breadth-first
search is a ___________.
11. Breadth-first search is an algorithm which travels in such a way that all
vertices are finished along every _____________.
12. The data structure which is used to track the working of Breadth-first
search is a ______________.
7.6 Topological Sorting
In the previous section, we studied about the Breadth-first search algorithm
and its working. In this section we will study about topological sorting, its
examples and also the uniqueness which it possesses.
Topological sort is done using a directed acyclic graph (DAG), which is a
linear ordering of all vertices G= (V, E) is an ordering of all vertices such that
if G contains an edge (u, v), then u appears before v in the ordering. A
topological sort of a particular graph can be looked upon as a horizontal line
where all directed edges travel from left to right. Thus, topological sort
differs from the usual sorting technique. Linear ordering cannot be
performed for cyclic graphs. Topological sorting can be used for scheduling
a sequence of jobs or tasks. The jobs are denoted by vertices, and there is
an edge from a to b, if job a must be completed before job b can be started
(for example, when starting a car, the ignition must be started first).
Therefore a topological sort gives an order in which to perform the jobs.
Directed acyclic graphs are used in many instances. They usually indicate
precedence among events. Topological sorting can be used to arrange
tasks under precedence constraints. Suppose we have a set of tasks to do,
but particular tasks have to be performed before other tasks, we can use
these precedence conditions to form a directed acyclic graph (DAG)., Any
topological sort (also known as a linear extension) describes an order to
perform these tasks such that each task is performed only after all of its
conditions are satisfied.

Example
Let us consider a set of five required assignments that a student has to
complete which are named as S1, S2, S3, S4, and S5. The assignments
should be completed within some time as long as certain conditions are met.
S1 and S2 have no conditions to be met. S3 requires S1 and S2 to be
completed. S4 requires S3 to be completed and S5 needs S3 to be
completed. The student can complete only one assignment in a day. The
order in which the student takes up a course is the main concern here.
Figure 7.2 illustrates the assignments .of the students represented as a
DAG.

Figure 7.2: The Five Assignments of Students Represented by a DAG
In the above figure, the vertices represent the assignments to be completed
by the student and the directed edges show the pre-required conditions to
be met. The main concern here would be whether ordering of vertices
should be possible in such a way that every vertex where the edge begins
must be listed before a vertex where the edge closes. This concept is called
as topological sorting. If the di-graph has a directed cycle present, the
problem will not have a solution.
7.6.1 Algorithm
One of the algorithms for topological sort was first described by Kahn
(1962). It proceeds by selecting vertices in the same order as the eventual
topological sort. For topological sorting to be performed a graph should be a
DAG.
We can see that a DAG also need not necessarily perform topological
sorting. If a graph has no directed cycles, topological sorting for that case
will have a solution. Depth-first search algorithm can be used to apply
topological sorting. We can note how the traversal is performed here. We
have to note how the traversal is performed until it reaches the dead end.
An algorithm which can be applied here uses the decrease by one
technique. Here; we identify a vertex which has no incoming edges, and
remove all the edges which are coming out of it. These steps have to be
checked visually in order to ensure that topological sorting is correctly
performed.
Small examples of topological sorting might prove to be wrong sometimes.
But, if we consider situations where big projects are involved, the pre-
specified conditions must be taken care such that they occur in sequence
and the project is successfully completed. For this to happen, topological
sorting must be performed using a di-graph. After this step, we can decide
about the situations that occur while performing topological sorting.
First, find a list of start nodes which doesnt have incoming edges and insert
them into a set A; at least one such node must exist if graph is acyclic.
Let us now discuss the pseudo code for topological sorting.

Pseudo code for topological sorting
//L Empty list that will contain the sorted elements
//S Set of all nodes with no incoming edges
While S is non-empty do
remove a node n from S
insert n into L
for each node m with an edge e from n to m do
remove edge e from the graph
if m has no other incoming edges then
insert m into S
if graph has edges then
output error message (graph has at least one cycle)
else
output message (proposed topologically sorted order:L)
If the graph was a DAG, a solution is contained in list L, else, the graph has
at least one cycle and therefore a topological sorting is not possible.
7.6.2 Uniqueness
If a topological sort has the property that every pair of consecutive vertices
in the sorted order is linked by edges, then these edges form a directed
Hamiltonian path A Hamiltonian path, also called a Hamilton path, is a path
between the vertices of a DAG that visits each vertex exactly once. If a
Hamiltonian path is present, the topological sort order is unique. On the
other hand, if a topological sort does not form a Hamiltonian path, the DAG
will have two or more valid topological orderings, in this case it is always
possible to form a second valid ordering by swapping two consecutive
vertices that are not connected by an edge to each other. Therefore, it is
possible to examine in polynomial time whether a unique ordering exists,
and whether a Hamiltonian path exists.
13. Topological ordering of a _____________ is a linear ordering of its
nodes.
14. In topological sorting the jobs are denoted by _________.
15. Being a DAG is also a necessary condition for ______________sorting
to be possible.
7.7 Algorithms for Generating Combinatorial Objects
In the previous section, we studied about topological sorting and its
uniqueness. In this section we will explain the algorithms for generating
combinatorial objects.
The most essential types of combinatorial objects are permutations and
combinations and subsets of a particular set. Combinatorial objects are
given importance in a mathematical subject called as combinatorics.
They usually arise in problems that require a consideration of different
choices such as Travelling Salesman problem (TSP), Knapsack problem
and so on.
To solve these problems we need to generate combinatorial objects.
7.7.1 Generating permutations
The set of numbers to be permuted are the number of integers which range
from 1 to n.
Let us presume the set whose elements need to be permuted is the set of
integers from 1 to n. They can be interpreted as indices of elements in an
n-element set {a
i
a
n
}
Now, let us see how a decrease-by-one technique solves the problem of
generating all n! Permutations:
Approach:
1. The smaller-by-one problem is to generate all (n-1)! Permutations.
2. Assume that the smaller problem is solved.
3. We can get a solution to the larger problem by inserting n in each of the
n possible positions among elements of every permutation of n-1
elements. The order of insertions depends on the total number of all
permutations which is n.(n-1) ! = n!
We can insert n in the previously generated permutations:
left to right
right to left

start 1

insert 2 12 21
right to left

insert 3 123 132 312 321 231 213
right to left left to right

Example: Traveling salesman problem
A difficulty which is closely connected to the topic of Hamiltonian circuits is
the Traveling-Salesman problem stated as follows: A sales person is
required to go to a certain number of cities in a trip. The distances between
cities are provided. The order in which he has to travel to every city and
return home with minimum distance covered to each city is the main
concern here.
The cities are represented by vertices and the roads between them are
shown as edges. A graph is obtained by doing this. In this graph, with every
edge s
i
there is a real number associated with it. The distance is measured
in miles here. This graph is called as a weighted graph; w(s
i
) being the
weight of edge s
i
.
If each of the cities is connected to another city by a road a complete
weighted graph is obtained. The graph has many Hamiltonian circuits, and
we must choose the one that has the smallest sum of distances or weights.
The total number of different Hamiltonian circuits excluding the disjoint ones
in a complete graph of x vertices can be shown to be (x-1)! / 2. This shows
from that starting from any vertex we have x-1 edges to pick from the first
vertex, x-2 from the second, x-3 from the third, and so on. These being
individual, results with (x-1)! choices. However, this number is, divided by 2,
because each Hamiltonian circuit is counted twice.
Theoretically, the traveling salesman problem can always be solved by
specifying all ((x-1)!)/2 Hamiltonian circuits, finding out the distance traveled
in each, and then choosing the shortest one. However, for a large value of x,
the cost and time taken is very high even for a digital computer.
The problem is to lay down a convenient algorithm for computing the
shortest route. Although many attempts have been made, there is no
efficient algorithm for problems of random size that has been found till date.
Since this problem has applications in operations research, some specific
large-scale examples have been worked out. There are also few methods
available that are suggested by expert mathematicians, but the shortest
possible path which is perfect has yet to be invented

Figure 7.3: Travelling Salesman Problem with 4 Cities
Figure 7.3 shows how travelling salesman problem is solved between four
cities P, Q, R, and S. The number above the edges indicates the distance
between the cities. Here, distance between two cities is same in each
direction which forms an undirected graph The minimum cost for the cities
travelled would be P->R->S->Q->P= 79.
Hope you are clear about algorithms for generating combinatorial objects.
16. Theoretically, the traveling salesman problem can always be solved by
specifying all _____________ Hamiltonian circuits.
17. The cities are represented by ____________ and the roads between
them are shown as edges.
18. Each of the cities is connected to another city by a road a complete
_____________________ is obtained.

7.8 Summary
Let us summarize what we have studied in this unit.
Decrease and conquer is a method by which we find a solution to a given
problem based upon the solution of a number of smaller problems. The
principle idea of decrease and conquer algorithm is to solve a problem by
reducing its instance to a smaller one and then extending the obtained
solution to get a solution to the original instance.
Insertion sort is one of the simplest algorithms that is implemented under the
decrease and conquer methodology. In this technique the element is
inserted at the suitable position.
Depth-first search (DFS) is an algorithm for searching a tree, tree structure,
or graph. We begin at the root and then explore along each branch.
Breadth-first search (BFS) is a graph search algorithm that begins at the
root node and explores all the neighbouring nodes. Then for each of those
nearest nodes, it explores their unexplored neighbour nodes, and so on,
until it finds the goal.
A topological sort of a particular graph can be looked upon as a horizontal
line where all directed edges travel from left to right. Thus, topological sort
differs from the usual sorting technique. Topological sorting can be used for
scheduling a sequence of jobs or tasks.
Algorithms for generating combinatorial objects define most important types
of combinatorial objects such as permutations and combinations.
7.9 Glossary
Term Description
Decrease-by-one technique Type of decrease and conquer, which can also be
termed as Decrease by constant.
Combinatorics Mathematical subject which involves solving
problems using figures.

1. Define the concept of decrease and conquer algorithm?
2. What are the best, worst and average cases in an insertion sort?
3. What are the major advantages of insertion sort?
4. Explain the working of DFS algorithm with example
5. Describe BFS algorithm.
6. Explain the travelling salesman problem in brief.
7.11 Answers
1. Top down or Bottom-up
2. Incremental
3. Solution
4. Two
5. Sorted
6. Left
7. Backtracking
8. Time and space
9. Backtracks
10. Graph
11. Uninformed
12. Goal
13. Directed acyclic graph
14. Vertices
15. Topological
16. Combinatorial
17. Subset
18. Decrease-by-one
Terminal Questions
1. Refer section 7.2 Concepts of Decrease and Conquer
2. Refer section 7.3.2 Best, worst and average cases
3. Refer section 7.3.3 Advantages of insertion sort
4. Refer section 7.4- Depth-first search
5. Refer section 7.5 Breadth-first search
6. Refer section 7.7.1 Generating permutations
References
Puntambekar, A. A. (2008). Design and Analysis of Algorithms, First
Anany Levitin (2008). The Design & Analysis of Algorithms, Second
edition, Pearson.
Pandey, Hari Mohan (2008). Design Analysis and Algorithm, First
edition. University science press, New Delhi.
Thomas H. Cormen (2001). Introduction to Algorithms, Second edition.
Mc Graw Hill.
E-References
http://www.cs.sunysb.edu/~algorith/files/topological-sorting.shtml

Unit 8 Transform and Conquer
Structure:
8.1 Introduction
Objectives
8.2 Presorting
8.3 Gaussian Elimination
Basics of Gaussian elimination
8.4 Balanced Search Trees
Fundamentals of balanced search trees
AVL trees
2-3 trees
8.5 Heaps and Heapsort
Max and min heaps
Architectural approach of heaps and algorithms
Heapsort
8.6 Problem Reduction
Computing the least common multiple
Counting paths in graphs
8.7 Summary
8.8 Glossary
8.10 Answers
8.1 Introduction
By now you must be familiar with the concepts of decrease-and-conquer
technique. In this unit we deal with a group of design methods that are
based on the idea of transformation. We call this general technique
transform-and-conquer. It is called so because these methods involve a two-
stage procedure.
A. Transform Here the problems instance is changed to a level where
the solution to the problem can be obtained easily.
B. Conquer In this stage the problem is solved.

The three variations of the transform and conquer approach which is
illustrated in figure 8.1 are:
1) Instance simplification: It is the transformation to a simpler and more
suitable instance of the same problem, such as
- presorting
- Gaussian elimination
2) Representation change: It is the transformation to a different
representation of the same instance, such as
- balanced search trees
- Horners rule fast exponentiation
- heaps and heapsort
3) Problem reduction: It is the transformation to an instance of a different
problem for which an algorithm is already available, such as
- computing least common multiple (lcm)
- counting paths in graphs

Figure 8.1: Transform and Conquer Strategy
Objectives:
- define the technique of transform and conquer
- describe the method of presorting
- explain Gaussian elimination technique
- explain the approaches of AVL and 2-3 trees in balanced search trees
- define heap and apply it to heapsort
- apply the problem reduction strategy

Solution
Problems
instance
Simpler instance
or
Another representation
or
Another problems instance
8.2 Presorting
Presorting is sorting before performing any operation. The time efficiency of
algorithms that involve sorting depends on the efficiency of the sorting
algorithm being used. The time spent in presorting should be offset by the
advantages of presorting.
Let us first define presorting.
Presorting is a process of preprocessing data in order to improve it so that it
can later be refined in a better way by any sorting algorithm.
Let us now consider the application of presorting to compute the mode.
Example: Computing a mode
A mode is a value that has the most number of occurrences in a given list of
numbers. For example, for the following list of numbers - 9, 3, 5, 9, 4, 9, 7,
8, 9, the mode is 9. (If you find that several different values occur most
often, then you can consider any of them as the mode.)
In the brute force approach to computing a mode, you would scan the list
and compute the frequencies of all its distinct values, then find the value
with the largest frequency. Whereas in order to implement this idea here,
you will have to store the values already encountered, along with their
frequencies, in a separate list. In all iterations, compare the i
th
element of the
original list with the values already encountered by traversing the separate
list. If a matching value is found, increment its frequency; otherwise, add the
current element to the list of distinct values seen so far with the frequency of
1. Hence considering the previous list of numbers 9, 3, 5, 9, 4, 9, 7, 8, 9,
let us sort the list first. We get the sorted list as 3, 4, 5, 7, 8, 9, 9, 9. Here
we can clearly see that 9 has the highest frequency, and therefore it is the
mode.
The worst case scenario that you are most likely to encounter here is to
encounter a list with no equal elements. For such a list, its i
th
element is
compared with i1 elements of the other list which stores the frequency of
distinct values seen so far before being added to the list with a frequency of
1. The worst-case number of comparisons made by this algorithm in
creating the frequency list is given in equation Eq: 8.1.
2
1
( 1)
( ) ( 1) 0 1 ( 1) ( )
2
n
i
n n
C n i n n
=
= = + + + = eO
Eq: 8.1
The additional n1 comparisons needed to find the largest frequency in the
auxiliary list do not change the quadratic worst-case efficiency class of the
algorithm. Hence we first sort the input. Then all equal values will be
adjacent to each other. To compute the mode, all we need to do is to find
the longest run of adjacent equal values in the sorted array.
Let us now discuss the algorithm Presort Computing Mode which presorts
an array of numbers and then computes the mode.
Algorithm: Presort Computing Mode (A [0...n 1])
//Computes the mode of an array by sorting it first
//Input: An array A [0...n 1] of orderable elements
//Output: The arrays mode
Sort the array A
i 0 //current run begins at position i
modefrequency 0 //highest frequency seen so far
while i n 1 do
runlength 1; runvalueA[i]
while i + runlength n 1 and A [i + runlength] = runvalue
runlengthrunlength+1
if runlength> modefrequency
modefrequency runlength; modevalue runvalue
i i + runlength
return modevalue
Let us now trace the algorithm to compute the mode using presort.
Algorithm Tracing for Computing Mode Using Presort
A[]=[1,3,3];
n=2;
while 0<= 2-1 do
runlength=1;
runvalue=A[0];
while 0+1<=2-1 and A[0+1]=A[0]
runlength=1+1
if 2>0
modefrequency=2; //modefrequency value changes from 0 to 2
modevalue=2; //modevalue variable is assigned the value 2
i=0+2 //value of i changes to 2
return 2 //modevalue has been assigned the value 2
1. _______, ________ & ________ are three instances of transform and
conquer technique.
2. The worst case efficiency of brute force algorithm is _______.
3. The searching requires _______ comparisons to in the worst case,
when the array is sorted first.
8.3 Gaussian Elimination
In the previous section we learnt about presorting. In this section we will
learn about Gaussian elimination which is another example of instance
simplification.
The algorithm of Gaussian elimination is used for solving systems of linear
equations, calculating the inverse of an invertible square matrix, and finding
the rank of a matrix.
8.3.1 Basics of Gaussian elimination
For many applications, we need to solve a system of n equations in n
unknowns as shown in the set of equations in Eq: 8.2
a
11
x
1
+ a
12
x
2
+ + a
1n
x
n
= b
1

a
21
x
1
+ a
22
x
2
+ + a
2n
x
n
= b
2

.
.
.
a
n1
x
1
+ a
n2
x
2
+ + a
n
x
n
= b
n
Eq: 8.2
Here n is a large number; hence to solve such systems of linear equations
we use the Gaussian elimination technique. The idea of Gaussian
elimination is to transform a system of n linear equations with n unknowns to
an equivalent system (i.e., a system with the same solution as the original
one) with an upper triangular coefficient matrix, a matrix with all zeros below
its main diagonal.

The system can be written as a matrix notation as shown in Matrix: 8.1.

11 12 1 1 1
21 22 2 2 2
1 2
n
n
n n nn n n
a a a x b
a a a x b
a a a x b
( ( (
( ( (
( ( (
=
( ( (
( ( (

Matrix: 8.1
To perform Gaussian elimination on a system you will first have to augment
the matrix. Your augmented matrix should look like Matrix: 8.2.

11 12 1 1 1
21 22 2 2 2
1 2
n
n
n n nn n n
a a a b x
a a a b x
a a a b x
( (
( (
( (
( (
( (
(

Matrix: 8.2
Now perform row operations so as to make the elements below the main
diagonal of the matrix to 0. This results in Matrix 8.3.

' ' ' ' '
11 12 1( 1) 1 1
' ' ' '
22 2( 1) 2 2
' '
( 1)( 1) ( 1)
' '
0
0 0
0 0 0
n n
n n
n n n n
n nn
a a a a b
a a a b
a a
b a

(
(
(
(
(
(
(

Matrix: 8.3
Solve the equation of the n
th
row for x
n
, then substitute back into the
equation of the (n-1)
st
row to obtain a solution for x
n-1
and so on.
Let us now discuss the algorithm for Gaussian elimination.
Algorithm: Gauss elimination (A [1...n, 1...n], b [1...n])
//Applies Gaussian elimination to matrix A of a systems coefficients,
//augmented with vector b of the systems right-hand side values.
//Input: Matrix A[1..n, 1,..n] and column-vector b[1..n]
//Output: An equivalent upper-triangular matrix in place of A with the
//corresponding right-hand side values in the (n + 1)st column.
for i 1 to n do A[i, n + 1]b[i] //augments the matrix
for i 1 to n 1 do
for j i + 1 to n do
for ki to n + 1 do
A[j , k] A[j , k] A[i, k] temp
Let us now trace the algorithm for Gauss elimination.
Algorithm Tracing for Gauss elimination n=2
//Applies Gaussian elimination to matrix A of a systems coefficients,
//augmented with vector b of the systems right-hand side values.
//Input: Matrix A[1..2, 1,..2] and column-vector b[1..2]
//Output: An equivalent upper-triangular matrix in place of A with the
//corresponding right-hand side values in the (2 + 1)st column.
for i 1 to 2 do A[1, 3]b[1] //augments the matrix
for i 1 to 1 do
for j 2 to 2 do
for k1 to 2 do
A[2, 1] A[2 , 1] A[1, 1] temp

Activity 1

(
(
(
(
=
7 3 2
3 9 4
2 4 2
A

Use Gaussian elimination to compute the inverse of the above matrix.
4. Gaussian elimination is an algorithm for solving systems of
__________ equations.
5. In Gaussian elimination we make the ________ coefficient matrix zero.
6. Gaussian elimination can be used to find the _________of a matrix.
8.4 Balanced Search Trees
In the previous section we discussed Gaussian elimination. In this section
we will discuss balanced search trees, an instance of representation change
variety of transform and conquer.
Many data structures use binary search trees. Operations on these search
trees are often proportional to the height of the tree. We can guarantee the
efficiency of such operations by ensuring that the height of the tree is
logarithmic in the number of nodes.
8.4.1 Fundamentals of balanced search tree
The binary search trees are one of the principle data structures for
implementing dictionaries. A balanced search tree is a binary tree whose
nodes contain elements of a set of orderable items, one element per node.
Hence all elements in the left sub-tree are smaller than the element in the
sub-trees root and all the elements in the right sub-tree are greater than it.
Thus there is a gain in the time efficiency of searching, insertion, and
deletion, which are all in (log n), but only in the average case. In the worst
case, these operations are in (n) because the tree can degenerate into a
severely unbalanced one with height equal to n 1.
There are two approaches to balanced search trees. The first approach is of
the instance simplification variety, where an unbalanced binary search tree
is transformed to a balanced one. Examples of these are AVL trees and red-
black trees. The second approach is of the representation change variety.
Specific cases of such trees are 2-3 trees, 2-3-4 trees and B-trees. In this
unit, we discuss only AVL and 2-3 trees.
8.4.2 AVL trees
An AVL tree is a binary search tree having a balance factor of every node,
which is defined as the difference between the heights of the nodes left and
right sub-trees, is either 0 or +1 or 1. (The height of the empty tree is
defined as 1.)
For example, the binary search tree in figure 8.2 (a) is an AVL tree but that
in 8.2 (b) is not. The numbers above the nodes indicate the nodes balance
factors.

Figure 8.2: (a) AVL Tree. (b) Binary Search Tree that is not an AVL Tree
If an insertion of a new node makes an AVL tree unbalanced, we transform
the tree by rotation. A rotation in an AVL tree is a local transformation of its
sub-tree rooted at a node whose balance has become either +2 or 2; if
there are several such nodes, we rotate the tree rooted at the unbalanced
node that is the closest to the newly inserted leaf. The rotations can be
classified as single rotations and double rotations.
Single rotations
Single rotations are the rotations performed to balance the AVL tree when a
new node is added. Here a node is rotated only once so as to balance the
tree. They are of two types.
R-rotation The first rotation type is called the single right rotation or
R-rotation. Figure 8.3 presents the single R-rotation in its most general form.
Note that this rotation is performed after a new key is inserted into the left
sub-tree of the left child of a tree whose root had the balance of +1 before
the insertion.
Figure 8.3: R-rotation
L-rotation The symmetric single left rotation or L-rotation is another type
of single rotation. It is the mirror image of the single R-rotation (Refer figure
8.4). It is performed after a new key is inserted into the right sub-tree of the
right child of a tree whose root had the balance of 1 before the insertion.

Figure 8.4: L-rotation
2
1 3
0
0 0

1
2
3
3
-1
-2
0
3
2
1
2
0
1
2
1 3
0
0 0
3
1
2
2
0
-1
2
1 3
0
0 0
2
1
3
-2
0
1
2
1 3
0
0 0
Double rotations
These rotations comprise of single rotation performed twice. They can be
classified as LR rotation and RL rotation.
LR-rotation The double left-right rotation (LR-rotation) is a combination of
two rotations: we perform, the L-rotation of the left sub-tree of root followed
by the R-rotation of the new tree rooted at (Refer figure 8.5). It is performed
after a new key is inserted into the right sub-tree of the left child of a tree
whose root had the balance of +1 before the insertion.

Figure 8.5: L-R-rotation
RL-rotation This is the mirror image of the double LR-rotation. It is
performed after a new key is inserted into the left sub-tree of the right child
of a tree whose root had the balance of -1 before the insertion (Refer figure
8.6).

Figure 8.6: R-L-rotation
The drawbacks of AVL trees are the need to maintain balances for the trees
nodes, frequent rotations, and overall complexity, especially of the deletion
operation.

Let us next discuss the algorithm to balance AVL trees.
Algorithm: AVL balance(x)
// Input: Node x
// Output: x and nodes above it are balanced
while x null do
if x.left.height > x.right.height + 1 then
if x.left.left.height < x.left.right.height
then LR-Rotation(x)
else R-Rotation(x)
else if x.left.height + 1 < x.right.height then
if x.right.left.height > x.right.right.height
then RL-Rotation(x)
else L-Rotation(x)
x x.parent
Let us now trace the algorithm to balance an AVL.
Algorithm Tracing of the Naive Algorithm to Balance an AVL
// Input: Node 2
// Output: 2 and nodes above it are balanced
while 2 null do
if 2.left.height > 2.right.height + 1 then
if 2.left.left.height < 2.left.right.height
then LR-Rotation(2)
else R-Rotation(2)
else if 2.left.height + 1 < 2.right.height then
if 2.right.left.height > 2.right.right.height
then RL-Rotation(2)
else L-Rotation(2)
2 2.parent
8.4.3 2-3 trees
A 2-3 tree is a data structure, which was introduced because the binary
search tree had no guarantee of being balanced when some random
insertion or deletion is made. A 2-3 tree has 2 types of nodes:
o 2-node
o 3-node

2-node

Figure 8.7: 2-Node
The figure 8.7 shows a 2-node structure. It has one data element and two
children. Every 2-node must have the following properties
1. Every value appearing in the child P must be X
2. Every value appearing in the child Q must be X
3. The length of the path from the root of a 2-node to every leaf in its child
must be the same.
3-node

Figure 8.8: 3-Node
The figure 8.8 shows a 3-node structure. It has 2 data elements and 3
children. Every 3-node must have the following properties:
1. Every value appearing in child P must be X
2. Every value appearing in child Q must be in between X and Y
3. Every value appearing in child R must be Y
4. The length of the path from the root of a 3-node to every leaf in its child
must be the same
Properties of 2-3 trees
We will discuss the operations done in a 2-3 tree using figure 8.9 which is a
2-3 tree with numerical keys. As explained earlier, if the key of the child is
smaller than the smallest key of its parent then the child is a left child i.e. it is
placed in the left branch in the sub-tree.
X
Q P
X | Y

P

Q

R


Figure 8.9: 2-3 Tree
Similarly if a child is larger than the largest key of its parent then it is the
right child, and if it is in between it takes the middle sub-tree. Let us now try
to insert a key 28 into the tree in the figure 8.9.

Figure 8.10: Inserting 28 into the Tree
Here in the figure 8.10 we see that the node containing 27 and 29 have
been split open to accommodate 28 which is now in a temporary node.
It should be remembered that it is always the middle value of the tree that is
pushed upwards. The insertion stops as soon as a node with only one key is
reached.

Figure 8.11: Further Splitting the Node Containing 25 and 31
Proceeding with the insertion, in the figure 8.11, we can see that the node
containing 25 and 31 have been split to accommodate 28. On doing this, the
node containing 21 and 23 becomes the left child of 25 and the node with 27
becomes the right child. Similarly 29 becomes the left child of 31 and the
node with 33 and 35 become the right child.

Figure 8.12: Balanced Tree
At this point (Refer figure 8.12) we see that the node with 9 and 19 has been
split. As 28 is greater than 19, it becomes the right child and 9 being smaller
becomes the left child. Here we can see that the tree has four levels, but is
balanced after insertion.
7. An AVL tree is a _________ tree.
8. The _________________ is the mirror image of the RL-rotation.
9. The two nodes of 2-3 tree are ___________ and ____________.
8.5 Heaps and Heapsort
In the previous section we learnt about balanced search trees. In this
section we will see another instance of representation change i.e. heaps
and heap sorts.
Heaps are data structures that are especially suitable for implementing
priority queues. A priority queue is a set of items with an orderable
characteristic called an items priority, with the following operations:
- finding an item with the highest (i.e., largest) priority
- deleting an item with the highest priority
- adding a new item to the set
The heap is the data structure that serves as a foundation of a theoretically
important sorting algorithm called heapsort which we will discuss after we
define the heap.
8.5.1 Max and min heaps
A heap can be defined as a binary tree with keys assigned to its nodes (one
key per node) provided the following two conditions are met:
The trees shape requirement: In a binary tree, all its levels must be full
except possibly the last level, where only some rightmost leaves may be
missing. This requirement is valid for both max and min heaps.

Figure 8.13: Max Heap
The parental dominance requirement for max heap: The key at each
node has to be greater than or equal to the keys at its children i.e.
key(parent) key(child). The figure 8.13 represents a max heap. All the
concepts in this unit utilizes the approach of max heaps.

Figure 8.14: Min Heap
The parental dominance requirement for min heap: The key at each
node has to be lesser than or equal to the keys at its children
i.e. key(parent) key(child). The figure 8.14 represents a min heap.

(a) (b)
Figure 8.15: Illustration of Heap
In figure 8.15, the first tree i.e. figure 8.15(a) is a heap, but the second tree
i.e. figure 8.15(b) is not, as the trees shape requirement is violated.
8.5.2 Architectural approach of heaps and algorithms
The two principal ways to construct a heap are:
1. Bottom-up heap construction algorithm
2. Top-down heap construction algorithm
Let us now discuss the bottom-up heap construction.
Bottom-up heap construction
It initializes the essentially complete binary tree with n nodes by placing
keys in the order given and then heapifies the tree as follows. Starting with
the last parental node and ending with the root, the algorithm checks
whether the parental dominance holds for the key at this node.

Figure 8.16: Checking for Parental Dominance
If it does not, the algorithm exchanges the nodes key K with the larger key
of its children and checks whether the parental dominance holds for K in its
new position (Refer to figures 8.16 and 8.17).

Figure 8.17: Continue Checking for Parental Dominance
This process continues until the parental dominance requirement for K is
satisfied. After completing the heapification of the sub-tree rooted at the
current parental node, the algorithm proceeds to do the same for the nodes
immediate predecessor. The algorithm stops after this is done for the trees
root to give the final heap in figure 8.18(a). The numbers above the nodes in
the tree indicate their position in the array which is shown by the
figure 8.18(b).
2
9 7
5 6 8
2
9 8
5 6 7
2
9 8
5 6 7
2
9 8
5 6 7
9
2 8
5 6 7

(a) (b)
Figure 8.18: Final Heap and Array Representation
Since the value of a nodes key does not change during the process of
shifting it down the tree, it need not be involved in intermediate swaps. The
empty nodes are swapped with larger keys in its children until a final
position is reached where it accepts the erased value again.
Let us now study the algorithm for bottom-up heap construction.
Algorithm: Heap Bottom-up (H [1...n])
//Constructs a heap from the elements of a given array
// by the bottom-up algorithm
//Input: An array H[1..n] of orderable items
//Output: A heap H[1..n]
for i n/2 down to 1 do
ki; vH[k]
heapfalse
while not heap and 2 * k n do
j 2 * k
if j <n //there are two children
if H[ j ]<H[ j + 1] j j + 1
if v H[j ]
heaptrue
else H[k]H[j ]; kj
H[k]v

9
6
8
2
5
7
[0]
[1]
[2]
[3]
[4]
[5]
5 4
1
3
0
9
6 8
5
2
7
2
Let us now trace the bottom-up heap construction algorithm.
Algorithm Tracing of Bottom-up Heap Construction
n=2
Algorithm: Heap bottom-up (H [1...2])

//Constructs a heap from the elements of a given array
// by the bottom-up algorithm
//Input: An array H[1..2] of orderable items
//Output: A heap H[1..2]
for i 2/2=1 down to 1 do
k1; vH[1]
heapfalse
while not heap and 2 * 1 2 do
j 2 * 1
if j <n //there are two children
if H[ 2 ]<H[ 2 + 1] 2 2 + 1
if v H[2 ]
heaptrue
else H[1]H[2 ]; 12
H[1]v
To find the efficiency in the worst case, assume n = 2k 1 so that a heaps
tree is full, i.e., the largest number of nodes occur on each level. Let h be
the height of the tree; according to the first property of heaps in the list at the
beginning of the section, h = log
2
n (or just log
2
(n + 1) 1= k 1 for the
specific values of n we are considering). In the worst case of the heap
construction algorithm each key on level i of the tree will travel to the leaf
level h.
Since moving to the next level down requires two comparison, one to find
the larger child and the other to determine whether the exchange is
required, the total number of key comparisons involving a key on level i will
be 2(h i). Equation Eq: 8.3 gives the total number of key comparisons in
the worst case.
1 1
2
0 level i keys 0
( ) 2( ) 2( )2 2( log ( 1))
h h
i
worst
i i
C n h i h i n n

= =
= = = +

Eq: 8.3
Here the validity of the last equality can be proved either by using the
closed-form formula for the sum
1
2
h
i
i
i
=
h or by mathematical induction in h.
Thus, with this bottom-up algorithm, a heap of size n can be constructed
with fewer than 2n comparisons.
Top-down heap construction algorithm
The top-down heap construction algorithm (which is less efficient) constructs
a heap by successive insertions of a new key into a previously constructed
heap shown in figure 8.19.

Figure 8.19: Pre-constructed Heap
To insert a new key into a heap, first attach a new node with key K in it after
the last leaf of the existing heap. Then shift K up to its appropriate place in
the new heap as in figure 8.20.

Figure 8.20: Inserting 9 into the Heap
Compare K with its parents key. Stop if the parent key is greater than or
equal to K (the structure is a heap), else, swap these two keys and compare
K with its new parent (Refer to figure 8.21). This swapping continues until K
is not greater than its last parent or it reaches the root. In this algorithm, too,
we can shift up an empty node until it reaches its proper position, where it
will get Ks value.

Figure 8.21: Check for Parental Dominance until Tree is Balanced
8
5 7
4 2 6 9
8
5 9
4 2 6 7
9
5 8
4 2 6 7
8
5 7
4 2 6
9
8 6
5 2 1
1
8 6
5 2
8
5 6
1 2
This insertion operation doesnt require more key comparisons than the
heaps height. Since the height of a heap with n nodes is about log
2
n, the
time efficiency of insertion is in O(log n).
Deleting the root key from a heap
The following steps indicate the procedure to delete the root key from a
heap in the figure 8.22.

Figure 8.22: Sample Heap
Step 1: Exchange the roots key with the last key K of the heaps in the
figure 8.23.

Figure 8.23: Exchanging the Root Key with the Smallest Key
Step 2: Decrease the heaps size by 1 (Refer figure 8.24).

Figure 8.24: Delete the Key Having the Original Root Key
Step 3: Heapify the smaller tree by shifting K down the tree exactly in the
same way we did it in the bottom-up heap construction algorithm. That is,
verify the parental dominance for K: if it holds, we are done (Refer figure
8.25); if not, swap K with the larger of its children and repeat this operation
until the parental dominance condition holds for K in its new position.

Figure 8.25: Heapified Tree
1
8 6
5 2 9
5
3 8
4 2 9 7
We can determine the efficiency of deletion by the number of key
comparisons needed to heapify the tree after the swap has been made
and the size of the tree is decreased by 1. Since it cannot require more key
comparisons than twice the heaps height, the time efficiency of deletion is
in O(log n) as well.
8.5.3 Heapsort
A heap is used to implement heapsort, which is an optimal sorting algorithm.
This can be done simply by inserting keys into the heap, then removing
them one by one.
Basics of heapsort
Heapsort is a comparison-based sorting algorithm, and belongs to the
selection sort family. Although it is slower in practice on most machines than
quick sort, it has a more favorable worst-case O(n log n) runtime.This is a
two-stage algorithm that works as follows.
Stage 1: (heap construction): Construct a heap for a given array.
Stage 2: (maximum deletions): Apply the root-deletion operation n1
times to the remaining heap.
We will see the implementation of heapsort using the following example.
Consider the following numbers 5,3,8,2,4,7,9. The tree representation of the
set of numbers can be seen in the figure 8.26.

Figure 8.26: Tree for the List
Let us now perform the first stage of heapification on the tree to make it
balanced as shown in the figure 8.27.

Figure 8.27: Process of Heapification
5
4 8
3 2 9 7
9
4 5
3 2 8 7
5
4 9
3 2 8 7
Now that we have got the heapified tree (Refer figure 8.28) we will perform
stage two which includes the deletion of the nodes

Figure 8.28: Heapified Tree
To perform the node deletion, now that we have the largest value on the top
of the heap we can just push it into an array and replace the value by the left
most element in the tree, which is deleted (Refer figure 8.29)

Figure 8.29: Node Deletion
We can now see in that in the figure 8.29 that the top most element which is
also the largest element is pushed into the array. So now we are left with a
tree which is not balanced. Therefore we will repeat the process of
hepification at the end of which we will have the largest element in the top
node of the tree. This element is now pushed again in the array, and
replaced by the bottom leftmost element. Hence we repeat this process
again and again which will finally give us the sorted array as in figure 8.30.

Figure 8.30: Sorted Array
Since we already know that the heap construction stage of the algorithm is
in O(n), we need to investigate just the time efficiency of the second stage.
For the number of key comparisons, C(n), needed for eliminating the root
keys from the heaps of diminishing sizes from n to 2, we get the inequality
shown in equation Eq: 8.4.
9

2
4 8
3 5 7
2 4 7 8 9 3 5
9
4 8
3 2 5 7

i n n n C
n
i
=
s + + + s
1
1
2 2 2 2
log 2 ] 1 [log 2 ) 2 ( [log 2 )] 1 ( [log 2 ) (

n n n n n
n
i
=
s = s
1
1
2 2 2
log 2 ) 1 ( log ) 1 ( 2 ) 1 ( log
Eq: 8.4
This means that C(n) O(n log n) for the second stage of heapsort. For both
stages, we get O(n) + O(n log n) = O(n log n). A more detailed analysis
shows that, in fact, the time efficiency of heapsort is in O(n log n) in both the
worst and average cases.
Activity 2
Construct a heap for the list 1, 8, 6, 5, 3, 7, 4 by the bottom-up algorithm.
10. _________ heap construction algorithm is less efficient than its
counterpart.
11. A heap can be defined as ________ with keys assigned to its nodes.
12. The time efficiency of heapsort is ________ in both worst and average
cases.
8.6 Problem Reduction
The previous section explained heap sort. This section explains the
technique of problem reduction.
If you need to solve a problem, reduce it to another problem that you know
how to solve.
This strategy can be used for actual problem solving, too. But it is difficult to
find a problem to which the problem at hand should be reduced to. We
should also ensure that our reduction-based algorithm is more efficient than
solving the original problem directly. This strategy can simply be shown by
the figure 8.31.

Figure 8.31: Problem Reduction Strategy
Problem 2
(can be solved using
algorithm A)
Problem 1
(to be solved)
Solution to
Problem 2
reduction Alg. A
8.6.1 Computing the least common multiple
You know that the least common multiple of two positive integers, denoted
as lcm (m, n), is defined as the smallest integer that is divisible by both m
and n. For example, lcm(24, 60) = 120, and lcm(11, 5) = 55. Given the prime
factors of m and n, lcm(m, n) can be computed as the product of all the
common prime factors of m and n times the product of ms prime factors that
are not in n times ns prime factors that are not in m.
For example consider equation Eq: 8.5
30 = 2 3 5
60 = 2 2 3 5
lcm(30, 60) = 2 2 3 5 = 60 Eq: 8.5
This algorithm for computing the least common multiple is inefficient and
requires a list of consecutive primes.
A much more efficient algorithm for computing the least common multiple
can be designed by using problem reduction. The Euclids algorithm helps to
find the greatest common divisor (gcd), which is a product of all the common
prime factors of m and n. It is not difficult to see that the product of lcm(m, n)
and gcd(m, n) includes every factor of m and n exactly once and hence is
simply equal to the product of m and n. This observation leads to the
formula in equation Eq: 8.6

) , gcd(
) , (
n m
n m
n m lcm

=
Eq: 8.6
Considering the equation Eq: 8.6, let us now solve the previous example in
Eq:8.5 to find the lcm(24,60). Here we can see that we get the same result
(Refer Eq: 8.7) as in Eq: 8.5.

60
5 x 3 x 2
60 x 30
) 60 , 30 ( lcm = =
Eq: 8.7
8.6.2 Counting paths in graphs
Let us now consider the problem of counting paths between two vertices in
a graph. It is not difficult to prove by mathematical induction that the number
of different paths of length k > 0 from the i
th
vertex to the j
th
vertex of a graph
(undirected or directed) equals the (i, j)
th
element of A
k
where A is the
adjacency matrix of the graph. Thus, we can solve the problem of counting a
graphs paths by using an algorithm for computing an appropriate power of
its adjacency matrix.
You can order the vertices of an adjacency matrix however you wish. This
means that you can have n! different orderings for a single graph. Usually,
to make things easier, we choose to order them alphabetically (a, b, c, d ).
You have a matrix which each column and row is labeled with the vertices.
Then you say how many edges connect i with j, and then you put the
answer in the (i,j)
th
position of the matrix. For example consider the graph in
the figure 8.32 its adjacency matrix can be written as matrix A.

Figure 8.32: Counting Paths in a Graph
Hence if we want to calculate the no of paths of length n we can find it by
calculating A
n
where A is the adjacency matrix. For example if

we want to
find the no of paths of length 4 from a to d in the graph from the figure 8.30,
we have to find the value at the position (1,4) in the matrix A
4
.
4
8 0 0 8
0 8 8 0
0 8 8 0
8 0 0 8
a b c d
a
b
A
c
d
(
(
(
=
(
(
(
(

Matrix: 8.4
Hence we find that in the matrix 8.4 in the position of the first row and fourth
column i.e. (1, 4) the value is 8. Therefore the no of paths of length 4 from a
to d is 8.
Activitiy 3
Prove the equality lcm(m, n) =
) n , m gcd(
n . m

b
d c
a
0 1 1 0
1 0 0 1
1 0 0 1
0 1 1 0
a b c d
a
b
A
c
d
(
(
(
=
(
(
(
(

13. Greatest common divisor can be computed easily by ________.
14. The problem of counting a graphs paths can be solved with an
algorithm for an appropriate power of its _______________.
15. What is the formula obtained to find the lcm in the problem reduction
approach to find the lcm?
8.7 Summary
In this unit we have learnt that the transform and conquer technique works
as two stage method i.e. Transform where the problem is modified and
Conquer where the problem is solved. We now know that there are three
principal varieties of the transform-and-conquer strategy: instance
simplification, representation change, and problem reduction.
We have learnt that presorting, balanced tree search, Gaussian elimination,
heaps, counting paths in a graph etc. are all strategies illustrating the 3
different varieties involved in the implementation of this technique. These
strategies help us solve complex problems encountered such as sorting
elements, and counting paths in graphs.
8.8 Glossary
Term Description
Adjacency matrix Representation of the vertices of a graph adjacent to the
other vertices.
Rank of a matrix The maximal number of linearly independent columns of A
is the column rank of a matrix A. Similarly, the maximal
number of linearly independent rows of A is the row rank of
the matrix.
1. Explain the algorithm to compute mode via presorting.
2. Briefly explain the four rotations in an AVL tree.
3. Differentiate between bottom-up and top-down heap construction.
4. What is heap sort?
5. Describe 2-3 trees.

8.10 Answers
1. Instance simplification, problem reduction , representation change
2. (n
2
)
3. [log
2
n] +1
4. Linear
5. Lower triangular
6. Rank
7. Binary search
8. LR-rotation
9. 2-node, 3-node
10. Top-down
11. Binary trees
12. O(n log n)
13. Euclids algorithm
14. Adjacent matrix
15.
) , gcd(
) , (
n m
n m
n m lcm

=

Terminal Questions
1. Refer to 8.2 Presorting
2. Refer to 8.4.2 AVL trees
3. Refer to 8.5.2 Architectural approach of heaps and algorithms
4. Refer to 8.5.3 Heapsort
5. Refer to 8.4.3 2-3 trees
References
- Anany V. Levetin. (2002). Introduction to the analysis and design of
algorithms. Addison-Wesley Longman Publishing Co.
- Thomas H. Cormen, Charles E. Leiserson, Ronald L Rivest, Clifford
Stein. (2006). Introduction to algorithms,2nd Edition, PHI
E-References
- http://lcm.csa.iisc.ernet.in/dsa/node118.html
- http://mathworld.wolfram.com/GaussianElimination.html
Unit 9 Space and Time Tradeoffs
Structure:
9.1 Introduction
Objectives
9.2 Sorting
Distribution counting
9.3 Input Enhancement in String Matching
Horspool's algorithm
Boyer-Moore algorithm
9.4 Hashing Methodology
Hash function
Collision resolution
9.5 Indexing Schemes
B-Tree technique
9.6 Summary
9.7 Glossary
9.9 Answers

9.1 Introduction
By now you must be familiar with different methods of problem solving using
algorithms. This unit deals with space and time tradeoffs in algorithms.
Algorithm analysis determines the amount of resources necessary to
execute the algorithm. The vital resources are time and storage. Most
algorithms are constructed to work with inputs of arbitrary length. Usually
the efficiency of an algorithm is stated as a function relating the input length
to the number of steps (time complexity) or storage locations (space
complexity).
A balance has to be maintained in the space and time aspects of computing.
Research in Computer Science these days is more towards achieving time
efficiency.
With continuous reduction in prices, space availability is perhaps not a
significant problem. But with respect to networking and robotics, however,
the necessity of a balance becomes apparent. Memory has to be used
conservatively.
Space-time or time-memory tradeoff in context with algorithms relates to the
execution of an algorithm. The execution of an algorithm can be done in a
short time by using more memory or the execution time becomes more
when memory used is less. Therefore selecting one alternative over the
other is the tradeoff here.
Many problems such as sorting or matrix-multiplication, have many choices
of algorithms to use, of which some are extremely space-efficient and some
extremely time-efficient. Space-time tradeoff proves that no algorithm exists
where efficiency is achieved by small space and small time simultaneously.
This unit includes the various approaches such as sorting by counting, input
enhancement in string matching, hashing, and B-Tree technique where time
is an important factor and the result of computation is quick at the cost of
space.
Objectives:
explain the importance of space-time tradeoff in programming
the process of sorting by counting
analyze the process of input enhancement in string matching
define the role of hashing in space and time tradeoff
explain B-Tree technique with respect to space and time tradeoff

9.2 Sorting
Input enhancement is based on preprocessing the instance to obtain
additional information that can be used to solve the instance in less time.
Sorting is an example of input enhancement that achieves time efficiency.
First, let us study distribution counting.
9.2.1 Distribution counting
Distribution counting is a sorting method that uses some associated
information of the elements and places the elements in an array at their
relative position. In this method elements are actually distributed in the array
from 0
th
to (n-1)
th
position. This method ensures that the elements do not get
over written. The accumulated sum of frequencies also called as distribution
in statistics is used to place the elements at proper positions. Therefore this
method of sorting is called distribution counting method for sorting.
6 4 2 6 4 4 2
Figure 9.1: Array a[ ]
The figure 9.1 depicts the values of array a[ ]. Observing the array we find
that the list contains only {2, 4, 6}. We will count the frequencies of each
element and the distribution count. Distribution count value represents the
last occurrence of corresponding element in the sorted array. These values
are given in table 9.1.
Table 9.1: Distribution Values of Elements of Array a[ ]
Elements 2 4 6
Frequencies 2 3 2
Distribution values 2 5 7
We will have two arrays as shown in figure 9.2, one to store distribution
values in an array called Dval [0..2] and another array to store the sorted list
of elements called b[06] and then scan the unsorted list of elements from
right to left. The element that is scanned first is 2 and its distribution value
is 2.
0 1 2 0 1 2 3 4 5 6
2 5 7

array Dval[ ] array b[ ]
Figure 9.2: Sorted Element Stored in Array b[ ]
Decrement the distribution value of 2, that is Dval[0]=Dval[0]-1. After which
move to left and scan the element at a[5]. So insert element 4 at 5-1 = 4
i.e b[4] position as shown in figure 9.3.
6 4 2 6 4 4 2

1 5 7

Figure 9.3: After Scanning Element a[5]
Now decrement the distribution value of 4 that is Dval 5-1 = 4. Scan a[4]
which is 4 then find the distribution value of 4 Dval[1] = 4 put 4 at 4 - 1 = 3,
at b[3] location. This is depicted in figure 9.4.
2
2 4

6 4 2 6 4 4 2

1 4 7

Figure 9.4: After Scanning Element a[4]
The array a[ ] is scanned through all the elements and sorted. In the last
step, a[0] which is 6, is scanned. The distribution value of 6 is Dval[2] =6.
Hence we insert 6 at 6-1=5 i.e. at b[5] position. This is shown in figure 9.5.
6 4 2 6 4 4 2

0 2 6

Figure 9.5: After Scanning All Elements in Array a[ ]
Thus the array is scanned from right to left. The following figure 9.6 depicts
the summation of all the above steps.

a[6] =1
a[5] =2
a[4]= 2
a[3] = 3
a[2] = 1
a[1] = 2
a[0] = 3

Figure 9.6: Depicting Array Dval[ ] and Array b[ ] After Scanning All Elements
The draw back of this method is that it depends on the nature of input,
requires additional auxiliary array for sorting list and an array for distribution
of values.

2 4 4
2 2 4 4 4 6 6
2 5 7
1 5 7
1 4 7
1 3 7
1 3 6
0 3 6
0 2 6
2
4
4
6
2
4
6
Let us now discuss the algorithm for distribution counting.
Algorithm for Distribution Counting
DistributionCounting(A[0...n-1],L,u)
//Sorts an array of integers from a limited range by distribution counting
//Input: An array A[0....n-1] of integers between L and u ( L<= u)
//Output: Array S[0...n-1] of A's elements sorted in nondecreasing order
for j = 0 to u-1 do D[]j] = 0 //initialize frequencies
for i = 0 to n-1 do D[A[i] - L] = D[A[i] -L] +1 //compute frequencies
for j =1 to u-L do D[j] = D[j-1] + D [j]
for 1=n - 1 downto 0 do
j = A[i] l
S[D[j] - 1] = A[i]
D[j] = D[j] 1
return S
Let us now trace this algorithm.
Algorithm Tracing for Distribution Counting
n=2
A[]=[1,2,1]
ALGORITHM DistributionCounting(A[0...1],2,3)
for j = 0 to 3 do D[0] = 0 //initialize frequencies
for i = 0 to 3 do D[1-2] = D[1-2] +1 //compute frequencies
for j =1 to 1 do D[1] = D[0] + D [1]
for 1=1 down to 0 do
j = A[0] l=0
S[0 - 1] = 1 ie S[1]=1
D[1] = D[1] 1
return 1

If we analyze the algorithm for distribution counting, we can divide the time
taken into four parts as given in the table 9.2.
Table 9.2: Analysis of Distribution Counting
Loop Complexity
for j = 0 to u-1
do D[]j] = 0
(u)

for i = 0 to n-1
do D[A[i] - L] = D[A[i] -L] +1
(n).
for j =1 to u-L
do D[j] = D[j-1] + D [j]
(u)
for 1=n - 1 downto 0 do
j = A[i] l
S[D[j] - 1] = A[i]
D[j] = D[j] 1 ,it takes O(n)
(n)
Total (u + n)

If u = O(n), then the time complexity of distribution counting will be (n).
1. Input enhancement is based on ___________________ the instance.
2. The information which is used to place the elements at proper positions
is accumulated sum of frequencies which is called as __________.
3. Sorting is an example of input enhancement that achieves ___________.

9.3 Input Enhancement in String Matching
Now, let us see some algorithms for string matching.
String matching is an important application of input enhancement. We know
that brute force method is the simplest method. But it is time consuming
because every character of pattern is matched with every character of text.
Hence faster algorithms have been developed and some of these are listed
below:
Horspools algorithm
The Boyer-Moore algorithm
The Knuth-Morris-Pratt algorithm
Two string matching algorithms the Boyer-Moore algorithm and Horspools
algorithm are discussed in this section. These algorithms are based on input
enhancement and preprocess the input pattern and get some information
about the input. They then arrange this information in a tabular form and
then make use of the information in the actual algorithm.
9.3.1 Horspools algorithm
How does the Horspools algorithm work?
This algorithm places the pattern beside the text. The characters from the
pattern are matched right to left with the characters from the text. If the
character of the pattern matches the character in the text, then the desired
substring is found as in the example shown in figure 9.7.

Shift pattern by
3 characters

A R
S H A R P E R
1 2 3
Figure 9.7: Example for Pattern Matching
The number of shifts that are made to match pattern against text is an
important activity. To speed up the algorithm we can precompute the shift
sizes and store them in a table. Such a table is called shift table.
The shift size can be computed by the following formula:
If T is not among first m-1 characters of pattern
Table (T) = the patterns length m
Else Table (T) = the distance from the rightmost T among
first m-1 characters of patterns to its last character.
The first step in this algorithm is to create the shift table for the pattern as
given in table 9.3.

A R
S H A R P E R
Initialize the shift table by length of pattern i.e. KIDS which is 4.
3 2 1
fill K column entry by 3,I column
entry by 2 and D column entry by 1
Table 9.3: Shift Table
A B C D E F G H I J K L M N
4 4 4 1 4 4 4 4 2 4 3 4 4 4

O P Q R S T U V W X Y Z
4 4 4 4 4 4 4 4 4 4 4 4
The table 9.4 illustrates the way the pattern is matched with the text.
Table 9.4: Matching Pattern Against Text

D A V I D L O V E S K I D S
K I D S
Scan is
done from
Right to left
IS, refer to
the shift
table I
indicates 2,
Hence shift
pattern by
2 positions
K I D S
K I D S
K I D S
Mismatch
occurs so
the entire
pattern is
shifted to
the right by
its length 4
K I D S
Now IS,
with
reference
to the shift
table the
pattern is
shifted 2
positions to
the right
K I D S

K I D S
All the
characters
from the
text are
matching
the
characters
in the
pattern,
hence the
search is
declared
successful
Let us next discuss the Horspools algorithm for string matching.
Horspool Algorithm for Matching Strings
Horspool_string(P[ ],T[ ],m,n)

//Input: The pattern P[ ], the text T[ ], length of pattern m, length of text n
//Output: The first positional index of matching substring
//If no matching substring is found then return -1 shift_table(P[0..m-1])
//Construct the shift table im-1
//set the position from right end in the pattern
While(i<=n-1) do
{
//initially assume that the number of matched characters are 0
match_ch0
while((match_ch<=m-1) AND (P [m-1-match_ch] ))
//Scan pattern from right to left to locate rightmost occurrence of
character
match_chmatch_ch+1
if (match_ch=m) then
return i-m+1//shifts pattern to its right if only last character is matching
else
ii+Table[T[i] ]//refer to the characters entry in the shift table
}
Return -1
Let us now trace this algorithm.
Algorithm tracing for Horspools Algorithm
n=2
P[ ]=[EE]
T[ ]=[GREEDY]
m=5
Horspool_string(P[ ],T[ ],m,n)
While(i<=1) do
{
//initially assume that the number of matched characters are 0
match_ch=0
while((match_ch<=4) AND (5 ))
//Scan pattern from right to left to locate rightmost occurrence of
character
match_ch=0+1
if (match_ch=5) then
return 1-5+1=5//shifts pattern to its right if only last character is
matching
else
0+Table[5]//refer to the characters entry in the shift table
}
Return -1
Time and space complexity of Horspools algorithm for string matching is
given in two parts.
Preprocessing phase time complexity O(m+n) and space complexity
O(n)
Searching phase time complexity O(m + n)
The average number of comparisons for one text character is between 1/
and 2/(+1) where is the number of storing characters.
Next let us study the Boyer- Moore algorithm.
9.3.2 Boyer-Moore algorithm
The Boyer-Moore algorithm uses two heuristics: good-suffix and bad-
character shift.
Bad character shift
We use this when mismatch occurs. We decide the number of places to shift
by using bad character shift.
As in Horspools algorithm if the rightmost character does not match,
then the pattern is shifted to the right by its length.
When the rightmost character of the pattern matches with that of the
text, then each character is compared from right to left. If at some point
a mismatch occurs after a certain number of k matches with texts
character T, then bad character shift is denoted by P.
P = max { text (T) k,1}

Good suffix Shift
This shift helps in shifting a matched part of the pattern, and is denoted by
Q. Good suffix shift Q is applied after 0 < k < m characters are matched.
Q = distance between matched suffix of size k and its rightmost occurrence
in the pattern that is not preceded by the same character as the suffix
When the substring is not found do the following:
If all the m characters of pattern are matching with the text then stop
If a mismatching pair occurs, then compute bad character shift P and
good suffix shift Q and use these to compute shift size.
R = max {P,Q} where k > 0
Consider
B H U T T O K N O W S T O R O N T O
T O R O N T O

6 5 4 3 2 1 0

The bad character shift table is as in table 9.5
Table 9.5 Bad Character Shift Table
A B C D E F G H I J K L M N O P Q R
7 7 7 7 7 7 7 7 7 7 7 7 7 2 3 7 7 4

S T U V W X Y Z
7 1 7 7 7 7 7 7
T O R O N T O
The Good- suffix table is as in Table 9.6
Table 9.6: Good-Suffix Table
k Q
1 3
2 5
3 5
4 5
5 5
6 5
Space means shift by 7

Since space is encountered while matching the pattern with the text, the
pattern is shifted by 7 positions as shown in table 9.7.
Table 9.7: After Shifting the Pattern by 7 Positions
T O R O N T O
With reference to T entry from table 9.5 i.e. bad character table which is 1,
shift the pattern to the right by 1 position as shown in table 9.8.
Table 9.8: Pattern Shifted to Right by 1 Position
T O R O N T O

k=2
There is mismatch in pattern because of the blank place in the text.
Therefore P = max [(text (-) k) ,1]
= max {(7 - 2),1}
P = 5

T O R O N T O
Compute Good suffix shift Q. As 2 characters from the pattern are matching
Q(2) = 5
Shift size R = max {P,Q}
= max {5,5}
= 5
Therefore we shift the pattern 5 positions ahead as shown in table 9.9.
Table 9.9: Depicting the Pattern in the Text
The required pattern is found in the text
Following are the steps that are followed to find the pattern:
Step 1: Construct a bad character table and obtain the bad character
shift P.
Step 2: Compute a good suffix shift Q.
Step 3: Compare the pattern from right to left by placing the pattern from
beginning of text.
Step 4: Repeat steps until pattern goes beyond last character of text or
matching.
The Boyer Moore algorithm is considered as one of the best pattern
matching algorithm with its best case time complexity as O (
m
n
). The
average and worst case time complexities are given as O(m+n) and O(mn).
Space complexity is given as O(m + k) where, k is the number of different
possible characters.
With the Boyer- Moore algorithm analyzed, let us move on to hashing.
Activity 1
Construct a good suffix table for the pattern TOMATO
4. Input enhancement is to _____________ the input pattern.
T O R O N T O
5. In Horspools algorithm, the characters from the pattern are matched
__________________.
6. The two heuristics in Boyre-Moore algorithm are _______________ and
________________.

9.4 Hashing Methodology
Hashing is the method by which a string of characters or a large amount of
data is transformed into a usually shorter fixed-length value or key that
represents the original string. This key is used to index and retrieve items
from a database. We can find items faster using the shorter hashed key
than by using the original value.
Hashing is performed on arbitrary data by a hash function. The code
generated is unique to the data it came from.
9.4.1 Hash function
What is hash function?
A hash function is a function that converts data to either a number or an
alphanumeric code. The hashing algorithm is called the hash function.
Hash functions are primarily used in hash tables, to locate a data record
given its search key. The hash function maps the search key to the hash.
The index gives the place where the corresponding record is stored.
Therefore, each slot in a hash table is related with a set of records, rather
than a single record. Each slot in a hash table is called a bucket, and hash
values are also called bucket indices.
A hash function returns hash values, hash codes, hash sums, checksums or
simply hashes. The hash function hints at the record's location it tells
where one should start looking for it.
Uses of Hash functions
Hash functions are used to speed up table lookup or data comparison tasks
such as finding items in a database, detecting duplicate or similar records
in a large file, and so on. Hash functions are also used to determine if two
objects are equal or similar, checksums over a large amount of data and
finding an entry in a database by a key value. The UNIX C-shell uses hash
table to store the location of executable programs.
9.4.2 Collision resolution
Hash collisions are unavoidable when hashing a random subset of a large
set of possible keys. A hash function can map two or more keys to the same
hash value. In many applications, it is desirable to minimize the occurrence
of such collisions, which means that the hash function must map the keys to
the hash values as evenly as possible. Therefore, hash table
implementations have some collision resolution strategy to handle such
events. The most common strategies are:
o Open Hashing (Separate Chaining)
o Closed Hashing (Open Addressing)
The keys (or pointers to them) need to be stored in the table, together with
the associated values in this method.
Load factor
The performance of collision resolution methods does not depend directly
on the number n of stored entries, but depends strongly on the table's load
factor, the ratio n/s between n and the size s of its bucket array.
Separate chaining
Hash collision is resolved by separate chaining also called open hashing or
closed addressing.
In this strategy, each slot of the bucket array is a pointer to a linked list
which contains the key-value pairs that are hashed to the same location.
Lookup scans the list for an entry with the given key. Insertion involves
adding a new entry record to both the ends of the list belonging to the
hashed slot. Deletion involves searching the list and removing the element.
Chained hash tables with lists which are linked is popular because they
require only basic data structures with simple algorithms, and can use
simple hash functions that are unsuitable for other methods.
A chained hash table is effective when the number of entries n is much
higher than the number of slots. The performance decreases (linearly) with
the load factor. For example, a chained hash table with 100 slots and 1000
stored keys (load factor 10) is five to ten times slower than a 1000-slot table
(load factor 1); but still 1000 times faster than a plain sequential list, and
possibly even faster than a balanced search tree.
In separate-chaining, the worst-case scenario is when all entries are
inserted into the same bucket, in which case the hash table is ineffective
and the cost is that of searching the bucket data structure. If the bucket data
structure is a linear list, the lookup procedure may have to scan all its
entries; hence the worst-case cost is proportional to the number n of entries
in the table.
Open addressing
Hash collision resolved by open addressing is called closed hashing. The
term "open addressing" indicates that the location ("address") of the item is
not determined by its hash value.
In open addressing, the entry records are stored in the bucket array itself.
When a new entry has to be made, the bucket is examined, starting with the
hashed-to slot and proceeds in some probe sequence, until an unoccupied
slot is found. When searching for an entry, the buckets are scanned, until
either the target record is found, or an unused array slot is found, which
indicates that there is no such key in the table. The popular probe
sequences are:
Double hashing the interval between probes is computed by another
hash function.
Linear probing the interval between probes is fixed (usually 1).
Quadratic probing the interval between probes is increased by
adding the successive outputs of a quadratic polynomial to the starting
value given by the original hash computation.
A drawback to the open addressing schemes is that the number of stored
entries cannot exceed the number of slots in the bucket array. In fact, even
with good hash functions, their performance decreases when the load factor
grows beyond 0.7 or so.
Open addressing schemes also put more strict requirements on the hash
function. The function must distribute the keys more uniformly over the
buckets and minimize clustering of hash values that are consecutive in the
probe order.
Open addressing saves memory if the entries are small (less than 4 times
the size of a pointer). Open addressing is a waste if the load factor is close
to zero (that is, there are far more buckets than stored entries), even if each
entry is just two words.
In the next section we will study the indexing schemes.
Activity 2
Compare the collision resolution strategies and list the disadvantages of
one strategy over the other

7. Each slot of a hash table is often called a _____________.
8. Collision occurs when a hash function maps two or more keys to the
____________________________.
9. When the interval between probes is computed by another hash function
it is ______________________.

9.5 Indexing Schemes
An effective indexing scheme will help in easy retrieval of data. Different
tradeoffs are involved in different indexing techniques. An indexing scheme
which is faster may require more storage. The B-tree is one such important
index organization.
9.5.1 B-Tree Technique
The B-Tree is an individual indexing scheme at the high speed end of the
spectrum. The tradeoff here is that in exchange for fast access times, you
pay in terms of code and memory buffer size and in disk space for the
B-Tree themselves.
It uses an index to the data records. A file that contains enough information
to describe the position and key of each record in the data file is built. The
index is organized into a branching tree structure and the tree is kept
balanced. The number of index accesses to reach a record is proportional to
the logarithm of the number of records in the file. That is to say access time
increases slowly as the number of records increases. As the branching
factor increases, the height of the tree decreases, thus making access
quicker.
A B-Tree of order m (the maximum number of children for each node) is a
tree which satisfies the following properties:
Every node has at most m children.
Every node (except root) has at least m2 children.
The root has at least two children if it is not a leaf node.
All leaves appear in the same level, and carry information.
A non-leaf node with k children contains k1 keys.
Internal nodes in a B-Tree nodes that are not leaf nodes are represented
as an ordered set of elements and child pointers. Every internal node has
maximum of U children and other than the root a minimum of L. The
number of elements is also restricted for the leaf nodes but these nodes
have no children or child pointers.
Root nodes have an upper limit on the number of children but no lower limit.
For example, when there are less than L1 elements in the entire tree, the
root will be the only node in the tree, and it will have no children at all.
B-Trees assign values to every node in the tree, and the same structure is
used for all nodes. However, since leaf nodes never have children, a
specialized structure for leaf nodes in B-Trees will improve performance.
Search
Searching is similar to searching a binary search tree. Starting from the root,
the tree is recursively traversed from top to bottom. At each level, the search
chooses the child pointer (sub tree) whose separation values are on either
side of the search value.
For example consider the figure 9.8. To search the number 5 in the tree we
start at the root node and traverse through the tree. The number 5 is
compared with root node value 13 and since it is less than 13 the searching
operation shifts to the left sub-tree. At the second level in the tree it again
compares with both the numbers 4, 7. The number 5 is between 4 and, so it
moves to the second element 7. The comparison continues in this way and
the node having number 5 is found.

Figure 9.8: B-tree Example for Search Operation
Binary search is typically used within nodes to find the separation values
and child tree of interest.
Insertion
To understand the process of insertion let us consider an empty B-tree of
order 5 and insert the following numbers in it 3, 14, 7, 1, 8, 5, 11, 17, 13. A
tree of order 5 has a maximum of 5 children and 4 keys. All nodes other
than the root must have a minimum of 2 keys. The figure 9.9 shows how the
1st four numbers gets inserted.

Figure 9.9: Insertion of First Four Numbers in the B-Tree

To insert the next number 8, there is no room in this node, so we split it into
2 nodes, by moving the median item 7 up into a new root node as depicted
in figure 9.10.

Figure 9.10: B-Tree After Inserting Number 8
Insertion of the next three numbers 5, 11, and 17 proceeds without requiring
any splits. This is illustrated in figure 9.11.

Figure 9.11: B-Tree After Inserting Numbers 5, 11 & 17
A split is to be done to insert 13. 13 is the median key and so it is moved up
into the parent node. This is depicted in figure 9.12.

Figure 9.12: B-Tree After Inserting Number 13
Deletion

Figure 9.13: B-Tree Before Deletion
Considering the B-tree in figure 9.13, let us understand the process of
deletion by the deleting the following numbers one by one
To delete 20, we find its successor 23 (the next item in ascending order)
because it is not a leaf node, and move 23 up to replace 20. Therefore we
can remove 23 from the leaf since this leaf has extra keys.

Figure 9.14: B-Tree After Deleting Number 20
Next, to delete 18, even though 18 is in a leaf, we can see from figure 9.6
that this leaf does not have an extra key; the deletion results in a node with
one key, which is not acceptable for a B-tree of order 5. If the sibling node to
the immediate left or right has an extra key, we then borrow a key from the
parent and move a key up from this sibling. In this specific case, the sibling
to the right has an extra key. So, the successor 23 of 19, is moved down
from the parent, and the 24 is moved up, and 19 is moved to the left so that
23 can be inserted in its place.

Figure 9.15: B-Tree After Deleting Number 18
To delete 5, there is another problem now. Refer to figure 9.15. Although 5
is a leaf, the leaf has no extra keys, nor do the siblings to the immediate
right or left. In this case the leaf has to be combined with one of the two
siblings. This results in moving down the parent's key that was between
those of these two leaves. Therefore let's combine the leaf containing 6 with
the leaf containing 1 3 and move down the 4. This results in an invalid
B-tree as shown in figure 9.16.

Figure 9.16: Invalid B Tree After Deleting 5
Now the parent node contains only one key, 7 which is not acceptable. If
this node had a sibling to its immediate left or right having a spare key, then
we could again "borrow" a key. Suppose the right sibling (the node with 17
24) had one more key. We would then move 13 down to the node with too
few keys and move the 17 up where the 13 had been. The 14 and 16 nodes
would be attached via the pointer field to the right of 13's new location.
Since in our example we have no way to borrow a key from a sibling, we
must again combine with the sibling, and move down the 13 from the parent.
In this case, the trees height reduces by one. The resulting B-tree is shown
in figure 9.17.

Figure 9.17: B-Tree After Deleting 5
10. As the ______________________ increases the height of the tree
decreases thus speeding access.
11. Access time increases slowly as the number of records __________.
12. The insertions in a B-Tree start from a ________.

9.6 Summary
Let us summarize the unit here.
We usually analyze the efficiency of an algorithm in terms of its time and
space requirements. Usually the efficiency of an algorithm is stated as a
function relating the input length to the number of steps (time complexity) or
storage locations (space complexity).
Distribution counting is an input enhancement method wherein a separate
array is used to store the information generated during the sorting process
and these arrays enhance the sorting process. Horspools and Boyre-Moore
algorithms are string matching algorithms where in the pattern is compared
with the text and shifting of the pattern is done by computing shift size. Thus
searching operation becomes faster.
Hashing is a technique that uses a hash key to find items. Collision occurs
when the hash key value of two items turns out to be the same value. This is
rectified by the two collision resolution methods - separate chaining and
open addressing. The branching factor in B-Tree technique speeds the
access time.
Thus by using all the above techniques, we can reduce the execution time
of an algorithm.

9.7 Glossary
Term Description
Leaf node A leaf node is a node in a tree data structure that has no child
nodes.
Linked list A linked list is a data structure which consists of a sequence of
data records, that in each record there is a field that contains a
reference

1. Explain distribution counting with an example.
2. Explain the two types of collision resolution in hashing.
3. Describe the algorithms based on input enhancement in string matching.
4. What is a hash function?
5. How does B-Tree technique enhance space and time tradeoff?

9.9 Answers
1. Preprocessing
2. Distribution
3. Time efficiency
4. Preprocess
5. Right to left
6. Good suffix and bad character shift
7. Bucket
8. Same hash value
9. Double hashing
10. Branching factor
11. Increases
12. Leaf node
Terminal Questions
1. Refer to 9.2.1 Distribution counting
2. Refer to 9.4.2 Collision resolution
3. Refer to 9.3 Input enhancement in string matching
4. Refer to 9.4.1 Hash function
5. Refer to 9.5.1 B-Tree technique
Reference
Puntambekar, A.A. (2008). Design and Analysis of Algorithms, First
Donald Adjeroh., & Timothy Bell., & Amar Mukherjee (2008). The
Burrows-Wheeler transform, Springer Publishing Company.
E-Reference
www.cs.ucr.edu/~jiang/cs141/ch07n.ppt
http://documentbook.com/horspool-ppt.html
http://www-igm.univ-mlv.fr/~lecroq/string/node14.html
Unit 10 Dynamic Programming 1
Structure:
10.1 Introduction
Objectives
10.2 Overview of Dynamic Programming
Dynamic programming approach
Concept of memoization
Dynamic programming Vs divide and conquer
10.3 Fibonacci Numbers
Introduction to Fibonacci numbers
Algorithms to find n
th
Fibonacci number
10.4 Binomial Coefficient
Definition of binomial coefficients
Computation of binomial coefficients
10.5 Warshalls and Floyds Algorithms
Overview of Warshalls and Floyds algorithm
Warshalls algorithm
Floyds algorithm
10.6 Summary
10.7 Glossary
10.9 Answers
10.1 Introduction
By now you must be familiar with the different algorithms with various
concepts and their space and time tradeoffs. This unit introduces you to the
concepts of dynamic programming.
Dynamic programming is a general algorithm design technique. It was
introduced in the 1950s by American mathematician Richard Bellman to
solve optimization problems.
Objectives:
- calculate the nth Fibonacci number
- explain the dynamic programming approach to compute binomial
coefficients
- explain Warshalls and Floyds algorithms
10.2 Overview of Dynamic programming
Dynamic programming is an optimization technique for particular classes of
backtracking algorithms which repeatedly solve sub-problems. When
dealing with algorithms, dynamic programming always refers to the
technique of filling in a table with values computed from the sub-problems.
The central idea of dynamic programming is to solve several smaller
(overlapping) sub-problems and record their solutions in a table so that each
sub-problem is only solved once. Hence the final state of the table will be (or
contain) the solution.
It sometimes happens that the natural way of dividing an instance
suggested by the structure of the problem leads us to consider several
overlapping sub-instances. If we solve each of these independently, they will
in turn create a large number of identical sub-instances. If, on the other
hand, we take advantage of the duplication and solve each sub-instance
only once, saving the solution for later use, then we get a more efficient
algorithm.
10.2.1 Dynamic programming approach
Dynamic programming has the following two approaches:
- Top-down approach In this approach, if the solution to any problem
can be formed recursively using the solution to its sub-problems, and if
its sub-problems are overlapping, then one can easily memoize or store
the solutions to the sub-problems in a table. Whenever we begin solving
a new sub-problem, we first check the table to see if it has been already
solved. If a solution has been recorded, then we can use it directly,
otherwise we solve the sub-problem first and add its solution to the
table.
Bottom-up approach In this approach, once we formulate the solution to a
problem recursively in terms of its sub-problems, we can try reformulating
the problem in a bottom-up fashion, i.e. try solving the sub-problems first
and use their solutions to arrive at solutions to bigger sub-problems. This is
also usually done in a table, where the results of the sub-problems are
stored in a table and are iteratively used to generate solutions to bigger sub-
problems.
Let us now analyze the concept of memoization.
10.2.2 Concept of memoization
Memoization has a specialized meaning in computing. Memoization is the
technique by which we make a function perform faster by trading space for
time. It can be achieved by storing the return values of the function in a
table. Hence when the function is called again, the values stored in the table
are used to further compute the values, instead of computing the value all
over again. The set of remembered associations can be a fixed-size set
controlled by a replacement algorithm or a fixed set, depending on the
nature of the function and its use. A function can be memoized only if it is
referentially transparent; that is, only if calling the function is exactly the
same as replacing that function call with its return value.
Memoization methodology
1) Start with a backtracking algorithm
2) Look up the problem in a table; if table has a valid entry for it, return that
value
3) Else, compute the problem recursively, and then store the result in the
table before returning the value
10.2.3 Dynamic programming Vs divide and conquer
At a glance the dynamic programming approach might look exactly same as
the divide-and-conquer technique, but they differ substantially in their
problem solving approach.
The divide-and-conquer algorithms divide the problem into independent sub-
problems, solve the sub-problems recursively, and hence combine their
solutions to solve the original problem. In contrast, the dynamic
programming approach is applicable when the sub-problems are not
independent, that is, when sub-problems share sub-problems. Therefore a
divide-and-conquer algorithm does more work than necessary by repeatedly
solving the common sub-problems. A dynamic-programming algorithm on
the other hand solves every sub-problem just once and then saves its
answer in a table, thereby avoiding the need to re-compute the answer
every time the sub-problem is encountered.
1. The ____________ and _____________ are the two approaches to
dynamic programming.
2. ____________ is a technique to store answers to sub-problems in a
table.
3. The ___________________ algorithm is similar to the dynamic
approach.
10.3 Fibonacci Numbers
In the previous section we got an overview of the dynamic programming. In
this section we will learn about the Fibonacci numbers.
Before we analyze the dynamic programming technique, it will be beneficial
to first discuss the Fibonacci numbers and the memoization technique used
to compute the n
th
Fibonacci number.
10.3.1 Introduction to Fibonacci numbers
In the Fibonacci sequence, after the first two numbers i.e. 0 and 1 in the
sequence, each subsequent number in the series is equal to the sum of the
previous two numbers. The sequence is named after Leonardo of Pisa, also
known as Fibonacci. These fascinating numbers can be found in the
branching of trees, the patterns on a pineapple, the florets of a sunflower,
the spirals of a pine cone, and even in the placement of leaves on the stems
of many plants. Fibonacci numbers are one example of patterns that have
intrigued mathematicians through the ages. A real life example of a
Fibonacci sequence is given below.
Beginning with a single pair of rabbits, if every month each productive pair
bears a new pair, which becomes productive when they are 1 month old,
how many rabbits will there be after n months?
Imagine that there are x
n
pairs of rabbits after n months. The number of
pairs in month n+1 will be x
n
(in this problem, rabbits never die) plus the
number of new pairs born. But new pairs are only born to pairs at least 1
month old, so there will be x
n-1
new pairs.
x
n+1
= x
n
+ x
n-1
Eq: 10.1
Equation Eq:10.1 is simply the rule for generating the Fibonacci numbers.
10.3.2 Algorithms to find n
th
Fibonacci number
Let us now try to write an algorithm to calculate n
th
Fibonacci number. By
definition, the nth Fibonacci number, denoted by F
n
is shown by Eq: 10.2
F
0
= 0
F
1
= 1
F
n
= F
n-1
+ F
n-2
Eq: 10.2
We will now try to create an algorithm for finding the nth Fibonacci-number.
Let's begin with the naive algorithm exactly coding the mathematical
definition:
Naive algorithm to calculate the nth Fibonacci number:
// fib -- compute Fibonacci(n)
function fib(integer n): integer
assert (n 0)
if n == 0: return 0 fi
return fib(n - 1) + fib(n - 2)
end
Let us now trace the naive algorithm to calculate the nth Fibonacci number
Trace of the naive algorithm to calculate the nth Fibonacci number
n=2
function fib( integer 2) //n should be an integer
assert (20)
return fib(2 - 1) + fib(2 - 2)=1 // the 2
nd
Fibonacci number
end
It should be noted that this is just an example because there is already a
mathematically closed form for F
n
as shown in Eq 10.3

5
) 1 (
) (
n n
n F
u u
=
Eq: 10.3
where

2
5 1+
= u
(Golden Ratio) Eq: 10.4
Fib(3) Fib(4)
Fib(2)
Fib(0)
Fib(2)
Fib(5)
Fib(6)
Fib(4)
Fib(2) Fib(1)
Fib(2)
Fib(1) Fib(0)
Fib(1) Fib(0)
Fib(1)
Fib(3)
Fib(1)
Fib(1)
Fib(2)
Fib(0)
Fib(1) Fib(3)
Fib(1) Fib(0)
Fib(3) Fib(4)
Fib(2)
Fib(0)
Fib(2)
Fib(5)
Fib(6)
Fib(4)
Fib(2) Fib(1)
Fib(2)
Fib(1) Fib(0)
Fib(1) Fib(0)
Fib(1)
Fib(3)
Fib(1)
Fib(1)
Fib(2)
Fib(0)
Fib(1) Fib(3)
Fib(1) Fib(0)
Fib(4)
Fib(2)
Fib(0)
Fib(2)
Fib(5)
Fib(6)
Fib(4)
Fib(2) Fib(1)
Fib(2)
Fib(1) Fib(0)
Fib(1) Fib(0)
Fib(1)
Fib(3)
Fib(1)
Fib(1)
Fib(2)
Fib(0)
Fib(1) Fib(3)
Fib(4)
Fib(2)
Fib(0)
Fib(2)
Fib(5)
Fib(6)
Fib(4)
Fib(2) Fib(1)
Fib(2)
Fib(1) Fib(0)
Fib(1) Fib(0)
Fib(1)
Fib(3)
Fib(1)
Fib(1)
Fib(2)
Fib(0)
Fib(4)
Fib(2)
Fib(0)
Fib(2)
Fib(5)
Fib(6)
Fib(4)
Fib(2) Fib(1)
Fib(2)
Fib(1) Fib(0)
Fib(1) Fib(0)
Fib(1)
Fib(3)
Fib(1)
Fib(1)
Fib(2)
Fib(0)
Fib(4)
Fib(2)
Fib(0)
Fib(2)
Fib(5)
Fib(6)
Fib(4)
Fib(2) Fib(1)
Fib(2)
Fib(1) Fib(0)
Fib(6)
Fib(4)
Fib(2) Fib(1)
Fib(2)
Fib(1) Fib(0)
Fib(2) Fib(1)
Fib(2)
Fib(1) Fib(0)
Fib(2)
Fib(1) Fib(0)
Fib(1) Fib(0)
Fib(1)
Fib(3)
Fib(1)
Fib(1)
Fib(2)
Fib(0)
Fib(1) Fib(3) Fib(1) Fib(3)
Fib(1) Fib(0)
Hence using the equation F(n) we can calculate the nth Fibonacci number
efficiently when n is small. However when n is large, this method is very
inefficient.
In the equation Eq: 10.4, is known as the Golden Ratio. It is an irrational
mathematical constant which is approximately 1.6180339887. This unique
ratio has served as an inspiration to thinkers of all discipline be it art,
mathematics, architecture, physiology, biology etc.

Figure 10.1: Fibonacci Tree
To analyze the running time of Fibonacci sequence we will look at a call tree
for the sixth Fibonacci number:
In the Figure 10.1 every leaf of the tree has the value 0 or 1, and the sum of
these values is the final result. Thus for any n, the number of leaves in the
call tree is Fib
n
itself. The closed form therefore tells us that the number of
leaves in fib(n) is approximately equal to equation Eq: 10.5
n 69 . 0 ) 618 . 1 log( n )
n
618 . 1 log( n
n
2 2 2 618 . 1
2
5 1
~ = = ~
|
|
.
|
\
|
+

Eq: 10.5
This means that there are far too many leaves, considering the repeated
patterns found in the figure 10.1 tree. (The algebraic manipulation used in
equation Eq: 10.5 to make the base of the exponent as 2 should be duly
noted.)
We can use a recursive memoization algorithm that can be turned bottom-
up into an iterative algorithm that would fill in a table of solutions to sub-
problems. Here some of the sub-problems solved might not be needed at
Fib(3) Fib(4)
Fib(2)
Fib(0)
Fib(2)
Fib(5)
Fib(6)
Fib(4)
Fib(2) Fib(1)
Fib(2)
Fib(1) Fib(0)
Fib(1) Fib(0)
Fib(1)
Fib(3)
Fib(1)
Fib(1)
Fib(2)
Fib(0)
Fib(1) Fib(3)
Fib(1) Fib(0)
Fib(3) Fib(4)
Fib(2)
Fib(0)
Fib(2)
Fib(5)
Fib(6)
Fib(4)
Fib(2) Fib(1)
Fib(2)
Fib(1) Fib(0)
Fib(1) Fib(0)
Fib(1)
Fib(3)
Fib(1)
Fib(1)
Fib(2)
Fib(0)
Fib(1) Fib(3)
Fib(1) Fib(0)
Fib(3) Fib(4)
Fib(2)
Fib(0)
Fib(2)
Fib(5)
Fib(6)
Fib(4)
Fib(2) Fib(1)
Fib(2)
Fib(1) Fib(0)
Fib(1) Fib(0)
Fib(1)
Fib(3)
Fib(1)
Fib(1)
Fib(2)
Fib(0)
Fib(1) Fib(3)
Fib(1) Fib(0)
Fib(4)
Fib(2)
Fib(0)
Fib(2)
Fib(5)
Fib(6)
Fib(4)
Fib(2) Fib(1)
Fib(2)
Fib(1) Fib(0)
Fib(1) Fib(0)
Fib(1)
Fib(3)
Fib(1)
Fib(1)
Fib(2)
Fib(0)
Fib(1) Fib(3)
Fib(4)
Fib(2)
Fib(0)
Fib(2)
Fib(5)
Fib(6)
Fib(4)
Fib(2) Fib(1)
Fib(2)
Fib(1) Fib(0)
Fib(1) Fib(0)
Fib(1)
Fib(3)
Fib(1)
Fib(1)
Fib(2)
Fib(0)
Fib(4)
Fib(2)
Fib(0)
Fib(2)
Fib(5)
Fib(6)
Fib(4)
Fib(2) Fib(1)
Fib(2)
Fib(1) Fib(0)
Fib(1) Fib(0)
Fib(1)
Fib(3)
Fib(1)
Fib(1)
Fib(2)
Fib(0)
Fib(4)
Fib(2)
Fib(0)
Fib(2)
Fib(5)
Fib(6)
Fib(4)
Fib(2) Fib(1)
Fib(2)
Fib(1) Fib(0)
Fib(6)
Fib(4)
Fib(2) Fib(1)
Fib(2)
Fib(1) Fib(0)
Fib(2) Fib(1)
Fib(2)
Fib(1) Fib(0)
Fib(2)
Fib(1) Fib(0)
Fib(1) Fib(0)
Fib(1)
Fib(3)
Fib(1)
Fib(1)
Fib(2)
Fib(0)
Fib(1) Fib(3) Fib(1) Fib(3)
Fib(1) Fib(0)
the end while computing the result (and this is where dynamic programming
differs from memoization), but dynamic programming can be very efficient
because it can use the result stored in a better manner and have less call
overhead.
The pseudocode to compute the nth Fibonacci number is given as follows:
Pseudocode to compute the n
th
Fibonacci number:
if n == 0 or n == 1:
return n
else-if f[n] != -1:
return f[n]
else
f[n] = fib(n - 1) + fib(n - 2)
return f[n]
fi
end
In the above code if the value of fib(n) already has been calculated it's
stored in fib[n] and then returned instead of calculating it again. That
means all the copies of the sub-call trees can be exempted from the
calculation.

Figure 10.2: Fibonacci Tree Dynamically Approached
In the Figure 10.2 the values in the boxes are values that already have been
calculated and the calls can thus be skipped. Hence it is a lot faster than the
straightforward recursive algorithm. Here every value less than n is
calculated once only. Therefore the first time you execute it, the asymptotic
running time is O(n). Any other calls to it will take O(1) time since the values
have been pre-calculated (assuming each subsequent call's argument is
less than n).
However the algorithm does consume a lot of memory. When we calculate
fib(n), the values of fib(0) to fib(n) are stored in main memory, though this
can be improved, there is no need as the memory cost has fallen drastically.
The O(1) running time of subsequent calls is lost since the values aren't
stored. The value of fib(n) only depends on fib(n-1) and fib(n-2) hence we
can discard the other values by going bottom-up. If we want to calculate
fib(n), we first have to calculate fib(2) = fib(0) + fib(1). We then need to
calculate fib(3) by adding fib(1) and fib(2). After this we can discard the
values of fib(0) and fib(1), since we no longer need them to calculate any
further values. Now we can calculate fib(4) from fib(2) and fib(3) and discard
fib(2), then we can calculate fib(5) and discard fib(3) and so on. The
pseudocode to do this is given below.
Pseudocode to calculate n
th
Fibonacci number (Dynamic
programming approach):
if n == 0 or n == 1:
return n
fi

let u := 0
let v := 1

for i := 2 to n:
let t := u + v
u := v
v := t
repeat

return v
end
We can rework the code to store the values in an array for subsequent calls,
but, we don't have to. This method is typical for dynamic programming in
which we first identify the sub-problems that we need to solve in order to
solve the entire problem. Then using iterative process on bottom-up we can
calculate the values.
Activity 1
Find the various instances in the natural world where you find the
presence of the Fibonacci number and discuss it with your friends.
4. The formula to calculate the n
th
Fibonacci series is _______________.
5. The asymptotic running time when we first run to calculate the n
th

Fibonacci number is _______.
6. To compute the n
th
Fibonacci number we followed the __________
dynamic approach.
10.4 Binomial Coefficients
In the previous section we discussed about Fibonacci numbers and how we
can calculate the nth Fibonacci number using the memoization technique. In
this section we will be learning about the binomial coefficients.
10.4.1 Definition of binomial coefficient
The binomial coefficient is the number of ways of picking k unordered
outcomes from n possibilities. It is also known as a combination or
combinatorial number. The binomial coefficient is represented as
|
|
.
|
\
|
k
n
or
n
C
k
and can be read as n choose k
The value of the binomial coefficient is given by equation Eq: 10.6
! )! (
!
k k n
n
k
n
C
k n
|
|
.
|
\
|
for n k s s 0 Eq: 10.6

Writing the factorial as a gamma function
) 1 ( ! + I = z z
allows the binomial
coefficient to be generalized to non-integer arguments, including complex n
and k.
10.4.2 Computation of binomial coefficients
There are several ways by which we can calculate the binomial coefficients.
Here we will learn to calculate binomial coefficients using Pascals triangle
and the dynamic programming approach.
Pascals triangle
Pascals triangle helps you determine the coefficients which arise in
binomial expansion. For example consider equation Eq: 10.7
(x+y)
2
= x
2
+ 2xy + y
2
= 1x
2
y
0
+ 2x
1
y
1
+ 1x
0
y
2
Eq: 10.7

You might notice that the coefficients are numbers in the row of Pascals
triangle shown in the Figure 10.3.
0: 1
1: 1 1
2: 1 2 1
3: 1 3 3 1
4: 1 4 6 4 1
5: 1 5 10 10 5 1
6: 1 6 15 20 15 6 1
7: 1 7 21 35 35 21 7 1
8: 1 8 28 56 70 56 28 8 1
Figure 10.3: Pascals Triangle
Hence in general when x+y is raised to a positive integer we can give the
following equation Eq: 10.8
n
n
1 n
1 n
2 2 n
2
1 n
1
n
0
n
y a xy a y x a y x a x a ) y x ( + + + + + = +

Eq: 10.8
where the coefficients a
i
in this expansion are exactly the numbers on row n
of Pascal's triangle and are written as equation Eq: 10.9, which is the
binomial theorem.

|
|
.
|
\
|
=
i
n
a
i
Eq: 10.9
You will notice that the entire right diagonal of Pascal's triangle is the
coefficient of y
n
in the binomial expansions, while the next diagonal is the
coefficient of xy
n1
and so on.
Dynamic programming approach
Computing a binomial coefficient is a standard example of applying dynamic
programming to a non-optimization problem. You may recall from your
studies of elementary combinatorics that the binomial coefficient, denoted
by C (n,k) or
|
|
.
|
\
|
k
n
is the number of combinations (subsets) of k elements from
an n-element set (0 k n). The name binomial coefficients comes from
the participation of these numbers in the binomial formula given in equation
Eq: 10.10
n k k n n n
b n n C b a k n C a n C b a ) , ( ) . ( ) 0 , ( ) ( + + + + = +

Eq: 10.10
Of the many properties of binomial coefficients, we concentrate on two given
by equation Eq: 10.11

) , 1 ( ) 1 , 1 ( ) , ( k n C k n C k n C + =
for n > k > 0
and Eq: 10.11

1 ) , ( ) 0 , ( = = n n C n C

The nature of recurrence, which expresses the problem of computing C(n,k)
in terms of smaller and overlapping problems of computing C(n-1,k-1) and
C(n-1,k), lends itself to solving by dynamic programming technique. To do
this, we record the values of the binomial coefficients in a table of n+1 rows
and k+1 columns, numbered from 0 to n and from 0 to k respectively.
To compute C(n,k), we fill the table in the Figure 10.4 row by row, starting
with row 0 and ending with row n. Each row i (0 i n) is filled left to right,
starting with 1 because C(n, 0)= 1. Rows 0 through k also end with 1 on the
tables main diagonal: C(i,i)=1 for 0 i k . We compute the other entries by
the formula to calculate C(n,k), by adding the contents of the cells in the
preceding row and the previous column and in the preceding row and the
same column. It is precisely the implementation of Pascals triangle.
Algorithm to find binomial coefficient (n, k)
//Computes C(n,k) by the dynamic programming algorithm
//Input: A pair of nonnegative integers n k 0
for i 0 to n do
for j 0 to min (i,k) do
if j = 0 or j = i
C[i,j] 1
else C[i,j] C[i-1 , j-1 ] + C[i-1 , j]
return C[n , k]

Let us now trace the algorithm of the binomial coefficient.
Tracing of the algorithm of binomial coefficient (4,3)
n=4,k=3
for i = 0 to 4 do //this loop will run from i = 0 to i = 4
for j = 0 to min (0,3) do //this loop will run from j = 0 to j = 0
if j = 0 or j = 0
C[0,0] = 1
else C[0,0] = C[0-1 , 0-1 ] + C[0-1 , 0] //will not perform this step
as the if condition is satisfied
return C[4 , 3]
We can clearly see that the algorithms basic operation is addition, so let A
(n,k) be the total number of additions made by the this algorithm in
computing C (n,k) . Note that computing each entry by formula of C (n,k)
requires just one addition. Also note that because the first k+1 rows of the
table form a triangle while the remaining n-k rows form a rectangle, we have
to split the sum expressing A (n,k) into two parts as shown by the equation
Eq: 10.13.

= + = + = = =
=
+ = + =
k
i
k
k i
n
k i
k
j
k
i
i
j
k i k n A
1 1 1 1 1
1
1
) 1 ( 1 1 ) , (

) ( ) (
2
) 1 (
nk k n k
k k
O e +
=
Eq: 10.13

0 1 2 k-1 k
0 1
1 1 1
2 1 2 1
:
k 1 1
:
n-1 1 C(n-1, k) C(n-1, k-1) C(n,k)
N 1
Figure 10.4: Table for Computing Binomial Coefficient C(n,k)
The figure 10.4 shows the table for computing binomial coefficient.
7. What formula can we use to find the value of the binomial coefficient?
8. The gamma function
) 1 ( ! + I = z z
allows the binomial coefficient to be
generalized to non-integer arguments like _________________.
9. Binomial coefficients are a study of _____________.
10.5 Warshalls and Floyds Algorithm
In the previous section we discussed the dynamic programming approach to
calculate binomial coefficients. In this section, to solve the all pairs shortest-
paths problem on a directed graph G=(V, E), we shall use the Warshall and
Floyd algorithm. This Warshall-Floyd algorithm runs in (V
3
) time. The
negative-weight edges may be present here, but we assume that there are
no negative-weight cycles. We shall follow the dynamic-programming
process here to develop the algorithm.
Transitive closure
The transitive closure of a directed graph (digraph) with n vertices is an
n n matrix such that T [i, j] is true if and only if, j is reachable from i by
some path. Our aim is to figure the transitive closure of a directed graph
shown in Figure 10.5 where 10.5 (a) is the digraph, 10.5 (b) the adjacency
matrix, and 10.5 (c) the transitive closure. The adjacency matrix is a
restriction of matrix T if we only allow paths of length 1.
(
(
(
(
(
(
(
=
d c b a
d
c
b
a
A
0
0
1
0
1
0
0
0
0
0
0
1
1
0
0
0

(
(
(
(
(
(
(
=
d c b a
d
c
b
a
T
1
0
1
1
1
0
1
1
1
0
1
1
1
0
1
1

(a) (b) (c)
Figure 10.5: (a) Digraph (b) Its Adjacency Matrix (c) Its Transitive Closure
We can generate transitive closure of the digraph with the help of depth-first
search or breadth-first search. Performing either traversal starting at the i
th

vertex gives the information about the vertices reachable from the i
th
vertex
and hence the columns that contain ones in the i
th
row of the transitive
closure. Thus by doing such a traversal for every vertex as a starting point
we obtain the transitive closure in its eternity.

a
c
b
d
10.5.1 Overview of Warshalls and Floyds algorithm
The sub-problems here are connectivity/distance information where only a
subset of vertices is allowed in the paths. They are very useful to obtain
information about all pairs of vertices even though they have complexity
(n
3
).
There are other algorithms if we only want the information about one pair,
(e.g. DFS for reachability). For sparse graphs there may be better
algorithms. Moreover you have to be careful to use 1 to represent the
absence of an edge.
10.5.2 Warshalls algorithm
Since the above mentioned method traverses the same digraph several
times, we use a better algorithm called the Warshalls algorithm named after
S. Warshall.
R
(0)
,.,R
(k-1)
,

R
(k)
,..,R
(n)
Eq: 10.13
Warshalls algorithm constructs the transitive closure of a given digraph with
n vertices through a series of n x n Boolean matrices. This series is shown
in equation Eq: 10.13

Figure 10.6: Rule for Changing Zeroes in Warshalls Algorithm
Each of these matrices provides certain information about directed paths in
the digraph (Refer figure 10.6). Specially, the element
) (k
ij
r
in the i
th
row and
j
th
column of matrix R
(k)
(k=0,1,,n) is equal to 1 if and only if there
exists a directed path (of a positive length ) from the i
th
vertex to the j
th
vertex
with each intermediate vertex, if any, not higher than k.
Let us consider the example in the figure 10.7. Here R
(k)
= R
(4)
where k=0, 1,
2, 3, 4. Hence, the series starts with R
(0)
(Refer Figure 10.7). It doesnt allow
any intermediate vertices in its path, hence R
(0)
is nothing but adjacency
matrix of the digraph. R
(1)
contains the information about the paths that can
use the first vertex as intermediate with more freedom. Therefore it may
contain more ones than R
(0)
. In general each subsequent matrix in the series
( )
1
1 1
j k
k
k
R
i
(
(
(
=
(
(
(
(

has one more vertex to use as intermediate for its paths than its
predecessor and hence may, but does not necessarily have to, contain
more ones. R
(4)
being the final matrix in the series, reflects paths that can
use all n vertices of the digraph as intermediate and hence is the digraphs
transitive closure.

Figure 10.7: Application of Warshalls Algorithm to the Digraph
The central part of the algorithm is that we can compute all the elements of
each matrix R
(k)
from its immediate predecessor R
(k-1)
in the series, like in
the example we can compute R
(4)
from R
(3)
. Let
) (k
ij
r
, the element in the i
th

row and the j
th
column of matrix R
(k)
, be equal to 1. This means that there
exists a path from the i
th
row and the j
th
vertex v
j
with each intermediate
vertex numbered not higher than k.
a
c
b
d
(
(
(
(
(
(
(
=
d c b a
d
c
b
a
R
1
0
1
1
1
0
1
1
1
0
1
1
1
0
1
1
) 4 (
v
i
, a list of intermediate vertices each numbered not higher than k, v
j.

Two situations regarding this path are possible. In the first, the list of its
intermediate vertices does not contain k
th
vertex. Then this path from v
i
to v
j
has intermediate vertices numbered not higher than k-1, and therefore
) 1 ( k
ij
r
is equal to 1 as well.
Warshalls Algorithm (A[1n,1n])
//Implements Warshalls algorithm for computing the transitive closure
//Input: The adjacency matrix A of a digraph with n vertices
//Output: The transitive closure of the digraph
R
(0)
A
for k 1 to n do
for i 1 to n do
for j 1 to n do
R
(k)
[i, j] R
(k-1)
[i, j] or (R
(k-1)
[i, k] and R
(k-1)
[k, j])
return R
(n)

Let us now trace Warshalls algorithm.
Tracing of Warshalls algorithm for A[3,3]
R
(0)
= A // A is an adjacency matrix of 3x3 which is assigned to R
(0)

for k = 1 to 3 do //this loop will run from k = 1 to k = 3
for j = 1 to 3 do //this loop will run from j = 1 to j = 3
R
(1)
[1, 1] R
(1-1)
[1, 1] or (R
(0-1)
[1, 1] and R
(1-1)
[1, 1])
return R
(3)

Several observations need to be made about Warshalls algorithm. First, it is
very concise. Still its time efficiency is only in (n
3
). In fact, for sparse graphs
represented by their adjacency lists, the traversal-based algorithm has a
better asymptotic efficiency than Warshalls algorithm.
We can speed up the implementation of Warshalls algorithm for some
inputs by restructuring its inner most loop. It can also be made to work faster
by considering matrix rows as bit strings and apply the bitwise-or operation.
As to the space efficiency of Warshalls algorithm, the situation is similar to
that of the two earlier examples of computing Fibonacci numbers and
binomial coefficients. Although we used separate matrices for recording
intermediate results of the algorithm, it is in fact unnecessary.
10.5.3 Floyds algorithm
We can generate the distance matrix with an algorithm that is very similar to
Warshalls algorithm. It is called Floyds algorithm, after its inventor R. Floyd.
It is applicable to both undirected and directed weighted graphs provided
that they do not contain a cycle of negative length.

(a)

(b) (c)
Figure 10.8: (a) Digraph (b) Its Weight Matrix (c) its Distance Matrix
The all-pairs-shortest paths problem asks to find the distances from each
vertex to all other vertices in a weighted connected graph. For our
convenience to record the lengths of the shortest paths we use an n x n
matrix D called the distance matrix. The element d
ij
in the i
th
row and the j
th

column of this matrix indicates the length of the shortest path from the i
th

vertex to the j
th
vertex (1 i, j n). An example of this is shown in Figure
10.8 where 10.8 (a) is the digraph, 10.8 (b) the weight matrix, and 10.8
(c) the distance matrix.
Floyds algorithm computes the distance matrix of a weighted graph with a
series of n x n matrices as given in equation Eq: 10.14
D
0
,..D
(k-1)
,D
(k)
,..,D
(n)
Eq: 10.14

Each of these matrices contains the length of the shortest paths with certain
constraints on the paths considered for the matrix in question. Specifically,
(
(
(
(
(
(
(
=
d c b a
d
c
b
a
W
0
1 0
3
7
0
6
2
0
(
(
(
(
(
(
(
=
d c b a
d
c
b
a
D
0
1
6
4
9
0
5
3
16
7
0
10
6
7
2
0
6
c d
3
2
7
1
a b
the element
) (k
ij
d in the i
th
row and the j
th
column of matrix D
(k)
(k=0,1,,n) is
equal to the length of the shortest path among all paths from the i
th
vertex to
the j
th
vertex with each intermediate vertex, if any, are not numbered higher
than k. In particular, the series starts with D
(0)
which does not allow any
intermediate vertices in its path; hence D
(0)
is nothing but the weight matrix
of the graph, like in the example in the figure 10.9. The last matrix in the
series, D
(n)
contains the length of the shortest paths among all the paths that
can use all n vertices as immediate. This is nothing but the distance matrix
being sought.
As in Warshalls algorithm, we can compute all the elements of each matrix
D
(k)
from its immediate predecessor D
(k-1)
in series. Let
) (k
ij
d be the element
in the i
th
row and the j
th
column of matrix D
(k)
. Hence
) (k
ij
d is the length of the
shortest path among all the paths from the i
th
vertex v
i
to the j
th
vertex v
j
with
their intermediate vertices numbered not higher than k.
v
i
, a list of intermediate vertices each numbered not higher than k, v
j
.

Figure 10.9: Application of Floyds Algorithm
We can partition all such paths into two disjoint subsets, those that do not
use the k
th
vertex v
k
as intermediate and those that do. Since the paths of
the first subset have their intermediate vertices numbered not higher than
k-1, the shortest of them is by definition of our matrices of length
) 1 ( k
ik
d .
If the graph does not contain a cycle of a negative length, our attention gets
focused only to the paths in the second subset that uses vertex v
k
as its
intermediate vertex exactly once. All such paths have the following form.
v
i
, vertices numbered k-1, v
k
, vertices numbered k-1, v
j
.
In other words, each of the paths is made up of a path from v
i
to v
k
with each
intermediate vertex numbered not higher than k-1 and a path from v
k
to v
j

with each intermediate vertex numbered not higher than k-1.
Since the length of the shortest path from v
i
to v
k
among the paths that use
the intermediate vertices numbered not higher than k-1 and a path from v
k
to
v
j
with each intermediate vertex numbered not higher than k-1 is equal to
) 1 ( k
kj
d and the length of the shortest path from v
k
to v
j
among the paths that
use intermediate vertices numbered not higher than k-1 is equal to
) 1 ( k
ik
d +
) 1 ( k
kj
d . Taking into account the lengths of the shortest paths in both
subsets, lead to the recurrence shown by equation Eq: 10.15.
) 1 ( k
ij
d = min {
) 1 ( k
ij
d ,
) 1 ( k
ik
d +
) 1 ( k
ikj
d } for k 1,
0
ij
d
=w
ij
Eq: 10.15
Putting it another way, the element in the i
th
row and the j
th
column of the
current distance matrix D
(k-1)
is replaced by the sum of elements in the same
row i and the k
th
column and in the same column j and the k
th
column if and
only if the latter sum is smaller than its existing value.
Floyds algorithm: W[1n,1n]
//Implements Floyds algorithm for all-pairs shortest-path problem
//Input: The weight matrix W of a graph with no negative-length cycle
//Output: The distance matrix of the shortest paths lengths
D W //is not necessary if W can be over written
for k 1 to n do
for i 1 to n do
For j 1 to n do
D[i , j] min{ D[i,j], D[i,k]+D[k,j]}
return D
Let us now trace the Floyds algorithm.
Tracing of Floyds algorithm

D = W //W is a weighted matrix of 3x3 assigned to D which is the distance
matrix
for k =1 to 3 do //this loop will run from k = 1 to k = 3
for j = 1 to 3 do //this loop will run from j = 1 to j = 3
D[1 ,1] = min{ D[1,1], D[1,1]+D[1,1]}
return D
Activity 2
Write a pseudocode to find the weight matrix and the distance matrix for
a digraph.
10. Both Warshalls and Floyds algorithms have the run time as ________.
11. The Warshalls algorithm is used to solve _______________ problems.
12. The Floyds algorithm is used to solve ________________ problems.
10.6 Summary
In this unit we have learned about the dynamic programming technique
which is a widely acclaimed tool used in applied mathematics and in
computer science wherein it is regarded as a general algorithm design.
In dynamic programming we have learned the technique to solve
overlapping problems by solving the sub-problem only once, and storing the
result in a table as we have seen in the case of Fibonacci numbers.
Similarly we have also learnt to apply the concept to find the binomial
coefficient and find solutions to the transitive closure and shortest path
problems utilizing Warshalls and Floyds algorithms respectively.
The remaining concepts like principle of optimality, optimal binary search,
Knapsack problem and memory functions will be covered in the next unit,
which will help you to broaden your horizon of dynamic programming.

10.7 Glossary
Term Description
Adjacency matrix Representation of the vertices of a graph adjacent to the
other vertices.
Combinatorics
Combinatorics is a branch of mathematics concerning the
study of finite or countable discrete structures.
Digraph A digraph is short for directed graph, and it is a diagram
composed of points called vertices (nodes) and arrows called
arcs going from a vertex to a vertex.
1. Differentiate between dynamic programming and Divide and Conquer
techniques.
2. Write the algorithm to find the n
th
Fibonacci number.
3. Explain the dynamic programming approach to find binomial coefficients.
4. What is Warshalls algorithm to find the transitive closure?
5. Explain the Floyds algorithm to find the shortest path.
10.9 Answers
1. Top-down, bottom-up
2. Memoization
3. Divide and conquer
4.
5
) 1 (
) (
n n
n F
u u
=

5. O(n)
6. Bottom-up
7.
! )! (
!
k k n
n
k
n
C
k n
|
|
.
|
\
|

8. Complex n and k
9. Combinatorics
10. (n
3
)
11. Transitive closure
12. Shortest path
Terminal Questions
1. Refer to10.2.3 Dynamic programming Vs divide and conquer
2. Refer to 10.3.2 Algorithm to find n
th
Fibonacci number
3. Refer to 10.4.2 Computation of binomial coefficients
4. Refer to 10.5.2 Warshalls algorithm
5. Refer to 10.5.3 Floyds algorithm
References
- Anany V. Levetin (2002). Introduction to the analysis and design of
algorithms. Addison-Wesley Longman Publishing Co.
- Cormen, Thomas H., &Charles E. Leiserson., &Ronald L Rivest.,
&Clifford Stein (2006). Introduction to algorithms, 2nd Edition, PHI
E-References
- http://mathworld.wolfram.com/BinomialCoefficient.html
- http://students.ceid.upatras.gr/%7Epapagel/project/kef5_6.htm
Unit 11 Dynamic Programming 2
Structure:
11.1 Introduction
Objectives
11.2 Principle of Optimality
11.3 Optimal Binary Search Trees
Solving binary search trees using dynamic programming
11.4 Knapsack Problem
Solving Knapsack problem using dynamic programming
11.5 Memory Functions
Solving Knapsack problem using memory functions
11.6 Summary
11.7 Glossary
11.9 Answers
11.1 Introduction
In the previous unit we studied some concepts of dynamic programming. As
you know, dynamic programming is defined as an optimization technique
that is used for particular classes of backtracking algorithms where the sub
problems are repeatedly solved.
The basic steps for dynamic programming are given below:
1) Develop a mathematical notation that is used to find a solution and sub
solutions for the problem.
2) Prove the solution using the Principle of Optimality.
3) Derive a recurrence relation that solves the sub solutions using the
mathematical notation in step 1.
4) Write an algorithm to compute the recurrence relation.
Sometimes, we have to solve problems optimally. At some other time a
non-optimal solution also gives a good solution. We cannot judge a problem
by a single criterion. Optimization of a problem is useful in any type of
problem we have to solve.
This unit defines the Principle of Optimality and analyzes binary search
trees using dynamic programming. It also introduces the Knapsack problem
and solves an instance of it using dynamic programming and memory
functions.
Objectives:
- define Principle of Optimality
- analyze optimal binary search trees with an example
- describe Knapsack problem with an example
- explain memory functions with an example
11.2 Principle of Optimality
Principle of Optimality is defined as a basic dynamic programming principle
which helps us to view problems as a sequence of sub problems.
Richard Ernest Bellman, a mathematician, invented the Principle of
Optimality. This principle explains that an optimal path has the property that
whatever the initial conditions and choices over some initial period, the
decision variables chosen over the remaining period must be optimal for the
remaining problem. Therefore, we can say that the optimal solution to a
problem is a combination of optimal solutions of its sub problems.
We might face a difficulty in converting the Principle of Optimality into an
algorithm, as it is not very easy to identify the sub problems that are relevant
to the problem under consideration. Bellman developed an equation for the
Principle of Optimality. We can study this equation only with the help of
dynamic programming concepts.
All optimization problems tend to minimize space and time, and maximize
profits. There is a mathematical function defined for this namely,
optimization function. Every instance of a dynamic programming problem
should be tracked as it involves the use of various different sub problems to
solve it. This information about the current situation required to make a
correct decision is known as a state. The variables we choose for solving
the problem at any point of time are called control variables. At every state
we have to choose the control variable with the help of the previous state.
The rule that determines the controls as a function of states is known as
policy function. The best possible value of the problem objective, written as
a function of the state, is called the value function.
Bellman formulated an equation for the Principle of Optimality during the
early stages in technology development. The computers used during that
stage were not as powerful as we use now. This principle may help in
advanced dynamic programming applications which support larger
dimensions than that used today.
We can derive the Bellman equation using a step by step recursive process
of writing down the relationship between the value function of one stage and
the value function of the next stage.
The Bellman equation for Principle of Optimality is given as:
V
N
(x
0
) =
( ) | | |
.
|
\
|
+
1 1 1 0 0 1 0 0 1 1 , , ) , ( max
1
x V x a x r a x x p N
x
a
Eq: 11.1
In the equation Eq: 11.1, V
N
(x
0
) is the value function where N is the number
of decision steps. We are aware that the value function explains the best
possible value of the objective, as a function of the state x
0
. Here
)] ( ) , , ( )[ , ( 1 1 1 0 0 1 0 0 1 1 x V x a x r a x x p N +
are the policies with respect to
distribution.
1. Principle of __________ is defined as a basic dynamic programming
principle which helps us to view problems as a sequence of sub
problems.
2. ______________, a mathematician, invented the Principle of Optimality.
3. All optimization problems tend to minimizing cost, time and maximizing
________.
11.3 Optimal Binary Search Trees
Now that we have discussed the basic Principle of Optimality, let us
optimize a binary search tree. First, let us define a binary search tree.
Binary search tree Binary search tree is defined as a binary tree with the
following properties:
- The values of elements in the left sub-tree of a node are lesser than the
nodes value.
- The values of elements in the right sub-tree of a node are greater than
the nodes value.
- The right and left sub-trees of a node are also binary search trees.
Binary search trees are node based data structures used in many system
programming applications for managing dynamic sets. An example for
binary search trees is given in figure 11.1.

Figure 11.1: Binary Search Tree
We use binary search trees for applications such as sorting, searching and
inorder traversal.
The four main operations that we perform on binary trees are:
Searching Here we match the searching element with the root node first,
left sub-tree, right sub-tree until we find the node or if no nodes are left. We
can search a binary search tree using the pseudocode given below.
Pseudocode for Searching a Binary Search Tree
find(Y, node){
if(node = NULL)
return NULL
if(Y = node:data)
return node
else if(Y < node:data)
return find(Y,node:leftChild)
else if(Y> node:data)
return find(Y,node:rightChild)
Insertion If the root node of the tree does not have any value, we can
insert the new node as the root node. For inserting a new element in an
existing binary search tree, first we compare the value of the new node with
the current node value. If the value of the new node is less than the current
node value, we insert it as a left sub-node. If the value of the new node is
greater than the current node value, then we insert it as a right sub-tree.
Let us now discuss the pseudocode for inserting a new element in a binary
search tree.
Pseudocode for Inserting a Value in a Binary Search Tree
//Purpose: insert data object Y into the Tree
//Inputs: data object Y (to be inserted), binary-search-tree node
//Effect: do nothing if tree already contains Y;
// otherwise, update binary search tree by adding a new node containing
data object Y
insert(Y, node){
if(node = NULL){
node = new binaryNode(Y,NULL,NULL)
return
}
if(Y = node:data)
return
else if(Y < node:data)
insert(Y, node:leftChild)
else // Y > node:data
insert(Y, node:rightChild)
}
Deletion If the node to be removed has no children, we can just delete it.
If the node to be removed has one child, then the node is deleted and the
child is connected directly to the parent node.
To remove a node which has two children, we adopt the following
procedure:
1) We find the minimum value in the right sub-tree
2) We replace the node to be removed with the minimum value found.
3) We then remove the duplicate value from the right sub-tree.

We can delete an existing element from a binary search tree using the
following pseudocode:
Pseudocode for Deleting a Value from a Binary Search Tree
//Purpose: delete data object Y from the Tree
//Inputs: data object Y (to be deleted), binary-search-tree node
//Effect: do nothing if tree does not contain Y;
// else, update binary search tree by deleting the node containing data
object Y
delete(Y, node){
if(node = NULL) //nothing to do
return
if(Y < node:data)
delete(Y, node:leftChild)
else if(Y > node:data)
delete(Y, node:rightChild)
else { // found the node to be deleted! Take action based on number of
node children
if(node:leftChild = NULL and node:rightChild = NULL){
delete node
node = NULL
return}
else if(node:leftChild = NULL){
tempNode = node
node = node:rightChild
delete tempNode}
else if(node:rightChild = NULL){
(similar to the case when node:leftChild = NULL)
}
else { //replace node:data with minimum data from right sub-tree
tempNode = findMin(node.rightChild)
node:data = tempNode:data
delete(node:data,node:rightChild)
}
}
}
Traversal Traversing a binary search tree involves visiting all the nodes in
the tree. First we visit the root node. Then we visit its left and right sub-trees.
We can visit a node only once. We can traverse a binary tree recursively
using the following pseudocode:
Pseudocode for Traversing a Binary Search Tree
PROCEDURE PreOrder (Binary_Tree_Node N)
BEGIN
ProcessNode(N)
If (N's left child is NOT NULL)
BEGIN
PreOrder(N's left child)
END
If (N's right child is NOT NULL)
BEGIN
PreOrder(N's right child)
END
END
The time taken to perform operations on a binary search tree is directly
proportional to the height of the tree. The insertion, deletion and search
operations has an average case complexity of O(log n), where n is the
number of nodes in the binary search tree.
If we know the frequency of operations performed on a binary search tree,
then, instead of modifying it, we can optimize it. The basic criterion for
optimizing a binary tree is that every binary tree should have a root node
and optimal sub-trees under it. We can use the principles of dynamic
programming to optimize a binary search tree.
Let us consider an example to search six elements P, Q, R, S, T and U with
probabilities 1/8, 1/32, 1/16, 1/32, 1/4 and 1/2. Here we can form 132
different binary search trees. One of them is shown in the figure 11.2.

Figure 11.2: Binary Search Trees
For the trees in the figure 11.2, we can find the average number
comparisons from the equation Eq: 11.2.
Average number comparisons for a tree =
x n* Eq: 11.2
Where, n is the value of the node and x is the level of the node in the tree.
For the tree of figure 11.2, the average number of comparisons is given as
(1/16 * 1) + (1/32 * 2) + (1/4 * 2) + (1/8 * 3) + (1/32 * 3) + (1/2 * 3)
~
2.6.
Here we can see that the tree is not optimized. We can generate all 132
binary search trees and analyze them to find the optimal one. If we start this
task using the binary search tree algorithms the task becomes exhaustive.
The total number of binary search trees with n elements is equal to the n
th
Catalan number, c(n), given in Eq 11.3.
c(n) =
1
1
2
+
|
|
.
|
\
|
n n
n
for n>0, c(0)
~
1 Eq: 11.3
The equation Eq: 11.3 reaches infinity as fast as 4
n
/n
1.5
.
Let us use dynamic programming approach to solve this problem. Let a
1
,
a
2
a
n
be the distinct elements given in ascending order and let p
1
, p2p
n

be the probabilities of searching the elements. Let c[i,j] be the smallest
average number of comparisons made in a binary search tree
j
i
T of
elements ai.aj, where i,j are some integer indices, 1
s
i
s
j
s
n.
Now let us find the values of C[i,j] for its the sub instances. We have to
choose a root a
k
for keys a
i
.a
j
, so as to derive the recurrence relation
using dynamic programming. For such binary search tree, the root consist of
the key a
k
, the left sub-tree
1 k
i
T contains keys aiak-1 optimally arranged
and the right sub-tree
j
k
T
1 +
contains keys a
k+1
a
j
also optimally arranged.
Here we are taking advantage of the Principle of Optimality. If we start
counting tree levels at 1 then we can derive the following recurrence
relation:
C[i,j] =
{
j k i
p
s s
min
k
. 1 +
=
1
1
k
s
p
s
. (level of a
s
in
1 k
i
T +1)
+
+ =
i
k s
p
1
s
. (level of a
s
in
j
k
T
1 +
+1)
}

=
{
j k i
p
s s
min
k
+
=
1
1
k
s
p
s
. level of a
s
in
1 k
i
T +
=
1 k
i s
p
s

+
+ =
i
k s
p
1
s
. level of a
s
in
j
k
T
1 +
+
+ =
j
k s
p
1
s
}

=
{
j k i s s
min

=
1
1
k
s
p
s
. level of a
s
in
1 k
i
T +
+
+ =
i
k s
p
1
s
. level of a
s
in
j
k
T
1 +
+
=
j
i s
p
s
}

=
} {
j k i
j k C k i C
s s
+ + ] , 1 [ ] 1 , [ min
+
=
j
i s
p
s
Thus, we have the recurrence relation given in Eq 11.3.
C[i,j] =
} {
j k i
j k C k i C
s s
+ + ] , 1 [ ] 1 , [ min
+
=
j
i s
p
s
for n j i s s s 1 . Eq: 11.4
In the recurrence relation given by Eq: 11.4, let us assume that C[i,i-1]
~
0
for 1
s
i
s
n+1. This we can interpret as the number of comparisons in the
empty tree. The figure 11.3 shows the values required to compute the C[i,j]
formula.

Figure 11.3: Dynamic Programming Algorithm for Optimal Binary Search Tree
In figure 11.3, we can find the values at row i and columns to the left of
column j and in column j and the rows below the row i. The arrows shown
point to the pairs of entries that are added up and the smallest one is
recorded as the value of C[i,j]. We have to fill the table along its diagonal,
starting with zeroes on the main diagonal and with probabilities given
as p
i
,
n i s s 1
, and moving toward the upper right corner.
This algorithm helps us to compute C[1,n], the average number of
comparisons for the successful searches in the optimal binary search tree.
We have to maintain another two dimensional table to record the value of
k for which the minimum is achieved. The table will be same as the one in
figure 11.3, and will be filled in the same manner. The table entries will start
at R[i,i] for
n i s s 1
and is used to find the optimal solution.

Let us next discuss the dynamic programming algorithm for binary search
tree optimization.
Dynamic Programming Algorithm for Binary Search Tree
Optimization
//Input: An array P[1..n] of search probabilities for a sorted list of n keys
//Output: Average number of comparisons in successful searches in the
//optimal binary search tree and table of sub trees roots in the optimal
//binary search tree
for i
1 to n do
C[i,i-1]
0
C[i,i]
P[i]
R[i,i]
i
C[n+1,n]
0
for d
1 to n-1 do//diagonal count

for i
1 to n - d do
j
i + d
minval

for k
I to j do
if C[i, k-1]+C[k+1, j]< minval
minval
C[I,k-1]+ C[k+1,j]; kmin
k
R[i,j]
kmin
sum
P[i];for s
i+1 to j do sum
sum + P[s]
C[I,j]
minval + sum
return c[1,n],R
Let us now trace the dynamic programming algorithm for binary search tree
optimization.
Algorithm Tracing for Binary Search Tree Optimization
P[5]={1,2,3,4,5}, n=5;
C[5,5]=0//array for comparisons in successful search
R[5,5]=0//root array
for i = 1 to 5 do //this loop will occur from i = 1 to i = 5
C[1,0]=0;
C[1,1]=P[1]=1// value of first element in the array is assigned to C[1,1]
R[1,1]=1;
C[6,5]=0;
For d=1 to 5-1// this loop will occur from d = 1 to d = 4
for i=1 to 5-1
j= 1+1
minval= infinite value
for k=1 to 2 do // this loop will occur from k = 1 to k = 2
if C[1,0] + C[2,2] < infinite value
minval = C[1,0] + C[2,2]=0+0
kmin=1;R[1,2]=1
Sum=P[1]=1;for s= 1+1 to 2 do
sum=P[1]+P[s]=1+2=3// s is assigned a value 2 in the previous step
C[1,2]=minval + sum = 0 + 3=3
return 3,1
The space efficiency of this algorithm is in quadratic terms and the time
efficiency is in cubic terms. We can also see that the values of the root table
are always non-decreasing along each row and column. This limits values
for R[i,j] to the range r[i,j-1],.r[i+1,j] and makes it possible to reduce the
running time of the algorithm to
O
(n
2
).
11.3.1 Solving binary search trees using dynamic programming
Let us illustrate the above mentioned algorithm using the four keys that we
used in the previous section. The keys and the probabilities are given in
table 11.1.
Table 11.1: Table of Keys and Probabilities
Key P Q R S T U
Probability 1/8 1/32 1/16 1/32 1/4 1/2

At initial stage the main table - table 11.2 is given as:
Table 11.2: Main Table

0 1 2 3 4 5 6
1 0 1/8
2 0 1/32
3 0 1/32
4 0 1/16
5 0 1/4
6 0 1/2
7 0
Let us compute C[1,2] as shown in equation Eq:11.5:
C[1,2] =
= + + =
+ + =
= + + = + + +
=
2
2
1
2 / 1 32 / 1 0
] 2 , 2 [ ] 0 , 1 [ : 1
2 / 1 0 8 / 1 ] 2 , 3 [ ] 1 , 1 [ : 1
min
s
s
Ps
C C k
Ps C C k =3/16 Eq:11.5
Thus, from the two possible binary trees P and Q, the root of the optimal
tree has index 2 and the average number of comparisons in a successful
search in the tree is 3/6.
Let us complete the above given table. The completed table 11.3 is the main
table.
Table 11.3: Main Table

0 1 2 3 4 5 6
1 0 1/8 3/16 9/32 15/32 31/32 63/32
2 0 1/32 3/32 7/32 19/32 47/32
3 0 1/32 1/8 15/32 21/16
4 0 1/16 3/8 19/16
5 0 1
6 0 1/2
7 0
Thus we can compute the average numbers of key comparisons in the
optimal tree to be 63/32. According to these probabilities, the optimal tree is
shown in the figure 11.4.
.
Figure 11.4: Optimal Binary Search Tree
4. ________________are node based data structures used in many
system programming applications for managing dynamic sets.
5. The Insertion, deletion and search operations of a binary search tree
has an average case complexity of _________.
6. The time taken to perform operations on a binary search tree is directly
proportional to the ________ of the tree.
11.4 Knapsack Problem
In this section we will define and analyze the Knapsack problem. Let us first
define the Knapsack problem.
If a set of items are given, each with a weight and a value, determine the
number of items that minimizes the total weight and maximizes the total
value.

Figure 11.5: Knapsack Example
Consider a situation where a thief breaks into a store and tries to fill his
knapsack with as much valuable goods as possible. The figure 11.5 given
above shows the number of goods with its value and weight. There are 3
items given with weights 10 kg, 20 kg and 30 kg with values Rs.60, Rs.100
and Rs. 120 respectively. The capacity of the knapsack is given as 50 kg.
We have to fill the knapsack with the items appropriately to get the
maximum value, but not exceeding the weight 50Kg
Let us try to fill the knapsack using different items as shown in the
figure 11.6.

Figure 11.6: Different Ways of Filling the Knapsack
Firstly we try to fill it using item 2 and 3 and the values add up to Rs 220.
Secondly we try to fill the knapsack using items 1 and 2 but these weights
do not fill the knapsack completely. Thirdly, we try to fill it with items 1 and 2.
This also does not fill the knapsack.
Now let us see the best possible solution for this problem from the
figure 11.7.

Figure 11.7: Optimal Solution for Knapsack Problem
Here we take items 1 and 2 as such and we take the 20/30 part of item 3.
Thus the values add up to Rs 240, which is the maximum value. Now let us
formally define the Knapsack problem.
Formal definition: There is a knapsack of capacity c > 0 and N items. Each
item has value v
i
> 0 and weight w
i
> 0. Find the selection of items (
i o
= 1 if
selected, 0 if not) that fit,
=
s
1 i
i iw No
c and the total value,
=1 i
i iv No
is
maximized. This is known as the 0-1 Knapsack problem or the Binary
Knapsack problem.
Let us see the Dynamic programming algorithm for the Knapsack problem:
Dynamic Programming Algorithm for Knapsack Problem
Dynamic knapsack (v,w,n,W)
FOR w = 0 TO W
DO c[0, w] = 0
FOR i=1 to n
DO c[i, 0] = 0
FOR w=1 TO W
DO IF wi w
THEN IF vi + c[i-1, w-wi]
THEN c[i, w] = vi + c[i-1, w-wi]
ELSE c[i, w] = c[i-1, w]
ELSE
c[i, w] = c[i-1, w]
Let us now trace the dynamic programming algorithm for Knapsack
problem.
Algorithm Tracing for Knapsack problem
v[3] = {1,2,3}, W= 5,n=3,C[5,5]=0 w[5]=0//wi and vi are arrays for weights
and values
FOR w = 0 TO 5// this loop will occur from w = 0 to w = 5
DO c[0, 0] = 0
FOR i=1 to 3 // this loop will occur from i = 1 to i = 3
DO c[1, 0] = 0
FOR w=1 TO 5 // this loop will occur from w = 1 to w = 5
DO IF 0 0
THEN IF v1 + c[1-1, w-w1] //this value is calculated as 1+0=1
THEN c[1, 0] = v1 + c[1-1, w-w1] //this value is calculated as
1+0=1
ELSE c[1, 0] = c[1-1, 0]
ELSE
c[1, 0] = c[1-1, 0]

The different types of Knapsack problems are:
Fractional Knapsack problem If we have materials of different values
per unit volume and maximum amounts, the Fractional Knapsack problem
finds the most valuable mix of materials which fit in a knapsack of fixed
volume. We have to take as much as possible material that is most valuable
per unit volume. Continue this process until the knapsack is full.
Bounded Knapsack problem If we have the types of items of different
values and volumes, find the most valuable set of items that fit in a
knapsack of fixed volume. Here the number of items of each type is
unbounded. This is an NP-hard optimization problem.
Now let us design a dynamic programming algorithm for the Knapsack
problem. We have n number of items with weights w
1
, w2w
n
and values
v
1
, v2.v
n
. The capacity of knapsack is given as W. We have to find the
most valuable subset of items that fit into the knapsack. Here, we assume
that the knapsack capacity and the weights given are positive integers and
the item values are not necessarily integers.
As we have done for every problem in dynamic programming, we have to
form a recurrence relation to solve the Knapsack problem. This recurrence
relation expresses the problem using its sub instances.
Let the instance defined by the first i items be 1 I n, the weights be
w
1
w
i
and the values be v
1
.v
i
. The capacity of knapsack is given as j,
where 1 j w. Let us also assume that V[i,j] be the value of the most
valuable subset of the first i items that fit into the knapsack with
capacity j. V[i,j] gives the optimal solution to the Knapsack problem. We can
split the i number of items that fit into the knapsack with capacity j into two.
These are as given below.
- We can have the subsets that do not include the i
th
item. Here the value
of the optimal subset is given as V[i-1,j].
- We can have the subsets that do include the i
th
item. An optimal subset is
made out of this item and another optimal subset from first i-1 items that
fit into the knapsack of capacity j-1. Here the value of the optimal subset
is given as v
i
+ V[i-1,j-w
i
].
The value of an optimal solution from these two feasible subsets of the
first i items is the maximum of these two values. If we cannot fit the i
th
item
in the knapsack, then the value of an optimal solution from the first i items is
the same as the value of an optimal subset selected from the first i-1 items.
Thus we can arrive at a recurrence relation as given in equation Eq: 11.6.
V[i,j] =

0 } , 1 [
0 ]} , 1 [ ], 1 [ max{
<
> +
i
i i i
w ifj j i V
w ifj w i V v i V
Eq:11.6
We can define the initial conditions as
V[0,j] = 0 for j
>
0 and v[i,0] = 0 for i
>
0.
Now we have to find V[n,W], the maximum value of a subset of the n given
items that fit into the knapsack of capacity W. This should be an optimal
subset. Table 11.4 illustrates the values computed from the equations. We
can fill the table either row wise or column wise.
To compute the entry in the i
th
row and the j
th
column, V[i,j]:
- We compute the maximum of the entry in the previous row and the same
column.
- We compute the sum of v
i
, the entry in the previous row and w
i
columns
to the left.
Let us compute the Knapsack problem using the table 11.4.
Table 11.4: Table for Solving the Knapsack Problem

0 j-w
i
J W
0 0 0 0 0
i-1 0 V[i-1,j-W
1
] V[i-1,j]
I 0 V[i,j]
N 0 goal

11.4.1 Solving Knapsack problem using dynamic programming
Let us solve an instance of Knapsack problem using dynamic programming.
Consider the following data given in table 11.5:
Table 11.5: Sample Data for Knapsack Problem
Item 1 2 3 4
Weight 5 4 6 3
Value Rs.10 Rs.40 Rs.30 Rs.50
Knapsack capacity is given as W=10.
If we apply the recurrence formulas to this set of data, then we will get the
following table 11.6.
Table 11.6: Example Table for Knapsack Problem
1 0 1 2 3 4 5 6 7 8 9 10
0 0 0 0 0 0 0 0 0 0 0 0
1 0 0 0 0 0 10 10 10 10 10 10
2 0 0 0 0 40 40 40 40 40 50 50
3 0 0 0 0 40 40 40 40 40 50 70
4 0 0 0 50 50 50 50 90 90 90 90
w
1
= 5, v
1
=10
w
2
= 4, v
2
= 40
w
3
= 6, v
3
= 30
w
4
= 3, v
4
= 50
We can compute the maximum value of V[4,10] as Rs.90. We can use the
table to track the optimal subset. Since V[4,10]
=
V[3,10], item 4 is included
in an optimal solution along with an optimal subset for filling 10-3=7
remaining units of the Knapsack capacity. This is represented as V[3,7].
Since V[3,7]=V[2,7], item, 3 is not a part of an optimal subset. Since
V[2,7]
=
V[1,7], item 2 is a part of an optimal solution. V[1,7-1] is left behind
as the remaining composition. Similarly, V[1,6]
=
V[0, 6] , therefore item 1 is
included in the solution.
We can find the time efficiency and space efficiency of the algorithm as
O
(nW).The time required to find the composition of an optimal solution is in
O
(n + W).
Activity 1
Item 1 2 3 4
Weight 3 5 2 4
Value (in Rs.) 10 15 25 45
Knapsack capacity is given as W=10. Analyze the Knapsack problem
using dynamic programming with the help of the values given above.
7. The __________ expresses the problem using its sub-instances.
8. ________________ is an NP-hard optimization problem.
9. The Knapsack problem minimizes the total _________ and maximizes
the total value.
11.5 Memory Functions
In the previous section we solved the Knapsack problem using dynamic
programming. In this section let us solve the Knapsack problem using
memory functions.
As you know, dynamic programming deals with problems whose solutions
satisfy a recurrence relation with overlapping sub problems. It uses a direct
top down approach to find a solution to such recurrence. This is a very
inefficient method. In the classic bottom up method, it fills a table with
solutions to all smaller sub problems. Sometimes, we do not need solutions
to all sub problems. This is where we use memory functions. The goal of
using memory functions is to solve only the sub problems which are
necessary. Memory functions use a dynamic programming technique called
memoization in order to reduce the inefficiency of recursion that might
occur.
We use memoization for finding solution to sub problems, so as to reduce
recalculation. We use it in algorithms which have lots of recursive calls to
the sub problems. Memory functions method solves problems using top
down approach, but maintains a table which is used for the bottom up
dynamic programming algorithms. We can initialize the table values to a
null symbol. When we have to compute a new value:
- The method checks the corresponding entry in the table
- If this entry is not null, it is retrieved
- If this entry is null, then the value is computed using recursive calls and
the results are entered in the table.
The algorithm for solving Knapsack problem using memory functions is
given below.
Algorithm for Solving Knapsack Problem Using Memory Functions
//Input: A nonnegative integer i indicating the number of the first items
used //and a non negative integer j indicating the Knapsacks capacity
//Output: The value of an optimal feasible subset of the first i items
//Note: uses as global variables input arrays weights[1..n],
values[1..n],and //table V[0..n,0..W] whose entries are initialized with Is
except for row 0 //and column 0 which are initialized as 0s.
If V[i,j]<0
If j< Weights[i]
value
MFKnapsack(i-1,j)
else
value
max[MFKnapscak(i-1,j),
Values[i] + MFKnapsack[i-1,,j-Weigths[i])]
V[i,j]
value
return V[i,j]
Let us now trace the above algorithm that uses memory functions.
Algorithm tracing for Knapsack Problem Using Memory Functions
i=2,j=2,weights[3]={1,2,3},values[3]={3,5,4}, V[5,5]= -1
n=5, W=5,V[0,0]=0
If V[2,2]<0//value of V[2,2]= -1,which is less than 0
If 2< Weights[2]//If 2<2 this condition is false , jump to else
value = MFKnapsack(2-1,2)//this is a recursive loop
else
value = max[MFKnapscak(2-1,2), Values[2]
+ MFKnapsack[2-1,,2-Weigths[2])]
// value = max[MFKnapsack(1,2), 5+ MFKnapsack[1,0]
V[22]
value
return V[2,2]
11.5.1 Solving Knapsack problem using memory functions
Now, let us solve the same instance given in the previous section by using
memory functions.
The table 11.7 gives the result. We can see that, here only 13 out of 40 non
trivial values are computed.
Table 11.7: Example Table for Knapsack Problem by Memory Functions
I 0 1 2 3 4 5 6 7 8 9 10
0 0 0 0 0 0 0 0 0 0 0 0
1 0 0 0 0 0 10 10 10 10 10 10
2 0 0 0 0 - - - 40 40 - 50
3 0 0 0 0 - - - - 40 - 70
4 0 0 0 - - - - - - - 90
w
1
= 5, v
1
=10
w
2
= 4, v
2
= 40
w
3
= 6, v
3
= 30
w
4
= 3, v
4
= 50
We cannot expect more than a constant factor gain in using the memory
function methods for the Knapsack problem. The time efficiency class of this
method is the same as the bottom up algorithm. The space efficiency is less
than that of bottom up algorithm.
Activity 2
Item 1 2 3 4
Weight 2 6 4 8
Value (in Rs.) 12 16 30 40
Knapsack capacity is given as W=12. Analyze the Knapsack problem
using memory functions with the help of the values given above.
10. The goal of using ________________ is to solve only the sub
problems which are necessary.
11. Memory functions use a dynamic programming technique called _____
in order to reduce the inefficiency of recursion that might occur.
12. Memory functions method solves the problem using ___________
approach.
11.6 Summary
In this unit we recollected the dynamic programming principle. Next we
defined the Principle of Optimality. Principle of Optimality is defined as a
basic dynamic programming principle which helps us to view problems as a
sequence of sub problems. Next we defined the binary search tree and
explained the various operations performed on the tree. We studied the
applicability of the Principle of Optimality on the binary search trees.
In the next section we studied the Knapsack problem. The problem is
defined for a set of items where each item has a weight and a value, and it
determines the number of items that minimizes the total weight and
maximizes the total value. We solved the Knapsack problem using the
dynamic programming technique. Next we discussed memory functions. It
uses a dynamic programming technique called memoization in order to
reduce the inefficiency of recursion that might occur. We also solved the
Knapsack problem using the memory functions.
11.7 Glossary
Terms Description
Recurrence relation Recurrence relation is an equation that recursively defines
a list where each term of the list is defined as a function of
the preceding terms.
NP hard problem NP hard problems are problems that are solved in non
deterministic polynomial time.

1. What is the basic Principle of Optimality?
2. What are the properties followed by a binary search tree?
3. Explain the steps for inserting an element in a binary search tree and
give its pseudocode.
4. Explain the dynamic programming algorithm for solving a binary search
tree.
5. Explain the algorithm to solve the Knapsack problem using the dynamic
programming method.
11.9 Answers
1. Optimality
2. Richard Ernest Bellman

3. Profits
4. Binary search trees
5. O(log n)
6. Height
7. Recurrence relation
8. Bounded Knapsack problem
9. Weight
10. Memory functions
11. Memoization
12. Top down
Terminal Questions
1. Refer section 11.2 Principle of optimization
2. Refer section 11.3 Optimal binary search trees
3. Refer section 11. 3 Optimal binary search trees
4. Refer section 11.4.1 Solving binary search trees using dynamic
programming
5. Refer section 11.5.1 Solving Knapsack problem using memory
functions
References
- Anany Levitin (2009). Introduction to Design and Analysis of Algorithms.
- Kellerer Hans, Pferschy Ulrich, Pisinger David(2004). Knapsack
problems. Springer, New York
E-References
- www.cecs.csulb.edu/~ebert/teaching/spring2002/cecs328/.../bst.pdf
- www.cs.ubc.ca/~nando/550-2006/lectures/l3.pdf
- http://lcm.csa.iisc.ernet.in/dsa/node91.html
- http://www.ehow.com/way_5172387_binary-tree-traversal-methods.html
- http://www.itl.nist.gov/div897/sqg/dads/HTML/knapsackProblem.html

Unit 12 Greedy Technique
Structure:
12.1 Introduction
Objectives
12.2 Introduction to Greedy Technique
Types of greedy algorithm
Greedy choice property
Applications of greedy technique
12.3 Prims Algorithm
Description
Correctness
Time complexity
12.4 Kruskals Algorithm
Description
Correctness
Time complexity
12.5 Dijkstras Algorithm
Description
Correctness
Time complexity
12.6 Huffman Trees
Huffman code
Constructing Huffman tree
Constructing Huffman code
12.7 Summary
12.8 Glossary
12.10 Answers
12.1 Introduction
In the previous unit we learnt about dynamic programming which is an
optimization technique. In this unit you will learn the concepts of Greedy
technique algorithms that are used for optimization problems such as
Kruskal's algorithm and Prim's algorithm for finding minimum spanning
trees. You will also learn about Dijkstra's algorithm for finding single-source
shortest paths, and the algorithm for finding optimum Huffman trees.
Objectives:
describe the Greedy technique
construct Prims, Kruskals and Dijkstras algorithm
check for correctness of Prims, Kruskals and Dijkstras algorithm
construct the algorithm for finding optimum Huffman trees
12.2 Introduction to Greedy Technique
The Greedy technique constructs a solution to an optimization problem
through a sequence of choices, each expanding a partially constructed
solution until a complete solution to the problem is reached.
The sequences of choices made should be:
Feasible, i.e. satisfying the constraints
Locally optimal with respect to a neighborhood definition
Greedy in terms of some measures and irrevocable
The greedy algorithm, always takes the best immediate solution while
finding an answer. Greedy algorithms find the globally optimal solution for
some optimization problems.
Now, let us see the different types of greedy techniques.
12.2.1 Types of greedy algorithm
Greedy algorithms can be classified as 'short sighted', and as 'non-
recoverable'. They are ideal for problems with optimal substructure. Greedy
algorithms are best suited for simple problems like giving change. The
greedy algorithm can also be used as a selection algorithm to prioritize
options within a search, or branch and bound algorithm. The following are a
few variations to the greedy algorithm:
Pure greedy algorithms
Orthogonal greedy algorithms
Relaxed greedy algorithms
Pure greedy algorithm
Pure greedy algorithm makes local choices in all iterations in order to find an
optimal solution. The Pure greedy algorithm chooses functions to use in
approximating.
Relaxed greedy algorithm
The Relaxed greedy algorithm provides the approximation order, and gives
a constructive proof of the estimate. There are several variants of the
relaxed greedy algorithm and their application for different dictionaries.
Orthogonal greedy algorithm
In Orthogonal greedy algorithm the best approximation from the functions
generated at each iteration is taken.
12.2.2 Greedy-choice property
In greedy algorithms a globally optimal solution is arrived by making a
locally optimal (greedy) choice. That is to say when considering which
choice to make, the choice that looks best in the current problem, without
considering results from sub problems is selected. The choices made by a
greedy algorithm can depend on choices of the past, but it cannot depend
on any future choices or on the solutions to sub problems. This is where
greedy algorithm differs from dynamic programming.
In dynamic programming, choices are made at each step, but the choice
usually depends on the solutions to sub problems. Dynamic-programming
solves problems in bottom-up manner that is solving from smaller sub
problems to larger sub problems.
Therefore a greedy strategy usually progresses in a top-down fashion,
making one greedy choice after another, reducing each given problem
instance to a smaller one.
There are various applications of greedy technique. Let us see those next.
12.2.3 Applications of greedy technique
If a greedy algorithm is proven to yield the global optimum solution for a
given problem class, it becomes the method of choice because it is faster
than other optimization methods hence such algorithms can be applied in
the following areas:
Optimal solutions:
o Change making for normal coin denominations
o Minimum spanning tree (MST)
o Single-source shortest paths
o Simple scheduling problems
o Huffman codes
Approximations/heuristics:
o Traveling salesman problem (TSP)
o Knapsack problem
o Other combinatorial optimization problems
1. The choices made in a greedy algorithm cannot depend on _________
choices.
2. The __________________ is greedy in the sense that at each iteration
it approximates the residual possible by a single function.
3. A greedy strategy usually progresses in a ___________ fashion.
12.3 Prims Algorithm
The previous section gave an introduction to the Greedy technique. Let us
now discuss Prims algorithm which is based on this technique. Prims
algorithm constructs a minimum spanning tree through a sequence of
expanding sub-trees. The minimum spanning tree is obtained by selecting
the adjacent vertices of already selected vertices. The tree starts from an
arbitrary root vertex and grows until the tree spans all the vertices in the
graph.
12.3.1 Description
This strategy is greedy since the tree is added at each step with an edge
that contributes the minimum amount possible to the tree's weight. After
every step, the current tree expands in the greedy manner by attaching it to
the nearest vertex not in the tree. The algorithm stops after all the vertices
have been included. Since the algorithm expands a tree by exactly one
vertex on each of its iterations, the total number of such iterations is n-1,
where n is the number of vertices.
Let us apply Prims algorithm to the graph considered in table 12.1 and
analyze the working of the algorithm
Matrix 1 is the matrix representation of the graph considered in table 12.1

0 7 14 6
7 0 5
14 5 0 8 4
6 8 0 6
4 6 0
E D C B A
E
D
C
B
A

Matrix 1
Table 12.1: Application of Prims Algorithm
Graph Description

Let us consider this weighted graph. The
numbers near the edges indicate their weight.

Choose Vertex C arbitrarily as the starting point.
Vertices A, B, E and D are connected to C
through a single edge. AC being the shortest arc
with length 4 is chosen as the second vertex
along with the edge AC.

The vertex nearest to either C or A is the next
vertex chosen. Arc AB is of length 6, arc CB is of
length 8, arc CE is of length 14 and arc CD is of
length 5. Therefore D is the smallest distance
away, so the vertex D and the arc CD are chosen
next.

Next we have to choose a vertex from the
remaining vertices which are nearest to either A,
C or D. Vertex B, connected through arc AB is of
length 6 is chosen.

E is the remaining vertex and the shortest arc BE
is chosen. The minimum spanning tree selected
now has a weight of 21.

Pseudocode of Prims Algorithm
1. (Initializations).
O= {1} (V (1) root of the T tree).
P= {2,..., n}
2. For every j belonging to P, e(j):=c[e(j1)] , p(j)=1
( all peaks connected to the root. By definition of the cost
function: e(j)=infinite when V(j) does not connect to V(1).).
3. Choose a k for which e(k)<=e(j) for every j belonging to P. In case of
tight choose the smaller one. Exchange the O set with the set
produced by the union of the O set and {k}. Exchange the P set with
the set produced by the difference of the P set and {k} (P<-P-{k}) If
P=0 then stop.
4. For every j belonging to P compare e(j) with c[e(kj)].If e(j) >c[e(kj)]
exchange e(j) <-c(e(kj)).Go back to Step 1.
12.3.2 Correctness
Proof of correctness is proved by induction that each sub-tree generated by
Prims algorithm is a part of some other minimum spanning tree. The basis
of the induction is that T (0) consisting of a single vertex must be a part of
any minimum spanning tree. Let us assume that T (i-1) is part of some
minimum spanning tree T, where i = 0 n-1.
This can be proved by contradicting and assuming that any minimum
spanning tree of the graph can contain T(i) let e=(u, v) be the minimum
weighted edge from a vertex in T(i-1) to a vertex not in T(i-1) used by the
algorithm to expand T(i-1) to T (i), by our assumption, e cannot belong to T,
but if we add e to T, a cycle must be formed. This cycle must contain
another edge (u1, v1) which is connecting a vertex v1 belonging to T(i-1) to
a vertex u1 not in T(i-1).
By deleting (u1,v1) we obtain another spanning tree of the entire graph
whose weight is less than or equal to the weight of T. Because the weight of
(u,v) is less than or equal to the weight of (u1,v1) this is a minimum
spanning tree which contradicts the assumption that no minimum spanning
tree contains T(i). This proves the correctness of Prims algorithm.

12.3.3 Time complexity
An implementation done by an adjacency matrix graph representation and
searching an array of weights to find the minimum weight edge requires
O(V) running time. Using a binary heap data structure and an adjacency list
representation, Prim's algorithm can be proved to run in time O(E log V)
where E is number of edges and V is number of vertices. Using a
sophisticated Fibonacci heap, this can be got to O(E + V log V), which is
asymptotically faster when the graph is dense enough and function of E is
(V).
4. The _______________________ is obtained by selecting the adjacent
vertices of already selected vertices.
5. Each _________________ generated by prims algorithm is a part of
some other minimum spanning tree.
6. The greedy strategy in prims algorithm is greedy since the tree is
added with an edge that contributes the __________ amount possible
to the tree's weight.
Activity 1
Apply Prims algorithm to the graph and find the minimum spanning tree
for the graph.

12.4 Kruskals Algorithm
In the previous section we discussed Prims algorithm. In this section we will
explain Kruskals algorithm which is also a greedy algorithm for minimum
spanning tree problem that yields an optimal solution.
12.4.1 Description
Kruskals algorithm finds a particular subset of the edges that are able to
form a tree that contain all the nodes (vertices) without forming a cycle
within the graph, but the total weight of the tree is minimized. If the graph is
not connected, then the algorithm yields a minimum spanning forest.
The theory of Kruskals algorithm: Create a forest in which each vertex is
an individual tree. Then create a set S that contains all of the graphs edges.
Search for the edge that has the minimum weight. If the edge connects two
different trees, then include it to the forest and combine the two trees into
one; otherwise, discard the edge.
Let us apply Kruskals algorithm to the graph considered in table 12.2 and
analyze the working of the algorithm
Matrix 2 is the matrix representation of the graph considered in table 12.2

0 10 8
10 0 7 5
7 0 14 4 6
5 14 0 8 4
4 0 7
6 8 7 0 6
4 6 0
G F E D C B A
G
F
E
D
C
B
A

Matrix 2
Table 12.2: Application of Kruskals Algorithm
Graph Description

Let us consider this graph for Kruskals
algorithm. The numbers near the arcs
indicate their weight.

AD and CE are the shortest arcs, with
length 4, let us choose AD arbitrarily.
Graph Description

CE is now the arc with shortest length
that does not form a cycle, with length
4, so it is chosen as the second arc.

The next arc, DF with length 5, is
chosen using the same method.

The next shortest arcs are AB and BE,
both with length 6. AB is chosen
arbitrarily, The arc BD is not included
because there already exists a path
between B and D; if included it would
form a cycle ABD.

The process chooses the next-smallest
arc, BE with length 6. Arcs BC, DE and
FE are not chosen as they would form
the loop BCE, DEBA, and FEBAD
respectively.

Finally, the process finishes with the arc
EG of length 8. Now all the edges are
included in the tree.

Pseudocode of Kruskals Algorithm
1 function Kruskal (G = <N, A>: graph; length: A R
+
): set of edges
2 Define an elementary cluster C(v) {v}.
3 Initialize a priority queue Q to contain all edges in G, using the weights
as keys.
4 Define a forest T //T will ultimately contain the edges of the
MST
5 // n is total number of vertices
6 while T has fewer than n-1 edges do
7 // edge u, v is the minimum weighted route from u to v
8 (u, v) Q.removeMin( )
9 // prevent cycles in T. add u, v only if T does not already contain a
path between u and v.
10 // the vertices has been added to the tree.
11 Let C(v) be the cluster containing v, and let C(u) be the cluster
containing u.
13 if C(v) C(u) then
14 Add edge (v, u) to T.
15 Merge C(v) and C(u) into one cluster, that is, union C(v) and C(u).
16 return tree T
12.4.2 Correctness
Kruskals algorithm can be proved by two parts. Firstly we prove that the
algorithm produces a spanning tree without forming a cycle and secondly
we prove that the constructed spanning tree is of minimal weight.
Spanning tree
Let P be a connected, weighted graph and Y the sub-graph of P produced
by the algorithm. Y will not have a cycle, since the last edge added to that
cycle would have been within one sub-tree and not between two different
trees. Y cannot be discarded, since the first encountered edge that joins two
components of Y would have been added by the algorithm. Thus, Y is a
spanning tree of P.
A real life application of spanning tree is in the phone network design.
Consider that you have to connect all your office branches with a telephone
network. The costs of these connections are based on the distance between
the cities where the branches are situated. We want a set of lines which
connects all the branch offices with a minimum cost. Here, you can use a
spanning tree to build a network.

Figure 12.1: Spanning Tree
The figure 12.1 shows a spanning tree with the office branches given as A,
B, C, D, E, F, G, H, I. Here the minimum distance between each branch is
chosen for the connection.
The time complexity of Kruskals algorithm depends completely on the
implementation of Union and Find-Set. The Union-Find data structure
implementation shows its efficiency only through amortized analysis.
Union-Find data structure
The Union-Find data structure is helpful for managing equivalence classes,
and is vital for Kruskals algorithm. This data structure helps us to store and
manipulate equivalence classes. Equivalence class has to satisfy reflexive,
symmetric and transitive properties. Each class has a representative
element. We can unite two equivalence classes together and hence create a
new equivalence class. The data structure therefore supports the operations
- Make set, Union and Find.
Makeset(x) initializes a set with element x. Union(x, y) will union two sets
together. Find(x) returns the set containing x. A simple implementation of
this data structure is a tree defined by a parent array. A set is stored as a
tree where the root represents the set, and all the elements in the set are
descendents of the root. Find(x) works by keeping track of the parent
pointers back until we reach the root parent. Makeset and Union are O(1)
operations but Find is an O(n) operation, because the tree can get long and
thin, depending on the order of the parameters in the calls to the Union.
Particularly it is bad to point the taller tree to the root of the shorter tree.
We can fix this by changing Union. Union(x, y) will not just set the parent of
x to y. Instead it will calculate which tree, x or y, has more number of nodes.
Then it points the parent of the tree with the fewer nodes to the root of the
tree with more nodes. This idea guarantees that the height of a tree is at
most log n. This means that the Find operation has become O(log n).
We can analyze Kruskals algorithm by implementing the Union-Find data
structure in this way. The sorting of the edges can be done in O(e log e)
which is O(e log n) for any graph. For each edge (u,v) we check whether u
and v are in the same tree. This is done with two calls to Find which is O(log
n), and we unite the two if necessary which is O(1). Therefore the loop is
O(e log n). Hence the total time complexity is O(e log n).
An amortized analysis path compression is another way to make the trees
even shorter and improve performance The p operations of Union and Find
using weighted union and path compression takes time O(p log*n).
Therefore each operation on the average is taking O(log*n) i.e. Kruskals
algorithm runs in time O(e log*n).
7. In Kruskals algorithm if the graph is not connected, then the algorithm
yields a __________________.
8. The Union-Find data structure is helpful for managing _____________
which is vital for Kruskals algorithm.
9. Correctness of Kruskals algorithm can be proved by saying that the
constructed spanning tree is of __________________.
12.5 Dijkstra's Algorithm
Let us now analyze Dijkstras algorithm which is also a greedy algorithm and
is similar to Prim's algorithm. It is applicable to both directed and undirected
graphs.
12.5.1 Description
Dijkstras algorithm finds solution for the single-source shortest path
problem for a tree or graph with nonnegative edge path costs. For a given
vertex in the graph, the algorithm finds the path with lowest cost between
the originating vertex and every other vertex. It is also used for finding costs
of shortest paths from a vertex to a destination vertex by ending the
algorithm once the shortest path to the destination vertex has been
determined.
The mechanism of Dijkstras algorithm is similar to prims algorithm. But they
solve different problems and the priorities are computed in different ways.
Dijkstras algorithm compares path lengths and adds the edge weights,
while prims algorithm compares the edge weights. The operation of the
Dijkstras algorithm is explained in Table 12.3.
Table 12.3: Illustrating the Operation of Dijkstras Algorithm
Graph Description

Let us consider this graph for Dijkstras
algorithm. The numbers near the arcs indicate
their weight

Let us choose A as the source vertex,
consider the two vertices connected to A,
comparing the path lengths. Vertex B is
chosen with path length AB.

To include vertices C and D, consider the path
length through the vertex B which is already
chosen. The shortest path length is included.
Therefore vertex D with path length 7 is
included to the tree

The remaining vertices are C and E, with path
lengths 9 and 12 respectively. Hence vertex C
is included to the tree.

The only remaining vertex is E, whose path
length through C is 16 and through D is 12.
Hence vertex E is included through D which is
of shorter length.

Let us see the pseudocode for Dijkstras algorithm.
Pseudocode for Dijkstras Algorithm
1 function Dijkstra(Graph, source):
2 for each vertex v in Graph: // Initializations
3 dist[v] := infinity
// Unknown distance function from source to v
4 previous[v] := undefined
// Previous node in optimal path from source
5 dist[source] := 0
// Distance from source to source
6 Q: = the set of all nodes in Graph
// All nodes in the graph are unoptimized
7 while Q is not empty:
// The main loop
8 u := vertex in Q with smallest dist[]
9 if dist[u] = infinity:
10 break
// all remaining vertices are inaccessible from source
11 remove u from Q
12 for each neighbor v of u:
// where v has not yet been removed from Q.
13 alt := dist[u] + dist_between(u, v)
14 if alt < dist[v]:
// Relax (u,v,a)
15 dist[v] := alt
16 previous[v] := u
17 return dist[]
12.5.2 Correctness
Let us show that the algorithm is correct with all edge weights being
nonnegative. Let (v,n) be the cost of the shortest path from the source
vertex v to node n.
Claim: If the partially constructed tree T, constructed after i edges has been
added, is a sub-tree of some shortest path tree, then the tree T
0
,
constructed after i+1 edges are added is also a sub-tree.
Proof of claim: Let T
short
be a shortest path tree that contains T as a sub-
tree. Let (m, n) be the (i+1)
st
edge added by the algorithm. If T
short
contains
(m,n), the proof is complete. Otherwise, let P be the path of T
short
from the
source v to n.
Let T
0
short
be the tree obtained from T
short
by removing the edge n of tree
T
short
. The sub-tree of T
short
rooted at n hangs beneath m in T
0
short
.
We can claim that the path from v to n in T
0
short
is not costly than the path
from v to n in T
short
and so T
0
short
is indeed a shortest path tree of the graph.
To see this, let c be the first node on path P that is not in tree T. Let b be the
predecessor of c on path P. Because we have assumed that all the weights
are non-negative, and a shortest path from v to n goes through c, then
(v, c) (v, n): Eq. 12.1
When x is added to the tree, the algorithm ensures that
val(c) val(b) + wt(b; c) = (v; c): Eq. 12.2
Also, at the point when v is added to the tree, then
val(n) val(c) Eq. 12.3
The algorithm always adds a node with minimum cost and n is selected for
addition before c. therefore by putting equations Eq 12.1, Eq 12.2, and
Eq 12.3, we have
val(n) val(c) (v;c) (v;n):
Now because parent (n) = m, we can say there is a shortest path from the
source to n that passes through m and so T
0
short
is certainly a shortest path
tree of the graph. Therefore this completes the proof.
The time complexity for the Dijkstra algorithm on a graph with n nodes and
m edges is O(n
2
) because it permits directed cycles. Finding the shortest
path from a source node to all other nodes in the graph has time complexity
of O(n
2
) for node selection and O(m) for distance updates. While O(n
2
) is the
complexity for dense graphs, the complexity can be improved notably for
sparse graphs.

10. Dijkstras algorithm solves the single-source ____________ problem
for a tree.
11. The algorithm finds the path with lowest cost between the ________
vertex and every other vertex.
12. The time complexity of Dijkstras algorithm can be improved for
_____________ graphs.
12.6 Huffman Trees
In the previous sections we learnt about algorithms which apply greedy
technique. But in this section we are going to learn about a tree which gives
optimal solution using greedy technique. A Huffman tree is a binary tree
which minimizes the weighted path length from the root to the leaves which
are predefined.
12.6.1 Huffman code
Huffman codes are digital data compression codes which are the outcome
of the brilliant piece of work by Prof. David A. Huffman (1925-1999).
Huffman codes give good compression ratios. Even today, after 50 years,
Huffman codes have not only survived but are unbeatable in many cases.
Huffman compression is a compression technique where there is no loss of
information when the data is compressed i.e. after we decompress the data,
the original information can be got. Hence it is named as lossless
compression. Lossless compression is desired in compressing text
documents, bank records etc.
Data encoding schemes fall into two categories,
1) Fixed length encoding In fixed length encoding all symbols are
encoded using the same number of bits. An example of fixed length
encoding is ASCII code which uses 7 bits to encode a total of 128
different symbols. The difficulty with fixed length codes is that the
probability of occurrence of the symbols to be encoded is not
considered. A symbol that occurs 1000 times is encoded with the same
number of bits as a symbol which comes only 10 times. This
disadvantage makes fixed length encoding inefficient for data
compression.
2) Variable length encoding Variable length encoding removes this
difficulty by assigning less number of bits to symbols which occur more
often and more number of bits to symbols whose frequency of
occurrence is less. .The Huffman Encoding scheme falls in the category
of variable length encoding i.e. code for the symbol depends on the
frequency of occurrence of that symbol.
Huffman coding is again classified into two different groups
3) Static Huffman coding Static Huffman coding is done with the help of
statistical symbol frequency tables in which symbol frequencies are
known before the actual coding takes place
4) Adaptive Huffman coding In adaptive Huffman compression the
symbol frequencies need not be known in advance. Symbols are
encoded as they are encountered.
12.6.2 Constructing Huffman tree
The following sequence of steps is to be followed to construct a Huffman
tree:
1. Input all symbols along with their respective frequencies
2. Create leaf nodes representing the symbols scanned
3. Let S be a set containing all the nodes created in step 2
4. To create the Huffman Tree:
i. Sort the nodes (symbols) in S with respect to their frequencies.
ii. Create a new node to combine the two nodes with least
frequencies.
iii. Frequency of this new combined node will be equal to sum of
frequencies of nodes which were combined. This newly created
combined node will be the parent of two nodes which were
combined.
iv. Replace, in S, the two nodes which were combined with the new
combined node.
After the 4th step you will be left with only one node, which is the root of the
Huffman tree, having frequency equal to sum of all frequencies of all
symbols. Thus a tree is generated with leaf nodes containing the basic
symbols whose code is to be found.

Table 12.4: Symbol Frequency Table
Symbol Frequency of occurrence
A 24
B 12
C 10
D 8
E 8
With the help of an example we will learn how to construct a Huffman tree
Using the symbols and frequencies from the table 12.4, we create the leaf
nodes and then sort them.
Symbols D and E have least frequency, 8; these 2 nodes are combined to
make a node DE having frequency 8+8=16. This new node DE is the
parent node of the nodes D and E, and DE replaces D and E.
Again we sort the nodes; now DE and C having least frequencies i.e. 16
and 10 each. This time we combine DE and C to create a new node DEC
having frequency 26.
Nodes DE and C are replaced by their parent DEC.
Combine B and DEC to create BDCE having frequency 12 of B and 26
of DEC. Hence BDEC becomes the parent of B and DEC with frequency
38.
At last only two nodes are left namely BDEC and A. We again sort them
and combine both of them to form ABDEC which has frequency count of
62.
After making ABDEC parent of A and BDEC and replacing them with
ABDEC; we have created the Huffman tree for the symbols in Table 12.4.
Node ABDEC is the root of the tree.
12.6.3 Constructing Huffman code
To assign codes start from root of the tree and assign 1 to every left branch
and 0 to every right branch. We will now explain the process of tree
creation. To find the code for a particular symbol, start from that symbol and
traverse in direction towards the root. As soon as you encounter a branch
assign the code to that branch (1/0); Suppose we need to find Huffman code
for C; we start from C and move toward root i.e. C -> EC -> DEC -> BDEC
->ABDEC; we get code 1, 1, 0, 1, 0 respectively (i.e. 1 Cor C->EC, 1 Cor EC
-> DEC, 0 C or DEC -> BDEC and so on). Note that this code that we have
is from LSB to MSB. So the final code for A will be 0. Table 12.2 shows
Huffman code for all the symbols.
Table 12.5: Symbol Frequency Table with Huffman Code
Symbol Frequency of occurrence Huffman code
A 24 0
B 12 100
C 10 101
D 8 110
E 8 111
From table 12.5, you can notice that no codeword is also a prefix of another
codeword. E.g. codeword for B is 100; now there is no other codeword
which begins with 100. Codes with this property are called as Prefix codes.
In prefix codes no codeword in the set is a prefix to another codeword.
Huffman codes are Prefix codes. This property makes Huffman codes easy
to decode. After studying much English prose, the frequency of characters
has been analyzed and a Huffman code has been assigned to every
character. Now suppose we want to code ADECBA; we can directly find
Huffman code for the each of the symbols from Table 12.2 i.e. for A, 0
forD,110 and so on. The code will look like 01101111011000.
There are 6 characters and it takes only 14 bits to transmit them. If we use
normal ASCII code then it will take 7*6= 42 bits to transmit the same string.
Thus the use of Huffman codes has saved 28 bits which is around 66%. In a
similar fashion Huffman codes can save from around 20% to 90%
depending on the pattern of data being compressed.
Activity 2
Obtain the Huffman code for the following data and encode the text.

Character P Q R S
Frequency 55 10 10 25

13. Huffman codes are digital _________________ codes.
14. The Huffman Encoding scheme falls in the category of ____________.
15. Static Huffman coding is done with the help of ___________ tables.
12.7 Summary
Optimization problems can be solved using greedy technique which involves
a sequence of steps that include choices which should be feasible, optimal,
and irrevocable. We discussed in this unit different algorithms that are
based on the greedy technique.
Prims algorithm is a greedy algorithm used to construct a minimum
spanning tree of a weighted graph.
Kruskals algorithm constructs a minimum spanning tree by selecting edges
in the increasing order and including them in the tree such that it does not
form a cycle.
Dijkstras algorithm solves single-source shortest problems. It is similar to
Prims algorithm but considers path lengths instead of edge lengths.
Huffman trees minimize the path length from the path to the leaves.
To conclude, if a problem is solved efficiently by a greedy algorithm then it is
widely accepted as the global optimal solution.
12.8 Glossary
Term Description
Equivalence class A set of things that are considered equivalent
Data compression Information is encoded using fewer bits
1. Describe greedy choice property.
2. Describe the working of Prims algorithm with an example.
3. Explain the time complexity in Kruskals algorithm and the method of
resolving it.
4. Explain the working of Dijkstras algorithm with an example.
5. How are Huffman codes constructed?
12.10 Answers
1. Future
2. Pure greedy algorithm
3. Top-down
4. Minimum spanning tree
5. Sub-tree
6. Minimum
7. Minimum spanning forest
8. Equivalence classes
9. Minimal weight
10. Shortest path
11. Originating
12. Sparse
13. Data compression
14. Variable length encoding
15. Statistical symbol frequency
Terminal Questions
1. Refer to 12.2.2 Greedy choice property
2. Refer to 12.3.1 Description
3. Refer to 12.4.3 Time complexity
4. Refer to 12.5 Dijkstras algorithm
5. Refer to 12.6.3 Constructing Huffman codes
Reference
Anany Levitin (2009). Introduction to Design and Analysis of Algorithms.
Cormen, H. Thomas (2001). Introduction to Algorithms MIT. Press,
McGraw-Hill Book Company
E-Reference
www.cs.cmu.edu/afs/cs/academic/class/15853-f00/.../compress1.ppt
http://www.cs.ubc.ca/~nando/320-2003/lectures/lecture9-2.pdf
http://www.devarticles.com/c/a/Development-Cycles/Greedy-Strategy-
as-an-Algorithm-Technique/2
http://www.mec.ac.in/resources/notes/notes/ds/kruskul.htm
http://www.programmersheaven.com/2/Art_Huffman_p1
Unit 13 Limitations of Algorithm Power
Structure:
13.1 Introduction
Objectives
13.2 Lower Bound Arguments
Trivial lower bounds
Information theoretic arguments
Adversary arguments
Problem reduction
13.3 Decision Trees
Decision trees for sorting algorithms
Decision trees for searching a sorted array
13.4 P, NP and NP Complete Problems
Non deterministic algorithm
NP hard and NP complete classes
Cooks theorem
13.5 Summary
13.6 Glossary
13.8 Answers
13.1 Introduction
So far in the previous units we have studied many algorithms, and learnt
how they play a significant role in solving a range of problems. But, it is not
possible to solve all problems using algorithms. The power of algorithms is
limited to some extent.
The reasons for these limitations are:
Some problems which can be solved using algorithms are not solved
within polynomial time.
Even if we can solve some problems within the polynomial time, the
efficiency of the algorithm is in lower bound.
This unit covers the limitations of algorithm power with respect to lower
bound arguments of algorithms. It explains decision trees with examples. It
also analyzes P, NP and NPcomplete problems.

Objectives:
explain the lowerbound arguments of algorithms
describe and implement decision trees
define P, NP and NPcomplete problems
13.2 Lower Bound Arguments
In this section we will discuss about the lower bound arguments in
algorithms. Lowerbound means calculating the minimum amount of work
required to solve the problem. While obtaining the lowerbound of the
algorithm we look for the limits of efficiency of any algorithm that can solve
the problem.
The following two methods help to make an algorithm more efficient:
1) First we verify the asymptotic efficiency class of the algorithm.
2) Then we check the class of the given problem to see where it fits in the
hierarchy of the efficiency classes (i.e., whether the problem lies in
linear, quadric, logarithmic or exponential category of efficiency class).
The efficiency of different algorithms is given in the table 13.1.
Table 13.1: Efficiency of Different Algorithms
Algorithms Efficiency
Insertion sort n
2

Quick sort n log n
Heap sort n log n
Linear search n/2
Binary search log
2
n
When we are finding the efficiency of an algorithm, it is better to compare it
with those algorithms that are used to solve similar kind of problems. For
example if we want to determine the efficiency of insertion sort then we have
to compare it with other sorting methods. We cannot determine the
efficiency of insertion sort if we compare it with the efficiency of Tower of
Hanoi problem because these are two different types of problems.
When we are determining the efficiency of an algorithm with respect to other
algorithms that are used to solve the same problem, it is better to know the
best possible efficiency of an algorithm which can solve that problem.
Knowing this helps us to improve the algorithm. If there is a gap between
the best lower bound and the fastest algorithm then there is a possibility of
improving the algorithm i.e. either there is an algorithm with the fastest
lower-bound or we can prove the better lower-bound for that algorithm can
be established.
Following are the different methods for obtaining the lowerbound of an
algorithm:
Trivial lower bounds
Information theoretic arguments
Adversary arguments
Problem reduction
13.2.1 Trivial lower bounds
This is the simple method used for obtaining lower bound class of an
algorithm. Trivial lower bound is obtained by counting the input data that the
algorithm reads and the output that it produces.
For example, the trivial lower bond for generating all permutation of n
numbers will be (n!) because the output size here is factorial n. This
algorithm is tight because good algorithms for generating permutations
spend a constant time on each of them expect the initial one.
Similarly if we calculate the trivial lower bound for finding the product of two
n n matrices, it is (n
2
). This is because this algorithm takes two n
elements as the inputs and produces n
2
elements as output. It is still not
known whether this bond is tight.
Limitations of this method
Trivial lower bounds are less useful. For instance let us consider the
problem of traveling salesman. We see that the trivial lower bound for this
algorithm is (n
2
) as the algorithm has n (n-1)/2 distances as the input and
produces n + 1 cities as the output. This trivial lower bound is not useful in
this case because there is no similar problem to compare it with.
There is one more problem in obtaining the lower-bound using this method.
When we are finding the lower-bound of an algorithm using this method it is
necessary to determine which part of the input has to be processed. For
instance, searching an element in the sorted array does not require
processing of all the input elements.
Let us consider another example, the problem of determining the
connectivity of an undirected graph using its adjacency matrix. It is possible
for such algorithms to check for the existence of n (n-1)/2 potential edges,
but the proof of this problem is not trivial.
13.2.2 Information theoretic arguments
While the trivial lower bound method takes into account the input and output
size of the problem, this method defines the lower bound of an algorithm
based on the amount of the information the algorithm produces.
For example let us consider the game of finding a positive integer in
between 1 and n. We get the integer by asking questions for which the
answer would be yes/no. Any algorithm used to solve this problem is
measured as log
2
n, which is the number of bits required to specify an
integer among the n possibilities. The answer to each question produces
one bit of information about the output of the algorithm. Therefore any
algorithm used to solve this problem requires log
2
n steps before it produces
the output.
This method is called as informationtheoretic arguments because it is
connected with information theory. We can find the lower bound of an
algorithm using this method with the help of a mechanism called decision
tree. We can use this method for finding the lower bound of those algorithms
which are based on comparison methods or searching or sorting.
Let us next discuss the adversary arguments method of finding a lower-
bound of an algorithm.
13.2.3 Adversary Arguments
An adversary is an opponent that a comparison algorithm uses. Its ultimate
goal is to minimize the number of comparisons that the algorithm makes
while adding items into the list of input to the problem.
Let us consider an example of comparing some array elements. According
to adversary arguments, if a[1] > a[2], then a[2] > a[1] will never be possible.
We use the codes as given below:
N Not used
T True once but never false
F False once but never true
TF True and false at least once
The table 13.2 gives all the possible outcomes.
Table 13.2: Possible Outcomes
Status Outcome New status Value
NN x>y T,F 2
T,N x>y T,F 1
TF,N x>y TF,F 1
F,N X<y F,T 1
T,T x>y T,TF 1
F,F x>y TF,F 1
T,F;TF,F;T,TF x>y N/C 0
F,T;F,TF,TF,T x<y N/C 0
TF,TF Consistent N/C 0
This problem requires 2n-2 bits of information to solve. All keys except one
must be false and all keys except one must true. Comparing N, N pairs
gives n/2 comparisons and n value bits. We also need n-2 additional bits.
Each element must be compared once. For lower bound, we require a total
of 3n/2-2 comparisons. To find the upper bound we have to group the true
condition and the false condition separately and find their maximum. This
will also be computed as total 3n/2-2 comparisons.
The above example demonstrates the adversary method of obtaining lower
bound. We measure the amount of comparisons required to minimize the
total comparisons to find the lower-bound.
Let us next analyze the method of problem reduction which is used to find
the lower-bound of an algorithm.
13.2.4 Problem reduction
In this method an unsolvable problem A is reduced to a solvable problem B
with a known algorithm. We can use the same reduction idea to find the
lower bound of an algorithm. If we have a problem A which is at least as
hard a problem B whose lower bound is known, we have to reduce B to A so
that any algorithm solving A would also solve B. then the lower bound for B
will also be the lower bound for A.
Let us consider the example of Euclidean minimum spanning tree problem:
Given n points in the Cartesian plane, construct a tree of minimum total
length whose vertices are the given points. Just adding 0 as a point to the y
coordinate that is (x
1,
o), (x
2
, 0), , ( x
n
, 0) in the Cartesian plane we can
transform the set of x
1
, x
2
,.. x
n
of n real numbers into a set of n points.
Let T be the minimum spanning tree for this set of points. While T must
contain the shortest edge, checking if T contains a zero length edge will
answer the question about the individuality of the given number. This
reduction means that (n log n) is the lower bound of the Euclidean
minimum spanning tree problem. The reduction method is used frequently to
compare the relative complexity of a problem as the final results of many
problems are not known.
Self Assessment Question
1. _____________________ means calculating the minimum amount of
work required to solve the problem.
2. Trivial lower bound is obtained by the count of the input data that the
algorithm reads and the output it produces.
3. _______________________ method defines the lower bound of an
algorithm based on the amount of the comparisons the algorithm
makes.
13.3 Decision Tree
In the previous section we have studied about lower bound and different
methods of obtaining lower bound. In this section we will study about
decision trees and the methods of implementing it.
Decision tree means to represent a program in the form of tree with
branches. Here each node represents a decision. First the root node is
tested and then the control is passed to one of its subtrees, depending on
the result of the test. This flow is continued till the leaf node with the element
of interest is reached.
A real life example for a decision tree is given in figure 13.2. Let us assume
that we are the income tax is calculated for salaried citizens under the age
on 60.

Figure 13.2: Example for Decision Tree
First we check if the citizen is a male or a female. If the citizen is a male
then check if the income is less than 2,00,000. If yes then the citizen is not
eligible for tax. If not, check if the income is less than 5,00,000. If yes then
the tax is 10% else check if the income is less than 10,00,000. If that is true
then the tax is 20% else it is 30%. If it is a female then first we check if the
less than 2,50,000 then proceed with the same process followed fro the
male citizen.
Some algorithms like sorting and searching need to compare their input
elements. To study such algorithms, we use decision trees.
Let us now discuss the implementation of decision trees for sorting
algorithms.
13.3.1 Decision trees for sorting algorithms
Sorting algorithms are comparison based. They compare the elements in a
list that has to be sorted. Studying the decision tree for comparison based
sorting algorithm gives the lower bound time efficiency for these algorithms.
We can deduce the output of a sorting algorithm as finding the permutation
of the indices of elements in the input list which is in ascending order. For
example let us consider the output of p < q < r achieved by sorting a list of
input elements p, q and r. Therefore, the possible output for sorting the n
elements in the input list is equal to factorial n. The height of the decision
tree for any comparison based sorting algorithm is h log
2
l where h is the
height of the tree and l is the no of leaves. The worst case number of
comparisons made by such algorithm is not less than log
2
n!. The figure 13.1
explains the decision tree for a three element selection sort.

Figure 13.1: Decision Tree for Three Element Selection Sort

C
worst
(n) log
2
n!
Using Stirlings formula for n! (i.e n! ~
n
e
n
n ) ( 2
) we get
log
2
n ! = log
2
2(n/e)
n
= n log
2
n - n log
2
e + log
2
n / 2 + log
2
2 / 2
= n log
2
n Eq: 13.1
Equation 13.1 gives the comparison for any sorting algorithm with n
elements in the input list. If we use merge sort for this comparison it gives
this comparison in its worst case. This means that the lower bound n log
2
n is
tight and cannot be improved. We however have to show that we can
improve the lower bound of log
2
n! for some values of n.
We can use the concept of decision trees for analyzing the average case of
comparison based sorting algorithms. We calculate the average number of
comparisons for an algorithm based on the average depth of its decision
tree. For example let us consider the insertion sort for three elements.
Figure 13.2 depicts the decision tree for a three element insertion sort.

Figure 13.2: Decision Tree for Three Element Insertion Sort
The lower bound of the average number of comparison C
avg
for any
comparison based sorting algorithm is given as
C
avg
(n) log
2
n !
According to equation 13.1 we have seen that the lower bound for this is n
log
2
n. Here the lower bound for both average and worst case are almost
same. However these lower bounds are obtained as a result of maximizing
the number of comparisons that are made for average and worst case. For
sorting algorithms the average case efficiencies are better than their worst
case efficiencies.
13.3.2 Decision trees for searching a sorted array
Let us now see how decision trees can be used for obtaining the lower
bound in searching a sorted array of n elements A [0] < A [1] < < A [n-1].
The basic algorithm for this problem is binary search algorithm. C
worst
gives
the number of comparisons made in the worst case. Equation 13.2 gives the
worst case for searching a sorted array problem.
C
worst
(n)

= log
2
n + 1
= log
2
(n + 1) Eq: 13.2
Now let us use a decision tree to establish whether this is the least possible
number of comparisons.
Here we are considering the three way comparison where the search
element key is compared with some element x to check if key <x, key=x or
key>x. The figure 13.3 shows the decision tree for the case n = 5. Consider
that the elements of the array are 1, 2, 3, 6 and 9. We will start the
comparison with the middle element, 3. The internal nodes of the tree signify
the elements of the array that are being compared with the search element
key. The leaves signify whether the search is successful or unsuccessful.
For an array of n elements such decision trees have 2n +1 leaves where n
for successful search and n + 1 for unsuccessful search. If the least height h
of a decision tree with l leaves is log
3
l then equation 13.3 gives the lower
bound of this problem based on the number of worst case comparisons.
C
worst
(n) log
3
(2n + 1) Eq: 13.3

This lower bound is smaller than the number of worst case comparisons for
binary search at least for higher value of n.

Figure 13.3: Decision Tree for Binary Search in a Five Element Array
Activity 1
Draw a decision tree, to sort three numbers using selection sort
algorithm.
Self Assessment Question
4. Comparison is the basic operation of _____________algorithm.
5. For sorting algorithms the average case efficiencies are better than
their worst case efficiencies.
6. We use a ________________ decision tree to represent an algorithm
for searching a sorted array with three way comparisons.
13.4 P, NP and NPComplete Problems
In the previous section we have discussed about decision trees and their
implementations. In this section let us discuss about P, NP, and
NPcomplete problems.
In the study of computational complexity the first concern for computer
professionals is to solve any problem within polynomial time.
These problems are classified into two different groups:
1) The first group consists of those problems that are solved within
polynomial time. Such problems are called tractable. For example
searching an element, sorting an array and so on.
2) The second group consists of those problems that are solved in non-
deterministic polynomial time. For example, knapsack problem and
traveling salesperson problem can be solved in non deterministic
polynomial time.
Definition of P Stands for polynomial time.
Definition of NP Stands for non-deterministic polynomial time.
The NP class problems are further divided into NPcomplete and NPhard
problems.
The following reasons justify the restriction of P to decision problems:
1) First it is reasonable to eliminate problems that we are not able to solve
in polynomial time because of the exponentially better output. For
example, generating the subsets of a given set of permutations of n
distinct items. But from the output we see that this cannot be solved in
polynomial time.
2) Secondly we can reduce some of problems that are not decision
problems to a sequence of decision problems that are easy to study. For
example let us consider colours of a graph. Here instead of asking for
the minimum number of colours required to color the vertices of a graph
so that no two vertices are coloured with the same colour, we can verify
whether there is any coloring of the graphs vertices with more than m
colors.
Not all decision problems are solved in polynomial time. Some decision
problems cannot be solved using any algorithm, such problems are called
undecidable.
There are many problems that have no polynomial time algorithms. Let us
see some examples of those algorithms that do not have polynomial time
algorithm:
Hamiltonian circuit A Hamiltonian circuit (or Hamiltonian Cycle) is
defined as a circuit in graph G starts and ends at the same vertex and
includes every vertex of G exactly once. If a graph G contains a Hamiltonian
cycle, it is called Hamiltonian Graph.
Traveling salesman If we have a set of cities and the distances between
them, this problem determines the shortest path starting from a given city,
passing through all the other cities and returning to the first city.
Knapsack problem If a set of items are given, each with a weight and a
value, this problem determines the number of items that minimizes the total
weight and maximizes the total value.
Partition problem This determines whether it is possible to partition the
given set of integers into two that have the same sum.
Bin packing Bin packing is a hard problem which has the goal to pack a
given number of objects into the minimum number of fixed-size bins.
Graph coloring This finds the chromatic number of the given graph.
Integer linear programming This finds the maximum or minimum values
of linear functions for a several integer valued variables subject to a finite
set of constraints.
Another common characteristic that we find in decision problems is that
solving such problems can be computationally difficult, whereas checking
whether a planned solution solves the problem is easy. For example let us
consider the Hamiltonian circuit; it is easy to check if the proposed list of
vertices for a graph with n vertices is in Hamiltonian circuit. We just have to
check whether the list contains n +1 vertices.
Let us first discuss Non-deterministic algorithms.
13.4.1 Nondeterministic algorithms
An algorithm which defines every operation exclusively is called
deterministic algorithm.
An algorithm where every operation may not have an exclusive result and
there is a specified set of possibilities for every operation is called non
deterministic algorithms.
Nondeterministic algorithm is a two staged algorithm. The two stages are
as follows:
Nondeterministic stage This is the guessing stage here a random string
is generates which can be thought as a candidate solution to the given
instance.
Deterministic stage This is the verification stage. In this stage it takes both
the candidate solution and the instance as the input and returns yes if the
candidate solution represents the actual solution for the instance.
Let us consider the following nondetermine algorithm.
Algorithm nondetermine
Algorithm nondetermine s)
// H is an array of n elements
// we have to determine the index p of H at which the search element s is
located
{// the following for loop is the guessing stage
for p = 1 to n do
H[p] = choose(p)
// Next is the verification - the deterministic stage
If (H [p] = s) then
{
write (p)
success ()
}
Else{
write (0)
fail ()
}
}
Let us now trace the nondetermine algorithm
Algorithm tracing for algorithm nondetermine
Let us consider n=4, s = 4, H[ ] = {1, 4, 6, 7}
// H is an array of 4 elements
// we have to determine the index p of H at which the search element s is
//located
for p = 1 to 4 do // this for loop is the guessing stage it executes for 2
times //from p = 1 to 4
H[1] = choose(1)
If (H [1] = s) then // H[1] is not equal to 4 so the else condition executes
//this is the verification stage
{
write (1)
success ()
}
Else{
write (0)
fail ()
}
We see that the above algorithm nondetermine has the following three
functions:
1) choose randomly chooses one of the elements form the given input
2) success signifies the successful completion
3) fail signifies the unsuccessful completion
The algorithm has non-deterministic complexity O(1). If A is ordered then
the deterministic search algorithm has the complexity as (n).
We say that a non-deterministic problem solves the decision problem if for
every instance yes of the problem it returns yes on some execution. If the
efficiency of the non deterministic algorithms verification stage is polynomial
than it is said to be a non deterministic polynomial
Let us next discuss NP-hard and NP-complete classes.
13.4.2 NPhard and NPcomplete classes
We know that NP stands for non-deterministic polynomial. These are the
problems that are solved using the non deterministic algorithms. The
NP complete class can be further classified in to two, they are:
1) NPcomplete
2) NPhard
NPcomplete problems
NPcomplete problems are problems that belong to class NP, i.e. they are
the subset of class NP. A problem Q is said to be NPcomplete if,
1) Q belongs to the class NP.
2) All other problems in class NP can be reduced to Q in polynomial time.
This implies that NPcomplete problems are tough to solve within
polynomial time. If we are able to solve NPcomplete problems in
polynomial time, then we can solve all other problems in class NP within
polynomial time.
NPhard problems
NP-hard problems are similar but more difficult than NPcomplete problems.
All problems in class NP can be reduced to NPhard.
Every problem in NP can be solved in polynomial time. If a NPhard
problem can be solved in polynomial time, then all NPcomplete problems
can also be solved in polynomial time. All NPcomplete problems are
NP hard but not all NPhard problems are NPcomplete.
13.4.3 Cooks theorem
Stephen Cook in 1971 stated that
Any NP problem can be converted into SAT (Satisfiability problem) in
polynomial time
Satisfiability problem SAT This is a decision problem whose instance
uses only AND, OR and NOT variables. Given a finite set of clauses, it
determines whether there is a true value assigned for the schematic letters
that appears in the clauses which makes all the clauses true.
To prove this, we need a consistent way of representing NP problems. What
makes a problem NP is the existence of a polynomial-time algorithm more
specifically, a Turing machine for checking candidate certificates. Cook used
a similar method to that of Turings which showed that the Entscheidungs
problem was equivalent to the Halting Problem. He showed how to encode
as propositional calculus clauses both the relevant facts about the problem
instance and the Turing machine which does the certificate checking, in a
way that the resulting set of clauses is satisfiable if and only if the original
problem instance is positive. Thus the problem of determining the latter is
reduced to the problem of determining the former.
Proof of Cooks theorem
Assume, then, that we are given an NP decision problem D. From the
definition of NP, there is a polynomial function P and a Turing machine M
which, when given any instance I for D, together with a candidate certificate
c, will check in time no greater than P(n), where n is the length of I, whether
or not c is a certificate of I.
Let us say that M has q conditions numbered 0; 1; 2,.., q 1, and a
tape alphabet a
1
; a
2
,. . We shall assume that the operation of the Turing
machine is governed by the functions T, U, and D. We also assume that the
first tape is inscribed with the problem instance on the squares 1; 2; 3,, n
and the putative certificate on the squares m,, 2; -1.
Square zero can be understood to contain a designated separator symbol.
We also assume that the machine halts scanning square 0, and that the
symbol in this square at that stage will be a1 if and only if the candidate
certificate is a true. Note that we must have m P (n). This is because with
a problem instance of length n the computation is completed in at most P (n)
steps. During this process, the Turing machine head cannot move more
than P(n) steps to the left of its starting point.
We define some propositions with their intended interpretations as follows:
1) For i = 0, 1, , P(n) and j = 0; 1,.., q 1, the proposition Q
ij

indicates that after i computation steps, M is in state of j.
2) For i = 0; 1,.., P(n), j = - P(n) ,.; P(n), and k = 1; 2,., s, the
proposition Sijk indicates that after i computation steps, square j of the
tape contains the symbol ak.
3) i = 0, 1, , P(n) and j = -P (n),, P (n), the proposition Tij indicates
that after i computation steps, the machine M is scans for square j of the
tape.
Now, we define some clauses to describe the computation executed by M:
1) In each calculation step, M is in at least at one state. For each
i = 0., P(n) we have the clause
Q
i0
v Q
i1
..Q
i (q -1)
;
which gives (P (n) + 1)q = O(P(n)) literals altogether.
2) In each computation step, M is in at most one at state. For each
i = 0., P(n) and for each pair j; k of different states, we have the
clause
(Q
ij
Q
ik
);
which gives a total of q (q - 1) (P(n) + 1) = O(P(n)) literals.
3) In each step, the tape square contains at least one alphabet symbol.
For each i = 0,.., P(n) and -P(n) j P(n) we have the clause
S
ij
1 v S
ij
2 .. S
ijs
;
Which gives (P (n) + 1) (2P (n) + 1) s = O(P (n)
2
) literals.
4) In each step, the tape square contains at most one alphabet symbol.
For each i = 0,. , P(n) and -P(n) j P(n), and each distinct pair
ak; al of symbols we have the clause
(S
ijk
S
ijl
);
which gives a total of (P (n) + 1)(2P(n) + 1)s(s - 1) = O(P(n)
2
) literals
altogether
5) In each step, the tape is scans at least one square. For each i = 0,
, P(n), we have the clause
T
i
(-P(n)) vT
i
(1-P(n)) v v T
i
(P(n)-1) v T
i
P (n);
Which gives (P (n) + 1)(2P(n) + 1) = O(P(n)
2
) literals.
6) In each step, the tape is scans at most one square. For each i = 0
., P(n), and each distinct pair of j and k of tape squares from -P(n)
to P(n), we have the clause
(T
ij
T
ik
);
which gives the total of 2P (n)(2P(n) + 1)(P(n) + 1) = O(P(n)3) literals.
7) Initially, the machine is in state 1 scanning square 1. This is
expressed by the two clauses
Q
-1; T-1;
this gives just two literals.
8) The configuration at every step after the first is determined from the
configuration of the previous step by the functions T, U, and D
defining the machine M. For each i = 0,........, P(n), -P(n) j P(n),
k = 0,, q - 1, and l = 1,., s, we have the clauses
T
ij
Q
ik
S
ijl
! Q
(i+1)T(k;l)

T
ij
Q
ik
S
ijl
! S
(i+1)jU(k;l)

T
ij
Q
ik
S
ijl
! T
(i+1)(j+D(k;l))

S
ijk
! T
ij
S
(i+1)jk

The fourth among these clauses ensures that the contents of any
tape square other than the currently scanned square remains the
same (to see this, note that the given clause is equivalent to the
formula S
ijk
:T
ij
S
(i+1)jk
). These clauses contribute a total of
(12s + 3)(P(n) + 1)(2P(n) + 1)q = O(P(n)2) literals.
9) Initially, the string a
i1,
a
i2
,., a
in
defining the problem instance I
is inscribed on squares 1; 2, , n of the tape. This is expressed by
the n clauses
S
01i1
; S
02i2
, ., S
0nin
;
a total of n literals.
10) By the P(n)
th
step, the machine has arrive at the stop state, and is
then is scanning for square 0, which contains the symbol a1. This is
expressed by the three clauses
Q
P(n)0,
S
P(n)01
, ., T
P(n)0
;
this gives another 3 literals.
On the whole, the number of literals involved in these clauses is O(P(n)
3
).
Note that q and s are constants and depend only on the machine and do
not vary with the problem instance. Also, it does not contribute to the growth
of the number of literals with increasing problem size, which is what the
notation O captures. The procedure for setting up these clauses, provided
the original machine M and the instance I of problem D, can be achieved in
polynomial time.
Now demonstrate that you have succeeded in converting D into SAT. Say, I
is a positive instance of problem D which means there is a certificate c
which halts the scanning symbol a1 on square 0 when M is run with inputs c;
I. This implies there is some sequence of symbols that can be placed initially
on squares -P(n),., -1 of the tape so that all the above clauses are
fulfilled.
In other words, if I is a negative instance of problem D then there is no
certificate for I, which means that when the computation halts, the machine
will not be scanning a1 on square 0. This implies that any symbols can be
placed on squares -P(n), ......., -1 of the tape and the set of above clauses
cannot be fulfilled, and thus forms a negative instance of SAT.
We can conclude the following from the instance I of problem D: In
polynomial time, a set of clauses forms a positive instance of SAT if and
only if I is a positive instance of D. In other words, the problem D is
converted into SAT in polynomial time. Since D was an arbitrary NP
problem, it follows that; any NP problem can be converted to SAT in
polynomial time.
Activity 2
Find some examples of NPhard and NPcomplete problems from the
Internet and analyze how they are solved.

7. Problems that are solved within polynomial time are called
_______________.
8. ____________ problem finds the chromatic number of the given graph.
9. An algorithm in which every operation is exclusively defined is called
______________ algorithm.
13.5 Summary
Limitations of algorithms power includes lower bound arguments, decision
trees and P, NP and NP complete problems
Lower bound arguments include different types of obtaining lower bounds
like trivial lower bound, informationtheoretic arguments, adversary
arguments and problem reduction.
Decision trees are used for sorting and searching algorithms which have to
compare their input elements.
We have also analyzed P, NP and NPcomplete problems. We have also
discussed the proof for Cooks theorem.
13.6 Glossary
Term Description
Node A node is a theoretical basic unit used to build linked data
structures such as trees, linked lists, and computer-based
representations of graphs.
Polynomial time An algorithm is in Polynomial time if its running time is
upper bounded by a polynomial in the size of the input for
the algorithm
Turing machine A Turing machine is a theoretical machine that
manipulates symbols contained on a strip of tape.
Chromatic number This is the minimum number of colors used for the vertices
of a given graph such that no two adjacent vertices are of
the same color.

1. What are the different types of lower bounds?
2. Explain trivial lower bound with example.
3. Explain the sorting problem with the help of a decision tree
4. What are non-deterministic algorithms?
5. Explain Cooks theorem.
13.8 Answers
1. Lower bound
2. True
3. Information theoretic
4. Sorting
5. True
6. Ternary
7. Tractable
8. Graph coloring
9. Deterministic
Terminal Questions
1. Refer section 13.2 Lower bound arguments
2. Refer section 13.2.1 Trivial lower bound arguments
3. Refer section 13.3.1 Decision tree for sorting algorithm
4. Refer section 13.4.1 Non-deterministic algorithms
5. Refer section 13.4.4 Cooks theorem
References
Puntambekar, A. A. (2008). Design and Analysis of Algorithms. Technical
Publication, Pune.
E-References
http://benchoi.info/Bens/Teaching/Development/Algorithm/PowerPoint/
CH05.ppt http://cs.baylor.edu/~maurer/aida/courses/adversar.ppt
www.inf.ed.ac.uk/teaching/courses/propm/papers/Cook.pdf
http://www.vidyasagar.ac.in/journal/maths/Vol12/JPS12-23.pdf
Unit 14 Coping with the Limitations of
Algorithm Power
Structure:
14.1 Introduction
Objectives
14.2 Backtracking
Outline of the algorithm
14.3 Branch and Bound
Outline of the algorithm
Effectiveness of the algorithm
14.4 Approximation Algorithms for NP-Hard Problems
Underlying principles
Approximation algorithms
14.5 Summary
14.6 Glossary
14.8 Answers
14.1 Introduction
In the earlier unit, you have learnt about the different limitations of algorithm
power. In this unit we will study about how to cope with some of these
limitations.
Combinatorial problems include counting of structures of a specific kind or
size, identifying the largest, smallest or optimal objects, constructing and
analyzing combinatorial structures. Backtracking and Branch and Bound
algorithm design techniques help in solving some of the large instances of
combinatorial problems. They define potential solutions, component by
component, and evaluate these partial solutions. They do not generate
solutions for the remaining components if they determine that these
components do not lead to a solution.
Both Backtracking and Branch and Bound construct state-space trees. The
nodes of these trees indicate the choices of solutions of a component. When
no solution can be obtained by consideration of the choices corresponding
to the descendants of the node, both these techniques terminate.
Branch and Bound is generally used for optimization problems as it
computes a bound on possible values of the objective function of the
problem. It usually uses the Best-first rule to construct space trees.
Backtracking is mostly used for non-optimization problems. This technique
constructs space trees using the Depth-First approach.
This unit also discusses approximation algorithms for NP-Hard problems like
the Traveling Salesman and the Knapsack problems. Greedy algorithms are
good approximation algorithms for the Knapsack problem.
Objectives:
analyze the algorithm technique of Backtracking
explain the solution strategy of Branch-and-Bound
discuss the approximation approach to cope with limitations of NP-Hard
problems
14.2 Backtracking
Problems that need to find an element in a domain that grows exponentially
with the size of the input, like the Hamiltonian circuit and the Knapsack
problem, are not solvable in polynomial time. Such problems can be solved
by the exhaustive search technique, which requires identifying the correct
solution from many candidate solutions. Backtracking technique is a
refinement of this approach. Backtracking is a surprisingly simple approach
and can be used even for solving the hardest Sudoku puzzle.
We can implement Backtracking by constructing the state-space tree, which
is a tree of choices. The root of the state-space tree indicates the initial
state, before the search for the solution begins. The nodes of each level of
this tree signify the candidate solutions for the corresponding component. A
node of this tree is considered to be promising if it represents a partially
constructed solution that can lead to a complete solution, else they are
considered to be non-promising. The leaves of the tree signify either the
non-promising dead-ends or the complete solutions.
We use the Depth-First-search method usually for constructing these state-
space-trees. If a node is promising, then a child-node is generated by
adding the first legitimate choice of the next component and the processing
continues for the child node. But if a node is non-promising, then the
algorithm backtracks to the parent node and considers the next promising
solution for that component. But if there are no more choices for that
component, the algorithm backtracks one more level higher. The algorithm
stops when it finds a complete solution.
This technique is illustrated by the figure 14.1. Here the algorithm goes from
the start node to node 1 and then to node 2. When no solution is found it
backtracks to node1 and goes to the next possible solution node 3. But
node 3 is also a dead-end. Hence the algorithm backtracks once again to
node 1 and then to the start node. From here it goes to node 4 and repeats
the procedure till node 6 is identified as the solution.

Figure 14.1: Backtracking Technique
14.2.1 Outline of the algorithm
The Backtracking algorithm constructs solutions for each component
sequentially and evaluates these partially constructed solutions. If the
algorithm can develop a partially constructed solution without violating the
problem constraints, it considers the first legitimate solution for the next
component. But if there is no legitimate solution for the next component or
for the remaining components, then the algorithm backtracks to replace the
last partially constructed solution with the next option.
The output of the Backtracking algorithm is an n-tuple (b
1
, b
2
,, b
n
) where
each co-ordinate b
i
is an element of a finite linearly ordered set S
i.
The tuple
may also have to satisfy some additional constraints depending on the
problem. The tuple can be of either fixed length or variable length depending
on the problem. A state-space tree is generated by all algorithms. The
nodes of this space tree represent the partially constructed tuples. The
algorithm finds the next element in S
i.+1
if a tuple (b
1
, b
2
,, b
i
) is not a
solution ensuring that it is consistent with the values of (b
1
, b
2
,, b
i
) and
the constraints of the problem. The algorithm backtracks to consider the
next value of b
i
if it does not find such an element.
The following pseudo code explains the Backtracking algorithm.
Pseudocode of Backtracking Algorithm
Backtrack (B[1..i])
// Input: B[1..i] indicates the first i promising parts of a solution
//Output: All the tuples which are the solution of the problem
If B[1..i] is a solution write B[1..i]
else
for each element e S i +1 consistent with B[1..i] and the constraints do
B[1..i] e
Backtrack (B[1..i+1])
In the worst case, Backtracking may have to generate all possible
candidates. We can reduce the size of the state space trees by the following
methods:
Considering the symmetry present in all combinatorial problems
Pre-assigning values to one or more components of a solution
Pre-sorting data
Let us now consider some examples of problems that can be solved by the
Backtracking technique.
Examples
The n-Queens problem, the Hamiltonian circuit and the subset-sum problem
are some examples of problems that can be solved by Backtracking. We will
first consider the n-Queens problem.
n-Queens problem
In the n-Queens problem, we place n queens on an n-by-n chessboard such
that no two queens can be in the same row or same column or in the same
diagonal. When n = 1, the problem has a trivial solution. The problem has
no solution when n = 2 or n = 3. Using the Backtracking technique, we need
to allot a column for each queen in such a way that each queen needs to be
placed in its own row.
Let us use the Backtracking technique to solve the 4-Queens problem. We
will denote the positions of the queen as (row, column). Initially, the first
queen is placed in (1, 4). The second queen cannot be placed in either the
first row or the fourth column. So, the second queen is placed in its first
acceptable position (2, 1). The third queen is then placed in (3, 3). But we
then see that the fourth queen will not have any acceptable position. So the
algorithm backtracks and places the second queen in its next acceptable
position (2, 2). But now we cannot find any acceptable position for the third
and fourth queen. So, we find that we have reached a dead-end. Once
again the algorithm backtracks and places the first queen in (1, 3). The
second queen moves to (2, 1), the third queen to (3, 4), and the fourth
queen to (4, 2). This is the solution to the problem. Figure 14.2 depicts the
state space tree for solving the 4-Queens problem.

Figure 14.2: State Space Tree for Solving the 4-Queens Problem
Let us next use the Backtracking technique to solve the Hamiltonian circuit
problem.

Hamiltonian circuit problem
The graph of the Hamiltonian circuit is shown in Figure 14.3. We assume
that the Hamiltonian circuit starts at vertex i, which is the root of the state-
space tree. From i, we can move to any of its adjoining vertices which are j,
k, and l. We first select j, and then move to k, then l, then to m and thereon
to n. But this proves to be a dead-end. So, we backtrack from n to m, then to
l, then to k which is the next alternative solution. But moving from k to m
also leads to a dead-end. So, we backtrack from m to k, then to j. From
there we move to the vertices n, m, k, l and correctly return to i. Thus the
circuit traversed is i-> j -> n-> m-> k-> l-> i.

Figure 14.3: Graph of Hamiltonian Circuit
Figure 14.4 shows the state-space tree for the above graph.

Figure 14.4: State-Space Tree for Finding a Hamiltonian Circuit
Let us next consider the Subset-Sum problem.
Subset-Sum problem
In the Subset-Sum problem, we have to find a subset of a given set
S = {s
1
,

s
2
,

.., s
n
} of n positive integers whose sum is equal to a positive
integer t. Let us assume that the set S is arranged in ascending order. For
example, if S = {2, 3, 5, 8} and if t = 10, then the possible solutions are
{2, 3, 5} and {2, 8}.
Figure 14.5 shows the state-space tree for the above set. The root of the
tree is the starting point and its left and right children represent the inclusion
and exclusion of 2. Similarly, the left node of the first level represents the
inclusion of 3 and the right node the exclusion of 3. Thus the path from the
root to the node at the i
th
level shows the first i numbers that have been
included in the subsets that the node represents. Thus, each node from
level 1 records the sum of the numbers S
sum
along the path upto that
particular node. If S
sum
equals t, then that node is the solution. If more
solutions have to be found, then we can backtrack to that nodes parent and
repeat the process. The process is terminated for any non-promising node
that meets any of the following two conditions:
S
sum
+ S
i+1
> t (the sum is too large)

n
S
sum
+ s
j
< t (the sum is too small)

j = i+ i

Figure 14.5: State-Space Tree of the Subset-Sum Problem
1. We can implement Backtracking by constructing the _______________.
2. Backtracking, in the _______ case may have to generate all possible
candidates in a problem state that is growing exponentially.
3. The n-Queens problem, the _____________ circuit and the Subset-
Sum problem are some examples of problems that can be solved by
Backtracking.
Activity 1
Construct a state-space tree for finding the subset of the instance
S = {2, 3, 5, 7, 9} that gives the sum t = 12.
14.3 Branch and Bound
Branch and Bound (BB) is a generic algorithm for finding optimal solutions
of various optimization problems, specifically in discrete and combinatorial
optimization. Let us now analyze this algorithm.
14.3.1 Outline of the algorithm
Backtracking cuts off a branch of the problems state-space tree as soon as
the algorithm deduces that it cannot lead to a solution. Branch and Bound
organizes details of all candidate solutions, and discards large subsets of
fruitless candidates by using upper and lower estimated bounds of the
quantity being optimized.
A feasible solution is a solution that satisfies all the constraints of a problem
and the one with the best value of objective function is considered an
optimal solution. Branch and Bound requires the following two additional
items when compared to Backtracking:
A method to provide, for every node of a state-space tree, a bound on
the best value of the objective function on any solution that can be
obtained by adding further components to partially constructed solution
indicated by the node.
The value of best solution that has been identified
A Branch and Bound procedure requires two tools. The first tool is a
procedure that splits a given set S of candidates into two or more smaller
sets S
1
, S
2
whose union covers S. Note that the minimum of f(x) over S is
min{ v
1
, v
2
.}, where each v
i
is the minimum of f(x) within Si. This step is
called branching as its recursive application defines a tree structure (the
search tree) whose nodes are the subsets of S. The second tool is a
procedure called bounding that computes the upper and lower bounds for
the minimum value of f(x) within a given subset S.
When the Branch and Bound algorithm identifies that the lower bound for
some tree node (set of candidates) A is greater than the upper bound for
some other node B, then it discards A from the search. This step is called
pruning, and is usually applied by maintaining a global variable m (shared
among all nodes of the tree) that records the minimum upper bound which is
found among all sub regions verified and discards any node whose lower
bound is greater than m.
Example: Assignment problem
The Branch and Bound approach is illustrated by applying it to the problem
of assigning 'n' people to n jobs so that the total cost of the assignment is
as small as possible. An instance of assignment problem is specified by
n-by-n cost matrix C so that the problem can be stated as follows. Select
one element in each of the matrix so that no two selected elements are in
the same column and their sum is the smallest possible.

Matrix C

This problem can be solved using branch and bound technique by
considering a small instance:

Figure 14.6: Level 0 and 1 of the State Space Tree for the Example
Assignment Problem
Figure 14.6 shows Levels 0 and 1 of the state space tree for the instance of
the assignment problem being solved with the best-first branch and bound
algorithm. The number above a node shows the order in which the node
was created. A nodes fields indicate the job number assigned to person a
and the lower bound value, lb, for this node.
We can find a lower bound on the cost of an optimal selection without
solving the problem. We know that the cost of any solution, including an
optimal one, cannot be smaller than the sum of smallest elements in each of
the matrixs rows. Therefore, here, the sum is 5+2+1+3=11.This is not the
cost of any valid selection. It is just a lower bound on the cost of any valid
selection .We will apply the same idea to partially constructed solutions. For
example, for any valid selection that selects from the first row, the lower
bound will be 8+4+1+3=16.
The problems state space tree deals with the order in which trees node will
be generated. Here we will generate all the children of the most promising
node among non terminated leaves in the current tree. We can tell about the
most promising nodes by comparing the lower bounds of the live nodes. It is
wise to consider a node with best bound as most promising, though this
does not prevent the possibility that an optimal solution will ultimately belong
to a different branch of the state-space tree. The variation of the strategy is
called Best-First-Branch-and-Bound and is shown in figure 14.7.

Figure 14.7: State Space Tree for Best-First-Branch-and-Bound
In the instance of the assignment problem given earlier in figure 14.6, we
start with the root that corresponds to no elements selected from the cost
matrix. As we already discussed, the lower bound value for the root,
denoted lb, is 11.The nodes on the first level of the tree correspond to
selections of an element in the first row of the matrix, that is, a job for person
a.in Matrix C.
Of the four live leaves (nodes 1 through 4) that can contain an optimal
solution, node 3 is the most promising because it has the lowest smaller
bound value. Following the Best-First Search strategy, we branch out from
that node first by considering three different ways of selecting an element
from the second row but not in the third column-the three different jobs that
can be assigned to person b in Matrix C.
Among the six live leaves (nodes 1, 2, 4, 5, 6, and 7) that may contain an
optimal solution, we again choose the one with the least lower bound, node
5. First, we consider selecting the second columns element from cs row
(assigning person c to job 2). We then have to select the element from
fourth column of ds row (assigning person d to job 4). This produces leaf 8
(figure 14.8), which corresponds to the acceptable solution - {a->3, b->1,
c->2, d->4} with the total cost of 11. Its sibling, node 9, corresponds to the
acceptable solution {a->2, b->1, c->4, d->3} with total cost of 23. As the cost
of node 9 is larger than the cost of the solution represented by leaf 8, node 9
is terminated.
When we examine all the live leaves of the last state-space tree (nodes 1, 2,
4, 6, and 7) of figure 14.8, we discover that their lower bound values are not
smaller than 11, the value of the best selection seen so far (leaf 8). Hence,
we end the process and identify the solution indicated by leaf 8 as the
optimal solution to the problem.

Figure 14.8: Complete Space Tree for the Instance of the Assignment Problem
14.3.2 Effectiveness of the algorithm
In Branch and Bound algorithm, the ratio of the number of solutions verified
largely decreases as the size of the problem increases. However; the
algorithm has one important limitation. Because a large number of solutions
must be kept in storage as the algorithm proceeds, the method is applicable
for problems which are of reasonable size and which are not likely to
increase to a large number of combinatorial possibilities. For problems
exceeding available storage, the Backtracking algorithm is suitable.
4. ___________________________ organizes details of all candidate
solutions, and discards large subsets of fruitless candidate solutions.
5. A _____________________ is a solution that satisfies all the
constraints of a problem.
6. In Branch and Bound algorithm, the ratio of the number of solutions
verified largely _______________ as the size of the problem increases.
14.4 Approximation Algorithms for NP Hard Problems
Combinatorial optimization problems lie within a finite but huge feasible
region. In this section, we focus on finding approximation algorithms for
optimization problems that are NP-Hard.
14.4.1 Underlying principles
An NP-Hard problem is one for which the algorithm can be translated to one
that can solve any NP-problem (non-deterministic polynomial time). Many
optimization problems do not have an algorithm that can find a solution for
all instances. Sometimes, when trying to find an optimal solution to some
problems we realize that it is NP-Hard. Such problems also do not have any
known polynomial-time algorithms. Exhaustive search algorithms can be
used to solve NP-Hard problems that have small instances. Dynamic
programming technique can also be used only if the instance parameters
are small. Hence we can use approximation algorithms to find a solution
which is near optimal to solve these problems. Even many real life
applications lack accurate data to operate with. In such situations, we can
only use approximation algorithms.
Most of these approximation algorithms are based on some heuristic which
is problem specific.
We also would like to determine how accurate the outputs of these
approximation algorithms are. The accuracy ratio of the approximation
algorithms is given in equation 14.1.
f (s
a
)
r(s
a
) = Eq: 14.1
f( s*)
Here sa is an approximate solution to the problem, s* is an exact solution to
the problem and r(sa) is the accuracy ratio. The closer r(sa) is to 1 the more
accurate is the approximate solution. But mostly, we do not know the value
of f(s*) the optimal value of the objective function. Hence, we should try to
obtain a good upper bound for the values of r(sa). We then can define
approximation algorithms in the following manner.
Definition: A polynomial approximation algorithm is said to be a
c-approximation algorithm, where c is greater than or equal to 1, if the
accuracy ratio of the approximation does not exceed c for any instance of
the problem.
This definition is reflected in Equation 14.2.
r( s
a
) < c Eq: 14.2
Finding approximate solutions with a reasonable level of accuracy is easier
for some problems. Some problems have real life applications which can be
solved by using approximation algorithms. The Traveling Salesman problem
is an example for this.
Combinatorial problems like the Traveling Salesman problems and the
Minimum Spanning Tree have at least a part of the input as integers.
Algorithms for these problems involve mathematical operations like addition
and comparison. There are no explicit bounds for these integers and hence
they can be very large. The time required for computations involving these
integers can grow logarithmically with the integers. So we bound these
operations by having an upper limit for the integers. We can also solve the
Knapsack problem by using an approximation algorithm.
14.4.2 Approximation algorithms
We shall now analyze the approximation solutions for the Traveling
Salesman and the Knapsack problems as they do not have optimal
solutions.

Approximation algorithms for the Traveling Salesman problem
There are several approximation algorithms for the Traveling Salesman
problem. Let us discuss a few of these approximation algorithms.
Nearest Neighbor algorithm
Let us analyze the simple greedy algorithm that is based on nearest
neighbor heuristic.
Step 1 Let us choose an arbitrary city as the start.
Step 2 We then go to a neighboring unvisited city which is nearest to the
city chosen. We repeat this operation till we visit all the cities.
Step 3 Then we return to the starting city.
Let us now consider an instance of the Traveling Salesman problem.
Consider the graph depicted for the Traveling Salesman problem in figure
14.9 which has a as the starting vertex.

Figure 14.9: Graph of the Instance of the Traveling Salesman Problem
Using the above described Nearest-Neighbor algorithm yields the tour of
length 12 say la: a-b-c-d-a. But if we want the optimal solution, we can
implement exhaustive search and the tour is of length 10 lb: a-b-d-c-a.
The accuracy ratio of this approximation is given in equation Eq: 14.3.
F(l
a
) = r(l
a
)/ r(l
b
) = 12/10=1.2 Eq: 14.3
In equation 14.3, F(la) is the accuracy ratio, r(la) is the tour length using
Nearest-Neighbor algorithm and r(lb) is the tour length using exhaustive
search algorithm. We conclude that although the above algorithm is very
simple it does not give us an accurate solution. Let us next analyze the
Multifragment-Heuristic algorithm to get a solution for the Traveling
Salesman problem.
Multifragment Heuristic algorithm
This algorithm gives more emphasis for the edges of a complete weighted
graph.
Step 1: We sort the edges in increasing order according to their weights.
Step 2: We repeat this step till we get a tour of length n where n is the
number of cities. We add the next edge to the sorted edge list of tour edges
provided we do not create a vertex of 3 degree or a cycle of length less than
n. If that is the case we can skip the edge.
Step 3: Then finally we return to the set of tour edges.
When we apply the Multifragmnent Heuristic algorithm to the graph in
Figure 14.9, we get the solution as {(a, b), (c, d), (b, c), (a, d)} which is very
similar to the tour produced by the Nearest Neighbor algorithm.
In general, the Multifragmnent Heuristic algorithm provides significantly
better tours than the Nearest Neighbor algorithm but the performance ratio
of the Multifragment Heuristic algorithms is unbounded.
We will next discuss the Minimum-Spanning Tree based algorithm.
Minimum-Spanning-Tree-based algorithm
There are some approximation algorithms that make use of the connection
between Hamiltonian circuit and spanning trees of the same graph. When
we remove an edge from a Hamiltonian circuit it yields a spanning tree.
Thus the Minimum Spanning Tree provides us a good basis for constructing
a shortest approximation tour.
Twice Around the Tree algorithm
Step 1: We should build a Minimum Spanning Tree of the graph according
to the given instance of the Traveling Salesman problem.
Step 2: We should start with an arbitrary vertex, walk around the Minimum
Spanning Tree and record all the vertices that we pass.
Step 3: We should scan the vertex list obtained in step 2 and eliminate all
the repeated occurrences of the same vertex except the starting one. We
form a Hamiltonian circuit of the vertices that are remaining on the list which
is the output of the algorithm.
Let us analyze the above algorithm with a graph as shown in Figure 14.10.

Figure 14.10: Graph Illustrating the Twice-Around the Tree Algorithm
We know that Minimum Spanning Tree is made up of edges (a, b), (b, c),
(b, d) and (d, e). Then the twice around the tree walk that starts and ends at
a is a, b, c, b, d, e, d, b, a
But if we eliminate the second b (to get a short cut way from c to d), the
second d and third b (to get a shortcut way from e to a) we get the
Hamiltonian circuit a, b, c, d, e, a which is of length 21.
Approximation algorithms for the Knapsack problem
Another well known NP-Hard problem is the Knapsack problem. In this
problem, we are given n items of known weights w1, w2, .., wn and values
v1, v2..vn and a knapsack which has the capacity of weight W. We then
have to find the most suitable subset of the items that can fit into the
knapsack. We consider many approximation algorithms for this problem
also. The Greedy algorithm for the Knapsack problem selects the items in
decreasing order of their weights in order to use the knapsack capacity
efficiently.
Now let us see an algorithm based on this Greedy heuristic.
Greedy algorithm for the Discrete Knapsack problem
Step 1: We compute value to weight ratios ri = vi/wi, where i=1..,n for the
items that are given to us.
Step 2: We should sort the items in non-increasing order of the ratios that
we already computed in step 1.
Step 3: We repeat the above operation till no item is left in the sorted list.
We place the current item on the list in the knapsack if it fits in else we
consider the next item.
Let us assume the instance of the Knapsack problem with its capacity equal
to 10 and the item information as given in Table 14.1.
Table 14.1: Item Information for the Knapsack problem
Item Weight Value
1 4 $30
2 5 $40
3 6 $18
We then compute value to weight ratios and sort the items in decreasing
order. The item information after sorting is given in Table 14.2.
Table 14.2: Sorted Item Information for the Knapsack problem
Item Weight Value Value/weight
1 4 $40 10
2 5 $30 6
3 6 $18 3
We compute the value to weight ratios and sort the items in decreasing
order. We select the first item weighing 4, skip the next item of weight 7,
select the next item of weight 5 and skip the last item of weight 3 using
Greedy algorithm. The solution we have found is optimal for the above
example. But Greedy algorithms do not always yield an optimal solution.
There is also no finite upper bound on the accuracy of these approximation
solutions.
Greedy algorithm for Continuous Knapsack problem
Step 1 We compute the value to weight ratios ri = vi/wi,i=1..,n for the
items that are given to us.
Step 2 We should sort the items in non increasing order of the ratios that
we already computed in step 1.
Step 3 We repeat the following procedure until we fill the knapsack to its
capacity or until no items remain in the sorted list. If the entire current item
can fit into the knapsack, place it in the knapsack and then consider the next
item, else place the largest fraction of the current item that can fit in the
knapsack and stop.
7. ________________ algorithms can be used to solve NP-Hard
problems that have small instances.
8. Minimum Spanning tree provides us a good basis for constructing a
_________ approximation tour.
9. We select the items in __________ order of their weights in order to
use the knapsack capacity efficiently.
Activity 2
Given the following information, solve the Knapsack problem using the
Greedy algorithm. The knapsack has a maximum capacity of 15.
Item: 1 2 3 4
Weight: 6 4 2 5
Value: 22 25 15 12
14.5 Summary
In this unit, we analyzed some solutions to cope with the limitations of some
algorithms. Backtracking and Branch and Bound algorithm design
techniques help in solving some of the large instances of combinatorial
problems.
The Backtracking algorithm constructs solutions for each component
sequentially and if it finds that it can develop a partially constructed solution
without violating the problem constraints, it considers the first legitimate
solution for the next component. But if there is no legitimate solution for the
next component or for the remaining components, then the algorithm
backtracks to replace the last partially constructed solution with the next
option. We also discussed how to solve the n-Queen problem, the
Hamiltonian circuit problem and the subset sum problem using the
backtracking approach.
Branch and Bound (BB) is a generic algorithm for finding optimal solutions
of various optimization problems, specifically in discrete and combinatorial
optimization. We discussed how to solve an instance of the Assignment
problem using the Branch and Bound approach.
We can use approximation algorithms to find a solution which is near
optimal to solve NP-Hard problems. We discussed some algorithms to solve
the Traveling Salesman problem and the Knapsack problem.
14.6 Glossary
Terms Description
Polynomial-time The execution time of a computation m(n) is said to be
in polynomial time when it is at most a polynomial
function of the problem size n.
Exhaustive search
algorithm
This algorithm produces the complete solution space for
the problem.
1. How will you solve the 4-Queens problem using the Backtracking
technique?
2. What is the basic principle of the Branch and Bound technique?
3. How can you solve the Traveling Salesman problem using the Nearest-
Neighbor algorithm?
4. Discuss the Greedy algorithm for the Discrete Knapsack problem.
14.8 Answers
1. State-space tree
2. Worst
3. Hamiltonian
4. Branch and Bound
5. Decreases
6. Optimal
7. Exhaustive search
8. Shortest
9. Decreasing
Terminal Questions
1. Refer section 14.2.1 Outline of the algorithm.
2. Refer section 14.3.1 Outline of the algorithm.
3. Refer section 14.4.2 Approximation algorithms
4. Refer section 14.4.2 Approximation algorithms
References
Christos, H. Papadamitrou., & Kenneth Steiglitz (1998). Combinatorial
Optimization. Algorithms and Complexity: Prentice Hall, New York
E-References
www2.siit.tu.ac.th/bunyarit/courses/its033/slides/ITS033x12x
LimitationxofxAlgorithm.ppt

__________________

MIT203.pdf For MSC IT

Caricato da

Informazioni sul documento

Titolo originale

Copyright

Formati disponibili

Condividi questo documento

Condividi o incorpora il documento

Opzioni di condivisione

Hai trovato utile questo documento?

Questo contenuto è inappropriato?

Copyright:

Formati disponibili

MIT203.pdf For MSC IT

Caricato da

Copyright:

Formati disponibili

Analysis and Design of Algorithms Unit 1

Sikkim Manipal University Page No. 1

for each entry which adds up to 3+4=7.

for n k s s 0 Eq: 10.6

1 to n-1 do//diagonal count

C[I,k-1]+ C[k+1,j]; kmin

Potrebbero piacerti anche