5/25/10

Problems, solu=ons, algorithms

CS420 lecture one Problems, algorithms, decidability, tractability
In this course we study ques=ons such as:
given a problem, how do we nd an (ecient) algorithm solving it? How do we measure the complexity (=me, space requirements) of an algorithm.

Problem: set of instances of a ques=on

is 2 a prime?, .., is 4 a prime?, is 2100000 a prime

Solu0on: set of answers

yes, .., no, no

Algorithm
Problem: ques=on + input(s) Algorithm: eec0ve procedure
mapping input to output

Is there an algorithm for each problem?

No
1. the problem needs to be eec0vely specied "how many angels can dance on the head of a pin?" not eec=ve 2. even if it is eec=vely specied, there is not always an algorithm to provide an answer

eec=ve: unambiguous, executable

Turing dened it as: "like a Turing machine" program = eec=ve procedure

the size of the problem: an integer n,

# inputs (for sor=ng problem) #digits of input (for prime problem) some=mes more than one integer

5/25/10

Ulam's problem
f(n): if (n==1) return 1 else if (odd(n)) return f(3*n+1) else return f(n/2)

Ulam's problem
f(n): if (n==1) return 1 else if (odd(n)) return f(3*n+1) else return f(n/2)

f(1) Try a few inputs, does f(n) always stop?

f(2), f(4), f(8), f(2n) .... f(3) f(5) f(7) f(9)

Ulam's problem
f(n): if (n==1) return 1
else if (odd(n)) return f(3*n+1) else return f(n/2) -Nobody has ever found an n for which f does not stop -Nobody has ever found a proof: n : f (n) stops (so there can be no algorithm deciding this.)

Hal=ng problem is undecidable

Given a program P an an input x will P(x) stop? we can prove: the hal0ng problem cannot be solved ie there is no algorithm Halt(P,x) that for any program P and input x decides whether P(x) stops.

5/25/10

Undecidability

Hal=ng problem is undecidable

Assume there is a program P(P1,D) that for any P1 and D, outputs Yes (if P1(D) halts) or No (if P1(D) loops) Then construct Q'(P2): if (P(P2,P2)) loop else halt Now execute Q'(Q') if P(Q', Q') says Q'(Q') halts it loops if P(Q', Q') says Q'(Q') loops it halts CONTRADICTION -> P does not exist!!

A problem P(x) is undecidable, if there is no algorithm solving P for any legal x

Verica=on/equivalence undecidable
Given any specica=on S and any program P, there is no algorithm that decides whether P executes according to S Given any two programs P1 and P2, there is no algorithm that decides x: P1(x)=P2(x) Does this mean we should not build program veriers?

Intractability
Suppose we have a program,
does it execute a in a reasonable =me? Eg, towers of Hanoi.
Three pegs, one with n smaller and smaller disks, move (1 disk at the =me) to another peg without ever placing a larger disk on a smaller

f(n) = # moves for tower of size n Monk: before a tower of Hanoi of size 100 is moved, the world will have vanished

5/25/10

Example: Tower of Hanoi, move all disks to third peg without ever

placing a larger disk on a smaller one.

Example: Tower of Hanoi, move all disks to third peg without ever

placing a larger disk on a smaller one.

13

14

Recursive Algorithms
Example: Tower of Hanoi, move all disks to third peg without ever

placing a larger disk on a smaller one.

Example: Tower of Hanoi, move all disks to third peg without ever

placing a larger disk on a smaller one.

f(n) = f(n-1) +

f(n) = f(n-1) + 1 +

15

16

5/25/10

f(n): #moves
Example: Tower of Hanoi, move all disks to third peg without ever

placing a larger disk on a smaller one.

f(n) = 2f(n-1) + 1, f(1)=1 f(1) = 1, f(2) = 3, f(3) = 7, f(4) = 15 f(n) = 2n-1 How can you show that?

f(n) = f(n - 1) + 1 + f(n - 1) f(n) = 2f(n - 1) + 1, f(1) = 1

Can you write an itera=ve Hanoi algorithm?

17

Was the monk right?

2100 moves, say 1 per second..... How many years???

Was the monk right?

2100 moves, say 1 per second..... 2100 ~ 1030 ~ 1025 days ~ 3.1022 years more than the age of the universe

5/25/10

Is there a beler algorithm?

Is there a beler algorithm?

Pile(n-1) must be o peg1 and on one other peg before disk n can be moved to its des=na=on so (induc=vely) all moves are necessary

Algorithm complexity
Measures in units of 0me and space Linear Search X in dic=onary D i=1; while (not at end and X!= D[i]) {i++} We don't know if X is in D, and we don't know where it is, so we can only give worst or average =me bounds We don't know the =me for atomic ac=ons, so we only determine Orders of Magnitude

=me complexity
In the worst case we search all of D, so the loop body is executed n =mes In the average case the loop body is executed about n/2 =mes. Why?

5/25/10

=me complexity
In the worst case we search all of D, so the loop body is executed n =mes In the average case the loop body is executed about n/2 =mes. In average case analysis we sum the products of the probability of each outcome and the cost of that outcome, here
n n

Units of =me
1 microsecond ? 1 machine instruc=on? #code fragments that take constant =me?

1/n * i = 1/n i = (n(n +1) /2) /n n /2

i=1 i=1

Units of =me
1 microsecond ? no, too specic and machine dependent 1 machine instruc=on? no, s0ll too specic #code fragments that take constant =me? yes bit?

unit of space

int?

5/25/10

unit of space
bit? very detailed but some0mes necessary int? nicer, but dangerous: we can code a whole program or array (or disk) in one arbitrary int, so we have to be careful with space analysis (take value ranges into account when needed)

Orders of magnitude
If an algorithm with input size n takes less than c.n2 steps to execute, we say it is an order n squared algorithm, nota=on O(n2) In general g(n) = O(f(n)) if there is a constant c such that g(n) <= c.f(n) for all but a nite number of values for n. In other words, there is a n0 such that g(n) <= c.f(n) for all n>n0

Worst / average case complexity

Worst case : maximal number of steps / bits / words an algorithm needs for inputs of size n Average case: the expected number of steps / bits / words an algorithm needs:

Is there a beler algorithm?

Is there a beSer algorithm than linear search? What does that mean?

P .C
i iI n

Pi : probability input i occurs Ci : complexity given input i In : all possible inputs of size n

5/25/10

Is there a beler algorithm?

Is there a beSer algorithm than linear search? What does that mean? 2*beler? beler order of magnitude? beler worst case? beler average case?

Is there a beler algorithm?

Is there a beSer algorithm than linear search? What does that mean? 2*beler? NO beler order of magnitude? beler worst case? beler average case? usually beSer worst case

Binary Search
Because dic=onaries are sorted BS(x,lo,hi): if (lo>hi) not found else { m = (lo+hi)/2; if (x==D[m]) found else if (x<D[m]) BS(x,lo,m-1) else BS(x,m+1,hi) }

Binary Search
Because dic=onaries are sorted BS(x,lo,hi): if (lo>hi) not found else { m = (lo+hi)/2; if (x==D[m]) found else if (x<D[m]) BS(x,lo,m-1) else BS(x,m+1,hi) } why m-1 and m+1?

5/25/10

Binary search worst case =me

Every 'chop' takes half of the remaining possibili=es away The number of =mes we can divide n>0 by 2 un=l it gets to 0: log2(n) so, worst case O(log(n)) why?

Is there an even beler algorithm?

uhhh...

Is there an even beler algorithm?

now we have to consider all possible algorithms Lower bound of a problem
given a set of "allowed steps"
taking constant =me

Searching in an ordered array

allowed steps:
comparison to /assignment of array element arithme=c on index some ac=on on found or not found

lower bound is minimal worst case complexity of any algorithm, using only the allowed steps, solving it

eg NOT if (isElement(x,D)) ... why NOT?

10

5/25/10

Event / decision trees

Given these steps each algorithm is associated with a set of event or decision trees:
comparison is an internal node with two children (not) found is a leaf other ops are branches leading to new decisions or leaves

event tree for linear search

compare not found compare not found found found ac=on found ... compare not found not found ac=on found found found ac=on found found found ac=on found

For some input a path through the tree to a par=cular leaf is taken

event tree for binary search

compare not found found ac=on found found compare not found compare not found found found ac=on found found found ac=on found

Height of the event tree

The event trees for any search are binary trees. Why? Any tree associated with the search has at least n+1 leaves (n found, 1 not found) Theorem: A binary tree with n leaves has a height of at least log(n). Check out some cases. There is at least one path from root to a leaf of length at least log(n), so any search algorithm takes at least O(log(n)) =me for some of its inputs. (In the next lecture we will use big Omega for lower bounds.)

11

5/25/10

Lower bound for search

Lower bound for search in an ordered array is O(log(n)) There is no asympto=cally beler search than binary search But what if I know more, eg search in an array with a xed range of values...

Upper and lower bounds

Upper bound of a problem:
the lowest worst case complexity of any known algorithm solving for the problem. eg search: O(log(n)) binary search

Lower bound of a problem

Maximum (the minimum number of steps we can prove any algorithm will need to execute) eg any search algorithm takes at least O(log(n)) steps

Closed problems, algorithmic gaps

Closed problem lower bound = upper bound eg search in ordered array Algorithmic gap lower bound < upper bound eg NP Complete problems

NP Complete problems
NPC: a certain class of problems with algorithmic gaps upper bound: exponen=al lower bound: polynomial Trial and error type algorithms are the only ones we have so far to nd an exact solu=on

12

5/25/10

Some NPC problems

TSP: Travelling Salesman given ci=es c1,c2,...,cn and distances between all of these, nd a minimal tour connec=ng all ci=es. SAT: Sa=sability given a boolean expression E with boolean variables x1,x2,...,xn determine a truth assignment to all xi making E true

Back tracking
Back tracking searches (walks) a state space, at each choice point it guesses a choice. In a leaf (no further choices) if solu=on found OK, else go back to last choice point and pick another move. A Non Determinis=c algorithm is one with the ability to select the sequence of right guesses (avoiding back track)

P vs NP

Coping with intractability

NP problems become intractable quickly NP is the class of problems solvable in TSP for 100 ci0es? How would you polynomial =me by a non determinis=c enumerate all possible tours? How many? algorithm Coping with intractability: P is the class of problems solvable in polynomial =me by a determinis=c algorithm Approxima=on: Find a nearly op=mal tour Randomiza=on: use a probabilis=c algorithm using the $1,000,000 ques0on: is P = NP ?
"coin tosses" (eg prime witnesses) a $1 ques=on: is Hanoi in NP?

13