Some Great Theoretical Ideas in Computer Science for

15-251

Probability II
(February 21, 2013)

If I randomly put 100 letters into 100 addressed envelopes, on average how many letters will end up in their correct envelopes?

On average, in class of size m, how many pairs of people will have the same birthday?

A new tool called Linearity of Expectation

Random Variable
To use this new tool, we will also need to understand the concept of a Random Variable

Random Variable
Let S be sample space in a probability distribution A Random Variable is a real-valued function on S Examples: X = value of white die in a two-dice roll X(3,4) = 3, Y(3,4) = 7, X(1,6) = 1 Y(1,6) = 7 Y = sum of values of the two dice

Notational Conventions
Use letters like A, B, E for events Use letters like X, Y, f, g for R.V.s R.V. = random variable

Two Views of Random Variables

Think of a R.V. as Input to the function is random

Two Coins Tossed

X: {TT, TH, HT, HH} {0, 1, 2} counts the number of heads Distribution on the reals

A function from S to the reals R Or think of the induced distribution on R Randomness is pushed to the values of the function

HH ! ! ! ! TT TH HT ! ! "

2 0 1

Its a Floor Wax And a Dessert Topping

Its a function on the sample space S

From Random Variables to Events

For any random variable X and value a, we can define the event A that X = a Pr(A) = Pr(X=a) = Pr({t ! S| X(t)=a})

Its a variable with a probability distribution on its values

You should be comfortable with both views

Two Coins Tossed

X: {TT, TH, HT, HH} {0, 1, 2} counts # of heads Pr(X = a) = Pr({t ! S| X(t) = a}) Pr(X = 1) = Pr({t ! S| X(t) = 1}) = Pr({TH, HT}) = "

From Events to Random Variables

For any event A, can define the indicator random variable for A: XA(t) = 1 if t ! A 0 if t " A
1
0.55

HH ! ! ! ! TT TH HT

X
! ! "

2 0 1

0.05 0 0.3 0.2

0.05 0.1 0.3

0
0.45

Distribution on X

Definition: Expectation
The expectation, or expected value of a random variable X is written as E[X], and is E[X] = # Pr(t) X(t) = # k Pr[X = k] t!S k X is a function on the sample space S X has a distribution on its values

A Quick Calculation
What if I flip a coin 3 times? What is the expected number of heads? E[X] = (1/8)#0 + (3/8)#1 + (3/8)#2 + (1/8)#3 = 1.5 But Pr[ X = 1.5 ] = 0 Moral: dont always expect the expected. Pr[ X = E[X] ] may be 0 !

Type Checking
A Random Variable is the type of thing you might want to know an expected value of If you are computing an expectation, the thing whose expectation you are computing is a random variable

Indicator R.V.s: E[XA] = Pr(A)

For any event A, can define the indicator random variable for A: XA(t) = 1 if t ! A 0 if t " A

E[XA] = 1 # Pr(XA = 1) = Pr(A)

Adding Random Variables

If X and Y are random variables (on the same set S), then Z = X + Y is also a random variable Z(t) = X(t) + Y(t) E.g., rolling two dice. X = 1st die, Y = 2nd die, Z = sum of two dice

Independence
Two random variables X and Y are independent if for every a,b, the events X=a and Y=b are independent How about the case of X=1st die, Y=2nd die? X = left arm, Y=right arm?

Linearity of Expectation
E[Z] =

# Pr[t] Z(t) t!S # Pr[t] (X(t) + Y(t)) t!S # Pr[t] X(t) + # Pr[t] Y(t)) t!S t!S

If Z = X+Y, then E[Z] = E[X] + E[Y]

Even if X and Y are not independent

= E[X] + E[Y]

Linearity of Expectation
E.g., 2 fair flips: X = heads in 1st coin, Y = heads in 2nd coin Z = X+Y = total # heads What is E[X]? E[Y]? E[Z]? 1,1,2 HH 0,0,0 TT

Linearity of Expectation
E.g., 2 fair flips: X = at least one coin is heads Y = both coins are heads, Z = X+Y Are X and Y independent? What is E[X]? E[Y]? E[Z]? 1,0,1 HT 1,1,2 HH 0,0,0 TT 1,0,1 TH

1,0,1 HT

0,1,1 TH

By Induction
E[X1 + X2 + + Xn] = E[X1] + E[X2] + . + E[Xn] The expectation of the sum = The sum of the expectations

If I randomly put 100 letters into 100 addressed envelopes, on average how many letters will end up in their correct envelopes?

Hmm $k k Pr(k letters end up in correct envelopes) = $k k (aargh!!)

Use Linearity of Expectation

Let Ai be the event the ith letter ends up in its correct envelope Let Xi be the indicator R.V. for Ai Xi = 1 if Ai occurs 0 otherwise So, in expectation, 1 letter will be in the same correct envelope Pretty neat: it doesnt depend on how many letters! Question: were the Xi independent? No! E.g., think of n=2

Let Z = X1 + + X100 We are asking for E[Z] E[Xi] = Pr(Ai) = 1/100 So E[Z] = 1

Use Linearity of Expectation

General approach: View thing you care about as expected value of some RV Write this RV as sum of simpler RVs (typically indicator RVs) Solve for their expectations and add them up!

Example
We flip n coins of bias p. What is the expected number of heads? We could do this by summing

$k k Pr(X = k) = $k k

n k p (1-p)n-k k

But now we know a better way!

Linearity of Expectation!
Let X = number of heads when n independent coins of bias p are flipped Break X into n simpler RVs: Xi = 1 if the ith coin is tails

What does this mean: E[X | A]? Is this true: Pr[ A ] = Pr[ A | B ] Pr[ B ] + Pr[ A | B ] Pr[ B ] Yes! Similarly: E[ X ] = E[ X | A ] Pr[ A ] + E[ X | A ] Pr[ A ]

0 if the ith coin is heads

E[ X ] = E[ #i Xi ] = np

Air Marshal Problem

Every passenger has an assigned seat There are n-1 passengers and n seats Before the passengers board, an air marshal sits on a random seat When a passenger enters the plane, if their assigned seat is taken, they pick a seat at random What is the probability that the last passenger to enter the plane sits in their assigned seat?

Several years ago Berkeley faced a law suit

1.! % of male applicants admitted to graduate school was 10% 2.! % of female applicants admitted to graduate school was 5% Grounds for discrimination? SUIT

Every department was more likely to admit a female than a male Berkeley did a survey of its departments to find out which ones were at fault
#of females accepted to department X #of female applicants to department X #of males accepted to department X #of male applicants to department X

>

Answer
Women tended to apply to departments that admit a smaller percentage of their applicants

How can this be ?

Women
Dept
Applied Accepted Applied

Men
Accepted

A B
total

99 1
100

4 1
5

1 99
100

0 10
10

Try to get a white ball

A single summary statistic (such as an average, or a median) may not summarize the data well !

Better

Choose one box and pick a random ball from it. Max the chance of getting a white ball 5/11 > 3/7

Try to get a white ball

Try to get a white ball

Better

Better

Better

6/9 > 9/14

Better

Try to get a white ball

Simpsons Paradox
Arises all the time Be careful when you interpret numbers

Better

Better Better 11/20 < 12/21 !!!

Department of Transportation requires that each month all airlines report their on-time record
# of on-time flights landing at nations 30 busiest airports # of total flights into those airports
http://www.bts.gov/programs/oai/

Different airlines serve different airports with different frequency

An airline sending most of its planes into fair weather airports will crush an airline flying mostly into foggy airports
It can even happen that an airline has a better record at each airport, but gets a worse overall rating by this method.

Alaska airlines
% on time # flights

America West
% on time # flights

Geometric Random Variable

Flip a coin with probability p of heads Let X = number of times the coin has to be flipped until we get a heads E[X] = E[X | flip 1 is H]Pr[flip 1 is H] + E[X | flip 1 is T]Pr[flip 1 is T] = p + E[X+1](1-p) = p + E[X](1-p) + (1-p)

LA Phoenix San Diego SF Seattle OVERALL

88.9 94.8 91.7 83.1 85.8 86.7

559 233 232 605 2146 3775

85.6 92.1 85.5 71.3 76.7 89.1

811 5255 448 449 262 7225

Alaska Air beats America West at each airport but America West has a better overall rating!

E[X] = 1/p

CMULand
All CMU SCS students fly off to space and colonize the moon Faced with the problem that there are more men than women, the authorities impose a new rule: When having kids, stop after you have a girl Will the number of new girls be higher than the number of new boys? What if the rule is: stop after having two girls? Heres What You Need to Know

Language of Probability
Events Pr [ A | B ] Independence

Random Variables
Definition Two Views of R.V.s

Expectation
Definition Linearity