University of Central Punjab

Faculty of Engineering
Department of Electrical Engineering
Advanced Probability Theory(Code: EE 5783
Dr. Muhammad Sarwar Ehsan

1 Specifying a Random Experiment

A random experiment is an experiment in which the outcome varies in an unpredictable
fashion when the experiment is repeated under the same conditions. A random ex-
periment is specified by stating an experimental procedure and a set of one or more
measurements or observations.

1.1 Examples
1. Select a ball from an urn containing balls numbered 1 to 50. Note the number of
the ball.

2. Toss a coin three times and note the sequence of heads and tails.

3. Toss a coin three times and note the number of heads.

4. Pick a number at random between zero and one.

5. Pick two numbers at random between zero and one.

1.2 The Sample Space

The set of all possible outcomes of a random experiment is called the sample space and
denoted by S or Ω.
An outcome or sample point of a random experiment as a result that cannot be
decomposed into other results denoted by ξ.
The sample space S or Ω can be specified compactly by using set notation. It can be
visualized by drawing tables, diagrams, intervals of the real line, or regions of the plane.
There are two basic ways to specify a set:

1. List all the elements, separated by commas, inside a pair of braces:

A = {0, 1, 2, 3}

1
 y

(0,1)

x x
0 1 (1,0)

Figure 1: Sample space for S4 and S5

2. Give a property that specifies the elements of the set:

A = {x : x is an interger such that 0 ≤ x ≤ 3}

The sample space corresponding to experiments in the above example are

S1 = {1, 2, ..., 50}

S2 = {HHH, HHT, HT H, T HH, T T H, T HT, HT T, T T T }
S3 = {0, 1, 2, 3}
S4 = {x : 0 ≤ x ≤ 1} = [0, 1]
S5 = {x1 , y2 : 0 ≤ x ≤ 1 and 0 ≤ y ≤ 1}

1.3 Discrete Sample Space

We call S a discrete sample space if S is countable; that is, its outcomes can be put into
one-to-one correspondence with the positive integers. S1 , S2 , S3 are examples of discrete
sample spaces.

1.4 Continuous Sample Space

We call S a continuous sample space if S is not countable. S4 , S5 are examples of continu-
ous sample spaces. Continuous sample space is a real line (or its portion), two-dimensional
plane or its portion as shown in the Fig. 1.

1.5 Event
Any subset of a sample space is called event. For example, the three tosses give the same
outcome, i.e., A3 = {HHH, T T T }.

2
 Two events of special interest are:

Certain Event , S, which consists of all outcomes and hence always occurs

Impossible or Null Event , ∅, which contains no outcomes and hence never occurs.

2 Axiomatic Definition of the Probability

Probabilities are numbers assigned to events that indicate how “likely” it is
that the events will occur when an experiment is performed.
Let E be a random experiment with sample space S. Let A is any event in sample
space S then a number P [A], called the probability of A, that satisfies the following
axioms:

Axiom I 0 ≤ P [A]

Axiom II P [S] = 1

Axiom III If A ∩ B = ∅ then P [A ∪ B] = P [A] + P [B]

Axiom III-A If A1 , A2 , ... is a sequence of events such that Ai ∩ Aj = ∅ for all i 6= j,

then "∞ #
[ ∞
X
P Ak = P [Ak ] = A1 + A2 + A3 + · · ·
k=1 k=1

Axioms I, II, and III are enough to deal with experiments with finite sample spaces. In
order to handle experiments with infinite sample spaces, Axiom III needs to be replaced
by Axiom III-A.

3 Corollaries
1. P [Ac ] = 1 − P [A]

2. P [A] ≤ 1

3. P [∅] = 0

4. P [A ∪ B] = P [A] + P [B] − P [A ∩ B]

5. If A ⊂ B then P [A] ≤ P [B]

3
4 Discrete Sample Space
Consider a countable sample space S = {a1 , a2 , ..., an } and let F consist of all subsets of
S. All distinct elementary events are mutually exclusive, so by Corollary 4 the probability
of any event B = {a01 , a02 , ...a0m } is given by

P [B] = P [{a01 , a02 , ...a0m }]

= P [a01 ] + P [a02 ] + ... + P [a0m ]

that is, the probability of an event is equal to the sum of the probabilities of the outcomes
in the event. Thus we conclude that the probability law for a random experiment with a
finite sample space is specified by giving the probabilities of the elementary events.

4.1 Equally Likely

If the sample space has n elements, S = {a1 , a2 , ..., an }, a probability assignment of
particular interest is the case of equally likely outcomes. The probability of the elementary
events is
1
P [a1 ] + P [a2 ] + ... + P [an ] =
n
The probability of any event that consists of k outcomes, say B = {a01 , a02 , ...a0k } is
k
P [B] = P [a01 ] + P [a02 ] + ... + P [a0k ] =
n
Thus if outcomes are equally likely, then the probability of an event is equal to the number
of outcomes in the event divided by the total number of outcomes in the sample space.

4.2 Example # 1
An urn contains 10 identical balls numbered 0, 1, 2, ..., 9. A random experiment involves
selecting a ball from the urn and noting the number of the ball. Find the probability of
the following events:

A = “number of ball selected is odd”

B = “number of ball selected is a multiple of 3” (1)
C = “number of ball selected is less than 5”

and of A ∪ B and A ∪ B ∪ C

4
4.3 Example # 2
A coin is tossed three times and we observe the sequence of heads and tails, then there
are eight possible outcomes S3 = {HHH, HHT, HT H, T HH, T T H, T HT, HT T, T T T }.
If we assume that the outcomes of S3 are equiprobable, then the probability of each of
the eight elementary events is 18 . This probability assignment implies that the probability
3
of obtaining two heads in three tosses is, by Corollary 3, is 8
Now suppose that we toss a coin three times but we count the number of heads in
three tosses instead of observing the sequence of heads and tails. The sample space is now
S4 = {0, 1, 2, 3}. If we assume the outcomes of S4 to be equiprobable, then each of the
elementary events of S4 has probability 41 . This second probability assignment predicts
that the probability of obtaining two heads in three tosses is 14 .
The first probability assignment implies that the probability of two heads in three
tosses is 3/8, and the second probability assignment predicts that the probability is 1/4.
Thus the two assignments are not consistent with each other. As far as the theory is
concerned, either one of the assignments is acceptable. It is up to us to decide which
assignment is more appropriate. Later in the chapter we will see that only the first
assignment is consistent with the assumption that the coin is fair and that the tosses are
“independent”. This assignment correctly predicts the relative frequencies that would be
observed in an actual coin tossing experiment.

4.4 Example # 3
A fair coin is tossed repeatedly until the first heads shows up; the outcome of the experi-
ment is the number of tosses required until the first heads occurs. Find a probability law
for this experiment.
It is conceivable that an arbitrarily large number of tosses will be required until heads
occurs, so the sample space is S = {1, 2, 3, ...}. Suppose the experiment is repeated n
times. Let Nj be the number of trials in which the jth toss results in the first heads. If
n is very large, we expect N1 to be approximately n/2 since the coin is fair. This implies
that a second toss is necessary about n − N1 ≈ n/2 times, and again we expect that
about half of these - that is, n/4 - will result in heads, and so on. Thus for large n, the
relative frequencies are  j
Nj 1
fj ≈ = j = 1, 2, .....
n 2
We therefore conclude that a reasonable probability law for this experiment is
 j
1
P [j tosses till first heads] = j = 1, 2, .....
2

5
5 Continuous Sample Space
Continuous sample spaces arise in experiments in which the outcomes are numbers that
can assume a continuum of values, so we let the sample space S be the entire real line R
(or some interval of the real line).

5.1 Example # 1
Consider the random experiment “pick a number x at random between zero and one”.
The sample space S for this experiment is the unit interval [0, 1], which is uncountably
infinite. If we suppose that all the outcomes S are equally likely to be selected, then we
would guess that the probability that the outcome is in the interval [0, 1/2] is the same
as the probability that the outcome is in the interval [1/2, 1]. We would also guess that
the probability of the outcome being exactly equal to 1/2 would be zero since there are
an uncountably infinite number of equally likely outcomes.

5.2 Example # 2
Consider Experiment E12 , where we picked two numbers x and y at random between
zero and one. The sample space is then the unit square. If we suppose that all pairs
of numbers in the unit square are equally likely to be selected, then it is reasonable
to use a probability assignment in which the probability of any region R inside the
unit square is equal to the area of R. Find the probability of the following events:
A = {x > 0.5}, B = {y > 0.5}, and C = {x > y}.

5.3 Example # 3
A player throws a penny of diameter of 3/4 to a large table ruled with unit squares. What
is the probability that the penny landed completely inside a square, assuming that it fell
on the table.

6 Counting Method (Permutations)

In many experiments with finite sample spaces, the outcomes can be assumed to be
equiprobable. The probability of an event is then the ratio of the number of outcomes in
the event of interest to the total number of outcomes in the sample space (Eq. (2.15)).
The calculation of probabilities reduces to counting the number of outcomes in an event.
In this section, we develop several useful counting (combinatorial) formulas. In general,

6
the number of distinct ordered k-tuples (x1 , ..., xk ) with components xi from a set with
ni distinct elements is

number of distinct ordered k-tuples = n1 n2 ...nk

6.1 Sampling with Replacement and with Ordering

Suppose we choose k objects from a set A that has n distinct objects, with replacement
- that is, after selecting an object and noting its identity in an ordered list, the object is
placed back in the set before the next choice is made. We will refer to the set A as the
“population”. The experiment produces an ordered k-tuple

(x1 , ..., xk )

where xi ∈ A and i = 1, ..., k. With n1 = n2 = ... = nk = n implies that number of

distinct ordered k-tuples= nk .

6.1.1 Example

An urn contains five balls numbered 1 to 5. Suppose we select two balls from the urn with
replacement. How many distinct ordered pairs are possible? What is the probability that
the two draws yield the same number? Equation (2.20) states that the number of ordered
pairs is 52 = 25. Table 2.1 shows the 25 possible pairs. Five of the 25 outcomes have the
two draws yielding the same number; if we suppose that all pairs are equiprobable, then
the probability that the two draws yield the same number is 5/25 = 0.2.

6.2 Sampling without Replacement and with Ordering

Suppose we choose k objects in succession without replacement from a population A of
n distinct objects. Clearly, k ≤ n. The number of possible outcomes in the first draw is
n1 = n; the number of possible outcomes in the second draw is n2 = n − 1, namely all
n objects except the one selected in the first draw; and so on, up to nk = n − (k − 1)
in the final draw. Equation (2.19) then gives number of distinct ordered k-tuples =
n(n − 1)...(n − k + 1). This is also called permutation, i.e.,
n!
(n)k =
(n − k)!

6.2.1 Example # 1

An urn contains five balls numbered 1 to 5. Suppose we select two balls in succession
without replacement. How many distinct ordered pairs are possible? What is the prob-
ability that the first ball has a number larger than that of the second ball? Equation

7
(2.21) states that the number of ordered pairs is 5(4) = 20. The 20 possible ordered pairs
are shown in Table 2.1(b). Ten ordered pairs in Tab. 2.1(b) have the first number larger
than the second number; thus the probability of this event is 10/20 = 1/2.

6.2.2 Example # 3

An urn contains five balls numbered 1, 2, ..., 5. Suppose we draw three balls with re-
placement. What is the probability that all three balls are different? From Eq. (2.20)
there are 53 = 125 possible outcomes, which we will suppose are equiprobable. The
number of these outcomes for which the three draws are different is given by Eq. (2.21):
5(4)(3) = 60. Thus the probability that all three balls are different is 60/125 = 0.48.

6.3 Permutations of n Distinct Objects

Consider sampling without replacement with k = n. This is simply drawing objects from
an urn containing n distinct objects until the urn is empty. Thus, the number of possible
orderings (arrangements, permutations) of n distinct objects is equal to the number of
ordered n-tuples in sampling without replacement with k = n. From Eq. (2.21), we have
number of permutations of n objects = n(n − 1)...(2)(1) , n!. (2.22) We refer to n! as n
factorial.

6.3.1 Example # 1

Find the number of permutations of three distinct objects (1, 2, 3). Equation (2.22) gives
3! = 3(2)(1) = 6. The six permutations are 123, 312, 231, 132, 213, 321.

6.3.2 Example # 2

Suppose that 12 balls are placed at random into 12 cells, where more than 1 ball is allowed
to occupy a cell. What is the probability that all cells are occupied?
The placement of each ball into a cell can be viewed as the selection of a cell number
between 1 and 12. Equation (2.20) implies that there are 12 12 possible placements of the
12 balls in the 12 cells. In order for all cells to be occupied, the first ball selects from any
of the 12 cells, the second ball from the remaining 11 cells, and so on. Thus the number
of placements that occupy all cells is 12!. If we suppose that all 1212 possible placements
are equiprobable, we find that the probability that all cells are occupied is
12!
Probability = = 5.3711 × 10−5
1212

8
This answer is surprising if we reinterpret the question as follows. Given that 12 airplane
crashes occur at random in a year, what is the probability that there is exactly 1 crash
each month? The above result shows that this probability is very small. Thus a model
that assumes that crashes occur randomly in time does not predict that they tend to
occur uniformly over time

6.3.3 Example # 3

Birthdays Problem: We want to determine the probability that in a class size of r two
or more birth dates match. We shall assume that the year consists of 365 days and the
year of birth is inconsequential. Here the 365 days in a year correspond to n cells and
the class size, to r. If the class size is over 365, then two or more birth dates will match
with probability 1. The number of ways r students can be placed into 365 birth dates
with replacement is (365)r . The number of ways they can be placed without replacement
is (365)r , meaning that all the birth dates are different. Thus, the probability of no two
birth dates match is
(365)r
P [no birthdays match] =
(365)r
The probability P(r) that two or more birth dates match is the complement of the pre-
ceding equation and is given by
(365)r
P [r] = 1 −
(365)r

Note that for two or more birthdays to match, there is about 50% chance for 23 students,
71% chance for 30 students, and 95% chance for 46 students. In other words, we need
only 46 students for 2 or more birthdays to match with 95% probability, but to get to
100% probability, we need to go to 366 students!

7 Combinations

7.1 Sampling without Replacement and without Ordering

Suppose we pick k objects from a set of n distinct objects without replacement and that
we record the result without regard to order. (You can imagine putting each selected
object into another jar, so that when the k selections are completed we have no record
of the order in which the selection was done.) We call the resulting subset of k selected
objects a “combination of size k”.

9
7.1.1 Example # 1

A batch of 50 items contains 10 defective items. Suppose 10 items are selected at random
and tested. What is the probability that exactly 5 of the items tested are defective?
The number of ways of selecting 10 items out of a batch of 50 is the number of combina-
tions of size 10 from a set of 50 objects:
 
50 50!
=
10 40!10!

The number of ways of selecting 5 defective and 5 nondefective items from the batch of
50 is the product N1 N2 , where N1 is the number of ways of selecting the 5 items from
the set of 10 defective items, and N 2 is the number of ways of selecting 5 items from the
40 nondefective items. Thus the probability that exactly 5 tested items are defective is
10 40



5 10!40!10!40!
50

5 = = 0.016
10
5!5!35!50!

7.1.2 Example # 2

Find the number of distinct permutations of k white balls and n − k black balls.
This problem is equivalent to the following sampling problem: Put n tokens numbered 1
to n in an urn, where each token represents a position in the arrangement of balls; pick a
combination of k tokens and put the k white balls in the corresponding positions. Each
combination of size k leads to a distinct arrangement (permutation) of k white balls and
n − k black balls. Thus the number of distinct permutations of k white balls and n − k
black balls is Ckn .
As a specific example let n = 4 and k = 2. The number of combinations of size 2 from a
set of four distinct objects is  
4 4!
C24 = = =6
2 2!2!
The 6 distinct permutations with 2 whites (zeros) and 2 blacks (ones) are

1100, 0110, 0011, 1001, 1010, 0101

7.1.3 Multinomial Coefficients

Suppose we partition a set of n distinct objects into J subsets B1 , B2 , ..., BJ , where BJ

is as signed k J elements and k1 + k2 + ... + kJ = n. Then the multinomial coefficient are
given as:
n!
k1 !k2 !...kJ
The binomial coefficient is the J = 2 case of the multinomial coefficient.

10
7.1.4 Example # 1

A six-sided die is tossed 12 times. How many distinct sequences of faces (numbers from
the set {1, 2, 3, 4, 5, 6} have each number appearing exactly twice? What is the probability
of obtaining such a sequence? The number of distinct sequences in which each face of the
die appears exactly twice is the same as the number of partitions of the set {1, 2, ..., 12}
into 6 subsets of size 2, namely
12! 12!
= 6 = 7, 484, 400
2!2!...2! 2
From Eq. (2.20) we have that there are 61 2 possible outcomes in 12 tosses of a die. If
we suppose that all of these have equal probabilities, then the probability of obtaining a
sequence in which each face appears exactly twice is
12!/26 7, 484, 400
= ≈ 3.4 × 10−3 .
612 2, 176, 782, 336

7.1.5 Sampling with Replacement and without Ordering

In the general case the form will involve kx ’s and n − 1/’s. Thus the number of different
ways of picking k objects from a set of n distinct objects with replacement and without
ordering is given by    
n−1+k n−1+k
=
k n−1

7.1.6 Example # 1

An urn contains red, blue, green, yellow, white, and black balls. In how many ways can
we choose

1. 10 balls? This is a direct application of Eq. (3.3.7) with n = 6, k = 10 and the number
of ways is      
6 − 1 + 10 15 15
= = = 3003
10 10 5
2. 20 balls with no more than 3 blue balls? If we draw 2 balls of each color, then we have
drawn 12 balls, leaving us to draw 20−12 = 8 balls to be drawn without restriction.
Hence, applying Eq. (3.2.7) with n = 6 and r = 8, we obtain the number of ways
as      
6−1+8 13 13
= = = 1287
8 8 5
3. 20 balls with no more than 3 blue balls? We first calculate the number of unrestricted
ways of drawing 20 balls as
   
6 − 1 + 20 25
= = 53, 130
21 5

11
 We next draw at least 4 blue balls out of 20 balls. Having drawn the 4 blue balls,
the number of ways to draw the remaining 16 balls is
     
6 − 1 + 16 21 21
= = = 20, 349
26 16 5

Thus the number of ways of drawing 20 balls with no more than 3 blue balls is
53, 130 − 20, 349 = 32, 781.

4. 20 balls with at least 4 red balls, at least 2 green balls, at least 6 white balls, and
no more than 3 brown balls? We first draw the lower bounded balls, namely,
4 + 2 + 6 = 12, before we go to the upper-bounded brown balls. We then draw the
remaining 8 balls in an unrestricted manner, and the number of ways is
   
6−1+8 13
= = 1287
8 5

We have to subtract from this number the number of ways 4 brown balls can be
picked from the 6 colors. This number is
   
6−1+4 9
= = 126
4 4

Hence the number of ways in which all the restrictions will be satisfied is 1287 −
126 = 1161.

12