EC 636 - Fall 2019 Introduction To Probability Theory: Dr. Nabil Drawil

EC 636 - Fall 2019
Introduction to Probability Theory
Dr. Nabil Drawil

University of Tripoli, Libya
Department of Computer Engineering
Nabil Drawil (University of Tripoli) EC 636 - Fall-2019 1 / 27

Outline
1 Sample Space and Event
2 Probability of An Event
3 Independent Events
Independent Events

Sample Space and Event
Experiment Sample Space and Event
Experiment (E): Any procedure that (1) can be repeated

theoretically an infinite number of times and (2) has a well-defined
set of possible outcomes.
Random experiment: conditions of the experiment only partially
determined the outcome
repeatable
variable outcomes
statistical regularity

Sample Space (S)
The set of all possible outcomes of an E

Some sample spaces
coin toss S = {H, T }, {0, 1}
roll of two dice S = {(1, 1), (1, 2), . . . , (6, 6)}
component lifetime S = {l ∈ (0, ∞)}
daily high temperature S = {(ti , i = 1, 2, . . . )}
noise waveform S = {(n(t), t ∈ <), each n(t) ∈ <}

Sample Space (S), cont.
Consider roll of two dice

die #1 s S
S
6
4
an event
A={dice sum to 7}
3
die #2
1 2 3 4 5 6
This is a product space

S1 = {1, 2, 3, 4, 5, 6}
S2 = {1, 2, 3, 4, 5, 6}
S = S1 × S2 = {(s1 , s2 ) : s1 ∈ S1 , s2 ∈ S2 }
where (s1 , s2 ) are ordered pairs

Types of Sample Space
Finite e.g., S = {0, 1} the binary alphabet

Countable e.g., S = {0, 1, 2, . . . } the non-negative integers
Uncountable (mostly continuous) e.g., S = [0, 1] the closed unit
interval
where finite and countable sample spaces are called discrete sample
space.

Event: a subset of S
Any designated collection of possible outcomes of an experiment

including individual outcomes, the entire sample space, and the null
set , constitutes an event.
An event is said to occur if the outcome is one of the constituent

members of that event.

Elementary Event
Typical form of an elementary event.

Single value s ∈ S
Vector s = (s1 , s2 , . . . , sk ), each si ∈ Si . i.e.,
S = S1 × S2 × · · · × Sn
Sequence s = (si , i = 1, 2, . . . ), each si ∈ Si
Waveform s = (s(t), t ∈ <)

Probability of An Event
n(A)
Relative frequency of an event A = n
n(A)
0≤ n ≤1
If A and B are disjoint events,
then n(A∪B)
n = n(A)
n + n
n(B)
Example
Experiment: toss a fair coin. S = {H, T }, A = {H}
n(A) 1
n −→ 2 as n −→ ∞

Probability of An Event (cont.)
n(A)
Probability of an event A : P(A) = lim n
n−→∞
It indicates the likelihood that event A will occur.
Review: set constructions P1-4

Axioms of Probability
1 0 ≤ P(A) ≤ 1 for all A ⊆ S and P(S) = 1: non negative and

normalized.
2 If A ∩ B = φ, A ⊂ S and B ⊂ S, then P(A ∪ B) = P(A) + P(B).
3 If A1 , A2 , . . . are finite or infinite disjoint events,
n n
!
[ X
P Ai = P(Ai ) finitely additive
i=1 i=1
∞ ∞
!
[ X
P Ai = P(Ai ) countably additive
i=1 i=1

Review: Set Constructions

n
S
Union Ai points in any set of collection
i=1
n
T
intersection Ai points in every set of collection
i=1
disjoint (or mutually exclusive ) collection has Ai ∩ Aj = φ for i 6= j.
n
S
exhaustive collaction has Ai = S.
i=1
Partition is a disjoint exhaustive collection.
A1 A2 A3
A4
An
S
A5
A6

A Special Case
Finite sample space:

S = {s1 , s2 , . . . , sn }
0 ≤ P(si ) ≤ 1
Pn
P(si ) = 1
i=1
S P
If A = si ; then P(A) = P(si )
i∈I i∈I

Basic Properties of probability
1 P(Ā) = 1 − P(A) for A ⊆ S.

2 P(A) ≤ P(B) for A ⊆ B. Bi
3 If B1 , B2 , . . . , Bn form a partition of S. then for any
A ⊆ S.
n
S
X
P(A) = P(A ∩ Bi )
i=1
4 P(A ∪ B) = P(A) + P(B) − P(A ∩ B)

for A, B ⊆ S.
5 Given A1 , A2 , . . . , An ⊆ S
A
 
n
[ n
X X
P Ai  = P(Ai ) − P(Ai ∩ Ai )
1 2
i=1 i=1 1≤i1 <i2 ≤n
X
+ P(Ai ∩ Ai ∩ Ai ) + . . .
1 2 3
1≤i1 <i2 <i3 ≤n
n−1
X
+ (−1) P(Ai ∩ Ai ∩ · · · ∩ Ain )
1 2
1≤i1 <i2 <···<in ≤n

Conditional Probability
A given B or A|B is an event in a new space defined by

B → P(A|B) should be normalized by P(B).
Outcomes in A|B are outcomes in A ∩ B
P(A ∩ B)
P(A|B) = , if P(B) > 0
P(B)
can be viewed as a cutting down of the original sample space
S→B
A → A/B
P(A ∩ B) = P(A|B)P(B) = P(B|A)P(A)

P(A ∩ B ∩ C) = P(A|B ∩ C)P(B ∩ C) = P(A|B ∩ C)P(B|C)P(C)

Example
Family has two children

A = {both are boys}
B = {at least one is a boy}
Child 2
girl
B
A
boy
Child 1
boy girl

Example
Family has two children

A = {both are boys}
B = {at least one is a boy}
Child 2
girl
B
A
boy
Child 1
boy girl
1 3 1 1
P(A) = 4 P(B) = 4 P(A ∩ B) = 4 P(A|B) = P(A ∩ B)/P(B) = 3
A and B are dependent events

Excercises
See problems on conditional

probability (e.g., 12 to 14).

Bayes’ Formula
If B1 , B2 , . . . , form a partition of S.
i.e., Bi ∩ Bj = φ for i 6= j - disjoint
∞
S
Bi = S - exhaustive
i=1
∞ ∞
S S
then A = A ∩ S = A ∩ Bi = (A ∩ Bi )
i=1 i=1

Bayes’ Formula (cont.)
Total probability formula

∞
[ ∞
X ∞
X
P(A) = P( (A ∩ Bi )) = P(A ∩ Bi) = P(A|Bi )P(Bi )
i=1 i=1 i=1
∞
P
i.e., P(A) = P(A|Bi )P(Bi )
i=1

Bayes’ Formula (cont..)
P(A ∩ Bj )
P(Bj |A) =
P(A)
P(A|Bj )P(Bj )
∴ P(Bj |A) = ∞
P
P(A|Bi )P(Bi )
i=1

Example
A satellite is sending a binary code {0, 1}. 70% of the message is 00 s. 80% chance of
a given 0 or 1 being received correctly. Find the probability that 0 was sent if 1 is
received.

Example
received.
80%
70% ‘0’
“0” 62%
20
%
20% error 14%

“1” 38%
30% ‘1’
80%
correct 24%

Example
received.
80%
70% ‘0’
“0” 62%
20
%
20% error 14%

“1” 38%
30% ‘1’
80%
correct 24%
Bi : i was sent (i = 0, 1)
Ai : i is received (i = 0, 1)
P(B0 ) = 0.7 ⇒ P(B1 ) = 0.3
P(A0 |B0 ) = 0.8 ⇒ P(A1 |B0 ) = 0.2
P(A1 |B1 ) = 0.8 ⇒ P(A0 |B1 ) = 0.2

Example
received.
80%
70% ‘0’
“0” 62%
20
%
20% error 14%

“1” 38%
30% ‘1’
80%
correct 24%
Using Bayes’ formula,

Bi : i was sent (i = 0, 1)
Ai : i is received (i = 0, 1)
P(B0 ) = 0.7 ⇒ P(B1 ) = 0.3 P(A1 |B0 )P(B0 )
P(B0 |A1 ) =
P(A0 |B0 ) = 0.8 ⇒ P(A1 |B0 ) = 0.2 P(A1 |B0 )P(B0 ) + P(A1 |B1 )P(B1 )
P(A1 |B1 ) = 0.8 ⇒ P(A0 |B1 ) = 0.2 0.2 × 0.7
= = 0.37
0.2 × 0.7 + 0.8 × 0.3

Independent Events Independent Events
Independent Events
If P(A|B) = P(A) or P(B|A) = P(B)
⇒ P(A ∩ B) = P(A)P(B)
then A and B are independent; otherwise, they are dependent.

Example1: role two dice
Physically unrelated events

A = { die #2 roll > 3} die #1 B
B = { die #1 roll > 3}
6
5 A
P(A) = 18/36 = 1/2 P(B) = 18/36 = 1/2
4
3
P(A ∩ B) = 9/36 = 1/4
2
die #2
∴ P(A ∩ B) = P(A)P(B) ⇒ B and A are 1 2 3 4 5 6
independent.

Example2: role two dice
Physically related events

B = { die #1 roll > 3} die #1 B C
C = { dice sum > 7}
6
5
P(B) = 1/2 P(C) = 15/36 = 5/12
4
3
P(B ∩ C) = 12/36 = 1/3
2
die #2
∴ P(B ∩ C) 6= P(B)P(C) ⇒ B and C are 1 2 3 4 5 6
dependent.

Remarks: Independent versus Disjoint
In fact, disjoint events of nonzero probability must be dependent.
disjoint: P(A ∪ B) = P(A) + P(B) where P(A ∩ B) = 0.

Independent: P(A ∩ B) = P(A)P(B).
Independence is often assumed if events have no obvious
physical connection.

Independence: Collection of Events
Consider events A1 , A2 , . . .

Q T
If they ere independent, then P Ai = P(Ai ), hence
i i
T
P Ai is non-increasing as i increases.
i
Events Ai , for i = 1, 2, . . . are independent if
!
\ Y
P Ai = P(Ai ), for any I = {i1 , i2 , . . . , ir }and any r > 1
i∈I i∈I
Events Ai , for i = 1, 2, . . . , are pairwise independent if

 
n
P(Ai ∩ Aj ) = P(Ai )P(Aj ) for any i 6= j. →   combinations
2

Independence: Collection of Events (cont.)
Note: independent ⇒ pairwise independent.

but pairwise independent 6⇒ independent.
Example
Draw ball from urn containing four balls numbered 1,2,3,4
A1 = {1, 2} P(A1 ) = 1/2 P(A1 ∩ A2 ) = 1/4 P(A1 ∩ A2 ∩ A3 ) = 1/4
A2 = {1, 3} P(A2 ) = 1/2 P(A1 ∩ A3 ) = 1/4 ⇒
A3 = {1, 4} P(A3 ) = 1/2 P(A2 ∩ A3 ) = 1/4
pairwise independent but not independent

EC 636 - Fall 2019 Introduction To Probability Theory: Dr. Nabil Drawil

Caricato da

Informazioni sul documento

Titolo originale

Copyright

Formati disponibili

Condividi questo documento

Condividi o incorpora il documento

Opzioni di condivisione

Hai trovato utile questo documento?

Questo contenuto è inappropriato?

Copyright:

Formati disponibili

EC 636 - Fall 2019 Introduction To Probability Theory: Dr. Nabil Drawil

Caricato da

Copyright:

Formati disponibili

EC 636 - Fall 2019

Introduction to Probability Theory

Dr. Nabil Drawil

Nabil Drawil (University of Tripoli) EC 636 - Fall-2019 1 / 27

1 Sample Space and Event

Nabil Drawil (University of Tripoli) EC 636 - Fall-2019 2 / 27

Experiment Sample Space and Event

Experiment (E): Any procedure that (1) can be repeated

Nabil Drawil (University of Tripoli) EC 636 - Fall-2019 3 / 27

Sample Space (S)

The set of all possible outcomes of an E

Nabil Drawil (University of Tripoli) EC 636 - Fall-2019 4 / 27

Sample Space (S), cont.

Consider roll of two dice

This is a product space

Nabil Drawil (University of Tripoli) EC 636 - Fall-2019 5 / 27

Types of Sample Space

Finite e.g., S = {0, 1} the binary alphabet

Nabil Drawil (University of Tripoli) EC 636 - Fall-2019 6 / 27

Any designated collection of possible outcomes of an experiment

An event is said to occur if the outcome is one of the constituent

Nabil Drawil (University of Tripoli) EC 636 - Fall-2019 7 / 27

Typical form of an elementary event.

Nabil Drawil (University of Tripoli) EC 636 - Fall-2019 8 / 27

Nabil Drawil (University of Tripoli) EC 636 - Fall-2019 9 / 27

Probability of An Event (cont.)

Review: set constructions P1-4

Nabil Drawil (University of Tripoli) EC 636 - Fall-2019 10 / 27

1 0 ≤ P(A) ≤ 1 for all A ⊆ S and P(S) = 1: non negative and

Nabil Drawil (University of Tripoli) EC 636 - Fall-2019 11 / 27

Review: Set Constructions

Nabil Drawil (University of Tripoli) EC 636 - Fall-2019 12 / 27

Finite sample space:

Nabil Drawil (University of Tripoli) EC 636 - Fall-2019 13 / 27

Basic Properties of probability

1 P(Ā) = 1 − P(A) for A ⊆ S.

4 P(A ∪ B) = P(A) + P(B) − P(A ∩ B)

Nabil Drawil (University of Tripoli) EC 636 - Fall-2019 14 / 27

A given B or A|B is an event in a new space defined by

P(A ∩ B) = P(A|B)P(B) = P(B|A)P(A)

Nabil Drawil (University of Tripoli) EC 636 - Fall-2019 15 / 27

Family has two children

Nabil Drawil (University of Tripoli) EC 636 - Fall-2019 16 / 27

Family has two children

A and B are dependent events

Nabil Drawil (University of Tripoli) EC 636 - Fall-2019 16 / 27

See problems on conditional

Nabil Drawil (University of Tripoli) EC 636 - Fall-2019 17 / 27

Nabil Drawil (University of Tripoli) EC 636 - Fall-2019 18 / 27

Bayes’ Formula (cont.)

Total probability formula

Nabil Drawil (University of Tripoli) EC 636 - Fall-2019 19 / 27

Bayes’ Formula (cont..)

Nabil Drawil (University of Tripoli) EC 636 - Fall-2019 20 / 27

Nabil Drawil (University of Tripoli) EC 636 - Fall-2019 21 / 27

20% error 14%

Nabil Drawil (University of Tripoli) EC 636 - Fall-2019 21 / 27

20% error 14%

Nabil Drawil (University of Tripoli) EC 636 - Fall-2019 21 / 27

20% error 14%

Using Bayes’ formula,

Nabil Drawil (University of Tripoli) EC 636 - Fall-2019 21 / 27

If P(A|B) = P(A) or P(B|A) = P(B)