Relationships Involving Joint, Marginal and Conditional

Probabilities





( ) ( | ) ( ) ( | ) ( ) P AB P A B P B P B A P A = =
, then (( ) | ) ( | ) ( | ) If AB P A B C P A C P B C | = = +
( ) ( ) ( | ) ( | )...... (Chain Rule) P ABC P A P B A P C AB =
1 2
1
If , ,......, are a set of mut exc and exh events, then
P(A)= ( | ) ( )
m
m
j j
j
B B B
P A B P B
=

1
( | ) ( )
( | ) ......(Bayes' Rule)
( | ) ( )
j j
j
m
j j
j
P A B P B
P B A
P A B P B
=

Sir Thomas Bayes arrived at the above formula, which is used in many applications,
particularly in interpreting the impact of additional information, A on the probability of
some event Bj.
Discuss Example 2.2 & 2.3 from Sam Shanmugan
Manufacturer Class of Defect
B1
None
B2
Critical
B3
Serious
B4
Minor
B5
Incidental
Total
M1 124 6 3 1 6 140
M2 145 2 4 0 9 160
M3 115 1 2 1 1 120
M4 101 2 0 5 2 110
Total 485 11 9 7 18 530
What is the probability of a component selected at random from the 530 components:-
(a) Being from manufacturer M2 and having no defects? 145/530
(b) Having a critical defect? 11/530
(c) Being from manufacturer M1? 140/530
(d) Having a critical defect given the component is from manufacturer M2? 2/160
(e) Being from manufacturer M1, given it has a critical defect? 6/11
Consider a Binary Channel as
shown on the right:
1. Prob that a 1 was recd ?
2. Prob that a 1 was transmitted,
when a 1 was recd ?
Txn Prob of the Bits
(i.e. source is
assymetric):
1. P(0) = 0.4;
2. Therefore P(1) = 0.6;
Diag on the right indicates
Channel Behaviour
From the Channel Characteristics, we can write the Channel transition
Probabilities as follows:
A 1 is recd whenever any of the
following two events occur:
A 0 is txn and a 1 is recd.
OR
A 1 is txn and a 1 is recd.
0.9
0.05
0.95
0.1
Prob that a 1 was recd ?
( ) ( | ) ( ) ( | ) ( ) P AB P A B P B P B A P A = =
Prob that a 1 was transmitted, when a 1 was recd ?
1
( | ) ( )
( | ) ......(Bayes' Rule)
( | ) ( )
j j
j m
j j
j
P A B P B
P B A
P A B P B
=

What is the Best Decision Rule ?

The following Decision Rules are possible (i.e. what should I assume, when I receive a
1 and what should I assume when I receive a 0:-
D-Rule 1 (Non-Inverting Rule): Whenever I rx a 1, I assume that a 1 was tx by
the Tx and when I rx a 0, I assume that a 0 was tx by the Tx.

D-Rule 2 (Inverting Rule): Whenever I rx a 1, I assume that a 0 was tx by the Tx
and when I rx a 0, I assume that a 1 was tx by the Tx.

D-Rule 3 (Based on the conditional probabilities): I calculate the following
probabilities a priori based on the known Channel Transition Probabilities and
the Source Probabilities.
Obviously, there would be an average probability of error for all the 3 Decision Rules
Calculation of the Bit-error
Probability in this case - Discuss
Bit-Error Prob Surface for Three Decision Rules for P(0) = 0.25
Bit-Error Prob Surface for Three Decision Rules for P(0) = 0.5
Bit-Error Prob Surface for Three Decision Rules for P(0) = 0.75
Bit-Error Prob Surface for Three Decision Rules for P(0) = 1
Relationships Involving Joint, Marginal and Conditional
Probabilities (contd)
The term independence is more obvious, when we consider the latter
equation. The probability of the event A
i
remains the same irrespective of
the outcome of event B
j
i.e. conditioning A
i
on B
j
does not affect the
probability of A
i
.
Also note that the latter equality follows from the first and vice versa.
It should be noted that statistical independence is quite different from
mutual exclusiveness. Indeed, if A
i
and B
j
are mutually exclusive, then
P(A
i
B
j
)=0, by definition.

Two events and are said to be statistically independent
if ( ) ( ) ( ) or when P( | )=P( )
i j
i j i j i j i
A B
P AB P A P B A B A =
Statistical Independence: Suppose that A
i
and B
j
are events associated
with the outcomes of two experiments. Suppose that the occurrence of
A
i
does not affect the occurrence of

B
j
and vice versa, then we say that the
events are statistically independent.

Relationships Involving Joint, Marginal and Conditional
Probabilities (contd)
Statistical Independence and Mutual Exclusiveness: Consider all the
cricket-matches played between India and England. The following table
indicates the venues and the winning team:
Venue Total Played at the
Venue
Won by India
(I)
Won by England
(E)
Bombay (B) 80 40 40
Delhi (D) 100 30 70
Lords (L) 70 10 60
Madras (M) 50 10 40
Total 300 90 210
Hence, event I and event D are statistically
independent.
Hence, event I and event M are statistically
dependent.
Hence, event M and event B are mutually
exclusive.
Introduction to RVs
It is often useful to describe the outcome of an experiment by a number. For
e.g. I can say that a Head corresponds to 1.34 and Tail corresponds to 2.56.
We can also have the following assignments for a random variable D, the
outcome of a Dice-throw.








Incidentally, if we see the probabilities, we find that in this case, the dice is
biased. It will also be noted that the assignment of the real numbers to the
respective outcomes (faces of the dice) of the experiment has been made
arbitrarily. Once the mapping has been done, it does not change.

{ 1.4, Two 0.3, Three 1.8, Four 2.9, Five 8.96, Six 11.4}
( 1.4) 0.1, ( 0.3) 0.2, (1.8) 0.2, ( 2.9) 0.3, (8.96) 0.1, ( 11.4) 0.1,
which we have got directly from the probability of the oc
One
and
P P P P P P

= = = = = =
currence of the various faces.
Introduction to RVs (contd)
The numerical quantity, associated with the random outcomes of an
experiment is loosely called a RV. The concept of the RV has been made for
mathematical convenience.

RV is a function whose domain is the set of outcomes of an experiment and
whose range is R
1
, the real line.




For every outcome of the sample space, the RV assigns a number on the real
line. When we see a RV, we only see a number on the real line but we should
remember that there is an underlying experiment in operation and giving rise
to certain outcomes.

The probability of occurrence of each outcome (of the underlying
experiment) determines the probability of occurrence of each of the mapped
numbers.
One
Two
Six -11.4
-0.3
-1.4
Domain, the
possible
outcomes
Range, the real
line
Types of Random Variables
Discrete RVs They take values which belong to a set of finite size. For
e.g. the RV representing the top face on a dice will take one out of max
possible six values. The sample space would be {-2, 3, 4.5, -9.0, 5.6, 8}.
Similarly, the age of a man (rounded off in years) and the number of
children of a couple would also be discrete RVs.


Continuous RVs A RV which may take infinite number of values
or a continuum of values is called a Continuous RV. For e.g. The height
of men in a town, the voltage at the front end of a receiver or the age of
a man (without any rounding-off)
Cumulative Distribution Function (cdf)
(2.0) ( 11.4) ( 2.9) ( 1.4) ( 0.3) (1.8)
( ) ( ) ( ) ( ) ( )
0.1 0.3 0.1 0.2 0.2
0.9
X
F P P P P P
P Six P Four P One P Two P Three
= + + + +
= + + + +
= + + + +
=
{ 1.4, Two 0.3, Three 1.8, Four 2.9, Five 8.96, Six 11.4}
( 1.4) 0.1, ( 0.3) 0.2, (1.8) 0.2, ( 2.9) 0.3, (8.96) 0.1, ( 11.4) 0.1,
which we have got directly from the probability of the oc
One
and
P P P P P P

= = = = = =
currence of the various faces.
Cumulative Distribution Function (cdf)
Fig
2
{ 1.4, Two 0.3, Three 1.8, Four 2.9, Five 8.96, Six 11.4}
( 1.4) 0.1, ( 0.3) 0.2, (1.8) 0.2, ( 2.9) 0.3, (8.96) 0.1, ( 11.4) 0.1,
which we have got directly from the probability of the oc
One
and
P P P P P P

= = = = = =
currence of the various faces.
(2.0) ( 11.4) ( 2.9) ( 1.4) ( 0.3) (1.8)
( ) ( ) ( ) ( ) ( )
0.1 0.3 0.1 0.2 0.2
0.9
X
F P P P P P
P Six P Four P One P Two P Three
= + + + +
= + + + +
= + + + +
=
Cumulative Distribution Function (cdf)
The nature of the cdf can be seen from fig . At times, the cdf is merely
called the Distribution Function. The cdf has the following properties
( ) 0
X
F =
( ) 1
X
F =
1 2 1 2
( ) ( ) if
X X
F x F x x x s <
1 2 2 1
( ) ( ) ( )
X X
P x X x F x F x < s =
Fig
2
Joint Occurrence of many RVs
Let us consider the case of joint occurrence of two or more RVs. A
good example is the position of a satellite in space. The position P is a
joint RV of the three co-ordinates X, Y and Z, which themselves are
random in nature and are therefore one-dimensional RVs. We may be
interested in the following probabilities:

What is the probability of the satellite of being in the general volume
(2km to 3km, 0 to -1km, 55 to 56km) in space i.e

P(2 X 3, 0 Y -1, 55 Z 56)=? s s s s s s
Joint Occurrence of many RVs
In a country, what is the probability of
a woman having age 30 years and
having two children i.e. P(A=30, C=2)=?


Though the position of the satellite in
space requires three co-ordinates, at
times we may merely be interested in
knowing what is the probability of the
satellite being in such a volume in space
that the Y-co-ord is between 5km and
6km. So here we are interested in
extracting the marginal probability of
the Y-co-ord from the joint probability
of the 3-D position, P. This aspect
would be discussed soon.
Sr Age
(A)
Children
(C)
1 32 2
2 29 1
3 32 3
4 29 1
5 30 1
6 30 2
7 29 2
8 29 1
10,000 32 2
Joint Occurrence of many RVs and their Joint cdf
Having considered joint RVs, we can now talk
about the joint cdf. It is given by:
,
( , ) [( ) ( )]
X Y
F x y P X x Y y = s s
, , ,
, , ,
( , ) 0, ( , ) 0, ( , ) ( ),
( , ) 0, ( , ) 1, ( , ) ( )
X Y X Y X Y Y
X Y X Y X Y X
F F y F y F y
F x F F x F x
= = =
= = =
From these definitions, it can be seen that:
Joint Occurrence of many RVs and their Joint cdf
,
,
1 6
(2, 3) (1,1) (1, 2) (1, 3) (2,1) (2, 2) (2, 3) 6*
12 12
1 6
(1, 6) (1,1) (1, 2) (1, 3) (1, 4) (1, 5) (1, 6) 6*
12 12
C D
C D
F P P P P P P
F P P P P P P
= + + + + + = =
= + + + + + = =
Fig
1
Fig
3
Consider the case when C={1,2,3}, a 3-sided coin and D={1,2,3,4}, a 4-sided Dice.
Discuss the following:
, , ,
, , ,
( , ) 0, ( , ) 0, ( , ) ( ),
( , ) 0, ( , ) 1, ( , ) ( )
X Y X Y X Y Y
X Y X Y X Y X
F F y F y F y
F x F F x F x
= = =
= = =