Documenti di Didattica
Documenti di Professioni
Documenti di Cultura
Jan Bouda
FI MU
1 / 39
Part I
Motivation
2 / 39
Communication system
Encoder
Xn
Yn
Decoder
3 / 39
Communication system
Definition
We define a discrete channel to be a system (X, p(y |x), Y) consisting of
an input alphabet X, output alphabet Y and a probability transition matrix
p(y |x) specifying the probability that we observe the output symbol y Y
provided that we sent x X. The channel is said to be memoryless if the
output distribution depends only on the input distribution and is
conditionally independent of previous channel inputs and outputs.
4 / 39
Channel capacity
Definition
The channel capacity of a discrete memoryless channel is
C = max I (X ; Y ),
(1)
5 / 39
Part II
Examples of channel capacity
6 / 39
7 / 39
a
Jan Bouda (FI MU)
1
p
1p
1p
d
May 12, 2010
8 / 39
Noisy Typewriter
Let us suppose that the input alphabet has k letters (input and
output alphabet are the same here).
Each symbol either remains unchanged (probability 1/2) or it is
received as the next letter (probability 1/2).
If the input has 26 symbols and we use every alternate symbol, we
select 13 symbols that can be transmitted faithfully. Therefore we see
that in this way we may transmit log 13 bits per channel use without
error.
The channel capacity is
C = max I (X ; Y ) = max[H(Y ) H(Y |X )] = max H(Y ) 1 =
X
(2)
= log 26 1 = log 13
since H(Y |X ) = 1 is independent of X .
Jan Bouda (FI MU)
9 / 39
1p
p
p
1
1p
10 / 39
p(x)H(Y |X = x) =
=H(Y )
(3)
1 H(p, 1 p).
Equality is achieved when the input distribution is uniform. Hence, the
information capacity of a binary symmetric channel with error probability p
is
C = 1 H(p, 1 p) bits.
11 / 39
1
Lecture 9 - Channel Capacity
12 / 39
(4)
It remains to determine the maximum of H(Y ). Let us defined E by
E = 0 Y = e and E = 1 otherwise. We use the expansion
H(Y ) = H(Y , E ) = H(E ) + H(Y |E )
(5)
13 / 39
Hence,
C = max H(Y ) H(, 1 ) =
X
(7)
= max(1 )H(, 1 ) = 1 ,
14 / 39
Symmetric channels
Let us consider channel with transition
0.3
p(y |x) = 0.5
0.2
matrix
0.2 0.5
0.3 0.2 ,
0.5 0.3
(8)
with the entry in xth row and y th column giving the probability that y is
received when x is sent. All the rows are permutations of each other and
the same holds for all columns. We say that such a channel is symmetric.
Symmetric channel may be alternatively specified e.g. in the form
Y = X + Z mod c,
where Z is some distribution on integers 0, 1, 2, . . . , c 1, input X has the
same alphabet as Z , and X and Z are independent.
Jan Bouda (FI MU)
15 / 39
Symmetric channels
We can easily find an explicit expression for the channel capacity. Let ~r be
(an arbitrary) row of the transition matrix:
I (X ; Y ) = H(Y ) H(Y |X ) = H(Y ) H(~r ) log Im(Y ) H(~r )
with equality if the output distribution is uniform. We observe that
1
uniform input distribution p(x) = Im(X
) achieves the uniform distribution
of the output since
X
1 X
1
1
p(y ) =
p(y |x)p(x) =
p(y |x) = c
=
,
Im(X )
Im(X )
Im(Y )
x
where c is the sum of entries in a single column of the probability
transition matrix.
Therefore, the channel (8) has capacity
C = max I (X ; Y ) = log 3 H(0.5, 0.3, 0.2)
X
16 / 39
Theorem
For a weakly symmetric channel,
C = log Im(Y ) H(~r ),
where ~r is any row of the transition matrix. It is achieved by the uniform
distribution on the input alphabet.
Jan Bouda (FI MU)
17 / 39
C 0, since I (X ; Y ) 0.
C log Im(Y ).
18 / 39
Part III
Typical Sets and Jointly Typical Sequences
19 / 39
(9)
20 / 39
Theorem (AEP)
If X1 , X2 , . . . are i.i.d. random variables, then for arbitrarily small 0
and sufficiently large n it holds that
1
P log p(X1 , X2 , . . . , Xn ) H(X ) 1
n
This theorem is sometimes presented in the alternative form
1
log p(X1 , X2 , . . . , Xn ) H(X ) in probability.
n
21 / 39
Proof.
The theorem follows directly from the weak law of large numbers, since
1
1X
log p(X1 , X2 , . . . , Xn ) =
log p(Xi )
n
n
i
and
E ( log p(X )) = H(X ).
22 / 39
Typical Set
Definition
(n)
Theorem
(n)
If (x1 , x2 , . . . , xn ) A , then
H(X ) n1 log p(x1 , x2 , . . . , xn ) H(X ) + .
(n)
23 / 39
Typical Set
Proof.
(n)
Property (1) follows directly from the definition of A , property (2) from
the AEP theorem.
To prove property (3) we write
X
X
X
1=
p(~x )
p(~x )
2n(H(X )+) =
~x (Im(X ))n
(n)
~x A
(n)
~x A
= A(n) 2n(H(X )+) .
The last property we get since for sufficiently large n we have
(n)
P(A ) 1 and
X
n(H(X ))
1 P(A(n)
2n(H(X )) = A(n)
.
)
2
(10)
(n)
~x A
Jan Bouda (FI MU)
24 / 39
The set A
where
p(~x , ~y ) =
n
Y
(11)
p(xi , yi ).
(12)
i=1
Jan Bouda (FI MU)
25 / 39
Joint AEP
Theorem (Joint AEP)
n ) be sequences of length n drawn i.i.d according to
Let (X n , YQ
p(~x , ~y ) = ni=1 p(xi , yi ). Then
1
2
(n)
P((X n , Y n ) A ) 1 as n .
(n)
A 2n(H(X ,Y )+) .
n , Y n ) p(~x )p(~y ), then
If (X
n(I (X ;Y )3)
n , Y n ) A(n)
P((X
.
)2
26 / 39
Joint AEP
Joint AEP.
1
n
P log p(X ) H(X ) < .
n
3
(13)
27 / 39
Joint AEP
Proof.
We also have that there exists n2 and n3 such that for all n > n2 (n3 )
1
n
P log p(Y ) H(Y ) <
(14)
n
3
1
n
n
P log p(X , Y ) H(X , Y ) < .
(15)
n
3
Finally, the probability that events (13), (14) and (15) hold
simultaneously, is at most . This gives the required result that the
probability of the complementary event is at least 1 .
28 / 39
Joint AEP
Proof.
2
We calculate
1=
p(x n , y n )
(x n ,y n )
p(x n , y n )
(n)
(x n ,y n )A
n(H(X ,Y )+)
A(n)
2
showing that
(n)
A 2n(H(X ,Y )+) .
29 / 39
Joint AEP
Proof.
3
We have
X
n , Y n ) A(n)
P((X
)=
p(x n )p(y n )
(n)
(x n ,y n )A
30 / 39
Joint AEP
Proof.
(n)
(x n ,y n )A
(n) n(H(X ,Y ))
A 2
and
(n)
A (1 )2n(H(X ,Y ))
31 / 39
Joint AEP
Proof.
By similar arguments as for the upper bound we get
X
n , Y n ) A(n)
p(x n )p(y n )
P((X
)=
(n)
(x n ,y n )A
32 / 39
Part IV
Channel Coding Theorem
33 / 39
34 / 39
35 / 39
Extension of a channel
Definition
The nth extension of the discrete memoryless channel (DMC) is the
channel (Xn , p(y n |x n ), Yn ), where
p(yk |x k , y k1 ) = p(yk |xk ),
k = 1, 2, . . . , n.
If the channel is used without feedback, i.e. the input symbols do not
depend on past output symbols p(xk |x k1 , y k1 ) = p(xk |x k1 ), then the
channel transition function for the nth extension of a discrete memoryless
(!) channel reduces to
p(y n |x n ) =
n
Y
p(yi |xi ).
i=1
36 / 39
Code
Definition
An (M, n) code for the channel (X, p(y |x), Y) is a triplet (M, X n , g )
consisting of
1
2
37 / 39
Error probability
Definition
Probability of an error for the code (M, X n , g ) and the channel
(X, p(y |x), Y) provided the ith index was sent is
X
i = P(g (Y n ) 6= i|X n = X n (i)) =
p(y n |x n (i))I (g (y n ) 6= i),
(16)
yn
Definition
The maximal probability of an error max for an (M, n) code is defined
as
max =
max i
i{1,2,...,M}
38 / 39
Error probability
Definition
(n)
i=1
(n)
39 / 39