Sei sulla pagina 1di 6

M. A.

Lokhandwala Page 1 of 6
Coding and Decoding
Coding:
If,
A= set of source alphabet ( source symbols ) ={
1
,

2
, ,
n
},
B= set of code alphabet (code words ) = { c
1,
c
2
, , c
n
},
then no. of code words in B = no. of source symbols in A.

So, coding is defined as a rule which assigns exactly one word from code alphabet to each
source symbol. i.e. a function K from A to the set of words in B. i.e.{ K(
1
)
,
K(
2
), , K(
n
) }=
{ c
1,
c
2
, , c
n
}
Code words: K(
1
)= c
1,
K(
2
)= c
2
, , K(
n
)= c
n
are code words.
Code: Set off all code words is called code.

Binary coding : There are two symbols in binary coding {0,1} i.e. code alphabet employs
two symbols.

Example-1:
If A = { 00, 01, 10, 11}, then
1
=00,
2
=01,
3
=10 and
4
=11.
and B = {000, 011, 101, 110} then,
K(
1
)= 000
,
K(
2
)= 011, K(
3
)= 101
,
K(
4
)= 110.

Note- all code words are distinct.

Example-2:

2-out-of-5 Even Parity Code:
In binary word of length 5, no. of two 1s =
|
|
.
|

\
|
2
5
=10.
each word can be assigned to 10 decimal symbols (source symbols) to form 2-out-of-5
code as shown in fig-1.below.

It is weighted binary code. Weight of five columns = D
4
D
3
D
2
D
1
D
0
=74210.

Source Symbol Code Word Source Symbol Code Word
1 00011 6 01100
2 00101 7 10001
3 00110 8 10010
4 01001 9 10100
5 01010 0 11000

Fig-1 : 2-out-of-5 code

Message 173 has code =

1 7 3
00011 10001 00110 .
Observations :
- No space is left between the code words as space is a code symbol ( e.g. Morse
code has code alphabet B={ , , space} ).
- Encoding/Decoding rule : Weight of five columns = D
4
D
3
D
2
D
1
D
0
=74210. Code
word digits are read as group of five and their weights are multiplied by their
values ( 0 or 1) and products are added to form source symbols. Zero is exception,
it can be decoded as first three consecutive zeros.
- D
0
digit of code word is used for maintaining even parity.

M. A. Lokhandwala Page 2 of 6

Message source : A generator which can generate source symbols continuously.

Example-3 : A decimal source can generate 100 symbols {
1,

2, ,

100
}

with
corresponding codes {00, 01, , 99}. Here, the code alphabet employs 10 symbols 0, 1,
, 9 and it

e.g. message generated may be {
1

100
}= {00010599}={ K(
1
) K(
2
) K(
6
) K(
100
)}.

Coding of source messages : For each coding K of source symbols, we define the coding of
source messages as the coding rule K*, which assigns to each word m
1
m
2
m
3
m
k


in the source
alphabet the word K*( m
1
m
2
m
3
m
k
) = K(m
1
) K(m
2
) . K(m
k
).

e.g. Two digit decimal coding of message word m
1
m
2
m
3
m
4
= a
1

6
a
100
= =00010599.

Uniquely decodable coding : Coding K is said to be uniquely decodable provided arbitrary two
distinct source messages (symbols) have distinct code words. In other words, provided that K* is
one-to-one.

Two digit decimal code above is uniquely decodable as all code words are distinct. Message m
1

m
2
m
3
m
4
= a
1

6
a
100
= 00010599 is uniquely decodable.

In contrast, the following coding is not uniquely decodable,
a 00 b 10 c 101 d 110 e 1001
Because for example, 10110 can be decoded as either bd or cb. Here symbols b, c & e are not
uniquely decodable.

Two important types of codes :
Block codes :
- Def : A code formed using pairwise distinct code words of a certain length n is called a
block code.
- Block codes are very convenient to decode as we know in advance, which code word
corresponds to which source symbol.
- Block codes are efficient when all source symbols appear with equal probability.
- Examples : Octal code. Hexa decimal code, ASCII code, Gray code, etc.

Instantaneous Codes :
- Codes of variable word lengths which are decodable symbols per symbol.
- Def : A coding is said to be instantaneous provided that no code word is prefix of another
code word; i.e. if a source symbol has a code b
1
b
2
.b
n
then no other source symbol has a
code b
1
b
2
.b
n
b
n+1
.b
m
.
- Preferable when probabilities of various source symbols vary substantially.
- Examples : Morse code : code alphabet = { , , space}. Space is used only at the end
of each code word. Decoding is simple: Look for the space to decide that the code is
terminated and then decode.

M. A. Lokhandwala Page 3 of 6
Construction of Instantaneous Code :

Let, Source alphabet = {
1
,

2
, ,
n
},
Length of Corresponding code words = d
1
, d
2,
. . . ., d
n
.
Assumption : d
1
d
2
, . . . ., d
n
.

Choose arbitrarily,
1.
K(
1
) = binary code word of length d
1.

2. K(
2
) = binary code word of length d
2,
But, avoid all those having prefix K(
1
).

The no. of all possible binary word of length d
2
= 2
d2
.

The no. of possible assignment as K(
2
) with those having K(
1
) as prefix = 2
d2-d1

( because, we have d
2
-d
1
digits remaining after the prefix K(
1
) ).

For Instantaneous binary word of length d
2
to exist, at least one binary word of length d
2

must exist which doent contain a code word K(
1
) of length d
1
(i.e. we should have at
least one choice of K(
2
)).

i.e. (2
d2
- 2
d2-d1
) 1.

2
d2
2
d2-d1
+ 1 is always satisfied.

3. K(
3
) = binary code word of length d
3,
But, avoid all those having prefix K(
1
) and K(
2
).
The no. possible word with prefix K(
1
) = 2
d3-d1
,
The no. possible word with prefix K(
2
) = 2
d3-d2

K(
3
), avoiding all those having prefix K(
1
) and K(
2
), can be selected if ( and only if ),
2
d3
2
d3-d1
+ 2
d3-d2
+1.
1 2
-d1
+ 2
-d2
+ 2-
d3
.
4. for K(
n
) = binary code word of length dn,

1 2
-d1
+ 2
-d2
+ . + 2
-dn
.
The above inequality is both necessary and sufficient condition for a construction of
instantaneous codes.

Procedure :
1. Select length ds of code words which satisfies above inequality,
2. Assign code words starting with that having smallest length keeping in mind that the
code word under consideration doesnt have any prefix which is a code word of length
less than its length.

Example 4 : Form the binary instantaneous code for the source alphabet {0,1,2,3}.
Let, d
1
= 1, d
2
= d
3
= d
4
= 2, then, 2
-1
+2
-2
+2
-2
+2
-2

= 1.25 > 1. Code not possible.
d
1
= 1, d
2
= d
3
= 2, d
4
= 3, then, 2
-1
+2
-2
+2
-2
+2
-3
= 1.125 > 1. Code not possible.
d
1
= 1, d
2
= 2, d
3
= d
4
= 3, then, 2
-1
+2
-2
+2
-3
+2
-3
= 1.0 Code is possible.

The instantaneous code is,
0 0 1 10 2 110 3 111.

Note: In above instantaneous coding, we have started with d
min
= d
1
= 1.

M. A. Lokhandwala Page 4 of 6
Example 4 : Form the binary instantaneous code for the source alphabet {0,1,2,3,4,5,6,7}.

n = d
max
= total no. of source alphabets.
Starting with d
min
= d
1
= 1,
d
1
= 1, d
2
= 2, d
3
= 3, d
4
= 4, d
5
= 5, d
6
= 6, d
7
= 7, d
8
= 7
then, 2^-1+2^-2+2^-3+2^-4+2^-5+2^-6+2^-7+2^-7 = 1.0.
Code is possible.
But, the total no. of bits required =
. 35 7 28 ) 1 8 ( ) 1 (
1 8
1
1
1
= + = + = +


=

= i
n
i
i n i

Starting with d
min
= d
1
= 2,
d
1
= 2, d
2
= 2, d
3
= 2, d
4
= 3, d
5
= 4, d
6
= 5, d
7
= 6, d
8
= 6
then, 2^-2+2^-2+2^-2+2^-3+2^-4+2^-5+2^-6+2^-6 = 1.0.
Code is possible.
The total no of bits required = . 30 6 18 6 ] [ ) 1 2 (
min
1
min
min
min
min
= + + = + +

+ =
d n i d
d n
d i
d


Starting with d
min
= 3,
d
1
= 3, d
2
= 3, d
3
= 3, d
4
= 3, d
5
= 3, d
6
= 3, d
7
= 3, d
7
= 3
then, 2^-3+2^-3+2^-3+2^-3+2^-3+2^-3+2^-3+2^-3 = 1.0.
Code is possible.
The total no of bits required = (3)(8) = 24. But, this will become a block code.

Important Note : Any block code of length d 2, satisfies above inequality, and hence can be
thought of as instantaneous code.
e.g. d
1
= d
2
= d
3
= d
4
= 2,
then, 2
-2
+2
-2
+2
-2
+2
-2

= 1.0.
d
1
= d
2
= d
3
= d
4
= d
5
= d
6
= d
7
= d
8
= 3,
then, 2
-3
+2
-3
+2
-3
+2
-3

+2
-3
+2
-3
+2
-3
+2
-3

= 1.0.

Observations :
1. Block codes required the minimum no. of bits for encoding.
2. then, what is the use of instantaneous code ?
3. In above example, minimum bit requirement comes from assumption of equal
probability of occurrence of each source symbol, which is not the case in practice.
4. In practice, source symbols have variable probability of occurrence. One with more
probability of occurrence can be assigned less bits and hence coding redundancy can be
removed. Here, instantaneous coding is employed.

M. A. Lokhandwala Page 5 of 6
Krafts Inequality

Theorem : Given a source alphabet of n symbols and a code alphabet of r symbols, then an
instantaneous code with given lengths d
1
, d
2
, , d
n
of code words exists, whenever the following
Krafts inequality

r
-d1
+ r
-d2
++ r
-dn
1
is satisfies.

Proof :

Let, source alphabet = {a
1
, a
2
,, a
n
},
Then, expected code words = {K(a
1
), K(a
2
),, K(a
n
)}
with corresponding lengths d
1
d
2
d
n
.

We can define the instantaneous coding K as follows:

The code word K(a
s
) for s
th
symbol must,
1. have length d
s
.
2. not have K(a
1
), K(a
2
),, K(a
s-1
) as prefix.

Thus, possible combinations of code words which can be assigned to s
th
symbol are,

1
1
s
i
d d d
i s s
r r



There must be at least one choice, so that code word can be formed.
i.e.
1
1
1
>

s
i
d d d
i s s
r r

multiplying both sides by
s
d
r

,

s i
d
s
i
d
r r

>

1
1
1

i.e.
=

=

= + >
s
i
d
s
i
d d
i i s
r r r
1
1
1
1


1
1
>

s
i
d
i
r

This is Krafts inequality.

This must hold for all symbols. The symbol with highest length will have largest summation.
Hence the inequality to be tested is

1
1
>

n
i
d
i
r

M. A. Lokhandwala Page 6 of 6
McMillan Theorem
Every uniquely decodable coding satisfies Krafts inequality.
Proof :
Let, d
i
= length of code word K(a
i
) for symbol a
i
, i= 1,2,,n.
Let, no. of symbols in code alphabet = r.
Let, source message = a
i1
a
i2
a
im
, which consists of n source symbols is emitted randomly.

Unique decidability means every source message must be uniquely decodable which is true
when no. of source messages of length j must be less than or equal to r
j
.

Therefore, the no. of source messages r
j
and
length of this message = j = d
i1
+d
i2
++d
im
.

Therefore, the no. of sums of the form
j = d
i1
+d
i2
++d
im
is smaller than or equal to r
j
.

For the message to be uniquely decodable, we need to prove that the number

=

=
m
i
d
i
r c
1
is less than or equal to 1.
and for this we will verify that the numbers
m
c
m
are bounded for m = 1, 2,
and this is true for c<1.

Let us compute the power of c.

) ( ) ( ) (
1 ,
) (
1 1
2

=
+
=

= =
m
j i
d d
m
j
d
m
i
d j i j
i
r r r c

and in general,

=
+ + +
=
m
i i i
d d d
m
m
m
i i i
r c
1 ,... ,
) ... (
2 1
2 1

In above equation, the summands are of the form r
-j
with j = d
i1
+d
i2
++d
im
.
j
min
= 1 and j
max
= m(d
max
), where d
max
= max(d
1
, d
2
,, d
m
).

So, above equation can be written by collecting all the summands of the form r
-j
as,

=
max
1
d m
j
j
j
m
r N c
where N
j
= no. of sums of the form r
-
.
j

Now, the no. of sums of the form r
-j

(N
j
) is always less than or equal to r
j
.

max
1 1
max max
1 d m r r c
d m
j
d m
j
j j m
= = s


=


Therefore, max
d
m
c
m
s
, means
m
c
m
is finite. This is true if
. 1 s c

Hence,
1
1
s =

m
i
d
i
r c
is proved.

Potrebbero piacerti anche