Quantifying Information

(Claude Shannon, 1948)

Given discrete random variable X
x1, x2, , xN
N possible values:
Associated probabilities: p1, p2, , pN
Information received when learning that choice
was xi:

!1$
I(xi ) = log 2 # &
" pi %
1/pi is proportional to the
uncertainty of choice xi.

Information is measured in
bits (binary digits) =
number of 0/1s required
to encode choice(s)

Information Conveyed by Data

Even when data doesnt resolve all the uncertainty

! 1 $
I(data) = log 2 #
&
" pdata %

!
$
1
& = 2 bits
e.g., I(heart) = log 2 #
# 13 &
" 52 %

Common case: Suppose youre faced with N equally

probable choices, and you receive data that
narrows it down to M choices. The probability that
data would be sent is M(1/N) so the amount of
information you have received is

"
%
"N%
1
I(data) = log 2 $
' = log 2 $ ' bits
#M &
# M (1 N ) &

Example: Information Content

Examples:
information in one coin flip:
N= 2

M= 1

Info content= log2(2/1) = 1 bit

card drawn from fresh deck is a heart:

N= 52 M= 13 Info content= log2(52/13) = 2 bits
roll of 2 dice:
N= 36 M= 1

Info content= log2(36/1) = 5.17

.17 bits ???

Probability & Information Content

Information
content

data
a heart
not the Ace of spades
a face card (J, Q, K)
the suicide king

pdata
13/52
51/52
12/52
1/52

log2(1/pdata)
2 bits
0.028 bits
2.115 bits
5.7 bits

Shannons definition for information

content lines up nicely with my
intuition: I get more information
when the data resolves more
uncertainty about the randomly
selected card.