RS Chapter 1 Random Variables

Chapter 1
Probability Theory:
Introduction

Definitions Algebra
Definitions: Semiring
A collection of sets F is called a semiring if it satisfies:
) F.
If A, B ) F, then A B ) F.
If A, B ) F, then there exists a collection of sets C1, C2, ..., Cn )
F, such that A \ B =U(I >= 1) Ci.
(A\B all elements of A not in B)
Definitions: Algebra
A collection of sets F is called an algebra if it satisfies:
) F.
If 1 ) F, then 1C ) F. (F is closed under complementation)
If 1 ) F & 2 ) F, then 1U 2 ) F. (F is closed under finite
unions).

RS Chapter 1 Random Variables

Definitions: Sigma-algebra
Definition: Sigma-algebra
A sigma-algebra (-algebra or -field) F is a set of subsets of s.t.:
) F.
If ) F, then C ) F. (C = complement of )
If 1, 2,, n, ) F, then U(I >= 1) i ) F.
Note that the set E = {{},{a},{b},{c},{a,,b},{b,c},{a,c},{a,b,c}}
is an algebra and a -algebra.
Sigma algebras are a subset of algebras in the sense that all algebras are algebras, but not vice versa. Algebras only require that
they be closed under pairwise unions while -algebras must be
closed under countably infinite unions.

Sigma-algebra
Theorem:
All -algebras are algebras, and all algebras are semi-rings.
Thus, if we require a set to be a semiring, it is sufficient to show instead
that it is a -algebra or algebra
Sigma algebras can be generated from arbitrary sets. This will be
useful in developing the probability space.
Theorem:
For some set X, the intersection of all -algebras, Ai, containing
X -that is, x )X =>x ) Ai for all i-- is itself a -algebra, denoted (X).
=>This is called the -algebra generated by X.

RS Chapter 1 Random Variables

Sample Space,
Definition: Sample Space
The sample space is the set of all possible unique outcomes of the
experiment at hand.
Example: If we roll a die, = {1; 2; 3; 4; 5; 6}.
In the probability space, the -algebra we use is (), the -algebra
generated by . Thus, take the elements of and generate the
"extended set" consisting of all unions, compliments, compliments of
unions, unions of compliments, etc. Include ; with this "extended set"
and the result is (), which we denote as .
Definition The -algebra generated by , denoted , is the collection
of possible events from the experiment at hand.

(, )
We have an experiment with = {1, 2}. Then,
={{},{1},{2},{1,2}}. Each of the elements of is an event. Think
of events as descriptions of experiment outcomes (: the nothing
occurs event).
Note that -algebras can be defined over the real line as well as over
abstract sets. To develop this notion, we need the concept of a topology.
Definition:
A topology on a set X is a collection of subsets of X satisfying:
1. ;X )
2. is closed under finite intersections.
3. is closed under arbitrary unions.
Any element of a topology is known as an open set.

RS Chapter 1 Random Variables

Borel -algebra
Definition: Borel -algebra (Emile Borel (1871-1956), France.)
The Borel -algebra (or, Borel field,) denoted B, of the topological space
(X; ) is the -algebra generated by the family of open sets. Its
elements are called Borel sets.
Lemma: Let C = {(a; b): a < b}. Then (C) = BR is the Borel field
generated by the family of all open intervals C.
What do elements of BR look like? Take all possible open intervals.
Take their compliments. Take arbitrary unions. Include and R. BR
contains a wide range of intervals including open, closed, and half-open
intervals. It also contains disjoint intervals such as {(2; 7] U (19; 32)}. It
contains (nearly) every possible collection of intervals that are imagined.

Measures
Definition: Measurable Space
A pair (X,) is a measurable space if X is a set and is a nonempty algebra of subsets of X.
A measurable space allows us to define a function that assigns realnumbered values to the abstract elements of .
Definition: Measure
Let (X,) be a measurable space. A set function defined on is called
a measure iff it has the following properties.
1. 0 (A) for any A .
2. () = 0.
3. (-additivity). For any sequence of pairwise disjoint sets {An})
such that Un=1 An ) , we have (Un=1 An ) = n=1 (An )

RS Chapter 1 Random Variables

Measures
Intuition: A measure on a set, S, is a systematic way to assign a number
to each suitable subset of that set, intuitively interpreted as its size. In
this sense, it generalizes the concepts of length, area, volume.

S
0

Examples of measures:
- Counting measure: (S) = number of elements in S.
- Lebesgue measure on R: (S) = conventional length of S.
That is, if S=[a,b] => (S) = [a,b] = b-a.

Measures & Measure Space

Note: A measure may take as its value. Rules:
(1) For any x )R, + x = , x * = if x > 0, x * = if x < 0,
and 0 * = 0;
(2) + = ;
(3) * a = for any a > 0;
(4) or / are not defined.
Definition: Measure Space
A triplet (X,,) is a measure space if (X,) is a measurable space and :
[0; ) is a measure.
If (X) = 1, then is a probability measure, which we usually use
notation P, and the measure space is a probability space.

RS Chapter 1 Random Variables

Lebesgue Measure
Lebesgue measure. There is a unique measure on (R, BR) that satisfies
([a, b]) = b a
for every finite interval [a, b], < a b < . This is called the
Lebesgue measure.
If we restrict to the measurable space ([0, 1], B[0,1]), then is a
probability measure.
Examples:
- Any Cartesian product of the intervals [a,b] x [c,d] is Lebesgue
measurable, and its Lebesgue measure is =(ba)(dc).
- ([set of rational numbers in an interval of R]) = 0.
Note: Not all sets are Lebesgue measurable. See Vitali sets.

Zero Measure
Definition: Measure Zero
A (-)measurable set E is said to have (-)measure zero if (E) = 0.
Examples: The singleton points in Rn, and lines and curves in Rn, n 2.
By countable additivity, any countable set in Rn has measure zero.
A particular property is said to hold almost everywhere if the set of
points for which the property fails to hold is a set of measure zero.
Example: a function vanishes almost everywhere; f = g almost
everywhere.
Note: Integrating anything over a set of measure zero produces zero.
Changing a function on a set of measure zero does not affect the value
of its integral.

RS Chapter 1 Random Variables

Measure: Properties
Let (, , ) be a measure space. Then, has the following
properties:
(i) (Monotonicity). If A B, then (A) (B).
(ii) (Subadditivity). For any sequence A1,A2, ...,
(Ui=1 Ai ) i=1 (Ai)
(iii) (Continuity). If A1A2A3 .... (or A1A2A3 ... and (A1)
< ), then
(lim n An ) = lim n (An ),
where
lim n An = Ui=1 Ai
(or i=1 Ai)

Probability Space
A measure space (, , ) is called finite if () is a finite real number
(not ). A measure is called -finite if can be decomposed into a
countable union of measurable sets of finite measure.
For example, the real numbers with the Lebesgue measure are -finite
but not finite.
Definition: Probability Space
A measure space is a probability space if ()=1. In this case, is a
probability measure, which we denote P.
Let P be a probability measure. The cumulative distribution function
(c.d.f.) of P is defined as:
F(x) = P ((, x]) , x ) R

RS Chapter 1 Random Variables

Product Space
Definition: Product space
Let i, i ) I, be sets, where I = {1, ..., k}, k is finite or
Define the product space as
i)I i = 1 n n n k = {(a1, ..., ak) : ai ) i, i ) I}
Example: R R = R2, R R R = R3
Let (i,

i), i ) I, be measurable spaces

i)I i is not necessarily a -field
(i)I i) is called the product -field on the product space i)Ii
(i)I i, (i)I i)) is denoted by i)I (i,i)
Example: i=1,...k(R, B) = (Rk, Bk).

Product Measure
Definition: Product measure
Consider a rectangle [a1,b1] [a2,b2] R2. The product measure of the
usual area [a1,b1][a2,b2]:
(b1 a1)(b2 a2) = ([a1, b1]) ([a2, b2])
Q: Is ([a1, b1]) ([a2, b2]) the same as the value of a measure defined
on the product -field?
A measure on (, ) is said to be -finite if and only if there exists
a sequence A1,A2, ..., such that KAi = and (Ai) < for all i
Any finite measure (such as a probability measure) is clearly -finite.
The Lebesgue measure on R is -finite, since R = KAn with An = (n,
n), n = 1, 2, ...
The counting measure in is -finite if and only if is countable.

RS Chapter 1 Random Variables

Product Measure & Joint CDF

Theorem: Product measure theorem
Let (i, i, i), i = 1, ..., k, be measure spaces with -finite measures,
where k 2 is an integer. Then there exists a unique -finite measure
on the product -field (A1 ... Ak), called the product measure and
denoted by
1 ... k(1 ... k)= (A1) ... (Ak)
for all Ai)i, i = 1, ..., k.
Let P be a probability measure on (Rk, Bk). The c.d.f. (or joint c.d.f.)
of P is defined by
F(x1 ,....,xk) = P((, x1] ... (, xk]),
xi ) R
There is a one-to-one correspondence between probability measures
and joint c.d.f.s on Rk.

Product Measure & Marginal CDF

If F(x1 ,....,xk) is a joint c.d.f., then
Fi(x) = lim xj,j=1,...,i1,i+1,...,k F(x1,..,xk)
is a cdf and is called the ith marginal cdf.
Marginal cdfs are determined by their joint c.d.f. But, a joint cdf
cannot be determined by k marginal cdfs.
If F(x1,..,xk) = F(x1) ... F(xk), then, the probability measure
corresponding to F is the product measure P1... Pk with Pi being the
probability measure corresponding to Fi

RS Chapter 1 Random Variables

Measurable Function
Definition: Inverse function
Let f be a function from to (often = Rk)
Let the inverse image of B under f:
f1(B) = {f ) B} = { ) : f())B}.
Useful properties:
f1(Bc ) = (f1(B))c for any B ;
f1(KBi) = Kf1(Bi) ) for any Bi , i = 1, 2, ...
Note: The inverse function f1 need not exist for f1(B) to be defined.
Definition: Let (, ) and (, G) be measurable spaces and f a
function from to . The function f is called a measurable function from
(, ) to (, G) if and only if f1(G).

Measurable Function
If f is measurable from (,) to (,G) then f1(G) is a sub--field of
. It is called the -field generated by f and is denoted by (f).
If f is measurable from (,) to (R,B), it is called a Borel function or a
random variable (RV).
A random variable is a convenient way to express the elements of
as numbers rather than abstract elements of sets.

RS Chapter 1 Random Variables

Measurable Function
Example: Indicator function for A .
IA() = 1
if ) A
0
if )AC
For any B R
IA1(B)
=&
if 0 not in B, 1 not in B
= A if 0 not in B, 1 ) B
= Ac if 0 ) B, 1 not in B
= if 0 ) B, 1 ) B
Then, (IA) = {&,A,Ac, } and IA is Borel if and only if A ) .
(f) is much simpler than .
Note: We express the elements of as numbers rather than abstract
elements of sets.

Measurable Function - Properties

Theorems: Let (, ) be a measurable space.
(i) f is a RV if and only if f -1(a,) ) for all a ) R.
(ii) If f and g are RVs, then so are fg and af + bg, where a and b ) R;
also, f/g is a RV provided g() 0 for any ) .
(iii) If f1, f2, ... are RVs, then so are supn fn, infn fn, limsupn fn, and liminfn
fn. Furthermore, the set
A = {): limn fn() exists }
is an event and the function
h() = limn fn()
)A
= f1()
)Ac
is a RV.

RS Chapter 1 Random Variables

Measurable Function - Properties

Theorems: Let (, ) be a measurable space.
(iv) (Closed under composition) Suppose that f is measurable from (,
) to (, G) and g is measurable from (, G) to (,H). Then, the
composite function g f is measurable from (, ) to (,H).
(v) Let be a Borel set in Rp. If f is a continuous function from to
Rp, then f is measurable.

Distribution
Definition
Let (,,) be a measure space and f be a measurable function from
(, ) to (, G). The induced measure by f, denoted by f 1, is a
measure on G defined as
f 1(B) = (f ) B) = ( f 1(B)),
B)G
If = P is a probability measure and X is a random variable or a
random vector, then P X1 is called the distribution (or the law) of X
and is denoted by PX.
The cdf of PX is also called the cdf (or joint cdf) of X and is denoted
by FX.

RS Chapter 1 Random Variables

Probability Space Definition and Axioms

Kolmogorov's axioms
Kolmogorov defined a list of axioms for a probability measure.
Let P: E [0; 1] be our probability measure and E be some -algebra
(events) generated by X.
Axiom 1: P[A] 1 for all A ) E
Axiom 2. P[X] = 1
Axiom 3. P[A1UA2U...UAn] = P[A1]+P[A2]+:::+P[An], where
{A1;A2; .... ;An} are disjoint sets in E.

Probability Space Properties of P

The three Kolmogorovs basic axioms imply the following results:
Theorem: P[AiC ] = 1 - P[Ai].
Theorem: P[] = 0
Theorem: P[Ai] ) [0,1].
Theorem: P[BAC] = P[B] - P[AB]
Theorem: P[AUB] = P[A] + P[B] - P[AB]
Theorem: A is in B => P[A] P[B]
Theorem: A = B => P[A] = P[B]
Theorem: P[A] = i=1P[A Ci] where {C1; C2; ..} forms a partition of E.
Theorem (Boole's Inequality, aka "Countable Subadditivity"):
P[Ui=1Ai] i=1P[Ai] for any set of sets {A1;A2; ::}

RS Chapter 1 Random Variables

Probability Space - (, , P)
Now, we have all the tools required to establish that (, , P) is a
probability space.
Theorem:
Let be the sample space of outcomes of an experiment, be the algebra of events generated from , and P: [0, ) be a probability
measure that assigns a nonnegative real number to each event in . The
space (, , P) satisfies the definition of a probability space.
Remark: The sample space is the list of all possible outcomes. Events are
groupings of these outcomes. The -algebra is the collection of all
possible events. To each of these possible events, we assign some "size"
using the probability measure P.

Probability Space
Example: Consider the tossing of two fair coins. The sample space is
{HH; HT; TH; TT}.
Possible events:
- The coins have different sides showing: {HT; TH}.
- At least one head: {HH;HT; TH}.
- First coin shows heads or second coin shows tails: {HH, HT, TT}
The sigma algebra generated from the sample space is the collection of
all possible such events: [, {H,H}, {HT}, {TH}, {TT}, {HH,HT},
{HH,TH}, {HT,TH}, {TH,TT}, {HH,HT,TH}, {HH,HT,TT},
{HH,TH,TT}, {HT,TH,TT}, {HH,HT,TH,TT}]
The probability measure P assigns a number from 0 to 1 to each of those
events in the sigma algebra.

RS Chapter 1 Random Variables

Probability Space
Example (continuation):
The probability measure P assigns a number from 0 to 1 to each of those
events in the sigma algebra. With a fair coin, we can assign the following
probabilities to the events in the above sigma algebra:
{0; 1/4; 1/4; 1/4; 1/4; 1/2; 1/2; 1/2; 1/2; 1/2; 1/2; 3/4; 3/4; 3/4; 3/4;
1}

But, we do not have to use these fair values. We may believe the coin is
biased so that heads appears 3/4 of the time. Then the following values
for P would be appropriate:
{0; 9/16; 3/16; 3/16; 1/16; 3/4; 3/4; 5/8; 3/8; 1/4; 1/4; 15/16; 13/16;
13/16; 7/16; 1}

Probability Space
As long as the values of the probability measure are consistent with
Kolmogorov's axioms and the consequences of those axioms, then we
consider the probabilities to be mathematically acceptable, even if they
are not reasonable for the given experiment.
Philosophical comment: Can the probability values assigned be
considered reasonable as long as they're mathematically acceptable?

RS Chapter 1 Random Variables

Random Variables Revisited

A random variable is a convenient way to express the elements of as
numbers rather than abstract elements of sets.
Definition: Measurable function
Let AX; AY be nonempty families of subsets of X and Y, respectively. A
function f: X Y is (AX;AY )-measurable if f -1 (A) AX for all A AY.
Definition: Random Variable
A random variable X is a measurable function from the probability space
(, ,P) into the probability space (, AX, PX), where in R is the
range of X (which is a subset of the real line) AX is a Borel field of X,
and PX is the probability measure on induced by X.
Specifically, X: ,.