Principles of Econometrics 2016: Exercise Session 1

Bernd Funovits

November 4, 2016

Matrix Algebra and Projections

1. Reminder matrix multiplication: Consider the matrices


A=

(a) Calculate

JA0

(b) Calculate

AJ

and
and

1
2


3
3
, B= 0
1
4

0
1

0 8
0
6 3 , J = 0
1 0
1

of

AB

1
0
0

JB
BJ

(c) How can one interpret the multiplication with the matrix
(d) Calculate

0
1
0

in a) and b)?

and write each line (column) of this matrix as a linear combination of the lines of

(columns

A).

(e) Compute
(f ) For

(AB)

C = (1, 2, 3)

and

B 0 A0

calculate

(AC)

and

C 0 A0

2. Reminder determinant: Consider the matrices


A=
and check that

det(AB) = det(A) det(B)



1
0
, B=
2
1

1
4

1
1

holds.

3. Reminder eigen decomposition: Remember that every quadratic matrix

which satises

AA0 = A0 A

(which holds

in particular for symmetric matrices) can be represented as

A = QDQ0
where

is a diagonal matrix and

have length

1.

is an orthogonal matrix, i.e. its columns are orthogonal to each other and

Consider the matrix


A=

(a) Calculate the eigenvalues of

(b) For both eigenvalues

and

2 ,


0
= 0.
1

A,

i.e. the solutions of

eigenvalue calculate the corresponding eigenvectors, i.e. solve

and the same for


3
.
1


1
det A
0

1
3

1 1
3

3
1 1

   
x
0
=
y
0

and normalize them such that they have lenght one.

(c) Verify that

A=

1
2
1
2

1
2
12

!

4
0


0
2

1
2
1
2

1
2
12

4. Scalar product: We dene for two vectors

x, y RN

that

hx, yi = x0 y
where the prime denotes the transpose of a matrix. Show that
(a)

hx, yi = hy, xi

(b)

hax + by, zi = a hx, zi + b hy, zi

(c) Cauchy-Schwarz inequality: Dene

(d) Triangle inequality:
(e) Pythagoras: If

a, b R and x, y, z RN
p
kxk = hx, xi. Show that |hx, yi| kxk kyk .

for

kx + yk kxk + kyk

x, y RN

are orthogonal, i.e.

hx, yi = 0,

then

kx + yk = kxk + kyk
5. Show that

PN

i=1

xi x0i = X 0 X

where

xi RK1

and

x01

..
.
0
N K

.
X=
xi R
..
.
x0N
1

PX = X (X0 X) X 0 , i.e. the projection

on the space spanned

N
is dened as v R
|

:
v
=
X
. Similary, we introduced


6. In the lecture we introduced the projection matrix

by the columns of the regressor matrix

which

MX = IN X (X 0 X)

the projection matrix

X0

which projects on the orthogonal complement of the space

above.
(a) Verify that

PX

and idempotent
(b) Verify that
(c) Dene

MX are indeed
(PX PX = PX ).

and

P X MX = 0

y = PX y

and that

orthogonal projections, i.e. show that they are symmetric (PX

MX PX = 0.

(the predictor) and

e = MX y

(the residual).

PN
y RN 1 , i.e. the scalar product hy, yi = y 0 y = i=1 yi2 ,
0
0
written as hy, yi = y PX y + y MX y = h
y , yi + he, ei by using the properties of PX and MX


0
N 1
0 1 0
Show that for = (1, . . . 1) R
, IN ( )
is a projection and interprete it.
i. Show that the square of the length of

(d)

0
= PX
)

can be

i. Hint: How does one calculate the mean of a variable?

7. Eigenvalues of projections:

We know that orthogonal projections are symmetric and thus have an eigenvalue

decomposition in an orthogonal basis, i.e.

P = V DV 0
D Rnn is a diagonal
0
0
matrix, i.e. V V = V V = In .
where

matrix with diagonal elements

to an eigenvector

in the eigenvalue decomposition above) satises

projections are idempotent, i.e.

8. The trace of a quadratic matrix

PP = P,

A Rnn

P v = v .

V Rnn is an
P are either zero

and

Show that all eigenvalues of an orthogonal projection

(a) Hint: Remember that an eigenvalue

dii , i {1, . . . , n},

v Rn1 , v 6= 0,

orthogonal
or one.

(one columns of the matrix

Use this together with the fact that orthogonal

to derive an equation for the eigenvalue.

is dened as the sum across the diagonal elements of

tr (A) =

n
X

A,

i.e.

aii .

i=1
(a) Cyclic permutation invariance of trace: Verify that for two matrices

tr(DC)

C R23

and

D R32

tr(CD) =

holds. (of course this property holds more general for all matrices of appropriate dimensions)

A, B Rnn

(b) Linearity of trace: Prove that for

i.
ii.

tr(A) = tr(A)
tr (A + B) = tr (A) + tr (B)

(c) Sum of eigenvalues: Assume that

A Rnn

is symmetric. We know from linear algebra that every symmetric

matrix has an eigenvalue decomposition in an orthogonal basis, i.e.

A = V DV 0
where

D Rnn

is a diagonal matrix with diagonal elements

orthogonal matrix, i.e.

V V = V V = In .

dii , i {1, . . . , n},

and

V Rnn

is an

Show that

tr (A) =

n
X

dii .

i=1
i. Hint: Use that

tr(AB) = tr(BA)

for matrices of appropriate dimensions.

ii. Comment: This property is used very often in the lecture. For example, we use it to obtain

Pn

2
i=1 ei

E (e0 e) =

= E [tr (e e)] = E (tr (ee )) = tr (E (ee ))

(d) Trace of orthogonal projection equals its rank: Use the preceding exercise to show that for an orthogonal
projection

the equation

tr(P ) = rank(P )

holds

i. Hint: Remember that the eigenvalues of orthogonal projections are either zero or one.
ii. Comments: This property is used extensively when we derive the degrees of freedom for
(for obtaining test statistics) and to obtain an unbiased estimator for the variance
9. Positive-(semi)denite matrices: A symmetric matrix
the inequality

v Av > 0

holds. If the inequality only

distributions

A is called positive denite if for all non-zero vectors v RN

0
holds weakly, i.e. v Av 0, then the matrix is called positive

semidenite. Show that

(a) Show that all eigenvalues of a positive semidenite matrix must be non-negative.
i. Hint: Represent

in its eigenvalue decomposition, assume that there is one negative eigenvalue and

show that in this case the inequality

(a) Hint: Write in

does not hold.

b = (X 0 X) X 0 y minimizes the distance between y and the

matrix X . Verify that S(b) is indeed a minimum by showing that
 
S S (b) for all RK .

10. We know from the lecture that

columns of the regressor

v 0 Av 0

 

0 

S = y X
y X

orthogonality arguments.

the term

space spanned by the



h
i
y X = y Xb X Xb

and use

Properties of Random Variables

1.

Rules for expectation and variance of vectors of random variables:

(column) random vectors,

and

Let

real vectors of appropriate dimensions and

and

and

be

m-

and

n-dimensional

real matrices of appropriate

dimensions. The covariance and the variance of random vectors are dened as

h
i
0
C (x, y) = E (x E(x)) (y E(y)) Rmn
V (x) = E [(x E(x)) (x E(x))] Rnn
Repeat the following:
(a) Linearity of expectations: Show that

E (a + Ax) = a + AE (x) , a Rp1 , A Rpm

C (a + Ax, b + By) = AC (x, y) B 0 , a Rp1 , A Rpm , b Rq1 , B Rqn
V (a + Ax) = AV(x)A, , A Rpm

i.
ii.
iii.

(b) Steiner's Lemma:

0

C (x, y) = E (xy 0 ) E (x) E (y)

i.

V (x) = E (xx0 ) E (x) E (x)

ii.
(c) If
2.

and

are uncorrelated, i.e.

C (x, y) = 0,

Independence of random variables:

then

V (x + y) = V(x) + V(y)

Repeat the following denitions and facts (for example from Gourieroux,

Monfort: Statistics and Econometric Models, Volume 2, page 461.)

(a) Two events

A, B

in a probability space are independent if

P (A B) = P (A) P(B).

x and y are independent if for all events A, B in a probability space P ([x A] [y B]) =
P ([x A]) P ([y B]).

(b) Two random variables

(c) If

and

are independent, then

g(x)

(d) If

and

are independent, then

E (g (x) h (y)) = E (g (x)) E (h (y)) for measurable functions g ()

g, h may be the identity or polynomials.

and

h(y)

are independent for measurable functions

g ()

and

h ().
and

h ()

such that the expectations exist. For example,

3.

Conditional expectations: (Hint:

A good summary of these results can be found in Gourieroux, Monfort:

Statistics and Econometric Models, Volume 2, page 461.)

In the lecture we discussed dierent assumptions

We mentioned that 1)E (|X)

that can be imposed in order to prove the Gau-Markov Theorem.

E () = 0

and that 2) if

and

are independent,

E () = 0

implies

E (|X) = 0.

= 0

implies

Here we will repeat some facts

about conditional expectations in order to understand these statements. Without giving a precise denition of a

y and x, the conditional expectation of y with

x, i.e. E (y|x), is a random variable as well and can be understood as the projection (in a certain space
random variables) of y on x, i.e. E (y|x) can be interpreted as the random variable containing the information
y that can be explained by x.

conditional expectation, we state that for two random variables

respect to
of
of

a,
E (a + AX + BY |Z) = a + AE (X|Z) + BE (Y |Z)

(a) The conditional expectation is linear: For scalar

(b) If

and

are independent, it follows that

(c) For any measurable function

(d) Law of iterated expectations:
(e) If

E (y|x) = 0,

then

g ()

matrices

A, B

and random variables

X, Y, Z

we have

E (|X) = E ().

such that the expectation exist, it follows that

E (yg(x)|x) = g(x)E (y|x)

E [E (|X)] = E ()

E (y g(x)) = 0

for any measurable function

g()

such that the expectation exists.