Sei sulla pagina 1di 20

Models Lineals

2-The L2 models

Pere Puig
Department de Matemtiques
Universitat Autnoma de Barcelona
ppuig@mat.uab.cat

Models Lineals

The size of the Earth


On 19 March 1791 the meter
was theoretically defined by
the French Academy of Science
as being equal to the ten
millionth part of one quarter
of the terrestrial meridian.
However, the length of the
meridian still had to be set up.
The persons in charge of this
mission were Pierre-Franois
Mechain and Jean-Baptiste
Delambre. The length of the
meridian would be determined
by triangulation between
Dunkirk and Barcelona.

Models Lineals

Models Lineals

Measurements of the French meridian arc, made in 1795


between Barcelona and Dunkirk:

Using this data set, Legendre in 1805 solved an


overdetermined system of equations by the method of least
squares.

Models Lineals

The L2 solutions
A general problem: Consider an overdetermined system of
equations
, where A is a nxm matrix, n>m, and
The L2 solution of the system is a vector
which minimizes
the quadratic difference between b and Ax:
where ai is the i-th row of the matrix A.
Theorem: Any solution of the mxm system of equations
(normal equations)
solves the L2 minimization problem.

Models Lineals

Proof: b is a vector in Rn and the columns of A define a vector


space C where Ax is an arbitrary vector in C. The minimum
distance b Ax is attained when b-Ax is orthogonal to Ax.

b Ax

C
Ax

Models Lineals

Example 1: fitting a simple line (simple linear regression).


Suppose there are n data points {yi, xi}, where i = 1, 2, , n.
The goal is to find the equation of the straight line
which would provide a "best" (L2) fit for the data points.
Here,

Models Lineals

Consequently, the normal equations are,

and the solutions,

They are usually expressed as,


where Sxy is the sample covariance of x and y, and S2x is the
sample variance of x.

Models Lineals

Example 2: fitting a simple line without intercept.


Now there are n data points {yi, xi}, where i = 1, 2, , n, and
the goal is to find the equation of the line passing through the
origin
which would provide a "best" (L2) fit.

The normal equation is,

and the solution:

Models Lineals

Example 3: the two variables predicting model.


Now there are n data points {zi, yi, xi}, where i = 1, 2, , n, and
the goal is to find the equation of the plane
which would provide a "best" (L2) fit for the data points.
Here,

Models Lineals

Consider the example of the sales staff :

Models Lineals

Now the predicting model is:

This is different from the L1 solution:

Note that the L2 solution has been expressed as,

but is

always invertible?

Models Lineals

Proposition: Let A a nxm matrix,


is invertible

Proof: Remember that a square matrix B is invertible


Remember also that rank(A) + dim(Ker(A)) = m, and therefore
rank(A) =m is equivalent to say that
The proof will be finished if we prove that
-

Models Lineals

Property: Consider an overdetermined system of equations


, where A is a nxm matrix, n>m, rank(A)=m, with a first column of
1s. Then, the L2 solution
, satisfies
where indicates the mean of the coefficients of the i-th column
of the matrix A, and is the mean of the components of vector b.
Proof: (Exercise)

How can it be interpreted?

Models Lineals

An application: the best local polynomial fit


Consider 2k+1 consecutive values of a time series, yt-k,...,yt,..., yt+k.
A local polynomial estimator of order p<2k+1 is the minimizer of
the least squares function,

Then, choosing u=0 we obtain a predictor of the central


observation yt . The local polynomial approach consists in
replacing yt by y*t=0

Models Lineals

The normal equations of this multiple regression


model are determined by,

However we are only interested in the value of 0


because we want to evaluate the polynomial at
u=0.

Models Lineals

Notice that,

Moreover, the coefficients (cu) do not depend of yt+u

This is a kind of linear filter.

17

Models Lineals

Example: Finding the local polynomial smoothing with p=2


and k=2 (5 consecutive points).
1
1
1
1
1

-2
-1
0
1
2

4
1
0
1
4

1
-2
4

1
-1
1

1
0
0

1
1
1

Matrix At
Matrix A
5
0
10

0
10
0

10
0
34

0,48571
0
-0,14286

At*A

0 -0,14286
0,1
0
0 0,071429
inv(At*A)

Les multiplicacions de matrius es fan amb MMULT


L'inversi d'una matriu es fa amb MINVERSA
-0,08571 0,3428571 0,48571 0,34286 -0,08571
-0,2
-0,1
0
0,1
0,2
0,142857 -0,071429 -0,14286 -0,0714 0,142857
inv(At*A)*At

18

1
2
4

Models Lineals

This linear filter is,


yt*=(-3 yt-2 + 12 yt-1 + 17 yt + 12 yt+1 -3 yt+2 )/35
For instance, it can be applied to the following time
series,

Lung diseases mortality for women from January 1974 to


December 1979.

19

Models Lineals

It can also be compared with the centered (simple) 5 points


moving average CMA(5).
1200
1000
800
600
400
200

en
e74
ju
l-7
4
en
e75
ju
l-7
5
en
e76
ju
l-7
6
en
e77
ju
l-7
7
en
e78
ju
l-7
8
en
e79
ju
l-7
9

Data

CMA(5)

20

LP2(5)