Pere Puig Department de Matemtiques Universitat Autnoma de Barcelona ppuig@mat.uab.cat
Models Lineals
The size of the Earth
On 19 March 1791 the meter was theoretically defined by the French Academy of Science as being equal to the ten millionth part of one quarter of the terrestrial meridian. However, the length of the meridian still had to be set up. The persons in charge of this mission were Pierre-Franois Mechain and Jean-Baptiste Delambre. The length of the meridian would be determined by triangulation between Dunkirk and Barcelona.
Models Lineals
Models Lineals
Measurements of the French meridian arc, made in 1795
between Barcelona and Dunkirk:
Using this data set, Legendre in 1805 solved an
overdetermined system of equations by the method of least squares.
Models Lineals
The L2 solutions A general problem: Consider an overdetermined system of equations , where A is a nxm matrix, n>m, and The L2 solution of the system is a vector which minimizes the quadratic difference between b and Ax: where ai is the i-th row of the matrix A. Theorem: Any solution of the mxm system of equations (normal equations) solves the L2 minimization problem.
Models Lineals
Proof: b is a vector in Rn and the columns of A define a vector
space C where Ax is an arbitrary vector in C. The minimum distance b Ax is attained when b-Ax is orthogonal to Ax.
b Ax
C Ax
Models Lineals
Example 1: fitting a simple line (simple linear regression).
Suppose there are n data points {yi, xi}, where i = 1, 2, , n. The goal is to find the equation of the straight line which would provide a "best" (L2) fit for the data points. Here,
Models Lineals
Consequently, the normal equations are,
and the solutions,
They are usually expressed as,
where Sxy is the sample covariance of x and y, and S2x is the sample variance of x.
Models Lineals
Example 2: fitting a simple line without intercept.
Now there are n data points {yi, xi}, where i = 1, 2, , n, and the goal is to find the equation of the line passing through the origin which would provide a "best" (L2) fit.
The normal equation is,
and the solution:
Models Lineals
Example 3: the two variables predicting model.
Now there are n data points {zi, yi, xi}, where i = 1, 2, , n, and the goal is to find the equation of the plane which would provide a "best" (L2) fit for the data points. Here,
Models Lineals
Consider the example of the sales staff :
Models Lineals
Now the predicting model is:
This is different from the L1 solution:
Note that the L2 solution has been expressed as,
but is
always invertible?
Models Lineals
Proposition: Let A a nxm matrix,
is invertible
Proof: Remember that a square matrix B is invertible
Remember also that rank(A) + dim(Ker(A)) = m, and therefore rank(A) =m is equivalent to say that The proof will be finished if we prove that -
Models Lineals
Property: Consider an overdetermined system of equations
, where A is a nxm matrix, n>m, rank(A)=m, with a first column of 1s. Then, the L2 solution , satisfies where indicates the mean of the coefficients of the i-th column of the matrix A, and is the mean of the components of vector b. Proof: (Exercise)
How can it be interpreted?
Models Lineals
An application: the best local polynomial fit
Consider 2k+1 consecutive values of a time series, yt-k,...,yt,..., yt+k. A local polynomial estimator of order p<2k+1 is the minimizer of the least squares function,
Then, choosing u=0 we obtain a predictor of the central
observation yt . The local polynomial approach consists in replacing yt by y*t=0
Models Lineals
The normal equations of this multiple regression
model are determined by,
However we are only interested in the value of 0
because we want to evaluate the polynomial at u=0.
Models Lineals
Notice that,
Moreover, the coefficients (cu) do not depend of yt+u
This is a kind of linear filter.
17
Models Lineals
Example: Finding the local polynomial smoothing with p=2
and k=2 (5 consecutive points). 1 1 1 1 1
-2 -1 0 1 2
4 1 0 1 4
1 -2 4
1 -1 1
1 0 0
1 1 1
Matrix At Matrix A 5 0 10
0 10 0
10 0 34
0,48571 0 -0,14286
At*A
0 -0,14286 0,1 0 0 0,071429 inv(At*A)
Les multiplicacions de matrius es fan amb MMULT
L'inversi d'una matriu es fa amb MINVERSA -0,08571 0,3428571 0,48571 0,34286 -0,08571 -0,2 -0,1 0 0,1 0,2 0,142857 -0,071429 -0,14286 -0,0714 0,142857 inv(At*A)*At
18
1 2 4
Models Lineals
This linear filter is,
yt*=(-3 yt-2 + 12 yt-1 + 17 yt + 12 yt+1 -3 yt+2 )/35 For instance, it can be applied to the following time series,
Lung diseases mortality for women from January 1974 to
December 1979.
19
Models Lineals
It can also be compared with the centered (simple) 5 points
moving average CMA(5). 1200 1000 800 600 400 200
en e74 ju l-7 4 en e75 ju l-7 5 en e76 ju l-7 6 en e77 ju l-7 7 en e78 ju l-7 8 en e79 ju l-7 9
