Econometric Theory/Ordinary Least Squares

(OLS)

Ordinary Least Squares or OLS is one of the simplest
(if you can call it so) methods of linear regression. The
goal of OLS is to closely t a function with the data.
It does so by minimizing the sum of squared errors from
the data.
Notice that the Sum of Model A is 5 + 10 5 10 = 0
and that the Sum of Model B is 3 3 + 3 3 = 0
Both Models sum to 0 and both are great ts! NO!!
So to account for the signs, whenever we sum errors, we
square the terms rst.

1 Why we Square Errors before

Summing 2 The Model
These two models each have an intercept term , and a
We are not trying to minimize the sum of absolute errors,
slope term (some textbooks use 0 instead of and 1
but rather the sum of squared errors. Lets take a brief
instead of , this is a much better approach once we move
look at our sweater story again.
to multivariate formulas). We can represent an arbitrary
single variable model with the formula: yi = +xi +ui
The y-values are related to the x-values given this for-
mula. We use the subscript i to denote an observation.
So y1 is paired with x1 , y2 with x2 , etc. The ut term is
the error term, which is the dierence between the eect
of xi and the observed value of yi .
Unfortunately, we don't know the values of , or ut .
We have to approximate them. We can do this by us-
ing the ordinary least squares method. The term least
squares means that we are trying to minimize the sum
of squares, or more specically we are trying to minimize
the squared error terms. Since there are two variables that
we need to minimize with respect to ( and ), we have
two equations:
f = u2i = (yi xi )2 f
= 2(yi
Model A xi ) = 0 = 2(yi xi )xi = 0 Call the
f

and . Solving we get:

solutions to these equations
(xi
x)yi
= y x
= (xi
x)2 Where y = yi and
n
x = x n . Computing these results can be left as an
i

exercise.
It is important to know that and are not the same as
and because they are based on a single sample rather
than the entire population. If you took a dierent sample,
you would get dierent values for and . Lets call


and the OLS estimators of and . One of the main
goals of econometrics is to analyze the quality of these
estimators and see under what conditions these are good
estimators and under which conditions they are not.
Once we have and , we can construct two more vari-
ables. The rst is the tted values, or estimates of y:
yi = i The second is the estimates of the error
+ x
Model B terms, which we will call the residuals:

1
2 2 THE MODEL

i = yi yi These two variables will be important later

u
on.
 3

3 Text and image sources, contributors, and licenses

3.1 Text
Econometric Theory/Ordinary Least Squares (OLS) Source: https://en.wikibooks.org/wiki/Econometric_Theory/Ordinary_Least_
Squares_(OLS)?oldid=3132464 Contributors: Ravster, Adrignola, Rob.s.brit, Ssjgotrunk and Anonymous: 5

3.2 Images
File:Sweater_sales2.JPG Source: https://upload.wikimedia.org/wikibooks/en/8/8e/Sweater_sales2.JPG License: Public domain Contrib-
utors: ? Original artist: ?
File:Sweater_sales3.JPG Source: https://upload.wikimedia.org/wikibooks/en/2/28/Sweater_sales3.JPG License: Public domain Contrib-
utors: ? Original artist: ?

3.3 Content license

Creative Commons Attribution-Share Alike 3.0