Ordered Probit

Econ 674
Purdue University

March 9, 2009

Justin L. Tobias (Purdue)

Ordered Probit

March 9, 2009

1 / 25

In some cases, the variable to be modeled has a natural ordinal interpretation. Some examples include:
1

Education, measured categorically, (e.g. 1 =< HS, 2 = HS, 3 = Some college, etc.). Income, also measured categorically. Survey responses, coded as a degree of opinion (e.g. 1 = Strongly Disagree, 2= Disagree, 3 = Agree, 4 = Strongly Agree. These are in contrast to other choices such as type of insurance or selected mode of transportation, for example, that are not ordered. In this lecture we discuss ordinal choice models, and focus on the ordered probit in particular.

2 3

Justin L. Tobias (Purdue)

Ordered Probit

March 9, 2009

2 / 25

The Ordered Probit Model

Suppose that the variable to be modeled, y takes on J dierent values, which are naturally ordered: 1 2 yi = . , i = 1, 2, . . . , n. . . J As with the probit model, we assume that the observed y is generated by a latent variable y , where

The link between the latent and observed data is given as follows:

Justin L. Tobias (Purdue)

Ordered Probit

March 9, 2009

3 / 25

The Ordered Probit Model

The j are called cutpoints or threshold parameters. They are estimated by the data and help to match the probabilities associated with each discrete outcome. Without any additional structure, the model is not identied. In particular, there are too many cutpoints and some restrictions are required. The most common way to achieve identication is to set:

and retain an intercept parameter in the model. What happens, for example when J = 2 and these restrictions are imposed?
Justin L. Tobias (Purdue) Ordered Probit March 9, 2009 4 / 25

The Ordered Probit Model

The likelihood for the ordered probit is simply the product of the probabilities associated with each discrete outcome:
n

L(, ) =
i=1

Pr(yi = j|xi ),

where = [3 4 J ]. The i th observations contribution to the likelihood is

Justin L. Tobias (Purdue)

Ordered Probit

March 9, 2009

5 / 25

The Ordered Probit Model

Therefore,
n

L(, ) =
i=1

(yi +1 xi ) (yi xi )

and

For purposes of computing the MLE, it can be useful to dene Zij = I (yi = j). Thus, we can write:
n J

L(, ) =
i=1 j=1

zij (log [(j+1 xi ) (j xi )]) .

(Some textbooks present the material this way, though we will not make use of this here).
Justin L. Tobias (Purdue) Ordered Probit March 9, 2009 6 / 25

The Ordered Probit Model

This yields the score for the parameter vector :

and likewise, we obtain the FOC for k , k = 3, 4, , J: Lk (, ) =

i:yi =k

(k xi ) (k+1 xi ) (k xi ) (k xi ) (k xi ) (k1 xi )

+
i:yi =k1

We do not report the Hessian here, as the expressions are rather lengthy. Nonetheless, standard MLE can be applied.
Justin L. Tobias (Purdue) Ordered Probit March 9, 2009 7 / 25

Marginal Eects
To x ideas, consider the case of an ordered probit model with J = 3, in which case we have:

From these, we obtain the category-specic marginal eects:

Justin L. Tobias (Purdue)

Ordered Probit

March 9, 2009

8 / 25

Marginal Eects
What do we learn from this simple model?
1

Like the probit, the marginal eects depend on x. We can evaluate these at sample means, or take a sample average of the marginal eects. Unlike the probit, the signs of the interior marginal eects are unknown and not completely determined by the sign of k . We can, however, sign the eects of the lowest and highest categories based on k . The others, however, can not be known by the reader simply by looking at a table of point estimates.

Justin L. Tobias (Purdue)

Ordered Probit

March 9, 2009

9 / 25

Interpretation
Continue to consider the case with J = 3 and suppose there are no covariates and only an intercept parameter is included. In this case we have Pr(yi = 1) = 1 () Pr(yi = 2) = ( ) [1 ()] = ( ) () Pr(yi = 3) = 1 ( ) What do you think will happen in terms of the MLEs? The likelihood is: L(, ) = [1 ()]
i:yi =1

[( ) ()]
i:yi =2

[1 ( )]
i:yi =3

which reduces to

Justin L. Tobias (Purdue)

Ordered Probit

March 9, 2009

10 / 25

Interpretation

In the last slide, we have dened nj as the number of observations for which yi = j and also note that n1 + n2 + n3 = n. Thus we obtain the log-likelihood L(, ) = n1 log[1()]+n2 log[()()]+n3 log[1()]. The FOC gives:

with Pj denoting the tted probability for category j.

Justin L. Tobias (Purdue)

Ordered Probit

March 9, 2009

11 / 25

Interpretation

Likewise, we get an FOC for : n1 () 1 () + n2 () ( ) ( ) + n3 = 0. ( ) () 1 ( )

Grouping terms, and using our FOC, the FOC can be shown to imply: n2 n1 = . P2 P1

Justin L. Tobias (Purdue)

Ordered Probit

March 9, 2009

12 / 25

interpretation
Noting that P1 + P2 + P3 = 1 and n1 + n2 + n3 = n, these two FOCs can be manipulated to yield:

That is, the parameters will be selected so that the tted values of each category exactly match the observed frequencies of outcomes in that category. Note that this generalizes to any J and any link function! For , for example, we obtain: 1 () = n1 /n or = 1
Justin L. Tobias (Purdue)

n n1 n

.
March 9, 2009 13 / 25

Ordered Probit

0.9

0.8

Pr(y=3) ( )

0.7

0.6

0.5

Pr(y=2) 1 ()

0.4

0.3

0.2

Pr(y=1)
0.1

0 2.5

1.5

0.5

0.5

1.5

2.5
14 / 25

Justin L. Tobias (Purdue)

Ordered Probit

March 9, 2009

Ordered Probit and the EM Algorithm

Suppose that J = 3 and consider the following model: yi = xi + i , and 1 2 yi = 3
i |xi iid

N (0, 1)

if yi 0 if 0 < yi if yi > .

Let

Justin L. Tobias (Purdue)

Ordered Probit

March 9, 2009

15 / 25

Ordered Probit and the EM Algorithm

This reparameterization denes an equivalent model:

where

Note that, in this representation, there are no unknown cutpoints.

Justin L. Tobias (Purdue)

Ordered Probit

March 9, 2009

16 / 25

Ordered Probit and the EM Algorithm

Step 1: E-Step Note that

and thus L(, 2 ; z ) = constant n 1 log( 2 ) 2 (z X ) (z X ). 2 2

The E-step is completed by taking expectations over z | = t , y : E [L(, 2 ; z )] constant n 1 log( 2 ) 2 Ez |=t ,y (z X ) (z X ). 2 2

Justin L. Tobias (Purdue)

Ordered Probit

March 9, 2009

17 / 25

Ordered Probit and the EM Algorithm

Step 2: M-Step: To implement the M step, we must evaluate this expectation and then maximize over and 2 . You will probably recognize the -part of this exercise. It will follow similarly to the probit, where:

with

Justin L. Tobias (Purdue)

Ordered Probit

March 9, 2009

18 / 25

To evaluate this mean, suppose

and we seek

That is,

Justin L. Tobias (Purdue)

Ordered Probit

March 9, 2009

19 / 25

Ordered Probit and the EM Algorithm

Applying this result, we can xi t 2 (t , t , yi ) = x t i x

i t

2 evaluate (t , t , y ) (xi (xt /t ) ) i t /t (xi t )([1xi ]/t + ([1x/t]/t )(xti t /)t ) i t ([1xi ]/t ) + 1([1xitt ]/t )

if yi = 1 if yi = 2 if yi = 3

Thus, the parameters are easily updated.

Justin L. Tobias (Purdue)

Ordered Probit

March 9, 2009

20 / 25

Ordered Probit and the EM Algorithm

As for the updating of 2 , note that it will be obtained as: 1 2 t+1 = Ez |=t ,2 =t2 ,yi (zi xi t+1 )2 . i n
i

Expanding this out, we obtain (dropping the subscript on the expectation for simplicity): 1 2 2 t+1 = E [zi ]2 2(t , t , yi )xi t+1 + (xi t+1 )2 . n
i

Only the rst term in the summation above requires further evaluation. We rst note that
2 2 E [zi ]2 = (xi t )2 + 2xi t E (vi |t , t , yi ) + E (vi2 |t , t , yi ).

We rst recognize that

2 E (vi |t , yi ) = (t , t , yi ) xi t .

Justin L. Tobias (Purdue)

Ordered Probit

March 9, 2009

21 / 25

Ordered Probit and the EM Algorithm

2 As for the E (vi2 |t , t , yi ) term, a little work gives:

2 E (vi2 |t , t , yi ) = 2 1 +

xi t /t (xi t /t ) [1 xi t ]/t ([1 xi t ]/t ) . ([1 xi t ]/t ) (xi t /t )

Justin L. Tobias (Purdue)

Ordered Probit

March 9, 2009

22 / 25

Ordered Probit and the EM Algorithm

So, we have everything we need to implement the EM algorithm. It would proceed as follows:
1 2

Pick some starting values.

2 Calculate (t , t , yi ) using the formula provided. Use this to update t to t+1 . 2 Calculate E (vi2 |t , t , yi ) i using the formula provided, and use it 2 2 2 [together with t+1 and (t , t , yi )] to update t to t+1 .

4 5

Iterate to convergence. Transform back by setting = 1 , = .

Justin L. Tobias (Purdue)

Ordered Probit

March 9, 2009

23 / 25

Ordered Probit and the EM Algorithm

We use n = 139 law school applications from 1985. The dependent variable is the rank of the law school with y = 1 if the rank is less than or equal to 25, y = 2 if the rank is between 25 and 50 and y = 3 if the rank exceeds 50. The independent variables include the applicants LSAT score, GPA and student/faculty ratio (the latter is rather questionable). In the following slides, we present the EM ordered probit estimates (which matched STATAs EXACTLY and were obtained faster!) We report some statistics evaluated at the sample mean of the xs and also setting LSAT and GPA to their maximum sample values.
Justin L. Tobias (Purdue) Ordered Probit March 9, 2009 24 / 25

Ordered Probit and the EM Algorithm

= [49.1 .24 2.73 .01],

= 1.00.

Category y =1 y =2 y =3

Fitted Probability xbar xmax .04 .99 .20 .01 .76 .00

LSAT/xbar .02 .05 -.08

Marginal Eect LSAT/ xmax GPA/ xbar .003 .25 -.003 .59 .000 -.85

Justin L. Tobias (Purdue)

Ordered Probit

March 9, 2009

25 / 25