logit and probit modelspersonal.strath.ac.uk/gary.koop/ec408/ec408_topic_2...random utility...

Logit and Probit Models

January 12, 2012

() Applied Econometrics: Topic 2 January 12, 2012 1 / 36

Introduction

Qualitative choice model is a general term for cases where dependentvariable in a regression is a choice

E.g. the consumer chooses between two brands of a product and youwish to explain how the consumer makes choice

Binary choice model when choice made between 2 things

Multinomial choice when choice is made between more than 2 things

Probit and logit are most popular binary choice models

Readings: Koop (2008) pages 277-287 and Gujarati chapter 8


Example: Choosing to Have an A¤air

Data set is taken from paper: Fair �A theory of extramarital a¤airs�published in Journal of Political Economy in 1978.

Data taken from a magazine survey of N = 601 observations on thefollowing variables:

AFFAIR = 1 if individual has had an a¤air (= 0 otherwise).

MALE = 1 if the individual is male (= 0 otherwise).

YEARS is the number of years the individual has been married.

KIDS = 1 if the individual has children from the marriage (= 0otherwise).

RELIG = 1 if the individual classi�es him or herself as religious (= 0otherwise).

EDUC is the years of schooling completed.

HAPPY = 1 if the individual person views his or her marriage ashappier than average (= 0 otherwise).


Idea is to see if we can explain choice to have an a¤air bycharacteristics of a person

AFFAIR is dependent variable � it is a dummy variable

Other variables are explanatory variables

What kind of model can we build in this case?


The Linear Probability Model

Why not run an OLS regression of AFFAIR on the explanatoryvariables?

This is sometimes done and called the linear probability model

However, it has some disadvantages

Remember classical assumptions

Assumption 4 assumed errors Normally distributed

This implies dependent variable is also Normally distributed

But AFFAIR is dummy variable �cannot have bell-shaped distribution

Linear probability model violates one of the classical assumptions


Can also show linear probability model will in general beheteroskedastic

Fitted value for the dependent variable won�t be zero or one (eventhough these are only values dep var can take on)

Sometimes interpret �tted values from this model as relating toprobability of making choice

E.g. each person has some probability of having an a¤air (e.g. person1 has 10% chance, person 2 has 80% chance, etc.)

We observe only the outcome (e.g. person 1 chooses not to have ana¤air, person 2 chooses to have an a¤air).

This might be a nice interpretation, but linear probability model canimply �tted values that are not between 0 and 1.

Since probabilities must lie between zero, this does not make sense

In short, there are several problems with linear probability model

Use logit or probit instead


Random Utility Framework

Before getting into probit and logit, introduce idea of random utilitymodel (RUM)

RUM uses some economic theory to motivate binary choice models

The set-up:

Individual has to make a choice between two alternatives.

Chooses alternative which maximizes utility

U1i is utility that individual i gets if alternative 1 is chosen

U0i is utility that individual i gets if alternative 1 is chosen

Have data for i = 1, ..,N individuals


Individual chooses alternative which gives highest utility.

Note: U1i � U0i is equivalent to U1i � U0i � 0, so can frame all interms of di¤erence in utility

LetY �i = U1i � U0i

If Y �i � 0 choose alternative 1, if Y �i < 0 choose alternative 0Y �i should depend on an individual�s characteristics (in example:MALE ,YEARS ,KIDS ,RELIG ,EDUC ,HAPPY )

Idea that a variable depends on some characteristics (explanatoryvariable) sounds like a regression

Y �i = α+ β1X1i + β2X2i + ..+ βkXki + εi


Use simple regression to simplify formula (without a¤ecting insights):

Y �i = βXi + εi

Econometrician wishes to estimate this regression model

Problem: Y �i is unobservable.

We will elaborate on this later, but note now:

probit model can be interpreted as this regression, where errors satisfyclassical assumptions.

logit model can also be interpreted as this regression, where errors areassumed satisfy all classical assumptions except one.

Exception: errors are assumed to have a logistic distribution.


Y �i is unobservable, but choice made by individual sheds some lighton Y �i .

If Yi = 1 we know Y �i � 0If Yi = 0 we know Y �i < 0

In words: if individual makes choice 1 it must be case that utility ofalternative 1 is highest

Econometric methods are based on combining:

the regression model for the unobservable Y �iand equations linking Yi and Y �i


Think in terms of the probability of making choice 1

Note: probability of making choice 0 will be one minus this probability.

In terms of probability formula:

Pr (Yi = 1) = Pr (Y �i � 0) = Pr (βXi + εi � 0) = Pr (εi � �βXi )

Remember: last lecture slides introduced some basic ideas aboutprobability distributions

Ideas can be used here as relating to random variable εi

Given an assumption about error distribution, can work outPr (εi � �βXi ).


Probit Model

Probit model assumes εi is Normal

Remember: probabilities obtained using Normal statistical tables (or acomputer software package).

I introduced probability in �rst lecture slides using idea of probabilitydensity function (p.d.f.)

A new concept in probability: the cumulative distribution function(c.d.f.)

De�nition: for any random variable Z and any point z , the c.d.f. is

Pr (Z � z)

If Z is standard Normal: N (0, 1), use notation for c.d.f. of Φ (z)


Probit Model (continued)

Using this notation, probit model has:

Pr (Yi = 1) = Pr (εi � �βXi ) = 1�Φ (�βXi ) = Φ (βXi )

Note 1: last equal sign arises since standard Normal distribution issymmetric about zero.

Note 2: Since Pr (Yi = 0) = 1� Pr (Yi = 1), can also sayPr (Yi = 0) = Φ (�βXi ).

We will use this result shortly to aid in interpreting estimation resultsin the probit model

But �rst: how do we estimate β?


Estimating the Probit Model

Maximum likelihood methods are used to estimate probit model

Remember: likelihood function is joint probability density function forY1, ..,YN , evaluated at the actual observations

If observations are independent of one another (as they will be forprobit due to adoption of classical assumptions for error): likelihoodfunction as:

L (β) = p (Y1, ..,YN ) =N

∏i=1p (Yi )

For probit, p (Yi ) is de�ned by noting that Pr (Yi = 1) = Φ (βXi )and Pr (Yi = 0) = Φ (�βXi ).


Estimating the Probit Model (continued)

Likelihood function is

L (β) =N

∏i=1p (Yi ) =

N

∏i=1

Φ (βXi )Yi Φ (�βXi )

1�Yi

Note: either Yi = 1 or Yi = 0, so, we will either have a Φ (βXi ) or aΦ (�βXi ) in the formula for the likelihood function.

Remember: MLE chooses estimate for β which maximizes L (β)

With the regression model of previous lecture slides MLE was equal toOLS estimator

But with probit there is no simple (analytical) solution to thismaximization problem

Gretl (or other econometrics software package) will �nd probit MLEusing numerical optimization


Logit Model

Logit model uses same RUM framework as probit, but assume εi hasa logistic distribution

Remember from my probit derivations: Pr (Yi = 1) = Φ (βXi )Probability of making a choice turned out to be the c.d.f.

With logistic distribution c.d.f. has a di¤erent form than c.d.f. ofNormal:

Pr (Yi = 1) =exp (βXi )

1+ exp (βXi )

Note: this implies

Pr (Yi = 0) =1

1+ exp (βXi )

Note: with probit there was no formula for Φ (βXi ) �need to usestatistical tables or computer

With logit there is a formula for c.d.f.


Estimating the Logit Model

With probit model we wrote out likelihood function and then saidnumerical optimization necessary to obtain MLE for β

Exactly same strategy for logit

Likelihood function for logit is:

L (β) =N

∏i=1

�exp (βXi )

1+ exp (βXi )

�Yi � 11+ exp (βXi )

�1�YiGretl will �nd MLE

Notation: bβ is MLE (with intercept and many explanatory variablesbα, bβ1, ..., bβk are MLEs)


Hypothesis Testing and Con�dence Intervals in Logit andProbit Models

I will not discuss the statistical theory used to derive these

Econometric software packages like Gretl provide the standardstatistical things I discussed in the previous lecture for regression.

E.g. in a probit or logit model involving many explanatory variables,estimates of α,β1, .., βk will be provided

P-values (to test whether individual coe¢ cients are signi�cant or not)and con�dence intervals will be presented

These statistical things same as in regression

Note: computer packages such as Gretl (see �robust�option) allowfor heteroskedasticity consistent estimation (HCE)

However, interpretation of results from probit or logit models requiressome new ideas


Interpretation of Probit Model Estimates

With regression models, β measures marginal e¤ect of explanatoryvariable on dependent variable.

With probit and logit, dependent variable is (unobserved) utilitywhich complicates interpretation

But you can use Pr (Yi = 1) = Φ (βXi ) result to calculateprobabilities

E.g. in our example can calculate things like: the probability that areligious man with 16 years of schooling, who has been marriedhappily for two years and has children will have an a¤air

That is: with many explanatory variables Φ�bα+ bβ1X1 + ...+ bβkXk�

is probability individual with X1, ..,Xk will make choice

Set X1, ..,Xk to values of interest and you can evaluate this probability


Interpretation of Probit Model Estimates (continued)

Simple regression example:

MLE is bβ, suppose bβ = �0.01X is explanatory variable and you are interested in individuals withvalues of 30, 60 and 120 for X

E.g. in textbook there is an example where Y is choice betweendriving to work and taking public transport and X is number ofminutes the drive is

Pr (Y = 1jX = 30) = Φ�30bβ� = Φ (�0.3) = 0.38

Pr (Y = 1jX = 60) = Φ�60bβ� = Φ (�0.6) = 0.27

Pr (Y = 1jX = 120) = Φ�120bβ� = Φ (�1.2) = 0.12

E.g. there is a 12% probability that a commuter facing a 120 minutedrive will take the car


Marginal E¤ects in the Probit Model

Marginal e¤ects interpretation in simple regression is: �how muchdoes Y change when you change X?" and β is the answer to this.

With qualitative choice models this becomes �how much doesunobserved utility change when you change X?" and β is the answerto this.

Sign of β can provide some information

If β is positive then utility of choice 1 relative to choice zero increaseswhen X increases

But magnitude tells us little

With qualitative choice models, marginal e¤ects of a di¤erent sort arepresented

�how much does the probability of making choice 1 change when youchange X?"

However, β does not answer this question.


Marginal E¤ects in the Probit Model (continued)

For the probit model, the marginal e¤ect of X on the probability ofmaking choice 1 can be shown to be:

φ (βX ) β

φ (.) is the formula for the Normal p.d.f.

Key thing: Gretl calculates this kind of marginal e¤ect

Problem: marginal e¤ect depends on X

E.g. marginal e¤ects for individuals with X = 30 and X = 120 aredi¤erent

Usual solution: evaluate marginal e¤ects at mean values for X


Marginal E¤ects in the Probit Model (continued)

In case with many explanatory variables:

marginal e¤ect of explanatory variable j on the probability of choice 1is:

φ�bα+ bβ1X 1 + ...+ bβkX k� bβj

where X j is average of the j th explanatory variable (i.e.∑Ni=1 XjiN ).


Marginal E¤ects in the Logit Model

Exactly the same issues arise with logit as with probit model (butformulae di¤er)

To repeat: direct interpretation of β is hard

But computer can calculate the marginal e¤ect of X on theprobability of making choice 1 (evaluated at mean of X )

For logit, this marginal e¤ect turns out to be:

exp (βXi )1+ exp (βXi )

11+ exp (βXi )

β

Can also calculate probability that an individual with certaincharacteristics will making choice 1 using result:

Pr (Yi = 1) =exp (βXi )

1+ exp (βXi )


Another Way of Interpreting Results in the Logit Model

De�nition: the odds ratio is the ratio of the probability of making thetwo choices:

Pr (Yi = 1)Pr (Yi = 0)

For logit model, this is:

Odds =Pr (Yi = 1)Pr (Yi = 0)

=

exp(βXi )1+exp(βXi )

11+exp(βXi )

= exp (βXi )

Note that (unlike with probit) this simpli�es nicely


The log of the odds ratio is called the logit

With the logit modelln (Odds) = βXi

The logit is simply βXiHence, β can be interpreted as a marginal e¤ect in terms of the logodds ratio

If you increase X by one unit, the log of the odds ratio will change byβ units


Relationship Between Probit and Logit

As we shall see in a future set of lecture slides, with more than twochoices there are important di¤erences between (multinomial) logitand probit

However, with two choices often they give similar results

Interpretation of β is di¤erent (as outlined in last few slides)

But if you want to make β comparable in logit and probit model thereis an approximate relationship:

Multiply probit�s β by 1.81 and it will be approximately the same aslogit�s β


Example: Choosing to Have an A¤air (continued)Logit Estimation

Table 9.1 presents computer output using the a¤airs data set

Maximum likelihood estimation used

Last columns repeat the exercise using a HCE (labelled �robust�)

Column labelled �odds ratio� is exp�

βj

�E¤ect of a one unit change in the j th explanatory variable on the oddsratio, holding the other explanatory variables constant)


Table 9.1: Logit Results for A¤air DataLogit Odds ratio Logit (robust)

Variable Coe¤P-val forβj = 0

Coe¤ Coe¤P-val forβj = 0

Intercept �1.29 0.07 �� 1.29 0.09MALE 0.25 0.26 1.28 0.25 0.27YEARS 0.05 0.03 1.05 0.05 0.03KIDS 0.44 0.12 1.55 0.44 0.13RELIG �0.89 0.00 0.41 �0.89 0.00EDUC 0.01 0.75 1.01 0.01 0.75HAPPY �0.87 0.00 0.42 �0.87 0.09


Example: Choosing to Have an A¤air (continued)Logit Estimation: Signi�cance of Coe¢ cients

Using the 5% level of signi�cance:

YEARS , RELIG and HAPPY are statistically signi�cant

MALE , KIDS and EDUC are not signi�cant

Note: if we use robust command, then HAPPY is no longersigni�cant (although it signi�cant at 10% level)


Example: Choosing to Have an A¤air (continued)Logit Estimation: Interpreting the Coe¢ cients

Since logit coe¢ cients do not directly measure marginal e¤ects, it ishard to interpret their magnitude.

We can interpret signs on the coe¢ cients.

Coe¢ cients on RELIG and HAPPY are negative which implies:

Individuals who are either religious or happily married are less likely tohave extramarital a¤airs.

Coe¢ cient on YEARS is positive:

Individuals who have been married longer are more likely to havea¤airs.


Example: Choosing to Have an A¤air (continued)Logit Estimation: Interpreting the Odds Ratios

Odds ratios more easy to interpret.

Coe¢ cient estimate for HAPPY is 0.42.

How do we interpret this number?

HAPPY is a dummy variable so a �one unit change�means a changefrom unhappy marriage to a happy one.

�if an individual�s marriages switches from an unhappy one to a happyone (holding other explanatory variables constant), then the oddsratio in favor of having an a¤air will be 42% of what it was before�.


E.g. suppose an individual initially had an odds ratio of 4.

This means Pr (Yi = 1) = 45 and Pr (Yi = 0) =

15

I.e. 80% chance individual will have an a¤air.

If this individual�s marriage becomes happy, odds ratio becomes 42%as high as before.

This means it becomes 4� 0.42 = 1.68.This odds ratio implies there is 63% chance individual will have a¤air.


Example: Choosing to Have an A¤air (continued)Probit Estimation

Table 9.2 presents probit results in same format as for logitOnly di¤erence is for logit we had odds ratio, here we have �MarginalE¤ects�Remember, in probit marginal e¤ects are

φ�bα+ bβ1X 1 + ...+ bβkX k� βj

Interpretation: e¤ect of explanatory variable j on the probability ofmaking choice 1Marginal e¤ect associated with KIDS is 0.07 which means:�if the number of children increases by one, the probability of havingan a¤air increases by 0.07 (holding all other explanatory variablesconstant at their average values)�Being religious lowers a¤air probability by 0.15Results/signi�cance of coe¢ cients very similar to logit so I will notrepeat discussion


Table 9.2: Probit Results for A¤air Data

ProbitMarginalE¤ects

Probit (robust)

Variable Coe¤P-val forβj = 0

Coe¤ Coe¤P-val forβj = 0

Intercept �0.74 0.08 �� 0.74 0.11MALE 0.15 0.23 0.05 0.15 0.24YEARS 0.03 0.03 0.01 0.03 0.02KIDS 0.25 0.12 0.07 0.25 0.13RELIG �0.51 0.00 �0.15 �0.51 0.00EDUC 0.01 0.81 0.00 0.01 0.81HAPPY �0.51 0.00 �0.17 �0.51 0.09


Summary

Probit and logit are most popular models with dummy dependentvariable (a binary choice, choice between 2 alternatives)

Random utility framework is a popular way of motivating logit andprobit

Probit and logit usually estimated by maximum likelihood

Con�dence intervals, P-values for testing whether βj = 0, etc.: sameas in regression

But interpretation of β not straightforward

Usually present Odds ratio (for logit) or marginal e¤ect (for probit)


logit and probit modelspersonal.strath.ac.uk/gary.koop/ec408/ec408_topic_2...random utility...

Documents