Class 4

Ordinary Least Squares

SKEMA Ph.D programme2010-2011

Lionel Nesta

Observatoire Français des Conjonctures Economiques

Lionel.nesta@ofce.sciences-po.fr

Introduction to Regression Ideally, the social scientist is interested not only in knowing the

intensity of a relationship, but also in quantifying the magnitude

of a variation of one variable associated with the variation of

one unit of another variable.

Regression analysis is a technique that examines the relation

of a dependent variable to independent or explanatory

variables.

Simple regression y = f(X)

Multiple regression y = f(X,Z)

Let us start with simple regressions

Scatter Plot of Fertilizer and Production

Scatter Plot of Fertilizer and Production

Scatter Plot of Fertilizer and Production

iPr ediction Y

i iError Y Y

Scatter Plot of Fertilizer and Production

Scatter Plot of Fertilizer and Production

Objective of Regression It is time to ask: “What is a good fit?”

“A good fit is what makes the error small”

“The best fit is what makes the error smallest”

Three candidates

1. To minimize the sum of all errors

2. To minimize the sum of absolute values of errors

3. To minimize the sum of squared errors

To minimize the sum of all errors

1

minn

i ii

y y

X

Y

–

–+

X

Y

– ++

Problem of sign

X

Y

+3

To minimize the sum of absolute values of errors

1

minn

i ii

y y

X

Y

–1

–1+2

Problem of middle point

To minimize the sum of squared errors

2

1

minn

i ii

y y

X

Y

–

–+

Solve both problems

22

1 1

min minn n

i ii i

y y

ε

ε²

Overcomes the sign problem

Goes through the middle point

Squaring emphasizes large errors

Easily Manageable

Has a unique minimum

Has a unique – and best - solution

To minimize the sum of squared errors

Scatter Plot of Fertilizer and Production

Scatter Plot of R&D and Patents (log)

Scatter Plot of R&D and Patents (log)

Scatter Plot of R&D and Patents (log)

Scatter Plot of R&D and Patents (log)

The Simple Regression Model

( )i i i

i i

y x

E y x

yi Dependent variable (to be explained)

xi Independent variable (explanatory)

α First parameter of interest

Second parameter of interest

εi Error term

The Simple Regression Model

iiy x

.

and are estimates of

the true - but unkown - and

2

1

minn

i ii

y y

ε

ε²

2 2

1 1

2

1

2

1

min min

0

0

n n

i i i ii i

n

i

n

i

y y y x

To minimize the sum of squared errors

2

1

minn

i ii

y y

ε

ε²

2

i i

i

y y x x

x x

y x

To minimize the sum of squared errors

Application to SKEMA_BIO Data using Excel

lnpat_assets lnrd_assetsNumerator Beta_Hat

Denominator Beta_Hat

-12.77 -2.28 -0.61 0.01 -0.01 0.00-12.51 -2.24 -0.35 0.05 -0.02 0.00-12.74 -2.20 -0.58 0.09 -0.05 0.01-12.52 -2.31 -0.36 -0.02 0.01 0.00-12.12 -2.25 0.04 0.04 0.00 0.00-12.53 -2.26 -0.37 0.03 -0.01 0.00-12.09 -2.25 0.07 0.04 0.00 0.00

Mean of y Mean of x Sum Sum-12.16 -2.29 448.75 256.55

Alpha_hat -8.148

Beta_hat 1.749

Deviation to the mean

Application to SKEMA_BIO Data using Excel

lnpat_assets lnrd_assetsNumerator Beta_Hat

Denominator Beta_Hat

-12.77 -2.28 -0.61 0.01 -0.01 0.00-12.51 -2.24 -0.35 0.05 -0.02 0.00-12.74 -2.20 -0.58 0.09 -0.05 0.01-12.52 -2.31 -0.36 -0.02 0.01 0.00-12.12 -2.25 0.04 0.04 0.00 0.00-12.53 -2.26 -0.37 0.03 -0.01 0.00-12.09 -2.25 0.07 0.04 0.00 0.00

Mean of y Mean of x Sum Sum-12.16 -2.29 448.75 256.55

Alpha_hat -8.148

Beta_hat 1.749

Deviation to the mean

Patent R&Dln 8.148 1.748 ln

Assets Assets i

InterpretationPatent R&D

ln 8.148 1.748 lnAssets Assets i

When the log of R&D (per asset) increases by one unit, the log of patent per asset increases by 1.748

Remember! A change in log of x is a relative change of x itself

A 1% increase in R&D (per asset) entails a 1.748% increase in the number of patent (per asset).

OLS with STATA

Stata Instruction : regress (reg)

reg y x1 x2 x3 … xk [if] [weight] [, options]

Options : noconstant : gets rid of constant

robust : estimates robust variances, even with heteroskedasticity

if : selects observations

weight : Weighted least squares

Application to Data using STATA

reg lpat_assets lrdi

_cons -8.150657 .2440936 -33.39 0.000 -8.630425 -7.670889 lrdi 1.748129 .1009131 17.32 0.000 1.549784 1.946475 lpat_assets Coef. Std. Err. t P>|t| [95% Conf. Interval]

Patent R&Dln 8.148 1.748 ln

Assets Assets i

predict newvar , [type]

Type means residual or predictions

Assessing the Goodness of Fit

It is important to ask whether a specification provides a good prediction on the dependent variable, given values of the independent variable.

Ideally, we want an indicator of the proportion of variance of the dependent variable that is accounted for – or explained – by the statistical model.

This is the variance of predictions (ŷ) and the variance of residuals (ε), since by construction, both sum to overall variance of the dependent variable (y).

Overall Variance

Decomposing the overall variance (1)

Decomposing the overall variance (2)

Coefficient of determination R² R2 is a statistic which provides information on the

goodness of fit of the model.

2

2

2

tot i

fit i tot fit res

res i i

SS y y

SS y y SS SS SS

SS y y

² fit

tot

SSR

SS

0 ² 1R

Fisher’s F Statistics Fisher’s statistics is relevant as a form of ANOVA on SSfit

which tells us whether the regression model brings significant (in a statistical sense, information.

Model SS df MSS F

(1) (2) (3) (2)/(3)

Fitted p

Residual N–p–1

Total N–1 2

iy y

2

i iy y

2

iy y

p: number of parametersN: number of observations

MSS

MSSfit

res

MSS fit

MSSres

STATA output

_cons -8.150657 .2440936 -33.39 0.000 -8.630425 -7.670889 lrdi 1.748129 .1009131 17.32 0.000 1.549784 1.946475 lpat_assets Coef. Std. Err. t P>|t| [95% Conf. Interval]

Total 1905.10212 430 4.43047005 Root MSE = 1.6165 Adj R-squared = 0.4102 Residual 1120.97039 429 2.61298459 R-squared = 0.4116 Model 784.131733 1 784.131733 Prob > F = 0.0000 F( 1, 429) = 300.09 Source SS df MS Number of obs = 431

. reg lpat_assets lrdi

.

What the R² is not

Independent variables are a true cause of the changes in the dependent variable

The correct regression was used

The most appropriate set of independent variables has been chosen

There is co-linearity present in the data

The model could be improved by using transformed versions of the existing set of independent variables

Inference on β

We have estimated

Therefore we must test whether the estimated parameter is significantly different than 0, and, by way of consequence, we must say something on the distribution – the mean and variance – of the true but unobserved β*

( )i iiE y y x Si 0, ( )iE y Si 0, ( ) iE y x

The mean and variance of β It is possible to show that is a good approximation,

i.e. an unbiased estimator, of the true parameter β*.

*ˆE

2 22

ˆ2

1

VAR where 1 1i in

i

y y nx x

The variance of β is defined as the ratio of the mean square of errors over the sum of squares of the explanatory variable

The confidence interval of β

We must now define de confidence interval of β, at 95%. To do so, we use the mean and variance of β and define the t value as follows: *

ˆt s

*.025

2

1

tn

i

x x

Therefore, the 95% confidence interval of β is:

If the 95% CI does not include 0, then β is significantly different than 0.

Student t Test for β We are also in the position to infer on β

H0: β* = 0

H1: β* ≠ 0

Rule of decision

Accept H0 is | t | < tα/2

Reject H0 is | t | ≥ tα/2

*

ˆ ˆ

ts s

STATA output

_cons -8.150657 .2440936 -33.39 0.000 -8.630425 -7.670889 lrdi 1.748129 .1009131 17.32 0.000 1.549784 1.946475 lpat_assets Coef. Std. Err. t P>|t| [95% Conf. Interval]

Total 1905.10212 430 4.43047005 Root MSE = 1.6165 Adj R-squared = 0.4102 Residual 1120.97039 429 2.61298459 R-squared = 0.4116 Model 784.131733 1 784.131733 Prob > F = 0.0000 F( 1, 429) = 300.09 Source SS df MS Number of obs = 431

. reg lpat_assets lrdi

.