lecture 4: multiple ols regression
Post on 26-Dec-2021
17 Views
Preview:
TRANSCRIPT
Lecture 4: Multiple OLS RegressionIntroduction to Econometrics,Fall 2020
Zhaopeng Qu
Nanjing University
10/15/2020
Zhaopeng Qu (Nanjing University) Lecture 4: Multiple OLS Regression 10/15/2020 1 / 79
1 Review for the previous lectures
2 Multiple OLS Regression: Introduction
3 Multiple OLS Regression: Estimation
4 Partitioned Regression: OLS Estimators in Multiple Regression
5 Measures of Fit in Multiple Regression
6 Categoried Variable as independent variables in Regression
7 Multiple Regression: Assumption
8 Properties of OLS Estimators in Multiple Regression
9 Multiple OLS Regression and Causality
Zhaopeng Qu (Nanjing University) Lecture 4: Multiple OLS Regression 10/15/2020 2 / 79
Review for the previous lectures
Section 1
Review for the previous lectures
Zhaopeng Qu (Nanjing University) Lecture 4: Multiple OLS Regression 10/15/2020 3 / 79
Review for the previous lectures
Simple OLS formula
The linear regression model with one regressor is denoted by
๐๐ = ๐ฝ0 + ๐ฝ1๐๐ + ๐ข๐
Where๐๐ is the dependent variable(Test Score)๐๐ is the independent variable or regressor(Class Size orStudent-Teacher Ratio)๐ข๐ is the error term which contains all the other factors besides ๐that determine the value of the dependent variable, ๐ , for a specificobservation, ๐.
Zhaopeng Qu (Nanjing University) Lecture 4: Multiple OLS Regression 10/15/2020 4 / 79
Review for the previous lectures
The OLS Estimator
The estimators of the slope and intercept that minimize the sum ofthe squares of ๏ฟฝ๏ฟฝ๐,thus
๐๐๐ ๐๐๐๐0,๐1
๐โ๐=1
๏ฟฝ๏ฟฝ2๐ = ๐๐๐
๐0,๐1
๐โ๐=1
(๐๐ โ ๐0 โ ๐1๐๐)2
are called the ordinary least squares (OLS) estimators of ๐ฝ0 and๐ฝ1.
OLS estimator of ๐ฝ1:
๐1 = ๐ฝ1 = โ๐๐=1(๐๐ โ ๐)(๐๐ โ ๐ )
โ๐๐=1(๐๐ โ ๐)(๐๐ โ ๐)
Zhaopeng Qu (Nanjing University) Lecture 4: Multiple OLS Regression 10/15/2020 5 / 79
Review for the previous lectures
Least Squares Assumptions
Under 3 least squares assumptions,1 Assumption 12 Assumption 23 Assumption 3
the OLS estimators will be1 unbiased2 consistent3 normal sampling distribution
Zhaopeng Qu (Nanjing University) Lecture 4: Multiple OLS Regression 10/15/2020 6 / 79
Multiple OLS Regression: Introduction
Section 2
Multiple OLS Regression: Introduction
Zhaopeng Qu (Nanjing University) Lecture 4: Multiple OLS Regression 10/15/2020 7 / 79
Multiple OLS Regression: Introduction
Violation of the first Least Squares Assumption
Recall simple OLS regression equation
๐๐ = ๐ฝ0 + ๐ฝ1๐๐ + ๐ข๐
Question: What does ๐ข๐ represent?Answer: contains all other factors(variables) which potentially affect๐๐.
Assumption 1๐ธ(๐ข๐|๐๐) = 0
It states that ๐ข๐ are unrelated to ๐๐ in the sense that,given a value of๐๐,the mean of these other factors equals zero.But what if they (or at least one) are correlated with ๐๐?
Zhaopeng Qu (Nanjing University) Lecture 4: Multiple OLS Regression 10/15/2020 8 / 79
Multiple OLS Regression: Introduction
Example: Class Size and Test Score
Many other factors can affect studentโs performance in the school.
One of factors is the share of immigrants in the class(school,district). Because immigrant children may have different backgroundsfrom native children, such as
parentsโeducation levelfamily income and wealthparenting styletraditional culture
Zhaopeng Qu (Nanjing University) Lecture 4: Multiple OLS Regression 10/15/2020 9 / 79
Multiple OLS Regression: Introduction
Scatter Plot: The share of immigrants and STR
14
16
18
20
22
24
26
0 25 50 75the share of immigrants
stud
entโ
teac
her
ratio
higher share of immigrants, bigger class sizeZhaopeng Qu (Nanjing University) Lecture 4: Multiple OLS Regression 10/15/2020 10 / 79
Multiple OLS Regression: Introduction
Scatter Plot: The share of immigrants and STR
630
660
690
0 25 50 75the share of immigrants
test
sco
res
higher share of immigrants, lower testscoreZhaopeng Qu (Nanjing University) Lecture 4: Multiple OLS Regression 10/15/2020 11 / 79
Multiple OLS Regression: Introduction
The share of immigrants as an Omitted Variable
Class size may be related to percentage of English learners andstudents who are still learning English likely have lower test scores.
In other words, the effect of class size on scores we had obtained insimple OLS may contain an effect of immigrants on scores.
It implies that percentage of English learners is contained in ๐ข๐, inturn that Assumption 1 is violated, more precisely,the estimates of
๐ฝ1 and ๐ฝ0 are biased and inconsistent.
Zhaopeng Qu (Nanjing University) Lecture 4: Multiple OLS Regression 10/15/2020 12 / 79
Multiple OLS Regression: Introduction
Omitted Variable Bias: IntroductionAs before, ๐๐ and ๐๐ represent STR and Test Score.
Besides, ๐๐ is the variable which represents the share of Englishlearners.
Suppose that we have no information about it for some reasons, thenwe have to omit in the regression.
Then we have two regression:True model(Long regression):
๐๐ = ๐ฝ0 + ๐ฝ1๐๐ + ๐พ๐๐ + ๐ข๐
where ๐ธ(๐ข๐|๐๐, ๐๐) = 0OVB model(Short regression):
๐๐ = ๐ฝ0 + ๐ฝ1๐๐ + ๐ฃ๐
where ๐ฃ๐ = ๐พ๐๐ + ๐ข๐Zhaopeng Qu (Nanjing University) Lecture 4: Multiple OLS Regression 10/15/2020 13 / 79
Multiple OLS Regression: Introduction
Omitted Variable Bias: Biasedness
Let us see what is the consequence of OVB
๐ธ[ ๐ฝ1] = ๐ธ[โ(๐๐ โ ๏ฟฝ๏ฟฝ)(๐ฝ0 + ๐ฝ1๐๐ + ๐ฃ๐ โ (๐ฝ0 + ๐ฝ1๐ + ๐ฃ))โ(๐๐ โ ๏ฟฝ๏ฟฝ)(๐๐ โ ๐)
]
= ๐ธ[โ(๐๐ โ ๏ฟฝ๏ฟฝ)(๐ฝ0 + ๐ฝ1๐๐ + ๐พ๐๐ + ๐ข๐ โ (๐ฝ0 + ๐ฝ1๐ + ๐พ๐ + ๐ข))โ(๐๐ โ ๏ฟฝ๏ฟฝ)(๐๐ โ ๐)
]
Skip Several steps in algebra which is very similar to procedures for provingunbiasedness of ๐ฝ1,using the Law of Iterated Expectation(LIE) again.At last, we get (Please prove it by yourself)
๐ธ[ ๐ฝ1] = ๐ฝ1 + ๐พ๐ธ[โ(๐๐ โ ๏ฟฝ๏ฟฝ)(๐๐ โ ๏ฟฝ๏ฟฝ)โ(๐๐ โ ๏ฟฝ๏ฟฝ)(๐๐ โ ๏ฟฝ๏ฟฝ) ]
Zhaopeng Qu (Nanjing University) Lecture 4: Multiple OLS Regression 10/15/2020 14 / 79
Multiple OLS Regression: Introduction
Omitted Variable Bias: Biasedness
As proving unbiasedness of ๐ฝ1,we can know
If ๐๐ is unrelated to ๐๐,then ๐ธ[ ๐ฝ1] = ๐ฝ1, because
๐ธ[โ(๐๐ โ ๏ฟฝ๏ฟฝ)(๐๐ โ ๏ฟฝ๏ฟฝ)โ(๐๐ โ ๏ฟฝ๏ฟฝ)(๐๐ โ ๏ฟฝ๏ฟฝ) ] = 0
If ๐๐ is not determinant of ๐๐, which means that
๐พ = 0
,then ๐ธ[ ๐ฝ1] = ๐ฝ1, too.
Only if both two conditions above are violated simultaneously, then๐ฝ1 is biased, which is normally called Omitted Variable Bias(OVB).
Zhaopeng Qu (Nanjing University) Lecture 4: Multiple OLS Regression 10/15/2020 15 / 79
Multiple OLS Regression: Introduction
Omitted Variable Bias(OVB): inconsistency
Recall: simple OLS is consistency when n is large, thus ๐๐๐๐ ๐ฝ1 = ๐ถ๐๐ฃ(๐๐,๐๐)๐ ๐๐(๐๐)
๐๐๐๐ ๐ฝ1 = ๐ถ๐๐ฃ(๐๐, ๐๐)๐ ๐๐๐๐
= ๐ถ๐๐ฃ(๐๐, (๐ฝ0 + ๐ฝ1๐๐ + ๐ฃ๐))๐ ๐๐๐๐
= ๐ถ๐๐ฃ(๐๐, (๐ฝ0 + ๐ฝ1๐๐ + ๐พ๐๐ + ๐ข๐))๐ ๐๐๐๐
= ๐ถ๐๐ฃ(๐๐, ๐ฝ0) + ๐ฝ1๐ถ๐๐ฃ(๐๐, ๐๐) + ๐พ๐ถ๐๐ฃ(๐๐, ๐๐) + ๐ถ๐๐ฃ(๐๐, ๐ข๐)๐ ๐๐๐๐
= ๐ฝ1 + ๐พ ๐ถ๐๐ฃ(๐๐, ๐๐)๐ ๐๐๐๐
Zhaopeng Qu (Nanjing University) Lecture 4: Multiple OLS Regression 10/15/2020 16 / 79
Multiple OLS Regression: Introduction
Omitted Variable Bias(OVB): inconsistency
Recall: simple OLS is consistency when n is large, thus ๐๐๐๐ ๐ฝ1 = ๐ถ๐๐ฃ(๐๐,๐๐)๐ ๐๐(๐๐)
๐๐๐๐ ๐ฝ1 = ๐ถ๐๐ฃ(๐๐, ๐๐)๐ ๐๐๐๐
= ๐ถ๐๐ฃ(๐๐, (๐ฝ0 + ๐ฝ1๐๐ + ๐ฃ๐))๐ ๐๐๐๐
= ๐ถ๐๐ฃ(๐๐, (๐ฝ0 + ๐ฝ1๐๐ + ๐พ๐๐ + ๐ข๐))๐ ๐๐๐๐
= ๐ถ๐๐ฃ(๐๐, ๐ฝ0) + ๐ฝ1๐ถ๐๐ฃ(๐๐, ๐๐) + ๐พ๐ถ๐๐ฃ(๐๐, ๐๐) + ๐ถ๐๐ฃ(๐๐, ๐ข๐)๐ ๐๐๐๐
= ๐ฝ1 + ๐พ ๐ถ๐๐ฃ(๐๐, ๐๐)๐ ๐๐๐๐
Zhaopeng Qu (Nanjing University) Lecture 4: Multiple OLS Regression 10/15/2020 16 / 79
Multiple OLS Regression: Introduction
Omitted Variable Bias(OVB): inconsistency
Thus we obtain
๐๐๐๐ ๐ฝ1 = ๐ฝ1 + ๐พ ๐ถ๐๐ฃ(๐๐, ๐๐)๐ ๐๐๐๐
๐ฝ1 is still consistentif ๐๐ is unrelated to X, thus ๐ถ๐๐ฃ(๐๐, ๐๐) = 0if ๐๐ has no effect on ๐๐, thus ๐พ = 0
If both two conditions above hold simultaneously, then ๐ฝ1 isinconsistent.
Zhaopeng Qu (Nanjing University) Lecture 4: Multiple OLS Regression 10/15/2020 17 / 79
Multiple OLS Regression: Introduction
Omitted Variable Bias(OVB):Directions
If OVB can be possible in our regression,then we should guess thedirections of the bias, in case that we canโt eliminate it.
Summary of the bias when ๐ค๐ is omitted in estimating equation
๐ถ๐๐ฃ(๐๐, ๐๐) > 0 ๐ถ๐๐ฃ(๐๐, ๐๐) < 0๐พ > 0 Positive bias Negative bias๐พ < 0 Negative bias Positive bias
Zhaopeng Qu (Nanjing University) Lecture 4: Multiple OLS Regression 10/15/2020 18 / 79
Multiple OLS Regression: Introduction
Omitted Variable Bias: Examples
Question: If we omit following variables, then what are the directionsof these biases? and why?
1 Time of day of the test2 Parking lot space per pupil3 Teachersโ Salary4 Family income5 Percentage of English learners(the Share of Immigrants)
Zhaopeng Qu (Nanjing University) Lecture 4: Multiple OLS Regression 10/15/2020 19 / 79
Multiple OLS Regression: Introduction
Omitted Variable Bias: ExamplesRegress Testscore on Class size
#### Call:## lm(formula = testscr ~ str, data = ca)#### Residuals:## Min 1Q Median 3Q Max## -47.727 -14.251 0.483 12.822 48.540#### Coefficients:## Estimate Std. Error t value Pr(>|t|)## (Intercept) 698.9330 9.4675 73.825 < 2e-16 ***## str -2.2798 0.4798 -4.751 2.78e-06 ***## ---## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1#### Residual standard error: 18.58 on 418 degrees of freedom## Multiple R-squared: 0.05124, Adjusted R-squared: 0.04897## F-statistic: 22.58 on 1 and 418 DF, p-value: 2.783e-06
Zhaopeng Qu (Nanjing University) Lecture 4: Multiple OLS Regression 10/15/2020 20 / 79
Multiple OLS Regression: Introduction
Omitted Variable Bias: ExamplesRegress Testscore on Class size and the percentage of English learners
#### Call:## lm(formula = testscr ~ str + el_pct, data = ca)#### Residuals:## Min 1Q Median 3Q Max## -48.845 -10.240 -0.308 9.815 43.461#### Coefficients:## Estimate Std. Error t value Pr(>|t|)## (Intercept) 686.03225 7.41131 92.566 < 2e-16 ***## str -1.10130 0.38028 -2.896 0.00398 **## el_pct -0.64978 0.03934 -16.516 < 2e-16 ***## ---## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1#### Residual standard error: 14.46 on 417 degrees of freedom## Multiple R-squared: 0.4264, Adjusted R-squared: 0.4237## F-statistic: 155 on 2 and 417 DF, p-value: < 2.2e-16
Zhaopeng Qu (Nanjing University) Lecture 4: Multiple OLS Regression 10/15/2020 21 / 79
Multiple OLS Regression: Introduction
Omitted Variable Bias: Examples่กจ 2: Class Size and Test Score
Dependent variable:testscr
(1) (2)str โ2.280โโโ โ1.101โโโ
(0.480) (0.380)el_pct โ0.650โโโ
(0.039)Constant 698.933โโโ 686.032โโโ
(9.467) (7.411)Observations 420 420R2 0.051 0.426
Note: โp<0.1; โโp<0.05; โโโp<0.01
Zhaopeng Qu (Nanjing University) Lecture 4: Multiple OLS Regression 10/15/2020 22 / 79
Multiple OLS Regression: Introduction
Warp Up
OVB bias is the most possible bias when we run OLS regression usingnonexperimental data.
OVB bias means that there are some variables which should havebeen included in the regression but actually was not.
Then the simplest way to overcome OVB: Put omitted variables intothe regression, which means our regression model is
๐๐ = ๐ฝ0 + ๐ฝ1๐๐ + ๐พ๐๐ + ๐ข๐
The strategy can be denoted as controlling informally, whichintroduces the more general regression model: Multiple OLSRegression.
Zhaopeng Qu (Nanjing University) Lecture 4: Multiple OLS Regression 10/15/2020 23 / 79
Multiple OLS Regression: Estimation
Section 3
Multiple OLS Regression: Estimation
Zhaopeng Qu (Nanjing University) Lecture 4: Multiple OLS Regression 10/15/2020 24 / 79
Multiple OLS Regression: Estimation
Multiple regression model with k regressors
The multiple regression model is
๐๐ = ๐ฝ0 + ๐ฝ1๐1,๐ + ๐ฝ2๐2,๐ + ... + ๐ฝ๐๐๐,๐ + ๐ข๐, ๐ = 1, ..., ๐ (4.1)
where
๐๐ is the dependent variable๐1, ๐2, ...๐๐ are the independent variables(includes one is our ofinterest and some control variables)๐ฝ๐, ๐ = 1...๐ are slope coefficients on ๐๐ corresponding.๐ฝ0 is the estimate intercept, the value of Y when all ๐๐ = 0, ๐ = 1...๐๐ข๐ is the regression error term, still all other factors affect outcomes.
Zhaopeng Qu (Nanjing University) Lecture 4: Multiple OLS Regression 10/15/2020 25 / 79
Multiple OLS Regression: Estimation
Interpretation of coefficients ๐ฝ๐, ๐ = 1...๐
๐ฝ๐ is partial (marginal) effect of ๐๐ on Y.
๐ฝ๐ = ๐๐๐๐๐๐,๐
๐ฝ๐ is also partial (marginal) effect of ๐ธ[๐๐|๐1..๐๐].
๐ฝ๐ = ๐๐ธ[๐๐|๐1, ..., ๐๐]๐๐๐,๐
it does mean that we are estimate the effect of X on Y when โotherthings equalโ, thus the concept of ceteris paribus.
Zhaopeng Qu (Nanjing University) Lecture 4: Multiple OLS Regression 10/15/2020 26 / 79
Multiple OLS Regression: Estimation
OLS Estimation in Multiple Regressors
As in a Simple OLS Regression, the estimators of Multiple OLSRegression is just a minimize the following question
๐๐๐๐๐๐ โ๐0,๐1,...,๐๐
(๐๐ โ ๐0 โ ๐1๐1,๐ โ ... โ ๐๐๐๐,๐)2
where ๐0 = ๐ฝ1, ๐1 = ๐ฝ2, ..., ๐๐ = ๐ฝ๐ are estimators.
Zhaopeng Qu (Nanjing University) Lecture 4: Multiple OLS Regression 10/15/2020 27 / 79
Multiple OLS Regression: Estimation
OLS Estimation in Multiple Regressors
Similarly in Simple OLS, based on F.O.C,the multiple OLS estimators๐ฝ0, ๐ฝ1, ..., ๐ฝ๐ are obtained by solving the following system of normal
equations
๐๐๐0
๐โ๐=1
๏ฟฝ๏ฟฝ2๐ = โ (๐๐ โ ๐ฝ0 โ ๐ฝ1๐1,๐ โ ... โ ๐ฝ๐๐๐,๐) = 0
๐๐๐1
๐โ๐=1
๏ฟฝ๏ฟฝ2๐ = โ (๐๐ โ ๐ฝ0 โ ๐ฝ1๐1,๐ โ ... โ ๐ฝ๐๐๐,๐)๐1,๐ = 0
โฎ =โฎ =โฎ๐
๐๐๐
๐โ๐=1
๏ฟฝ๏ฟฝ2๐ = โ (๐๐ โ ๐ฝ0 โ ๐ฝ1๐1,๐ โ ... โ ๐ฝ๐๐๐,๐)๐๐,๐ = 0
Zhaopeng Qu (Nanjing University) Lecture 4: Multiple OLS Regression 10/15/2020 28 / 79
Multiple OLS Regression: Estimation
OLS Estimation in Multiple RegressorsSimilar to in Simple OLS, the fitted residuals are
๐ข๐ = ๐๐ โ ๐ฝ0 โ ๐ฝ1๐1,๐ โ ... โ ๐ฝ๐๐๐,๐
Therefore, the normal equations also can be written as
โ ๐ข๐ = 0โ ๐ข๐๐1,๐ = 0
โฎ =โฎโ ๐ข๐๐๐,๐ = 0
While it is convenient to transform equations above using matrixalgebra to compute these estimators, we can use partitionedregression to obtain the formula of estimators without usingmatrices.
Zhaopeng Qu (Nanjing University) Lecture 4: Multiple OLS Regression 10/15/2020 29 / 79
Partitioned Regression: OLS Estimators in Multiple Regression
Section 4
Partitioned Regression: OLS Estimators inMultiple Regression
Zhaopeng Qu (Nanjing University) Lecture 4: Multiple OLS Regression 10/15/2020 30 / 79
Partitioned Regression: OLS Estimators in Multiple Regression
Partitioned regression: OLS estimatorsA useful representation of ๐ฝ๐ could be obtained by the partitionedregression.
The OLS estimator of ๐ฝ๐; ๐ = 1, 2...๐ can be computed in threesteps:
1 Regress ๐๐ on ๐1, ๐2, ...๐๐โ1, ๐๐+1, ๐๐, thus๐๐,๐ = ๐พ0 + ๐พ1๐1๐ + ... + ๐พ๐โ1๐๐โ1,๐ + ๐พ๐+1๐๐+1,๐... + ๐พ๐๐๐,๐ + ๐ฃ๐๐
2 Obtain the residuals from the regression above,denoted as ๏ฟฝ๏ฟฝ๐,๐ = ๐ฃ๐๐3 Regress ๐ on ๏ฟฝ๏ฟฝ๐,๐
The last step implies that the OLS estimator of ๐ฝ๐ can be expressedas follows
๐ฝ๐ = โ๐๐=1 (๏ฟฝ๏ฟฝ๐๐ โ ๏ฟฝ๏ฟฝ๐๐)(๐๐ โ ๐ )
โ๐๐=1 (๏ฟฝ๏ฟฝ๐๐ โ ๏ฟฝ๏ฟฝ๐๐)2
= โ๐๐=1 ๏ฟฝ๏ฟฝ๐๐๐๐
โ๐๐=1 ๏ฟฝ๏ฟฝ2
๐๐
Zhaopeng Qu (Nanjing University) Lecture 4: Multiple OLS Regression 10/15/2020 31 / 79
Partitioned Regression: OLS Estimators in Multiple Regression
Partitioned regression: OLS estimatorsSuppose we want to obtain an expression for ๐ฝ1.
Then the first step: regress ๐1,๐ on other regressors, thus
๐1,๐ = ๐พ0 + ๐พ2๐2,๐ + ... + ๐พ๐๐๐,๐ + ๐ฃ๐
Then, we can obtain
๐1,๐ = ๐พ0 + ๐พ2๐2,๐ + ... + ๐พ๐๐๐,๐ + ๏ฟฝ๏ฟฝ1,๐
where ๏ฟฝ๏ฟฝ1,๐ is the fitted OLS residual,thus ๏ฟฝ๏ฟฝ๐,๐ = ๐ฃ1๐
Then we could prove that
๐ฝ1 = โ๐๐=1 ๏ฟฝ๏ฟฝ1,๐๐๐
โ๐๐=1 ๏ฟฝ๏ฟฝ2
1,๐
Zhaopeng Qu (Nanjing University) Lecture 4: Multiple OLS Regression 10/15/2020 32 / 79
Partitioned Regression: OLS Estimators in Multiple Regression
Proof of Partitioned regression result(1)
Recall ๐ข๐ are the residuals for the Multiple OLS regression equation,thus
๐๐ = ๐ฝ0 + ๐ฝ1๐1,๐ + ๐ฝ2๐2,๐ + ... + ๐ฝ๐๐๐,๐ + ๏ฟฝ๏ฟฝ๐
Then we haveโ ๏ฟฝ๏ฟฝ๐ = โ ๏ฟฝ๏ฟฝ๐๐๐๐ = 0, ๐ = 1, 2, ..., ๐
Likewise,๏ฟฝ๏ฟฝ1๐ are the residuals for the partitioned regression equation on๐2๐..., ๐๐๐, then we have
โ ๏ฟฝ๏ฟฝ1๐ = โ ๏ฟฝ๏ฟฝ1๐๐2,๐ = ... = โ ๏ฟฝ๏ฟฝ1๐๐๐,๐ = 0
Additionally, because๏ฟฝ๏ฟฝ1,๐ = ๐1,๐ โ ๐พ0 โ ๐พ2๐2,๐ โ ... โ ๐พ๐๐๐,๐, then
โ ๏ฟฝ๏ฟฝ๐๏ฟฝ๏ฟฝ๐๐ = 0
Zhaopeng Qu (Nanjing University) Lecture 4: Multiple OLS Regression 10/15/2020 33 / 79
Partitioned Regression: OLS Estimators in Multiple Regression
Proof of Partitioned regression result(2)
โ๐๐=1 ๏ฟฝ๏ฟฝ1,๐๐๐
โ๐๐=1 ๏ฟฝ๏ฟฝ2
1,๐= โ ๏ฟฝ๏ฟฝ1,๐( ๐ฝ0 + ๐ฝ1๐1,๐ + ๐ฝ2๐2,๐ + ... + ๐ฝ๐๐๐,๐ + ๏ฟฝ๏ฟฝ๐)
โ ๏ฟฝ๏ฟฝ21,๐
= ๐ฝ0โ๐
๐=1 ๏ฟฝ๏ฟฝ1,๐โ๐
๐=1 ๏ฟฝ๏ฟฝ21,๐
+ ๐ฝ1โ๐
๐=1 ๏ฟฝ๏ฟฝ1,๐๐1,๐โ๐
๐=1 ๏ฟฝ๏ฟฝ21,๐
+ ...
+ ๐ฝ๐โ๐
๐=1 ๏ฟฝ๏ฟฝ1,๐๐๐,๐โ๐
๐=1 ๏ฟฝ๏ฟฝ21,๐
+ โ๐๐=1 ๏ฟฝ๏ฟฝ1,๐๏ฟฝ๏ฟฝ๐
โ๐๐=1 ๏ฟฝ๏ฟฝ2
1,๐
= ๐ฝ1โ๐
๐=1 ๏ฟฝ๏ฟฝ1,๐๐1,๐โ๐
๐=1 ๏ฟฝ๏ฟฝ21,๐
Zhaopeng Qu (Nanjing University) Lecture 4: Multiple OLS Regression 10/15/2020 34 / 79
Partitioned Regression: OLS Estimators in Multiple Regression
Proof of Partitioned regression result(3)
๐โ๐=1
๏ฟฝ๏ฟฝ1,๐๐1,๐ =๐
โ๐=1
๏ฟฝ๏ฟฝ1,๐( ๐พ0 + ๐พ2๐2,๐ + ... + ๐พ๐๐๐,๐ + ๏ฟฝ๏ฟฝ1,๐)
= ๐พ0 โ 0 + ๐พ2 โ 0 + ... + ๐พ๐ โ 0 + โ ๏ฟฝ๏ฟฝ21,๐
= โ ๏ฟฝ๏ฟฝ21,๐
Thenโ๐
๐=1 ๏ฟฝ๏ฟฝ1,๐๐๐โ๐
๐=1 ๏ฟฝ๏ฟฝ21,๐
= ๐ฝ1โ๐
๐=1 ๏ฟฝ๏ฟฝ1,๐๐1,๐โ๐
๐=1 ๏ฟฝ๏ฟฝ21,๐
= ๐ฝ1
Identical argument works for ๐ = 2, 3, ..., ๐, thus
๐ฝ๐ = โ๐๐=1 ๏ฟฝ๏ฟฝ๐,๐๐๐
โ๐๐=1 ๏ฟฝ๏ฟฝ2
๐,๐๐๐๐ ๐ = 1, 2, .., ๐
Zhaopeng Qu (Nanjing University) Lecture 4: Multiple OLS Regression 10/15/2020 35 / 79
Partitioned Regression: OLS Estimators in Multiple Regression
The intuition of Partitioned regression
Partialling Out
First, we regress ๐๐ against the rest of the regressors (and aconstant) and keep ๏ฟฝ๏ฟฝ๐ which is the โpartโ of ๐๐ that is uncorrelated
Then, to obtain ๐ฝ๐ , we regress Y against ๏ฟฝ๏ฟฝ๐ which is โcleanโ fromcorrelation with other regressors.
๐ฝ๐ measures the effect of ๐1 after the effects of ๐2, ..., ๐๐ have beenpartialled out or netted out.
Zhaopeng Qu (Nanjing University) Lecture 4: Multiple OLS Regression 10/15/2020 36 / 79
Partitioned Regression: OLS Estimators in Multiple Regression
Test Scores and Student-Teacher Ratios
Now we put two additional control variables into our OLS regressionmodel
๐ ๐๐ ๐ก๐ ๐๐๐๐ = ๐ฝ0 + ๐ฝ1๐๐ ๐ + ๐ฝ2๐๐๐๐๐ก + ๐ฝ3๐๐ฃ๐๐๐๐ + ๐ข๐
elpct: the share of English learners as an indicator for immigrants
avginc: average income of the district as an indicator for familybackgrounds
Zhaopeng Qu (Nanjing University) Lecture 4: Multiple OLS Regression 10/15/2020 37 / 79
Partitioned Regression: OLS Estimators in Multiple Regression
Test Scores and Student-Teacher Ratios(2)
tilde.str <- residuals(lm(str ~ el_pct, data=ca))mean(tilde.str) # should be zero
## [1] -1.0111e-16sum(tilde.str) # also is zero
## [1] -4.240358e-14
Zhaopeng Qu (Nanjing University) Lecture 4: Multiple OLS Regression 10/15/2020 38 / 79
Partitioned Regression: OLS Estimators in Multiple Regression
Test Scores and Student-Teacher Ratios(3)
Multiple OLS estimator
๐ฝ๐ = โ๐๐=1 ๏ฟฝ๏ฟฝ๐,๐๐๐
โ๐๐=1 ๏ฟฝ๏ฟฝ2
๐,๐๐๐๐ ๐ = 1, 2, .., ๐
sum(tilde.str*ca$testscr)/sum(tilde.str^2)
## [1] -1.101296
Zhaopeng Qu (Nanjing University) Lecture 4: Multiple OLS Regression 10/15/2020 39 / 79
Partitioned Regression: OLS Estimators in Multiple Regression
Test Scores and Student-Teacher Ratios(4)reg3 <- lm(testscr ~ tilde.str,data = ca)summary(reg3)
#### Call:## lm(formula = testscr ~ tilde.str, data = ca)#### Residuals:## Min 1Q Median 3Q Max## -48.693 -14.124 0.988 13.209 50.872#### Coefficients:## Estimate Std. Error t value Pr(>|t|)## (Intercept) 654.1565 0.9254 706.864 <2e-16 ***## tilde.str -1.1013 0.4986 -2.209 0.0277 *## ---## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1#### Residual standard error: 18.97 on 418 degrees of freedom## Multiple R-squared: 0.01154, Adjusted R-squared: 0.009171## F-statistic: 4.878 on 1 and 418 DF, p-value: 0.02774Zhaopeng Qu (Nanjing University) Lecture 4: Multiple OLS Regression 10/15/2020 40 / 79
Partitioned Regression: OLS Estimators in Multiple Regression
Test Scores and Student-Teacher Ratios(5)reg4 <- lm(testscr ~ str+el_pct,data = ca)summary(reg4)
#### Call:## lm(formula = testscr ~ str + el_pct, data = ca)#### Residuals:## Min 1Q Median 3Q Max## -48.845 -10.240 -0.308 9.815 43.461#### Coefficients:## Estimate Std. Error t value Pr(>|t|)## (Intercept) 686.03225 7.41131 92.566 < 2e-16 ***## str -1.10130 0.38028 -2.896 0.00398 **## el_pct -0.64978 0.03934 -16.516 < 2e-16 ***## ---## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1#### Residual standard error: 14.46 on 417 degrees of freedom## Multiple R-squared: 0.4264, Adjusted R-squared: 0.4237## F-statistic: 155 on 2 and 417 DF, p-value: < 2.2e-16Zhaopeng Qu (Nanjing University) Lecture 4: Multiple OLS Regression 10/15/2020 41 / 79
Partitioned Regression: OLS Estimators in Multiple Regression
Test Scores and Student-Teacher Ratios(6)่กจ 3: Class Size and Test Score
Dependent variable:testscr
(1) (2)tilde.str โ1.101โโ
(0.499)str โ1.101โโโ
(0.380)el_pct โ0.650โโโ
(0.039)Constant 654.157โโโ 686.032โโโ
(0.925) (7.411)Observations 420 420R2 0.012 0.426Adjusted R2 0.009 0.424
Note: โp<0.1; โโp<0.05; โโโp<0.01Zhaopeng Qu (Nanjing University) Lecture 4: Multiple OLS Regression 10/15/2020 42 / 79
Measures of Fit in Multiple Regression
Section 5
Measures of Fit in Multiple Regression
Zhaopeng Qu (Nanjing University) Lecture 4: Multiple OLS Regression 10/15/2020 43 / 79
Measures of Fit in Multiple Regression
Measures of Fit: The ๐ 2
Decompose ๐๐ into the fitted value plus the residual ๐๐ = ๐๐ + ๏ฟฝ๏ฟฝ๐
The total sum of squares (TSS): ๐ ๐๐ = โ๐๐=1(๐๐ โ ๐ )2
The explained sum of squares (ESS): โ๐๐=1( ๐๐ โ ๐ )2
The sum of squared residuals (SSR): โ๐๐=1( ๐๐ โ ๐๐)2 = โ๐
๐=1 ๏ฟฝ๏ฟฝ2๐
And๐ ๐๐ = ๐ธ๐๐ + ๐๐๐
The regression ๐ 2 is the fraction of the sample variance of ๐๐explained by (or predicted by) the regressors.
๐ 2 = ๐ธ๐๐๐ ๐๐ = 1 โ ๐๐๐
๐ ๐๐
Zhaopeng Qu (Nanjing University) Lecture 4: Multiple OLS Regression 10/15/2020 44 / 79
Measures of Fit in Multiple Regression
Measures of Fit in Multiple RegressionWhen you put more variables into the regression, then ๐ 2 alwaysincreases when you add another regressor. Because in general theSSR will decrease.
Consider two models๐๐ = ๐ฝ0 + ๐ฝ1๐1๐ + ๐ข๐
๐๐ = ๐ฝ0 + ๐ฝ1๐1๐ + ๐ฝ2๐2๐ + ๐ฃ๐
Recall: about two residuals ๏ฟฝ๏ฟฝ๐ and ๐ฃ๐, we have๐
โ๐=1
๏ฟฝ๏ฟฝ๐ =๐
โ๐=1
๏ฟฝ๏ฟฝ๐๐1๐ = 0๐
โ๐=1
๐ฃ๐ =๐
โ๐=1
๐ฃ๐๐1๐ =๐
โ๐=1
๐ฃ๐๐2๐ = 0
Zhaopeng Qu (Nanjing University) Lecture 4: Multiple OLS Regression 10/15/2020 45 / 79
Measures of Fit in Multiple Regression
Measures of Fit in Multiple Regression
we will show that ๐โ๐=1
๏ฟฝ๏ฟฝ2๐ โฅ
๐โ๐=1
๐ฃ2๐
therefore ๐ 2๐ฃ โฅ ๐ 2
๐ข, thus ๐ 2 the regression with one regressor is lessor equal than ๐ 2 that corresponds to the regression with tworegressors.
This conclusion can be generalized to the case of ๐ + 1 regressors.
Zhaopeng Qu (Nanjing University) Lecture 4: Multiple OLS Regression 10/15/2020 46 / 79
Measures of Fit in Multiple Regression
Measures of Fit in Multiple Regression
At first we would like to know โ๐๐=1 ๏ฟฝ๏ฟฝ๐ ๐ฃ๐
๐โ๐=1
๏ฟฝ๏ฟฝ๐ ๐ฃ๐ =๐
โ๐=1
(๐๐ โ ๐ฝ0 โ ๐ฝ1๐1๐) ๐ฃ๐
=๐
โ๐=1
๐๐ ๐ฃ๐ โ ๐ฝ0๐
โ๐=1
๐ฃ๐ โ ๐ฝ1๐
โ๐=1
๐1 ๐ฃ๐
=๐
โ๐=1
๐๐ ๐ฃ๐ โ ๐ฝ0 โ 0 โ ๐ฝ1 โ 0
=๐
โ๐=1
( ๐ฝ0 + ๐ฝ1๐1๐ + ๐ฝ2๐2๐ + ๐ฃ๐) ๐ฃ๐
= โ๐=1
๐ฃ๐ ๐ฃ๐
Zhaopeng Qu (Nanjing University) Lecture 4: Multiple OLS Regression 10/15/2020 47 / 79
Measures of Fit in Multiple Regression
Measures of Fit in Multiple Regression
Then we can obtain
๐โ๐=1
๏ฟฝ๏ฟฝ2๐ โ
๐โ๐=1
๐ฃ2๐ =
๐โ๐=1
๏ฟฝ๏ฟฝ2๐ +
๐โ๐=1
๐ฃ2๐ โ 2
๐โ๐=1
๐ฃ2๐
=๐
โ๐=1
๏ฟฝ๏ฟฝ2๐ +
๐โ๐=1
๐ฃ2๐ โ 2
๐โ๐=1
๏ฟฝ๏ฟฝ๐ ๐ฃ๐
=๐
โ๐=1
(๏ฟฝ๏ฟฝ๐ โ ๐ฃ๐)2 โฅ 0
- Therefore ๐ 2๐ฃ โฅ ๐ 2
๐ข, thus ๐ 2 the regression with one regressor is less orequal than ๐ 2 that corresponds to the regression with two regressors.
Zhaopeng Qu (Nanjing University) Lecture 4: Multiple OLS Regression 10/15/2020 48 / 79
Measures of Fit in Multiple Regression
Measures of Fit: The Adjusted ๐ 2
the Adjusted ๐ 2,is a modified version of the ๐ 2 that does notnecessarily increase when a new regressor is added.
๐ 2 = 1 โ ๐ โ 1๐ โ ๐ โ 1
๐๐๐ ๐ ๐๐ = 1 โ ๐ 2
๏ฟฝ๏ฟฝ๐ 2
๐
because ๐โ1๐โ๐โ1 is always greater than 1, so ๐ 2 < ๐ 2
adding a regressor has two opposite effects on the ๐ 2.๐ 2 can be negative.
Remind: neither ๐ 2 nor ๐ 2 is not the golden criterion for good orbad OLS estimation.
Zhaopeng Qu (Nanjing University) Lecture 4: Multiple OLS Regression 10/15/2020 49 / 79
Measures of Fit in Multiple Regression
Standard Error of the Regression
Recall: SER(Standard Error of the Regression)SER is an estimator of the standard deviation of the ๐ข๐, which aremeasures of the spread of the Yโs around the regression line.
Because the regression errors are unobserved, the SER is computedusing their sample counterparts, the OLS residuals ๐ข๐
๐๐ธ๐ = ๐ ๏ฟฝ๏ฟฝ = โ๐ 2๏ฟฝ๏ฟฝ
where ๐ 2๏ฟฝ๏ฟฝ = 1
๐โ๐โ1 โ ๐ข2๐ = ๐๐๐ ๐โ๐โ1
๐ โ ๐ โ 1 because we have ๐ + 1 stricted conditions in the F.O.C.Inanother word,in order to construct ๐ข2๐, we have to estimate ๐ + 1parameters,thus ๐ฝ0, ๐ฝ1, ..., ๐ฝ๐
Zhaopeng Qu (Nanjing University) Lecture 4: Multiple OLS Regression 10/15/2020 50 / 79
Measures of Fit in Multiple Regression
Example: Test scores and Student Teacher Ratios
Zhaopeng Qu (Nanjing University) Lecture 4: Multiple OLS Regression 10/15/2020 51 / 79
Categoried Variable as independent variables in Regression
Section 6
Categoried Variable as independent variables inRegression
Zhaopeng Qu (Nanjing University) Lecture 4: Multiple OLS Regression 10/15/2020 52 / 79
Categoried Variable as independent variables in Regression
A Special Case: Categorical Variable as ๐
Recall if ๐ is a dummy variable, then we can put it into regressionequation straightly.
What if ๐ is a categorical variable?Question: What is a categorical variable?
For example, we may define ๐ท๐ as follows:
๐ท๐ =โง{โจ{โฉ
1 small-size class if ๐๐ ๐ in ๐๐กโ school district < 182 middle-size class if 18 โค ๐๐ ๐ in ๐๐กโ school district < 223 large-size class if ๐๐ ๐ in ๐๐กโ school district โฅ 22
(4.5)
Zhaopeng Qu (Nanjing University) Lecture 4: Multiple OLS Regression 10/15/2020 53 / 79
Categoried Variable as independent variables in Regression
A Special Case: Categorical Variable as ๐
Naive Solution: a simple OLS regression model
๐ ๐๐ ๐ก๐๐๐๐๐๐ = ๐ฝ0 + ๐ฝ1๐ท๐ + ๐ข๐ (4.3)
Question: Can you explain the meanning of estimate coefficient ๐ฝ1?
Answer: It doese not make sense that the coefficient of ๐ฝ1 can beexplained as continuous variables.
Zhaopeng Qu (Nanjing University) Lecture 4: Multiple OLS Regression 10/15/2020 54 / 79
Categoried Variable as independent variables in Regression
A Special Case: Categorical Variables as ๐The first step: turn a categorical variable(๐ท๐) into multiple dummyvariables(๐ท1๐, ๐ท2๐, ๐ท3๐)
๐ท1๐ = {1 small-sized class if ๐๐ ๐ in ๐๐กโ school district < 180 middle-sized class or large-sized class if not
๐ท2๐ = {1 middle-sized class if 18 โค ๐๐ ๐ in ๐๐กโ school district < 220 large-sized class or small-sized class if not
๐ท3๐ = {1 large-sized class if ๐๐ ๐ in ๐๐กโ school district โฅ 220 middle-sized class or small-sized class if not
Zhaopeng Qu (Nanjing University) Lecture 4: Multiple OLS Regression 10/15/2020 55 / 79
Categoried Variable as independent variables in Regression
A Special Case: Categorical Variables as ๐
We put these dummies into a multiple regression
๐ ๐๐ ๐ก๐๐๐๐๐๐ = ๐ฝ0 + ๐ฝ1๐ท1๐ + ๐ฝ2๐ท2๐ + ๐ฝ3๐ท3๐ + ๐ข๐ (4.6)
Then as a dummy variable as the independent variable in a simpleregression The coefficients (๐ฝ1, ๐ฝ2, ๐ฝ3) represent the effect of everycategorical class on ๐ก๐๐ ๐ก๐ ๐๐๐๐ respectively.
Zhaopeng Qu (Nanjing University) Lecture 4: Multiple OLS Regression 10/15/2020 56 / 79
Categoried Variable as independent variables in Regression
A Special Case: Categorical Variables as ๐
In practice, we canโt put all dummies into the regression, but onlyhave ๐ โ 1 dummies unless we will suffer perfect multi-collinearity.
The regression may be like as
๐ ๐๐ ๐ก๐๐๐๐๐๐ = ๐ฝ0 + ๐ฝ1๐ท1๐ + ๐ฝ2๐ท2๐ + ๐ข๐ (4.6)
The default intercept term, ๐ฝ0,represents the large-sized class.Then,the coefficients (๐ฝ1, ๐ฝ2) represent ๐ก๐๐ ๐ก๐ ๐๐๐๐ gaps betweensmall_sized, middle-sized class and large-sized class,respectively.
Zhaopeng Qu (Nanjing University) Lecture 4: Multiple OLS Regression 10/15/2020 57 / 79
Multiple Regression: Assumption
Section 7
Multiple Regression: Assumption
Zhaopeng Qu (Nanjing University) Lecture 4: Multiple OLS Regression 10/15/2020 58 / 79
Multiple Regression: Assumption
Multiple Regression: Assumption
Assumption 1: The conditional distribution of ๐ข๐ given ๐1๐, ..., ๐๐๐has mean zero,thus
๐ธ[๐ข๐|๐1๐, ..., ๐๐๐] = 0
Assumption 2: (๐๐, ๐1๐, ..., ๐๐๐) are i.i.d.Assumption 3: Large outliers are unlikely.Assumption 4: No perfect multicollinearity.
Zhaopeng Qu (Nanjing University) Lecture 4: Multiple OLS Regression 10/15/2020 59 / 79
Multiple Regression: Assumption
Perfect multicollinearity
Perfect multicollinearity arises when one of the regressors is a perfectlinear combination of the other regressors.
Binary variables are sometimes referred to as dummy variables
If you include a full set of binary variables (a complete and mutuallyexclusive categorization) and an intercept in the regression, you willhave perfect multicollinearity.
eg. female and male = 1-femaleeg. West, Central and East China
This is called the dummy variable trap.
Solutions to the dummy variable trap: Omit one of the groups or theintercept
Zhaopeng Qu (Nanjing University) Lecture 4: Multiple OLS Regression 10/15/2020 60 / 79
Multiple Regression: Assumption
Perfect multicollinearityregress Testscore on Class size and the percentage of English learners
#### Call:## lm(formula = testscr ~ str + el_pct, data = ca)#### Residuals:## Min 1Q Median 3Q Max## -48.845 -10.240 -0.308 9.815 43.461#### Coefficients:## Estimate Std. Error t value Pr(>|t|)## (Intercept) 686.03225 7.41131 92.566 < 2e-16 ***## str -1.10130 0.38028 -2.896 0.00398 **## el_pct -0.64978 0.03934 -16.516 < 2e-16 ***## ---## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1#### Residual standard error: 14.46 on 417 degrees of freedom## Multiple R-squared: 0.4264, Adjusted R-squared: 0.4237## F-statistic: 155 on 2 and 417 DF, p-value: < 2.2e-16Zhaopeng Qu (Nanjing University) Lecture 4: Multiple OLS Regression 10/15/2020 61 / 79
Multiple Regression: Assumption
Perfect multicollinearityadd a new variable nel=1-el_pct into the regression
#### Call:## lm(formula = testscr ~ str + nel_pct + el_pct, data = ca)#### Residuals:## Min 1Q Median 3Q Max## -48.845 -10.240 -0.308 9.815 43.461#### Coefficients: (1 not defined because of singularities)## Estimate Std. Error t value Pr(>|t|)## (Intercept) 685.38247 7.41556 92.425 < 2e-16 ***## str -1.10130 0.38028 -2.896 0.00398 **## nel_pct 0.64978 0.03934 16.516 < 2e-16 ***## el_pct NA NA NA NA## ---## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1#### Residual standard error: 14.46 on 417 degrees of freedom## Multiple R-squared: 0.4264, Adjusted R-squared: 0.4237## F-statistic: 155 on 2 and 417 DF, p-value: < 2.2e-16
Zhaopeng Qu (Nanjing University) Lecture 4: Multiple OLS Regression 10/15/2020 62 / 79
Multiple Regression: Assumption
Perfect multicollinearity่กจ 4: Class Size and Test Score
Dependent variable:testscr
(1) (2)str โ1.101โโโ โ1.101โโโ
(0.380) (0.380)nel_pct 0.650โโโ
(0.039)el_pct โ0.650โโโ
(0.039)Constant 686.032โโโ 685.382โโโ
(7.411) (7.416)Observations 420 420R2 0.426 0.426
Note: โp<0.1; โโp<0.05; โโโp<0.01Zhaopeng Qu (Nanjing University) Lecture 4: Multiple OLS Regression 10/15/2020 63 / 79
Multiple Regression: Assumption
Multicollinearity
Multicollinearity means that two or more regressors are highly correlated,but one regressor is NOT a perfect linear function of one or more of theother regressors.
multicollinearity is NOT a violation of OLS assumptions.It does not impose theoretical problem for the calculation of OLSestimators.
But if two regressors are highly correlated, then the the coefficient onat least one of the regressors is imprecisely estimated (high variance).
to what extent two correlated variables can be seen as โhighlycorrelatedโ?
rule of thumb: correlation coefficient is over 0.8.
Zhaopeng Qu (Nanjing University) Lecture 4: Multiple OLS Regression 10/15/2020 64 / 79
Multiple Regression: Assumption
Venn Diagrams for Multiple Regression Model
In a simple model (y on X),OLS uses โBlueโ + โRedโ toestimate ๐ฝ.When y is regressed on Xand W: OLS throws awaythe red area and just usesblue to estimate ๐ฝ.Idea: Red area iscontaminated(we do notknow if the movements iny are due to X or to W).
Zhaopeng Qu (Nanjing University) Lecture 4: Multiple OLS Regression 10/15/2020 65 / 79
Multiple Regression: Assumption
Venn Diagrams for Multicollinearity
less information (compare the Blue and Green areas in both figures) isused, the estimation is less precise.
Zhaopeng Qu (Nanjing University) Lecture 4: Multiple OLS Regression 10/15/2020 66 / 79
Multiple Regression: Assumption
Multiple Regression: Test Scores and Class Size่กจ 5: Class Size and Test Score
testscr(1) (2) (3)
str โ2.280โโโ โ1.101โโโ โ0.069(0.480) (0.380) (0.277)
el_pct โ0.650โโโ โ0.488โโโ
(0.039) (0.029)avginc 1.495โโโ
(0.075)Constant 698.933โโโ 686.032โโโ 640.315โโโ
(9.467) (7.411) (5.775)N 420 420 420R2 0.051 0.426 0.707Adjusted R2 0.049 0.424 0.705
Notes: โโโSignificant at the 1 percent level.โโSignificant at the 5 percent level.โSignificant at the 10 percent level.
Zhaopeng Qu (Nanjing University) Lecture 4: Multiple OLS Regression 10/15/2020 67 / 79
Properties of OLS Estimators in Multiple Regression
Section 8
Properties of OLS Estimators in MultipleRegression
Zhaopeng Qu (Nanjing University) Lecture 4: Multiple OLS Regression 10/15/2020 68 / 79
Properties of OLS Estimators in Multiple Regression
Properties of OLS estimators: Unbiasedness(1)Use partitioned regression formula
๐ฝ1 = โ๐๐=1 ๏ฟฝ๏ฟฝ1,๐๐๐
โ๐๐=1 ๏ฟฝ๏ฟฝ2
1,๐
Substitute๐๐ = ๐ฝ0 + ๐ฝ1๐1,๐ + ๐ฝ2๐2,๐ + ... + ๐ฝ๐๐๐,๐ + ๐ข๐, ๐ = 1, ..., ๐,then
๐ฝ1 = โ ๏ฟฝ๏ฟฝ1,๐(๐ฝ0 + ๐ฝ1๐1,๐ + ๐ฝ2๐2,๐ + ... + ๐ฝ๐๐๐,๐ + ๐ข๐)โ ๏ฟฝ๏ฟฝ2
1,๐
= ๐ฝ0โ๐
๐=1 ๏ฟฝ๏ฟฝ1,๐โ๐
๐=1 ๏ฟฝ๏ฟฝ21,๐
+ ๐ฝ1โ๐
๐=1 ๏ฟฝ๏ฟฝ1,๐๐1,๐โ๐
๐=1 ๏ฟฝ๏ฟฝ21,๐
+ ...
+ ๐ฝ๐โ๐
๐=1 ๏ฟฝ๏ฟฝ1,๐๐๐,๐โ๐
๐=1 ๏ฟฝ๏ฟฝ21,๐
+ โ๐๐=1 ๏ฟฝ๏ฟฝ1,๐๐ข๐
โ๐๐=1 ๏ฟฝ๏ฟฝ2
1,๐Zhaopeng Qu (Nanjing University) Lecture 4: Multiple OLS Regression 10/15/2020 69 / 79
Properties of OLS Estimators in Multiple Regression
Properties of OLS estimators: Unbiasedness(2)
Because๐
โ๐=1
๏ฟฝ๏ฟฝ1,๐ =๐
โ๐=1
๏ฟฝ๏ฟฝ1,๐๐๐,๐ = 0 , ๐ = 2, 3, ..., ๐๐
โ๐=1
๏ฟฝ๏ฟฝ1,๐๐1,๐ = โ ๏ฟฝ๏ฟฝ21,๐
Therefore๐ฝ1 = ๐ฝ1 + โ๐
๐=1 ๏ฟฝ๏ฟฝ1,๐๐ข๐โ๐
๐=1 ๏ฟฝ๏ฟฝ21,๐
Zhaopeng Qu (Nanjing University) Lecture 4: Multiple OLS Regression 10/15/2020 70 / 79
Properties of OLS Estimators in Multiple Regression
Properties of OLS estimators: Unbiasedness(3)Recall Assumption 1: ๐ธ[๐ข๐|๐1๐, ๐2๐...๐๐๐] = 0 and ๏ฟฝ๏ฟฝ1๐ is afunction of ๐2๐...๐๐๐
Then take expectations of ๐ฝ1 and The Law of IteratedExpectations again
๐ธ[ ๐ฝ1] = ๐ธ[๐ฝ1 + โ๐๐=1 ๏ฟฝ๏ฟฝ1,๐๐ข๐
โ๐๐=1 ๏ฟฝ๏ฟฝ2
1,๐] = ๐ฝ1 + ๐ธ[โ๐
๐=1 ๏ฟฝ๏ฟฝ1,๐๐ข๐โ๐
๐=1 ๏ฟฝ๏ฟฝ21,๐
]
= ๐ฝ1 + ๐ธ[โ๐๐=1 ๏ฟฝ๏ฟฝ1,๐๐ธ[๐ข๐|๐2๐...๐๐๐]
โ๐๐=1 ๏ฟฝ๏ฟฝ2
1,๐]
= ๐ฝ1
Identical argument works for ๐ฝ2, ..., ๐ฝ๐, thus
๐ธ[ ๐ฝ๐] = ๐ฝ๐ where ๐ = 1, 2, ..., ๐Zhaopeng Qu (Nanjing University) Lecture 4: Multiple OLS Regression 10/15/2020 71 / 79
Properties of OLS Estimators in Multiple Regression
Properties of OLS estimators: Consistency(1)
Recall๐ฝ1 = โ๐
๐=1 ๏ฟฝ๏ฟฝ1,๐๐๐โ๐
๐=1 ๏ฟฝ๏ฟฝ21,๐
Similar to the proof in the Simple OLS Regression,thus
๐ฝ1 = โ๐๐=1 ๏ฟฝ๏ฟฝ1,๐๐๐
โ๐๐=1 ๏ฟฝ๏ฟฝ2
1,๐=
1๐โ2 โ๐
๐=1 ๏ฟฝ๏ฟฝ1๐๐๐1
๐โ2 โ๐๐=1 ๏ฟฝ๏ฟฝ2
1๐= (
๐ ๏ฟฝ๏ฟฝ1๐๐ 2
๏ฟฝ๏ฟฝ1
)
where ๐ ๏ฟฝ๏ฟฝ1๐ and ๐ 2๏ฟฝ๏ฟฝ1
are the sample covariance of ๏ฟฝ๏ฟฝ1 and ๐ and thesample variance of ๏ฟฝ๏ฟฝ1.
Zhaopeng Qu (Nanjing University) Lecture 4: Multiple OLS Regression 10/15/2020 72 / 79
Properties of OLS Estimators in Multiple Regression
Properties of OLS estimators: Consistency(2)
Base on L.L.N(the law of large numbers) and random sample(i.i.d)
๐ ๏ฟฝ๏ฟฝ21
๐โถ ๐๏ฟฝ๏ฟฝ2
1= ๐ ๐๐(๏ฟฝ๏ฟฝ1)
๐ ๏ฟฝ๏ฟฝ1๐๐
โถ ๐๏ฟฝ๏ฟฝ1๐ = ๐ถ๐๐ฃ(๏ฟฝ๏ฟฝ1, ๐ )
Combining with Continuous Mapping Theorem,then we obtain theOLS estimator ๐ฝ1,when ๐ โถ โ
๐๐๐๐ ๐ฝ1 = ๐๐๐๐(๐ ๏ฟฝ๏ฟฝ1๐๐ 2
๏ฟฝ๏ฟฝ1
) = ๐ถ๐๐ฃ(๏ฟฝ๏ฟฝ1, ๐ )๐ ๐๐(๏ฟฝ๏ฟฝ1)
Zhaopeng Qu (Nanjing University) Lecture 4: Multiple OLS Regression 10/15/2020 73 / 79
Properties of OLS Estimators in Multiple Regression
Properties of OLS estimators: Consistency(3)
๐๐๐๐ ๐ฝ1 = ๐ถ๐๐ฃ(๏ฟฝ๏ฟฝ1, ๐ )๐ ๐๐(๏ฟฝ๏ฟฝ1)
= ๐ถ๐๐ฃ(๏ฟฝ๏ฟฝ1, (๐ฝ0 + ๐ฝ1๐1๐ + ... + ๐ฝ๐๐๐๐ + ๐ข๐))๐ ๐๐(๏ฟฝ๏ฟฝ1)
= ๐ถ๐๐ฃ(๏ฟฝ๏ฟฝ1, ๐ฝ0) + ๐ฝ1๐ถ๐๐ฃ(๏ฟฝ๏ฟฝ1, ๐1๐) + ... + ๐ฝ๐๐ถ๐๐ฃ(๏ฟฝ๏ฟฝ1, ๐๐๐) + ๐ถ๐๐ฃ(๏ฟฝ๏ฟฝ1, ๐ข๐)๐ ๐๐(๏ฟฝ๏ฟฝ1)
= ๐ฝ1 + ๐ถ๐๐ฃ(๏ฟฝ๏ฟฝ1, ๐ข๐)๐ ๐๐(๏ฟฝ๏ฟฝ1)
Zhaopeng Qu (Nanjing University) Lecture 4: Multiple OLS Regression 10/15/2020 74 / 79
Properties of OLS Estimators in Multiple Regression
Properties of OLS estimators: Consistency(4)
Based on Assumption 1: ๐ธ[๐ข๐|๐1๐, ๐2๐...๐๐๐] = 0And ๏ฟฝ๏ฟฝ1๐ is a function of ๐2๐...๐๐๐
Then๐ถ๐๐ฃ(๏ฟฝ๏ฟฝ1, ๐ข๐) = 0
Then we can obtain๐๐๐๐ ๐ฝ1 = ๐ฝ1
Identical argument works for ๐ฝ2, ..., ๐ฝ๐,thus
๐๐๐๐ ๐ฝ๐ = ๐ฝ๐ where ๐ = 1, 2, ..., ๐
Zhaopeng Qu (Nanjing University) Lecture 4: Multiple OLS Regression 10/15/2020 75 / 79
Properties of OLS Estimators in Multiple Regression
The Distribution of Multiple OLS Estimators
Recall from last lecture:Under the least squares assumptions,the OLS estimators ๐ฝ1 and ๐ฝ0, areunbiased and consistent estimators of ๐ฝ1 and ๐ฝ0.In large samples, the sampling distribution of ๐ฝ1 and ๐ฝ0 is wellapproximated by a bivariate normal distribution.
Similarly as in the simple OLS, the multiple OLS estimators areaverages of the randomly sampled data, and if the sample size issufficiently large, the sampling distribution of those averages becomesnormal.
Because the multivariate normal distribution is best handledmathematically using matrix algebra, the expressions for the jointdistribution of the OLS estimators are deferred to Chapter 18(SWtextbook).
Zhaopeng Qu (Nanjing University) Lecture 4: Multiple OLS Regression 10/15/2020 76 / 79
Multiple OLS Regression and Causality
Section 9
Multiple OLS Regression and Causality
Zhaopeng Qu (Nanjing University) Lecture 4: Multiple OLS Regression 10/15/2020 77 / 79
Multiple OLS Regression and Causality
Independent Variable v.s Control VariablesGenerally, we would like to pay more attention to only oneindependent variable(thus we would like to call it treatmentvariable), though there could be many independent variables.
Because ๐ฝ๐ is partial (marginal) effect of ๐๐ on Y.
๐ฝ๐ = ๐๐๐๐๐๐,๐
which means that we are estimate the effect of X on Y when โotherthings equalโ, thus the concept of ceteris paribus.
Therefore,other variables in the right hand of equation, we call themcontrol variables, which we would like to explicitly hold fixed whenstudying the effect of ๐1 on Y.
More specifically,regression model turns into๐๐ = ๐ฝ0 + ๐ฝ1๐ท๐ + ๐พ2๐ถ2,๐ + ... + ๐พ๐๐ถ๐,๐ + ๐ข๐, ๐ = 1, ..., ๐
Transform it into๐๐ = ๐ฝ0 + ๐ฝ1๐ท๐ + ๐ถ2...๐,๐๐พโฒ
2...๐ + ๐ข๐, ๐ = 1, ..., ๐
Zhaopeng Qu (Nanjing University) Lecture 4: Multiple OLS Regression 10/15/2020 78 / 79
Multiple OLS Regression and Causality
Good Controls and Bad Controls
Questions: Are โmore controlsโ always better (or at least never worse)?Answer: It depends on. Because there are such things as Bad Controls,which means that when add a control(put a variable into the regression), itmay overcome the OVB bias but also bring more other bias.Good controls, are variables determining the treatment and correlated withthe outcome. And we can think of them as having been fixed at the timethe treatment variable was determined.Irrelevant controls, are variables uncorrelated with the treatment butcorrelated with the outcome. It may help reducing standard errors.Bad controls are not the factors we would like make fixed but theoutcomes of treatment.We will soon come back to discuss this topic again in Lecture 7.
Zhaopeng Qu (Nanjing University) Lecture 4: Multiple OLS Regression 10/15/2020 79 / 79
top related