Bivariate linear regression

ASW, Chapter 12

Economics 224 – Notes for November 12, 2008

Regression line

• For a bivariate or simple regression with an independent variable x and a dependent variable y, the regression equation is y = β0 + β1 x + ε.

• The values of the error term, ε, average to 0 so E(ε) = 0 and E(y) = β0 + β1 x.

• Using observed or sample data for values of x and y, estimates of the parameters β0 and β1 are obtained and the estimated regression line is

where is the value of y that is predicted from the estimated regression line.

xbby 10ˆ

xy 10

y

Bivariate regression line

x

yE(y) = β0 + β1x

yi

ε or error term

xi

y = β0 + β1x + ε

E(yi)

Observed scatter diagram and estimated least squares line

x

y

ŷ = b0 + b1x

y (actual)

ŷ (estimated)deviation

Example from SLID 2005

• According to human capital theory, increased education is associated with greater earnings.

• Random sample of 22 Saskatchewan males aged 35-39 with positive wages and salaries in 2004, from the Survey of Labour and Income Dynamics, 2005.

• Let x be total number of years of school completed (YRSCHL18) and y be wages and salaries in dollars (WGSAL42).

Source: Statistics Canada, Survey of Labour and Income Dynamics, 2005 [Canada]: External Cross-sectional Economic Person File [machine readable data file]. From IDLS through UR Data Library.

ID# YRSCHL18 WGSAL421 17 625002 12 155003 12 675004 11 95005 15 380006 15 360007 19 700008 15 470009 20 80000

10 16 2800011 18 6500012 11 4800013 14 7250014 12 3300015 14.5 600016 13.5 6250017 15 7750018 13 4200019 10 3600020 12.5 2100021 15 4100022 12.3 52500

YRSCHL18 is the variable “number of years of schooling”

WGSAL42 is the variable “wages and salaries in dollars, 2004”

x

y

Plot of WGSAL42 with YRSCHL18

Total Number of years of schooling compl

222018161412108

Wa

ge

s a

nd

sa

lari

es

be

fore

de

du

ctio

ns

100000

80000

60000

40000

20000

0

x

y

Mean of x is 14.2 and sd is 2.64 years.

Mean of y is $45,954 and sd is $21,960.

n = 22 cases

Plot of WGSAL42 with YRSCHL18

Total Number of years of schooling compl

222018161412108

Wa

ge

s a

nd

sa

lari

es

be

fore

de

du

ctio

ns

100000

80000

60000

40000

20000

0

y

x

xy 181,4493,13ˆ

Analysis and results

H0: β1 = 0. Schooling has no effect on earnings.

H1: β1 > 0. Schooling has a positive effect on earnings.

From the least squares estimates, using the data for the 22 cases, the regression equation and associate statistics are:

y = -13,493 + 4,181 x. R2 = 0.253, r = 0. 503.

Standard error of the slope b0 is 1,606. t = 2.603 (20 df), significance = 0.017.

At α = 0.05, reject H0, accept H1 and conclude that schooling has a positive effect on earnings.

Each extra year of schooling adds $4,181 to annual wages and salaries for those in this sample.

Expected wages and salaries for those with 20 years of schooling is -13,493 + (4,181 x 20) = $70,127.

Equation of a line

• y = β0 + β1 x. x is the independent variable (on horizontal) and y is the dependent variable (on vertical).

• β0 and β1 are the two parameters that determine the equation of the line.

• β0 is the y intercept – determines the height of the line.

• β1 is the slope of the line.

– Positive, negative, or zero.

– Size of β1 provides an estimate of the manner that x is related to y.

Positive Slope: β1 > 0

x

y

β0Δx

Δy 0 slope1

x

y

Example – schooling (x) and earnings (y).

Negative Slope: β1 < 0

x

y

β0 Δx

Δy 0 slope1

x

y

Example – higher income (x) associated with fewer trips by bus (y).

Zero Slope: β1 = 0

x

y

β0Δx

0 slope1

x

y

Example – amount of rainfall (x) and student grades (y)

Infinite Slope: β1 =

x

y

x

y slope1

Infinite number of possible lines can be drawn. Find the straight line that best fits the points in the scatter diagram.

Plot of WGSAL42 with YRSCHL18

Total Number of years of schooling compl

222018161412108

Wa

ge

s a

nd

sa

lari

es

be

fore

de

du

ctio

ns

100000

80000

60000

40000

20000

0

Least squares method (ASW, 469)• Find estimates of β0 and β1 that produce a line that fits

the points the best. • The most commonly used criterion is least squares. • The least squares line is the unique line for which the

sum of the squares of the deviations of the y values from the line is as small as possible.

• Minimize the sum of the squares of the errors ε.• Or, equivalent to this, minimize the sum of the squares of

the differences of the y values from the values of E(y). That is, find b0 and b1 that minimize:

2

1022 ˆ xbbyyy iii

Least squares line

• Let the n observed values of x and y be termed xi and yi, where i = 1, 2, 3, ... , n.

• ∑ε2 is minimized when b0 and b1 take on the following values:

21

xx

yyxxb

i

ii

xbyb 10

Province Income AlcoholNewfoundland 26.8 8.7Prince Edward Island 27.1 8.4Nova Scotia 29.5 8.8New Brunswick 28.4 7.6Quebec 30.8 8.9Ontario 36.4 10Manitoba 30.4 9.7Saskatchewan 29.8 8.9Alberta 35.1 11.1British Columbia 32.5 10.9

Income is family income in thousands of dollars per capita, 1986. (independent variable)

Alcohol is litres of alcohol consumed per person 15 years of age or over, 1985-86. (dependent variable)

Is alcohol a superior good?

Sources: Saskatchewan Alcohol and Drug Abuse Commission,Fast Factsheet, Regina, 1988 Statistics Canada, EconomIc Families – 1986 [machine-readable data file, 1988.

Hypotheses

H0: β1 = 0. Income has no effect on alcohol consumption.

H1: β1 > 0. Income has a positive effect on alcohol consumption.

Province x y x-barx y-bary (x-barx)(y-bary) x-barx sq

Newfoundland 26.8 8.7 -3.88 -0.6 2.328 15.0544PEI 27.1 8.4 -3.58 -0.9 3.222 12.8164

Nova Scotia 29.5 8.8 -1.18 -0.5 0.59 1.3924

New Brunswick 28.4 7.6 -2.28 -1.7 3.876 5.1984Quebec 30.8 8.9 0.12 -0.4 -0.048 0.0144Ontario 36.4 10 5.72 0.7 4.004 32.7184Manitoba 30.4 9.7 -0.28 0.4 -0.112 0.0784

Saskatchewan 29.8 8.9 -0.88 -0.4 0.352 0.7744Alberta 35.1 11.1 4.42 1.8 7.956 19.5364

British Columbia 32.5 10.9 1.82 1.6 2.912 3.3124

sum 306.8 93 -6.8E-14 -7.1E-15 25.08 90.896mean 30.68 9.3

b1 0.275919732b0 0.834782609

xy 276.0835.0ˆ

SUMMARY OUTPUT

Regression StatisticsMultiple R 0.790288R Square 0.624555Adjusted R Square 0.577624Standard Error 0.721104

Observations 10

ANOVA

df SS MS FSignificance

F

Regression 1 6.920067 6.920067 13.30803 0.006513Residual 8 4.159933 0.519992Total 9 11.08

CoefficientsStandard

Error t Stat P-valueIntercept 0.834783 2.331675 0.358018 0.729592

X Variable 1 0.27592 0.075636 3.648018 0.006513

Analysis. b1 = 0.276 and its standard error is 0.076, for a t value of 3.648. At α = 0.01, the null hypothesis can be rejected (ie. with H0, the probability of a t this large or larger is 0.0065) and the alternative hypothesis accepted. At 0.01 significance, there is evidence that alcohol is a superior good, ie. that income has a positive effect on alcohol consumption.

Uses of regression line

• Draw line – select two x values (eg. 26 and 36) and compute the predicted y values (8.1 and 10.8, respectively). Plot these points and draw line.

• Interpolation. If a city had a mean income of $32,000, the expected level of alcohol consumption would be 9.7 litres per capita.

771.10)36276.0(832.0276.0835.0ˆ

091.8)26276.0(832.0276.0835.0ˆ

xy

xy

667.9)32276.0(832.0276.0835.0ˆ xy

Extrapolation

• Suppose a city had a mean income of $50,000 in 1986. From the equation, expected alcohol consumption would be 14.6 litres per capita.

• Cautions:– Model was tested over the range of income values from

26 to 36 thousand dollars. While it appears to be close to a straight line over this range, there is no assurance that a linear relation exists outside this range.

– Model does not fit all points – only 62% of the variation in alcohol consumption is explained by this linear model.

– Confidence intervals for prediction become larger the further the independent variable x is from its mean.

635.14)50276.0(832.0276.0835.0ˆ xy

Change in y resulting from change in x

1

10

/

ˆ

bdxdy

xbby

1in change

in changeb

x

y

Estimate of change in y resulting from a change in x is b1.

For the alcohol consumption example, b1 = 0.276.

A 10.0 thousand dollar increase in income is associated with a 2.76 per litre increase in annual alcohol consumption per capita, at least over the range estimated.

This can be used to calculate the income elasticity for alcohol consumption.

Goodness of fit (ASW, 12.3)

• y is the dependent variable, or the variable to be explained.

• How much of y is explained statistically from the regression model, in this case the line?

• Total variation in y is termed the total sum of squares, or SST.

• The common measure of goodness of fit of the line is the coefficient of determination, the proportion of the variation or SST that is “explained” by the line.

2SST yyi

SST or total variation of y

)ˆ()ˆ()(

)ˆ()ˆ(

yyyyyy

yyyyyy

iiii

iiii

Difference of any observed value of y from the mean is the difference between the observed and predicted value plus the difference of the predicted value from the mean of y. From this, it can be proved that:

Difference from mean

“Error” of prediction

Value of y “explained” by the line

222ˆˆ yyyyyy iiii

SST= Total variation of y

SSE = “Unexplained” or “error” variation of y

SSR = “Explained” variation of y

29

Variation in y

x

y ŷ = b0 + b1xyiŷi

xi

y

yyi

30

x

y ŷ = b0 + b1xyiŷi

xi

Variation in y “explained” by the line

y

yyi ˆ

yyi

“Explained” portion

31

Variation in y that is “unexplained” or error

x

y

yiŷi

xi

yi – ŷi

y

ŷ = b0 + b1x

yyi

yyi ˆ

‘Unexplained” or error

Coefficient of determination

The coefficient of determination, r2 or R2 (the notation used in many texts), is defined as the ratio of the “explained” or regression sum of squares, SSR, to the total variation or sum of squares, SST.

2

222

)(

)ˆ(

SST

SSR

yy

yyRr

i

i

The coefficient of determination is the square of the correlation coefficient r. As noted by ASW (483), the correlation coefficient, r, is the square root of the coefficient of determination, but with the same sign (positive or negative) as b1.

Calculations for:

Province x y Predicted Y Residuals SSE SSR SST

Nfld 26.8 8.7 8.229431 0.470569 0.221435 1.146117 0.36

PEI 27.1 8.4 8.312207 0.087793 0.007708 0.975734 0.81

NS 29.5 8.8 8.974415 -0.17441 0.03042 0.106006 0.25

NB 28.4 7.6 8.670903 -1.0709 1.146833 0.395763 2.89

Que 30.8 8.9 9.33311 -0.43311 0.187585 0.001096 0.16

Ont 36.4 10 10.87826 -0.87826 0.771342 2.490907 0.49

Man 30.4 9.7 9.222742 0.477258 0.227775 0.005969 0.16

SK 29.8 8.9 9.057191 -0.15719 0.024709 0.058956 0.16

Alb 35.1 11.1 10.51957 0.580435 0.336905 1.487339 3.24

BC 32.5 10.9 9.802174 1.097826 1.205222 0.252179 2.56

4.159933 6.920067 11.08

R squared 0.624555xy 276.0835.0ˆ

SUMMARY OUTPUT

Regression StatisticsMultiple R 0.790288R Square 0.624555Adjusted R Square 0.577624Standard Error 0.721104

Observations 10

ANOVA

df SS MS F Significance F

Regression 1 6.920067 6.920067 13.30803 0.006513Residual 8 4.159933 0.519992Total 9 11.08

Interpretation of R2

• Proportion, or percentage if multiplied by 100, of the variation in the dependent variable that is statistically explained by the regression line.

• 0 R2 1.

• Large R2 means the line fits the observed points well and the line explains a lot of the variation in the dependent variable, at least in statistical terms.

• Small R2 means the line does not fit the observed points very well and the line does not explain much of the variation in the dependent variable.

– Random or error component dominates.

– Missing variables.

– Relationship between x and y may not be linear.

How large is a large R2?

• Extent of relationship – weak relationship associated with low value and strong relationship associated with large value.

• Type of data– Micro/survey data associated with small values of R2.

For schooling/earnings example, R2 = 0.253. Much individual variation.

– Grouped data associated with larger values of R2. In income/alcohol example, R2 = 0.625. Grouping averages out individual variation.

– Time series data often results in very high R2. In consumption function example (next slide), R2 = 0.988. Trends often move together.

GDP

Consumption

Consumption (y) and GDP (x), Canada, 1995 to 2004, quarterly data

Beware of R2

• Difficult to compare across equations, especially with different types of data and forms of relationships.

• More variables added to model can increase R2. Adjusted R2 can correct for this. ASW, Chapter 13.

• Grouped or averaged observations can result in larger values of R2.

• Need to test for statistical significance.

• We want good estimates of β0 and β1, rather than high R2.

• At the same time, for similar types of data and issues, a model with a larger value of R2 may be preferable to one with a smaller value.

Next day

• Assumptions of regression model.

• Testing for statistical significance.