chapter 5: regression1 chapter 5 relationships: regression

Chapter 5: Regression 1

Chapter 5

Relationships: Regression

2

Objectives (BPS chapter 5)Regression

Regression lines

The least-squares regression line

Facts about least-squares regression

Residuals

Influential observations

Cautions about correlation and regression

Association does not imply causation

3

Correlation tells us about

strength (scatter) and direction

of the linear relationship

between two quantitative

variables.

We would like to have a numerical description of how the variables vary

together. We would also like to make predictions based on the observed

association between those two variables.

We wish to find the straight line that best fits our data. But which line best describes our data?

4

A regression lineA regression line is a straight line that describes how a response variable

(y) changes as an explanatory variable (x) changes. We often use a

regression line to predict the value of y for a given value of x.

5Chapter 5: Regression 5

Linear Regression We wish to quantify the linear relationship between an

explanatory variable and a response variable.

We can then predict the average response for all subjects with a given value of the explanatory variable.

Regression equation: y = a + bx– x is the value of the explanatory variable– y is the average value of the response variable

– note that a and b are just the y-intercept and slope of a straight line

6Chapter 5: Regression 6

0

20

40

60

80

0 20 40 60Axis Title

Axis Title

Thought Question 1

How would you draw a line through the points? How would you draw a line through the points? How do you determine which line ‘fits best’? How do you determine which line ‘fits best’?

7Chapter 5: Regression 7

Y

Y = mX + b

b = Y-intercept

X

Changein Y

Change in X

m = Slope

Linear Equations

High School TeacherHigh School Teacher

8Chapter 5: Regression 8

The Linear Model Remember from Algebra that a straight

line can be written as: In Statistics we use a slightly different

notation:

We write to emphasize that the points that satisfy this equation are just our predicted values, not the actual data values.

y

y mx b

y

= a + bxy

9Chapter 5: Regression 9

Example: Fat Versus Protein

The following is a scatterplot of total fat versus protein for 30 items on the Burger King menu:

We wish to fit a straight line through the data.

10Chapter 5: Regression 10

Residuals

The model won’t be perfect, regardless of the line we draw.

Some points will be above the line and some will be below.

The estimate made from a model is the predicted value (denoted as ).

y

y

11Chapter 5: Regression 11

Residuals (cont.)

The difference between the observed value and its associated predicted value is called the residual.

To find the residuals, we always subtract the predicted value from the observed one:

ˆresidual observed predicted y y

12Chapter 5: Regression 12

Residuals (cont.)

A negative residual means the predicted value is too big (an overestimate).

A positive residual means the predicted value is too small (an underestimate).

13Chapter 5: Regression 13

“Best Fit” Means Least Squares Some residuals are positive, others are negative,

and, on average, they cancel each other out. So, we can’t assess how well the line fits by

adding up all the residuals. Similar to what we did with the standard

deviation, we square the residuals and add the squares.

The smaller the sum, the better the fit. The line of best fit is the line for which the sum of

the squared residuals is smallest.

14Chapter 5: Regression 14

Least Squares

Used to determine the “best” line

We want the line to be as close as possible to the data points in the vertical (y) direction (since that is what we are trying to predict)

Least Squares: use the line that minimizes the sum of the squares of the vertical distances of the data points from the line

15Chapter 5: Regression 15

The Linear Model (cont.)

We write b and a for the slope and intercept of the line. The b and a are called the coefficients of the linear model.

The coefficient b is the slope, which tells us how rapidly changes with respect to x. The coefficient a is the intercept, which tells where the line hits (intercepts) the y-axis.

y

y

16

First we calculate the slope of the line, b. We already know how to calculate r, sx and sy.

r is the correlation (the slope has the same sign as r)sy is the standard deviation of the response variable ysx is the the standard deviation of the explanatory variable x

Once we know b, the slope, we can calculate a, the y-intercept:

where x and y are the sample means of the x and y variables

How to:

This means that we don’t have to calculate a lot of squared distances to find the least-squares regression line for a data set. We can instead rely on the equation.

Some calculators can calculate r, a and b.

b rsy

sx

17Chapter 5: Regression 17

Example

Fill in the missing information in the table below:

18

Facts about least-squares regression

1. The distinction between explanatory and response variables is

essential in regression.

2. The correlation coefficient (r) and the slope (b) of the least-squares

line have the same sign. The direction of the association determines

the sign of the slope of the regression line.

3. The least-squares regression line always passes through the point

.

4. The correlation r describes the strength of a straight-line relationship.

The square of the correlation, r2, is the fraction of the variation in the

values of y that is explained by the least-squares regression of y on x.

The square of the correlation is called Coefficient of Determination.

,x y

19

The distinction between explanatory and response variables is crucial in

regression. If you exchange y for x in calculating the regression line, you

will get the wrong line.

Regression examines the distance of all points from the line in the y

direction only.

Data from the Hubble

telescope about galaxies

moving away from Earth:

These two lines are the two

regression lines calculated

either correctly (x = distance,

y = velocity, solid line) or

incorrectly (x = velocity, y =

distance, dotted line).

20Chapter 5: Regression 20

Interpretation of the Slope and Intercept

The slope indicates the amount by which

changes when x changes by one unit.

The intercept is the value of when x = 0.

It is not always meaningful.

y

y

y

21Chapter 5: Regression 21

Example

The regression line for the Burger King data is

Interpret the slope and the intercept.

Slope: For every one gram increase in protein, the fat

content increases by 0.97g.

Intercept: A BK meal that has 0g of protein contains

6.8g of fat.

y

22Chapter 5: Regression 22

In predicting a value of y based on some given value of x ...

1. If there is no linear correlation, the best predicted y-value is y.

Predictions

2. If there is a linear correlation, the best predicted y-value is found by substituting the x-value into the regression equation.

23Chapter 5: Regression 23

Example: Fat Versus ProteinThe regression line for the

Burger King data fits the data

well:– The equation is

– The predicted fat content for a BK Broiler chicken sandwich that contains 30g of protein is

6.8 + 0.97(30) = 35.9 grams of fat.

24Chapter 5: Regression 24

Prediction via Regression Line

Hand, et al., A Handbook of Small Data Sets, London: Chapman and Hall

The regression equation is y = 3.6 + 0.97x– y is the average age of all husbands who have wives

of age x

Suppose we know that an individual wife’s age is 30. What would we predict her husband’s age to be?

For all women aged 30, we predict the average husband age to be 32.7 years:

3.6 + (0.97)(30) = 32.7 years

Husband and Wife: Ages

^

25

Extrapolation is the use of a

regression line for predictions outside

the range of x values used to obtain

the line.

This can be misleading, as

seen here.

Caution!Beware of Extrapolation

26Chapter 5: Regression 26

Caution !Beware of Extrapolation

Sarah’s height was plotted against her age

Can you predict her height at age 42 months?

Can you predict her height at age 30 years (360 months)?

80

85

90

95

100

30 35 40 45 50 55 60 65

age (months)

hei

gh

t (c

m)

27Chapter 5: Regression 27

A CautionBeware of Extrapolation

Regression line: = 71.95 + .383 x

height at age 42 months? = 88 cm.

height at age 30 years? = 209.8 cm.– She is predicted to

be 6' 10.5" at age 30.70

90

110

130

150

170

190

210

30 90 150 210 270 330 390

age (months)

hei

gh

t (c

m)

28Chapter 5: Regression 28

Residuals Revisited Residuals help us to see whether the

model makes sense. When a regression model is

appropriate, nothing interesting should be left behind.

After we fit a regression model, we usually plot the residuals in the hope of finding no apparent pattern.

29Chapter 5: Regression 29

Residual Plot Analysis

A residual plot is a scatterplot of the regression residuals against the explanatory variable.

If a residual plot does not reveal any pattern, the regression equation is a good representation of the association between the two variables.

If a residual plot reveals some systematic pattern, the regression equation is not a good representation of the association between the two variables.

30Chapter 5: Regression 30

Residuals Revisited (cont.) The residuals for the BK menu

regression look appropriately boring:

Plot

31

Residuals are randomly scattered—good!

A curved pattern—means the relationship you are looking at is not linear.

A change in variability across plot is a

warning sign. You need to find out what it

is and remember that predictions made in

areas of larger variability will not be as

good.

32Chapter 5: Regression 32

Coefficient of Determination (R2)

Measures usefulness of regression prediction R2 (or r2, the square of the correlation):

measures the percentage of the variation in the values of the response variable (y) that is explained by the regression line r=1: R2=1: regression line explains all (100%) of

the variation in y r=.7: R2=.49: regression line explains almost

half(50%) of the variation in y

33Chapter 5: Regression 33

Along with the slope and intercept for a regression, you should always report R2 so that readers can judge for themselves how successful the regression is at fitting the data.

Statistics is about variation, and R2 measures the success of the regression model in terms of the fraction of the variation of y accounted for by the regression.

R2 (cont)

34

r = −1r2 = 1

Changes in x

explain 100% of

the variations in y.

y can be entirely

predicted for any

given value of x.

r = 0r2 = 0

Changes in x

explain 0% of the

variations in y.

The value(s) y

takes is (are)

entirely

independent of

what value x

takes.

Here the change in x only

explains 76% of the change in

y. The rest of the change in y

(the vertical scatter, shown as

red arrows) must be explained

by something other than x.

r = 0.87r2 = 0.76

35Chapter 5: Regression 35

Caution with regression Since regression and correlation are closely

related, we need to check the same conditions for regression as we did for correlation:

– Quantitative Variables Condition

– Straight Enough Condition

– Outlier Condition

36

Caution with regression Do not use a regression on inappropriate data.

Pattern in the residuals

Presence of large outliers Use residual plots for help.

Clumped data falsely appearing linear

Recognize when the correlation/regression is performed on

averages.

A relationship, however strong, does not imply causation.

Beware of lurking variables.

Avoid extrapolating (predicting outside the observed x data range).

37Chapter 5: Regression 37

1. If there is no linear correlation, don’t use the regression equation to make predictions.

2. When using the regression equation for predictions, stay within the scope of the

available sample data.

3. A regression equation based on old data is not necessarily valid now.

4. Don’t make predictions about a population that is different from the population from which the sample data were drawn.

Guidelines for Using The Regression Equation

38Chapter 5: Regression 38

Vocabulary Marginal Change – refers to the slope; the amount the

response variable changes when the explanatory variable changes by one unit.

Outlier - A point lying far away from the other data points.

Influential Point - An outlier that that has the potential to change the regression line.

- Points that are outliers in either the x or y direction of a scatterplot are often influential for the correlation.

- Points that outliers in the x direction are often influential for the least-squares regression line. Try

39

All data Without child 18 Without child 19

Outlier in y-direction

Influential

Are these

points

influential?

40

Vocabulary: Lurking vs. Confounding

LURKING VARIABLE

A lurking variable is a variable that is not among the explanatory

or response variables in a study and yet may influence the

interpretation of relationships among those variables.

CONFOUNDING

Two variables are confounded when their effects on a response

variable cannot be distinguished from each other. The confounded

variables may be either explanatory variables or lurking variables.

41

Lurking variablesLurking variables can falsely suggest a relationship.

What is the lurking variable in these examples?How could you answer if you didn’t know anything about the topic?

Strong positive association between

the number firefighters at a fire site and

the amount of damage a fire does

Negative association between moderate

amounts of wine drinking and death rates

from heart disease in developed nations

42Chapter 5: Regression 42

Correlation Does Not Imply Causation

Even very strong correlations may not correspond to a real

causal relationship.

43Chapter 5: Regression 43

Evidence of Causation A properly conducted experiment

establishes the connection Other considerations:

– A reasonable explanation for a cause and effect exists

– The connection happens in repeated trials – The connection happens under varying

conditions– Potential confounding factors are ruled out– Alleged cause precedes the effect in time

44Chapter 5: Regression 44

Evidence of Causation An observed relationship can be used

for prediction without worrying about causation as long as the patterns found in past data continue to hold true.

We must make sure that the prediction makes sense.

We must be very careful of extreme extrapolation.

45Chapter 5: Regression 45

Reasons Two Variables May Be Related (Correlated)

Explanatory variable causes change in response variable

Response variable causes change in explanatory variable

Explanatory may have some cause, but is not the sole cause of changes in the response variable

Confounding variables may exist Both variables may result from a common cause

– such as, both variables changing over time The correlation may be merely a coincidence

46Chapter 5: Regression 46

Common Response(both variables change due to

common cause)

Both may result from an unhappy marriage.

Explanatory: Divorce among men Response: Percent abusing alcohol

47Chapter 5: Regression 47

Both Variables are Changing Over Time

Both divorces and suicides have increased dramatically since 1900.

Are divorces causing suicides? Are suicides causing divorces??? The population has increased

dramatically since 1900 (causing both to increase). Better to investigate: Has the rate of divorce

or the rate of suicide changed over time?

48Chapter 5: Regression 48

The Relationship May Be Just a Coincidence

Sometimes we see some strong correlations (or apparent associations) just by chance, even when the variables are not related in the population.

49Chapter 5: Regression 49

A required whooping cough vaccine was blamed for seizures that caused brain damage– led to reduced production of vaccine (due to lawsuits)

Study of 38,000 children found no evidence for the accusations (reported in New York Times)– “people confused association with cause-and-effect”– “virtually every kid received the vaccine…it was

inevitable that, by chance, brain damage caused by other factors would occasionally occur in a recently vaccinated child”

Coincidence (?)Vaccines and Brain Damage

50

Quiz1. The least-squares regression line is

A) the line that best splits the data in half, with half of the points above the line and half below the line.

B) the line that makes the square of the correlation in the data as

large as possible.

C) the line that makes the sum of the squares of the vertical distances of the data points from the line as small as possible.

D) all of the above.

51

Quiz2. The fraction of the variation in the values of a response y

that is explained by the least-squares regression of y on x is

A) the intercept of the least-squares regression line

B) the correlation coefficient.

C) the slope of the least-squares regression line.

D) the square of the correlation coefficient.

52

Quiz3. A researcher obtained the average SAT scores of all

students in each of the 50 states, and the average teacher salaries in each of the 50 states of the US. He found a negative correlation between these variables. The researcher concluded that a lurking variable must be present. By lurking variable he means

A) the true cause of a response. B) the true variable, which is explained by the explanatory variable. C) a variable that is not among the variables studied but which affects the response variable. D) any variable that produces a large residual. .

53

1. Find the least-squares regression line of John’s height on age based on the data.

2. Interpret the slope of the regression line.

3. If John’s height at 54 months has a residual of 1.2 in, what was his actual height?

4. What percentage of the variation in John’s height is explained by the regression model?

Example:

54Chapter 5: Regression 54

Key Concepts Least Squares Regression Equation Interpretation of the slope and intercept Prediction – avoid extrapolations Residual Plot Analysis R2

Correlation does not imply causation Reasons variables may be correlated