chap. 11: simple linear regression

EPI 809/Spring 2008EPI 809/Spring 2008 11

Chapter 11Chapter 11

Regression and Correlation Regression and Correlation methodsmethods


Learning ObjectivesLearning Objectives

1.1. Describe the Linear Regression ModelDescribe the Linear Regression Model

2.2. State the Regression Modeling StepsState the Regression Modeling Steps

3.3. Explain Ordinary Least SquaresExplain Ordinary Least Squares

4.4. Compute Regression CoefficientsCompute Regression Coefficients

5.5. Understand and check model assumptionsUnderstand and check model assumptions

6.6. Predict Response VariablePredict Response Variable

7.7. Comments of SAS OutputComments of SAS Output


Learning Objectives… Learning Objectives…

8.8. Correlation ModelsCorrelation Models

9.9. Link between a correlation model and a Link between a correlation model and a regression modelregression model

10.10. Test of coefficient of CorrelationTest of coefficient of Correlation


ModelsModels


What is a Model?What is a Model?

1.1. Representation of Representation of Some PhenomenonSome Phenomenon

Non-Math/Stats ModelNon-Math/Stats Model

1.1. Representation of Representation of Some PhenomenonSome Phenomenon

Non-Math/Stats ModelNon-Math/Stats Model


What is a Math/Stats Model?What is a Math/Stats Model?

1.1. Often Describe Relationship between Often Describe Relationship between VariablesVariables

2.2. TypesTypes- Deterministic Models (no randomness)Deterministic Models (no randomness)

- Probabilistic Models (with randomness)Probabilistic Models (with randomness)


Deterministic ModelsDeterministic Models1.1. Hypothesize Exact RelationshipsHypothesize Exact Relationships2.2. Suitable When Prediction Error is NegligibleSuitable When Prediction Error is Negligible3.3. Example: Body mass index (BMI) is measure of Example: Body mass index (BMI) is measure of

body fat basedbody fat based

Metric Formula: BMI = BMI = Weight in KilogramsWeight in Kilograms (Height in Meters) (Height in Meters)22

Non-metric Formula: BMI = BMI = Weight (pounds)x703Weight (pounds)x703 (Height in inches)(Height in inches)22


Probabilistic ModelsProbabilistic Models

1.1. Hypothesize 2 ComponentsHypothesize 2 Components• DeterministicDeterministic• Random ErrorRandom Error

2.2. Example: Systolic blood pressure of newborns Example: Systolic blood pressure of newborns Is 6 Times the Age in days + Random ErrorIs 6 Times the Age in days + Random Error

• SBPSBP = 6xage(d) = 6xage(d) + + • Random Error May Be Due to Factors Random Error May Be Due to Factors

Other Than age in days (e.g. Birthweight)Other Than age in days (e.g. Birthweight)


Types of Types of Probabilistic ModelsProbabilistic Models

ProbabilisticModels

RegressionModels

CorrelationModels

OtherModels


Regression ModelsRegression Models


Types of Types of Probabilistic ModelsProbabilistic Models

ProbabilisticModels

RegressionModels

CorrelationModels

OtherModels


Regression ModelsRegression Models

Relationship between one Relationship between one dependentdependent variablevariable and and explanatory variable(s)explanatory variable(s)

Use equation to set up relationshipUse equation to set up relationship• NumericalNumerical Dependent (Response) Variable Dependent (Response) Variable• 1 or More Numerical or Categorical Independent 1 or More Numerical or Categorical Independent

(Explanatory) Variables(Explanatory) Variables Used Mainly for Prediction & EstimationUsed Mainly for Prediction & Estimation


Regression Modeling Steps Regression Modeling Steps 1.1. Hypothesize Deterministic Hypothesize Deterministic

ComponentComponent• Estimate Unknown ParametersEstimate Unknown Parameters

2.2. Specify Probability Distribution of Specify Probability Distribution of Random Error TermRandom Error Term

• Estimate Standard Deviation of ErrorEstimate Standard Deviation of Error

3.3. Evaluate the fitted ModelEvaluate the fitted Model 4.4. Use Model for Prediction & Use Model for Prediction &

Estimation Estimation


Model SpecificationModel Specification


Specifying the deterministic Specifying the deterministic componentcomponent

1.1. Define the dependent variable and Define the dependent variable and independent variableindependent variable

2.2. Hypothesize Nature of RelationshipHypothesize Nature of Relationship Expected Effects (i.e., Coefficients’ Signs)Expected Effects (i.e., Coefficients’ Signs) Functional Form (Linear or Non-Linear)Functional Form (Linear or Non-Linear) InteractionsInteractions


Model Specification Model Specification Is Based on TheoryIs Based on Theory

1.1. Theory of Field (e.g., Theory of Field (e.g., Epidemiology)Epidemiology)

2.2. Mathematical TheoryMathematical Theory 3.3. Previous ResearchPrevious Research 4.4. ‘Common Sense’‘Common Sense’


Thinking Challenge: Thinking Challenge: Which Is More Logical?Which Is More Logical?

Years since seroconversionCD+ counts

CD+ counts

Years since seroconversion



CD+ counts

CD+ counts


OB/GYN Study OB/GYN Study


Types of Types of Regression ModelsRegression Models



RegressionModels



RegressionModels

Simple

1 ExplanatoryVariable



RegressionModels

2+ ExplanatoryVariables

Simple Multiple




RegressionModels

Linear


Simple Multiple




RegressionModels

Linear Non-Linear


Simple Multiple




RegressionModels

Linear Non-Linear


Simple Multiple

Linear




RegressionModels

Linear Non-Linear


Simple Multiple

Linear


Non-Linear


Linear Regression Linear Regression ModelModel



RegressionModels

Linear Non-Linear


Simple

Non-Linear

Multiple

Linear



YY = mX + b

b = Y-interceptX

Changein Y

Change in X

m = Slope

Linear EquationsLinear Equations

© 1984-1994 T/Maker Co.

YY XXii ii ii 00 11

Linear Regression ModelLinear Regression Model

1.1. Relationship Between Variables Is Relationship Between Variables Is a Linear Functiona Linear Function

Dependent Dependent (Response) (Response) VariableVariable(e.g., CD+ c.)(e.g., CD+ c.)

Independent Independent (Explanatory) Variable (Explanatory) Variable (e.g., Years s. serocon.)(e.g., Years s. serocon.)

Population Population SlopeSlope

Population Population Y-InterceptY-Intercept

Random Random ErrorError


Population & Sample Population & Sample Regression ModelsRegression Models



PopulationPopulation



Unknown Relationship

PopulationPopulation

Y Xi i i 0 1




PopulationPopulation Random SampleRandom Sample

Y Xi i i 0 1




PopulationPopulation Random SampleRandom Sample

Y Xi i i 0 1

Y Xi i i 0 1


Y

X

Population Linear Regression Population Linear Regression ModelModel

Y Xi i i 0 1

iXYE 10

ObservedObservedvaluevalue

Observed valueObserved value

ii = Random error= Random error


Y

X

Y Xi i i 0 1

Sample Linear Regression Sample Linear Regression ModelModel

Y Xi i 0 1

Unsampled Unsampled observationobservation

ii = Random = Random errorerror

Observed valueObserved value

^̂


Estimating Parameters:Estimating Parameters:Least Squares MethodLeast Squares Method


0204060

0 20 40 60X

Y

Scatter plotScatter plot

1.1. Plot of All (Plot of All (XXii, , YYii) Pairs) Pairs 2.2. Suggests How Well Model Will FitSuggests How Well Model Will Fit


Thinking ChallengeThinking Challenge

How would you draw a line through the How would you draw a line through the points? How do you determine which line points? How do you determine which line ‘fits best’? ‘fits best’?

0204060

0 20 40 60X

Y


Thinking ChallengeThinking ChallengeHow would you draw a line through the How would you draw a line through the points? How do you determine which line points? How do you determine which line ‘fits best’?‘fits best’?

0204060

0 20 40 60X

YSlope changed

Intercept unchanged



0204060

0 20 40 60X

Y

Slope unchanged

Intercept changed



0204060

0 20 40 60X

YSlope changed

Intercept changed


Least SquaresLeast Squares 1.1. ‘Best Fit’ Means Difference Between ‘Best Fit’ Means Difference Between

Actual Y Values & Predicted Y Values Are Actual Y Values & Predicted Y Values Are a Minimum. a Minimum. ButBut Positive Differences Off- Positive Differences Off-Set Negative onesSet Negative ones



Actual Y Values & Predicted Y Values is a Actual Y Values & Predicted Y Values is a Minimum. Minimum. ButBut Positive Differences Off-Set Positive Differences Off-Set Negative ones. Negative ones. So square errors!So square errors!

n

ii

n

iii YY

1

2

1

2ˆˆ



Actual Y Values & Predicted Y Values Are Actual Y Values & Predicted Y Values Are a Minimum. a Minimum. ButBut Positive Differences Off- Positive Differences Off-Set Negative. So square errors!Set Negative. So square errors!

2.2. LS Minimizes the Sum of the Squared LS Minimizes the Sum of the Squared Differences (errors) (SSE)Differences (errors) (SSE)

n

ii

n

iii YY

1

2

1

2ˆˆ


Least Squares GraphicallyLeast Squares Graphically

2

Y

X

1 3

4

^^

^^

Y X2 0 1 2 2

Y Xi i 0 1

LS minimizes ii

n2

112

22

32

42


Coefficient EquationsCoefficient Equations Prediction equationPrediction equation

Sample slopeSample slope

Sample Y - interceptSample Y - intercept

ii xy 10ˆˆˆ

21̂xx

yyxxSSSS

i

iixx

xy

xy 10 ˆˆ


Derivation of Parameters (1)Derivation of Parameters (1) Least Squares (L-S): Least Squares (L-S):

Minimize squared errorMinimize squared error

xy 10 ˆˆ

220 1

0 0

0 1

0

2

i i iy x

ny n n x

220 1

1 1

n n

i i ii i

y x


Derivation of Parameters (1)Derivation of Parameters (1) Least Squares (L-S): Least Squares (L-S):

Minimize squared errorMinimize squared error

220 1

1 1

0 1

1 1

0

2

2

i i i

i i i

i i i

y x

x y x

x y y x x

1

1

1̂

i i i i

i i i i

xy

xx

x x x x y y

x x x x x x y y

SSSS


Computation TableComputation Table

Xi Yi Xi2 Yi

2 XiYi

X1 Y1 X12 Y1

2 X1Y1

X2 Y2 X22 Y2

2 X2Y2

: : : : :Xn Yn Xn

2 Yn2 XnYn

Xi Yi Xi2 Yi

2 XiYi


Interpretation of CoefficientsInterpretation of Coefficients



1.1. Slope (Slope (11)) Estimated Estimated YY Changes by Changes by 11 for Each 1 Unit for Each 1 Unit

Increase in Increase in XX• If If 11 = 2, then = 2, then YY Is Expected to Increase by 2 for Is Expected to Increase by 2 for

Each 1 Unit Increase in Each 1 Unit Increase in XX

^̂

^̂

^̂



1.1. Slope (Slope (11)) Estimated Estimated YY Changes by Changes by 11 for Each 1 Unit for Each 1 Unit

Increase in Increase in XX• If If 11 = 2, then = 2, then YY Is Expected to Increase by 2 for Each Is Expected to Increase by 2 for Each

1 Unit Increase in 1 Unit Increase in XX

2.2. Y-Intercept (Y-Intercept (00)) Average Value of Average Value of YY When When XX = 0 = 0

• If If 00 = 4, then Average = 4, then Average YY Is Expected to Be 4 Is Expected to Be 4 When When XX Is 0 Is 0

^̂

^̂

^̂^̂

^̂


Parameter Estimation ExampleParameter Estimation Example Obstetrics:Obstetrics: What is the What is the relationshiprelationship between between

Mother’s Estriol level & Birthweight using the Mother’s Estriol level & Birthweight using the following data?following data?EstriolEstriol BirthweightBirthweight

(mg/24h)(mg/24h) (g/1000)(g/1000) 11 11

22 1133 2244 2255 44


01234

0 1 2 3 4 5 6

Scatterplot Scatterplot Birthweight vs. Estriol level Birthweight vs. Estriol level

Birthweight

Estriol level


Parameter Estimation Solution Parameter Estimation Solution TableTable

Xi Yi Xi2 Yi

2 XiYi

1 1 1 1 12 1 4 1 23 2 9 4 64 2 16 4 85 4 25 16 20

15 10 55 26 37


Parameter Estimation SolutionParameter Estimation Solution

10.0370.02ˆˆ

70.0

51555

5101537

ˆ

10

2

1

2

12

11

11

XY

n

XX

n

YXYX

n

i

n

ii

i

n

ii

n

iin

iii


Coefficient Interpretation Coefficient Interpretation SolutionSolution



1.1. Slope (Slope (11)) Birthweight (Birthweight (YY) Is Expected to Increase by .7 ) Is Expected to Increase by .7

Units for Each 1 unit Increase in Estriol (Units for Each 1 unit Increase in Estriol (XX))

^̂



1.1. Slope (Slope (11)) Birthweight (Birthweight (YY) Is Expected to Increase by .7 ) Is Expected to Increase by .7

Units for Each 1 unit Increase in Estriol (Units for Each 1 unit Increase in Estriol (XX))

2.2. Intercept (Intercept (00)) Average Birthweight (Average Birthweight (YY) Is -.10 Units When ) Is -.10 Units When

Estriol level (Estriol level (XX) Is 0) Is 0• Difficult to explainDifficult to explain• The birthweight should always be positiveThe birthweight should always be positive

^̂

^̂


SAS codes for fitting a simple linear SAS codes for fitting a simple linear regressionregression

DataData BW; /*Reading data in SAS*/ BW; /*Reading data in SAS*/ input input estriol birthwestriol birthw@@;@@; cards;cards; 11 11 2 2 1 1 33 22

44 2 2 5 5 44 ; ; runrun;;

PROC REGPROC REG data=BW data=BW; /*Fitting linear regression models*/; /*Fitting linear regression models*/ model model birthwbirthw==estriolestriol;; runrun; ;


Parameter EstimatesParameter Estimates

Parameter Standard Variable DF Estimate Error t Value Pr > |t|

Intercept 1 -0.10000 0.63509 -0.16 0.8849 Estriol 1 0.70000 0.19149 3.66 0.0354

Parameter Estimation Parameter Estimation SAS Computer OutputSAS Computer Output

0̂̂ 1̂^


Parameter Estimation Thinking Parameter Estimation Thinking ChallengeChallenge

You’re a Vet epidemiologist for the county You’re a Vet epidemiologist for the county cooperative. You gather the following data:cooperative. You gather the following data:

Food (lb.)Food (lb.) Milk yield (lb.)Milk yield (lb.) 4 4 3.03.0 6 6 5.55.51010 6.56.51212 9.09.0

What is the What is the relationshiprelationship between cows’ food intake and milk yield?between cows’ food intake and milk yield?

© 1984-1994 T/Maker Co.


02468

10

0 5 10 15

Scattergram Scattergram Milk Yield vs. Food intake*Milk Yield vs. Food intake*

M. Yield (lb.)M. Yield (lb.)

Food intake (lb.)Food intake (lb.)


Parameter Estimation Solution Parameter Estimation Solution Table*Table*

Xi Yi Xi2 Yi

2 XiYi

4 3.0 16 9.00 12 6 5.5 36 30.25 3310 6.5 100 42.25 6512 9.0 144 81.00 10832 24.0 296 162.50 218


Parameter Estimation Solution*Parameter Estimation Solution*

80.0865.06ˆˆ

65.0

432296

42432218

ˆ

10

2

1

2

12

11

11

XY

n

XX

n

YXYX

n

i

n

ii

i

n

ii

n

iin

iii


Coefficient Interpretation Coefficient Interpretation Solution*Solution*



1.1. Slope (Slope (11)) Milk Yield (Milk Yield (YY) Is Expected to Increase ) Is Expected to Increase

by .65 lb. for Each 1 lb. Increase in Food by .65 lb. for Each 1 lb. Increase in Food intake (intake (XX))

^̂



1.1. Slope (Slope (11)) Milk Milk Yield (Yield (YY) Is Expected to Increase by .65 ) Is Expected to Increase by .65

lb. for Each 1 lb. Increase inlb. for Each 1 lb. Increase in Food intake Food intake ( (XX))

2.2. Y-Intercept (Y-Intercept (00)) Average Milk yield (Average Milk yield (YY) Is Expected to Be 0.8 ) Is Expected to Be 0.8

lb. When Food intake (lb. When Food intake (XX) Is 0) Is 0

^̂

^̂

chap. 11: simple linear regression

Documents