chapter 1: linear regression with one predictor variable
TRANSCRIPT
BSTT523: Kutner et al., Chapter 1 1
Chapter 1: Linear Regression with One Predictor Variable
also known as: Simple Linear Regression
Bivariate Linear Regression
Introduction:
Β· Functional relation between two variables:
π = π(π)
Value of X β Value of Y
Example: Β°F = 32Β° + (9/5)Β°C
is a deterministic relationship
1 value of X β 1 unique value of Y
Β· Statistical relation between two variables:
1 value of X β a distribution of values of Y
Y = Dependent / Response / Outcome Variable
X = Independent / Explanatory / Predictor Variable
BSTT523: Kutner et al., Chapter 1 2
Linear Equation: General equation for a straight line
π = π0 + π1π
π0: Intercept = value of Y when X=0
π1: Slope = change in Y per unit change in X
π1 =πβππππ ππ πβπ£πππ’π
πβππππ ππ πβπ£πππ’π="πππ π"
"ππ’π"
What if X increases by 1 unit?
π = π0 + π1(π + 1) = {π0 + π1π} + π1
Y increases by π1 units
{ Other than linear: e.g. curvilinear π = π0 + π1π + π2π2 }
Regression of Y on X
Β· Observe data points {(π1, π1), . . . , (ππ, ππ)}
Β· At each point ππ there is a distribution of ππβs
BSTT523: Kutner et al., Chapter 1 3
Example: Y = head circumference (cm), X = gestational age (wks)
in a sample of 100 low birth weight infants
Qs: Does average head circumference change with gestational age?
What is the form of the relationship? (linear? curvilinear?)
How to estimate the relationship, given the data?
How to make predictions for new observations?
15
20
25
30
35
40
20 22 24 26 28 30 32 34 36
Y =
He
ad C
ircu
mfe
ren
ce (
cm)
X = Gestational Age (weeks)
BSTT523: Kutner et al., Chapter 1 4
Descriptive Data:
Β· Scatterplot
Β· Correlation Coefficients
Some examples:
BSTT523: Kutner et al., Chapter 1 5
Population Correlation Coefficient:
Random Variables X and Y with parameters ππ, ππ, ππ2, ππ
2
π =πΆππ£(π,π)
ππππ=πΈ[(πβππ)(πβππ)]
ππππ , β1 β€ π β€ +1
π measures the direction and strength of linear association
between X and Y
Maximum likelihood estimator of π is
the Pearson Correlation Coefficient:
π =β(ππβπ)(ππβπ)
ββ(ππβπ)2β(ππβπ)
2=β(ππβπ)(ππβπ)
(πβ1)π ππ π
Inference on π:
If X and Y are both Normal,
π»0: π = 0 vs. π»π: π β 0
π =πβπβ2
β1βπ2 ~ π‘(πβ2) under π»0: π = 0
critical value = Β±π‘(πβ2,πΌ 2β )
BSTT523: Kutner et al., Chapter 1 6
Spearman Rank Correlation Coefficient:
For X or Y non-Normal
π ππ= rank of ππ
π ππ= rank of ππ
π π = π π =π+1
2 means of ranks π ππ or π ππ
Spearman rank correlation coefficient is
ππ =β(π ππβπ π)(π ππβπ π)
ββ(π ππβπ π)2β(π ππβπ π)
2 , β1 β€ ππ β€ +1
π»0: There is no association between X and Y
π»π: There is association between X and Y
π =ππ βπβ2
β1βππ 2 ~ π‘(πβ2) under π»0
critical value = Β±π‘(πβ2,πΌ 2β )
BSTT523: Kutner et al., Chapter 1 7
The Simple Linear Regression Model
ππ = π½0 + π½1ππ + ππ , i = 1, . . . ,n observations
ππ value of response for ith observation
ππ value of predictor for ith observation
Population parameters (unknown):
π½0 Population intercept
π½1 Population regression coefficient
ππ is ith random error term
Mean: πΈ(ππ) = 0
Variance: πππ(ππ) = π2
Independence: ππ and ππ are uncorrelated for π β π
Normality: ππ~π(0, π2) π. π. π. for all i
β ππ = π½0 + π½1ππβ πΆπππ π‘πππ‘
+ ππβπ πππππ,π.π.π.π(0,π2)
β πΈ(ππ) = πΈ(π½0 + π½1ππ + ππ) = π½0 + π½1ππ
πππ(ππ) = πππ(π½0 + π½1ππ + ππ) = πππ(ππ) = π2
β ππ ~ π(ππ, π2) π. π. π.
where ππ = π½0 + π½1ππ
BSTT523: Kutner et al., Chapter 1 8
How to obtain οΏ½ΜοΏ½0 and οΏ½ΜοΏ½1, estimates for π½0 and π½1?
Least Squares Estimators (LSE):
LSEs minimize the sum of squared deviations of ππ from πΈ(ππ)
Least Squares Criterion:
π = β [ππ β πΈ(ππ)]2π
π=1 = β [ππ β (π½0 + π½1ππ)]2π
π=1
Minimize Q: set first derivatives w.r.t. each parameter = 0
First derivatives are:
ππ
ππ½0= β2β(ππ β π½0 β π½1ππ) (1)
ππ
ππ½1= β2βππ(ππ β π½0 β π½1ππ) (2)
Normal Equations: set (1)=0 and (2)=0; call solutions οΏ½ΜοΏ½0 and οΏ½ΜοΏ½1
β2β(ππ β οΏ½ΜοΏ½0 β οΏ½ΜοΏ½1ππ) = 0
β2βππ(ππ β οΏ½ΜοΏ½0 β οΏ½ΜοΏ½1ππ) = 0
β
βππ = ποΏ½ΜοΏ½0 + οΏ½ΜοΏ½1βππ
βππππ = οΏ½ΜοΏ½0βππ + οΏ½ΜοΏ½1βππ2
BSTT523: Kutner et al., Chapter 1 9
Solution to Normal Equations:
Least Squares Estimators (LSE):
οΏ½ΜοΏ½π =β(πΏπβπΏ)(ππβπ)
β(πΏπβπΏ)π
οΏ½ΜοΏ½π = π β οΏ½ΜοΏ½ππΏ
Properties of LSE:
Unbiased estimators (accuracy)
πΈ(οΏ½ΜοΏ½0) = π½0 , πΈ(οΏ½ΜοΏ½1) = π½1
Minimum variance (precision)
Robust against Normality assumption
Note:
functions are called βestimatorsβ
calculated values from data are called βestimatesβ
Interpretation:
Intercept (οΏ½ΜοΏ½0) Value of Y when X=0
(not always meaningful!)
Slope (οΏ½ΜοΏ½1) Average change in Y per unit increase in X
βEffectβ of X on Y; βregression coefficientβ
BSTT523: Kutner et al., Chapter 1 10
Example: X = gestational age (wks), Y = head circumference (cms)
Formula for least squares regression line is:
Y = 3.91 + 0.78X
Intercept: not meaningful! (extrapolation to X = 0 weeks)
Slope: For every increase of one week gestational age,
there is an increase of about 0.78 cm head circumference.
15
20
25
30
35
40
20 22 24 26 28 30 32 34 36
Y =
He
ad C
ircu
mfe
ren
ce (
cm)
X = Gestational Age (weeks)
BSTT523: Kutner et al., Chapter 1 11
Another approach:
Method of Maximum Likelihood
The MLE maximizes the likelihood function
(the likelihood of the observed data, given the model parameters)
Q. Under which parameter values is the sample data
most likely to occur?
[see explanation of MLE on p.27-29]
For simple linear regression:
ππ = ππ β π½0 β π½1ππ ~ π(0, π2)
β π(ππ) =1
β2ππππ₯π {β
1
2π2(π¦π β π½0 β π½1π₯π)
2}
β likelihood = πΏ = β π(ππ)ππ=1
β πππππΏ = ππ {1
(2ππ2)π 2β ππ₯π [β1
2π2β (π¦π β π½0 β π½1π₯π)
2ππ=1 ]}
= βπ
2ππ(2ππ2) β
1
2π2β (π¦π β π½0 β π½1π₯π)
2ππ=1
ππππππΏ
ππ½0= 0 ,
ππππππΏ
ππ½1= 0 ,
ππππππΏ
ππ= 0
β same solution as LSE (please prove for yourself!)
same nice properties
BSTT523: Kutner et al., Chapter 1 12
After calculating the fitted regression line:
Fitted value οΏ½ΜοΏ½π οΏ½ΜοΏ½π = οΏ½ΜοΏ½0 + οΏ½ΜοΏ½1ππ
On the fitted line for the value ππ
Fitted Y- values are estimates of the Mean Response Function
οΏ½ΜοΏ½π is an unbiased estimator of the mean response at ππ
The fitted line is an unbiased estimator of the mean response
function
Note: the point (π, π) is ALWAYS on the fitted regression line,
i.e,
π = οΏ½ΜοΏ½0 + οΏ½ΜοΏ½1π
BSTT523: Kutner et al., Chapter 1 13
The ith residual ππ:
ππ = ππ β οΏ½ΜοΏ½π = ππ β (οΏ½ΜοΏ½0 + οΏ½ΜοΏ½1ππ)
Β· it is the vertical distance between (ππ , ππ) and (ππ , οΏ½ΜοΏ½π)
Β· it is the estimate of the ith error term, ππ = ποΏ½ΜοΏ½
Β· β ππ = 0ππ=1
Proof:
βππ = β[ππ β (οΏ½ΜοΏ½0 + οΏ½ΜοΏ½1ππ)]
= βππ β ποΏ½ΜοΏ½0 β οΏ½ΜοΏ½1βππ
= 0 (by normal equation 1)
BSTT523: Kutner et al., Chapter 1 14
Error Sum of Squares:
πππΈ = β (ππ β οΏ½ΜοΏ½π)2π
π=1 = β ππ2π
π=1
minimum when residuals are from LSE or MLE.
associated degrees of freedom ππ = π β 2
(generally ππ = π β π where p = # of parameters in the model)
Mean Squared Error: unbiased estimator of π2
πππΈ =πππΈ
ππ=
πππΈ
πβ2 πΈ(πππΈ) = π2
BSTT523: Kutner et al., Chapter 1 15
Example: X = gestational age and Y = head circumference
100 observations
scatterplot, fitted line, fitted values
15
20
25
30
35
40
20 22 24 26 28 30 32 34 36
Y =
He
ad C
ircu
mfe
ren
ce (
cm)
X = Gestational Age (weeks)
BSTT523: Kutner et al., Chapter 1 16
EXCEL: SUMMARY OUTPUT
Regression Statistics Multiple R 0.780691936 R Square 0.609479899 Adjusted R
Square 0.605495 Standard
Error 1.590413353 Observations 100
ANOVA df SS MS F Significance F
Regression 1 386.8673658 386.8674 152.9474 1.00121E-21 Residual 98 247.8826342 2.529415
Total 99 634.75
Coefficients Standard
Error t Stat P-value Intercept 3.914264144 1.82914689 2.13994 0.034842 X Variable 1 0.780053162 0.063074406 12.36719 1E-21
BSTT523: Kutner et al., Chapter 1 17
SAS output:
The REG Procedure
Model: MODEL1
Dependent Variable: headcirc
Number of Observations Read 100
Number of Observations Used 100
Analysis of Variance
Sum of Mean
Source DF Squares Square F Value Pr > F
Model 1 386.86737 386.86737 152.95 <.0001
Error 98 247.88263 2.52941
Corrected Total 99 634.75000
Root MSE 1.59041 R-Square 0.6095
Dependent Mean 26.45000 Adj R-Sq 0.6055
Coeff Var 6.01290
Parameter Estimates
Parameter Standard
Variable DF Estimate Error t Value Pr > |t|
Intercept 1 3.91426 1.82915 2.14 0.0348
gestage 1 0.78005 0.06307 12.37 <.0001