inference for regression parameters, correlation & analysis...

Inference for Regression Parameters, Correlation &Analysis of Variance

9.4, 9.5 & 11

Cathy Poliak, [email protected] Fleming 11c

Department of MathematicsUniversity of Houston

Lecture 13 - 3339

Cathy Poliak, Ph.D. [email protected] Office Fleming 11c (Department of Mathematics University of Houston )Sections 9.4, 9.5 & Chapter 11 Lecture 13 - 3339 1 / 41

Outline

1 Inference for Regression Parameters

2 F-test

3 Estimating Correlation

4 Comparing More Than Two Means

5 ANOVA

6 Pairwise Tests


Least-Squares regression

The least-squares regression line (LSRL) of Y on X is the linethat makes the sum of the squares of the vertical distances of thedata points from the line as small as possible.The linear regression model is: Y = β0 + β1x + ε

I Y is dependent variable (response).I x is the independent variable (explanatory).I β0 is the population intercept of the line.I β1 is the population slope of the line.I ε is the error term which is assumed to have mean value 0. This is

a random variable that incorporates all variation in the dependentvariable due to factors other than x .

I The variability: σ of the response y about this line. More precisely,σ is the standard deviation of the deviations of the errors, εi in theregression model.

We will gather information from a sample so we will have the leastsquares estimates model: Y = β0 + β1x .


Principle of Least Squares

The vertical deviation of the point (xi , yi) from the line y = b0 + b1x is

hieght of point − height of line = yi − (b0 + b1xi)

The sum of the square vertical deviations from the points(x1, y1), (x2, y2), . . . , (xn, yn) to the line is then

f (b0,b1) =n∑

i=1

[yi − (b0 + b1xi)]2

The point estimates of β0 and β1, denoted by β0 and β1 and called theleast squares estimates, are those values that minimize f (b0,b1).


Estimating the Regression Parameters

In the simple linear regression setting, we use the slope b1 andintercept b0 of the least-squares regression line to estimate theslope β1 and intercept β0 of the population regression line.The standard deviation, σ, in the model is estimated by theregression standard error

s =

√∑(yi − yi)2

n − 2=

√∑all residuals2

n − 2

Recall that yi is the observed value from the data set and yi is thepredicted value from the equation.In R s is the called the Residual Standard Error in the lastparagraph of the summary.


The Least - Squares Estimates

Recall ei = observed Y - predicted Y is the i th residual. Think of itas an estimate of the unobservable true random error εi .

The method of least squares selects estimators β0 and β1 thatminimizes the residual sum of squares:

SS(resid) = SSE =n∑

i=1

e2i =

n∑i=1

(Yi − Yi

)Where the estimate of the slope coefficient β1 is:

β1 =

∑(xi − x)(yi − y)∑

(xi − x)2 =Sxy

Sxx

The estimate for the intercept β0 is:

β0 = y − β1x


t Test for Significance of β1

HypothesisH0 : β1 = β10 versus Ha : β1 6= β10

Usually β10 = 0Test statistic

t =observed− hypothesized

standard deviation of observedobserved = b1

hypothesized = 0

standard error = SEb1 =s√∑

(xi − x)2

With degrees of freedom df = n − 2.P-value: based on a t distribution with n − 2 degrees of freedom.Decision: Reject H0 if p-value ≤ α.Conclusion: If H0 is rejected we conclude that the explanatoryvariable x can be used to predict the response variable y .


Conditions for regression inference

The sample is an SRS from the population.

There is a linear relationship in the population.

The standard deviation of the responses about the population lineis the same for all values of the explanatory variable.

The response varies Normally about the population regressionline.


Example: Real Estate

Can the size of the house help us predict the list price?Explanatory Variable: Price (List Price of a home per $1,000)

Response Variable: Size (Size of the house in square feet)

Data: 55 houses for sale in a certain neighborhood https://www.math.uh.edu/~cathy/Math3339/data/homes.xlsx.


https://www.math.uh.edu/~cathy/Math3339/data/homes.xlsx

https://www.math.uh.edu/~cathy/Math3339/data/homes.xlsx

R output

> attach(homes)> plot(Size,Price)> homes.lm=lm(Price~Size)> summary(homes.lm)

Call:lm(formula = Price ~ Size)

Residuals:Min 1Q Median 3Q Max

-99.157 -31.898 4.873 36.389 105.515

Coefficients:Estimate Std. Error t value Pr(>|t|)

(Intercept) 92.529120 25.701237 3.60 7e-04 ***Size 0.099106 0.008001 12.39 <2e-16 ***---Signif. codes:0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 46.69 on 53 degrees of freedomMultiple R-squared: 0.7432,Adjusted R-squared: 0.7384F-statistic: 153.4 on 1 and 53 DF, p-value: < 2.2e-16


Height

Because elderly people may have difficulty standing to have theirheights measured, a study looked at predicting overall height fromheight to the knee. Here are data (in centimeters, cm) for five elderlymen:

Knee Height (cm) 57.7 47.4 43.5 44.8 55.2Overall Height(cm) 192.1 153.3 146.4 162.7 169.1

1. We want to test if the relationship of the measurement of theheight is more than double that of the measurement from the floorto the knee. Test H0 : β1 = 2 versus Ha : β1 > 2.

2. Give a conclusion for the relationship between using knee lengthto predict overall height.


Confidence Intervals for β1

If we want to know a range of possible values for the slope we can usea confidence interval.

Remember confidence intervals are

estimate± t∗ × standard error of the estimate

Confidence interval for β1 is

b1 ± tα/2,n−2 × SEb1

Where t∗ is from table D with degrees of freedom n − 2 where n =number of observations.In R we can get this by confint(name.lm,level = 0.9) For a 90%confidence interval.> confint(shelf.lm,level = 0.9)

5 % 95 %(Intercept) 105.519121 184.48088space 4.516724 10.28328


Determine a 90% Confidence Interval for CoffeeExample

From the output in R:


(Intercept) 145.000 21.783 6.657 5.66e-05 ***space 7.400 1.591 4.652 0.000906 ***


Definitions of Regression Output

1. The error sum of squares, denoted by SSE is

SSE =∑

(yi − yi)2

2. A quantitative measure of the total amount of variation in observedvalues is given by the total sum of squares, denoted by SST .

SST =∑

(yi − y)2

3. The regression sum of squares, denoted SSR is the amount oftotal variation that is explained by the model

SSR =∑

(yi − y)2

4. The coefficient of determination, r2 is given by

r2 =SSRSST


Finding these values using R

> anova(shelf.lm)Analysis of Variance Table

Response: soldDf Sum Sq Mean Sq F value Pr(>F)

space 1 20535 20535 21.639 0.0009057 ***Residuals 10 9490 949---Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1


F-distribution

The F distribution with ν1 degrees of freedom in the numeratorand ν2 degrees of freedom in the denominator is the distribution ofa random variable

F =U/ν1

V/ν2,

where U ∼ χ2(df = ν1) and V ∼ χ2(df = ν2) are independent.That F has this distribution is indicated by F ∼ F (ν1, ν2).Notice U = SSR

σ2 ∼ χ2(df = 1) and V = SSEσ2 ∼ χ2(df = n − 2) are

independent.Let MSE = SSE/(n − 2) and MSR = SSR/1. Then

F =MSRMSE

=SSR/1

SSE/(n − 2)∼ F (1,n − 2)

Then we can use the F-distribution to test the hypothesisH0 : β1 = 0 versus Ha : β1 6= 0.


F-test for Shelf Space

Analysis of Variance Table

Response: soldDf Sum Sq Mean Sq F value Pr(>F)

space 1 20535 20535 21.639 0.0009057 ***Residuals 10 9490 949---Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Note: F = t2 and the p-value is the same.


Estimating the Correlation Coefficient

Notice that β1 = ρσyσx

Where ρ is the population correlation, and σis the population standard deviation.

This means that the regression null hypothesis H0 : β1 = 0 isequivalent to H0 : ρ = 0.

Recall that we can use the sample correlation, to estimate theslope,

β1 = Rsy

sxThus this can be rewritten as:

R = β1sx

sy

Recall the student t statistic for testing H0 : β1 = 0,

t =β1s√∑(xi − x)2


Test Statistic for H0 : ρ = 0

From the test statistic of H0 : β1 = 0 we get the test statistic for testingH0 : ρ = 0.

T =R√

n − 2√1− R2

Therefore, if X and Y have a bivariate normal distribution, a test ofH0 : ρ = 0 against Ha : ρ 6= 0 is to reject H0 when |T | > tα/2(n − 2)or when the p-value based on the the student-t distribution TDist(df = n − 2) is smaller than the chosen significance level.

Note: We have three equivalent tests in the case of a bivariatenormal distribution:

1. the T test for H0 : β1 = 02. the T test for H0 : ρ = 03. the F test for significance of regression


Test H0 : ρ = 0 for Shelf Space Example

Given: R = 0.827 and n = 12 find the test statistic T and the p-value.


R output> cor.test(sold,space)

Pearson’s product-moment correlation

data: sold and spacet = 4.6517, df = 10, p-value = 0.0009057alternative hypothesis: true correlation is not equal to 095 percent confidence interval:0.4817419 0.9500109sample estimates:cor0.8270006

> summary(shelf.lm)

Call:lm(formula = sold ~ space)

Residuals:Min 1Q Median 3Q Max

-42.00 -26.75 5.50 21.75 41.00


(Intercept) 145.000 21.783 6.657 5.66e-05 ***space 7.400 1.591 4.652 0.000906 ***---Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 30.81 on 10 degrees of freedomMultiple R-squared: 0.6839,Adjusted R-squared: 0.6523F-statistic: 21.64 on 1 and 10 DF, p-value: 0.0009057


Weight Loss

From: http://sphweb.bumc.bu.edu/otlt/MPH-Modules/BS/BS704_HypothesisTesting-ANOVA/BS704_HypothesisTesting-Anova_print.html

Is there a difference in the mean weight loss among differentprograms?A clinical trial is run to compare weight loss programs andparticipants are randomly assigned to one of the comparisonprograms and are counseled on the details of the assignedprogram.Participants follow the assigned program for 8 weeks.Three popular weight loss programs are considered.

I Low calorie diet.I Low fat dietI Low carbohydrate dietI Control group


http://sphweb.bumc.bu.edu/otlt/MPH-Modules/BS/BS704_HypothesisTesting-ANOVA/BS704_HypothesisTesting-Anova_print.html



Results

Response variable = weight loss = weight at the end of 8 weeks -weight at beginning of the studyThe observed weight losses of twenty people in this study are asfollows:

Low Calorie Low Fat Low Carbohydrate Control8 2 3 29 4 5 26 3 4 -17 5 2 03 1 3 3

Is there a statistically significant difference in the mean weight lossamong the four diets?


Box Plots of Weight Loss

calorie carb control fat

02

46

8


Hypotheses

We want to know if there is a "statistically significant difference" inthe mean weight loss among the four diets.

Null hypothesis: mean weight loss is the same among the fourdiets

H0 : µcalorie = µcarb = µcontrol = µfat

Alternative hypothesis is that at least one of the mean weight lossamong the four diets is different.

Rejecting H0 is evidence that the mean of at least one group isdifferent from the other means.


Assumptions

The assumptions of analysis of variance are the same as those of thetwo sample t-test, but they must hold for all k groups.

The measurements in every group is a SRS.

We have a Normal distribution for each of the k populations.

The variance is the same in all k populations.


Analysis: ANOVA

ANalysis Of VArianceWe can estimate how much variation among group means oughtto be present from sampling error alone if the null hypothesis istrue.ANOVA lets us determine whether there is more variance amongthe sample means than we would expect by chance alone.


The Formulas

Let Xi. =∑ni

j=1 Xij

nidenote the average of the observations in the i th

group.

Let N =∑M

i=1 ni , be the total number of observations in all the Mgroups.

Let X.. =∑M

i=1 ni Xi.N be the average of all the observations (the

grand average)

The treatment sum of squares (between groups) isSS(betw) =

∑Mi=1 ni(Xi. − X..)2.

The error sum of squares (residual) isSSE =

∑Mi=i∑ni

n=1(Xij − Xi.)2 =

∑Mi=1(n − 1)S2

i


Diets Example

Low Calorie Low Fat Low Carbohydrate Control8 2 3 29 4 5 26 3 4 -17 5 2 03 1 3 3


The F Test

The mean square for treatments is MSTr = SSTrM−1 .

The mean square for error is MSE = SSEN−M .

The test statistic is F = MSTrMSE .

This test statistic has an F distribution with parameters "numeratordegrees of freedom" = M - 1 and "denominator degrees offreedom" = N - M. Where N is the total number of observationsand M is the number of groups.


The ANOVA Table

Source of degrees of Sum of Mean FVariation freedom Squares SquareTreatments M - 1 SSTr MSTr MSTr

MSEError N - M SSE MSETotal N - 1 SST


ANOVA for Diets

Source of degrees of Sum of Mean FVariation freedom Squares SquareDiets

Error

Total

p-value = 1 - pf(f,M - 1, N - M)


R Code

> diet.lm=lm(Loss~Diet,data=diet)> anova(diet.lm)Analysis of Variance Table

Response: LossDf Sum Sq Mean Sq F value Pr(>F)

Diet 3 75.75 25.25 8.5593 0.001278 **Residuals 16 47.20 2.95---Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1


Tukey’s Method (The T Method)

The T-method is used to determine which pair (or pairs) of meansdiffers significanctly.

1. Select α, determine Qα,k ,N−k in R it is qtukey(1− α, k ,N − k )where, k = number of groups and N = total sample size.

2. Calculate w = Qα,k ,N−k√

MSE/j . Where j = the number ofelements in each group.

3. List the sample means in increasing order and underline thosepairs that differ by less than w .

4. Any pair of sample means not underscored by the same linecorresponds to a pair of population or treatment means that arejudged significantly different.


Tukey’s Method for Diet Example


Multiple Comparisons

If our p-value is small for the ANOVA F test, this implies that atleast one of the means is different from the other.

Which one(s) are different?

We could do a t-test for each pair of means.

Problem: when we do multiple t-tests our P(Type 1 error) becomesgreater than α.

Solution: There are methods of adjustments to reduce thesignificance level of the pairwise test enough so that theprobability of one or more type I errors in the whole set ofcomparisons is less than α.


The Bonferroni Method

The Bonferroni Method of adjustments reduces the significancelevel for the pairwise test to α/k , where k is the number ofcomparisons.R Code:> attach(diet)> pairwise.t.test(Loss,Diet,"bonferroni")

Pairwise comparisons using t tests with pooled SD

data: Loss and Diet

calorie carb controlcarb 0.05695 - -control 0.00083 0.35914 -fat 0.02632 1.00000 0.70193

P value adjustment method: bonferroni


Example MPG

Is there a difference in the average miles per gallon for different makesof automobiles? The following table shows the mean mpg of threedifferent makes of automobiles. The data is on the data sets list calledmpg.

Make n X SHonda 5 29.9 1.468Toyota 6 33.04 2.1173Nissan 4 29.3 1.3115


inference for regression parameters, correlation & analysis...

Documents