lecture 6: multiple regression laura mcavinue school of psychology trinity college dublin
Post on 21-Dec-2015
216 views
TRANSCRIPT
Lecture 6:Multiple Regression
Laura McAvinue
School of Psychology
Trinity College Dublin
Previous Lectures
• Relationship between two variables
• Correlation– Measure of strength of association between two variables
• Simple linear regression– Measure of the ability of one variable (X) to predict the other
variable (Y)– Computes a regression equation that describes the relationship
between the response variable (Y) and the predictor variable (X) by expressing Y as a function of X
Multiple Regression
• Used when there is more than one predictor variable
• Two purposes– To predict Y, given a combination of predictor
variables– To assess the relative importance of each predictor
variable in explaining the response variable Y
Regression Equations
Simple Linear Regression ˆ Y bX a
Multiple Regression Y = a + b1X1 + b2X2 +… + bkXk
b1 = Regression coefficient for first predictor variable, X1
b2 = Regression coefficient for second predictor variable, X2
a = Intercept, value of Y when all predictor variables are 0
Statistical Models
• Running a regression analysis is not a simple matter of inputting data, clicking a button and obtaining a ‘fixed’ model of the data
• You create the model of your data– Subjective process in many respects– You shape the model you create– Your job is to create the model that best describes the data
Multiple Regression
• Assessing the relative contribution of each predictor variable to the response variable– Which variable contributes most?– Which is the second biggest predictor?– Which variables don’t seem to contribute to prediction?
• Problem– The order with which you input the variables into the analysis
influences the model– Variable entered first is attributed more variance– By the time the last variable is entered, there might be very little
variance left to explain
Variance in Y
related to X2
Variance in Y
related to X1
Variance in Y related to shared variance between X1 & X2
Which variable gets credit, X1 or X2?
Multiple CorrelationThe predictor variables are correlated with each other and with the response variableWhich predictor variable gets credit for this shared variance?
Different Methods of Multiple Regression
• Hierarchical Regression
• Entry / Standard Regression
• Sequential Methods– Forward Addition– Backward Selection– Stepwise
• Combinatorial Approach
Hierarchical Regression
• You decide the order in which the variables are entered
• Based on theory / prior research
• Allows you to assess whether each predictor adds anything to the model, given the predictors that are already in the model
Entry / Standard Regression
• Computer package enters all predictor variables into the model simultaneously– Creates a regression equation including all predictor variables– Allows us to assess the unique contribution of each predictor
variable when all other variables are held constant
• Advantages & Disadvantages– Easy to see which variables significantly predict the response
variable– May not create the best model for predicting Y as it will include
variables that don’t significantly predict Y
Sequential Models
• Aim to create the ‘best model’– The combination of variables that best predicts the response
variable
• Build several models in a series of steps, adding or deleting variables at each step, depending on their contribution to predicting the response variable
• Final model includes only variables which significantly and uniquely predict the response variable
Sequential Methods
• Forward Addition
• Begins with only one variable in the model– The variable that makes the biggest contribution to the response
variable (highest r)
• Adds the variable with the next highest contribution
• Continues to add variables until there are no more variables that make a significant contribution to the response variable over and above the variables that are already in the equation
Sequential Methods
• Backward Selection• Begins with all predictor variables in the model and
successively deletes variables until only significant ones remain
• Stepwise Regression• Similar to previous two but more versatile• Generally moves forward, adding significant variables,
but can move backward to eliminate a variable if it no longer significantly predicts when another variable is added
Sequential Methods
• Drawbacks– Inclusion in the model depends on mathematical
criterion rather than psychological theory or research
– Variable selection could depend upon tiny differences in correlation between each predictor variable and the response variable
• Slight numerical differences could therefore lead to major differences in theoretical interpretation
– Difficult to replicate results
Combinatorial Methods
• Best Subsets Method
• Computes models with all possible combinations of the predictor variables and chooses the model that explains most variance in the response variable
Critical Considerations for MR
• Sample size
• Distribution requirements: Residuals– Data must be normally distributed
• Outliers
• Multi-collinearity
Sample Size
• Ratio of cases to predictors should be substantial
• Stevens (1996) advised about 15 participants per predictor variable– Size matters: The more people in your sample the better the
chance of the results being replicated
• However, an even bigger ratio is needed when– Response variable has skewed data distribution– Poor reliability in measures - substantial measurement error
reduces size of true relationships of variables– Stepwise methods (45-50 participants per predictor)
Residuals
• Recall the Method of Least Squares– Fits the regression line by minimising the prediction error of the
line
– Minimises the sum of squares of the residuals (Y-Y’)2 • Fits a line of the form
Y = a + b1X1 + b2X2 +… + bkXk + e
Assumes: Y = Fit + noise
Residuals
• Method of Least Squares models the noise (e) in the data using the normal distribution
– Assumes the noise is normally distributed with mean of 0 and
variance σ2
• If this assumption is violated, the results of your regression analysis may not be valid
– You need to check this by plotting the residuals– Standardised Residual Plots
• Histogram• Normal Probability Plot
Histogram
0.00 200.00 400.00 600.00 800.00 1000.00
variable
0
10
20
30
40
50
60
70
Fre
qu
ency
Mean = 72.5765Std. Dev. = 123.52254N = 85
Normal Probability Plot
Plots the residual value that was obtained for eachdata point (observed) against the value you wouldexpect if the residuals werenormally distributed (expected)
Should be a straight diagonalline
Outliers
• Data points that lie far from the rest of the data and have large residuals
• Big influence on regression analysis
• You can check for outliers
– Scatterplots examining relationship between response variable and predictor variables separately
– Casewise diagnostics in SPSS– Plots of the standardised residuals
Plot of Standardised Residuals
Plot of Standardized Predicted ValuesX
Studentised Deleted Residuals(Residual scores divided by their standard deviation, which is calculatedleaving out any suspiciously outlyingdata points)
Based on the assumption of normality:99.9% of residuals should lie within+3 & - 3 standard deviations
Any point outside this range is an outlier
-3 -2 -1 0 1 2 3
0
-1
1
3
2
4
-2
-3
Multi-Collinearity
• Occurs when predictor variables are highly correlated with one another
– High bivariate correlations (.7 / .8 or above)– High multivariate correlation
• Not a desired feature of the dataset
– Some predictor variables are redundant– Statistically, leads to unstable results
Multi-Collinearity
• To assess whether multi-collinearity is present
– Examine the bivariate correlations between predictor variables– Tolerance Statistic
• 1 – Multiple correlation (correlation between each predictor variable and all others)
• If low, then multiple correlation must be high and multi-collinearity is a problem
• Solution
– Leave out one of the predictor variables– Combine two highly correlated predictor variables
Let’s take an example
• Interested in a theory which suggests that a person’s level of optimism (X1) and the social support (X2) that he/she has in his/her life predicts how long he/she will survive (Y) after being diagnosed with cancer.
• Three steps to Regression Analysis:
– A. Examine the relationship between the predictor and response variables separately
– B. Perform and interpret the multiple regression
– C. Assess the appropriateness of the regression analysis
Let’s take an example
• Open the following dataset• Software / Kevin Thomas / Multiple Regression Dataset• Run Correlations between…
– Survival & Optimism– Survival & Social Support
Correlations
1 .599**
.000
200 200
.599** 1
.000
200 200
Pearson Correlation
Sig. (2-tailed)
N
Pearson Correlation
Sig. (2-tailed)
N
survival in weeks
Optimism
survival inweeks LOT score
Correlation is significant at the 0.01 level (2-tailed).**.
Correlations
1 .326**
.000
200 200
.326** 1
.000
200 200
Pearson Correlation
Sig. (2-tailed)
N
Pearson Correlation
Sig. (2-tailed)
N
survival in weeks
Social Support
survival inweeks SS score
Correlation is significant at the 0.01 level (2-tailed).**.
Create Scatterplots & fit regression line
Graphs / Scatter / Simple Scatter / y = Survival, X = Predictor Variable
Fit regression line: Double click on chart, then Elements / Fit line at total
Step 2: The Multiple Regression
• Analyse, Regression, Linear
– Dependent variable: Survival– Independent variable: Social, optimism
• Method: Enter (gives a standard multiple regression)
• Statistics
– Regression Coefficients• Estimates • Model fit • Descriptives
Answer the questions on your worksheet
1. Does this model (i.e. combination of social support and optimism) significantly predict the response variable (survival in months)?
ANOVAb
528045.0 2 264022.487 67.733 .000a
767907.0 197 3898.005
1295952 199
Regression
Residual
Total
Model1
Sum ofSquares df Mean Square F Sig.
Predictors: (Constant), SS score, LOT scorea.
Dependent Variable: survival in weeksb.
Yes, F (2, 199) = 67.73, p < .001
Answer the questions on your worksheet
2. What percentage of variance in the response variable, survival in months, is explained by this model?
40.1%
Model Summary
.638a .407 .401 62.43401Model1
R R SquareAdjustedR Square
Std. Error ofthe Estimate
Predictors: (Constant), SS score, LOT scorea.
R Square adjusted = Estimate of the population proportion of variation in survival due to optimism & support
Penalises for number of variables in the model
Answer the questions on your worksheet
3. Write the regression equation
Survival in months = 3.67(optimism) + 12.99(social support) + 4.34
Coefficientsa
4.340 22.760 .191 .849
3.670 .367 .558 10.005 .000
12.987 3.226 .225 4.026 .000
(Constant)
LOT score
SS score
Model1
B Std. Error
UnstandardizedCoefficients
Beta
StandardizedCoefficients
t Sig.
Dependent Variable: survival in weeksa.
Answer the questions on your worksheet
4. What does this equation tell us about the relationship between months of survival and social support?
As social support increases by one unit, survival in months increases by almost 13 months
Coefficientsa
4.340 22.760 .191 .849
3.670 .367 .558 10.005 .000
12.987 3.226 .225 4.026 .000
(Constant)
LOT score
SS score
Model1
B Std. Error
UnstandardizedCoefficients
Beta
StandardizedCoefficients
t Sig.
Dependent Variable: survival in weeksa.
Answer the questions on your worksheet
5. Do both variables significantly predict survival in months?
Yes, for optimism, t = 10, p < .001 & for social support, t = 4.026, p < .001
Coefficientsa
4.340 22.760 .191 .849
3.670 .367 .558 10.005 .000
12.987 3.226 .225 4.026 .000
(Constant)
LOT score
SS score
Model1
B Std. Error
UnstandardizedCoefficients
Beta
StandardizedCoefficients
t Sig.
Dependent Variable: survival in weeksa.
Answer the questions on your worksheet
6. Which of the predictor variables contributes most to the response variable?
Optimism has a Beta value of .558 and so, contributes more than social support, which has a Beta value of .225
Coefficientsa
4.340 22.760 .191 .849
3.670 .367 .558 10.005 .000
12.987 3.226 .225 4.026 .000
(Constant)
LOT score
SS score
Model1
B Std. Error
UnstandardizedCoefficients
Beta
StandardizedCoefficients
t Sig.
Dependent Variable: survival in weeksa.
Beta = Standardized Regression Coefficient (B / Std. Error)
Can be used to compare strength of contribution of predictor variables
Answer the questions on your worksheet
7. Use the regression equation to make the following prediction: If a person has an optimism score of 10 and a social support score of 2, how long would you expect them to survive?
Survival in months = 3.67(optimism) + 12.99(social support) + 4.34
Survival in months = 3.67(10) + 12.99(2) + 4.34
Survival in months = 36.7 + 25.98 + 4.34
Survival in months = 67.02
67 months!
Answer the questions on your worksheet
8. What is the standard error of this prediction?
62.43 months
Model Summary
.638a .407 .401 62.43401Model1
R R SquareAdjustedR Square
Std. Error ofthe Estimate
Predictors: (Constant), SS score, LOT scorea.
Step 2: Assess the appropriateness of the Analysis
• Distribution of Residuals• Outliers• Multi-collinearity
• Re-run regression but this time…– Statistics
• Collinearity Diagnostics• Residuals, casewise diagnostics
– Outliers outside 3 standard deviations
– Plots• Histogram• Normal Probability Plot• Plot of Standardized Predicted Values (Y: ZPRED) by
Studentized Deleted Residuals (X: SDRESID)
Distribution of Residuals
Outliers
Residuals Statistics a
59.9595 326.4863 193.2000 51.5121 200-182.7696 162.2596 5.400E-15 62.1195 200
-2.587 2.587 .000 1.000 200
-2.927 2.599 .000 .995 200
Predicted ValueResidualStd. Predicted Value
Std. Residual
Minimum Maximum Mean Std. Deviation N
Dependent Variable: length of survival (months)a.
All residuals lie within -3 and 3 standard deviations
Note that you expect 1% of cases to lie outside this area so in a large sample, if you have one or two, that could be ok
Outliers
All residuals lie within -3 and 3 standard deviations
Multi-Collinearity
Bivariate correlations seem to be low (r = .182) even though significant (p = .01)
Tolerance is high, meaning that the multiple correlation is small, meaning that multi-collinearity is not a feature of this dataset
Correlations
1 .182**
.010
200 200
.182** 1
.010
200 200
Pearson Correlation
Sig. (2-tailed)
N
Pearson Correlation
Sig. (2-tailed)
N
SS score
LOT score
SS score LOT score
Correlation is significant at the 0.01 level (2-tailed).**.
Coefficientsa
4.340 22.760 .191 .849
3.670 .367 .558 10.005 .000 .967 1.034
12.987 3.226 .225 4.026 .000 .967 1.034
(Constant)
LOT score
SS score
Model1
B Std. Error
UnstandardizedCoefficients
Beta
StandardizedCoefficients
t Sig. Tolerance VIF
Collinearity Statistics
Dependent Variable: survival in weeksa.
Summary
• Multiple Regression– To predict Y given a combination of predictor variables– To assess the relative importance of each predictor variable in
explaining the response variable
• Statistical modelling– Different Methods
• Three steps– Examine the relationship between the predictor and response
variables separately– Perform and interpret the multiple regression– Assess the appropriateness of the regression analysis
• There are a number of critical considerations