multiple regression with p > 2...
TRANSCRIPT
MR and Multiple Predictors 11.1
CHAPTER 11:
MULTIPLE REGRESSION WITH P > 2 PREDICTORS
The MR Equation with Multiple Predictors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
The Overall Relationship . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
Significance of Individual Predictors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
Strength of Unique Contribution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
A Four Predictor Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
Overall Regression Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
Unique Contribution of Predictors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
Using SPSS Menus for Regression with Multiple Predictors . . . . . . . . . . . . . . . . . . . . . 14
Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
MR and Multiple Predictors 11.2
DATA LIST FREE / subj grde abil stdy wght.BEGIN DATA 1 83 112 22 133 2 75 113 16 128 3 70 103 21 154 4 71 106 26 189 5 48 67 27 136 6 81 103 18 157 7 48 95 20 138 8 62 73 27 153 9 61 121 14 137 10 58 85 22 12911 80 123 12 120 12 66 89 18 12113 81 114 25 167 14 61 71 29 16815 59 96 17 143 16 41 75 19 16617 69 112 20 146 18 63 108 13 11519 67 120 14 146 20 71 100 16 14421 54 88 20 154 22 72 107 18 15423 69 86 25 115 24 71 92 22 140END DATA.
Box 11.1. Study Data with Three Predictors.
Previous examples of multiple regression have involved designs with only two predictors,
so the essential question has been whether one, neither, or both predictors contribute to the
prediction of y, and if so, how to apportion the variability in y to the two predictors. The basic
methods that we have used also apply to studies with more than two variables, but the analytic
and conceptual problems increase. This chapter begins to examine these issues. First, let us see
how the methods developed in preceding chapters can be extended to multiple predictor studies.
Earlier we discussed a hypothetical study of the relation between grades, ability, and
study time. Equally-hypothetical critics of the study questioned the conclusion that study time
had a positive effect on grades
controlling for ability. The critics
argued that statistics can prove
anything and that the complicated
methods used by the researchers
would probably show that weight of
students predicts grades. To answer
this charge, the researchers obtained
data for 24 high school students on
grades, ability, study time, and
weight. The data are shown in Box 11.1 (two cases per row).
Once data is entered into SPSS, regression and supplementary analyses are informative
about the relationship between grades and all three predictors, and about the unique contribution
of each individual predictor. We previously saw that ability and study time contributed
significantly to grades (see also later analyses in this chapter), so our focus will be primarily on
whether weight adds anything new to the prediction. At the same time, we want to appreciate
how information about the overall effect of predictors can be generalized to three or more
predictors.
MR and Multiple Predictors 11.3
THE MR EQUATION WITH MULTIPLE PREDICTORS
Determining a number of statistics becomes much more complicated when we have more
than two predictors. The slopes, for example, must take into consideration the correlation
between grades and all three predictors, as well as the three correlations among the three
predictors (a total of six rs). We therefore obtain these slopes from the SPSS output rather than
compute them directly. Box 11.2 shows the SPSS commands to produce and analyze the three
predictor equation. The only difference from earlier SPSS commands is that we now list four
variables (i.e., one dependent variable and three predictors), rather than three (i.e., one dependent
variable and two predictors). The /ENTER command instructs SPSS to use all but the dependent
variable as predictors (i.e., the remaining three variables). We examine in turn the overall
relationship and then the unique contribution of each predictor (focussing on Weight, our new
variable). Box 11.2 also shows the first three cases with the derived scores, as well as the
REGRESS /VARI = grde abil stdy wght /DEP = grde /ENTER /SAVE PRED(prdg.asw) RESI(resg.asw).
Model R R Square Adjusted R Square Std. Error of the Estimate 1 .762(a) .580 .517 7.6198780
Model Sum of Squares df Mean Square F Sig. 1 Regression 1605.374 3 535.125 9.216 .000(a) Residual 1161.251 20 58.063 Total 2766.625 23
Coefficients(a) Unstandardized Coefficients Standardized Coefficients t Sig. Model B Std. Error Beta 1 (Constant) -12.456 20.550 -.606 .551 ABIL .677 .131 1.013 5.169 .000 STDY 1.389 .501 .601 2.773 .012 WGHT -.111 .099 -.186 -1.120 .276
Residuals Statistics(a) Minimum Maximum Mean Std. Deviation N Predicted Value 46.203117 80.817650 65.875000 8.3545723 24 Residual -16.244148 16.237690 .000000 7.1055728 24
LIST. SUBJ GRDE ABIL STDY WGHT PRDG.ASW RESG.ASW 1.0000 83.0000 112.000 22.0000 133.000 79.08295 3.91705 2.0000 75.0000 113.000 16.0000 128.000 71.97994 3.02006 3.0000 70.0000 103.000 21.0000 154.000 69.26472 .73528
...
Box 11.2. SPSS Commands and Output for 3-Predictor Regression Equation.
MR and Multiple Predictors 11.4
�g.asw = -12.455881 + .6767a + 1.3894s -.1114w
Mean Std Dev SSGrades 65.8750 10.968 2766.831 � SSTot
�g.asw 65.8750 8.3546 1605.385 � SSReg
y-�g.asw .0000 7.1056 1161.260 � SSRes
Source df SS MS FRegression 3 1605.37420 535.12473 9.21635 p = .0005Residual 20 1161.25080 58.06254Total 23 2766.625
Rg.asw = .76175 = ry.� R2g.asw = .58026 = 1605.3742/2766.625
FR = (.761752/3)/((1-.761752)/(24-3-1)) = 9.216 = FReg
Box 11.3. MR Analysis of Expanded Study Time Results.
original variables.
The predicted score for subject (case) one is y’ = -12.456 + .677x112 + 1.389x22 -
.111x133 = 79.163 �79.08295, and the residual score is y - y’ = 83.00 - 79.08295 = 3.91705.
Predicted scores for all subjects could be calculated by substituting each subject’s scores on the
predictor variables into the best-fit prediction equation. These scores contain a certain amount of
the variability in grades, namely that variability predictable from ability, study time, and weight.
The remaining variability in grades is contained in the residual scores.
The Overall Relationship
The Unstandardized Coefficients column shows the slopes for each of the three
predictors, controlling for the other two predictors included in the equation, as well as the
intercept (i.e., the Constant in SPSS terms). This equation can be used to generate predicted and
residual scores, as just shown, which are saved by SPSS. The SAVE also generates descriptive
statistics for the predicted and residual scores, which can be used to calculate SSRegression and
SSResidual.
The statistical results from the multiple regression of grades on the three predictors are
expanded in Box 11.3. The analysis produces a regression equation, now with three predictors.
The regression coefficients indicate that grades are positively related to ability and study time,
but somewhat negatively related to weight. These regression coefficients take into consideration
the relations among the threee predictors as well as the relations between predictors and the
criterion
variable,
grades.
The
equation
partitions
the total
variability in
grades
(SSGrades =
MR and Multiple Predictors 11.5
2766.625) into variability that can be predicted (SSRegression = SS� = 1605.374) and variability that
cannot be predicted (SSResidual = SSy-� = 1161.251) on the basis of Ability, Study Time, and Weight.
These quantities are calculated in Box 11.3 from the standard deviations of the Grade, Predicted,
and Residual scores. The three variables predict 58.026% of the variation in grades, R2 =
SSRegression/SSTotal.= 1605.3742/2766.625 = .5803. This gives Rg.aws= �.58026 = .76175, also shown
in the regression output in Box 11.2.
Box 11.4 shows descriptive statistics and correlations for the original and derived scores.
Note first in Box 11.4 that the descriptive statistics provide again the necessary information for
computing SSTotal, SSRegression, and SSResidual. Note also that the correlations between the residual
scores and all three predictors are 0. All the variability in Grades related to the predictors has
been captured by the regression equation and put in the Predicted Grade scores. As shown
previously, the predicted and residual scores also share zero variability. With respect to the
partitioning of the variability in Grades into predicted and residual components, the correlation
between Grades and PRDG.ASW is .762, our multiple R as calculated above.
The correlation between Grades and RESG.ASW represents the remaining (unpredicted)
variability in Grades, SQRT(1 - R2) = SQRT(1 - .7622) = .648. That is, 1 - R2 = 1 - .5803 = .4197
� .64792 = r2 between GRDE and RESG.ASW. Another way of considering what has happened
to the variability in Grades is to note
that .7622 + .6482 = 1.00 (i.e., all the
variability in Grades can be
accounted for by what can be
predicted and what cannot be
predicted). Correlational analysis of
original and derived scores has
shown that residual scores account
for the variability in y not predicted
by the 3-equation model, that
predicted and residual scores are
independent, and that residual scores are independent of the three predictors. These observations
CORR grde abil stdy wght prdg.asw resg.asw /STAT = DESCR. Mean Std. Deviation N GRDE 65.875000 10.9675906 24 ABIL 98.291667 16.4117645 24 STDY 20.041667 4.7409334 24 WGHT 143.875000 18.3286720 24 PRDG.ASW 65.8750000 8.35457229 24 RESG.ASW .0000000 7.10557284 24
Correlations GRDE ABIL STDY WGHT PRDG.ASW ABIL .646 STDY -.139 -.649 WGHT -.049 -.127 .442 PRDG.ASW .762 .849 -.182 -.064 RESG.ASW .648 .000 .000 .000 .000
Box 11.4. Descriptive Statistics and Correlations for
Original and Derived Scores.
MR and Multiple Predictors 11.6
about the overall regression are analogous to those for a two-predictor model (or for a 10-
predictor model).
The significance of the overall relationship between grades and the three predictors is
determined using analysis of variance with either SSs or R2. Given MSRegression = SSRegression / p =
1605.3742 / 3 = 535.1247 and MSResidual = SSResidual / (n - p - 1) = 1161.2508 / (24 - 3 -1) = 58.0625,
F = MSRegression / MSResidual = 535.1247 / 58.06254 = 9.21635 = (R2 / p) / ((1 - R2) / (n - p - 1)) =
(.761752 / 3) / ((1 - .761752) / (24 - 3 - 1)), df = 3, 20. The results confirm that this strong relation
is significant, F = 9.216, p = .0005, and we can reject H0: �g.asw = 0.
Significance of Individual Predictors
As for two-predictor regression, the unique contributions of individual predictors can be
assessed in various ways. We focus on the statistics for weight, the new variable, but the general
points also apply to ability and study time. The Coefficients section of Box 11.2 shows several
REGRESS /VARI = grde abil stdy wght /STAT = DEFAULT ZPP CHANGE /DEP = grde /ENTER abil stdy /ENTER.
Model Summary R R Adjusted Std. Error of Change Statistics Square R Square the Estimate Model R Square Change F Change df1 df2 Sig. F Change 1 .744(a) .554 .511 7.6660706 .554 13.038 2 21 .000 2 .762(b) .580 .517 7.6198780 .026 1.255 1 20 .276
Model Sum of Squares df Mean Square F Sig. 1 Regression 1532.484 2 766.242 13.038 .000(a) Residual 1234.141 21 58.769 Total 2766.625 23
2 Regression 1605.374 3 535.125 9.216 .000(b) Residual 1161.251 20 58.063 Total 2766.625 23
Coefficients(a) Unstand Coeff Standard Coeff t Sig. Correlations Model B Std. Error Beta Zero-order Partial Part 1 (Constant) -19.737 19.613 -1.006 .326 ABIL .642 .128 .961 5.017 .000 .646 .738 .731 STDY 1.122 .443 .485 2.532 .019 -.139 .484 .369 WGHT 2 (Constant) -12.456 20.550 -.606 .551 ABIL .677 .131 1.013 5.169 .000 .646 .756 .749 STDY 1.389 .501 .601 2.773 .012 -.139 .527 .402 WGHT -.111 .099 -.186 -1.120 .276 -.049 -.243 -.162
Excluded Variables(b) Beta In t Sig. Partial Collinearity Statistics Correlation Model Tolerance 1 WGHT -.186(a) -1.120 .276 -.243 .760
Box 11.5. Unique Contribution of Weight to Three-Predictor Equation.
MR and Multiple Predictors 11.7
measures of the contribution of weight to the prediction equation; these also appear in the output
for Model 2 in Box 11.5. The REGRESSION in Box 11.5 has added Weight separately to the
regression, and includes the CHANGE and ZPP options.
The significance of the slope for weight can be determined with the SE of the slope, which
is calculated using the following formula: SEw.as3 = SQRT(MSE / ((1 - R2w.as) x SSw)). MSE =
58.063 from the ANOVA for Model 2 in Box 11.5. Using sWeight from Box 11.4, SSw = (24 -
1)18.3286722 = 7726.625. A regression with Weight as the dependent variable and Ability and
Study Time as predictors determined that R2w.as = .2395. This gives a value of .099 as SE for
weight with all predictors in the equation (i.e., Model 2). The t-test for weight when ability and
study time are also in the equation is not significant, tgw.as = -.111/.099 = -1.12, p = .2758,
indicating that weight makes no additional contribution to the prediction of grades over and above
what Ability and Study Time can predict. Box 11.5 shows this same statistic, as well as a more
complete regression analysis that focusses on the unique contribution of Weight.
Conceptually, the t-test on the slope for Weight is testing the significance of the unique
contribution of Weight over and above the variability in Grades already predicted by Ability and
Study Time. This is calculated as SSChange; specifically, SSg’w.as = SSg’.was - SSg’.as = 1605.374 -
1532.484 = 72.89. The significance of this unique contribution is F = (72.89/1)/58.063 = 1.255,
which appears as FChange for Model 2 in the regression output. As well, F = 1.1202 = t2,
demonstrating the equivalence between the F test for the change when Weight was added to the
other predictors, and the t test for the slope for Weight when all predictors were in the equation.
The F would be tested for significance with dfNumerator = 1 and dfDenominator = 20, and the t with df =
20. If critical values for F and t were obtained, FAlpha = t2Alpha (assuming a non-directional or two-
tailed test was conducted for t). Both FChange and the t-test for the slope have a significance of
.276, indicating that the contribution of Weight is not significant when Aptitude and Study time
are controlled statistically.
Strength of Unique Contribution
Researchers are interested not only in the significance of the unique contribution, but also
in the strength. The part r2 is the proportion of total variability in Grades predicted uniquely by
Weight; hence, r2g(w.as) = 72.89 / 2766.625 = .026, or equivalently, = R2
g.was - R2
g.as = .580 - .554 =
MR and Multiple Predictors 11.8
.026. This value appears as R2Change for Model 2 in the Change section of the printout. In addition,
rg(w.as) = SQRT(.026) = .162, which appears in the Part column for Weight in the Coefficients
section of the output. The part r has a negative sign because the slope for Weight is negative in
the multiple regression equation with all three predictors.
The part r can also be computed using residual scores, as shown in Box 11.6. Two
regression are shown (in part). In the first regression, Weight is the dependent variable and
Ability and Weight are the predictors. Residual Weight scores are saved as resw.as and then
REGRESS /VARI = wght abil stdy /DEP = wght /ENTER /SAVE RESI(resw.as).
Model R R Square Adjusted R Square Std. Error of the Estimate 1 .489(a) .240 .167 16.7272520
Model Sum of Squares df Mean Square F Sig. 1 Regression 1850.805 2 925.402 3.307 .056(a) Residual 5875.820 21 279.801 Total 7726.625 23
...
Residuals Statistics(a) Minimum Maximum Mean Std. Deviation N Predicted Value 129.958389 160.637802 143.875000 8.9704948 24 Residual -36.989136 31.821617 .000000 15.9834427 24
CORR resw.as WITH grde abil stdy. GRDE ABIL STDY RESW.AS Pearson Correlation -.162 .000 .000
REGRESS /VARI = grde abil stdy /DEP = grde /ENTER /SAVE RESI(resg.as).
Model R R Square Adjusted R Square Std. Error of the Estimate 1 .744(a) .554 .511 7.6660706
Model Sum of Squares df Mean Square F Sig. 1 Regression 1532.484 2 766.242 13.038 .000(a) Residual 1234.141 21 58.769 Total 2766.625 23
...Residuals Statistics(a) Minimum Maximum Mean Std. Deviation N Predicted Value 49.747364 81.526260 65.875000 8.1627031 24 Residual -15.714190 14.391576 .000000 7.3251841 24
CORR resg.as WITH resw.as abil stdy. RESW.AS ABIL STDY RESG.AS Pearson Correlation -.243 .000 .000
Box 11.6. Part and Partial rs from Residual Scores.
MR and Multiple Predictors 11.9
correlated with GRDE, ABIL, and STDY. As expected, resw.as correlates 0 with both Ability
and Study time; that is, the variability in resw.as represents the variability in Weight that is
completely independent of the other two predictors. The correlation between resw.as and Grades
is our part correlation. The unique variation in weight (i.e., the variation in weight independent of
the other two predictors) predicts -.1622 = .026 of the total variability in Grades, as previously
shown.
The partial correlation is another statistic that describes the unique contribution of each
predictor. The partial correlation represents how much is uniquely predicted by Weight of the
variability in Grades not already predicted by Ability and Study Time. Ability and Study Time
account for 1532.484 units of variability in Grades, leaving 1234.141 units as the residual. This
becomes the denominator for the partial r2; that is, r2gw.as = 72.89 / 1234.141 = .059, and rgw.as =
SQRT(.059) = .243, which appears as -.243 in the Partial column of the three-predictor equation.
The partial correlation can also be computed using residual scores, and the relevant
analyses are shown in the bottom half of Box 11.6. The dependent variable Grades is regressed
on Ability and Study Time, and a residual Grade score saved as resg.as. This residual score has
1234.141 units of variability, the same as the denominator in the preceding calculation of the
partial. The correlation between resw.as and resg.as is the partial correlation. That is, Weight
can uniquely predict -.2432 (or 5.9%) of the 1234.141 units of variability in Grades not already
explained by Ability and Study Time. Although still modest, 5.9% is considerably larger than
2.6% because Ability and Study Time together account for over half of the variability in Grades.
That is, the denominator for the partial r2 is much smaller than the denominator for the part r2.
A FOUR PREDICTOR EXAMPLE
Regression analyses with numerous predictors operate on the same principles. The
significance and part correlations for one variable in a 6-predictor model represent the
contributions of that variable over the effects of the other 5 variables. That is, the tests reflect the
differences in R2 and SSRegression between a 6-predictor regression with that variable and a 5-
predictor regression without that variable.
MR and Multiple Predictors 11.10
WORD RT AGE CON FRQ LEN WORD RT AGE CON FRQ LEN
1 800 5 1 56 8 21 1020 12 5 37 11 2 1213 10 7 67 10 22 982 7 4 60 5 3 1052 7 7 47 10 23 1150 14 6 58 13 4 1103 11 7 37 10 24 889 8 4 55 6 5 1016 5 6 52 4 25 1071 7 4 59 8 6 1020 5 5 21 6 26 896 7 6 27 9 7 874 6 7 47 5 27 1041 8 6 53 8 8 891 6 4 44 9 28 1104 9 5 35 8 9 944 9 6 37 9 29 910 7 3 36 810 1060 6 3 28 11 30 906 7 4 47 1011 1163 12 3 81 8 31 1098 7 4 63 1012 874 8 5 61 8 32 1001 7 4 43 913 1148 11 7 68 9 33 889 9 7 60 614 951 8 4 36 5 34 1060 11 4 68 1015 889 5 2 47 9 35 1119 12 3 110 1016 1064 8 4 48 9 36 979 10 6 68 917 1119 12 6 75 7 37 989 7 6 64 518 770 8 2 62 8 38 1113 9 6 50 819 1034 6 2 59 8 39 708 4 3 25 820 1054 12 3 53 9 40 987 12 9 79 5
Box 11.7. Raw Data for Naming Study.
We illustrate these points for a 4-predictor regression problem that involves picture
naming reaction times (RT) obtained for 40 words, along with measures of Concreteness (CON),
Length (LEN), Frequency in print (FRQ), and rated Age of acquisition (AGE). The data are
shown in Box 11.7. Note that
for the CON variable low
scores indicate concrete words
and high scores abstract
words, and that for the FRQ
variable, low scores indicate
more frequent words and high
scores less frequently
occurring words. These scales
were reversed so that all
variables would be expected to
have positive relationships with RT. That is, naming would be slower for words that were
abstract, were long, appeared infrequently in print, and were learned at a later age. The following
analyses focus on the Age predictor.
The SPSS regression commands and
preliminary output appear in Box 11.8. The
dependent variable and all four predictors
appear on the VARIABLES line and age is
entered after the other three predictors because
that will be our primary focus in the following
discussion. We have also requested ZPP and
CHANGE, as well as the DEFAULT statistics.
All four predictors correlate positively with
RT, as expected (recall that CON and FRQ are
reversed). The correlation matrix also shows,
however, that the predictors are correlated with
REGRESS /VARI = rt age con frq len /DESCR /STAT = DEFAU ZPP CHANGE /DEP = rt /ENTER con frq len /ENTER age /SAVE PRED(pr.acfl) RESI(rr.acfl).
Descriptive Statistics Mean Std. Deviation N RT 998.775000 114.0576749 40 AGE 8.350000 2.4966644 40 CON 4.750000 1.7795130 40 FRQ 53.075000 17.3772434 40 LEN 8.200000 1.9767884 40
RT AGE CON FRQ AGE .585 CON .318 .355 FRQ .349 .524 .055 LEN .323 .344 -.160 -.030
Box 11.8. Regression Command and
Preliminary Output for Four Predictors.
MR and Multiple Predictors 11.11
one another to a considerable degree. In particular, age correlates with all three of the other
predictors.
Box 11.9 shows the remaining printout from the regression in Box 11.8. In the final
regression equation, the overall contribution of all four predictors is both highly significant and a
strong effect, F = 6.011, p = .001, R2 = .407. With respect to the individual predictors, however,
the unique contribution is not significant for any of the predictors, ps > .10, and the part
correlations are all modest, part rs < .22. This seemingly anomalous outcome is explained later.
Overall Regression Results
The best-fit regression equation with all four predictors is: y = 622.601 + 15.073C +
1.147F + 14.700L + 14.751A. The predicted scores based on this equation (not shown) account
Model Summary(c) R R Adjusted Std. Error of Change Statistics Square R Square the Estimate Model R Square Change F Change df1 df2 Sig. F Change 1 .601(a) .361 .308 94.9137441 .361 6.773 3 36 .001 2 .638(b) .407 .339 92.6989896 .046 2.741 1 35 .107
Model Sum of Squares df Mean Square F Sig. 1 Regression 183046.697 3 61015.566 6.773 .001(a) Residual 324310.278 36 9008.619 Total 507356.975 39
2 Regression 206598.382 4 51649.595 6.011 .001(b) Residual 300758.593 35 8593.103 Total 507356.975 39
Coefficients(a) Unstand Coeff Stand Coeff t Sig. Correlations Model B Std. Error Beta Zero-order Partial Part 1 (Constant) 584.562 94.505 6.186 .000 CON 23.240 8.664 .363 2.682 .011 .318 .408 .357 FRQ 2.237 .876 .341 2.554 .015 .349 .392 .340 LEN 22.570 7.791 .391 2.897 .006 .323 .435 .386 AGE
2 (Constant) 622.601 95.116 6.546 .000 CON 15.073 9.795 .235 1.539 .133 .318 .252 .200 FRQ 1.147 1.080 .175 1.062 .296 .349 .177 .138 LEN 14.700 8.972 .255 1.638 .110 .323 .267 .213 AGE 14.751 8.910 .323 1.656 .107 .585 .269 .215
Excluded Variables(b) Beta In t Sig. Partial Collinearity Statistics Correlation Model Tolerance 1 AGE .323(a) 1.656 .107 .269 .445
Residuals Statistics(a) Minimum Maximum Mean Std. Deviation N Predicted Value 873.093933 1177.169556 998.775000 72.7832014 40 Residual -189.455215 135.995087 .000000 87.8166203 40
Box 11.9. Regression Results for Four-Predictor Regression from Box 11.8.
MR and Multiple Predictors 11.12
for (40 - 1)72.78322 = 206598.382 units of variability in RT and fail to account for (40 -
1)87.816622 = 300758.593 units of variability in RT. SSRegression + SSResidual = 507356.975 = SSRT.
The predicted variability represents 40.72% of the total variability in RT; that is, R2r.cfla =
206598.382 / 507356.975 = .4072, and Rr.cfla = SQRT(.4072) = .638. These values appear in the
model summary section. We cannot predict 59.28% of the variability in RT; that is, 1 - R2 = 1 -
.4072 = .5928, and SQRT(.5928) = .770.
Box 11.10 shows the
descriptive statistics and correlations
for the original and derived scores.
SSRegression, SSResidual, and SSTotal could
be obtained from these statistics. The
correlation between RT and the
predicted scores (pr.acfl) = .638 =
Rr.acfl. The correlation between RT
and the residual scores (rr.acfl ) =
.770 = SQRT(1 - R2r.acfl). The
correlation matrix also shows that the residual scores correlate 0 with all four predictors and with
the predicted scores themselves. These relationships have been seen many times before.
The significance of the overall regression is tested using Analysis of Variance, and the
relevant results are shown in Box 11.9. SSRegression is divided by 4, its df = p, to give MSRegression and
SSResidual is divided by 35, its df = n - p - 1, to give MSResidual. F = MSRegression / MSResidual = 6.011,
which is highly significant. F could also be calculated using our formula based on R2, and the
results would be identical, except perhaps for rounding. We can clearly reject the null hypothesis
of no relationship in the population between RT and the collective effects of all four predictors.
Adjusted R2 and Standard Error of the Estimate. Two entries in the Box 11.9 regression
output have not previously been explained. The Adjusted R2 adjusts for chance relations between
the predictors and y; this is particularly important when multiple predictors are used. There are
several ways to think about why this adjustment is necessary. First, remember that multiple
regression minimizes SSResidual (i.e., maximizes SSRegression) for this sample of data. It is expected
CORR rt pr.acfl rr.acfl age con frq len /STAT = DESCR.
Mean Std. Deviation N RT 998.775000 114.0576749 40 PR.ACFL 998.7750000 72.78320138 40 RR.ACFL .0000000 87.81662030 40 AGE 8.350000 2.4966644 40 CON 4.750000 1.7795130 40 FRQ 53.075000 17.3772434 40 LEN 8.200000 1.9767884 40
RT PR.ACFL RR.ACFL AGE CON FRQ LEN PR.ACFL .638 1 .000 .917 .499 .547 .506 RR.ACFL .770 .000 1 .000 .000 .000 .000 AGE .585 .917 .000 1 .355 .524 .344 CON .318 .499 .000 .355 1 .055 -.160 FRQ .349 .547 .000 .524 .055 1 -.030 LEN .323 .506 .000 .344 -.160 -.030 1
Box 11.10. Correlation of Original and Derived Scores.
MR and Multiple Predictors 11.13
that the relationship in the population will not benefit from this minimization to a particular set of
sample data, and hence R2 in the population is expected to be somewhat smaller than that
computed for the sample. Second, multiple regression is so powerful that it can capitalize on
patterns in the data that essentially are due to random covariation, and multiple regression will be
particularly sensitive to this chance variation when n is small or p is large. The formula for
R2Adjusted = 1 - (1 - R2) × ((n - 1)/(n - p - 1)) = 1 - (1 - .407) x ((40 - 1) / (40 - 4 - 1) = 1 - .593 x
1.114 = .339, as shown in Box 11.9. R2 adjusted is always smaller than R2, and the most dramatic
reductions occur when n is small and p is large because the magnitude of the adjustment
(reduction) depends on the sample size n and the number of predictors p.
The other statistic that we have not discussed is the Standard Error for the overall
regression results. This value is simply the square root of MSResidual (i.e., �8593.103 = 92.699 �
Standard Error). If one wanted to report a standard deviation associated with the error or residual
variation, this would be the appropriate statistic to use.
Unique Contribution of Predictors
The story with respect to individual predictors appears to be complex. As noted
previously, none of the predictors is significant in the four-predictor equation, despite the high
level of overall significance. Briefly (more extensively in the next chapter), this occurs because
what the predictors can explain is due to their shared variability with one another, and not from
anything unique to any of the predictors. Let us examine more closely the significance of Age,
although any of the predictors could be examined in this manner.
The significance of the unique contribution of Age can be calculated from a t-test on the
slope, t = 14.751 / 8.910 = 1.656, p = .107, as shown on the Age line of the Coefficient section of
the printout. With df = 35, this difference would be marginally significant using a one-tailed test,
that is, p = .107/2 = .0535. We could also perform this test as an F test. SSPredicted increased by
SSr’.acfl - SSr’.clf = 206598.382 - 183046.697 = 23551.685, seemingly a large amount except that
SSTotal is a very large number. F = (23551.685/1) / 8593.103 = 2.741 = 1.6562. This F and its
corresponding p = .107 appear as FChange and Significance of F Change for Model 2 in the Change
section of the printout. Despite the overall relationship being highly significant, Age (and the
other predictors) do not uniquely predict significant amounts of variability in RT.
MR and Multiple Predictors 11.14
Figure 11.1. Using Menu to Obtain Unique Contribution
with Multiple Predictors.
There are several measures of the
relative strength of the unique contribution
of Age, including the part correlation,
r2r(a.cfl) = 23551.685 / 507356.975 = .046 =
R2r’.acfl - R
2r’.cfl. This part r2 appears in the
R2 Change section of the printout. The part
r is the square root of this value; that is,
rr(a.cfl) = .215, which appears in the Part
column for the Age predictor. Note that
the part r is considerably smaller than the simple r between RT and Age, rr.a = .585. The part r for
Age can also be obtained by computing a residual Age score that is uncorrelated with the other
predictors. Box 11.11 shows the procedure. Other measures of strength shown in Box 11.9
include the partial r of .269 and the standardized slope of .323.
Using SPSS Menus for Regression with Multiple Predictors
Regression analysis with
multiple predictors can be done by
SPSS either using the syntax
approach, as emphasized in this
chapter, or using the menu system.
To obtain the unique contribution
of Age, for example, first enter a
block of predictors that includes
Concreteness, Frequency, and
Length (see Figure 11.1 for this
first step). Then click on Next to
initiate a new block of predictors.
This will bring up the screen
shown in Figure 11.2. Age can
then be selected as an additional predictor to add to those already in the equation.
REGRE /VARI = age con frq len /DEP = age /ENTER /SAVE RESI(ra.cfl)....CORR ra.cfl WITH rt age con frq len /STAT = DESCR.
Mean Std. Deviation N RA.CFL .0000000 1.66588503 40 AGE 8.350000 2.4966644 40 ... RT AGE CON FRQ LEN RA.CFL .215 .667 .000 .000 .000
Box 11.11. Part r for Age by Residual Scores.
MR and Multiple Predictors 11.15
Figure 11.2. Block 2 and Statistics Options.
Once Age as been selected,
as shown in bottom menu screen in
Figure 11.2, select Statistics to
request Change and Part and Partial
rs, as shown in top menu screen in
Figure 11.2. Then select Continue
and OK to run the analysis.
The syntax generated by
SPSS is shown in Box 11.12.
Although somewhat more detailed
than syntax commands we have
entered because defaults are
specified, Box 11.12 shows essential
commonalities with
commands used earlier. If you
do use the menu system for
regression analyses, remember
to save the syntax commands
produced by SPSS in a syntax
file in case the analysis needs
to be redone.
CONCLUSIONS
This chapter has shown that our approach to the overall regression results with two
predictors generalizes directly to multiple regression with three or more predictors. Moreover,
tthe unique contribution of predictors is also quite general and can be extended to three, four, or
more predictors. SSChange for the last predictor entered into the equation is central to one approach.
This allows us to calculate FChange and R2Change statistics relevant to the significance and strength of
the unique contribution of each predictor. A second approach to part and partial correlation
REGRESSION /MISSING LISTWISE /STATISTICS COEFF OUTS R ANOVA CHANGE ZPP /CRITERIA=PIN(.05) POUT(.10) /NOORIGIN /DEPENDENT RT /METHOD=ENTER CON FRQ LEN /METHOD=ENTER AGE .
Box 11.12. Syntax Generated by Menu Commands.
MR and Multiple Predictors 11.16
R2Adj'1&(1&R2)× n&1
n&p&1
Equation 11.1. Formula to Calculate R2 Adjusted.
SE1.23...p
'
MSResidual
SS1(1 &R2
1.23...p)
Equation 11.3. Formula for SE Slope with More than TwoPredictors.
SSy1.23 ÿp
'SSy.123 ÿp
&SSy.23 ÿp
R2y1.23 ÿp
'
SSy1.23 ÿp
SSTotal
'R2y.123 ÿp
&R2y.23 ÿp
Equation 11.2. Equations for SSChange and R2Change for Multiple Predictor Equations.
strength of the unique contribution of each predictor. A second approach to part and partial
correlation computes a residual predictor score that is independent of the other predictors in the
equation and, for the partial correlation, a residual dependent score that reflects variability in the
criterion variable that is independent of other predictors in the equation.
Equation 11.1 shows the formula for R2 Adjusted, which provides a more accurate
estimate of R2 in the population than does the unadjusted R2. The unadjusted R2 capitalizes on
chance relationships in the data and over-estimates the magnitude of the population R2.
Equation 11.2 summarizes some of the extensions for the unique contribution of a
predictor, although a conceptual understanding is ultimately more important than formula at this
stage of our understanding of multiple regression.
Equation 11.3 shows the
slightly modified formula for the
standard error of the regression
coefficients. Now the standard
error is based on the variability in
each predictor that is independent of all of the other predictors in the equation. To get R21.234..., we
would need to run the appropriate multiple regression analysis (e.g., regress x1 on predictors x2 to
xp). Note that, other things being equal, SE becomes smaller as the unique variation in SS
increases, and SE becomes larger as the unique variation in SS decreases. This means that for the
purpose of testing the unique contribution of predictors, it is desirable to have more unique
variability in the predictors, rather than less.
SPSS and Multiple Predictors 12.1
CHAPTER 12
MORE ON REGRESSION ANALYSES WITH MULTIPLE PREDICTORS
Automated Strategies for Variable Selection in MR . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
Forward Selection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
Backward Selection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
Stepwise Selection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
Examples of Automated Regression-Building Procedures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
MR for the Extended Study Time Study . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
Automated Procedures using SPSS Menus . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
Word Attributes and Naming Latency Study . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
Interpretation of Unique Contribution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
SPSS Regression Options and Menus . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
Complexities in Regression Model Building . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
Redundant Predictors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
Coincident Outcomes as Predictors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
Suppressor Effects . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
Testing All Possible Equations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
SPSS and Multiple Predictors 12.2
This chapter examines additional aspects of multiple regression for designs with more
than two predictors. Following a brief introduction, we examine several automated ways to enter
predictors into a multiple regression analysis, including the SPSS commands to perform the
various selection methods.
The preceding analyses and those in earlier chapters have demonstrated that MR assesses
the independent contribution of predictors beyond what other predictors already explain; does x1
add anything to what x2 to xp can predict? Equivalently, does the residual of x1, after being
regressed on x2 to xp, contribute anything unique to the prediction of y? One implication of this
approach is that the predictors in a multiple predictor design compete with one another for
significance. Two correlated variables that account only for the same variation in y could both be
not significant because of their redundancy with the other predictor. Alternatively, a predictor
might only be significant when some other predictor or set of predictors is also in the equation
because its contribution is masked by off-setting influences.
Because of these subtleties, multiple predictors often forces researchers to decide which
variables, if any, should be included in the regression equation. Depending on the choice of
predictors, it is possible that some variables will not contribute uniquely to the overall regression,
either because they are not related to the criterion variable, because they are redundant with other
variables in the set, or because a critical covariate has not been entered as an additional predictor.
Because of the complex ways in which combinations of variables influence one another,
selection of the "best" set of MR predictors is a challenging problem.
Selection of predictors for an MR model is further complicated by the fact that the
number of different combinations of predictors increases dramatically as the number of
predictors increases. With four predictors (1 to 4), there are 16 (24 = 2 × 2 × 2 × 2) possible
combinations of the four predictors: 1 set of no predictors, 4 singles (1, 2, 3, 4), 6 pairs (12, 13,
14, 23, 24, 34), 4 triplets (123, 124, 134, 234), and 1 quartet (1234). With 6 predictors, there are
26 = 64 combinations: 1 empty set, 6 singles, 15 pairs, 20 triplets, 15 quartets, 6 pentads, and 1
hexad. In general, the number of combinations of predictors is 2p, where p is the number of
predictors. Choosing the "best" of these 2p possibilities is not easy when more than a few
predictors are involved, as is the case for the increasingly complex MR studies used in
SPSS and Multiple Predictors 12.3
contemporary research.
AUTOMATED STRATEGIES FOR VARIABLE SELECTION IN MR
A variety of strategies have been developed to assist with the selection problem, and
some strategies have been implemented in computer packages such as SPSS. Several approaches
are introduced here and illustrated for a few designs in later parts of the chapter. Readers should
be forewarned, however, that blind statistical approaches to predictor selection in MR have many
limitations and dangers. To avoid these potential pitfalls, researchers need to understand the
fundamental nature of multiple regression, one objective of this book. Even more importantly,
researchers always require sound theoretical and empirical knowledge of the domain being
studied, and should use that knowledge when developing regression models for complex
phenomena. This more sophisticated and reasoned type of model building is presently beyond
the capabilities of basic multiple regression, as we have been doing with SPSS. With these
caveats in ming, the three automated methods that we consider are Forward, Backward, and
Stepwise selection.
Forward Selection
The Forward selection method selects the predictor at each stage that (a) adds the most to
the prediction of the criterion, and (b) reaches some specified probability level (called PIN in
SPSS to enter the equation (by default, PIN = .05). Selection proceeds at each step of the
regression until no more variables reach the probability level specified for entry into the equation.
At step 1, the predictor most strongly related to y would be entered in the equation if its
probability was less than the entry p value; let us say this is predictor x1. After x1 is entered,
each of the remaining predictors is re-analyzed for its contribution to the equation over and above
x1 (i.e., ty2.1, ty3.1, ..., typ.1). The strongest of these predictors is entered as long as its probability is
less than the critical p value to enter the equation; suppose this predictor is x2. The contribution
of each remaining variable when included with x1 and x2 is evaluated (i.e., ty3.12, ty4.12, ..., typ.12)
and the process continues until none of the variables have a low enough probability to enter the
equation or until all predictors are in the regression equation. Note in particular about this brief
description that the probability associated with the unique contribution of a predictor must be
SPSS and Multiple Predictors 12.4
recalculated anew at each stage. This is because the probability being considered is the
probability associated with each predictor if it alone was added to predictors aleady in the
equation.
The preceding paragraph referred deliberately to "probability to enter" rather than
significance level. Significance refers to the probability of a Type I error and MR selection
procedures complicate determining the "true" probability of a Type I error (i.e., �). The
probability normally specified as � assumes that researchers are performing one test of
significance. But MR calculates probabilities for multiple predictors and then selects the variable
with the lowest probability. The probability that one of p predictors falls in the rejection region
is greater than the nominal probability used for each of the p statistical tests. The actual
probability of rejecting a true H0 can therefore be much larger than the nominal probability level
when multiple predictors are involved, as in the Forward procedure and in other statistical
procedures for selecting predictors. This is one of a number of problems with automatic
regression-building procedures.
Backward Selection
An alternative to Forward selection is Backward selection, which removes variables from
the equation rather than adding them. Backward selection begins with all predictors in the
equation (i.e., R2y.12...p). The significance level for each predictor is determined with all of the
other predictors included (e.g., ty1.2...p, ty2.1...p, typ.1...p-1). A predictor is then removed if it (a) adds
least to the prediction of the criterion, and (b) fails to reach some specified probability level
(called POUT in SPSS) to remain in the equation (by default, POUT = .10, unless some user-
specified value is provided). This variable (e.g., x1) is removed from the equation and a
probability is calculated for the contribution of each predictor without x1 in the equation (e.g.,
ty2.3...p, ...). If the probability level for the weakest predictor is greater than the probability to
remain, then that predictor is removed. The procedure continues until the remaining variables all
have probabilities lower than the probability to remain or all variables have been removed. As in
the Forward procedure, probabilities must be recalculated anew at each stage of the analysis.
Forward and Backward selection procedures do NOT necessarily result in the same
equation, although they might, depending on the relations among the predictors and between the
SPSS and Multiple Predictors 12.5
predictors and the criterion variable. One factor leading to differences is that SPSS uses different
default values for Entering and Removing predictors; more on this shortly. A more fundamental
factor is that the significance of a predictor at each stage depends on the other predictors in the
equation (i.e., the context), which can differ for Forward and Backward analyses. The
reservations stated earlier about interpreting probabilities as significance levels apply to the
Backward procedure as well as to the Forward procedure.
Stepwise Selection
A third approach to variable selection is essentially a combination of the Forward and
Backward procedures and is called Stepwise selection. At each step, Stepwise selection involves
a Forward selection of the predictor that has the lowest probability at that point and that
surpasses some specified probability level (PIN) to enter the equation. The procedure then
examines the variables currently in the equation to determine if one or more should be removed
because their probability levels have fallen below the value specified for remaining in the
equation (POUT). Variables that no longer meet the criterion to remain in the equation are
successively removed. The next variable for entry is then selected if its probability is less than
the probability value to enter the equation. All predictors in the equation are again reevaluated
for removal, and so on until no more predictors can be entered or removed.
In Stepwise selection, the probability value to enter the equation (e.g., default PIN = .05)
must be less than the probability to remain in the equation (e.g., default POUT = .10); that is, the
probability to remove a predictor will be greater than the probability to enter a predictor. If the
probability to remain were equal to or lower than the probability to enter, then predictors could
be entered and removed from the equation in an endless, repeating cycle. The Stepwise method
is prone to the same misuse of probabilities to represent significance levels as are the Forward
and Backward selection procedures. In fact the significance problem may be even more serious
with stepwise regression.
Forward, Backward, and Stepwise methods illustrate the general statistical considerations
involved in selection of predictors in an MR study with multiple predictors. Similar statistical
considerations underlie other selection procedures. With a modest number of predictors, for
example, it is practical to perform regression for all-possible prediction equations, but some
SPSS and Multiple Predictors 12.6
criteria (e.g., R2Change) must then be used by the researcher to choose the best of the resulting
equations. Because the various selection methods are all calculation-intensive, the procedures
are illustrated only using SPSS, although in principle simple examples could be done
“manually.”
EXAMPLES OF AUTOMATED REGRESSION-BUILDING PROCEDURES
This section illustrates the automated procedures using SPSS and sample problems that
we have discussed in previous chapters. Refer to those earlier chapters for details about the
studies and for fuller explanation of the basic analyses. Here we concentrate on the regression-
building procedures discussed in the preceding section.
Note that automated procedures require that all predictors be specified for the regression
analysis at some point, and not simply introduced as each stage of the equation is constructed.
That is, researchers must use the /VARIABLES = option or list multiple predictors when
specifying the method being used (e.g., /METHOD = FORWARD x1 x2 x3 ...). The automated
procedures select from among some set of predictors and need to know what variables are to be
included in the candidate set.
MR for the Extended Study Time Study
We earlier examined a study in which grades (GRDE) for 24 students were regressed on
study time (STDY), ability (ABIL), and weight (WGHT). The following analyses do not provide
much additional information beyond what has already reported for this study. The difference is
that instead of researchers specifying what variables are entered (with the ENTER keyword),
SPSS automatically enters or removes predictors according to criteria specified by the user.
Box 12.1 shows a FORWARD regression for the extended Study Time data. The
researcher specifies /METHOD = FORWARD (instead of /ENTER varnames; here the user could
have simply entered /FORWARD) and SPSS proceeds to enter and remove variables according
to default probability values for entering (PIN = .05) and removing (POUT = .10) the predictors.
These default values for entering and removing predictors are reported early in the output (the
first line in our edited version).
SPSS selects the predictor with the lowest probability that is less than PIN (i.e., the most
SPSS and Multiple Predictors 12.7
REGRESS /VARI = grde abil stdy wght /DESCR /STAT = DEFAU CHANGE ZPP /DEP = grde /METHOD = FORWARD.
grde abil stdy wght abil .646 1.000 -.649 -.127 stdy -.139 -.649 1.000 .442 wght -.049 -.127 .442 1.000
Variables Entered/Removed(a) Model Variables Variables Method Entered Removed 1 abil . Forward (Criterion: Probability-of- F-to-enter <= .050) 2 stdy . Forward (Criterion: Probability-of- F-to-enter <= .050)
Model R R Adjusted Std. Error of Change Statistics Square R Square the Estimate R Square Change F Change df1 df2 Sig. F Change 1 .646(a) .418 .391 8.5567541 .418 15.786 1 22 .001 2 .744(b) .554 .511 7.6660706 .136 6.409 1 21 .019
a Predictors: (Constant), abilb Predictors: (Constant), abil, stdy
Model Sum of Squares df Mean Square F Sig. 1 Regression 1155.828 1 1155.828 15.786 .001(a) Residual 1610.797 22 73.218 Total 2766.625 23
2 Regression 1532.484 2 766.242 13.038 .000(b) Residual 1234.141 21 58.769 Total 2766.625 23
a Predictors: (Constant), abilb Predictors: (Constant), abil, stdy
Coefficients(a) Model Unstandardized Standardized t Sig. Correlations Coefficients Coefficients B Std. Error Beta Zero-order Partial Part 1 (Constant) 23.419 10.828 2.163 .042 abil .432 .109 .646 3.973 .001 .646 .646 .646
2 (Constant) -19.737 19.613 -1.006 .326 abil .642 .128 .961 5.017 .000 .646 .738 .731 stdy 1.122 .443 .485 2.532 .019 -.139 .484 .369
Excluded Variables(c) Model Beta In t Sig. Partial Collinearity Correlation Statistics Tolerance 1 stdy .485(a) 2.532 .019 .484 .579 wght .033(a) .199 .844 .043 .984
2 wght -.186(b) -1.120 .276 -.243 .760
Box 12.1. Forward Regression of Study Time Study.
"significant" predictor) to enter first. As shown by the correlation matrix, the ability variable has
the strongest simple relationship with grades. It is therefore selected to enter at Step 1; its
probability is .0006, which is less than .05 (PIN) and is also lower than the probabilities for
Study Time and Weight, the other predictors being considered. Statistics are also computed for
both STDY and WGHT as second predictors in addition to Ability, and these statistics are
printed in the "Excluded Variables" section for Model 1. In essence, SPSS is reporting here the p
value for study time if added to ability (.019), and for weight if added to ability (.844) .
SPSS and Multiple Predictors 12.8
REGRESS /VARI = grde abil stdy wght /CRITERIA PIN (.30) POUT (.40) /DEP = grde /METHOD = STEP.
Model Variables Variables Method Entered Removed 1 abil . Stepwise (Criteria: Probability-of-F-to-enter <=.300, Probability-of-F-to-remove >=.400). 2 stdy 3 wght
Model Summary Model R R Square Adjusted R Std. Error of Square the Estimate 1 .646(a) .418 .391 8.5567541 2 .744(b) .554 .511 7.6660706 3 .762(c) .580 .517 7.6198780a Predictors: (Constant), abilb Predictors: (Constant), abil, stdyc Predictors: (Constant), abil, stdy, wght
ANOVA(d) Model Sum of Squares df Mean Square F Sig. 1 Regression 1155.828 1 1155.828 15.786 .001(a) Residual 1610.797 22 73.218 Total 2766.625 23
2 Regression 1532.484 2 766.242 13.038 .000(b) Residual 1234.141 21 58.769 Total 2766.625 23
3 Regression 1605.374 3 535.125 9.216 .000(c) Residual 1161.251 20 58.063 Total 2766.625 23
Model Unstandardized Standardized t Sig. Coefficients Coefficients B Std. Error Beta 1 (Constant) 23.419 10.828 2.163 .042 abil .432 .109 .646 3.973 .001
2 (Constant) -19.737 19.613 -1.006 .326 abil .642 .128 .961 5.017 .000 stdy 1.122 .443 .485 2.532 .019
3 (Constant) -12.456 20.550 -.606 .551 abil .677 .131 1.013 5.169 .000 stdy 1.389 .501 .601 2.773 .012 wght -.111 .099 -.186 -1.120 .276
Excluded Variables(c) Model Beta In t Sig. Partial Collinearity Correlation Statistics Tolerance 1 stdy .485(a) 2.532 .019 .484 .579 wght .033(a) .199 .844 .043 .984
2 wght -.186(b) -1.120 .276 -.243 .760
Box 12.2. Stepwise Regression with PIN=.3 and POUT=.4.
The probability for STDY, when added to Ability, is .019, which is less than PIN and less
than .844, the probability for Weight when added to Ability. Therefore, SPSS enters STDY into
the prediction equation at step 2. After entering Study Time, SPSS computes statistics for the
remaining variable (or variables if p > 3); the statistics are reported in the Excluded Variables
section for Model 2. Although Weight now has a much lower p value than before, .276 vs. .844,
it is still not less
than .05, the
criterion that
SPSS uses to
admit new
predictors. SPSS
stops at the two
predictor
equation.
SPSS has
selected the two-
predictor
equation as the
most suitable
model for this
particular study.
Almost any
selection method
and researchers
themselves are
likely to agree
that Weight is not
a valid predictor
of Grades and
SPSS and Multiple Predictors 12.9
that the model including only Ability and Study Time is the most theoretically useful. It is
critical to remember, however, that judgments about theoretical meaningfulness were not part of
the statistical selection procedure, and that there is no necessary correspondence between
statistical and theoretical results. The ease of Stepwise and other selection procedures sometimes
blinds researchers to the fact that such methods ignore conceptual meaningfulness, an essential
component of model and theory building.
Box 12.2 shows a Stepwise analysis of the data, but now an additional command is
included to modify the default PIN and POUT values. The CRITERIA subcommand permits
researchers to control the criteria used to enter and remove variables from the equation. There are
several criteria available (e.g., F to enter or remove), but we consider only the probabilities
associated with entering and removing predictors (i.e., PIN and POUT). The desired probabilities
are entered in parentheses after the PIN and POUT keywords as shown in Box 12.2. POUT must
be greater than PIN, as shown here, or SPSS will increase its value (to 1.1 times PIN).
The results of the new analysis are identical to those in Box 12.1 for the first two steps.
Ability and then Study Time are successively selected for entry into the equation. After Step 2,
however, the PIN check now finds that the probability for Weight (.276) is less than the new PIN
(.30); indeed, the PIN of .30 was chosen so as to permit Weight to enter the equation. Weight is
entered, and because .276 is also less than POUT (.40), Weight remains in the equation once
entered. The final equation is equivalent to that resulting from the /ENTER option, which forces
all predictors into the equation.
Researchers sometimes want to see the equation with all p predictors, even when Stepwise
or other selection procedures are used. To force the equation with all predictors, follow
/STEPWISE with /ENTER. When Stepwise selection is finished, /ENTER then adds any variables
omitted from the Stepwise equation and reports the results with all variables in the final equation.
If the entry of excluded variables does not modify the probabilities from the Stepwise regression
(e.g., including Weight did not increase or decrease the probabilities for Ability and Study Time),
researchers might feel more certain that excluded variables were unrelated to the criterion. If
significant predictors become more or less significant when the final variables are added,
however, then excluded variables could have some relation with the criterion that is
SPSS and Multiple Predictors 12.10
Figure 12.1. Stepwise Procedure via Menus.
overshadowed by other predictors, or excluded variables could somehow weaken or enhance the
contribution of the other predictors, or there may be some complex inter-relationship among the
included and excluded predictors that is not readily picked up by automated procedures that
consider only one predictor at a time.
In the
preceding
example, the
various
statistical
procedures led
to the same
final equation.
But this is by
no means
always the case. Box 12.3 shows
FORWARD and BACKWARD analyses
for the first Grades study, with just
Ability and Study Time as predictors.
Recall that earlier analyses revealed that
neither predictor alone was significant,
although both were significant when the
two predictors were together in the
equation. Box 12.3 shows the
implications of this for the automated
procedures. /FORWARD and
/STEPWISE would stop without entering
any predictors because neither predictor
is significant at the .05 level, whereas
/BACKWARDS stops after both predictors
REGRE /VARI = grde abil stdy /DEP = grde /FORWARD.
Variables Entered/Removed(a) a Dependent Variable: grde
REGRE /VARI = grde abil stdy /DEP = grde /BACKWARD.
Model Variables Variables Method Entered Removed 1 stdy, abil(a) . Enter
Model R R Square Adjusted R Std. Error of Square the Estimate 1 .649(a) .421 .332 8.386014928... Model Unstandardized Standardized t Sig. Coefficients Coefficients B Std. Error Beta 1 (Constant) -8.535 25.161 -.339 .740 abil .447 .178 .662 2.516 .026 stdy 1.358 .465 .769 2.922 .012
Box 12.3. Analysis of First Grades Study.
SPSS and Multiple Predictors 12.11
are entered because both predictors remain significant at the .10 level (indeed at a much higher
level of significance).
Automated Procedures using SPSS Menus
Figure 12.1 shows several screens relevant to specifying automated procedures using the
SPSS Menu system. Selecting Analyze | Regression | Linear activates the Linear Regression box
shown in Figure 12.1. The user specifies the Dependent variable (grde in this example) and all of
the predictors to be processed automatically by SPSS (abil, stdy, and wght here). Clicking
Method shows a series of options, including Stepwise, Forward, and Backwards. Stepwise has
been selected in Figure 12.1. Selecting Options opens the Linear Regression: Options dialogue
box also shown in Figure 12.1. Here users can control the p values for entering and removing
predictors (i.e., PIN and POUT). Clicking Continue and Ok produces the requested analysis,
which would agree with earlier printouts.
SPSS would also produce syntax for this menu command, which illustrates another feature
of the Regression syntax. Specifically, it is possible to omit the /VARIABLES= portion of the
regression command if the dependent variable is identified by the /DEPENDENT= command and
the independent variables are explicitly listed on the /METHOD= line. That is, users can write
commands such as: REGRESSION /DEP = grde /STEP abil stdy wght.
Word Attributes and Naming Latency Study
An earlier chapter described a study in which researchers measured naming reaction time
(RT) and various word characteristics related to ease of naming: concreteness (CON), frequency
(FRQ), length (LEN), and rated age of word acquisition (AGE). The results appear in Box 12.3.
Before considering the underlying structure of these data, which are known for this artificial case,
let us see what MR reveals about the data.
SPSS and Multiple Predictors 12.12
REGR /VAR = rt TO len /DESC /DEP = rt /ENTER
Correlations: RT AGE CON FRQAGE .585CON .318 .355FRQ .349 .524 .055LEN .323 .344 -.160 -.030
Multiple R .63813 Analysis of VarianceR Square .40721 DF Sum Squares MeanSquareAdj. R Square .33946 Regression 4 206598.3814 51649.5954Stand. Error 92.69899 Residual 35 300758.5933 8593.1027
F = 6.01059 Signif F = .0009
Variable B SE B Beta T Sig TLEN 14.699981 8.972093 .254772 1.638 .1103FRQ 1.146724 1.079876 .174709 1.062 .2956CON 15.073202 9.794688 .235170 1.539 .1328AGE 14.751414 8.910410 .322901 1.656 .1068(Constant) 622.600736 95.116180 6.546 .0000
Box 12.4. MR Analyses of Naming Data and ENTER Results.
REGRESS /DEP = RT /FORWARD age con frq len.
Model R R Square Adjusted R Std. Error of Square the Estimate 1 .585(a) .343 .325 93.6742945
Model Sum of Squares df Mean Square F Sig. 1 Regression 173911.784 1 173911.784 19.819 .000(a) Residual 333445.191 38 8774.873 Total 507356.975 39
Model Unstandardized Standardized t Sig. Coefficients Coefficients B Std. Error Beta 1 (Constant) 775.439 52.307 14.825 .000 AGE 26.747 6.008 .585 4.452 .000
Excluded Variables(b) Model Beta In t Sig. Partial Collinearity Correlation Statistics Tolerance 1 CON .127(a) .898 .375 .146 .874 FRQ .059(a) .375 .710 .062 .726 LEN .138(a) .984 .331 .160 .882
Box 12.5. Forward Selection Procedure.
Box 12.4 shows the MR results when all predictors are forced into the equation with
ENTER. Several aspects of the analysis warn us to interpret the data carefully. Although the
overall regression is highly
significant and the predictors
account for over 40% of the
variation in naming RT, no
predictor is individually
significant. This outcome
indicates redundancy among
the predictors. The
correlation matrix shows that
the age of acquisition
variable (AGE) may be the
problem. AGE correlates with concreteness, frequency, and length, which are relatively
independent, and all predictors are correlated with the criterion variable RT. AGE has a higher
correlation with RT than do the other predictors.
SPSS and Multiple Predictors 12.13
REGRE /VARI = rt age con frq len /DEP = rt /STEPWISE.
Model Variables Variables Method Entered Removed 1 AGE . Stepwise(Criteria: Probability-of-F-to-enter <= .050, Probability-of- F-to-remove >= .100).
Model R R Square Adjusted R Std. Error of Square the Estimate 1 .585(a) .343 .325 93.6742945
Model Sum of Squares df Mean Square F Sig. 1 Regression 173911.784 1 173911.784 19.819 .000(a) Residual 333445.191 38 8774.873 Total 507356.975 39
Model Unstandardized Standardized t Sig. Coefficients Coefficients B Std. Error Beta 1 (Constant) 775.439 52.307 14.825 .000 AGE 26.747 6.008 .585 4.452 .000
Excluded Variables(b) Model Beta In t Sig. Partial Collinearity Correlation Statistics Tolerance 1 CON .127(a) .898 .375 .146 .874 FRQ .059(a) .375 .710 .062 .726 LEN .138(a) .984 .331 .160 .882
Box 12.6. Stepwise Selection Method.
Box 12.5 shows the results of the Forward selection procedure, which is requested in
SPSS by using the /FORWARD keyword in place of /ENTER or /STEP. The results of this
analysis are straightforward. AGE enters as a predictor at Step 1, because AGE has the strongest
relation with RT. Once AGE is in the equation, none of the other predictors meet the PIN
probability level, the closest is Length (LEN) with a p of only .331. SPSS settles for an equation
with just one predictor. Note in Box 12.5 that the predictors were listed on the /FORWARD
subcommand rather than being entered on a separate /VARIABLES subcommand. This approach
is available for the various automated methods available in SPSS.
Box 12.6 reports the results for the Stepwise selection procedure. Since Stepwise is a
combination of Forward and Backward methods, we should not be surprised that the resulting
equation has a single predictor, age of acquisition. The output from the Stepwise and Forward
SPSS and Multiple Predictors 12.14
REGRE /VARI = rt age con frq len /DEP = rt /BACKWARD.
Model Variables Variables Method Entered Removed 1 LEN, FRQ, CON, . Enter AGE(a) 2 . FRQ Backward (criterion: Probability of F-to-remove >= .100). 3 . CON Backward (criterion: Probability of F-to-remove >= .100). 4 . LEN Backward (criterion: Probability of F-to-remove >= .100).
Model R R Square Adjusted R Std. Error of Square the Estimate 1 .638(a) .407 .339 92.6989896 2 .623(b) .388 .337 92.8631792 3 .600(c) .360 .325 93.7129335 4 .585(d) .343 .325 93.6742945
Model Sum of Squares df Mean Square F Sig. 1 Regression 206598.382 4 51649.595 6.011 .001(a) Residual 300758.593 35 8593.103 Total 507356.975 39
2 Regression 196908.453 3 65636.151 7.611 .000(b) Residual 310448.522 36 8623.570 Total 507356.975 39
3 Regression 182418.761 2 91209.380 10.386 .000(c) Residual 324938.214 37 8782.114 Total 507356.975 39
4 Regression 173911.784 1 173911.784 19.819 .000(d) Residual 333445.191 38 8774.873 Total 507356.975 39
Model Unstandardized Standardized t Sig. Coefficients Coefficients B Std. Error Beta 1 (Constant) 622.601 95.116 6.546 .000 AGE 14.751 8.910 .323 1.656 .107 CON 15.073 9.795 .235 1.539 .133 FRQ 1.147 1.080 .175 1.062 .296 LEN 14.700 8.972 .255 1.638 .110
2 (Constant) 675.100 81.402 8.293 .000 AGE 20.524 7.073 .449 2.902 .006 CON 12.237 9.440 .191 1.296 .203 LEN 11.485 8.461 .199 1.357 .183
3 (Constant) 728.287 70.948 10.265 .000 AGE 24.580 6.401 .538 3.840 .000 LEN 7.957 8.084 .138 .984 .331
4 (Constant) 775.439 52.307 14.825 .000 AGE 26.747 6.008 .585 4.452 .000
Excluded Variables(d) Model Beta In t Sig. Partial Collinearity Correlation Statistics Tolerance 2 FRQ .175(a) 1.062 .296 .177 .626 3 FRQ .106(b) .655 .517 .109 .676 CON .191(b) 1.296 .203 .211 .784 4 FRQ .059(c) .375 .710 .062 .726 CON .127(c) .898 .375 .146 .874 LEN .138(c) .984 .331 .160 .882
Box 12.7. Backward Selection Analysis.
selection methods are identical for this study. Age of acquisition is the only predictor that gets
entered into the equation. Perhaps the Backward procedure will produce different results.
SPSS and Multiple Predictors 12.15
Box 12.7 shows the Backward selection results, requested in SPSS using the
/BACKWARD keyword. Backward selection begins with all variables in the equation, and this is
shown as steps 1 to 4 at the first block. The initial regression is identical to that in Box 12.4,
when all predictors were forced into the equation with /ENTER. Now, however, SPSS has been
requested to remove variables whose probabilities are less than .10, the default value for POUT.
FRQ is removed first because it has the highest probability level when all predictors are in
the equation, p = .2956, and its probability is greater than POUT = .10. SPSS proceeds to remove
in turn CON, p = .2031, and finally LEN, p = .3314. Note that p values are recomputed after each
predictor is removed from the equation. All predictors but AGE end up being removed, and AGE
becomes increasingly significant as more of the competing predictors are excluded. The net result
is an equation with just one predictor, age of acquisition, the same outcome as occurred with the
Forward selection procedure. To the extent that these methods are adequate for building
meaningful regression models, the one-predictor equation looks increasingly plausible.
All three selection methods resulted in a single equation with age of acquisition as the
predictor. Researchers who put excessive faith in blind selection methods might believe that this
is in fact the most sensible model for the data. But in the present case, we have the advantage of
knowing how the data were generated, which means that the results of the automated regression
analyses can be compared to the "true" underlying structure.
The data were actually produced as follows. Random scores were produced for the item
attributes of Concreteness, Frequency, and Length. These scores (plus some random variation)
were then used to generate both the Reaction Time and the Age of Acquisition scores. Age of
Acquisition scores were NOT used directly in the generation of RT scores; that is, there is no
direct causal relation between AGE and the RT data. Rather, AGE is highly correlated with RT
because both AGE and RT were determined by Concreteness, Frequency, and Length. In a sense,
the high correlation between AGE and RT fooled the regression program and its statistical criteria
into choosing AGE as the primary and indeed the only predictor of RT.
Box 12.8 shows the results of a regression analysis that the automated procedures never
got to; namely, the regression of RT on CON, FRQ, and LEN without AGE in the equation. The
prediction is very good, slightly stronger than AGE alone comparing R2s, and each of the
SPSS and Multiple Predictors 12.16
REGRE /STAT = DEFAU ZPP /DEP = rt /ENTER con frq len.
Model R R Square Adjusted R Std. Error of Square the Estimate 1 .601(a) .361 .308 94.9137441
Model Sum of Squares df Mean Square F Sig. 1 Regression 183046.697 3 61015.566 6.773 .001(a) Residual 324310.278 36 9008.619 Total 507356.975 39
Model Unstandardized Standardized t Sig. Correlations Coefficients Coefficients B Std. Error Beta Zero-order Partial Part 1 (Constant) 584.562 94.505 6.186 .000 CON 23.240 8.664 .363 2.682 .011 .318 .408 .357 FRQ 2.237 .876 .341 2.554 .015 .349 .392 .340 LEN 22.570 7.791 .391 2.897 .006 .323 .435 .386
Box 12.8. MR Analysis for ENTER without AGE.
predictors makes a highly significant contribution to the regression equation. These significant
effects were overshadowed by the domineering AGE variable, and its spurious correlation with
RT.
Perhaps the most appropriate conclusions from this hypothetical study are that
Concreteness, Length, and Frequency contribute significantly to both RT and Age of Acquisition,
but that RT and AGE are not themselves directly linked. The relations between AGE and the
other predictors makes sense if subjects in the Age of Acquisition rating task did not actually
remember at what age they learned the words (a plausible assumption?), but rather made
inferences about Age of Acquisition from other word attributes. That is, subjects know that
concrete, short, and familiar words are learned earlier than abstract, long, and less familiar words.
This example involved only four predictors, yet still presented analytic challenges for the
standard MR selection procedures. The theoretical and empirical difficulties become even more
subtle and complex as the number of predictors increases, so users of MR must prepare for careful
theoretical and empirical analysis of the relations among variables in MR studies. Only
occasionally will the relations be sufficiently simple to trust automated analysis using purely
statistical selection criteria, and unfortunately, researchers never know which occasions are the
simple ones. Some of these complexities are described shortly.
SPSS and Multiple Predictors 12.17
Interpretation of Unique Contribution
Irrespective of how the final model is arrived at, the interpretation of the unique
contribution of each predictor follows the basic principles described previously. Boxes 12.9 and
12.10 demonstrate the application of our principles to understanding the unique contribution of
CON in the regression analysis of Box 12.8. Box 12.9 shows the change analysis with CON
added after FRQ and LEN are already in the equation.
The difference between SSRegression for Models 1 and 2 (or equivalently between SSResidual)
would give SSR’C.FL, which divided by 9008.619 would give FChange = 7.196 = t2 = 2.6822. Dividing
SSR’C.FL by SSTotal would give the part r2, r2R(C.FL) = .128 = .361 - .233 = .3572. Dividing SSR’C.FL by
389133.844 would give the partial r2, r2RC.FL = .4082.
REGRE /STAT = DEFAU CHANGE ZPP /DEP = rt /ENTER frq len /ENTER con.
Model R R Adjusted Std. Error of Change Statistics Square R Square the Estimate R Square Change F Change df1 df2 Sig. F Change 1 .483(a) .233 .192 102.5530639 .233 5.621 2 37 .007 2 .601(b) .361 .308 94.9137441 .128 7.196 1 36 .011
Model Sum of Squares df Mean Square F Sig. 1 Regression 118223.131 2 59111.565 5.621 .007(a) Residual 389133.844 37 10517.131 Total 507356.975 39
2 Regression 183046.697 3 61015.566 6.773 .001(b) Residual 324310.278 36 9008.619 Total 507356.975 39
Model Unstandardized Standardized t Sig. Correlations Coefficients Coefficients B Std. Error Beta Zero-order Partial Part 1 (Constant) 715.919 87.334 8.197 .000 FRQ 2.356 .945 .359 2.492 .017 .349 .379 .359 LEN 19.246 8.311 .334 2.316 .026 .323 .356 .333
2 (Constant) 584.562 94.505 6.186 .000 FRQ 2.237 .876 .341 2.554 .015 .349 .392 .340 LEN 22.570 7.791 .391 2.897 .006 .323 .435 .386 CON 23.240 8.664 .363 2.682 .011 .318 .408 .357
Box 12.9. Change Statistics for Unique Contribution of CON.
SPSS and Multiple Predictors 12.18
Box
12.10 shows the
creation of
residual
predictor and
criterion
variables that
also produce
the part and partial rs (shown
in italics and bold in the
correlation matrix).
It will always be the
case that our methods for
understanding the unique
contribution of predictors in
multiple regression can be
applied to regression
analyses of any sort (i.e.,
using ENTER, FORWARD,
BACKWARD, or
STEPWISE).
SPSS Regression Options
and Menus
Figure 12.2 shows
again how Stepwise
regression would be accessed
by the menu system. The
various independent
variables are selected and
REGRESS /DEP = con /ENTER frq len /SAVE RESI(resc.fl).REGRESS /DEP = rt /ENTER frq len /SAVE RESI(resr.fl).VARIABLE LABELS resc.fl '' resr.fl ''.CORR rt frq len resc.fl resr.fl.
RT FRQ LEN resc.fl resr.fl RT Pearson 1 .349 .323 .357 .876 FRQ Pearson .349 1 -.030 .000 .000 LEN Pearson .323 -.030 1 .000 .000 resc.fl Pearson .357 .000 .000 1 .408
resr.fl Pearson .876 .000 .000 .408 1
Box 12.10. Part and Partial Using Residual Variables.
Figure 12.2. Stepwise Regression Using Menus (and
Corresponding Syntax).
SPSS and Multiple Predictors 12.19
moved into the Independent(s) field and the Stepwise Method is selected. The resulting syntax is
shown in the output screen above the Regression screen. Rather than being listed on a
/VARIABLES = subcommand, the predictors are included on the /METHOD = STEPWISE
subcommand. The actual analysis would be identical to that shown in Box 12.6.
COMPLEXITIES IN REGRESSION MODEL BUILDING
Although computers make possible the calculation-intensive but automatic selection
procedures described above, no purely statistical method is completely satisfactory for the
selection of predictors in MR designs. Researchers must interpret findings with an understanding
of the theoretical meaning of the constructs and relations being examined, something automated
procedures do not do, and with an adequate understanding of the significance tests and other
statistics reported by MR for the individual predictors and used by the various selection
procedures. In short, care with both theory and data analysis is essential for reducing the
likelihood of inappropriate interpretations from MR designs. We consider three simple examples
of problems that can arise: redundant predictors, coincident criterion variables as predictors, and
suppressor effects.
Redundant Predictors
Consider first a study in which two highly overlapping predictors are included in a
regression analysis; perhaps the variables are alternative measures of the same underlying
construct. To illustrate, educational researchers in the Study Time project could have measured
both Study Time and self-reported Effort for each student. It is likely that Study Time and Effort
would be highly correlated, since amount of studying would be a major contributor to or
manifestation of effort.
Regressing grades on both Study Time and Effort and examining their individual
contribution would be equivalent to producing residual scores for Study Time partialling out
Effort, and residual scores for Effort partialling out Study Time. The residual scores for Study
Time would be independent of Effort, and the residual scores for Effort would be independent of
Study Time. Such residuals could be conceptually meaningless. It seems possible, perhaps even
probable, that we would remove the variability in one or both of the scores that is responsible for
SPSS and Multiple Predictors 12.20
its relation with grades. That is, Study Time independent of Effort and Effort independent of
Study Time might be uncorrelated with grades, and might also represent unusual and perhaps
uninteresting aspects of the data.
What could happen with blind use of an automated selection procedure is that one of the
variables would be selected for entry into the equation and the other variable excluded.
Whichever variable happened to have the highest simple or partial correlation with the criterion
grades would be entered, no matter how slight its superiority. The other predictor would not be
selected because it no longer met the probability for entry, once its correlated partner was already
in the equation. Naive researchers might conclude that the omitted variable (e.g., Study Time)
was not important for grades, when in fact a more correct interpretation would be that the critical
variability in Study Time has already been credited to the Effort variable.
A more adequate solution to this problem would be to combine the Study Time and Effort
measures together to obtain a single predictor that reflected some underlying composite or general
construct, perhaps a revised construct of Effort that included study effort. An alternative approach
would be to identify aspects of Effort that are potentially independent of Study Time (e.g., talking
to teacher or other students about course) and using both Study Time and this "purer" and perhaps
less correlated measure of non-studying Effort as predictors.
Coincident Outcomes as Predictors
A related problem arises when one predictor is determined by a collection of the other
predictors, all of which have relations to the criterion. Consider the previously-mentioned study
of picture naming latency with mean naming reaction time (RT) for a large set of words as the
dependent variable. Each variable has scores for length in letters, frequency in print,
concreteness, and rated age of word acquisition, and these are used to predict naming RT.
Stepwise or some other selection procedure might result in an equation with age of acquisition as
the sole predictor, because its correlation with RT tends to be higher than each of the other
variables. Given age of acquisition is in the equation, the other predictors make no additional
contribution to the prediction of RT.
SPSS and Multiple Predictors 12.21
Figure 12.3. Relation of Predictors to RT and
Age of Acquisition.
Figure 12.4. Negative Relation Between
Predictors Positively Related to Criterion.
Any conclusion that gives undue
weight to this finding may be incorrect if age
of acquisition is itself dependent on length,
frequency, and concreteness. That is, perhaps
subjects do not rate age of acquisition directly,
but rather give early ratings to words that are
short, frequent, and concrete, and late ratings
to words that are long, infrequent, and abstract. The hypothesized model is diagrammed in Figure
12.3. Age of acquisition is the best predictor of RT not because age of acquisition contributes
directly to naming RT, but because age is a proxy for all of the other variables that do contribute
to RT. That is, age includes predictive variability associated with length, concreteness, and
frequency, not leaving anything to be accounted for by those variables, the "true" predictors in this
hypothetical situation.
Suppressor Effects
Other problems, such as suppressor effects, may be specific to particular selection
procedures. For example, two negatively correlated predictors can have hidden positive effects on
a criterion variable. Their positive effects could be hidden or masked because of their negative
relation with the other predictor that is also positively related to the criterion. One example of
such a situation, which we analyzed in earlier chapters, is illustrated in Box 12.3. Grades are
positively determined by ability and study time, which are negatively related to one another.
Such underlying structures can pose
problems for some automated selection
methods. If the simple relations with grades
not controlling for the other variables are
nonsignificant, then the two variables might
never get in the equation together using the
Forward or Stepwise selection procedures.
The simple correlations (or part correlations
with all but the critical remaining predictor) never achieve the requisite probability level to be
SPSS and Multiple Predictors 12.22
entered because of the masking effect of the confounded predictor. The Backward selection
procedure would not have this difficulty as long as the probabilities for the effects both remained
low enough to remain in the equation. If the probability for one of the variables increased to
above the critical value, however, then it and subsequently the other predictor would be ejected.
Redundant predictors, coincident outcomes, and suppressor effects illustrate just a few of
the complexities of multiple regression. Overcoming these and other difficulties requires an
adequate conceptual understanding of the domain being investigated, knowledge about the
statistical methods that underlie MR, experience, and a healthy dose of skepticism about the
adequacy of automated selection procedures.
Testing All Possible Equations
Another approach to regression modelling that is also flawed but sometimes used by
researchers is to examine all-possible equations to determine the best combination of predictors.
Although this approach has the
potential to identify some of the
complex relationships just
discussed, it is also extremely
susceptible to the problem of
inflated Type I errors. That is,
all-possible regressions is prone
to identifying predictors as
significant simply because of the
inflated probability of a Type I
error. Although SPSS does not
automatically examine all-
possible equations, it is relatively
simple to obtain relevant basic
statistics except for exceptionally
large numbers of predictors.
Box 12.11 shows one way to efficiently get basic statistics (in particular R and R2) for all
*All Possible Regressions: # equations = 2^4 - 1 = 15. Model R R Square Adjusted R Std. Error of Square the EstimateREGR /STAT = R /DEP = rt /ENTER age. 1 .585(a) .343 .325 93.6742945REGR /STAT = R /DEP = rt /ENTER con. 1 .318(a) .101 .078 109.5332346REGR /STAT = R /DEP = rt /ENTER frq. 1 .349(a) .122 .099 108.2802034REGR /STAT = R /DEP = rt /ENTER len. 1 .323(a) .104 .081 109.3570377REGR /STAT = R /DEP = rt /ENTER age con. 1 .597(a) .357 .322 93.9146992REGR /STAT = R /DEP = rt /ENTER age frq. 1 .588(a) .345 .310 94.7517013REGR /STAT = R /DEP = rt /ENTER age len. 1 .600(a) .360 .325 93.7129335REGR /STAT = R /DEP = rt /ENTER con frq . 1 .460(a) .212 .169 103.9639084REGR /STAT = R /DEP = rt /ENTER con len. 1 .495(a) .245 .204 101.7493731REGR /STAT = R /DEP = rt /ENTER frq len. 1 .483(a) .233 .192 102.5530639REGR /STAT = R /DEP = rt /ENTER age con frq. 1 .601(a) .362 .309 94.8428311REGR /STAT = R /DEP = rt /ENTER age con len. 1 .623(a) .388 .337 92.8631792REGR /STAT = R /DEP = rt /ENTER age frq len. 1 .606(a) .367 .314 94.4441804REGR /STAT = R /DEP = rt /ENTER con frq len. 1 .601(a) .361 .308 94.9137441REGR /STAT = R /DEP = rt /ENTER age con frq len. 1 .638(a) .407 .339 92.6989896
Box 12.11. All Possible Regression Statistics.
SPSS and Multiple Predictors 12.23
possible regression equations for the naming study. Although 15 separate regressions must be
done, including /STATISTICS = R greatly reduces the output. Examining the R2s in Box 12.11
reveals that the amount of variation accounted for is roughly equivalently for the following 7
equations: any equation involving Age, either alone or with one, two, or all three other predictors,
and the equation with just the three other predictors of Con, Frq, and Len. This pattern of results
could lead to an interpretation such as that suggested earlier, namely that Age overlaps with all
three other predictors and is therefore the best single predictor of the four. Although in this case
the simple correlation matrix helps to support this same interpretation, other more complex
patterns of relationships among predictors might not be discovered without examining all-possible
combinations (or subsets) of predictors, as in Box 12.9.
SPSS has another regression option that can help with deciphering complex regressions.
The /TEST option in Regression allows users to specify sets of predictors for which basic change
statistics will be computed (in particular, R2). But it is important to understand the way in which
/TEST works. Specifically, it tests whether removal of the specified set of predictors results in a
REGRE /DEP = rt /TEST (age) (con frq len).
Model Sum of Squares df Mean Square F Sig. R Square Change
1 Subset AGE 23551.684 1 23551.684 2.741 .107(a) .046 Tests CON, FRQ, LEN 32686.598 3 10895.533 1.268 .300(a) .064
Regression 206598.382 4 51649.595 6.011 .001(b) Residual 300758.593 35 8593.103 Total 507356.975 39
REGR /STAT = DEFAU CHANGE /DEP = rt /ENTER con frq len /ENTER age.
Model R R Adjusted Std. Error of Change Statistics Square R Square the Estimate R Square Change F Change df1 df2 Sig. 1 .601(a) .361 .308 94.913744124 .361 6.773 3 36 .001 2 .638(b) .407 .339 92.698989556 .046 2.741 1 35 .107
REGR /STAT = DEFAU CHANGE /DEP = rt /ENTER age con frq len /REMOVE age.
Model R R Adjusted Std. Error of Change Statistics Square R Square the Estimate R Square Change F Change df1 df2 Sig. 1 .638(a) .407 .339 92.698989556 .407 6.011 4 35 .001 2 .601(b) .361 .308 94.913744124 -.046 2.741 1 35 .107
REGR /STAT = DEFAU CHANGE /DEP = rt /ENTER age /ENTER con frq len.
Model R R Adjusted Std. Error of Change Statistics Square R Square the Estimate R Square Change F Change df1 df2 Sig. 1 .585(a) .343 .325 93.674294525 .343 19.819 1 38 .000 2 .638(b) .407 .339 92.698989556 .064 1.268 3 35 .300
Box 12.12. Using /TEST Option to Test Contribution of Single Predictor.
SPSS and Multiple Predictors 12.24
substantive change in R2 or F. Box 12.12 demonstrates the procedure for the word naming study,
as well as the more familiar regression commands that would produce equivalent statistics. The
first regression uses the /TEST option to determine the contribution of the two subsets listed to the
overall regression. Here we see that Age does not contribute significantly over and above Con,
Frq, and Len, F = 2.741, p = .107, R2Change = .046.
Simultaneously, we see that Con, Frq, and Len together (df = 3) do not contribute
significantly over and above Age alone, F = 1.268, p = .300, R2Change = .064. Note that these
statistics are equivalent to those shown for Model 2 in the Change Statistics section of the last
regression analysis. Model 2 was when Con, Frq, and Len were added to Age, hence dfNumerator =
3. Although this is one of the few times that we have examined change statistics for more than
one additional predictor, the procedures would be similar to those we have done before; namely,
SSChange = SSy.ACFL - SSy.A, and dfNumerator for SSChange = number of additional predictors = 3.
Box 12.13 illustrates the use of /TEST to examine the removal of all-possible
combinations of predictors. This is the converse of the analyses in Box 12.9, where all possible
combinations of predictors were entered into the equation. Two analyses were used in Box 12.13
REGRE /DEP = rt /TEST (age) (con) (frq) (len) (age con) (age frq) (age len) (con frq) (con len) (frq len).
Model Sum of Squares df Mean Square F Sig. R Square Change 1 Subset AGE 23551.684 1 23551.684 2.741 .107(a) .046 CON 20350.722 1 20350.722 2.368 .133(a) .040 FRQ 9689.929 1 9689.929 1.128 .296(a) .019 LEN 23067.261 1 23067.261 2.684 .110(a) .045 AGE, CON 88375.251 2 44187.625 5.142 .011(a) .174 AGE, FRQ 82299.999 2 41149.999 4.789 .015(a) .162 AGE, LEN 99155.694 2 49577.847 5.769 .007(a) .195 CON, FRQ 24179.621 2 12089.811 1.407 .258(a) .048 CON, LEN 31423.148 2 15711.574 1.828 .176(a) .062 FRQ, LEN 25580.324 2 12790.162 1.488 .240(a) .050
Regression 206598.382 4 51649.595 6.011 .001(b) Residual 300758.593 35 8593.103 Total 507356.975 39
REGRE /DEP = rt /TEST (age con frq) (age con len) (con frq len) (age con frq len).
Model Sum of Squares df Mean Square F Sig. R Square Change 1 Subset AGE, CON, FRQ 153681.951 3 51227.317 5.961 .002(a) .303 AGE, CON, LEN 144776.300 3 48258.767 5.616 .003(a) .285 CON, FRQ, LEN 32686.598 3 10895.533 1.268 .300(a) .064 AGE, CON, FRQ, 206598.382 4 51649.595 6.011 .001(a) .407 LEN
Regression 206598.382 4 51649.595 6.011 .001(b) Residual 300758.593 35 8593.103 Total 507356.975 39
Box 12.13. Using /TEST to Examine Removal of Combinations of Predictors.
SPSS and Multiple Predictors 12.25
because of memory limitations (i.e., SPSS would not examine all possible combinations without
increasing the amount of computer memory available for the analysis). Note that the final R2s in
Boxes 12.11 and 12.13 equal .407, whether it is the R2 for entering all four predictors (Box 12.11)
or for removing all four predictors (Box 12.13). Box 12.13 reveals that the effect of removing
each predictor individually, including Age, is modest, as is the effect of removing various
combinations of Con, Frq, and Len. But removing Age plus any combination of the other
predictors results in a more substantial decrease. It is perhaps more challenging, but one can
through thoughtful examination of the R2s in Box 12.13 arrive at the combination of Con, Frq, and
Len as arguably the most intelligible combination of predictors.
Although all-possible subsets of predictors and SPSS’s /TEST procedure are susceptible to
many of the same flaws as the automated procedures (i.e., STEPWISE, ...), there is one important
difference — researchers can use their judgment and knowledge of theory and previous findings
to inform their selection of the “most appropriate” model. Selection is not simply a function of
some uninformed statistical criterion.
CONCLUSIONS
Chapter 11 demonstrated that the interpretation of individual predictors in multiple
predictor studies involves a straightforward extension of the approaches developed for the two-
predictor MR design. Each predictor's contribution controls statistically for all other predictors in
the equation. Unfortunately, not all aspects of multiple regression with many predictors are as
straightforward, and Chapter 12 has described some special tools that have been constructed to
help with deciphering the most meaningful combination(s) of predictors. However, the number
and subtle influences on one another of multiple predictors present additional complexities for the
selection of appropriate models using Forward, Backward, Stepwise, or other approaches to
predictor selection. Purely statistical methods must be balanced by thoughtful consideration of the
theory that underlies the variables and relations of interest.
SPSS and Multiple Predictors 12.26
SET SEED = 27395137INPUT PROGRAMLOOP SUBJ = 1 TO 24COMP #z1 = NORMAL(1)COMP #z2 = NORMAL(1)COMP grde = RND(65+10*(#z1*.4+#z2*.4+NORMAL(1)*SQRT(1-.68**2)))COMP abil = RND(100 + 15*#z1)COMP stdy = RND( 20 + 5*(#z1*-.5 + #z2*SQRT(1-.5**2)))COMP wght = RND(140 + NORMAL(20))END CASEEND LOOPEND FILEEND INPUT PROGRAM
SET SEED = 39513773INPUT PROGRAMLOOP WORD = 1 TO 40COMP #c = NORM(1) /* concretenessCOMP #f = NORM(1) /* frequencyCOMP #l = NORM(1) /* lengthCOMP rt = RND(1000 +100*(#c*.5+#f*.5+#l*.5+NORM(1)*SQRT(1-.75**2)))COMP age = RND( 8 + 2*(#c*.5+#f*.5+#l*.5+NORM(1)*SQRT(1-.75**2)))COMP con = RND( 5 + 1.5*(#c*.8 + NORM(1)*SQRT(1-.8**2)))COMP frq = RND( 50 + 15 *(#f*.8 + NORM(1)*SQRT(1-.8**2)))COMP len = RND( 8 + 2 *(#l*.8 + NORM(1)*SQRT(1-.8**2)))END CASEEND LOOPEND FILEEND INPUT PROGRAM
Notes
Note 12.1. The following SPSS program was used to generate the new study time data,
including weight as an additional predictor. The program assumes that grades are dependent
positively on both ability and study time, that study time is negatively related to ability, and that
weight is independent of ability, study time, and grades. Twenty-four observations are randomly
selected from a population of scores with these characteristics.
Note 12.2. The following SPSS program was used to generate the data for the naming
study. See the text for discussion of the underlying structure, which in essence creates both
reaction time and age of acquisition scores as a function of concreteness, word length, and
frequency.
MR and Non-Linear Relationships 13.1
CHAPTER 13 -
MULTIPLE REGRESSION AND NON-LINEAR RELATIONSHIPS
Using Plots to Identify Nonlinear Relationships . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
Plot of Dependent Variable Against Predictor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
Plot of Residuals Against Predictor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
Polynomial Regression . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
Plotting Nonlinear (Quadratic) Equations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
Value of Changing Slope . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
Transforming Variables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
Using COMPUTE to Transform Predictors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
Using SPSS’s Curve Estimation Procedure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
Discussion of Transformations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
Imitation Study . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
MR and Non-Linear Relationships 13.2
Standard correlation and regression methods are designed for linear relationships between
the criterion and predictor variables. Linear relations are those in which changes on the predictor
are associated with constant changes on the criterion (i.e., y increases or decreases by the same
amount when x changes from 0 to 1 as when x changes from 10 to 11). Changes in y that vary
across different levels of a single predictor indicate a nonlinear relation. Unless prepared for
nonlinear relationships, researchers might wrongly conclude that there is a linear relation
between the variables or, in extreme cases of curvilinearity, that there is no relationship between
the criterion and the predictor. Chapter 13 examines several ways of determining whether a
nonlinear relationship exists, and if so, of analyzing such data.
USING PLOTS TO IDENTIFY NONLINEAR RELATIONSHIPS
Consider a study of the
effect on reaction time (RT) of
number of alternatives
(uncertainty or U). In each of
8 groups, 4 subjects pressed
different keys as rapidly as
possible when numbers appeared on a computer screen (e.g., key 1 for the number 1, key 2 for 2,
and so on). Subjects in the lowest uncertainty group (U = 1) only saw one number and pressed
one key. Subjects in U = 2, had two stimuli and responses, and so on up to U = 8, where subjects
saw one of 8 numbers and pressed one of 8 keys. RT to press the key was determined in ms
(milliseconds or 1000ths of a second). The results are shown in Box 13.1, along with the SPSS
command to plot RT as a function of uncertainty (u).
DATA LIST FREE / u rt.BEGIN DATA1 303 1 297 1 296 1 312 2 318 2 320 2 323 2 3363 342 3 340 3 342 3 347 4 363 4 364 4 353 4 3495 363 5 366 5 379 5 349 6 373 6 367 6 389 6 3707 374 7 367 7 377 7 396 8 392 8 390 8 387 8 386END DATA.GRAPH /SCATTERPLOT(BIVAR) = u WITH rt.
Box 13.1. RTs as a Function of Stimulus Uncertainty.
MR and Non-Linear Relationships 13.3
Uncertainty
1086420
Reaction T
ime
400
380
360
340
320
300
280
R^2 = .88
Figure 13.1. Observed and Predicted (Linear)
RTs.
Plot of Dependent Variable Against Predictor
Figure 13.1 shows the plot of the
observed RTs at each uncertainty level, as
well as the predicted RTs from a linear
regression of RT on Uncertainty. Actual
regression analyses are presented later.
Figure 13.1 shows that RT increased (people
got slower) as the number of alternative
responses increased. This result agrees with
several theories of choice RT because, as
uncertainty increases, the mind must make
more decisions to identify which stimulus
occurred and to select the appropriate
response. Simple regression indicates a strong linear component to the relation between
Uncertainty and RT, as indicated by R2 = .88. The fit to the observed data is quite good, as
would be expected in a predictor that explains 88% of the variability in RT.
Despite the robust linear effect, however, Figure 13.1 reveals that the effect on RT of
increasing Uncertainty is not uniform across all levels of Uncertainty. Changes in U at the low
end (1 to 2 to 3) produce larger increases in RT than changes in U at the upper end (6 to 7 to 8).
The rate of increase in RT levels off or decreases as Uncertainty increases, leading to a flattening
of the relationship at the upper levels of uncertainty. These systematic deviations from linearity
can be seen by comparing the observed RTs to the RTs predicted by the best-fit linear equation.
As shown in Figure 13.1, the linear equation predicts RTs that are too high (slow) at the low and
high ends of U and RTs that are too low (fast) at the middle levels of U. The observed values
start out lower than the predicted values, rise above the predicted values in the middle range of
U, and then curve below again. Such systematic deviations from predicted values indicate a
nonlinear relation. If the relation between RT and U were linear, deviations from the predicted
values (i.e., residual scores) would be scattered randomly about the best-fit linear equation over
the entire range of Uncertainty. That is, the best-fit line would pass through the center of the
MR and Non-Linear Relationships 13.4
Uncertainty
1086420
Re
sid
ua
l S
co
re (
Lin
ea
r)
20
10
0
-10
-20
Figure 13.2. Plot of Linear Residuals Against U.
observations at all levels of uncertainty.
Plot of Residuals Against Predictor
This non-linear effect can be seen even more clearly by plotting residuals from the linear
regression against the predictor, U.
Figure 13.2 shows the plot. The
horizontal dashed line at 0 indicates the
mean of the residual scores; recall that
residual scores sum to 0 and hence have
a mean of 0. If there were no nonlinear
relationship between RT and U, then the
residuals in Figure 13.2 would be
scattered randomly about the mean of 0
over the full range of the predictor U.
But clearly, the residual scores are not
random. In fact, low and high values of
U are associated with predictions that are too high (i.e., the residuals fall below the overall mean
of 0), and middle values of U are associated with predictions that are too low (i.e., these residuals
fall above their overall mean of 0).
Figures 13.1 and 13.2 show visual ways to determine whether a relationship is linear or
non-linear; that is, plot the original y scores or the residual scores from a linear regression against
the original predictor scores. These procedures are particularly robust when nonlinear relations
are quite striking, although they still involve some degree of subjective judgment. The plots of
residuals are generally somewhat more sensitive than plots of the original y scores. But better yet
is to perform statistical procedures to determine whether deviations from linearity are significant
and how significant they are. Such procedures are described in the following sections.
MR and Non-Linear Relationships 13.5
POLYNOMIAL REGRESSION
To determine whether the apparent deviation from linearity is significant or should be
attributed to chance, researchers can use multiple regression with polynomial predictors.
Polynomial predictors are similar to interaction terms (discussed in a later chapter) that involve
multiplying predictors together to generate additional predictors sensitive to differences in the
slope for one variable at different levels of another variable. In the case of non-linear regression,
a single predictor is multiplied by itself one or more times. That is, in addition to x in the
equation, we include x2 (x × x) and occasionally higher powers (i.e., x3, x4, ...). The variable x is
the linear term, x2 the quadratic term, x3 the cubic term, and so on. Including these powers of x
in the prediction equation provides one way to accommodate nonlinear relationships.
Box 13.2 shows the polynomial regression for the uncertainty study. A new predictor, U2,
COMPUTE u2 = u**2.REGRESS /VARI = rt u u2 /STAT = DEFAU CHANGE ZPP /DEP = rt /ENTER u /ENTER u2 /SAVE PRED(prdr.uu2) RESI(resr.uu2).
R R Adjusted Std. Error of Change Statistics Square R Square the Estimate Model R Square Change F Change df1 df2 Sig. 1 .938(a) .880 .876 10.1966576 .880 220.605 1 30 .000 2 .962(b) .925 .920 8.1910931 .045 17.489 1 29 .000
Model Sum of Squares df Mean Square F Sig. 1 Regression 22936.720 1 22936.720 220.605 .000(a) Residual 3119.155 30 103.972 Total 26055.875 31
2 Regression 24110.149 2 12055.074 179.674 .000(b) Residual 1945.726 29 67.094 Total 26055.875 31
Unstand Coeff Stand Coeff t Sig. CorrelationsModel B Std. Error Beta Zero Partial Part1 (Constant) 301.482 3.973 75.891 .000 U 11.685 .787 .938 14.853 .000 .938 .938 .938 U2
2 (Constant) 281.661 5.714 49.294 .000 U 23.577 2.913 1.893 8.093 .000 .938 .833 .411 U2 -1.321 .316 -.978 -4.182 .000 .870 -.613 -.212
Residuals Statistics(a) Minimum Maximum Mean Std. Deviation N Predicted Value 303.916656 385.708344 354.062500 27.8881110 32 Residual -17.511906 14.047619 .000000 7.9224593 32
Box 13.2. Polynomial Regression with U and U2 as Predictors.
MR and Non-Linear Relationships 13.6
Uncertainty
1086420
Re
actio
n T
ime
400
380
360
340
320
300
280
Figure 13.3. Polynomial Prediction Equation.
is first computed and then a regression analysis includes both U and U2 as predictors of RT. U
and U2 have been added sequentially to demonstrate more fully the additional contribution of the
non-linear component. Adding U2 increases R2 by .045 (part r = -.212), which is a highly
significant change, FChange = 17.489, p = .000, or equivalently, tU2 = -4.182, p = .000.
Note that the coefficient for U is much steeper in Model 2, BU = 23.577 than in Model 1
where BU = 11.685, and that the slope for U2 is negative, BU2 = -1.321. Essentially what this
equation does is start out with a much steeper increase in RT as a function of initial increases in
U, but the slope becomes increasingly shallow as U increases. Specifically, the multiple
regression of RT on U and U2 produces � = 281.661 + 23.577 × U -1.321 × U2. The coefficient
for U is now almost twice as large as it was when RT was regressed on U alone. Moreover, R2
has increased significantly, from .876 to .920, and the negative coefficient for U2 is significant, as
shown by tbu2 (tU2 = -4.182, p = .0002) or the equivalent FChange. We later discuss these statistics
and others for U2 in more detail.
Plotting Nonlinear (Quadratic) Equations
This effect, and the improvement in the prediction can be seen in Figure 13.3, which
shows the predicted scores given the non-linear equation including both U and U2. The
polynomial prediction equation goes more clearly through the center of all the data points across
the entire range of the Uncertainty predictor.
There is no obvious tendency for the equation
to under-predict or over-predict at different
locations.
Figure 13.3 was produced in two steps.
First a scattergram was requested using either
the menu system, Graph | Scatter | Simple, or
the following syntax: GRAPH
/SCATTERPLOT(BIVAR) = u WITH RT. Then
the Chart Editor was activated by double-
clicking on the original graph. One option
available in the Chart Editor is to fit either
MR and Non-Linear Relationships 13.7
linear or quadratic equations to the data. Figure 13.3 shows the Quadratic fit (Figure 13.1 shows
the Linear fit). The curve in Figure 13.3 generates predicted values closer to the actual data than
does the straight line in Figure 13.1. In particular, there is no longer any systematic under- or
over-predicting across the range of uncertainty values. Predictions fall in the “middle”
throughout, with residuals both above and below the predicted values.
Value of Changing Slope
Because U2 = U × U, the best-fit regression equation can be rewritten as: � = 281.661 +
(23.577 - 1.321 × U) × U. The value in parentheses (23.577 - 1.321 × U) becomes smaller as U
increases; for example, it will be 23.588 - 1.321 × 1 = 22.256 for U = 1 and 23.577 - 1.321 × 8 =
13.01 for U = 8. Because the adjusted value in parentheses is then multiplied by U, the predicted
change in RT as U increases becomes smaller as U gets larger. For U = 1, � = 281.661 +
22.256×1 = 303.917. For U = 8, � = 281.661 + 13.01×8 = 385.74. Thus the polynomial term
reflects the change in the slope as the predictor increases, and the predicted values level off at the
upper end of U.
Although the preceding description provides an intuitive sense that the slope will change
as U increases, it does not in fact provide the proper way to calculate what the actual slope will
be for any particular value of x. The formula to determine the slope for a quadratic regression
equation is: b1 + 2b2X, where X is a particular point on the predictor variable (U in the present
example). Here are some values for the slope at different values of U in the present example: bu=1
= 23.577 + 2 x -1.321 x 1 = 20.935, bu = 4 = 23.577 + 2 x -1.321 x 4 = 13.009, bu = 8 = 23.577 + 2 x
-1.321 x 8 = 2.441. Note how the slope decreases systematically from U = 1 to 4 to 8.
The interaction term, U2, in this study was negative, indicating that the slope became
shallower as the predictor increased. If the effect of U on RT increased as U got larger (i.e., the
slope got steeper), then the coefficient for U2 would be positive. If the effect of U on RT was
constant across levels of U, then the coefficient for U2 would be approximately zero, and not
significant. This would indicate a linear relationship or a more complex nonlinear function that
was not fit well by x2. Polynomial regression is one powerful method for the analysis of
nonlinear relations.
Although polynomial regression provides a way to accommodate nonlinear relationships,
MR and Non-Linear Relationships 13.8
the approach does have some limitations. A close look at Figure 13.3, for example, shows that
predicted values are actually starting to decrease at the highest levels of uncertainty. That is, the
slope has become negative, rather than positive. Extrapolating beyond the top value in Figure
13.3 would make this effect even stronger. But it is highly unlikely from a theoretical
perspective that actual reaction times would start to decrease. Rather RTs are likely to continue
increasing at a decreasing rate. But polynomial regression does not allow for such levelling off,
rather than actual changes in the sign of the slope. Alternative approaches are less vulnerable to
this problem, including more sophisticated nonlinear regression approaches not discussed in this
book and transforming predictors.
TRANSFORMING VARIABLES
A second approach to nonlinear relationships is to transform the criterion variable or the
predictors to make the relation more linear. In Figure 13.1, for example, the relation would be
more linear if the U scale were compressed so that the higher values of U were pushed closer to
the lower values, with little or less compression at the lower levels of U. There are various
operations that compress variables, including a square root transformation. To illustrate, original
values of 1, 4, and 9 would become 1, 2, and 3. Note that the original differences of 3 between 1
and 4, and of 5 between 9 and 4 have been equalized; that is, both differences are now 1 (2 vs 1
and 3 vs 2).
Other transformations produce even more compression. Logarithms to the base 10, for
example, would make the difference between 1 and 10 exactly the same as the difference
between 10 and 100; that is 1, 10, and 100 would become 0, 1, and 2. Reciprocals (one divided
by the values of the original predictor) also produce compression; for example, .3333, .5, and 1
would become 1/.333 = 3, 1/.5 = 2, and 1/1 = 1 (note the reversal in ordering using the reciprocal
transformation).
These are common transformations to compress predictors. Transformations that
compress predictors involve powers less than 1 (square root is power of .5, logarithm � power of
0, and reciprocals are powers of -1). Transformations using powers greater than 1 spread
predictors out; for example, squaring 1, 2, and 3 produces 1, 4, and 9, so that now the distance
MR and Non-Linear Relationships 13.9
Figure 13.4. Plot of RT Against LOGU.
between the 2nd and 3rd levels of the predictor is greater than the distance between the 1st and 2nd
levels.
When predictors are transformed in the preceding ways, they are used by themselves or
with other predictors in the regression analysis. This is different than the polynomial regression
approach, which includes both the original predictor (e.g., U) and the newly created predictor
(e.g., U2). In the approach being discussed here, transformed predictors replace the original
predictor, rather than complementing it in the regression analysis.
Figure 13.4 shows a plot of the observed data using log uncertainty, instead of the
original uncertainty levels; that is,
log(1) = 0, log(2) = .301, log(3) = .477,
..., log(8) = .903. The relation between
mean RT and log U is clearly more
linear than the relation between RT and
the original Uncertainty scores.
Predicted RTs from the best-fit
equation using log U are also plotted in
Figure 13.4. The observed and
predicted values are much closer, and
there are now few systematic
deviations about the regression line. R2
has also increased from .88 (see Figure
13.1 or Model 1 in Box 13.2) to .925
(see later analyses for origin of this
new value).
MR and Non-Linear Relationships 13.10
COMP lgu = LG10(u)REGR /VAR = rt lgu u /DESC /DEP = rt /ENTER lgu
Correlation: RT UU .938LGU .962 .959
Multiple R .96187R Square .92519Adjusted R Square .92270Standard Error 8.06043
DF Sum of Squares Mean SquareRegression 1 24106.75793 24106.75793Residual 30 1949.11707 64.97057
F = 371.04120 Signif F = .0000
Variable B SE B Beta T SigTLGU 96.052116 4.9865 .9619 19.262 .000(Const) 298.766251 3.2049 93.223 .000
Box 13.3. Regression of RT on Log U.
Figure 13.5. SPSS’s Curve Estimation Dialogue
Box.
Using COMPUTE to Transform Predictors
Box 13.3 shows the regression of RT on log Uncertainty scores that were created by the
COMPUTE statement on the
first line. Various measures
from the regression analysis
indicate a stronger relation
between RT and log U than
between RT and U. For
example, R2 = .9252, as seen
previously in Figure 13.4,
instead of .8803, indicating
that an additional 4.5% of the
variability is predicted by log
U. This increase is due to the
compression of the high end
of U. Note in Figure 13.4 that the points, which represent levels 1 to 8 on U, become closer
along the horizontal axis to the immediately preceding point as log U increases. For example,
log(8) - log(7) = .903 - .845 = .058, whereas log(2) - log(1) = .301 - 0 = .301. The distance
between 7 and 8 on the log scale is about 1/5th the distance between 1 and 2 on the log scale. On
the original scale, the distance between 7
and 8 equalled the distance between 1 and
2.
Using SPSS’s Curve Estimation
Procedure
In addition to its standard Linear
Regression option, SPSS provides a Curve
Estimation procedure that can
automatically conduct analyses for certain
kinds of transformations. The Curve
MR and Non-Linear Relationships 13.11
Estimation Dialogue box shown in Figure 13.5 can be accessed by the following menu
commands: Analyze | Regression | Curve Estimation. In Box 13.5, RT has been identified as the
Dependent variable, U as the Independent or predictor variable, and the Logarithmic
transformation has been selected. Other selected options are Display ANOVA table, Include
constant in equation, and Plot models. Clicking Ok runs the analysis.
The output for
the Curve Estimation
analysis is shown in
Box 13.4, along with
the syntax commands
equivalent to the
options in Figure 13.5.
Note the many
similarities to our
earlier analysis using computed logarithm scores, including: R2 = .92519, F = 371.04120, t =
19.262, intercept = 298.766251.
There is one notable difference between the results in Box 13.3 and 13.4, the value of the
slope for U. Specifically, B = 41.7149 in Box 13.4 versus B = 96.0521 in Box 13.3. This
difference in slopes arises from the different bases used to calculate the logarithms in the two
analyses. We computed logu = lg10(u), which uses 10 as the base for the logarithm. To
illustrate, lg10(2) = .30103 because 10.301 = 2.00. The Curve Estimation procedure, however,
computes what are called natural logarithms, using e = 2.71828 as the base. The logarithm of 2 to
the base e is .69315; that is, 2.71828.693 = 2.00. Because the natural logarithms are .69315/.30103
= 2.3026 times larger, the slope for the base 10 logarithms is 96.0521/41.7149 = 2.3026 times
larger. To make the two analyses equivalent, we would just need to change the compute in Box
13.3 to: COMPUTE logu = ln(u); ln is SPSS’s way of specifying natural logarithms. Under
some circumstances, it could be meaningful to specify logarithms using other bases (e.g., base 2 logarithms).
CURVEFIT /VARIABLES=RT WITH U /CONSTANT /MODEL=LOGARITHMIC /PRINT ANOVA /PLOT FIT.
Multiple R .96187R Square .92519Adjusted R Square .92270Standard Error 8.06043
DF Sum of Squares Mean SquareRegression 1 24106.758 24106.758Residuals 30 1949.117 64.971F = 371.04120 Signif F = .0000
-------------------- Variables in the Equation --------------------Variable B SE B Beta T Sig TU 41.714904 2.165610 .961870 19.262 .0000(Constant) 298.766251 3.204860 93.223 .0000
Box 13.4. Curve Estimation Analysis.
MR and Non-Linear Relationships 13.12
280
300
320
340
360
380
400
1 2 3 4 5 6 7 8
U
Observed
Logarithmic
RT
Figure 13.6. Plot of Logarithmic Function.
Figure 13.6 shows the plot of the logarithmic function to the data. The fit using the
logarithm of u generates
predicted values (the points on
the line) that are much closer to
the observed values than does a
straight linear equation. Like the
polynomial regression, the use of
logarithms for the predictor
scores allows the slope to be
steeper when U is small and
become gradually shallower as U
increases. Unlike the polynomial
fit, however, the predicted value
will not start to decrease if U is
extrapolated to higher values.
Finally, note in Figure
13.5 that one of the options on the Curve Estimation procedure is to do Linear and Quadratic fits
to the data. These options allow users to conduct polynomial regressions equivalent to that
discussed earlier.
Discussion of Transformations
People sometimes object to transformations of data because numbers are thought of as
having an absolute meaning that should not be violated. For example, how can a log
transformation change 1, 10, and 100 into 1, 2, and 3 without violating some aspect of the
original data? Such questions are important and researchers should consider them when
transforming scores. But carried to extreme, these concerns fail to appreciate that numbers can
be somewhat arbitrary with respect to the particular psychological or physical dimension being
measured. People expressing these concerns also fail to appreciate that some commonly used
scales are in fact transformations from the original scale; the decibel scale of loudness, for
example, is a logarithmic scale rather than a scale of equal intervals of loudness.
MR and Non-Linear Relationships 13.13
And often numerical scores depend upon the composition of the measure. Consider a test
of arithmetic ability with 30 questions. If the test included 10 easy questions, 10 medium
questions, and 10 difficult questions each worth one mark, then students would fall into four
clusters: those with scores around 0, those with scores around 10, those with scores around 20,
and those with scores around 30. People would be spaced out fairly evenly and the differences
between the groups would be about the same (and meaningfully so). But imagine a second test
with 14 easy questions, 14 medium questions, and only 2 difficult questions. Now people would
fall at 0, 14, 28, or 30 instead of 0, 10, 20, and 30. But the difference between 28 and 30 on the
second test represents the same difference in ability as the difference between 20 and 30 on the
first test. Such complications mean that numbers should not be treated absolutely and that they
can often be transformed without violating the data. In this example, a transformation that
expanded the difference between 28 and 30 relative to differences at the lower end of the scale
might be appropriate.
The malleable nature of numbers is shown clearly in cases where equally acceptable
alternative measures lead to somewhat different conclusions. Consider 3 subjects who are given
as long as necessary to complete 20 arithmetic problems. Subjects 1, 2, and 3 take 10, 20, and 30
minutes, respectively. Using time as the dependent measure, it took subjects an average of 20
minutes to complete the problems, subject 2 is exactly at average, and subjects 1 and 2 are above
and below subject 2 and the average by the same absolute amount (10 minutes).
Rather than using time as the dependent variable, the researchers could report the data in
terms of the rate or speed with which the problems were solved (i.e., #problems / time). This
new measure produces scores for subjects 1, 2, and 3 of 2.000 (20/10), 1.000 (20/20), and .667
(20/30), respectively. That is, subjects 1, 2, and 3 solved 2, 1, and .667 problems per minute.
The mean of these scores is 1.222, approximately 1 and 1/5th items per minute. But notice that
subject 2 is no longer at the average; subject 2 only solved 1 item per minute, whereas the mean
was 1.222 items per minute. Moreover, subjects 1 and 3 are no longer equal distance from
subject 2 (2 - 1 = 1 > 1 - .667 = .333) or from the mean (2 - 1.222 = .778 > 1.222 - .667 = .555),
which was the case for the time scores.
MR and Non-Linear Relationships 13.14
0
1
2
10 20 30
Time to Solve 20 Problems
Figure 13.7. Time and Rate Measures of
Performance.
NUM STOP RES.NUM NUM2 NUMROOT 1 5 -15.36962 1.00 1.00 1 7 -13.36962 1.00 1.00 2 18 -4.51008 4.00 1.41 2 23 .48992 4.00 1.41 4 46 19.20901 16.00 2.00 4 40 13.20901 16.00 2.00 8 38 2.64718 64.00 2.83 8 45 9.64718 64.00 2.83 16 50 -2.47648 256.00 4.00 16 43 -9.47648 256.00 4.00
Box 13.5. Data from Imitation Study.
These differences between the two
measures occur because the transformation to
rate is a nonlinear transformation, as shown in
Figure 13.7. Note that the rate measure on
the vertical axis decreases more as time
increases from 10 to 20 than as time increases
from 20 to 30. Similar nonlinear
transformations are legitimate in many other
circumstances, although the acceptability of
alternative scales may not be as obvious as for
measures of time and rate.
Neither the time nor the rate scale is more correct than the other, since both are sensible
measures for these data. It is equally reasonable to ask how long it took to solve the 20 problems
(i.e., the time measure) and how many problems were solved in each minute (i.e., the rate
measure). The two measures just don't provide perfectly equivalent results because of their
nonlinear relationship to one another.
IMITATION STUDY
To examine the relation between
number of people performing an action
and the percent of passers-by imitating
the action, social psychologists had
varying numbers of confederates (NUM
= 1, 2, 4, 8, or 16) stand on a street
looking up. The experiment was
performed twice on different days and at
different locations. The percent of passers-by who stopped (STOP) was determined for each
level of NUM. The 10 observations are shown in Box 13.5, along with several variables created
during the analyses.
MR and Non-Linear Relationships 13.15
Figure 13.8. Scattergram of Stopping as a
Function of Number of Observers.
Figure 13.9. Plot of Residuals from Linear
Regression with Predictor.
The researchers hypothesized that
the effect of increased numbers of
bystanders would be larger at lower levels
than at higher levels. That is, increasing the
number of observers from 2 to 3 was
expected to have a larger effect than the
increase from 12 to 13 observers.
Consistent with prediction, the plot in
Figure 13.8 shows that the relation between
STOP and NUM is definitely nonlinear.
The effect of number of observers appears
to level off at 4 observers, with few
additional stoppers as the observers
increased from 4 to 16. Figure 13.8 shows
that the linear regression (dashed line) overpredicts Stopping at low and high values of the
predictor, and underpredicts stopping in the middle range of the predictor.
The nonlinear nature of the relationship is
shown even more clearly in Figure 13.9, which
plots residuals from a simple regression of STOP
on NUM (see Box 13.6 for the analysis). The
mean for residuals is zero, since the sum of
residuals above and below the best-fit straight line
cancel out. The mean of 0 is shown in Figure
13.9 by the dashed line. For a linear relation, the
residuals should scatter randomly about this
average value. Instead, the residuals in this study
vary systematically, curving from below to above
the mean and then back below as NUM increases.
This curvilinear pattern indicates a nonlinear
MR and Non-Linear Relationships 13.16
REGR /DEP=stop /ENTER num /SAVE=RESID(res.num)
Multiple R .73288R Square .53712
DF Sum of Squares Mean SquareRegression 1 1363.47110 1363.47110Residual 8 1175.02890 146.87861F = 9.28298 Signif F = .0159
Variable B SE B Beta T Sig TNUM 2.14046 .70253 .73288 3.047 .0159(Constant) 18.22917 5.80170 3.142 .0138
COMP num2 = num*numREGR /DEP = stop /ENTER num num2
Multiple R .89777R Square .80599 > R²S.N = .53712
DF Sum of Squares Mean SquareRegression 2 2046.01392 1023.00696Residual 7 492.48608 70.35515F = 14.54061 Signif F = .0032
Variable B SE B Beta T Sig TNUM2 -.37935 .12179 -2.29977 -3.115 .0170NUM 8.68422 2.15645 2.97344 4.027 .0050(Constant) 3.52941 6.19649 .570 .5868
Box 13.6. Simple and Polynomial Regression for Imitation Study.
relation between NUM and STOP, which is consistent with the impression from the original plot
in Figure 13.8. The solid line in Figure 13.9 demonstrates that there is a considerable curvilinear
relationship with Number of Observers after the linear effect is removed. This line was added in
SPSS’s chart editor by requesting a polynomial fit to the residuals.
Box 13.6 shows the simple and polynomial regression of STOP on NUM and NUM2.
The prediction equation with both NUM and NUM2 is clearly an improvement over the simple
regression with NUM alone. R2 has increased dramatically, as reflected in the significant t for
NUM2. Other statistics could have been requested that would also have confirmed the significant
nonlinear effect (e.g., FChange, part r).
The
best-fit
nonlinear
equation is: �
= 3.53 +
8.68N -
.38N2, which
can be
rewritten as �
= 3.53 +
(8.68 -
.38N)N. This
second
version
shows how
the
coefficient
for the final N varies across different values of N. For NUM = 1, � = 3.53 + (8.68 -.38×1)N =
3.53+8.3N = 11.83, whereas, for NUM = 16, � = 3.53 + (8.68 -.38×16)N = 3.53+2.6N = 45.13.
Number of bystanders is multiplied by 8.3 for NUM=1 and by only 2.6 for NUM=16. As noted
MR and Non-Linear Relationships 13.17
Figure 13.10. Quadratic Fit to Stopping Data.
earlier, the actual slope at any given point on x could be obtained by calculating b1 + 2 x b2 x X.
The nonlinear nature of the
polynomial equation permits a better fit
to the data than the linear equation, as
shown in Figure 13.10. The linear
equation in Figure 13.8 predicts too
many passers-by will stop at the low and
high ends of NUM, and that too few will
stop at moderate levels of NUM. It is
these systematic deviations from the
predicted values that were plotted in
Figure 13.9. The polynomial predictions
bend with the data and come much closer
to the observed values, as expected given
the larger R2.
Although the nonlinear equation is an improvement for the fitted values, Figure 13.8
suggests some possible difficulties for the model with N and N2, as noted earlier. Specifically,
the predicted value from the nonlinear equation actually declines somewhat from NUM = 8 to
NUM = 16, and this decline would be even more marked if higher levels of NUM were
examined. It seems unlikely, however, that the percent of people stopping would actually
decrease as crowd size increased beyond 16. The nonlinear equation using just NUM and NUM2
would therefore provide an increasingly poor fit for higher numbers of onlookers.
Instead of polynomial regression, the researchers could transform the predictor variable
NUM, perhaps using a square root transformation (power of .5 or 1/2) to compress the upper
values of NUM (i.e., 1, 2, 4, 8, and 16 become 1, 1.414, 2, 2.828, and 4). Or the logarithm to the
base 2 might also be a good choice: 20 = 1, 21 = 2, 22 = 4, 23 = 8, 24 = 16.
MR and Non-Linear Relationships 13.18
COMP numroot = sqrt(num) /* Compresses number scaleREGR /DEP = stop /ENTER numroot
Multiple R .82417R Square .67925 > R²S.N = .53712
Adjusted R Square .63916Standard Error 10.08849
DF Sum of Squares Mean SquareRegression 1 1724.27920 1724.27920Residual 8 814.22080 101.77760F = 16.94164 Signif F = .0034
Variable B SE B Beta T Sig TNUMROOT 12.27631 2.98257 .82417 4.116 .0034(Constant) 3.89638 7.42653 .525 .6140
Box 13.7. Square Root Transformation of NUM.
Box 13.7 shows the regression analysis for the square root transformation. Although R2
for the �NUM is an improvement over the R2 for NUM alone, the transformed variable does not
achieve as good a fit to the data as the polynomial regression. The researchers might try other
meaningful transformations, such as logarithmic, to see if any provides an adequate depiction of
the data.
Box 13.8
shows that in fact the
logarithmic
transformation with
base = 2 does a much
better job than the
square root
transformation. The
first line computes the transformed scores. To compute logarithms to any desired base, simply
divide lg10(x) by lg10(base). Now our R2 becomes .820, even better than the value we obtained
using polynomial regression, although the transformation only uses a single (albeit transformed) predictor.
COMP log2num = lg10(num)/lg10(2).REGRE /DEP = stop /ENTER log2num.
Model R R Square Adjusted R Std. Error of Square the Estimate 1 .905(a) .820 .797 7.563894500
Model Sum of Squares df Mean Square F Sig. 1 Regression 2080.800 1 2080.800 36.370 .000(a) Residual 457.700 8 57.212 Total 2538.500 9
Model Unstandardized Standardized t Sig. Coefficients Coefficients B Std. Error Beta 1 (Constant) 11.100 4.143 2.679 .028 log2num 10.200 1.691 .905 6.031 .000
Box 13.8. Logarithm (Base 2) Transformation.
MR and Non-Linear Relationships 13.19
0
10
20
30
40
50
0 5 10 15
NUM
Observed
Linear
Logarithmic
STOP
Figure 13.11. Linear and Logarithmic
Models.
Box 13.9
shows the results of
the linear and
logarithmic fits to the
stopping study using
the Curve Estimation
(CURVEFIT) in
SPSS. The
logarithmic
regression clearly
provides the better fit
as indicated by R2, F,
or the observed p
value for the fit. This
analysis could also
have been requested
via menus: Analyze | Regression | Curve Estimation.
A plot of both equations appears in Figure
13.11 and very clearly shows the marked difference
between the fit of the linear and logarithmic models.
The logarithmic model allows for a change in the
slope as number of viewers increases, and this change
in slope more closely corresponds to the pattern
demonstrated by participants in this study. That is, the
likelihood of a passerby stopping increases quite
markedly as number of viewers initially increases, but
eventually number of viewers reaches some level
above which there is a much diminished influence on
passersby. Note as well that predictions from the logarithmic fit flatten out at upper levels of
CURVEFIT /VARIABLES=STOP WITH NUM /CONSTANT /MODEL=LINEAR LOGARITHMIC /PRINT ANOVA /PLOT FIT.
Dependent variable.. STOP Method.. LINEAR
Multiple R .73288R Square .53712Adjusted R Square .47926Standard Error 12.11935
DF Sum of Squares Mean SquareRegression 1 1363.4711 1363.4711Residuals 8 1175.0289 146.8786F = 9.28298 Signif F = .0159
-------------------- Variables in the Equation --------------------Variable B SE B Beta T Sig T
NUM 2.140457 .702527 .732883 3.047 .0159(Constant) 18.229167 5.801696 3.142 .0138
Dependent variable.. STOP Method.. LOGARITH
Listwise Deletion of Missing DataMultiple R .90537R Square .81970Adjusted R Square .79716Standard Error 7.56389
DF Sum of Squares Mean SquareRegression 1 2080.8000 2080.8000Residuals 8 457.7000 57.2125F = 36.36967 Signif F = .0003
-------------------- Variables in the Equation --------------------Variable B SE B Beta T Sig TNUM 14.715489 2.440085 .905371 6.031 .0003(Constant) 11.100000 4.142916 2.679 .0280
Box 13.9. Linear and Quadratic Fit for Stopping Data.
MR and Non-Linear Relationships 13.20
NUM, but do not decrease, as was the case for the quadratic regression.
CONCLUSIONS
Chapter 13 has demonstrated how MR methods can be used to analyze nonlinear relations
between a single predictor and a criterion. The presence of nonlinear relations can be determined
by plotting original or residual scores against the predictor variable. These plots will reveal any
systematic deviations from a linear relation, although regression analyses with polynomial or
transformed predictors may be necessary to determine whether the deviations are significant.
The procedures discussed here all made use of basic linear regression, but with accommodations
(i.e., additional polynomial terms, transformations) that allowed for nonlinear relationships. In
addition to the techniques discussed here, there are more advanced ways to analyze nonlinear
relationships that actually allow researchers to fit truly nonlinear equations to data.
Categorical Predictors and Regression 14.1
CHAPTER 14
CATEGORICAL PREDICTORS AND REGRESSION
Multiple Regression and Categorical Predictors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
Categorical Predictor with k = 2 (p = 1) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
Categorial Predictors with k > 2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
ANOVA Example for k = 4 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
Indicator Variables as Patterns Correlated with Y . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
Two Groups: k = 2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
Three Groups: k = 3 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
Predictors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
Age Effects on Memory Independent of Exercise . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
Age Effects on Memory Accounted for by Exercise . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
Note 14.1:
Alternative Indicator Variables for Treatment Study . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
Note 14.2:
Table of Indicator Variables (IVs) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
Categorical Predictors and Regression 14.2
Although multiple regression is generally presented as a way to analyze non-experimental
datasets involving numerical predictors, the techniques are equally well-suited to the analysis of
categorical predictors, either from experimental or non-experimental studies. Categorical
predictors have levels that differ qualitatively from one another; hence, numbers assigned to the
different levels of the predictor are entirely or partly arbitrary (e.g., Religion: 1 = Roman Catholic,
2 = Protestant, 3 = Other; Gender: 1 = Male, 2 = Female; Treatment Group: 1 = Control, 2 =
Treatment A, 3 = Treatment B). This chapter demonstrates how multiple regression
accommodates categorical predictors, a topic touched on briefly in an earlier chapter and
elaborated further in the second half of the course covering analysis of variance.
MULTIPLE REGRESSION AND CATEGORICAL PREDICTORS
The standard statistical approach to categorical predictors is to determine whether the
means on the criterion (or dependent) variable differ significantly from one another using either a
t-test or analysis of variance (ANOVA). ANOVA may be followed by additional tests to make
comparisons between specific groups. ANOVA was discussed briefly in an earlier chapter, and
full descriptions are available elsewhere, including later sections of this text. This chapter
considers only the case of a single categorical predictor, beginning with categorical variables with
only two levels (i.e., two conditions) and then more than two levels of a single categorical
variable.
Categorical Predictor with k = 2 (p = 1)
ANOVA terminology uses k to indicate the number
of levels of a predictor variable (also known as a factor in
ANOVA). When there are two groups being compared, k =
2. Such datasets can also be analyzed using t-tests that are
equivalent to the corresponding ANOVA. Box 14.1 shows
data from a comparison of verbal ability scores for males
and females. There were 8 subjects in each group. Previous
research generally shows a difference favouring females on
verbal ability tests.
Males Females31 3137 4530 3631 3124 2827 4133 3632 34
Box 14.1. Verbal Ability Scores
for Males and Females.
Categorical Predictors and Regression 14.3
Box 14.2 shows the independent groups t-test and one-way ANOVA results for these data
(the data for these analyses were input as pairs of scores, with the gend score being 1 for males
and 2 for females, and the verb score being the verbal ability scores). Using non-directional (i.e.,
two-tailed) tests, the difference is only marginally significant (note that p = .076 for the
independent groups t and for the ANOVA F statistics). If a one-tailed test were deemed
appropriate, then p = .076/2 = .038, and the difference would be judged to be significant. The
equivalence of t and F can also be demonstrated by squaring t to obtain F, or taking the square
root of F to obtain t; that is, t2 = F. The critical values of t and F (assuming a non-directional test
or appropriate selection of F) would also be equivalent using the same transformation.
The df for the numerator of the F test (i.e., df between groups) is k - 1 = 2 - 1 = 1, which is
why the F and t are equivalent. No such equivalence is possible when there are more than two
groups (i.e., when dfNumerator > 1 for F). The fact that df = 1 for the F test also provides some
indication of the kind of regression analysis that would be necessary to duplicate these results;
specifically, a single predictor would suffice, since df for FRegression is p, the number of predictors.
The formula for calculating t and F for this design were presented in earlier chapters. For
the t-test, t = (y�1 - y�2) / SQRT(sp2(1/n1 + 1/n2)), where sp
2 = (SS1 + SS2)/(n1 + n2 - 2). For the F
test, F = MSBetween / MSWithin, where MSWithin = sp2 as computed for the t-test, and MSBetween = njs y�
2,
where s y�2 = �(y�j - y�G)2 / (k - 1). With more than two groups, sp
2 = �SSj / (N - k). Although not
TTEST /GROUP = gend /VARI = verb.
GEND N Mean Std. Deviation Std. Error Mean VERB 1.0000 8 30.625000 3.8890873 1.3750000 2.0000 8 35.250000 5.5997449 1.9798088
Levene's Test t-test for Equality of Means Equal Variances
F Sig. t df Sig. Mean Std. Error (2-tailed) Difference Difference VERB Equal variances 1.079 .317 -1.919 14 .076 -4.625000 2.4104497
ONEWAY verb BY gend.
Sum of Squares df Mean Square F Sig. Between Groups 85.563 1 85.563 3.682 .076 Within Groups 325.375 14 23.241 Total 410.938 15
Box 14.2. T-test and ANOVA Results for Verbal Ability Study.
Categorical Predictors and Regression 14.4
emphasized earlier, SSWithin in the ANOVA summary table is SS1 + SS2 = �SSj. SSBetween is nj�(y�j
- y�G)2, with df = k - 1.
Box 14.3 shows the comparable regression analysis. It is actually very straightforward
because we simply regress the verbal ability scores (verb) on the gender scores (gend). Note first
that the values for the F and t tests for the regression analysis duplicate those from the
independent groups t-test and the ANOVA. Moreover, p = .076 for both F and t in Box 14.3.
The reason for the equivalence between the regression analyses and t-tests or ANOVA on
the differences between means is also apparent. The regression coefficient for Gender is 4.625,
REGRESS /VARI = verb gend /DEP = verb /ENTER /SAVE PRED(prdv.g) RESI(resv.g).
Model R R Square Adjusted R Square Std. Error of the Estimate 1 .456(a) .208 .152 4.8208994
Model Sum of Squares df Mean Square F Sig. 1 Regression 85.563 1 85.563 3.682 .076(a) Residual 325.375 14 23.241 Total 410.938 15
Coefficients(a) Unstandardized Coefficients Standardized Coefficients t Sig. Model B Std. Error Beta 1 (Constant) 26.000 3.811 6.822 .000 GEND 4.625 2.410 .456 1.919 .076
Residuals Statistics(a) Minimum Maximum Mean Std. Deviation N Predicted Value 30.625000 35.250000 32.937500 2.3883397 16 Residual -7.250000 9.750000 .000000 4.6574313 16
LIST. SUBJ GEND VERB PRDV.G RESV.G 1.0000 1.0000 31.0000 30.62500 .37500 2.0000 1.0000 37.0000 30.62500 6.37500 3.0000 1.0000 30.0000 30.62500 -.62500 4.0000 1.0000 31.0000 30.62500 .37500 5.0000 1.0000 24.0000 30.62500 -6.62500 6.0000 1.0000 27.0000 30.62500 -3.62500 7.0000 1.0000 33.0000 30.62500 2.37500 8.0000 1.0000 32.0000 30.62500 1.37500 9.0000 2.0000 31.0000 35.25000 -4.2500010.0000 2.0000 45.0000 35.25000 9.7500011.0000 2.0000 36.0000 35.25000 .7500012.0000 2.0000 31.0000 35.25000 -4.2500013.0000 2.0000 28.0000 35.25000 -7.2500014.0000 2.0000 41.0000 35.25000 5.7500015.0000 2.0000 36.0000 35.25000 .7500016.0000 2.0000 34.0000 35.25000 -1.25000
Box 14.3. Regression Analysis for Verbal Ability Study.
Categorical Predictors and Regression 14.5
the difference between the means for the two groups. This occurs because there is exactly a one-
unit difference between the numerical codes for the two groups; that is, Males = 1 on gend and
Females = 2. The PRDV.G scores listed below the regression analysis show this basis for the
correspondence; specifically, the predicted score for the 8 males is 30.625, M for Males, and the
predicted score for the 8 females is 35.250, M for Females, a difference of 4.625 units. Since the
predicted scores are the group means, SSRegression represents variation due to the differences
between the group means, which is equal to SSBetween in the ANOVA and has the same df. The
variability in the RESV.G scores represents deviations from the predicted scores (i.e., the group
means), and hence is equivalent to SSWithin in ANOVA or SS1 + SS2 in the t-test.
Although we used the original Gender scores as our predictor (i.e., 1 and 2), we could
actually have recoded these scores to any number of alternatives and obtained the equivalent
statistical results (see Box 14.4) as long as members of group 1 were coded as one value and
members of group 2 as a second value. If we had used 0 for Males and 1 for Females, for
RECODE gend (1 = 0) (2 = 1) INTO genddum.REGRE /DEP = verb /ENTER genddum.
Model R R Square Adjusted R Std. Error of Square the Estimate 1 .456(a) .208 .152 4.8208994
Model Sum of Squares df Mean Square F Sig. 1 Regression 85.563 1 85.563 3.682 .076(a) Residual 325.375 14 23.241 Total 410.938 15
Model Unstandardized Standardized t Sig. Coefficients Coefficients B Std. Error Beta 1 (Constant) 30.625 1.704 17.968 .000 genddum 4.625 2.410 .456 1.919 .076
RECODE gend (1 = -1) (2 = +1) INTO gendeff.REGRE /DEP = verb /ENTER gendeff.
Model R R Square Adjusted R Std. Error of Square the Estimate 1 .456(a) .208 .152 4.8208994
Model Sum of Squares df Mean Square F Sig. 1 Regression 85.563 1 85.563 3.682 .076(a) Residual 325.375 14 23.241 Total 410.938 15
Model Unstandardized Standardized t Sig. Coefficients Coefficients B Std. Error Beta 1 (Constant) 32.938 1.205 27.329 .000 gendeff 2.313 1.205 .456 1.919 .076
Box 14.4. Alternative Indicator Variables (Predictors) for Gender.
Categorical Predictors and Regression 14.6
example, then the intercept would have been MMales and the slope would have been MFemales - MMales
(the same slope as in Box 14.3). If we had used -1 for Males and +1 for Females, then the
intercept would have been the overall Mean for the two groups (called the grand Mean in
ANOVA terminology), and the slope would have been the deviation of the group Means from the
grand Mean. Because the denominator values would also change proportional to the change in the
numerators, the tests of significance are equivalent to those in Box 14.3 (see Box 14.4). These
tests would be the same using any of a number of alternative ways to code the gender predictor.
This example illustrates that regression can easily be used to analyze categorical variables
with only two groups (i.e., k = 2) because a single predictor variable encodes the difference
between the two groups. The situation becomes slightly more complex with more than two
groups, although still relatively simple given our understanding of multiple regression. The main
difference for k > 2 is that more predictors are necessary; specifically, k - 1 predictors are needed
to perform regression analyses equivalent to analysis of variance on k group means. Predictors
that encode differences between categorical groupings are often called indicator variables.
Categorial Predictors with k > 2
The added complexity with k > 2 can be derived from our earlier observation that the
dfRegression (i.e., p) must equal dfNumerator for ANOVA, which is k - 1. This means that researchers
need k - 1 predictors to duplicate ANOVA on k groups (i.e., 2 predictors for k = 3, 3 predictors for
k = 4, and so on). In short, there will be p = k - 1 indicator variables for analysis of the differences
between k groups, and hence the dfs for the numerators of regression and anova will be
equivalent.
Box 14.6 shows data for a comparison of verbal ability scores of 15 children in three grade
levels (grades 1, 2, and 3), along with some scores generated for the purpose of and resulting from
the multiple regression analysis. We would expect that verbal ability would differ across grade
levels, presumably with scores getting higher as children advance in grade.
Categorical Predictors and Regression 14.7
Box 14.5 shows the standard ANOVA results for these data. The means for Grades 1 (M
= 22.333), 2 (M = 28.500), and 3 (M = 43.333) differ significantly, F(2, 15) = 37.162, p = .000.
Verbal ability
scores clearly
increase with
Grade level,
although the
increase from
grade 1 to 2 is
considerably
smaller than the
increase from grade 2 to 3 (i.e., the relationship between verbal ability and grade is not linear, as
revealed in a subsequent analysis).
To duplicate these ANOVA results, we will need to use k - 1 = 2 predictors or indicator
variables. There are many possible ways to generate two predictors that will produce the overall F
and related statistics. There are several advantages to using indicator variables that have a mean
of 0 (i.e., the sum of the predictor values is 0), and there are also advantages to using predictors
that are uncorrelated with one another (i.e., they are orthogonal in ANOVA terminology). For
example, we could compare Grade 1 children with Grade 2 children using one predictor (p1 = -1
+1 0 for Grades 1, 2, and 3, respectively) and compare Grade 1 children with Grade 3 children
using the other predictor (p2 = -1 0 +1 for Grades 1, 2, and 3, respectively). These predictors
clearly sum to 0; however, they are not uncorrelated. Alternatively, we could use predictor one to
compare Grade 1 children with Grades 2 and 3 combined (i.e., p1 = -2 +1 +1 for Grades 1, 2, and
3, respectively) and use predictor two to compare Grade 2 children Grade 3 (i.e., p2 = 0 -1 +1, for
Grades 1, 2, and 3, respectively). Here the two predictors are uncorrelated. A third type of
categorical predictor for three groups reflects linear (-1, 0, +1) and quadratic (-1, +2, -1) patterns
in the data. These are called polynomial coefficients and are also independent of one another.
ONEWAY verb BY grade /STAT = DESCR.
N Mean Std. Std. 95% Confidence Interval for Mean Deviation Error Lower Bound Upper Bound 1.0000 6 22.333333 3.8297084 1.5634719 18.314301 26.352366 2.0000 6 28.500000 4.9295030 2.0124612 23.326804 33.673196 3.0000 6 43.333333 4.1793141 1.7061979 38.947412 47.719255
Total 18 31.388889 9.9418242 2.3433104 26.444936 36.332842
Sum of Squares df Mean Square F Sig. Between Groups 1398.111 2 699.056 37.162 .000 Within Groups 282.167 15 18.811 Total 1680.278 17
Box 14.5. Verbal Ability by Grades ANOVA Results.
Categorical Predictors and Regression 14.8
Box 14.6 shows a regression analysis using polynomial predictors. Polynomial predictors
examine the linear and non-linear components of the categorical predictor and hence are only
appropriate when there is a meaningful ordering of the levels (as is the case here for grade level).
The linear predictor for three groups is -1, 0, +1 for Grades 1, 2, and 3, respectively. The non-
RECODE grade (1 = -1) (2 = 0) (3 = +1) INTO lin.RECODE grade (1 = -1) (2 = +2) (3 = -1) INTO qua.REGRESS /VARI = verb lin qua /DEP = verb /ENTER /SAVE PRED(prdv.lq) RESI(resv.lq).
Model R R Square Adjusted R Square Std. Error of the Estimate 1 .912(a) .832 .810 4.3371778
Model Sum of Squares df Mean Square F Sig. 1 Regression 1398.111 2 699.056 37.162 .000(a) Residual 282.167 15 18.811 Total 1680.278 17
Coefficients(a) Unstandardized Coefficients Standardized Coefficients t Sig. Model B Std. Error Beta 1 (Constant) 31.389 1.022 30.705 .000 LIN 10.500 1.252 .887 8.386 .000 QUA -1.444 .723 -.211 -1.998 .064
Residuals Statistics(a) Minimum Maximum Mean Std. Deviation N Predicted Value 22.333334 43.333332 31.388889 9.0687281 18 Residual -7.500000 7.500000 .000000 4.0740691 18
LIST. SUBJ GRADE VERB LIN QUA PRDV.LQ RESV.LQ 1.0000 1.0000 24.0000 -1.0000 -1.0000 22.33333 1.66667 2.0000 1.0000 26.0000 -1.0000 -1.0000 22.33333 3.66667 3.0000 1.0000 25.0000 -1.0000 -1.0000 22.33333 2.66667 4.0000 1.0000 18.0000 -1.0000 -1.0000 22.33333 -4.33333 5.0000 1.0000 24.0000 -1.0000 -1.0000 22.33333 1.66667 6.0000 1.0000 17.0000 -1.0000 -1.0000 22.33333 -5.33333 7.0000 2.0000 30.0000 .0000 2.0000 28.50000 1.50000 8.0000 2.0000 27.0000 .0000 2.0000 28.50000 -1.50000 9.0000 2.0000 21.0000 .0000 2.0000 28.50000 -7.5000010.0000 2.0000 27.0000 .0000 2.0000 28.50000 -1.5000011.0000 2.0000 30.0000 .0000 2.0000 28.50000 1.5000012.0000 2.0000 36.0000 .0000 2.0000 28.50000 7.5000013.0000 3.0000 49.0000 1.0000 -1.0000 43.33333 5.6666714.0000 3.0000 42.0000 1.0000 -1.0000 43.33333 -1.3333315.0000 3.0000 39.0000 1.0000 -1.0000 43.33333 -4.3333316.0000 3.0000 48.0000 1.0000 -1.0000 43.33333 4.6666717.0000 3.0000 40.0000 1.0000 -1.0000 43.33333 -3.3333318.0000 3.0000 42.0000 1.0000 -1.0000 43.33333 -1.33333
Box 14.6. Regression Analysis Using Polynomial Predictors.
Categorical Predictors and Regression 14.9
linear (or quadratic) predictor for three groups is -1, +2, -1 for Grades 1, 2, and 3, respectively.
These contrasts sum to 0 and are uncorrelated. The RECODE commands in Box 14.5 generate the
linear (lin) and quadratic (qua) predictors.
The REGRESSION command regresses the dependent variable, verb, on the lin and qua
predictors, and saves the predicted and residual scores. Note first the many equivalencies between
the statistics for the overall regression and the previous ANOVA: FRegression = FANOVA, SSRegression =
SSBetween, SSResidual = SSWithin, and between the corresponding dfs and MSs. These correspondences
occur because the best-fit regression equation produces the group means as the predicted scores
(compare the PRDV.LQ column with the group means from the ANOVA), and the deviations
from the group mean as the residual scores.
Briefly, the tests of significance for the two predictors are also informative. Note that the
linear effect is highly significant (p = .000) and the quadratic (i.e., non-linear) effect is marginally
significant by a two-tailed test (p = .064). This suggests that there may be a significant non-linear
component to the effect of Grade on verbal ability (keep in mind the small n for this study,
something that works against finding a significant effect). In fact, using categorical predictors is a
third way to analyze non-linear predictors (in addition to the two methods discussed in the
previous chapter, namely, polynomial regression and transformations of predictors). Table T5 in
the Appendix shows coefficients that would be used to test linear and various non-linear
components for categorical variables with varying numbers of groups or levels.
ANOVA Example for k = 4
As a final
example of how
regression handles
categorical predictors,
consider a study in
which a Control Group
(Treat = 1) is being
compared with three
different treatment
LIST. SUBJ TREAT ASSERT P1 P2 P3 PRDA.123 RESA.123 1.0000 1.0000 29.0000 -3.0000 .0000 .0000 28.25000 .75000 2.0000 1.0000 31.0000 -3.0000 .0000 .0000 28.25000 2.75000 3.0000 1.0000 30.0000 -3.0000 .0000 .0000 28.25000 1.75000 4.0000 1.0000 23.0000 -3.0000 .0000 .0000 28.25000 -5.25000 5.0000 2.0000 34.0000 1.0000 -1.0000 -1.0000 32.00000 2.00000 6.0000 2.0000 27.0000 1.0000 -1.0000 -1.0000 32.00000 -5.00000 7.0000 2.0000 35.0000 1.0000 -1.0000 -1.0000 32.00000 3.00000 8.0000 2.0000 32.0000 1.0000 -1.0000 -1.0000 32.00000 .00000 9.0000 3.0000 26.0000 1.0000 -1.0000 1.0000 33.50000 -7.5000010.0000 3.0000 32.0000 1.0000 -1.0000 1.0000 33.50000 -1.5000011.0000 3.0000 35.0000 1.0000 -1.0000 1.0000 33.50000 1.5000012.0000 3.0000 41.0000 1.0000 -1.0000 1.0000 33.50000 7.5000013.0000 4.0000 44.0000 1.0000 2.0000 .0000 39.50000 4.5000014.0000 4.0000 37.0000 1.0000 2.0000 .0000 39.50000 -2.5000015.0000 4.0000 34.0000 1.0000 2.0000 .0000 39.50000 -5.5000016.0000 4.0000 43.0000 1.0000 2.0000 .0000 39.50000 3.50000
Box 14.7. Original and Derived Scores for Assertiveness Study.
Categorical Predictors and Regression 14.10
conditions (Treat = 2, 3, and 4) for Assertiveness (ASSERT in the datafile), with the expectation
that all of the conditions will produce higher Assertiveness scores than the control group and that
treatment 4 will be better than the other two treatment conditions. The data for the four subjects
in each of the 4 groups are shown in Box 14.7, along with some scores generated for and by the
regression analysis in Box 14.9.
The ANOVA results are shown in Box 14.8. The overall difference among the four means
is significant, F(3, 12) = 3.999, p = .035, and the pattern of differences among the means (see the
Mean column in Box 14.8) is consistent with the predictions.
Box 14.9 shows the corresponding regression analysis, using predictors that map onto the
ONEWAY assert BY treat /STAT = DESCR.
N Mean Std. Std. 95% Confidence Interval for Mean Deviation Error Lower Bound Upper Bound 1.0000 4 28.250000 3.5939764 1.7969882 22.531181 33.968819 2.0000 4 32.000000 3.5590261 1.7795130 26.336795 37.663205 3.0000 4 33.500000 6.2449980 3.1224990 23.562815 43.437185 4.0000 4 39.500000 4.7958315 2.3979158 31.868762 47.131238 Total 16 33.312500 5.9185443 1.4796361 30.158730 36.466270
Sum of Squares df Mean Square F Sig. Between Groups 262.688 3 87.563 3.999 .035 Within Groups 262.750 12 21.896 Total 525.438 15
Box 14.8. ANOVA Results for Assertiveness Study.
RECODE treat (1 = -3) (2 = +1) (3 = +1) (4 = +1) INTO p1.RECODE treat (1 = 0) (2 = -1) (3 = -1) (4 = +2) INTO p2.RECODE treat (1 = 0) (2 = -1) (3 = +1) (4 = 0) INTO p3.REGRESS /VARI = assert p1 p2 p3 /DEP = anx /ENTER /SAVE PRED(prda.123) RESI(resa.123).
Model R R Square Adjusted R Square Std. Error of the Estimate 1 .707(a) .500 .375 4.6792984
Model Sum of Squares df Mean Square F Sig. 1 Regression 262.687 3 87.562 3.999 .035(a) Residual 262.750 12 21.896 Total 525.437 15
Unstandardized Coefficients Standardized Coefficients t Sig. Model B Std. Error Beta 1 (Constant) 33.313 1.170 28.476 .000 P1 1.687 .675 .510 2.499 .028 P2 2.250 .955 .481 2.356 .036 P3 .750 1.654 .093 .453 .658
Box 14.9. Regression Analysis for Assertiveness Study.
Categorical Predictors and Regression 14.11
expected pattern of results. Specifically, predictor one (p1) compares the control group (treat =1)
to the three treatment groups; that is, p1 = -3 +1 +1 +1 for groups 1 to 4, respectively. Predictor
two (p2) compares the last treatment group (which was expected to do better) to the other two
treatment groups; that is, p2 = 0 -1 -1 +2 for groups 1 to 4, respectively. The third and final
predictor compares groups 2 and 3 to one another; that is, p3 = 0 -1 +1 0 for groups 1 to 4,
respectively.
The overall analysis in Box 14.9 agrees exactly with the ANOVA in Box 14.8, just as we
saw for the earlier analyses with k = 2 and 3. In addition, the significance tests for the individual
predictors confirm the predictions. The comparison between the Control group and the three
Treatment groups (i.e., p1) is significant, p = .028, as is the comparison between the last treatment
group and groups 2 and 3 (i.e., p2), p = .036. Alternative analyses for this study using different
indicator variables are shown in the Note 14.1 at the end of this chapter, and Note 14.2 shows
various alternative indicators for incorporating categorical predictors in multiple regression.
INDICATOR VARIABLES AS PATTERNS CORRELATED WITH Y
One can think about indicator variables, such as those generated for the preceding
examples, as patterns that researchers are looking for in the data. The unique effect of each
predictor represents the strength and significance of the relationship between the data and that pre-
defined pattern. Here we develop briefly this idea that indicator variables encode patterns as
numerical codes that can be correlated with our dependent variable, y.
Two Groups: k = 2
With only two groups, the idea of a “pattern” is limited to one group being greater or less
than another; that is, the two group means
differing significantly. Even so it is still
informative to see what happens when we
correlate y with our simple pattern scores (1
vs. 2, 0 vs. 1, -1 vs. +1). Box 14.10 shows the
correlation between verbal scores and the
gender indicator variable. Note first that the p
CORR verb gend /STAT = DESCR.
Mean Std. Deviation N VERB 32.93750000 5.234102916 16 GEND 1.50000000 .516397779 16
VERB GEND GEND Pearson .456 1 Correlation Sig. (2-tailed) .076 .
Box 14.10. Correlation with Gender ‘Scores”.
Categorical Predictors and Regression 14.12
value for the correlation coefficient, p = .076, is exactly the same as the p reported by the
independent groups t-test and the equivalent analysis of variance. Those tests were in essence
tests of the correlation between verbal scores and the dichotomous gender variable. Second, a t or
F statistic for this correlation coefficient would equal the t and F for the difference between
means: tr = (.456 - 0) / SQRT((1 - .4562)/(16 - 2)) = 1.917, and Fr = (.4562/1) / ((1 - .4562)/(16 - 2)
= 3.675 = tr2. Third, SSRegression = .4562 x SSTotal = .4562 x (16 - 1) 5.23412 = 85.449 = SSBetween. In
short, the tests of the difference between the two means is essentially testing the strength and
significance of the correlation between y, the verbal ability scores, and the gender indicator
variable.
Three Groups: k = 3
Our second example involved verbal
ability scores across three grade levels. We
tested the results for linear and non-linear (or
quadratic) patterns in the data. The
coefficients for the linear indicator variable
were: -1 0 +1; that is, the linear indicator
variable was ordered and its values were equal
distances apart (defining a linear relationship). The quadratic coefficients were: -1 +2 -1, which
defines a curvilinear or quadratic effect (inverted u or an opposite u-shaped pattern).
Box 14.11 shows the correlations between our two indicator variables and the verbal
scores. The significance tests are not shown in Box 14.11 because the computed tests are not
appropriate. We want to know the significance of lin and qua in the multiple regression equation
with both predictors in the equation, not their separate effects when alone. Nonetheless, the
correlations in Box 14.11 do represent the unique contribution of each predictor in the earlier
regression analysis. The simple rs represent the unique contribution because the two indicator
variables are completely independent of each other; note that r = 0 between lin and qua in Box 14.11.
CORR verb lin qua /STAT = DESCR.
Mean Std. Deviation N VERB 31.38888889 9.941824243 18 LIN .00000000 .840168050 18 QUA .00000000 1.455213750 18
VERB LIN QUA LIN Pearson .887 1 .000 QUA Pearson -.211 .000 1
Box 14.11. Correlation of Scores with Indicator
Variables for the Three Grade Levels.
Categorical Predictors and Regression 14.13
Figure 14.1. Fit of Data to Linear Coefficients.
Let us demonstrate the equivalence for lin. Lin accounts for .8872 × 1680.278 (i.e., SSTotal)
= 1321.991 units of variability in verbal
scores. Using MSError from the regression
analysis, this produces F = (1321.991/1)
/ 18.811 = 70.278, the square root of
which is 8.383, which is approximately
equal to tLin from the regression analysis.
Figure 14.1 shows the plot relating
verbal scores to the linear component of
the grade effect. Clearly the linear
pattern specified by the linear
coefficients accounts for considerable
variability in the verbal scores.
Figure 14.1 also reveals that there
is some systematic, albeit slight,
deviation from a strictly linear pattern.
Specifically, the increase in verbal scores from -1 to 0 (i.e., from grade 1 to grade 2) is not quite as
large as the increase from 0 to +1 (i.e., from grade 2 to grade 3). This deviation from the linear
pattern is captured by the quadratic coefficients and is almost significant, p = .064 in Box 14.5.
Categorical Predictors and Regression 14.14
In a somewhat different fashion, Box 14.12 shows for the four-group study that each
indicator variable is sensitive to the degree of relationship between itself and the dependent
variable, assertiveness scores in this example. The analysis focusses on p1, which has values of -
3 1 1 1, and looks for a pattern of results in which the Control group (group 1) differs from the
three treatment groups. Note in particular that the simple r of .510 is identical to the part r
because the three predictors are independent, and that .5102 = .260 = R2Change, which is tested for
significance by either FChange for Model 2 or tP1. And because the indicator variables are
uncorrelated, R2 = .5102 + .0932 + .4812 = .500.
COMBINING CATEGORICAL AND NUMERICAL PREDICTORS
In the preceding examples, regression analysis was used to replicate results that could have
been produced using ANOVA. This nicely demonstrates the equivalence of regression (the
General Linear Model) and ANOVA. The primary benefit of accommodating categorical
REGRE /STAT = DEFA ZPP CHANGE /DESCR /DEP = assert /ENTER p2 p3 /ENTER p1.
Mean Std. Deviation N ASSERT 33.31250000 5.918544303 16 P2 .00000000 1.264911064 16 P3 .00000000 .730296743 16 P1 .00000000 1.788854382 16
ASSERT P2 P3 P1 P2 .481 1.000 .000 .000 P3 .093 .000 1.000 .000 P1 .510 .000 .000 1.000
Model R R Adjusted Std. Error of Change Statistics Square R Square the Estimate R Square Change F Change df1 df2 Sig. F Change 1 .490(a) .240 .123 5.543100354 .240 2.050 2 13 .168 2 .707(b) .500 .375 4.679298380 .260 6.243 1 12 .028
Model Sum of Squares df Mean Square F Sig. 1 Regression 126.000 2 63.000 2.050 .168(a) Residual 399.437 13 30.726 Total 525.437 15
2 Regression 262.687 3 87.562 3.999 .035(b) Residual 262.750 12 21.896 Total 525.437 15
Model Unstandardized Standardized t Sig. Correlations Coefficients Coefficients B Std. Error Beta Zero-order Partial Part ... 2 (Constant) 33.313 1.170 28.476 .000 P2 2.250 .955 .481 2.356 .036 .481 .562 .481 P3 .750 1.654 .093 .453 .658 .093 .130 .093 P1 1.687 .675 .510 2.499 .028 .510 .585 .510
Box 14.12. Simple and Part Correlations for k = 4 Study.
Categorical Predictors and Regression 14.15
variables in regression analysis, however, is to allow researchers to include both categorical and
numerical predictors in their analysis. Here we illustrate using a study of the relationship between
age, exercise, and memory performance.
Age Effects on Memory Independent of Exercise
The first version of the study involved 24 participants from 3 age
levels (1 = Young, 2 = Middle Age, and 3 = Old). Box 14.13 shows
scores on age, exercise, and memory for the 24 participants. The
researchers were interested in whether the categorical (although ordered)
variable age was related to memory performance, and whether any
relationship between age and memory performance might be mediated by
the amount of exercise people engaged in. If exercise mediated the
relationship between age and memory, then using multiple regression to
control for differences in exercise should eliminate any differences in
memory performance across the three age groups. If exercise does not
mediate the relationship between age and memory, then controlling
statistically for exercise should have no effect on differences across ages
in memory performance.
Preliminary analyses were conducted to determine whether there
were indeed any differences across age in memory or exercise. These analyses were done using
both ANOVA and regression approaches, although the two analyses are equivalent as we saw
earlier in the chapter. Because age is an ordered variable, polynomial indicator variables (linear
and quadratic) were used for the regression analyses.
s age exe mem 1 1 22 33 2 1 16 41 3 1 24 47 4 1 19 44 5 1 22 42 6 1 14 37 7 1 24 48 8 1 25 43 9 2 24 4310 2 21 3511 2 20 4112 2 20 4513 2 17 3014 2 18 4915 2 17 3516 2 19 3617 3 18 2918 3 14 2919 3 23 3020 3 24 3521 3 20 3522 3 20 3223 3 21 4124 3 20 28
Box 14.13. Results for
Age and Memory
Study.
Categorical Predictors and Regression 14.16
Box 14.14 shows the results of the analyses to determine whether memory performance
differs significantly across ages. The ANOVA demonstrates clearly that the three group means
differ significantly, F = 6.897, p = .005. The regression analysis using linear and quadratic
indicator variables replicates this analysis exactly. The reason is that using these (or other)
indicator variables generates the group means as the predicted values, as shown for the first two
subjects in each group at the bottom of Box 14.14.
ONEWAY mem BY age /STAT = DESCR.
N Mean Std. Std. 95% Confidence Interval for Minimum Maximum Deviation Error Mean Lower Bound Upper Bound 1 8 41.88 4.970 1.757 37.72 46.03 33 48 2 8 39.25 6.296 2.226 33.99 44.51 30 49 3 8 32.38 4.406 1.558 28.69 36.06 28 41
Total 24 37.83 6.499 1.327 35.09 40.58 28 49
Sum of Squares df Mean Square F Sig. Between Groups 385.083 2 192.542 6.897 .005 Within Groups 586.250 21 27.917 Total 971.333 23
RECODE age (1 = -1) (2 = 0) (3 = 1) INTO agelin.RECODE age (1 3 = -1) (2 = 2) INTO agequa.REGRESS /DEP = mem /ENTER agelin agequa /SAVE PRED(prdm.lq) RESI(resm.lq).
Model R R Square Adjusted R Std. Error of Square the Estimate 1 .630(a) .396 .339 5.284
Model Sum of Squares df Mean Square F Sig. 1 Regression 385.083 2 192.542 6.897 .005(a) Residual 586.250 21 27.917 Total 971.333 23
Model Unstandardized Standardized t Sig. Coefficients Coefficients B Std. Error Beta 1 (Constant) 37.833 1.079 35.079 .000 agelin -4.750 1.321 -.610 -3.596 .002 agequa .708 .763 .157 .929 .364
Residuals Statistics(a) Minimum Maximum Mean Std. Deviation N Predicted Value 32.38 41.88 37.83 4.092 24 Residual -9.250 9.750 .000 5.049 24
LIST. s age exe mem agelin agequa prdm.lq resm.lq 1 1 22 33 -1.0000 -1.0000 41.87500 -8.87500 2 1 16 41 -1.0000 -1.0000 41.87500 -.87500... 9 2 24 43 .0000 2.0000 39.25000 3.7500010 2 21 35 .0000 2.0000 39.25000 -4.25000...17 3 18 29 1.0000 -1.0000 32.37500 -3.3750018 3 14 29 1.0000 -1.0000 32.37500 -3.37500...
Box 14.14. ANOVA for Memory and Equivalent Regression Analysis.
Categorical Predictors and Regression 14.17
Box 14.15 shows the corresponding analyses for the relationship between age and
exercise. Here we see that exercise does not in fact differ significantly across the three age levels,
F = .306, p = .740 for both the ANOVA and regression analyses. Again, predicted scores are
equal to the group means, which explains the correspondence between the two analyses.
Together these analyses suggest that any differences across age in memory performance
probably are not due to differences in exercise. The rational is that there are no significant
ONEWAY exe BY age /STAT = DESCR.
N Mean Std. Std. 95% Confidence Interval for Minimum Maximum Deviation Error Mean Lower Bound Upper Bound 1 8 20.75 4.027 1.424 17.38 24.12 14 25 2 8 19.50 2.330 .824 17.55 21.45 17 24 3 8 20.00 3.071 1.086 17.43 22.57 14 24
Total 24 20.08 3.120 .637 18.77 21.40 14 25
Sum of Squares df Mean Square F Sig. Between Groups 6.333 2 3.167 .306 .740 Within Groups 217.500 21 10.357 Total 223.833 23
RECODE age (1 = -1) (2 = 0) (3 = 1) INTO agelin.RECODE age (1 3 = -1) (2 = 2) INTO agequa.REGRESS /DEP = exe /ENTER agelin agequa /SAVE PRED(prde.lq) RESI(rese.lq).
Model R R Square Adjusted R Std. Error of Square the Estimate 1 .168(a) .028 -.064 3.218
Model Sum of Squares df Mean Square F Sig. 1 Regression 6.333 2 3.167 .306 .740(a) Residual 217.500 21 10.357
Total 223.833 23
Model Unstandardized Standardized t Sig. Coefficients Coefficients B Std. Error Beta 1 (Constant) 20.083 .657 30.572 .000 agelin -.375 .805 -.100 -.466 .646 agequa -.292 .465 -.135 -.628 .537
Residuals Statistics(a) Minimum Maximum Mean Std. Deviation N Predicted Value 19.50 20.75 20.08 .525 24 Residual -6.750 4.500 .000 3.075 24
LIST s, age, exe agelin agequa prde.lq rese.lq. s age exe agelin agequa prde.lq rese.lq 1 1 22 -1.0000 -1.0000 20.75000 1.25000 2 1 16 -1.0000 -1.0000 20.75000 -4.75000... 9 2 24 .0000 2.0000 19.50000 4.5000010 2 21 .0000 2.0000 19.50000 1.50000...17 3 18 1.0000 -1.0000 20.00000 -2.0000018 3 14 1.0000 -1.0000 20.00000 -6.00000
Box 14.15. ANOVA and Equivalent Regression Analysis for Relation between Age and
Exercise.
Categorical Predictors and Regression 14.18
differences across age in exercise that could potentially account for the memory differences. To
formally test the hypothesis, we conduct a multiple regression in which age and exercise are
included together as predictors of memory.
Box 14.16 shows the relevant analysis, with agelin and agequa entered together after
exercise, so that we can isolate the overall effect of age. FChange = 7.277, pChange = .004, indicating
that the differences in memory performance among the three groups remain significant even when
exercise is controlled. Note in the change statistics that the df for FChange = 2 because we entered
two predictors into the equation at once (agelin and agequa). This is one illustration of a situation
in which it makes sense to test the significance of the change when more than one predictor is
added to the equation.
The conclusion from this analysis is that there are indeed significant differences among the
ages in memory performance and these differences cannot be accounted for by differences in
exercise. Let us now change the data set “slightly,” and see how the analyses would differ.
REGRE /STAT = DEFA CHANGE /DEP = mem /ENTER exe /ENTER agelin agequa.
Model R R Adjusted Std. Error of Change Statistics Square R Square the Estimate R Square Change F Change df1 df2 Sig. F Change 1 .335(a) .112 .072 6.260 .112 2.786 1 22 .109 2 .697(b) .486 .409 4.995 .374 7.277 2 20 .004
Model Sum of Squares df Mean Square F Sig. 1 Regression 109.189 1 109.189 2.786 .109(a) Residual 862.144 22 39.188 Total 971.333 23
2 Regression 472.325 3 157.442 6.310 .003(b) Residual 499.008 20 24.950 Total 971.333 23
Model Unstandardized Standardized t Sig. Coefficients Coefficients B Std. Error Beta 1 (Constant) 23.806 8.500 2.801 .010 exe .698 .418 .335 1.669 .109
2 (Constant) 25.114 6.878 3.651 .002 exe .633 .339 .304 1.870 .076 agelin -4.512 1.255 -.579 -3.595 .002 agequa .893 .728 .199 1.227 .234
Box 14.16. Effect of Age Controlling for Exercise.
Categorical Predictors and Regression 14.19
Age Effects on Memory Accounted for by Exercise
Box 14.17 shows the modified data set. In essence, as we will
see shortly, the change that has been introduced is to insert differences in
exercise across age levels. This change raises the possibility that
differences in memory performance across age may now become non-
significant when exercise is controlled in a multiple regression analysis.
First, we repeat the ANOVA and regression analyses for the relationship
between age and exercise to demonstrate that there are indeed significant
differences in exercise across the ages. We do not present here the
corresponding analyses for memory, which would be identical to those
presented in Box 14.14 (note that the memory data is exactly the same in
the two data sets; only the data for exercise has been modified for this
second variant of the study.
s age exe mem 1 1 22 33 2 1 16 41 3 1 24 47 4 1 19 44 5 1 22 42 6 1 14 37 7 1 24 48 8 1 25 43 9 2 21 4310 2 18 3511 2 17 4112 2 17 4513 2 14 3014 2 15 4915 2 14 3516 2 16 3617 3 12 2918 3 8 2919 3 17 3020 3 18 3521 3 14 3522 3 14 3223 3 15 4124 3 14 28
Box 14.17. Modified
Memory Data Set.
ONEWAY exe BY age /STAT = DESCR.
N Mean Std. Std. 95% Confidence Interval for Minimum Maximum Deviation Error Mean Lower Bound Upper Bound 1 8 20.75 4.027 1.424 17.38 24.12 14 25 2 8 16.50 2.330 .824 14.55 18.45 14 21 3 8 14.00 3.071 1.086 11.43 16.57 8 18
Total 24 17.08 4.190 .855 15.31 18.85 8 25
Sum of Squares df Mean Square F Sig. Between Groups 186.333 2 93.167 8.995 .002 Within Groups 217.500 21 10.357 Total 403.833 23
REGRESS /DEP = exe /ENTER agelin agequa /SAVE PRED(prde2.lq) RESI(rese2.lq).
Model R R Square Adjusted R Std. Error of Square the Estimate 1 .679(a) .461 .410 3.218
Model Sum of Squares df Mean Square F Sig. 1 Regression 186.333 2 93.167 8.995 .002(a) Residual 217.500 21 10.357 Total 403.833 23
Model Unstandardized Standardized t Sig. Coefficients Coefficients B Std. Error Beta 1 (Constant) 17.083 .657 26.005 .000 agelin -3.375 .805 -.672 -4.195 .000 agequa -.292 .465 -.101 -.628 .537
LIST s, age, exe, mem, agelin, agequa, prde2.lq, rese2.lq.
s age exe mem agelin agequa prde2.lq rese2.lq 1 1 22 33 -1.0000 -1.0000 20.75000 1.25000 2 1 16 41 -1.0000 -1.0000 20.75000 -4.75000... 9 2 21 43 .0000 2.0000 16.50000 4.5000010 2 18 35 .0000 2.0000 16.50000 1.50000...17 3 12 29 1.0000 -1.0000 14.00000 -2.0000018 3 8 29 1.0000 -1.0000 14.00000 -6.00000
Box 14.18. Analyses of Relationship between Age and Exercise in Revised Data Set.
Categorical Predictors and Regression 14.20
Box 14.18 shows the results of the (equivalent) ANOVA and regression analyses of the
relationship between age and exercise. Amount of exercise now decreases significantly from
20.75 to 16.50 to 14.00 as age level increases from young to middle-age to old. This difference is
highly significant. The critical question now is whether the difference across age in memory
performance will remain significant when we control for differences in exercise.
Box 14.19 shows the relevant results. Exercise is entered first and now there is no
significant change in prediction of memory performance when the two age predictors are added to
the equation (nor is either individual predictor significant in the Coefficient section of the
analysis).
The conclusion from this analysis is markedly different from that with the original data set.
Now we would conclude that exercise might indeed be the mediating factor in deteriorating
memory performance across ages. When we control statistically for exercise (i.e., examine the
effect of age independent of exercise), we no longer observe a relationship between age and
memory performance. Our conclusion must, of course, be tentative because this is a non-
experimental study and does not allow strong conclusions about the presence or absence of causal
relationships.
CONCLUSIONS
This chapter has briefly demonstrated how it is possible to accommodate categorical
REGRE /STAT = DEFA CHANGE /DEP = mem /ENTER exe /ENTER agelin agequa.
Model R R Adjusted Std. Error of Change Statistics Square R Square the Estimate R Square Change F Change df1 df2 Sig. F Change 1 .614(a) .377 .348 5.246 .377 13.289 1 22 .001 2 .697(b) .486 .409 4.995 .110 2.135 2 20 .144
Model Sum of Squares df Mean Square F Sig. 1 Regression 365.775 1 365.775 13.289 .001(a) Residual 605.558 22 27.525 Total 971.333 23
2 Regression 472.325 3 157.442 6.310 .003(b) Residual 499.008 20 24.950 Total 971.333 23
Model Unstandardized Standardized t Sig. Coefficients Coefficients B Std. Error Beta 1 (Constant) 21.575 4.587 4.704 .000 exe .952 .261 .614 3.645 .001
2 (Constant) 27.014 5.875 4.598 .000 exe .633 .339 .408 1.870 .076 agelin -2.612 1.693 -.335 -1.543 .138 agequa .893 .728 .199 1.227 .234
Box 14.19. Nonsignificant Age effect in Revised Memory Study.
Categorical Predictors and Regression 14.21
predictors in multiple regression. Essentially, researchers define k - 1 predictors to represent,
ideally, meaningful comparisons among the k conditions. The capacity to include a mix of both
numerical and categorical predictors makes multiple regression an extremely general and
powerful data analytic technique, including the possibility of examining categorical predictors
controlling for relationships with other predictors that are numerical in nature.
Categorical Predictors and Regression 14.22
NOTE 14.1:
ALTERNATIVE INDICATOR VARIABLES FOR TREATMENT STUDY
RECODE treat (1 3 4 = 0) (2 = 1) INTO dum12.RECODE treat (1 2 4 = 0) (3 = 1) INTO dum13.RECODE treat (1 2 3 = 0) (4 = 1) INTO dum14.REGRE /DESCR /DEP = assert /ENTER dum12 dum13 dum14 /SAVE PRED(prddum) RESI(resdum).
Mean Std. Deviation N assert 33.312500 5.9185443 16 dum12 .250000 .4472136 16 dum13 .250000 .4472136 16 dum14 .250000 .4472136 16
assert dum12 dum13 dum14 dum12 -.132 1.000 -.333 -.333 dum13 .019 -.333 1.000 -.333 dum14 .623 -.333 -.333 1.000
Model R R Square Adjusted R Std. Error of Square the Estimate 1 .707(a) .500 .375 4.6792984
Model Sum of Squares df Mean Square F Sig. 1 Regression 262.687 3 87.562 3.999 .035(a) Residual 262.750 12 21.896 Total 525.437 15
Model Unstandardized Standardized t Sig. Coefficients Coefficients B Std. Error Beta 1 (Constant) 28.250 2.340 12.074 .000 dum12 3.750 3.309 .283 1.133 .279 dum13 5.250 3.309 .397 1.587 .139 dum14 11.250 3.309 .850 3.400 .005
Residuals Statistics(a) Minimum Maximum Mean Std. Deviation N Predicted Value 28.250000 39.500000 33.312500 4.1847939 16 Residual -7.5000000 7.5000000 .0000000 4.1852917 16
LIST assert treat dum12 dum13 dum14 prddum resdum.
assert treat dum12 dum13 dum14 prddum resdum 29.0000 1.0000 .0000 .0000 .0000 28.25000 .75000 31.0000 1.0000 .0000 .0000 .0000 28.25000 2.75000 30.0000 1.0000 .0000 .0000 .0000 28.25000 1.75000 23.0000 1.0000 .0000 .0000 .0000 28.25000 -5.25000 34.0000 2.0000 1.0000 .0000 .0000 32.00000 2.00000 27.0000 2.0000 1.0000 .0000 .0000 32.00000 -5.00000 35.0000 2.0000 1.0000 .0000 .0000 32.00000 3.00000 32.0000 2.0000 1.0000 .0000 .0000 32.00000 .00000 26.0000 3.0000 .0000 1.0000 .0000 33.50000 -7.50000 32.0000 3.0000 .0000 1.0000 .0000 33.50000 -1.50000 35.0000 3.0000 .0000 1.0000 .0000 33.50000 1.50000 41.0000 3.0000 .0000 1.0000 .0000 33.50000 7.50000 44.0000 4.0000 .0000 .0000 1.0000 39.50000 4.50000 37.0000 4.0000 .0000 .0000 1.0000 39.50000 -2.50000 34.0000 4.0000 .0000 .0000 1.0000 39.50000 -5.50000 43.0000 4.0000 .0000 .0000 1.0000 39.50000 3.50000
Categorical Predictors and Regression 14.23
RECODE treat (1 = -1) (2 = 1) (3 4 = 0) INTO eff12.RECODE treat (1 = -1) (3 = 1) (2 4 = 0) INTO eff13.RECODE treat (1 = -1) (4 = 1) (2 3 = 0) INTO eff14.REGRE /DESCR /DEP = assert /ENTER eff12 eff13 eff14 /SAVE PRED(prdeff) RESI(reseff).
Mean Std. Deviation N assert 33.312500 5.9185443 16 eff12 .000000 .7302967 16 eff13 .000000 .7302967 16 eff14 .000000 .7302967 16
assert eff12 eff13 eff14 eff12 .231 1.000 .500 .500 eff13 .324 .500 1.000 .500 eff14 .694 .500 .500 1.000
Model R R Square Adjusted R Std. Error of Square the Estimate 1 .707(a) .500 .375 4.6792984
Model Sum of Squares df Mean Square F Sig. 1 Regression 262.688 3 87.563 3.999 .035(a) Residual 262.750 12 21.896 Total 525.437 15
Model Unstandardized Standardized t Sig. Coefficients Coefficients B Std. Error Beta 1 (Constant) 33.313 1.170 28.476 .000 eff12 -1.313 2.026 -.162 -.648 .529 eff13 .187 2.026 .023 .093 .928 eff14 6.188 2.026 .763 3.054 .010
Residuals Statistics(a) Minimum Maximum Mean Std. Deviation N Predicted Value 28.250000 39.500000 33.312500 4.1847939 16 Residual -7.5000000 7.5000000 .0000000 4.1852917 16
LIST assert treat eff12 eff13 eff14 prdeff reseff.
assert treat eff12 eff13 eff14 prdeff reseff 29.0000 1.0000 -1.0000 -1.0000 -1.0000 28.25000 .75000 31.0000 1.0000 -1.0000 -1.0000 -1.0000 28.25000 2.75000 30.0000 1.0000 -1.0000 -1.0000 -1.0000 28.25000 1.75000 23.0000 1.0000 -1.0000 -1.0000 -1.0000 28.25000 -5.25000 34.0000 2.0000 1.0000 .0000 .0000 32.00000 2.00000 27.0000 2.0000 1.0000 .0000 .0000 32.00000 -5.00000 35.0000 2.0000 1.0000 .0000 .0000 32.00000 3.00000 32.0000 2.0000 1.0000 .0000 .0000 32.00000 .00000 26.0000 3.0000 .0000 1.0000 .0000 33.50000 -7.50000 32.0000 3.0000 .0000 1.0000 .0000 33.50000 -1.50000 35.0000 3.0000 .0000 1.0000 .0000 33.50000 1.50000 41.0000 3.0000 .0000 1.0000 .0000 33.50000 7.50000 44.0000 4.0000 .0000 .0000 1.0000 39.50000 4.50000 37.0000 4.0000 .0000 .0000 1.0000 39.50000 -2.50000 34.0000 4.0000 .0000 .0000 1.0000 39.50000 -5.50000 43.0000 4.0000 .0000 .0000 1.0000 39.50000 3.50000
Categorical Predictors and Regression 14.24
RECODE treat (1 = -1) (2 = 1) (3 4 = 0) INTO or1v2.RECODE treat (1 2 = -1) (3 = 2) (4 = 0) INTO or12v3.RECODE treat (1 2 3 = -1) (4 = 3) INTO or123v4.REGRE /DESCR /DEP = assert /ENTER or1v2 or12v3 or123v4 /SAVE PRED(prdorth) RESI(resorth).
Mean Std. Deviation N assert 33.312500 5.9185443 16 or1v2 .000000 .7302967 16 or12v3 .000000 1.2649111 16 or123v4 .000000 1.7888544 16
assert or1v2 or12v3 or123v4 or1v2 .231 1.000 .000 .000 or12v3 .240 .000 1.000 .000 or123v4 .623 .000 .000 1.000
Model Summary(b) Model R R Square Adjusted R Std. Error of Square the Estimate 1 .707(a) .500 .375 4.6792984
Model Sum of Squares df Mean Square F Sig. 1 Regression 262.687 3 87.562 3.999 .035(a) Residual 262.750 12 21.896 Total 525.437 15
Model Unstandardized Standardized t Sig. Coefficients Coefficients B Std. Error Beta 1 (Constant) 33.313 1.170 28.476 .000 or1v2 1.875 1.654 .231 1.133 .279 or12v3 1.125 .955 .240 1.178 .262 or123v4 2.062 .675 .623 3.054 .010
Residuals Statistics(a) Minimum Maximum Mean Std. Deviation N Predicted Value 28.250000 39.500000 33.312500 4.1847939 16 Residual -7.5000000 7.5000000 .0000000 4.1852917 16
LIST assert treat or1v2 or12v3 or123v4 prdorth resorth.
assert treat or1v2 or12v3 or123v4 prdorth resorth 29.0000 1.0000 -1.0000 -1.0000 -1.0000 28.25000 .75000 31.0000 1.0000 -1.0000 -1.0000 -1.0000 28.25000 2.75000 30.0000 1.0000 -1.0000 -1.0000 -1.0000 28.25000 1.75000 23.0000 1.0000 -1.0000 -1.0000 -1.0000 28.25000 -5.25000 34.0000 2.0000 1.0000 -1.0000 -1.0000 32.00000 2.00000 27.0000 2.0000 1.0000 -1.0000 -1.0000 32.00000 -5.00000 35.0000 2.0000 1.0000 -1.0000 -1.0000 32.00000 3.00000 32.0000 2.0000 1.0000 -1.0000 -1.0000 32.00000 .00000 26.0000 3.0000 .0000 2.0000 -1.0000 33.50000 -7.50000 32.0000 3.0000 .0000 2.0000 -1.0000 33.50000 -1.50000 35.0000 3.0000 .0000 2.0000 -1.0000 33.50000 1.50000 41.0000 3.0000 .0000 2.0000 -1.0000 33.50000 7.50000 44.0000 4.0000 .0000 .0000 3.0000 39.50000 4.50000 37.0000 4.0000 .0000 .0000 3.0000 39.50000 -2.50000 34.0000 4.0000 .0000 .0000 3.0000 39.50000 -5.50000 43.0000 4.0000 .0000 .0000 3.0000 39.50000 3.50000
Categorical Predictors and Regression 14.25
NOTE 14.2:
TABLE OF INDICATOR VARIABLES (IVS)
Number of Groups (k) 2 3 4
Dummy Coding D1 D1 D2 D1 D2 D3 1 0 0 0 0 0 0
Group 2 1 1 0 1 0 0 3 0 1 0 1 0 4 0 0 1
Effect Coding E1 E1 E2 E1 E2 E3 1 -1 -1 -1 -1 -1 -1
Group 2 1 1 0 1 0 0 3 0 1 0 1 0 4 0 0 1
Orthogonal Coding O1 O2 O1 O2 O3 1 -1 -1 -1 -1 -1
Group 2 1 -1 1 -1 -1 3 0 2 0 2 -1 4 0 0 3
Polynomial Coding L Q L Q C 1 -1 1 -3 1 -1
Group 2 0 -2 -1 -1 3 3 1 1 1 -1 -3 4 3 1 1
Dummy Coding
One group coded 0 on all IVs (Group 1 above) and each of k - 1 other groups coded 1 on different IVs
Not a contrast (i.e., do not sum to 0) and not orthogonal (i.e., rs not equal to 0)
Compares each of k - 1 groups to group coded 0 on all indicator variables
Effect Coding
One group coded -1 on all IVs (Group 1 above) and each of k - 1 other groups coded 1 on different IVs
A contrast (i.e., sum to 0) but not orthogonal (i.e., rs not equal to 0)
Compares each of k - 1 groups coded 1 to grand mean (equivalently to average of other groups)
Orthogonal Coding
Numerous alternative kinds of Orthogonal IVs (e.g., see Polynomial codes)
Above, Group 1 compared to Group 2, Groups 1 & 2 to 3, and so on to Groups 1 to k - 1 compared to Group k
A contrast (i.e., sum to 0) and orthogonal (i.e., rs equal 0)
Equivalent to test of difference between means for group(s) coded -1 and group coded non-zero
Polynomial Coding (Orthogonal)
Contrasts corresponding to Linear (equal steps), Quadratic (u-shaped or inverted u), and higher patterns
A contrast (i.e., sum to 0) and orthogonal (i.e., rs equal 0)
Testing for Linear, Quadratic, and higher patterns in ordered categorical predictor
For higher values of k, see Table T.5 in Appendix of Tables
Categorical Predictors and Regression 14.26
Interactions and Regression 15.1
CHAPTER 15
INTERACTION AND REGRESSION
MR and Statistical Interactions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
Separate Regression Analyses . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
Combined Regression Analysis with Interaction Term . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
SPSS and Interaction Terms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
Interaction Between Numerical Predictors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
Graphing Interactions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
Interactions and Nonlinear Regression . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
Interactions and Regression 15.2
The previous chapter demonstrated how multiple regression can accommodate categorical
predictors. Chapter 15 examines the related question of interactions and how multiple regression
can incorporate such effects. Interaction has a specific meaning in statistics; specifically, an
interaction occurs when the effect of one variable varies across the levels of another variable.
For example, a non-smoking program might benefit people who smoke relatively few cigarettes a
day but not help people who smoke a lot. Because the Program effect varies across levels of the
Amount Smoked variable (i.e., the program has little effect for heavy smokers and a robust effect
for light smokers), the Program and Amount Smoked variables demonstrate a statistical
interaction. Or a treatment might be effective for Men but not for Women (or vice versa).
Multiple regression can be used to examine such interactions, but researchers must create special
predictor variables that are sensitive to the interactions. This approach also provides yet another
way to incorporate non-linear relationships into regression analyses, and it can be extended to
interactions between numerical predictors.
MR AND STATISTICAL INTERACTIONS
It is often the case in Psychology and other disciplines that the effect of one predictor
depends on or interacts with another predictor. One common type of interaction in educational
research, for example, is the Aptitude-Treatment interaction, which occurs when certain methods
of teaching (the Treatment) work better with certain types of students (the Aptitude). To
illustrate, stronger students might learn better from unstructured, participatory instruction than
from traditional lectures, whereas weaker students might learn better from structured lectures.
This is an interaction between Type of Instruction and Student Aptitude. A second example of
an interaction in educational research would be attitudes toward mathematics having more of an
effect on performance in Mathematics courses for one gender than the other.
Interactions and Regression 15.3
6
8
10
12
14
2 4 6 8
Social Class
Closed Society
Open Society
Figure 15.1. Interaction Between Social Class
and Societal Value on Education.
In regression analyses, interactions
appear as differences in regression
coefficients between two or more groups.
Consider the relation between social class and
educational achievement in different
societies, as shown in Figure 15.1. In a
closed society that does not emphasize
education and includes monetary barriers to
schooling, social class might be a strong
predictor of educational achievement.
Members of disadvantaged groups would lack the financial resources and perhaps the
encouragement to pursue higher education and would therefore be more likely to terminate their
education earlier than advantaged members of that society. In an open society that values
education for all its members and has few financial barriers, the relation between social class and
educational achievement would be weaker and perhaps even eliminated.
Such an interaction would be revealed by differences between the two societies in the
regression coefficients for predicting educational achievement from some measure of social
class. In Figure 15.1, the slope for the closed society is steeper than for the open society. The
interaction could also be described in terms of the differing effect of society type (Open or
Closed) for lower and upper social classes. Society type has little or no effect on the upper
classes; they achieve high levels of education irrespective of whether the society is open or
closed. On the other hand, the effect of society type is quite dramatic for the lower classes,
whose members achieve more schooling in the open society than in the closed society. The
effect of society type for intermediate social classes falls between these extremes.
Our hypothetical example of an interaction involves the effect on romantic intimacy of
the duration of the relationship and external stress. Researchers theorized that many couples
become less intimate as they spend longer in a relationship. The researchers also proposed,
however, that this negative relation could be partly overridden by stressful events that reduce
intimacy and have a larger absolute effect on new couples.
Interactions and Regression 15.4
Statistics (S) Yes (S=0) No (S=1)SB Yrs Int SB Yrs Int 1 17 19 11 13 26 2 13 26 12 20 22 3 9 15 13 10 26 4 11 23 14 11 38 5 12 16 15 16 29 6 12 20 16 13 29 7 16 22 17 15 27 8 11 15 18 10 34 9 12 18 19 17 2510 20 10 20 18 30
Box 15.1. Data for Intimacy Study.
No Statistics Students
� = 39.55 - .766 × Yrs
R = .574 R2 = .33
DF SS MS F pReg 1 63.421 63.421 3.934 .08Res 8 128.979 16.122Tot 9 192.400
tb1 = -1.983 = �3.934
Statistics Students
� = 22.81 - .33 × Yrs
R = .238 R2 = .057
DF SS MS F pReg 1 11.011 11.011 .480 .51Res 8 183.389 22.92358Tot 9 194.400
tb1 = - .693 = �.480
Box 15.2. Relation of Intimacy to Years for Two Groups.
To examine this theory, the researchers
obtained scores for length in the relationship (Yrs) and
amount of intimacy (Int) for 20 university students, 10
of whom were experiencing the stress of taking
statistics and 10 of whom were not taking statistics.
The results are shown in Box 15.1 (SB is simply a
subject code). The basic question is whether the
relationship between Years in the relationship (YRS)
and Intimacy (INT) is the same for the Statistics (S = 0)
and No Statistics (S = 1) groups. That is, do subjects 1 to 10 (S = 0) show the same relationship
between YRS and INT as subjects 11 to 20 (S = 1)?
Separate Regression Analyses
Box 15.2 shows the
results from simple
regressions of Intimacy on
Years, separately for the
Statistics and No Statistics
groups. The No Statistics
results confirm the research
hypothesis about the relation
between Intimacy and Years
in the relationship: the
negative relationship with
Years accounts for 32.96%
of the variability in
Intimacy. The (equivalent)
F and t tests demonstrate
that the effect is significant by a directional test, p = .08/2 = .04. The best-fit regression equation
is � = 39.553 - .766 × Y. The No Statistics group starts out quite high on the Intimacy scale
Interactions and Regression 15.5
Figure 15.2. Regression of Intimacy on Length
of Relationship for Two Groups.
(Intercept = 39.553), but decreases by .766 units with each additional year in the relationship.
The Statistics group shows less evidence for a negative relation between Years and
Intimacy. The weak negative relation accounts for only 5.7% of the variation in intimacy and
does not approach significance, p = .51. Intimacy declines less with years for the Statistics group
than for the No Statistics group, primarily because the Statistics group has a lower level of
intimacy than the No Statistics group for relations in their early Years (i.e., difference in b0 =
39.553 - 22.811 = 16.742 units).
The two regression equations are
plotted in Figure 15.2, and show the
somewhat steeper slope for the No Statistics
group than for the Statistics group. When
plotted in this way, interactions appear as
non-parallel regression lines. Sometimes the
interaction is so robust that the regression
lines actually intersect or cross. Other times,
as here, the regression lines do not cross, at
least for the range of Years plotted. The lines
would, however, converge and eventually
cross at higher levels for the duration of
relationship variable if the trends of each line
continued unchanged.
The slope for the No Statistics group is significantly different from 0 and the slope for the
Statistics group is not significantly different from 0, but these separate significance tests do not
directly determine whether the difference between the two coefficients (i.e., between -.766 and
-.330) is significant. It is possible that -.766 differs significantly from 0, but does not differ
significantly from -.330. In terms of the non-parallel regression lines in Figure 15.2, the
differences between the slopes may have occurred by chance. To determine the significance of
the difference between the regression coefficients, the results for both groups must be analyzed
together, with the interaction between the two groups (i.e., the difference between their
Interactions and Regression 15.6
regression coefficients) coded somehow by a predictor variable.
Combined Regression Analysis with Interaction Term
The previous chapter demonstrated how codes can be used to indicate group membership
for categorical predictors, such as Statistics vs. No Statistics in the present study. A multiple
regression analysis for all 20 scores could include Years as a numerical predictor and another
variable S as a categorical predictor for the Statistics (S = 0) vs. No Statistics (S = 1) Groups.
The two-predictor equation would indicate whether Years (Y) and Statistics (S) predict Intimacy
jointly and independently.
This two-predictor equation would not, however, provide any measure of the difference
between the regression coefficients for Years in the Statistics and No Statistics groups. Testing
the interaction directly requires a third predictor that is the product of Years times the Statistics
code (i.e., SY = S × Y). The regression coefficient for SY reflects the difference in the
regression coefficients of the separate Statistics and No Statistics equations.
As shown shortly, the best-fit regression equation for our three-predictor equation is: � =
22.811 + 16.742 × S - .332 × Y - .434 × SY. Because SY = S × Y, this equation can be rewritten
as: � = (22.811 + 16.742 × S) - (.332 + .434 × S)×Y. The revised equation can be used to
determine two separate equations, by replacing S with 0 for the Statistics group (i.e., � = 22.811 -
.332 × Y) and by replacing S with 1 for the no statistics group (i.e., � = 39.553 - .766 × Y).
These two equations correspond to the simple equations for the Statistics and No Statistics
groups, as reported in Box 15.2. The single multiple regression equation therefore contains the
simple equations for both of the groups.
The coefficients in the 3-predictor equation and their statistical tests are directly related to
the simple equations or to differences between the simple equations. The intercept in the MR
equation is b0 = 22.811, which is b0 for the Statistics group (S = 0). The coefficient for Years is
bY = -.332, which is bY for the Statistics group (S = 0). Thus, b0 and bY in the MR equation
define the simple equation for the Statistics group, the group coded zero by the dummy variable.
The remaining regression coefficients represent differences between the Statistics and No
Statistics groups. The coefficient for the S indicator variable is bS = 16.742, which is the
difference in the b0s from the separate regressions (i.e., 39.553 = 22.811 + 16.742). This term
Interactions and Regression 15.7
essentially represents differences between groups at a value of zero on the predictor variable (i.e.,
Years = 0 in our example). Interpretation of this coefficient and its significance depends on a
number of factors, such as whether the interaction term is also significant, and whether the
intercept value of the predictor is meaningful. Here the difference in intercepts would be for
years in relationship equal to 0, probably not that meaningful in considering the impact of
statistics on intimacy for the two groups.
Of primary importance here is the coefficient for the SY interaction term, bSY = -.434.
This coefficient is the difference in the bYs from the simple regressions (i.e., -.766 = -.332 +
-.434). The SY regression coefficient reflects the differences between the slopes for the two
groups, and the t-test for the contribution of SY to the MR equation is therefore a test of the
significance of the difference in the separate slopes. In the present study, tSY = -.709, p = .49, so
we do not reject H0: ßIYwithinStats = ßIYwithinNoStats (withinStats indicates the regression coefficient for
the Statistics group and withinNoStats indicates the regression coefficient for the No Statistics
group). Concretely, -.766 and -.332 do not differ significantly from one another because
differences in slopes this large could occur too often (> .05) by chance.
The unique feature of these analyses for interaction is simply that one of the predictors is
the product of other predictors. This method can also be used to study interactions between two
numerical variables; the numerical variables are multiplied together and the regression
coefficient for the resulting product term reflects the interaction between the two numerical
variables that were multiplied together. If the interaction coefficient is nonsignificant, then the
slope for one predictor does not vary across levels of the other predictor. If the interaction is
significant, the slope for one predictor either increases or decreases across levels of the other
predictor.
Interactions and Regression 15.8
LIST
SUBJ STAT YEARS INTIM STXYR PRD.ALL RES.ALL 1.00 .00 17.00 19.00 .00 17.17283 1.82717 2.00 .00 13.00 26.00 .00 18.49950 7.50050 3.00 .00 9.00 15.00 .00 19.82617 -4.82617 4.00 .00 11.00 23.00 .00 19.16284 3.83716 5.00 .00 12.00 16.00 .00 18.83117 -2.83117 6.00 .00 12.00 20.00 .00 18.83117 1.16883 7.00 .00 16.00 22.00 .00 17.50450 4.49550 8.00 .00 11.00 15.00 .00 19.16284 -4.16284 9.00 .00 12.00 18.00 .00 18.83117 -.8311710.00 .00 20.00 10.00 .00 16.17782 -6.1778211.00 1.00 13.00 26.00 13.00 29.59574 -3.5957412.00 1.00 20.00 22.00 20.00 24.23404 -2.2340413.00 1.00 10.00 26.00 10.00 31.89362 -5.8936214.00 1.00 11.00 38.00 11.00 31.12766 6.8723415.00 1.00 16.00 29.00 16.00 27.29787 1.7021316.00 1.00 13.00 29.00 13.00 29.59574 -.5957417.00 1.00 15.00 27.00 15.00 28.06383 -1.0638318.00 1.00 10.00 34.00 10.00 31.89362 2.1063819.00 1.00 17.00 25.00 17.00 26.53191 -1.5319120.00 1.00 18.00 30.00 18.00 25.76596 4.23404
Box 15.3. Original and Derived Scores for Intimacy
Study.
SPSS and Interaction Terms
The hypothetical study just
discussed examined the relationship
between scores on a measure of
intimacy (INTIM) and the duration of
interpersonal relationships (YEARS),
as qualified by whether or not a person
is taking statistics (STAT = 0 for
Statistics group and STAT = 1 for No
Statistics group). The original data are
presented in Box 15.3, along with
variables created during the analysis.
The basic questions to be addressed
are: (a) does intimacy decrease or
increase or remain constant with years in a relationship, and (b) is the relation between intimacy
and years the same for STAT and NOSTAT subjects?
Box 15.4 reports separate regressions of INTIM on YEARS for STAT and NOSTAT
participants; TEMPORARY and SELECT IF statements were used to analyze the two groups
separately (alternatively, SPSS’s SPLIT FILE command could have been used to obtain these
separate analyses). When the STAT group alone is studied (STAT = 0), there is little evidence
for any relation between YEARS and INTIM: R = .238, R² = .057, and the equivalent F and t
tests do not approach significance, p = .5079.
Interactions and Regression 15.9
TEMPORARY.SELECT IF STAT=0. /* Statistics GroupREGR /VAR = intim years /DEP = intim /ENTER.
Multiple R .23800R Square .05664Adjusted R Square -.06128Standard Error 4.78786
DF Sum of Squares Mean SquareRegression 1 11.01139 11.01139Residual 8 183.38861 22.92358
F = .48035 Signif F = .5079
Variable B SE B Beta T Sig TYEARS -.331668 .478547 -.237998 -.693 .5079(Constant) 22.811189 6.542275 3.487 .0082
TEMPORARY.SELECT IF STAT=1. /* No Statistics GroupREGR /VAR = intim years /DEP = intim /ENTER.
Multiple R .57414R Square .32963Adjusted R Square .24584Standard Error 4.01526
DF Sum of Squares Mean SquareRegression 1 63.42128 63.42128Residual 8 128.97872 16.12234
F = 3.93375 Signif F = .0826
Variable B SE B Beta T Sig TYEARS -.765957 .386190 -.574136 -1.983 .0826(Constant) 39.553191 5.666609 6.980 .0001
Box 15.4. Separate Regressions for Statistics and No Statistics Groups.
The
results for the
NOSTAT
group reveal
a stronger
relation
between YRS
and INTIM:
R = .574, R2
= .3296. The
F- and t-tests
here
approach
significance,
p = .0826,
and would be
significant by
a one-tailed
test. A one-
tailed or
directional test is appropriate given the predictions of the researchers. With respect to intercepts,
the NOSTAT group is initially higher on Intimacy than the STATS group (39.553 - 22.811 =
16.742), but the scores decrease more with years for the NOSTAT group (-.766 vs. -.332).
Interactions and Regression 15.10
COMP stxyr = stat*years.REGR /VAR = intim stat years stxyr /DESC /STAT = DEFA ZPP /DEP = intim /ENTER /SAVE PRED(prd.all) RESID(res.all).
Mean Std Dev LabelINTIM 23.500 6.909STAT .500 .513YEARS 13.800 3.350STXYR 7.150 7.714
Correlation: INTIM STAT YEARSSTAT .757YEARS -.148 .153STXYR .638 .951 .366
Multiple R .80969 R Square .65560
DF Sum of Squares Mean SquareRegression 3 594.63267 198.21089Residual 16 312.36733 19.52296
F = 10.15271 Signif F = .0006
Variable B SE B Beta Corr Part Partl T SigTSTXYR -.4343 .6129 -.4849 .6384 -.1040 -.1744 -.709 .489YEARS -.3317 .4416 -.1608 -.1478 -.1102 -.1845 -.751 .464STAT 16.7420 8.6796 1.2431 .7573 .2833 .4344 1.929 .072Const 22.8112 6.0375 3.778 .002
Box 15.5. MR Analysis with Interaction Term.
To test whether the slopes for STAT and NOSTAT subjects differ significantly, the group
variable (i.e., STAT) is multiplied times the YEARS variable to generate a new predictor
variable called STXYR. Intimacy scores are then regressed on three predictors: Years, the
Statistics indicator variable, and the interaction or product term STXYR. The SPSS commands
and results for this analysis are shown in Box 15.5. The initial COMPUTE statement creates the
interaction
variable. The
best-fit
regression
equation
agrees with
that presented
earlier. Of
particular
interest is the
STXYR
term, which
represents the
difference
between the
slopes from
the simple
regressions for STAT and YEARS (see Box 15.3 for these scores).
The three predictors account for much of the variability in Intimacy scores, R2 = .656 and
FRegression = 10.15, p = .0006. Despite this substantial prediction, none of the individual predictors
are significant in the 3-predictor model, although p = .07 for STAT by a two-tailed test.
Redundant predictors is a serious problem with interaction variables, however, because product
terms tend to correlate highly with one or more of the predictors from which they were formed.
The correlation matrix in Box 15.5 shows an r of .951 between STAT and the interaction term.
Interactions and Regression 15.11
Multiple R .80299 R Square .64480Adjusted R Square .60301 Standard Error 4.35329
DF Sum of Squares Mean SquareRegression 2 584.83016 292.41508Residual 17 322.16984 18.95117
F = 15.42992 Signif F = .0002
Variable B SE B Beta Corr Part Partl T SigTYEARS -.5572 .3017 -.2701 -.1478 -.2669 -.4088 -1.847 .082STAT 10.7572 1.9701 .7987 .7573 .7893 .7980 5.460 .000Const 25.8102 4.2422 6.084 .000
Box 15.6. MR Without Interaction Term.
There are ways to reduce the problem of correlated predictors (e.g., center the predictors by
subtracting their Ms before computing the cross-product or interaction term).
The significance tests associated with bS and bSY test the differences between the separate
regression equations. The marginally significant t for bS = 16.742, t = 1.929, p = .072, suggests
that the difference between intercepts for the STATS and NOSTATS groups almost reaches
conventional levels of significance. The nonsignificant t for bSY = -.434 (t = -.709, p = .489)
indicates that the difference between the regression coefficients for the STATS and NOSTATS
groups is not significant.
Given the nonsignificant interaction between STAT and YEARS, the interaction term
could be omitted and intimacy regressed on STAT and YEARS alone. The two-predictor
regression results are presented in Box 15.6.
STAT now makes a highly significant contribution to prediction, t = 5.46, p = .000, with
students not taking statistics (STAT = 1) showing more intimacy than those taking statistics
(STAT = 0). The highly correlated interaction term masked this significant effect in the three-
predictor equation. Intimacy declines somewhat with years (bI = -.557, t = -1.847, p = .082,
nondirectional test), and the decline is presumed to be the same for both groups in this analysis
(i.e., there is no interaction term to accommodate different slopes). Together the two variables
account for 64.48% of the variation in INTIM scores, only slightly less than the 65.56% when the
interaction was included.
When the emphasis is on the effect of the categorical variable (i.e., the STAT dummy
variable), the analysis in Box 15.6 is an example of Analysis of Covariance. Partialling out the
Interactions and Regression 15.12
effect of the numerical predictor (the covariate) reduces the error term for the test of differences
between groups, and can statistically control for differences on the covariate. This analysis is
identical to earlier analyses in Chapter 14 involving categorical and numerical predictors without
interaction terms.
INTERACTION BETWEEN NUMERICAL PREDICTORS
The preceding example involved an interaction between a categorical predictor (STAT =
0 or 1) and a numerical predictor (YEARS). But it is also possible to test interactions between
two numerical
variables. To
illustrate, the
relation of
negative
affect
(NEGAFF) to
social support
(SUPP) and
external
stressors
(STRESS)
was studied to
determine whether social support ameliorated (i.e., reduced) the harmful effect of external
stressors on negative affect. That is, more stress is generally associated with more negative
affect, but the researchers hypothesized that this relation would be stronger for people with little
social support and weaker for people with good social supports. The hypothetical data are
presented in Box 15.7 along with some additional variables produced for and by the regression
analysis described below.
To test the predicted interaction (i.e., that the effect of stress would vary as a function of
level of social support), the researchers first centered the two predictor variables by subtracting
LIST. SUBJ STRESS SUPP NEGAFF STR2 SUP2 STRXSUP PRDN.SSX RESN.SSX 1.0000 1.0000 1.0000 52.0000 -2.0000 -2.0000 4.0000 52.36000 -.36000 6.0000 2.0000 1.0000 51.0000 -1.0000 -2.0000 2.0000 57.32000 -6.3200011.0000 3.0000 1.0000 63.0000 .0000 -2.0000 .0000 62.28000 .7200016.0000 4.0000 1.0000 70.0000 1.0000 -2.0000 -2.0000 67.24000 2.7600021.0000 5.0000 1.0000 71.0000 2.0000 -2.0000 -4.0000 72.20000 -1.20000 2.0000 1.0000 2.0000 55.0000 -2.0000 -1.0000 2.0000 52.52000 2.48000 7.0000 2.0000 2.0000 58.0000 -1.0000 -1.0000 1.0000 56.89000 1.1100012.0000 3.0000 2.0000 67.0000 .0000 -1.0000 .0000 61.26000 5.7400017.0000 4.0000 2.0000 62.0000 1.0000 -1.0000 -1.0000 65.63000 -3.6300022.0000 5.0000 2.0000 71.0000 2.0000 -1.0000 -2.0000 70.00000 1.00000 3.0000 1.0000 3.0000 55.0000 -2.0000 .0000 .0000 52.68000 2.32000 8.0000 2.0000 3.0000 56.0000 -1.0000 .0000 .0000 56.46000 -.4600013.0000 3.0000 3.0000 64.0000 .0000 .0000 .0000 60.24000 3.7600018.0000 4.0000 3.0000 61.0000 1.0000 .0000 .0000 64.02000 -3.0200023.0000 5.0000 3.0000 64.0000 2.0000 .0000 .0000 67.80000 -3.80000 4.0000 1.0000 4.0000 50.0000 -2.0000 1.0000 -2.0000 52.84000 -2.84000 9.0000 2.0000 4.0000 51.0000 -1.0000 1.0000 -1.0000 56.03000 -5.0300014.0000 3.0000 4.0000 58.0000 .0000 1.0000 .0000 59.22000 -1.2200019.0000 4.0000 4.0000 68.0000 1.0000 1.0000 1.0000 62.41000 5.5900024.0000 5.0000 4.0000 69.0000 2.0000 1.0000 2.0000 65.60000 3.40000 5.0000 1.0000 5.0000 56.0000 -2.0000 2.0000 -4.0000 53.00000 3.0000010.0000 2.0000 5.0000 56.0000 -1.0000 2.0000 -2.0000 55.60000 .4000015.0000 3.0000 5.0000 55.0000 .0000 2.0000 .0000 58.20000 -3.2000020.0000 4.0000 5.0000 60.0000 1.0000 2.0000 2.0000 60.80000 -.8000025.0000 5.0000 5.0000 63.0000 2.0000 2.0000 4.0000 63.40000 -.40000
Box 15.7. Original and Derived Scores for Negative Affect Study.
Interactions and Regression 15.13
their Ms, which in the present case were both 2.00, and then creating the cross-product of the two
centered variables. These operations produced the STR2, SUP2, and STRXSUP variables shown
in Box 15.7.
The commands to produce these new predictors are shown at the beginning of Box 15.8,
which also shows the resulting regression analysis. In the analysis, the interaction variable was
entered separately (last) to allow for a clearer impression of the benefits of including an
interaction component in the regression equation.
COMP str2 = stress-3.COMP sup2 = supp -3.COMP strxsup = str2*sup2.REGRE /VARI = negaff str2 sup2 strxsup /DESCR /STAT = DEFAU CHANGE ZPP /DEP = negaff /ENTER str2 sup2 /ENTER /SAVE PRED(prdn.ssx) RESI(resn.ssx).
Mean Std. N Deviation NEGAFF 60.240000 6.6035344 25 STR2 .000000 1.4433757 25 SUP2 .000000 1.4433757 25 STRXSUP .000000 2.0412415 25
NEGAFF STR2 SUP2 STRXSUP STR2 .826 1.000 .000 .000 SUP2 -.223 .000 1.000 .000 STRXSUP -.182 .000 .000 1.000
R R Adjusted Std. Error Change Statistics Square R Square of the Model Estimate R Square F Change df1 df2 Sig. F Change Change 1 .856(a) .732 .708 3.5682947 .732 30.097 2 22 .000 2 .875(b) .766 .732 3.4178105 .033 2.980 1 21 .099
Model Sum of df Mean Square F Sig. Squares 1 Regression 766.440 2 383.220 30.097 .000(a) Residual 280.120 22 12.733 Total 1046.560 24
2 Regression 801.250 3 267.083 22.864 .000(b) Residual 245.310 21 11.681 Total 1046.560 24
Unstandardized Standardized t Sig. Correlations Coefficients Coefficients Model B Std. Error Beta Zero-order Partial Part 1 (Constant) 60.240 .714 84.410 .000 STR2 3.780 .505 .826 7.491 .000 .826 .848 .826 SUP2 -1.020 .505 -.223 -2.021 .056 -.223 -.396 -.223
2 (Constant) 60.240 .684 88.127 .000 STR2 3.780 .483 .826 7.820 .000 .826 .863 .826 SUP2 -1.020 .483 -.223 -2.110 .047 -.223 -.418 -.223 STRXSUP -.590 .342 -.182 -1.726 .099 -.182 -.353 -.182
Box 15.8. Regression Analysis for Negative Affect Study.
Interactions and Regression 15.14
65
50
41
60
70
2Stress3
80
Affect
Social Support3 24 15 6
Figure 15.3. 3-D Graph of Negative Affect
Data.
The Model 2 regression analysis in Box 15.8 demonstrates a highly significant overall
effect, F = 22.864, p = .000, R2 = .766, with Stress being a particularly strong predictor, t =
7.820, p = .000, pr2 = .8262 = .682, Beta = .826. The contribution of Social Support was more
modest, t = -2.11-, p = .047, pr2 = -.2232 = .050, and the interaction between Stress and Social
Support was marginally significant, t = -1.726, p = .099 (two-tailed), pr2 = -.1822 = .033, and in
the direction expected. The negative slope for the interaction means that the effect of Stress
became smaller as level of Social Support increased.
The nature of the interaction can be seen if one carefully examines the raw scores and the
predicted scores in Box 15.7. The scores are listed sorted first by level of Social Support (i.e.,
the 5 people with the lowest level of social support are presented first), and then sorted by Stress
within levels of Social Support (i.e., cases increase in stress levels from 1 to 5 within each level
of Social Support). Looking now at the NEGAFF column, note that the effect of Stress (i.e., the
increase in NEGAFF scores) is greater for lower levels of Social Support than for higher levels.
This is true both for the observed and predicted scores.
Graphing Interactions
Figure 15.3 shows a 3-dimensional
graph of the data for this hypothetical study.
One can see the interactive effect of Social
Support (left-right axis) and Stress (front-back
axis) on Negative Affect (the vertical axis) by
comparing the effects of each predictor across
levels of the other predictor. At high levels of
Social Support, for example, the effect of
Stress is less than at low levels of social
support. At Social Support = 1, Negative
Affect increases from just over 50 units to over
70 units. At Social Support = 5, however, Negative Affect scores increase more modestly from
about 56 to about 63. Equivalently, note that the effect of social support varies across levels of
stress. For low stress levels, there is little effect of social support as negative affect scores
Interactions and Regression 15.15
Figure 15.4. Predicted Negative Affect
Including Interaction Term.
Figure 15.5. Another Visual Representation
of the Interaction.
remain about 55 or so. At high levels of stress, however, negative affect decreases with social
support, from about 75 units for level 1 of social support to about 60 units at level 5. In general,
then, the effect of each predictor varies with levels of the other predictor. This is the defining
characteristic of an interaction.
Figure 15.4 shows a three-dimensional plot
of the prediction equation including the interaction
between stress and social support. The interaction
is shown even more clearly than the plot of the
observed data in Figure 15.3. The slope for social
support begins flat at low stress levels and becomes
increasingly negative with increases in stress.
Conversely, the slope for stress is steeper for low
levels of social support than for high levels of
social support.
Figure 15.5 shows a second way to
represent the interaction visually. Here a two-
dimensional scattergram is presented, with Social
Support on the horizontal axis and Negative Affect
on the vertical axis. The levels of the stress
variable are represented by separate symbols and
line styles for the five levels of stress. The graph
clearly shows that the effect of social support is
helpful for the highest three stress levels;
specifically, negative affect decreases with
increasing social support. But for the lowest two
levels of stress, there is no marked effect of Social
Support, and if anything increases in social support
are associated with slight increases in levels of
negative affect, rather than decreases.
Interactions and Regression 15.16
There are several reasons why the interaction term may have reached only marginal levels
of significance, including factors that we have discussed before (e.g., small sample size). To
illustrate the importance of sample size, Box 15.9 shows a regression analysis of the same data,
but with scores
now doubled
simply by
duplicating the
scores shown in
Box 15.7. Note
now that the
interaction effect
is highly
significant, although the values for the intercept and slope are identical to those in Box 15.8. The
extra 25 subjects is sufficient to greatly strengthen the significance of the effect.
In addition to ensuring that sample sizes are adequate, researchers must appreciate that
obtaining significance for interactions can be more challenging than for the overall effect of a
predictor. This topic is discussed more fully in texts on ANOVA for factorial studies, and in
books devoted to interaction analysis in regression.
INTERACTIONS AND NONLINEAR REGRESSION
Chapter 13 showed various ways for researchers to accommodate nonlinear relationships
within a regression analysis: polynomial regression, transformations of predictors, and multiple
indicator variables for linear and nonlinear components of the relationship. Interaction analysis
provides another way of examining nonlinear effects. We illustrate with the reaction time study
analyzed in Chapter 13.
REGRE /DEP = negaff /ENTER strdev supdev strxsup.
Model R R Square Adjusted R Std. Error of Square the Estimate 1 .875(a) .766 .750 3.265831008
Model Sum of Squares df Mean Square F Sig. 1 Regression 1602.500 3 534.167 50.083 .000(a) Residual 490.620 46 10.666 Total 2093.120 49
Model Unstandardized Standardized t Sig. Coefficients Coefficients B Std. Error Beta 1 (Constant) 60.240 .462 130.430 .000 strdev 3.780 .327 .826 11.574 .000 supdev -1.020 .327 -.223 -3.123 .003 strxsup -.590 .231 -.182 -2.555 .014
Box 15.9. Analysis of Interaction with Double the Sample Size.
Interactions and Regression 15.17
Figure 15.6. Predicted Values Using Interaction Terms.
Box
15.10 shows a
simple linear
regression
analysis for
this study, R2 =
.880, a strong
and highly
significant
effect.
Although
highly
significant,
plots of the original data or the residual scores from a linear regression reveal a marked bend in
the data. In essence, the line tends to flatten out as uncertainty (U) increases.
Box 15.10 illustrates how to
accommodate this bend using
interaction terms. First, the levels of u
are divided into low (lohi = -1) and
high (lohi = +1) levels. Second, the
original u score is centered by
subtracting out its mean. Finally, we
compute the interaction between these
two predictors. A regression with
these three predictors increases the
strength of the relationship, R2 = .928.
The plot of the predicted scores in
Figure 15.5 shows that the equation
produced in Box 15.10 is actually two
REGRE /DEP = rt /ENTER u. Model R R Square Adjusted R Std. Error of Square the Estimate 1 .938(a) .880 .876 10.196657560...
RECODE u (1 2 3 4 = -1) (5 6 7 8 = +1) INTO lohi.COMP udev = u - 4.5.COMP uxlohi = udev * lohi.REGRE /DEP = rt /ENTER udev lohi uxlohi /SAVE PRED(prdint).
Model R R Square Adjusted R Std. Error of Square the Estimate 1 .963(a) .928 .920 8.178259507
Model Sum of Squares df Mean Square F Sig. 1 Regression 24183.125 3 8061.042 120.523 .000(a) Residual 1872.750 28 66.884 Total 26055.875 31
Model Unstandardized Standardized t Sig. Coefficients Coefficients B Std. Error Beta 1 (Constant) 364.763 2.963 123.112 .000 udev 13.075 1.293 1.050 10.111 .000 lohi -3.650 2.963 -.128 -1.232 .228 uxlohi -5.350 1.293 -.210 -4.137 .000
Box 15.10. Interaction Analysis for Nonlinear Relationships.
Interactions and Regression 15.18
straight lines, with somewhat different slopes. The slope is steeper for the low values of u and
shallower for the high values of u. It is this bend that allows additional variation to be captured.
And that additional variation is significant, as shown by the significance of the interaction term
in Box 15.10, t = -4.137, p = .000. In principle, it would be possible to fit a series of straight
lines with different slopes to the data, but much beyond two or three segments are likely to be
difficult to explain theoretically and difficult to justify statistically.
CONCLUSIONS
This chapter has briefly demonstrated how it is possible to accommodate interaction
effects in multiple regression (i.e., that the effect of one variable depends on the level of other
variables in the analysis). We illustrated the use of cross-products of predictors to measure and
test interaction effects. The inclusion of categorical predictors (Chapter 14) and interactions
(Chapter 15) makes multiple regression an extremely general and powerful data analytic
technique.