Equations in Simple Regression Analysis

The Variance

sx

nx2

2

1

The standard deviation

S sx x 2

The covariance

sxy

nxy 1

The Pearson product moment correlation

rs

s sxyxy

x y

The normal equations (for the regressions of y on x)

bxyx

s

s

xy

xy

x

2

2

a Y b Xyx -

The structural model (for an observation on individual i)

Y a b X ei yx i i

The regression equation

( )

( )

Y a b X

Y b X b X

Y b X X

Y b X

yx

yx yx

yx

yx

Partitioning a deviation score, y

y Y Y

Y Y Y Y Y Y

Y Y Y Y

( ) ( )

( ) ( )

Partitioning the sum of squared deviations (sum of squares,

SSy)

y Y Y

Y Y Y Y

Y Y Y Y

SS SS

2 2

2

2 2

( )

[( ) ( )]

( ) ( )

reg res

Calculation of proportions of sums of squares due to regression and due to

error (or residual)

y

y

SS

y

SS

y

SS

y

SS

y

2

2

21

reg

2

res

2

reg

2

res

Alternative formulas for computing the sums of squares due to regression

SS Y Y

Y bx Y

bx

b x

xy

xx

xy

x

xy

xxy

b xy

reg

( )

( )

( )

( )

( )

( )

2

2

2

2 2

2

2 22

2

2

2

Test of the regression coefficient, byx, (i.e. test the null hypothesis that byx =

0)First compute the variance of estimate

s est

Y Y

N kSS

N k

y x y

2 2

2

1

1

( )

( )

res

Test of the regression coefficient, byx, (i.e. test the null hypothesis that byx =

0)Then obtain the standard error of estimate

Then compute the standard error of the regression coefficient, Sb

s sy x y x 2

ss

x n

s

x Nb

y x y x

2

2 21 1( ) / ( ) ( ) / ( )

The test of significance of the regression coefficient (byx)

The significance of the regression coefficient is tested using a t test with (N-k-1) degrees of freedom:

tb

s

b

S

S n

yx

b

yx

y x

x

1

Computing regression using correlations

The correlation, in the population, is given by

The population correlation coefficient, ρxy, is estimated

by the sample correlation coefficient, rxy

xy

x yN

rz z

Ns

s s

xy

x y

xyx y

xy

x y

2 2

Sums of squares, regression (SSreg)

Recalling that r2 gives the proportion of variance of Y accounted for (or explained) by X, we can obtain

or, in other words, SSreg is that portion of SSy predicted or explained by the regression of Y on X.

SS r y

SS r y

reg

res

2 2

2 21( )

Standard error of estimate

From SSres we can compute the variance of estimate and standard error of estimate as

(Note alternative formulas were given earlier.)

sr y

N ks s

y x

y x y x

22 21

1( )

Testing the Significance of r

The significance of a correlation coefficient, r, is tested using a t test:

With N-2 degrees of freedom.

21

2

r

Nrt

Testing the difference between two correlations

To test the difference between two Pearson correlation coefficients, use the “Comparing two correlation coefficients” calculator on my web site.

Testing the difference between two regression coefficients

This, also, is a t test:

Where

was given earlier. When the variances, , are unequal, used the pooled estimate given on page 258 of our textbook.

22

21

21 bb SS

bbt

2bS

2bS

Other measures of correlation

Chapter 10 in the text gives several alternative measures of correlation:Point-biserial correlationPhi correlationBiserial correlationTetrachoric correlationSpearman correlation

Point-biserial and Phi correlation

These are both Pearson Product-moment correlationsThe Point-biserial correlation is used when

on variable is a scale variable and the other represents a true dichotomy.For instance, the correlation between an

performance on an item—the dichotomous variable—and the total score on a test—the scaled variable.

Point-biserial and Phi correlation

The Phi correlation is used when both variables represent a true dichotomy.For instance, the correlation between two

test items.

Biserial and Tetrachoric correlation

These are non-Pearson correlations.Both are rarely used anymore.The biserial correlation is used when

one variable is truly a scaled variable and the other represents an artificial dichotomy.

The Tetrachoric correlation is used when both variables represent an artificial dichotomy.

Spearman’s Rho Coefficient and Kendall’s Tau Coefficient

Spearman’s rho is used to compute the correlation between two ordinal (or ranked) variables.It is the correlation between two sets of

ranks.

Kendall’s tau (see pages 286-288 in the text) is also a measure of the relationship between two sets of ranked data.