statistics for the social sciences psychology 340 spring 2005 effect sizes & statistical power

Statistics for the Social Sciences

Psychology 340Spring 2005

Effect sizes & Statistical Power


Outline

• Effect size: Cohen’s d• Error types• Statistical Power Analysis


Performing your statistical test

Real world (‘truth’)

H0 is correct

H0 is wrong

Experimenter’s conclusions

Reject H0

Fail to Reject

H0

There really isn’t an effect

There really isan effect




H0 is correct

H0 is wrong




H0 is correct

H0 is wrong


H0 is correct

H0 is wrong

Type I error

Type II error

α

β

So there is only one distribution


H0 is correct

H0 is wrong

Type I error

Type II error

α

β

So there are two distributionsThe original

(null) distribution

The new (treatment) distribution

The original (null) distribution




H0 is correct

H0 is wrong

So there is only one distribution


H0 is correct

H0 is wrong

Type I error

Type II error

α

β


(null) distribution




Effect Size

H0 is wrong


H0 is correct

H0 is wrong

Type I error

Type II error

α

β


(null) distribution


• Hypothesis test tells us whether the observed difference is probably due to chance or not

• It does not tell us how big the difference is– Effect size tells us how much the two populations don’t overlap


Effect Size



• Figuring effect size

– Effect size tells us how much the two populations don’t overlap

μ1 − μ 2

μ2 μ1

But this is tied to the particular units of measurement

But this is tied to the particular units of measurement


Effect Size



d =μ1 −μ2

σ

• Standardized effect size


μ2 μ1

– Puts into neutral units for comparison (same logic as z-scores)

Cohen’s d


Effect Size



• Effect size conventions– small d = .2– medium d = .5– large d = .8


sm-m

= 21d

μ2 μ1


Error types


H0 is correct

H0 is wrong


Reject H0

Fail to Reject H0

I conclude that there is an effect

I can’t detect an effect

There really isn’t an effect

There really isan effect


Error types


H0 is correct

H0 is wrong


Reject H0

Fail to Reject H0

Type I error Type

II error

α

β

Type I error (): concluding that there is a difference between groups (“an effect”) when there really isn’t.

Type I error (): concluding that there is a difference between groups (“an effect”) when there really isn’t.

Type II error (): concluding that there isn’t an effect, when there really is.

Type II error (): concluding that there isn’t an effect, when there really is.


Statistical Power

• The probability of making a Type II error is related to Statistical Power– Statistical Power: The probability that the study will produce a statistically significant results if the research hypothesis is true (there is an effect)

Power =1−

• So how do we compute this?


Statistical Power

H0: is true (is no treatment effect)Real world (‘truth’)

Fail to reject H0

Reject H0


H0 is correct

H0 is wrong

Type I error

Type II error

α

β

= 0.05



Statistical Power

H0: is false (is a treatment effect)

= 0.05

Reject H0


H0 is correct

H0 is wrong

Type I error

Type II error

α

β The original (null) distribution


The new (treatment) distributionThe new (treatment) distribution

Fail to reject H0


Statistical Power

Fail to reject H0

Reject H0


H0 is correct

H0 is wrong

Type I error

Type II error

α

β

= probability of a Type II error





= 0.05

Failing to Reject H0, even though there is

a treatment effect


a treatment effect


Statistical Power

Fail to reject H0

Reject H0


H0 is correct

H0 is wrong

Type I error

Type II error

α

β

Power = 1 -





= probability of a Type II error

= 0.05


a treatment effect


a treatment effectProbability of

(correctly) Rejecting H0

Probability of (correctly)

Rejecting H0


Statistical Power

1) Gather the needed information: mean and standard deviation of the Null Population and the predicted mean of Treatment Population

• Steps for figuring power

μ2 μ1

μ2 = 55;σ = 2.5 μ1 = 60;σ = 2.5


Statistical Power

2) Figure the raw-score cutoff point on the comparison distribution to reject the null hypothesis


= 0.05From the unit normal table: Z = -1.645

Transform this z-score to a raw score

raw score =μ1 +σ(Z) =60 + (2.5)(−1.645) = 55.89

μ1

μ1 = 60;σ = 2.5


Statistical Power

3) Figure the Z score for this same point, but on the distribution of means for treatment Population


55.89Transform this raw score to a z-score

Z =X−μσ

=55.88 − 55

2.5

=0.355

μ2 = 55;σ = 2.5

Remember to use the properties of the

treatment population!

Remember to use the properties of the

treatment population!


Statistical Power

4) Use the normal curve table to figure the probability of getting a score more extreme than that Z score


= probability of a Type II error =0.355

From the unit normal table: Z(0.355) = 0.3594

Power = 1 -

Power =1−0.3594 =0.64The probability of detecting this an effect of this size from these populations is 64%

55.89


Statistical Power

-level

– Sample size

– Population standard deviation σ

– Effect size

– 1-tail vs. 2-tailed

Factors that affect Power:


Statistical Power

= 0.05

Fail to reject H0

Reject H0

Power = 1 -


-levelChange from = 0.05 to 0.01


Statistical Power

= 0.05

Fail to reject H0

Reject H0

Power = 1 -



= 0.01


Statistical Power

= 0.05

Fail to reject H0

Reject H0

Power = 1 -



= 0.01

So as the level gets smaller, so does the

Power of the test

So as the level gets smaller, so does the

Power of the test


Statistical Power

= 0.05

Fail to reject H0

Reject H0

Power = 1 -


Sample sizeChange from n = 25 to 100

€

σX

=σ

n

Recall that sample size is related to the spread of the distribution


Statistical Power

= 0.05

Fail to reject H0

Reject H0

Power = 1 -




Statistical Power

= 0.05

Fail to reject H0

Reject H0

Power = 1 -



As the sample gets bigger, the standard error gets smaller and the Power gets larger


Statistical Power

= 0.05

Fail to reject H0

Reject H0

Power = 1 -


Population standard deviationChange from σ = 25 to 20

€

σX

=σ

n

Recall that standard error is related tothe spread of the distribution


Statistical Power

= 0.05

Fail to reject H0

Reject H0

Power = 1 -




Statistical Power

= 0.05

Fail to reject H0

Reject H0

Power = 1 -


As the σ gets smaller, the standard error gets smaller and the Power gets larger



Statistical Power

= 0.05

Fail to reject H0

Reject H0

Power = 1 -


Effect sizeCompare a small effect (difference) to a big effect

μtreatment μno treatment


Statistical Power

= 0.05

Fail to reject H0

Reject H0

Power = 1 -




Statistical Power

= 0.05

Fail to reject H0

Reject H0

Power = 1 -



As the effect gets bigger, the Power gets larger


Statistical Power

= 0.05

Fail to reject H0

Reject H0

Power = 1 -


1-tail vs. 2-tailedChange from = 0.05 two-tailed

to = 0.05 two-tailed


Statistical Power

= 0.05

Fail to reject H0

Reject H0

Power = 1 -




p = 0.025

p = 0.025


Statistical Power

Fail to reject H0

Reject H0

Power = 1 -




p = 0.025

p = 0.025

= 0.05


Statistical Power

Fail to reject H0

Reject H0

Power = 1 -




p = 0.025

Two tailed functionally cuts the -level in half, which decreases the power.p = 0.025

= 0.05


Statistical Power

Factors that affect Power: -level: So as the level gets smaller, so does the Power of the test

– Sample size: As the sample gets bigger, the standard error gets smaller and the Power gets larger

– Population standard deviation: As the population standard deviation gets smaller, the standard error gets smaller and the Power gets larger

– Effect size: As the effect gets bigger, the Power gets larger

– 1-tail vs. 2-tailed: Two tailed functionally cuts the -level in half, which decreases the power


Why care about Power?

• Determining your sample size– Using an estimate of effect size, and population standard deviation, you can determine how many participants need to achieve a particular level of power

• When a result if not statistically significant– Is is because there is no effect, or not enough power

• When a result is significant– Statistical significance versus practical significance


Ways of Increasing Power

statistics for the social sciences psychology 340 spring 2005 effect sizes & statistical power

Documents

correct h

social sciences effect

effect slide

wrong real world truth

wrong slide

wrong type

social sciences psychology

populations dont overlap