experiment basics: designs psych 231: research methods in psychology

Experiment Basics: Designs

Psych 231: Research Methods in Psychology

Announcements

Due this week in labs - Group project: Methods sections

• Recommended/required: • Questionnaires/examples of stimuli, etc. – things that

you want to have ready for pilot week (week 10)

IRB worksheet (including a consent form)• Doing this as an in-lab exercise

Group Project ratings sheet Exam 2 a week and a half away Fun (and informative) site: ThePsychFiles.com podcast

Experimental designs

So far we’ve covered a lot of the general details of experiments

Now let’s consider some specific experimental designs. Some bad (but not uncommon) designs (and potential fixes) Some good designs

• 1 Factor, two levels• 1 Factor, multi-levels• Factorial (more than 1 factor)• Between & within factors

Poorly designed experiments

Bad design example 1: Does standing close to somebody cause them to move? (theory of personal space)

“hmm… that’s an empirical question. Let’s see what happens if …”

So you stand closely to people and see how long before they move

Problem: no control group to establish the comparison group (this design is sometimes called “one-shot case study design”)

Very Close (.2 m) Close (.5 m) Not Close (1.0 m)

Fix: introduce a (or some) comparison group(s)


Bad design example 2: Does a relaxation program decrease the urge to

smoke?

2 groups• relaxation training group• no relaxation training

group

The participants choose which group to be in

Traininggroup

No training (Control) group


Non-equivalent control groups

participants

Traininggroup

No training (Control) group

Measure

Measure

Self Assignment

Independent Variable

Dependent Variable

RandomAssignment Problem: selection bias for the two groups

Fix: need to do random assignment to groups

Problem: selection bias for the two groups

Fix: need to do random assignment to groups

Bad design example 2:


Bad design example 3: Does a relaxation program decrease the urge to

smoke?

Pre-test desire level Give relaxation training program Post-test desire to smoke


One group pretest-posttest design

participantsPre-test Training

groupPost-testMeasure

Independent Variable Pre vs. Post

Dependent Variable

Dependent Variable

Problems include: history, maturation, testing, and more

Pre-test No Training group

Post-testMeasure

Fix: Add another factor

Bad design example 3:



Now let’s consider some specific experimental designs. Some bad (but not uncommon) designs Some good designs


1 factor - 2 levels

Good design example How does anxiety level affect test performance?

• Two groups take the same test• Grp1(low anxiety group): 5 min lecture on how good

grades don’t matter, just trying is good enough

• Grp2 (moderate anxiety group): 5 min lecture on the importance of good grades for success

1 Factor (Independent variable), two levels• Basically you want to compare two treatments (conditions)

• The statistics are pretty easy, a t-test

What are our IV and DV?

1 factor - 2 levels

participants

Low

Moderate Test

Test

Random Assignment

Anxiety Dependent Variable



anxiety

low moderate

8060

low moderatete

st p

erf

orm

an

ce

anxiety

One factor

Two levels

Use a t-test to see if these points are statistically different

T-test = Observed difference between conditions

Difference expected by chance

1 factor - 2 levels

Advantages: Simple, relatively easy to interpret the results Is the independent variable worth studying?

• If no effect, then usually don’t bother with a more complex design

Sometimes two levels is all you need• One theory predicts one pattern and another predicts a

different pattern

1 factor - 2 levels

low moderatete

st p

erf

orm

ance

anxiety

What happens within of the ranges that you test?Interpolation

Disadvantages: “True” shape of the function is hard to see

• Interpolation and Extrapolation are not a good idea

1 factor - 2 levels

Extrapolation

low moderate

test

perf

orm

an

ce

anxiety

What happens outside of the ranges that you test?

Disadvantages: “True” shape of the function is hard to see

• Interpolation and Extrapolation are not a good idea

1 factor - 2 levels

high



Now let’s consider some specific experimental designs. Some bad (but not uncommon) designs Some good designs


1 Factor - multilevel experiments

For more complex theories you will typically need more complex designs (more than two levels of one IV)

1 factor - more than two levels Basically you want to compare more than two

conditions The statistics are a little more difficult, an ANOVA

(Analysis of Variance)

Good design example (similar to earlier ex.)

How does anxiety level affect test performance?• Groups take the same test

• Grp1(low anxiety group): 5 min lecture on how good grades don’t matter, just trying is good enough

• Grp2 (moderate anxiety group): 5 min lecture on the importance of good grades for success


• Grp3 (high anxiety group): 5 min lecture on how the students must pass this test to pass the course

1 factor - 3 levels

participants

Low

Moderate Test

Test

Random Assignment

Anxiety Dependent Variable

High Test


anxiety

low mod high

8060 60

low mod

test

perf

orm

an

ce

anxiety

high


Advantages Gives a better picture of the relationship

(functions other than just straight lines)

Generally, the more levels you have, the less you have to worry about your range of the independent variable

low moderatete

st

perf

orm

ance

anxiety

2 levels

highlow mod

test

perf

orm

ance

anxiety

3 levels


Disadvantages Needs more resources (participants and/or

stimuli) Requires more complex statistical analysis

(ANOVA [Analysis of Variance] & follow-up pair-wise comparisons)

Pair-wise comparisons

The ANOVA just tells you that not all of the groups are equal. If this is your conclusion (you get a “significant ANOVA”)

then you should do further tests to see where the differences are

• High vs. Low

• High vs. Moderate

• Low vs. Moderate


So far we’ve covered a lot of the about details experiments generally

Now let’s consider some specific experimental designs. Some bad (but common) designs Some good designs

• 1 Factor, two levels• 1 Factor, multi-levels• Factorial (more than 1 factor) • Between & within factors

Factorial experiments

Two or more factors Some vocabulary

• Factors - independent variables• Levels - the levels of your independent variables

• 2 x 4 design means two independent variables, one with 2 levels and one with 4 levels

• “Conditions” or “groups” is calculated by multiplying the levels, so a 2x4 design has 8 different conditions

A1

A2

B1 B2 B3 B4

Factorial experiments

Two or more factors (cont.) Main effects - the effects of your independent variables ignoring

(collapsed across) the other independent variables Interaction effects - how your independent variables affect each

other• Example: 2x2 design, factors A and B• Interaction:

• At A1, B1 is bigger than B2

• At A2, B1 and B2 don’t differ

Everyday interaction = “it depends on …”

Interaction effects

Rate how much you would want to see a new movie (1 no interest, 5 high interest) Ask men and women – looking for an effect of gender

Not much of a difference

Interaction effects

Maybe the gender effect depends on whether you know who is in the movie. So you add another factor:

Suppose that George Clooney might star. You rate the preference if he were to star and if he were not to star.

Effect of gender depends on whether George stars in the movie or not

This is an interaction

A video lecture from ThePsychFiles.com podcast

Results of a 2x2 factorial design

The complexity & number of outcomes increases:• A = main effect of factor A

• B = main effect of factor B

• AB = interaction of A and B

• With 2 factors there are 8 basic possible patterns of results: 1) No effects at all

2) A only

3) B only

4) AB only

5) A & B

6) A & AB

7) B & AB

8) A & B & AB

2 x 2 factorial design

Condition

mean

A1B1

Condition

mean

A2B1

Condition

mean

A1B2

Condition

mean

A2B2

A1 A2

B2

B1

Marginal means

B1 mean

B2 mean

A1 mean A2 mean

Main effect of B

Main effect of A

Interaction of ABWhat’s the effect of A at B1?What’s the effect of A at B2?

Main effect of AMain effect of BInteraction of A x B

A

B

A1 A2

B1

B2

Main Effect of A

Main Effect of B

60

45

45

30

6030

6030

A

A1 A2

Dependent

Vari

able

B1

B2

✓X

X

Examples of outcomes


A

B

A1 A2

B1

B2

Main Effect of A

Main Effect of B

45

60

30

45

3030

6060

A

A1 A2

Dependent

Vari

able

B1

B2

✓X

X



A

B

A1 A2

B1

B2

Main Effect of A

Main Effect of B

45

45

45

45

6030

3060

A

A1 A2

Dependent

Vari

able

B1

B2

✓X

X



A

B

A1 A2

B1

B2

Main Effect of A

Main Effect of B

45

45

30

30

3030

6030

A

A1 A2

Dependent

Vari

able

B1

B2

✓

✓✓


Anxiety and Test Performance

test

perf

orm

ance

highlow mod

anxiety

easymedium

hard

80 80 80

8065 65

anxiety

low mod high

8035 3550

70

80

main effect of difficulty

8060 60

main effect of anxiety

Let’s add another variable: test difficulty.

easy

medium

hard

Test

diffi

cult

y

Interaction ? Yes: effect of anxiety depends on level of test difficulty

Factorial Designs

Advantages Interaction effects

– Always consider the interaction effects before trying to interpret the main effects

– Adding factors decreases the variability– Because you’re controlling more of the variables that

influence the dependent variable– This increases the statistical Power of the statistical tests

– Increases generalizability of the results– Because you have a situation closer to the real world (where

all sorts of variables are interacting)

Factorial Designs

Disadvantages Experiments become very large, and unwieldy The statistical analyses get much more complex Interpretation of the results can get hard

• In particular for higher-order interactions

• Higher-order interactions (when you have more than two interactions, e.g., ABC).


So far we’ve covered a lot of the about details experiments generally

Now let’s consider some specific experimental designs. Some bad (but common) designs Some good designs

• 1 Factor, two levels• 1 Factor, multi-levels• Factorial (more than 1 factor) • Between & within factors

Example

What is the effect of presenting words in color on memory for those words?

Two different designs to examine this question

Clock

Chair

Cab

Clock

Chair

Cab

Clock

Chair

Cab

Clock

Chair

Cab

So you present lists of words for recall either in color or in black-and-white.

participants

Coloredwords

BWwords

Test

2-levels

Each of the participants is in only one level of the IV

Between-Groups Factor

Clock

Chair

Cab

Clock

Chair

Cab

Clock

Chair

Cab

Clock

Chair

Cab

levelslevels

participantsColoredwords

BWwords

TestTest

2-levels, All of the participants are in both levels of the IV

Clock

Chair

Cab

Clock

Chair

Cab

Clock

Chair

Cab

Clock

Chair

Cab

levelslevels

Sometimes called “repeated measures” design

Within-Groups Factor

Between vs. Within Subjects Designs

Within-subjects designs

All participants participate in all of the conditions of the experiment.

participants

Coloredwords

BWwords

Test participantsColoredwords

BWwords

TestTest

Between-subjects designs Each participant

participates in one and only one condition of the experiment.



participants

Coloredwords

BWwords

Test participantsColoredwords

BWwords

TestTest




Between subjects designs

Advantages:

Independence of groups (levels of the IV)• Harder to guess what the experiment is about without

experiencing the other levels of IV • Exposure to different levels of the independent variable(s)

cannot “contaminate” the dependent variable• Sometimes this is a ‘must,’ because you can’t reverse the

effects of prior exposure to other levels of the IV• No order effects to worry about

• Counterbalancing is not required

participants

Coloredwords

BWwords

Test

Clock

Chair

Cab

Clock

Chair

Cab

Clock

Chair

Cab

Clock

Chair

Cab

Between subjects designs

Disadvantages

Individual differences between the people in the groups

• Excessive variability• Non-Equivalent groups

participants

Coloredwords

BWwords

Test

Clock

Chair

Cab

Clock

Chair

Cab

Clock

Chair

Cab

Clock

Chair

Cab

Individual differences

The groups are composed of different individuals

participants

Coloredwords

BWwords

Test



participants

Coloredwords

BWwords

Test

Excessive variability due to individual differences Harder to detect the effect of the IV if there

is one

RNR R



participants

Coloredwords

BWwords

Test

Non-Equivalent groups (possible confound) The groups may differ not only because of the IV, but also because the

groups are composed of different individuals

Dealing with Individual Differences

Strive for Equivalent groups Created equally - use the same process to

create both groups Treated equally - keep the experience as

similar as possible for the two groups Composed of equivalent individuals

• Random assignment to groups - eliminate bias• Matching groups - match each individuals in one

group to an individual in the other group on relevant characteristics

Matching groups

Group A Group B Matched groups Trying to create

equivalent groups Also trying to reduce

some of the overall variability

• Eliminating variability from the variables that you matched people on

RedShort21yrs

Bluetall

23yrs

Greenaverage22yrs

Browntall

22yrs

ColorHeight

Age

matchedRed

Short21yrs

matched Bluetall

23yrs

matchedGreen

average22yrs

matchedBrown

tall22yrs



participants

Coloredwords

BWwords

Testparticipants

Coloredwords

BWwords

TestTest




Within subjects designs

Advantages: Don’t have to worry about individual differences

• Same people in all the conditions• Variability between conditions is smaller (statistical

advantage)

Fewer participants are required


Disadvantages Range effects Order effects:

• Carry-over effects • Progressive error

• Counterbalancing is probably necessary to address these order effects


Range effects – (context effects) can cause a problem The range of values for your levels may impact

performance (typically best performance in middle of range).

Since all the participants get the full range of possible values, they may “adapt” their performance (the DV) to this range.

testCondition 2Condition 1

test

Order effects

Carry-over effects Transfer between conditions is possible Effects may persist from one condition into

another• e.g. Alcohol vs no alcohol experiment on the effects on

hand-eye coordination. Hard to know how long the effects of alcohol may persist.

How long do we wait for the effects to wear off?

Order effects

Progressive error Practice effects – improvement due to repeated

practice Fatigue effects – performance deteriorates as

participants get bored, tired, distracted

Dealing with order effects

Counterbalancing is probably necessary This is used to control for “order effects”

• Ideally, use every possible order • (n!, e.g., AB = 2! = 2 orders; ABC = 3! = 6 orders, ABCD = 4! = 24 orders, etc).

All counterbalancing assumes Symmetrical Transfer

• The assumption that AB and BA have reverse effects and thus cancel out in a counterbalanced design

Counterbalancing

Simple case Two conditions A & B Two counterbalanced orders:

• AB• BA

participants

Coloredwords

BWwords

TestTest

Coloredwords

BWwords

TestTest

Counterbalancing

Often it is not practical to use every possible ordering Partial counterbalancing

• Latin square designs – a form of partial counterbalancing, so that each group of trials occur in each position an equal number of times

Partial counterbalancing

Example: consider four conditions Recall: ABCD = 4! = 24 possible orders

1) Unbalanced Latin square: each condition appears in each position (4 orders)

DCBA

ADCB

BADC

CBAD

Order 1

Order 2

Order 3

Order 4

Partial counterbalancing

2) Balanced Latin square: each condition appears before and after all others (8 orders)

A B D C

B C A D

C D B A

D A C B

A B C D

B C D A

C D A B

D A B C

Example: consider four conditions Recall: ABCD = 4! = 24 possible orders

Mixed factorial designs

Mixed factorial designs Treat some factors as within-subjects

(participants get all levels of that factor) and others as between-subjects (each level of this factor gets a different group of participants).

This only works with factorial (multi-factor) designs

experiment basics: designs psych 231: research methods in psychology

Documents

levels1 factor

groupsbad design example

factorbad design example

chance1 factor

levelsgood design examplehow

testgrp1low anxiety

relaxation program

importance of good grades