Research Problems

Hawthorne Effect

Western Electric’s Hawthorne Plant 1939 study of light intensity

productivity went…up Potential Solutions:

run experiment for a longer perioduse a control group

John Henry Effect

legend of black railway worker control group overcompensates Potential Solutions:

don’t do threatening experiments don’t set up obviously competitive situations don’t tell control group that they are control group

• conduct in another school somewhere else unfortunately, produces new variable of different

school, neighbourhood, etc.!

Placebo Effect

introduce placebo in attempt to make conditions in treatment and control groups identical

placebo effects are reactions that (after taking the placebo) cannot be explained by the chemical or medical effects of the placebo.—psychological factors

Placebo Effect Potential Solutions double-blind experiment

secrecy but then violate principle of informed consent

screen out or balance number of placebo reactors in treatment & control groups

the Pygmalion effect

Rosenthal and Jacobson, 1968 self-fulfilling prophecy Potential solution:

do not tell subjects what experiment is about but what happens to principle of informed consent?

Demand Characteristics

rumour setting instructions status and personality of researcher unintentional cues from experimenter experimental procedure itself

Demand Characteristics

Potential Solutions

reduce clarity of demand characteristics

generate alternative demand characteristics

reduce subjects’ motivation to respond to demand characteristics

Validity

Internal Validity– control experiment to eliminate

extraneous variables External Validity

– outcome can be generalized to other populations in other settings

Threats to Internal Validity History

change producing events in addition to the experimental treatment

Maturationsubjects grow older, learn more, can do

more

Threats to Internal Validity

Testingin pre-test/post-test designs — subjects may

remember questions from pre-test

Instrumentationchanges in measurementchanges in observerchanges in subjects (testing effects)

Threats to Internal Validity

Differential SelectionDifferences between treatment and control

groupE.g., volunteers vs non-volunteers

Experimental Mortality (or Attrition)subjects that drop out of experiment may

differ from others in important ways

Threats to Internal Validity

Testing & Experimental Treatment InteractionIn pre-test/post-test design, pretest may

interact with experimental treatment to exaggerate result

Statistical Regression

Threats to Internal Validity

Not Paying Attention to What’s Really Going On

External Validity

1. Population Validity– can results be generalized from specific

sample to the population from which sample was drawn

2. Ecological validity– can results be generalized from contrived

conditions created by experimenter to another set of environmental conditions (i.e., real world)

Experimental Design

Experimental Design

Purpose is to design an experiment that controls for as many extraneous variables as possible

– Campbell and Cook, classic 1968 paper categorized experimental designs by how many variables they controlled

Weak Designs

Single Group Designs

The One-Shot Case Design

X 0X = treatment

o = observation or measurement

The One-Shot Case Design

hardly experiment at all– can’t be sure result was result of treatment, not

history, maturation, etc. no control over group selection could have problem with subject mortality (e.g.,

transient student population) since only tested once, can’t even measure gain —

maybe students knew it before we started consequently, results are largely meaningless

One Group Pre-test/Post-test Design

O1 X 02

One Group Pre-test/Post-test Design

Uncontrolled Extraneous Factors (all of them!) history e.g., Hawthorne effect maturation testing

Pre-test may have contributed to higher scores on post-test due to greater familiarity with types of questions or focus on certain topics

instrumentation pre-test and post-test the same? Observer the same?

selection: could be an atypical group mortality interaction of testing & experimental treatment statistical regression

One Group Pre-test/Post-test Design useful for studying stable dependent variable justified when

– extraneous factors can be estimated with a high degree of certainty or

– can be safely assumed to be nonexistent• E.g., not maturation because change is too dramatic for

maturation to explain it

Control Group Designs

Posttest-Only Nonequivalent Groups Design

X O1

-----------------------------------------

O2

(dotted line means not random selection)

Posttest-Only Nonequivalent Groups Design also called “static group comparisons” still a relatively weak design advantage over single one shot design that one can

compare, so cancel out maturation, etc.

Problems: Differential selection experimental mortality

Non-Equivalent Control Group Design

Expt O1 X O2

-----------------------------------------

Cntl O3 O4

can compare average (mean) gain O2-O1

with average gain O4-O3

Non-Equivalent Control Group Design stronger design (weakest of the strong

designs) common design in education because usually

can’t randomize assignment of students to classes

pre-test measures whether initial groups are similar on tested variable– could also match subjects based on pre-test but may miss other factors which could impact

results (e.g., better teacher in one group)

Non-Equivalent Control Group Design Factors controlled by inclusion of a control group

history maturation testing instrumentation (assuming same for both groups) statistical regression

Factors still NOT controlled differential selection experimental mortality (if any) testing and experimental treatment interaction

Pre-test/Post-test Control Group Design

R O1 X O2

R O3 O4

where R = randomly selected groups

Pre-test/Post-test Control Group Design still two uncontrolled factors Intersession History

– events that are specific to one group and not the other which occur during the group sessions

– To control for interssion history, need to balance (control) such factors as

• experimenters• time of day• day of week

by randomly assigning to groups of subjects

Pre-test/Post-test Control Group Design Interaction of testing and treatment

– Cannot be avoided when there is a pretest– eliminating the pre-test would therefore

improve design

The Post-test Only Control Group Design

R X O1

R O2

The Post-test Only Control Group Design do not need pre-test with random selection disadvantages

– random assignment may not be fully successful in eliminating initial differences between control and experimental groups

– cannot form subgroups (i.e., high, medium, & low) to determine whether the experimental treatment has a different effect on subjects at different levels of the variable as measured by the pretest (because no pretest)

Solomon Four-Group Design

R O X OR O OR X OR O

Solomon Four-Group Design basically, a pretest and non-

pretest experiment at the same time

controls for everything, except differential mortality

Time Series Design

O O O X O O O

Time Series Design

Similar to one-group pre-test /post-test design (weak #2) but additional pre- and post- measurements add power

additional measurements enable researcher to rule out maturation

testing effects as sources of influence shift from pretest to posttest