Producing DataDesigning Experiments

3 Basic Principals of Experimental Design

• Control• Control the effects of lurking variables

on the response, most simply by comparing 2 or more treatments

• Randomize• Use impersonal chance to assign

experimental units to treatments• Replicate

• Repeat each treatment on many units to reduce chance variation in results

CRR

Control• The main idea in this part of the design is to

eliminate or minimize the effect of lurking variables• Our goal is to show that the response was caused by

our treatment and not “confounded” by other variables Comparison is an aspect of control

• Compare two “groups” by giving one the treatment and one a placebo

• Looking for the “placebo effect” - where subject responds similarly despite receiving the placebo

• Control Group – group that receives the placebo treatment

When you have a control group, both groups get the lurking variables, so you can focus on the

effectiveness of your treatment!!!

Randomization

• This is the rule used to assign the experimental units the treatment• Chance should be used to divide experimental units

into groups (SRS – random #’s)• This ensures the similarity of the groups receiving the

different treatments• The groups must be similar to further eliminate lurking

variables and their influences• If groups are assigned by the experimenter either by

choice or through matching, bias is introduced

Consider our caffeine study: Problems among the groups could occur if there are individuals that have a higher tolerance for caffeine, or

are of different age or gender or race, etc. By randomizing the assignment of individuals to

the groups, we look to “even” out the differences by creating similar groups which

will reduce the effects of these variables.

To “help out” the effectiveness of random assignment we should:

1)Use enough experimental units

2)Make sure Groups are of equal size

3)Stratify if necessary

Drawing Experimental Diagrams

A food company wants to asses the nutritional quality of a new “instant breakfast”. They are going to examine a rat’s weight gain over a 28 day period by giving a control group of rats a standard diet.

Random Assignment

Group 1

15 Rats

Group 2

15 Rats

Treatment 1

New Diet

Treatment 2

Standard Diet

Compare Weight Gain

What’s the Goal of the Big 3?• Control

• Control the effects of lurking variables on the response, most simply by comparing 2 or more treatments

• Randomize• Use impersonal chance to assign experimental

units to treatments

• Replicate• Repeat each treatment on many units to reduce

chance variation in results

To produce an experiment with a result that is:

STATISTICALLY SIGNIFICANT

What’s Statistically Significant?

An observed effect that is so large that it would

rarely occur by CHANCE!!

This means that we have good evidence to support

the desired effect

“There is not statistically significant evidence that caffeine causes a freak to go crasy!!! Thanks to

Mummy!

Matched Pairs Design

• Matched Pairs• Match subjects in a comparative

experiment to ensure results are truly comparative• Control vs. Experimental to 2 similar

subjects

By matching a subject with

another of similar characteristics, I limit the effect of lurking variables (differences in

subjects)That gives me a better chance of

showing a CAUSAL relationship!!!

Making Inferences about the Population

• The Better the Experiment…• The More Valid a Generalization can Be• Generalizations about Causation must

be examined on a case by case basis

Each experiment should exhibit some element of the big 3:

CRR

ontrol

andomization

eplication

Select your Design based on your

experiment… Just be sure to include

all three elements!!