Experimental Design

Experimental designs

• Experiment is probably the strongest design with respect to internal validity

• Recall that internal validity is at the center of all causal or cause-effect inferences

• When you want to determine whether some program or treatment causes some outcome to occur, then you are interested in having internal validity

Experimental design

• To really show that there is a causal relationship, you have to simultaneously address the two propositions:

• If X, then Y

• and

• If not X, then not Y

In other words…

• If the program is given, then the outcome occurs

• and

• If the program is not given, then the outcome does not occur

If you are able to provide evidence for both of these propositions, then you've isolated the program from all of the other potential causes of the outcome. That points to the causal effectiveness of the program.

Logic of experimental design • In the simplest type of experiment, we create

two groups that are "equivalent" to each other• One group (treatment group) gets the program

and the other group (control group) does not• The people are assigned randomly to each

group • Now, if we observe differences in outcomes

between these two groups, then the differences must be due to the only thing that differs between them -- that one got the program and the other didn't.

Babbie 2005

Components of Experiments

• Independent and dependent variables

• Pre-testing and post-testing

• Experimental and control groups

Independent and dependent variables

• Treatment or Independent Variable the stimulus, manipulation, or intervention that the researcher creates or delivers to one set of participants or clients

• Dependent Variable is the outcome or condition that may change as a result of being subjected to or exposed to the treatment or an independent variable

Pre-testing and post-testing

• Pre-test is the measurement of the dependent variable prior to treatment or intervention

• Post-test is measurement of the dependent variable after the treatment/ independent variable has been applied

Copyright @ Allyn & Bacon 2003

Experimental and Control groups

• Must be as similar as possible.

• An Experimental Group are those who receive the treatment or are exposed to the independent variable under study.

• The Control Group are those who do not receive the treatment or independent variable under study.

NOTATIONS

• Observations or Measures

• Treatments or Programs

• Groups

• Assignment to Group

• Time

Observations or Measures• These are symbolized by an 'O' in design

notation• An O can refer to a single measure (e.g., a

measure of body weight), a single instrument with multiple items (e.g., a 10-item self-esteem scale), a complex multi-part instrument (e.g., a survey), or a whole battery of tests or measures given out on one occasion

• If you need to distinguish among specific measures, you can use subscripts with the O, as in O1, O2, and so on.

Because all of the Os have a subscript of 1, there is some measure that is collected for both groups on both occasions

But the design also has two Os with a subscript of 2, both taken at the posttest. This means that there was some measure that were collected only at the posttest.

Treatments or Programs

• These are symbolized with an 'X' in design notations

• The X can refer to a simple intervention (e.g., a one-time surgical technique) or to a complex program (e.g., an employment training program)

• Usually, a no-treatment control or comparison group has no symbol for the treatment (some researchers use X+ and X- to indicate the treatment and control respectively)

Groups

• Each group in a design is given its own line in the design structure

• If the design notation has three lines, there are three groups in the design

Assignment to Group

• Assignment to group is designated by a letter at the beginning of each line (i.e., group) that describes how the group was assigned. The major types of assignment are:

• R = random assignment • N = nonequivalent groups • C = assignment by cutoff

Time

• Time moves from left to right. Elements that are listed on the left occur before elements that are listed on the right.

Babbie 2005

Diagram of Classical Experimental Design

Types of Designs

Types of Designs

The Nonequivalent Groups Design

• The term means that assignment to group was not random

• In other words, the researcher did not control the assignment to groups through the mechanism of random assignment.

• As a result, the groups may be different prior to the study

The Nonequivalent Groups Design

• We most often use intact groups that we think are similar as the treatment and control groups

• In education - we might pick two comparable classrooms or schools

• In community-based research, we might use two similar communities

• We try to select groups that are as similar as possible, but we can never be sure the groups are comparable

• Because it's often likely that the groups are not equivalent, this design was named the nonequivalent groups design to remind us.

Posttest only nonexperimental design

• You might use this design if you want to study the effects of a natural disaster like a flood or tornado and you want to do so by interviewing survivors

• Notice that in this design, you don't have a comparison group (e.g., interview in a town down the road that didn't have the tornado to see what differences the tornado caused) and you don't have multiple waves of measurement (e.g., a pre-tornado level of how people in the ravaged town were doing before the disaster)

Example

• Effect of the decision to conduct crackdown on drinking and driving by a local police force (planned intervention)

Posttest only nonexperimental design

• Does it make sense to do the non-experimental study?

• Of course! You could gain lots of valuable information by well-conducted post-disaster interviews

• But you may have a hard time establishing which of the things you observed are due to the disaster rather than to other factors like the peculiarities of the town or pre-disaster characteristics.

Types of Designs

• A randomized experiment generally is the strongest of the three designs when your interest is in establishing a cause-effect relationship

• A non-experiment is generally the weakest in this respect

• The simplest form of non-experiment is a one-shot survey design that consists of nothing but a single observation O

• This is probably one of the most common forms of research and, for some research questions -- especially descriptive ones -- is clearly a strong design

Expanding Across Time

We can add to the basic design by including additional observations either before or after the program The addition of such pretests provides a "baseline" which, for instance, helps to assess the potential of a maturation or testing threat. If a change occurs between the first and second pre-program measures, it is reasonable to expect that similar change might be seen between the second pretest and the posttest even in the absence of the program.

Expanding across time

Similarly, additional postprogram measures could be added

This would be useful for determining whether an immediate program effect decays over time, or whether there is a lag in time between the initiation of the program and the occurrence of an effect

• We might also add and remove the program over time

• This design is frequently used in clinical psychology and psychiatry. The design is particularly strong against a history threat. When the program is repeated it is less likely that unique historical events can be responsible for replicated outcome patterns.

Expanding Across Groups • Often, it will be to our advantage to add

additional groups to a design in order to rule out specific threats to validity

• If this design were implemented within a single institution where members of the two groups were in contact with each other one might expect that intergroup communication, group rivalry, or demoralization of a group which gets denied a desirable treatment

Expanding Across Groups

• In such a case, one might add an additional nonequivalent group from a similar institution which consists of persons unaware of the original two groups:

The Solomon Four-Group Design

Note that two of the groups receive the treatment and two do not. Further, two of the groups receive a pretest and two do not. Within each treatment condition we have a group that is pretested and one that is not. By explicitly including testing as a factor in the design, we are able to assess experimentally whether a testing threat took place.

Switching Replications Design

• This is a two group design with three waves of measurement. The implementation of the treatment is repeated or replicated.

• And in the repetition of the treatment, the two groups switch roles -- the original control group becomes the treatment group in phase 2 while the original treatment acts as the control.

• By the end of the study all participants have received the treatment.

Switching Replications Design• This design works especially well in schools that are on

a semester system.• All students are pretested at the beginning of the

school year. • During the first semester, Group 1 receives the

treatment and during the second semester Group 2 gets it.

• The design also enhances organizational efficiency in resource allocation

• Schools only need to allocate enough resources to give the program to half of the students at a time.

Strengths Weaknesses

• The only method that allows us to test the causal relationships between variables

• Random assignment of subjects to experimental and control groups allows us to test our hypotheses

• In real life, only rarely one variable actually a cause of another one

• Difficult to test very complex hypotheses (difficult to manipulate and control more than one or two variables)

• Ethical issues