Experimental and Single-Subject DesignExperimental and Single-Subject Design

PSY440

May 27, 2008

Definitions:Definitions:

Consider each of the following terms and generate a definition for each:

• Research

• Empirical

• Data

• Experiment

• Qualitative Research

DefinitionsDefinitions

Research

1. Scholarly or scientific investigation or inquiry

2. Close and careful study (American Heritage Dictionary)

3. Systematic (step-by-step)

4. Purposeful (identify, describe, explain, predict)

DefinitionsDefinitions

Empirical:Relying upon or derived from observation or experiment; capable of proof or verification by means of observation or experiment. (American Heritage Dictionary)

Data: Information; esp. information organized for analysis or used as the basis of a decision. (American Heritage Dictionary)

Experiment:A method of testing an hypothesized causal relationship between two variables by manipulating one variable and observing the effect of the manipulation on the second variable.

Overview of Experimental Overview of Experimental DesignDesign

Based on Alan E. Kazdin. (1982). Single-Case Research Designs:

Methods for Clinical and Applied Settings. Chapter IV.

Independent & Dependent VariablesIndependent & Dependent Variables

The independent variable (IV) is the variable that is manipulated in an experiment.

The dependent variable (DV) is the variable that is observed to assess the effect of the manipulation of the IV.

What are some examples of IV’s and DV’s that might be studied experimentally?

Internal and External ValidityInternal and External Validity

Internal validity refers to the extent to which a study is designed in a way that allows a causal relation to be inferred. Threats to internal validity raise questions about alternative explanations for an apparent association between the IV and DV.

External validity refers to the generalizability of the findings beyond the experimental context (e.g. to other persons, settings, assessment devices, etc).

Threats to Internal ValidityThreats to Internal Validity

History

Maturation

Testing

Instrumentation

Statistical Regression

Attrition

Diffusion of Treatment

HistoryHistory

Any event other than the intervention occurring at the time of the experiment that could influence the results.

Example in intervention research: Participant is prescribed medication during the time frame of the psychosocial treatment

Other examples?How can this threat be ruled out or reduced by

the experimental design?

MaturationMaturation

Any change over time that may result from processes within the subject (as opposed to the IV)

Example: Client learns how to read more effectively, so starts behaving better during reading instruction.

Other examples?How can this threat be ruled out or reduced by

the experimental design?

TestingTesting

Any change that may be attributed to effects of repeated assessment

Example: Client gets tired of filling out weekly symptom checklist measures, and just starts circling all 1’s or responding randomly.

Other examples?

How can this threat be ruled out or reduced by the experimental design?

InstrumentationInstrumentation

Any change that takes place in the measuring instrument or assessment procedure over time.

Example: Teacher’s report of number of disruptive incidents drifts over time, holding the student to a higher (or lower) standard than before.

Other examples?How can this threat be ruled out or reduced by

the experimental design?

Statistical RegressionStatistical Regression

Any change from one assessment occasion to another that might be due to a reversion of scores toward the mean.

Example: Clients are selected to be in a depression group based on high scores on a screening measure for depression. When their scores (on average) go down after the intervention, this could be due just to statistical regression (more on this one later in the course : )

How can this threat be ruled out or reduced by the experimental design?

Selection BiasesSelection Biases

Any differences between groups that are due to the differential selection or assignment of subjects to groups.

Example: Teachers volunteer to have their classes get social skills lessons, and their students are compared to students in classrooms where the teachers did not volunteer (both teacher and student effects may be present).

Other examples?How can this threat be ruled out or reduced by the

experimental design?

AttritionAttrition

Any change in overall scores between groups or in a given group over time that may be attributed to the loss of some of the participants.

Example: Clients who drop out of treatment are not included in posttest assessment - may inflate treatment group posttest score.

Other examples?

How can this threat be ruled out or reduced by the experimental design?

Diffusion of TreatmentDiffusion of Treatment

The intervention is inadvertently provided to part or all of the control group, or at the times when the treatment should not be in effect.

Example: Teacher starts token economy before finishing the collection of baseline data.

Other examples?

How can this threat be ruled out or reduced by the experimental design?

Internal validity and single-subject designsInternal validity and single-subject designs

In single-subject research, the participant is compared to him/herself under different conditions (rather than comparing groups).

The participant is his/her own controlSelection biases and attrition are automatically ruled

out by these designsWell designed single-subject experiments can rule

out (or reduce) history, maturation, testing, instrumentation, and statistical regression

Threats to External ValidityThreats to External Validity

Generality Across• Participants• Settings• Response Measures• Times• Behavior Change Agents

Reactive Experimental ArrangementsReactive AssessmentPretest SensitizationMultiple Treatment Interference

Generality Across Subjects, Settings, Responses, & TimesGenerality Across Subjects, Settings, Responses, & Times

Results do not extend to participants, settings, behavioral responses, and times other than those included in the investigation

Example: Couple uses effective communication skills in session, but not at home.

Other examples?How can this threat be ruled out or reduced by

the experimental design?

Generality Across Behavior Change AgentsGenerality Across Behavior Change Agents

Intervention results do not extend to other persons who can administer the intervention (special case of previous item)

Example: Parents are able to use behavior modification techniques successfully but child care providers are not (child responds differently to different person)

Other Examples?How can this kind of threat to external validity be

ruled out or reduced?

Reactive Experimental ArrangementsReactive Experimental Arrangements

Participants may be influenced by their awareness that they are participating in an experiment or special program (demand characteristics)

Example: Social validity of treatment is enhanced by the association with a university

Other examples?How can this threat be ruled out or reduced by

the experimental design?

Reactive AssessmentReactive Assessment

The extent to which participants are aware that their behavior is being assessed and that this awareness may influence how they respond (Special case of reactive arrangements).

Example: Child complies with adult commands when the experimenter is observing, but not at other times.

Other examples?

How can this threat be ruled out or reduced by the experimental design?

Pretest SensitizationPretest Sensitization

Assessing participants before treatment sensitizes them to the intervention that follows, so they are affected differently by the intervention than persons not given the pretest.

Example: Pretest administered before parenting group makes participants pay attention to material more closely and learn more

Other examples?How can this threat be ruled out or reduced by the

experimental design?

Multiple Treatment InterferenceMultiple Treatment Interference

When the same participant(s) are exposed to more than one treatment, the conclusions reached about a particular treatment may be restricted.

Example: Clients are getting pastoral counseling at church and CBT at a mental health center.

Other examples?

How can this threat be ruled out or reduced by the experimental design?

Evaluating a Research DesignEvaluating a Research Design

No study is perfect, but some studies are better than others. One way to evaluate an experimental design is to ask the question:

How well does the design minimize threats to internal and external validity? (In other words, how strong a causal claim does it support, and how generalizable are the results?)

Internal/External Validity Trade-OffInternal/External Validity Trade-Off

Many designs that are well controlled (good internal validity), are more prone to problems with external validity (generality across settings, behavioral responses, interventionists, etc. may be more limited).

Random selection/assignmentRandom selection/assignment

Random: Chance of being selected is equal for each participant and not biased by any systematic factor

Group designs can reduce many of the threats to internal validity by using random assignment of participants to conditions.

They can (in theory) also limit some threats to external validity by using a truly random sample of participants (but how often do you actually see this?)

Single-Subject DesignsSingle-Subject Designs

More modest in generalizability claimsCan be very strong (even perfect) in reducing threats

to internal validityExamples:

Reversal (ABAB)Multiple BaselineChanging CriterionMultiple Treatment

Reversal Design (ABAB)Reversal Design (ABAB)

A=Baseline

B=Treatment

A=Return to Baseline

B=Reintroduce Treatment

Reversal DesignReversal Design

Baseline:

Stable baseline allows stronger causal inference to be drawn

Stability refers to a lack of slope, and low variability (show examples on white board)

ABAB Design: BaselineABAB Design: Baseline

If trend is in reverse direction from expected intervention effect, that’s OK

If trend is not too steep and a very strong effect is expected, that may be OK

For reversal design, relatively low variability makes it easier to draw strong conclusions

Threats to internal validity?Threats to internal validity?

History & maturation (return to baseline rules these out)

Testing, instrumentation, and statistical regression (return to baseline rules these out, but use of reliable measures, trained & monitored observers strengthens the design)

Selection bias & attrition (not an issue with single-subject research!)

Diffusion of treatment (a concern but can be controlled in some cases)

What about generalizability?What about generalizability?

Can’t easily generalize beyond the case.

Need to replicate under different circumstances and with different participants, or follow-up with group design.

Disadvantages to Reversal DesignDisadvantages to Reversal Design

Diffusion of treatment:If the intervention “really worked” the removal of the intervention should not result in a return to baseline behavior!

Ethical concerns:If the behavior change is clinically significant, it may be unethical to remove it, even temporarily!

Multiple Baseline DesignMultiple Baseline Design

Collect baseline data on more than one dependent measure simultaneously

Introduce interventions to target each dependent variable in succession

Multiple Baseline DesignMultiple Baseline Design

Each baseline serves as a control for the other interventions being tested (Each DV is a “mini” AB experiment of the previous baseline)

May be done with one participant when multiple behaviors are targeted, or with multiple participants, each receiving the same intervention in succession.

Example with one participantExample with one participant

Multiple baseline design across behaviors to increase assertive communication:– Increase eye contact– Increase speech volume– Increase number of requests made

Example across participantsExample across participants

• More than one client receives the same intervention, beginning baseline all at one time, and introducing the intervention in succession, so that each participant serves as a control for the others.

• Example from textbook: intervention to increase appropriate mealtime behavior in preschoolers

Example across situations, settings, and timeExample across situations, settings, and time

• Measure target behavior in multiple settings, and introduce the intervention into each setting in succession, while collecting baseline data on all settings

Advantages of Multiple BaselineAdvantages of Multiple Baseline

+ No need to remove intervention

+ Interventions can be added gradually (practical utility in clinical setting)

Disadvantages of Multiple BaselineDisadvantages of Multiple Baseline

- Interdependence of baselines (change in one may result in change in another)

- Inconsistent effects of interventions (some are followed by changes in behavior and others are not)

- Prolonged baselines (ethical & methodological concerns)

Changing Criterion DesignsChanging Criterion Designs

Intervention phase requires increasing levels of performance at specified times.

If the performance level increases as expected over several intervals, this provides good evidence for a causal relationship between intervention and outcome.

Multiple Treatment DesignsMultiple Treatment Designs

More than one treatment is implemented during the same phase, in a manner that allows the effects of the treatments to be compared to each other