Connecting the dots…

• Our 1st exposure to research studies – Experimental and confounding variables– Covered between/within experimental

research designs– Implementing Tx/IV(s) within the same

subject/group or between subject/group

• Moving the topic covariance and describing the split-plot design and the nested design.

Examples of Case Scenario Psychological Research

• Description of social behavior– Are people who grow up in warm climates different

from those in cold climates?

• Establish a relationship between cause & effect– Does heat cause higher amounts of aggression?

• Develop theories about why people behave the way that they do– We dislike Democrats to feel better about ourselves

• Application– Creating effective therapeutic treatments, more

successful negotiation tactics, and greater understanding amongst groups of people

Review of Advanced Research

The “final push”

The Process of Doing Research

• First, select a topic– Good theory:

• Has predictive power• Is simple & straightforward

• Then, search the literature– Find out what others have done

that may be applicable to your area of interest

The Process of Doing Research

• Next, formulate hypotheses– Hypothesis: specific statement of

expectation derived from theory• State the relationship between two

variables

– Variable: can be any event, characteristic, condition, or behavior

Let’s take a closer look . . .at variables

• Dependent variable (outcome variable)– Dependent on the influence of other factor(s)– How do we operationalize?

• Independent variable (predictor variable)– Factor(s) that change the outcome variable– How do we operationalize & manipulate?– Control group

The Process of Doing Research

• Then pick your research method– Experimental vs. correlational (DesignDesign)– Field vs. laboratory (SettingSetting)

• Finally, collect & analyze your

data

Let’s take a closer look . . . at research methods

• Experimental vs. correlational designs– Correlational: observe the relationship between

two variables• Describe patterns of behavior

– Types include• Naturalistic observation• Case studies• Surveys

Correlational research

• Advantages– Sometimes manipulation of variables is

impossible or unethical– Efficient – look at lots of data

• Disadvantages– CANNOT DETERMINE CAUSATION– Could be a lurking variable

The Goal of Research

To seek the truth.

Experimentation is one mechanism for identifying

causation, which is a step toward understanding how one set of

factors influence another set of factors

Key Characteristics of Experimental Designs

• Random assignment

• Control over extraneous variables

• Manipulation of the treatment conditions

• Outcome Measures

• Group Comparisons

• Threats to validity

Experimental Research

• Researcher manipulates one variable (IV) to see effect on other variable (DV)– Try to hold everything else constant

• True experiments have– Random sampling: selecting subjects

randomly from population– Random assignment: chance assignment to

condition

Types of Experimental Designs

• Pre-experimental designs: One group designs and designs that compare pre-existing groups

• Quasi-experimental designs: Experiments that have treatments, outcome measures, and experimental conditions but that do not use random selection and assignment to treatment conditions.

• True experimental designs: Experiments that have treatments, outcome measures, and experimental conditions and use random selection and assignment to treatment conditions. This is the strongest set of designs in terms of internal and external validity.

Pre-Experimental Designs

One-Shot Case Study: A single group is studied once after some intervention/treatment that is presumed to cause change. – For example, a training program is

implemented and participants are given a posttest at the conclusion of the training.

X O

Pre-Experimental Designs

One-Group Pretest-Posttest Design: One group, not randomly selected nor randomly assigned, is given a pretest, followed by a treatment/intervention, and finally a posttest. There is no comparison group. Generally done with intact groups.– For example, a classroom teacher gives her students

a pretest then implements an instructional strategy followed by a posttest.

O1 X O2

Pre-Experimental Designs

The Static-Group Comparison: One group which has experienced a treatment/intervention (X) is compared to another group that has not had the intervention. The groups are not randomly selected nor randomly assigned and are generally pre-existing groups. There is no pre-observation/pretest. – For example, comparison of GRE scores for students who

attended a rural high school versus those who attended an urban high school.

X1 O X2 O

Experimental vs. Quasi-experimental Research Designs

• Experimental research design: The researcher has control over the experiment in terms of sample selection, treatment, environment, etc.

• Experimental designs are typical in psychology, medicine, education, etc.

• Quasi-experiments: The researcher does not have control over the experiment, rather the experiment occurs in a “natural” setting.

• Quasi-experimental design are typical in economics, sociology, public administration, urban planning, political sciences, etc.

Advanced Research Methods

PS504-02

Kevin Wickes

Unit 7 Seminar

(Covariance and Nested/Split Plot)

Overview• Covariate?

– How can measuring this type of variable help to control for potential confounds in a study?

– Can you think of any covariates that you would want to measure in your hypothetical research study? Why?

– How would this help you obtain a clearer picture of your results?

• Split-plot design and the nested design– How are both of these approaches a unique combination of

both a between subjects and a within subjects design?

– Since nested designs use pre-existing groups of subjects, why isn’t this approach considered quasi-experimental?

Control over extraneous variables

• Extraneous Variables: influences in participant selection, procedures, statistics, or the design likely to affect the outcome and provide an alternative explanation results than what was expected.

• Random assignment helps to control for extraneous variables

• Done before the experiment begins

Control over extraneous variables

Other control procedures– pretest/posttest– covariates– matching participants– selecting homogenous samples– using blocking variables

Pre-Test and Post-Tests

Time 1 Time 2

Pre-Test Post-Test

Intervention

Controlling for Covariates

DependentVariable

IndependentVariable

No Covariates

Covariate Introduced

Covariate:Parents Who

Smoke

VarianceRemovedVariance

DependentVariable:Rates of Smoking

IndependentVariable: Typeof Instruction

Matching Process Based on Gender

ExperimentalGroup

ControlGroup

JohnJimJamesJoshJacksonJaneJohannaJulieJeanJeb

Manipulation of the treatment conditions

• Identify a treatment variable

• Identify the conditions or levels of the treatment variable

• Manipulate the treatment conditions

The Experimental Manipulation of a Treatment Group

Independent Variables1. Age (can’t manipulate)

2. Gender (can’t manipulate)

3. Types of Instruction (can manipulate)

a. Lecture (control)

b. Lecture + Hazard Instruction (Comparison)

c. Lecture + Hazard Instruction + slides of damaged lungs (experiment)

Dependent Variable

Frequency of

Smoking

What is ANCOVA?

• Analysis of Covariance

• Extension of ANOVA, using ‘regression’ principles

• Assess effect of – one variable (IV) on – another variable (DV) – after controlling for a third variable (CV)

Combining Experimental and Correlational Designs

• Covariates in experimental designs– Measure your subjects on a covariate—a variable that you believe

may be correlated with your dependent variable– If left unmeasured these covariates add error variance and might

obscure significant effects– Measuring the covariate allows you to use correlational statistical

techniques in your analysis (e.g., Analysis of Covariance or ANCOVA) to “subtract out” the error variance associated with the covariate, thereby increasing the statistical power of your experiment

– Example: measuring IQ in a learning experiment

Why use ANCOVA?

• Reduces variance associated with covariate (CV) from the DV error (unexplained variance) term

• Increases power of F-test

• May not be able to achieve experimental over a variable (e.g., randomisation), but can measure it and statistically control for its effect.

Why use ANCOVA?

• Adjusts group means to what they would have been if all P’s had scored identically on the CV.

• The differences between P’s on the CV are removed, allowing focus on remaining variation in the DV due to the IV.

• Make sure hypothesis (hypotheses) is/are clear.

ANCOVA Example

• Does Teaching Method affect Academic Achievement after controlling for motivation?

• IV = teaching method• DV = academic achievement• CV = motivation• Experimental design - assume students

randomly allocated to different teaching methods.

ANCOVA example 1

AcademicAchievement

(DV)

TeachingMethod(IV)

Motivation(CV)

ANCOVA example 1

AcademicAchievement

TeachingMethod

Motivation

Summary of ANCOVA• Use ANCOVA in survey research when

you can’t randomly allocate participants to conditionse.g., quasi-experiment, or control for extraneous variables.

• ANCOVA allows us to statistically control for one or more covariates.

Summary of ANCOVA• We can use ANCOVA in survey research

when can’t randomly allocate participants to conditions e.g., quasi-experiment, or control for extraneous variables.

• ANCOVA allows us to statistically control for one or more covariates.

Summary of ANCOVA

• Decide which variable is IV, DV and CV.

• Check Assumptions:– normality– homogeneity of variance (Levene’s test)– Linearity between CV & DV (scatterplot)– homogeneity of regression (scatterplot –

compares slopes of regression lines)

• Results – does IV effect DV after controlling for the effect of the CV?

Common types of ANOVA research designs

Nested

Split-plot

NESTED DESIGNS

Definition

• In certain multifactor experiments, the levels of one factor are similar but not identical for different levels of another factor (is unique to that particular factor).

Aim

• Nested experiments are commonly used to identify the important sources of variation in a system.

• Such sources of variation if not well addressed, might make it impossible to guarantee some level of precision.

Nested design

In this example, Cognitive Behavioral Therapy (CBT) type is “nested within” Personality Disorders (PD).

The nested factor is always random

No CBT CBT1 - DBT CBT2 - CT

PD A PD B PD C PD D PD E PD F

Personal GROWTH

Maximize effective behaviors and Minimize dysfunctional behaviors

No CBT CBT1 - DBT CBT2 - CT

PD A PD B PD C PD D PD E PD F

Variance: Subgroup within a group

Variance: Among all subgroups

Grand mean

Variance: Group

Personal GROWTH

Maximize effective behaviors and Minimize dysfunctional behaviors

Explanation

• a nested design with factor A “nested within” with factor B.

• In other words, A is subgroup (Personality Disorder), B (CBT) is group.

SPLIT-PLOT DESIGNS

(Mixed)

Definition

In some multifactor designs involving randomized blocks, we may be unable to completely randomize the order of the runs within the block. This often results in a generalization of the randomized block design called split-plot design.

Situations leading to Split-plot

• Some of the factors of interest may be 'hard to vary' while the remaining factors are easy to vary. As a result, the order in which the treatment combinations for the experiment are run is determined by the ordering of these 'hard-to-vary' factors

• Experimental units are processed together as a batch for one or more of the factors in a particular treatment combination

• Experimental units are processed individually, one right after the other, for the same treatment combination without resetting the factor settings for that treatment combination.

Definition of Mixed Models by their component effects

1. Mixed Models contain both fixed and random effects

2. Fixed Effects: factors for which the only levels under consideration are contained in the coding of those effects

3. Random Effects: Factors for which the levels contained in the coding of those factors are a random sample of the total number of levels in the population for that factor.

Examples of Fixed and Random Effects

1. Fixed effect:

2. Sex where both male and female genders are included in the factor, sex.

3. Agegroup: Minor and Adult are both included in the factor of agegroup

4. Random effect: 1. Subject: the sample is a random sample of

the target population

Classification of effects

1. There are main effects: Linear Explanatory Factors

2. There are interaction effects: Joint effects over and above the component main effects.

Conceptualizing the Design• This is a very popular design because you

are combining the benefits of each design

• Requires that you have one between groups IV and one within subjects IV

• Often called “Split-plot” designs, which comes from agriculture

• In the simplest 2 x 2 design you would have

Conceptualizing the Design• In the simplest 2 x 2

design you would have subjects randomly assigned to one of two groups, but each group would experience 2 conditions (measurements)

GRE - before GRE - afterS1 S1

S2 S2

S3 S3

S4 S4

S5 S5

S6 S6

S7 S7

S8 S8

S9 S9

S10 S10

Kaplan

Princeton

Conceptualizing the Design• Advantages

–First, it allows generalization of the repeated measures over the randomized groups levels

–Second, reduced error (although not as reduced as purely WS) due to the use of repeated measures

• Disadvantages–The addition of each of their respective

complexities

Conceptualizing the Design

• Types of Mixed Designs– Other than the mixture

of any number of BG IVs and any number of WS IVs…

– Pretest Posttest Mixed Design to control for testing effects

Again… from another view

Mixed design is the best of both worlds of w/in and between

RESEARCH DESIGN

Mixed Designs A mixed ("split-plot") design combines between-groups and within-subjects methodologies. – Counterbalanced designs can be considered a type

of mixed design because they permit comparisons both between groups and within subjects.

– A design is also a mixed design when it includes two or more independent variables and at least one variable is a between-groups variable and another is a within-subjects variable.

RESEARCH DESIGN

Example: In the example study, the psychologist would be using a mixed design if therapy approach is treated as a between-groups variable (patients receive only one type of therapy), while phenothiazines is treated as a within-subjects variable (the placebo, low dose, and high dose are administered sequentially to each patient).

RESEARCH DESIGN

Mixed designs are common in research studies that involve measuring the dependent variable over time or across trials.

In this type of study, time or trials is an additional IV and is considered a within-subjects variable because comparisons on the dependent variable will be made within subjects across time or across trials.

RESEARCH DESIGN

Example: In our example study, the psychologist decides to compare the effects of four levels of therapy (family therapy, individual therapy, a combination of the two, and no therapy) by assigning patients to one of the levels and measuring the short- and long-term effects of therapy by administering the BPRS at two-month intervals for 24 months after therapy begins. Because the study includes a between-groups variable (therapy) and a within-subjects variable (time), it is utilizing a mixed design.

Combining Experimental and Correlational Designs

• Quasi-independent variable in experimental designs– “Quasi” means “kind of, but not really”– Similar to including a covariate, except

• measurement of covariate is used to assign Ss to groups

• Covariate is thus treated as an quasi-independent variable

– Quasi-independent variables are referred to as “quasi” because they cannot be manipulated, they are essentially dependent variables (measures) that are treated as independent variables in the experimental design and analysis

Controlling for Threats/Variances/Errors

IV and 1-DV