Sampling & Experimental Control

Psych 231: Research Methods in Psychology

Sampling

Population

Everybody that the research is targeted to be about

The subset of the population that actually participates in the research

Sample

Errors in measurement Sampling error

Why do we do we use sampling methods? Typically don’t have the

resources to test everybody, so we test a subset

Sampling

Sample

Inferential statistics used to generalize back

Sampling to make data collection manageable

Population

Allows us to quantify the Sampling error

Sampling

Goals of “good” sampling:– Maximize Representativeness:

– To what extent do the characteristics of those in the sample reflect those in the population

– Reduce Bias:– A systematic difference between those in

the sample and those in the population

Key tool: Random selection

Sampling Methods

Probability sampling Simple random sampling Systematic sampling Stratified sampling

Non-probability sampling Convenience sampling Quota sampling

Have some element of random selection

Susceptible to biased selection

Simple random sampling

Every individual has a equal and independent chance of being selected from the population

Systematic sampling

Selecting every nth person

Cluster sampling

Step 1: Identify groups (clusters) Step 2: randomly select from each group

Convenience sampling

Use the participants who are easy to get

Quota sampling

Step 1: identify the specific subgroups Step 2: take from each group until desired number of

individuals

Extraneous Variables

Control variables Holding things constant - Controls for excessive random

variability Random variables – may freely vary, to spread variability

equally across all experimental conditions Randomization

• A procedure that assures that each level of an extraneous variable has an equal chance of occurring in all conditions of observation.

Confound variables Variables that haven’t been accounted for (manipulated,

measured, randomized, controlled) that can impact changes in the dependent variable(s)

Co-varys with both the dependent AND an independent variable

Colors and words

Divide into two groups: men women

Instructions: Read aloud the COLOR that the words are presented in. When done raise your hand.

Women first. Men please close your eyes. Okay ready?

BlueGreenRedPurpleYellowGreenPurpleBlueRedYellowBlueRedGreen

List 1

Okay, now it is the men’s turn. Remember the instructions: Read aloud the

COLOR that the words are presented in. When done raise your hand.

Okay ready?

BlueGreenRedPurpleYellowGreenPurpleBlueRedYellowBlueRedGreen

List 2

Our results

So why the difference between the results for men versus women?

Is this support for a theory that proposes: “Women are good color identifiers, men are not” Why or why not? Let’s look at the two lists.

BlueGreenRedPurpleYellowGreenPurpleBlueRedYellowBlueRedGreen

List 2Men

BlueGreenRedPurpleYellowGreenPurpleBlueRedYellowBlueRedGreen

List 1Women

Matched Mis-Matched

What resulted in the perfomance difference? Our manipulated independent variable

(men vs. women) The other variable match/mis-match?

Because the two variables are perfectly correlated we can’t tell

This is the problem with confounds

BlueGreenRedPurpleYellowGreenPurpleBlueRedYellowBlueRedGreen

BlueGreen

RedPurple

YellowGreenPurple

BlueRed

YellowBlueRed

Green

IVDV

Confound

Co-vary together

Experimental Control

Our goal: To test the possibility of a relationship

between the variability in our IV and how that affects the variability of our DV.

• Control is used to minimize excessive variability.• To reduce the potential of confounds.

Sources of variability (noise)

Nonrandom (NR) Variability - systematic variation A. (NRexp) manipulated independent variables (IV)

i. our hypothesis is that changes in the IV will result in changes in the DV

Sources of Total (T) Variability:

T = NRexp + NRother + R

B. (NRother) extraneous variables (EV) which covary with IV i. Condfounds

Sources of variability (noise)

Non-systematic variationC. Random (R) Variability

• Imprecision in manipulation (IV) and/or measurement (DV) • Randomly varying extraneous variables (EV)

Sources of Total (T) Variability:

T = NRexp + NRother + R

Sources of variability (noise)

Sources of Total (T) Variability:

T = NRexp + NRother + R Goal: to reduce R and NRother so that we can detect NRexp.

That is, so we can see the changes in the DV that are due to the changes in the independent variable(s).

Weight analogy

Imagine the different sources of variability as weights

RNRexp

NRother

RNRother

Treatment group control group

The effect of the treatment

Weight analogy

If NRother and R are large relative to NRexp then detecting a difference may be difficult

RNRexp

NRother

RNRother

Difference Detector

Weight analogy

But if we reduce the size of NRother and R relative to NRexp then detecting gets easier

RNRother

RNRexpNR

other

Difference Detector

Things making detection difficult

Potential Problems Confounding Excessive random variability Dissimulation

Difference Detector

Potential Problems

Confounding If an EV co-varies with IV, then NRother component

of data will be "significantly" large, and may lead to misattribution of effect to IV

IVDV

EV

Co-vary together

Confounding

RNRexp

NRother

Hard to detect the effect of NRexp because the effect looks like it could be from NRexp but is really (mostly) due to the NRother

R

Difference Detector

Potential Problems

Excessive random variability If experimental control procedures are not applied

• Then R component of data will be excessively large, and may make NRexp undetectable

So try to minimize this by using good measures of DV, good manipulations of IV, etc.

Excessive random variability

RNRexp

NRother

NRother

R

Hard to detect the effect of NRexp

Difference Detector

Potential Problems

Potential problem caused by experimental control Dissimulation

• If EV which interacts with IV is held constant, then effect of IV is known only for that level of EV, and may lead to overgeneralization of IV effect

This is a potential problem that affects the external validity

Controlling Variability

Methods of Experimental Control Comparison Production Constancy/Randomization

Methods of Controlling Variability

Comparison An experiment always makes a comparison, so it must have at

least two groups• Sometimes there are control groups

• This is typically the absence of the treatment• Without control groups if is harder to see what is really

happening in the experiment• It is easier to be swayed by plausibility or inappropriate

comparisons

• Sometimes there are just a range of values of the IV

Methods of Controlling Variability

Production The experimenter selects the specific values of the

Independent Variables • Need to do this carefully

• Suppose that you don’t find a difference in the DV across your different groups

• Is this because the IV and DV aren’t related?• Or is it because your levels of IV weren’t different enough

Methods of Controlling Variability

Constancy/Randomization If there is a variable that may be related to the DV that you

can’t (or don’t want to) manipulate• Control variable: hold it constant

• Random variable: let it vary randomly across all of the experimental conditions

But beware confounds, variables that are related to both the IV and DV but aren’t controlled

Experimental designs

So far we’ve covered a lot of the about details experiments generally

Now let’s consider some specific experimental designs. Some bad designs Some good designs

• 1 Factor, two levels

• 1 Factor, multi-levels

• Factorial (more than 1 factor)

• Between & within factors

Poorly designed experiments

Example: Does standing close to somebody cause them to move? So you stand closely to people and see how long before

they move

Problem: no control group to establish the comparison group (this design is sometimes called “one-shot case study design”)

Poorly designed experiments

Does a relaxation program decrease the urge to smoke? One group pretest-posttest design Pretest desire level – give relaxation program – posttest desire

to smoke

Poorly designed experiments

One group pretest-posttest design

Problems include: history, maturation, testing, instrument decay, statistical regression, and more

participants Pre-test Training group

Post-testMeasure

Independent Variable

Dependent Variable

Dependent Variable

Poorly designed experiments

Example: Smoking example again, but with two groups. The subjects get to choose which group (relaxation or no program) to be in

Non-equivalent control groups Problem: selection bias for the two groups, need to do random

assignment to groups

Poorly designed experiments

Non-equivalent control groups

participants

Traininggroup

No training (Control) group

Measure

Measure

Self Assignment

Independent Variable

Dependent Variable

“Well designed” experiments

Post-test only designs

participants

Experimentalgroup

Controlgroup

Measure

Measure

Random Assignment

Independent Variable

Dependent Variable

“Well designed” experiments

Pretest-posttest design

participants

Experimentalgroup

Controlgroup

Measure

Measure

Random Assignment

Independent Variable

Dependent Variable

Measure

Measure

Dependent Variable