overview: threats to validity 1) people are different 2) people change 3) process of studying...

Overview: Threats to Validity 1) People are Different 2) People Change 3) Process of Studying People Changes Them 4) Variables with Treatment Change People

1) People are Different:Individual Differences People are different

Can’t determine if outcome is due to IV or to individual differences

Pitfall of one group experiments

Threat to internal validity

1) People are Different:Selection Bias/ Nonresponse Bias Selection

Choosing participants from a nonrepresentative sample Threat to external validity

President Alf Landon?

Nonresponse bias When a substantial proportion of those invited to

participate refuse to do so Threat to external validity

2) People Change History

Events occurring between first and second measurements but not part of the manipulation.

SubjectsSmoking Measure

TrainingProgram

Smoking Measure

DV: Pretest DV: PosttestIndependent

Variable

Pitfall of one group pretest-posttest designThreat to internal validity

2) People Change Maturation People change over time

Grow older wiser, stronger , healthier, more tired, more bored…

Third graders

Math ExamTrainingProgram

Math Exam

Pitfall of one group pretest-posttest designThreat to internal validity

3) People Change Regression Toward The Mean AKA statistical regression Extreme scores (high or

low) change over time (regress to the mean) High scores become lower Low scores become higher

Hex of the Sports Illustrated cover

Pitfall of one group

pretest-posttest design

Does Studying Something Change it? "The more precisely

the POSITION is determined,the less precisely the MOMENTUM is known"

The Hawthorne effect An increase in worker productivity produced by

the obtrusive observation of that process

3) Studying People Changes People Testing Effects Simply taking the pretest changes the participant!

May cue people in on what is being studied Make them more adept at a skill

StudentsScienceExam

ScienceProgram

Science Exam

Pitfall of one group pretest-posttest designThreat to internal validity

3) Studying People Changes People Mortality (Attrition) No, this isn’t that people in your study are dying Participants drop out of the study before completing

it

CouplesRelationship

StatusMarriage

CounselingRelationship

Status

Homogeneous attritionAttrition rates equal across experimental conditionsThreat to external validity

Heterogeneous attritionAttrition rates different across experimental conditionsThreat to internal validity

3) Studying People Changes People Participant Reaction Bias Research participants realize they are being

studied and behave in a way they normally would not Good subject role Participant reactance Evaluation apprehension

Threats to internal validity

Applicant Hiring Experiment

INSTRUCTIONS This study assesses job applicants capacity to handle stress You will be interviewing an unemployed person Your task is to read questions to the job applicant and then give

negative responses After the first eight questions you should read the 'Stress Remarks'

SCENARIO The applicant appears well-qualified and experienced The answers to the first eight questions are intelligent, lucid and to the

point. Stress Remark:

"This job is much too difficult for you according to the test".

Jane Jane remembered the mix of male and female

participants in the waiting area and noted the fact that the researcher was male. She decided that the experimenter had predicted that female subjects will be less likely to make the stress remarks and she so simply ignored them and said "well done you are doing well, aren't you!" in order to help the researcher's project.

Good-Subject Role Participants consciously/unconsciously try to

behave consistent with experimenters hypothesis

Demand characteristics Feature of an experiment that may inform

participants of the purpose of the study

Su Su remembered the mix of male and female

participants in the waiting area and noted the fact that the researcher was male. She also decided that the experimenter has predicted that female subjects will be less likely to make the stress remarks and made a point of not only making the stress remarks, but saying it in an aggressive manner - in order to disabuse the researcher of female stereotypes

Participant Reactance Participants try to disconfirm perceived

hypotheses

John John looked at the highly professional instruction

cards and the expensive video recording equipment and decided this was a test of management skills. So, he attempted to stay calm and read out the questions in a controlled, professional manner in order to show the experimenter that he was made of stern stuff and competent at handling difficult interpersonal situations. However, he was so nervous about doing the task well that he mixed up the order of questions and read them out too quickly.

Evaluation Apprehension Participants try to behave in a way that will

portray them most favorably.

Controlling for Demand Characteristics Keep participants in the dark

Cover stories Schacter and Singer’s emotion study

Unobtrusive observations One-way mirror Hot sauce Radio stations

Ensure anonymity

4) Treatment Changes People Experimenter Bias AKA expectancy effects Experimenter expectations biasing results Two sources of experimenter bias

Unintentionally treat participants differently Experimenters record the behaviors differently

Threat to internal validity The story of Clever Hans

Expectancy Effects Rosenthal and Jacobson (1966)

Students IQ tested Told teachers some students were going to be “intellectual

bloomers” These students actually chosen at random At end of school year their IQ tested again Results

“Ready to bloomers” saw marked improvements Self-fulfilling prophecy

Teacher expectations are communicated to students who in turn respond to those cues by adjusting their behavior.

The result: the original expectation becomes true. 

Solution to Expectancy Effects 1) Practice makes perfect (or nearly so) 2) Run all participants in one fell swoop 3) Go high tech! 4) Ignorance can be bliss

Unaware of hypotheses Unaware of condition

Single blind Double blind

4) Treatment Changes People Confounds Some additional variable varies systematically with

the independent variable

Really, all threats we have discussed the problem was that something else was accompanying the independent variable.