overview: threats to validity 1) people are different 2) people change 3) process of studying...
TRANSCRIPT
Overview: Threats to Validity 1) People are Different 2) People Change 3) Process of Studying People Changes Them 4) Variables with Treatment Change People
1) People are Different:Individual Differences People are different
Can’t determine if outcome is due to IV or to individual differences
Pitfall of one group experiments
Threat to internal validity
1) People are Different:Selection Bias/ Nonresponse Bias Selection
Choosing participants from a nonrepresentative sample Threat to external validity
President Alf Landon?
Nonresponse bias When a substantial proportion of those invited to
participate refuse to do so Threat to external validity
2) People Change History
Events occurring between first and second measurements but not part of the manipulation.
SubjectsSmoking Measure
TrainingProgram
Smoking Measure
DV: Pretest DV: PosttestIndependent
Variable
Pitfall of one group pretest-posttest designThreat to internal validity
2) People Change Maturation People change over time
Grow older wiser, stronger , healthier, more tired, more bored…
Third graders
Math ExamTrainingProgram
Math Exam
Pitfall of one group pretest-posttest designThreat to internal validity
3) People Change Regression Toward The Mean AKA statistical regression Extreme scores (high or
low) change over time (regress to the mean) High scores become lower Low scores become higher
Hex of the Sports Illustrated cover
Pitfall of one group
pretest-posttest design
Does Studying Something Change it? "The more precisely
the POSITION is determined,the less precisely the MOMENTUM is known"
The Hawthorne effect An increase in worker productivity produced by
the obtrusive observation of that process
3) Studying People Changes People Testing Effects Simply taking the pretest changes the participant!
May cue people in on what is being studied Make them more adept at a skill
StudentsScienceExam
ScienceProgram
Science Exam
Pitfall of one group pretest-posttest designThreat to internal validity
3) Studying People Changes People Mortality (Attrition) No, this isn’t that people in your study are dying Participants drop out of the study before completing
it
CouplesRelationship
StatusMarriage
CounselingRelationship
Status
Homogeneous attritionAttrition rates equal across experimental conditionsThreat to external validity
Heterogeneous attritionAttrition rates different across experimental conditionsThreat to internal validity
3) Studying People Changes People Participant Reaction Bias Research participants realize they are being
studied and behave in a way they normally would not Good subject role Participant reactance Evaluation apprehension
Threats to internal validity
Applicant Hiring Experiment
INSTRUCTIONS This study assesses job applicants capacity to handle stress You will be interviewing an unemployed person Your task is to read questions to the job applicant and then give
negative responses After the first eight questions you should read the 'Stress Remarks'
SCENARIO The applicant appears well-qualified and experienced The answers to the first eight questions are intelligent, lucid and to the
point. Stress Remark:
"This job is much too difficult for you according to the test".
Jane Jane remembered the mix of male and female
participants in the waiting area and noted the fact that the researcher was male. She decided that the experimenter had predicted that female subjects will be less likely to make the stress remarks and she so simply ignored them and said "well done you are doing well, aren't you!" in order to help the researcher's project.
Good-Subject Role Participants consciously/unconsciously try to
behave consistent with experimenters hypothesis
Demand characteristics Feature of an experiment that may inform
participants of the purpose of the study
Su Su remembered the mix of male and female
participants in the waiting area and noted the fact that the researcher was male. She also decided that the experimenter has predicted that female subjects will be less likely to make the stress remarks and made a point of not only making the stress remarks, but saying it in an aggressive manner - in order to disabuse the researcher of female stereotypes
Participant Reactance Participants try to disconfirm perceived
hypotheses
John John looked at the highly professional instruction
cards and the expensive video recording equipment and decided this was a test of management skills. So, he attempted to stay calm and read out the questions in a controlled, professional manner in order to show the experimenter that he was made of stern stuff and competent at handling difficult interpersonal situations. However, he was so nervous about doing the task well that he mixed up the order of questions and read them out too quickly.
Evaluation Apprehension Participants try to behave in a way that will
portray them most favorably.
Controlling for Demand Characteristics Keep participants in the dark
Cover stories Schacter and Singer’s emotion study
Unobtrusive observations One-way mirror Hot sauce Radio stations
Ensure anonymity
4) Treatment Changes People Experimenter Bias AKA expectancy effects Experimenter expectations biasing results Two sources of experimenter bias
Unintentionally treat participants differently Experimenters record the behaviors differently
Threat to internal validity The story of Clever Hans
Expectancy Effects Rosenthal and Jacobson (1966)
Students IQ tested Told teachers some students were going to be “intellectual
bloomers” These students actually chosen at random At end of school year their IQ tested again Results
“Ready to bloomers” saw marked improvements Self-fulfilling prophecy
Teacher expectations are communicated to students who in turn respond to those cues by adjusting their behavior.
The result: the original expectation becomes true.
Solution to Expectancy Effects 1) Practice makes perfect (or nearly so) 2) Run all participants in one fell swoop 3) Go high tech! 4) Ignorance can be bliss
Unaware of hypotheses Unaware of condition
Single blind Double blind
4) Treatment Changes People Confounds Some additional variable varies systematically with
the independent variable
Really, all threats we have discussed the problem was that something else was accompanying the independent variable.