Experimental Validity

Granada, Sarah Jane P.

Experimental Validity

• If a study is valid then it truly represents what it was intended to represent.  Experimental validity refers to the manner in which variables that influence both the results of the research and the generalizability to the population at large.  It is broken down into two groups: (1) Internal Validity and (2) External Validity.

• Internal Validity.  Internal validity refers to a study’s ability to determine if a causal relationship exists between one or more independent variables and one or more dependent variables.  

•   Researchers must be aware of aspects that may reduce the internal validity of a study and do whatever they can to control for these threats.  

• These threats, if left ignored, can reduce validity to the point that any results are meaningless rendering the entire study invalid.  There are eight major threats.

• History.  History refers to any event outside of the research study that can alter or effect subjects’ performance.  

• Maturation.  While not a major concern in very short studies such as a survey study, maturation can play a major role in longer-term studies.

• Testing.  People tend to perform better at any activity the more they are exposed to that activity. 

• Statistical Regression.  Statistical regression, or regression to the mean, is a concern especially in studies with extreme scores. 

• Instrumentation.   If the measurement device(s) used in your study changes during the course of the study, changes in scores may be related to the instrument rather than the independent variable. Selection. 

•  Selection refers to the manner in which subjects are selected to participate in a study and the manner in which they are assigned to groups. 

• Experimenter Bias.  We engage in research in order to learn something new or to support a belief or theory.  Therefore, we as researchers may be biased toward the results we want.

• Mortality.  Mortality, or subject dropout, is always a concern to researchers.

• External Validity.  External validity refers to the generalizability of a study.  In other words, can we be reasonable sure that the results of our study consisting of a sample of the population truly represents the entire population?  Threats to external validity can result in significant results within a sample group but an inability for this to be generalized to the population at large.  

• Demand Characteristics.  Subjects are often provided with cues to the anticipated results of a study.  When asked a series of questions about depression, for instance, subjects may become wise to the hypothesis that certain treatments work better in treating mental illness.  When subjects become wise to anticipated results (often called a placebo effect), they can begin to exhibit performance that they believe is expected of them.  Making sure that subjects are not aware of anticipated outcomes (referred to as a blind study) reduces the possibility of this threat.

• Hawthorne Effects.  Similar to a placebo, research has found that the mere presence of others watching your performance causes a change in your performance.  If this change is significant, can we be reasonably sure that it will also occur when no one is watching?  Addressing this issue can be tricky but employing a control group to measure the Hawthorne effect of those not receiving any treatment can be very helpful.  In this sense, the control groups is also being observed and will exhibit similar changes in their behavior as the experimental group therefore negating the Hawthorne effect.

• Order Effects (or Carryover Effects).  Order effects refer to the order in which treatment is administered and can be a major threat to external validity if multiple treatments are used.  If subjects are given medication for two months, therapy for another two months, and no treatment for another two months, it would be possible, and even likely, that the level of depression would be least after the final no treatment phase.  Does this mean that no treatment is better than the other two treatments?  It likely means that the benefits of the first two treatments have carried over to the last phase, artificially elevating the no treatment success rates.

• Treatment Interaction Effects.  The term interaction refers to the fact that treatment can affect people differently depending on the subject’s characteristics.