thursday validity, reliability and generalizabilityupdated
TRANSCRIPT
-
8/14/2019 Thursday Validity, Reliability and Generalizabilityupdated
1/16
Validity, Reliability andValidity, Reliability and
GeneralizabilityGeneralizability
-
8/14/2019 Thursday Validity, Reliability and Generalizabilityupdated
2/16
ValidityValidity
HOW MIGHT YOU BE WRONG?HOW MIGHT YOU BE WRONG?
Internal ValidityInternal Validity
External ValidityExternal ValidityConstruct ValidityConstruct Validity
-
8/14/2019 Thursday Validity, Reliability and Generalizabilityupdated
3/16
ValidityValidity
Are your findings really about what theyAre your findings really about what they
appear to be about or did something elseappear to be about or did something else
cause them?cause them?
-
8/14/2019 Thursday Validity, Reliability and Generalizabilityupdated
4/16
Internal ValidityInternal Validity
Did your treatment actually cause theDid your treatment actually cause the
outcomes?outcomes?
Did anything else potentially cause theDid anything else potentially cause the
outcome instead of your treatment?outcome instead of your treatment?
-
8/14/2019 Thursday Validity, Reliability and Generalizabilityupdated
5/16
INTERNAL VALIDITYINTERNAL VALIDITY
Internal validity addresses the "true" causes of theInternal validity
addresses the "true" causes of theoutcomes that you observed in your study.outcomes that y
ou observed in your study. StrongStronginternal validity means that you not only haveinternal validity means that you not only havereliable measures of your independent andreliable measures of your independent and
dependent variables BUT a strong justification thatdependent variables BUT a strong justification thatyou are able to rule out extraneous variables, oryou are able to rule out extraneous variables, oralternative, often unanticipated, causes for youralternative, often unanticipated, causes for yourfindings.findings. Thus strong internal validity refers to theThus strong
internal validity refers to theunambiguous assignment of causes to effects.unambig
uous assignment of causes to effects.
Internal validity is aboutInternal validity is about causal controlcausal control..
-
8/14/2019 Thursday Validity, Reliability and Generalizabilityupdated
6/16
EXTERNAL VALIDITYEXTERNAL VALIDITY
External validity addresses the abilityExternal validity
addresses the abilityto generalize your study to otherto g
eneralize your study to other
people and other situations.
people and other situations. To haveTo havestrong external validity (ideally), youstrong external validity (ideally), youneed a probability sample of subjectsneed a probability sample of subjects
or respondents drawn using "chanceor respondents drawn using "chancemethods" from a clearly definedmethods" from a clearly definedpopulationpopulation
-
8/14/2019 Thursday Validity, Reliability and Generalizabilityupdated
7/16
External ValidityExternal Validity
When you have strong external validity,When you have strong external validity,
you can generalize to other people andyou can generalize to other people and
situations with confidence.situations with confidence.
External validity refers to the population weExternal validity refers to the population we
use and the setting. If I do a study with fifthuse and the setting. If I do a study with fifth
graders, then the results can be applied to agraders, then the results can be applied to a
similar set of fifth graders.similar set of fifth graders.
-
8/14/2019 Thursday Validity, Reliability and Generalizabilityupdated
8/16
Construct ValidityConstruct Validity
Does your test measure what you think itDoes your test measure what you think it
really measures. What are you measuringreally measures. What are you measuring
or observing?or observing?
Keep it simple and account for it.Keep it simple and account for it.
-
8/14/2019 Thursday Validity, Reliability and Generalizabilityupdated
9/16
CONSTRUCT VALIDITYCONSTRUCT VALIDITY
Construct validity is about the correspondenceConstruct validity is about the correspondencebetween your concepts (constructs) and the actualbetween your concepts (constructs) and the actualmeasurements that you use.measurements that you use.A measure with highA measure with high
concept validity accurately reflects the abstract conceptconcept validity accurately reflects the abstract conceptthat you are trying to study. Since we can only knowthat you are trying to study. Since we can only knowabout our concepts through the concrete measures thatabout our concepts through the concrete measures thatwe use, you can see that construct validity is extremelywe use, you can see that construct validity is extremelyimportant. It also becomes clear why it is so important toimportant. It also becomes clear why it is so important to
have very clear conceptual definitions of our variables.have very clear conceptual definitions of our variables.
-
8/14/2019 Thursday Validity, Reliability and Generalizabilityupdated
10/16
Construct ValidityConstruct Validity
Construct validity is often establishedConstruct validity is often established
through the use ofthrough the use ofa multi-trait, multi-methoda multi-trait, multi-method
matrix.matrix.At least two constructs are measured.At least two constructs are measured.
Each construct is measured at least two differentEach construct is measured at least two differentways, and the type of measures is repeatedways, and the type of measures is repeated
across constructs. For example, each constructacross constructs. For example, each construct
first might be measured using a questionnaire,first might be measured using a questionnaire,
then each construct would be measured using athen each construct would be measured using asimilar set of behavioral observation categories.similar set of behavioral observation categories.
-
8/14/2019 Thursday Validity, Reliability and Generalizabilityupdated
11/16
RELIABILITYRELIABILITY
In order to make any kind of causalIn order to make any kind of causal
assessments in your research situation,assessments in your research situation,
you must first haveyou must first have reliablereliable measures,measures,
i.e., stable and/or repeatable measures. Ifi.e., stable and/or repeatable measures. If
the random error variation in yourthe random error variation in your
measurements is so large that there ismeasurements is so large that there is
almost no stability in your measures, youalmost no stability in your measures, youcan't explain anything!can't explain anything!
-
8/14/2019 Thursday Validity, Reliability and Generalizabilityupdated
12/16
ReliabilityReliability
Researcher Bias-How might you haveResearcher Bias-How might you have
effected the study? Account for it!effected the study? Account for it!
-
8/14/2019 Thursday Validity, Reliability and Generalizabilityupdated
13/16
ReliabilityReliability
Reliability is required to make statements aboutReliability is required to make statements aboutvalidity. However, reliable measures could bevalidity. However, reliable measures could bebiasedbiased and hence "untrue" measures of aand hence "untrue" measures of aphenomenon)phenomenon) or confoundedor confounded with other factorswith other factorssuch as acquiescence response set. Picture asuch as acquiescence response set. Picture ascale that always weighs five pounds too light.scale that always weighs five pounds too light.The results are reliable, but inaccurate orThe results are reliable, but inaccurate orbiased. Or, picture an intelligence test on whichbiased. Or, picture an intelligence test on which
women or people of color always score lowerwomen or people of color always score lower((even if this doesn't occur on other testeven if this doesn't occur on other tests). Again,s). Again,the measure may be reliable but biased.the measure may be reliable but biased.
-
8/14/2019 Thursday Validity, Reliability and Generalizabilityupdated
14/16
ReliabilityReliability
Identifying and controlling threats toIdentifying and controlling threats to
reliability and validity in qualitativereliability and validity in qualitative
research is critical. Asking the question ofresearch is critical. Asking the question of
whether another researcher, going into thewhether another researcher, going into the
field and asking the same interviewfield and asking the same interview
questions would get the same answers isquestions would get the same answers is
the overarching premise of reliability.the overarching premise of reliability.
-
8/14/2019 Thursday Validity, Reliability and Generalizabilityupdated
15/16
GeneralizabilityGeneralizability
Can you apply your findings across similarCan you apply your findings across similar
populations? If so how and whatpopulations? If so how and what
boundaries exist?boundaries exist?
-
8/14/2019 Thursday Validity, Reliability and Generalizabilityupdated
16/16
Generalizability (Be Honest)Generalizability (Be Honest)
The study is generalizable to other school headsThe study is generalizable to other school headsfrom international schools in Asia. However, onefrom international schools in Asia. However, onelimit to this study was that the study will belimit to this study was that the study will beconducted with schools only in Asia. Therefore,conducted with schools only in Asia. Therefore,
the findings will not apply universally. A secondthe findings will not apply universally. A secondlimit to the study is that the entire study waslimit to the study is that the entire study wasfunded by the researcher, with limited funds forfunded by the researcher, with limited funds forair travel and miscellaneous expenses.air travel and miscellaneous expenses.Therefore, schools were selected within a fourTherefore, schools were selected within a four
hour flight range of the researchers home town.hour flight range of the researchers home town.Telephone interviews were deemed less reliableTelephone interviews were deemed less reliableas they did allow for interpretation of bodyas they did allow for interpretation of bodylanguage and visual clues.language and visual clues.