11

Lecture 3Lecture 3

Theory and Measurement: Theory and Measurement: Causation, Validity and ReliabilityCausation, Validity and Reliability

22

Assignment 1Assignment 1

Stating a research problem.Stating a research problem.

Providing a short justification for the Providing a short justification for the problem.problem.

Providing a few hypotheses that come Providing a few hypotheses that come from the research problem.from the research problem.

33

Today’s LectureToday’s Lecture

Discussion of CausationDiscussion of Causation

Discussion of ValidityDiscussion of Validity

Discussion of ReliabilityDiscussion of Reliability

Time permitting, Issues in data preparation Time permitting, Issues in data preparation and error-checkingand error-checking

44

Reminder from Lectures 1 and 2: Reminder from Lectures 1 and 2: Causation versus CorrelationCausation versus Correlation

Correlation:Correlation: Non-directional relationship between two variables.Non-directional relationship between two variables. Increase in X associated with Increase in Y, but could Increase in X associated with Increase in Y, but could

also be stated as Increase in Y associated with also be stated as Increase in Y associated with increase in X.increase in X.

Causation:Causation: Directional relationship between at least two Directional relationship between at least two

variables.variables. Increase in X leads to increase in Y, but reverse may Increase in X leads to increase in Y, but reverse may

or may not be true.or may not be true.

55

CausationCausation

Key to causation is Key to causation is directionalitydirectionality

Must be able to establish directionality Must be able to establish directionality either theoretically, methodologically, or either theoretically, methodologically, or both.both.

66

Porter’s (1997) Three Criteria for Porter’s (1997) Three Criteria for CauseCause

Independent variable must precede the Independent variable must precede the dependent variable.dependent variable.

Independent variable must be related to Independent variable must be related to the dependent variable.the dependent variable.

There must be no third variable that could There must be no third variable that could explain why the independent variable is explain why the independent variable is related to the dependent variable.related to the dependent variable.

77

Example: Age and IncomeExample: Age and Income

We know they are correlated, so does age We know they are correlated, so does age cause income to increase?cause income to increase? We know that Income cannot ‘cause’ age.We know that Income cannot ‘cause’ age. They certainly seem related and the direction They certainly seem related and the direction

seems clear… so is it not clear that age seems clear… so is it not clear that age causes income to increase?causes income to increase?

““Third Variable” problem: age related to Third Variable” problem: age related to education, job experience, other factors.education, job experience, other factors.

88

Five Approaches to Quantitative Research Five Approaches to Quantitative Research and Implications for Causalityand Implications for Causality

DescriptiveDescriptive

AssociationalAssociational

ComparativeComparative

Quasi-ExperimentalQuasi-Experimental

Randomized ExperimentalRandomized Experimental

99

Research types and causality: Research types and causality: DescriptiveDescriptive

DescriptiveDescriptive Summarize dataSummarize data Statistics: histograms, means, percentagesStatistics: histograms, means, percentages Cannot show causalityCannot show causality

1010

Research types and causality: Research types and causality: AssociationalAssociational

AssociationalAssociational Only to relate variablesOnly to relate variables Predictions only made to show that a Predictions only made to show that a

relationship existsrelationship exists Statistics: Correlation, Multiple RegressionStatistics: Correlation, Multiple Regression To some degree, To some degree, regression can partially be regression can partially be

used to infer causalityused to infer causality

1111

Research types and causality: Research types and causality: ComparativeComparative

ComparativeComparative Compares two or more groupsCompares two or more groups Looking for difference between groupsLooking for difference between groups Statistics: t-tests, ANOVA (inferential Statistics: t-tests, ANOVA (inferential

statistics)statistics) Not well suited for establishing causeNot well suited for establishing cause b/c it b/c it

does not meet Porter’s (1997) 3does not meet Porter’s (1997) 3rdrd condition condition (extraneous variables)(extraneous variables)

1212

Research types and causality: Research types and causality: Quasi-experimentalQuasi-experimental

Quasi-experimentalQuasi-experimental Compares groupsCompares groups ‘‘quasi-experimental’ b/c it does not have quasi-experimental’ b/c it does not have

random assignment to groupsrandom assignment to groups.. Can examine causalityCan examine causality Statistics: t-tests, ANOVA (inferential Statistics: t-tests, ANOVA (inferential

statistics)statistics)

1313

Research types and causality: Research types and causality: randomized experimentalrandomized experimental

Randomized experimentRandomized experiment To determine causesTo determine causes Compares groupsCompares groups Has random assignment to groupsHas random assignment to groups Best way to determine Best way to determine exact causesexact causes Statistics: t-tests, ANOVA (inferential Statistics: t-tests, ANOVA (inferential

statistics)statistics)

1414

In Summary…In Summary…

You just need to know that the You just need to know that the type of type of researchresearch that you do will affect your ability that you do will affect your ability to describe to describe causalitycausality..

Whenever possible, choose a research Whenever possible, choose a research method that will allow you to have the method that will allow you to have the most explanatory power.most explanatory power.

1515

Validity in Research:Validity in Research:

The ‘quality’ or merit of The ‘quality’ or merit of researchresearch

1616

Validity: Internal, External, Validity: Internal, External, MeasurementMeasurement

Internal Validity: “the approximate validity with which we Internal Validity: “the approximate validity with which we can infer that a relationship is causal” (Cook and can infer that a relationship is causal” (Cook and Campbell 1979).Campbell 1979).

External Validity: “external validity asks the question of External Validity: “external validity asks the question of generalizability: to what populations, settings, treatment generalizability: to what populations, settings, treatment variables, and measurement variables can this effect be variables, and measurement variables can this effect be generalized?” (Campbell and Stanley 1966).generalized?” (Campbell and Stanley 1966).

Measurement Validity: Do our measures capture what Measurement Validity: Do our measures capture what we want them to capture?we want them to capture?

1717

Internal ValidityInternal Validity

Two major threats to internal validity (is Two major threats to internal validity (is our study our study causalcausal)?:)?:

Equivalence of groups on participant Equivalence of groups on participant characteristicscharacteristics

Control of extraneous experience or Control of extraneous experience or environment variables.environment variables.

1818

Equivalence of GroupsEquivalence of Groups

If looking at a specific cause (X affects Y), If looking at a specific cause (X affects Y), then the groups must not vary significantly then the groups must not vary significantly on other key variables.on other key variables.

Example: Looking at the effect of computer Example: Looking at the effect of computer use on intelligence.use on intelligence.

But what if computer users and non-computer But what if computer users and non-computer users differ on employment, age, education, etc?users differ on employment, age, education, etc?

1919

Control of Extraneous Experience Control of Extraneous Experience or Environment Variablesor Environment Variables

If looking at a specific cause (X affects Y), then If looking at a specific cause (X affects Y), then one or more groups cannot receive unknown one or more groups cannot receive unknown stimuli or information that could affect outcome.stimuli or information that could affect outcome. Problem is particularly troublesome if it affects groups Problem is particularly troublesome if it affects groups

differentially.differentially.

Example: Study of two classrooms, one with Example: Study of two classrooms, one with information technology and one without such information technology and one without such technology. technology.

What if one of the classes also has teaching assistants who What if one of the classes also has teaching assistants who help the students?help the students?

2020

Other types of Internal Validity Other types of Internal Validity ProblemsProblems

Statistical RegressionStatistical Regression Because of statistical variation, some individuals may be placed Because of statistical variation, some individuals may be placed

in wrong group (extremes regress to mean)in wrong group (extremes regress to mean)

Experimental MortalityExperimental Mortality Some individuals ‘leave’ study– if this is systematic for certain Some individuals ‘leave’ study– if this is systematic for certain

groups, it’s a problem.groups, it’s a problem.

SelectionSelection Process for assigning to different groups.Process for assigning to different groups.

Interactions with Participant AssignmentInteractions with Participant Assignment Biases in assignment to groups can also have interactions Biases in assignment to groups can also have interactions

between groups (i.e., environmental factors that differentially between groups (i.e., environmental factors that differentially affect certain individuals who were not randomly assigned to affect certain individuals who were not randomly assigned to groups).groups).

2121

External ValidityExternal Validity

How generalizable is a given study?How generalizable is a given study?

Two major types:Two major types: Population external validityPopulation external validity Ecological external validityEcological external validity

2222

Population External ValidityPopulation External Validity

Population External Validity:Population External Validity: Does the actual sample of participants represent the Does the actual sample of participants represent the

theoretical or target population?theoretical or target population?

To evaluate, you must know:To evaluate, you must know: The theoretical populationThe theoretical population The accessible populationThe accessible population The sampling designThe sampling design The selected sampleThe selected sample The The actualactual sample who complete study sample who complete study

2323

Ecological External ValidityEcological External Validity

Ecological External ValidityEcological External Validity Are the conditions, settings, procedures, Are the conditions, settings, procedures,

questions, etc representative of real life?questions, etc representative of real life? Often, ecological external validity in Often, ecological external validity in

competition with experimental controls that competition with experimental controls that attempt to isolate specific variables.attempt to isolate specific variables.

Example:Example: Study of sharing behavior in P2P-like systems Study of sharing behavior in P2P-like systems

(Cheshire 2005)(Cheshire 2005)

2424

Why care about external validity?Why care about external validity?

1930’s Literary Digest poll:1930’s Literary Digest poll: Franklin Roosevelt predicted to lose the 1936 Franklin Roosevelt predicted to lose the 1936

presidential election by a landslide.presidential election by a landslide. Oops… he Oops… he wonwon by a landslide. by a landslide.

How could this happen?How could this happen? Sample was selected from automobile Sample was selected from automobile

registrations, telephone directories…during registrations, telephone directories…during the middle of the Great Depression.the middle of the Great Depression.

2525

Related point: outliers in sampleRelated point: outliers in sample

What is the best undergraduate major if you want a high What is the best undergraduate major if you want a high income (UNC-Chapel Hill survey)?income (UNC-Chapel Hill survey)?

Geography was #1Geography was #1

Maybe not time to switch majors just yet…Maybe not time to switch majors just yet… One outlier, Michael Jordan, accounted for the huge skew in One outlier, Michael Jordan, accounted for the huge skew in

average salaries for graduates (he makes $80 million/year)average salaries for graduates (he makes $80 million/year)

Key Point: you have to try and make every effort to make Key Point: you have to try and make every effort to make your sample generalizable to the population of interest. your sample generalizable to the population of interest.

Non-representative samples will lead to inaccurate Non-representative samples will lead to inaccurate conclusions!!!conclusions!!!

2626

Measurement ValidityMeasurement Validity

Deals with whether the variables are Deals with whether the variables are appropriately defined and representative appropriately defined and representative of the concepts or constructs under of the concepts or constructs under investigation. Also called construct validity.investigation. Also called construct validity.

Examples:Examples: How do you measure life happiness?How do you measure life happiness? How do you measure technical proficiency?How do you measure technical proficiency? How do you measure one’s social network?How do you measure one’s social network?

2727

Example of Measurement Validity Example of Measurement Validity ProblemProblem

Operational definition of ‘supervision’ is Operational definition of ‘supervision’ is defined as a supervisor being 10 feet or less defined as a supervisor being 10 feet or less from a worker (example from Cook and from a worker (example from Cook and Campbell 1979)Campbell 1979)

Problem: the way that supervision is defined, Problem: the way that supervision is defined, it may be relevant to the construct of ‘stress’ it may be relevant to the construct of ‘stress’ rather than just “supervision”.rather than just “supervision”.

2828

Validity: SummaryValidity: Summary

Internal Validity: Internal Validity: Has the causal link between our concepts (or Has the causal link between our concepts (or

variables) been established?variables) been established?

External Validity: External Validity: Is the study generalizable, and to what group(s)?Is the study generalizable, and to what group(s)?

Measurement Validity: Measurement Validity: Do our measures capture what we want them to Do our measures capture what we want them to

capture?capture?

2929

Reliability in ResearchReliability in Research

3030

ReliabilityReliability

Reliability deals with the consistency of Reliability deals with the consistency of your research instrument (i.e., survey your research instrument (i.e., survey questions, experimental manipulations, questions, experimental manipulations, etc.)etc.)

3131

ReliabilityReliability

Are the findings (or a specific measure) Are the findings (or a specific measure) consistent if you were to do the study over consistent if you were to do the study over again?again?

A study can be reliable, but not valid. A study can be reliable, but not valid. Furthermore, it cannot be valid unless it is Furthermore, it cannot be valid unless it is reliable.reliable.

Thus, reliability is Thus, reliability is absolutely requiredabsolutely required. Validity . Validity is equally important, but the degree of validity is equally important, but the degree of validity (such as external validity) may not be very high (such as external validity) may not be very high depending on the nature of the study.depending on the nature of the study.

3232

Reliability: the problem of errorReliability: the problem of error

Error is the difference between the Error is the difference between the observed score and the ‘true’ score.observed score and the ‘true’ score.

Random error occurs:Random error occurs: Due to observers…Due to observers… Due to individual variation (age, mood, etc)Due to individual variation (age, mood, etc) Due to inconsistent situations during data Due to inconsistent situations during data

collection (i.e., survey on patriotism after 9/11)collection (i.e., survey on patriotism after 9/11)

3333

Methods of Measuring ReliabilityMethods of Measuring Reliability

Split-half or item performanceSplit-half or item performance Analyze half of survey/instrument and Analyze half of survey/instrument and

compare to overall analysis to see if it is compare to overall analysis to see if it is consistent.consistent.

Cronbach’s alpha is a related and common Cronbach’s alpha is a related and common way to measure reliability (correlating way to measure reliability (correlating performance on each item with overall score)performance on each item with overall score)

3434

Three More Methods of Measuring Three More Methods of Measuring ReliabilityReliability

Test-retestTest-retest Administering test to same group at different times, Administering test to same group at different times,

correlate the two scores.correlate the two scores.

Multiple or Parallel formsMultiple or Parallel forms Mixing same items on a survey and giving to same Mixing same items on a survey and giving to same

group twice.group twice.

Inter-rater reliabilityInter-rater reliability Agreement between different interviewers or coders Agreement between different interviewers or coders

on same subjects/responses.on same subjects/responses.

3535

Summary: ReliabilitySummary: Reliability

Basically, reliability just deals with the Basically, reliability just deals with the consistency of your measures. If you can consistency of your measures. If you can show that they are consistent, then you show that they are consistent, then you have this covered.have this covered.

3636

Class Survey: Data Class Survey: Data Preparation and Error-Preparation and Error-

CheckingChecking