measurement issues general steps –determine concept –decide best way to measure –what...
TRANSCRIPT
Measurement Issues
• General steps– Determine concept– Decide best way to measure–What indicators are available– Select intermediate, alternate or indirect
measures
Measurement Issues
• General steps– Consider limitations of measures
selected– Collect or secure info/data– Summarize findings in writing
• What is the relation between concepts, variables, instruments & measures?
Concepts
• Program is based on conceptual basis of why people behave the way they do
• Why do you think people behave the way they do?
• Think of food and nutrition issues
Variables
• A theory has variables
• Variables define concepts
• Theory states how the variables interact or are related
Variables
• Variables of the theory are what you measure
• Variables are the verbal or written abstractions of the ideas that exist in the mind
• Why should an intervention be based on a theory?
Why use theory?
• Know what you are to address in the intervention
• Makes evaluation easier
• Know what to measure to evaluate
• Figure 6.1 A simple social learning theory model for reducing salt in the diet
• Comes next
Fig. 6.1 Social learning theory
• Need measurements and instruments to assess changes in the variables of interest
Instruments
• Something that produces a measure of an object
• Series of questions to measure the variable, concept
• Includes instructions
Measures
• The numbers that come from the person answering questions on the instrument
• Figure 6.2 Relation among models, variables, measures, and an instrument
• Comes next
Fig. 6.2
• Based on why you think people behave the way the do, list possible variables to consider to measure this variable.
• What might be variables of the social learning theory?
• What about variables that would verify if a change has or has not taken place?
• Figure 6.1 A simple social learning theory model for reducing salt in the diet
• Comes next
• See how the program links with the theory & what measure
Fig. 6.1 Social learning theory
Reliability
• The extent to which an instrument will produce the same result (measure or score) if applied two different or more times.
Reliability
• X = T + E
• X is measure
• T is true value
• E is random error
Reliability
• Measurement error reduces the ability to have reliable and valid results.
Reliability
• Random error is all chance factors that confound the measurement.
• Always present
• Effects reliability but doesn’t bias results
Reliability
• Figure 6.5 Distribution of scores of multiple applications of a test with random error
• A is true score
• a is measure
Fig. 6.5 Distribution of scores of multiple applications of a test and random error
Distribution
• Can have the same mean with two different distributions
• Figure 6.6 next
Fig. 6.6 Two distributions of scores aroundthe true mean
• Which distribution has less variability?
• Which distribution has less random error?
Sources of Random Error
• Day-to-day variability
• Confusing instructions
• Unclear instrument
• Sloppy data collector
Sources of Random Error
• Distracting environment
• Respondents
• Data-management error
• What can you do to reduce random error and increase reliability?
Variability & the Subject
• What you want to measure will vary from day to day and within the person
Variability & the Subject
• Intraindividual variability– variability among the true scores within
a person over time
• Figure 6.7 True activity scores (A, B, C) for 3 days with three measures (a, b, c) per day
• Comes next
Fig. 6.7 True activity (A, B, C) for 3 days with three measures (a, b, c) per day
Variability & the Subject
• Interindividual variability– variability between each person in the
sample
• Figure 6.8 Interindividual (A, X) and intraindividual (A1, A2, A3) variability for two people (A, X) in level of physical activity
• Comes next
Fig. 6.8 Interindividual (A, X) and intraindividual (A1, A2, A3) variability for two people (A, X) in levelof physical activity
Assessing Reliability
• Need to know the reliability of your instruments
• Reliability coefficient of 1 is highest, no error
• Reliability coefficient of 0 is lowest, all error
Factors of Reliability
• Type of instrument– observer– self-report
• Times instrument applied– same time– different time
• Figure 6.9 Types of reliability
• Comes next
Fig. 6.9 Types of reliability
Assessing Reliability
• Interobserver reliability– have 2 different observers rate same
action at same time– reproducibility
Assessing Reliability
• Intraobserver reliability– 1 observer assesses same person at
two different times– video tape the action & practice
Assessing Reliability
• Repeat method– self-report or survey– repeat the same item/question at 2
points in survey
Assessing Reliability
• Internal consistency– average inter-item correlation among
items in an instrument that are cognitively related
Assessing Reliability
• Internal consistency– Cronbach’s alpha– 0.70 & above a good score
Assessing Reliability
• Test-retest reliability (internal consistency method)– same survey/test at 2 different times to
same person
Validity
• Degree to which an instrument measures what the evaluator wants it to measure
Bias
• Systematic error that produces a systematic difference between an obtained score and the true score
• Bias threatens validity
Bias
• Figure 6.10 Distribution of scores of multiple applications of a test with systematic error
• Comes next
Fig. 6.10 Distribution of scores of multiple applications of a test with systematic error
• What will basis do to your ability to make conclusions about your subjects?
• Figure 6.11 Effect of bias on conclusions
• Comes next
Fig. 6.11 Effect of bias on conclusions
Types of Validity
• Face
• Content
• Criterion
Face Validity
• Describes the extent to which an instrument appears to measure what it is suppose to measure
• How many veg did you eat yesterday?
Content Validity
• Extent to which an instrument is expected to cover several domains of the content
• Consult a group of experts
Criterion Validity
• How accurate is a less costly way to measure the variable compared to the valid and more expensive instrument
What can lower validity?
• Guinea pig effect– awareness of being tested
• Role selection– awareness of being measured may
make people feel they have to play a role
What can lower validity?
• Measurement as a change agent– act of measurement could change
future behavior
What can lower validity?
• Response sets– respond in a predictable way that has
nothing to do with the questions
What can lower validity?
• Interviewer effects– characteristics of the interviewer affects
the receptivity and answers of the respondent
What can lower validity?
• Population restrictions– if people can’t use the method of data
collection, can’t generalize to others
• End of reliability and validity
• Questions
• Look at CNEP Survey