1 epi 225 measurement in clinical research fall 2009 anita l. stewart, ph.d. institute for health...
TRANSCRIPT
1
EPI 225
Measurement in Clinical Research
Fall 2009
Anita L. Stewart, Ph.D.Institute for Health & Aging
University of California, San Francisco
2
Overview of Course How to find, critique and select good self-
report measures for your research Weekly homework with weekly feedback I am available for questions/discussion:
– after class: 4:30-5:30– by email
Focus on one concept of interest to you– Health outcome or a determinant of health– Measurable (need to find 2-3 measures to review)
3
Importance of Appropriate Measures
Inadequate measures can result in: Conceptual inadequacy Poor data quality (e.g. missing data) Poor variability Poor reliability and validity Inability to detect associations
– correlations, mean differences
5
Overview of Class 1
Role of concepts in measurement Terminology, depicting concepts Complexity of most concepts Concept development methods Importance of defining concepts prior to
selecting existing measures
6
Typical Sequence of Developing New Self-Report Measures
Develop/define concept
Create item pool
Pretest/revise
Field survey
Psychometric analyses
Final measures
7
Typical Sequence of Developing New Self-Report Measures
Develop/define concept
Create item pool
Pretest/revise
Field survey
Psychometric analyses
Final measures
8
Process of Selecting Good Measures for Your Studies
Define concept (variable)
Identify potential measures
Review measures’ properties--conceptual adequacy
--psychometric adequacy
Pretest best 1-2 measures
Select final measure
9
Process of Selecting Good Measures for Your Studies
Define concept (variable)
Identify potential measures
Review measures’ properties--conceptual adequacy
--psychometric adequacy
Pretest best 1-2 measures
Select final measure
10
Overview of Class 1
Role of concepts in measurement Terminology, depicting concepts Complexity of most concepts Concept development methods Importance of defining concepts prior to
selecting existing measures
11
Concept/Construct
A variable that is relatively abstract as opposed to concrete– e.g. health status, stress, acculturation as
opposed to height, body temperature An abstraction based on observations of
certain behaviors or characteristics Cannot be assessed directly
12
Concept/Construct: Other Terms
Variable, latent trait, latent variable Terms defined differently by different
investigators/authors Similar meaning
– often used interchangeably
13
Latent Variable or Trait
Latent - present but not visible, unobservable
Latent trait - unobservable set of characteristics that can be empirically inferred and estimated through answers to a set of questions
14
Measures of Concepts
Concepts are defined and operationalized in terms of observed indicators
Measures are “proxies” for the latent variables we cannot directly observe
17
Overview of Class 1
Role of concepts in measurement Terminology, depicting concepts Complexity of most concepts Concept development methods Importance of defining concepts prior to
selecting existing measures
18
Complexity of Concepts
By definition, due to abstract nature of concepts, most are complex – Hard to define – Multidimensional
» Concepts within concepts
Some examples– Health– Quality of life– Interpersonal processes of care – Shared decision making
19
“Health” as a Concept or Latent Variable
What comes to mind when you think of the word “health”?
Health
20
Some Common Terminology of “Health”
Health status Health outcomes Clinical status Functional status Functioning and well-being Quality of life Health-related quality of life
22
Perspectives on Health Bench
Scientist Clinician Individual
Molecular, genetic, cellular X ??Anatomic, physiologic signs XSymptoms X XDiagnosed conditions X XSeverity of conditions X XFunctioning in daily life XWell-being X
23
Contradictory or Different Perspectives?
Patient 1 MD - signs of illness
Pt – perceives very good health
Patient 2 MD - no signs of illness
Pt – perceives poor health
Patient 3 Lab: cellular abnormality
Pt - no symptoms
24
Dimension, Domain, Sub-domain, Component
Terms somewhat interchangeable Dimension: a distinct component of a
multidimensional construct that can be theoretically or empirically specified (e.g., physical and mental health)
25
Medical Outcomes Study (MOS)
Large study of patients with chronic disease– 1989 forward
Extensive measurement development part of MOS– Health, functioning, and well-being
27
Physical Health
MOS Physical Health: Dimensions
Physicalfunctioning
Physicalfunctioning
Health perceptions
Health perceptions
PainPainEnergy &
fatigueEnergy &
fatigue
Role limitationsdue to physical
health
Role limitationsdue to physical
health
28
Physical Health
MOS Physical Health: Dimensions, Sub-domains
Physicalfunctioning
Physicalfunctioning
Health perceptions
Health perceptions
PainPainEnergy &
fatigueEnergy &
fatigue
Role limitationsdue to physical
health
Role limitationsdue to physical
health
Painfrequency
Painfrequency
Painseverity
Painseverity
29
MOS Framework: Indicators of Physical and Mental Health
Physical Mental
Indicators Health Health Physical functioning X
Pain X Energy/fatigue X X Sleep problems X X Cognitive functioning X Psychological distress/well-being X Role limitations due to physical health X Role limitations due to emotional problems X Health perceptions X X RD Hays and AL Stewart, Psychological Assessment:
A Journal of Consulting and Clinical Psychology, 2:22-30, 1990.
30
What is Perfect Health?
An ideal toward which people are oriented – not a state they expect to attain
“A healthy person is someone who has been inadequately studied” (p. 31)
AC Twaddle, The concept of health status, Soc Sci Med 1974;8
33
What are normal health states?
Illness
(abnormal)
Perfecthealth
(abnormal)
Less thanperfecthealth?
(normal)
34
Concept of Quality of Life
What comes to mind when you think of the word “quality of life”?
Quality of Life
35
Quality of Life Terminology
Satisfaction with life “as a whole” Well-being Global well-being Subjective well-being Sense of well-being Global happiness
36
Dimensions/Domains of Quality of Life
Marriage Family life Health Neighborhood Friendships Job
City Housing Standard of living National government Local government
…. and many more
37
Health-Related Quality of Life?
Domains of life relevant to health, disease, and medical care
Well-being– Physical (malaise, symptoms)– Mental (psychological well-being)
Functioning– Limited in activities because of health problems
or treatments
39
Shared Decision Making (DM):Some “Definitions”
MD propensity to encourage shared DM Pt desire for active role in DM MD encourages pts to take active role Pt plays active role in deciding Pt and MD share process of deciding on
treatment
40
Varying Definitions Suggest Complex Concept
Process of shared decision making
MD propensityto encourage shared DM
Patient desire for active role in DM
41
Concepts within Concepts
Process of shared decision making
MD propensityto encourage shared DM
Patient desire for active role in DM
42
MD Propensity to Encourage Shared DM
“Participatory Decision-Making Style”: Propensity (of physicians) to
– offer pts choices among treatment options
– to give pts a sense of control of care
– to give pts a sense of responsibility for care
Kaplan SH et al. Ann Int Med, 1996;124:497
43
Varying Definitions Suggest Complex Concept
Process of shared decision making
MD propensityto encourage shared DM
Patient desire for active role in DM
44
Process of Shared Decision Making in ICU: Multidimensional
Information exchange – MD provides relevant medical information – Family member provides info on pt’s values/preferences
Deliberation– MD + family member discuss appropriate treatment options
Treatment decision– MD + family member clarify best treatment choice
Doug White, MD, Dept Med, Div Pulmonary and Critical Care
45
Overview of Class 1
Role of concepts in measurement Terminology, depicting concepts Complexity of most concepts Concept development methods Importance of defining concepts prior to
selecting existing measures
46
Concept Development Methods
Primarily qualitative methods (formative research)– ethnography
– direct observation
– in-depth interviews
– focus groups
47
Qualitative Methods to Develop Concepts and Measures
Focus groups, in-depth interviews Data consist of words (text) Useful for
– Concept development
– Item construction
48
Focus Groups
Open-ended guided group discussion with probing of responses
Listen to perspective of likely “subjects”– How they discuss “concept” in their own
words Participants stimulate comments of others
49
Who Provides Input?
Patients or representatives of the population of interest
“Experts” with extensive clinical or personal experience with population of interest
50
Focus Group Methods
Led by experienced moderator– usually 1 - 2 hours
Purposeful sampling of 6-10 homogenous participants per group
Interview guide – set of open ended questions
Audio-record and transcribe discussion
51
Focus Group Methods Resources
Good overview of focus group methods and application - Vogt 2004 reading– They refer to “enhancing content validity” which is
another way of discussing “concept development” Handout: Using focus groups to develop
structured surveys– Annotated bibliography
52
Focus Groups to Develop Concept of Expectations Regarding Aging (ERA)
How older adults think about aging– Do they expect to decline on various
dimensions?
– Would their expectations affect decisions about health and health care?
53
ERA Concept Development : Qualitative Methods
Conducted 5 focus groups of older adults 10 vignettes describing older adults
experiencing common age-associated changes– Probed “does this describe changes you would
expect to occur with aging?” Analyzed transcripts using qualitative methods Results: 26 “domains”
CA Sarkisian et al. Med Care. 2003;39:1025-36.
54
Domains Resulting from ERA Focus Groups
Physical function Cognitive function Social function Pain Sexual function Personal autonomy Loneliness
Appearance Energy Urinary incontinence Falling Happiness Spirituality Being treated differently
55
Example: Interview Guide Exploring Depression Concept in Diverse Groups
Sample: White, South Asian, and Black Caribbean individuals in U.K.
Questions: – What does the word depression mean to you?
– Do you consider depression to be an illness?
– What do you think might cause depression?
V Lawrence et al., Gerontologist, 2006;46:23-32
56
More Advanced Concept Development Methods
Iterative process Integrates qualitative and quantitative
methods (“mixed methods”)
57
Mixed Methods: Iterative Sequence to Develop Structured Measures
Focus groups In-depth interviews Input from quantitative studies Develop preliminary items Review by expert panel Cognitive interview pretest Pilot study Nationwide survey Psychometric testing
N Krause, J Geront: Soc Sci, 2002;57B:S263-274
58
Mixed Methods: Iteratively Refine Concept Based on Quantitative Results
Develop/define concept
Create item pool
Pretest/revise
Field survey
Psychometric analyses
Final measures
59
Iterative Concept Development: Interpersonal Processes of Care (IPC)
Interpersonal processes between physicians and patients, emphasis on vulnerable patients
Two “rounds” of concept development Round one – initial concept based on:
– Review of literature– Content of various satisfaction with care measures– Clinical experience of investigators
IN READINGS: AL Stewart et al: Milbank Quarterly, 1999
60
Initial Hypothesized “Multidimensional” IPC Conceptual Framework
I. COMMUNICATION III. INTERPERSONAL STYLE General clarity Friendly, courteous Elicitation of Respectful patient issues Perceived discrimination Explanations of Cultural sensitivity -condition Emotional support -self care -process of care II. DECISION MAKING Responsiveness to patient preferences Consideration of ability to comply
Stewart et al., Milbank Mem Q,1999, Table 1
61
IPC Concept Development (cont)
Initial concept modified after measurement analyses
Differences from hypothesized concept:– No cultural sensitivity scale
– Content of many dimensions changed
Stewart et al., Milbank Mem Q, 1999, Tables 2 and 3
62
“Round Two” IPC Concept Development
Draft IPC II
conceptualframework
IPC framework (Milbank Quarterly)
19 new focus groups -African American, Latino,and White adults
Updated literature review of quality of care in diverse groups
63
IPC “Round Two”Hypothesized Conceptual Framework
I. COMMUNICATION III. INTERPERSONAL STYLE General clarity Friendly, courteous Elicitation Respectful Explanations of Perceived discrimination --condition Cultural sensitivity --process of care Emotional support --self-care Empowerment Office staff: friendly II. DECISION MAKING courteous, respectful Responsive to pt preferences Consider ability to comply
Stewart et al., Health Serv Res, 2007;42:3(Part I):1235-1255, Table 1
64
IPC “Round Two”Measures Developed and Tested
Develop/define concept
Create item pool
Pretest/revise
Field survey N=1,664
Psychometric analyses
Final measures
65
IPC “Round Two”Final Conceptual Framework
I. COMMUNICATION III. INTERPERSONAL STYLE General clarity Emotional support Hurried, distracted Respectful Elicited concerns Assumptions re SES Explained results, Discriminated due to R/E medications Disrespectful office staff II. PATIENT-CENTERED DECISION MAKING Considered ability to comply Decided together Stewart et al., Health Serv Res, 2007 - Fig 1
66
IPC Concept Development: Cultural Sensitivity of Physicians
Hypothesized scale did not work in either of our IPC studies
For minority patients and those with limited English proficiency– Sensitivity of providers to their cultural
perspective
67
New Qualitative Analyses of Prior Focus Groups
Two open-ended focus group questions:– What does the word culture mean to you?– What do your doctors understand about your
culture that might affect your visits? Transcripts analyzed using qualitative
methods
Nápoles-Springer et al. Health Expectations. 2005;8:4-17.
68
Results: Multiple Dimensions of Cultural Factors Affecting the Medical Encounter
Examples … Values, beliefs, and
attitudes Communication
including sensitivity to language access
Complementary and alternative medicine
Privacy and modesty Ethnic concordance
of physician and patient
Empowerment and submissiveness
69
Another Concept Development Method:Review Existing Measures
Work “backwards” by organizing items in published measures
Can identify underlying “concept” implied by measures
70
Physical Functioning Concept “Derived” from Four Measures
AIMS MOS HAQ SIP
Walking X X X X
Climbing stairs or steps X X X X
Bending, kneeling X X X X
Lifting, carrying X
Getting out of bed X
Bathing X X X
Running errands X
Opening jars X
Vigorous activities X X X
71
Depression Concept “Derived” from Four Measures
CES-D20
HRSD21
Beck21
GDS15
Personal functioning, behavior
-sleep problems, substance abuse
3 3 4 3
Social functioning, behavior -cut down work, withdraw
1 1 0 0
Physical symptoms
-fatigue, low libido
2 8 5 1
Emotional status
-depressed mood, lonely, happy
9 3 4 5
Cognitions/perceptions
-sense of failure, suicidal thoughts
5 6 8 6
72
Depression Concept “Derived” from Four Measures (cont)
CES-D20
HRSD21
Beck21
GDS15
Personal functioning, behavior
-sleep problems, substance abuse
3 3 4 3
Social functioning, behavior -cut down work, withdraw
1 1 0 0
Physical symptoms
-fatigue, low libido
2 8 5 1
Emotional status
-depressed mood, lonely, happy9 3 4 5
Cognitions/perceptions
-sense of failure, suicidal thoughts
5 6 8 6
73
Concept of Body Awareness
Interested in measuring body awareness as an outcome of integrated medicine treatments/therapies
Reviewed 12 measures of concepts related to body awareness– e.g., Body Intelligence Scale, Body Awareness
Measure, Scale of Body Connection
IN READINGS: W Mehling et al., PLosOne, 2009;4(5):1-18.
74
Adapted From Mehling’s Table 6: Concept Derived from Measures?
BIS BRQ BAM TQ SBA SBC
Body sensations
Stress, worry X X X
Well-being X X X
Pain, symptoms X X
Attention quality
Amount of attention
X X X X X X
Thinking vs experiencing
X
Emotional awareness
X X X
75
Multidimensional Concept: Result of Measures Review (Mehling’s Table 5)
Dimension Sub-Domain
Perceived body sensation
Sensations of distress, worry, pain, tension
Sensations of well-being
Neutral sensations
Affective aspect of sensation
Attention quality Intensity of attention/ignoring sensation
Self-efficacy in attention control
Thinking/labeling vs. present sensation
Attitude Trusting
Catastrophizing
Mind-body integration
Emotional awareness
Sense of embodied self vs disconnected
76
Overview of Class 1
Role of concepts in measurement Terminology, depicting concepts Complexity of most concepts Concept development methods Importance of defining concepts prior to
selecting existing measures
77
Process of Selecting Good Measures for Your Studies
Define concept (variable)
Identify potential measures
Review measures’ properties--conceptual adequacy
--psychometric adequacy
Pretest best 1-2 measures
Select final measure
78
Process of Selecting Good Measures for Your Studies
Define concept (variable)
Identify potential measures
Review measures’ properties--conceptual adequacy
--psychometric adequacy
Pretest best 1-2 measures
Select final measure
79
State Role of Concept in Your Research
Evaluate intervention (outcome) Determinant (e.g., social support) of an
outcome Predict future event (e.g. functioning to
predict hospitalization) Describe population Covariate
80
Define Concept for Your Research
Define it first from your point of view – How you would define it based on your experience
and understanding of your research question For dependent variables
– Describe how intervention or independent variables might affect it - types of changes you expect
For independent (predictor) variables– Describe how it might be predictive
81
Measure of Concept Needs to Match Your Definition
Concept definition is the basis for selecting appropriate measure
Measure must reflect all key dimensions of your concept
82
Measure of Concept Needs to Match Your Definition (cont)
Label or name of a measure may not reflect what it actually contains
May find measure of your concept with the right label– But mismatched
» Measures something different than you need» Measures only part of what you need (missing
elements)
83
Homework: Define Concept
Summarize briefly a key research question of interest to you
Choose one key concept from this question and describe its role (e.g. dependent variable)
Define key concept from your point of view including its dimensions
84
Send Homework by Email
Include your last name in the file name of your homework
Send to [email protected]
85
Use Syllabus!
Print out syllabus– A pdf file of the syllabus is posted along with
class 1 readings Syllabus contains all details (readings,
handouts, homework)