1

EPI 225

Measurement in Clinical Research

Fall 2008

Anita L. Stewart, Ph.D.Institute for Health & Aging

University of California, San Francisco

2

Overview of Entire Class How to identify, critique and select good

measures for your research Weekly homework (paper copy) w/weekly

feedback Two discussion sessions (optional)

– 2:15-3:00 and 4:30-5:00 Focus on one concept of interest to you

– Health dimension or a determinant of health– Measurable (need to find 2-3 measures to review)

3

Importance of Appropriate Measures

Inadequate measures can result in: Conceptual inadequacy Poor data quality (e.g. missing data) Poor variability Poor reliability and validity Inability to detect associations

– correlations, mean differences

4

Class 1

Concept Development and Concept Definitions

September 18, 2008

5

Overview of Class 1

Role of concepts in measurement Complexity of most concepts Concept development methods Importance of defining concepts prior to

selecting existing measures

6

Typical Sequence of Developing New Self-Report Measures

Develop concept

Create item pool

Pretest/revise

Field survey

Psychometric analyses

Final measures

7

Typical Sequence of Developing New Self-Report Measures

Develop concept

Create item pool

Pretest/revise

Field survey

Psychometric analyses

Final measures

8

Process of Selecting Good Measures for Your Studies

Define concept (variable)

Identify potential measures

Review measures’ properties--conceptual adequacy

--psychometric adequacy

Pretest best 1-2 measures

Select final measure

9

Process of Selecting Good Measures for Your Studies

Define concept (variable)

Identify potential measures

Review measures’ properties--conceptual adequacy

--psychometric adequacy

Pretest best 1-2 measures

Select final measure

10

Concept/Construct

A variable that is relatively abstract as opposed to concrete– e.g. health status, stress, acculturation vs.

height, body temp An abstraction based on observations of

certain behaviors or characteristics Cannot be assessed directly

11

Concept/Construct: Other Terms

Variable, latent trait, latent variable Terms defined differently by different

investigators/authors Similar meaning

– often used interchangeably

12

Latent Variable or Trait

Latent - present but not visible, unobservable

Latent trait - unobservable set of characteristics that can be empirically inferred and estimated through answers to a set of questions

13

Measures of Concepts

Concepts are defined and operationalized in terms of observed indicators

Measures are “proxies” for the latent variables we cannot directly observe

14

Depicting Latent Variables and Measures

CONCEPTVariable A

Measure A

CONCEPTVariable B

Measure B

15

Depicting Latent Variables and Measures

Stress

Perceived Stress Scale

Health status

SF-36

16

Overview of Class 1

Role of concepts in measurement Complexity of most concepts Concept development methods Importance of defining concepts prior to

selecting existing measures

17

Complexity of Concepts

By definition, due to abstract nature of concepts, most are complex – Hard to define

– Multidimensional» Concepts within concepts

18

Example: Shared Decision Making

How would you define “shared decision making”?

19

Shared Decision Making (DM): Some “Definitions”

MD propensity to encourage shared DM MD encourages pts to take active role MD responsive to pt preferences regarding

participating in decision Pt desire for active role in DM Pt plays active role in deciding Pt and MD share process of deciding on

treatment

20

Varying Definitions Suggest Complex Concept

Process of shared decision making

MD propensityto encourage shared DM(participatory DM style)

Patient desire for shared DM

(active patient orientation)

21

Varying Definitions Suggest Complex Concept

Process of shared decision making

MD propensityto encourage shared DM(participatory DM style)

Patient desire for shared DM

(active patient orientation)

22

Participatory Decision-Making Style

Propensity (of physicians) to– offer pts choices among treatment options

– to give pts a sense of control of care

– to give pts a sense of responsibility for care

Kaplan SH et al. Ann Int Med, 1996;124:497

23

Varying Definitions Suggest Complex Concept

Process of shared decision making

MD propensityto encourage shared DM(participatory DM style)

Patient desire for shared DM

(active patient orientation)

24

Process of Shared Decision Making in ICU: Multidimensional

Information exchange – MD provides relevant medical information – Surrogate provides information on pt’s values/preferences

Deliberation– MD and surrogate discuss which treatment option is most

appropriate given pt’s values and condition Treatment decision

– MD and surrogate clarify what they think is correct treatment choice

Doug White, MD, Dept Med, Div Pulmonary and Critical Care

25

“Health” as a Concept or Latent Variable

What comes to mind when you think of the word “health”?

Health

26

Some Common Terminology of “Health”

Health indicators Health status Health outcomes Clinical status Functional status Functioning and well-being Quality of life Health-related quality of life

27

Definition Varies by Perspective

Health

Bench scientist

Individual

Populationscientist Clinician

28

Perspectives on Health Bench

Scientist Clinician Individual

Molecular, genetic, cellular X ??Anatomic, physiologic signs XSymptoms X XDiagnosed conditions X XSeverity of conditions X XFunctioning in daily life XWell-being X

29

Contradictory or Different Perspectives?

Patient 1 Signs of illness (MD)

Perception: very good health (pt)

Patient 2 No signs of illness (MD)

Perception: poor health (pt)

Patient 3 Cellular abnormality (lab)

No signs of illness (MD)

30

Dimension, Domain, Sub-domain, Component

Terms somewhat interchangeable Dimension: a distinct component of a

multidimensional construct that can be theoretically or empirically specified (e.g., physical and mental health)

31

MOS Framework: Physical and Mental Health are “Latent Variables”

Physicalhealth

Mentalhealth

32

Physical Health

MOS Physical Health: Dimensionality

Physicalfunctioning

Physicalfunctioning

Health perceptions

Health perceptions

PainPainEnergy &

fatigueEnergy &

fatigue

Role limitationsdue to physical

health

Role limitationsdue to physical

health

33

Physical Health

MOS Physical Health: Dimensionality

Physicalfunctioning

Physicalfunctioning

Health perceptions

Health perceptions

PainPainEnergy &

fatigueEnergy &

fatigue

Role limitationsdue to physical

health

Role limitationsdue to physical

health

Painfrequency

Painseverity

34

MOS Framework: Indicators of Physical and Mental Health

Physical Mental

Indicators Health Health Physical functioning X Pain X Energy/fatigue X X Sleep problems X X Cognitive functioning X Psychological distress/well-being X Social activity limitations due to health X X Role limitations due to physical health X Role limitations due to emotional problems X Current health perceptions X X

35

What is Perfect Health?

An ideal toward which people are oriented – not a state they expect to attain

“A healthy person is someone who has been inadequately studied” (p. 31)

AC Twaddle, The concept of health status, Soc Sci Med 1974;8

36

Health as a continuum?

Illness Perfecthealth

37

Health as a continuum?

Illness Perfecthealth

Less thanperfecthealth?

38

What are normal health states?

Illness

(abnormal)

Perfecthealth

(abnormal)

Less thanperfecthealth?

(normal)

39

Concept of Quality of Life

What comes to mind when you think of the word “quality of life”?

Quality of Life

40

Quality of Life Terminology

Satisfaction with life “as a whole” Well-being Global well-being Subjective well-being Sense of well-being Global happiness

41

Dimensions/Domains of Quality of Life

Marriage Family life Health Neighborhood Friendships Job

City Housing Standard of living National government Local government

…. and many more

42

Health-Related Quality of Life?

Domains of life relevant to health, disease, and medical care

Well-being– Physical (malaise, symptoms)– Mental (psychological well-being)

Functioning– Limited in activities because of health problems

or treatments

43

Overview of Class 1

Role of concepts in measurement Complexity of most concepts Concept development methods Importance of defining concepts prior to

selecting existing measures

44

Concept Development Methods

Primarily qualitative methods (formative research)

Many types– ethnography, participant-observation, direct

observation, focus groups, in-depth interviews

45

Qualitative Methods to Develop Concepts and Measures

Focus groups, in-depth interviews Data consist of words (text), not numbers Useful for

– Concept development

– Item construction

46

Focus Groups

Open-ended guided group discussion with probing of responses

Listen to perspective of likely “subjects”– How they discuss “concept” in their own

words Participants stimulate comments of others

47

Who Provides Input?

Patients or representatives of the population of interest

“Experts” with extensive clinical or personal experience with population of interest

48

Focus Group Methods

Led by experienced moderator– usually 1 - 2 hours

Purposeful sampling of 6-10 homogenous participants per group

Interview guide – set of open ended questions

Audio-record and transcribe discussion

49

Example: Interview Guide for Concept of Depression in Diverse Groups

Sample: White, South Asian, and Black Caribbean individuals in U.K.

Questions: – What does the word depression mean to you?

– Do you consider depression to be an illness?

– What do you think might cause depression?

V Lawrence et al., Gerontologist, 2006;46:23-32

50

Advanced Concept Development Methods

Iterative process Integrating qualitative and quantitative

methods (“mixed methods”)

51

Nine Step Iterative Sequence to Develop Structured Measures

Focus groups In-depth interviews Input from quantitative studies Develop preliminary items Review by expert panel Cognitive interview pretest Pilot study Nationwide survey Psychometric testing

N Krause, J Geront: Soc Sci, 2002;57B:S263-274

52

Iteratively Refine Concept Based on Qualitative Methods

Develop concept

Create item pool

Pretest/revise

Field survey

Psychometric analyses

Final measures

53

Mixed Methods: Iteratively Refine Concept Based on Quantitative Results

Develop concept

Create item pool

Pretest/revise

Field survey

Psychometric analyses

Final measures

54

Concept Development: Interpersonal Processes of Care

Interpersonal processes between physicians and patients, emphasis on vulnerable patients

Two “rounds” of concept development Round one – initial concept and items based on:

– Review of literature– Satisfaction with care measures content– Clinical experience of investigators

55

Initial Hypothesized “Multidimensional” Conceptual Framework of IPC

I. COMMUNICATION III. INTERPERSONAL STYLE General clarity Friendly, courteous Elicitation/responsiveness Respectful to pt issues Perceived discrimination Explanations of Cultural sensitivity -condition Emotional support -self care -process of care Empowerment II. DECISION MAKING Responsiveness to patient preferences Consideration of ability to comply

Stewart et al., Milbank Mem Q,1999 in readings, Table 1

56

Concept Development: Interpersonal Processes of Care (cont)

Initial concept modified slightly after measurement analyses

Differences:– No cultural sensitivity scale

– Content of many dimensions changed

Stewart et al., Milbank Mem Q, 1999, Tables 2 and 3

57

Developing “Round Two” IPC Concept

Draft IPC II

conceptualframework

IPC Round One frameworkin Milbank Quarterly

19 new focus groups -African American, Latino,and White adults

Updated literature review of quality of care in diverse groups

58

IPC “Round Two” Hypothesized Conceptual Framework

I. COMMUNICATION III. INTERPERSONAL STYLE General clarity Friendly, courteous Elicitation/responsiveness Respectful Explanations of Perceived discrimination --condition Cultural sensitivity --processes of care Emotional support --self-care Empowerment Office staff: friendly II. DECISION MAKING courteous, respectful Responsive to pt preferences Consider ability to comply

Stewart et al., Health Serv Res, 2007;42:3(Part I):1235-1255, Table 1

59

Round Two: Interpersonal Processes of Care Concept

Develop concept

Create item pool

Pretest/revise

Field survey N=1,664

Psychometric analyses

Final measures

60

Final “Round Two” Conceptual Framework

I. COMMUNICATION III. INTERPERSONAL STYLE General clarity Emotional support Hurried, distracted Respectful Elicited concerns Assumed SES Explained results Discriminated due to R/E Explained medications Disrespectful office staff II. PATIENT-CENTERED DECISION MAKING Consider ability to comply Decided together Stewart et al., Health Serv Res, 2007 - Fig 1

61

Concept Development: Cultural Sensitivity of Physicians

For minority patients and those with limited English proficiency– Sensitivity of providers to their cultural

perspective hypothesized to be an important aspect of interpersonal processes of care

Hypothesized scale did not work in either of our IPC studies

62

New Qualitative Analyses

19 focus groups of ethnically diverse adults– Groups homogeneous on ethnicity, language

(English/Spanish), gender, and age Two open-ended focus group questions:

– What does the word culture mean to you?– What do your doctors understand about your culture

that might affect your visits? Transcripts analyzed using qualitative methods

Nápoles-Springer et al. Health Expectations. 2005;8:4-17.

63

Results: Multiple Dimensions of Cultural Factors Affecting the Medical Encounter

Examples … Values, beliefs, and

attitudes Communication

including sensitivity to language access

Complementary and alternative medicine

Privacy and modesty Ethnic concordance

of physician and patient

Empowerment and submissiveness

64

Overview of Class 1

Role of concepts in measurement Complexity of most concepts Concept development methods Importance of defining concepts prior to

selecting existing measures

65

Process of Selecting Good Measures for Your Studies

Define concept (variable)

Identify potential measures

Review measures’ properties--conceptual adequacy

--psychometric adequacy

Pretest best 1-2 measures

Select final measure

66

Process of Selecting Good Measures for Your Studies

Define concept (variable)

Identify potential measures

Review measures’ properties--conceptual adequacy

--psychometric adequacy

Pretest best 1-2 measures

Select final measure

67

State Role of Concept in Your Research

Evaluate intervention (outcome) Predict an outcome Describe population Predict future event (e.g. health, use of

emergency care) Covariate

68

Define Concept for Your Research

Define it first from your point of view – How you would define it based on your experience

and understanding of your research question For dependent variables

– Describe how intervention or independent variables might affect it - types of changes you expect

For independent (predictor) variables– Describe how it might be predictive

69

Importance of Defining Concept Your Way

Distinction between “label” or name of a measure and what it actually contains

May find “good” measure of your concept with the right label– But mismatched

» Measures something different than you need» Measures only part of what you need (missing

elements)

70

Four Physical Functioning Measures: Content Areas

AIMS MOS HAQ SIP

Walking X X X X

Climbing stairs or steps X X X X

Bending, kneeling X X X X

Lifting, carrying X

Getting out of bed X

Bathing X X X

Running errands X

Opening jars X

Vigorous activities X X X

71

Four Depression Measures: Content Areas

CES-D20

HRSD21

Beck21

GDS15

Personal functioning, behavior

-sleep problems, substance abuse

3 3 4 3

Social functioning, behavior -cut down work, withdraw

1 1 0 0

Physical symptoms

-fatigue, low libido

2 8 5 1

Emotional status

-depressed mood, lonely, happy

9 3 4 5

Cognitions/perceptions

-sense of failure, suicidal thoughts

5 6 8 6

72

Four Depression Measures: Content Areas as Concept Definitions?

CES-D20

HRSD21

Beck21

GDS15

Personal functioning, behavior

-sleep problems, substance abuse

3 3 4 3

Social functioning, behavior -cut down work, withdraw

1 1 0 0

Physical symptoms

-fatigue, low libido

2 8 5 1

Emotional status

-depressed mood, lonely, happy9 3 4 5

Cognitions/perceptions

-sense of failure, suicidal thoughts

5 6 8 6

73

Homework: Define Concept

Summarize briefly a key research question of interest to you

Choose one key concept and describe its role (e.g. dependent variable)

Define concept from your point of view– What are the dimensions of your

definition?