more practical metrics for standardizing health outcomes in effectiveness research
DESCRIPTION
More Practical Metrics for Standardizing Health Outcomes in Effectiveness Research. John E. Ware, Jr., PhD, Professor and Chief Division of Measurement Sciences, Department of Quantitative Health Sciences, University of Massachusetts Medical School, Worcester, MA - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: More Practical Metrics for Standardizing Health Outcomes in Effectiveness Research](https://reader035.vdocuments.mx/reader035/viewer/2022081515/56812f58550346895d94e8b2/html5/thumbnails/1.jpg)
More Practical Metrics for Standardizing Health Outcomes in Effectiveness Research
John E. Ware, Jr., PhD, Professor and Chief
Division of Measurement Sciences, Department of Quantitative Health Sciences, University of Massachusetts Medical School, Worcester, MA
Track A - Patient Reported Outcome Measurement and Comparative Effectiveness Research to Reform: Achieving Health System Change
AHRQ 2009 Annual Conference, Bethesda MD September 13-16, 2009
![Page 2: More Practical Metrics for Standardizing Health Outcomes in Effectiveness Research](https://reader035.vdocuments.mx/reader035/viewer/2022081515/56812f58550346895d94e8b2/html5/thumbnails/2.jpg)
2
What is the Relationship Between Health Care Expenditures &
Outcomes?
Expenditures for Health Care ($)
HealthOutcome
![Page 3: More Practical Metrics for Standardizing Health Outcomes in Effectiveness Research](https://reader035.vdocuments.mx/reader035/viewer/2022081515/56812f58550346895d94e8b2/html5/thumbnails/3.jpg)
3
More Health Care is Not Always Better
HealthOutcome
“Flat of the Curve”
Expenditures for Health Care ($)
Health Insurance Experiment Revealed:
![Page 4: More Practical Metrics for Standardizing Health Outcomes in Effectiveness Research](https://reader035.vdocuments.mx/reader035/viewer/2022081515/56812f58550346895d94e8b2/html5/thumbnails/4.jpg)
4
When the Same Outcome Costs More,
HealthOutcome
Payers & Consumers:
Expenditures for Health Care ($)
Want to Pay Less
![Page 5: More Practical Metrics for Standardizing Health Outcomes in Effectiveness Research](https://reader035.vdocuments.mx/reader035/viewer/2022081515/56812f58550346895d94e8b2/html5/thumbnails/5.jpg)
5
Who is Most Vulnerable with Aggressive Cost Containment?
HealthDecline
Expenditures for Health Care ($)
Cost Containment
WellWell offYoung
Health Insurance Experiment (HIE)(1974-1981)Medical Outcomes
Study (MOS)(1986-1990)
Most vulnerable inthe MOS:• Chronically ill• Elderly• Poor• Non- white
![Page 6: More Practical Metrics for Standardizing Health Outcomes in Effectiveness Research](https://reader035.vdocuments.mx/reader035/viewer/2022081515/56812f58550346895d94e8b2/html5/thumbnails/6.jpg)
6
4-Year Physical Health Outcomes Favored FFS > HMO for Chronically-
Ill Medicare in the MOS
Source: Ware, Bayliss, Rogers et al., JAMA, 1996; 276:1039-1047
These percentages, better & worse would be only about 5% due to measurement error
![Page 7: More Practical Metrics for Standardizing Health Outcomes in Effectiveness Research](https://reader035.vdocuments.mx/reader035/viewer/2022081515/56812f58550346895d94e8b2/html5/thumbnails/7.jpg)
7
When Outcomes Vary at the Same Price
Expenditures for Health Care ($)
HealthOutcome
Payers & ConsumersWant the Best Outcomes
![Page 8: More Practical Metrics for Standardizing Health Outcomes in Effectiveness Research](https://reader035.vdocuments.mx/reader035/viewer/2022081515/56812f58550346895d94e8b2/html5/thumbnails/8.jpg)
8
To Compare Health Care EffectivenessWe Need Health Outcomes “Rulers”
Expenditures for Health Care ($)
Better
Same
Worse
HealthOutcome 4
6
7
1
2
3
5
![Page 9: More Practical Metrics for Standardizing Health Outcomes in Effectiveness Research](https://reader035.vdocuments.mx/reader035/viewer/2022081515/56812f58550346895d94e8b2/html5/thumbnails/9.jpg)
9
Continuum of Disease-specific and Generic Health Measures
9
Specific Symptoms
Adapted from: Wilson and Cleary, JAMA, 1995 Ware, Annual Rev. Pub. Health, 1995
(1) (2) (3) (4)
![Page 10: More Practical Metrics for Standardizing Health Outcomes in Effectiveness Research](https://reader035.vdocuments.mx/reader035/viewer/2022081515/56812f58550346895d94e8b2/html5/thumbnails/10.jpg)
1010
Specific Symptoms
Adapted from: Wilson and Cleary, JAMA, 1995 Ware, Annual Rev. Pub. Health, 1995
(1) (2) (3) (4)
dd
Spirometry
Over the last 4 weeks I have had shortness of breath Almost every day Several days a week A few days a month Not at all
Shortness of Breath
Continuum of Disease-specific and Generic Health Measures
![Page 11: More Practical Metrics for Standardizing Health Outcomes in Effectiveness Research](https://reader035.vdocuments.mx/reader035/viewer/2022081515/56812f58550346895d94e8b2/html5/thumbnails/11.jpg)
1111
Specific Symptoms
Adapted from: Wilson and Cleary, JAMA, 1995 Ware, Annual Rev. Pub. Health, 1995
(1) (2) (3) (4)
dd
Over the last 4 weeks I have had shortness of breath Almost every day Several days a week A few days a month Not at all
How much did your lung/respiratory problems limit your usual activities or enjoyment of everyday life? Not at all A little Moderately Extremely
Spirometry Shortness of Breath
Respiratory-specific
Continuum of Disease-specific and Generic Health Measures
![Page 12: More Practical Metrics for Standardizing Health Outcomes in Effectiveness Research](https://reader035.vdocuments.mx/reader035/viewer/2022081515/56812f58550346895d94e8b2/html5/thumbnails/12.jpg)
1212
Specific Symptoms
Adapted from: Wilson and Cleary, JAMA, 1995 Ware, Annual Rev. Pub. Health, 1995
(1) (2) (3) (4)
dd
Over the last 4 weeks I have had shortness of breath Almost every day Several days a week A few days a month Not at all
How much did your lung/respiratory problems limit your usual activities or enjoyment of everyday life? Not at all A little Moderately Extremely
In general, would you say your health is…
Excellent Very good Good Fair Poor
Spirometry Shortness of Breath
Respiratory-specific
Generic
Continuum of Disease-specific and Generic Health Measures
![Page 13: More Practical Metrics for Standardizing Health Outcomes in Effectiveness Research](https://reader035.vdocuments.mx/reader035/viewer/2022081515/56812f58550346895d94e8b2/html5/thumbnails/13.jpg)
13
Specific Symptoms
(1) (2) (3) (4)
There is More to the Continuum
![Page 14: More Practical Metrics for Standardizing Health Outcomes in Effectiveness Research](https://reader035.vdocuments.mx/reader035/viewer/2022081515/56812f58550346895d94e8b2/html5/thumbnails/14.jpg)
14
Prediction and Risk Management:
PROs are among the Best Predictors
Health-Related QOL (HR-QOL)
Future health
Inpatient expenditures
Outpatient expenditures
Job loss
Response to treatment
Return to work
Work productivity
Mortality
(3) (4)
![Page 15: More Practical Metrics for Standardizing Health Outcomes in Effectiveness Research](https://reader035.vdocuments.mx/reader035/viewer/2022081515/56812f58550346895d94e8b2/html5/thumbnails/15.jpg)
What Do We Need for Comparative Effectiveness
Research?
• Outcomes that matter to patients
• Practical measures
• Coverage of a wide range
• Greater precision
• Comparability of scores
• Ease of interpretation
15
Physical activity limitationsSymptoms of psychological distressPhysical well-beingLife satisfactionEmotional behaviorRole disability due to physical problemsPsychological well-beingGeneral health perceptionsPhysical mobilityRole disability due to emotional problemsSatisfaction with physical conditionSocial activities with friends/relatives
![Page 16: More Practical Metrics for Standardizing Health Outcomes in Effectiveness Research](https://reader035.vdocuments.mx/reader035/viewer/2022081515/56812f58550346895d94e8b2/html5/thumbnails/16.jpg)
16
Content of Widely-Used Patient-Reported Outcome Measures
Source: Adapted from Ware, 1995
Reported health transition
SIP = Sickness Impact Profile (1976)HIE = Health Insurance Experiment surveys (1979)NHP = Nottingham Health Profile (1980)QLI = Quality of Life Index (1981)COOP = Dartmouth Function Charts (1987)DUKE = Duke Health Profile (1990)MOS FWBP = MOS Functioning and Well-Being
Profile (1992)
MOS SF-36 = MOS 36-Item Short-Form Health Survey (1992)
QWB = Quality of Well-Being Scale (1973)EUROQOL = European Quality of Life Index (1990)HUI = Health Utility Index (1996)SF-6D = SF-36 Utility Index (Brazier, 2002)
Psychometric Utility RelatedSIP HIE NHP COOP DUKE MOS
FWBPMOSSF-36
QWB EURO-QOL
HUI SF-6DCONCEPTS
Physical functioning Social functioning Role functioning Psychological distress Health perceptions (general) Pain (bodily) Energy/fatigue Psychological well-being Sleep Cognitive functioning Quality of life
PROMIS = Patient Reported Outcomes Measurement Information System
= Quality of Well-Being Scale (1973)
PROMIS
![Page 17: More Practical Metrics for Standardizing Health Outcomes in Effectiveness Research](https://reader035.vdocuments.mx/reader035/viewer/2022081515/56812f58550346895d94e8b2/html5/thumbnails/17.jpg)
What Do We Need for Comparative Effectiveness Research?
• Outcomes that matter to patients
• Practical measures
• Coverage of a wide range
• Greater precision
• Comparability of scores
• Ease of interpretation17
![Page 18: More Practical Metrics for Standardizing Health Outcomes in Effectiveness Research](https://reader035.vdocuments.mx/reader035/viewer/2022081515/56812f58550346895d94e8b2/html5/thumbnails/18.jpg)
What Do We Need for Comparative Effectiveness Research?
• Outcomes that matter to patients
• Practical measures
• Coverage of a wide range
• Greater precision
• Comparability of scores
• Ease of interpretation 18
“Ceiling Effect”
![Page 19: More Practical Metrics for Standardizing Health Outcomes in Effectiveness Research](https://reader035.vdocuments.mx/reader035/viewer/2022081515/56812f58550346895d94e8b2/html5/thumbnails/19.jpg)
19Ware JE, Jr, et al. Med Care. 2000;38:1173-82.
Skewed 5-Item Headache Pain Measure
“Ceiling Effect”
r = 0.536N = 1016
Dynamic 5-Item HeadachePain Measure
r = 0.938N = 1016
CriterionScore
CriterionScore
A Practical Solution in 1999: Computerized Dynamic Health
Assessment
3 SD units
NoNoDisabilityDisability
IRT/CAT will spawn a new generation of
static tools
![Page 20: More Practical Metrics for Standardizing Health Outcomes in Effectiveness Research](https://reader035.vdocuments.mx/reader035/viewer/2022081515/56812f58550346895d94e8b2/html5/thumbnails/20.jpg)
What Do We Need for Comparative Effectiveness Research?
• Outcomes that matter to patients
• Practical measures
• Coverage of a wide range
• Greater precision
• Comparability of scores
• Ease of interpretation 20
Cri
teri
on
Cri
teri
on
VASVAS
![Page 21: More Practical Metrics for Standardizing Health Outcomes in Effectiveness Research](https://reader035.vdocuments.mx/reader035/viewer/2022081515/56812f58550346895d94e8b2/html5/thumbnails/21.jpg)
What Do We Need for Comparative Effectiveness Research?
• Outcomes that matter to patients
• Practical measures
• Coverage of a wide range
• Greater precision
• Comparability of scores
• Ease of interpretation 21
![Page 22: More Practical Metrics for Standardizing Health Outcomes in Effectiveness Research](https://reader035.vdocuments.mx/reader035/viewer/2022081515/56812f58550346895d94e8b2/html5/thumbnails/22.jpg)
22
Practical Solution in 2000:Cross-Calibration of Headache Pain Disability Measures
Note: Direction of scoring shown with arrows Source: Ware, Bjorner & Kosinski, Medical Care, 2000
Scales 20 30 40 50 60 70
HDI 16 43 73 91 98 100
HIMQ 74 53 31 17 8 2
MIDAS 58 28 5 1 0 0
MSQ 31 53 79 92 96 99
DYNHA-5 (+) 23 32 41 51 58 66
Theta (θ) [Best Possible Estimate]
![Page 23: More Practical Metrics for Standardizing Health Outcomes in Effectiveness Research](https://reader035.vdocuments.mx/reader035/viewer/2022081515/56812f58550346895d94e8b2/html5/thumbnails/23.jpg)
23
52 centimeters = 20.5 inches
and Public-Private Partnerships That Meetthe Needs of Research and Business
We Need the Health Equivalent of a Two-Sided Tape Measure
![Page 24: More Practical Metrics for Standardizing Health Outcomes in Effectiveness Research](https://reader035.vdocuments.mx/reader035/viewer/2022081515/56812f58550346895d94e8b2/html5/thumbnails/24.jpg)
What Do We Need for Comparative Effectiveness Research?
• Outcomes that matter to patients
• Practical measures
• Coverage of a wide range
• Greater precision
• Comparability of scores
• Ease of interpretation24
What do the results mean?
![Page 25: More Practical Metrics for Standardizing Health Outcomes in Effectiveness Research](https://reader035.vdocuments.mx/reader035/viewer/2022081515/56812f58550346895d94e8b2/html5/thumbnails/25.jpg)
25
PRO Validation Must be Comprehensive
• Diagnosis
• Disease severity
• Responders
• Treatments
MeasuresIn
Question
GoldStandard
• Work productivity• Costs of care• Mortality• Self- evaluated
health
OtherMeasures& Methods
• Diagnosis
• Disease severity
• Responders
• Treatments
• Work productivity• Costs of care• Mortality• Self-evaluated
health
GoldStandard
OtherMeasures& Methods
Adapted from: Ware JE, Jr. and Keller SD: Interpreting general health measures, in: Quality of Life and Pharmacoeonomics in Clinical Trials. Philadelphia, PA: Lippincott-Raven Publishers; 1995: Chapter 47.
Causes Consequences
![Page 26: More Practical Metrics for Standardizing Health Outcomes in Effectiveness Research](https://reader035.vdocuments.mx/reader035/viewer/2022081515/56812f58550346895d94e8b2/html5/thumbnails/26.jpg)
26
What Do Differences in Treatment Effectiveness Mean?
30 40 50
Chronic Lung Disease
Physical Component Summary (PCS)
DiabetesType II
Congestive Heart Failure
Average Adult
Asthma After
Rx
Asthma Before
Rx
Treatment
Average Well Adult
50% reduction in disease burden 33% reduction in hospitalization Substantial increase in work productivity Subsequent cost savings
![Page 27: More Practical Metrics for Standardizing Health Outcomes in Effectiveness Research](https://reader035.vdocuments.mx/reader035/viewer/2022081515/56812f58550346895d94e8b2/html5/thumbnails/27.jpg)
Matching Methods to Applications:“Choosing the Right Horse for the Course”
• Population monitoring
• Group-Level outcomes monitoring
• Patient-level measurement/management
27
![Page 28: More Practical Metrics for Standardizing Health Outcomes in Effectiveness Research](https://reader035.vdocuments.mx/reader035/viewer/2022081515/56812f58550346895d94e8b2/html5/thumbnails/28.jpg)
28
1
2
3
4
5
6
7
1
3
5
7
Single-Item
1
2
3
4
5
Multi-ItemScale
PopulationMonitoring
Group-LevelOutcomesMonitoring
Patient-LevelManagement
“Item Pool” (CAT Dynamic)
7
6
Most Functionally Impaired
NoisyIndividual
Classification
Very AccurateIndividual
Classification
Matching Methods to Applications
![Page 29: More Practical Metrics for Standardizing Health Outcomes in Effectiveness Research](https://reader035.vdocuments.mx/reader035/viewer/2022081515/56812f58550346895d94e8b2/html5/thumbnails/29.jpg)
Solutions
• Improved psychometrics (Item response theory – IRT)
• Computerized adaptive testing (CAT) software
• The Internet (and other connectivity)
29
Business Week. November 26, 2001.
![Page 30: More Practical Metrics for Standardizing Health Outcomes in Effectiveness Research](https://reader035.vdocuments.mx/reader035/viewer/2022081515/56812f58550346895d94e8b2/html5/thumbnails/30.jpg)
30
First, Construct Better Metrics
Source: Business Week 11/26/01
• Comprehensive Item “Pools”• IRT Cross Calibration of Items
ADLSIPFIM
Physical Functioning (PF)
PF-10
+ =
NEWPF
% @ Ceiling:
1980 “PF Ruler” > 75% @ Ceiling
1990 “PF Ruler” > 30% @ Ceiling
2008 “PF Ruler” < 3 % @ Ceiling
![Page 31: More Practical Metrics for Standardizing Health Outcomes in Effectiveness Research](https://reader035.vdocuments.mx/reader035/viewer/2022081515/56812f58550346895d94e8b2/html5/thumbnails/31.jpg)
31
RheumatoidArthritis
Physical Function (PF), Mean = 50 Source: Rose M, Bjorner JB, Becker J, Fries JF and Ware JE. Evaluation of a preliminary physical function item bank supported expected advantages of the Patient-Reported Outcomes Measurement Information System (PROMIS). Journal of Clinical Epidemiology, 2008, 61, 17-33.
6.0
5.0
4.0
3.0
2.0
1.0
0 10 20 30 40 50 60 70 80
StandardError
0.75
0.90
0.95
Reliability
PF “Criterion”(Item Bank)
PF-10 (“Static”)
PF-2 (“Static”)
PF CAT-10
PF-1 (“Static”)
Precision Varies Across “Static” and Dynamic Forms and Across Score Levels
![Page 32: More Practical Metrics for Standardizing Health Outcomes in Effectiveness Research](https://reader035.vdocuments.mx/reader035/viewer/2022081515/56812f58550346895d94e8b2/html5/thumbnails/32.jpg)
32
CAT
2nd Solution, Assess Health Dynamically
Patient scores
here
CAT = Computerized Adaptive Testing
![Page 33: More Practical Metrics for Standardizing Health Outcomes in Effectiveness Research](https://reader035.vdocuments.mx/reader035/viewer/2022081515/56812f58550346895d94e8b2/html5/thumbnails/33.jpg)
33
What are the Advantages of Dynamic Assessments?
• More accurate risk screening • Reliable enough to monitor individual
outcomes• Brevity of a short form –
90% reduction in respondent burden• Elimination of “ceiling” & “floor” effects • Can be administered using various data
collection technologies• Markedly reduced data collection costs• Monitor data quality in real time
![Page 34: More Practical Metrics for Standardizing Health Outcomes in Effectiveness Research](https://reader035.vdocuments.mx/reader035/viewer/2022081515/56812f58550346895d94e8b2/html5/thumbnails/34.jpg)
35
Reference – Headache Impact: MS Bayliss, JE Dewey, R Cady etal., A.Study of the Feasibility of Internet Administration of a computerized health survey: The Headache Impact Test (HIT), Quality of Life Research, 2003, 12: 953-961
References – Asthma Control: Nathan RA, Sorkness CA, Kosinski M et al., “Development of the Asthma Control Test: A survey for assessing asthma control. Journal of Allergy and Clinical Immunology. 2004;113: 59-65.
www.amIhealthy.com www.asthmacontroltest.com
/
3rd Solution: The Internet
![Page 35: More Practical Metrics for Standardizing Health Outcomes in Effectiveness Research](https://reader035.vdocuments.mx/reader035/viewer/2022081515/56812f58550346895d94e8b2/html5/thumbnails/35.jpg)
September 15, 2009
More Practical Metrics for Standardizing Health Outcomes in Effectiveness Research ([email protected])
Conclusions
• Patient-reported outcomes (PROs) are very useful
• Standardization of concepts & metrics is enabling comparisons across treatments & settings
• Increasing widespread use proves that more practical tools will be adopted
• Promising technological advances include: item response theory (IRT), computerized adaptive testing (CAT) and Internet-based data capture