multiple criteria hta into practice local hta program - multi... · 1 presenter: paule poulin, phd...
TRANSCRIPT
![Page 1: Multiple Criteria HTA Into Practice Local HTA Program - Multi... · 1 Presenter: Paule Poulin, PhD Authors: P. Poulin, L. Austen et al., Health Technology & Innovation Department](https://reader035.vdocuments.mx/reader035/viewer/2022071107/5fe231ca1aa48910ef060c6f/html5/thumbnails/1.jpg)
1
Presenter: Paule Poulin, PhDAuthors: P. Poulin, L. Austen et al.,
Health Technology & InnovationDepartment of Surgery
Calgary Research Portfolio
Multiple CriteriaHTA Into Practice
Local HTA Program
CADTH Symposium April 3-5, 2011Vancouver, BC
InnovationCREATE
OR DIE!
![Page 2: Multiple Criteria HTA Into Practice Local HTA Program - Multi... · 1 Presenter: Paule Poulin, PhD Authors: P. Poulin, L. Austen et al., Health Technology & Innovation Department](https://reader035.vdocuments.mx/reader035/viewer/2022071107/5fe231ca1aa48910ef060c6f/html5/thumbnails/2.jpg)
2
Objectives:
1. Local HTA Program Overview
2. Multiple Criteria for Technology Evaluation
3. Lessons Learned
![Page 3: Multiple Criteria HTA Into Practice Local HTA Program - Multi... · 1 Presenter: Paule Poulin, PhD Authors: P. Poulin, L. Austen et al., Health Technology & Innovation Department](https://reader035.vdocuments.mx/reader035/viewer/2022071107/5fe231ca1aa48910ef060c6f/html5/thumbnails/3.jpg)
3
Objectives:
1. Local HTA Program Overview
2. Multiple Criteria for Technology Evaluation
3. Lessons Learned
![Page 4: Multiple Criteria HTA Into Practice Local HTA Program - Multi... · 1 Presenter: Paule Poulin, PhD Authors: P. Poulin, L. Austen et al., Health Technology & Innovation Department](https://reader035.vdocuments.mx/reader035/viewer/2022071107/5fe231ca1aa48910ef060c6f/html5/thumbnails/4.jpg)
4
Local decision-makers need to consider
• Local priority-setting• Local operations• Local population health needs• Local alternatives• Local presence of trained personnel• Local infrastructure impact• Local budget impact……..• As well as HTAs by HTA agencies
HTA into Practice“Local HTA Decision-Support Program”
![Page 5: Multiple Criteria HTA Into Practice Local HTA Program - Multi... · 1 Presenter: Paule Poulin, PhD Authors: P. Poulin, L. Austen et al., Health Technology & Innovation Department](https://reader035.vdocuments.mx/reader035/viewer/2022071107/5fe231ca1aa48910ef060c6f/html5/thumbnails/5.jpg)
5
SIGNIFICANT CHANGEUncertainties about clinical safety, effectiveness
and/or cost or resources impact
Local HTA Committeeevaluation
MINOR CHANGENo significant clinical, contractual,
cost or resource issues
Referredfor further assessment
Forms to gather information
+ +
Clinical, Financial, or Economic Impact
Forms E-G
TechnologyRequest
Form A
Support of Technology Request
Form B
Contract-Costing Check
Form C
Local HTACheck
Form D
Approvedfor purchase/implementation
![Page 6: Multiple Criteria HTA Into Practice Local HTA Program - Multi... · 1 Presenter: Paule Poulin, PhD Authors: P. Poulin, L. Austen et al., Health Technology & Innovation Department](https://reader035.vdocuments.mx/reader035/viewer/2022071107/5fe231ca1aa48910ef060c6f/html5/thumbnails/6.jpg)
6
SIGNIFICANT CHANGEUncertainties about clinical safety, effectiveness
and/or cost or resources impact
Referredfor further assessment
Tools for Decision Making
RecommendationMade by Local HTA Committee
Evaluation Criteria
Tool 2
Evaluation Worksheet
Tool 3Decision
Made by Executive Committee
MINOR CHANGENo significant clinical, contractual, cost or
resource issues
ApprovedFor purchase/implementation
Screening Guide
Tool 1
Recommendation and Decision Guide
Tool 4
Local HTA Committee evaluation
Prioritization Tool
Tool 5
![Page 7: Multiple Criteria HTA Into Practice Local HTA Program - Multi... · 1 Presenter: Paule Poulin, PhD Authors: P. Poulin, L. Austen et al., Health Technology & Innovation Department](https://reader035.vdocuments.mx/reader035/viewer/2022071107/5fe231ca1aa48910ef060c6f/html5/thumbnails/7.jpg)
7
Objectives:
1. Local HTA Program Overview
2. Multiple Criteria for Technology Evaluation
3. Lessons Learned
![Page 8: Multiple Criteria HTA Into Practice Local HTA Program - Multi... · 1 Presenter: Paule Poulin, PhD Authors: P. Poulin, L. Austen et al., Health Technology & Innovation Department](https://reader035.vdocuments.mx/reader035/viewer/2022071107/5fe231ca1aa48910ef060c6f/html5/thumbnails/8.jpg)
8
Criteria Development (supported by CADTH grant)
Criteria list from literature
Criteria revision by selected group
Group retreat (criteria revision)
Member survey (criteria ranking)
Final List of Criteria
Revised
Forms
Revised
Tools
Criteria cross-checked withinformation-gathering Forms
Criteria used to formalizeTechnology Evaluation and Decision
Tools
Mod
ified
Del
phi P
roce
ss
![Page 9: Multiple Criteria HTA Into Practice Local HTA Program - Multi... · 1 Presenter: Paule Poulin, PhD Authors: P. Poulin, L. Austen et al., Health Technology & Innovation Department](https://reader035.vdocuments.mx/reader035/viewer/2022071107/5fe231ca1aa48910ef060c6f/html5/thumbnails/9.jpg)
9
Tool 2: Evaluation Criteria
Domain (5)
Health Gain
Service Delivery
Strategic Fit
Innovation
Financial
![Page 10: Multiple Criteria HTA Into Practice Local HTA Program - Multi... · 1 Presenter: Paule Poulin, PhD Authors: P. Poulin, L. Austen et al., Health Technology & Innovation Department](https://reader035.vdocuments.mx/reader035/viewer/2022071107/5fe231ca1aa48910ef060c6f/html5/thumbnails/10.jpg)
10
Domain (5)
Health Gain
Service Delivery
Strategic Fit
Innovation
Financial
Tool 2: Evaluation Criteria
![Page 11: Multiple Criteria HTA Into Practice Local HTA Program - Multi... · 1 Presenter: Paule Poulin, PhD Authors: P. Poulin, L. Austen et al., Health Technology & Innovation Department](https://reader035.vdocuments.mx/reader035/viewer/2022071107/5fe231ca1aa48910ef060c6f/html5/thumbnails/11.jpg)
11
Domain (5)
Health Gain
Service Delivery
Strategic Fit
Innovation
Financial
Tool 2: Evaluation Criteria
![Page 12: Multiple Criteria HTA Into Practice Local HTA Program - Multi... · 1 Presenter: Paule Poulin, PhD Authors: P. Poulin, L. Austen et al., Health Technology & Innovation Department](https://reader035.vdocuments.mx/reader035/viewer/2022071107/5fe231ca1aa48910ef060c6f/html5/thumbnails/12.jpg)
12
Domain (5)
Health Gain
Service Delivery
Strategic Fit
Innovation
Financial
Tool 2: Evaluation Criteria
“CREATE” OR
“DIE”
![Page 13: Multiple Criteria HTA Into Practice Local HTA Program - Multi... · 1 Presenter: Paule Poulin, PhD Authors: P. Poulin, L. Austen et al., Health Technology & Innovation Department](https://reader035.vdocuments.mx/reader035/viewer/2022071107/5fe231ca1aa48910ef060c6f/html5/thumbnails/13.jpg)
13
Domain (5)
Health Gain
Service Delivery
Strategic Fit
Innovation
Financial
Tool 2: Evaluation Criteria
![Page 14: Multiple Criteria HTA Into Practice Local HTA Program - Multi... · 1 Presenter: Paule Poulin, PhD Authors: P. Poulin, L. Austen et al., Health Technology & Innovation Department](https://reader035.vdocuments.mx/reader035/viewer/2022071107/5fe231ca1aa48910ef060c6f/html5/thumbnails/14.jpg)
14
Domain (5) Criteria (12) Example Clarifying Questions
Health Gain Efficacy (patient) • evidence for health gain?
Population Health • addresses burden of disease?
Standard of Care • new Standard of Care?
Service Delivery Safety • as safe as current practice?
Training • requires training or credentialing?
Access • improves access to the under-served?
Service Coordination • impact on other health services?
Strategic Fit Strategic Fit • aligned with strategic goals?
Innovation Knowledge & Research • knowledge generation or transfer?
Financial Cost • start-up and on-going costs?
Sustainability • can we afford it ?
Economic Analysis • cost-effective?
Tool 2: Evaluation Criteria
![Page 15: Multiple Criteria HTA Into Practice Local HTA Program - Multi... · 1 Presenter: Paule Poulin, PhD Authors: P. Poulin, L. Austen et al., Health Technology & Innovation Department](https://reader035.vdocuments.mx/reader035/viewer/2022071107/5fe231ca1aa48910ef060c6f/html5/thumbnails/15.jpg)
15
SIGNIFICANT CHANGEUncertainties about clinical safety, effectiveness
and/or cost or resources impact
Referredfor further assessment
Tools 2 & 3: Evaluation Criteria and Worksheet
RecommendationMade by Local HTA Committee
Evaluation Criteria
Tool 2
Evaluation Worksheet
Tool 3Decision
Made by Executive Committee
MINOR CHANGENo significant clinical, contractual, cost or
resource issues
ApprovedFor purchase/implementation
Screening Guide
Tool 1
Local HTA Committee evaluation
![Page 16: Multiple Criteria HTA Into Practice Local HTA Program - Multi... · 1 Presenter: Paule Poulin, PhD Authors: P. Poulin, L. Austen et al., Health Technology & Innovation Department](https://reader035.vdocuments.mx/reader035/viewer/2022071107/5fe231ca1aa48910ef060c6f/html5/thumbnails/16.jpg)
16
For each criterion:
A. Score the QUALITY and COMPLETENESS of the information provided by the applicant: Adequate, Inadequate, NA
B. Score the SIGNIFICANCE and IMPACT of the technology on a scale of 0-5
C. Make a RECOMMENDATION
Tool 3: Evaluation Worksheet
![Page 17: Multiple Criteria HTA Into Practice Local HTA Program - Multi... · 1 Presenter: Paule Poulin, PhD Authors: P. Poulin, L. Austen et al., Health Technology & Innovation Department](https://reader035.vdocuments.mx/reader035/viewer/2022071107/5fe231ca1aa48910ef060c6f/html5/thumbnails/17.jpg)
17
Tool 3A: “ Device X” – Quality and Completeness of Information
Domain Criteria Example Clarifying Questions SCORE
Health Gain Efficacy (patient) • Evidence for efficacy? Indications for use? Inadequate
Population Health • Adequate projections of prevalence? Adequate
Standard of Care • Potential for new standard of care? Inadequate
Service Delivery Safety • Complications and risks addressed? Inadequate
Training • Training implications described? Adequate
Access • Improves access? NA
Service Coordination • Impact on other health services described? Adequate
Strategic Fit Strategic Fit • Aligned with strategic goals? Adequate
Innovation Knowledge & Research
• Innovative characteristics described?• Outcome measures described?
Inadequate
Financial Cost • Information on resources and impact complete? Adequate
Sustainability
Economic Analysis • Evidence to support cost-benefits? Adequate
![Page 18: Multiple Criteria HTA Into Practice Local HTA Program - Multi... · 1 Presenter: Paule Poulin, PhD Authors: P. Poulin, L. Austen et al., Health Technology & Innovation Department](https://reader035.vdocuments.mx/reader035/viewer/2022071107/5fe231ca1aa48910ef060c6f/html5/thumbnails/18.jpg)
18
For each criterion:
A. Score the QUALITY and COMPLETENESS of the information provided by the applicant: Adequate, Inadequate, NA
B. Score the SIGNIFICANCE and IMPACT of the technology on a scale of 0-5
C. Make a RECOMMENDATION
Tool 3: Evaluation Worksheet
![Page 19: Multiple Criteria HTA Into Practice Local HTA Program - Multi... · 1 Presenter: Paule Poulin, PhD Authors: P. Poulin, L. Austen et al., Health Technology & Innovation Department](https://reader035.vdocuments.mx/reader035/viewer/2022071107/5fe231ca1aa48910ef060c6f/html5/thumbnails/19.jpg)
19
Domain (5) Criteria (12) Example Clarifying Questions
Health Gain Efficacy (patient) • evidence for health gain?
Population Health • addresses burden of disease?
Standard of Care • new Standard of Care?
Service Delivery Safety • as safe as current practice?
Training • requires training or credentialing?
Access • improves access to the under-served?
Service Coordination • impact on other health services?
Strategic Fit Strategic Fit • aligned with strategic goals?
Innovation Knowledge & Research • knowledge generation or transfer?
Financial Cost • start-up and on-going costs?
Sustainability • can we afford it ?
Economic Analysis • cost-effective?
Tool 2: Evaluation Criteria
![Page 20: Multiple Criteria HTA Into Practice Local HTA Program - Multi... · 1 Presenter: Paule Poulin, PhD Authors: P. Poulin, L. Austen et al., Health Technology & Innovation Department](https://reader035.vdocuments.mx/reader035/viewer/2022071107/5fe231ca1aa48910ef060c6f/html5/thumbnails/20.jpg)
20
Tool 2: Criteria - Scoring Significance and Impact
CRITERIA 0 POINTS 1 POINT 3 POINTS 5 POINTS
Efficacy No improvement in patient health gain compared with current practices
Minimal improvement
Moderate improvement
Vast improvement
Population Health
Addresses a condition with very low prevalence (rate/100,000 < 1)
Low prevalence (rate/100,000 is 1-10)
Moderate prevalence (rate/100,000 is 10-1000)
High prevalence (rate/100,000 is 1,000-10,000)
Standard of Care
Does not represent Standard of Care in Alberta
Represents Standard of Care in some health regions in Alberta
Represents Standard of Care in most regions in Alberta
Represents a new Standard of Care in our region or Alberta
![Page 21: Multiple Criteria HTA Into Practice Local HTA Program - Multi... · 1 Presenter: Paule Poulin, PhD Authors: P. Poulin, L. Austen et al., Health Technology & Innovation Department](https://reader035.vdocuments.mx/reader035/viewer/2022071107/5fe231ca1aa48910ef060c6f/html5/thumbnails/21.jpg)
21
Tool 3B: “ Device X” – Scoring Significance and Impact
POTENTIAL IMPROVEMENT
Domain Criteria None: 0 Minimal: 1 Moderate: 3 Vast: 5
Health Gain Efficacy (patient) X
Population Health X
Standard of Care X
Service Delivery Safety X
Training (vast to minimal) X
Access X
Service Coordination X
Strategic Fit Strategic Fit X
Innovation Knowledge & Research X
Outcome Measures (quality) X
Financial Cost (viability) X
Sustainability
Economic Analysis X
![Page 22: Multiple Criteria HTA Into Practice Local HTA Program - Multi... · 1 Presenter: Paule Poulin, PhD Authors: P. Poulin, L. Austen et al., Health Technology & Innovation Department](https://reader035.vdocuments.mx/reader035/viewer/2022071107/5fe231ca1aa48910ef060c6f/html5/thumbnails/22.jpg)
22
SIGNIFICANT CHANGEUncertainties about clinical safety, effectiveness
and/or cost or resources impact
Referredfor further assessment
Tool 4: Recommendation and Decision Guide
RecommendationMade by Local HTA Committee
Evaluation Criteria
Tool 2
Evaluation Worksheet
Tool 3Decision
Made by Executive Committee
MINOR CHANGENo significant clinical, contractual, cost or
resource issues
ApprovedFor purchase/implementation
Screening Guide
Tool 1
Local HTA Committee evaluation
Recommendation and Decision Guide
Tool 4
![Page 23: Multiple Criteria HTA Into Practice Local HTA Program - Multi... · 1 Presenter: Paule Poulin, PhD Authors: P. Poulin, L. Austen et al., Health Technology & Innovation Department](https://reader035.vdocuments.mx/reader035/viewer/2022071107/5fe231ca1aa48910ef060c6f/html5/thumbnails/23.jpg)
23
Tool 4: Recommendation and Decision Guide
DecisionMade by Executive Committee
1. Rejected (No)
2. Approved (Yes)
3. Conditional (Yes but…a. Single caseb. Clinical auditc. Other (e.g. training)
4. Research Usea. Clinical trialb. Support of project
5. Referred
Recommendation and Decision Guide
Tool 4
RecommendationMade by Local HTA Committee
C r
i t e
r i
a
![Page 24: Multiple Criteria HTA Into Practice Local HTA Program - Multi... · 1 Presenter: Paule Poulin, PhD Authors: P. Poulin, L. Austen et al., Health Technology & Innovation Department](https://reader035.vdocuments.mx/reader035/viewer/2022071107/5fe231ca1aa48910ef060c6f/html5/thumbnails/24.jpg)
24
Tool 4: Recommendation and Decision Guide
RECOMMENDATIONOR DECISION
CRITERIA & RATIONALE(e.g., may include)
1. Rejected• Negative, poor, or no data on efficacy• Insufficient evidence of safety• Decreases or worsens service delivery
2. Approved • Efficacy and Safety well established• Enhanced population health is likely• Strategic fit is strong• Will likely improve service delivery• Financial impact is the same or better than current
practice (Cost Neutral)• Cost-effectiveness is likely the same or better than
current practice
![Page 25: Multiple Criteria HTA Into Practice Local HTA Program - Multi... · 1 Presenter: Paule Poulin, PhD Authors: P. Poulin, L. Austen et al., Health Technology & Innovation Department](https://reader035.vdocuments.mx/reader035/viewer/2022071107/5fe231ca1aa48910ef060c6f/html5/thumbnails/25.jpg)
25
Tool 4: Recommendation and Decision Guide
RECOMMENDATIONOR DECISION
CRITERIA & RATIONALE
3a. Conditional –Single Case
• Urgent/emergent approval of a single case where options are limited
3b. Conditional –Clinical Audit
• Evidence promising for efficacy, but limited for effectiveness
• Good evidence for safety• Cost-effectiveness may be uncertain• Advantage over current practice needs to be established• Cost needs to be established
3c. Conditional –Other
• May require new training protocols• Patient population may need to be defined
![Page 26: Multiple Criteria HTA Into Practice Local HTA Program - Multi... · 1 Presenter: Paule Poulin, PhD Authors: P. Poulin, L. Austen et al., Health Technology & Innovation Department](https://reader035.vdocuments.mx/reader035/viewer/2022071107/5fe231ca1aa48910ef060c6f/html5/thumbnails/26.jpg)
26
Tool 4: Recommendation and Decision Guide
RECOMMENDATIONOR DECISION
CRITERIA & RATIONALE
4a. Research Use –Clinical Trial
• Efficacy uncertain or controversial• Safety uncertain• Population health is uncertain• Effect on service delivery is uncertain• May be innovative• Clinical trials protocols must be observed
4b. Research Use –Support of Project
• Innovative project proposed by applicant• Funding and research protocols in place• Acceptable use of resources
5. Referred • Technology has potential impact beyond the scope of surgery
• Working group required
![Page 27: Multiple Criteria HTA Into Practice Local HTA Program - Multi... · 1 Presenter: Paule Poulin, PhD Authors: P. Poulin, L. Austen et al., Health Technology & Innovation Department](https://reader035.vdocuments.mx/reader035/viewer/2022071107/5fe231ca1aa48910ef060c6f/html5/thumbnails/27.jpg)
27
Tool 4: “Device X” - Decision
RECOMMENDATIONOR DECISION
CRITERIA & RATIONALE
4a. Research Use –Clinical Trial
• Efficacy uncertain or controversial• Safety uncertain• Population health is uncertain• Effect on service delivery is uncertain• May be innovative• Clinical trials protocols must be observed
![Page 28: Multiple Criteria HTA Into Practice Local HTA Program - Multi... · 1 Presenter: Paule Poulin, PhD Authors: P. Poulin, L. Austen et al., Health Technology & Innovation Department](https://reader035.vdocuments.mx/reader035/viewer/2022071107/5fe231ca1aa48910ef060c6f/html5/thumbnails/28.jpg)
28
Prioritize Technologies for funding or purchaseRecommendationsRecommendationsRecommendationsRecommendations
Prioritize Technologies for funding or purchase
DecisionsDecisionsDecisionsDecisions
Prioritization Tool
Tool 5
Proposal forTechnology OneTechnology OneTechnology OneTechnology One
Proposal forTechnology TwoTechnology TwoTechnology TwoTechnology Two
Proposal forTechnology ThreeTechnology ThreeTechnology ThreeTechnology Three
Tool 5: Prioritization Tool
![Page 29: Multiple Criteria HTA Into Practice Local HTA Program - Multi... · 1 Presenter: Paule Poulin, PhD Authors: P. Poulin, L. Austen et al., Health Technology & Innovation Department](https://reader035.vdocuments.mx/reader035/viewer/2022071107/5fe231ca1aa48910ef060c6f/html5/thumbnails/29.jpg)
29
Step 1. Screening Guide
Step 2. Criteria Review
Step 3. Criteria Weighting
Step 4. Criteria Rating Scales
Step 5. Technology Scoring
Step 6. Technology Ranking
Step 7. Additional Checks
Adapted from Drs. Craig Mitton and Cam Donaldson
Tool 5: Prioritization Tool
![Page 30: Multiple Criteria HTA Into Practice Local HTA Program - Multi... · 1 Presenter: Paule Poulin, PhD Authors: P. Poulin, L. Austen et al., Health Technology & Innovation Department](https://reader035.vdocuments.mx/reader035/viewer/2022071107/5fe231ca1aa48910ef060c6f/html5/thumbnails/30.jpg)
30
Objectives:
1. Local HTA Program Overview
2. Multiple Criteria for Technology Evaluation
3. Lessons Learned
![Page 31: Multiple Criteria HTA Into Practice Local HTA Program - Multi... · 1 Presenter: Paule Poulin, PhD Authors: P. Poulin, L. Austen et al., Health Technology & Innovation Department](https://reader035.vdocuments.mx/reader035/viewer/2022071107/5fe231ca1aa48910ef060c6f/html5/thumbnails/31.jpg)
31
Lessons Learned:
WEAKNESSES
•Criteria have not been weighted
•Prioritization Tool has not been tested
•Patient interests not well represented
•Administrators and practitioners sometimes resistant to change
![Page 32: Multiple Criteria HTA Into Practice Local HTA Program - Multi... · 1 Presenter: Paule Poulin, PhD Authors: P. Poulin, L. Austen et al., Health Technology & Innovation Department](https://reader035.vdocuments.mx/reader035/viewer/2022071107/5fe231ca1aa48910ef060c6f/html5/thumbnails/32.jpg)
32
Lessons Learned:
STRENGTHS
•Bottom-Up approach – User develop ensures buy-in
•Reviewers satisfied with criteria and technology evaluation tools
•Provides a systematic, consistent and transparent approach to technology evaluation and decision-making
![Page 33: Multiple Criteria HTA Into Practice Local HTA Program - Multi... · 1 Presenter: Paule Poulin, PhD Authors: P. Poulin, L. Austen et al., Health Technology & Innovation Department](https://reader035.vdocuments.mx/reader035/viewer/2022071107/5fe231ca1aa48910ef060c6f/html5/thumbnails/33.jpg)
33
Acknowledgements
Supported by:• Canadian Agency for Drugs and Technology in Health• The Department of Surgery & Surgical Services – Calgary• The Calgary Health Research Portfolio – Calgary• Surgery Clinical Network HTAI - AHS
Participating Departments from the former Calgary H ealth Region:• Surgery & Surgical Services• Anesthesia • Cardiac Sciences• Critical Care Medicine• Internal Medicine• Professional Practice & Development• Medical Device Safety & Risk Management• Pharmacy• Physician Leadership Portfolio, and the • Calgary Health Research Portfolio
![Page 34: Multiple Criteria HTA Into Practice Local HTA Program - Multi... · 1 Presenter: Paule Poulin, PhD Authors: P. Poulin, L. Austen et al., Health Technology & Innovation Department](https://reader035.vdocuments.mx/reader035/viewer/2022071107/5fe231ca1aa48910ef060c6f/html5/thumbnails/34.jpg)
34
Multiple CriteriaHTA Into Practice
Local HTA Program
InnovationCREATE
OR DIE!
Contact information:
Dr. Paule Poulin
Alberta Health Services - Calgary
(403) 944-1652