measuring the impact of risk-based monitoring · 2020-03-26 · risk based monitoring continues to...
TRANSCRIPT
Tuesday 24March2020
9:00am-10:30 am (Eastern)
Measuring the Impact of Risk-Based Monitoring
Copyright ©2015 TransCelerate BioPharma Inc., All rights reserved. 2
2
Julie Day
• TransCelerate BioPharma Inc.
• Program Director
Jenni Sullivan
TransCelerate BioPharma Inc.
Program Manager
Gloria McHugh
TransCelerate BioPharma Inc.
Project Manager
TransCelerate Team
Please report feedback or technical issues to: [email protected]
Copyright ©2015 TransCelerate BioPharma Inc., All rights reserved. 3
Expert Presenters
3
Samantha Hewlett
• Risk-Based Study Execution Capability Lead, Amgen
• RBM Team Member
Rodrigo Perez Marentes
Clinical Studies Unit Risk-Based Monitoring Business Lead, Sanofi
RBM Team Member
Mary Arnould
Director, Clinical Science Management Operations & RBM Lead, Astellas
RBM Team Member
Copyright ©2015 TransCelerate BioPharma Inc., All rights reserved. 4
Logistics for this Webinar
4
All participants will be muted for this call.
For audio: Connect to audio to listen to presentations via your computer or phone
You can reduce control panel for a better view of
presentation by clicking the orange and red arrow
To submit a question to the presenters:Type your question in the Questions panel and click Send.
NOTE: this webinar may be recorded in whole or in part.
If you do not consent to be recorded, please send a note
using the Questions panel.
Copyright ©2015 TransCelerate BioPharma Inc., All rights reserved. 5
Logistics for this Live Polling
5
To help make this session interactive and facilitate sharing of useful and
appropriate information, we will conduct live polls throughout the session.
Your participation is completely voluntary and all responses will be
anonymous. Answers will be shared only as a percentage of respondents.
Instructions: When the
blue poll question
appears, enter your
response and submit.
Copyright ©2015 TransCelerate BioPharma Inc., All rights reserved. 6
Ground Rules
6
We want to make this discussion helpful and answer as many of your questions as we can, so some quick ground rules:
• Participation is voluntary, as is using TransCelerate assets / tools
– The responsibility for compliance with laws and regulations is owned by solution adopter.
• You don’t have to identify what company you work for
• Things we would ask you not to discuss:
– What vendors / sites / CROs you are using or not using
– Any issues you have with any vendors / sites / CROs
– Your long-term development plans– Anything related to costs
• We can’t answer questions about:– Vendors– Costs of using / implementing TransCelerate assets / tools – Which member companies are using the assets / tools
Copyright ©2015 TransCelerate BioPharma Inc., All rights reserved. 7Copyright ©2017 TransCelerate BioPharma Inc., All rights reserved. * Confidential - NOT FOR DISTRIBUTION *
AGENDA
1) Overview of TransCelerate [Julie Day, TransCelerate ] (5 mins)
2) RBM Metrics: Measuring the Impact of Risk Based Monitoring [Sam Hewlett, Amgen] (15 mins)
3) Member Company Perspectives with RBM Metrics (30 mins)
▪ Challenges in Implementing RBM Metrics-Experience from 2 Member Companies [Mary Arnould, Astellas]
▪ How Amgen Has Evolved Their Metrics [Sam Hewlett, Amgen]
▪ What’s Next for Risk Based Monitoring [Rodrigo Perez, Sanofi]
4) Q&A (focused on today’s webinar metrics topics) [Sam Hewlett, Amgen] (40 mins)
7
1. Overview of
TransCelerate
Julie Day•TransCelerate BioPharma Inc.
•Program Director
Copyright ©2020 TransCelerate BioPharma Inc., All rights reserved. 9
Our Shared Vision:
To improve the health of people around
the world by accelerating and simplifying
the research and development of
innovative new therapies.
TransCelerate:
A Not-for-Profit Entity Created to Foster Collaboration
Copyright ©2020 TransCelerate BioPharma Inc., All rights reserved.
2 1 MEMBERCOMPANIES
Current state of organization
Today
25+including 5
pharmacovigilance initiatives
INITIATIVES
focus on preclinical research
ENHANCING INDUSTRY COLLABORATION
With an effective and proven governance
structure have increased the ease and desire to
collaborate
FACILITATING FUTURE PLATFORM
TRIALS
12+ initiatives deliver solutions that facilitate future platform
trials
BREADTH & DEPTH
Over 60 solutions being delivered across 25+ initiatives, across 3 strategic
priorities
TransCelerate Founded
2012
5 INITIAL INITIATIVES
10 MEMBERCOMPANIES
10
platform to enable data sharing
BioCelerate Founded
2016
Bayer most recent
member
Copyright ©2020 TransCelerate BioPharma Inc., All rights reserved.
The Reach of our Global Membership is Expanding
11
Membership is available to biopharmaceutical research and development organizations
that engage in innovative discovery, development and manufacturing of new medicines*.
* to be eligible for membership, companies must meet
specified eligibility criteria.
Thousands of people have contributed to the design,
development and deployment of TransCelerate solutions
2. RBM Metrics: Measuring the
Impact of Risk Based Monitoring
Samantha Hewlett
Risk-Based Study Execution Capability Lead, Amgen
RBM Team Member
Copyright ©2019 TransCelerate BioPharma Inc., All rights reserved. 13
Timeline of Risk Based Monitoring Initiative
PAST PRESENT FUTURE
2012-2017
Active Phase: Deliver & Measure
• 8 Methodology Framework
Papers
• Multiple Supporting Tools and
Resources for Industry Use
• TransCelerate collects, blinds and
aggregates RBM metrics data
from member companies.
Implementation increases most
significantly from 2015-2016.
2018-2019
Mature Phase: Facilitate Adoption
2012TransCelerate forms. One of its first initiatives is Risk Based
Monitoring, in response to FDA guidance.
• Focus on Facilitating Awareness, Adoption and Best Practice Sharing
• Ongoing High Visibility Engagements across Industry (Healthcare Authorities, Conferences, Site Advocacy Groups, Contract Research Organizations)
• Perception is that RBM is improving quality and efficiency, however quantifying the magnitude of improvement remains challenging
2019TransCelerate Metrics Survey. 18 member
companies are invited to participate in an anonymous survey to better understand
how the value of RBM implementation is defined and assessed.
2020 on …
Risk Based Monitoring continues to
evolve. Multiple factors including
ICH E6, E8 revisions, evolving
quality management systems, and
technology advances have the
potential to impact Risk Based
Monitoring
2020 and beyond…
Evolution
Copyright ©2019 TransCelerate BioPharma Inc., All rights reserved. 14
Collecting RBM Data Quarterly | Methods
• Per FDA suggestion, data was collected from participating
member companies quarterly from 2013-2017, resulting in
>1000 observations.
• Guided by a defined set of RBM metrics, member
companies voluntarily reported data to TransCelerate on
clinical trials where they were implementing RBM.
• Member companies were asked to rate change over time
for each metric as “better,” “worse” or “about the same”
(as compared to each company’s internal baseline
expectation).
• Data was blinded, aggregated and reported by
TransCelerate each quarter.
METRICS
Average number of major/critical audit findings
Percentage of unreported, confirmed SAEs as
compared to total SAEs as discovered through
any method
Number of Significant Protocol Deviations
Average Monitoring (all types) cost per site
Average interval between on-site monitoring visits
per site
Median number of days from issue open to close
Median number of days from patient visit to eCRF
data entry
Median number of days from query open to close
In 2018 & 2019, we have further aggregated the data to analyze risk based monitoring trends and produce cumulative observations.
Copyright ©2019 TransCelerate BioPharma Inc., All rights reserved. 15
Cumulative Data Observations (Relative %)
• Audit Findings: Quality appears to increase
in more mature studies.
• SAE Reporting: Early on in studies, there
appears to be an initial rise in SAE
reporting, but as the study progresses, SAE
reporting appears to improve, overall.
• Significant Protocol Deviations:
Compliance appears to improve during
the study; however, data becomes too
limited in later maturities to prove this.
• Overall Monitoring Cost & On-Site Visit
Interval: Cost and On-site Visit Intervals are
highly correlated.
• Issue Open to Close, eCRF Data Entry,
Query Open to Close: Those implementing
took care to focus on data flow,
recognizing it is critical to success.
• All 8 metrics showed at least 50% “better”
• >70% were “about the same” or “better”
Copyright ©2019 TransCelerate BioPharma Inc., All rights reserved. 16
Understanding Current State | What do RBM metrics look like today?
2019
RBM Metrics Survey
of Member Companies
(n=18)
Objectives:
(1) analyze current use of the original metrics
(2) assess how benefits of RBM are currently measured
(3) determine whether new metrics have been developed by members to further define the value of RBM
Results:
(1) Interpretation of experiences of member companies
(2) Not “one-size-fits-all”
Copyright ©2019 TransCelerate BioPharma Inc., All rights reserved. 17
Understanding Current State2019 Member Company Survey Results
Key Decision Drivers Behind RBM Adoption
Both Quality/Compliance and
Efficiency/Cost are primary
drivers for implementing an
RBM model.
1 Participants could select only 1 response
Copyright ©2019 TransCelerate BioPharma Inc., All rights reserved. 18
Understanding Current State2019 Member Company Survey Results
Types of Trials in Scope of RBM
❖ Implementation of RBM is now seen across
all phases of Interventional Clinical Trials.
❖ 14 companies report utilizing RBM in all
phases of their studies currently. The others
have taken various, staggered
approaches.
❖ Member companies approaches to
implementation have evolved over time
to expand to current state.
14
21 1
0
2
4
6
8
10
12
14
16
All Clinical Trials Phase I/II/III
Only
Phase II/III
Only
None
(not
implemented
RBM)
# of Member Companies
Copyright ©2019 TransCelerate BioPharma Inc., All rights reserved. 19
Understanding Current State2019 Member Company Survey Results
Components Implemented with RBM
• Risk assessments, reduced Site Data
Verification (SDV) and a more remote
approach to analysis and monitoring are
the major components consistently seen
across companies who have adopted a
RBM model.
• The original focus areas of the
TransCelerate RBM model (e.g., site
focused) appear to be well-established.
• The risk-based approach appears to be
spreading to other, related areas (e.g. data
cleaning).
2 Participants could select more than one response
Copyright ©2019 TransCelerate BioPharma Inc., All rights reserved. 20
Understanding Current State2019 Member Company Survey Results
How long did it take to see/measure the impact(s) of RBM
implementation?
1 Participants could select only 1 response
➢ Whilst it appears that RBM contributed
positively to the development process
within 1-3 years of implementation,
few companies are tangibly
measuring its direct impact through
metric analysis, with many are still
developing their metrics and
collecting data.
➢ While quality is a key decision driver
for implementation, organizations
struggle to measure quality directly
and note significant change
management efforts are necessary to
support the process.
Copyright ©2019 TransCelerate BioPharma Inc., All rights reserved. 21
Are the original metrics being used to
evaluate the implementation of RBM?
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Use of RBM Metrics
Currently Use Previously Used Never Used
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Usefullness of RBM Metrics
Extremely Useful Somewhat Useful Neutral Not Very Useful
Are the original metrics
considered useful?
Audience Poll
Question: Has your company implemented
metrics to assess the value of RBM? (select one)
❑ Yes
❑ No
❑ I’m not sure
Audience Poll
Question: How helpful have your company metrics
been in measuring the value of RBM? (select one)
❑ Very helpful
❑ Somewhat helpful
❑ Not very helpful
❑ Not at all helpful
❑ Neutral, a mix, or un-determined at this time
3. Member Company
Perspectives with RBM Metrics
Challenges in Implementing
RBM Metrics-Experience from 2
Member Companies
Mary Arnould
Director, Clinical Science Management Operations & RBM Lead, Astellas
RBM Team Member
Copyright ©2015 TransCelerate BioPharma Inc., All rights reserved. 26
TransCelerate Metrics
Metric Category
Average number of major/critical audit findings
Quality
Percentage of unreported, confirmed SAEs as
compared to total SAEs as discovered through any
method
Number of Significant Protocol Deviations
Median number of days from issue open to close
Cycle TimeMedian number of days from patient visit to eCRF data
entry
Median number of days from query open to close
Average interval between on-site monitoring visits per
site Cost
Average Monitoring (all types) cost per site
Copyright ©2015 TransCelerate BioPharma Inc., All rights reserved. 27
A Tale of Two Companies Implementation of RBM Metrics
Company A
• Large pharma
• Insourced central and site monitoring
• Insourced site monitoring
• Primary driver for implementation was quality
• Implementation was “all in” from outset
• RBM Metrics developed from the outset and tracked quarterly
Company B
• Medium pharma
• Insourced central monitoring
• Outsourced site monitoring
• Primary driver for implementation was quality
• Implementation began with pilots and progressed to full implementation
• RBM Metrics not initiated until full implementation was well underway
Company Characteristics:
Both companies sought to develop all 8 TransCelerate metrics and
developed a non-RBM comparison for each metric
Copyright ©2015 TransCelerate BioPharma Inc., All rights reserved. 28
The “Easy” Metrics
Median number of days from patient visit to eCRF data entry
Median number of days from query open to close
Copyright ©2015 TransCelerate BioPharma Inc., All rights reserved. 29
The “Challenging” Metrics
Average number of major/critical audit findings
• Are all studies audited?
• Have audit plans changed over time?
• Are audits consistently performed across phases and TAs?
Number of Significant Protocol Deviations
• Have categorizations of PDs changed over time?
• Data source may present challenges if PDs collected by vendor partners
• Study specific PD identification or management may impact overall metric
Median number of days from issue open to close
• Data sources may present challenges if collected by vendor partners
Copyright ©2015 TransCelerate BioPharma Inc., All rights reserved. 30
The “Challenging” Metrics
Average interval between on-site monitoring visits per site
• Relies on CTMS data for monitoring visit dates
• Has direct impact on Monitoring Cost
• Helpful to visualize utilization of on-site and remote visits if using adaptive approach
Average Monitoring (all types) cost per site
• Actual costs may not reflect potential cost savings because of difficult to control variables (e.g. Portfolio characteristics, vendor relationships, Increasing cost over time)
• Easier to estimate when all monitoring activities are performed by sponsor
Copyright ©2015 TransCelerate BioPharma Inc., All rights reserved. 31
The “Difficult” Metric
Percentage of unreported, confirmed SAEs as compared to total SAEs as discovered through any method
• Difficulty exists in measuring the unknown
• Relies on effective SDR-what is impact when SDR is performed on less than 100% of data?
• Two examples:
• Responses to question on monitoring visit report related to sufficient SAE reporting
• Measuring SAE reporting time (Site awareness date to reporting date) as a surrogate indicator
Copyright ©2015 TransCelerate BioPharma Inc., All rights reserved. 32
Conclusion
Common challenges:
• Access to the data
• Establishing and maintaining the non-RBM baseline
• Number of non-RBM studies will decrease over time as number of RBM studies increases
• Visualization tool-Tableau, Excel, others?
• Are these the metrics that best reflect the value of RBM to the organization?
Evolution of Amgen
Metrics Over Time
Samantha Hewlett
Risk-Based Study Execution Capability Lead, Amgen
RBM Team Member
Copyright ©2019 TransCelerate BioPharma Inc., All rights reserved. 34
Evolution of Metrics Over Time
I’m looking in generally
the right area
Precise detection
Copyright ©2019 TransCelerate BioPharma Inc., All rights reserved. 35
Evolution of Metrics Over Time
▪ Amgen implemented first version of RBM in 2012
▪ Primarily focussed on reduced SDR/SDV with enhanced data review
▪ Supported by many of the metrics already discussed – data entry, query resolution,
protocol deviations and SDV backlog
▪ Metrics were reviewed bi-weekly by CRAs and drove actions according to pre-defined
guidance
Challenge
▪ Not reproduceable
▪ No ability to trend data
▪ Not flexible
▪ Very granular in nature
Copyright ©2019 TransCelerate BioPharma Inc., All rights reserved. 36
Evolution of Metrics Over Time
Centralized Statistical
Monitoring (CSM) detects
sites with unusual clinical &
operational data and flags
potential issues for GCTMs to
investigate
Central Monitors review
site operational Key Risk
Indicators (KRIs) and
recommend actions to CRAs
Central Monitoring Data Monitoring
Enhanced Data Review Levels of data review, including
ongoing, real-time data review
by Medical Data Reviewers
Site Monitoring
Flexible Monitoring
Schemas based on risk &
site performance
Risk-Based Source
Data Verification and
increased focus on Source
Data Review
Risk ManagementCT-RACT used to identify and manage risks & critical processes
RBM
Copyright ©2019 TransCelerate BioPharma Inc., All rights reserved. 37
Evolution of Metrics Over Time - Conclusion
▪ Internal and external factors WILL continue to change
▪ Sophistication in data analysis is evolving at a rapid rate
▪ New data sources continue to be introduced
▪ We can not afford to be complacent and must continue to ensure that
even with the “need for speed” data integrity is maintained
WHY?
Copyright ©2019 TransCelerate BioPharma Inc., All rights reserved. 38
Evolution of Metrics Over Time - Conclusion
▪ Internal and external factors WILL continue to change
▪ Sophistication in data analysis is evolving at a rapid rate
▪ New data sources continue to be introduced
▪ We can not afford to be complacent and must continue to ensure that
even with the “need for speed” data integrity is maintained
WHY – because patients matter
4. Future of RBM Metrics
Rodrigo Perez Marentes
Clinical Studies Unit Risk-Based Monitoring Business Lead, Sanofi
RBM Team Member
Copyright ©2019 TransCelerate BioPharma Inc., All rights reserved. 40
Current Challenges
<50% of MC use historical controls/baseline data to demonstrate the value of RBM implementation
Measuring RBM value is not necessarily one-size-fits-all. Sponsors prefer fit for purpose methods.
Multiple changes are introduced in trial conduct in parallel – Masks true impact of RBM on each trial.
Limitations to consistently define, measure and benchmark quality in a scalable manner
Challenges to quantifying the value of RBM and other quality-focused initiatives.
The existing measurement processes prevent us from quantifying the benefits of RBM accurately.
Copyright ©2019 TransCelerate BioPharma Inc., All rights reserved. 41
How Are RBM Metrics Evolving?
Proxy Metrics
The original RBM metrics were proxy metrics that measured for quality but did not always directly measure the impact attributable to RBM
Original Metrics
New Metrics
Several MCs have defined new/novel Key Performance Indicators
(KPIs) to measure the value of RBM.
KPIs
Tailor Made Metrics
Best practices appear to include defining measures tailor-made to the chosen RBM model and the individual organization’s operational design.
Other quality
initiatives
The art and
science of
measuring &
demonstrating the benefits of
RBM are still
evolving
Copyright ©2019 TransCelerate BioPharma Inc., All rights reserved. 42
What’s Next For Risk Based Monitoring Metrics?
QTLs
Regulatory Inspections
ICH E6, E8 Revisions
Technology
Guidance needed
Insight & experience needed
Continued evolution
Continued industry communications to share
learningsIncreasing levels of data intelligence & capability
for feedback loops
2020 on… RBM Continues to evolve.
Multiple factors including ICH E6, E8
revisions, evolving quality management
systemes, and technology advances
have the potential to impact RBM.
Copyright ©2019 TransCelerate BioPharma Inc., All rights reserved. 43
These Metric Narratives illustrate the expectationsof RBM’s impact as well as alternative observations
RANK PROCESS COMPLIANCE DATA
Extremely Useful
• Time from data “cut” to Action3
• Ratio of on-site to off-site monitoring visits
• SDR & SDV backlog4
• % queries resolved in 7 days3
• % pages submitted in 7 days3
• Important Protocol Deviation
Incidence3
• Missed Assessments2
• Dosing Deviation
Incidence3
Somewhat Useful
• Reports for centralized monitoring – user statistics to indicate
frequency and duration of use3
• Ratio of Data correction XX days after initial data entry3
• RBM user satisfaction survey3
• Query rates2
• Qualitative interviews with HQ trial teams2
• Survey for use and usefulness of site risk indicator report2
• TMF Compliance2 • SAE/AE rates2
• Query rate (per 1000
data points)3
Neutral• Action item aging3
• External data review status2
• CAPAs close on time (site)3
• CAPA # overall3
• eTMF status2
• Ratio of number of AE emerging
per subject4
• Query aging3
• Ratio of number of AE
emerging per subject4
Not very Useful• On-site vs remote visit ratio: Ratio calculated as number on-
site to remote visits4
• Ratio of Missing data for the
primary endpoint4
• Ratio of Missing data for
the primary endpoint4
KEY PERFORMANCE INDICATORS (KPIs)
Examples of new KPIs for RBM developed by Member Companies. The 9 MCs that responded provided 28 new/novel KPIs and ranked them by usefulness.
1 These are the direct answers from survey respondents, we have not summarized, word-smithed, or tried to categorize them. Some information may be contradictory since it was submitted by different companies. 2 Qualitative Indicator 3 Quantitative Indicator 4 Duplicate
Copyright ©2019 TransCelerate BioPharma Inc., All rights reserved. 44
What’s next for Sanofi?
Sanofi’s Perspective – Future of metrics
➢ Continuous improvement mode - Performance Metrics to be fine tuned and
re-defined.
➢ Set new target of monitoring visit adaptive approach - Shift from “one size
fits all” to “fit for purpose”.
➢ Process simplification for metrics calculation methodology – to be tackled
by department.
➢ Leverage Technology: Automation of performance indicators measurement
– still too manual and not consistent.
➢ Implementation of QTLs – Participation in TransCelerate QTLs workstream.
➢ From a targeted RBM into a more holistic approach of global Risk Based
Quality Management.
Audience Poll
Question: What do you think is THE MOST
IMPORTANT issue facing RBM implementers today? (select one)
❑ Continued technological advancements to support RBM
❑ Inspection Findings feedback from marketing applications
❑ More guidance from Regulatory Authorities
❑ QTLs
❑ Other
Q & A or Panel DiscussionType your questions into the Questions panel on your
GoToWebinar screen, click “Submit”
Please note that Q & A session are not recorded.
Please stay on the line to participate in the live Q & A.
Samantha Hewlett
Risk-Based Study Execution Capability Lead, Amgen
RBM Team Member
Please focus your
questions on today’s
Metrics topic
Audience Poll
How much value did this Webinar provide?
No ValueA Little Value
Moderate Value
A Lot of Value
Visit us, for more information:www.TransCelerateBioPharmaInc.com
Watch our “About Us” Video
Sign up for our Newsletter, Accelerate to Innovate
@TransCelerat
e
TransCelerate
BioPharma Inc.
Thank you