requisiti formali e criteri di valutazione€¦ · •the z zlinked evidene approah involves the...

Centro Regionale Biomarcatori

Requisiti formali e criteri di valutazione per l’utilizzo clinico di un test

diagnostico in vitro

Ospedale SS Giovanni e Paolo, Venezia, Aulss3 Serenissima

Massimo Gion

Criteri di valutazione per introdurre un biomarcatore nella pratica clinica

Tre successivi livelli di valutazione sono stati riconosciuti dalla comunità scientifica come requisiti per introdurre un biomarcatore nella pratica clinica:

• Validità Analitica

• Validità Clinica

• Utilità Clinica

Validità Analitica

• La Validità Analitica si riferisce all’accuratezza con la quale un dato biomarcatore è identificato da un dato test di laboratorio

• Riguarda i requisiti tecnici e la performance analitica (sensibilità e specificità analitiche, esattezza, accuratezza, precisione) del test di laboratorio considerato

Validità Clinica

• La Validità Clinica si riferisce all’accuratezza con cui un dato test è associato:

• a una particolare condizione clinica (sensibilità e specificità diagnostica, valore predittivo positivo negativo), oppure …

• … a un particolare esito clinico (prognosi, risposta a un trattamento, …)

Utilità Clinica

• L’Utilità Clinica si riferisce ai rischi e ai benefici che risultano dall’utilizzo del test

• La determinazione dell’Utilità Clinica richiede la valutazione degli esiti medici e sociali associati con la esecuzione del test e con i conseguenti interventi eseguiti sia in caso di valori positivi che negativi

• L’Utilità Clinica implica che esistono forti evidenze per sostenere che l’uso del biomarcatore produce per il paziente esiti migliori che se il test non fosse disponibile

• Un test non può avere Utilità Clinica se non ha Validità Analitica e Validità Clinica, ma ….

… Validità Analitica e Validità Clinica da sole non sono sufficienti per introdurre un test nella pratica clinica

Validità Analitica e Validità Clinica vs. Utilità Clinica

I requisiti formali richiesti dagli Enti Regolatori per immettere un IVD sul mercato garantiscano che l’IVD sia conforme a criteri di valutazione basati sulla utilità clinica?

Criteri di valutazione per introdurre un biomarcatore nella pratica clinica

La normativa vigente

• DIRETTIVA 98/79/CE DEL PARLAMENTO EUROPEO E DEL CONSIGLIO del 27 ottobre 1998 relativa ai dispositivi medico-diagnostici in vitro

• DECRETO LGS. 8 SETTEMBRE 2000, N.332, emendato con lD. lgs. 25.01.2010, n.37 – Recepimento Direttiva 2007/47/CE attuazione della direttiva 98/79/CE relativa ai dispositivi medico-diagnostici in vitro

ALLEGATO I – REQUISITI ESSENZIALI

A. REQUISITI GENERALI – I dispositivi … devono fornire le prestazioni previste dal fabbricante … in

termini di sensibilità analitica, sensibilità diagnostica, specificità analitica, specificità diagnostica, esattezza, ripetibilità, riproducibilità, … interferenze note e limiti di rilevazione

B. REQUISITI RELATIVI ALLA PROGETTAZIONE E ALLA FABBRICAZIONE – Caratteristiche chimiche e fisiche – Infezione e contaminazione microbica – Caratteristiche relative alla fabbricazione e all'ambiente – Protezione contro le radiazioni – Informazioni fornite dal fabbricante (obblighi per l’etichettatura e le

informazioni da inserire nel manuale/istruzioni per l’uso)

Non si applica ai dispositivi fabbricati ed utilizzati unicamente nell'ambito della stessa struttura sanitaria (Test in house)

Nuovo Regolamento Europeo – IVDR

• REGOLAMENTO (UE) 2017/746 DEL PARLAMENTO EUROPEO E DEL CONSIGLIO del 5 aprile 2017 relativo ai dispositivi medico-diagnostici in vitro e che abroga la direttiva 98/79/CE

ALLEGATO XIII - VALUTAZIONE DELLE PRESTAZIONI, STUDI DELLE PRESTAZIONI

1. VALUTAZIONE DELLE PRESTAZIONI (7 pg) – Piano di valutazione delle prestazioni (13 punti) – Dimostrazione della validità scientifica nonché delle prestazioni

analitiche e della prestazione clinica (12 punti) – Evidenze cliniche e rapporto di valutazione delle prestazioni (8 punti)

2. STUDI DELLA PRESTAZIONE CLINICA – Scopo degli studi della prestazione clinica – Considerazioni etiche sugli studi della prestazione clinica – Metodi per gli studi della prestazione clinica (29 punti)

Si applica anche ai dispositivi fabbricati ed utilizzati unicamente nell'ambito della stessa struttura sanitaria (Test in house).

Implementation Timelines – Reality

Implementation Timelines – Reality

26 November 2019

• Today marks the half-way point of the EU IVD Regulation’s (IVDR) five-year transition period.

Winding down to the EU IVD and Medical Devices Regulations Deadlines: The Finish Lines in Sight?

MedTech Europe is the European trade association for the medical technology industry including diagnostics, medical devices and digital health.

• 85% of over 50,000 IVDs will need certification by Notified Bodies in the next 30 months (more than 330 IVDs per week).

• Only about 10 organizations have applied so far to be Notified Bodies under that Regulation (only two have received their license to operate under the Regulation).

• We remain seriously held back by the slow and piecemeal implementation of the new regulatory framework.

Winding down to the EU IVD and Medical Devices Regulations Deadlines: The Finish Lines in Sight?

I requisiti formali richiesti dagli Enti Regolatori per immettere un IVD sul mercato NON garantiscono che l’IVD sia conforme a criteri di valutazione basati sulla utilità clinica.

Criteri di valutazione per introdurre un biomarcatore nella pratica clinica

I requisiti formali richiesti dagli Enti Regolatori per immettere un IVD sul mercato NON garantiscono che l’IVD sia conforme a criteri di valutazione basati sulla utilità clinica.

Clinici e decisori devono basare le proprie scelte sulle evidenze prodotte dalla ricerca.

Criteri di valutazione per introdurre un biomarcatore nella pratica clinica

Evidence for clinical utility

Patient outcomes in randomized comparisons of diagnostic tests are still the ultimate judge (1).

• (1) Siontis CK et al, Journal of Clinical Epidemiology 69 (2016) 248e268 - Department of Medicine College of Medicine Mayo Clinic Rochester, MN 45110, USA)

Randomized trials?

Evaluation of clinical study The issue of the study design

(Ferrante di Ruffano L, J Clinical Epidemiol, 2012)

• There are very few randomized trials of diagnostic/prognostic/predictive tests, in general, (in the range of few hundreds (0.1%) of all randomized trials performed on all interventions, a number that now exceeds half a million).

• Those involving biomarkers are only a small fraction of them.

Clinici e decisori devono fare ricorso a un grado più basso di evidenza per basare le proprie scelte sulla selezione e l’utilizzo dei test.

Studi sull’utilità clinica dei biomarcatori Oltre i trial randomizzati

Clinici e decisori devono fare ricorso a un grado più basso di evidenza per basare le proprie scelte sulla selezione e l’utilizzo dei test.

Studi sull’utilità clinica dei biomarcatori Oltre i trial randomizzati

• Selezione: criteri di analisi della performance

• Utilizzo: criteri di analisi della utilità clinica

Studi sull’utilità clinica dei biomarcatori Oltre i trial randomizzati

Selezione dei test

Draw the pathway

• Step 1. Define intended benefits

• Step 2. Map current practice

• Step 3. Propose test role

• Step 4. Link clinical performance requirements to intended benefits

• Step 5. Set minimum acceptable clinical performance levels

(Test Evaluation Working Group of the EFCCLM, Annals Clin. Bioch. 2019)

Step 1

Step 2

Step 3

Step 4

Step 5

(Test Evaluation Working Group of the EFCCLM, Annals Clin. Bioch. 2019)

Step 1. Define intended benefits

• What patient or other benefits do you hope to

achieve by using the biomarker

Step 1

(Test Evaluation Working Group of the EFCCLM, Annals Clin. Bioch. 2019)

Step 2. Map current practice

• What patient group will be tested?

• What test/s do they currently receive to detect the

target condition?

• What key actions are based on the results (further

tests, treatments)?

• What are the benefits and harms of these actions?

Step 2

Step 3. Propose test role

Remap the clinical pathway with the biomarker as the new test:

How will the biomarker be positioned to alter the current clinical pathway?

• add-on: positioned with or after existing test

• triage: positioned before existing test

• replacement: positioned to replace existing test

• new pathway: created if there is no existing test

Step 3

Step 4 Link clinical performance

requirements to intended benefits

Step 5 Set minimum acceptable

clinical performance levels

(Test Evaluation Working Group of the EFCCLM, Annals Clin. Bioch. 2019)

Stop

+

(Test Evaluation Working Group of the EFCCLM, Annals Clin. Bioch. 2019)

Step 4 Link clinical performance

requirements to intended benefits

Step 5 Set minimum acceptable

clinical performance levels

Stop

Step 3

(Test Evaluation Working Group of the EFCCLM, Annals Clin. Bioch. 2019)

Step 3

Step 4 Link clinical performance

requirements to intended benefits

Step 5 Set minimum acceptable

clinical performance levels

(or New pathway)

Step 1

Step 2

Step 3

Step 4

Step 5

(Test Evaluation Working Group of the EFCCLM, Annals Clin. Bioch. 2019)

• Selezione: criteri di analisi della performance

• Utilizzo: criteri di analisi della utilità clinica

Studi sull’utilità clinica dei biomarcatori Oltre i trial randomizzati

Nuovi modelli di studio clinico

Studi sull’utilità clinica dei biomarcatori Oltre i trial randomizzati

• Linked Evidence Approach (LEA)

• Pragmatic studies

• Practice Based Research

• Linking of acquired evidence assessing each component of a test-treatment pathway with the aim to predict the likely impact of testing on patient health outcomes.

• Evidence of test accuracy would be considered a sufficient proxy for diagnostic effectiveness.

(Merlin 2013)

Linked Evidence Approach (LEA)

(Merlin 2013)

The use of linked evidence in the evaluation of diagnostic tests

Explanatory vs. pragmatic studies

• Explanatory trials aim at verifying a biological hypothesis

• Pragmatic trials intend to offer help in deciding between two treatment options

(Bossuyt PM et al, Journal of Clinical Epidemiology 2020)

Features Explanatory Pragmatic

Purpose Understanding Decision making

Index test Biomarker or prototype Assay or device

Positivity threshold Explored Prespecified

Recruitment Selective Representative

Sample size Smaller Larger

Analysis Per protocol Intention-to-test

Explanatory vs. pragmatic studies

(Bossuyt PM et al, Journal of Clinical Epidemiology 2020)

Features Explanatory Pragmatic

Purpose Understanding Decision making

Index test Biomarker or prototype Assay or device

Positivity threshold Explored Prespecified

Recruitment Selective Representative

Sample size Smaller Larger

Analysis Per protocol Intention-to-test

Explanatory vs. pragmatic studies

(Bossuyt PM et al, Journal of Clinical Epidemiology 2020)

Features Explanatory Pragmatic

Purpose Understanding Decision making

Index test Biomarker or prototype Assay or device

Positivity threshold Explored Prespecified

Recruitment Selective Representative

Sample size Smaller Larger

Analysis Per protocol Intention-to-test

Explanatory vs. pragmatic studies

(Bossuyt PM et al, Journal of Clinical Epidemiology 2020)

Features Explanatory Pragmatic

Purpose Understanding Decision making

Index test Biomarker or prototype Assay or device

Positivity threshold Explored Prespecified

Recruitment Selective Representative

Sample size Smaller Larger

Analysis Per protocol Intention-to-test

Explanatory vs. pragmatic studies

(Bossuyt PM et al, Journal of Clinical Epidemiology 2020)

Features Explanatory Pragmatic

Purpose Understanding Decision making

Index test Biomarker or prototype Assay or device

Positivity threshold Explored Prespecified

Recruitment Selective Representative

Sample size Smaller Larger

Analysis Per protocol Intention-to-test

Explanatory vs. pragmatic studies

(Bossuyt PM et al, Journal of Clinical Epidemiology 2020)

Features Explanatory Pragmatic

Purpose Understanding Decision making

Index test Biomarker or prototype Assay or device

Positivity threshold Explored Prespecified

Recruitment Selective Representative

Sample size Smaller Larger

Analysis Per protocol Intention-to-test

Explanatory vs. pragmatic studies

(Bossuyt PM et al, Journal of Clinical Epidemiology 2020)

Practice Based Research

• The use of research-inspired principles, designs and information gathering techniques within existing forms of practice.

• Demonstrate whether interventions with proven efficacy are truly effective and sustainable when provided in the real world setting.

(Westfall JM et al, JAMA 2007)

2017

Conclusioni

• I requisiti formali (EU, USA) non sono operativi o non sono adeguati.

• I criteri di valutazione finora utilizzati sono risultati poco efficienti.

• Per la scelta, la valutazione e l’utilizzo di nuovi biomarcatori è necessario e raccomandabile utilizzare modelli innovativi, basati su criteri finalizzati ad accelerare l’acquisizione di evidenze e inquadrati in nuovi contesti teorici.

Obiettivo biomarcatori: trasferire alla pratica clinica una crescente complessità in tempi utili e con costi sostenibili.

Grazie per l’attenzione

Linked Evidence Approach (LEA)

• The ‘‘linked evidence approach’’ involves the narrative linking of systematically acquired evidence assessing each component of a test-treatment pathway with the aim to predict the likely impact of testing on patient health outcomes.

Linked Evidence Approach (LEA)

• False-positive and false-negative test results will be reflected in the measured health outcomes of patients.

• The ‘‘linked evidence approach’’ involves the narrative linking of systematically acquired evidence assessing each component of a test-treatment pathway with the aim to predict the likely impact of testing on patient health outcomes.

• Evidence of test accuracy would be considered a sufficient proxy for diagnostic effectiveness if:

1. there is reasonable justification to assume that the population receiving the new test is to all intents and purposes the same population that would receive treatment for the condition

2. there is good evidence that treatment impacts positively on the health outcomes in this population.

(Transferability assumption)

(Merlin 2013)

Linked Evidence Approach (LEA)

• Evidence of test accuracy would be considered a sufficient proxy for diagnostic effectiveness if:

1. there is reasonable justification to assume that the population receiving the new test is to all intents and purposes the same population that would receive treatment for the condition

2. there is good evidence that treatment impacts positively on the health outcomes in this population.

(Transferability assumption)

(Merlin 2013)

Linked Evidence Approach (LEA)

Lo scenario: la ricerca

• Tecnologie innovative in rapido sviluppo (omics)

• Software capaci di esplorare i big data ed estrarre informazioni

• Nuove matrici (exosomes)

• Conoscenze sepre più dettagliate dei meccanismi

• Migliaia di biomarcatori “candidati”

Lo scenario: l’applicazione clinica

• Marcatori circolanti

– Diagnostici

– Prognostici e/o predittivi

• Marcatori nel tessuto del tiumore

– Prognostici e/o predittivi

New drugs authorised each year with and without a predictive biomarker in the indication or contraindication

(Malottki K et al. BMJ Open 2014)

An evaluation of the indications and contraindications of all drugs considered by the European Medicines Agency (EMA) and published in 883 European public assessment reports and pending decisions found mentions of only 37 predictive biomarkers for 41 drugs.

(Malottki K et al. BMJ Open 2014)

List of Cleared or Approved Companion Diagnostic Devices (In Vitro and Imaging Tools)

17 markers (47 commercial products)

Updated 08.02.2019

(Hayes DF, JAMA, 2017)

The regulatory issues (USA)

• The regulatory environment of laboratory assays, including tumor biomarker tests, is at best inconsistent.

• While the Office of In Vitro Diagnostics is superb in assessing the analytical validity of tumor biomarker tests, their hands are tied in regards to insisting on clinical utility as a criterion for clearance or approval of a tumor biomarker test.

• Therefore, approval of a tumor biomarker test by the FDA does not necessarily imply that it should be used to direct patient care.

(Monaghan PJ et al, JIFCC 2018)

IVDD 98/79/EC IVDR 2017/746

Legislative framework

Directive Requires transposition in each member state

Regulation Immediately applicable and enforceable by law

in all Member States

Pages 37 163

Recitals (Why) 35 101

Articles (What) 24 113

Annexes (How) 10 15

Definitions 10 74

Enhanced Regulatory Framework

• 107 original-research articles published in 2006 that assessed the diagnostic value of a molecular- or "omics"-based test were identified

• studies citing the 107 original-research (published before Dec.31 2016) were located

(Parker LA et al, Clin Chem 2018 )

In the 10-year period analyzed (2006-2016),

• 4257 articles cited the 107 diagnostic studies

• 118 (2.8%) were diagnostic studies of the same test, and of these, 25 (21.2%) did not constitute progress toward validation of the test for use in clinical practice (potential research waste).

• 28 (26.2%) appeared to have made progress toward clinical application.

• 4 (9.1%) had made progress toward clinical application.

Articles evaluating molecular diagnostic tests are numerous in biomedical journals but few tests have made progress toward clinical application in the 10 years following their discovery.

(Parker LA et al, Clin Chem 2018 )

(Korevaar DA et al, Evid Based Med, 2014)

• Extensive

• Indirect

• Probabilistic

• Relative

• Contextual and/or sequential

(Bossuyt PMM et al, Clin Chem, 2012)

1. Intended clinical use The dimensions of clinical utility

Clinical utility of biomarkers is indirect

• The majority of tests simply generate information, and information itself does not generate a health benefit.

• Improvements in health outcomes from diagnostic testing will be generated by the way test results are used to guide downstream management.

• Therefore, the effects of testing will depend not just on the test but also on the effectiveness of downstream management.

(Bossuyt PMM et al, Clin Chem, 2012)

Clinical utility of biomarkers is probabilistic

There is no certainty that testing will lead to the desired outcomes in the individual patient. That may be due to:

• variability in the test results (individual biological variability)

• differences in the downstream management

• variability in response

The utility can descrive in aggregate terms as the expected outcome in a group of people for whom we consider using the test.

Clinical utility is relative

• It cannot be defined in an absolute sense but must be evaluated relative to a comparator strategy (the current best testing strategy or not using any form of diagnostic testing all).

• The consequences of using markers depends on the clinical context.

• With changes in the available management options and with new and alternative forms of testing, the utility of a diagnostic test may change

• New interventions could lead to the disappearance of a previously useful diagnostic test, but they could also make a new test interesting, owing to the potential for the test and the management strategy concerned with improving health outcomes.

Clinical utility is contextual and consequential

2. Performance that could confer clinical utility

• By prespecifying the minimum clinical performance levels needed to achieve the proposed clinical benefits, researchers can design more purposeful clinical performance studies to determine if test performance is sufficient for intended use

• To draw the link between test accuracy and health outcomes, one must define the clinical consequences of true/false positive and true/false negative results for the target condition, relative to current practice without the new test.

• Rischi

• Criteri di ottimizzazione

Studi sull’utilità clinica dei biomarcatori Oltre i trial randomizzati

• An audit of diagnostic test accuracy studies has documented that few state an explicit hypothesis about the accuracy level that would favour clinical uptake, and researchers frequently over-interpret the results.

• Such unwarranted optimism can motivate subsequent futile studies, wasting research resources.

• Moreover, it can lead to marketing of biomarkers as medical tests with no clinical benefits for patients or the potential for harm

Studi sull’utilità clinica dei biomarcatori Oltre i trial randomizzati – I Rischi

Studi sull’utilità clinica dei biomarcatori Oltre i trial randomizzati – I Rischi

Selective reporting

• We evaluated 1915 articles included on cancer prognostic markers published.

• Only 26 articles (1.4%) were fully ‘negative’ for all presented results in the abstract and without efforts to expand on non-significant trends or to defend the importance of the marker with other arguments

Selective reporting

(Kyzas PA met al, EJC, 2007)

• Close to 100% of published study reports may show statistically significant results, although this is totally implausible given the fact that most of these studies are of small sample size and thus of limited power to show statistically significant associations with the outcomes of interest

• Published reports are often incomplete.

• It is very likely that a large number of biomarker discovery explorations are not reported at all

(Ioannidis JPA and Bossuyt PMM, Clin Chem 2017)

Selective reporting

Selective reporting

Common issues involve

• The selection of favorable subgroup or subset analyses and

• The manipulation of optimal cutoffs for continuous measurements

(Ioannidis JPA and Bossuyt PMM, Clin Chem 2017)

(Enzmann H et al, 2016)

The regulatory issues (EU)

• Innovative medicinal products receive a marketing authorization from the European Commission based on a positive benefit–risk assessment by the EMA.

• IVD may be sold in the EU with a CE mark after assessment and approval from a notified bodies (NB). For IVD the certification will focus on the technical features and technical quality of the products.

(Hayes DF, JAMA, 2017)

The regulatory issues (USA)

• The regulatory environment of laboratory assays, including tumor biomarker tests, is at best inconsistent.

• While the Office of In Vitro Diagnostics is superb in assessing the analytical validity of tumor biomarker tests, their hands are tied in regards to insisting on clinical utility as a criterion for clearance or approval of a tumor biomarker test.

• Therefore, approval of a tumor biomarker test by the FDA does not necessarily imply that it should be used to direct patient care.

Key notions for the design of biomarker studies

• The clinical pathway should be draw to show where the new test will be positioned to achieve the intended benefits.

• The design should be built around the intended clinical use for that biomarker

• The biomarker’s performance should be measured against performance that could confer clinical utility in practice

(Test Evaluation Working Group of the EFCCLM, Annals Clin. Bioch. 2019)

• behavioral effects

• early resolution of uncertainty

• help patients and the patient’s relatives cope with their illness

Clinical utility of biomarkers is extensive

(Bossuyt PMM et al, Clin Chem, 2012)

A test may have clinical utility in the absence of effective clinical treatment.

A test may have clinical utility without actually improving the primary health outcomes.

• If the introduction of a marker leads to health outcomes comparable to those obtained with current standard diagnostic strategies, but these outcomes are achieved in a simpler way, there may be clinical utility.

Step 4. Link clinical performance requirements to intended benefits

• Will the intended benefits stem from detecting more or earlier TP or TN, …

• … or from other test attributes without compromising accuracy?

Step 5. Set minimum acceptable clinical performance levels

What trade-off are you prepared to accept for benefits versus harms?

• What is the maximum FP you would tolerate for one additional TP?

• What is the maximum FN you would tolerate per 100 patients avoiding further testing/treatment? per 100 patients with the condition?