1

Generating Appropriate and Reliable Evidence for the Value Assessment of Medical Devices: An 1

ISPOR Medical Devices and Diagnostics Special Interest Group Report, Part 2 2

ABSTRACT 3

Creating a value assessment framework for medical devices has become important as payers 4

increasingly require evidence for funding, coverage, pricing, and reimbursement decisions that goes 5

beyond a medical device’s safety and effectiveness. While several frameworks are currently employed to 6

assess the value of medical devices, they are largely based on pharmaceutical value assessment, which 7

does not fully capture the multiple domains of device product value. There is a need to develop an 8

evaluation framework that accounts for the vast diversity of medical devices, challenges in evidence 9

generation, and importantly, optimally supports the evolving requirements of increasingly cost-conscious, 10

value-oriented stakeholders. Part 1 of this article details the unique characteristics of devices that make it 11

imperative to assess medical devices differently from pharmaceuticals. Part 2 proposes a value 12

assessment framework that builds on an evidence-generation strategy that will allow data integration from 13

heterogeneous sources, fit the intended purpose of the device, and is designed to employ attributes 14

beyond clinical value to include evidence in the real world care continuum. More significantly, the 15

framework is designed to align with global reimbursement policy frameworks, which will allow device 16

manufacturers to create value claims and messages about the core device attributes, thus leading toward 17

an effective market access strategy. 18

INTRODUCTION 19

Economic and clinical evaluations have played a pivotal role in assessing and communicating the value of 20

pharmaceuticals (Drummond 1997, Fry 2003). Furthermore, economic and clinical data can influence 21

value based pricing (VBP) arrangements (McGuire 2008), as well as inform coverage policies and 22

reimbursement decisions (Sullivan 2009). The increasing reliance on economic and clinical evidence in 23

health care decision making in many developed countries has attempted to facilitate rational decisions 24

about product adoptions and patient access (Massetti 2015). However, if this trend is to be fully 25

extrapolated to medical devices, clinical and economic evidence should be fit-for-purpose to incorporate 26

2

the unique attributes of medical device evaluation. Fit-for-purpose in the context of medical device 27

evaluation refers to the quality, quantity and consistency of the evidence used to communicate the value 28

of a medical product to stakeholders at a particular point in the product’s lifecycle. There is sufficient 29

evidence demonstrating that transplanting the same processes and methods of evaluating a 30

pharmaceutical does not work for medical devices (MedTech HTA, Drummond 2009, EUNetHTA). 31

Therefore, this paper will build on part 1 of this article which detailed the unique characteristics of medical 32

devices that make it imperative to revisit de facto evidentiary methodologies; part 2 addresses 33

methodological issues in generating appropriate and reliable evidence for value assessment of medical 34

devices. Note that diagnostics and drug-device combination products for which the primary mode of 35

action is a drug or biologic are not in the scope of this paper, as they require even further differentiated 36

frameworks. 37

MEASURING VALUE 38

In health care, value is commonly defined as outcomes relative to costs (Porter 2010). Measuring value in 39

health care seeks to transform health care delivery by linking cost, quality, safety, outcomes, and 40

efficiency to the care and services provided. Emerging new technologies have the potential to impact 41

patient care in positive ways, and in many cases, provide even greater opportunities to extract efficiencies 42

from the organizational and administrative drivers of health care spending. The need to determine the 43

value for money for medical devices, understand the opportunity costs in resource allocation and the 44

benefits for all stakeholders when incorporating a new medical device into an existing health system, 45

requires the generation and collection of evidence from research. This has led to a growing interest in and 46

application of evidence-based health technology assessment (HTA) for assessing medical devices (Huot 47

2012, Schreyögg 2009, Tarricone 2011). In Europe, HTA institutions play a role in the diffusion of new 48

technologies into the health care system; however, device assessment represents a minority of HTAs 49

conducted in countries where there are such processes in place (Wilsdon T, Serota A). Medical devices 50

account for approximately 6-8% of health care spending (MedTech Europe), and while HTA decisions do 51

not necessarily influence pricing and reimbursement decisions in the same way as they do for 52

pharmaceuticals, the methodology and process of HTA and how it is used in appraising medical devices 53

3

continues to facilitate decision-making regarding acquisition, implementation or discontinuation at other 54

levels within the health care system. These may include, hospital based HTA’s, procurement 55

assessments in the form of tendering, as well as reimbursement decisions for devices through 56

Medicare’s bundled payment systems for out-patient services. Thus, HTA’s for medical devices can often 57

supplement pricing and coverage decisions and be a useful tool in optimizing the path to market and 58

patient access. 59

The key question enabling effective medical device adoption decisions is whether existing value 60

assessment tools are fit-for-purpose and whether they fully reflect the value of these new technologies. Is 61

there a need for improvement in the evaluation methods for medical devices? Physicians, hospitals, 62

industry, insurers, regulators, public agencies, and patients are increasingly requiring that clinical and 63

economic information be linked to pricing, such that the benefits of health technology justify its payments 64

(Eisenberg 1989, Buxton 2006). This increased interest in evidence to support value-based assessment 65

of medical devices for coverage, reimbursement and pricing will need to formally rely on methods specific 66

to medical device assessment in order to balance access to effective new medical technologies and 67

resource allocation decisions. To overcome some of these challenges, a clear framework to assess the 68

value of medical devices is needed and should describe the manner in which economic evaluations 69

should be designed, analyzed, and reported. This article, which continues from from part 1 of the ISPOR 70

Medical Devices and Diagnostics Special Interest Group on the value assessment of medical devices, will 71

underscore the need for a fit-for-purpose framework in the area of medical device evaluation. 72

METHODOLOGICAL CHALLENGES TO MEDICAL DEVICE VALUE ASSESSMENT 73

a. WHAT COUNTS AS APPROPRIATE AND RELIABLE EVIDENCE 74

In assessing the value of medical devices, the question emerges on what constitutes appropriate and 75

reliable evidence and to what extent existing methods for evaluating evidence of effectiveness – primarily 76

rooted in pharmaceutical efficacy studies – can help. The terms appropriate and reliable are two 77

important concepts used in appraising the quality of evidence (Bowman CE & Ligensa T 2013; Campbell 78

et al. 2002; Oxman 2004; Boaz 2003; Leung L et al. 2015). Evidence is said to be appropriate if it is 79

4

suitable or proper in a given context; and considered reliable when it consistently yields comparable 80

results of good quality. Thus, the appropriateness and reliability of evidence can be influenced by the 81

research question, study design, data source, data collection and analysis, timing, and relevance in a 82

particular context. 83

The debate as to what constitutes reliable evidence— replicability of the processes and the results— is 84

centered around the hierarchical system of classifying evidence (Rosen L 2006; GRADE 2004; Süt N 85

2014; Lie RK 2011), also known as the levels of evidence. The establishment of a hierarchy of evidence 86

was first defined in a Canadian Task Force report on the Periodic Health Examination, published in 1979 87

(Canadian Task Force 1979). In this report, evidence from randomized controlled trials (RCTs) evaluating 88

the effectiveness of an intervention was considered to have the highest quality evidence, followed by 89

evidence obtained from well-designed cohort or case-control studies. Hierarchical systems have 90

traditionally placed RCTs— the gold standard of medical studies —at this level (Sacket 1989; Canadian 91

Task Force 1979) because of its high internal validity. Other versions of the evidence pyramid have 92

described systematic reviews of RCTs and RCTs themselves at the higher levels of evidence, followed by 93

cohort and case–control studies (OCEBM Levels of Evidence Working Group; NHMRC). 94

Unlike Europe, RCTs in the US are the most commonly used type of evidence in premarket and post-95

market regulatory decisions for high risk medical devices (Sorenson & Drummond 2014). The reliance on 96

RCTs to provide an adequate assessment of benefit-risk to support regulatory decisions of medical 97

devices (e.g., class III high risk devices) for premarket approval (PMA) and, in some cases, post-approval 98

to assess the continued safety and effectiveness of a device has faced many challenges (Yue 2007, 99

Zannad 2014, FDA 2013, FDA 2015). Alternative trial designs (e.g., adaptive trials, Bayesian methods) 100

have been developed to support device approval and overcome challenges related to conducting double-101

blind RCTs in medical device studies such as randomization, blinding, sample size, choice of control or 102

comparator group, etc. (Yue 2007, Bernard 2014). These alternative clinical trial designs are likely to 103

succeed within an effective regulatory framework that allows for prospectively planned modifications 104

based on accumulating study data necessary to properly characterize the safety and clinical effect of the 105

device in both the premarketing and post-marketing settings. For example, the United States Food and 106

5

Drug Administration’s (FDAs) regulatory framework for medical devices provides clarity on when and how 107

to consider the use of group sequential designs, sample size adaptation, and group sequential design 108

with sample size reassessment for mid-trial design adaptations within the context of a benefit-risk 109

assessment framework (FDA CDRH 2016) for certain medical devices and during the post-market phase 110

when assessing the long-term benefit-risk profile. There are, however, major challenges that still exist in 111

the collection of post-market data as part of post-market controls (e.g., post-approval studies and post-112

market surveillance) (Kramer et al. 2013, White & Carolan-Rees 2013, Lemmens 2014) and value 113

assessment (Tarricone & Drummond 2011, Taylor & Iglesias 2009, Yong et al. 2010). 114

To streamline data collection—specifically collection of data to demonstrate value—and integrate 115

premarket and post-market data on device performance, medical device manufacturers managing their 116

products within a regulatory framework should establish a process that consistently collects evidence of 117

value throughout the entire lifecycle, alongside the generation of safety and effectiveness data. Turning 118

data collection into evidence gathering over the device’s life-cycle requires a clear strategy that supports 119

the value to stakeholders. The starting point for coordinating evidence generation activities should be 120

early—while still in the product design phase. A robust evidence-generation strategy will allow data 121

integration from heterogeneous sources (e.g., patient registries, RCT, etc.) and promote the development 122

of better evidence that aligns with reimbursement policy frameworks. Ideally, users should have access 123

to multiple, heterogeneous data sources that fit to particular user’s information needs, thereby enabling 124

the user to have a unified view of the data. Data of varying sources can be extracted at all stages of the 125

value chain to contribute to building the target product profile (TPP) and evidence development plan. At 126

its simplest level, this will ensure that only products posing an acceptable trade-off between the costs and 127

outcomes are adopted into common medical practice. 128

b. DESIGNING FIT-FOR-PURPOSE STUDIES 129

Strategies to tackle and mitigate some of the issues prevailing in drug development—develop drugs more 130

rapidly, more efficiently, more cost-effectively— with alternative clinical trial designs (e.g., dose finding to 131

adaptive trial designs) have applicability to the challenges in device innovation and development (Bates et 132

6

al. 2015, Loudon et al. 2015, Burns et al. 2010). Designing device studies that are fit-for-purpose requires 133

that the most appropriate design is chosen for the intended purpose. For instance, a pivotal study may 134

be more appropriate when gathering evidence to support the evaluation of the safety and effectiveness of 135

a medical device; whereas a feasibility study may be indicated when evaluating the device design 136

concept with respect to initial clinical safety and device functionality. Tailoring protocol design also applies 137

to post-approval studies, post-market surveillance studies, non-clinical studies (e.g., bench, animal or 138

measurement studies and, for in vitro diagnostic devices, analytical validation studies), etc. Design 139

decisions that are consistent with the intended purpose of a trial should also consider other elements that 140

may influence trial methodological quality and efficiency, such as whether a device is intended for use as 141

a therapeutic, aesthetic or diagnostic device, in addition to whether a device has more than one intended 142

use (e.g., an interventional laparoscopic device that both diagnoses a condition and then provides 143

therapy for that condition). 144

Conducting randomized and quasi-randomized controlled trials 145

The use of randomized controlled trials to demonstrate the efficacy and safety of a medical product for 146

regulatory approval will continue to meet the definition of a rigorous clinical trial methodology 147

in terms of internal validity; however, adaptations to trial and/or statistical procedures (e.g., group 148

sequential design ) offers some advantages due to its flexibility and efficiency when applied in early 149

clinical development (Chow et al), and in some cases, the application of nonrandomized designs (e.g., 150

single-arm trials) supported by appropriate statistical analyses when it is not logistically feasible or ethical 151

to conduct a randomized controlled trial. For example, a single-arm study was sufficient to support the 152

clinical development of Crizotinib in for the treatment of patients with advanced anaplastic lymphoma 153

kinase (ALK)-positive non-small cell lung cancer (NSCLC) (Selaru et al. 2016). The flexibility of alternative 154

designs, however, may introduce possible biases that can reduce the study’s reliability and 155

generalizability.To address concerns that trial design decisions do not match the intentions of the trial, the 156

Pragmatic–Explanatory Continuum Indicator Summary tool (revised in 2015 as PRECIS-2) has been 157

developed by a group of trialists and methodologists. The PRECIS tool helps trial designers think about 158

the trial elements that separate explanatory approaches from pragmatic approaches by allowing trialists 159

to prospectively consider the design of their trial along 9 domains—eligibility criteria, recruitment, setting, 160

7

organization, flexibility (delivery), flexibility (adherence), follow-up, primary outcome, and primary 161

analysis—scored from 1 (very explanatory) to 5 (very pragmatic) to facilitate domain discussion and 162

consensus (Loudon et al. 2015). Other published scientific recommendations (Bernard et al 2008, 163

Campbell et al. 2008, Li & Yue 2007) with a particular focus on design challenges to the clinical 164

evaluation of a new medical device have provided similar guidance along the domains contained in the 165

PRECIS tool. Bernard et al. explored various methodologies to support clinical development of medical 166

devices when faced with the specific challenges of timing of assessment, eligible patient population and 167

recruitment, acceptability, blinding, choice of comparator group, and the learning curve. The authors 168

discussed alternative experimental designs, their limitations, and their applications when conventional 169

RCTs cannot be applied to the clinical development of medical devices (MDs) (Bernard et al. 2014). The 170

Medical Device Innovation Consortium (MDIC) has taken a somewhat different approach to trial design by 171

promoting the simplification of all clinical trials to large simple trials (LST) and pragmatic trial designs, 172

whenever possible (Alpert et al. 2016). While there are some differences in methodological challenges 173

between devices and drugs that may impact design considerations for pivotal clinical studies (Saksena, 174

Sanjeev, et al. 1995) and data collection, special consideration should also be given to the surgical 175

technique and clinician skills, as well as any learning curve, that may influence trial design and study 176

outcomes (Rathi, Vinay K., et al 2015, FDA Design Considerations 2013). 177

Even with the use of alternative designs to improve clinical trial efficiency, there continues to be a paucity 178

of high-quality clinical evidence– a concern that could very well create additional barriers to device 179

procurement and reimbursements in a data-driven, value-seeking global payer environment. A study by 180

Boudard et al. (2013), reviewing hospital-based HTAs for innovative medical devices, found that only 47 181

of 215 (22%) clinical studies included in the assessments provided high-quality clinical evidence on the 182

Sackett scale (levels 1-2), with only 33 (15%) of those being RCTs. A majority (52.1%) of studies included 183

fewer than 30 patients, and only 14 of the 47 high-quality studies reported the amount of missing data. A 184

follow-up period was mentioned in only 84 (71.8%) studies of implantable medical devices, averaging 185

18.9 months. Interestingly, the methodological quality did not increase with the risk level of the medical 186

device. In addition to the gaps and barriers to high-quality evidence for premarket requirements, if a 187

8

device has a short product life cycle, or if the effects are only observed in the very long term, the product 188

life cycle evidence generation strategy will not gather sufficient evidence to further support device 189

evaluation. In such circumstances, the concept of adaptive licensing (AL) (Eichler, H‐G., et al 2015) may 190

permit further data collection that will support an application for marketing approval or licensure and 191

assessment of the long-term benefit-risk profile. 192

Conducting non-randomized controlled trials 193

Randomized experiments are generally considered to be the preferrred methodological design for 194

obtaining causal inferences (Rubin 2008) by which claims about causal relationships can be made. Its 195

application in evaluating the safety and effectiveness of medical devices have proven to be beneficial in 196

satisfying statutory requirements within the premarket approval process; however, recent regulatory 197

decisions (FDA RWE Guidance) and the passing of the 21st Century Cures Act have opened the door to 198

the use of real-world evidence (RWE) to support benefit-risk evaluation of devices at various points in the 199

product life cycle. RCTs will likely remain the gold standard as the basis of medical product approvals 200

(Downing et al. 2014), but their usefulness has been shown to be limited when it comes to extrapolating 201

data to patients seen in real world clinical practice settings (Kennedy-Martin et al. 2015). In addition, they 202

are labor intensive, time consuming, expensive, and methodologically challenging to conduct for medical 203

devices; this often does not fit with shorter product life cycles (PLC’s). Furthermore, the specific 204

characteristics of patients enrolled in a RCT may also limit applicability of the results to a broader 205

population. As such, more pragmatic designs may be reasonable options for demonstrating the value of 206

medical devices and informing practice. 207

The use of observational studies as an alternative (or complement) to RCTs are becoming increasingly 208

relevant in clinical investigations, health policy and health services research. Observational studies that 209

are properly designed have been shown to approximate randomized experiments (Rubin 2008, Concato 210

et al. 2000). For instance, in a review conducted by Concato et al. which examined published reports of 211

RCTs and observational studies assessing the same clinical topic (clinical intervention and outcome), the 212

investigators found that the well-designed observational studies (with either a cohort or a case–control 213

9

design) did not systematically overestimate the magnitude of treatment effects as compared with those in 214

randomized controlled trials. 215

Clinical evidence available for the assessment of MDs is frequently inadequate; thus increasing the 216

quality of observational data sources and other RWE-based studies (e.g., post-market surveillance 217

studies, pragmatic trials, registries, and economics studies) will enable researchers and policy makers to 218

address questions about the long-term effectiveness and safety and other important outcomes of medical 219

devices. These RWE sources can also augment existing clinical evidence needed as part of pay-for-220

performance and risk-based reimbursement models—and ultimately support the expected product value 221

profile. For observational studies to be robust, we propose the following table (Table 1.) to build on the 222

challenges associated with RCT’s: 223

Table 1 224

10

Methods Usefulness

Study planning

Studies can be initiated by healthcare

providers who approach the industry for

provision of hardware/software; or by the

industry who approaches providers to

coordinate research using their devices.

Compared to pharmaceuticals, devices can be more

costly and need to be delivered in accordance with

indication criteria for the specified target population.

Cooperation between providers and industry can

reduce the burden, improve the correct utilization of

new devices and ensure that patients derive a benefit.

A research protocol should be created

whether data collection is retro- or

prospective.

Developing a research protocol is particularly important

when there are many outcomes of interest and

resource use is limited; and when different providers

are collecting and reporting data, such as in the

creation of patient registries.

The clinical outcomes identified should be

appropriate for measuring device

performance and patient benefit. It could be

useful to correlate device performance with

several PRO measures to assess which

measures are more sensitive.

Demonstrating patient benefit next to device

performance is particularly important for obtaining

reimbursement when many devices are available for

the same indications (Dawisha 2011).

For conducting an economic evaluation

measures of device use, resource utilization,

valid and reliable functional status

questionnaires, generic and disease-specific

health-related QOL measures should be

considered. It is recommended to use both a

disease-specific and a generic instrument

that are appropriate and sensitive to the

health condition of interest (EUnetHTA

2015).

Collection of resource utilization allows for the ability

to properly assign costs to the items and services

used during the treatment pathway.

While a disease-specific instrument gives more

information on the impact of the specific health

condition; generic instruments can illustrate how the

specific health condition can impact other aspects of

health that would otherwise remain unknown. Also, the

latter allows a comparison of different interventions

intended for similar target populations and/or

indications of use (EUnetHTA 2015).

Study design

Consider including a comparison group in

the study, even when the study aim is to

describe device outcomes. This cohort could

include patients who have not yet been

exposed to the medical device or those

representing a different pathway or indication

of use.

A direct comparison makes it possible to assess

whether patients really benefit from a new device or

technological feature. These study designs provide

higher level of evidence and are more useful for

demonstrating medical device value, particularly when

a RCT is not possible.

Blinding participants to the allocated

treatment may be possible in CER of

diagnostic and therapeutic devices, but is

not always feasible in practice, particularly

for implantable devices.

Consider including an independent

statistician who is blinded to the data.

This helps to reduce bias and uncertainty in study

results.

Table 1. Methods for Implementing and Improving the Quality of Observational Studies

11

Table 1 Continued

To improve the efficiency and effectiveness of

medical device development and timely

access, identify situations where non-RCT

studies can be used to support regulatory

and coverage decisions. The adaptive

liscensing (AL) approach is considered a

flexible regulatory pathway that balances

early patient access with data collection.

The generation of evidence is shifting from

predominantly premarket to continual data collection

throughout the total product life cycle. This approach

will be useful for medical devices with continual new

product introductions and short product lifecycle.

Ethical approval and voluntary consent

Ethics approval and voluntary consent

should be sought by providers even if

research is of a retrospective nature.

As data protection laws are becoming more stringent

on the use of patient-level data (Miani 2014) there is a

growing importance of obtaining ethical approval and

voluntary consent particularly for retrospective

analysis.

Data collection and analysis

At study initiation the characteristics of

study participants should be collected and

recorded to allow covariate analysis.

Covariate analysis assesses the sensitivity of study

outcomes, and helps quantify and reduce the

uncertainty in RWE.

Study monitoring by a third-party is crucial

for regulatory studies and should also be

considered for post-market studies. Third

parties can also collect data and supply the

anonymized information to the respective

stakeholders.

Monitoring by regulators and/or other third parties can

reduce bias and confirm the research protocol is being

correctly implemented, which would carry particular

importance in early stages of developing registries.

Data collection by third parties can also remove

barriers to patient-level data (Miani 2014).

Study reportingAny kind of industry involvement should be

disclosed.

Industry involvement does not necessarily imply a

conflict of interest. Most industry partners are subject

to government regulations that impose stringent

criteria on medical device research studies to secure

clinical, economic and quality of life evidence. This

would depend on how the industry is involved in

research and to what degree.

Decisions on ethical appraisal and voluntary

consent should be disclosed.

The lack of such information particularly in

retrospective studies can degrade the reliability of the

study and the quality of the publication.

If data is collected at several intervals the

results from not just the last follow-up visit,

but also from all intervals should be reported.

Long-term studies help illustrate the stability and

reliability of device benefit. This kind of data supports

decisions on reimbursement.

Individual-level outcome data should always

be provided either in the report or as

supplementary material.

This increase transparency and allows further

statistical analysis as part of technology appraisals.

Dissemination of results

Upon completing the research, results

should be published and disseminated to all

involved parties as soon as possible.

Studies showing a large effect of an intervention may

be crucial for making decisions on device utilization ,

and may even be requested by payers and decision

makers early on in the data collection and analysis

process (Polly 2011).

c. DEVICE EFFECTS VERSUS IMPACT IN THE REAL WORLD 225

Post-market surveillance programs help to address certain safety and effectiveness questions that may 226

not be fully resolved at the time of approval because of new or expanded conditions of use for existing 227

devices; significant changes in device characteristics; longer term follow-up or evaluation of rare events; 228

or public health concern(s) resulting from reported or suspected problems in marketed devices (FDA 229

Guidance PMS 1998). Data collected outside of traditional clinical trials as a part of a post-market 230

surveillance and risk management program, post-approval study, or captured during routine care creates 231

an opportunity to collect additional information regarding benefits or risks to augment the clinical trial. For 232

12

many medical devices, technical performance demonstrates safety from harm, and therefore 233

demonstrates a base-level value. The real value will be derived from device use and performance in 234

clinical practice. This represents somewhat of a divergence from pharmaceutical value assessment. This 235

shift in data collection from within the context of a clinical trial in the premarket phase to greater reliance 236

on post-market data collection in clinical practice or real world settings will allow the identification and 237

collection of outcomes data that further demonstrates value (Segal 2013). For example, the expanded 238

indication given under the FDA de novo program for Medtronic Duet External Drainage and Monitoring 239

System on the basis of extensive bench testing and clinical data from published evidence supporting the 240

“off-label” use or the advantage of a post-market surveillance system such as the Transcatheter Valve 241

Therapy (TVT) registry linked to CMS claims data to support an expansion of an approved indication 242

(Faris and Shuren 2017). 243

244

Epidemiologic methods in observational comparative effectiveness research (CER) can be applied to 245

medical devices to assess outcomes that are important to patients and clinicians (Jalbert et al. 2014). 246

These methods offer a tailored way of assessing the effectiveness of medical devices in clinically realistic 247

settings (Fischer 2012). CER results are, however, not helpful in identifying the appropriate target 248

population, i.e., which subgroup of patients benefit the most. Besides, there remains a methodological 249

challenge in how to compare benefits from alternative treatment pathways (with and without a new 250

device) as they are setting-specific and multivariate. The choice of comparator, duration of follow-up, and 251

sample size are also challenging decisions (Jalbert 2014; Price 2015). Moreover, the unit of comparison 252

is often the device itself (device selection may be an independent outcome variable) rather than the 253

performance of the treatment pathway as a whole. Whether a control or comparator group is used, 254

medical devices should be assessed in the actual settings and with the actual populations in which they 255

are used e.g., inpatient, outpatient and/or the home environment. Additionally, focusing on proving the 256

effectiveness of a device in a test setting inevitably requires excluding other elements of the treatment 257

pathway that are deemed as externalities, the value of medical innovation may, also, rely on other 258

characteristics (e.g., user’s experience, training, teamwork and coordination, a hospital’s volume profile, 259

digital and technical infrastructure, organizational readiness, etc.) (Abrishami 2015). 260

13

d. CAPTURING DEVICE EFFECTS AND REAL WORLD EVIDENCE 261

RWE is considered an umbrella term for data regarding the effects of health interventions collected 262

outside of conventional controlled trials. Real-world data (RWD) is collected either prospectively or 263

retrospectively during observations of routine clinical practice. RWE is evidence derived from the analysis 264

and/or synthesis of RWD (GetReal 2015). RWD as part of post-market surveillance can strengthen the 265

medical device post-market surveillance system and provide a way to demonstrate actual improvement in 266

outcomes during the course of implementation (Reynolds 2014). RWD obtained from electronic medical 267

records (EMRs), insurance claims data, patient registries, and other public or private databases, together 268

with technology and healthcare data partners, can also leverage outcomes and big data to demonstrate 269

product value. Big databases can provide information for a period longer than a product lifetime and give 270

insight into comparative effectiveness of incremental product development. 271

For example, with the collection of data on treatment costs alongside clinical outcomes, one can examine 272

the actual cost profile and cost-saving potential of the new device relative to alternatives by collecting 273

data on the resources used and outcomes gained. RWE to demonstrate medical device value is 274

increasingly considered by payers, HTA agencies, regulators and policy makers, such as NICE in the UK; 275

however, the methods applied for data collection are cautiously evaluated and data availability prior to 276

decisions on device uptake is extremely limited. Critics often argue about the uncertainty and low 277

reliability of RWD and consider it to provide low level evidence. The quality of research will ultimately 278

depend on its methodology. RWE will not replace the traditional evidentiary standards used in regulatory 279

decision-making, but may be used to augment the information needed to support clearance or approval. 280

However, with precise implementation of an a priori research protocol, RWE can create a better 281

understanding of a device benefit-risk profile, provide a vast amount of information on value, and describe 282

how and to what extent the use of the device has improved outcomes. Key considerations in 283

implementing observational studies and for improving validity and reliability are listed in Table 1. 284

To take full advantage of data gathered from real-world data sources, a multipurpose data repository, 285

14

data warehouse, or a set of linked data sources should have the capability to follow patients across the 286

care continuum and integrate with other data source systems (e.g., capturing UDI data in EHRs linked to 287

healthcare administrative claims data). There are opportunities for generating new evidence across the 288

total product lifecycle of a medical device, but concerns remain over data privacy, data ownership, data 289

sharing and transparency, etc. (Kostkova et al. 2016). 290

CONSIDERATIONS FOR ESTABLISHING A FRAMEWORK FOR MEDICAL DEVICES: 291

Payers are interested in evidence that goes beyond the device’s safety and effectiveness and the need 292

for creating a value assessment framework for medical devices has become increasingly important for 293

funding, coverage, pricing, and reimbursement decisions. Creating a fit-for-purpose value framework that 294

is functional at all levels of value assessment will be important in providing an organized, logical structure 295

of the key domains of perceived total product value (i.e., a more holistic approach that includes defining 296

and measuring value across the care continuum), which will allow device manufacturers to build an 297

effective market access strategy and create value claims and messages about core device attributes. 298

The assessment of safety and performance of medical devices by regulatory agencies is insufficient to 299

quantify the entire product value proposition and to allow for successful market access and adoption in a 300

value-driven health care system (Pham 2014). For a value-based access and adoption process, device 301

manufacturers need to demonstrate the value of their devices to other stakeholders (e.g., physicians, 302

payers, providers, patients) whose needs are important and valued differently (World Health Organization 303

2010). Despite greater attention by medical device manufacturers to the value proposition in the early 304

stages of the product life cycle, significant barriers remain (Bergsland 2014), particularly in how product 305

value is measured and assessed. In generating the required evidence to demonstrate product value to 306

stakeholders, device manufacturers should consider (Sorenson 2008) the data collection process, study 307

design, relevant endpoints, real-world performance, how patients may engage/interact with the device, 308

timing of assessment, and pricing when selecting a framework intended to contribute to medical device 309

evaluation. Table 2 provides a list of questions to help identify the appropriate methods for medical device 310

value assessment. 311

15

Table 2 312

Subset of value assessment Questions for generating relevant evidence

Identification of target

indicationsWhich patient population could benefit the most?

How does the new device improve the performance/ outcome of the whole treatment

pathway?

How does the new device fit into the whole treatment pathway?

Does it replace another treatment? Which part of the treatment pathway will become

obsolete?

How does the new device affect the costs of the whole treatment pathway?

What are the average costs of the treatment pathway including the new device?

How can the device improve healthcare delivery and reduce total staff time and/or

resource allocation compared to standard of care (SOC)?

Prospective risk inventory What are the potential risks (safety, financial, legal)?

What is the reasonable time frame for outcome improvement and/or cost reduction

given the product life cycle and the state of medical practice?

Considering the fact that users may improve as they becomes more familiar with a

technology, how could a possible/expected learning curve be best reflected in the

assessment?

Which research design and instruments are in place for generating relevant evidence?

Is conducting RCT necessary or feasible? If not, why?

What are relevant outcome measures?

What is the relevant comparator?

(Pre)requirements for

delivering value

Under which circumstances could the new device deliver value? How feasible and

realistic are the preconditions on the return on investment, infrastructural adjustments,

operator training, logistics, liability, etc.?

Price assessment To what extent is the price elastic to volume and scale?

Table 2. Relevant Questions to Address in Order to Demonstrate the Value of Medical Devices for

Pricing and Reimbursement Decisions

Added value (clinical)

Protocol for evidence

generation and data collection

Added value (economic)

Time frame of the assessment

VALUE ASSESSMENT FRAMEWORKS 313

Value frameworks are commonly used to assess treatment value. All of the value frameworks that have 314

been recently developed partly define value in terms of the favorable effect of an intervention (i.e., clinical 315

benefit or efficacy). With new payment models, pricing and reimbursement decisions will likely be 316

influenced by the overall product value. Consequently, reimbursement decisions will not reflect device 317

value solely according to safety (i.e., risk associated with use) and effectiveness (i.e., does it work). It 318

may encompass other quality metrics or measures on health outcomes such as usability, device ease of 319

use, user satisfaction, relative effectiveness, and cost-effectiveness. It will also reflect on how effectively a 320

device impacts non-health outcomes that focus on cost such as the financial consequence of adoption 321

(e.g., budgetary impact) and efficiency (e.g., time motion analysis). Despite limitations in these 322

frameworks for assessing the value of prescription drugs (Neumann 2015), these various frameworks do 323

not consider other important dimensions of device product value (e.g. usability, convenience, ease of use, 324

quality). Value frameworks should consider all relevant domains of product value in an appropriate multi-325

16

dimensional framework. 326

Table 3 327

17

Organization Factors Considered Description

Payers

Hospital

Purchasers

American College of

Cardiology–American

Heart Association

(ACC–AHA)

Clinical benefit vs. risks

Magnitude of net benefit

Precision of estimate based on

quality of evidence

Value (cost-effectiveness)

Magnitude of treatment effect ranges from class I

(“benefit [greatly exceeds] risk,” “procedure or

treatment is useful or effective”) to class III (“no

benefit, or harm,” “procedure or treatment is not

useful or effective and may be harmful”).

Precision of treatment effect ranges from level A

(“data derived from multiple randomized trials or

meta-analyses”) to level C (“only consensus

opinion of experts, case studies, or standard of

care”). Value corresponds to cost-effectiveness

thresholds (high: less than $50,000 per QALY;

intermediate: $50,000 to $100,000 per QALY;

low: more than $150,000 per QALY). The

framework lists the clinical benefit and value

designations without combining them.

American Society of Clinical

Oncology (ASCO)

Clinical benefit

Overall survival

Progression-free survival

Response rate

Toxicity

Bonus factors

Palliation

Time off all treatment

Cost per month

A therapy can be awarded up to 130 points.

Clinical benefit (≤80 points) reflects end point and

magnitude of benefit, with preference given to

evidence on overall survival if available. Toxicity

(±20 points) reflects the rate of grade 3 to 5 toxic

effects with treatment relative to standard of care.

Bonus point score reflects palliation (10 points if

therapy improves symptoms) and increased time

off all treatment

(≤20 points). The framework doesn’t combine

each drug’s point score and cost.

Institute for Clinical and

Economic Review

(ICER)

Incremental cost-effectiveness plus

care value components

Comparative clinical effectiveness

Other benefits and disadvantages

Contextual considerations

Budget impact

Cost-effectiveness ratio must not exceed a

threshold ranging from $100,000 to $150,000 per

QALY. Selection of final threshold is based on:

(a) comparative clinical effectiveness, reflecting

“judgments of the health benefit magnitude” and

“strength of a body of evidence”; (b) other benefits

and disadvantages, including such outcomes as

factors influencing adherence or return to work;

and (c) contextual considerations, including

“ethical, legal, or other issues” (e.g., high burden

of illness, availability of alternative treatments).

Budget impact is acceptable if a drug’s

introduction is compatible with an annual health

care budget increase of GDP growth plus 1%.

ICER reverse-engineers a “value-based price

benchmark” that independently satisfies both the

cost-effectiveness and budget-impact criteria.

Memorial Sloan Kettering

Cancer Center

Efficacy (survival)

Toxicity

Novelty

Research and development cost

Rarity

Population health burden

Framework assigns values to each domain.

Efficacy is assessed as improvement in overall

survival, if available. Efficacy score also reflects

evidence quality. Toxicity is a drug’s impact on

probability of severe side effects and treatment

discontinuation. Novelty is scored as 1 (novel

mechanism of action), 0.5 (“known target but

different mechanism of targeting”), or 0 (“next-in-

class”). Research and development cost

corresponds to the “number of human subjects

enrolled in the approval trials for the first

indication.” Rarity is the 2015 projected disease

incidence. Population health burden is the annual

years of life lost to the targeted disease in the

United States. “Fair price” is the product of the

scores, each of which is scaled by a user-

adjusted weight.

Table 3. Summary of Value Frameworks

Use of Accepted Framework

for Device Assessment

18

Table 3 Continued

National Comprehensive

Cancer Network

(NCCN)

Efficacy

Safety

Evidence quality

Evidence consistency

Affordability

Each area is scored on a scale of 1 to 5, with 1

indicating least favorable and 5 most favorable.

The framework presents the scores separately.

There is no explicit synthesis. Stakeholders

judge acceptability on the basis of their overall

impression of the listed factors.

European network for health

technology assessment

(EUnetHTA) Joint Action

Health problem and current use of technology

Description and technical characteristics

Safety

Clinical effectiveness

Costs and economic evaluation

Ethical analysis

Organisational aspects

Patient and social aspects

Legal aspects

In this model, two types of assessments can be

identified: the Rapid Relative Effectiveness

Assessment (REA) covers the clinical domains

and measures the medical/therapeutic added

value of a technology; the Full HTA Assessment

also includes other domains (cost-effectiveness,

budget impact, ethical, and legal considerations

as well as impact on patients and the

organisation of health care systems).

Advanced Medical

Technology Association

(AdvaMed)

Clinical impact

Non-clinical patient impact

Care delivery revenue and cost impact

Public/population impact

These four categories consider the impact on the

effectiveness and efficiency of care delivered

under new value-based performance metrics and

reimbursement models. The categories intend to

align value assessments with health reform

initiatives to improve the patient care experience,

improve population health, and reduce the per-

capita cost of health care. The results of the

value assessment do not assign higher values to

one value driver category over another, and do

not sum impacts across categories of value.

Table adapted from Neumann, Peter J., and Joshua T. Cohen. "Measuring the value of prescription drugs." New England Journal of Medicine 373.27 (2015): 2595-2597.

* GDP denotes gross domestic product, and QALY quality-adjusted life-year.

ELEMENTS OF VALUE ASSESSMENT FRAMEWORKS FOR DEVICE ASSESSMENT 328

Total Product Life Cycle (TPLC) is a conceptual framework for looking at a given device from initial 329

conception, through pre-market development, to widespread market use, and finally to obsolescence and 330

replacement by subsequent generations of products (Hausman 2004). The TPLC of a medical device is 331

often characterized as having rapid innovation, short product life cycles, broad product diversity, and 332

highly globalized; which poses a very different pre-and-post approval challenge, primarily for predicate 333

devices covered by a patent, than pharmaceuticals with a longer patent life and few, if any, modifications. 334

For pharmaceuticals, payers making reimbursement and formulary status decisions may expect post-335

market data collection spanning over three or more years. However, the short commercial life cycle of 336

roughly 18-24 months on average may negatively impact the post-market data collection for medical 337

devices. For example, short commercial life cycle may hinder the need for post-market data collection in 338

the form of medical device reporting (MDR), post-approval studies, post-market surveillance studies, as 339

well as the collection of data to address a wider range of practical or real-world questions. 340

Furthermore, lifecycle extension strategies can be quite effective for extending the life cycle of 341

19

pharmaceuticals, which may enable the collection of post-market data on long-term outcomes (Vernaz et 342

al. 2013). These tactics can be broadly divided into: marketing strategies (e.g., pricing, promotion, 343

divestiture, differentiation, over-the-counter drugs, and branded generics); R&D strategies (new 344

indications, reformulations, combination drugs, and next-generation drugs); and legal strategies (generic 345

settlements and patenting); however, these re-innovation tactics may not necessarily be applicable to 346

medical devices (Kappe 2014). Therefore, for device manufacturers to make a strong case for total 347

product value, their products have to go beyond simply safety and effectiveness. Understanding medical 348

device total product value allows the device to be viewed along other attributes/domains such as, quality, 349

ease-of-use, convenience, etc—and assessed and appraised using a multicriteria decision analysis 350

(MCDA) approach that allows decision-makers to evaluate alternatives against multiple criteria or 351

attributes. 352

Evidence from clinical and safety data are the backbone of substantiating domains of value. Other key 353

domains include economic, organizational, societal and ethical aspects. These must ensure the patient is 354

the intended recipient, whether directly or indirectly through a doctor or other health care practitioners. 355

The increasing use of medical devices has led some HTA bodies to develop methodology guidelines 356

specific to medical devices. For instance, some of the challenges in assessing the value of medical 357

devices are acknowledged and addressed in these guidelines (HAS 2009, NICE 2011). Several medical 358

device assessment agencies have been established to provide policy makers with information on the 359

clinical and economic value of innovative and costly medical devices (Ciani 2015). Given the differences 360

between drugs and devices, this article makes the case that a distinctive value assessment framework 361

tailored to devices should be developed, that contains broader and real world attributes of value (Table 362

4). Table 4 provides an illustrative example of how a comprehensive value assessment framework for 363

medical devices can be used as a transparent analytical tool to assess the value of a medical device.This 364

provides support for assessing a device`s clinical, economic and non-clinical impact; as well as the 365

quality, efficiency, reproducibility and generalizability of results, and performance along the care 366

continuum in a real world context. 367

Table 4368

20

Example Device Device Class Convenience Ease of use Safety Efficacy/Effectiveness Cost-effectiveness Budget Impact Efficiency

Strength

of the

evidence

PRO UsabilityAdherence/c

omplianceQuality Social Impact

Educational

necessity

Organizational

impact

Wheelchair, mechanical 1

External transcutaneous

cardiac pacemaker 2

Replacement heart

valve 3

Fetal cardiac monitor 2

Electroconvulsive

therapy device 3

Table 4. Value Assessment Framework

Domains/Attributes

369

21

CONCLUSION 370

Developing a value assessment framework that limits assessment of medical device value with 371

information on safety and effectiveness data from RCTs is impractical. Instead, choosing an alternative 372

clinical trial design and evidence generation strategy, in a more cost-effective manner, will likely balance 373

premarket and post-market data collection efforts, facilitate timely access to new important medical 374

devices, decrease the burden of expensive RCTs, spur investment in evidence generation systems that 375

captures evidence that substantiates medical product value and encourage collaboration with multiple 376

healthcare data partners. While there may not be a perfect single clinical trial design for evaluating all 377

medical devices, evidence should be fit for purpose and able to adequately inform all levels of value 378

assessment, from regulatory approval to procurement and reimbursement decisions. Thus, if the new 379

evidence is guided by a 'fit for purpose' framework, then the level and volume of evidence gathered 380

should consider all key stakeholder needs (regulatory agencies, providers and payers) and hence be a 381

function of medical device safety, performance characteristics, clinical efficacy/effectiveness and where 382

appropriate a health economic value assessment. 383

22

Reference List 384

1. Drummond M, Jönsson B, Rutten F. The role of economic evaluation in the pricing and reimbursement 385

of medicines. Health Policy 1997;40(3):199-215. 386

2. Fry RN, Avey SG, Sullivan, SD. The Academy of Managed Care Pharmacy format for formulary 387

submissions: An evolving standard—A foundation for Managed Care Pharmacy Task Force Report. 388

Value Health 2003;6:505–521. 389

3. McGuire A, Raikou M, Kanavos P. Pricing pharmaceuticals: Value based pricing in what sense. 390

Eurohealth 2008;14(2):3-5. 391

4. Sullivan SD, Watkins J, Sweet B. et al. 2009. Health Technology Assessment in Health‐Care Decisions 392

in the United States. Value in Health. 2009; 12(s2): S39-S44. 393

5. Massetti M, Aballéa S, Videau Y, Rémuzat C, Roïz J, Toumi M. A comparison of HAS & NICE 394

guidelines for the economic evaluation of health technologies in the context of their respective 395

national health care systems and cultural environments. J Mark Access Health Policy 2015;3:24966. 396

6. Huot L, Decullier E, Maes-Beny K, Chapuis FR. Medical device assessment: scientific evidence 397

examined by the French national agency for health – a descriptive study. BMC Public Health 398

2012;12:585. 399

7. Schreyögg J, Bäumler M, Busse R. Balancing adoption and affordability of medical devices in Europe. 400

Health Policy 2009;92(2):218-24. 401

8. Tarricone, R., Drummond, M. Challenges in the clinical and economic evaluation of medical devices: 402

The case of transcatheter aortic valve implantation. J Med Market 2011;11(3):221-9. 403

9. Eisenberg JM. Clinical Economics: A guide to the economic analysis of clinical practices. JAMA 404

1989;262(20):2879-86. 405

10. Buxton MJ. Economic evaluation and decision making in the UK. Pharmacoeconomics 406

2006;24(11):1133-42. 407

11. Robinson JC. Value-based purchasing for medical devices. Health Affairs 2008;27(6):1523-31. 408

12. Kirisits A, Redekop WK. The economic evaluation of medical devices: Challenges ahead. Appl Health 409

Econ Health Policy 2013;11(1):15-26. 410

13. Ventola CL. Challenges in evaluating and standardizing Medical Devices in Health Care facilities. 411

23

Pharm Therapeutics 2008;33(6):348-359. 412

14. Craig JA, Carr L, Hutton J, Glanville J, Iglesias CP, Sims AJ. A review of the economic tools for 413

assessing new Medical Devices. Appl Health Econ Health Policy 2015;13:15–27. 414

15. KNAW (Royal Netherlands Academy of Arts and Sciences). Evaluation of new technology in health 415

care: In need of guidance for relevant evidence. Amsterdam: KNAW, 2014. Available from: 416

https://www.knaw.nl/en/news/publications/evaluation-of-new-technology-in-health-care-417

1/@@download/pdf_file/verkenning-new-technology-health-care.pdf [Accessed September 9, 2015] 418

16. Daniel GW, Rubens EK, McClellan M. Coverage with evidence development for Medicare 419

beneficiaries: challenges and next steps. JAMA internal medicine 2013; 173(14):1281-1282. 420

17. Garrison L, Towse A, Briggs A, de Pouvourville G, Grueger J, Mohr P, Siviero P, Miguel Sleeper AC. 421

Performance-Based Risk-Sharing Arrangements-Good Practices for Design, Implementation and 422

Evaluation: An ISPOR Task Force Report Value Health 2013; 16: 703-719 423

18. Walker S, Sculpher M, Claxton K, Palmer S. Coverage with evidence development, only in research, 424

risk sharing, or patient access scheme? A framework for coverage decisions. Value Health 2012; 425

15(3):570-579. 426

19. Neumann PJ, Chambers JD, Simon F, Meckley LM. Risk-sharing arrangements that link payment for 427

drugs to health outcomes are proving hard to implement. Health Affairs 2011; 30(12):2329-2337. 428

20. OCEBM Levels of Evidence Working Group. “The oxford 2011 Levels of Evidence” Oxford Centre for 429

Evidence-Based Evidence Available from http://www.cebm.net/ocebm-levels-of-evidence/. [Accessed 430

March 20, 2013]. 431

21. NHMRC additional levels of evidence and grades for recommendations for developers of guidelines. 432

Available from 433

http://faculty.nhmrc.gov.au/_files_nhmrc/file/guidelines/developers/nhmrc_levels_grades_evidence_1434

20423.pdf [Accessed May 6, 2014]. 435

22. Fischer MA, Avorn J. Academic detailing can play a key role in assessing and implementing 436

comparative effectiveness research findings. Health Aff (Millwood) 2012;31:2206-12. 437

23. Jalbert JJ, Ritchey ME, Mi X, et al. Methodological considerations in observational comparative 438

effectiveness research for implantable medical devices: an epidemiologic perspective. Am J 439

24

Epidemiol 2014;180:949-58. 440

24. Price D, Graham C, Parkin CG, et al. Are systematic reviews and meta-analyses appropriate tools for 441

assessing evolving Medical Device Technologies? J Diabetes Sci Technol 2015; pii: 442

1932296815607863. [Epub ahead of print]. 443

25. Backhouse ME, Wonder M, Hornby E, et al. Early dialogue between the developers of new 444

technologies and pricing and reimbursement agencies: A pilot study. Value Health 2011;14:608-15. 445

26. U.S. Food and Drug Association (FDA). Investigational Device Exemptions (IDEs) for Early Feasibility 446

Medical Device Clinical Studies, Including Certain First in Human (FIH) Studies (2013). Available 447

from: 448

http://www.fda.gov/downloads/MedicalDevices/DeviceRegulati%20onandGuidance/GuidanceDocume449

nts/UCM279103.pdf2 [Accessed April 22, 2016]. 450

27. Abrishami P, Boer A, Horstman K. How can we assess the value of complex medical innovations in 451

practice? Expert Rev Pharmacoecon Outcomes Res 2015;15:369-71. 452

28. Reynolds IS, Rising JP, Coukell AJ, et al. Assessing the safety and effectiveness of devices after US 453

Food and Drug Administration approval: FDA-mandated postapproval studies. JAMA Intern Med 454

2014;174:1773-9. 455

29. Pham B, Tu HAT, Han D et al. Early economic evaluation of emerging health technologies: protocol 456

of a systematic review. Syst Rev 2014;3:81. 457

30. World Health Organization (WHO). Clinical evidence for medical devices: Regulatory processes 458

focussing on Europe and the United States of America: background paper 3, August 2010. Available 459

from: http://apps.who.int/iris/bitstream/10665/70454/1/WHO_HSS_EHT_DIM_10.3_eng.pdf 460

[Accessed April 22, 2016]. 461

31. Bergsland J, Elle OJ, Fosse E. Barriers to medical device innovation. Med Devices 2014;7:205–209. 462

32. Sorenson C, Drummond M, Kanavos P. Ensuring value for money in health care: the role of health 463

technology assessment in the European Union. WHO Regional Office Europe; 2008. 464

33. Dawisha SM, Kline Leidy N. Patient reported outcomes in decision making and communication. In: 465

Ackerman SJ, Dix Smith M, Ehreth J, Eldessouki R, Sullivan E, eds., Therapeutic and Diagnostic 466

Device Outcomes Research. USA: International Society for Pharmacoeconomics and Outcomes 467

25

Research, 2011. 468

34. European Network for Health Technology Assesment (EUnetHTA).Guidelines for Therapeutic 469

medical devices, 2015. Available from http://www.eunethta.eu/eunethta-guidelines [Accessed 470

February 1, 2016]. 471

35. Miani C, Robin E, Horvath V et al. Health and healthcare: Assessing the real-world data policy 472

landscape in Europe. RAND Europe, 2014. 473

36. Polly DW. Therapeutic and diagnostic device clinical outcomes research practical considerations. In: 474

Ackerman SJ, Dix Smith M, Ehreth J, Eldessouki R, Sullivan E., eds., Therapeutic and Diagnostic 475

Device Outcomes Research. USA: International Society for Pharmacoeconomics and Outcomes 476

Research, 2011. 477

37. Haute Autorite de Sante (HAS). Medical Device assessment in France. 2009. Available from: 478

http://www.has-sante.fr/portail/upload/docs/application/pdf/2010-03/guide_dm_gb_050310.pdf 479

[Accessed October 03, 2015]. 480

38. National Institute for Health and Care Excellence (NICE). Medical Technologies Evaluation 481

Programme Methods guide. 2011. Available from http://www.nice.org.uk/Media/Default/About/what-482

we-do/NICE-guidance/NICE-medical-technologies/Medical-technologies-evaluation-programme-483

methods-guide.pdf [Accessed April 22, 2016]. 484

39. Ciani O, Wilcher B, Blankart CR et al. Health technology assessment of medical devices: A survey of 485

non-European union agencies. Int J Technol Assess Health Care 2015;31:154-65. 486

40. GetReal. Glossary of definitions of common terms. Innovative Medicines Initiative (IMI) GetReal. 487

Available from: http://www.imi-488

getreal.eu/Portals/1/Documents/Publications/D1.3%20GetReal%20Glossary.pdf [Accessed February 489

22, 2016]. 490

41. Sullivan E, Mathias SD. Patient reported outcomes in therapeutic and diagnostic device research. In: 491

Ackerman SJ, Dix Smith M, Ehreth J, Eldessouki R, Sullivan E, eds., Therapeutic and Diagnostic 492

Device Outcomes Research. USA: International Society for Pharmacoeconomics and Outcomes 493

Research, 2011. 494

42. Bernard A, Vaneau M, Fournel I. et al. Methodological choices for the clinical development of medical 495

26

devices. Med Devices (Auckl). 2014;7:325-34. 496

43. Zelen M. A new design for randomized clinical trials. N Engl J Med. 1979;300:1242-45. 497

44. Devereaux PJ, Bhandari M, Clarke M et al. Need for expertise based randomised controlled trials. 498

BMJ 2005;330(7482):88. 499

45. Lilford RJ, Braunholtz DA, Greenhalgh R, Edwards SJ. Trials and fast changing technologies: the 500

case for tracker studies. BMJ 2000; 320(7226):43-6. 501

46. Campbell MJ. Cluster randomized trials in general (family) practice research. Stat Methods Med Res 502

2000;9(2):81-94. 503

47. Hills M., Armitage P. (1979). The two-period cross-over clinical trial. Br J Clin Pharmac;8:7-20. 504

48. Boudard A, Martelli N, Prognon P, Pineau J. Clinical studies of innovative medical devices: What level 505

of evidence for hospital-based health technology assessment? J Eval Clin Pract 2013;19(4):697-702. 506

49. Suter LG, Paltiel AD, Rome BN, Solomon DH, Thornhill TS, Abrams SK, Katz JN, Losina E. Placing a 507

price on medical device innovation: The example of total knee Arthroplasty. PLoS One 508

2013;8(5):e62709. 509

50. Sedrakyan A, Marinac-Dabic D, Normand SL, Mushlin A, Gross T. A framework for evidence 510

evaluation and methodological issues in implantable device studies. Med Care 2010;48(6 511

Suppl):S121-8. 512

51. Burns, Daniel K., et al. "Designing pharmacogenomic studies to be fit for purpose." 513

Pharmacogenomics 11.12 (2010): 1657-1667. 514

52. Loudon, Kirsty, et al. "The PRECIS-2 tool: designing trials that are fit for purpose." bmj 350 (2015): 515

h2147. 516

53. Bates, Susan E., et al. "Advancing clinical trials to streamline drug development." (2015): 4527-4535. 517

54. Lemmens, Trudo, and Shannon Gibson. "Decreasing the Data Deficit: Improving Post-Market 518

Surveillance in Pharmaceutical Regulation." (2014). 519

55. White, Judith, and Grace Carolan-Rees. "Current state of medical device nomenclature and 520

taxonomy systems in the UK: spotlight on GMDN and SNOMED CT." JRSM short reports 4.7 (2013): 521

1-7. 522

56. Schneeweiss, S. "Developments in post‐marketing comparative effectiveness research." Clinical 523

27

Pharmacology & Therapeutics 82.2 (2007): 143-156. 524

57. Porter, Michael E. "What is value in health care?." New England Journal of Medicine 363.26 (2010): 525

2477-2481. 526

58. Zannad, Faiez, et al. "Current challenges for clinical trials of cardiovascular medical devices." 527

International journal of cardiology 175.1 (2014): 30-37. 528

59. Bernard A, Vaneau M, Fournel I, Galmiche H, Nony P, Dubernard JM. Methodological choices for the 529

clinical development of medical devices. Medical Devices (Auckland, NZ). 2014;7:325-334. 530

doi:10.2147/MDER.S63869. 531

60. U.S.Department of Health and Human Services, Food and Drug Administration, Center for Devices 532

and Radiological Health, Center for Biologic Evaluation and Research. Design considerations for 533

pivotal clinical investigations for medical devices: guidance for industry, clinical investigators, 534

institutional review boards, and Food and Drug Administration staff. 535

https://www.fda.gov/downloads/MedicalDevices/DeviceRegulationandGuidance/GuidanceDocuments536

/UCM373766.pdf; Accessed 2/1/2017. 537

61. U.S. Department of Health and Human Services, Food and Drug Administration, Center for Devices 538

and Radiological Health, Center for Biologic Evaluation and Research. Adaptive Designs for Medical 539

Device Clinical Studies: Guidance for Industry and Food and Drug Administration Staff. 540

https://www.fda.gov/downloads/medicaldevices/deviceregulationandguidance/guidancedocuments/uc541

m446729.pdf; Accessed 2/1/2017. 542

62. U.S. Department of Health and Human Services, Food and Drug Administration, Center for Devices 543

and Radiological Health, Center for Biologic Evaluation and Research. Factors to Consider Regarding 544

Benefit-Risk in Medical Device Product Availability, Compliance, and Enforcement Decisions: 545

Guidance for Industry and Food and Drug Administration Staff. 546

https://www.fda.gov/downloads/medicaldevices/deviceregulationandguidance/guidancedocuments/uc547

m506679.pdf Accessed 2/1/2017. 548

63. Yong, Pierre L., LeighAnne Olsen, and J. Michael McGinnis. "Approaches to Assessing Value—549

Illustrative Examples." (2010). 550

64. Yue*, Lilly Q. "Special Issue on Medical Device Clinical Studies–Guest Editor's Note." Journal of 551

28

Biopharmaceutical Statistics 18.1 (2007): 1-3. 552

65. Cynthia P Iglesias. (2015) Does assessing the value for money of therapeutic medical devices require 553

a flexible approach?. Expert Review of Pharmacoeconomics & Outcomes Research 15:1, pages 21-554

32. 555

66. Sackett DL. Rules of evidence and clinical recommendations on the use of antithrombotic agents. 556

Chest. 1989;95:2S–4S. 557

67. Leung L. Validity, reliability, and generalizability in qualitative research. Journal of Family Medicine 558

and Primary Care. 2015;4(3):324-327. doi:10.4103/2249-4863.161306. 559

68. Rosen L, Manor O, Engelhard D, Zucker D: In defense of the randomized controlled trial for health 560

promotion research. American Journal of public Health. 2006, 96: 1181-1186. 561

69. GRADE Working Group. Grading quality of evidence and strength of recommendations. BMJ : British 562

Medical Journal. 2004;328(7454):1490. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC428525/ 563

70. Süt N. Study Designs in Medicine. Balkan Medical Journal. 2014;31(4):273-277. 564

71. The periodic health examination. Canadian Task Force on the Periodic Health Examination. Can Med 565

Assoc J. 1979; 121:1193–1254. [PubMed: 115569] 566

https://www.ncbi.nlm.nih.gov/pmc/articles/PMC1704686/pdf/canmedaj01457-0037.pdf 567

72. Burns PB, Rohrich RJ, Chung KC. The Levels of Evidence and their role in Evidence-Based 568

Medicine. Plastic and reconstructive surgery. 2011;128(1):305-310. 569

doi:10.1097/PRS.0b013e318219c171. 570

https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3124652/pdf/nihms288127.pdf 571

73. Evans D. Hierarchy of evidence: a framework for ranking evidence evaluating healthcare 572

interventions. Journal of clinical nursing. 2003 Jan 1;12(1):77-84. 573

74. The periodic health examination. Canadian Task Force on the Periodic Health Examination CMAJ 574

November 3, 1979 121:1193-1254 575

75. Wilsdon T, Serota A. A comparative analysis of the role and impact of Health Technology 576

Assessment. London: Charles River Associates; 2011. 577

http://medicinesaustralia.com.au/files/2010/01/HTA-Comparison-Report-Updated-July-26–2011-578

STC.pdf (accessed April 4, 2017). 579

29

76. OCEBM Levels of Evidence Working Group*. “The Oxford Levels of Evidence 2”. Oxford Centre for 580

Evidence-Based Medicine. http://www.cebm.net/index.aspx?o=5653 581

77. Saksena, Sanjeev, et al. "Clinical investigation of antiarrhythmic devices: a statement for healthcare 582

professionals from a joint task force of the North American Society of Pacing and Electrophysiology, 583

the American College of Cardiology, the American Heart Association, and the Working Groups on 584

Arrhythmias and Cardiac Pacing of the European Society of Cardiology." Journal of the American 585

College of Cardiology 25.5 (1995): 961-973. 586

78. Taylor, Rod S., and Cynthia P. Iglesias. "Assessing the clinical and cost-effectiveness of medical 587

devices and drugs: are they that different?." Value in Health 12.4 (2009): 404-406. 588

79. Rathi, Vinay K., et al. "Characteristics of clinical studies conducted over the total product life cycle of 589

high-risk therapeutic medical devices receiving FDA premarket approval in 2010 and 2011." Jama 590

314.6 (2015): 604-612. 591

80. Eichler, H‐G., et al. "From adaptive licensing to adaptive pathways: delivering a flexible life‐span 592

approach to bring new drugs to patients." Clinical Pharmacology & Therapeutics 97.3 (2015): 234-593

246. 594

81. Segal, Courtney, Erin Holve, and Raj Sabharwal. "Collecting and Using Patient-Reported Outcomes 595

(PRO) for Comparative Effectiveness Research (CER) and Patient-Centered Outcomes Research 596

(PCOR): Challenges and Opportunities." (2013). 597

82. Faris O and Shuren J. An FDA Viewpoint on Unique Considerations for Medical-Device Clinical 598

Trials. N Engl J Med 2017; 376:1350-1357. 599

83. Jessica J. Jalbert, Mary Elizabeth Ritchey, Xiaojuan Mi, Chih-Ying Chen, Bradley G. Hammill, Lesley 600

H. Curtis, Soko Setoguchi; Methodological Considerations in Observational Comparative 601

Effectiveness Research for Implantable Medical Devices: An Epidemiologic Perspective. Am J 602

Epidemiol 2014; 180 (9): 949-958. 603

84. Neumann, Peter J., and Joshua T. Cohen. Measuring the value of prescription drugs. New England 604

Journal of Medicine 373.27 (2015): 2595-2597. 605

85. Alpert, Susan, Wes Cetnarowski, and B. Braun. A Framework for Simplification of Clinical Trials in 606

Regulated Medical Devices. (2016). 607

30

85. Selaru, P., et al. Sufficiency of Single‐Arm Studies to Support Registration of Targeted Agents in 608

Molecularly Selected Patients with Cancer: Lessons from the Clinical Development of Crizotinib. Clinical 609

and translational science 9.2 (2016): 63-73. 610

86. Chow, Shein-Chung, and Mark Chang. Adaptive design methods in clinical trials–a review. Orphanet 611

journal of rare diseases 3.1 (2008): 11. 612

Kostkova, Patty, et al. Who owns the data? Open data for healthcare. Frontiers in public health. 2016:4. 613