tntsfnii

A Clinical Approach to TechnologyAssessment: How Do We and HowShould We Choose the Right Treatment?Anthony Zietman, MD,* and Geoffrey Ibbott, PhD†

The evidence required to support the use of new technology in medicine differs from thatrequired for new drugs. On one extreme, very little may be required for small devices, buton the other strong evidence is required to support the use of truly novel, potentiallydangerous, and high-cost machines. The randomized controlled trial is built into theevaluation of drugs and suits them well. It is not so well suited to the evaluation of majordevices in which installation costs and return on investment are important. We discusswhere the randomized controlled trial may still play a role and what alternatives may existwhen this is not possible. We also discuss the role that independent bodies may have indetermining whether or not a new device is not only safe but also adds to the medicallandscape in a way that justifies its cost.

Semin Radiat Oncol 22:11-17 © 2012 Elsevier Inc. All rights reserved.

New medical devices enter the marketplace every day,perhaps even every hour. This cascade covers every-

hing from minor upgrades on preexisting equipment (ie, aew grip on a scalpel or a more sensitive pulse oximeter) toruly revolutionary new technology with huge biological andocietal implications, such as artificial hearts or carbon ionacilities. All need some degree of assessment although theeeds vary. The US Food and Drug Administration (FDA), in

ts evaluation of devices, takes a very different approach fromts evaluation of drugs.1 Every drug is a new biological agentand needs testing, at least against placebo and ideally againstpreexisting alternatives. The randomized phase III trial isthought to be the best method to conduct head-to-head com-parisons with all variables controlled. Clinical effectivenesscan be determined and side effects, usually seen early, de-tected. The FDA mandates these trials, and the huge pharma-ceutical industry, recognizing that it needs them to gain ap-proval, has developed a machinery to conduct them. TheFDA prevents many drugs with inadequate evidence of effi-cacy from ever coming to market and continues to collectpostapproval data on others that do. Postapproval data gath-ering, sometimes known as a “phase IV” study, has been the

*Department of Radiation Oncology, Massachusetts General Hospital, Bos-ton, MA.

†M.D. Anderson Cancer Center, Houston, TX.Address reprint requests to Anthony Zietman, MD, Department of Radiation

Oncology, Massachusetts General Hospital, Fruit Street, Boston, MA

02114. E-mail: azietman@partners.org

1053-4296/12/$-see front matter © 2012 Elsevier Inc. All rights reserved.doi:10.1016/j.semradonc.2011.09.008

undoing of several high-profile drugs in recent years, withVioxx (Rofecoxib, Merck and Co. Inc, Whitehouse Station,New Jersey) one of the most spectacular examples.

The FDA takes a different approach to technology, assum-ing that most technologies are simply tools and that the ma-jority are evolutions of tools already in existence. If one con-siders major advances in surgery, such as fiber-optics androbotic tools, or in radiation oncology, such as a faster treat-ment delivery systems, or new on-treatment imaging devices,the truth of that statement is clear. Radiation oncology has arich history of delivery system evolution from kV x-ray units,through cobalt irradiators, to the linear accelerator, withoutany being tested by randomized trials. The role of the FDA fordevices is simply to determine that the device does what itsays it does (referred to as “efficacy”) and does so safely. Thereis a very reasonable fear that if all devices needed lengthyrandomized testing before approval, then the delay wouldrender them obsolete on arrival. In addition, with thousandsof new devices coming into use annually, where does onedraw the line? A pocket calculator is a medical device used ina clinical context, but few would recommend randomizedtesting and, if so, against what, the slide rule? Furthermore,many devices are expensive to purchase and install, and noinstitutions would buy them if they did not have a reasonablehope of amortizing their costs and making a profit in thefuture. In short, an onerous system to test the clinical efficacyof all devices would cause the engine of innovation to grind to

a halt.

11

utepatcmcmo

mpdtctfds

12 A. Zietman and G. Ibbott

Simple devices, with no risk of injury to the patient, areclassified by the FDA as class I, for which the approval pro-cess is simple. Devices perceived as presenting a moderaterisk to the patient, such as a new design of radioactive source,are designated as class II. When the new device can be de-scribed as functionally identical to a previously approved(predicate) device, a straightforward approval process calleda 510 k is permitted. When the device is considered to pres-ent significant risk and there is no predicate device, it isdesignated class III, and its safety and effectiveness must beshown. Usually this is done through the use of a limitedclinical trial. When the FDA is satisfied that both safety andeffectiveness have been shown, a premarket approval is is-sued. The decision to use or not to use a device, which hasbeen approved by the FDA for safety, is left entirely to clini-cians, physicists, and, increasingly, the hospital businessmanagers. There is a somewhat naive assumption that themarket will determine whether or not a device has clinicalmerit and deserves to live or die. If it is useful and reducesoperating room time or increases a surgeon’s field visibility, itwill survive. If it is needlessly complex or buggy, it will dis-appear.

Of course, this idealized view does not always hold true,and in recent years many expensive new devices have comeinto widespread use with a strong marketing drive and littleevidence of value to the patient or worse. In addition, somenew devices are of such enormous cost with hospitals makingmassive capital investments that other, potentially more valu-able, clinical services may suffer. When such substantial “op-portunity costs” are engendered, stronger evidence of valuemay be necessary. Policymakers in Washington DC recog-nize this.2 They know that new technology is a huge contrib-

tor to the health care budget deficit and are seeking to “bendhe cost curve” through the more rigorous application ofvidence-based medicine. The insurance companies, underressure from the employers and the dictates of competition,lso need to find ways to prune medical costs. These actors,ogether with many health policy experts, both academic andommercial, have come to form the “value movement” inedicine and are looking closely at the intersection between

ost and clinical efficacy. They ask hard questions about thearginal benefits of expensive technologies, and radiation

ncology sits squarely in their sights.In this article, we attempt to resolve the needs of policy-akers, industry, clinicians, and above all patients. We pro-ose evidence that will be required on several levels and atifferent stages of device development, roll out, and use. Weier it for the type of device and attempted to work out howlinicians can, through judicious data collection and applica-ion of their collective conscience, come to control the marketor the good of the patient. We categorize four forms of evi-ence: proof of concept, evidence of efficacy, evidence ofafety, and evidence of value.

Proof of ConceptIn the early 1990s, Neil Postman, a sociologist from New

York University, wrote a book describing what he saw as the

3 ages of man.3 First came the “Tool Age,” in which man usedsimple tools to solve simple problems. This was followed bythe “Age of Technology,” in which tools were used creativelyto solve bigger problems and became instruments of powerand prestige. He cites bridge building and architecture, en-gines, and electricity as examples from this age. Finally,comes the “Age of Technopoly,” when the technology itselfbecomes more interesting than its stated purpose and humanthought is consumed by the technological wizardry ratherthan its consequence or value. In radiation therapy, it couldbe argued that radium needles and molds represented oursimplest tools, the cobalt machine, the linear accelerator, andthe after-loader; our period of power and creative expansion;and, finally, the age we have reached today when a host ofnew smart delivery systems have been developed although itis unclear what clinical benefits, if any, some of them bring. Itis unclear whether they are bringing added value or justincreasing complexity and cost and thus decreasing safetyand efficacy.

The pharmaceutical industry is arguably more adept atdeveloping “me-too” drugs for the purpose of market share orextending patents than it is at creating novel blockbusters.This has added greatly to the cost of health care. The deviceindustry is somewhat similar, only it does so without a phaseIII testing process to at least help identify a “me-too” deviceand generate data to determine whether or not physiciansand physicists should choose to use it.

Proof of concept, either in terms of simplicity or novelty, isa prerequisite for a new device, but it does not determinewhether that device is developed to solve a problem or issimply looking for a problem to solve. If the former is correct,then we have a device with a known application for which itcan be approved and tested. If the latter is the case, then thepotential exists that, once FDA approved, it can be used “offlabel.” Examples of the former are a rectal balloon and aradiation targeting system; examples of the latter are a roboticoperating system and a proton beam delivery system.

In radiation therapy, concepts may be judged in the fol-lowing 3 ways:

1. Technical: these are systems or devices that improvethe targeting of radiation beams (eg, fiducial markers orcone beam radiography) or that may speed the deliveryand convenience of radiation delivery (eg, volumetric-modulated arc therapy, high-output linear accelerators,and high-dose rate brachytherapy). Simply showinggreater speed and convenience without sacrificingsafety is in itself justification for the application of newtechnology. The testing of patient outcome is not nec-essary.

2. Dosimetric: many systems theoretically allow for thesafer delivery of high doses of radiation through greateraccuracy and the exclusion of more normal tissue fromthe high- and low-dose volumes. The dose volume his-togram has become the comparator by which these de-vices are judged. For brachytherapy sources, dosimetryhas not been required, and the FDA grants a 510 k

approval on the basis of a predicate device. The Amer-

tp

A clinical approach to technology assessment 13

ican Association of Physicists in Medicine (AAPM) andRadiation Therapy Oncology Group (RTOG) have to-gether developed a system of approval requiring peer-reviewed dosimetry publications. This is a good exam-ple of our community stepping in to fill a regulatorygap. Although a dosimetric endpoint for both telether-apy and brachytherapy makes great sense to radiationoncologists and physicists, payers are far less impressedby it, and, mindful of cost, they demand better evidencethat patients benefit in a meaningful way. Indeed, con-cerns have been expressed that overconfidence basedon dosimetry may lead to more marginal misses of tu-mor. In 1999, a randomized trial comparing 2-dimen-sional treatment of prostate cancer with 3-dimensionaltreatment was published, clearly showing a reductionin radiation proctitis in the patients treated 3-dimen-sionally.4 No such trial has yet been performed for in-tensity-modulated radiation therapy (IMRT) or protonbeam in any clinical situation, and reimbursement mayultimately suffer as a consequence.

3. Biological: some radiation therapies potentially bringunique biology to the clinic. These include neutrons,protons (which may have a higher radiobiological ef-fectiveness [RBE] at the end of their range), carbonions, and high doses per fraction. It is sometimes saidthat radiation is unlike any drug. However, in thesescenarios, it may be remarkably similar, and significantlevels of testing will be necessary because new biologybrings unpredictability and the unforeseen.

Evidence of EffectivenessHierarchies of evidence have been developed to assist deci-sion makers in their decisions about new therapies and ther-apeutic equipment.

Randomized Controlled TrialTraditionally, the randomized controlled trial (RCT) hasbeen placed at top of the evidence hierarchy. These trialswere introduced in the mid-20th century as a method tocompare the effects of 2 different treatments in identical pa-tient groups contemporaneously. They not only compare 2therapies but can assess the magnitude of the effect in acontrolled environment. They are used to assess drug efficacyboth before FDA approval and then, often more rigorously,after approval. They validate therapy, and the validity is re-inforced by subsequent RCTs. Many in the evidence-basedmovement and in independent and payer technology assess-ment groups hold RCTs to be superior to all other forms ofclinical evidence. However, it is becoming clear that RCTShave their limitations,5 and holding all new approaches tohis standard would nullify 50% to 80% of current clinicalractice. The limitations include the following:

1. Null hypothesis: the starting assumption of the nullhypothesis is that the treatment has no benefit, and the

study is set up to prove or disprove this hypothesis.

This method ignores prior data or studies and can limitthe investigator’s ability to find small but meaningfuleffects

2. P � .05: randomized trials may become slave to the useof the “P value” to determine that which is true fromthat which is chance at an arbitrary probability of 1 in20. Trials may be stopped prematurely and use sub-group analyses inadequate to sort out real effects. Somenow advocate a “Bayesian approach,” which reversesconventional thinking and looks at the probability ofthe tested hypothesis itself being true conditional onprior and accumulated data although this is not with-out pitfalls.

3. “Generalizability”: randomized trials use purified pop-ulations with strict and limited eligibility and highlycontrolled therapy. Thus, it is not clear how generaliz-able the results will be to a real-world population.

4. Endpoints: RCTs are typically powered for reasonablynumerous and meaningful events like metastases ordeath but not for the rarer, and especially late, adverseevents. This is a particularly important consideration inradiation therapy in which serious complications mayoccur sporadically and years after therapy.

5. Costs and scale: much of the cost of RCTs is now causedby increasing regulation, with extensive paperwork be-ing required at participating sites, necessitating largecosts per patient. The speed of the trial is limited byexclusion criteria and the rate of accrual.

6. Randomization between technologies has proven to beenormously challenging as many patients come to thephysician with their own biases and preconceptions.One of the best examples of this is the concept of aproton against photon trial, in which many patients,and indeed many physicians, are unwilling to acceptrandomization because they are convinced that protonbeam will provide a superior outcome. The vendorsthemselves are understandably unwilling to put theirdevice through a trials process that may delay approvaland the recovery of investment by years. There is alsothe risk that the technology itself will be obsolete by thetime the trial reports.

Observational StudiesThis is a distinct class of evidence whose perceived validity ison the rise. Data on patients treated differently are collectedprospectively, but there is no randomization and, thus, thereis a substantial risk of bias and confounding. Advocates con-tend that eliminating the challenge of randomization allowsfor wide and substantial accrual, and these great numbersallow deeper scrutiny that may act to offset the confounders.This class of evidence contains many types of study and hasits own hierarchy. At the bottom are the case series and casereports, and at the top are case-controlled studies and con-temporaneously treated cohorts. In the middle stand studieswith historical controls and studies with before-and-after de-signs.

A newer concept in data collection that lends itself to this

pdpmlisotpcdl

14 A. Zietman and G. Ibbott

kind of investigation is known as “the registry.”6 Registriesrospectively collect data on all patients with a particularisease from a number of institutions. They collect patientarameters, information on treatment technique, and infor-ation on patient outcome in a comprehensive but nonse-

ective way. There are many examples of successful registriesn medicine that may be readily interrogated for prompt an-wers and that have particular value connecting process withutcome. The Northern New England Cardiothoracic Regis-ry has a long and successful track record of determining therocess modifications required to improve outcome afteroronary artery bypass surgery and has served to raise stan-ards of practice and reduce variation, at first regionally and

ater nationally.7 This form of data collection is particularlywell suited to new technologies. There are many such regis-tries in existence, including the collection of data on implant-able defibrillators and positron emission tomography scan-ners.

Of course, observational studies are not without their ownweaknesses. In particular, their size and heterogeneity createsstatistical “noise” that can either hide biases or drown out realeffects. It has been argued that their sheer size washes awaybias, but when compared with a well-stratified randomizedtrial with tight eligibility, they are a less rigorous method fordetecting true effects.

Endpoints relevant to radiation technologies include thefollowing:

1. Cancer control: this endpoint is important in most can-cer drug trials and in trials in which radiation therapy iscompared with surgical or pharmacologic alternatives.It could theoretically be an endpoint in dose escalationstudies if the new technology is used to deliver higherradiation doses than the standard alternative. In thislatter circumstance, it is likely that the new technologywould have been phase I to II tested for that higher dosealready.

2. A reduction in morbidity: many new technologies inour field claim to offer greater accuracy, thus allowing areduction in the normal tissue volume receiving highdoses of radiation. For many technologies, this willlikely have been shown already by comparative plan-ning studies. Some radiation complications are solidlydefined and subject to little subjectivity, such as blind-ness after irradiation of the optic chiasm or rectal bleed-ing requiring transfusion. However, most are some-what subjective, and when left to the physician, whomay be rushed and unable to ask nuanced questions, orworse the principal investigator of the study, are notscientifically rigorous.

3. Patient-reported outcomes (PROs): over the last de-cade, it has been well recognized that physician-re-ported outcomes are unreliable for many endpoints,and many validated questionnaire “instruments” nowexist to accurately determine morbidity and its conse-quence on the patient’s well-being. These may be as-cribed numeric values and used as sensitive measures

in comparative studies of any kind, whether they are

randomized or observational. These have become thebest method to assess whether a new technology claim-ing reduced morbidity on the basis of improved dosim-etry makes a meaningful difference to the patient. Al-though more reflective of the real outcome thanphysician-reported outcomes, these endpoints can alsobe distorted in tests of technology. Talcott et al8 studiedPROs in a trial in which a proton beam had been used todeliver a boost dose of radiation to men with prostatecancer. Ten years later, a substantial minority of menhad, by all objective criteria, poor urinary and boweloutcomes, yet when their degree of distress was asked,it was in many cases remarkably low. This dissonancecertainly may arise because over 10 years men canadapt to their disability. However, it is equally likelythat men receiving a therapy that they actively soughtover weeks or months of their own research may bemore likely to suppress the morbid consequences ofthat treatment knowing that they had “received thenewest and best treatment possible.” This serves as awarning as to how difficult it may be to assess trueoutcome when devices with novel and exciting con-cepts and brand names are marketed directly to thepatient using creative and seductive advertising tools.

4. Patient-reported outcomes require money and staff tocollect: regular mailings of lengthy questionnaires arenecessary although in the future secure, web-basedtools or “tablets” are likely to make the job easier. How-ever, computer literacy is not uniform across the pop-ulation, with both generational as well as socioeco-nomic dependencies. This will change as the babyboomers become the largest cancer patient populationand the economics of computer literacy continue toevolve.

5. Maintaining follow-up: patients’ age can also be diffi-cult, and significant consequences of late effects may bemissed.

Evidence of SafetySafety is theoretically ensured by the FDA, which evaluatessafety data provided by the manufacturer of the device. To apoint, this is true. No patient will be electrocuted when theyget on the treatment table, and radiation should emerge fromthe head of the machine in a reliable fashion. The FDA’simpact on safety is very important, but there are several ad-ditional factors, all of which have become very evident re-cently and are the subject of other articles in this edition ofSeminars in Radiation Oncology. Radiation planning and deliv-ery systems are complex, and buried deep within them therecan be faults that lay dormant only to emerge in a particularset of circumstances and at some later time with catastrophicconsequences. These devices do not work in isolation andmust be interconnectable with software and hardware thatmay not have been designed by the vendor. Apparent inter-connectability is insufficient, and unless the novel device hasbeen formally tested across the spectrum of other devices and

software with which it must work in the “real world,” safety is

acsdrtrplkat

cwdiatasvtmb

aptmnbd

eawabbwsetwpbrsoTWtpod

cHeowNigUsdchicitstsukfeb

A clinical approach to technology assessment 15

not ensured and the patient is at risk. Finally, these devicesmust work with human beings with differing levels of sophis-tication training and alertness. Complexity raises the risk ofhuman error. This, by itself, argues against the rapid prolif-eration of new technologies with minimal clinical testing. Ifnothing else, observational studies or a randomized trial actas a brake on proliferation, allowing time for these problemsto emerge.

Robotic technology has, in the last few years, spread dra-matically in use across surgery. It is now used in the vastmajority of radical prostatectomies performed in the UnitedStates.9 Hu et al10 have used the Surveillance, Epidemiology,nd End-results (SEER)-Medicare database to report the out-omes of radical prostatectomy performed using either theurgical robot or with an open, traditional procedure. Theata suggest that outcomes may actually be worse after theobotic procedure versus the open procedure. There appearo be higher rates of incontinence and, using positive marginates as a surrogate, worse cancer control. Whether thisoorer outcome relates to deficiencies in the device or the

earning curve of the surgeons who use the device is un-nown, but it does raise questions and is a good example ofn observational study being used to provide relatively rapidechnology assessment in a postmarketing fashion.

Evidence of Value:The Integration ofEfficacy and Safety With CostPhysicians and patients have traditionally only worried about2 things: is this treatment effective, and is it more effectivethan the alternatives? The urgent need to avoid Medicarebankruptcy makes expensive, emerging technologies a primetarget for scrutiny. It is no longer enough to say this treatmentis superior; we need to know by how much is it superior andat what economic price. Some states adopted Certificate ofNeed legislation a number of years ago in an attempt to con-tain costs. The intent was to require a facility to show theneed for expensive medical equipment based on the popula-tion served and the inadequacy of existing resources. TheCertificate of Need system has, for the most part, been a failedexperiment because it considered only cost and was easilyand frequently circumvented by the politically adept.

Health care economists talk of the quality-adjusted lifeyear as a measure of value.11 Complex models take into ac-ount additional life gained from the technology or drug andeigh it against the suffering that comes from progressiveisease with or without that treatment and from the morbid-

ty of the treatment itself. The costs of treatment are modeled,nd a cost for each additional year of quality life emerges. Inhe United States, it has been arbitrarily stated that a quality-djusted life year estimate of greater than $50,000 corre-ponds to a level above which treatment is of questionablealue.12 This bar may rise or fall according to the society andhe sensibilities of the policymakers. It was originally deter-ined for renal dialysis patients and is now applied more

roadly across medicine. In prostate cancer, both radiation c

nd surgical treatments struggle to fall under this bar. Therincipal determinants of this are the long life expectancy ofhose who receive no treatment at all and the significantorbidities of the treatments themselves. It is unlikely thatew and more expensive radiation technologies will fare anyetter. However, a low-cost treatment like brachytherapyoes.Other health care economists express these data in differ-

nt, more comprehensible ways. The Institute for Clinicalnd Economic Review develops 2-dimensional matrices inhich evidence of clinical efficacy is set on 1 axis and cost on

nother.13 The societal value of the technology is determinedy its position in the matrix. Technologies, such as protoneams, which are high-cost and low-evidence, fare poorlyhen assessed for prostate cancer. Brachytherapy and active

urveillance fare better. This situation could change if newvidence for superior efficacy was produced or the cost of theechnology came down, which are both likely. A problemith these models is the method used to assess evidence. Therocess begins with a literature review that is strictlyounded to publications predetermined to be of quality andelevance. The parameters may be set so tightly that thou-ands of papers not meeting eligibility are dismissed, andnly a limited number of “high-quality” publications survive.hus, the model is only as good as the evidence that enters it.e as a specialty are to blame when we have not generated

he evidence. The policy makers may equally undermine thisrocess by holding the RCT in excessively high esteem whenne considers its limitations and the potential to gather evi-ence other ways.In the United Kingdom, a nationally funded organization

alled the National Institute for Clinical Effectiveness andealth has been established. It performs drug and technology

vidence reviews according to a methodology that admitsbservational as well as randomized data, and it determineshether value is present for the patient and the payer (theational Health Service). This model is now widely adopted

n Europe and has undoubtedly served to constrain therowth in use of new technology. The philosophy in thenited States is very different. There is a great fear that any

uch process will stifle innovation, restrain physician free-om, reduce patient choice, and constitute a form of healthare rationing. Medicare in the United States is a nationalealth care system similar to that in the United Kingdom, but

t has, for the political reasons described, stayed away fromoverage decision based on price and value. The commercialnsurers, under pressure from employers, are now less reluc-ant. Many are returning with a critical eye to coverage deci-ions they made in the past (eg, for IMRT and proton beamherapy) and looking far more critically at new technologies,uch as stereotactic, extracranial radiation treatment, partic-larly when used for prostate cancer. Advisory companiesnown as benefit managers are being used to set up criteriaor use, according to their own metrics, and they then strictlynforce their criteria. In the face of these developments, theest outcome for the patient can only come from our spe-

ialty constraining its urge to rush into new advanced tech-

16 A. Zietman and G. Ibbott

nologies and seeking to gather meaningful evidence to offsetthis loss of control.

Evidence and EmergingTechnology in Radiation OncologyIn radiation oncology, a pragmatic dichotomy exists. Mostnew technologies represent evolutions of greater or lesserdegree, but a few represent revolutions. For these 2 scenarios,the standards of evidence must differ.

If small evolutionary tools perform their tasks as specified,in some way improving the efficiency of treatment, and theydo so at little added cost, then there is little need for evidenceover and above that of safety. The FDA does this job well, andthe technology may be adopted without additional research.

If the device delivers radiation to standard doses but doesso using a technology that is sufficiently novel that unpredict-able factors are conceivable (eg, intraoperative, electronicbrachytherapy), then more extensive phase I to II testingshould be a minimum requirement. The FDA covers this withits requirements for class III devices.

When new technologies are developed that offer increasedaccuracy or may be used to deliver higher doses than stan-dard, then deeper evaluation is required to determine theextent and balance of these effects. It is unlikely that a seriesof RCTs could keep up with the speed at which this form oftechnology is currently emerging. The entry of all patientswith 1 disease into a registry regardless of whether they aretreated conventionally or with the new tool would be partic-ularly helpful in this situation. The collection of both cancerendpoints and PROs will enable a determination of compar-ative effectiveness. Medicare’s coverage with evidence devel-opment policy exists to reimburse physicians and their insti-tutions for their investment in the new technology but drivesthem into approved studies. New technology assessmentwould be a perfect application of this policy as it allows theadoption of the technology but on the conditions of furtherresearch.

However, revolutionary changes are different, and hereadoption should ideally be delayed until the supportive evi-dence is accumulated. The RCT may well have its role in thissetting. How might “revolutionary” be defined? We haveadapted the following criteria developed elsewhere to thefield of radiation oncology:

1. Where the therapy is sufficiently novel that there is aneed for the retraining or recredentialing of physiciansor physicists: this has not yet happened in radiationoncology although it is conceivable. Such a situationhas occurred, however, in radiology with cardiac com-puted tomography scanning and in urology with thelaparoscopic prostatectomy.

2. Where the risk of benefit may be matched or offset by apotential risk of harm: in retrospect, this would havebeen the justification for an RCT comparing 3-dimen-sional radiation with IMRT in 1 or more clinical situa-tions. Better targeting improved the dose volume histo-

gram (DVH) with IMRT, but larger-dose inhomogeneity

within the target volume and increased low dose “bath”outside were of unknown consequence and potentiallyharmful. This category also includes those technologiesthat introduce a new biology in all its unpredictability(eg, the ultra-hypofractionated ablative therapies nowused in the central nervous system, lung, liver, andprostate). They could be tested in a randomized fashionagainst standard radiotherapy or surgery or any of thenonradiation ablative therapies. To date, determiningthe dose and fractionation for these techniques haslargely been guesswork in all sites, particularly theprostate. Manipulations of the alpha-beta fractionationmodel have been used in support, but this model isfraught with uncertainty, and even its strongest sup-porters urge caution in its interpretation and extrapo-lation.14

3. Particle therapy may also introduce new and unpredict-able biology as the RBE may not be known with accu-racy for either tumors or normal tissues. The RBE ofproton beam, the best established of all the particletherapies, is itself uncertain, and ranges of 1.05 to 1.15are commonly quoted.15 Although being wrong maynot matter at lower doses, once the beam is used toescalate dose close to tolerance, then small differencesof 5% to 10% start to matter.

4. When the price of the innovation is so high to thehealth system that major opportunity costs are engen-dered: in other words, the health of others may becompromised by this diversion of resources. Protonbeam and carbon ion facilities clearly fit this category.Therefore, their value must be high for them to bejustified. This means there should be strong evidence ofsuperiority compared with alternative therapies.

Proton beam technology is a variegated situation thatshows clearly the complexity of this issue. It shows the needto assess not just a technology but also its specific applicationand the need to define “on-label” and “off-label” uses. A ra-tional payment system that incentivized this would be help-ful. For pediatric cancers, the likelihood of superiority forproton beam is so high that there are few physicians withequipoise. The consequences of low-dose radiation to thenormal tissues of a child are predictably awful in a progres-sive and cumulative fashion starting with low doses. In thissituation, the proton beam is like imatinib, the novel drugthat had an effect so dramatic in gastrointestinal stromal tu-mors that no RCT was believed to be ethical.16 This may beconsidered an “on-label” use. At the opposite end are thecases in which little advantage can result from treatment withproton beam technology. Examples would include skin can-cers in which there are no large volumes of normal tissue tobe spared and lymphomas in which only low doses are nor-mally required. It is in the gray zone in between that RCTs or,at the least, rapid, high-volume observational studies mayprove very helpful. For prostate cancer, the benefits are likelysmall because many patients need no treatment and becausealternatives with low morbidity exist. For lung cancer, the

technical issues of lung movement, inhomogeneity, and mar-

ncdl

Aflphtn

A clinical approach to technology assessment 17

gins make for a “2-sided” situation in which proton beammay be either better or worse than the alternatives.

ConclusionsThere is no one-size-fits-all solution to the question of how toperform the assessment of new technologies in radiation on-cology. In the United States, we have a market-based healthcare system and are likely to have one for the foreseeablefuture. New technology is, together with new pharmaceuti-cals, the major driver of spiraling cost. In addition, new andhighly complex technology may carry safety risks. It is clearthat a brake needs to be placed on the adoption of newtechnology, but it needs to be pressed gently and selectivelyto avoid arresting our national “engine of discovery.” We, andothers,17,18 conclude that the overexuberant use of new tech-

ology may be restrained in a fashion consistent with ouronsciences as health care providers and according to evi-ence-based principles on which most could agree. The fol-

owing changes should occur:

1. Expanded role for FDA: the FDA should move beyondbasic safety and into the testing of interconnectability.Complex systems carry risks, and the FDA could deter-mine the minimum conditions under which a devicecan be used, which devices or classes of device it may beused with, and mandate a national or better yet aninternational system for the reporting of errors or mal-functions.

2. External regulation through review panels and internalcontrol through society guidelines: MedPAC has thepotential to determine centers for medicare and med-icaid services (CMS) approval for payment on the basisof comparative effectiveness evidence. To date, Med-PAC’s coverage decisions have not taken cost into ac-count. Private insurance companies are also developingtheir own evidence criteria for payment but in an adhoc, poorly informed, and self-serving fashion. Ideally,they would take their guidance from a fortified andindependent CMS or from specialty societies likeAmerican Society for Radiation Oncology (ASTRO)that developed their own criteria for use following anestablished and respected methodology. A robust set ofguidelines developed by ASTRO would frustrate someradiation oncology practices, but, in encouraging therational and justified use of new technologies, providecredibility and cover for the specialty.

3. A change in the reimbursement system: coverage forany FDA-approved device has been easy to come by.The use of coverage with evidence development wouldboth constrain rampant use of the “revolutionary” tech-nologies, particularly by those facilities unwilling or

unable to enter patients into observational studies, and

at the same time force the rapid accumulation of mean-ingful patient-oriented evidence.

4. Responsible marketing: vendors, practices, and hospi-tals have all been guilty of using new technology, notonly as a revenue source but also as a tool for self-promotion. Truth in advertising standards would freeclinicians from the urge to use hyperbolic marketingand allow us to more thoughtfully and selectively useour new treatments.

Ultimately, we believe that a responsible initiative led bySTRO, the AAPM, and other concerned societies can help to

end off rash refusals by payers to cover new technology. If weeverage our own consciences, become selective users, and allarticipate in evidence development, then the credibility andealth of both our field and our patients can be ensured. Ashe former Chair of the Federal Reserve Bank once warned,othing good results from “irrational exuberance.”

References1. Available at: http://www.fda.gov. Accessed May 10, 2011.2. Emanuel Z, Fuks VR: The perfect storm of over-utilization. JAMA 299:

2789-2791, 20083. Technopoly PN: The Surrender of Culture to Technology. New York,

NY, Vintage Books, Random House, Inc, 19934. Dearnaley DP, Khoo VS, Norman AR, et al: Comparison of radiation

side-effects of conformal and conventional radiotherapy in prostatecancer: A randomized trial. Lancet 353:267-272, 1999

5. Rawlins M: De testimonio: On the evidence for decisions about the useof therapeutic interventions. Lancet 372:2152-2161, 2008

6. Shah BR, Drozda J, Peterson ED: Leveraging observational registries toinform comparative effectiveness research. Am Heart J 160:8-15, 2010

7. Nugent WC, Nugent WC, Ross CS, et al: Building and supportingsustainable improvement in cardiac surgery: The Northern New Eng-land experience. Semin Cardiothorac Vasc Anesth 9:119-121, 2005

8. Talcott JA, Rossi C, Shipley WU, et al: Patient-reported outcomes afterconventional and high-dose combined proton and photon radiation forearly prostate cancer (PROG 9509): Long-term results from a random-ized trial. JAMA 303:1046-1053, 2010

9. Makarov DV, Yu JB, Desai RA, et al: The association between diffusionof the surgical robot and radical prostatectomy rates. Med Care 49:333-339, 2011

10. Hu JC, Gu X, Lipsitz SR, et al: Comparative effectiveness of minimallyinvasive vs open radical prostatectomy. JAMA 302:1557-1564, 2009

11. Shepard DS, Weinstein MC: Utility functions for life years and healthstatus. Oper Res 28:206-224, 1980

12. Cleemput I, Neyt M, Thiry N, et al: Using threshold values for cost perquality-adjusted life-year gained in healthcare decisions. Int J TechnolAssess Health Care 27:71-76, 2011

13. Available at: http://www.icer-review.org. Accessed May 10, 2011.14. Bentzen S, Ritter MA: The alpha/beta ratio for prostate cancer: What is

it really? Radiother Oncol 76:1-3, 200515. Dale RG, Carabe-Fernandez A, Cárabe-Fernández A: Why more needs

to be known about RBE effects in modern radiotherapy. Appl RadiatIsot 67:387-392, 2009

16. rann A, Grann VR, Daniels S, et al: The case for randomized trials incancer treatment: New is not always better. JAMA 293:970-978, 2005

17. Joensuu H, Roberts BJ, Sarlomo-Rikala M, et al: Effect of the tyrosinekinase inhibitor ST1571 in a patient with a metastatic gastrointestinalstromal tumor. N Engl J Med 344:1052-1056, 2001

18. Zietman A, Tepper J, Goitein M: Technology evolution: Is it survival of

the fittest? J Clin Oncol 28:4275-4279, 2010