DocumentationofvaluecreatedbyRealWorldEvidence(RWE):TVTRegistrycasestudyand

proposedmetricsGregoryPappasMDPhD

AssociateDirectorforNESTFDA/CDRH

Overview

• MDEpiNetWorkgroup• Framingthequestion:DoesRWEprovidea“better,faster,cheaper”waytogenerateevidenceforregulatorydecisionmaking?• Methods• ResultsoftheTVTcasestudy• Discussion• NextSteps

MDEpiNetworkgroupwasformedtoaddresstheissues• Includesrepresentativesfromclinicalcenters,registries,researchscientists,andmanycompaniesincludingallthreeofthecompaniesthatmakethevalvesforTVT.• TheTVTCRNwasselectedasfirstcasestudybecausehavegreatestamountofexperiencetodateusingRWE.• VascularQualityInitiative(VQI)wasselectedassecondcasestudy.

• WhilefewerdecisionshavebeenmadewiththeVQI,itisprobablyamoretypicalormodelforevaluationofROI.

• Co-Chairofworkgroup,JesseBerlinJnJ• EditorofJournaloftheAmericanCollegeofCardiologyreadthedraftwhitepaperandshowedinterestinreviewingafinalmanuscriptwhichiscurrentlyundergoingfinaldraftingandrevision.• ThankstoJohnLaschinger,Changfu Wu,DougDumont,ErikaTang,andTianeyZiegler.

AreCRN(coordinatedregistrynetwork)studies“Better,Faster,Cheaper”?• OneoftheoriginalreasonsforpromotingtheuseofRWEwastobringdownthecostandtimeofevidencegeneration,andtoovercomesomeofthelimitationsofthetoolsandmethodswecurrentlyuseandovercomelimitationsoftraditionalpostmarketstudies.• LiteraturedocumentingfasterenrollmentusingCRNsexists• E.g.SAFE-PCIforWomen

• Itmakescommonsensebuttherehasnotbeenaclearmethodanddemonstration.

Definingtermsandparameters

• CRN- CoordinatedRegistryNetwork(DougkeepssayingweshouldrebrandthemasCoordinatedRWENetworks!)• TVT– Transcatheter ValveTherapy• TheTVTCRNcasestudyisusedbecauseitisthemostmatureexampleoftheuseofRWEfordeviceevaluationandregulatorydecisionmaking• Returnoninvestment(ROI)isestimatedforsponsorsasagroupandfocusesonevidencegeneration.• Dayssavediscalculatedseparatelyforreasonsexplained(dayssavedmaycontributetoROIinsomecircumstances)

Background

• TheTranscatheter ValveTherapyCoordinatedRegistryNetwork(TVTCRN)hasbeenusedtosupportover20pre-marketandpost-marketregulatorydecisions.AmethodtoevaluatethevaluecreatedbytheTVTCRNcomparedtotraditionalstudydesignsusedfordeviceevaluationisproposed.

Methods:Datasourcesandlinkage

Methods:Datasourcesandlinkage

Petersonet.al

Methods:comparingCRNcostsandtimeswith“counterfactual”studiesMethods:ReturnonInvestment(ROI)andDaysSavedwerecalculatedcomparingstudiesthatwereconducteduseoffortheTVTCRNwith“counterfactualstudies,”studiesthatwouldhavebeendoneiftheTVTregistrydidnotexist. .• BoththecounterfactualcostofstudiesandtimelengthofstudieswereprojectedusingdesignspecificationsdeterminedbyFDAreviewers• CostofthestudiesdonewiththeTCTCRNwasprovidedbyACC.• CostofthecounterfactualstudiesareestimatedusingResnic model1

[1]Wimmer,NeilJ.etal.“AssessingtheCostBurdenofUnitedStatesFDA-MandatedPost-ApprovalStudiesforMedicalDevices.” Journalofhealthcarefinance 2016.SpecFEATURES(2016):http://www.healthfinancejournal.com/~junland/index.php/johcf/article/view/82/83.Print.

Method:calculatingcostsofcounterfacutalstudiesusingtheResnic model1• Briefly,thecostestimationmodelwasdevelopedthroughiterativestructuredinterviewswith12domainareaexpertsinthedesignandmanagementofpost-approvalclinicalstudiesselectedfromindustry,academiaandclinicaltrialsiteleadership.• ModifiedDelphiapproach,consensusamongthedomainexperts• Thedriversandestimatesofstudycostsinclude:• Plannedenrollment• Numberofparticipatingcenters,• Durationofproposedstudy• Intensityoffollowup.

[1]Wimmer,NeilJ.etal.“AssessingtheCostBurdenofUnitedStatesFDA-MandatedPost-ApprovalStudiesforMedicalDevices.” Journalofhealthcarefinance 2016.SpecFEATURES(2016):http://www.healthfinancejournal.com/~junland/index.php/johcf/article/view/82/83.Print.

Resnic spreadsheet

Key 1=yes,0=no 1=yes,0=no 1=yes,0=no;2=historical # 0%- 100% # # # # 1=yes,0=no #

1=imaging,2=invasive3=other

0=cardiac,1=non-cardiac

Scenario

Total#Subjects

Recruitment Randomized ControlGroup TotalNumSites OutsideUS% StudyDuration(yrs)

#Yrs>2(Extension)

#AddPhoneCallperYr

#In-PersonperYr

RequiredProcedure?

TotalProcedureCount

TypeofProcedure(ifyes)

OrganSystem

Example 1000 0 0 0 100 0% 3.0 1 0 0 1 1 2 0P100009 2000 1 0 0 39 0% 5.0 3 0 1 1 5 1 0P100041 1100 1 0 0 27 0% 5.0 3 0 1 1 5 1 0

P100041/S039 100 1 0 0 27 0% 5.0 3 0 1 1 5 1 0P110021 1700 1 0 0 27 0% 5.0 3 0 1 1 5 1 0

P110021/S026 100 1 0 0 27 0% 5.0 3 0 1 1 5 1 0P130009 1000 1 0 0 28 0% 5.0 3 0 1 1 5 1 0

P130009/S034 200 1 0 0 28 0% 5.0 3 0 1 1 5 1 0P130009/S057 1000 1 0 0 57 0% 10.0 8 0 1 1 7 1 0

P140031 1000 1 0 0 29 0% 5.0 3 0 1 1 5 1 0P140031/S010 1000 1 0 0 57 0% 10.0 8 0 1 1 7 1 0

P130021 1000 1 0 0 41 0% 5.0 3 0 1 1 5 1 0P130021/S002 1000 1 0 0 45 0% 5.0 3 0 1 1 5 1 0P130021/S010 200 1 0 0 37 0% 5.0 3 0 1 1 5 1 0P130021/S014 250 1 0 0 24 0% 5.0 3 0 1 1 5 1 0P130021/S016 250 1 0 0 24 0% 5.0 3 0 1 1 5 1 0P130021/S025 100 1 0 0 41 0% 5.0 3 0 1 1 5 1 0P130021/S033 1000 1 0 0 65 0% 10.0 8 0 1 1 7 1 0

Methods:Time-savingsanalysis(Days-saved)TimeframefromstudydesigntoMarketingApplicationSubmission.

CET=(numberofpatientsforstudybydesign/rateofenrollment)/studydurationspecifiedbydesign

Methods:Cost-savingsanalysis

• ROI=(CostSavingsfromInvestment)/(CostofInvestment)*100• CostSavingsfromInvestment(CSI)=costoftraditionalstudies(CTS)overtimeperiodminus costofinvestment(X)

Methods:days-savedmetric

• Thetimeperiodformeasurementisthe“evidencegenerationperiod”,thetimefromthefirstenrolledsubjecttotheendofthestudy.Wearecallingthistherealscenariotimeforevidencegeneration.• Othertimefactorswereexcludedfromthemetric

• TimeforIRBvariesbroadlyfordifferentCRNandcanbecombinedwithothertimeintervals• TimeforsiteenrollmentcouldnotbemeasuredwithFDArecordsandthereforeintroducesanunderestimateinto“dayssaved”.

• Thecounterfactualtimeforevidencegenerationistherateofenrollmentpercite,timethenumberofsites,timethestudysizeplusthelengthofthestudyasspecifiedbythedesign.• RateofenrollmentistakenfromthepivotalstudyfortheTVTthatusedtraditionalmethodstoenrollpatient

• Dayssavedisthenthedifferencebetweentherealscenariotimeandthecounterfactualtime

Results:ROI

• TVTCRNwasusedtosupport23(pre-marketandpost-market)decisionssubmittedbythreemedicaldevicecompanieswereidentified. In21studies.• ThreecompaniesinvestedinTVT,fundsgiventoACCtoconductstudies.• ThecoststothesestudiesconductedintheTVTCRNis$25,050,000.• Theestimatedcostsofthecounterfactualstudiesisatotalofover$127millionusingResnic Model.ROIisestimatedtobegreaterthan400%.• Dayssavedvaryamongthestudiesconductedandrevealtimesavingsfrommonthstoyears.

Results:ROI

• Costofindividualcounterfactualstudiesvarybetween$2.2to$18.9• Totalofthecounterfactualstudies=$127million• CostofInvestment(COI)=$25,050,000• CostSavingsfromInvestment(CSI)=$127million- $25,050,000• =$102,171,539• TVTReturnonInvestment(ROI)=(CSI/COI)*100• =($102million/25,050,000)*100• =408%

Results:Days-savedStudy StudyType Total

Counterfactualstudytime(days)

Realscenario-Daysspecifiedbydesign

DaysSaved

YearsSaved

A PAS 4256 1825 2431 6.7B PAS 3756 1825 1931 5.3C LabelExpansion 2001 1825 176 0.5D PAS 4810 1825 2985 8.2E Label

expansion2001 1825 176 0.5

F PAS 3518 1825 1693 4.6G PAS 2164 1825 339 0.9H PAS 4482 3650 832 2.3I PAS 3460 1825 1635 4.5J PAS 4482 3650 832 2.3K PAS 2981 1825 1156 3.2L PAS 2878 1825 1053 2.9M PAS 2081 1825 256 0.7M PAS 2319 1825 494 1.4O PAS 2319 1825 494 1.4P PAS 1941 1825 116 0.3Q PAS 4379 3650 729 2.0R LabelExpansion 610 365 245 0.7S PAS 1988 1825 163 0.4

Discussion:ROICaveats

• SavingsonevidencegenerationisnottheonlyconsiderationforindustrywhenconsideringuseofCRN.Controloverdataisafactorthatcannotbeeasilyquantifiedinthesemetrics.• Factorswhichdriveunder-estimationofthisROI

• Combinationofstudies• UsemodelforPAScosts(Resnic et.al)knowingthatlabelexpansionstudiescostmorethanPAS

• TimerequiredbystafftomanageormonitorstudiesmaybegreaterfortraditionalmethodscomparedtoCRN

• FastertimetomarketcontributestofinancialbenefitnotincludedinthisROI(butreflectedindayssavedmetric)

• CRNprovideapossiblereplacementforsomeMDR,future

Discussion:ROI

• Whatarethedriversofthesavings?• Inherentsavings– usingclaimsdatatherewasnoneedtoconductsite-basedfollow-upofindividualsforoutcomes;fasterenrollmentwithsomanysites.• ***Leverage,reuseofdatacollectedinregistriesforotherpurposes(e.g.,qualitycontrol).

• TheROIforusingregistryforhospitalsandhealthcaresystemsisnotaddressedhere• Thereisatleastonestudythatcalculatesthebenefitofregistryusebyhospitals• TheKaiser-Permanenteregistriesmorethanpayforthemselves.KPheavilyusesitsdatatoguidemanagementofthesystem.(personalcommunication)

Discussion:Days-saved

• Days-savedmaybeanunderestimatebecauseourmetricdidnotcaptureeliminationofsiteenlisting.• Themeaningofdays-saveddiffersforpreandpostmarketstudies.• ForthePAS,thedays-savedisameasureofefficiency,butdoesnothaveaneffectontheROI.• Forlabelexpansionstudiesthedays-savedmaytranslatetofastertimetomarket.ThiswouldeffecttheROI.• Becausetimingfortakingaproducttomarketisabusinessdecision,itmaynotbethesameasdays-saved.AlsotheROIoffastertimetomarketinvolvesthesizeoftheincreasedmarketduetothelabelexpansionandthemarginonaproduct.

Publichealthbenefit

• PublichealthbenefitsofCRNinclude:• Largerstudysizedthatcaptureheterogeneityinthepopulation• Largernumberofstudysitescaptureheterogeneityamongoperators• CRNprovidesnearrealtimesurveillance,improvinguponpassivesurveillancecurrentlyreliedupon.

• ThepublichealthbenefitsoftheCRNapplytoallstakeholders.• Forthedevicemanufacturesthepublichealthbenefitstranslateto:• Fasterpotentialrecalls,andsmallerrecalls• Protectionofbrandandfewrecallsduetodetectionofproblemsearly• Fastertimetomarketmeansthatlivesavingtechnologygetstopatientsfaster

Conclusions:

• Onecasestudycannotbewidelygeneralized.ThispaperdemonstratesamethodologythatmaybeusefulforstudyofothercasesandusedtomonitorRWEuseovertime.

Nextsteps

• Casestudyofvasculardevices(VQICRN)isunderway.ThiswillprovidegetterinsightbyallowingacomparisonofcasestudiesanddiscussionofthedriversofROIandtimesaving• ThemetricsmaybeusefultotheNESTccasatooltomonitorRWEevidencegeneration.ThiswassetoutasaroleoftheCoordinatingCenterbythePlanningBoard.