#CMIMI18#CMIMI18

An update on FDA perspectives for

machine learning in medical image

interpretation Berkman Sahiner PhD, Aria Pezeshk PhD, Nicholas Petrick PhD

FDA/CDRH/OSEL/DIDSRberkman.sahiner@fda.hhs.gov

#CMIMI18

Outline FDA premarket process for medical devices FDA and machine learning for image interpretation FDA guidances Recent DeNovo devices → New paths established for the market

FDA's software pre-certification program DIDSR research related to machine learning for image interpretation

2

#CMIMI18

Office of Science and Engineering LabsMission Conduct laboratory-based regulatory research to facilitate the development and

innovation of safe and effective medical devices Provide expertise, data, and analyses to support regulatory process Collaborate with academia, industry, government, and standards development

organizations to develop, translate, and disseminate information regarding regulated products

3

#CMIMI18

Division of Imaging, Diagnostics and Software Reliability Part of OSEL Support the mission of OSEL the through

investigating issues related to Medical imaging Computer-assisted diagnosis Software reliability

• Some relevant research topics to this meeting• Making most efficient use of small datasets for

training/testing of machine learning algorithms• Continuously-learning systems

• Website: Search for FDA DIDSRhttps://github.com/DIDSR

4

#CMIMI18

FDA Premarket Process for Medical Devices

5

#CMIMI18

Medical Device Classification Intended Use / Indications for Use Risk-Based Paradigm Medical devices are classified and regulated according to their degree of risk to the

public

Class I Class II Class III

Low Risk High Risk

6

#CMIMI18

Device Class and Pre-Market Requirements

7

Device Class Controls Premarket ReviewProcess

Class I(lowest risk) General Controls Most are exempt

Class II General Controls & Special Controls

Mostly Premarket Notification[510(k)]

Class III(highest risk)

General Controls &Premarket Approval

Mostly Premarket Approval[PMA]

#CMIMI18

General Controls

All or some may be required, depending on device class Establishment registration and device listing Good Manufacturing Practices (GMPs) Labeling Record keeping Reporting of device failures Prohibition against adulterated or misbranded devices Premarket notification 510(k)

8http://www.fda.gov/MedicalDevices/DeviceRegulationandGuidance/Overview/GeneralandSpecialControls/

#CMIMI18

Special Controls

Regulatory requirements for class II devices Devices for which general controls alone are insufficient to provide reasonable

assurance of the safety and effectiveness of the device

Generally device-specific Examples of special controls Premarket data requirements Performance standards Guidelines Special labeling requirements Postmarket surveillance

9

#CMIMI18

510(k) Premarket Notification Path to market for the majority of medical devices Requires determination that a new device is substantially equivalent to a

legally marketed device (predicate device)

10

#CMIMI18

Substantial EquivalenceOption 1 Has the same intended use as the predicate; and Has the same technological characteristics as the predicate;

11

#CMIMI18

Substantial Equivalence

Option 2 Has the same intended use as the predicate;

and Has different technological characteristics and the information submitted to FDA,

including appropriate clinical or scientific data where necessary, demonstrates that the device: Does not raise different questions of safety and effectiveness than the predicate; and Appropriate clinical or scientific data demonstrate that the device is at least as safe and effective as

the predicate

12

#CMIMI18

Performance Data

Performance data needed to support a premarket submission depends on the intended use of the device

13

“as a second reader, this device assists radiologists in minimizing oversights by identifying image areas that may need as second review”

≠“this device analyzes screening mammograms and identifies mammograms that do not need any further review by a radiologist”

#CMIMI18

Premarket Approval (PMA)

Class III devices Demonstrate reasonable assurance of safety and effectiveness Very device specific Standalone submission

No comparison to a predicate

When FDA finds a device safe and effective for its intended use, it is “approved”

14

#CMIMI18

De NovoNovel devices that have not previously been classified are Class III by default

(and hence, PMA devices) De novo is a petition for down classification (Class III to Class II or Class I) De novo petition must propose controls that would be needed to assure the

safety and effectiveness of the device

15

#CMIMI18

De Novo A granted de novo establishes a new device type, a new device classification, a

new regulation, and necessary general (and special) controlsOnce the de novo is granted, the device is eligible to serve as a predicate All the followers are 510(k) devices

16

#CMIMI18

PreSubmissions (Qsubs) Informal interaction with FDA (usually non-binding) Informational Meeting Study Risk Determination (FDA decision is final) Early Collaboration Meeting Submission Issue Meeting

Helps avoid delays in device submission or repeating clinical studies

Example question: Does FDA agree that the proposed test plan is sufficient to support the stated indication?

17

#CMIMI18

Machine Learning in Image Interpretation

Computer-aided detection (CADe) Sequential reading Concurrent reading

Computerized detection Prioritization / triage Image interpretation devices used

in intra-operative procedures

• Computer-aided diagnosis (CADx)• Presence/absence of disease• Severity, stage, prognosis, response

to therapy• Recommendation for intervention

• Quantitative imaging• Many other possibilities

18

Breast BrainLung Colon Liver Urinary T. Heart Prostate

#CMIMI18

CADe Guidances

Computer-Assisted Detection Devices Applied to Radiology Images and Radiology Device Data – Premarket Notification [510(k)] Submissions

http://www.fda.gov/RegulatoryInformation/Guidances/ucm187249.htm

19

• Clinical Performance Assessment: Considerations for Computer-Assisted Detection Devices Applied to Radiology Images and Radiology Device Data

http://www.fda.gov/RegulatoryInformation/Guidances/ucm187277.htm

#CMIMI18

New DeNovo Devices for Image Interpretation

20

#CMIMI18

QuantX

Computer-aided diagnosis (CADx) device used to assist radiologists in the assessment and characterization of breast abnormalities using MR image data DEN170022

https://angel.co/quantitative-insights

#CMIMI18

QuantX FDA concluded that this device, and substantially equivalent devices of this

generic type, should be classified into Class II Radiological computer-assisted diagnostic (CADx) software for lesions

suspicious for cancer Aids in the characterization of lesions as suspicious for cancer identified on acquired

medical images such as MRI Mammography Radiography CT

Characterizes lesions based on features or information extracted from the images Provides information about the lesion(s) to the user Diagnostic and patient management decisions are made by the clinical user

22

#CMIMI18

QuantX Regulation number: 21 CFR 892.2060 Regulation Name: Radiological CADx software for lesions suspicious for

cancer Special Controls: Device and algorithm description Description of performance testing protocols and dataset used to assess whether the

device will improve reader performance Standalone performance testing protocols and results Results from performance testing protocols that demonstrate that the device improves

reader performance Appropriate software documentation

23

#CMIMI18

OsteoDetect

Analyze wrist radiographs using machine learning techniques to identify and highlight distal radius fractures Detect, identify and characterize findings

based on features or information extracted from images (CADe + CADx) DEN180005 Special controls parallel those of CADe

and CADx

https://www.slashgear.com/osteodetect-ai-tool-finds-wrist-fractures

#CMIMI18

Viz.Ai AI algorithm to analyze computed

tomography (CT) images and notify specialists regarding a potential stroke DEN170073General class: Notify an appropriate

medical specialist of pre-specified clinical conditions in parallel to standard of care image interpretation Identification of suspected findings is not

for diagnostic use beyond notification Images previewed through the mobile

application are compressed and for informational purposes only

https://twitter.com/viz_ai

#CMIMI18

Viz.Ai Regulation Number: 21 CFR 892.2080 Regulation Name: Radiological Computer Aided Triage and Notification

Software Special controls: Device and algorithm description Description of pre-specified performance testing protocols and datasets used to assess

whether the device will provide effective triage Standalone performance testing protocols and results of the device Results from performance testing that demonstrate that the device will provide

effective triage Appropriate software documentation

https://twitter.com/viz_ai

#CMIMI18

IDx-DR Automatically detect more than mild diabetic retinopathy (DR) in adults

diagnosed with diabetes who have not been previously diagnosed with DR DEN180001

Computerized detection

https://xtalks.com/idx-dr-becomes-first-fda-approved-ai-based-diagnostic-for-diabetic-retinopathy-1274/

Indicated for use with Topcon NW400 Retinal Camera

#CMIMI18

IDx-DR

28

Designed an tested for use in a primary care setting

#CMIMI18

IDx-DR Regulation Number: 21 CFR 886.1100 Regulation Name: Retinal diagnostic software device Special controls: Software verification and validation documentation Clinical performance data, including

Sensitivity, specificity, positive predictive value, and negative predictive value Variability in output performance due to both the user performing the acquisition and the image

acquisition device used User training program Human factors validation testing

29

#CMIMI18

FDA’s Software Pre-Certification Program

30

#CMIMI18

IMDRF and Software as Medical Device (SaMD) IMDRF: International Medical Device Regulators Forum SaMD: Software intended to be used for one or more medical purposes that

perform these purposes without being part of a hardware medical device Not all SaMD is AI Many uses of AI in medicine according to IMDRF’s definition is SaMD

IMDRF reports on SaMD, especially that for SaMD evaluation, are helpful to guide our thinking on AI

31

#CMIMI18

IMDRF and SaMD

SaMD: Key Definitions Dec. 2013

SAMD: Possible Framework for Risk Categorization and Corresponding Considerations Sept. 2014

SaMD: Application of Quality Management System Oct. 2015

SaMD: Clinical Evaluation Sept. 2017

32

#CMIMI18

IMDRF Final Document on Clinical Evaluation and FDA Guidance SAMD: Clinical Evaluation Adopted as FDA guidance Dec. 2017

Harmonized principles for individual jurisdictions to adopt based on their own regulatory framework, not regulations FDA intends to consider the principles of this guidance in the development of

regulatory approaches for SaMD and digital health technologies

https://www.fda.gov/downloads/medicaldevices/deviceregulationandguidance/guidancedocuments/ucm524904.pdf

33

#CMIMI18

SAMD Includes AI/ML Algorithms

34

#CMIMI18

Software Precertification Pilot Program Voluntary pathway Streamlined and efficient regulatory oversight of software based-medical

devices for manufacturers who demonstrate Robust culture of quality Organizational excellence (CQOE) Commitment to monitoring real-world performance

Latest working model: June 2018 https://www.fda.gov/downloads/MedicalDevices/DigitalHealth/DigitalHealthPreCertP

rogram/UCM611103.pdf

FDA will continue to build and refine the current working model by considering public comments and regularly seeking additional public input throughout the development of this program.

35

#CMIMI18

Key Program Components

36

#CMIMI18

Excellence Appraisal (#1) Designed for organizations of all

sizesOrganizations demonstrate

excellence based on outcomes achieved by Processes Operations Capabilities

Applies a least burdensome approach Recognizes organizations

following existing standards

37

Five principlesProduct QualityPatient SafetyClinical ResponsibilityCybersecurity

ResponsibilityProactive Culture

#CMIMI18

Review Pathway Determination (#2)

38

#CMIMI18

Streamlined Review (#3): Iterative, InteractiveUnderstanding the product FDA would work interactively with the program participant to understand the details of

the software functions

Premarket review: Interactive review of supporting information Software’s analytical performance Clinical performance Appropriate safety measures

Various options for conducting the review, possibly screen sharing, access to the development environment, and testing logs

Marketing authorization

39

#CMIMI18

Real-World Performance (#4)

40

RWPA demonstrate safety, effectiveness, and performance of the SaMD product, and should be representative of the data domains outlined here

#CMIMI18

Next Steps Comments and feedback received at the public docket reviewed every other

week Working model is iteratively refined by incorporating comments

Please provide feedback by reviewing the working model and the challenge questions! https://www.fda.gov/downloads/MedicalDevices/DigitalHealth/DigitalHealthPreCertPr

ogram/UCM611103.pdf https://www.fda.gov/downloads/MedicalDevices/DigitalHealth/DigitalHealthPreCertPr

ogram/UCM605686.pdf

Pilot program with nine companies ongoingGoal: Establish the framework of Pre-Cert 1.0 by the end of 2018 Have a limited number of organizations precertified in 2019.

41

#CMIMI18

DIDSR Research Related to Machine Learning for Image Interpretation

42

#CMIMI18

Related Research at DIDSR To prepare for the rapid progress in AI systems in radiology, DIDSR is actively

engaged on multiple ML projects.

Overarching research questions: How can a limited imaging data set be best used for accurate AI training?

Augmentation, synthetic data How can a test data set be best used for accurate and precise performance estimation

for an AI algorithm? What are some of the best and least burdensome practices for the validation of

continuously learning systems?

43

#CMIMI18

Related Research at DIDSR

Selected research projects Computational lesion insertion For algorithm training For algorithm testing

Synthetic images and lesions for algorithm training 3D CNNs for lung nodule detection Test data reuse for continuous

learning

44

Research questions Best use of limited data set for

training Accurate and precise performance

estimation Continuous learning

#CMIMI18

Computational Lesion Insertion for Training

45

Source Target Cut & Paste Seamless Insertion

Pezeshk et al.IEEE TBME 2016IEEE TMI 2017

#CMIMI18

Computational Lesion Insertion for Testing Develop a methodology to reduce the clinical data burden in the assessment

of a mammographic computer-aided detection (CADe) device under a modification in the image acquisition system Develop an image blending technique that will allow users to blend lesions

imaged using the original acquisition system into normal mammograms acquired with the new system Blend microcalcifications (µcalcs) imaged using an original acquisition system into

normal mammograms acquired with the same system

46

#CMIMI1847

Microcalcification Cluster Insertion Example

Target mammogram Insertion resultSource (original) mammogram

#CMIMI18

Reader Study A radiologist experienced in mammographic image interpretation was asked

to distinguish between original and inserted µcalc clusters

48

A rating between 0 and 100

0: Definitely inserted100: Definitely original

ROC methodology:AUC=1: original and inserted are perfectly distinguishableAUC=0.5: original and inserted are indistinguishable

AUC = 0.62 ± 0.05

#CMIMI18

Computational Lesion Insertion for Testing Purpose: Compare the sensitivity of a commercial CADe system (iCAD SecondLook v. 7.2) on

mammograms with original and inserted microcalcifications

Expectation: Similar sensitivity in original and inserted µcalc clusters

49

Detection type

Cluster type

# of cases

Detection Sensitivity (%), [95% CI]Original Inserted

By-caseMalignant 12 83.3% (10/12), [54, 96.5] 91.7% (11/12), [62.5, 100]

Benign 56 85.7% (48/56), [74, 92.8] 83.9 % (47/56), [72, 91.5]

Overall 68 85.3% (58/68), [74.8, 92] 85.3 % (58/68), [74.8, 92]

Ghanian et al.SPIE 2018Submitted to JMI

#CMIMI18

Synthetic Images for Algorithm Training

50

VICTRE (virtual imaging clinical trials for regulatory evaluation)

Breast phantom generation

Mass simulation and insertion

Monte Carlo simulation based X-ray projection

Mammographic image

preprocessing

Augment training data set

#CMIMI18

Synthetic Images for Algorithm Training

51

Mammoprojection w/o lesion

Breast Model

Mammoprojection with lesion

Projection of synthetic lesion only

Test result on independent real

data set

Cha et al.Accepted for RSNA 2018

#CMIMI18

3D DCNNs for Pulmonary Nodule Detection

Screening Discrimination Mapping of detections

Pezeshk et al.SPIE Med Imaging 2017Submitted to IEEE JBHI

Full CT volume

#CMIMI18

Pezeshk et al.SPIE Med Imaging 2017Submitted to IEEE JBHI

3D DCNNs for Pulmonary Nodule Detection

#CMIMI18

Test Data Reuse for Continuous LearningHigh-quality test data sets with reference truth are often limited in medicine If you get to interrogate the test set many times, your so-called “test” set

becomes part of training Performance seems to improve for the so-called “test” set as the device

adapts If you collect a fresh sample from the true population, you may find that

performance has actually declined

Possible mitigation strategy: Thresholdout (Dwork et al., Science 2015) In essence, add noise to reported performance each time you access the test

set Differential privacy from cryptography

54

#CMIMI18

Test Data Reuse for Continuous Learning

55

Gossmann et al.SPIE 2018Submitted to JMLR

#CMIMI18

Summary A number of DeNovo devices for image analysis that use machine learning Opens the path for similar devices through 510(k) pathway

Software precertification pilot program Aims at tailored, pragmatic, and least burdensome regulatory oversight Needs your input

FDA researchers are active in the design of tools and methods related to machine learning in medical imaging to improve public health

56