artificial intelligence in medical ... › 2018meeting › handouts › allen_ai_value.pdf · •...

© 2017 | DATA SCIENCE INSTITUTE™: AMERICAN COLLEGE OF RADIOLOGY | ALL RIGHTS RESERVED.

<<November 3, 2018>>

Artificial Intelligence In Medical Imaging: Leveraging The Value of Radiology Professionals

Bibb Allen, MD FACR

Chief Medical Officer

ACR Data Science Institute


Disclosures: Dr. Allen has no relationships with ACCME-defined commercial interests.

Affiliations:

• Chief Medical Officer American College of Radiology Data Science Institute

• Former Board Chair and President ACR

Acknowledgements:

• Keith Dreyer, DO, PhD Chief Science Officer American College of Radiology Data Science Institute and Chair ACR Commission on Informatics

• ACR Data Science Institute team

CONFLICTS


1. Introduction & Background

2. Intro to AI and Imaging

3. The Value of AI in Imaging

4. The Challenges of Using AI in

Image Interpretation

5. Augmented Intelligence for

Radiologists

6. The ACR Data Science Institute

(DSI)

7. Case Study of AI in Image

Interpretation

8. An Example of a DSI Use Case

9. Concluding Slides & Discussion

Agenda


Innovation in healthcare can be lightning fast but often fraught with errors and missed opportunities that cost lives.

X-Ray as a Case Study in Rapid Adoption


Röntgen Tesla

1895


Wilhelm Roentgen discovers X-rays Wuerzburg Germany - 1895

First x-ray image in Ohio 4 months later


1st description of Physicists and physician partnership

in imaging research?

Thanks to Dr. Jeffrey Duerk

Diagnostic Imaging History

4 months later Kenyon College, Ohio judge- nail in hand.

1896

Hand of Roentgen’s wife.*Note wedding ring

1895


No innovation is without risk or unintended consequences!


Improving the health of

populations

Improving the individual

experience of care

Reducing the per capita

costs of care

Improving the work life of those who deliver care

ESTABLISHING A VALUE PROPOSITION HEALTHCARE REFORM AND FOR RADIOLOGY

Berwick DM, Nolan TW, Whittington J. The triple aim: care, health, and cost. Health

affairs. 2008 May;27(3):759-69.

Bodenheimer T, Sinsky C. From triple to quadruple aim: care of the patient requires care of

the provider. The Annals of Family Medicine. 2014 Nov 1;12(6):573-6.

May, 2008

The Quadruple Aim

What Does It Mean For

Radiology?

Why do AI if not to improve quality and value?


3 Key Actions:

Imaging 3.0 is a vision and game plan for providing optimal imaging care.

IMAGING 3.0: VALUE-BASED RADIOLOGY

11

“Our goal is to deliver all the imaging care that is beneficial and necessary and none that is not.”

Culture Change

Portfolio of IT Tools

Alignment of Incentives


3 Key Actions:

Imaging 3.0 is a vision and game plan for providing optimal imaging care.

IMAGING 3.0: VALUE-BASED RADIOLOGY

12

Culture Change

Portfolio of IT Tools

Alignment of Incentives

Clinical Decision Support for Ordering PhysiciansProviding >24 Million examinations per month

Structured ReportingIncorporated in all VR reporting platforms

Artificial Intelligence

RegistriesRadiation Exposure / Patient Outcomes / Quality

Image SharingRSNA / NIH / Vendors

Clinical Decision Support for Image InterpretationIntegrated into >75% of radiologists desktops

IMAGING

PROTOCOL

CLINICAL

CARE

INTERPRET

RADIOLOGY AI DEVELOPMENT CYCLE

Detection, Segmentation, Quantification, ClassificationQA, Workflow, Hanging Protocols, Priors Management

Artifact Reduction, Findings OptimizationDose and Contrast Optimization

Patient Positioning, Exam Protocolling,Priors Management

Exam Clinical Decision SupportPopulation Health Management

INTERPRETATION

IMAGE ACQUISITION

PRE-ACQUISITION

CLINICAL CAREAI

USE CASES

AIUSE CASES

AIUSE CASES

AIUSE CASES

There is currently a limited use of AI in clinical care

Why?

While there is significant research and preliminary applications of AI in healthcare…

HEALTHCARE DATA SCIENCE CHALLENGES

USE CASE

DEFINITION

DATA

ENGINEERING

AI MODEL

CREATION

AI/HUMAN

INTERFACE

REGULATORYBUSINESS MODEL

AI MODEL

VALIDATION

USER ADOPTIONCONTINUOUS LEARNING

CLINICAL

INTEGRATION


Infrastructural Components

Applications and Services

Third-party Applications and Services

Apple Ecosystem


Ecosystem Primary Beneficiary Other Beneficiaries

Apple Apple’s Shareholders Apple Users, Apple Partners and Employees

AI in Medical Imaging Patients Those who serve the patients (Providers, Vendors, Insurers, Regulators, Associations/Societies, Doctors)


The Imaging Lifecycle

Referring PhysicianImaging Modality

Operations

Appropriateness Determination and Patient Scheduling

Imaging Protocol Optimization

CommunicationInterpretation and

Reporting

Data Mining & Business Intelligence

Patient


Validation

Payment Models

Fear of change

Hype that can’t live up to reality

Clinical Integration

Complex AI and medical informatics

The path doesn’t look like an expedition party climbing Mt. Everest.

Regulation

Data Engineering


DATA SCIENCE AND ARTIFICIAL INTELLIGENCE ECOSYSTEM


Infrastructural Components

Applications and Services

Third-party Applications and Services

Apple Ecosystem


Ecosystem Primary Beneficiary Other Beneficiaries

Apple Apple’s Shareholders Apple Users, Apple Partners and Employees

AI in Medical Imaging Patients Those who serve the patients (Providers, Vendors, Insurers, Regulators, Associations/Societies, Doctors)


ACR DSI MISSION

Leverage the value of radiology professionals as AI evolves through the development of appropriate use cases and workflow integration

Protect patients through leadership roles in the regulatory process with government agencies and verification of algorithms

Establish industry relationships by providing credible use cases, help with FDA and other government agencies, and pathways for clinical integration

Educate radiology professionals, other physicians and all stakeholders about AI and the ACR’s role in data science for the good of our patients

AIECOSYSTEM

EDUCATION

http://acrdsi.org/media-library/pdf/Strategic-Plan-Final.pdf

AIECOSYSTEM


Advance data science as core to clinically relevant,

safe and effective radiologic care

• Establish the ACR as a global leader in advancing appropriate data science solutions

• Define, communicate and educate about the benefits of data science for all radiology professionals, patients and the greater community

• Facilitate the development of AI solutions that are free of unintentional bias

• Ensure that data science integrates into all facets of the ACR

• Develop external relationships that support and extend the ACR’s data science goal

• Promote radiology medical education that includes the skills needed to adapt to and implement data science solutions

The ACR Strategic Plan For Data Science Adopted October 2017


Establish the ACR as a global leader in advancing appropriate data science solutions

• The ACR DSI led the development of a collaboration between RSNA, ESR, RANZCR, CAR, and other organizations to develop joint statements and other position papers, as well as shared activities to advance AI for the benefit of radiologists and our patients

• Keith Dreyer presented information about the ACR DSI at the 2018 European Congress of Radiology (ECR)

• Contributed a textbook chapter, “The Role of an AI Ecosystem” for Springer’s Artificial Intelligence in Medical Imaging: Opportunities, Applications and Risks, edited by Erik Ranschaert and Sergey Morozov. Authors include an array of international AI experts

• The ACR DSI has a memorandum of understanding with Medical Image Computing and Computer Assisted Intervention (MICCAI) to provide educational content and support including the use of ACR DSI Use Cases for activities such as AI challenges.

ACR DSI AND THE ACR STRATEGIC PLAN FOR DATA SCIENCE


Define, communicate and educate about the benefits of data science for all radiology professionals, patients and the greater community

• Presentations at multiple national meetings including RSNA, ARRS, SIIM and ACR annual meetings

• Collaboration with RLI for presentations at state chapters and at the ACR-RBMA Practice Leaders Forum (January 2019)

• Collaborations with ACR Quality and Safety

• Presentations at state chapter meeting

• JACR– Data Science and Radiological Science Column

– JACR Special Issue on Artificial Intelligence

– Contribution to JACR Special issue on Health Equity

– JACR Special Issue on Data Science and Quality for July 2019

• AI Journal Advisor to begin Fall 2018



NIH / NIBIB WORKSHOP ON AI IN MEDICAL IMAGING


Facilitate the development of AI solutions that are free of unintentional bias

• ACR DSI Senior Scientist Raym Geis is heading up the ACR DSI ethics in AI project with multiple other organizations

• ACR DSI Use Cases and validation process designed to obtain training and validation across a diverse range of practices

• JACR Special Issue on Health Equity

Ensure that data science integrates into all facets of the ACR

• RLI programming as above

• Quality and Safety programming as above

• Economics program “Economics of AI” at SIIM 2018

• Collaboration with Patient and Family Centered Care Commission around issue such as data sharing and other issues regarding ethics of AI



Promote radiology medical education that includes the skills needed to adapt to and implement data science solutions

• ACR leaders are participating in the RSNA/SIIM informatics curriculum for residents (https://imaging-informatics-course.appspot.com/niic/)

• Kathy Andriole is leading an education effort for ACR members about understanding and integrating AI into clinical practice


https://imaging-informatics-course.appspot.com/niic/


• Lead - ACR to become a global leader in data science– Do the right thing: Advance data science solutions that are appropriate and free of unintentional bias

– Believe in what we do: Integrate data science into all facets of the ACR

• Define - ACR to define the beneficial uses of data science in radiology– Standards: Create standard methodologies to expand beneficial radiological data science throughout healthcare

– Relationships: Develop external relationships to support and extend our data science goals

• Educate - ACR to educate on the use of appropriate data science in radiology – Promote: Radiology medical education that includes the skills needed to adapt and implement data science solutions

– Socialize: With radiology professionals, patients and the greater community on the value of beneficial data science

The ACR Strategic Plan For Data Science - Adopted Oct, 2017

Advance data science as core to clinically relevant, safe and effective radiologic care

Lead, Define, Educate


AIMODELS

DATA

ENGINEERING

AICONCEPTS

AIAPPLICATIONS

AI DEVELOPMENT CYCLE

ACR DATA SCIENCE INSTITUTE

AIMODELS

DATA

ENGINEERING

AICONCEPTS

AIAPPLICATIONS

ACR DATA SCIENCE INSTITUTE

Define standard methods toaggregate and annotate

data for AI model training and testing

Define standard methods to integrate and monitor, AI models in clinical practice

Define standards for Use Cases considering

clinical needs and technical capabilities

Define standardizedmethods for AI model

validation consistent with regulatory processes

ACR TOUCH-AI

ACR ASSESS-AI ACR CERTIFY-AI

ACR DATA-AI


DSI SUPPORT STAFF

Chris Treml

ACR DSI Director of Operations

Current Staffing• Director of Operations• Clinical informatics analyst• 2 data engineers• Data science analyst• Communications specialist

SCALABLE


Structured Data Elements

• Training and testing AI algorithms

• Validating AI algorithms

• Monitoring algorithms in clinical practice

AI Use Case Standard• Authored by experts, used by machines• Converts human language to machine readable language• Open source authoring platform• Trusted partnerships with industry and regulators• Ensure patient safety

MOVING CLINICALLY EFFECTIVE AI USE CASES TO CLINICAL PRACTICE

ACR® TOUCH-AITOUCH-AI Technically Oriented Use Cases for Healthcare AI


ACR DATA-AIACR TOUCH-AI

The Radiology AI EcosystemIdeas To Clinical Practice

WHAT SHOULD DEVELOPERS BUILD?

EXPERT PANELS, PRIORITIZE CLINICAL NEEDS, TECHNICAL SPECS, DATA

PARAMETERS, PUBLIC INPUT



RadElement.org

ACR TOUCH-AI


ACR DATA-AI

Common Data Elements (CDEs)

STANDARDIZATION AND COMMON DATA ELEMENTS

COLLABORATION, STANDARDIZATION, INTEROPERABLE



ACR DATA-AIACR TOUCH-AI


ACR DSI DATA SCIENCE SUBSPECIALTY PANELS: ACR DSI USE CASE CREATION

ACR DSI USE CASES: AUTHORED BY EXPERTS – USED BY MACHINES


• Over 40 use cases near completion• Currently undergoing preliminary industry review• Plan to publish October 2018TOUCH-AI

ACR DSI AI USE CASE DIRECTORY


ACR TOUCH-AI

STANDARD SPECIFICATIONS FOR DATA ACCESS

• TOUCH-AI allows multiple institutions to create datasets that developers can use for algorithm training and testing

• The ACR DSI will house a freely available public directory of institutions that have created these datasets

• Using multiple sites provides technical, geographic and patient diversity to prevent unintended bias in algorithm development

• Allows more individuals and institutions to participate in AI development

ACR DSI AI-Data Directory

A Directory of Datasets For AI Training Available To Developers

AI-Data Directory



ACR DATA-AI

HOW DO WE MAKE IT?RESOURCES FOR AI DEVELOPERS

STANDARDIZATION, DIVERSITY, AVAILABILITY


DATA SCIENCE AND HEALTH EQUITY

• AI algorithms outside of healthcare have been shown to incorporate ethnic, gender and social bias

• The physician community should work with developers and regulators develop pathways to ensure algorithms marketed for widespread clinical practice are safe, effective and free of unintended bias

• Structured use cases with standards for developing datasets for training and testing

• ACR DSI validation and monitoring services, ACR Certify-AI and ACR Assess-AI, incorporate standards to mitigate algorithm bias and promote health equity

• Work with the payer and developer communities to ensure payment models for AI do not limit access to AI tools based on the socioeconomic status of our patients or the resources of their health systems.

March 2019

Health Equity

WHAT ELSE CAN THE DSI TO PROMOTE HEATH EQUITY?


ACR TOUCH-AI

ACR Certify-AI

VALIDATING AI ALGORITHMS

Specifications For Algorithm Validation and Certification

• Centralized performance assessment of AI algorithm according to statistical metrics specified in the TOUCH-AI use case

• Embargoed validation datasets are created at multiple institutions to ensure geographic, technical and patient diversity

• Guidelines for data quality to ensure “ground truth” consistency between sites

• Reports are generated for developers, agencies and customers

ACR ASSESS-AI

ACR DATA-AI


ACR CERTIFY-AI

HOW DO WE VALIDATE AI ALGORITHMS

FOR MARKETING IN CLINICAL PRACTICE?

EVALUATION METHODOLOGY, HIGHLY DIVERSE DATA, REPRODUCIBLE, HONEST

BROKER, COMPARABLE,


CERTIFY-AI WORKFLOW

GEOGRAPHIC AND TECHNICAL DIVERSITY IN VALIDATION DATASETS


PROCESS FOR ESTABLISHING PERFORMANCE OF AN AI ALGORITHM

Step Description MDDT Tool Pneumothorax Example

1 Define the use case, specifying the trigger, the measure and,

and the clinical context

TOUCH-AI Presence or absence of

Pneumothorax

2 Identify sources of variability in the algorithm’s measurements TOUCH-AI TAI-THOR0000118

3 Determine the performance metrics critical to the specific

clinical role

Certify-AI CIs for sensitivity and

specificity

4 Identify the reference data set for evaluation Certify-AI CAI- THOR00001

5 Define the minimum acceptance criteria for the metrics

identified in step 3

Certify-AI Lower bound for sensitivity is

>0.95 and the lower bound for

specificity is >0.90

6 Test the algorithm’s performance using criteria defined in step 5 Certify-AI Report See example

VALIDATION METRICS DEVELOPED AS PART OF EACH USE CASE


CERTIFY-AI VALIDATION REQUIREMENTS

ACR DSI AI USE CASES SET REQUIREMENTS FOR THE ALGORITHM AND VALIDATION PROCESS


ACCOUNTING FOR SOURCES OF VARIABILITY

ANTICIPATE REAL-WORLD VARIABILITY IN PATIENT POPULATIONS

© 2017 | DATA SCIENCE INSTITUTE™: AMERICAN COLLEGE OF RADIOLOGY | ALL RIGHTS RESERVED.PERFORMANCE CRITERIA AND STATISTICAL METRICS SPECIFIED FOR FOR EACH USE CASE

STATISTICAL SPECIFICATIONS


CERTIFY-AI REPORTS FOR DEVELOPERS AND REGULATORS


ACR TOUCH-AI

MONITORING ALGORITHM PERFORMANCE IN CLINICAL PRACTICE

Specifications For Monitoring In Clinical Practice

• Radiologist input (e.g. agree/disagree) is gathered as the case is being reported

• Specified metadata about the exam such as equipment vendor, slice thickness and exposure are also transmitted to the registry

• Assessment reports include algorithm performance metrics and the exam parameters

• Assessment reports used by the developers for algorithm improvement, continuous learning, and post-market surveillance reporting to the FDA

ACR Assess-AI

ACR CERTIFY-AI

ACR DATA-AI


ACR ASSESS-AI

HOW DO WE MAKE SURE IT WORKS IN THE

REAL WORLD?

MONITORING, SUCCESS RATE, FAILURE CONDITIONS, SCALABLE


Lung RADS 37 mm nodule

with…..

Rad Report

EHRRegistry

Other

XML Reporting Framework

(CARDS)

Full Initial LungCancer Screening AI

Visualization And Reporting

UI

Cloud / On-prem Modality

PACSTranscription

Detect and Localize

Quantify and Characterize

Classify

Registry

PerformanceAnalytics And

Quality Improvement

Registries

Developers

End Users

Regulators

AlgorithmPerformanceAssessment

AI Output

7 mm

Solid

Lung-RADS 3

AI IN CLINICAL PRACTICE WITH REGISTRY REPORTING FOR MONITORING WITH REAL-WORLD DATA

INTEGRATING AI INTO ROUTINE CLINICAL PRACTICE


ACR TOUCH-AIACR Assess-AI

ACR CERTIFY-AI

ACR DATA-AI


ACR ASSESS-AI

ACR Certify-AI

ACR DSI AI-Data DirectoryAI-Data Directory

ACR TOUCH-AITOUCH-AI Structured AI Use Cases Define Parameters For

Testing, Training, Validation And Monitoring AI

Diverse Multi-Institution Data For Testing And Training

Embargoed Multi-Institution Data For Algorithm Validation

Monitoring AI Performance In The Wild Supplements Algorithm Validation

MONITORING ALGORITHM PERFORMANCE IN CLINICAL PRACTICE


DSI ACTIVITY TIMELINE


DSI ACTIVITIES (JUN-DEC, 2018)

• ACR DSI Economics Summit (Jun)

• ACR/RSNA Common Data Elements Workshop (Jul)

• ACR/RSNA Standards Workshop (Aug)

• ACR/RSNA/NIBIB NIH AI Workshop (Aug)

• ACR DSI FDA Meeting (Sep)

• MICCAI DSI Conference (Sep)

• AAPM Webinar (Sep)

• Q&S/DSI Joint Conference (Oct)

• RSNA ML Showcase (Nov)

• FDA NEST Pilot of Certify-AI Completion (Dec)

• ACR RLI Practice Leaders Forum (Jan ‘19)

DSI ACTIVITY TIMELINE


RSNA 2018 MACHINE LEARNING SHOWCASE








Radiologists


(DSI)


Interpretation



Agenda


Radiologists who use AI will replace those who don’t.








Radiologists


(DSI)


Interpretation



Agenda


Possible Applications of AI in Medical Imaging

Image interpretation

• Quantification of findings

• Quantified comparison between multiple studies

• Multiparametric analysis across multiple modalities

• Volumetric analysis

• Textural analysis

• Automation of Region Of Interest targeting and measuring



Patient care and safety

• Detection and prioritization of potentially critical results

• Radiation dose optimization

• Pre-test probability assessment of patient risk of positive findings and contrast reactions

• Cancer and mammography screening

• Automatic protocoling of studies from EMR data



Practice optimization for productivity and quality• Automated transcription of audio narration

• Automated population of structured reports

• Optimization for case assignment across teams

• Increased accuracy of coding

• Smarter PACS hanging protocols and synchronization protocols

• Communication and tracking of primary and incidental findings

• Decreased patient waiting times

• Quality improvement in scanning

• Prediction and prevention of missed patient appointments

• Preventing imaging machine outages


1999 2015

Improving Diagnosis In Health Care


Radiologist Relevance In Error Reduction


Introducing the team of the Radiology Professional + AI

Receiver Operating Characteristic (ROC) Curves

Specificity

Sen

siti

vity

A Long Term Goal for

Radiology

0

1

1


Why I need AI to help me do better Diagnostic Radiology in Breast Imaging


1. 2. 3.



Tests are not perfect: Mammographic sensitivity decreased from a level of 85.7%–88.8% in patients with almost entirely fatty tissue to 62.2%–68.1% in patients with extremely dense breast tissue.

85.7%–88.8%

62.2%–68.1%



• A good population screening tool needs to be widely available and even in the USA we are not screening every woman who is eligible

• Management of probably benign findings costs money and human capital

• Every year 41,000 American women die of breast cancer

• Guidelines are confusing: give me more time to speak with patients


Disease progressionEarly Late

Diagnostic Imaging, AI & Population Health

PoorGoodOutcome

Det

ecti

on

Symptomatic

Asymptomatic


Disease progressionEarly Late

Symptomatic

Asymptomatic

Det

ecti

on

Pre-symptomatic

Diagnostic Imaging, AI & Population Health

PoorGoodOutcome


Radiologists Making the Most of Data Science and Artificial Intelligence

2 Help sick patients get healthy as soon as possible

1 Prevent illness

3 Stabilize & manage patients with chronic conditions

Doing Better With Less…

Improving the health of populations

Improving the individual experience of care

Reducing the per capita costs of care

Improving the work life of those who deliver care

… Through Imaging.








Radiologists


(DSI)


Interpretation



Agenda


Another challenge to using AI is that we don’t really understand how AI arrives at a particular conclusion.


Why is the algorithm effective?What’s inside the black box?

What’s in the black box?

Neonatal Intraventricular Hemorrhage

Explicability


What’s in the Black Box?


“…machine prediction is a complement to human

judgment. And cheaper prediction will generate

more demand for decision-making, so there will be

more opportunities to exercise human judgment. ”


• The patient’s prior radiation dose exposure is unknown, which

can this impact a decision of CT vs. MRI?

• Does the patient have to drive 3 hours to get to a more advanced

imaging machine?

• Does the patient have claustrophobia that makes it hard to go in

certain machines?

• The patient is losing her insurance at the end of the month, so a

follow-up exam in the future may not be feasible.

• The patient suffers from multiple, co-morbid conditions so how

sure can we be that any one condition is the cause of the finding?

• How much might we learn from an immediate follow-up study

and what are the cost-benefit factors of how this might impact

decisions about the course of treatment?

Some rewards that computer has a hard time weighing:


Anatomy

(e.g., body part)

Using Representative/Diverse Training Data: Multiple Dimensions of Image Variation

Patient demographics

(e.g., gender, age)

Pathology

(e.g., degree of tear)

Modality

(e.g., X-Ray, MRI, CT, PET, Ultrasound)


Using Representative/Diverse Training Data: Multiple Dimensions of Image Variation

Modality

(e.g., X-Ray, MRI, CT, PET, Ultrasound)

Modality-specific variations

• MRI – For example:• Techniques

(Pulse Sequences, Field of View)• Anatomic planes

(axial, sagittal, coronal)• Equipment variation

(Manufacturer, Product Version and Firmware/Software Version, Field Strengths, Signal-to-Noise Ratio)

• CT – For example:• Exposure parameters• Slice thickness• Number of detectors• Equipment variation


Goal of Image Interpretation

The Triangulation Approach to Radiographic Diagnosis

Step #1 Step #2

Correlation of radiographic findings and Gamut with patients’ clinical and lab findings to arrive at the most likely diagnosis

Step #3


4,600Unique imaging findings

www.gamuts.net contains:

13,000Unique conditions that cause findings

57,000Linkages between findings and conditions








Radiologists


(DSI)


Interpretation



Agenda


What will it be like when AI is an indispensable tool for radiology professionals?


Case Study for AI Adoption in Imaging



• Partnered with company developing algorithms looking for five

findings on chest, abdomen, and pelvis CT scans:

1) coronary calcium scores, 2) pulmonary emphysema, 3) liver

steatosis, 4) spine compression fractures, and 5) bone mineral

density

• Automatically scans images when received by the PACS and notifies

radiologists when they enter the case with a green light/red light

indicator if it identifies something

• Phased testing and adoption to obtain confidence in software and

buy-in from clinicians

• Benefits found with:

• Incidental findings that can be overlooked

• Potentially problematic bone mineral density readings that

are too early stage to be identified by the human eye


“It’s like having an extra set of

eyes to help us provide additional

information to referring physicians

for optimal patient care.”

– Dr. Arun Krishnaraj


Deep Learning: A Modern Approach to Early Breast Cancer Detection

Connie Lehman MD PhDProfessor of RadiologyHarvard Medical SchoolDirector of Breast ImagingMassachusetts General Hospital


Modern technology is better but wide variation across radiologists


Technology advances are limited by variable reader performance


2012 2013 2014


- +

+

-

+

+

-- -

--

-

-

+

-

+ or -

Pixel

Pixel

Deep Learning Methods

Pixel

Pixel

+ or -Race

Age

Family

Menopause

Traditional Methods

- +

+

-

+

+

-- -

--

-

-

+

-

High-Risk Benign Breast Lesions: Some Patients Can Avoid Surgery


Bahl et al, Radiology (2017)

100% Excised | 87% Benign Surgery Reduction

Reducing Overtreatment: High Risk Lesions

ML Model

Benign / Malignant

30%


Wide Variation in Radiologists’ Assessment of Mammograms as “Dense”


Density

Deep Learning density assessment to reduce human variation

Connie Lehman MD PhD MGHRegina Barzilay PhD MIT

Bahl et al, Radiology (2017)








Radiologists


(DSI)


Interpretation



Agenda


How can we make AI an indispensable tool for radiology professionals, referring physicians and patients?


ACR Data Science Institute: Participation

• Industry vendors

• Data scientists

• Physicians

• Informaticists

• Patient advocates

• Healthcare executives

• Regulators and policy makers

• Insurers

• Patients


The ACR: Leading the Way

109


The Tragedy of the Commons


ACR Data Science Institute: Participation

http://www.acrdsi.org/


ACR DSI Mission

Ensure the value of radiologists as AI evolves through the development of appropriate use cases and workflow integration

Protect patients through leadership roles in the regulatory process with government agencies and validation of algorithms


Educate radiologists, other physicians and all stakeholders about AI and the ACR’s role in data science for the good of our patients


Building AI

MARKETS

IDEAS

NEEDS SOLUTIONS

Assess

Assess

Concept

Concept Create

Create

Produce

Produce

AI DEVELOPMENT

CYCLE

DogsCats


Building AI

APPLICATION ENVIRONMENT(DEPLOYMENT)

RESEARCH ENVIRONMENT(IDEAS)

CLINICAL ENVIRONMENT(NEEDS)

COMPUTE ENVIRONMENT(SOLUTIONS)

Assess

Assess

Concept

Concept Create

Create

Produce

Produce

AI DEVELOPMENT

CYCLE


Clinical Data Science: Considerations

Use cases

Regulatory

Validation

Economics

Standards

Education

Commercialization

Legal

Ethical

Implementation

Content

Compute


Leverage the value of radiology professionals as AI evolves through the development of appropriate use cases and workflow integration

Protect patients through leadership roles in the regulatory process with government agencies and verification of algorithms


Educate radiology professionals, other physicians and all stakeholders about AI and the ACR’s role in data science for the good of our patients

ACR DSI Mission

EDUCATIONAIECOSYSTEM


ABDOMINAL IMAGING

BREAST IMAGING

CARDIAC IMAGING

EMERGENCY IMAGING

MUSCULOSKELETAL

NEURORADIOLOGY

NUCLEAR MEDICINE

PEDIATRIC IMAGING

THROACIC IMAGING

INTERVENTIONAL

MAGNETICRESONANCE

COMPUTEDTOMOGRAPHY

POSITRONEMISSION

RADIOGRAPHY ANGIOGRAPHY ULTRASOUND FLUOROSCOPY

ANATOMY ANATOMY ANATOMY ANATOMY ANATOMY ANATOMY ANATOMY

FINDINGS

FINDINGS

FINDINGS

FINDINGS

FINDINGS

FINDINGS

FINDINGS

FINDINGS

FINDINGS

FINDINGS

PCL Tear Use Case


Building AI

PRODUCTS

IDEAS

NEEDS SOLUTIONS

Clinical Use

Clinical Use

AI Use Case

AI Use Case

Clinical Data

Clinical Data

AI Models

AI Models

?

No standard AI use cases (annotation, validation, integration and surveillance)

No standard methods for clinical integration of AI

No standard methods for AI model validation

No standard method for AI model training and testing

AI DEVELOPMENT

CYCLE


While there are promising publication and initial applications of AI in healthcare, there is currently a limited use of AI in clinical care.

Possible Reasons Current Impact

1 Clinically effective uses for AI have been poorly defined

2 No standards for clinical integration / care management

3 Large, annotated training sets are difficult to create

4 Currently no successful economic/business models

5 Limitations in current AI/human UX/UI

6 Inconsistent results and explicability between models

7 Healthcare regulatory hurdles are challenging

8 Resulting inference models are too brittle in practice

9 Data science algorithms are limited for healthcare use

10 Poor acceptance of technology in healthcare

Healthcare AI Challenges


Building AI

CLINICAL PRODUCTS

IDEAS

NEEDS ALGORITHMS

Clinical Use

Clinical Use

AI Use Case

AI Use Case

Clinical Data

Clinical Data

AI Models

AI Models

Bring great ideas and clinical needs together

Standardized methods to annotate, or aggregate, data for

AI model training and testing

AI DEVELOPMENT

CYCLE

Mechanisms to integrate and monitor, AI models in clinical practice – using real world experience

* Standardized methods for AI model validation*


• Algorithms useful, safe and effective

• Clinically validated

• Transparency in algorithm output

• Monitored in practice

• Free of unintended bias

• Medicare and insurance coverage issues

Protecting Patients From Unintended Consequences Of AI

!


Applications of AI in Medical Imaging

Image interpretation

• Quantification of findings

• Quantified comparison between multiple studies

• Multiparametric analysis across multiple modalities

• Volumetric analysis

• Textural analysis

• Automation of Region Of Interest targeting and measuring



Patient care and safety

• Detection and prioritization of potentially critical results

• Radiation dose optimization

• Pre-test probability assessment of patient risk of positive findings and contrast reactions

• Cancer and mammography screening

• Automatic protocoling of studies from EMR data



Radiologist and optimization for productivity and quality• Automated transcription of audio narration

• Automated population of structured reports

• Optimization for case assignment across teams

• Smarter PACS hanging protocols and synchronization protocols

• Communication and tracking of primary and incidental findings

• Decreased patient waiting times

• Quality improvement in scanning

• Prediction and prevention of missed patient appointments


ACR DSI Data Science

Panels

ACR DSIUse CaseDirectory(Public)

ACR DSIDataset

AuthoringUtilities(Public)

ACR DSIPerformance

Analytics ServiceNRDR

AI Registry

ACR DSIAlgorithm Validation

Service

AI modelDeployment

Directory(Public)

Testing/TrainingDatasetsDirectory(Public)

ACR Assist-AITM

ClinicalDirectory(Public)


Radiology’s Value Proposition

• Trusted partnerships with industry and regulators

• Ensure patient safety

• Increase radiology professionals’ value in healthcare

Use Case Development• Use case authoring platform• Human language to machine language

Moving Clinically Effective AI Use Cases To Clinical Practice: Radiology AI Ecosystem

ACR®TOUCH-AI


RadElement.org

AI Data Elements

Common Data Elements



Panels


ACR DSIDataset


ACR DSIPerformance


AI Registry


Service

AI modelDeployment

Directory(Public)


ACR Assist-AITM


Making Datasets For AI Training Available To Developers

Standard Specifications For Data Access

ACR DSI Data Access

Directory

Specifications For Data Access

• Standardized definitions and data elements allow multiple institutions to use these standards to create datasets that developers can use for algorithm training and testing.

• Specifications include standardized tools and methods for image annotation.

• Using multiple sites as data sources for these datasets provides technical, geographic and patient diversity to prevent unintended bias in algorithm development.

• Allows more individuals and institutions to participate in AI development.

• The ACR DSI will house a freely available public directory of institutions that have created these datasets around ACR DSI Use Cases to inform the developer community.



Panels


ACR DSIDataset


ACR DSIPerformance


AI Registry


Service

AI modelDeployment

Directory(Public)


ACR Assist-AITM


ACR Data Science Institute Certified Algorithms

Validating AI Algorithms

Specialty society certification of AI

algorithms provides an “honest broker”

partnership with radiology, developers

and government regulators

Specifications For Algorithm Validation

• Centralized assessment of algorithm performance will be performed according to the statistical metrics metrics specified in the use case using novel datasets.

• These validation datasets are created at multiple institutions to ensure geographic, technical and patient diversity within the validation dataset.

• Multiple readers and guidelines for data quality to ensure “ground truth” consistency between sites, consistent metrics for measuring performance across sites and standards to protect developers’ intellectual property, ensure patient privacy and diminish bias.

• Reports are generated for developers


ACR DSIPerformance


AI Registry


Service

AI modelDeployment

Directory(Public)

ACR Assist-AITM


Monitoring Algorithm Performance In Clinical Practice

Specifications For Monitoring In Clinical Practice

• Data elements in each use case specify how the algorithm will be monitored in clinical practice.

• Radiologist input is gathered as the case is being reported, and if the radiologist does not incorporate the algorithm inferences into the report, this change is captured in the background by the reporting software. If the radiologists agrees, changes the output of the agrees with algorithm, this is also noted and transmitted to the registry.

• Specified metadata about the exam such as equipment vendor, slice thickness and exposure are also transmitted to the registry.

• Algorithm assessment reports include algorithm performance metrics and the exam parameters affecting the algorithms’ performance.

• These reports are used by the developers to report to the FDA and for algorithm improvement.

AI Monitoring Program

• Patient safety and FDA surveillance

• Algorithm transparency and radiologist acceptance

• Developer improvements



AI Algorithm Assessment Reports For Each Developer

And Site

Raw Data Captured In Reporting Software And Transmitted To

An AI Registry

Raw Data From Reports And Modalities

Aggregated In Registry

Working Example of Monitoring Algorithm Performance Using An AI Data Registry

• This example is from a pediatric bone age classification algorithm. The reporting software, PACS or the modality transmits information about the radiologist’s agreement or disagreement with the algorithm along metadata about the examination to the AI data registry.

• The raw data are complied in the registry and reports are aggregated and developer specific reports are generated for developers for use in FDA post-market surveillance reports and to improve the algorithm.

• Site reports are provided to provide AI performance metrics to the clinical practices.



PostmarketSurveillance

NationalEvaluation

System

“Real World” Data

TIME TO MARKET

Expedited AccessPathway

PremarketReview

Prem

arket Decisio

n

Benefit Risk

INFORMATION FLOW

“Safety Net”

Courtesy Greg Pappas, FDA


Validating AI Algorithms – Regulatory Collaborations

Office of Science And Engineering Labs FDA Center For Devices And Radiological Health


Validating AI Algorithms – Regulatory Update

Academic Partners And Industry Partners

NEST will evaluate program for using real word data to assess AI algorithms

Individual components of the validation process will support applications for MDDT


Integrating AI Into Clinical Workflow


with…..

Rad Report

Lung RADS 3 nodule with…..

Rad ReportRegistry

ReportUI

XML Reporting

Framework

ReportUI

XML Reporting

Framework

Classic Radiologist Decision Support

Hybrid Radiologist Decision Support With AI

AI

Rad Report

Full integrated AI

7 mm

Solid

Lung-RADS 3

Radiologist Input

7 mm

Solid

Lung-RADS 3

Radiologist Input


Integrating AI Into Clinical Workflow: DSI Use Case Implementation


with…..

Rad Report

EHRRegistry

Other

XML Reporting Framework

(CARDS)

Full Initial LungCancer Screening AI

Visualization And Reporting

UI

Cloud / On-prem Modality

PACSTranscription

Detect and Localize

Quantify and Characterize

Classify

Registry

PerformanceAnalytics And

Quality Improvement

Registries

Developers

End Users

Regulators

AlgorithmPerformanceAssessment

AI Output

7 mm

Solid

Lung-RADS 3


The Importance Of Transparency


Use Cases Content Validation Implementation Regulatory Safety

Economics Standards Education Facilitation Legal Ethical

DSI and Healthcare AI IndustryServices to assist industry deliver successful AI solutions to clinical practices• AI Use Case Development (ACR TOUCH-AI)• AI model Certification (ACR CERTIFY-AI)• AI model Integration (ACR ASSIST)• AI model Assessment (ACR DSI ASSESS and ACR AI REGISTRY)

ACR DSI Activities And Relationships: Industry

Moving AI From Concept To Clinical Practice


ACR DSI Use Cases

Highest clinical value

Solvable by artificial

intelligence

USE CASES

Use Case Prioritization

AI


ACR Data Science Institute Use Case Development: Data Science Subspecialty Panels


“The ACR is actively creating use cases for imaging AI and will be working with MICCAI under this memorandum of understanding to leverage this knowledge base in MICCAI’s imaging AI competitions.

ACR will also work with MICCAI to promote learning on a global scale…”


TOUCH-AI: Common Data Elements (CDE) and RadElements

RadElement.org

AI Data Elements

TOUCH-AI

ACR

MonitorData elements for monitoring

in clinicalpractice

ConceptNarratives and

Flowcharts

ValidateData elements and metrics for

validation

BuildData elements

to annotate,train, and test

IntegrateData elements

for clinicalintegration


• Useful

• Safe and effective in clinical practice

• Performance monitored and improvements made based on real world data

• Transparent

• Ensure diversity and preventing unintended bias

Summary Of ACR DSI Objectives








Radiologists


(DSI)


Interpretation



Agenda


NORMAL GRADE I GRADE II GRADE III GRADE IV

Mortality + + ~20% ~90%

DSI Pediatric Panel

Neonatal Intraventricular Hemorrhage


Is there PVHemorrhage?

(Y/N)

Is there GM hemorrhage?

(Y/N)

Is there IV hemorrhage?

(Y/N)

Is thereHydrocephalus?

(Y/N)

PV Hemorrhage

GM BleedAI Use Case

IV BleedAI Use Case

HydrocephalusAI Use Case

Calculate IVH Grade

IVHGrade(I-IV)

ClinicalManagement

Saliency maps

Saliency maps

ACR Assist (TOUCH-AI) Module: Intraventricular Hemorrhage Reporting

Neonatal Germinal Matrix Hemorrhage Detection System

Header• Purpose - To detect germinal matrix hemorrhages in newborns on imaging.• Description - Long narrative• TOUCH-AI-ID - TAI.5001• Types - Ultrasound (US) Head• Referenced Clinical Algorithms — Papile IVH Grading System• Age – Neonates• Sex – All• Logic – External

Data• Input

• Mandatory - US Head• Optional — Birth weight

• Output• Mandatory - Presence of germinal matrix hemorrhage. (CDE-II) • Optional - Quantification Of GM hemorrhage. (CDE-12)• Optional - Saliency map of GM hemorrhage. (CDE-13)


AI could help the radiologist to detect and/or quantify the following:

Abdominal Imaging• Liver steatosis

Breast Imaging• Malignant breast lesions

Cardiac Imaging• Coronary calcium scores• Risk for aortic aneurysms• Quantify LV/RV stroke volume, ejection

fraction, cardiac output and mass (to speed analysis)

Neuroradiology & Emergency Imaging• Brain bleed locations and assess severity (to

automatically move a case to the top of the radiology group’s worklist)

Musculoskeletal• Spine compression fractures• Bone mineral density

Nuclear Medicine• Alzheimer's disease with beta-amyloid PET/CT

(well before the onset of symptoms)

Pediatric Imaging• Tiny fractures in any bone

Thoracic Imaging• Lung nodules in Chest CTs• Pulmonary emphysema• Risk for pulmonary hypertension








Radiologists


(DSI)


Interpretation



Agenda


“The future is already here.It’s just not evenly distributed.”

William Gibson


Summary


AI will persistently and pervasively enhance all aspects of radiology

• It’s not about Human vs AI.

• It is about Human augmented by AI vs.

Human working without AI


AI will expand today’s decision-making capabilities

• Earlier and better detection leads to

better treatment options and improved

outcomes


Meaningful AI will improve quality, efficiency and outcomes

• Utilizing all available data to optimize

patient care


My patients (i.e., your friends, family, colleagues and neighbors)thank you for all of your great work!


Discussion and Q&A

Q? A!

155

artificial intelligence in medical ... › 2018meeting › handouts › allen_ai_value.pdf · •...

Documents