MedicalMultimediaInformationSystems
KlausSchoeffmann1, BerndMünzer1, Pål Halvorsen2,MichaelRiegler2
1 InstituteofInformationTechnology
KlagenfurtUniversity,Austria
2 SimulaResearchLaboratory
Norway
• Introduction&Overview• MultimediaDatainMedicine• CharacteristicsofEndoscopicVideo• DifferentFieldsandCommunities
• Application1:Post-ProceduralUsageofSurgeryVideos• Domain-SpecificStorageforlong-termArchiving• VideoContentAnalysis• Visualization,Interaction&Annotation
• Application2:DiagnosticDecisionSupport• Knowledgetransfer• Analysis• Feedback
• Conclusions&Outlook
Agenda
ACMMultimedia2017Tutorial MedicalMultimediaInformationSystems(MMIS) 2
Introduction
ACMMultimedia2017Tutorial MedicalMultimediaInformationSystems(MMIS) 3
Inspectionsandinterventionproducemanykindsofdata• Medicaltext
• ORreports,Patientrecords…
• Sensorsignals• ECG,EEG,vitalsigns
• Medicalimages(radiology)• Ultrasound,x-ray• CT,MRI,PET,…
• Medicalvideo• Opensurgery• Microscopicsurgery• Endoscopicinspections• Endoscopicsurgery
Multimedia Data in Medicine
ACMMultimedia2017Tutorial MedicalMultimediaInformationSystems(MMIS) 4
Communities:
• SignalProcessing
• MedicalImaging
• Computer-AssistedSurgery/Robotics
• Multimedia
„HumanEEGwithout alpha-rhythm“by Andrii Cherninskyi /CCBY-SA
„Pankreatitis“by Hellerhoff/CCBY-SA„Ultrasound“,PublicDomain
• Traditionalopensurgery?
• Minimallyinvasiveinterventions
• Reducedtraumaforpatient
• Inherentlyavailablevideosignal
• Usefulfordocumentation
• Microscopicsurgery
Video Data Sources in Medicine
ACMMultimedia2017Tutorial MedicalMultimediaInformationSystems(MMIS) 5
„Laparoscopy“,PublicDomain
„KussmaulGastroscopy“,PublicDomain
Diagnostic Endoscopy
ACMMultimedia2017Tutorial MedicalMultimediaInformationSystems(MMIS) 6
• Diagnosis/Inspections• Gastroenterology(colonoscopy,gastroscopy)
• Bronchoscopy
• Hysteroscopy
• …
• Flexibleendoscope
• Naturalorifices
• WCE(Wirelesscapsuleendoscopy)„Colonoscopy“,PublicDomain
„Kolontransversum“by J.Guntau /CCBY-SA
Therapeutic Endoscopy
ACMMultimedia2017Tutorial MedicalMultimediaInformationSystems(MMIS) 7
• Therapy/Surgery• Laparoscopy
• Cholecystectomy
• GynecologicalSurgery
• UrologicalSurgery
• …
• Arthroscopy
• …
• Rigidendoscope
• SmallIncisions„Laparoscopy“by BruceBlaus /CCBY
„Arthroscopy“,PublicDomain
Endoscopic Video Examples
ACMMultimedia2017Tutorial MedicalMultimediaInformationSystems(MMIS) 8
Domain-specific Characteristics & Challenges
ACMMultimedia2017Tutorial MedicalMultimediaInformationSystems(MMIS) 9
• FullHDor4K(evenstereo3D)
• Singleshot recordings
• Up to multiplehours
• Homogenous color distribution
• Visually very similar content
• Circular content area
• Restricted motion
• Geometric distortion
• Specular reflections
• Occlusions
• Smoke
• Noise,motion blur,blood,flying particles
ResearchFieldsandCommunities
ACMMultimedia2017Tutorial MedicalMultimediaInformationSystems(MMIS) 10
Overview
ACMMultimedia2017Tutorial MedicalMultimediaInformationSystems(MMIS) 11
Münzer,Bernd,KlausSchoeffmann,and LaszloBöszörmenyi."Content-based processing and analysis of endoscopicimages and videos:Asurvey."MultimediaToolsand Applications (2017):1-40.
Pre-Processing
• ImageEnhancement• Contrastenhancement,colormisalignmentcorrection…
• Cameracalibrationanddistortioncorrection• Specularreflectionremoval• Combstructureremoval&superresolution• …
• InformationFiltering• FrameFiltering• ImageSegmentation
ACMMultimedia2017Tutorial MedicalMultimediaInformationSystems(MMIS) 12
T.Stehle.Removalofspecularreflectionsinendoscopicimages.ActaPolytechnica:JournalofAdvancedEngineering,46(4):32–36,2006.
J.Barreto,J.Roquette,P.Sturm,andF.Fonseca.AutomaticCamera Calibration AppliedtoMedicalEndoscopy.In20thBritishMachineVisionConference(BMVC’09),2009.
B.Münzer,K.Schoeffmann,andL.Böszörmenyi.RelevanceSegmentationofLaparoscopicVideos.In2013IEEEInternationalSymposium onMultimedia(ISM),pages84–91,Dec.2013.
A.Chhatkuli,A.Bartoli,A.Malti,andT.Collins.Liveimageparsinginuterinelaparoscopy.InIEEEInternationalSymposiumonBiomedicalImaging(ISBI),2014.
Real-time Support at Intervention Time
Applications
§ Diagnosissupport
§ Robot-assistedsurgery
§ ContextAwareness
§ Augmentedreality
ACMMultimedia2017Tutorial MedicalMultimediaInformationSystems(MMIS) 13
“Roboticsurgicalsystem”,PublicDomain
T.Collins,D.Pizarro,A.Bartoli,M.Canis,andN.Bourdel.Computer-AssistedLaparoscopicmyomectomybyaugmentingtheuteruswithpre-operativeMRIdata.In2014IEEEInternationalSymposiumonMixedandAugmentedReality(ISMAR),pages243–248,Sept.2014.
„DaVinciSurgical System“by Cmglee /CCBY-SA
Slightlymodifiedfrom:M.P.Tjoa,S.M.Krishnan,etal.Featureextractionfortheanalysisofcolonstatusfromtheendoscopicimages.BioMedical EngineeringOnLine,2(9):1–17,2003.
• 3Dreconstruction
• Deformingtissuetracking
• ImageRegistration
• Instrumentdetectionandtracking
• Surgicalworkflowunderstanding
Enabling Techniques
ACMMultimedia2017Tutorial MedicalMultimediaInformationSystems(MMIS) 14
L.Maier-Hein,P.Mountney,A.Bartoli,H.Elhawary,D.Elson,A.Groch,A.Kolb,M.Rodrigues,J.Sorger,S.Speidel,andD.Stoyanov.Opticaltechniques for 3Dsurface reconstruction incomputer-assisted laparoscopic surgery.MedicalImageAnalysis,17(8):974–996,Dec.2013.
S.Giannarou,M.Visentini-Scarzanella,andG.Z.Yang.Affine-invariantanisotropic detector for softtissue tracking inminimally invasivesurgery.InBiomedicalImaging:From Nanoto Macro,2009.ISBI’09.IEEEInternationalSymposiumon,pages 1059–1062,2009.
Post-Procedural Applications
Managementand Retrieval• Compression and storage• Content-based retrieval• Temporalvideo segmentation• Videosummarization• Visualization &Interaction
QualityAssessment§ Skillsassessment
§ Education&Training
§ ErrorRating
§ Assessmentof intervention quality
ACMMultimedia2017Tutorial MedicalMultimediaInformationSystems(MMIS) 15
M.Lux,O.Marques,K.Schöffmann,L.Böszörmenyi,andG.Lajtai.Anovel tool for summarization of arthroscopic videos.MultimediaToolsandApplications,46(2-3):521–544,Sept.2009.
D.Liu,Y.Cao,W.Tavanapong,J.Wong,J.H.Oh,andP.C.deGroen.Quadrantcoveragehistogram:anewmethodformeasuringqualityofcolonoscopic procedures.InEngineeringinMedicineandBiologySociety,2007.EMBS
2007.29thAnnualInternationalConferenceoftheIEEE,pages3470–3473,2007.
J.Muthukudage,J.Oh,W.Tavanapong,J.Wong,andP.C.d.Groen.ColorBasedStoolRegionDetectioninColonoscopyVideosforQualityMeasurements.InY.-S.Ho,editor,AdvancesinImageandVideoTechnology,number7087inLectureNotesin
ComputerScience,pages61–72.SpringerBerlinHeidelberg,Jan.2012.
• Vision• Archivetogetherallrelevanttext,image,andvideodata• Usedataforinformationretrieval• Supportsurgeonsatdiagnosis,surgeryplanning,teaching,…• Combinedifferentkindofdata(e.g.,radiology-supportedsurgery)
• Challenges• Isolatedsystems/separationofdata• VeryBigData• Alotofirrelevantcontent• Veryspecificdomaincharacteristics• Needfor domain expertknowledge• Differentcommunities and views
Medical Multimedia Information Systems
ACMMultimedia2017Tutorial MedicalMultimediaInformationSystems(MMIS) 16
Post-ProceduralUseofSurgeryVideos
ACMMultimedia2017Tutorial MedicalMultimediaInformationSystems(MMIS) 17
• Videorecordingsofendoscopicsurgeriesshowthesameimagesthesurgeonusedforoperation
• Valuableinformationforpost-proceduraluse:• Laterinspectionofspecificmoments• Discussionofcriticalmoments(e.g.,withOPteam)• Informationtopatients• Preparationoffutureinterventions• Forensics&investigations(e.g.,comparisons)• Trainingandteaching• Surgicalqualityassessment(technicalerrors)
Video as the ’’Eye of the Surgeon’’
ACMMultimedia2017Tutorial MedicalMultimediaInformationSystems(MMIS) 18
Full Storage of Endoscopic Videos
• Exemplaryhospital• 5departments(Lap,Gyn,Arthro,GI,ENT)• 2operationrooms,each4ops/day,eachopca.1-2h• à i.e.40interventionsperday,each~90mins.
• 60hoursvideoperday!• Assumption:HD1920x1080,H.264/AVC• 270GB/day(1h=4.5GB)• 1.9TB/week• 100TB/year(200TBMPEG-2) 4Kabouttwiceasmuch!
(unlessencodedwithH.265/HEVC)
ACMMultimedia2017Tutorial MedicalMultimediaInformationSystems(MMIS) 19
Greatchallengeforahospital’sITdepartment!
How to Reduce Storage Requirements?
1. Spatial compression optimization
2. Temporal compression optimization
3. Perceptual quality based optimization
Transcoding
ACMMultimedia2017Tutorial MedicalMultimediaInformationSystems(MMIS) 20
upto30%
upto40%upto93%
Study on Video Quality
• Subjectivequalityassessment• CatharinaHospitalEindhoven,NL• 37participants
• 19experiencedsurgeonsand18trainees• 7women,30men,averageage:40years
• Subjectivetestsregardingmaximumcompression1) Perceivablequalityloss
• Double-Stimulus(ITU-RBT.500-11)• Switchbetweenreferenceandtestvideo
2) Perceivablesemanticinformationloss• SingleStimulus(ITU-RP.910)• Assessingrandomvideos(incl.reference)
Münzer,B.,Schoeffmann,K.,Böszörmenyi,L.,Smulders,J.F.,&Jakimowicz,J.J.(2014,May).Investigationoftheimpactofcompressionontheperceptionalqualityoflaparoscopicvideos.In2014IEEE27thInternationalSymposiumonComputer-BasedMedicalSystems (pp.153-158).IEEE.
Session1 Session2
ACMMultimedia2017Tutorial MedicalMultimediaInformationSystems(MMIS) 21
Assessment of Video Quality (Session 1)
-5
0
5
10
15
20
25
30
35
0
3000
6000
9000
12000
15000
18000
21000
24000
20 22 24 26 28 18 20 22 24 26 18 18
Diffe
renceMeanOpinion
Score(D
MOS)
Bitrate(Kb/s)
TestConditions
Averagebitrate Ratingdifference
1920x1080 1280x720 960x540 640x360
subjectivelybetterthanreference
Referencevideo(MPEG-2,HD,20(35)Mbit/s)
“lossless”
ACMMultimedia2017Tutorial MedicalMultimediaInformationSystems(MMIS) 22
crf(constantratefactor)
Assessment of Video Quality (Session 2)
1. Visuallylosslesswith8Mbit/sQ1(incomparisonto20Mbit/s)Reduction:60%datavs.0%MOS
2. Goodqualitywith2,5Mbit/sandQ2reducedresolution(1280x720)Reduction:88%datavs.7%MOS
3. Acceptablequalitywith1,4Mbit/sQ3andlowerresolution(640x360)Reduction:93%datavs.31%MOS
1
23
ACMMultimedia2017Tutorial MedicalMultimediaInformationSystems(MMIS) 23
Example Videos
1280x720Weakcompression
16MB
(crf 18)
640x360Strongcompression
0,8MB
(crf 26)
20x
ACMMultimedia2017Tutorial MedicalMultimediaInformationSystems(MMIS) 24
EndoscopicVideoContentAnalysis
ACMMultimedia2017Tutorial MedicalMultimediaInformationSystems(MMIS) 25
1000frames(sampledfrom17minwith1fps)
ACMMultimedia2017Tutorial
MedicalMultimediaInformationSystems(MMIS)
26
Content Relevance Filtering / Instrument Recognition
ACMMultimedia2017Tutorial MedicalMultimediaInformationSystems(MMIS) 27
Münzer,B.,Schoeffmann,K.,&Böszörmenyi,L.(2013,December). Relevancesegmentationoflaparoscopicvideos.InMultimedia(ISM),2013IEEEInternationalSymposiumon(pp.84-91).IEEE.
Primus,M.J.,Schoeffmann,K.,&Böszörmenyi,L.(2015,June).Instrumentclassificationinlaparoscopicvideos.InContent-BasedMultimediaIndexing(CBMI),201513thInternationalWorkshopon(pp.1-6).IEEE.
Instrumentdetectionforcontentunderstanding(e.g.,opphasesegmentation,followinginstrumentsinrobot-assistedsurgery)
Out-of-patientScenes BlurryScenes BorderArea
Phase Segmentation (Cholecystectomy)
ACMMultimedia2017Tutorial MedicalMultimediaInformationSystems(MMIS) 28
ManfredJ.Primus,KlausSchoeffmann andLaszloBöszörmenyi.“TemporalSegmentationofLaparoscopicVideosintoSurgicalPhases“,inProceedingsofthe 14thInternationalWorkshoponContent-BasedMultimediaIndexing(CBMI2016),Bucharest,Romania,2016
à Phasesegmentationthroughinstrumentrecognition(coloranalysis,imagemoments,rules/heuristics)
Instrument Recognition/Tracking
ACMMultimedia2017Tutorial MedicalMultimediaInformationSystems(MMIS) 29
Classification of OP Scene (Cataract Surgeries)
ACMMultimedia2017Tutorial MedicalMultimediaInformationSystems(MMIS) 30
ManfredJ.Primus,DorisPutzgruber-Adamitsch,MarioTaschwer,BerndMünzer,Yosuf El-Shabrawi,LaszloBöszörmenyi,andKlausSchoeffmann.“Frame-BasedClassificationofOperationPhasesinCataractSurgeryVideos“. Proceedingsofthe24thInternationalConferenceonMultimediaModeling2018(MMM2018),Bangkok,Thailand,2018,pp.1-12, toappear
Learning Medical Semantic (e.g., Surgical Actions)
ACMMultimedia2017Tutorial MedicalMultimediaInformationSystems(MMIS) 31
1.105Segments/823.000Frames/9hannotatedVideo (outof111interventions)
Dissection – 58Segs /35.517Pics Coagulation – 212Segs /84.786Pics Cutting cold – 271Segs /26.388Pics
Cutting – 106Segs /92.653Pics Hysterectomy – 25Segs /68.466Pics Injection – 52Segs /52.355Pics
Suturing – 92Segs /321.851PicsSuction &Irrigation– 173Segs /73.977Pics
Petscharnig,S.,&Schöffmann,K.(2017).Learninglaparoscopicvideoshotclassificationforgynecologicalsurgery.MultimediaToolsandApplications,1-19.
WHY?• structurevideocontent,• automaticindexingforretrieval,• automaticsupervisionofsurgeries
Deep Learning Surgical Actions
ACMMultimedia2017Tutorial MedicalMultimediaInformationSystems(MMIS) 32
ConfidenceThresholdslow high
Petscharnig,S.,&Schöffmann,K.(2017).Learninglaparoscopicvideoshotclassificationforgynecologicalsurgery.MultimediaToolsandApplications,1-19.
Deep Learning Surgical Actions
ACMMultimedia2017Tutorial MedicalMultimediaInformationSystems(MMIS) 33
R...RecallP...Precision
Smoke Detection
ACMMultimedia2017Tutorial MedicalMultimediaInformationSystems(MMIS) 34
Cauterizationin90%surgeries
Instruments:LaserorHF
(100° - 1200° C)
Currentfiltrationsystemmanual!
à AutomaticSmokeDetection&Removal?(Real-Time)
Automatic Smoke Detection
ACMMultimedia2017Tutorial MedicalMultimediaInformationSystems(MMIS) 35
AchievablePerformancewithSaturationPeakAnalysis(SPA)
AndreasLeibetseder,ManfredJ.Primus,StefanPetscharnig,andKlausSchoeffmann.“Image-basedSmokeDetectioninLaparoscopicVideos“.Proceedingsof ComputerAssistedandRoboticEndoscopyandClinicalImage-BasedProcedures:4thInternationalWorkshop,CARE2017,and6thInternationalWorkshop,CLIP2017,heldinConjunctionwithMICCAI 2017,QuebecCity,QC,Canada,September14,2017,pp.70-87
Automatic Smoke Detection - Performance
ACMMultimedia2017Tutorial MedicalMultimediaInformationSystems(MMIS) 36
20Kimages(DSA) 10Kimages(DSA)4.5Kimages(DSB)
SPA: SaturationPeakAnalysisGLNRGB:GoogLeNet usingRGBimagesGLNSAT:GoogLeNet usingsaturationonlyimages
Deep Learning
AndreasLeibetseder,ManfredJ.Primus,StefanPetscharnig,andKlausSchoeffmann.“Image-basedSmokeDetectioninLaparoscopicVideos“.Proceedingsof ComputerAssistedandRoboticEndoscopyandClinicalImage-BasedProcedures:4thInternationalWorkshop,CARE2017,and6thInternationalWorkshop,CLIP2017,heldinConjunctionwithMICCAI 2017,QuebecCity,QC,Canada,September14,2017,pp.70-87
Real-Time Smoke Detection Prototype
ACMMultimedia2017Tutorial MedicalMultimediaInformationSystems(MMIS) 37
AndreasLeibetseder,ManfredJ.Primus,StefanPetscharnig,andKlausSchoeffmann.“Image-basedSmokeDetectioninLaparoscopicVideos“.Proceedingsof ComputerAssistedandRoboticEndoscopyandClinicalImage-BasedProcedures:4thInternationalWorkshop,CARE2017,and6thInternationalWorkshop,CLIP2017,heldinConjunctionwithMICCAI 2017,QuebecCity,QC,Canada,September14,2017,pp.70-87
VideoInteractionTools
ACMMultimedia2017Tutorial MedicalMultimediaInformationSystems(MMIS) 38
Desired Status
ACMMultimedia2017Tutorial MedicalMultimediaInformationSystems(MMIS) 39
BerndMünzer,KlausSchoeffmann andLaszloBoeszoermenyi.“EndoXplore:AWeb-basedVideoExplorerforEndoscopicVideos“. ProceedingsoftheIEEEInternationalSymposiumonMultimedia2017(ISM2017),Taipei,Taiwan,2017,pp.1-2
Special Content Visualization
ACMMultimedia2017Tutorial MedicalMultimediaInformationSystems(MMIS) 40
Special Interaction Tools
ACMMultimedia2017Tutorial MedicalMultimediaInformationSystems(MMIS) 41
MarcoA.Hudelist,SabrinaKletz,andKlausSchoeffmann.2016.AMulti-VideoBrowserforEndoscopicVideosonTablets.In Proceedingsofthe2016ACMonMultimediaConference (MM'16).ACM,NewYork,NY,USA,722-724.
MarcoA.Hudelist,SabrinaKletz,andKlausSchoeffmann.2016.ATabletAnnotationToolforEndoscopicVideos.In Proceedingsofthe2016ACMonMultimediaConference (MM'16).ACM,NewYork,NY,USA,725-727.
Surgical Quality Assessment (SQA) Software
ACMMultimedia2017Tutorial MedicalMultimediaInformationSystems(MMIS) 42
• Integratingratingfeatures• Moreefficientvideonavigation/browsing
MarcoA.Hudelist,HeinrichHusslein,BerndMuenzer,SabrinaKletz andKlausSchoeffmann.“ATooltoSupportSurgicalQualityAssessment“,inProceedingsofthe ThirdIEEEInternationalConferenceonMultimediaBigData (BigMM),LagunaHills,CA,USA,2017,pp.238-239.
DiagnosticDecisionSupport
43ACMMultimedia2017Tutorial MedicalMultimediaInformationSystems(MMIS)
ChallengesandRequirements
44ACMMultimedia2017Tutorial MedicalMultimediaInformationSystems(MMIS)
45ACMMultimedia2017Tutorial MedicalMultimediaInformationSystems(MMIS)
Medicalknowledgetransfer AutomaticDataanalysis/detection Feedback/visualization
• Medicalknowledgetransfers– needDATAw/GroundTruth
• Highdetectionaccuracy
• Fastandefficient:real-timefeedbackandlargescale
• Fitthenormalexaminationprocedures
• Adheretoethical,legal,privacychallenges®ulations
46ACMMultimedia2017Tutorial MedicalMultimediaInformationSystems(MMIS)
Key Challenges & Requirements
Gastrointestinal(GI)CaseStudy(challenges,systemsupport,datasets,diagnosticdecisionsupport,...)
47ACMMultimedia2017Tutorial MedicalMultimediaInformationSystems(MMIS)
• Manytypesofdiseasescanpotentiallyaffectthehumangastrointestinal(GI)tract– thedigestivesystem
• about2.8millionsofnewluminalGIcancers(esophagus,stomach,colorectal)aredetectedyearly• themortalityisabout65%
• ScreeningoftheGItractusingdifferenttypesofendoscopy…• iscostly(colonoscopyaccordingtoNYTimes:$1100/patient,$10billiondollars)• consumesvaluablemedicalpersonneltime(1-2hours)• doesnotscaletolargepopulations• isintrusivetothepatient• …
• Currenttechnologymaypotentiallyenableautomaticalgorithmicscreeningandassistedexaminationsà atrueinterdisciplinaryactivitywithhighchancesofsocietalimpact
48ACMMultimedia2017Tutorial MedicalMultimediaInformationSystems(MMIS)
GI Tract Challenges and Potential
49ACMMultimedia2017Tutorial MedicalMultimediaInformationSystems(MMIS)
WHO: Colorectal Cancer Mortality 2012
Women
Men
Colorectalcanceristhethirdmostcommoncauseofcancermortalityforbothwomenandmen,anditisaconditionwhereearlydetectionisimportantforsurvival,i.e.,a5-yearsurvivalprobabilityofgoingfromalow10-30%ifdetectedinlaterstagestoahigh90%survivalprobabilityinearlystages.
Colonoscopyitisnottheidealscreeningtest.Relatedtothecancerexample,onaverage20%ofpolyps(possiblepredecessorsofcancer)aremissedorincompletelyremoved.Theriskof gettingcancerlargelydependontheendoscopists abilitytodetectandremovepolyps.A1%increaseindetectioncandecreasetheriskofcancerwith3%.
ACMMultimedia2017Tutorial MedicalMultimediaInformationSystems(MMIS)
Live Automatic Detection
• Systemtoassistdoctorsduringliveendoscopyprocedures
• detectionaccuracydependonexperienceandskills
• havea“secondeye”,“better”detection
• automatictagging,annotationoflesions
• Betterprocedurefordocumentation,automaticreportgeneration
50
51MedicalMultimediaInformationSystems(MMIS)
Video Capsule (PillCam)
§ Standardcolonoscopy:§ expensive§ doesnotscale§ intrusive
§ WirelessVideoCapsuleendoscopy:§ betterscale§ lessintrusive§ possibletocombine
examinations
§ watchhoursofvideo§ lessexpensive?
52ACMMultimedia2017Tutorial MedicalMultimediaInformationSystems(MMIS)
System Overview
MedicalKnowledgeTransfer(DataCollection)
53ACMMultimedia2017Tutorial MedicalMultimediaInformationSystems(MMIS)
• Needmoredataandthereforetoolstoefficientlyannotateandtagdata
54ACMMultimedia2017Tutorial MedicalMultimediaInformationSystems(MMIS)
Available GI Datasets
Name Contain Annotation Size Type Usage
CVC-ClinicDB Polyps GTmasks 612images Trad. ©,bypermission
ETIS-Larib PolypDB Polyps,Normal GTmasks 1500images Trad. ©,bypermission
ASU-MayoClinicDB Polyps,Normal GTmasks 18videos Trad. ©,bypermission
ColonoscopyVideosDB VariousLesions Sorted 76videos Trad. Academic
CapsuleEndoscopyDB VariousLesionsandFindings Sorted 3170images, 47videos VCE Academic, byrequest
GastroAtlas VariousLesionsandFindings Sorted,Textannotations 4449 videos Trad. Academic
WEOAtlas VariousLesionsandFindings Sorted,Textannotations ? Trad. Academic
GASTROLAB VariousLesionsandFindings Sorted,Textannotations ? Trad. Academic
AtlasofGE VariousLesions Sorted,Textannotations 669 images Trad. ©,bypermission
• Whichimageisnotfromthesameclass?
…anditgetsworse…
• Makingamistakebetweencatsanddogsmaynotmatter,butamisclassificationheremayhavelethalconsequences
Why Can’t CS People Do the Annotation!?
ACMMultimedia2017Tutorial MedicalMultimediaInformationSystems(MMIS) 55
PylorusZ-line Z-line Z-line Z-line Z-line
• Simpleandefficient
• Web-based
• Assistedobjecttracking
56ACMMultimedia2017Tutorial MedicalMultimediaInformationSystems(MMIS)
Video Annotation Subsystem"ExpertDrivenSemi-SupervisedElucidationToolforMedicalEndoscopicVideos"
ZenoAlbisser,et.al.ProceedingsoftMMSys,Portland,OR,USA,March2015
• Forlargecollectionofimages• VV/Kvasir dataset• Fullycleaned
• Featureextractionmechanisms
• Differentunsupervisedclusteringalgorithms
• Hierarchicalimagecollectionvisualization
• Opensource:ClusterTaghttps://bitbucket.org/mpg_projects/clustertag
57ACMMultimedia2017Tutorial MedicalMultimediaInformationSystems(MMIS)
ClusterTag: Image Clustering and Tagging Tool"ClusterTag:InteractiveVisualization,ClusteringandTaggingToolforBigImageCollections"
KonstantinPogorelov,et.al.ProceedingsofICMR,Bucharest,Romania,June2017
• Multi-ClassImageDatasetforComputerAidedGIDiseaseDetection• GIendoscopyimages• Someimagescontainthepositionandconfigurationoftheendoscope(scopeguide)• 8differentanomaliesandanatomicallandmarks
• v1:500imagesperclass,6pre-extractedglobalfeatures• v2:1000imagesperclass
• Newinformationaddedinthefuture:http://datasets.simula.no/kvasir/
ACMMultimedia2017Tutorial MedicalMultimediaInformationSystems(MMIS)
The Kvasir Dataset"Kvasir:AMulti-ClassImage-DatasetforComputerAidedGastrointestinalDiseaseDetection"
KonstantinPogorelov,etal.ProceedingsofMMSYS,Taiwan,June2017
• BowelPreparationQualityVideo
• 21GIendoscopyvideos ofcolon
• Someframescontainthepositionandconfigurationoftheendoscope(scopeguide)
• 4classesshowingfour-scoreBBPS-definedbowel-preparationquality
• 0- verydirty• …• 3- veryclean
• http://datasets.simula.no/nerthus/
ACMMultimedia2017Tutorial MedicalMultimediaInformationSystems(MMIS)
The Nerthus Dataset"Nerthus:ABowelPreparationQualityVideoDataset"
KonstantinPogorelov,etal.ProceedingsofMMSYS,Taiwan,June2017
GIAnomalyDetectionSystem
60ACMMultimedia2017Tutorial MedicalMultimediaInformationSystems(MMIS)
• Easytoextendwithnewdiseases• Easytoextendwithnewalgorithms• Easytotrain
• Resultsareexplainable?
• DiseaseLocalization?
• Real-time?
61ACMMultimedia2017Tutorial MedicalMultimediaInformationSystems(MMIS)
Detection and Automatic Analysis subsystem
62ACMMultimedia2017Tutorial MedicalMultimediaInformationSystems(MMIS)
State-of-The-Art: Some Example Detection Systems
Polyp-Alert• detectspolypsusingedgesandtexture• nearreal-timefeedbackduringcolonoscopy(10fps)• detected97.7%(42of43)ofpolypshotson53randomlyselected
(notperframedetection)• only4.3%ofafull-lengthcolonoscopyprocedurewronglymarked• oneofthefewend-to-endsystems• Wallapak Tavanapong – fromMMcommunity
• Featuresextractionusingopen-sourceLIRE(Lucene ImageRetrieval)• Indexer:
• IndexingimagesbyLIREfeaturesfor“training”
• Classifier:• Built-inbenchmarkingfunctionality• Outputtoconsole&JSON/HTML
• Verifiedwithdifferentdatasetsandusecases,e.g.,life-logging,recommendersystems,networkanalysis,etc.
• Opensourceproject– OpenSea
63ACMMultimedia2017Tutorial MedicalMultimediaInformationSystems(MMIS)
Global Features (GF)-Based Detection”EIR- EfficientComputerAidedDiagnosisFrameworkforGastrointestinalEndoscopies"
MichaelRiegler,et.al.ProceedingsofCBMI,Bucharest,Romania,June2016
• Searchforanoptimalcombinationofglobalimagefeaturedescriptors
• Combiningresultsbylatefusion• LIREimagefeaturedescriptorsJCDandTamuraarethebestchoice
64ACMMultimedia2017Tutorial MedicalMultimediaInformationSystems(MMIS)
Global Features (GF)-Based Detection
Originalpolyp Colorfeature Edgeandcolor Texture Edge
65ACMMultimedia2017Tutorial MedicalMultimediaInformationSystems(MMIS)
Global Features (GF)-Based Detection
Featureextractors
Features
Features
Polyps
Cancer
Featureextractors
FeaturesNormal
Distancetothetrainingimages
Classselectionforeachfeature
Distance
Distance
Polyps
Cancer
DistanceNormal
Indexofthetrainingset
LatefusionImageclass
• WithmanyenoughCPUs,thedetectionrunsinreal-time
• GPU-acceleration
66ACMMultimedia2017Tutorial MedicalMultimediaInformationSystems(MMIS)
Global Features (GF)-Based Detection
Java CUDAC++
""GPU-acceleratedReal-timeGastrointestinalDiseasesDetection"KonstantinPogorelov,et.al.
ProceedingsofCBMS,Dublin,Ireland/Belfast,NorthernIreland,June2016
• Tensorflow asbackend• BasedonInceptionv3• Lastlayersremoved• Modelretrainedonmedicaldata• Applyingsimpletransformationstoincreasesizeoftrainingset
• Verylongtrainingtime• Applyingmodelisfast
67ACMMultimedia2017Tutorial MedicalMultimediaInformationSystems(MMIS)
Basic CNN-Based Detection“Efficientdiseasedetectioningastrointestinalvideos- globalfeaturesversusneuralnetworks"
KonstantinPogorelov,et.al.MultimediaToolsandApplications,2017
Performance(accuracyandspeed)
68ACMMultimedia2017Tutorial MedicalMultimediaInformationSystems(MMIS)
§Mayodataset(18781images/frames)§ masksforallpolyps
• GF:• recall98.50%,precision93.88%,fps~300
• CNN:• ModifiedInceptionv3:recall95.86%,precision80.78%,fps:~30• Inceptionv3+WEKA:recall:88.87%,precision:89.16%,fps:~30
69ACMMultimedia2017Tutorial MedicalMultimediaInformationSystems(MMIS)
ASU Mayo Dataset: Polyp Detection ”EIR- EfficientComputerAidedDiagnosisFrameworkforGastrointestinalEndoscopies"
MichaelRiegler,et.al.ProceedingsofCBMI,Bucharest,Romania,June2016
• ResourceconsumptionandprocessingperformanceofGF:
• Neuralnetworks(alsoincludingGPUsupport)?• testssofar:~30fps(sameGPUasabove)
• butaddinglayers,morenetworks,…!??(newerGPU)• Inceptionv3TFL:66fps,plainCNN:~40-45fps• GAN:~12fps(for160x160)
70ACMMultimedia2017Tutorial MedicalMultimediaInformationSystems(MMIS)
ASU Mayo Dataset: Polyp Detection
• Processonlyframescontainingpolyps
• Performsimageenhancement
• Detectscurve-shapedobjectsandlocalmaximums
• Buildsenergymapandselects4possiblelocations
• Localizationperformance:• recall31.83%,• precision32.07%• ~30fps
• laterbetterGPU:~75fps(detection:300fps;localization100fps)
71ACMMultimedia2017Tutorial MedicalMultimediaInformationSystems(MMIS)
ASU Mayo Dataset: First Try for Polyp Localization
• Vestre Viken (VV)multi-diseasedataset(250imagesperclass)
• GF:• recall90.60%• precision91.40%• fps~30
• CNN:• recall:87.20%• precision:87.90%• fps:~30
72ACMMultimedia2017Tutorial MedicalMultimediaInformationSystems(MMIS)
VV Dataset: Multi-Disease Detection""Efficientdiseasedetectioningastrointestinalvideos- globalfeaturesversusneuralnetworks"
KonstantinPogorelov,et.al.MultimediaToolsandApplications,2017
• GF
• CNN
73ACMMultimedia2017Tutorial MedicalMultimediaInformationSystems(MMIS)
VV Dataset: Multi-Disease Detection""Efficientdiseasedetectioningastrointestinalvideos- globalfeaturesversusneuralnetworks"
KonstantinPogorelov,et.al.MultimediaToolsandApplications,2017
• 7differentalgorithms• Convolutionalneuralnetworks(CNN)(2)– trainedfromscratch
• 3-layers• 6-layers
• Transferlearning(1)– retrainedInceptionv3• Globalfeatures(4)
• 2globalfeatures(JCD,Tamura)• 6globalfeatures(JCD,Tamura,ColorLayout,EdgeHistogram,AutoColorCorrelogram andPHOG)• 2differentalgorithms(Randomforestandlogisticmodeltree)
• 2baselines• RandomForrestwithoneglobalfeature• Majorityclass
• 2-foldedcrossvalidation
ACMMultimedia2017Tutorial MedicalMultimediaInformationSystems(MMIS)
Kvasir Dataset v1: Multi-Disease Detection
75ACMMultimedia2017Tutorial MedicalMultimediaInformationSystems(MMIS)
Kvasir Dataset v1: Multi-Disease Detection
76ACMMultimedia2017Tutorial MedicalMultimediaInformationSystems(MMIS)
Kvasir Dataset v1: Multi-Disease Detection
Dyed
and
Lifted
Polyp
Dyed
Resectio
nMargin
77ACMMultimedia2017Tutorial MedicalMultimediaInformationSystems(MMIS)
Kvasir Dataset v1: Multi-Disease Detection
Cecum
Pylorus
• UsingsameGFandsomenewdeepfeatures,i.e.,• Pre-trainedImageNet datasetInceptionv3• ResNet50models
• UseddifferentMLclassifications;• randomtree(RT)• randomforest(RF)• logisticmodeltree(LMR)– performedbest
• Usesweightsof1000pre-definedconceptsasfeatures
• Toplayerinputasfeaturesvector(16384forInceptionv3and2048forResNet50)
ACMMultimedia2017Tutorial MedicalMultimediaInformationSystems(MMIS)
Kvasir Dataset v1 à v2: Multi-Disease Detection
Pretrainedmodel
Outputortop-layerinputweights
WEKAforclassification
78
Team Approaches F1 FPS
SCL-UMD Global-features anddeep-features extraction,Inception-V3and VGGNet CNNmodels,followedby
machine-learning-basedclassificationusingRT,RF,SVMandLMR classifiers
0.848 1.3
FAST-NU-DS Global andlocalfeaturescombinedfollowedbydatasizereductionbyapplyingK-means clusteringandthanusing logisticregression model fortheclassification
0.767 2.3
ITEC-AAU TwodifferentcustomInception-likeCNNmodels 0.755 1.4
HKBU Amanifoldlearningmethod(bidirectionalmarginalFisheranalysis)learningacompactrepresentationofthedata,thenmachine-learning-basedmulti-classsupport
vectormachineisusedfortheclassification
0.703 2.2
SIMULA GF-featuresextraction,ResNet50 and Inception-V3CNNmodels andfollowedbymachine-learning-basedclassificationusingRT,RFandLMR classifiers
0.826 46.0
• 7differentalgorithms• Convolutionalneuralnetworks(CNN)(2)– trainedfromscratch
• 3-layers• 6-layers
• Transferlearning(1)– retrainedInceptionv3• Globalfeatures(4)
• 2globalfeatures(JCD,Tamura)• 6globalfeatures
(JCD,Tamura,ColorLayout,EdgeHistogram,AutoColorCorrelogram andPHOG)• 2differentalgorithms(Randomforestandlogisticmodeltree)
• 2baselines• RandomForrestwithoneglobalfeature• Majorityclass
• 2-foldedcrossvalidation
79ACMMultimedia2017Tutorial MedicalMultimediaInformationSystems(MMIS)
Nerthus Dataset: Bowel Cleanness Level
80ACMMultimedia2017Tutorial MedicalMultimediaInformationSystems(MMIS)
Nerthus Dataset: Bowel Cleanness Level
81ACMMultimedia2017Tutorial MedicalMultimediaInformationSystems(MMIS)
Nerthus Dataset: Bowel Cleanness Level
• Toolittledata• Blurryimagesduetocameramotion• Objectstooclosetocamera• Underoroverscenelighting• Flares• Artificialobjectsandnatural“contaminations”• Lowresolutionofcapsularendoscopes• …
82ACMMultimedia2017Tutorial MedicalMultimediaInformationSystems(MMIS)
Data Challenges: Preprocessing
83ACMMultimedia2017Tutorial MedicalMultimediaInformationSystems(MMIS)
Data Enhancements for CNN Training
84ACMMultimedia2017Tutorial MedicalMultimediaInformationSystems(MMIS)
Data Enhancements for CNN Training
DetectionFeedback
85ACMMultimedia2017Tutorial MedicalMultimediaInformationSystems(MMIS)
86ACMMultimedia2017Tutorial MedicalMultimediaInformationSystems(MMIS)
Detection Subsystem Outputs
• Visualizetheoutputofthesystemtothemedicaldoctors• Simpleandeasytounderstand• Livesupport• Useableforautomaticreports,etc.
• Polyps• Input:CameraorVideofiles
• Output:LivestreamandPerformancereports
• FullHD• Real-time:30FPS
87ACMMultimedia2017Tutorial MedicalMultimediaInformationSystems(MMIS)
Real-time Detection Feedback
So,allproblemssolved!!??
88ACMMultimedia2017Tutorial MedicalMultimediaInformationSystems(MMIS)
• Improvedetection,localizationandsystemperformance(retrieval,machinelearning,features,search,real-time,distributedcomputing,scale,visualization,neuralnetworks,userinteraction,objecttracking,…)
1. Exploitingdomainexpertknowledge– builddatasets2. Integrationofvariousdata,multi-modality– newsensors3. ExplainableAI4. Automatedreportsystem5. Fullsystemintegration6. Patientcontextinformation7. Visualization,decisionsupport8. Integrationofdatafromvarioussources/systems9. Otherareasinmedicine10. …
Manymore…
89ACMMultimedia2017Tutorial MedicalMultimediaInformationSystems(MMIS)
Many Open Challenges…"MultimediaandMedicine:TeammatesforBetterDiseaseDetectionandSurvival"
MichaelRiegler,et.al.ProceedingsACMMM,Amsterdam,TheNetherlands,October2016
• Wehavegivenseveralcase-specificexamples,butingeneral,theyarecommonforMMIS
• Doctorswanttouseallthedataforgeneralsupport:analysis,diagnostics,reporting,teaching,statistics,similaritysearch/comparisons,…
• Currently,…• moreandmorehighqualitydataisrecorded/produced
• dataanalysismethodsare(only)promising• goodvisualizationtoolsexist,butnotused(e.g.,AR,VR,…)• sometoolsaremissing• many(other)areasproduceseparate(isolated)methods• …
• but,weneedacompleteintegratedsystem!
Ø Ourmultimediacommunityisneeded
Summary
ACMMultimedia2017Tutorial MedicalMultimediaInformationSystems(MMIS) 90
91ACMMultimedia2017Tutorial MedicalMultimediaInformationSystems(MMIS)
The End…