Advanced Interaction Techniques for Medical Intervention and Diagnosis

Chris DesomerLars Jacobs

Prof. Dr. Ir. P. Suetens Prof. Dr. Ir. T. Tuytelaars

Dr. Ir. S. De BuckDr. Ir. P. SlagmolenDr. Ir. J. GemmekeDr. Ir. M. Pedersoli

Supervisors:

Advisors:

Assessors:

07/04/2023 Slide 2 of 32

Outline• Context• Autostereoscopic visualisation• Gesture interface• Evaluation• Discussion & conclusion• Demo

Context

07/04/2023 Slide 4 of 32

Introduction (I)

07/04/2023 Slide 5 of 32

Introduction (II)

07/04/2023 Slide 6 of 32

Problem statement

07/04/2023 Slide 7 of 32

Purpose • Implementation of autostereoscopic visualisation

• Implementation of gesture interface

• Added value?

Autostereoscopic visualisation

07/04/2023 Slide 9 of 32

3D perception • Humans sees 3D if each eye gets another image

07/04/2023 Slide 10 of 32

Lenslet

07/04/2023 Slide 11 of 32

Multiview

07/04/2023 Slide 12 of 32

2D+Depth

07/04/2023 Slide 13 of 32

Real time aspects • Network optimisation

• Code optimisationo Block based instead of pixel basedo memcpy(), memset()

• Hardware optimisation

07/04/2023 Slide 14 of 32

4D visualisation • 3D + time

• Demo 4D

Gesture interface

07/04/2023 Slide 16 of 32

Hardware/framework gesture interface• Advanced interaction 3D hand/finger detection required• Kinect

o Far field detectiono Frameworks for 3D

hand/finger detection (e.g. Forth)

• Leap Motiono Near field detectiono Designed for 3D hand/finger

detection

07/04/2023 Slide 17 of 32

Comparison Kinect versus Leap MotionKinect Leap Motion

Price € 250 € 901

Position in the OR Outside sterile zone Inside sterile zone2

Interaction zone 57° horizontal Forth: 2m distance from sensor gives a

range of ≈ 2m

150° vertical 0,25 m distance from sensor gives a range of

≈ 2mResolution Kinect: 30 fps

Kinect + Forth: 20 fps 115 fps

Accuracy in ideal interaction zone 2cm 0,01mm

Conclusion: Leap Motion

1 Excl. price of sterilization 2 Sterilization can be done by a protective cover for ultrasound transducers

07/04/2023 Slide 18 of 32

Requirements Requirement Gesture Translation Closed hand movement

Scaling Pinch to zoom

Rotation Open hand rotation

Reset Swipe

On/off mechanism Virtual push button

Settings Static gesture

07/04/2023 Slide 19 of 32

Leap Motion

Gesture recognition

using heuristics

Action

Flow

Leap Motion

Gesture recognition using

correlation

Action

Data frame

Parameters

07/04/2023 20

Virtual push button

Evaluation

07/04/2023 Slide 22 of 32

Data models• Meshes

• Tracts

• Volume data o Filtering

07/04/2023 Slide 23 of 32

Data models• Meshes

• Tracts

• Volume data o Filtering

07/04/2023 Slide 24 of 32

Data models• Meshes

• Tracts

• Volume data o Filtering

Autostereoscopic visualisation

07/04/2023 Slide 26 of 32

Gesture interfaceGesture Intuitiveness DifficultyTranslation 3.4 3.0

Scaling 4.2 3.0

Rotation 3.5 2.9

Swipe 4.7 4.2

Airbutton 4.3 3.6

Static gestures 3.6 3.5

1: very unintuitive/very difficult5: very intuitive/very easy

07/04/2023 Slide 27 of 32

System Usability Scale (SUS)

50% gives at least a SUS score of 74 (≈70%)25% gives at least a SUS score of 79 (≈88%)

An average system has a SUS score of 68

Discussion & conclusion

07/04/2023 Slide 29 of 32

Discussion (I)• Autostereoscopic visualisation

+ Added value- Low resolution

• Gesture interface+ Dynamic 3D gesture set+ Intuitive- Training needed

07/04/2023 Slide 30 of 32

Discussion (II)• Forensic science

o Virtual crime sceneo No sterility

• Cardiovascular surgery o Combination = added valueo Catheter visualisation

07/04/2023 Slide 31 of 32

Conclusion• First general prototype

o Autostereoscopic visualisationo Gesture interface

• Nexto Application specific prototypeo Optimisations per medical domain

Demo

07/04/2023 Slide 34 of 32

References (I)• Article, R., & Ar, W. I. S. (2013). Augmented Reality in

Surgery, 139.• Dodgson, N. A. (2005). Autostereoscopic 3D displays.

Computer, 38(8), 31–36. doi:10.1109/MC.2005.252• Delude, C., Is Heart Surgery History?, 2007, (‘http://

protomag.com/assets/is-heart-surgery-history?page=4’)• Ector, B. J., Buck, S. De, Sc, M., Ph, D., Heidbuchel, H., &

Ph, D. (2008). Improved Efficiency in the EP Lab with syngo DynaCT Cardiac, (October), 26–31.

07/04/2023 Slide 35 of 32

References (II)• Oikonomidis, I., Kyriazis, N., & Argyros, A. (2011). Efficient

model-based 3D tracking of hand articulations using Kinect. Procedings of the British Machine Vision Conference 2011, 101.1–101.11. doi:10.5244/C.25.101

• Ruijters, D. (2008). Integrating autostereoscopic multi-view lenticular displays in minimally invasive angiography. Workshop proceedings, 87–94. Retrieved from https://lirias.kuleuven.be/handle/123456789/198103

• Sansoni, G., & Docchio, F. (2013). Biomedical 2D and 3D Imaging : State of Art and Future Perspectives, 3–19.

• Solutions, P. (n.d.). 3D Interface Specifications White Paper.

07/04/2023 Slide 36 of 32

2D+Depth

07/04/2023 Slide 37 of 32

Hardware optimisations Windows 8 laptop

Windows 7 desktop

iMac

CPU Intel Core i7 Q720 @ 1.60GHz

Intel Core i5 2500 @ 3.30GHz

Intel Core i5 4670 @ 3.40GHz

RAM memory 8Gb SO-DIMM DDR3 @ 533MHz

16Gb DDR3 @ 666MHz

32Gb DDR3 @ 1600MHz

GPU AMD Radeon HD6570M 1Gb RAM

AMD Radeon HD6450 1Gb RAM

NVIDIA GeForce GTX 780M 4GB RAM

Refresh rate ± 2.5 fps ± 9 fps ± 20 fps