Harnessing the GPU for Surgical Training and Preoperative PlanningJosh Passenger, David Conlan, Hans de Visser, Mario Cheng, David Hellier, Juan Borda, Tobias Reichl, Christoph Russ, John Taylor and Olivier Salvado

Introduction

• Surgical simulation and preoperative planning systems

• Need high performance computing

• Solve complex spatial and physically based problems• Often consume huge datasets

• CUDA provides the raw power we need

• Success stories so far

• Colonoscopy Simulator

• Non-linear soft tissue model

• Free-hand ultrasound from Computed Tomography (CT)• Large data-set visualisation

Why is CSIRO developing a colonoscopy simulator?

• Colorectal cancer is a priority research focus for the CSIRO National Preventative Health Flagship within a wide range of projects

• Australia has the second highest incidence of colorectal cancer in the world, with less than 40% early detection

• Screening programs are being phased in slowly as positives usually require a colonoscopy with limited resources available

• Colonoscopy is a difficult procedure to master with hundreds of cases required to reach an expert skill level

Why is CSIRO developing a colonoscopy simulator?

• Training currently happens on real patients with increased risk and cost over patient-free simulation based training

• Existing simulators rate poorly for realism and complexity and are under-utilised

• Development of a high fidelity, portable simulator has the potential to significantly reduce cost and risk of training and provide a platform for future certification

The challenges

• Surgical simulation for colonoscopy poses many computational andtechnical challenges including:

• High density colon organ meshes required for realism• Physically based modelling of the colon (highly mobile)• Physically based modelling of the colonoscope (long and flexible)• Tool and organ tissue interactions (at haptic frequencies)• Realistic organ and tool surface shading (wet surfaces)• Collision detection between many dynamic systems• Simulation and interaction of viscous fluids within the colon• Realistic force rendering to the user

• With limited CPU resources we must rely heavily on the computational power of the GPU

• Not all processes are suitable for GPU acceleration

The colon and colorectal cancer

Anatomy of an Olympus Colonoscope

Insertion tip

Control handle

Insertion shaft

Umbilical socket

Umbilical

Generating realistic cases from CT data

Patient CT scans are processed The colon surface is extracted

Mesh Generation using Marching Cubes

• Mesh generated using Marching Cubes• Smoothed and decimated• No texture coordinates!

Mesh Texture Coordinates

• Texture coordinates are required for surface shading• Allow mapping of 2D textures onto 3D meshes• We use tricky maths to pre-calculate texture coordinates

U

V

Texture mapping

High fidelity visualisation GLSL shading

Virtual colonReal colon

Integrated haptic device - EPFL

Fluid and tissue simulation

Soft tissue interaction

Modelling of tools and surface bleeding

Colon physical model editor

A fully featured colon physical model editor is under development to allow ranged definition of model parameters for inclusion in the clinical database.

Mass Spring Model of the Colon

High to low mapping

• Simulates insufflation and preserves soft tissue appearance using high low mapping

• Solved completely on the GPU using CUDA

We need to simulate endoscopic looping

• Requires an accurate endoscope model interacting with virtual colon model at haptic rates (>300 hertz)

Dynamic endoscope intubation

Software framework for CUDA memory

• CUDA Data Pointer system implemented as C++ templates• Provides management of CUDA device pointers with

associated host memory• Provides a low code overhead for uploading and downloading

data from the device• Bind() and Unbind() methods invoked before and after use• Allows binding of data to OpenGL vertex buffers automatically• Data pointers can be wrapped with textures in CUDA for better

cache utilisation• Greatly simplifies management of complex memory structures • Previous implementations showed that a large amount of code

was devoted to managing device data buffers• Most bugs were related to buffer misuse

Mapping surface meshes to physical models

low resolution physical model

The problem: Mesh stretching and discontinuity during mapping

high resolution mesh

Mapping surface meshes to physical models

• “Batch movement” of high resolution vertices in which low resolution mesh is recognizable

• Solution:• Calculate smooth spline (surface)

through low resolution nodes (“predict” high res. surface)

• Map high resolution vertices onto smooth surface

Mapping surface meshes to physical models

• The benefit of our GPU mapping kernel is that small surface details are preserved while gross low resolution movements are smoothly interpolated onto the high resolution mesh.

• The mapping results are written directly to an OpenGL vertex buffer for rendering, no CPU transfer for high resolution meshes.

Mass Spring Solver on the GPU

• Using masses and springs to represent flexible structures is cheap to compute but has limited physical accuracy.

• CUDA kernel structure:1. Calculate the spring forces based on

spring length2. Calculate the mass accelerations

based on forces exerted by attached springs

• Stores mass positions and spring configurations as textures on the GPU.

• The CUDA solver easily computes a 2K node structure in 0.2 ms which takes 10.6 ms to solve on the CPU.

Simple mass-spring network

Non-Linear Soft Tissue Physical Model

• The mechanical response of organs is known to be non-linear

• Less expensive and simpler models have been used previously including Linear Finite Element

• Linear FEM solvers not only have limited accuracy but are only valid for small deformations (<10%)

• A better solution would capture all tissue properties and still solve at haptic rates (~1000 Hz)

• Previously these solutions were only available offline• Solution: use the GPU!

TLED Non-Linear Soft Tissue Solver

• Previously an efficient CPU formulation for a non-linear FEM was proposed by Karol Miller

• This was accurate but did not run in real-time for large models

• Reformulated the solver into a GPGPU solver using CUDA with support for tetrahedral and hexahedral elements

• CUDA Kernel organisation• Kernel 1 (Elements) computes element stresses based on

current configuration and converts them into nodal force contributions

• Kernel 2 (Nodes) sums the previously calculated element force contributions at each node

TLED Non-Linear Soft Tissue Solver

• Accurately representing soft tissue properties allows us to tackle all non-linear tissue properties

• This includes viscoelasticity where tissue strain is time dependent

Isotropic viscoelasticity Anisotropic viscoelasticity

TLED Non-Linear Soft Tissue Solver

• Ratio of CPU to GPU solution time

• For larger models the GPU solver is up to 56.3 times faster than the CPU solver!

• Our solver can compute a hexahedral model with 30K nodes in 1 millisecond!

(DOF = 3 * node count)

Prostate Cancer

• In developed countries 15% of all male cancers are prostate cancer

• Second most common cancer in Australia in males after skin cancer

• Second most common cause of cancer death• 14% prevalence for men over 70 years

Transrectal needle biopsy

Sampling areas of the gland, where tumors are more prevalent

Use of transrectal ultrasound• Guide the position of the needle• Volume Measurement

Simulation of transrectal needle biopsy

• Difficult procedure that requires training• A simulator will improve skills for a surgeon• Minimises potential risk for the patient

• Biopsy protocol yet to be defined• A simulator will potentially help to develop ways to

improve the procedure• Improve diagnosis efficacy

Simulation of transrectal needle biopsy

TRUS simulator from 3D Computed Tomography (CT) Images

TRUS Simulation

Ultrasound and Interaction with tissues• Acoustic properties

Probe (linear arrray)

Boundary tissueReflection

Refraction

Attenuation

Transmitted

x

Intensity

Tissue 2

Tissue1AbsorptionDiffusion

Transducer

A[dB]f[Hz]

2-18Mhz

TRUS Simulation

Ultrasound and Interaction with tissues• Acoustic properties

Real TRUS ImageA[dB]f[Hz]

Tissue 2

Tissue1

Transducer2-18Mhz

Non-homogeneous tissues

Specular reflection

Scattering

Diffusion

Linear scanning

Tiltscanning

Tilt Scanning with side-firing TRUS transducer

Rotational scanning

TRUS Simulation

• Geometry of acquisition

Simulation of acquisition

CT scan

Probe Position(x,y,z,φ)

Geometry of acquisition

(0,0,0.9)

(0,1,0.4)(0,1,0)

v

(0,0,0)

(1,1,0)

(1,0,1)(0,0,1)w(1,0,0.5)

(1,1,0.2)

(1,1,1)

(1,0,0)u

CT scan

Probe Position(x,y,z,φ)

Transmission

• Ray casting to compute the transmitted intensity

Transmission

Transmission Absorption ReflectionNoise

Method

Probe Position(x,y,z,φ)

Absorption

2D slice CT

AcousticImpedance

3D scanner CT

Reflection

Gradient 3D Noise

2D Noise

Blending

Acquisition Settings(gain, frequency,

geometry)

Transmission Image synthesis

Simulated US image

Preoperative planning

• Using the GPU we can visualise large 3D medical image data sets

• Loads medical images onto GPU as 3D OpenGL textures• Slices the textures using quads aligned with the screen but

modulated 3D texture coordinates• Real-time ray casting is used for iso-surfacing, maximum

intensity projection and transfer function 3D rendering• Platform developed which allows point picking from any view• This data is then exported for use in building patient specific

finite element models and planning surgical procedures• Data is pre-processed to calculate an image gradient and this,

along with the original image intensity is uploaded as a 3D RGBA OpenGL image

Interactive iso-surface generation using GLSL raycasting

Transfer function rendering using GLSL

Conclusion

• NVIDIA CUDA is allowing us to solve problems in real time• We are now using denser meshes, higher fidelity solvers, larger

data sets• We are tackling problems that were not possible previously due

to computational constraints• The possibilities include:

• High fidelity, patient free training systems for surgeons• Better preoperative data for surgeons

• Expected outcomes:• Reduced cost and improved safety with training in repeatable

virtual environments

• Improved surgical outcomes through advanced preoperative planning

• Increased patient safety!

Thank you

Project Team

Josh Passenger, Project Leader

Olivier Salvado, Team Leader

Contact UsPhone: 1300 363 400 or +61 3 9545 2176

Email: enquiries@csiro.au Web: www.csiro.au

Hans de Visser, Scientist

David Conlan, Software Engineer

Olivier Comas, PhD student

Mario Cheng, PhD student

David Hellier, PhD student

Tobias Reichl, Trainee

Juan Borda, Trainee

Christoph Russ, Trainee