university of california, san diego san diego supercomputer center computational radiology...
Post on 22-Dec-2015
217 views
TRANSCRIPT
![Page 1: University of California, San Diego San Diego Supercomputer Center Computational Radiology Laboratory Brigham & Women’s Hospital, Harvard Medical School](https://reader030.vdocuments.mx/reader030/viewer/2022032523/56649d7e5503460f94a619ce/html5/thumbnails/1.jpg)
University of California, San Diego
San Diego Supercomputer CenterComputational Radiology LaboratoryBrigham & Women’s Hospital, Harvard Medical School
ICCS2005ICCS2005
A Dynamic Data Driven Grid System for Intra-operative Image Guided
Neurosurgery
A Majumdar1, A Birnbaum1, D Choi1, A Trivedi2, S. K. Warfield3, K. Baldridge1, and Petr Krysl2
1 San Diego Supercomputer Center University of California San Diego
2 Structural Engineering Dept University of California San Diego
3 Computational Radiology Lab Brigham and Women’s Hospital
Harvard Medical School
Grants: NSF: ITR 0427183,0426558; NIH:P41 RR13218, P01 CA67165, LM0078651, I3 grant (IBM)
![Page 2: University of California, San Diego San Diego Supercomputer Center Computational Radiology Laboratory Brigham & Women’s Hospital, Harvard Medical School](https://reader030.vdocuments.mx/reader030/viewer/2022032523/56649d7e5503460f94a619ce/html5/thumbnails/2.jpg)
University of California, San Diego
San Diego Supercomputer CenterComputational Radiology LaboratoryBrigham & Women’s Hospital, Harvard Medical School
ICCS2005ICCS2005
TALK SECTIONS
1. PROBLEM DESCRIPTION AND DDDAS2. GRID ARCHITECTURE3. ADVANCED BIOMECHANICAL MODEL4. PARALLEL AND END-to-END TIMING5. SUMMARY
![Page 3: University of California, San Diego San Diego Supercomputer Center Computational Radiology Laboratory Brigham & Women’s Hospital, Harvard Medical School](https://reader030.vdocuments.mx/reader030/viewer/2022032523/56649d7e5503460f94a619ce/html5/thumbnails/3.jpg)
University of California, San Diego
San Diego Supercomputer CenterComputational Radiology LaboratoryBrigham & Women’s Hospital, Harvard Medical School
ICCS2005ICCS2005
1. PROBLEM DESCRIPTION AND DDDAS
![Page 4: University of California, San Diego San Diego Supercomputer Center Computational Radiology Laboratory Brigham & Women’s Hospital, Harvard Medical School](https://reader030.vdocuments.mx/reader030/viewer/2022032523/56649d7e5503460f94a619ce/html5/thumbnails/4.jpg)
University of California, San Diego
San Diego Supercomputer CenterComputational Radiology LaboratoryBrigham & Women’s Hospital, Harvard Medical School
ICCS2005ICCS2005
Neurosurgery Challenge• Challenges :
• Remove as much tumor tissue as possible• Minimize the removal of healthy tissue• Avoid the disruption of critical anatomical structures• Know when to stop the resection process
• Compounded by the intra-operative brain shape deformation that happens as a result of the surgical process – preoperative plan diminishes
• Important to be able to quantify and correct for these deformations while surgery is in progress by dynamically updating pre-operative images in a way that allows surgeons to react to these changing conditions
• The simulation pipeline must meet the real-time constraints of neurosurgery – provide images approx. once/hour within few minutes during surgery lasting 6 to 8 hours
![Page 5: University of California, San Diego San Diego Supercomputer Center Computational Radiology Laboratory Brigham & Women’s Hospital, Harvard Medical School](https://reader030.vdocuments.mx/reader030/viewer/2022032523/56649d7e5503460f94a619ce/html5/thumbnails/5.jpg)
University of California, San Diego
San Diego Supercomputer CenterComputational Radiology LaboratoryBrigham & Women’s Hospital, Harvard Medical School
ICCS2005ICCS2005
Intraoperative MRI Scanner at BWH
![Page 6: University of California, San Diego San Diego Supercomputer Center Computational Radiology Laboratory Brigham & Women’s Hospital, Harvard Medical School](https://reader030.vdocuments.mx/reader030/viewer/2022032523/56649d7e5503460f94a619ce/html5/thumbnails/6.jpg)
University of California, San Diego
San Diego Supercomputer CenterComputational Radiology LaboratoryBrigham & Women’s Hospital, Harvard Medical School
ICCS2005ICCS2005
Brain Shape Deformation
Before surgeryBefore surgery After surgeryAfter surgery
![Page 7: University of California, San Diego San Diego Supercomputer Center Computational Radiology Laboratory Brigham & Women’s Hospital, Harvard Medical School](https://reader030.vdocuments.mx/reader030/viewer/2022032523/56649d7e5503460f94a619ce/html5/thumbnails/7.jpg)
University of California, San Diego
San Diego Supercomputer CenterComputational Radiology LaboratoryBrigham & Women’s Hospital, Harvard Medical School
ICCS2005ICCS2005
Overall Process• Before image guided neurosurgery
• During image guided neurosurgery
Segmentation and Visualization
Preoperative Planning ofSurgical Trajectory
Preoperative
Data Acquisition
Preoperative data
Intraoperative MRISegmentation Registration
Surfacematching
Solve biomechanicalModel for volumetricdeformation
Visualization Surgicalprocess
![Page 8: University of California, San Diego San Diego Supercomputer Center Computational Radiology Laboratory Brigham & Women’s Hospital, Harvard Medical School](https://reader030.vdocuments.mx/reader030/viewer/2022032523/56649d7e5503460f94a619ce/html5/thumbnails/8.jpg)
University of California, San Diego
San Diego Supercomputer CenterComputational Radiology LaboratoryBrigham & Women’s Hospital, Harvard Medical School
ICCS2005ICCS2005
Timing During Surgery
Time (min)
Before surgery During surgery
0 10 20 30 40
Preop segmentation
Intraop MRI
Segmentation
Registration
Surface displacement
Biomechanical simulation
Visualization
Surgical progress
![Page 9: University of California, San Diego San Diego Supercomputer Center Computational Radiology Laboratory Brigham & Women’s Hospital, Harvard Medical School](https://reader030.vdocuments.mx/reader030/viewer/2022032523/56649d7e5503460f94a619ce/html5/thumbnails/9.jpg)
University of California, San Diego
San Diego Supercomputer CenterComputational Radiology LaboratoryBrigham & Women’s Hospital, Harvard Medical School
ICCS2005ICCS2005
Current Prototype DDDAS Inside Hospital
Pre and Intra-op 3D MRI (once/hr)Pre and Intra-op 3D MRI (once/hr)
Local Local computercomputer
at BWHat BWH
Crude linear elastic FEM Crude linear elastic FEM solutionsolution
Merge pre and intra-op vizMerge pre and intra-op viz
Intr
a-op
sur
gica
l In
tra-
op s
urgi
cal
deci
sion
and
ste
erde
cisi
on a
nd s
teer
Segmentation, Registration, Segmentation, Registration, Surface Matching for BCSurface Matching for BC
Once every hour or twoOnce every hour or twofor a 6 or 8 hour surgeryfor a 6 or 8 hour surgery
![Page 10: University of California, San Diego San Diego Supercomputer Center Computational Radiology Laboratory Brigham & Women’s Hospital, Harvard Medical School](https://reader030.vdocuments.mx/reader030/viewer/2022032523/56649d7e5503460f94a619ce/html5/thumbnails/10.jpg)
University of California, San Diego
San Diego Supercomputer CenterComputational Radiology LaboratoryBrigham & Women’s Hospital, Harvard Medical School
ICCS2005ICCS2005
Current Prototype DDDAS System
• Receives 3-D MRI from operating room once/hour or so• Uses displacement of known surface points as BC to
solve a crude linear elastic biomechanical FEM material model on compute system located at BWH
• This crude inaccurate model is solvable within the time constraint of few minutes once an hour on local computers at BWH
• Dynamically updates pre-op images with biomechanical volumetric simulation based intra-op images
• Time critical updates shown to surgeons for intra-op surgical navigation
![Page 11: University of California, San Diego San Diego Supercomputer Center Computational Radiology Laboratory Brigham & Women’s Hospital, Harvard Medical School](https://reader030.vdocuments.mx/reader030/viewer/2022032523/56649d7e5503460f94a619ce/html5/thumbnails/11.jpg)
University of California, San Diego
San Diego Supercomputer CenterComputational Radiology LaboratoryBrigham & Women’s Hospital, Harvard Medical School
ICCS2005ICCS2005
Two Research Aspects
• Grid Architecture – grid scheduling, on demand remote access to multi-teraflop machines, data transfer• Data transfer from BWH to SDSC, solution of detail
advanced biomechanical model, transfer of results back to BWH for visualization need to be performed in a few minutes
• Development of detailed advanced non-linear scalable viscoelastic biomechanical model• To capture detail intraoperative brain deformation
![Page 12: University of California, San Diego San Diego Supercomputer Center Computational Radiology Laboratory Brigham & Women’s Hospital, Harvard Medical School](https://reader030.vdocuments.mx/reader030/viewer/2022032523/56649d7e5503460f94a619ce/html5/thumbnails/12.jpg)
University of California, San Diego
San Diego Supercomputer CenterComputational Radiology LaboratoryBrigham & Women’s Hospital, Harvard Medical School
ICCS2005ICCS2005
Example of visualization: Intra-op Brain Tumor with Pre-op fMRI
![Page 13: University of California, San Diego San Diego Supercomputer Center Computational Radiology Laboratory Brigham & Women’s Hospital, Harvard Medical School](https://reader030.vdocuments.mx/reader030/viewer/2022032523/56649d7e5503460f94a619ce/html5/thumbnails/13.jpg)
University of California, San Diego
San Diego Supercomputer CenterComputational Radiology LaboratoryBrigham & Women’s Hospital, Harvard Medical School
ICCS2005ICCS2005
2. GRID ARCHITECTURE
![Page 14: University of California, San Diego San Diego Supercomputer Center Computational Radiology Laboratory Brigham & Women’s Hospital, Harvard Medical School](https://reader030.vdocuments.mx/reader030/viewer/2022032523/56649d7e5503460f94a619ce/html5/thumbnails/14.jpg)
University of California, San Diego
San Diego Supercomputer CenterComputational Radiology LaboratoryBrigham & Women’s Hospital, Harvard Medical School
ICCS2005ICCS2005
Queue Delay Experiment on TeraGrid Cluster
• TeraGrid is a NSF funded grid infrastructure across multiple research and academic sites
• Queue delays at SDSC and NCSA TG were measured over 3 days for 5 mins wall clock time on 2 to 64 CPUs
• Single job submitted at a time• If job didn’t start within 10 mins, job terminated, next one
processed• What is the likelihood of job running• 313 jobs to NCSA TG cluster and 332 to SDSC TG
cluster – 50 to 56 jobs of each size on each cluster
![Page 15: University of California, San Diego San Diego Supercomputer Center Computational Radiology Laboratory Brigham & Women’s Hospital, Harvard Medical School](https://reader030.vdocuments.mx/reader030/viewer/2022032523/56649d7e5503460f94a619ce/html5/thumbnails/15.jpg)
University of California, San Diego
San Diego Supercomputer CenterComputational Radiology LaboratoryBrigham & Women’s Hospital, Harvard Medical School
ICCS2005ICCS2005
% of submitted tasks that run, as a fn of CPUs requested
![Page 16: University of California, San Diego San Diego Supercomputer Center Computational Radiology Laboratory Brigham & Women’s Hospital, Harvard Medical School](https://reader030.vdocuments.mx/reader030/viewer/2022032523/56649d7e5503460f94a619ce/html5/thumbnails/16.jpg)
University of California, San Diego
San Diego Supercomputer CenterComputational Radiology LaboratoryBrigham & Women’s Hospital, Harvard Medical School
ICCS2005ICCS2005
Average queue delay for tasks that began running within10 mins
![Page 17: University of California, San Diego San Diego Supercomputer Center Computational Radiology Laboratory Brigham & Women’s Hospital, Harvard Medical School](https://reader030.vdocuments.mx/reader030/viewer/2022032523/56649d7e5503460f94a619ce/html5/thumbnails/17.jpg)
University of California, San Diego
San Diego Supercomputer CenterComputational Radiology LaboratoryBrigham & Women’s Hospital, Harvard Medical School
ICCS2005ICCS2005
Queue Delay Test Conclusion
• There appears to be a direct relationship between the size of request and the length of the queue delay
• Two clusters exhibit different performance profiles
• This behavior of queue systems clearly merits further study
• More rigorous statistical characterization ongoig on much larger data sets
![Page 18: University of California, San Diego San Diego Supercomputer Center Computational Radiology Laboratory Brigham & Women’s Hospital, Harvard Medical School](https://reader030.vdocuments.mx/reader030/viewer/2022032523/56649d7e5503460f94a619ce/html5/thumbnails/18.jpg)
University of California, San Diego
San Diego Supercomputer CenterComputational Radiology LaboratoryBrigham & Women’s Hospital, Harvard Medical School
ICCS2005ICCS2005
Data Transfer• We are investigating grid based data transfer mechanisms such as
globus-url-copy, SRB• All hospitals have firewalls for security and patient data privacy –
single port of entry to internal machines
Transfer direction
Globus-url-copy
SRB Scp Scp –C
TG to BWH 50 49 68 31
BWH to TG 9 12 40 30
Transfer time in seconds for 20 MB fileTransfer time in seconds for 20 MB file
![Page 19: University of California, San Diego San Diego Supercomputer Center Computational Radiology Laboratory Brigham & Women’s Hospital, Harvard Medical School](https://reader030.vdocuments.mx/reader030/viewer/2022032523/56649d7e5503460f94a619ce/html5/thumbnails/19.jpg)
University of California, San Diego
San Diego Supercomputer CenterComputational Radiology LaboratoryBrigham & Women’s Hospital, Harvard Medical School
ICCS2005ICCS2005
3. ADVANCED BIOMECHANICAL MODEL
![Page 20: University of California, San Diego San Diego Supercomputer Center Computational Radiology Laboratory Brigham & Women’s Hospital, Harvard Medical School](https://reader030.vdocuments.mx/reader030/viewer/2022032523/56649d7e5503460f94a619ce/html5/thumbnails/20.jpg)
University of California, San Diego
San Diego Supercomputer CenterComputational Radiology LaboratoryBrigham & Women’s Hospital, Harvard Medical School
ICCS2005ICCS2005
Mesh Model with Brain Segmentation
![Page 21: University of California, San Diego San Diego Supercomputer Center Computational Radiology Laboratory Brigham & Women’s Hospital, Harvard Medical School](https://reader030.vdocuments.mx/reader030/viewer/2022032523/56649d7e5503460f94a619ce/html5/thumbnails/21.jpg)
University of California, San Diego
San Diego Supercomputer CenterComputational Radiology LaboratoryBrigham & Women’s Hospital, Harvard Medical School
ICCS2005ICCS2005
Current and New Biomechanical Model• Current linear elastic material model – RTBM• Advanced model under development - FAMULS• Advanced model is based on conforming
adaptive refinement method – FAMULS package (AMR)
• Inspired by the theory of wavelets this refinement produces globally compatible meshes by construction
• First task is to replicate the linear elastic result produced by the RTBM code using FAMULS
![Page 22: University of California, San Diego San Diego Supercomputer Center Computational Radiology Laboratory Brigham & Women’s Hospital, Harvard Medical School](https://reader030.vdocuments.mx/reader030/viewer/2022032523/56649d7e5503460f94a619ce/html5/thumbnails/22.jpg)
University of California, San Diego
San Diego Supercomputer CenterComputational Radiology LaboratoryBrigham & Women’s Hospital, Harvard Medical School
ICCS2005ICCS2005
FEM Mesh : FAMULS & RTBM
RTBM (Uniform)RTBM (Uniform)FAMULS (AMR)FAMULS (AMR)
![Page 23: University of California, San Diego San Diego Supercomputer Center Computational Radiology Laboratory Brigham & Women’s Hospital, Harvard Medical School](https://reader030.vdocuments.mx/reader030/viewer/2022032523/56649d7e5503460f94a619ce/html5/thumbnails/23.jpg)
University of California, San Diego
San Diego Supercomputer CenterComputational Radiology LaboratoryBrigham & Women’s Hospital, Harvard Medical School
ICCS2005ICCS2005
Deformation Simulation After Cut
No – AMR FAMULSNo – AMR FAMULS 3 level AMR FAMULS3 level AMR FAMULS RTBM RTBM
![Page 24: University of California, San Diego San Diego Supercomputer Center Computational Radiology Laboratory Brigham & Women’s Hospital, Harvard Medical School](https://reader030.vdocuments.mx/reader030/viewer/2022032523/56649d7e5503460f94a619ce/html5/thumbnails/24.jpg)
University of California, San Diego
San Diego Supercomputer CenterComputational Radiology LaboratoryBrigham & Women’s Hospital, Harvard Medical School
ICCS2005ICCS2005
Advanced Biomechanical Model
• The current solver is based on small strain isotropic elastic principle
• The new biomechanical model will be inhomogeneous scalable non-linear viscoelastic model with AMR
• We also want to increase resolution close to the level of MRI voxels i.e. millions of FEM meshes
• Since this complex model still has to meet the real time constraint of neurosurgery it requires fast access to remote multi-tflop systems
![Page 25: University of California, San Diego San Diego Supercomputer Center Computational Radiology Laboratory Brigham & Women’s Hospital, Harvard Medical School](https://reader030.vdocuments.mx/reader030/viewer/2022032523/56649d7e5503460f94a619ce/html5/thumbnails/25.jpg)
University of California, San Diego
San Diego Supercomputer CenterComputational Radiology LaboratoryBrigham & Women’s Hospital, Harvard Medical School
ICCS2005ICCS2005
4. PARALLEL AND END-to-END TIMING
![Page 26: University of California, San Diego San Diego Supercomputer Center Computational Radiology Laboratory Brigham & Women’s Hospital, Harvard Medical School](https://reader030.vdocuments.mx/reader030/viewer/2022032523/56649d7e5503460f94a619ce/html5/thumbnails/26.jpg)
University of California, San Diego
San Diego Supercomputer CenterComputational Radiology LaboratoryBrigham & Women’s Hospital, Harvard Medical School
ICCS2005ICCS2005
Parallel Registration Performance
0
500
1000
1500
2000
2500
3000
1 2 3 4
# of CPUs
Ela
pse
d T
ime
(sec
)
patient1
patient2
![Page 27: University of California, San Diego San Diego Supercomputer Center Computational Radiology Laboratory Brigham & Women’s Hospital, Harvard Medical School](https://reader030.vdocuments.mx/reader030/viewer/2022032523/56649d7e5503460f94a619ce/html5/thumbnails/27.jpg)
University of California, San Diego
San Diego Supercomputer CenterComputational Radiology LaboratoryBrigham & Women’s Hospital, Harvard Medical School
ICCS2005ICCS2005
Parallel Rendering Performance
![Page 28: University of California, San Diego San Diego Supercomputer Center Computational Radiology Laboratory Brigham & Women’s Hospital, Harvard Medical School](https://reader030.vdocuments.mx/reader030/viewer/2022032523/56649d7e5503460f94a619ce/html5/thumbnails/28.jpg)
University of California, San Diego
San Diego Supercomputer CenterComputational Radiology LaboratoryBrigham & Women’s Hospital, Harvard Medical School
ICCS2005ICCS2005
Parallel RTBM Performance
(43584 meshes, 214035 tetrahedral elements)
-
10.00
20.00
30.00
40.00
50.00
60.00
1 2 4 8 16 32
# of CPUs
Ela
pse
d T
ime
(sec
)
IBM Power3
IA64 TeraGrid
IBM Power4
![Page 29: University of California, San Diego San Diego Supercomputer Center Computational Radiology Laboratory Brigham & Women’s Hospital, Harvard Medical School](https://reader030.vdocuments.mx/reader030/viewer/2022032523/56649d7e5503460f94a619ce/html5/thumbnails/29.jpg)
University of California, San Diego
San Diego Supercomputer CenterComputational Radiology LaboratoryBrigham & Women’s Hospital, Harvard Medical School
ICCS2005ICCS2005
End to End (BWH SDSCBWH) Timing
• RTBM – not during surgery
• Rendering - during Surgery
![Page 30: University of California, San Diego San Diego Supercomputer Center Computational Radiology Laboratory Brigham & Women’s Hospital, Harvard Medical School](https://reader030.vdocuments.mx/reader030/viewer/2022032523/56649d7e5503460f94a619ce/html5/thumbnails/30.jpg)
University of California, San Diego
San Diego Supercomputer CenterComputational Radiology LaboratoryBrigham & Women’s Hospital, Harvard Medical School
ICCS2005ICCS2005
End-to-end Timing of RTBM
• Timing of transferring ~20 MB files from BWH to SDSC, running simulations on 16 nodes (32 procs), transferring files back to BWH = 9* + (60** + 7***) + 50* = 124 sec.
• This shows that the grid infrastructure can provide biomechanical brain deformation simulation solutions (using the linear elastic model) to surgery rooms at BWH within ~ 2 mins using TG machines
• This satisfies the tight time constraint set by the neurosurgeons
![Page 31: University of California, San Diego San Diego Supercomputer Center Computational Radiology Laboratory Brigham & Women’s Hospital, Harvard Medical School](https://reader030.vdocuments.mx/reader030/viewer/2022032523/56649d7e5503460f94a619ce/html5/thumbnails/31.jpg)
University of California, San Diego
San Diego Supercomputer CenterComputational Radiology LaboratoryBrigham & Women’s Hospital, Harvard Medical School
ICCS2005ICCS2005
End-to-end Timing of Rendering
• MRI data from BWH was transferred to SDSC during a surgery
• Parallel rendering was performed at SDSC• Rendered viz was sent back to BWH (but not
shown to surgeons)• Total time (for two sets of data) in sec =
2*53 (BWH to SDSC) + 2* 7.4 (render on 32 procs) + 0.2 (overlapping viz) + 13.7 (SDSC to BWH) = 148.4 sec
DURING SURGERYDURING SURGERY
![Page 32: University of California, San Diego San Diego Supercomputer Center Computational Radiology Laboratory Brigham & Women’s Hospital, Harvard Medical School](https://reader030.vdocuments.mx/reader030/viewer/2022032523/56649d7e5503460f94a619ce/html5/thumbnails/32.jpg)
University of California, San Diego
San Diego Supercomputer CenterComputational Radiology LaboratoryBrigham & Women’s Hospital, Harvard Medical School
ICCS2005ICCS2005
5. SUMMARY
![Page 33: University of California, San Diego San Diego Supercomputer Center Computational Radiology Laboratory Brigham & Women’s Hospital, Harvard Medical School](https://reader030.vdocuments.mx/reader030/viewer/2022032523/56649d7e5503460f94a619ce/html5/thumbnails/33.jpg)
University of California, San Diego
San Diego Supercomputer CenterComputational Radiology LaboratoryBrigham & Women’s Hospital, Harvard Medical School
ICCS2005ICCS2005
Ongoing and Future DDDAS Research
• Continuing research and development in grid architecture, on demand computing, data transfer
• Continuing development of advanced biomechanical model and parallel algorithm
• Moving towards near-continuous DDDAS instead of once an hour or so 3-D MRI based DDDAS
• Scanner at BWH can provide one 2-D slice every 3 sec or three orthogonal 2-D slices every 6 sec
• Near-continuous DDDAS architecture• Requires major research, development and implementation work in
the biomechanical application domain • Requires research in the closed loop system of dynamic image driven
continuous biomechanical simulation and 3-D volumetric FEM results based surgical navigation and steering