s. guatelli, a. mantero, j. moscicki, m. g. pia geant4 medical simulations in a distributed...
TRANSCRIPT
S. Guatelli, A. Mantero, J. Moscicki, M. G. Pia
Geant4 medical simulations Geant4 medical simulations in a distributed computing in a distributed computing
environmentenvironment
4th Workshop onGeant4 Bio-medical Developments
Geant4 Physics ValidationINFN Genova, 13-20 July 2005
OutlineOutline
Problem: how to obtain quick response Brief introduction of DIANE How to parallelize a Geant4 application Project
– Parallelization of two medical physics Geant4 applications• Brachytherapy • IMRT application
– Study of the performance• Using a dedicated cluster• Using the GRID
First results Work in progress
Execution time - BrachyExecution time - Brachy
Number of events for a sufficient statistic for a dosimetric study of a single brachytherapic source: 20 M events
Execution time of 20 M events on a Pentium IV, 3 GHz – 16650 s ~ 5 h
Clinical use: quick response means order of minutes
Execution times – IMRT Execution times – IMRT
Number of events for a sufficient statistic: 109 events
Execution time of 109 events: – 822890. s ~ 228 h ~ 9 days and half
Quick response required for clinical use
Speed adequate for clinic useSpeed adequate for clinic useSpeed adequate for clinic useSpeed adequate for clinic use
Transparent configuration in sequential or parallel mode
Transparent access to the GRID through an Transparent access to the GRID through an intermediate software layerintermediate software layer
Parallelisation
Access to distributed computing resources
ProjectProject
Parallelization of the Geant4 IMRT and Brachytherapy application
Parallelization through DIANE
Performance test– Run on a single machine– Run on a dedicated cluster– Run on the GRID
DIANE DIANE DIDIstributed stributed ANANalysis alysis EEnvironmentnvironment
speed OK but expensive hardware
investment + maintenance
IRCC LAN
SWITCH
Node01
Node02
Node03
Node04 IMRT
Geant4 Simulation and Anaphe Analysis on a dedicated Beowulf Cluster
S. Chauvie et al., IRCC Torino, Siena 2002
Previous studies for parallelization of a Geant4 based medical application
Alternative strategy
DIANEDIANE
Parallelisation Access to the GRID
Transparent access to a distributed computing environment
DIANE DIANE DIDIstributed stributed ANANalysis alysis EEnvironmentnvironment
prototype for an intermediate layer between applications and the GRID
Parallel cluster processingParallel cluster processing– make fine tuning and customisation easy– transparently using GRID technology– application independentapplication independent
Hide complex
details of underlying technology
Developed by J. Moscicki, CERNhttp://cern.ch/DIANE
Practical examplePractical example
How to “dianize” the Geant4 application:
– Look the Geant4 extended example: ExDIANEin the parallel directory
– Completely transparent to the user: same G4 code
– Documentation at www.cern.ch/diane/ specific for Geant4 applications available
Run through DIANERun through DIANE#-*-python-*-Application = "G4Analysis"WorkerInitData = {
'G4ApplicationComponentName' : "G4MedLinac", ‘initMacroFile' : ""“/control/verbose 1/run/verbose 1/control/saveHistory/run/initialize/tracking/storeTrajectory 1/Jaws/X1/DistanceFromAxis -5.0 cm/Jaws/X2/DistanceFromAxis 5.0 cm/Jaws/Y1/DistanceFromAxis -5.0 cm/Jaws/Y2/DistanceFromAxis 5.0 cm/Jaws/update/energy 6.0 MeV/sourceType 0.127 MeV """}JobInitData = {
'runParams' : { 'seed' : 0 ,
'eventNumber' : 100000, 'macroFileTemplate': "/run/beamOn " },
'eventsPerWorker' :10000, 'workerInit' : WorkerInitData }
Example of a macro file
Run on parallel mode on a Run on parallel mode on a dedicated clusterdedicated cluster
Type the command:– Diane.startjob –j macrofileName.mac
w2@cluster –wms=IPLIST
IPLIST if a file name containing the list of the names of the machines the user intends to use
Practical ExamplePractical Example example: Geant4 simulation with analysis
• the total number of events of the simulation is divided in tasks
• each task produces a file with histograms
• job result = sum of histograms produced by tasks
master-worker model• client starts a job
• workers perform tasks
and produce histograms
• master integrates
the results
Resources of this projectResources of this project
Dedicated cluster of 4 pc’s in Genova (Pentium IV, 3 GHz)– For preliminary tests
Dedicated 30 pc’s (biprocessors) cluster (Xeon, 2.8 GHz)– Thanks to H.C. Lee, Academia Sinica Computing Center, Taiwan
LSF cluster at CERN – To study a real case of running on a cluster, used by more users
GRID– Run on a distributed computing environment
Running on a single CPURunning on a single CPU
To study the efficiency of DIANE
Plot of:
with respect to the number of events
Execution time using DIANE means running sequentially the dianized simulation, dividing the job in tasks
Useful study to optimize the number of events for task
Brachy-Iridium source simulation
Overhead of DIANEdianenosequentialtimeExecution
dianewithtimeExecution
____
___
Preliminary
Preliminary resultsPreliminary results
Running on a dedicated cluster– Divide the total
number of events in tasks
– Dispatch the tasks on more workers
Execution times of the Brachy with respect to the number of CPU’s used
No merging of the output files of the simulations
Preliminary resultsPreliminary resultsRunning on a
dedicated cluster
Efficiency:
The efficiency is higher with higher number of tasks
neparalleltimeExecution
esequentialtimeExecution
mod___
mod___
N is the number of CPUs
Preliminary
Performance OptimizationPerformance OptimizationWhy the efficiency is higher with higher number of
tasks
• The execution time is bigger because there is still one task to end
• Splitting the job in more tasks increases the balance in execution times of the workers
Work in progressWork in progress
Optimization of the method to merge the output files of the tasks– In the present situation the merging introduces a
significant overhead on the results – We found problems with adding histograms with PI– A. Pfeiffer and L. Moneta are helping in this last task
Last step: running on the GRID
Refine the results