parallel computational steering and analysis for hpc applications using a paraview interface and the...
TRANSCRIPT
1Parallel Computational Steering and Analysis for HPC Applications using a ParaView Interface
Parallel Computational Steering and Analysis for HPC Applications using a ParaView Interface and the HDF5 DSM Virtual File DriverJ. Biddiscombe, J. Soumagne, G. Oger, D. Guibert, J.G. Piccinali
Monday, 11 April 2011
2Parallel Computational Steering and Analysis for HPC Applications using a ParaView Interface
Project/Funding
Monday, 11 April 2011
NextMuSE (http://nextmuse.cscs.ch)
Next generation Multi-mechanics Simulation Environment
European Community’s Seventh Framework Programme (FP7/2007-2013)
Information and Communication Technologies (ICT)
Small or medium-scale focused research project - Specific Targeted Research Project (STREP)
Future and Emerging Technologies Open Scheme (FET-Open)
3Parallel Computational Steering and Analysis for HPC Applications using a ParaView Interface
Project : Naval Partners (branching out)
Monday, 11 April 2011
4Parallel Computational Steering and Analysis for HPC Applications using a ParaView Interface
leafletshinges
closedopen
25 mm
Project : Mechanical heart valves
Reynolds number 6000
Design static since early 1980s
Monday, 11 April 2011
5Parallel Computational Steering and Analysis for HPC Applications using a ParaView Interface
Project : TurboMachinery Partners
Monday, 11 April 2011
6Parallel Computational Steering and Analysis for HPC Applications using a ParaView Interface
Requirements
In-Situ Visualization Computational Steering Two way exchange of potentially large data
(simulation has no meshing capability)
Multiple codes/languages Treat Cluster/Supercomputer like workstation extension
Constraints Partners who are far away ... and who can’t program their way out of a paper bag
(because they’re physicists really)
Monday, 11 April 2011
7Parallel Computational Steering and Analysis for HPC Applications using a ParaView Interface
Ideal Solution
Sim Node
Sim Node
Sim Node
Sim Node
Vis Code
Vis Code
Vis Code
Vis Code
Sim Node
Sim Node
M Nodes
Vis Code
Vis Code
Visit/ParaView
Use this approach for in-situ visualization
Memory pressures
(especially for time-dependent data)
If Vis/Analysis code doesn’t scale as well as
simulation code, possible problem
Render Server
Render Server
Render Server
Render Server
Data Server
Monday, 11 April 2011
8Parallel Computational Steering and Analysis for HPC Applications using a ParaView Interface
Parallel IO (c.f. Lustre)
Sim Node
Sim Node
Sim Node
Sim Node
OSS Node
OSS Node
OSS Node
OSS Node
Sim Node
Sim Node
M Nodes N OSS Nodes
All IO goes through OSS nodes
HDF5 Model already
looks like this
Monday, 11 April 2011
9Parallel Computational Steering and Analysis for HPC Applications using a ParaView Interface
DSM Implementation
Sim Node
Sim Node
Sim Node
Sim Node
Vis Node
Vis Node
Vis Node
Vis Node
DSM
DSM
DSM
DSM
Sim Node
Sim Node
M Nodes N Nodes
DSM
DSM
HDF5/MPI
HDF IO intercepted and routed to DSM
HDF5/MPI
DSM Model could
look like this
Just replace the IO
layer in HDF5
IO Forwarding
c.f. ADIOS etc
Monday, 11 April 2011
10Parallel Computational Steering and Analysis for HPC Applications using a ParaView Interface
Analysis/Vis
Machine
Simulation
Machine
(Infiniband)
Switch
Simulation
Nodes
Analysis/Vis
Nodes
DS
M
DS
M
DSM may be on
either side
Render
Workstation
GU
I
GUI may be
separate
Workstation
DS
M
Simulation
Machine
DS
M
DS
M
GU
I
Render
Render
GU
I
GU
I
Render
Analysis/Vis
Workstation
GU
I
Render
3
2
2
Monday, 11 April 2011
Configurations
11Parallel Computational Steering and Analysis for HPC Applications using a ParaView Interface
H5FDdsm Bandwidth
J. Soumagne, J. Biddiscombe, “Computational Steering and Parallel Online Monitoring Using RMA through the HDF5 DSM Virtual File Driver”, International Conference on Computational Science, ICCS 2011
J. Soumagne, J. Biddiscombe, J. Clarke, “An HDF5 MPI virtual file driver for parallel in-situ post-processing”, EuroMPI 2010, Sep 2010. vol. 6305 of Lecture Notes in Computer Science. Springer Berlin/Heidelberg, 2010, pp. 62–71
Talk to Jerome for further info on numbersMonday, 11 April 2011
12Parallel Computational Steering and Analysis for HPC Applications using a ParaView Interface
Our driver: H5FDdsm Data sent into HDF5 is then automatically redirected to this driver Currently uses flat memory model (may be improved by scattering)
more links = more BW
HDF5 – Virtual File Drivers
Monday, 11 April 2011
…start
end of file (eof)
local length 𝟐× ( local length )(𝑵 −𝟏 )×(local length )𝟎 𝑵 ×(local length )
eof + metadata =total length of DSMNetwork
network layer = plugin
TCP / MPI / MPI_RMA
(Gemini/DMAPP)
13Parallel Computational Steering and Analysis for HPC Applications using a ParaView Interface
Performance – Write test with a 20GB DSM distributed among 8 post-processing nodes on a Cray XT5
Monday, 11 April 2011
16 32 64 128 256 5120
2000
4000
6000
8000
10000
12000
Parallel File System (Lustre)H5FDdsm (Socket over SeaStar2+ Interconnect)
Number of PE used for writing to the DSM
Dat
a w
rite
rate
(GB/
s)
Saturation of the network
14Parallel Computational Steering and Analysis for HPC Applications using a ParaView Interface
Performance – Steering overhead
16 32 640
2000
4000
6000
8000
10000
12000
Write test with a 20GB DSM distributed among 8 post-processing nodes on a Regular cluster
Parallel File System (GPFS)H5FDdsm (Socket over Infiniband QDR 4x)H5FDdsm (MVAPICH2 over Infiniband QDR 4x)
Number of PE used for writing to the DSM
Data
writ
e ra
te (G
B/s)
Monday, 11 April 2011
Series10
100
200
300
400
500
600
700
800
900
Steering overhead using a unique time step of an arbitrary simulation (here GADGET-2)
File System H5FDdsm (No Steering)H5FDdsm (With Steering)
Tim
e (m
s)
15Parallel Computational Steering and Analysis for HPC Applications using a ParaView Interface
Current developments
Scalability on large systems Dynamic MPI communication not
supported Use MPMD job instead of two
separate SPMD jobs
Additional communicator designed for Gemini systems RMA here does not perform so well
DMAPP communicator (reachable bandwidth of 5GB/s between nodes)
16 64128
256512
10241536
20483072
0
5,000
10,000
15,000
20,000
25,000
30,000
35,000
H5FDdsm write test of 96GB to 176DSM PEs (44 nodes) using MPI RMA
on Cray XE6 (palu)
Number of PE used for writing to the DSM
Da
ta w
rite
ra
te (
MB
/s)
Monday, 11 April 2011
16Parallel Computational Steering and Analysis for HPC Applications using a ParaView Interface
In-Situ Visualization – How to use the driver
Create a file access property list
Instead of using the MPI-IO driver
Insert our DSM driver
Then carry on as usual
Comment out any MPIO_COLLECTIVE dataset transfer properties
Monday, 11 April 2011
H5Pset_fapl_dsm(fapl_id, MPI_COMM_WORLD, NULL);
H5Pset_fapl_mpio(fapl_id, MPI_COMM_WORLD, MPI_INFO_NULL);
file_id = H5Fcreate(filename, H5F_ACC_TRUNC, H5P_DEFAULT, acc_plist_id);H5Pclose(acc_plist_id);H5Fclose(file_id);
fapl_id = H5Pcreate(H5P_FILE_ACCESS);
17Parallel Computational Steering and Analysis for HPC Applications using a ParaView Interface
In-Situ Visualization – Create Template
Monday, 11 April 2011
Built on top of Xdmf model and format
Generate Xdmf files to drive Xdmf reader in plugin
Uses in-memory H5dump to get the missing (metadata) information
Much simpler to write Allows variable number of
elements
<Grid> <Topology TopologyType="Polyvertex"> </Topology> <Geometry GeometryType="XYZ"> <DataItem>/fluid/position</DataItem> </Geometry> <Attribute> <DataItem>/fluid/density</DataItem> </Attribute> <Attribute> <DataItem>/fluid/velocity</DataItem> </Attribute> <Attribute> <DataItem>/fluid/vmob</DataItem> </Attribute> <Attribute> <DataItem>/fluid/w</DataItem> </Attribute></Grid>
18Parallel Computational Steering and Analysis for HPC Applications using a ParaView Interface
In-Situ Visualization – Turbine
Monday, 11 April 2011
19Parallel Computational Steering and Analysis for HPC Applications using a ParaView Interface
Selectable Arrays
ParaView doesn’t have ‘contracts’, so from the template, we can generate on/off switches for each exported dataset/array
Simulation can check if an array is ‘enabled’ and not send it. (better than turning it off in the XdmfReader)
We don’t use HDF5 for this, just a metadata flag maintained by the H5FDdsm library.
Monday, 11 April 2011
H5FD_dsm_steering_is_enabled(const char *name); // /grid/group/name
20Parallel Computational Steering and Analysis for HPC Applications using a ParaView Interface
Note on Steering Architecture
Monday, 11 April 2011
ParaView
Servers
…
Set Steering Commands and Disabled Objects in
metadata section
Read simulation data – write user defined steering commands and data into HDF5 “file” section
Parallel
Simulation
Write output data – Read user defined steering commands and data
Get Steering Commands and Disabled Objects
21Parallel Computational Steering and Analysis for HPC Applications using a ParaView Interface
Computational Steering
Each simulation requires custom controls
And different analysis requirements
Monday, 11 April 2011
Vis + GUI +
AnalysisHeart
Valve
Sloshing
Ship
Turbine
Deforming
Hull
22Parallel Computational Steering and Analysis for HPC Applications using a ParaView Interface
Automatic GUI Generation : Interactions XML
<DataExportProperty
name="ModifiedBodyNodes"
command="SetSteeringArray"
label="Modified Body Node Data">
<DataExportDomain name="data_export"
full_path="/Mesh_DataSet"
geometry_path="/Mesh_Nodes#1/NewXYZ"
topology_path="/Mesh_Nodes#1/NewCo..."
command_property="ReloadFreeBodyMesh">
</DataExportDomain>
</DataExportProperty>
Monday, 11 April 2011
Reuse ServerManager syntax from ParaView
Added some extra Properties
Can also link GUI 3D widgets using ‘hints’
23Parallel Computational Steering and Analysis for HPC Applications using a ParaView Interface
No ‘app-specific’ knowledge required
Generate ServerManager XML from template
parse hints/commands and other stuffRegister it internally on the flyNo need to (re)compile paraview plugin
Unless you have some custom heart valve adjustment widget that is specially designed for that app
Engineer only modifies simulation ‘Names’ and XML template
Monday, 11 April 2011
24Parallel Computational Steering and Analysis for HPC Applications using a ParaView Interface
H5FDdsm Steering API
Initialize the steering extension (buffer is optional)
Get steering commands and disabled objects
Test if “name” is a disabled object
Get/Set user defined scalar parameter
Get/Set user defined vector parameter
Monday, 11 April 2011
H5FD_dsm_steering_init( MPI_Comm comm, void *buffer);
H5FD_dsm_steering_update();
H5FD_dsm_steering_is_enabled(const char *name);
H5FD_dsm_steering_scalar_get(const char *name, hid_t mem_type, void *data);
H5FD_dsm_steering_vector_get(const char *name, hid_t mem_type, hsize_t
number_of_elements, void *data);
25Parallel Computational Steering and Analysis for HPC Applications using a ParaView Interface
H5FDdsm Steering API
Test if a user defined parameter is set
Wait for an incoming “play” command
Begin/End query and avoid multiple metadata accesses
Get/Free dataset handle for data specific parallel read
Monday, 11 April 2011
H5FD_dsm_steering_is_set(const char *name, int *set);
H5FD_dsm_steering_begin_query();
H5FD_dsm_steering_get_handle(const char *name, hid_t *handle);
H5FD_dsm_steering_free_handle(hid_t handle);
H5FD_dsm_steering_wait();
26Parallel Computational Steering and Analysis for HPC Applications using a ParaView Interface
Can switch between
modes arbitrarily
(or every N iterations)
Calculation
Write
Initialization
Analysis
+ GUI
Interaction
CalculationWait
mode
Analysis
+ GUI
Interaction
File
Wait
mode
Wait Mode
Monday, 11 April 2011
Computation/Analysis not overlapped
Define some pipeline in the
ParaView GUI which does
something interesting
(to send data back)
H5FD_dsm_steering_wait();
27Parallel Computational Steering and Analysis for HPC Applications using a ParaView Interface
GUI
Write
Initialization
Ana
lysi
s
File
Calculation
Ana
lysi
s
Ana
lysi
s
Free Mode
Monday, 11 April 2011
Computation/Analysis overlapped
Can switch between
modes arbitrarily
(or every N iterations)
File Create wipes DSM
clean
So may want to add more
sync optionsCan store N
time steps
28Parallel Computational Steering and Analysis for HPC Applications using a ParaView Interface
Steering demo (recorded)
Monday, 11 April 2011
29Parallel Computational Steering and Analysis for HPC Applications using a ParaView InterfaceMonday, 11 April 2011
Steering demo (recorded)
30Parallel Computational Steering and Analysis for HPC Applications using a ParaView Interface
Animated Wedge
Monday, 11 April 2011
31Parallel Computational Steering and Analysis for HPC Applications using a ParaView Interface
Conclusion
Minimally intrusive way to steer applications Supports Parallel for Sim and Vis/Analysis Allows remeshing/exchange of large data
Can do stuff that we couldn’t do before!
Further work Now developing new MPI mode for large core counts
DMAPP/Gemini layer
Turbine mesh next target for manipulation
Monday, 11 April 2011
32Parallel Computational Steering and Analysis for HPC Applications using a ParaView Interface
Software available on HPCforge
Monday, 11 April 2011
H5FDdsm: https://hpcforge.org/projects/h5fddsm
ICARUS: (ParaView plugin) https://hpcforge.org/projects/icarus
XdmfGenerator: https://hpcforge.org/projects/xdmfgenerator
Questions: