code saturne a h.p.c point of view - welcome to...
TRANSCRIPT
Code_Saturnea H.P.C point of
viewDaresbury meeting
June, 26th 2008
Presentation of Code_Saturne
June, 26th 2008J. Bonelle - EDF R&D - Code_Saturne : a H.P.C. point of view http://rd.edf.com/code_saturne3
Code_SaturneEDF General purpose CFD software
Code_SaturneEDF General purpose CFD software
Open source since March 2007
http://rd.edf.com/code_saturne (source code and manuals)
[email protected] (support)
Basic capabilities :
Simulation of incompressible or expandable flows with or without heat transfer and turbulence (mixing length, 2-equation models, v2f, RSM, LES, …)
1998Prototype
20001.0
Basic modelling
Wide range of meshes
2001Qualification for nuclear application
20041.1
Parallelism
L.E.S
20061.2
State of the art in turbulence
2007Open source
20081.3
Massively parallel
ALE
Code coupling
June, 26th 2008J. Bonelle - EDF R&D - Code_Saturne : a H.P.C. point of view http://rd.edf.com/code_saturne4
Code_SaturneEDF General purpose CFD software
Code_SaturneEDF General purpose CFD software
Co-located Finite Volume approach
Meshes with any type of cells (tetrahedral to polyhedral)
PWR lower plenum
Example of mesh with stretched cells and hanging nodes
3D polyhedral cells
Example of composite mesh
June, 26th 2008J. Bonelle - EDF R&D - Code_Saturne : a H.P.C. point of view http://rd.edf.com/code_saturne5
Code_SaturneEDF General purpose CFD software
Code_SaturneEDF General purpose CFD software
Main application area :
Nuclear power plant optimisation in terms of lifespan, productivity and safety.
But also applications in :
Combustion (gas and coal)
Electric arc and joule effect
Atmospheric flows
Radiative heat transfer
Other functionalities :
Fluid structure interaction
Deformable meshes : Arbitrary Lagragian Eulerianmethod (ALE)
Dispersed particle tracking (Lagrangian approach)
June, 26th 2008J. Bonelle - EDF R&D - Code_Saturne : a H.P.C. point of view http://rd.edf.com/code_saturne6
Code_SaturnePortability and parallelism
Code_SaturnePortability and parallelism
Available on a wide range of operating systems : Linux, Unix (HP-UX, Solaris, Cray, OSF1, Mac OSX …)
Available on a wide range of architectures :
IBM BlueGene/L and BlueGene/P (PowerPC 4)
IBM Cluster – MareNostrum, Barcelona Supercomputing Centre (PowerPC 5)
Bull Novascale (Intel Itanium2)
Cray XT3 and XT4 – Hector, Edinburgh (AMD Opteron)
Cluster of PCs (Intel Xeon, AMD Opteron, …)
Parallel on distributed memory machines thanks to MPI implementations (OpenMPI, MPICH2, …)
Tested so far from 1 to 8192 processors
Periodicity capability : translation and rotation
Domain splitting thanks to METIS
June, 26th 2008J. Bonelle - EDF R&D - Code_Saturne : a H.P.C. point of view http://rd.edf.com/code_saturne7
Code_Saturne architectureCode_Saturne architecture
FVMlibrary
Code_Saturne
Preprocessor (Envelope)
mesh import mesh pasting
domain decomposition
Post-processing
output
Restart files
Meshes
Parallel Kernelghost cells creation
periodicityCFD Solver
code coupling
parallel treatmentCode_Saturne
SYRTHESCode_Aster
Salome platform...
GUI
Xml data file
External libraries (EDF, LGPL):• BFT: Base Functions and Types• FVM: Finite Volume Mesh
BFT library
Serial I/Omemory management
Parallel meshManagementParallel I/O
June, 26th 2008J. Bonelle - EDF R&D - Code_Saturne : a H.P.C. point of view http://rd.edf.com/code_saturne8
Code_SaturneCode coupling capabilities
Code_SaturneCode coupling capabilities
All these softwares are under Gnu GPL or LGPL version 2
SALOME : http://www.salome-platform.org
Code_Aster : http://www.code-aster.org
SYRTHES : soon available
Parallel code coupling capabilities thanks to FVM (EDF’s “Finite Volume Mesh” library)
SALOME Platform
Code_Saturne
Code_Saturne
SYRTHESC.A.D, mesh, visualization, coupling platform
CFD + conjugate heat transfer and radiation
CFD + CFD = RANS/LES or Chimera approach
CFD + structural mechanics analysis
Code_Saturne’sHPC performance
June, 26th 2008J. Bonelle - EDF R&D - Code_Saturne : a H.P.C. point of view http://rd.edf.com/code_saturne10
Code_SaturneBenchmark test cases
Code_SaturneBenchmark test cases
Father
200 iterations
Turbulence = L.E.S
Number of cells in the mesh in million
Hypi
200 iterations
Turbulence = L.E.S
GRILLE
50 iterations
Turbulence = k-ε
Industrial studies Exploratory studies
1 10 100
June, 26th 2008J. Bonelle - EDF R&D - Code_Saturne : a H.P.C. point of view http://rd.edf.com/code_saturne11
Code_Saturne’s benchmark on EDF’s HPC resourcesCode_Saturne’s benchmark on EDF’s HPC resources
47.7022.900.662.40Theoretical
performance in Tflops
InfinibandIBMMyrinetInfinibandInterconnect
552
1.8 and 2.6 GHz
AMD Opteron
Tantale
8192
0.7 GHz
IBM PowerPC 440
BlueGene/L
Intel Itanium2 dual-coreAMD OpteronProcessor
1.6 GHz2.6 GHzFrequency in GHz
7456128Number of
proc. or cores
PlatineCluster ChatouMachine’s name
June, 26th 2008J. Bonelle - EDF R&D - Code_Saturne : a H.P.C. point of view http://rd.edf.com/code_saturne12
Code_SaturneFATHER test case (1M)
Code_SaturneFATHER test case (1M)
10
100
1 000
10 000
100 000
1 10 100 1 000 10 000
Number of processors
Elap
sed
time
(s)
Opteron 2.6 GHz- Myrinet
Tantale :Opteron 1.8/2.6 GHz - Infiniband
Platine : Itanium2 1.6 GHz - Infiniband
BlueGene/L : PowerPC440 0,7 GHz - IBM
June, 26th 2008J. Bonelle - EDF R&D - Code_Saturne : a H.P.C. point of view http://rd.edf.com/code_saturne13
Code_SaturneHYPI test case (10M)
Code_SaturneHYPI test case (10M)
100
1000
10000
100000
1000000
1 10 100 1000 10000
Number of processors
Elap
sed
time
(s)
Opteron 2.6 GHz- Myrinet
Tantale :Opteron 1.8/2.6 GHz - Infiniband
Platine : Itanium2 1.6 GHz - Infiniband
BlueGene/L : PowerPC440 0,7 GHz - IBM
June, 26th 2008J. Bonelle - EDF R&D - Code_Saturne : a H.P.C. point of view http://rd.edf.com/code_saturne14
Code_SaturneGRILLE test case (100M)
Code_SaturneGRILLE test case (100M)
100
1000
10000
100000
100 1000 10000
Number of processors
Elas
ped
time
(s)
Platine : Itanium2 1.6 GHz - Infiniband
BlueGene/L : PowerPC440 0,7 GHz - IBM
Cache effect
Code_Saturne’sHPC prospects
June, 26th 2008J. Bonelle - EDF R&D - Code_Saturne : a H.P.C. point of view http://rd.edf.com/code_saturne16
Code_SaturneWhy more and more flops ?
Code_SaturneWhy more and more flops ?
To make numerical studies with finer and larger meshes
Bigger computational domain (from a geometrical point of view)
Refined computational domain (simulation of very small scales)
To make “easily” many computations
“Monte Carlo” approaches : treatment of uncertainties
To increase the simulated physical time of phenomena
Computation of transitional phenomena
To take more physics into account
Use of LES turbulence modelling instead of k-εmodelling if necessary
Mixing different approaches : eulerian and lagrangian approaches
Multi-physics studies : fluid structure interactions, coupling codes
17
Code_Saturne – HPC Roadmap: application examples
2003 2010 201520072006Consecutive to the Civauxthermal fatigue event
Computations enable to better understand the wall thermal loading in an injection.
Knowing the root causes of the event ⇒ define a new design to avoid this problem.
Part of a fuel assembly3 grid assemblies
Computation with an L.E.S. approach for turbulent modelling
Refined mesh near the wall.
9 fuel assemblies
No experimental approach up to now
Will enable the study of side effects implied by the flow around neighbour fuel assemblies.
Better understanding of vibration phenomena and wear-out of the rods.
The whole vessel reactor
106 cells3.1013 operations
108 cells1016 operations
1010 cells5.1018 operations
109 cells3.1017 operations
107 cells6.1014 operations
Fujistu VPP 5000
1 of 4 vector processors
2 month length computation
Cluster, IBM Power5
400 processors
9 days
# 1 Gb of storage
2 Gb of memory
IBM Blue Gene/L « Frontier »
8000 processors
# 1 month
30 times the power of
IBM Blue Gene/L « Frontier »
# 1 month
# 15 Gb of storage
25 Gb of memory
# 10 Tb of storage
25 Tb of memory
# 1 Tb of storage
2,5 Tb of memory
# 200 Gb of storage
250 Gb of memory
Power of the computer Pre-processing not parallelized Pre-processing not parallelized
Mesh generation
resolved
… ibid. …
… ibid. …
Visualisation
500 times the power of
IBM Blue Gene/L « Frontier »
# 1 month
Computations with smaller and smaller scales in larger and larger geometries ⇒ a better understanding of physical phenomena ⇒ a more effective help for decision making
⇒ A better optimisation of the production (margin benefits)
Power of the computer
resolved
… ibid. …
Scalability / Solver
Appendix
June, 26th 2008J. Bonelle - EDF R&D - Code_Saturne : a H.P.C. point of view http://rd.edf.com/code_saturne19
Code_SaturneBenchmark results (1/2)
Code_SaturneBenchmark results (1/2)
Interest of using BLAS library
Example on platine : local dot product for the HYPI test case
233,7143,026,619,7With BLAS library (GFlop/s)
5,56,94,03,6Without BLAS library (GFlop/s)
2561283216Number of processors
Performance of the local matrix vector product
Example on 32 processors for the FATHER test case
4.1
5.3
Platine(Opt. BULL)
1,34,03,45,3Performancein GFlop/s
1.03.12.64.1Ratio
PowerPC 440 0,7 GHzItanium2 1,6 GHzOpteron 1,8 GHzOpteron 2,6 GHzProcessor
BlueGenePlatineTantaleCluster ChatouMachine
June, 26th 2008J. Bonelle - EDF R&D - Code_Saturne : a H.P.C. point of view http://rd.edf.com/code_saturne20
Code_SaturneBenchmark results : I/O issue (1/2)
Code_SaturneBenchmark results : I/O issue (1/2)
Elapsed time in outputting post-processing files and restart files
Grille test case : comparison between Platine and BlueGene/L
54 s
156 s
X
256
99 s
169 s
556 s
1024
199 sBlueGene/L: After I/O upgrade
180 sPlatine
487 sBlueGene/L: Before I/O upgrade
2048Number of processors
Elapsed time in outputting post-processing files and restart files increases with the number of processors
Because of our algorithm = send all data to write to rank 0.
Solution : use of MPI I/O
Criteria to evaluate I/O quality:
Intrinsic quality of each I/O node and their number
Interconnect and MPI implementation
Filesystem (Lustre Vs GPFS)
June, 26th 2008J. Bonelle - EDF R&D - Code_Saturne : a H.P.C. point of view http://rd.edf.com/code_saturne21
Code_SaturneHYPI test case (10M)
Code_SaturneHYPI test case (10M)
Opteron 2.6 GHz- Myrinet
Tantale :Opteron 1.8/2.6 GHz - Infiniband
Platine : Itanium2 1.6 GHz - Infiniband
BlueGene/L : PowerPC440 0,7 GHz - IBM
0
10
20
30
40
50
60
1 10 100 1 000 10 000
Number of processors
Elap
sed
time
(s)