hpc and combustion - genci cea...hpc and combustion thierry poinsot imft, université de toulouse,...
TRANSCRIPT
HPC and combustion
Thierry POINSOT
IMFT, Université de Toulouse, CNRS and CERFACS
Contributions from G. Staffelbach, L. Gicquel, B. Cuenot, O. Vermorel, F. Duchaine, E. Riber (CERFACS), L. Selle (IMFT), V. Moureau (CORIA), S. Mendez (UM2)
1
OUTLINE
• COMBUSTION RESEARCH AND HPC
• PERFORMING HPC COMBUSTION SIMULATION ON TIER0/TIER1 SYSTEMS FOR INDUSTRY:
!HOW CAN THIS BE DONE ? WHAT DID THE FRENCH COMMUNITY DO ?
!WHAT ARE THE WEAKEST PARTS OF OUR ORGANIZATION TO DO THIS ?
2
COMBUSTION OVERVIEW
Just remember two equations:
3
ENERGY ON EARTH TODAY = COMBUSTION
ENERGY ON EARTH = COMBUSTIONCOMBUSTION IS PRODUCING MORE THAN 90 PERCENT OF THE ENERGY TODAY. THIS WILL DECREASE... BUT NOT TOMORROW
4
COMBUSTION OVERVIEW
Just remember two equations:
5
ENERGY ON EARTH TODAY = COMBUSTION
ENERGY ON EARTH TOMORROW = COMBUSTION
CLIMATE CHANGE AND ENERGY MARKET: 2010/2030
•COMBUSTION SCIENCE MUST ALLOW THIS WITHOUT INCREASING EMISSIONS, WASTING FOSSIL FUELS OR MAKING CLIMATE CHANGE WORSE (!...)
6
OPTIMIZATION AND SIMULATION NEEDED
SIMULATION OF COMBUSTION: MULTISCALE -MULTIPHYSICS
7
Contrails
Engines!
Heat transfer, radiation, flow, turbulence, chemistry, fatigue, vibration, acoustics
COMBUSTION: MULTISCALE -MULTIPHYSICS
8
Within the combustion chamber: nanoseconds and nanometers
1 cm
1 cm
Fields of density in a H2-O2 engine
COMBUSTION: MULTISCALE -MULTIPHYSICS
9
Outside the engine: miles and days !
2 miles
WHICH EQUATIONS ?
•The Navier Stokes equations (5 + N unknowns: density, velocities and energy, N species). Partial differential equations -> non local, intense communication required
•Kinetics : N = 10 to 300 species reacting through 3000 reactions (everything local)
•Heat transfer through the walls, radiation, noise, soot
•All these flows are turbulent10
WHAT DO WE COMPUTE ?
•FINITE VOLUME CODES USING DOMAIN DECOMPOSITION AND MPI.
•Typically 100 Mcells with 100 variables at each cell (3 velocities, density, energy + 5 to 90 species) over 1000000 time steps. -> 10^16 unknowns
11
WHAT ABOUT PARALLELISM ?
•THE NAVIER STOKES EQUATIONS ARE ‘EMBARRASINGLY DIFFICULT’ TO PARALLELIZE -> THIS PROBLEM HAS BEEN IDENTIFIED AND IS REMODELING OUR COMMUNITY IN LARGER COLLABORATIVE TEAMS.
•THIS IS NOT A ‘ONE-PROFESSOR ONE-CODE’ SHOW ANY MORE BECAUSE THE CODES ARE USED FOR INDUSTRY APPLICATIONS ON A DAILY BASIS
12
13
ok
A joint initiative of French labs for the promotion of SUper-Computing for the modeling of Combustion, mixing and complex fluids in rEal SyStems.
The SUCCESS scientific groupSUCCESS was created in 2012 to help the promotion of super-computing in the area of Computational Fluid Dynamics (CFD) for complex geometries. It is coordinated by the CORIA lab and is composed of 8 French public labs
Our objectivesDistribute in the labs research HPC codes for CFD in complex geometriesEnsure the training of usersManage the development roadmapShare databases of high-resolution simulationsPromote super-computing
Some facts8 French public labsAround 120 researchers and students2 PRACE proposals accepted over the recent yearsSeveral prizes related to !"##$!! codes: Bull-Joseph Fourier prize, IBM faculty award, ...
The codes
A massively-parallel finite-volume and finite-element 3D code for the simulation of
compressible turbulent reactive and two-phase flows.
A massively-parallel finite-volume 3D code for the simulation of turbulent reactive and two-
phase flows at low-Mach number.
The labsCNRS labs: CORIA, EM2C, I3M, LEGI, IMFT, LMAEPIC labs: CERFACS, IFP-EN
!"##$!!%%http://success.coria-cfd.fr
LMA
AVBP Strong scaling examples
14
65536
57344
49152
40960
32768
24576
16384
8192
0
equi
vale
nt p
erfo
rman
ce
6553657344491524096032768245761638481920cores
ANL INTREPID, Bluegene P(1)
PRACE/TGCC, CURIE, BullX(2)
GENCI/CINES, JADE, SGI Altix ICE(3)
PRACE/JSC, JUQUEEN, Bluegene Q(4)
INCITE/ARNL, INTREPID, Bluegene P(5)
HLRS/PRACE, HERMIT, CRAY XE6 (5)
Ideal
(1) 93M Tetrahedra case - 1 step Chemistry - 2 tasks per node(2) 200M Tetrahedra case - 2 step Chemistry(3) 29M Tetrahedra case - 7 step Chemistry(4) 75M Tetrahedra case - No chemistry - 64 tasks per nod(5) 75M Tetrahedra case - No chemistry - 4 tasks per node
1 billion cells (BG/Q)
V. Moureau, P. Domingo, L. Vervisch, D. VeynanteDNS analysis of a Re = 40,000 swirl burner
0 2048 4096 6144 8192 10240 12288
Number of cores
0 0
2048 2048
4096 4096
6144 6144
8192 8192
10240 10240
12288 12288
Scal
e-up
linearYALES2
YALES2 scale-up on Babel @ IDRIS (Blue Gene/P)Up to 12288 cores and 2.6 billion tetrahedrons
2.6B tets
878M tets329M tets
41M tets
14M tets
2008
YALES2 solver(CORIA Rouen)
Juelich Workshop 2010
ORGANIZATION OF THE FRENCH NUMERICAL COMBUSTION COMMUNITY
16
SNECMA SNECMA DMSTURBOMECA
RHODIAALSTOM
AIR LIQUIDEHERAKLES
AIRBUSSIEMENSANSALDO
17
!"#$%&'()*()+,#-./#0)-1(23'(4567(89&:;<
CER
FAC
SIn
stitu
t Fra
nçai
s du
Pétro
leG6S'564'58+*Q!"#$%&$'()'(*+,-./!0%&/'(+1,
2"3-%&-+1456)+%7658*,!/792%&"65*+8))+,$:"%&"(18;<,$:%281=<'>+1
$?2@$2"0A/!A.@2/0:18>B%76(
"'=+)*
-'=+:*+5*Q
C@2-"0%&D8))65';<+,$:/E."2-0%&E'5=+*,C@2-"0%&D+51'1,C7C%&E'5=+6(F,
/2@0:G$7C0
0GC$."[email protected]!2"2@C#2//0/!
0!/%G!H:!A2I0J%A2%#/0@-2
K
-'=+
@++=*
-'LM(4+5%;'LM618+*%N-'LM(O1P%;+14+5*Q%-20R%E65;+)'16R%I2@-!=>?=@!
A4+,)--'B
C$0@#./A
=>@!C!D@=>E(@F(GFCH(G=@I(@I!(JD4(@FF1GFCH=>E(6F@I(=>(7C4?!(4>K(=>?=@!(7CFL!?@D
MN
!"#$%&'()$*+,-.*/$01#2)&34$#5$67*6!+$85$9:;<
THIS ORGANIZATION ALLOWS TO DO OTHER THINGS IN OTHER FIELDS:
19
S. Mendez, F. NicoudYALES 2
Université de Montpellier
TWO TYPICAL PRACE EXAMPLES
• PRECCINSTA: A laboratory burner using a Turbomeca injection system. Developed in EC project PRECCINSTA and used as the classical benchmark for LES codes in the world. Most important problems in the GT industry can be reproduced on the PRECCINSTA burner. CH4 + Air at 1 bar.
• MASCOTTE: a high pressure experiment corresponding to rocket combustion (60 to 100 bars). Very powerful flames: H2/O2. Very extreme conditions (flames of the order or 10 microns)
20
21
PRECCINTSA (2005): THE FIRST LES OF SWIRLED BURNERS
3 millions cells
Roux, Lartigue, Poinsot, Meier and Bérat Comb and Flame 141, 2005
22
COLD FLOW - PRECCINSTA
23
All LES are compared with measurements DLR (LDA):• Velocity profiles are compared at five stations along the burner. • Comparison for axial, tangential and radial velocities (mean and RMS)
X= 5 mm
X= 1,5 mm
X= 25 mm
X= 15 mm
X= 35 mm
FROM SWIRLER
TO OUTLET
U
W
Comparison of mean velocity fields
24
-40
-20
0
20
40
Dis
tance
fro
m a
xis
[mm
]
151050
-40
-20
0
20
40
151050
-40
-20
0
20
40
151050
-40
-20
0
20
40
151050
-40
-20
0
20
40
151050
x=1.5 mm x=5 mm x=15 mm x=25 mm x=35 mm
Uxp Profiles: Red solid: CDP - Black solid: TTGC - Black dotted: TTGC_SSS - Circles: Exp.RMS velocity profiles: Stanford code (red), CERFACS code (black)
and DLR experiments (symbols)
PRECCINSTA:•First computed with LES in 2004: 1
Million cells (AVBP)
•Repeated in 2007 with 10 Mcells (AVBP)
•In 2009, repeated with 100 Mcells, 500 Mcells (YALES)
•In 2010 and 2011, on PRACE machines: JUGENE: 2 billion and CURIE 12 billion cells (YALES)
25
3T
DLR PRECCINSTA BURNER, Experiments: W. Meier et al., Combust. Flame, 150(1/2):2–26, 2007
=84>%?84%'#5$#@$(#1'&)4$>485A$3B)$C$&183)18#5$85$D,+**67/!.$EF1$G#>1)%>H$*I,6.J
3U
More details in:Roux et al, Combustion and Flame (2005)Moureau et al, Journal of Computational Physics (2007) (2 papers)Galpin et al, Combustion and Flame (2008)Moureau et al, Combustion and Flame (2011)Franzelli et al, Combustion and Flame (2011)
IS THIS ENOUGH ?:•At 20 billion cells, we are almost reaching
what we need in terms of resolution for the large flow structures but near walls and within the flame front, this is not enough
• This is a single burner at atmospheric pressure with gaseous fuel (no liquid phase). Real combustors have 16 to 24 burners, working at 20 to 100 bars, with liquid fuels.
•We still miss a 10^6 factor in power28
ROCKET COMBUSTION:
• PRACE Project by EM2C and CERFACS
• ROCKET COMBUSTION IS A VERY SPECIFIC AND NARROW FIELD: IT IS ALSO DIFFICULT SINCE FLUIDS IN THESE ENGINES ARE IN SUPERCRITICAL CONDITIONS (100 bars)
• CERFACS AND EM2C HAVE DEVELOPED SUPERCRITICAL CAPABILITIES IN AVBP IN THE LAST FIVE YEARS
• PRACE WORK PERFORMED IN 2012: ACOUSTIC EXCITATION OF SUPERCRITICAL FLAMES IN THE CONFIGURATION MASCOTTE OF ONERA
29
! T. Schmitt, H. Layal, M. Boileau, S. Ducruix, S.Candel (EM2C), ! A. Ruiz, G. Staffelbach, B. Cuenot and T. Poinsot (CERFACS)Large-Eddy Simulation of high-frequency instabilities under transcritical conditionsObjective: Observe and understand the flame behavior in transcritical flows of an experimental setup submitted to artificial acoustic transverse perturbation.
!
CEA - CERFACS 30
8.5M hours at TGCC
CEA - CERFACS
WITH FORCING:
31
WHAT IS GOING ON IN THE USA IN THE FUELD OF COMBUSTION AND HPC ?
32
In the last 15 years: the ASCI projects. Most ASCI projects in the USA were actually combustion projects: gas turbines, scramjets, rockets, fires, galaxies...
Now: the CODESIGN Centers. Example: Sandia CODESIGN Center. Main idea: «Compute combustors before you build them»
FOR THE FUTURE IN EUROPE
• EESI and PRACE have opened the path for a fast HPC evolution in Europe.
• Machines are already here (Tier1 and Tier0).
• The main questions today are:
(1) the codes: see GIS SUCCESS lead by CNRS
(2) the money: from industry and the EC
(3) the users !
34
Money: the ERC (European Research Council) INTECOCIS advanced grant at Institut de
Mécanique des Fluides and CERFACS 2013-2018 (intecocis.inp-toulouse.fr)
35
• Five year, 2.5 Meuros project on HPC tools for combustion instabilities. Coordinator: IMFT
• Ten researchers on numerical combustion, 5 years
• Collaboration with GENCI, SAFRAN, ANSALDO, SIEMENS
USERS / FORMATION: ‘ignored’ problem ?
36
The last ten years using AVBP at CERFACS and the las t two years suppor t ing an exp los ive development of AVBP and YALES at SAFRAN have shown that building an efficient HPC CFD team in combustion required multiple experts:
HPC experts- develop the code- port it on new machines- do the first demos
Combustion experts- know the physics- write the models- use the code but do not address HPC
CERFACS
37
HPC experts- develop the code- port it on PRACE/INCITE machines- do the first demos
Combustion experts- know the physics- write the models- use the code but do not address HPC
CERFACS
Industry experts:- know combustors- know ‘old style’ CFD and learn HPC
Students:- know nothing
Sous traitants:- know a little bit on CFD and the combustors
Combustion labs:- know theory and experiments- dont know the code details or HPC
38
HPC experts- develop the code- port it on PRACE machines- do the first demos
Combustion experts- know the physics- write the models- use the code but do not address HPC
CERFACS today
Industry experts:- know the engine- know ‘old style’ CFD and learn HPC
Students:- know nothing
Sous traitants:- know a little bit on CFD and the engine
Combustion labs:- know theory and experiments- dont know the code details or HPC
Interface / formation- interfaces to access the code- formation to learn the basics
INTERFACES: DISTRIBUTING HPC CODES TO INDUSTRY
39
C’est bientôt
fini?Mmmh...
La taille de ce tube est le vrai point critique aujourd’hui
A . Dauptain Copyright
DISTRIBUTING LES CODES:
40
00
• Codes CANNOT be used in industry without additional interfaces (it works with labs but not with industry).
• These interfaces must integrate a very large set of information coming from industrial needs
• The time required to write these interfaces is VERY large: the codes MUST be modified and simplified to adjust to the interfaces
INTERFACES: THE C3S EXAMPLE
• In the last 4 years, CERFACS has developed an interface called C3S for AVBP users in labs and industry
41
00
• C3S installed and working at CERFACS, IMFT, EM2C, IFPEN, SNECMA, Villaroche, SNECMA Vernon, TURBOMECA Bordes, etc
EXAMPLE :
42
BUT THIS IS NOT ENOUGH!CREATING ONE INTERFACE FOR ONE CODE IS NOT
ENOUGH:
• Keeping up with AVBP developments in C3S is difficult:-> must coordinate code and interface evolutions. The interface MUST evolve with the code (or vice versa ?)
• If you write a piece of code, you should also, at the same time, write the interface
!In 2011: a new initiative -> C3Sm (A. Dauptain)
43
C3SM • Is an engine... which writes interfaces:
!developed by CERFACS
!used by all french groups developing AVBP or any other code linked to AVBP, to automatically produce and update the interface and the documentation: YALES, PRISSMA (radiation), AVTP (heat transfer), N3S (RANS), Coolant (SAFRAN dedicated software for cooling)...
44
!"#$%&'&(&)*#+,-"+./'%-0)-/0'-&"1#2345
2345
2&"'*".#657 5&'*.% 895
2*#$%&'&(&)*#%*"+#$&//-:)*#1• #)0#(%;0'-&"#+,895#$0%#)*/#+;<*)&$$*.%/#+*#(&+*• #)0#(%;0'-&"#+*##=&'*.%/#895#$*%>&%=0"'/#*'#(&=$0'-:)*/
?<0"'#1#2&"'%-:.'-&"#=&"&)-'@-A.*
B;<*)&$$*.%/#+*#(&+*/#/(-*"'-CA.*/
!'-)-/0'*.%/#+*#(&+*/#/(-*"'-CA.*/
2%;0'*.%/#+,-"'*%>0(*/
7*#(%;0'*.%#+,-"'*%>0(*/#*/'#0.#=-)-*.#+.#+-/$&/-'->D#-)#*/'#-"(&"'&.%"0:)*D#*'#(%;*#+*#),0''*"'*E
!"#$% &'%
?$%F/#1#2&"'%-:.'-&"#%;$0%'-*
B;<*)&$$*.%/#+*#(&+*/#/(-*"'-CA.*/
!'-)-/0'*.%/#+*#(&+*/#/(-*"'-CA.*/
2%;0'*.%/#+,-"'*%>0(*/
7*/#+;<*)&$$*.%/#*'#.'-)-/0'*.%/#$*.<*"'#>0-%*#;<&).*%#)*.%#&.'-)#(@*G#*.HD#/0"/#$0//*%#$0%#),-"'*%=;+-0-%*#+.#(%;0'*.%#
+,-"'*%>0(*E
2345
&()*"&%
WHAT ABOUT FORMATION ?
48
• PhDs are of course a good solution:
!EM2C, CORIA, IMFT, IFP, CERFACS produce more than 20 PhD per year. They go to industry and they know HPC.
!But this is not fast enough
• Systematic courses on HPC/CFD/Combustion are required: in 2011, formation cycles at CERFACS
49
!"#$%&'()*#$+,-./*0-%*1(%2$*3..4*5&#")(6-7*"/&72*89:;*
!"#!$%&'(&)*+%,"!!%,#-%./+01%,"!,%
<-.$*'-"=)&72*"/&72*>=$7;81?*%!2#!3%&'(&)*+%,"!!%!4#,"%5.67.+8%,"!,%!9#!:%576*%,"!,%
8.@(7'$.*#$/,*2$7$%(6-7*"/&72*<3!A8BC* ,-#,2%5.67.+8%,"!,%
D(+(*(//&#&)(6-7*"/&72*>=$7;81?*9"%5.67.+8#!;(%<*)+7.+8%
,"!,%
;%-2%(##&72*&7*E-%+%(7*FG*** !9#!$%=.+'>%,"!,%
<-#="+$%*(%',&+$'+"%$/*(7.*-=6#&/(6-7* ,,%=.+'>%,"!,%
;(%())$)*=%-2%(##&72*+--)/* ,9#%,$%./+01%,"!,%
D-H7/'()&72*')&#(+$*/'$7(%&-/*H&+,*+,$*H$(+,$%I+4=&72*J(/$.*#$+,-.-)-24*D5<1K?*
,:#,-%./+01%,"!,%
5-)"6-7*#$+,-./*0-%*-=6#&/(6-7*=%-J)$#/* 3#!!%=.8%,"!,%
E"7.(#$7+()/*+-*"7.$%/+(7.*(7.*(7()4L$*,&2,*M.$)&+4*<ED*/&#")(6-7/*
,!#,:%=.8%,"!,%
<-.$*'-"=)&72*"/&72*>85K5* %9"#9!%=.8#!;(%576*%,"!,%
!"#$%&'()*#$+,-./*0-%*+,$*$)$'+%-#(27$6'*=%-J)$#*"/&72*<35<*
-#4%%576*%,"!,%
5-)"6-7*#$+,-./*0-%*)&7$(%*/4/+$#/* !!#!,%576*%,"!,%
*K7+%-."'6-7*+-*.(+(*(//&#&)(6-7* !4#!3%576*%,"!,%
5-)"6-7/*-0*=(%6()*.&N$%$76()*$O"(6-7/P*(==)&'(6-7/*+-*+,$*H(@$*=%-=(2(6-7*=%-J)$#*
,:%#%,-%%576*%,"!,%
8.@(7'$.*()2-%&+,#/*0-%*.(+(*(//&#&)(6-7* ,2#,3%576*%,"!,%
!"#$%%&&&'()*+,(-'+*%.*,/0/01%
%%%%(+.0606?@'*+<.';A<+%
B6% ,"!!C% DEFGHDI% 060J.(*;% .% (+.0606?%/+&?+.=%<&+%'&=/7(*+%;'0*6'*%(.+?*J6?%;(7K*6(;C% L>M;C% *6?06**+;% .6K%+*;*.+'>*+;C% 06% .'.K*=0'% .6K% .//10*K%+*;*.+'>A%%N>*% (+.0606?% ;*;;0&6;% .+*% (.7?>(% )8%DEFGHDI%+*;*.+'>*+;%.6K%*6?06**+;A%%L+.'J'.1% (+.0606?% ;*;;0&6;%K*K0'.(*K% (&%(>*% '&K*;% K*O*1&/*K% &+% '&#K*O*1&/*K%)8% DEFGHDI% .+*% /.+(% &<% (>*% /+&?+.=A%DEFGHDI% 0;% .7(>&+0P*K% (&% ?0O*% (+.0606?%;*;;0&6;% Q0(>06% (>*% B6K0O0K7.1% +0?>(% (&%(+.0606?%RMBGSA%
50
www.cerfacs.fr/elearning/combustion/
51
CONCLUSIONS• DANS 10 ANS, LES GROUPES QUI AURONT CONTINUE A
DEVELOPPER DES CODES DE PETITE TAILLE EN COMBUSTION, AURONT BEAUCOUP DE MAL A PUBLIER MAIS L’EFFORT POUR CONSTRUIRE DES GRANDS CODES EST ENORME
• SUCCESS: REPONSE ‘LOGICIEL’ DES COMBUSTIONNISTES FRANCAIS. COUVRE LES LABOS ET LES INDUSTRIELS
• L’ARRIVEE DE GENCI A PERMIS A LA COMMUNAUTE ‘COMBUSTION NUMERIQUE’ DE DISPOSER DES MOYENS NÉCESSAIRES POUR DEMONTRER SON SAVOIR FAIRE
• PRACE ET INCITE COMPLETENT CE DISPOSITIF
• LA COMBUSTION EST UN EXEMPLE OU CES OUTILS HPC SONT UTILISES AUJOURD’HUI PAR L’INDUSTRIE QUI SUPPORTE LEUR DEVELOPPEMENT DE FACON TRES FORTE
52