direct numerical simulation of turbulent boundary layers at high reynolds numbers

8/6/2019 Direct Numerical Simulation of Turbulent Boundary Layers at High Reynolds Numbers

http://slidepdf.com/reader/full/direct-numerical-simulation-of-turbulent-boundary-layers-at-high-reynolds-numbers 1/21

Direct Numerical Simulation of Direct Numerical Simulation of Turbulent Boundary Layers atTurbulent Boundary Layers at

High Reynolds Numbers.High Reynolds Numbers.

G. Borrell, J.A. Sillero, J. JiménezG. Borrell, J.A. Sillero, J. Jiménez

XXIII ParCFD. May 20th. BarcelonaXXIII ParCFD. May 20th. Barcelona



DNS as a tool to obtainDNS as a tool to obtain

correctcorrect

datasets for walldatasets for wall

bounded turbulent flowbounded turbulent flow researchresearch..



DNS of a TBLDNS of a TBL

● Standard for future turbulence research.Standard for future turbulence research.

● DNS data is used for decades.DNS data is used for decades.

● Scalability, scalability and scalability.Scalability, scalability and scalability.

● 32K cores of Blue Gene/P.32K cores of Blue Gene/P.

● Huge amount of resources.Huge amount of resources.

● 73M CPUh from DoE INCITE grant73M CPUh from DoE INCITE grant

● 40M CPUh from PRACE project40M CPUh from PRACE project



Spectral / spectral likeSpectral / spectral like

High orderHigh order

● Better resolutionBetter resolution

●

Less pointsLess points

● Coupled Coupled

Low orderLow order

● Less computations perLess computations perpoint.point.

● Better flops countBetter flops count● Less coupledLess coupled

●

Tightier control on the numerical errors.Tightier control on the numerical errors.● Linear analysis suggests that high orderLinear analysis suggests that high order

methods are effectivelymethods are effectively faster.faster.



● Spectral and spectral-like operators need contiguous arrays.Spectral and spectral-like operators need contiguous arrays.

● Planes for z and y-wise operations.Planes for z and y-wise operations.

● Pencils for x-wise operations.Pencils for x-wise operations.

● Variables have to be transposed.Variables have to be transposed.

● Results are transposed back to planes.Results are transposed back to planes.

●

Not the most practical domain decompositionNot the most practical domain decomposition● 2 or 3 planes fit on a BG/P2 or 3 planes fit on a BG/P nodenode

● MPI+OpenMP is a need, not a feature.MPI+OpenMP is a need, not a feature.

● Global transposeGlobal transpose isis the bottleneck of the algorithm.the bottleneck of the algorithm.



The global transposeThe global transpose

● UsesUses alltoallvalltoallv over the torusover the torusnetworknetwork

● Great performance and scalabilityGreat performance and scalability

● Data is casted to float to saveData is casted to float to save

bandwidth.bandwidth.

● Planes are realigned to increasePlanes are realigned to increasethe message sizethe message size

● Casting and transposes useCasting and transposes useOpenMPOpenMP

●

Transposes use blocking as wellTransposes use blocking as well● Optimized for cache performanceOptimized for cache performance

Latency Analysis



ScalabilityScalability

Threads Efficiency1 1

2 0,984

4 0,923

OpenMP

MPI



Running the simulationRunning the simulation



:-(:-(



Dual boundary layerDual boundary layer

● BL1 runs at a lower resolutionBL1 runs at a lower resolution

● Generates correct inflow for BL2Generates correct inflow for BL2

● <10% of the overall computational cost. 1/3 of the volume.<10% of the overall computational cost. 1/3 of the volume.

● 2 MPI groups2 MPI groups

● 3 Communicators3 Communicators



:-):-)



:'-(:'-(



Node mapping on the BG/PNode mapping on the BG/P

● No previous experience with torus networkNo previous experience with torus network

● Standard [XYZ,T] topology not goodStandard [XYZ,T] topology not goodenough.enough.

● [8,8,8] block for BL1[8,8,8] block for BL1

● The rest of the nodes are populated inThe rest of the nodes are populated in[XYZ] order.[XYZ] order.

● BL2 is not affectedBL2 is not affected

Topology Nodes BL1 BL2

[8,8,8] 512 22,77 -

[32,32,8] 8192 160,22 85,44

[32,32,8]' 8192 79,59 86,09



Parallel I/OParallel I/O

● These simulations will generate ~400 TiBThese simulations will generate ~400 TiB

● I/O is a serious bottleneck if data is to be writtenI/O is a serious bottleneck if data is to be writtenin serialin serial

● ChoicesChoices● POSIX calls to the file system (TACC)POSIX calls to the file system (TACC)

● Sionlib (Jülich)Sionlib (Jülich)

●

Parallel HDF5Parallel HDF5● Similar performance, HDF5 is more convenient.Similar performance, HDF5 is more convenient.

● 8192 nodes. 26GiB/s read, 2GiB/s write.8192 nodes. 26GiB/s read, 2GiB/s write.



Some resultsSome results



Conclusions and future workConclusions and future work

● Two different simulations are runningTwo different simulations are running

● Intrepid (ALCF, Argonne National Laboratory)Intrepid (ALCF, Argonne National Laboratory)

● Jugene (Jülich Forschungszentrum) Jugene (Jülich Forschungszentrum)

● Producing resultsProducing results

● No further tuning expectedNo further tuning expected

● Limited speedups <5%Limited speedups <5%

● Post processing and visualisation has justPost processing and visualisation has juststarted.started.



Thanks for your attention.Thanks for your attention.

[email protected]@torroja.dmt.upm.es

mailto:[email protected]



direct numerical simulation of turbulent boundary layers at high reynolds numbers

Documents