Industrial application of Petascale Computations. 3D visualization of

Petascale Modeling.

Dr. Miroslav Iliev, Dr. Valentin Pavlov, Dimitar Dimitrov, Veselin Slavchev

Rila Solutions EAD, Sofia 1113, Acad. G. Bonchev str., bl. 27

Supercomputing - introduction

• Solving problems with great scientific or sociological importance and impact

• Using computers, providing top-of-the line performance for information processing

• Driving the forefront of intellectual and technological development

• Providing the means for effective and efficient scientific and industrial development

Supercomputing – architecture evolution

• Monolitic high-performance custom architecture computers (1960’s-1970’s)

• Multi-CPU high-performance custom architecture computers (1970’s-1980’s)

• Massive-parallel computing based on custom and general-purpose nodes (1990’s-present)

• Supercomputer networks and grids

Supercomputing – architecture example

Components of a massively parallel supercomputer - IBM Blue Gene/P

Supercomputing – performance evolution

• 1 MFLOP/s - CDC6600 in 1964• 8.126 PFLOP/s - Fujitsu K computer in 2011• Exponential performance increase, matching

closely Moore’s law

Supercomputing – application overview

• fast, accurate and detailed process simulations

• provide "alive" models and detailed view over their process development

• knowledge validation by comparing simulation results with data from same type of real events

• fast/real-time processing of high-volume data streams

Industrial applications and business benefits

• Operational cost reduction

– validation and tuning of business or technological model accuracy

– simulate many what-if scenarios in a fast and accurate manner, enabling model parameter tuning toward desirable business goal

Operational cost reduction – example• Project: INCITE project – 3D simulations of a turbulent CO/H2 jet flame• Paper: Direct numerical simulation of turbulent combustion: fundamental

insights towards predictive models - Evatt R Hawkes, Ramanan Sankaran, James C Sutherland and Jacqueline H Chen

• Supported by: Division of Chemical Sciences, Geosciences and Biosciences, Office of Basic Energy Sciences, DOE, SciDAC

HO2 dissipation OH dissipation

100 million grid run

Industrial applications and business benefits

• Research & development cycle acceleration

– detailed and accurate results on properties and behavior of product or process before prototype implementation

– improved effectiveness and efficiency of business activities and abilities for fast adaptation to quick market changes

R&D cycle acceleration - example• Project: Research - Turboinštitut• Paper: Numerical Prediction of Efficiency, Cavitation and Unsteady

Phenomena in Water Turbines - Dragica Jost, Andrej Lipej, Peter Meznar• Supported by: Turboinštitut - Slovenia

Rotating cavitation rope in the draft tube, numerical simulation(left) and experiment on the test rig (right)

Industrial applications and business benefits

• Disaster mitigation and response analysis

– simulate events which cannot be controlled and are neither desirable nor economically sane to recreate in real life

– allows improving our understanding of the causes for disaster events and their development in time

– devise measures to prevent disasters from happening and establish adequate response strategies

Disaster mitigation and response - example

• Project: M8• Paper: Scalable Earthquake Simulation on Petascale Supercomputers - Y.

Cui et al.• Supported by: NSF

Peak horizontal ground velocity and seismograms

Industrial applications and business benefits

• Ecology impact optimization

– provides window of opportunities for detailed simulation of industrial processes and their impact on the environment

– possibility for process optimization in order to limit undesirable output while maintaining efficiency

Ecology impact - example

• Project: SPRINTAS – Fukushima NPP pollution estimation• Paper: A numerical simulation of global transport of atmospheric particles

emitted from the Fukushima Daiichi Nuclear Power Plant - Toshihiko Takemura et al.

а OO:OOUTC 18M A R 20 11 BON 70N г- -...- ...-с :.._ . ." ., . " . .. . 60N 50N 40N :30N 20N 10N

1е-09 1е-08 1е-07

ь OO:OOUTC 21 MAR20 11 BON 70N 60N 50N 40N :30N 20N 10N

1е-09 1е-08 1е-07

с BON 70N 60N 50N 40N :30N 20N 10N

120Е 180 120W бОW

OO:OOUTC 24M A R 20 11

о 60Е S PR IN TA R S

1е-09 1е-08 1е-07 1е-06 1е-05 0.0001 0.001 0.01 0.1 1

Near-surface concentration of particles from Fukushima Daiichi NPP

Industrial applications and business benefits

• Health and safety analysis

– detailed simulations of processes involving people and living organisms without putting anyone in a risky situation

– modeling of events and interactions involving hazardous, toxic, virulent or otherwise dangerous materials and substances

– fast/realtime analysis of time- and decision-critical data streams and repositories

Health and safety analysis - example

• Project: Automotive crash simulation• Supported by: AUDI AG (Volkswagen Group)

Visualization of simulation data streams

• Data presentation – transformation of the simulation data in a meaningful and comprehensible form which will allow him/her to understand the results and take decision

• Focus – creating meaningful perceptional model using one or several senses (vision, hearing, touch, vestibular sense, etc.)

Visualization of simulation data streams

• Typical data presentation

– based on vision (covers ~80% of human perceptional basis)

– using more dimensions as the complexity of the chosen data presentation increases

– utilizing standard graphics display technology– easily augmented to true visual 3D perception

experience through stereoscopic image pairs

Visualization of simulation data streams

• Challenges– exponential increase of computational power lead

to similar increase in the amount of simulation output data. Contemporary Petascale computing generates data from tens to thousands of terabytes.

– transforming extremely large data streams into acceptable data presentation is a serious computational task comparable to the actual Petascale simulations.

Visualization of simulation data streams

• Solution: use the existing supercomputing infrastructure to tackle the visualization task

– ensures top-of-the-class computing capabilities

– solves the task efficiently with minimal amount of data transfer to external systems

VisIt – Petascale capable visualization system

• Open Source multi-platform massively-parallel visualization framework

• Highly scalable - runs on computing clusters, including supercomputers and supercomputing grids

• Workflow enables easy parallel task execution

• Has been used to visualize data sets with trillion elements (8 trillion in 2009)

VisIt – workflow overview

• Establish access to the initial data sources

• Define plots over the data sources

• Define operators that operate over the defined plots

• Definition of custom and derivative data analysis presentation

• Set additional visualization and presentation options

• Define presentation output path

VisIt – work environment example

VisIt - architecture

• Multi-component architecture where each component can have multiple instances on different computing hosts

• Components communicate with each other through a network.

• Could work independently or integrated with the simulation software

VisIt - architecture

BGSC –global Petascale initiative partner

• Bulgarian Supercomputing Centre (BGSC) works with and provides access to a supercomputer IBM Blue Gene/P, consisting of 2048 computing nodes (8192 PowerPC cores @ 850 MHz, 4TB RAM)

• Connection between the computing nodes and the rest of the infrastructure: 16 channels x 10 Gb/s

• Disk storage capacity: 12 TB

• Front-end OS: SLES10, externally accessible through SSH

• Performance rating: 27.85 Tflops

• Energy efficiency: 371.67 Mflops/W

BGSC – software packages• bio-informatics (GROMACS, GAMESS, NAMD, CP2K,

LAMPPS, OpenAtom, DL_PLOY, etc.)

• fluid and thermo-dynamics (Code_Saturne, Syrthes, etc.)

• earth sciences (SPECFEM3D)

• material sciences (PLANETICS)

• mathematics (PETSc, ParMETIS, LAPACK, ScaLAPACK, SuperLU, FFTW, HYPRE, Trilinos, ParFE, GotoBLAS, etc.)

• Visualization (VisIt, Salome-Platform)

• Custom end-users code and packages, related to user’s domain of expertise.

Conclusion• Many of challenges in the Industry may be solved by

simulations and modeling.

• Supercomputers are particularly suitable for modeling, due to the low-cost of operation and large processing power they offer.

• The Important task of visualization needs same range of scalability as the simulations.

• BGSC has the necessary infrastructure and capabilities in order to meet the demand of forward-thinking and innovative industry members. The Center maintains partnerships and participates in projects with similar facilities around the world in order to expands its capabilities even further.