modelling core-collapse supernovae · motivation: physics supernovae serve as “laboratories”...

1

SCICOMP 10© 2004 Eric S. Myra 1

Modelling Core-Collapse Supernovae:

The Development of Parallel Iterative Solvers for Large-Scale Simulation of Radiation Hydrodynamics

Eric S. MyraDEPT. OF PHYSICS & ASTRONOMY

STATE UNIVERSITY OF NEW YORK AT STONY BROOK

ScicomP 10

9-13 August 2004

Austin, TX


Ground to be covered…

• Motivation (why is the problem worth working on)

• A glimpse of the supernova scenario

• A glimpse of how we turn physics into a mathematical model

• A glimpse at parallel solvers and their performance

• Computational bottlenecks and challenges of the next stages

2


• Doug Swesty (SUNY at Stony Brook)

• Jim Lattimer (SUNY at Stony Brook)• Dennis Smolarski (Santa Clara Univ, Univ of Illinois at U-C)

• Polly Baker (Indiana Univ)

• Ed Bachta (Indiana Univ)

Collaborators

Terascale Supernova InitiativeTony Mezzacappa, P.I.


The Storyline of the Project:

…into this!How to turn this…

3


Motivation: Physics

Supernovae serve as “laboratories” for fundamental physics.

• Strong Interaction:– serve as constraints on nuclear free-energy parameters through

neutron stars mass ranges

• incompressibility, symmetry parameters

• Weak Interaction:– reaction rates

– neutrino masses and mixing

• Gravitational Interaction:

– expected to be visible to gravitational wave detectors


Motivation: The Nat’l. Acad. Sci. CPU’s Eleven Big Questions

1. What is dark matter?2. What is the nature of dark energy?3. How did the universe begin?

4. Did Einstein have the last word on gravity?5. What are the masses of neutrinos and how have they shaped the

evolution of the universe?6. How do cosmic accelerators work, and what are they accelerating?7. Are protons unstable?

8. What are the new states of matter at exceedingly high density and temperature?

9. Are there additional space-time dimensions?

10. How were the elements from iron to uranium made?11. Is a new theory of matter and light needed at the highest energies?

4


Supernovae are needed to explain the universe’s structure and make-up.

• The “we are star stuff” argument.– Supernovae are the primary source of heavy elements.

• Supernovae are among the most energetic explosions in the cosmos.

• Affect the ISM (heating and compression) out to 100 light years.

• SN1987A commenced the science of extra-solar-system neutrino astronomy.– some day: to be repeated by not-too-close observation – no neutrino-radiation poisoning, please!

• Supernovae are a primary prospective source for gravitational-wave astronomy.

Motivation: Astronomy


Motivation: Computation and Modelling

Supernova modelling is a challenging problem algorithmically.

• Diverse branches of physics:

– relativistic hydrodynamics; non-equilibrium radiation

– nuclear and weak-interaction physics

• Diverse coupling between branches:

– matter-radiation coupling

– multi-phase statistical mechanics

• Diverse timescales (strong, weak, hydrodynamic, diffusive, etc.)

• Diverse physical regimes (e.g., densities can vary by 15 orders)

• A 1% problem!

%1erg10erg10

B.E.starneutron-protoenergyexplosionSN

53

51≈≈

5


Motivation: Computation and Modelling

Supernova modelling is a challenging problem computationally.

Accurate modelling requires: • multi-dimensional computations

• careful energy bookkeeping

• many nested iterative loops• scalable pre-conditioners and sparse linear solvers for large

parallel computers

• a tera- or petaFLOP-scale solution

The best methods (for the physics and the computation) require the fastest computers available.


Initial Conditions: The Pre-Collapse of Massive Stars

• Evolution by fusing lighter elements to form heavier elements.

• High mass stars produce elements up to iron.

• Star is like an onion, with layers of successively heavier elements.

• Each burning stage progresses more rapidly.

• When the iron core mass grows to about 1.2−1.4 Msunthe core collapses under its own weight

H → He

He → C,O

C→Ne, Mg

O→ S, Si

S,Si→ Fe

Fe

6


Core Collapse and Explosion: The Nickel Tour

Onset of collapseInverse decay and collapse runaway

Core bounce and rebound

Shock formation and propagation

The shock hits the neutrinosphere

Somehow…matter is ejected.

This is the tricky bit!

npe e +→+− ν


The strong force causes the core to bounce

• Nuclear matter goes through a series of phase transitions at about 1014 g/cm3

– pasta phases, bubble phases, etc.

• Above nuclear density, strong force is repulsive and dominant

• Collapse stops at center– soon after, stops everywhere else in sonic

region

• A shock wave forms on the boundary of subsonic and supersonic material

7


The shock starts to advance…

• The force of the core bounce starts the shock advancing.

• Supersonic material in the outer core…

• falls through the shock.• is dissociated (costing 8 MeV/baryon ∼ 2 x 1051 erg/0.1Msun).

• Once the shock is beyond the neutrinosphere, ’s freely stream.

• Dissociation and emission deplete the shock of energy.

• Need to deliver about 1051 ergs to the mantle for an explosion.

• This does not happen in realistic calculations.

“Prompt”shocks eventually stall!


R vs. t plots (Strong shocks vs. weak shocks)

A strong shock propagates.Matter is ejected.

A weak shock stalls. Matter falls back.

8


Neutrino-driven explosions: The “Wilson” Effect

Net ν heating in this region

Net ν cooling in this region

ν heating α ρ

ν cooling α ρ2

→ net heating beyond some radius

Does enough of this happen to make a difference?

Can one model it correctly?


Giving Up on Spherical Symmetry:How convection complicates the picture

Convection may occur in the proto-neutron star.

• may boost overall ν luminosity.

Convection may occur behind the shock.

• negative entropy gradient may form from ν heating.

• may boost shock radius.

9


Some Recent Evidence for Proto-Neutron Star Convection

• 2D run 1024 processors on seaborg at NERSC

• Visualization by

Ed Bachta, Pervasive Technology Labs, Indiana University

Velocity visualized with LEA showing flow and color showing magnitude

Time variant LEA composited with an entropy colormap


The Physical Model

• Matter (“heavy” nuclei, He nuclei, p, n, e-, e+)

– baryons interact through strong force, gravity, weak force, E&M– leptons interact through weak force, E&M, gravity

– matter modelled as a multi-phase quantum gas of arbitrary degeneracy

• Neutrino radiation (not in equilibrium)

– neutrinos interact through weak force only– modelled via the Boltzmann Transport Equation (BTE) or some

approximation thereof

10


The Mathematical Model

• Matter is tracked through Lagrangian (1d only) or Eulerianhydrodynamics (1-3d)

• Neutrinos are tracked via the BTE (2-6d)

– usually solved in some approximation

– must solve for the neutrino spectra for all species at all timesand places


Hydrodynamic Evolution of Matter (Newtonian)

CONSERVATION OF MASS

CONSERVATION OF MOMENTUM

GAS ENERGY

EQUATION

CONSERVATION OF

ELECTRIC CHARGE

Need an equation of state and neutrino distributions to close the system.

11


Radiative-Hydrodynamic Evolution of Neutrinos (O(v/c))

• An moment approximation to the Boltzmann Equation is used.

• Multi-spatial dimensions and one (spectral energy) momentum-space dimension

• 6 such equations at each point in phase space: 3 neutrino flavors (e, µ, τ) and their anti-neutrinos

COUPLING ACROSS

SPECIES

NON-LINEAR EQUATIONS

HIGHER MOMENTS (F,P) CLOSED WITH FLUX-LIMITED DIFFUSION APPROXIMATION


EQUATION OF STATE

EMISSIVITY

ABSORPTION OPACITY

DIFFUSION COEFFICIENT

SCATTERINGCOEFFICIENTS

PAIR EMISSIVITY

Microphysics

T = temperature

ρ = densityYe = electron fractionEe = neutrino energy density

Tabular EOS obtained viathermodynamicallyconsistent interpolationscheme; also maintains continuity in derivativesof EOS w.r.t. T & ρ

Bi-quintic hermite interpolationin T & ρ; linear interpolationin Ye

Neutrino microphysicscoefficients fromtri-linear interpolationscheme

12


Solution Scheme

Sparing many tedious details….

• Equations are differenced

• Material Hydro is solved explicitly (flea on the tail of the dog)– uses ZEUS-like staggered-mesh algorithm

• Radiation is solved implicitly (the meat of the problem)– CFL condition too restrictive on timestep if done explicitly


Forming the Difference Equations

• Differencing in time

– connects only consecutive timesteps

• Differencing in space

– connects only nearest neighbors (physics is local!)

• Differencing in momentum space

– everything couples to everything else

1+→ nn tt

1 1 +− iii

1

1

−

+

j

j

j

. . . 2 1 1 . . . ++− kkkk. . .

1

1

2...

−

+

+

k

k

k

k

In the zeroth moment of the BTE, only energies are present to couple.

In the full solution all energies and all angles couple.

13


The Matrices Are Sparse

DENSE BLOCKS OF COUPLED ENERGY GROUPS DOWN

MAIN DIAGONAL

OUTLIERS CORRESPOND TO

SPATIAL COUPLING

BLACK AREAS ARE WHERE THE MATRIX

HAS ENTRIES OF ZERO.

ENERGY COUPLING DECREASES FURTHER

OUT IN THE STAR (LOWER DENSITY OF

SCATTERING CENTERS)

MORE BANDS OF OUTLIERS IN HIGHER SPATIAL DIMENSIONS


The Matrices

• Matrices currently have about 2M unknowns

• Matrices are sparse (10 –5 to 10 –4 of elements occupied)

• Equations are non-linear (solution requires nested iterations)

– iterative solver for the equation– wrapped by Newton-Raphson iteration for non-linearity

• Done for each neutrino flavor (e, µ, τ)

• This process is repeated at each timestep.

– 10 5 – 10 6 timesteps per model

Ax = b

14


The Growing Magnitude of the Problem

Assumes a matrix-free formulation and no tables for microphysics

Scaling this approach to 3D requires 150 GB of memory.

Solving the 3D BTE with angles (16 x 16) requires 39 TB!

spatial points

2 x 256x256 x 40 x ( 6 + 8 ) x 8 = 600 MB

neutrino species bytes/variable

energy bins scratch vectors TOTAL MEMORYNEEDED (2D)

previous &current

timestep


Spatialdomain

decomposition

Global Problem Domain

Problem split up intoone spatial sub-domain

per processor

Adjacent processes in

cartesiantopology must

exchange ghost zones

when neededby algorithms

Parallel Implementation

15


Matrices arising from local spatial operators require only nearest-neighbor communicationto implement matrix-vector multiplies

Colors also indicate how the block banded matrix and the vectorsarising from difference eqns. are distributed among processors

Colors indicatethe ownership ofzones by differentprocesses

I,j-1 I+1,j-1

I+2,j

I+2,j+1

I+1,jI,j

I+1,j+1I,j+1

I-1,j

I-1,j+1

I,j+2 I+1,j+2

Decomposition


Sparse System Solution: Krylov Subspace Algorithms

• Idea:

– avoid direct solve for A x = b• Instead:

– minimize some function of x

– e.g. φ(x) = 1/2 x T A x - x T b– algorithms designed to reduce the residual: r = A x - b

• We are actively working on several versions of Krylov algorithms

– currently use BiCGSTAB algorithm

• does not require transpose• 2 matvecs and 4 inner products per iteration

– all methods benefit from preconditioning (parallelization issue)

16


Benefits of Krylov Subspace Algorithms

• Requires only:

– matrix-vector multiplies– vector additions (embarrassingly parallel)

– vector inner products (global reduction)

• Solution benefits if matrix is in operator form

– avoids storage of the matrix (big issue for the future)– no ordering defined

– preserves numerical symmetries identically– if mesh is partitioned and load balanced, then solver is too

– can be extended to block-structured AMR and unstructured grids.


N-K solution via Newton-BiCGSTAB

Currently using sparse parallel approximate inverse preconditioning (physics based) -- highly parallel

Transport Courant # ~ 10-20 means we can predict where to place elements of approximate inverse

Typically 2-3 Newton iterations

Typically 10-20 BiCGSTAB iterationsper Newton iteration

Implicit solvers have to be ultra-fast-- 105 - 106 timesteps to complete

simulation

Newton-Krylov Solver Performance (on seaborg)

17


For science to advance, better science must be enabled...

• Helped by larger grids

• Helped by computing more physics

• Helped by storing and post-processing more model-generated data

Where Do We Go From Here?

B L


Requirements for Larger Grids

• Require some combination of

– faster processors– faster interconnect

– lower interconnect latency

– more memory• Larger problems will show correspondingly better scaling

– nothing surprising here…

18


Requirements for Doing More Physics

• Generally the perfectly parallel part of the problem

– Benefits from faster processors, larger caches, more registers, more on-chip parallelism

• In this field, vector architectures are ancient history

– Real-world physics is branchy– Table look-ups more easily optimized for superscalar than for

vector architectures (at least for our tables)

B L


Requirements for More Data-Processing Capability

• Efficient parallel I/O essential

• Support for parallel formats (e.g. parallel HDF and netCDF important)• File Sizes:

– Large for both checkpointing and visualization dumps– Currently ~ 1 GB– Near Future ~ 10s of GB

• File Abundances:– Many (typically 1000’s) of files from a single batch job

• File Access Frequency:– Low (often only once per run)– post-processed/analyzed on non-ultrascale platform

19


Conclusions

• State-of-the-art computers are lending new insight to a longstanding problem.

• Mathematicians and computational scientists have the opportunityto try their new methods on “real” problems.

• Hard problems are good. You learn much more from a hard problem than an easy one.

• We’re looking forward to new generations of technology!


Questions?

modelling core-collapse supernovae · motivation: physics supernovae serve as “laboratories”...

Documents