senr/nrpy: a next-generation, dynamical reference metric...
TRANSCRIPT
SENR/NRPy: A Next-Generation, Dynamical SENR/NRPy: A Next-Generation, Dynamical Reference Metric Numerical Relativity CodeReference Metric Numerical Relativity Code
Zachariah B. EtienneZachariah B. EtienneIan RuchlinIan Ruchlin
in collaboration within collaboration with
Thomas W. BaumgarteThomas W. Baumgarte
SENR/NRPy: Code OverviewSENR/NRPy: Code Overview
● NRPy: Python+sympy code generation for NRNRPy: Python+sympy code generation for NR● Similar to Kranc, but with no Mathematica!Similar to Kranc, but with no Mathematica!
– Equations Equations at your fingertipsat your fingertips, even on HPC systems!, even on HPC systems!– Input: Einstein notation + simple syntax Python codeInput: Einstein notation + simple syntax Python code– Output: Efficient, compiler-vectorizable C code (AVX)Output: Efficient, compiler-vectorizable C code (AVX)
● SENR: Simple, Efficient Numerical Relativity codeSENR: Simple, Efficient Numerical Relativity code● Contains NRPy wrappers, diagnostics, MoL, BCs for Contains NRPy wrappers, diagnostics, MoL, BCs for
solving BSSN equations in arbitrary coord systemssolving BSSN equations in arbitrary coord systems– Log-Spherical Polar, Cylindrical, Cartesian, Bispherical-likeLog-Spherical Polar, Cylindrical, Cartesian, Bispherical-like
https://tinyurl.com/senrcode
SENR/NRPy: MotivationSENR/NRPy: Motivation
● SENR/NRPy = Simple, efficient, open (BSD-licensed, SENR/NRPy = Simple, efficient, open (BSD-licensed, Python-based) infrastructure for numerical relativity codes Python-based) infrastructure for numerical relativity codes and code generationand code generation
● Goal: When solving problem, choose the best coordinate Goal: When solving problem, choose the best coordinate grid for the task!grid for the task!● Black hole, neutron star: Log-Spherical polar coordsBlack hole, neutron star: Log-Spherical polar coords● Compact binary: Dynamical, Bispherical-like coordsCompact binary: Dynamical, Bispherical-like coords
● Better coordinate grids = Giant efficiency gain over AMR!Better coordinate grids = Giant efficiency gain over AMR!● At least At least ~160x decrease in # of gridpoints → use desktop for BHB~160x decrease in # of gridpoints → use desktop for BHB● Single grid patch = ~25x better scalability than AMR!Single grid patch = ~25x better scalability than AMR!
https://tinyurl.com/senrcode
Enormous Inefficiencies Exist in Enormous Inefficiencies Exist in Numerical Relativity (NR) SimulationsNumerical Relativity (NR) Simulations
BHBHBHBHdxdx dxdx
2 dx2 dx
4 dx4 dx
8 dx8 dx
16 dx, etc16 dx, etc
AMRAdaptive Mesh Refinement
(Most Popular Method in NR)
BHBHBHBHdxdx dxdx
2 dx2 dx
4 dx4 dx
AMRAdaptive Mesh Refinement
(Most Popular Method in NR)
8 dx8 dx
16 dx, etc16 dx, etc
Enormous Inefficiencies Exist in Enormous Inefficiencies Exist in Numerical Relativity (NR) SimulationsNumerical Relativity (NR) Simulations
BHBHdxdx
Enormous Inefficiencies Exist in Enormous Inefficiencies Exist in Numerical Relativity (NR) SimulationsNumerical Relativity (NR) Simulations
BHBHBHBHdy=dx=drdy=dx=dr
Near-Spherical Near-Spherical ObjectObject
● Highest res needed Highest res needed in radial dirn, need in radial dirn, need ~1/3 points in ~1/3 points in angular directions angular directions Cost: Cost: Nr*Ntheta*Nphi ~ Nr*Ntheta*Nphi ~ 1/9 Nr1/9 Nr33
● Cartesian grid: Cartesian grid: need dx=dy=dz=dr. need dx=dy=dz=dr. Cost:Cost:Nx*Ny*Nz ~ NrNx*Ny*Nz ~ Nr33
● So far, spherical So far, spherical polar grid ~polar grid ~9x 9x more more efficient than efficient than CartesianCartesian
Enormous Inefficiencies Exist in Enormous Inefficiencies Exist in Numerical Relativity (NR) SimulationsNumerical Relativity (NR) Simulations
BHBHBHBHdy=dx=drdy=dx=dr
Enormous Inefficiencies Exist in Enormous Inefficiencies Exist in Numerical Relativity (NR) SimulationsNumerical Relativity (NR) Simulations
What about dr What about dr along diagonal?along diagonal?
● Cube diagonal = Cube diagonal = √√3*sidelength → to 3*sidelength → to get dr resolution in get dr resolution in all directions, need all directions, need to reduce dx,dy,dz to reduce dx,dy,dz by by √√33
● Since cost in Since cost in memory ~1/dxmemory ~1/dx33, , “fitting the round “fitting the round peg in a square peg in a square hole” increases hole” increases cost by another cost by another factor of factor of ((√√3)3)33~5.2x!~5.2x!
Near-Spherical Near-Spherical ObjectObject
● Highest res needed Highest res needed in radial dirn, need in radial dirn, need ~1/3 points in ~1/3 points in angular directions angular directions Cost: Cost: Nr*Ntheta*Nphi ~ Nr*Ntheta*Nphi ~ 1/9 Nr1/9 Nr33
● Cartesian grid: Cartesian grid: need dx=dy=dz=dr. need dx=dy=dz=dr. Cost:Cost:Nx*Ny*Nz ~ NrNx*Ny*Nz ~ Nr33
● So far, spherical So far, spherical polar grid ~polar grid ~9x 9x more more efficient than efficient than CartesianCartesian
BHBHBHBHdy=dx=drdy=dx=dr
Inefficiencies so far: Inefficiencies so far: ~47x~47x
Enormous Inefficiencies Exist in Enormous Inefficiencies Exist in Numerical Relativity (NR) SimulationsNumerical Relativity (NR) Simulations
What about dr What about dr along diagonal?along diagonal?
● Cube diagonal = Cube diagonal = √√3*sidelength → to 3*sidelength → to get dr resolution in get dr resolution in all directions, need all directions, need to reduce dx,dy,dz to reduce dx,dy,dz by by √√33
● Since cost in Since cost in memory ~1/dxmemory ~1/dx33, , “fitting the round “fitting the round peg in a square peg in a square hole” increases hole” increases cost by another cost by another factor of factor of ((√√3)3)33~5.2x!~5.2x!
Near-Spherical Near-Spherical ObjectObject
● Highest res needed Highest res needed in radial dirn, need in radial dirn, need ~1/3 points in ~1/3 points in angular directions angular directions Cost: Cost: Nr*Ntheta*Nphi ~ Nr*Ntheta*Nphi ~ 1/9 Nr1/9 Nr33
● Cartesian grid: Cartesian grid: need dx=dy=dz=dr. need dx=dy=dz=dr. Cost:Cost:Nx*Ny*Nz ~ NrNx*Ny*Nz ~ Nr33
● So far, spherical So far, spherical polar grid ~polar grid ~9x 9x more more efficient than efficient than CartesianCartesian
BHBHBHBH AMR Box side-AMR Box side-length = 2length = 2
AMR Box AMR Box Boundary is a Boundary is a
Cube...Cube...
● … … but fields fall off but fields fall off radially!radially!
● →→region outside region outside orange circle is orange circle is over-resolved by 2xover-resolved by 2x
● Total volume of Total volume of over-resolved over-resolved region = 8-4/3 pi ~ region = 8-4/3 pi ~ 3.8 = about half the 3.8 = about half the cube!cube!
● Bispherical Bispherical coordinate system: coordinate system: Gain another ~1.7xGain another ~1.7x
Enormous Inefficiencies Exist in Enormous Inefficiencies Exist in Numerical Relativity (NR) SimulationsNumerical Relativity (NR) Simulations
BHBHBHBH AMR Box side-AMR Box side-length = 2length = 2
AMR Box AMR Box Boundary is a Boundary is a
Cube...Cube...
● … … but fields fall off but fields fall off radially!radially!
● →→region outside region outside orange circle is orange circle is over-resolved by 2xover-resolved by 2x
● Total volume of Total volume of over-resolved over-resolved region = 8-4/3 pi ~ region = 8-4/3 pi ~ 3.8 = about half the 3.8 = about half the cube!cube!
● Bispherical Bispherical coordinate system: coordinate system: Gain another ~1.7xGain another ~1.7x
Inefficiencies so far: Inefficiencies so far: ~80x~80x
Enormous Inefficiencies Exist in Enormous Inefficiencies Exist in Numerical Relativity (NR) SimulationsNumerical Relativity (NR) Simulations
BHBHBHBHdxdx dxdx
2 dx2 dx
4 dx4 dx
AMRAdaptive Mesh Refinement
(Most Popular Method in NR)
8 dx8 dx
AMRAMR
● Information must be interpolated Information must be interpolated across refinement boundaries.across refinement boundaries.
● Interpolation → grids must overlap Interpolation → grids must overlap ● Overlap regions (grey) can take up Overlap regions (grey) can take up
50% of overall computational domain!50% of overall computational domain!
Enormous Inefficiencies Exist in Enormous Inefficiencies Exist in Numerical Relativity (NR) SimulationsNumerical Relativity (NR) Simulations
BHBHBHBHdxdx dxdx
2 dx2 dx
4 dx4 dx
AMRAdaptive Mesh Refinement
(Most Popular Method in NR)
8 dx8 dx
AMRAMR
● Information must be interpolated Information must be interpolated across refinement boundaries.across refinement boundaries.
● Interpolation → grids must overlap Interpolation → grids must overlap ● Overlap regions (grey) can take up Overlap regions (grey) can take up
50% of overall computational domain!50% of overall computational domain!
Enormous Inefficiencies Exist in Enormous Inefficiencies Exist in Numerical Relativity (NR) SimulationsNumerical Relativity (NR) Simulations
BHBHBHBH
AMR Box AMR Box Boundary is a Boundary is a
Cube...Cube...
● … … but fields fall off but fields fall off radially!radially!
● →→region outside region outside orange circle is orange circle is over-resolved by 2xover-resolved by 2x
● Total volume of Total volume of over-resolved over-resolved region = 8-4/3 pi ~ region = 8-4/3 pi ~ 3.8 = about half the 3.8 = about half the cube!cube!
● Bispherical Bispherical coordinate system: coordinate system: Gain another ~1.7xGain another ~1.7x
High-order finite High-order finite difference with difference with
AMRAMR
● → → Enormous Enormous number of ghost number of ghost zones at refinement zones at refinement boundaries!boundaries!
● Ghost zones can Ghost zones can take up 50% of take up 50% of overall overall computational computational domain!domain!
● Bispherical Bispherical coordinate system: coordinate system: Gain another ~2xGain another ~2x
Enormous Inefficiencies Exist in Enormous Inefficiencies Exist in Numerical Relativity (NR) SimulationsNumerical Relativity (NR) Simulations
BHBHBHBH
AMR Box AMR Box Boundary is a Boundary is a
Cube...Cube...
● … … but fields fall off but fields fall off radially!radially!
● →→region outside region outside orange circle is orange circle is over-resolved by 2xover-resolved by 2x
● Total volume of Total volume of over-resolved over-resolved region = 8-4/3 pi ~ region = 8-4/3 pi ~ 3.8 = about half the 3.8 = about half the cube!cube!
● Bispherical Bispherical coordinate system: coordinate system: Gain another ~1.7xGain another ~1.7x
AMR Inefficiencies: AMR Inefficiencies: ~160x~160x
High-order finite High-order finite difference with difference with
AMRAMR
● → → Enormous Enormous number of ghost number of ghost zones at refinement zones at refinement boundaries!boundaries!
● Ghost zones can Ghost zones can take up 50% of take up 50% of overall overall computational computational domain!domain!
● Bispherical Bispherical coordinate system: coordinate system: Gain another ~2xGain another ~2x
Enormous Inefficiencies Exist in Enormous Inefficiencies Exist in Numerical Relativity (NR) SimulationsNumerical Relativity (NR) Simulations
AMR Inefficiencies: AMR Inefficiencies: ~160x~160x
Enormous Inefficiencies Exist in Enormous Inefficiencies Exist in Numerical Relativity (NR) SimulationsNumerical Relativity (NR) Simulations
AMR Inefficiencies: AMR Inefficiencies: ~160x (estimated)~160x (estimated)
Enormous Inefficiencies Exist in Enormous Inefficiencies Exist in Numerical Relativity (NR) SimulationsNumerical Relativity (NR) Simulations
AMR Inefficiencies: AMR Inefficiencies: ~160x (estimated)~160x (estimated)
Enormous Inefficiencies Exist in Enormous Inefficiencies Exist in Numerical Relativity (NR) SimulationsNumerical Relativity (NR) Simulations
Inefficiency Inefficiency MeasurementMeasurement
● Single black holeSingle black hole● Moderate spin:Moderate spin:
● a/M = 0.5a/M = 0.5● Set up AMR Set up AMR
(Carpet) grid, (Carpet) grid, measure H measure H constraint violationconstraint violation
● Adjust SENR grids:Adjust SENR grids:● H_SENR < H_AMR H_SENR < H_AMR
at all pointsat all points
AMR Inefficiencies: AMR Inefficiencies: ~160x (estimated)~160x (estimated)
Enormous Inefficiencies Exist in Enormous Inefficiencies Exist in Numerical Relativity (NR) SimulationsNumerical Relativity (NR) Simulations
Inefficiency Inefficiency MeasurementMeasurement
● Single black holeSingle black hole● Moderate spin:Moderate spin:
● a/M = 0.5a/M = 0.5● Set up AMR Set up AMR
(Carpet) grid, (Carpet) grid, measure H measure H constraint violationconstraint violation
● Adjust SENR grids:Adjust SENR grids:● H_SENR < H_AMR H_SENR < H_AMR
at all pointsat all points
AMR Grid:AMR Grid:● 10 GB10 GB
SENR's Log-SENR's Log-Spherical Spherical Polar Grid:Polar Grid:
● 40 MB40 MB● (un-optimized grid (un-optimized grid
structure, another structure, another 4-10x drop 4-10x drop possible)possible)
AMR Inefficiencies: AMR Inefficiencies: ~160x (estimated)~160x (estimated)
Enormous Inefficiencies Exist in Enormous Inefficiencies Exist in Numerical Relativity (NR) SimulationsNumerical Relativity (NR) Simulations
Inefficiency Inefficiency MeasurementMeasurement
● Single black holeSingle black hole● Moderate spin:Moderate spin:
● a/M = 0.5a/M = 0.5● Set up AMR Set up AMR
(Carpet) grid, (Carpet) grid, measure H measure H constraint violationconstraint violation
● Adjust SENR grids:Adjust SENR grids:● H_SENR < H_AMR H_SENR < H_AMR
at all pointsat all points
AMR Grid:AMR Grid:● 10 GB10 GB
SENR's Log-SENR's Log-Spherical Spherical Polar Grid:Polar Grid:
● 40 MB40 MB● (un-optimized grid (un-optimized grid
structure, another structure, another 4-10x drop 4-10x drop possible)possible)
AMR Inefficiencies: AMR Inefficiencies: 250x (measured)250x (measured)
AMR Inefficiencies: AMR Inefficiencies: ~160x (estimated)~160x (estimated)
Enormous Inefficiencies Exist in Enormous Inefficiencies Exist in Numerical Relativity (NR) SimulationsNumerical Relativity (NR) Simulations
Inefficiency Inefficiency MeasurementMeasurement
● Single black holeSingle black hole● Moderate spin:Moderate spin:
● a/M = 0.5a/M = 0.5● Set up AMR Set up AMR
(Carpet) grid, (Carpet) grid, measure H measure H constraint violationconstraint violation
● Adjust SENR grids:Adjust SENR grids:● H_SENR < H_AMR H_SENR < H_AMR
at all pointsat all points
AMR Grid:AMR Grid:● 10 GB10 GB
SENR's Log-SENR's Log-Spherical Spherical Polar Grid:Polar Grid:
● 40 MB40 MB● (un-optimized grid (un-optimized grid
structure, another structure, another 4-10x drop 4-10x drop possible)possible)
AMR Inefficiencies: AMR Inefficiencies: 250x (measured)250x (measured)
Stable long-term BH evolutions!Stable long-term BH evolutions!But does it converge?But does it converge?
SENR Results: SENR Results: Exponential convergence of Exponential convergence of
numerical errors!numerical errors!
Simulating Simulating black black hole without hole without
excisionexcision: :
Numerical errors Numerical errors converge to zero converge to zero
exponentiallyexponentially with with increased increased
polynomial polynomial approximation order!approximation order!
SENR/NRPy: SummarySENR/NRPy: Summary● Open Source, Open DevelopmentOpen Source, Open Development → → Greater AdoptionGreater Adoption
● http://tinyurl.com/senrcodehttp://tinyurl.com/senrcode● Algorithmic SimplicityAlgorithmic Simplicity → → More Science FasterMore Science Faster
● Easier to debug & extendEasier to debug & extend● Build on tried & true algorithmsBuild on tried & true algorithms
– BSSN in Spherical Polar Coords techniques pioneered by T. BSSN in Spherical Polar Coords techniques pioneered by T. Baumgarte et alBaumgarte et al
● SENR: Extend ideas to support arbitrary, SENR: Extend ideas to support arbitrary, dynamicaldynamical coords coords● Memory Efficiency Is Key FocusMemory Efficiency Is Key Focus: : Unlock the DesktopUnlock the Desktop
● Get public involved → ~10,000x more GW throughput!Get public involved → ~10,000x more GW throughput!
● Bottom line: Bottom line: Maximize science with minimal human & computational Maximize science with minimal human & computational resourcesresources