owls: overwhelmingly large simulations joop schaye, craig booth, claudio dalla vecchia, alan duffy,...
TRANSCRIPT
![Page 1: OWLS: OverWhelmingly Large Simulations Joop Schaye, Craig Booth, Claudio Dalla Vecchia, Alan Duffy, Marcel Haas, Volker Springel, Tom Theuns, Luca Tornatore,](https://reader035.vdocuments.mx/reader035/viewer/2022062519/5697bff01a28abf838cbacae/html5/thumbnails/1.jpg)
OWLS: OverWhelmingly Large Simulations
Joop Schaye, Craig Booth, Claudio Dalla Vecchia, Alan Duffy, Marcel Haas, Volker Springel,
Tom Theuns, Luca Tornatore, Rob Wiersma, …
The formation of galaxies and the evolution of the intergalactic medium
![Page 2: OWLS: OverWhelmingly Large Simulations Joop Schaye, Craig Booth, Claudio Dalla Vecchia, Alan Duffy, Marcel Haas, Volker Springel, Tom Theuns, Luca Tornatore,](https://reader035.vdocuments.mx/reader035/viewer/2022062519/5697bff01a28abf838cbacae/html5/thumbnails/2.jpg)
Outline
1. Introducing cosmological simulations and OWLS
2. Preparing for and coping with lots of data
3. Data format and units
4. Microsimulations
5. Virtual observations of simulations
6. Discussion points
![Page 3: OWLS: OverWhelmingly Large Simulations Joop Schaye, Craig Booth, Claudio Dalla Vecchia, Alan Duffy, Marcel Haas, Volker Springel, Tom Theuns, Luca Tornatore,](https://reader035.vdocuments.mx/reader035/viewer/2022062519/5697bff01a28abf838cbacae/html5/thumbnails/3.jpg)
Example science questions• What determines the SF history of the
universe?• Where are the baryons and how can
they be detected?• Where are the metals and how can they
be detected?• How does galaxy formation depend on
environment?• What are the large-scale effects of
galactic winds driven by stars and AGN?• Virtual observations
![Page 4: OWLS: OverWhelmingly Large Simulations Joop Schaye, Craig Booth, Claudio Dalla Vecchia, Alan Duffy, Marcel Haas, Volker Springel, Tom Theuns, Luca Tornatore,](https://reader035.vdocuments.mx/reader035/viewer/2022062519/5697bff01a28abf838cbacae/html5/thumbnails/4.jpg)
Cosmological hydro simulations
• Evolution from z>~100 to z ~< 10 of a representative part of the universe
• Boundary conditions: periodic• Expansion solved analytically and scaled
out• Initial conditions from the CMB • Components: cold dark matter, gas, stars,
radiation (optically thin)• Scales ~< kpc to ~ 100 Mpc• Sub-grid modules are a crucial part of the
game• Discretizaton: time and mass (SPH)
![Page 5: OWLS: OverWhelmingly Large Simulations Joop Schaye, Craig Booth, Claudio Dalla Vecchia, Alan Duffy, Marcel Haas, Volker Springel, Tom Theuns, Luca Tornatore,](https://reader035.vdocuments.mx/reader035/viewer/2022062519/5697bff01a28abf838cbacae/html5/thumbnails/5.jpg)
The OWLS simulations• LOFAR IBM Bluegene/L (12k CPUs, PowerPC
440, 700 MHz, 500 MB per 2 CPUs)• Cosmological (default: WMAP3)• Hydrodynamical (SPH)• Gadget III• 2xN3 particles, N = 512 for most• Two sets:
– L = 25 Mpc/h to z=2 – L = 100 Mpc/h to z=0
• Runs repeated many times with varying physics/numerics
• Status: runs nearly done, starting analysis
![Page 6: OWLS: OverWhelmingly Large Simulations Joop Schaye, Craig Booth, Claudio Dalla Vecchia, Alan Duffy, Marcel Haas, Volker Springel, Tom Theuns, Luca Tornatore,](https://reader035.vdocuments.mx/reader035/viewer/2022062519/5697bff01a28abf838cbacae/html5/thumbnails/6.jpg)
OWLS subgrid physics improvements
• New star formation• New wind model• Added chemodynamics• New cooling
![Page 7: OWLS: OverWhelmingly Large Simulations Joop Schaye, Craig Booth, Claudio Dalla Vecchia, Alan Duffy, Marcel Haas, Volker Springel, Tom Theuns, Luca Tornatore,](https://reader035.vdocuments.mx/reader035/viewer/2022062519/5697bff01a28abf838cbacae/html5/thumbnails/7.jpg)
OWLS Numerical Resolution
• 25 Mpc/h (run to z=2):– m_bar = 1x10^6 Msun/h – Softening = 2 kpc/h comoving < 0.5 kpc/h proper
• 100 Mpc/h (run to z=0):– m_bar = 7x10^7 Msun/h– Softening = 8 kpc/h comoving
< 2.0 kpc/h proper
![Page 8: OWLS: OverWhelmingly Large Simulations Joop Schaye, Craig Booth, Claudio Dalla Vecchia, Alan Duffy, Marcel Haas, Volker Springel, Tom Theuns, Luca Tornatore,](https://reader035.vdocuments.mx/reader035/viewer/2022062519/5697bff01a28abf838cbacae/html5/thumbnails/8.jpg)
Gas density evolution
![Page 9: OWLS: OverWhelmingly Large Simulations Joop Schaye, Craig Booth, Claudio Dalla Vecchia, Alan Duffy, Marcel Haas, Volker Springel, Tom Theuns, Luca Tornatore,](https://reader035.vdocuments.mx/reader035/viewer/2022062519/5697bff01a28abf838cbacae/html5/thumbnails/9.jpg)
![Page 10: OWLS: OverWhelmingly Large Simulations Joop Schaye, Craig Booth, Claudio Dalla Vecchia, Alan Duffy, Marcel Haas, Volker Springel, Tom Theuns, Luca Tornatore,](https://reader035.vdocuments.mx/reader035/viewer/2022062519/5697bff01a28abf838cbacae/html5/thumbnails/10.jpg)
Gas density zoom
![Page 11: OWLS: OverWhelmingly Large Simulations Joop Schaye, Craig Booth, Claudio Dalla Vecchia, Alan Duffy, Marcel Haas, Volker Springel, Tom Theuns, Luca Tornatore,](https://reader035.vdocuments.mx/reader035/viewer/2022062519/5697bff01a28abf838cbacae/html5/thumbnails/11.jpg)
Zoom
CDV, OWLS project
![Page 12: OWLS: OverWhelmingly Large Simulations Joop Schaye, Craig Booth, Claudio Dalla Vecchia, Alan Duffy, Marcel Haas, Volker Springel, Tom Theuns, Luca Tornatore,](https://reader035.vdocuments.mx/reader035/viewer/2022062519/5697bff01a28abf838cbacae/html5/thumbnails/12.jpg)
Outline
1. Introducing cosmological simulations and OWLS
2. Preparing for and coping with lots of data
3. Data format and units
4. Microsimulations
5. Virtual observations of simulations
6. Discussion points
![Page 13: OWLS: OverWhelmingly Large Simulations Joop Schaye, Craig Booth, Claudio Dalla Vecchia, Alan Duffy, Marcel Haas, Volker Springel, Tom Theuns, Luca Tornatore,](https://reader035.vdocuments.mx/reader035/viewer/2022062519/5697bff01a28abf838cbacae/html5/thumbnails/13.jpg)
OWLS data volume
• 7 - 42 floats per particle up to 168 bytes/particle
• 268 M particles per snapshot 26 GB/snapshot
• Up to 35 snapshots per simulations 900 GB/simulation
• ~60 large simulations 54 TB of snapshot data
![Page 14: OWLS: OverWhelmingly Large Simulations Joop Schaye, Craig Booth, Claudio Dalla Vecchia, Alan Duffy, Marcel Haas, Volker Springel, Tom Theuns, Luca Tornatore,](https://reader035.vdocuments.mx/reader035/viewer/2022062519/5697bff01a28abf838cbacae/html5/thumbnails/14.jpg)
How do we analyze 50+ TB of data?
• First analyze lower resolution versions• Use hdf5 only read what is actually needed• Use fast visualization software (e.g. avoid SPH
interpolation)• Produce as much as possible on the fly:
– Logs (e.g. SF histories)– Grids (e.g. baryon distribution) – Sight lines (e.g. QSO absorption spectra)– Images (e.g. for videos)– Zooms saved on the fly (e.g. most massive object)– Diagnostic particle arrays (e.g. max temperature)– Group catalogues
![Page 15: OWLS: OverWhelmingly Large Simulations Joop Schaye, Craig Booth, Claudio Dalla Vecchia, Alan Duffy, Marcel Haas, Volker Springel, Tom Theuns, Luca Tornatore,](https://reader035.vdocuments.mx/reader035/viewer/2022062519/5697bff01a28abf838cbacae/html5/thumbnails/15.jpg)
Analysis that can be done on a notebook:
• Low-resolution versions• Evolution of globally averaged (or
gridded) quantities• Halo integrated properties• Halo profiles• Absorption spectra• Most things related to zooms
![Page 16: OWLS: OverWhelmingly Large Simulations Joop Schaye, Craig Booth, Claudio Dalla Vecchia, Alan Duffy, Marcel Haas, Volker Springel, Tom Theuns, Luca Tornatore,](https://reader035.vdocuments.mx/reader035/viewer/2022062519/5697bff01a28abf838cbacae/html5/thumbnails/16.jpg)
Outline
1. Introducing cosmological simulations and OWLS
2. Preparing for and coping with lots of data
3. Data format and units
4. Microsimulations
5. Virtual observations of simulations
6. Discussion points
![Page 17: OWLS: OverWhelmingly Large Simulations Joop Schaye, Craig Booth, Claudio Dalla Vecchia, Alan Duffy, Marcel Haas, Volker Springel, Tom Theuns, Luca Tornatore,](https://reader035.vdocuments.mx/reader035/viewer/2022062519/5697bff01a28abf838cbacae/html5/thumbnails/17.jpg)
Hdf5 – Hierarchical Data Format
• Binary fast I/O and compressed• Machine-independent• Intuitive hierarchical “directory-like”
structure• Individual elements can be read, modified
and saved • Easy to add meta-data• Utilities available to view the data in ascii
or graphical form (incl. tables, images)• Very easy to read/write from IDL, C, F90• Free
![Page 18: OWLS: OverWhelmingly Large Simulations Joop Schaye, Craig Booth, Claudio Dalla Vecchia, Alan Duffy, Marcel Haas, Volker Springel, Tom Theuns, Luca Tornatore,](https://reader035.vdocuments.mx/reader035/viewer/2022062519/5697bff01a28abf838cbacae/html5/thumbnails/18.jpg)
Lay-out of an hdf5 file
![Page 19: OWLS: OverWhelmingly Large Simulations Joop Schaye, Craig Booth, Claudio Dalla Vecchia, Alan Duffy, Marcel Haas, Volker Springel, Tom Theuns, Luca Tornatore,](https://reader035.vdocuments.mx/reader035/viewer/2022062519/5697bff01a28abf838cbacae/html5/thumbnails/19.jpg)
hdf5 groups
![Page 20: OWLS: OverWhelmingly Large Simulations Joop Schaye, Craig Booth, Claudio Dalla Vecchia, Alan Duffy, Marcel Haas, Volker Springel, Tom Theuns, Luca Tornatore,](https://reader035.vdocuments.mx/reader035/viewer/2022062519/5697bff01a28abf838cbacae/html5/thumbnails/20.jpg)
hdf5 data sets
![Page 21: OWLS: OverWhelmingly Large Simulations Joop Schaye, Craig Booth, Claudio Dalla Vecchia, Alan Duffy, Marcel Haas, Volker Springel, Tom Theuns, Luca Tornatore,](https://reader035.vdocuments.mx/reader035/viewer/2022062519/5697bff01a28abf838cbacae/html5/thumbnails/21.jpg)
Data format: Units• Data should use code units to
allow easy debugging, restarting• But data should be in physically
sensible units to allow analysis by external users
• Cosmological data should clarify h and aexp dependence for external users
Solution: use code units but includeconversion factors to cgs and aexp and hdependencies as meta-data.
![Page 22: OWLS: OverWhelmingly Large Simulations Joop Schaye, Craig Booth, Claudio Dalla Vecchia, Alan Duffy, Marcel Haas, Volker Springel, Tom Theuns, Luca Tornatore,](https://reader035.vdocuments.mx/reader035/viewer/2022062519/5697bff01a28abf838cbacae/html5/thumbnails/22.jpg)
Attributes to data
sets
![Page 23: OWLS: OverWhelmingly Large Simulations Joop Schaye, Craig Booth, Claudio Dalla Vecchia, Alan Duffy, Marcel Haas, Volker Springel, Tom Theuns, Luca Tornatore,](https://reader035.vdocuments.mx/reader035/viewer/2022062519/5697bff01a28abf838cbacae/html5/thumbnails/23.jpg)
Outline
1. Introducing cosmological simulations and OWLS
2. Preparing for and coping with lots of data
3. Data format and units
4. Microsimulations
5. Virtual observations of simulations
6. Discussion points
![Page 24: OWLS: OverWhelmingly Large Simulations Joop Schaye, Craig Booth, Claudio Dalla Vecchia, Alan Duffy, Marcel Haas, Volker Springel, Tom Theuns, Luca Tornatore,](https://reader035.vdocuments.mx/reader035/viewer/2022062519/5697bff01a28abf838cbacae/html5/thumbnails/24.jpg)
Outline
1. Introducing cosmological simulations and OWLS
2. Preparing for and coping with lots of data
3. Data format and units
4. Microsimulations
5. Virtual observations of simulations
6. Discussion points
![Page 25: OWLS: OverWhelmingly Large Simulations Joop Schaye, Craig Booth, Claudio Dalla Vecchia, Alan Duffy, Marcel Haas, Volker Springel, Tom Theuns, Luca Tornatore,](https://reader035.vdocuments.mx/reader035/viewer/2022062519/5697bff01a28abf838cbacae/html5/thumbnails/25.jpg)
Virtual Observations: Uses
• Guide the design of observational campaigns and instruments
• Test data analysis pipelines• Determine selection effects• Test theoretical models• PR/education
![Page 26: OWLS: OverWhelmingly Large Simulations Joop Schaye, Craig Booth, Claudio Dalla Vecchia, Alan Duffy, Marcel Haas, Volker Springel, Tom Theuns, Luca Tornatore,](https://reader035.vdocuments.mx/reader035/viewer/2022062519/5697bff01a28abf838cbacae/html5/thumbnails/26.jpg)
Virtual Observations: Types• Galaxy magnitudes
– Population synthesis, which depends on• Assumed IMF• Age and composition of star particle
– Dust column and assumed reddening law• Stellar light images• Absorption spectra• Gas emission maps (2-D: images, 3-D:
IFU datacubes)• Other maps: e.g. lensing, SZ, dust
absorption/emission
![Page 27: OWLS: OverWhelmingly Large Simulations Joop Schaye, Craig Booth, Claudio Dalla Vecchia, Alan Duffy, Marcel Haas, Volker Springel, Tom Theuns, Luca Tornatore,](https://reader035.vdocuments.mx/reader035/viewer/2022062519/5697bff01a28abf838cbacae/html5/thumbnails/27.jpg)
Creating Virtual Observations
• Compute emission and/or absorption of each resolution element
• Project and grid data, optionally combine different time slices (e.g. lightcones)
• Observe with chosen instrument (easiest step)
![Page 28: OWLS: OverWhelmingly Large Simulations Joop Schaye, Craig Booth, Claudio Dalla Vecchia, Alan Duffy, Marcel Haas, Volker Springel, Tom Theuns, Luca Tornatore,](https://reader035.vdocuments.mx/reader035/viewer/2022062519/5697bff01a28abf838cbacae/html5/thumbnails/28.jpg)
Virtual Observations – Star light
• Population synthesis– Save initial mass, age, and composition of
individual star particles– Assume IMF
• Dust column– Save mass and composition of individual
gas particles– Assume reddening law– Require fast calculation of SPH column
densities
![Page 29: OWLS: OverWhelmingly Large Simulations Joop Schaye, Craig Booth, Claudio Dalla Vecchia, Alan Duffy, Marcel Haas, Volker Springel, Tom Theuns, Luca Tornatore,](https://reader035.vdocuments.mx/reader035/viewer/2022062519/5697bff01a28abf838cbacae/html5/thumbnails/29.jpg)
Virtual Observations – Gas absorption
• Require gas mass, position, velocity, density, temperature, composition
• Assume ionization equilibrium and use ionization balance tables (e.g. CLOUDY)– Collisional ionization temperature– Photo-ionization density,
temperature, radiation field
![Page 30: OWLS: OverWhelmingly Large Simulations Joop Schaye, Craig Booth, Claudio Dalla Vecchia, Alan Duffy, Marcel Haas, Volker Springel, Tom Theuns, Luca Tornatore,](https://reader035.vdocuments.mx/reader035/viewer/2022062519/5697bff01a28abf838cbacae/html5/thumbnails/30.jpg)
Virtual Observations – Gas emission
• Require gas mass, position, density, temperature, composition, (velocity)
• Assume ionization equilibrium and use emissivity tables (e.g. CLOUDY)
• Dust columns
![Page 31: OWLS: OverWhelmingly Large Simulations Joop Schaye, Craig Booth, Claudio Dalla Vecchia, Alan Duffy, Marcel Haas, Volker Springel, Tom Theuns, Luca Tornatore,](https://reader035.vdocuments.mx/reader035/viewer/2022062519/5697bff01a28abf838cbacae/html5/thumbnails/31.jpg)
On-demand Virtual Observations of cosmological simulations
• Computed from physical properties (e.g. gas density, temperature)
• Computed from pre-calculated “ideal” observables
– Example: narrow band gas emission for various assumed radiation fields, chemical compositions, or density/temperature cuts– Allows one to vary both physical assumptions and instrumental characteristics– Computationally expensive
– Example: narrow band gas emission for a fixed radiation field and chemical composition– Cannot vary physical assumptions– Relatively cheap
![Page 32: OWLS: OverWhelmingly Large Simulations Joop Schaye, Craig Booth, Claudio Dalla Vecchia, Alan Duffy, Marcel Haas, Volker Springel, Tom Theuns, Luca Tornatore,](https://reader035.vdocuments.mx/reader035/viewer/2022062519/5697bff01a28abf838cbacae/html5/thumbnails/32.jpg)
VOs from physical properties • Challenge: Analysis of 3-D simulations is
typically memory-intensive (512^3 1 single precision array is 0.5 GB)
• Solution: Prepare data with reduced
dimensionality, calculate on demand–1-D: Absorption spectra, dust columns–2-D: Images from integrated quantities
Problem: Emission & (gas) absorption depends on local 3-D rather than projected 2-D properties
![Page 33: OWLS: OverWhelmingly Large Simulations Joop Schaye, Craig Booth, Claudio Dalla Vecchia, Alan Duffy, Marcel Haas, Volker Springel, Tom Theuns, Luca Tornatore,](https://reader035.vdocuments.mx/reader035/viewer/2022062519/5697bff01a28abf838cbacae/html5/thumbnails/33.jpg)
On-demand VOs
• Feasible now or in the near future:– (VOs based on) object catalogues– Absorption spectra– SZ effect– Images of stellar light (though maybe
not full lightcones)– Other VOs from pre-calculated
observables
![Page 34: OWLS: OverWhelmingly Large Simulations Joop Schaye, Craig Booth, Claudio Dalla Vecchia, Alan Duffy, Marcel Haas, Volker Springel, Tom Theuns, Luca Tornatore,](https://reader035.vdocuments.mx/reader035/viewer/2022062519/5697bff01a28abf838cbacae/html5/thumbnails/34.jpg)
Discussion points:1. Are raw cosmological simulation data
sets too large to put in the VO? 2. Is there a place for non-VO, processed
data products in the VO? (e.g. physical properties)
3. Must VO software for non-simulators be like a black box?
4. Is it desirable to create black boxes for non-simulators?
5. How to prevent wasting of resources by wrong use of VO software?