horizon project - cds.u-strasbg.fr

HORIZON ProjectHORIZON Projecthttp://www.projethttp://www.projet--horizon.frhorizon.fr

H. Wozniak on behalf of the Horizon ProjectH. Wozniak on behalf of the Horizon Project

2008/04/072008/04/07 H. Wozniak (CRAL)H. Wozniak (CRAL) 22

Project goalsProject goals

•• Understanding the formation mechanisms of:Understanding the formation mechanisms of:–– LargeLarge--scale structurescale structure–– Clusters of galaxiesClusters of galaxies–– Galaxies (also evolution)Galaxies (also evolution)–– MilkyMilky--WayWay

•• Sharing best practices in: Sharing best practices in: –– physical algorithmsphysical algorithms–– ParallelismParallelism–– Software developmentsSoftware developments–– visualizationvisualization

•• Producing mock observations/catalogues/light cones for: Producing mock observations/catalogues/light cones for: –– Science preparation for forthcoming instrumentsScience preparation for forthcoming instruments–– Analysis of large surveys (Analysis of large surveys (egeg VVDS)VVDS)


Alimi Jean-MichelArbey Alexandre Athanassoula Lia Aubert Dominique Audit Edouard Baek SunghyeBlaizot Jeremy Bonhomme Nicolas Bournaud FredericChampavert NicolasChevallier LoicChilingarian IgorColombi StéphaneCombes Françoise Courtin JeromeCourtois HeleneCourty Stéphanie Da Silva Antonio Depardon BenjaminDevriendt Julien

Ocvirk Pierre Peirani SebastienPichon Christophe Pier Stefano CorasanitiPrunet Simon Rasera Yann Revaz YvesRimes Chris Roy Fabrice Sauvageot Jean-Luc Semelin BenoitSlyz AdrianneSousbie Thierry Stéphane FayTeyssier RomainThiebaut JéromeTiret OlivierTweed Dylan Wozniak HervéZidani Djilali

Dimatteo PaolaDubois YohanDurier Fabrice Elad ZingerEmsellem Eric Forero Jaime Füzfa AndréGay ChristopheGrasseau GillesGuiderdoni Bruno Honoré Pierre-François Jablonka Pascale Lambert Jean-Charles Lavaux GuilhemLe Fèvre Jean-Paul Leborgne DamienLegrand FrançoisMelchior Anne-LaureMichel-Dansac Leo

WhoWho’’s whos who

•• ~ ~ 2/3 on large2/3 on large--scale structure and galaxy formationscale structure and galaxy formation•• ~ ~ 1/3 on galaxy evolution1/3 on galaxy evolution•• Several Several ‘‘methodologicalmethodological’’ originsorigins

–– Code developers (AMR, Code developers (AMR, TreeSPHTreeSPH, PMSPH, , PMSPH, radiativeradiative transfer, MHD, etc.)transfer, MHD, etc.)–– Complementary Complementary approchesapproches (direct simulation (direct simulation vsvs semisemi--analytics)analytics)–– Programming techniques (MPI, Programming techniques (MPI, OpenMPOpenMP, Computing, Computing--centrescentres expertise, etc.)expertise, etc.)

•• A distributed network (both people and material)A distributed network (both people and material)

5 partners5 partnersLab Lab CoCo--IILUTHLUTH J.J.--M. M. AlimiAlimi MeudonMeudonIAP IAP C. C. PichonPichon ParisParisLERMALERMA F. F. CombesCombes ParisParisCRALCRAL B. Guiderdoni LyonB. Guiderdoni LyonSAPSAP R. R. TeyssierTeyssier SaclaySaclay


Computing centres (CCRT, IDRIS, BSC)

HP Meso-machine

Paris Meudon Lyon

Soumission de jobs

Post-treatment, raw archives, backup

Visualization,Post-treatment andadditional archives

Saclay Marseille

Horizon-GRID

HardwareHardwareWeb Horizon:Web Horizon:http://www.projethttp://www.projet--horizon.frhorizon.frGalMerGalMer database:database:http://galmer.obspm.frhttp://galmer.obspm.frGalICS database: (not opened)GalICS database: (not opened)http://horizonhttp://horizon--vo.univvo.univ--lyon1.frlyon1.fr


Another largeAnother large--scale simulation?scale simulation?

Horizon XL Horizon L Horizon S

•• Cosmological simulations:Cosmological simulations:–– Initial conditions:Initial conditions:

•• MPGraficMPGrafic ((PrunetPrunet et al. 2008)et al. 2008)Three box size (20, 100, 500 Three box size (20, 100, 500 Mpc/hMpc/h) were chosen and correspond to three "different" power ) were chosen and correspond to three "different" power spectra. spectra.

•• Low resolution versions of the ICs were also produced by averagiLow resolution versions of the ICs were also produced by averaging the different fields over 2ng the different fields over 233

cells, leading to a hierarchy of 512cells, leading to a hierarchy of 51233, 256, 25633, 128, 12833, 64, 6444, 32, 3233 ICs boxes.ICs boxes.–– Dark matter largeDark matter large--scale simulation (scale simulation (Horizon 4Pi Horizon 4Pi simulation)simulation)

•• 4096409633, 2 , 2 Gpc/hGpc/h periodic box periodic box ~ 50% of the observable Universe~ 50% of the observable Universe•• CEACEA--Bull computer with 6144 proc. Bull computer with 6144 proc.

–– Hydro+NHydro+N--body + SF largebody + SF large--scale simulation (scale simulation (Mare NostrumMare Nostrum simulation)simulation)•• 1024102433, 4 billions AMR cells, 50 , 4 billions AMR cells, 50 Mpc/hMpc/h boxbox•• Mare Nostrum computer (Barcelona) with 2048 proc.Mare Nostrum computer (Barcelona) with 2048 proc.

–– Simulations with Simulations with ‘‘quintessencequintessence’’, DM, DM--only only ‘‘smallsmall--sizesize’’ simulationssimulations……•• ‘‘zoomzoom’’--simulationssimulations

–– Cluster of galaxiesCluster of galaxies–– ‘‘milkymilky--wayway’’ like galaxieslike galaxies

•• ‘‘idealizedidealized’’ simulationssimulations–– GalMerGalMer (galaxies mergers) sub(galaxies mergers) sub--projectproject–– Chemodynamical simulation of isolated discsChemodynamical simulation of isolated discs


Generation of density fields and 3D velocities in Fourier space Generation of density fields and 3D velocities in Fourier space with random with random phases, from WMAP spectrum.phases, from WMAP spectrum.

Standard process: white noise generation in the real space + Standard process: white noise generation in the real space + -- convolution by parallel FFT (FFTW)convolution by parallel FFT (FFTW)

MpgraficMpgrafic codecodehttp://www2.iap.fr/http://www2.iap.fr/usersusers//pichonpichon//mpgrafic.htmlmpgrafic.htmlPrunetPrunet et al., 2008, MNRAS, in presset al., 2008, MNRAS, in pressTested up to 4096 processorsTested up to 4096 processors4096409633: 256 Go by field: 256 Go by field

Analytical transfer function or Analytical transfer function or produced by a produced by a BoltzmannBoltzmann code code including including ““wiggles baryonswiggles baryons””

Density field 40963 produced by MPgrafic

Initial conditionsInitial conditions


SoftwaresSoftwares

•• MPGraficMPGrafic ((PrunetPrunet et al. 2008): Initial conditions generatoret al. 2008): Initial conditions generator•• RAMSES: AMR code (RAMSES: AMR code (TeyssierTeyssier 2002): 2002): ‘‘extremeextreme’’ and and ‘‘zoomzoom’’ simulationssimulations•• GadgetGadget--2 (2 (SpringelSpringel et al. 2005): et al. 2005): ‘‘zoomzoom’’•• TreeSPHTreeSPH ((CombesCombes, , SemelinSemelin, et al.): , et al.): ‘‘idealizedidealized’’ mergersmergers•• PMSPH (Wozniak et al.): chemodynamical evolutionPMSPH (Wozniak et al.): chemodynamical evolution•• GalICS suite: GalICS suite: HalomakerHalomaker, , galaxymakergalaxymaker etc. (etc. (BlaizotBlaizot et al.): semiet al.): semi--analytical analytical

modellingmodelling•• ……


MilestonesMilestones

•• Jun 03: first discussionsJun 03: first discussions•• Feb 04: first approval by Scientific Advisory committeesFeb 04: first approval by Scientific Advisory committees•• Sep 04: KickSep 04: Kick--off meetingoff meeting•• Sep 05: opening of Sep 05: opening of mesomeso--machine hosted by HPC1 (HPmachine hosted by HPC1 (HP--France)France)•• Oct 05: Selection by DEISAOct 05: Selection by DEISA•• Feb 06: opening of the Horizon miniFeb 06: opening of the Horizon mini--GRID (distributed data storage)GRID (distributed data storage)•• Oct 06: start of Mare Nostrum simulationOct 06: start of Mare Nostrum simulation•• Jan 07: installation of CEAJan 07: installation of CEA--BULL computer (43 BULL computer (43 TflopsTflops))•• JulJul--Aug 07: Horizon 4Pi run (70 10Aug 07: Horizon 4Pi run (70 1099 of particles down to z=0)of particles down to z=0)•• Sep 07: end of Mare Nostrum simulation (150 10Sep 07: end of Mare Nostrum simulation (150 1066 stellar particles down to stellar particles down to

z=1.55)z=1.55)•• Dec 07: GALMER database openingDec 07: GALMER database opening•• FevFev 08: 08: BlueGeneBlueGene installed in IDRIS (139 installed in IDRIS (139 TflopsTflops))•• SummerSummer--Fall 08: GalICS database opening?Fall 08: GalICS database opening?•• Fall 08: end of the project (HorizonFall 08: end of the project (Horizon--II?)II?)


70 billions particles N-body simulation with 140 billions AMR cells for a 2 Gpc/h periodic box in a LCDM universe.

We use a new French supercomputer BULL Novascale 3045 recently commissioned at CCRT (Centre de CalculRecherche et Technologie, CEA).

We ran RAMSES in pure N-body mode with 6144 processors for 2 months. Starting with a base grid of 40963 cells, we used 6 additional level of refinements for a formal resolution of 2621443.

Using our light cone, we have computed a full sky convergence map for simulating future weak-lensing surveys like DUNE or LSST.

6 billions light years

13 billions light years

4Pi simulation4Pi simulation


The first Full-Sky weak-lensing map with 0.9 arcmin2

resolution : 4 decade in angular scales !!!

Preparation for cosmology surveys using wide field imaging in space

Teyssier et al., 2008 (submitted)

4Pi simulation4Pi simulation


GADGET team: G. GADGET team: G. YepesYepes, S. , S. GoettloberGoettlober, M. , M. HoeftHoeft, A. , A. KhalatyanKhalatyan……RAMSES team: RAMSES team: R. R. TeyssierTeyssier, D. Aubert, E. Audit, J. , D. Aubert, E. Audit, J. DevriendtDevriendt, C. , C. PichonPichon……withwith strongstrong support support fromfrom BSC BSC andand IDRIS.IDRIS.

GADGET GADGET RAMSESRAMSESSPHSPH z=5.7z=5.7 AMRAMR

50 h50 h--11 MpcMpc box box withwith 1024102433 particlesparticles andand 4 billion AMR 4 billion AMR cellscells1024102433 base base gridgrid +5 +5 levelslevels ofof refinementrefinement ((smallestsmallest cellcell isis 11--2 2 kpckpc physicalphysical))N body + N body + gasgas, , coolingcooling, star formation, , star formation, polytropepolytrope, supernovae blast , supernovae blast waveswaves, , metalsmetalsRAMSES RAMSES runrun: 4 : 4 weeksweeks ((fromfrom sept. 2006 to sept. 2007): 1.5 sept. 2006 to sept. 2007): 1.5 MhrMhr withwith 2048 proc.2048 proc.

MareNostrumMareNostrum simulationsimulation


50h-1Mpc 12,5h-1Mpc 3125h-1kpc 780h-1kpc

DarkDarkmattermatterdensitydensity

GasGastemperaturetemperature

GasGasdensitydensity

StellarStellardensitydensity

190h-1kpc


T map, average 3DT map, cut T map, XMM

UsingUsing thethe XMM instrument simulator of J.XMM instrument simulator of J.--L. L. SauvageotSauvageot, , derivederivemockmock XX--rayray images to images to assessassess feasibilityfeasibility of of studyingstudying shockshockwaveswaves physicsphysics..

SolovyevaSolovyeva et al. (2007, 2008)et al. (2007, 2008)

ZoomZoom--simulations: clusters of galaxiessimulations: clusters of galaxies


Properties of galaxiesProperties of galaxies

•• ‘‘Mare NostrumMare Nostrum’’ simulationsimulation–– Formed by small scale star formation Formed by small scale star formation

recipesrecipes–– Formed by semiFormed by semi--analytical analytical modellingmodelling

•• ‘‘GalMerGalMer’’ galaxiesgalaxies–– Formed by mergersFormed by mergers–– True color images and spectra with True color images and spectra with

PegasePegase--HRHR

•• ‘‘GalICSGalICS’’ galaxies in DM simulationsgalaxies in DM simulations–– Formed by semiFormed by semi--analytical analytical modellingmodelling

•• Need catalogues of DM halosNeed catalogues of DM halos•• Merging treesMerging trees•• GalICSGalICS


z=2.4

MareNostrumMareNostrum simulationsimulation


Down to Galactic Scales

GALICSGALICSGALMERGALMER

OnOn--line cataloguesline catalogues


I. Chilingarian, P. di Matteo, A.-L. Melchior, B. Sémelin, F. Combeshttp://galmer.obspm.frGalMerGalMer


GalICS is coming soonGalICS is coming soon……J. Blaizot, A. Cattaneo, J. Devriendt, J. Forero-Romero,

B. Guiderdoni, S. Hatton, B. Lanzoni, D. Tweedhttp://horizon-vo.univ-lyon1.fr/...


TheoryTheory--VO issuesVO issues

•• Data access:Data access:–– Initial condition in Initial condition in GraficGrafic file format with I/O routinesfile format with I/O routines–– Raw snapshots available for small simulations only (Raw snapshots available for small simulations only (cosmocosmo=tens of Tb of data)=tens of Tb of data)–– PostPost--processed data only (catalogues thru database access)processed data only (catalogues thru database access)

•• TheoryTheory--theory comparisonstheory comparisons–– Same Initial conditions but different codes (AMR Same Initial conditions but different codes (AMR vsvs SPH)SPH)

•• How comparing results? Correlation function, positions, phaseHow comparing results? Correlation function, positions, phase--space,space,…… ??–– Same codes but different resolutions and box sizesSame codes but different resolutions and box sizes

•• TheoryTheory--observation interfaceobservation interface–– Statistical approach only (surveys)Statistical approach only (surveys)

•• A lot of work (beyond the raw simulationsA lot of work (beyond the raw simulations……))–– Via databasesVia databases

•• Needs exhaustive documentation, friendly interface, etc.Needs exhaustive documentation, friendly interface, etc.–– Via web interfacesVia web interfaces

•• Needs forms?Needs forms?

Thank youThank you

horizon project - cds.u-strasbg.fr

Documents