visualization challenges for large scale astrophysical

Ultrascale Visualization WorkshopNovember 13, 2011

Seattle, WA

Ralf Kähler (KIPAC/SLAC)

Tom Abel (KIPAC/Stanford)Marcelo Alvarez (CITA)Oliver Hahn (Stanford)

Hans-Christian Hege (ZIB)Ji-hoon Kim (KIPAC)

Stuart Marshall (SLAC) Matthew Turk (Columbia) Risa Wechsler (Stanford)John Wise (Georgia Tech)

Visualization Challenges for Large Scale Astrophysical Simulation Data

COMPUTATIONAL ASTROPHYSICS

COMPUTATIONAL PHYSICS DEPARTMENT

Evolution of First Galaxies

Re-ionization EpochLarge-Scale Structure Formation

Evolution of First StarsSimulation: Wise & Abel

Simulation: Wu, Hahn & Wechsler

Simulation: Kim & Abel

Simulation: Alvarez & Abel

Large-Scale Structure FormationSimulation: Wu, Hahn & Wechsler

Evolution of First Stars

Evolution of First GalaxiesSimulation: Kim & Abel

Simulation: Turk & Abel

Reionization EpochSimulation: Alvarez & Abel

Numerical Astrophysics

High dynamical range



Formation of First Stars

- Protogalaxy > 10,000 light years

Simulation: T. Abel



Formation of First Stars

- Protogalaxy > 10,000 light years

- Protostar ~ 10 solar radii

Simulation: T. Abel


> 10 spatial orders of magnitude

Simulation: T. Abel

STRUCTURED ADAPTIVE MESH REFINEMENT

Simulation Data: Tom Abel

TEMPORAL REFINEMENT

AMR schemes perform refinement in time

Stability condition for solvers:

- Δt proportional to Δx

- global Δt determined by smallest Δx

➔ Computational overhead on coarse levels

TEMPORAL REFINEMENT

0 1 2 3 4 TIME

Level 0

Level 1

Level 2

enzo - Adaptive Mesh Refinement

enzo

- Astrophysical Adaptive Mesh Refinement code

- Cosmological structure formation simulations

http://code.google.com/p/enzo/



DATA SIZES

10-40 levels of refinement

10,000-100,000 time steps

up to 100 million patches

- less than 32^3 cells per patch

up to billions of cells per time step

- different fields ( gas density, gas temperature, etc. )

10s of TByte of data

VISUALIZATION SOFTWARE

C++, JavaScript

Graphical User Interface (Qt)

OpenGL

Available as Open Source soon

Simulation data: Kim & Abel



Modular Design: add vis-routines & readers

Readers for Enzo, Gadget, RAMSES


Modular Design: add vis-routines & readers

Readers for Enzo, Gadget, RAMSES

Data nodes:Grid-based (AMR)Point-based (unstructured)Image data Simulation data: Kim & Abel


Visualization methods:



Real-Time Rendering:OpenGL & OpenGL Shading Language

Customize rendering at run-time by modifying OpenGL Shaders



Visualization methods:SlicingDirect Volume RenderingStreamlines...



Simulation Data: Mike Norman



Simulation Data: John Wise & Tom Abel



Simulation Data: Ji-hoon Kim & Tom Abel



Simulation Data: Wu, Hahn & Wechsler

Pre-Rendering for American Museum of Natural History Show “The Big Bang”, Narrated by Liam NeesonSimulation data: Ji-hoon Kim (Stanford), Tom Abel (Stanford/SLAC)

DATA ACCESS PATTERNS

0 1 2 3 4 TIME

Level 0

Level 1

Level 2

Generation of Proxy-Grid Structure

Time A Time B

Generation of Proxy-Grid Structure

DATA STORAGE

HDF5 for data I/O

- http://www.hdfgroup.org/HDF5/

Enzo HDF5 output

- each processor writes stream of separate HDF5 groups

- no spatial or temporal relations

GROUP "grid-0" { ATTRIBUTE "origin" { ... } ATTRIBUTE "dims" { .... } ATTRIBUTE "level" { .... } ... DATASET "Density" { ... } DATASET "Temperature" { ... } ...}


GROUP "grid-n" { ATTRIBUTE "origin" { ... } ATTRIBUTE "dims" { .... } ATTRIBUTE "level" { .... } ... DATASET "Density" { ... } DATASET "Temperature" { ... } ...}...

http://www.hdfgroup.org/HDF5/




HDF5 INDEX FILE

Post-processing:

- create HDF5 index file

- match the AMR grid structure

HDF5 "./rho.a5" {

GROUP "/" {

GROUP "globalMetaData" { DATASET "times" { ... } DATASET "timesteps" { ... }

... }

GROUP "time-0" { GROUP "level-0" {

ATTRIBUTE "predecessor" { ... } ATTRIBUTE "successor" { ... } } GROUP "level-1" { GROUP "grid-0" { ATTRIBUTE "origin" { ... } ATTRIBUTE "dims" { ... } ATTRIBUTE "data_reference" { ... } … } GROUP "grid-1" { ATTRIBUTE "origin" { ... }

ATTRIBUTE "dims" { ... } ATTRIBUTE "data_reference" { ... } ... } ... } GROUP "level-2" { ... } ... } GROUP "time-1" {

...

}

HDF5 INDEX FILE

HDF5 "./rho.a5" {

GROUP "/" {


... }




...

}

available time steps

HDF5 INDEX FILE

time groups

HDF5 "./rho.a5" {

GROUP "/" {


... }




...

}

HDF5 INDEX FILE

level groups

HDF5 "./rho.a5" {

GROUP "/" {


... }




...

}

HDF5 INDEX FILE

grids on this level

HDF5 "./rho.a5" {

GROUP "/" {


... }




...

}

HDF5 INDEX FILE

grid meta data

HDF5 "./rho.a5" {

GROUP "/" {


... }




...

}

HDF5 INDEX FILE

previous & next time steps

HDF5 "./rho.a5" {

GROUP "/" {


... }




...

}

HDF5 INDEX FILE

links to datasets

HDF5 "./rho.a5" {

GROUP "/" {


... }




...

}

HDF5 INDEX FILE

HDF5 "./rho.a5" {

GROUP "/" {


... }



ATTRIBUTE "dims" { ... } ATTRIBUTE "data_reference" { ... } ... }



GROUP "grid-n" { ATTRIBUTE "origin" { ... } ATTRIBUTE "dims" { .... } ATTRIBUTE "level" { .... } ... DATASET "Density" { ... } DATASET "Temperature" { ... } ...}...

links to datasets

POSTGRES DATABASE

Alternative:

- Index-structure via databases

Meta data in database, data (fields) in original HDF5 files

QtSQL Module: http://doc.qt.nokia.com/4.7/qtsql.html

PostgreSQL plugin

http://doc.qt.nokia.com/4.7/qtsql.html

http://doc.qt.nokia.com/4.7/qtsql.html

COMPARISON

File sizes

- Postgres database ~ 20% of HDF5 index file

Metadata access

- Postgres twice as fast as HDF5

GPU-Assisted Ray Casting

[Stegmaier, et al. 2005]

Input data ➔ 3D texture

Render front faces

Fragment Shader for each covered pixel:

Compute ray-direction

Sampling and color mapping

Resulting intensities --> frame buffer

GPU-Assisted Ray Casting for AMR Data

Problem: Overlapping regions

➤ Decomposition of data domain

Adaptive kD-tree

- Nodes represent of non-overlapping blocks of cells

- View-dependent sorting (front-to-back or back-to-front)

GPU-Assisted Ray Casting for AMR Data

View-dependent selection of nodes:

- Resolution level based on distance to viewpoint

~200,000 nodes ~20,000 nodes

Simulation data: John Wise (Georgia Tech), Tom Abel (Stanford/KIPAC)

Pre-Rendering for California Academy of Sciences, Dome Show “Life: A Cosmic Journey”, Narrated by Jodie Foster, November 2010Simulation data: John Wise (Princeton), Tom Abel (Stanford/SLAC)

POINT-BASED DATASETS

Unstructured point datasets

- Dark matter density

- Single stars and star clusters

- ...

Point attributes

- Position

- Mass

- Age

- Accretion rate

- ...

Simulation: Wu, Hahn & Wechsler

Solution for “opaque” point representation

1. Pass

Enable depth-buffer updates

Render points

2. Pass

Perform ray-tracing up to pixels depth value

3. Blending step

GPU-Raycasting of Combined Grid- and Point-based Data


Usually semi-transparent point representation

gaussian opacity profile for galaxies, stars, DM, etc.

Potential solution:

Resampling of point data to grid structure

Simultaneous rendering of both (grid) data sets


Requires highly resolved grids

High (texture-)memory consumption

and/or sacrifices data resolution

Low-pass filtering

Rendering artifacts

GPU-Raycasting ofCombined Grid- and Point-based Data

• Point data ➤ Octree structure- Efficient GPU-representation- Implicit bounding-box information

• Recursive refinement of octree nodes

• Stopping criteria- Number of inserted points < threshold

Combining Grid and Point Data

Node texture

3D-RGBA texture

One texel per node

Alpha-channel stores node type

Internal node Leaf node

RGB-channel

Index of first child node Index into data texture


Combining Grid and Point Data

Data texture

3D-RGBA floating point

RGB-channels: center

Alpha-channel: radius

Radius=0 indicates end of list



1. Grid data ➤ 3D-texture

2. Point set ➤ two 3D-textures (GPU-octree representation)

3. Render front faces of bounding box

4. In Fragment shader

• Compute ray-direction

• Sampling of grid-based data

• Sampling of point-based data

• Combination of partial intensities

• Combination with total intensity

5. Resulting intensity ➤ frame-buffer

Combined Point & Grid Raycasting

Summation of Intensities

• Relation between opacity and extinction coefficient

• Combination of opacities for segment si

• Combination with accumulated intensity/opacity of ray


Performance Optimizations

• Example: Gaussian profiles



• ROI around camera location



• Point and grid data inside ROI rendered with GPU approach



• Outside ROI: - Process grid data until depth of first pass- blending with point splats



• Correct result if only one splat hit by ray outside ROI



• Partially incorrect depth sorting for more than one splat• Artifacts usually not visible in far-field


Comparison of Rendering Quality

ROI = 0% of volume ROI = 25% of volumeROI = 100% of volume



Numerical Simulation: Marcelo Alvarez (CITA), Tom Abel (KIPAC/Stanford)

Visit us as the SLAC booth (303) and watch some visualizations in 3D !

visualization challenges for large scale astrophysical

Documents