o ak r idge n ational l aboratory u. s. d epartment of e nergy 1 enabling supernova computations by...
TRANSCRIPT
1
OAK RIDGE NATIONAL LABORATORYU. S. DEPARTMENT OF ENERGY
Enabling Supernova Computations by Integrated Transport and Provisioning Methods
Optimized for Dedicated Channels
Nagi Rao, Bill Wing, Tony MezzacappaOak Ridge National Laboratory
Malathi VeeraraghavanUniversity of Virginia
DOE MICS PI Meeting: High-Performance Networking ProgramSeptember 14-16, 2004
Fermi National Laboratory
OAK RIDGE NATIONAL LABORATORYU. S. DEPARTMENT OF ENERGY
Outline
Background ORNL Tasks
Preliminary Results
UVA Tasks Preliminary Results
OAK RIDGE NATIONAL LABORATORYU. S. DEPARTMENT OF ENERGY
Terascale Supernova Initiative - TSI
Science Objective: Understand supernova evolutions DOE SciDAC Project: ORNL and 8 universities Teams of field experts across the country collaborate on
computations Experts in hydrodynamics, fusion energy, high energy
physics Massive computational code
Terabyte/day generated currently Archived at nearby HPSS Visualized locally on clusters – only archival data
Current Networking Challenges Limited transfer throughput
Hydro code – 8 hours to generate and 14 hours to transfer out
Runaway computations Find out after the fact that parameters needed adjustment
OAK RIDGE NATIONAL LABORATORYU. S. DEPARTMENT OF ENERGY
Data and File Transfers (terabyte – petabyte) Move data from computations on supercomputers Supply data to visualizations on clusters and supercomputers
Interactive Computations and Visualization Monitor, collaborate and steer computations Collaborative and comparative visualizations
Visualization channel
Visualization control channel
Steering channel
TSI Desired Capabilities
Computation orvisualization
OAK RIDGE NATIONAL LABORATORYU. S. DEPARTMENT OF ENERGY
Background on NSF CHEETAH Project Circuit-switched High-speed End-to-End Transport
arcHitecture (CHEETAH) Team: UVA, ORNL, NCSU, CUNY Concept:
Share bandwidth on a dynamic call-by-call basis End-to-end circuit:
Ethernet - Ethernet over SONET - Ethernet Network
Second NICs at hosts in a compute cluster/viz cluster Connected to MSPPs that perform Ethernet-SONET mapping GMPLS-enabled SONET crossconnects
Transport protocols and middleware To support file transfers on dedicated circuits To support remote visualization and computational steering
Applications to support TSI scientists SFTP Ensight + new visualization programs
OAK RIDGE NATIONAL LABORATORYU. S. DEPARTMENT OF ENERGY
Current DOE ORNL-UVA Project:Complementary Roles
•Project Components:•Provisioning for UltraScience Net - GMPLS•File transfers for dedicated channels•Peering – DOE UltraScience Net and NSF CHEETAH•Network optimized visualizations for TSI•TSI application support over UltraScience Net + CHEETAH
Peering
ORNL UVA
VisualizationTSI Application
ProvisioningFile Transfers
This project leverages two projects•DOE UltraScience Net•NSF CHEETAH
OAK RIDGE NATIONAL LABORATORYU. S. DEPARTMENT OF ENERGY
Peered UltraScienceNet-CHEETAH
CERN
Chicago
Sunnyvale
Atlanta
ANLFNAL
ORNL
CalTech
SLAC
LBL
NERSC
PNNL
10 Gbps
10 Gbps
DOE Science UltraNet + NSF CHEETAH
Seattle
BNL
JLab
University
DOE National Lab
Future Connections
UltraNetCHEETAH
UVa
NCSU
CUNY
Enables coast-to-coast dedicated channels
Phase I: TL1-GMPLS cross conversion
Phase II: GMPLS-based
OAK RIDGE NATIONAL LABORATORYU. S. DEPARTMENT OF ENERGY
ORNL: Year 1 Activities
• Peering CHEETAH - UltraScienceNet• Visualization
• Decomposable visualization pipeline• Analytical formulation• First implementation
• TSI support• Monitoring Visualizations
OAK RIDGE NATIONAL LABORATORYU. S. DEPARTMENT OF ENERGY
ORNL Personnel
Conference Papers• M. Zhu, Q. Wu, N. S. V. Rao, S. S.Iyengar, “Adaptive Visualization Pipeline Partition and
Mapping on Computer Network”, International Conference on Image Processing and Graphics, ICIG2004.
• M. Zhu, Q. Wu, N. S. V. Rao, S. S.Iyengar, “On Optimal Mapping of Visualization Pipeline On Optimal Mapping of Visualization Pipeline onto Linear Arrangement of Network Nodes”, International Conference on Visualization and onto Linear Arrangement of Network Nodes”, International Conference on Visualization and Data Analysis, 2005Data Analysis, 2005
Publications
Nagi Rao, Bill Wing, Tony Mezzacappa (PIs)
Qishi Wu (Post-Doctoral Fellow)
Mengxia Zhu (Phd Student – Louisiana State Uni.)
OAK RIDGE NATIONAL LABORATORYU. S. DEPARTMENT OF ENERGY
Modules of Visualization Pipeline
Visualization Modules Pipeline consists of several modules Some modules are better suited to certain network nodes
Visualization clusters Computation clusters Power walls
Data transfers between modules are of varied sizes and rates
Note:Commercial tools do not support efficient decomposition
filtering
transformation(topological surface
construction, volumetrictransfer function)
renderingframebufferfiltered data
transformed data(geometric model,volumetric values)raw data
Datasource
Display
OAK RIDGE NATIONAL LABORATORYU. S. DEPARTMENT OF ENERGY
Grouping Visualization Modules
Grouping Decompose the pipeline into modules Combine the modules into groups
Transfers on single node are generally faster Between node transfers take place over the network
Align bottleneck network links between modules with least data requirements
OAK RIDGE NATIONAL LABORATORYU. S. DEPARTMENT OF ENERGY
Optimal Mapping of Visualization Pipeline:Minimization of Total Delay
Dynamic Programming Solution Combine modules into groups Align bottleneck network links between modules with least
data requirements Polynomial-time solvable – not NP-complete
1
11 ,
( ) ,
( )( ) min
min ( ( ) )
m m m
vm
mm ton v V m m m m
u adj v v u v
c mT v pT v
c m c mT u p b
Note: 1. Commercial tools (Ensight) are not readily amenable to optimal
network deployment2. This method can be implemented into tools that provide appropriate
hooks
( )O n E
[ ], [ 1]
1 1
11 1 1 2 1[ ] [ ], [ 1]
( )1( )
i P i P i
i
q q q qi
total computing transport G L j ji i i j G and j iP i P i P i
m GT Path P of q nodes T T T T c m
p b
OAK RIDGE NATIONAL LABORATORYU. S. DEPARTMENT OF ENERGY
Optimal Mapping of Visualization Pipeline:Maximization of Frame Rate
Dynamics Programming Solution Align bottleneck network links between modules with least
data requirements Polynomial-time solvable – not NP-complete( )O n E
[ ], [ 1]
1,2, , 1
12[ ]
[ ], [ 1]1,2, , 1
12[ ]
( )
max ( ), ( ), ( )
1,
( )max ,
1
i
q
bottleneck
computing i transport P i P i computing qPath P of q nodesi q
j jj G and jP i
i
Path P of q nodesP i P ii q
j jj G and jP q
T Path P of q nodes
T G T L T G
c mp
m G
b
c mp
(2)
11
1 ,1
( ) ,
max ( ), ,
( ) min (6)
min max ( ), ,
i
i
i i i
mi m mm
iv
mi
m ton v Vm m m m
u adj v v u v
GS v c mF v p
F vc m mF u p b
OAK RIDGE NATIONAL LABORATORYU. S. DEPARTMENT OF ENERGY
First Implementation
Client/Server OpenGL implementation (leveraged from CHEETAH) Case 1: small cube geometry or frame-buffer Case 2: small geometry Case 3: small geometry CT scan: raw image or frame-buffer
Computer
Computer
LSU
NCSU
ORNL
Headnode
Slavenode
Slavenode
Slavenode
Slavenode
Dimension
Estimated bandwidth
Minimum delay
Raw data size/delay
Geometry size/delay
FB size/delay
Cube1
10x6x8 0.284Mbps 0.032sec 8 K / 0.257sec 1K / 0.032sec 1.8M/50.73sec
Cube2
50x20x39 0.300Mbps 0.034sec 610K / 16.3sec
16K / 0.46sec 1.8M/48.03sec
Cube3
150x210x139
0.277Mbps 0.033sec 71.6M / 34.4min
2.4M / 69.34sec
1.8M/52.01sec
Hand 256x256x80
0.239Mbps 0.033sec 81.9M / 45.69min
NA 1.8M/60.28sec
OAK RIDGE NATIONAL LABORATORYU. S. DEPARTMENT OF ENERGY
Requirements Light-weight server located at the computation site Remote client provides constant monitoring of variables
Our first implementation OpenGL server and client Client
Geometric operations Point, iso-surface, vector view
Commercial Visualization tools Not light weight – server on supercomputers Expensive – collaborative visualization by team Not optimized for network deployment
Monitoring Visualization
OAK RIDGE NATIONAL LABORATORYU. S. DEPARTMENT OF ENERGY
ORNL: Year 2 Activities
• MPLS Peering CHEETAH• Visualizations
• Computational Monitoring• Collaborative Visualization
• TSI support• Collaborative Steering• Integrated Data Transfers