the doe ultrascience network (aka ultranet)

OAK RIDGE NATIONAL LABORATORYU. S. DEPARTMENT OF ENERGY

The DOE UltraScience Network(aka UltraNet)

GNEW2004

March 15, 2004

Wing/Rao


Talk Outline

What are we trying to do? Why do it? How (we all sort of know don’t we)? How can we get there from here? …Where will that be?


DOE Science

DOE has bigger problems (drivers) … (and less money) than any other three-letter agency

But with it, we are trying to do things like: Climate Modeling Supernova modeling Genomics (and proteomics)


Climate Modeling

North American model at 400Kmresolution

North American model at 60Kmresolution

The Rockies block the atmospheric circulation and are importantgeneses regions for the weather systems that form the North AtlanticStorm Track. They are poorly represented in low resolution models.


Type II CoreCollapse Supernova

QuickTime™ and aVideo decompressor

are needed to see this picture.


Molecular dynamics of a lipopolysaccharide (LPS)

Nanosecond molecular dynamics of the LPS membrane of Pseudomonas aeruginosa

QM/MM molecular dynamics of membrane plus mineral

Computational Molecular BiologyComputational Molecular BiologyMembrane with possible uses for bioremediationMembrane with possible uses for bioremediation

.25TF

11TF

Energy transduction across membranes is still out of reach at 1.0PF

400 TF>1.0 PF


Billions of Bases in GenBank

1982 1986 1990 1994 1998 2002

According to the GOLD database, there are 146 published genomes, 344 prokaryotic ongoing genomes projects, and 243 eukaryotic ongoing genome projects.


Growth of Proteomic Data vs. Sequence Data

0.00001

0.0001

0.001

0.01

0.1

1

10

100

1000

198819901992199419961998200020022004200620082010201220142016

Years

PetaBytes

Proteomic data

GenBank


PetaScale Science Climate modeling today => 50-100 TB data sets

Still adding carbon chemistry Still refining resolution

You saw DOE Road Map table showing PB requirements in 2-3 years

Supernova modeling just going from 2 1/2 dimensions to 3D - this is a PB problem today

Genomics and Proteomics will dwarf these …and then of course there is High Energy

…but


We run hard into network limits…

There is some VERY nice work being done on TCP But that will just emphasize router delay And expose congestion, even where it coexists with it

There is nice work being done to extend MPLS But that does not eliminate jitter And imposes additional overhead

And there is still the tyranny of layers


Jitter - ORNL / Atlanta


2.5 Gbps

Science Applications - Bit Rate BudgetEnd-to-end

40 Gbps

Optical Backbone Host

SONET(One Channel)

Site SecurityFirewall

10 Gbps

5.0 Gbps

1.0 Gbps

TCP/IP(One Flow)

DWDM(One

Lambda)

IP Network Services

OC-768

100 Gbps

Research Opportunities

Bit-Rate

OC-48

OC-192

GigE

Apps

ESnet.65 Gbps

End-to-end Performance

Gap

ESnet

SITE LAN

Host

ESnet Cluster

One Host

ESnet


What are we trying to do?

We are trying to break out of the limitations of conventional networks TCP/IP Firewalls, and layers Small-frame technologies

Take advantage of the real throughput available in optical networks today at the SONET layer

Develop the techniques needed within 2-3 years for DOE Peta-scale science

Use switched-circuits as a solution we can start on today


Restating the Problem -

Networks will soon offer 40 Gbs channels Typical multi-stream throughput limited to

5-6Gbs, and programming multi-steam is hard - hero efforts yield 9-10Gbs

Typical single-steam throughput to a single application ~1Gbs, even on a tuned network - hero effort yields 5-6Gbs

Firewalls throw it away anyway


Solution - We won’t have line-rate optical packet-switching anytime

soon No photonic transistors No photonic memory buffers

We can do optical circuit switching Useless for most commercial networks

Doesn’t aggregate for economy of scale But bulk P2P purchases can be extremely attractive

A good match for DOE high-performance science applications which require both bandwidth and precision Petabyte file transmission in 2-3 years Tera-scale interactive visualization ASAP Remote real-time instrument control now


UltraNet Research Testbed

Build a sparse, lambda-switching, dedicated channel-provisioning testbed

Connect hubs close to DOE’s largest Science users (but let the user labs pay last-mile costs)

Provide an evolving matrix of switching capabilities

Separately fund research projects (e.g., high-performance protocols, control, visualization) that will exercise the network and directly support applications at the host institutions


The Resources:

Off hours capacity from Esnet Probably 2 x OC48 between Sunnyvale and

Chicago in the northern route

Dedicated Lambdas on NLR Two 10G lambdas between Chicago and

Sunnyvale Possibly more in year two or three

Two dedicated lambdas on the ORNL Chicago-Atlanta connector


Resources Cont.

A progression of switching technologies Start with Ciena, Cisco, or Sycamore Migrate to Calient, Glimmerglass all-optical or a hybrid

Some Local Storage Logistical Storage Depot (prior to local application

development) Progression of experimental point-to-point

transport technologies (plus legacy) Fiber channel Infiniband

Migrate to the Production ESnet environment


CERN

StarLight

Sunnyvale

SOX

ANLFNAL ORNL

CalTech

SLAC

LBLNERSC

PNNL2 x10 Gbps

2 x10 Gbps

Seattle

BNL

JLab

Research Exchange Point

DOE University Partners

DOE National LabAccess via production networks

Research 10 Gbps lambda

Science UltraNet Phase I


The Physical View From 50,000 feet

NLR

ESnet

CERN etc.

ORNL Connector


Network OverviewC

hic

ag

oC

hic

ag

o

Sc

hn

eid

er

Sc

hn

eid

er

Mo

no

nM

on

on

Cra

wfo

rds

vil

leC

raw

ford

sv

ille

Ind

ian

ap

oli

sIn

dia

na

po

lis

Bo

wli

ng

Gre

en

Bo

wli

ng

Gre

en

Na

sh

vil

leN

as

hv

ille

Ch

ris

tia

na

Ch

ris

tia

na

Se

ym

ou

rS

ey

mo

ur

Lo

uis

vil

leL

ou

isv

ille

Up

ton

Up

ton

Ste

ve

ns

on

Ste

ve

ns

on

Ch

att

an

oo

ga

Ch

att

an

oo

ga

Da

lto

nD

alt

on

Ac

wo

rth

Ac

wo

rth

Ata

lnta

Ata

lnta

CICI

SOADM conversionSOADM conversion

2 x OC192CHI – ATL

Tx ETF

2 x OC192CHI – ATL

Tx ETF

OLA1OLA1

KnoxvilleKnoxville

OakridgeOakridge

CICI

2 x OC192CHI – ORNL

Ultranet

2 x OC192CHI – ORNL

Ultranet

OLA2OLA2OLA - 2OLA - 2

MemphisMemphis

OLA -1OLA -1

OLA - 3OLA - 3

20 x Gig-E???20 x Gig-E???

GbE / 10GbE / OC102GbE / 10GbE / OC102

Intro 10GLAN PHY – CDIntro 10GLAN PHY – CD

10GbEOC192

10GbEOC192

10GbEOC192

10GbEOC192

10GbEOC192

10GbEOC192

10GbEOC192

10GbEOC192

OC192 10GbE

OC192 10GbE

10GbEOC192

10GbEOC192

10GbEOC192

10GbEOC192


The Logical View

Dedicated Lambdas, NLR

Dedicated Lambdas, NLR

Scheduled Lambdas, ESnet

MSPP

MSPP

MSPP SONET switch

SONET ADM

MSPP

MSPP

ORNL

Chicago

Atlanta

PNNL

Sunnyvale

SONET ADM

SONET switch


Schedule

Oak Ridge Connector to be up in June Fall-back circuits via NLR if necessary

Chicago-Sunnyvale paced by NLR PNNL fiber schedule will pace their connection

Expect first user-traffic in June


Future Plans

ORNL is half the team building CHEETAH Nagi Rao and Malathi Veeraraghavan are PI’s Funded by NSF Will Connect ORNL, Atlanta, NC State, possibly CUNY Explicitly developing signaling protocols

Want to partner with HOPI …and all of you


Thank You

the doe ultrascience network (aka ultranet)

Documents