o ak r idge n ational l aboratory u. s. d epartment of e nergy doe-ultrascience net (& network...

OAK RIDGE NATIONAL LABORATORYU. S. DEPARTMENT OF ENERGY

DOE-UltraScience Net (& network infrastructure) Update

JointTechs MeetingFebruary 15, 2005

W. R. Wing


Motivation - DOE Needs Extreme Networking 160-200 Gbs throughput by 2008

Probably only achievable by circuit bonding

0.1% packet re-order and jitter control Probably only achievable at SONET layer

Line rate, provably-secure, connections Only available at SONET or optical layer


The Well Known Bandwidth Problem

DOE needs 160 - 200 Gbs by 2008 Typical multi-stream throughput limited to

25+/- Gbs, and programming multi-steam is hard - hero efforts yield ~30 Gbs

Typical single-steam throughput to a single application ~1Gbs, even on a tuned network - hero effort yields 5-6+ Gbs

Networks will soon offer 40 Gbs channels Firewalls throw it away anyway


Even Without Firewalls -We Have a Technology Problem

• DOE needs 160-200 Gbs by 2008• This is on the “blue line”

• This isn’t a problem routers are going to fix - bonded IP channels ONLY work with VERY large numbers of streams (as Teragrid has discovered)• Bonding at layer-2 (Ethernet) is just as bad• Solution is circuit switching and Layer-1 bonding


Jitter Control

Needed to deconstruct data sets, needed for remote vis, and for remote instrument control or steering

Routers and switch-routers are path deterministic, but not time-deterministic

Only at layer-1 (or SONET) do you get time-determinism


Security

Security requirements will get more and more onerous

Solutions involving IPSec, VPN’s, and encryption have more and more trouble running at line speed

Alternate solution is to use optical/SONET circuits with provable, known end-points and inherent immunity to injected traffic

Again - only available at layer-1


In Summary Only at layer-1 do you get:

Zero packet re-ordering Zero jitter Zero drops due to Congestion Known (by definition) paths and end points

SONET (or below) can do this Ethernet switches and switch/routers can’t MPLS can’t

Layer-1 circuits can: Bypass firewalls Carry non-IP frames (e.g., Fiber-Channel over SONET) Easily (transparently) support parallel, bonded circuits


MPLS Jitter - ORNL / Atlanta


How do we get there from here?

It requires a research network we control at least down to the SONET layer

It requires a research network with significant span

It requires a research network with at least two lambdas


UltraScience Net Research Testbed

Building an extended-regional lambda-switching testbed

Connect to NLR in Atlanta and Chicago Use asset-trading to extend reach (Sunnyvale and

East Coast) Provide an evolving matrix of switching

capabilities Separately fund research projects (e.g., high-

performance protocols, control, visualization) that will exercise the network and directly support applications at the host institutions


In More Detail -C

hic

ag

oC

hic

ag

o

Sc

hn

eid

er

Sc

hn

eid

er

Mo

no

nM

on

on

Cra

wfo

rds

vil

leC

raw

ford

sv

ille

Ind

ian

ap

oli

sIn

dia

na

po

lis

Bo

wli

ng

Gre

en

Bo

wli

ng

Gre

en

Na

sh

vil

leN

as

hv

ille

Ch

ris

tia

na

Ch

ris

tia

na

Se

ym

ou

rS

ey

mo

ur

Lo

uis

vil

leL

ou

isv

ille

Up

ton

Up

ton

Ste

ve

ns

on

Ste

ve

ns

on

Ch

att

an

oo

ga

Ch

att

an

oo

ga

Da

lto

nD

alt

on

Ac

wo

rth

Ac

wo

rth

Atl

an

taA

tla

nta

OLAOLA

KnoxvilleKnoxville

ORNLORNL

2x10GbE ORNL TeraGRID2x10GbE ORNL TeraGRID

OLA - 2OLA - 2

MemphisMemphis

OLA -1OLA -1

2xO

C19

2 –

UT

Ter

aGR

ID2x

OC

192

– U

T T

eraG

RID

2xOC192 – UT TeraGRID

2xOC192 – UT TeraGRID

2x10

Gb

E –

UT

Ter

aGR

ID2x

10G

bE

– U

T T

eraG

RID

T640T640

T320T320

T640T640

CICI

CDCD

CICI

2xOC192 – ORNL TeraGRID2xOC192 – ORNL TeraGRID 2x10GbECD bypass for future 10GbECD switching

2x10GbECD bypass for future 10GbECD switching

20 x Gig-E???20 x Gig-E???

ONLINE M

etro

???`

ONLINE M

etro

???`

ORNL TeraGRIDORNL TeraGRID

UT TeraGRIDUT TeraGRID

N x

10

Gb

E

N x

10

Gb

E

N x 10GbE

N x 10GbE

N x 10GbE, UltranetN x 10GbE, Ultranet

TN.eduTN.edu

F10F10

GbE / 10GbEGbE / 10GbE

Gb

E /

10

Gb

E

Gb

E /

10

Gb

E

Gb

E /

10

Gb

E

Gb

E /

10

Gb

E

Gb

E /

10

Gb

E

Gb

E /

10

Gb

E

GbE / 10GbEGbE / 10GbE

OC192’sOC192’s


Connections to National Nets

ORNL

ORNL-TVA

Internet2

NLR

ORNL

ESnetTeraGrid


Research Projects A progression of switching approaches

Study/compare SONET, MPLS(GMPLS), and all optical New all-optical technologies coming (e.g., laser tuning)

Fast Local Storage Storage Depots Transport-optimized storage

Progression of experimental point-to-point transport technologies Fiber channel Infiniband LAN-PHY Ethernet initial transport

Circuit-Switched backbone, Frame-Switched edge


Digression: LAN-PHY vs. WAN-PHY

Conventional Wisdom: LAN-PHY will win Cheaper, faster, etc. (based on POS experience) False cost model (extra processing) Ignore SONET advantages

New Transponders are transparent Software selectable OC192 (WAN–PHY) or LAN–

PHY Costs now equal Advantage now WAN-PHY

DCC and OAM non-trivial considerations


PoS vs. Ethernet

DATAIP IPPOS PAD POS

DATAIP IPE EPAD

DATAIP IP

DATAIP IPPOS PAD POSSONET SONET

DATAIP IPE EPAD

DATAIP IP

DATAIP IPE EPADSONET SONET


Status and Schedule

All contracts signed Fiber, Co-Lo space and power, equipment and

installation, smart-hands Equipment shipping, built-out started

Chicago-Sunnyvale paced by NLR cross-connects in Chicago (Level(3) to Starlight) NLR supplying 10Gig-E (LAN-PHY) initially PNNL fiber schedule will pace their connection

First Chicago-Sunnyvale traffic this month Details on following VuGraphs -


Phase-1 (February)

Force10E600

Sunnyvale

Chicago

Force10E600

Seattle

10G WanPHY

OC-192 SONET

OC-192 SONET (integrated optics)

10G LanPHY

1G Ethernet


Phase-2 (February-March)

VL2020CienaCD-CI

Force10E600

Sunnyvale

Chicago

Atlanta

CienaCD-CI

Force10E600

CienaCD-CI

SycamreSN16000

CienaCD-CI

SycamreSN16000

TurinTR2000

Seattle

ORNL

10G WanPHY

OC-192 SONET


10G LanPHY

1G Ethernet

N x 1Gig-E

ORNLM160

SOXM320


Phase-3 (March - April)

VL2020CienaCD-CI

Force10E600

Sunnyvale

Chicago

Atlanta

CienaCD-CI

Force10E300

Force10E600

CienaCD-CI

SycamreSN16000

Force10E300

SycamreSN16000

TurinTR2000

Seattle

ORNL

10G WanPHY

OC-192 SONET


10G LanPHY

1G Ethernet


Phase-4 (April - May)

Force 10E300

Sunnyvale

Chicago

Atlanta

CienaCD-CI

CienaCD-CI

CienaCD-CI

SycamreSN16000

SycamreSN16000

CienaCD-CI

Force 10E300

Force 10E300

Force 10E300

VL2020

10G WanPHY

OC-192 SONET


10G LanPHY

1G Ethernet

Seattle

PNNL

ORNL

TurinTR2000


Thank You

http://www.csm.ornl.gov/ultranet

o ak r idge n ational l aboratory u. s. d epartment of e nergy doe-ultrascience net (& network...

Documents