High Performance HybridOptical-Packet Networks:

Developments and Potential Impacts

SubOptic 2007May 15, 2007

Baltimore, MD

Dr. Don Riley, Professor, University of Maryland

SURA IT Fellow; Chair, IEEAF

Jerry Sobieski, Director of Network Research

Mid-Atlantic Crossroads (MAX)

2

Overview of Global R&ENetworking Direction

• The international research and education community hasmounted significant efforts to develop a new generation ofnetworking capabilities that have at the core infrastructureowned/controlled, provisioned and managed by the R&Ecommunity:– all-optical with multiple point-to-point wavelengths, currently at

the 10Ge level.– Services provisioned across these backbones are multi-layer,

multi-service in nature, including Layer 1, 2 and 3 services.– Efforts include “control plane” issues of how you manage and

monitor provision of “hybrid” services across multiple servicelayers, across multiple network domains, often on an internationalbasis.

3

Needs of Global R&E Collaborations areDriving the Networking Development

• The need for such new services is driven by theapplications community - the research community that istypically a globally-distributed collaboration.

• U.S. “CyberInfrastructure” one such effort focused on theinfrastructural needs, and is now a new program office ofthe National Science Foundation.

• New research collaborations include the need to movemassive amounts of data in real time, control expensiveand unique equipment, etc. Example projects include:– the high energy physics community’s Large Halon Collider

(LHC) project coming on line next year at CERN in Geneva, theextended very long baseline interferometry (eVLBI) radioastronomy project with antenna arrays located around the world.

– The eVLBI (very long base-line interferrometry) internationalcollaboration in radio astronomony

4

Overview of Global R&ENetworking Direction

• This tutorial will provide some insight into thesedevelopments and the driving motivations – and potentialfuture developments and possible implications for thesubmarine cable industry.

• The presentation will include overviews of GLIF andDRAGON objectives and technology developments, andother similar, related projects and initiatives.

• One goal of this tutorial is to stimulate discussion of areasof needed development/collaboration regarding submarinesystems technology - so that these important links betweenglobal R&E communities do not become the “disconnect”.

Internet2 Today

Internet2 HOPI Testbed Topology usingDRAGON Technology ~ 2006, showing overlaywith NLR Backbone

Current Abilene Backbone Topology Abilene Weather Map – Jan 23, 2007

Abilene Logical Map – January 2007

Internet2 Tomorrow - “NewNet”

to replace Abilene summer 2007

Internet2 Tomorrow - “NewNet”Internet2 DWDM Layer1 - Optical Circuit Services Internet2 Optical Circuit Services

Internet2 IP Services

Internet2 HOPI Testbed Topology usingDRAGON Technology

NLR Today

NLR Topology, showing also POP-territories(gray areas unserved) & key Federal lab sites

9

NLR Today

NLR Layer-1 Wave Availability Jan 23, 2007NLR Layer-2 Network Status Monday, Jan 23,2007

NLR Layer-3 Network Status Jan 23, 2007

CA*net4FranceGLORIAD (Russia, China)Korea (Kreonet2)

Japan (SINet)Australia (AARNet)Canada (CA*net4Taiwan (TANet2)Singaren

ATL

DC

MAE-EPAIX-PAEquinixMAE-West

Connecting DOE Labs to the World’s R&E and Commercial Nets:ESnet’s Domestic, Commercial, and International Connectivity (Spring 2006)

Abilene

Abilene

CERN(USLHCnet

CERN+DOE funded)

GÉANT - France, Germany, Italy, UK, etc

NYC

Starlight C

HI-S

L

Ab

ilene

SNV

SDSC

MAXGPoP

SoXGPoP

High Speed International Connection

Commercial and R&E peering points

Abilene High-speed peering points with Abilene

ESnet core hubs IP SDN

CHI

MRENNetherlandsStarTapTaiwan (TANet2)UltraLight

NGIX-W

Australia

SEA

SINet (Japan)Russia (BINP)

AMPATH(S. America)

AMPATHS. America

MA

N L

AN

Ab

ilen

e

ESnet provides:• High-speed peerings with Abilene, CERN,

and US and international R&E networks• Management of the full complement of global

Internet routes (about 180,000 unique IPv4routes) in order to provide DOE scientists richconnectivity to all Internet sites

Australia

Equinix

Equinix

PNWGPoP/PacificWave

NGIX-E

PacificWave

UNM ALB

SNV

USN

USN

10-50 Gb/s circuitsProduction IP coreScience Data Network coreMetropolitan Area NetworksInternational connections

MetropolitanArea Rings

Primary DOE Labs

IP core hubs

possible hubs

SDN hubs

Europe(GEANT)

Asia-Pacific

New York

Chi

cago

Washington, DC

Atl

anta

CERN

Seattle

AlbuquerqueAu

s.A

ust

ralia

San Diego

LA

Su

nn

yval

e

Denver

South America(AMPATH)

South America(AMPATH)

Canada(CANARIE)

CERN

Loop offBackbone

Canada(CANARIE)

Europe(GEANT)

SDN Core

IP Core

ESnet Target Architecture: IP Core+Science Data Network Core+Metro Area Rings

ESnet3: A National IP Network Built onVarious Circuit Infrastructure

Primary DOE Labs

Backbone hubs

Europe

Asia-Pacific

New York

Chi

cago

Washington, DC

Atl

anta

CERN

Seattle

AlbuquerqueAu

s.A

ust

ralia

10 Gbps circuitsProduction IP coreNLR coreMetro Area NetworksLab suppliedInternational connections

Su

nn

yval

e

AMPATH(S. America)

AMPATH

Canada Russia andChina

Canada CERN

San Diego

Major research andeducation (R&E)network peering points

R&E

R&E

R&E

R&E

R&E

R&E

R&E

ESnet network architecture consists of1) Circuits2) Circuits interconnects

• hubs with routers and switches3) Connected sites4) Connections to other networks

• US R&E, international, and commercial

Qwest supplied10Gbps backbone

NLR supplied10Gbps circuits

2700 miles / 4300 km

1200

mile

s / 1

900

km

ESne

t Sc

ienc

e Dat

a Net

work

(SDN) co

re

TWC

SNLL

YUCCA MT

BECHTEL-NV

PNNLLIGO

INEEL

LANL

SNLAAlliedSignal

PANTEX

ARM

KCP

NOAA

OSTI ORAU

SRS

JLAB

PPPLLab DCOffices

MIT

ANL

BNL

FNALAMES

NR

EL

LLNL

GA

DOE-ALB

OSC GTNNNSA

International (high speed)10 Gb/s SDN core10G/s IP core2.5 Gb/s IP coreMAN rings (≥ 10 G/s)Lab supplied linksOC12 ATM (622 Mb/s)OC12 / GigEthernetOC3 (155 Mb/s)45 Mb/s and less

Office Of Science Sponsored (22)NNSA Sponsored (12)Joint Sponsored (3)

Other Sponsored (NSF LIGO, NOAA)Laboratory Sponsored (6)

42 end user sites

ESnet IP core

SINet (Japan)Russia (BINP)CA*net4

FranceGLORIAD (Russia, China)Korea (Kreonet2

Japan (SINet)Australia (AARNet)Canada (CA*net4Taiwan (TANet2)Singaren

ESnet IP core: Packet overSONET Optical Ring and Hubs

ELP

ATL

DC

commercial and R&E peering points

MAE-E

PAIX-PAEquinix, etc.

PN

WG

Po

P/

PA

cifi

cWav

e

ESnet3 Layer 2 Architecture Provides Global High-Speed Internet Connectivityfor DOE Facilities and Collaborators (spring, 2006)

ESnet core hubs IP

Abilene high-speed peering points with Internet2/Abilene

Abilene

CERN(USLHCnet

CERN+DOE funded)

GÉANT - France, Germany, Italy, UK, etc

NYC

Starlight

Chi NAP

SNV

Ab

ilene

SNV SDN

JGI

LBNL

SLACNERSC

SDSC

Equinix

MA

N L

AN

Ab

ilen

e

MAXGPoP

SNV

ALB

ORNL

CHI

MRENNetherlandsStarTapTaiwan (TANet2, ASCC)

NA

SA

Am

es

AU

AU

SEA

CH

I-SL

LHC Tier 0, 1, and 2 Connectivity Requirements Summary

Denver

Su

nn

yval

e

LA

KC

Dallas

Albuq.

CE

RN

-1G

ÉA

NT

-1G

ÉA

NT

-2C

ER

N-2

Tier 1 Centers

ESnet IP core hubs

ESnet SDN/NLR hubs

Cross connects with Internet2/Abilene

CE

RN

-3

Abilene/GigaPoP nodes

USLHC nodes

ESnetSDN

Abilene / GigapopFootprint

Seattle

FNAL(CMS T1)

BNL(Atlas T1)

New York

Wash DC

Jacksonville

Boise

San DiegoAtlanta

Vancouver

Toronto

Tier 2 Sites

Chicago

ESnetIP Core

TRIUMF(Atlas T1,Canada)

CANARIE

GÉANT

USLHCNet

Virtual Circuits

• Direct connectivity T0-T1-T2

• USLHCNet to ESnet to Abilene

• Backup connectivity

• SDN, GLIF, VCs

15

Esnet4 Plan for 2012 Build-out

16

Quilt Fiber-based RON Map

NSF Funding for InternationalResearch Network Connections

(IRNC)• NSF funding new links from U.S. to International R&E

Networks(2.5 to 10 Gbps “lambdas”)– To Europe– To Asia-Pacific– To South America– To China and Russia– Nothing to African yet

• Purpose is to support US researchers - in their research andtheir international collaborations

One view of NSF IRNC

GLORIAD: Global Ring to China, Korea and RussiaGLORIAD: Global Ring to China, Korea and Russia

To EuropeTo EuropeTo Japan,To Japan,HongKongHongKong,,SingaporeSingapore P-WaveP-Wave

To Hawaii,To Hawaii,AustraliaAustralia

To AustraliaTo Australia

To Latin AmericaTo Latin America

19

GEANT2 & dark fiber

20

European Dark Fiber Efforts

21

CESNET (CZ) Dark Fiber Efforts

22

Dark Fiber in European NRENs

23

GEANT2 & Global Research Networking

International GLIF Initiative:Global Lambda Integrated Facility

www.glif.is Visualization courtesy ofBob Patterson, NCSA.

A globally integrated set of “light path” facilities: optical waves(lambdas - typically 10G), open exchange points, international peerings

GLIFLightpaths

andGOLEs

International: Atlantic Wave

The DRAGON Testbed

28

DRAGON/HOPI Control PlaneProvisioning Environment

• GMPLS Multi-layer, Multi-Domain• Ethernet Service Provisioning• Dynamic dedicated VLAN based connections

EthernetLayer

Switched WDMOptical Layer

DRAGONMulti-Layer GMPLS Network

HOPIDynamic Ethernet Network

DomainBoundary

GMPLS Provisioned LSPDedicated Ethernet VLAN “Circuit”

GWU CLPK

LA

SEADC

CHI

Static Optical Layer

MCLNARLG DCNE

NY

HOUEthernet Layer

ENNI

IGP-TE IGP-TE

UNIUNI

29

Heterogeneous Network TechnologiesComplex End to End Paths

EndSystem

AS 1AS2 AS 3

VLSR

Ethernet SegmentVLSR EstablishedVLAN

Ethernet overWDM

Ethernet overSONET

EndSystem

Ethernet SegmentVLSR EstablishedVLAN

VLSR

Router MPLSLSP

IP Control Plane

IP Control Plane

IP Control Plane

Ethernet

Router

Lambda Switch

SONET Switch

“horizontal” multi-layer adaptations for multi-domain

30

DRAGON-HOPI SC05 Demo:Intercontinental eVLBI Collaboration

31

DRAGON Application Specific Topologies

• Identify endpoints, control plane sets up topologies

• Set up global multi-link topologies

• ~30 seconds

Grids as the eScience Enabler

Regional GridInfrastructure

Program

SCOOPTele-MedicineBio-Informatics

Connectivity

Last Mile

High PerformanceComputing

Standards &Specifications

NetworkResearch

New SimulationAnd Modeling Applications

(energy,weatherPopulation,etc)

33

What’s next?What can we do together?

How can we help?

Followup Contact or Questions:

Don Riley <drriley@umd.edu>

Jerry Sobieski <jerrys@maxgigapop.net>