High Performance Communications Based on Dynamic Lightpaths: TransLight/GLIF

Joe Mambretti, Director, (j-mambretti@northwestern.edu)International Center for Advanced Internet Research (www.icair.org)

Director, Metropolitan Research and Education Network (www.mren.org)Partner, StarLight/STAR TAP, PI-OMNINet (www.icair.org/omninet)

JET Workshop on Optical Network TestbedsNational Science Foundation

Arlington, Va.

April 19, 2005

Perspective: Large Scale Science Is Global

ATLAS

Sloan Digital Sky Survey

LHC

ALMA

Source:Larry Smarr

Overview of Initiatives

• Ref: Earlier Presentation on Emerging New Communications Architecture, New Technologies Migrating to Global Infrastructure

• Dynamic Lightpaths – Globally Implemented• TransLight (First Instantiation of the GLIF)• GLIF – Global Lambda Integrated Facility (www.glif.is)• StarLight – An Initial GLIF Facility TransLight/StarLight IRNC

(www.startap.net/starlight)• TransLight/PacificWave IRNC• GLORIAD and GLORIAD IRNC• APATH and WHREN IRNC• TransPAC IRNC

*Lights and TransLight

• *Lights are bandwidth concentrators and exchanges (StarLight, NetherLight, UKLight, NorthernLight, T-LEX, CzechLight, HKLight, TaiwanLight etc…)

• *Lights also deliver Grid services such as computing, storage and visualization support

• *Lights disrupt old pricing structures from the traditional telcos, which kept the prices high

• *Lights goal is understanding the big data streams and how to map them economically to the least services they need

• TransLight is the initial infrastructure part of GLIF, the Global Lambda Integrated Facility, an direct result of Euro-Link activities

Source: Tom DeFanti

What is GLIF?

• GLIF is a consortium of institutions, organizations, consortia and country National Research & Education Networks who voluntarily share optical networking resources and expertise to develop the Global LambdaGrid for the advancement of scientific collaboration and discovery

• GLIF is under the leadership of SURFnet and University of Amsterdam in The Netherlands

• GLIF Is Developing Architectural Models for Global GLIF Services and Facilities, such as Open Exchange Points

• GLIF Is a Key Organizer of iGRID 2005• www.glif.is

Source: Tom DeFanti

What is the LambdaGrid?

• Today’s Grids enable scientists to schedule computer resources and remote instrumentation over today’s “best effort” networks.

• LambdaGrids enable scientists to also schedule bandwidth. Wave Division Multiplexing (WDM) technology divides white light into individual wavelengths (or “lambdas”) on optical fiber, creating parallel networks.

• LambdaGrids provide deterministic networks with known and knowable characteristics. – Guaranteed Bandwidth (data movement)– Guaranteed Latency (collaboration, visualization, data

analysis)– Guaranteed Scheduling (remote instruments)

Source: Tom DeFanti

Global Participation: GLIF at StarLight

DWDM SURFnet

10 Gbit/s

SURFnet10 Gbit/s

SURFnet10 Gbit/s

IEEAF10 Gbit/s

DwingelooASTRON/JIVE

DwingelooASTRON/JIVE

PragueCzechLight

PragueCzechLight

2.5 Gbit/s

NSF10 Gbit/s

StockholmNorthernLightStockholm

NorthernLight

CA*net4

2.5 Gbit/s

New YorkMANLANNew YorkMANLAN

TokyoWIDETokyoWIDE

10 Gbit/s

10 Gbit/s10 Gbit/s

10 Gbit/sIEEAF10 Gbit/s

10 Gbit/s

10 Gbit/s

2.5 Gbit/s

2.5 Gbit/s

TokyoAPANTokyoAPAN

AmsterdamNetherLightAmsterdamNetherLight

GenevaCERN

GenevaCERN

LondonUKLightLondonUKLight

ChicagoStarLightChicagoStarLight

Source: Kees Neggers, SURFnet

Global Lambda Integrated FacilityWorld Map – December 2004

Predicted international Research & Education Network bandwidth, to be made available for scheduled application and middleware research experiments by December 2004.

www.glif.is Visualization courtesy of Bob Patterson, NCSA.

TransLight/StarLight IRNC Award SCI-0441094

Tom DeFanti (Principal Investigator) Maxine Brown (Co-Principal Investigator)

Joe Mambretti (Investigator)

• In cooperation with US and European national research and education networks, TransLight/StarLight will implement a strategy to best serve established production science

• TransLight/StarLight will support usage by those scientists, engineers and educators who have persistent large-flow, real-time, and/or other advanced application requirements

Scope of Work

• Procure and maintain (minimally) one OC-192 circuit that connects UCAID’s Abilene network in New York City (NYC) to TERENA’sGÉANT L3 network in Amsterdam.

• Evaluate feasibility of a second OC-192 for L1/L2 production services for high-flow data/collaboration-intensive applications.

• Ensure that appropriate traffic monitoring, measurement and analysis are accomplished through NSF-designated partners.

• Provide related management, documentation, engineering support and education and outreach efforts, and attend relevant networking and/or discipline science meetings and conferences.

• Promote and support availability to the advanced science, engineering and education communities who require ongoing transatlantic production services.

• Also Ref: IRNC Translight/PacificWave (TL/PW) PI John Silvester(USC), Co-PI Ron Johnson (UW), GLORIAD IRNC, Greg Cole, APATH/WHREN IRNC,

TransLight/StarLight IRNC

HPIIS Euro-Link OC-192 connection in place until June 30, 2005

Tender out for new link(s); responses due in April

New link(s) to be live on July 1, 2005

US TransLight/StarLight (black)10-20Gb NYC—Amsterdam

US GLORIAD IRNC (Dark Blue)–~1-10 Gb Chicago—Amsterdam

GEANT/Internet2 (red) –30Gb London, etc.—NYC/DC?

UKLight to US (magenta)–10Gb London—Chicago

SURFnet to US (Green)–10Gb Amsterdam—NYC–10Gb Amsterdam—Chicago

CERN to US (cyan)–10Gb CERN—Chicago

IEEAF (blue)–10Gb NYC—Amsterdam

Canadian CA*net4 to US (white)–40Gb Chicago-Canada-NYC–10Gb NYC—Amsterdam

European (not GEANT) (yellow)–20Gb Amsterdam—CERN –10Gb Prague—Amsterdam–10Gb Stockholm—Amsterdam–10Gb London—Amsterdam

Leveraging ~12xOC-192s Trans-Atlantic

Northern Light

CERN

UKLight

Manhattan Landing

StarLight – “By Researchers For Researchers”A Key TransLight/GLIF Facility

Abbott Hall, Northwestern University’sChicago downtown campusView from StarLight

StarLight is an experimental optical infrastructure andproving ground for network services optimized forhigh-performance applicationsGE+2.5+10GEExchangeSoon:Multiple 10GEsOver Optics –World’s “Largest”10GE Exchange!

StarLight InfrastructureStarLight is a large research-friendly co-location facility with space, power and fiber that is being made available to university and national/international network collaborators as a point of presence in Chicago

StarLight

• StarLight Provides Next Generation Advanced Services and Facilities in Cooperation With Multiple Regional, National, and International Networks

• StarLight Provides Facilities for Advanced Prototypes and Early Implementations

• StarLight Also Provides an Exceptional Opportunity for Leading Edge Innovation and Experimentation

• Future Evolution Will Implement a Range of Advanced Technologies

Where is StarLight?

• Located in Northwestern’s Downtown Campus: 710 N. Lake Shore DriveCarrier POPsChicago NAP

CA*net4 has 2x10Gb and Equipment at StarLight

Source: CANARIE

UCLP general operation

CA*net 4 UCLPSwitch AgentsSignal Control Plane Agents

Standard CLI or TL1 interface

Customer A and sub- partition

Customer B

Customer C

DWDM EastboundDWDM Westbound

X

X

OSPF

GMPLS

ISIS

Customer A signaling plane

Subtended Lightpaths to User

Customer B signaling plane

Grooming agents

Customer C signaling plane

X X

Customer A UCLP Server

MonFox TL1 Proxy

OXC

X

Customer C signaling plane

Customer C

CANARIE provides APN to NRC

New YorkChicagoSeattle

Victoria

Vancouver

Edmonton

CalgaryRegina

Saskatoon

Winnipeg

Toronto

Ottawa

Montreal

Fredericton

HalifaxCA*net 4 router2G Lightpath WSGbE interface WS

NRC partitions APN

New YorkChicagoSeattle

Victoria

Vancouver

Edmonton

CalgaryRegina

Saskatoon

Winnipeg

Toronto

Ottawa

Montreal

Fredericton

Halifax

UCLP intended for projects like National LambdaRail

CAVEwave acquires a separate wavelength between Seattle and Chicago and wants to manage it as part of its network including add/drop, routing, partition etc

NLR Condominium lambda network

OriginalCAVEwave

SURFnet5 – Soon “6” Connected to StarLight via Production Circuit and NetherLight Research Circuit

• Partners BT and Cisco

• 15 PoPs connected by thirty 10 Gbit/s lambdas

• Dual stack IPv4 and IPv6

• 500,000 users

• 84 institutes connected at Gbit/s level

UvA-VLEUvA-MM

VUULeidenTUDelft

STARPLANEDWDM Backplane

CPU’s

RCPU’sR

CPU’s

R

CPU’

s

R

CPU’s

R

NOC

CdLSource: Cees de Laat

CANARIE2xGigE circuitsSURFnet 2xGigE

circuits

NetherLightStarLight

Source: Tom DeFanti

SuperJANETR&D NetworkSuperJANETR&D Network

GEANT

UKLightLondon

StarLightChicago

NetherLightAmsterdam

CERN

CzechLightLocal Research Equipment

International Point-of-Access

CA*net

Abilene

UKLight is Connected to StarLight with 10Gb and Equipment

10Gb/s

10Gb/s

10Gb/s10Gb/s

10Gb/s

2.5Gb/s

Source: Peter Clarke, David Salmon, UKLight

GLORIAD: Worldwide Ring Now 622 Mb StarLight to Moscow, 10Gb Coming

JGN II Network Topology Map

KanazawaSendai

Sapporo

Nagano

KochiNagoya

Fukuoka

Okinawa

Okayama

<１G>・Teleport Okayama (Okayama)・Hiroshima University (Higashi

Hiroshima)<１００M>・Tottori University of

Environmental Studies (Tottori)・Techno Ark Shimane (Matsue)・New Media Plaza Yamaguchi

(Yamaguchi)

<１０G>・Kyoto University (Kyoto)・Osaka University (Ibaraki)

<１G>・NICT Kansai Advanced Research Center (Kobe)

<１００M>・Lake Biwa Data Highway AP * (Ohtsu)・Nara Prefectural Institute of Industrial

Technology (Nara)・Wakayama University (Wakayama)・Hyogo Prefecture Nishiharima Technopolis

(Kamigori-cho, Hyogo Prefecture)

<１０G> ・Kyushu University (Fukuoka)

<１００M>・NetCom Saga (Saga)・Nagasaki University

(Nagasaki)・Kumamoto Prefectural Office

(Kumamoto)・Toyonokuni Hyper Network AP

*(Oita)・Miyazaki University (Miyazaki)・Kagoshima University

(Kagoshima)

<１００M>・Kagawa Prefecture Industry Promotion

Center (Takamatsu)・Tokushima University (Tokushima)・Ehime University (Matsuyama)・Kochi University of Technology

(Tosayamada-cho, Kochi Prefecture)

<１００M>・Nagoya University (Nagoya)・University of Shizuoka (Shizuoka)・Softopia Japan (Ogaki, Gifu Prefecture)・Mie Prefectural College of Nursing (Tsu)

<１０G>・Ishikawa Hi-tech Exchange Center

(Tatsunokuchi-machi, Ishikawa Prefecture)<１００M>・Toyama Institute of Information Systems (Toyama)・Fukui Prefecture Data Super Highway AP * (Fukui)

<１００M>・Niigata University

(Niigata)・Matsumoto Information

Creation Center (Matsumoto,

Nagano Prefecture)

<１０G>・Tokyo University

(Bunkyo Ward, Tokyo)・NICT Kashima Space Research Center

(Kashima, Ibaraki Prefecture)<1G>・Yokosuka Telecom Research Park

(Yokosuka, Kanagawa Prefecture)<100M>・Utsunomiya University (Utsunomiya)・Gunma Industrial Technology Center

(Maebashi)・Reitaku University

(Kashiwa, Chiba Prefecture)・NICT Honjo Information and

Communications Open Laboratory(Honjo, Saitama Prefecture)

・Yamanashi Prefecture Open R&D Center

(Nakakoma-gun, Yamanashi Prefecture)

<１G>・Tohoku University (Sendai)・NICT Iwate IT Open Laboratory

(Takizawa-mura, Iwate Prefecture)<１００M>・Hachinohe Institute of Technology

(Hachinohe, Aomori Prefecture)・Akita Regional IX * (Akita)・Keio University Tsuruoka Campus

(Tsuruoka, Yamagata Prefecture)・Aizu University (Aizu Wakamatsu)

<１００M>・Hokkaido Regional Network Association

AP * (Sapporo)

NICT Tsukuba Research Center

20Gbps10Gbps1GbpsOptical testbeds

Core network nodesAccess points

Osaka

Otemachi

USA

NICT Keihannna Human Info-Communications

Research Center

NICT Kita Kyushu IT Open Laboratory

NICT KoganeiHeadquarters

[Legends ]

*IX:Internet eXchange

AP:Access Point

10 Gbps StarLight

Source: JGN II

AARNET’s International Connections

Starlight

TWARENLight Path

Taipei

Hsinchu

Taichung

Tainan

NTU

NSYSUNCKU

CCU

NCNU

NCHU

NTHU

NCTUNCU

SinicaNTU

StarLightStarLight CANARIECANARIECANARIECANARIEPacific WavePacific Wave

LALALANLALALAN

MAN LANMAN LAN

2.5G

2.5G

STM

-4c 2.5G

2.5G(1.25G)

2.5G(1.25G)

2.5G(1.25G)

NETHERNETHERLandLand

2.5G(1.25G)

Source: TWAREN

Kreonet

Europe

Kwangju Busan

SuwonChonan

JeonjuChang-

won

Pohang

K R E O N E TK R E O N E T

APII

Seoul

KREONet2

High-Performance S&T Facilities(High-Performance Cluster/Supercomputers, Storage, Experimental Facilities, Visualization, Access Grid, DB Servers, etc.)

e-Science based KREONET/KREONet2/SuperSIReN

→ joining global Cyber Infrastructure

GEANT

Singapore/SingAREN

Daegu

China/CSTNET

Incheon

Japan/SINET

APII

TEIN

DaejeonSuperSIReN

Source: Kreonet

WHREN - LILA Project

• Joint response by FIU and CENIC to NSF IRNC solicitation

• 2.5Gbps persistent high-performance research network for South America to support U.S. and international science and engineering research and education communities

• Collaboration with research network operators and exchanges in the Americas

• Phased implementation over 5 years

Source: AMPATH

TeraGrid has 10x10Gb over I-WIRE and Equipment at StarLight

NCSA/UIUC

ANL

UICStarlight / NW Univ

Multiple Carrier Hubs

Ill Inst of Tech

Univ of Chicago

I-WIRE

StarLightInternational Optical Peering Point

(see www.startap.net)

Los Angeles

San Diego

TeraGrid Backbone

Chicago

Urbana

OC-48 (2.5 Gb/s, Abilene)Multiple 10 GbE (Qwest)Multiple 10 GbE (I-WIRE Dark Fiber)

$7.5M Illinois DWDM Initiative

Pittsburgh

Source: Dan Reed

ETF Connected via StarLight

www.teragrid.org Source: TeraGrid

The First Link of the National Lambda Rail:StarLight <=> PSC for ETF

Source: John Silvester, Dave Reese, Tom West, CENIC

TWCJGISNLL

LBNL

SLACFix-W

YUCCA MTBECHTEL

PNNLLIGO

INEEL

LANL

SNLAAlliedSignal

PANTEX

ARM

AlliedSignal

NOAA

OSTIORAU

SRS

ORNLJLAB

MAE-E

PPPL

ANL-DCINEEL-DCORAU-DC

LLNL/LANL-DC

MIT

ANL

BNL

FNALAMES

NY-NAP

NevisYale

PAIX-W

ATL HUB

4xLAB-DC

Brandeis

SEA HUB

SNV

SNV

SNVELP

ELP

NERSC PAIX-E

NR

EL

Mae-W

SNV

ALBHUB

NYC HUBS

DC HUB

ELP HUB

CHI HUBSNV HUBLLNL

GA

DOE-ALB

SDSC

Japan

GTN&NNSA

SAN

International (high speed)OC192 (10G/s optical)OC48 (2.5 Gb/s optical)Gigabit Ethernet (1 Gb/s)OC12 ATM (622 Mb/s)OC12 OC3 (155 Mb/s)T3 (45 Mb/s)T1-T3T1 (1 Mb/s)

Office Of Science Sponsored (22)NNSA Sponsored (12)Joint Sponsored (3)Other Sponsored (2 LIGO, NOAA)Laboratory Sponsored (6)

QWESTATM

42 end user sites

ESnet IP

GEANT- Germany- France- Italy- UK- etc

CA*net4CERNMRENNetherlandsRussiaStarTapTaiwan (ASCC)

CA*net4KDDI (Japan)FranceSwitzerlandTaiwan (TANet2)

AustraliaCA*net4Taiwan (TANet2)Singaren

DOE’s ESnet Connects Facilities and Collaborators at StarLight

STARLIGHTCHI NAP

Sinet (Japan)Japan –Russia(BINP)

Source: DOE

DOE’s “Ultra Science Net” is at StarLight

http://www.csm.ornl.gov/ultranet/topology.htmlSource DOE

UltraLight Network: PHASE III

• Move into production

• Optical switching fully enabled amongst primary sites

• Integrated international infrastructure

Source:

UltraLight Network

ARC/NGIX-West

JPL

GRC

MSFC

NGIX-StarLight

GSFC

NPN Sites OC-3 ATMNREN Sites OC-12ATMHybrid Ground Station (35 Mbps)

NGIX-East

NASA WAN Testbed:• NASA Research and Education Network

(NREN)• NASA Prototyping Network (NPN)

NASA WAN

Testbed

NASA’s NREN Network Testbed comes to StarLight

Source: NASA

NISN

NGIX WestNew Generation IP Connections to

West Coast Regional networks,Tier 1 ISP networks,

the PacRim, Japan, and Australia

MAE WestMost Important IP Connections to

West Coast Regional networks,Tier 1 ISP networks,

the PacRim, Japan, and Australia

Commercial Ameritech NAPStarlight

Most Important Connections toMid-West & SouthEast

Regional networks, Tier 1 ISP networks

NGIX EastNew Generation IP Connections to

East Coast Regional networks,Tier 1 ISP networks, and Europe

MAE EastMost Important Connections toEast Coast Regional networks,Tier 1 ISP networks, and Europe

NASA’s NISN is at StarLight

Source: NASA

DREN Network Is At StarLight

September 26-30, 2005University of California, San Diego

California Institute for Telecommunications and Information Technology [Cal-(IT)2]United States

GLIF Will Meet During “World Of Tomorrow 2005”

iGrid 2oo5THE GLOBAL LAMBDA INTEGRATED FACILITY

Co-Organizers: Tom DeFanti, Maxine Brown

www.startap.net/starlight

Thanks to All Participants, NSF,and National and International Supporters!