GigaPoP Transport Options:

I-WIRE Positioning for the Bandwidth Tsunami

Virtual Internet2 Member Meeting

Oct 4, 2001

Linda WinklerArgonne National Laboratory

winkler@mcs.anl.gov

I-WIRE Background

•State Funded Dark Fiber Optical Infrastructure to support Networking and Applications Research• $7.5M Total Funding

•$4M FY00-01 (in hand)•$2.5M FY02 (approved 1-June-01)•Additional $0.5M in FY03, FY04

• Application Driven•Access Grid: Telepresence & Media•Computational Grids: Internet Computing

•Data Grids: Information Analysis

• New Technologies Proving Ground•Optical Switching•Dense Wave Division Multiplexing•Advanced middleware infrastructure

UIC

ANL

NCSA/UIUC

UC

NU / Starlight

Star Tap

IIT

For more information see www.iwire.org

Dark Fiber

•Location, location, location•Metro

•Roughly $2 per meter per strand

•Lateral challenges•Regional

•May require regeneration; regen, reshape, retime (3R) is expensive!

•Wide area•Expensive due to 3R requirements•Key is location of carrier runs•$100K-$1M/month for 2200 mile OC-192 link

Dark Fiber (cont.)

•Key is the one time up front cost for the purchase of an IRU

•Maintenance and management are the buyers problem

•Obtain fiber characteristics as soon as possible (SMF vs. NZDSF, OTDR shots)

•Rapid provisioning possible allowing more adaptive networks

•The fiber industry is immature and underdeveloped, allowing sophisticated customers to negotiate much more attractive deals than would result from a standard RFP pricing exercise.

Indefeasible Right to Use (IRU) Services

•Terms 10, 15, 20 yrs•Alternatives

•Long term capital lease•Short term lease•Managed service

•Considered as a physical asset which can be re-sold, traded or used a collateral.

•Cost can be amortized over lifetime which results in a monthly cost substantially below traditional telecommunication services.

•Be sure of contract conditions due to shaky nature of some vendors financial situation.

Dense Wavelength Division Multiplexing

•To ring or not to ring (ring vs. mesh)• Redundancy (at what cost?)• Survivability• Protection

•Mesh topology benefits• Migration, scaling• Deployment speed• Capacity Utilization• Network Restoration• Operating Costs

•Laser reach-3R issues and OEO• Large portion of the cost

•No standards• Number and spacing of lambdas are design variables

•One transponder per wavelength• Beware OC-192/10GbE WAN PHY/10GbE LAN PHY are not all equal

TeraGrid Backplane

NCSA/UIUC

ANL

UICMultiple Carrier Hubs

Starlight / NW Univ

Ill Inst of Tech

Univ of Chicago Indianapolis (Abilene NOC)

St Louis GigaPoP

I-WIRE

StarLightInternational Optical Peering Point

(see www.startap.net)

Los Angeles

San Diego

TeraGrid Backplane

Abilene

Chicago

IndianapolisSt. LouisUrbana

OC-48 (2.5 Gb/s, Abilene)

Multiple 10 GbE (Qwest)

Multiple 10 GbE (I-WIRE Dark Fiber)

• Solid lines in place and/or available by October 2001• Dashed I-WIRE lines planned for Summer 2002

Charlie Catlett – Argonne National Laboratory (catlett@mcs.anl.gov)

TeraGrid Proposed Backplane Architecture

ChicagoLos Angeles

Site Border Switch

Cluster Aggregation

Switch

One Wilshire(Carrier Fiber Collocation Facility)

DTF Backbone

Core Switch

Qwest San Diego POP

Vendor TBDLong-Haul DWDM(Operated by site)

Vendor TBDMetro DWDM(operated by site)

Ciena CoreStream™Long-Haul DWDM(Operated by Qwest)

Vendor TBDMetro DWDM

Vendor TBDSwitch/Router*(256 Gb/s crossbar)

DTF Local Site Resources and External Network Connections

2200mi

140mi 25mi

2mi

115mi15mi

20mi

455 N. Cityfront Plaza(Qwest Fiber Collocation Facility)

Vendor POP at JPL

Caltech SDSC NCSA ANL

Caltech Cluster(64p)

SDSC Cluster(250p)

NCSA Cluster(2000p)

ANL Cluster(128p)

4 x 10 GbE

*Initial Phase using IP Switch/Routers. This design will be evaluated beginning October 2001. Phase 2 (optical mesh) will also be evaluated prior to full DTF cluster deployment in early 2002..Charlie Catlett – Argonne National Laboratory (catlett@mcs.anl.gov)

Wavelength Services

•OC48, OC192 vs. 10GbE•Be sure of handoff specifications•Management- determine level required•Qwest/Teleglobe/(3)Link Global/Global Crossing service offerings

•Benefits•Lower cost•Customer responsible for protection•Share cost of electronics-save capital investment•Customer empowerment•Potential for higher utilization of network•Transparency of signal•Flexibility

IP Routers and Switches

•Interoperability •Which PHY (LAN vs. WAN) interface between DWDM and

CPE?

•$$$•WAN PHYs tend to be pricey

•Is 10GbE really 10,000 Mb/s, or 9.3 Gb/s, or maybe 8 Gb/s?

•What is the largest individual stream you must support?•Aggregates vs. large streams

•Currently no visibility into the optical layer

Next Steps: Optical Mux / Wavelength Router / Optical Wavelength Cross-connect System

GigE / 2.5 Gb/s 10 Gb/s

FDP

Customer Interface

x

OTU

OTU

1

n

2.5 / 10 / 40 Gb/s

DWDM

Optical Mux/Router/

Cross-Connect

Optical Switches

•Current tech is O-E-O•Incoming signals are converted from Optical to Electrical•Signal is switched electrically•Outgoing signals are converted back to optical•Up to 64x64 at 2.5 Gb/s•Smart but slow

•Future tech will be O-O-O•Operate in the all optical domain•Bit rate independent•A win at 40 Gb/s•Below 10 Gb/s electronic switching will be hard to beat•Challenge will be in management•Fast but dumb

Optical Internetworking Progress

•Optical Internetworking Forum (OIF)•UNI 1.0 specification in progress•Based on domain services model

•Internet Engineering Task Force (IETF)•Generalized Multiprotocol Label Switching (GMPLS)•Incorporates Domain and Peer Models

•OIF and IETF are in sync•OIF UNI 1.0 based on GMPLS•Start with Domain Model and evolve

•Stay tuned. The ending of this story has not yet been written.

Wavelengths and the Future

• Wavelength services are causing a network revolution:• Core long distance SONET Rings will be replaced by meshed

networks using wavelength cross-connects • Re-invention of pre-SONET network architecture

• Improved transport infrastructure will exist for IP/packet services

• Electrical/Optical grooming switches will emerge at edges

• Automated Restoration (algorithm/GMPLS driven) becomes technically feasible.

• Operational implementation will take beyond 2003

StarLight Optical Peering Point & Co-lo Facility

•710 N. Lake Shore Dr. Chicago, IL •Northwestern Campus•Central downtown location•Near carrier services serving Chicago loop•Telephone switch room•Colo space available•Multiple carriers access

•Ameritech, AT&T, Qwest, Global Crossing, Global Crossing, MCI Worldcom

•Other builds possible