Update on Dark Fiber Techniques in the USUpdate on Dark Fiber Techniques in the US

Steve Cotter, Internet2

CEF WorkshopMay 17, 2005

Steve Cotter, Internet2

CEF WorkshopMay 17, 2005

BackgroundBackground

•Director, Network Services at Internet2•Responsibilities include:

• Abilene Network• Manhattan Landing Exchange Point (MAN LAN)• FiberCo

• Also working on:• Next-generation Abilene• HOPI Project (Hybrid Optical Packet Infrastructure)

•Previously had responsibility for managing National LambdaRail’s fiber acquisition and facilities / optronics buildout

Networking in the USNetworking in the US

The model of a facilities-based infrastructure built with owned assets has gained momentum on both the regional and national scales• What’s driving this?

• Desire to move away from the traditional GigaPoP model of provisioned, high-capacity services• A distressed telecom market has created a contrarian

opportunity to own what is viewed by many as a strategic asset – dark fiber – but the window is closing!• Increasing capabilities / expertise in deploying and managing

networks within the community• Research needs - Grid computing wants to view the network as a

schedulable resource• On par with CPU, memory, storage, and visualization capabilities

All at a time when carriers have not made major capital expenditures in new optical infrastructure

Current LandscapeCurrent Landscape

The result:• U.S. research universities have been aggressively acquiring

raw assets & ancillary services from wholesale telecom providers • National LambdaRail launched to provide a national scale

infrastructure for research and experimentation in networking technologies and applications• Upwards of 30,000 route-miles of inter-city dark fiber are now

held by this community• < 40% held by NLR

• More than 25 RONs have emerged • mostly state-based• many with strong gubernatorial support• often with concurrent economic development objectives

• Active examination of new network service models is underway• Moving beyond the common bearer service view of best-effort IP

State and Regional Optical NetworksState and Regional Optical Networks

• Alabama*• Arizona (CENIC)• Arkansas*• California (CALREN)• Colorado (FRGP/BRAN)• Connecticut (Conn. Education

Network)• Florida (Florida LambdaRail)• Georgia (Southern Light Rail)• Great Plains Network*• Indiana (I-LIGHT)• Illinois (I-WIRE)• Louisiana* (LONI)• Maryland, D.C. & northern

Virginia (MAX)• Michigan (MiLR)• Minnesota

• New England region (NEREN)• New Mexico* (NMSU, UNM)• New York (NYSERNet, Cornell)• North Carolina (NC LambdaRail)• Ohio (Third Frontier Network)• Oklahoma (OneNet)• Oregon• Pacific Northwest (Lariat – NIH

BRIN, PNNL)• Rhode Island (OSHEAN)• SRON* (southeastern U.S.)• Tennessee* (OneTN)• Texas (LEARN)• Virginia (MATP)• Wisconsin (WiscNet)• Wyoming

(RONs in red have made dark fiber acquisitions through FiberCo)

(*RONs with RFx’s issued or in process of acquiring fiber)

States with Regional Optical NetworksStates with Regional Optical Networks

States with a RON

FiberCo OverviewFiberCo Overview

•Tool designed to support optical initiatives • Spun off from NLR governance discussions• Internet2 took responsibility for forming the organization• Patterned on the success of Quilt commodity Internet project •Operates on behalf of U.S. higher education and affiliates – Internet2 membership

•Not an operating entity•Will not light the fiber – only a holding company

•Functions•Market maker• Assignment vehicle for both national & regional optical initiatives

FiberCo OverviewFiberCo Overview

• Fiber partner• National-scale IRU and O&M pricing available

through March, 2006 on Level(3)’s traditional footprint

• Exploring compatible agreements with other providers - WilTel, Qwest, etc.

• Level(3) chosen because of their overall wholesale approach• Open fiber interconnection policy• Co-location space availability• Homogenous fiber type - G.655 (Corning LEAF)• No AUP/CoU restrictions• Ease of interconnection with NLR

FiberCo OverviewFiberCo Overview

• Bifurcated contracts designed to survive a carrier bankruptcy event• 20 yr IRU – can be assigned to other non-profit entities• O&M Contract – renewable on 5 yr terms

• FiberCo assigns both the fiber IRU asset and recurring O&M agreements through an assignment agreement• 3-way among FiberCo, Level(3) & assignee• Transaction fee (1/yr) charged to recover costs

• Ongoing bilateral relationship directly between Level(3) and assignee

• Assignees become responsible for any recurring IRU tax liabilities and potential exemptions

Current Product PortfolioCurrent Product Portfolio

• Dark Fiber - ~7500 route miles assigned to date• 20-year IRU agreements• 5-year renewable agreement for fiber O&M and

equipment co-location & power• Pre-negotiated contracts customized for the R&E

community• Collaborative effort with Level(3) on interconnects

and lateral builds • Builds at cost plus 15%

• Metro Fiber• National-scale pricing for metro fiber with Level(3)• Volume discounts on metro fiber• 50% paid for temporary lit services are credited toward

cost of metro IRU• Access to the Level(3) footprint in Europe at

FiberCo prices

Work in ProgressWork in Progress

• Closer collaboration with NLR and its regionals to help facilitate more deployments

• Formalized agreements with additional fiber providers so FiberCo can cover more geography, continue beyond March ‘06

• Professional services agreement taking advantage of aggregate demand for:• Network engineering & design • Installation, test & turn up services• Shared NOC services• Remote hands & eyes

National LambdaRail Fiber MapNational LambdaRail Fiber Map

DAL

SYR

TUL

PENELP

PHO

BAT

ALB

HOU

WDC

OGDCLE

NYC

SAA

4

4

JAC4

4

4

44

4

4

4

4

4

4

4

4

4

4

4

44

44

4

44

4 4

SLC4

4

Level3 fiberWilTel fiberOther fiber

LAX4

4

RAT4

Cisco 15808 terminalCisco 15808 OADMCisco 15454 terminalCisco 15454 OADM

8

4

4

8

STAR

CHIKAN

PIT

BOI

CLE

ATL

POR

RAL

DENSVL

SEA

8

8

8

8

8 8

8

8

8

8

88

8

8

88

88

8

8

8

8

8

4

4

LAX

SAN

88

8

8

8

8

NLR Layer 2 NetworkNLR Layer 2 Network

HOU

TUL

ELP

KAN

PHO

BAT

LAXALB

PIT

WDC

CLE

ATL

RAL

CHINYC

DENSVL

SEA

JAC

10GE wave

Cisco 6509 switch

NLR Layer 3 NetworkNLR Layer 3 Network

ALBPHO

BAT

RAL

JAC

TUL

PIT

HOU

LAX

WDC

ATL

CHINYC

DEN

SEA

Cisco CRS-1 router

10GE wave

NLR Equipment LayoutNLR Equipment Layout

CRS-1

6509

1545415808

15500

1550015500

EastWest

Metro and Regional Optical Networks

NLR demarc

DWDM

1GE10GE or OC192

Lessons LearnedLessons Learned

• No single carrier has all routes• Build ongoing relationships with multiple carriers

• The last mile (including the campus) is still the most difficult• Dark fiber is real estate

• Transactions can be time consuming (up to 6 months)• Definite advantage to pre-negotiated contracts

• Owning the fiber is only ~10% of the total cost of ownership• Fiber plant design (hut spacing, topology), power, optical gear

maintenance have great impact on recurring costs• Know the capabilities of the fiber you are acquiring

• Fiber type, characteristics, and method of deployment must allow you to reach your networking goals (i.e. high channel count DWDM, 40G)

• Because of the uncertainty with funding, regionals in the US need to develop a sustainable business model

• You are now a service provider • Need to remain aware of the capabilities of the equipment and new

feature releases in order to deliver advanced services

Lessons LearnedLessons Learned

• Need a mechanism to facilitate research collaboration on a regional basis between universities and the private/public sectors

• Creative solutions are sometimes best• Combine lit services with deploying your own equipment to

expand reach and minimize costs (i.e. adding commodity Internet, wave services for backhaul to the deal)

• Consider owning a pool of cards that can be deployed and redeployed on carrier gear where and when you choose

• Anchor tenant approach – carrier owns the chassis, you own up to X% channel cards within that chassis

• Lease fiber with option to buy• Rent conduit, pull your own fiber

ObservationsObservations

• The window for acquiring fiber at current pricing levels in the US is almost over – the economy is recovering and market prices going up!

• Potentially disruptive technologies are on the horizon

• Consolidation of the telecom industry in the US does not bode well for customer-owned networks

For more informationFor more information

• Internet2 http://www/internet2.edu

• FiberCo http://www.fiberco.org

• National LambdaRail http://www.nlr.net