nlr layer2/layer3 users bof nlr status update philadelphia internet2 member meeting 19 september...

NLR Layer2/Layer3 Users NLR Layer2/Layer3 Users BOFBOF

NLR status updateNLR status update

Philadelphia Internet2 Member Philadelphia Internet2 Member MeetingMeeting

19 September 200519 September 2005

Brent Sweeny, Indiana UniversityBrent Sweeny, Indiana University

9/19/20059/19/2005 NLR Layer2/Layer3 users BOFNLR Layer2/Layer3 users BOF 22

National LambdaRail designNational LambdaRail design

NLR WaveNet PoP NLR WaveNet & FrameNet PoP NLR WaveNet, FrameNet & PacketNet PoP

NLR owned fiberBAT

SAN

STA

CHISLC

HOU

DAL

SYR

TUL

PENELP

KAN

PHOLAX

ALB

PIT

WDC

OGD

BOI

CLE

ATL

POR

RAL

NYC

SAA

DENSVL

SEA

JAC


Generic NLR L1, L2 and L3 PoP Layout

CRS-1

Colo

EastWest

NLR demarc

15808

6509

15454

DWDM

1G wave, link or port10G wave, link or port


DAL

SYR

TUL

PENELP

PHO

BAT

ALB

HOU

WDC

OGDCLE

NYC

SAA

4

4

JAC4

4

4

44

4

4

4

4

4

4

4

4

4

4

4

44

44

4

44

4 4

SLC4

4

Level3 fiberWilTel fiber

4

4

RAT4

Cisco 15808 terminalCisco 15808 OADMCisco 15454 terminalCisco 15454 OADM

8

4

4

8

STA

CHIKAN

PIT

BOI

CLE

ATL

POR

RAL

DENSVL

SEA

8

8

8

8

8 8

8

8

8

8

88

8

8

88

88

8

8

8

8

8

4

4

LAX 8

8

NLR Layer 1NLR Layer 1


Layer 1 Phase 2 Layer 1 Phase 2 DeploymentDeployment

DAL

SYR

TUL

PENELP

KAN

PHO

BAT

ALB

HOU

WDC

OGDCLE

NYC

SAA

DEN

4

4

JAC4

4

4

44

4

4

4

4

4

4

4

4

4

4

4

44

44

4

44

4 4

SLC4

4

Level3 fiberWilTel fiber

LAX4

4

RAT4

Cisco 15808 terminalCisco 15808 OADMCisco 15454 terminalCisco 15454 OADM

8

4

4

8


Layer 1 baselineLayer 1 baseline

Opportunity to connect into lambda fabricOpportunity to connect into lambda fabric Point to pointPoint to point Other endpoint could be anywhereOther endpoint could be anywhere

Early examples:Early examples: HOPIHOPI UltralightUltralight iGRIDiGRID SC05 (Supercomputing)SC05 (Supercomputing)

Experiments support center (for all layers)Experiments support center (for all layers)


Layer 2 Network DesignLayer 2 Network Design

HOU

TUL

ELP

KAN

BAT

LAXALB

PIT

WDC

CLE

ATL

RAL

CHINYC

DENSVL

SEA

JAC

10GE wave10GE managed waveYellow sites are done

Cisco 6509 switch

SVL


Layer 2 installation statusLayer 2 installation statusSwitch installations completed:

Sunnyvale Denver Kansas City ChicagoCleveland Pittsburg Raleigh Washington DCJacksonville Atlanta Los Angeles TulsaEl Paso Houston

New York City scheduled for this week (Sept 20-21).

Baton Rouge waiting for Katrina side-effects to calm down. (Was scheduled for the week of August 29, now tentatively Oct 11-12.)

Albuquerque scheduled for mid-Oct.

Interconnections being completed now.


Layer 2 service baselineLayer 2 service baseline

1GE Connection into local 65001GE Connection into local 6500 Access to “national peering VLAN”Access to “national peering VLAN” Additional Options:Additional Options:

Dedicated point to point Etherrnet, Nx1GEDedicated point to point Etherrnet, Nx1GE Best-effort point to multipoint (no dedicated Best-effort point to multipoint (no dedicated

bw)bw) Soon:Soon:

10GE ports10GE ports Dedicated point-to-multipointDedicated point-to-multipoint


GoalGoal

Provide circuit-like options for users Provide circuit-like options for users who can’t use, can’t afford, or don’t who can’t use, can’t afford, or don’t need, a 10G Layer1 wave via point-need, a 10G Layer1 wave via point-to-point layer2 VLANs.to-point layer2 VLANs.

Experiment with large-scale layer2 Experiment with large-scale layer2 capabilities.capabilities.


Dimensions of NLR Layer2 Dimensions of NLR Layer2 capabilitiescapabilities

point-to-point or multipointpoint-to-point or multipoint public or privatepublic or private bandwidth guaranteed or best effortbandwidth guaranteed or best effort resilient (spanning tree) or nailed upresilient (spanning tree) or nailed up temporary or permanenttemporary or permanent experimental or stable/production-experimental or stable/production-

orientedoriented

Some of the above dimensions are either/or, some Some of the above dimensions are either/or, some are more spectrum-like.are more spectrum-like.

The following slides show some expected uses of The following slides show some expected uses of NLR layer 2, and how these dimensions may NLR layer 2, and how these dimensions may relate to each type of project.relate to each type of project.


National Peering FabricNational Peering Fabric Multipoint, public, best-effort, resilient, Multipoint, public, best-effort, resilient,

permanent, stablepermanent, stable NLR allocated addresses, peer with any other NLR allocated addresses, peer with any other

member across the layer 2 fabric policy-free.member across the layer 2 fabric policy-free. Possible to have more than one (say by MTU).Possible to have more than one (say by MTU). Ready to go today.Ready to go today.

Backup connections to networks such as Backup connections to networks such as Abilene, commodity providers, etc.Abilene, commodity providers, etc.

Point-to-point, private, permanent, stablePoint-to-point, private, permanent, stable Could be best-effort or guaranteed.Could be best-effort or guaranteed. Could be nailed-up or resilient.Could be nailed-up or resilient. Load-balance or leave idle until needed.Load-balance or leave idle until needed. Ready to go today, though we only have pricing for Ready to go today, though we only have pricing for

the guaranteed nailed-up case.the guaranteed nailed-up case.


To enable a flexible topology for NLR To enable a flexible topology for NLR layer3layer3

Best-effort, private, temporary, experimentalBest-effort, private, temporary, experimental We have 8 layer3 nodes, but the topology between We have 8 layer3 nodes, but the topology between

them can be made much more interesting by creating them can be made much more interesting by creating various connections over the layer2 network.various connections over the layer2 network.

Enables layer3 experimentation.Enables layer3 experimentation.

To provide members with a second path To provide members with a second path into the NLR layer3 networkinto the NLR layer3 network

Point-to-point, private, permanent, experimentalPoint-to-point, private, permanent, experimental Connect to a second node on the layer3 backbone.Connect to a second node on the layer3 backbone. Load-balance or leave idle until needed.Load-balance or leave idle until needed. Included in membership.Included in membership.


Temporary connections for special Temporary connections for special projectsprojects

Guaranteed, private, temporary, stableGuaranteed, private, temporary, stable For remote instrumentation, where the member For remote instrumentation, where the member

only has the remote resource reserved for a only has the remote resource reserved for a limited window.limited window.

For conferences, demos, and other special For conferences, demos, and other special events.events.

Provides a low latency/jitter path if needed.Provides a low latency/jitter path if needed. Nailed-up if latency is critical, probably resilient Nailed-up if latency is critical, probably resilient

if not.if not. Could be point-to-point or multipoint.Could be point-to-point or multipoint. Technically, this is possible today, but we have Technically, this is possible today, but we have

no pricing model for it.no pricing model for it.


Bootstrapping circuit-like researchBootstrapping circuit-like research Point-to-point, private, guaranteed, Point-to-point, private, guaranteed,

temporary, nailed-up, experimentaltemporary, nailed-up, experimental To enable a researcher to get started while waiting To enable a researcher to get started while waiting

for funding or provisioning of a layer1 circuit.for funding or provisioning of a layer1 circuit. Similar to a special event, but more experimental, Similar to a special event, but more experimental,

a probably a stronger need for it to be nailed-up.a probably a stronger need for it to be nailed-up. Technically, this is possible today, but we have no Technically, this is possible today, but we have no

pricing model for it.pricing model for it.

Provide control plane network for optical Provide control plane network for optical experimentsexperiments

Permanent, resilient, experimentalPermanent, resilient, experimental A topology could be created for the oob A topology could be created for the oob

management network needed for some dynamic management network needed for some dynamic optical networking experiments (GMPLS, etc.)optical networking experiments (GMPLS, etc.)


Cluster/Grid LANCluster/Grid LAN Multipoint, private, guaranteed, experimentalMultipoint, private, guaranteed, experimental

To enable remote clusters to appear on the same LAN.To enable remote clusters to appear on the same LAN. It is not known if spanning tree would be wanted.It is not known if spanning tree would be wanted. It could evolve into a more production-like It could evolve into a more production-like

requirement.requirement. Technically, this is possible today, but we have no Technically, this is possible today, but we have no

pricing model for it.pricing model for it.

Experiment directly with Layer 2Experiment directly with Layer 2 Could be of any type (experimental, obviously)Could be of any type (experimental, obviously)

Web-based provisioning, direct user requests, etc.Web-based provisioning, direct user requests, etc. Concern about interaction with more production-like Concern about interaction with more production-like

requirements.requirements.


ALBPHO

BAT

RAL

JAC

TUL

PIT

HOU

LAX

WDC

ATL

CHINYC

DEN

SEA

Cisco CRS-1 router

10GE waveYellow sites are installed

Layer 3 NetworkLayer 3 Network

LAX


Layer 3 installation statusLayer 3 installation status

With the work this week in NYC, all layer3 router installations are complete.

Waves are being built between routers and for layer3 backhaul now.


Layer3 service baselineLayer3 service baseline

Each member gets two routed connectionsEach member gets two routed connections ““local” 10GE local” 10GE VLAN backhauled to 2VLAN backhauled to 2ndnd node node

BGP peering with NLR L3 networkBGP peering with NLR L3 network IPv4 unicastIPv4 unicast IPv4 multicast (MBGP/PIM/MSDP)IPv4 multicast (MBGP/PIM/MSDP) IPv6 unicast (multicast later)IPv6 unicast (multicast later) An ‘experimental’ (changeable, changing) An ‘experimental’ (changeable, changing)

L3 networkL3 network


Layer 3 coming servicesLayer 3 coming services

Likely eventual logical routersLikely eventual logical routers More 1GE optionsMore 1GE options More 10G optionsMore 10G options Pre-emptable connectionsPre-emptable connections MPLSMPLS More user control—scheduling, testing, More user control—scheduling, testing,

etcetc User access to measurement dataUser access to measurement data


NLR Engineering/Support NLR Engineering/Support OrganizationOrganization

A very A very distributed, coordinateddistributed, coordinated organization: organization: Service desk at IndianaService desk at Indiana Layer1 NOC and engineering at CENICLayer1 NOC and engineering at CENIC Layer2/3 NOC and engineering at Layer2/3 NOC and engineering at

IndianaIndiana Experiments support center at North Experiments support center at North

CarolinaCarolina


NLR User ResourcesNLR User Resources http://noc.nlr.nethttp://noc.nlr.net [email protected]@nlr.net [email protected]@nlr.net [email protected]@nlr.net

http://noc.nationallambdarail.org/

mailto:[email protected]




NLR Layer 2/3 - discussion NLR Layer 2/3 - discussion

What do users want/need?What do users want/need?Tools?Tools?User groups? User groups? Instruction/workshops?Instruction/workshops?Monitoring and measurement ability? Specify?Monitoring and measurement ability? Specify?Full routes? Control over route propagation?Full routes? Control over route propagation?Control over protocols and timers?Control over protocols and timers?What would you do with logical routers?What would you do with logical routers?Direct access to login and configure routers?Direct access to login and configure routers?Commodity access or ISP collaboration?Commodity access or ISP collaboration?Collaboration with projects like PlanetLab and WAIL? Collaboration with projects like PlanetLab and WAIL?

Which others?Which others?


Summary from July ‘05 (Joint Techs Summary from July ‘05 (Joint Techs BOF)BOF)

CostsCosts More deeply-technical descriptions of the More deeply-technical descriptions of the

architecture, service offeringsarchitecture, service offerings L2 and L3 service-design docsL2 and L3 service-design docs Standardized interface into resourcesStandardized interface into resourcesAnd some thoughts about possibilities:And some thoughts about possibilities: "matchmaking" service, across all layers?"matchmaking" service, across all layers? Logical routers: separate forwarding/control planes, Logical routers: separate forwarding/control planes,

possibly enabling customer logical networks (RONs)possibly enabling customer logical networks (RONs) Customer edge controlCustomer edge control Native non-IP, non-Ethernet protocols?Native non-IP, non-Ethernet protocols? Commodity internet peering?Commodity internet peering? What’s NLR trying to be? (“dancing-elephants or What’s NLR trying to be? (“dancing-elephants or

dancing-gerbils”, “facile/flexible”)dancing-gerbils”, “facile/flexible”)

nlr layer2/layer3 users bof nlr status update philadelphia internet2 member meeting 19 september...

Documents

layer2layer3 users bofto

nlr layer3besteffort

nx1gebesteffort point

expected uses of nlr

point etherrnet

ge portsdedicated point

largescale layer2

pointtopoint layer2