[ 1 ]

Network Services: Today & Tomorrow

Next Generation Infrastructure

Evolution UpdateRob Vietzke

rvietzke@internet2.edu

March 6, 2019

[ 2 ]

AgendaWhat is the NGI Program?

Member-Driven Use Case Stories

Program Status Update& Next Steps

Discussion

[ 3 ]

Internet2 Network Today• Services:

– Advanced Layer 1 Spectrum & Waves– Advanced Layer 2 Ethernet VLANS

• Point to Point vlan networks• Point to Multipoint vlan Networks• Point to Cloud Direct connect Networks• Portal and API Driven on demand networks

– Advanced Layer 3 Virtual Networks• Research and Education Network• Internet2 Peer Exchange (formerly TR-CPS)• Internet2 Cloud Connect Multicloud VRF• MANRS Support• Portal and API Driven on demand networks

– Other: Private Networks, dDOS, etc– With regional & global partners, Internet2

extends throughout the US and to Africa, Asia, Australia, Europe & South America

15,500 Miles Dark Fiber • 65 Ciena 6500 Optical Add Drops • 45

Juniper MX960 MPLS Nodes • 180-200 petabytes moved per month •

1.85 exabytes moved in 2018

[ 4 ]

What is NGI?• The Next Generation Infrastructure Program is a full set

of activities to review and update the services, value and supporting technology of the Internet2 infrastructure portfolio (and relationships in the larger ecosystem)– Includes the services and service models through which the community adopts

Internet2 infrastructure services

– Includes a number of infrastructure upgrade projects

– Includes new features, primarily driven by software, automation and systems virtualization to allow the infrastructure to be more readily integrated in to the broader campus, regional and cloud environment around us.

[ 5 ]

Background: Community tells its stories

• Community discussion about shared future since 2016.– 7+ F2F meetings with community leaders

– RECINNS Paper Process

– 14+ community leaders calls

– 2022 Expectations Paper (requirements)

• https://internet2.box.com/v/NGI2022

– Pilots/Proofs of Concept - optical, router slicing,

cloud completed

– Decision to move to implementation, May, 2018

Guiding principles:

Ecosystem approach

- Focus on joint service delivery model

campus, regional, Internet2

Experimentation

- Try stuff, short term commit

- No impact on current production service

Target research end users

- Push service delivery edge close to user

[ 6 ]

2022 Community Requirements:

Support of ResearchAutomated & Programmable

Cloud & Peer ConnectedEmbedded Security

Regional/National IntegrationInfrastructure Sharing

Academic Enterprise SupportGlobal Reachability

Operational TelemetryResearch Data

Resilient

Economics & ScalabilityOperations

Secure

[ 7 ]

5 Stories we have heard:

A researcher wants to move increasingly large file sets between collaborators in her field of study. She values fast transfers from her lab instruments to compute resources in the cloud and secure dissemination of results to students and other scientists.

Valuing simplicity in her own workflow, she appreciates campus IT/Security is positioned to move data as fast as possible, that the data is secured to only her collaborators and that she can be alerted if there are any anomalies in the data movement, security, etc.

Support the Data-Centric Researcher

As an end user of R&E infrastructure, operators and sophisticated research teams want to see software interfaces that can provision, change and support their own private network needs across the whole R&E ecosystem. Portal-driven configuration changes, customized telemetry for the private networks, and API–driven programmability allow them to build, monitor and change their own extended networks from their local compute cluster to their global collaborators and providers.

Support Software-Driven Infrastructure

A Cloud Architect plans increasing reliance on public cloud resources to augment and supplant the campus data center. She needs assurance that her campus has the reliability, resiliency, security and economics that enable rapidly evolving architectures. She plans to use the R&E networks that her campus has invested in, but needs new agility and end-to-end visibility for success.

Support Cloud Migration for Research and Administration

Campus & Regional networking needs evolve rapidly. Operators need capacity, visibility & agility to deliver services beyond traditional borders. By example, after establishing local content peering, remote peering become critical to resiliency and scalability. Sharing infrastructure within the R&E Community to extend the regional to national sites while reducing cost & increasing capacity.

Support Campus/Regional Network Ecosystem

[ 12 ]

To support continued growth in utilization with flat annual contributions, Internet2 must have updated equipment that brings efficiencies in power draw, space used, automation and maintenance.

Reset Internet2’s Operating &

Scale Economics

400G MOTRc 2018

400G line side1-slot

35 watts / 100G$262 per year

QSFP-DDc 2019

400G line sidePluggable

3.8 watts / 100g

$29 per year

OCI/OCLDc 2011

100G line side4-slots

363 watts

$1.361 per year

OTRc 2015

100G line side1-slots

195 watts$1463 per year

CFP2-DCOc 2018

200G line sidePluggable19 watts

$71.25 per year

9.5 watts / 100G

47.4

80.3 104.

120.

6

265.

1

351.

8

575.

4

694.

5

1099

.6

1298

.9 1842

.8

R² = 0.9943

0

1,000

2,000

3,000

4,000

5,000

6,000

CY 0

8

CY 0

9

CY 1

0

CY 1

1

CY 1

2

CY 1

3

CY 1

4

CY 1

5

CY 1

6

CY 1

7

CY 1

8

YTD

19

CY 2

0

CY 2

1

CY 2

2

CY 2

3

Internet2 Network Total PetaBytes Carried Per Year (Calendar Year)

Peta Bytes per year Poly. (Peta Bytes per year)

[ 13 ]

Story Solutions = R&E Value• Support the Data Intensive Researcher• Support Software Driven Infrastructure• Support Cloud for Research and Administration• Deliver Ecosystem-wide Solutions• Reset Internet2 Economics for Scale

[ 14 ]

2020 Ecosystem Expectations Taskforce

[ 15 ]

Proposed Activity: Community Expectations Task Force

• Draft Charter: A task force of leading architects from across the community will be convened to establish a set of criteria that each part of the community should aspire to support in order to fully participate in this new environment. This would include infrastructure, policy, operations, and service level expectations for the various constituencies across the community. The task force’s eventual report should lay out the criteria as well as potential implementation expectations

• Proposed Participants• Campus Cloud Architects Research Facilitators / CI Architects

• Network Architects Security Officers or Analysts

• Regional Network Leadership Campus IT and/or Research Leadership

• I2 Staff Liaisons to Quilt (regional) and Utah (research) efforts

[ 16 ]

Community Expectations Task Force - Examples

1999

• Multicast

• Jumbo Frames

• IPv6

• QoS

2019 Example Expectation Areas

• Private L2/L3 Networks

• Resiliency

• Headroom

• Security

• Telemetry

• Performance/Measurement

• DTN

• Cloud Connectivity

• Operations

• Network Basics: Jumbos, v6

• Others…

Lab

Campus

Regional

National

Global

[ 17 ]

Lab Campus Regional NationalSupported Research Data Transfer

Support for Cloud L2/L3 Networks

Data & Routing Security

Performance Management

Operations

Network Basics: IPv6, Jumbo Frames, etc

What do we expect from leading community members in 2022?

Illustrative Example

[ 18 ]

Next Generation Infrastructure 2019 and Beyond

[ 19 ]

ØService Requirements & Value ImprovementsØService Model (fees) Discussion

ØOptical RFP -> ImplementationØPacket RFI->Workshop->RFP

ØNetwork AutomationØService Orchestration

ØTestbed

19

Community Service Requirements

& Service Definitions

Automation &Software Planning

Member-Facing Service

Internal Operations

Infrastructure Planning

Optical, L2/3 Hardware Programs

Com

mun

ity

Colla

bora

tion Experim

ents

Pilots & Testbeds

Management& Governance

NGI Background: Where are we?

[ 20 ]

• Cloud Connect Portal– On Demand Layer 3 connectivity to Big-3 providers– Major value proposition according to campus CIO’s

– Over 50 campuses already piloting service

• TR-CPS “Cap” raised to 50G / Peering Exchange Upgrades to 100G in progress

• Optical RFP released, received and under evaluation

• Router Slicing, Routed Service Futures, Alien Wave Pilots, Open Science Grid/Stashcache, NRP Pilot

NGI: Wins since Global Summit 2018

[ 21 ]

Interconnects Internet2 members to cloud and commercial peers at 7 locations across the country

• Procured hardware to upgrade the two largest sites to move peering ports from dozens of 10G to 48 x100G

• Reclaimed 100G hardware will move to the other 5 interconnection sites

• Enables upgrades of our fastest growing hardware

• New Hardware costs 1/6 the cost per 100G of current platform, 1/3 less maintenance costs

NGI: Interconnection Upgrade (Underway 4Q-18 to 1Q-19)

[ 22 ]

New opportunity to automate operations & service delivery; reduce time to service; enable sharing

• Released cloud-connect portal (OESS 2.0) - Added self-service cloud provisioning to Amazon and Google Direct connect & API features that reduce time to results and enable infrastructure sharing

• Demonstrated new telemetry-based path control at TechEX with Arista Networks

NGI: Software, Systems, Automation

[ 23 ]

New opportunity to automate operations & service delivery; reduce time to service; enable sharing

• Released cloud-connect portal (OESS 2.0) - Added self-service cloud provisioning to Amazon and Google Direct connect & API features that reduce time to results and enable infrastructure sharing

• Demonstrated new telemetry-based path control at TechEX with Arista Networks

NGI: Software, Systems, Automation

Cloud Connect Portal 2.02

[ 24 ]

Cloud Connect PortalOESS 2.02

Demonstrations“Mint” Room

Wednesday, March 62:15-2:45

Thursday, March 710:00-10:30

Thursday, March 72:30-3:15

[ 25 ]

Where are we?

Optic

al

Pack

et

Auto

matio

n

Self-

Serv

ice P

orta

l &

Tele

metr

y

[ 26 ]

2019 Infrastructure Activities

27

Com

munity R

equirements & Pilots

ProcessProgram

Organization &

Developm

entPeering Im

provements

Optical R

FPC

loud-Connect Service Introduction

REC

INN

S / Routed Services Futures

& Node Slicing Pilot R

eportsN

ational Research Platform

PilotTR

-CPS C

AP Increase / Interconnect Infrastructure Im

provements

2016-2018 2019 2020 2021

Introduce Additional Self Service Functions

Finish Optical Upgrade / Major

progress on Packet Infrastructure & Sharing

Progress on implem

enting new

service model

CapitalBudget: $9M $15M $4.2M

Capital Budget Buckets

Cloud Connect Portal Now Production

Orchestration / Automation Strategy

Settled

Interconnect Upgrades Begun

Optical Upgrade Begins

Final Orchestration Components

Completed

Final Packet and Optical Upgrades Com

plete

[ 28 ]

Today:

10, 40 & 100G waves

“Dark Channel” with Ciena Equipment

Fiber/System Sharing in bulk agreements

Fixed 50 Ghz spectrum

Distance & hardware based pricing

Advanced Layer 1 ServiceLikely Tomorrow:

100, 200, 400, 800 G waves

Support 3rd party transponders

Fiber/System sharing in bulk agreements

Flex 50+ Ghz spectrum

Custom Telemetry

Updated segment fees

Available 2020

[ 29 ]

Underlying photonic transport system for nationwide community wavelength sharing

• Implement new nationwide optronic kit to support:– >50Ghz media channels– 200-800G transponders – inter-network optical connections and foreign waves

• Drive out scaling costs / find efficiencies

• Address sub-optimal OSNR on some paths

NGI: Optical Photonics Investment (2019 Activity)

• >50 ghz / 35 Gbpd• Disaggregated• Programmable• Efficient

[ 30 ]

• 13 proposals, 8 manufacturers received Feb 5• Community team is reviewing

– Optical Capabilities Costs (Opex, Capex)– Software & Support Systems Partnership

• Looking more deeply at optical characteristics of proposals, particularly to carry pluggable optics (launch power) and for non-premium foreign waves (OSNR and Raman/EDFA)

• Also discussing implementation services

• Targeting BAFO process in a few weeks

NGI: RFP Evaluation & Shift to Implementation

• >50 ghz / 35 Gbpd• Disaggregated• Programmable• Efficient

[ 31 ]

• Open Line System Upgrade First– Request for Proposal Responses Received February 5, 2019

• 12 Proposals representing 8 Equipment Manufacturers Under Evaluation• Next Step is Best and Final Offers for equipment with top 3-4 potential partners• Also reviewing operational impact and professional services to smooth transition

• Optical Transponder & Pluggable Capacity will come next– New 200G-800G Transponder Platforms part of RFP responses– Also expect pluggable 400G DWDM modules to begin shipping in late 2019

• Expect to have first segments upgraded late in 2019

Advanced Layer 1 Service

[ 32 ]

Today:

10 & 100G access

Uniform delivery typically by local router

R&E, Peering, Special Services blended in to uniform connection-based pricing

Cloud Connect Portal to AWS, Google at L2 and L3

Advanced Layer 2/3 ServiceLikely Tomorrow:

10, 100, Nx100G, 400 G connection offerings

Differentiation of connector platform and peering portfolio fee models

Backhaul and “local router” options

On demand, portal and API driven layer 2 and layer 3 services, including cloud connect

Enhanced self-service & telemetry

[ 33 ]

• High Level Request for Information to be released shortly (next week?)– Asks leading providers to envision a 2022 R&E Network– Asks providers to describe an integrated, automated, efficient platform for the

ecosystem (packet platform, optronics, controllers, portals, telemetry, etc)

• Possible Community Collaboration Workshop, mid-summer, to finalize design options prior to release of Request for Proposals late summer

• Ideally evaluating proposals in the 3rd quarter for award in 4th quarter

Advanced Layer 2/3 Service

[ 34 ]

Focus: Peering Portfolio Building BlocksEnhanced Cloud Delivery & Cloud Performance Services- Continue training, outreach, topology expansion and service enhances to Cloud Connect Service- perfSONAR or other performance tool on demand at cloud-edge

Rapid Deployment 10G & 100G PNI Ports- low cost 10G and 100G dedicated ports at the peering points for connectors to do direct-connects at remote sites

Cloud Router / Shared Router / Dedicated Router- Several options for connectors to deploy a virtual or physical router at a remote peering point at lower TCO

Virtual Network Function Hosting- Virtual machines available at peering sites for connectors to run software of their choice on their private network

Clean Pipe Services- Internet2 services delivered “scrubbed” with appropriate reports on attacks, threats, etc.

Enhanced self-service & telemetry- Great self-configuration & telemetry associated with services and API’s to drive the network from applications.

Targeted Telemetry

Targeted Portals

Rapid

Agile

Automated

[ 35 ]CCP/OESS2

Building Peer and Cloud Connectivity…

Internet2Regional

Campus

Campus

I2PXCloud Portfolio Options Today

[ 36 ]CCP/OESS2

Building Peer and Cloud Connectivity…

Internet2Regional

Campus

Campus

I2PX

Regional Peering

New Shared Cloud Portfolio Options

[ 37 ]

Building Peer and Cloud Connectivity…

Rapid Deploy 10G

Internet2Regional

Campus

Campus

Cloud Router orDedicated Router

I2PXNew Dedicated Cloud Portfolio Options

Regional Peering

[ 38 ]

Building Peer and Cloud Connectivity…Example 1: A regional wants to establish 20G connectivity in Chicago

and 20G in Sunnyvale for peering and dedicated cloud connect• Use 4 10G Layer 2 Rapid Deploy ports and leverage L3VRF Portal• Internet2 backbone backhauls traffic to the regional ports• No remote colo, power, hardware or transport• Regional pays peering/cross connects as if they were there• Cost estimate for service is <$50K/yr for 40G

Example 2: A regional wants 100G peering in Chicago w/their own remote router• Add a pair of resilient 100G Internet2 Rapid Deploy port in Chicago• Internet2 provides partial rack space, management Ethernet port• Internet2 provides 20 hours/year remote hands• Internet2 backbone to backhaul traffic to the regional• Regional provides router• Regional pays peering/cross connects as if they were there• Cost estimate for service is <$60K/yr for 2x100G

[ 39 ]

Service Model (Fee) Subcommittee Activities

[ 40 ]

• Implementation of NGI provides opportunity for community to consider changes to the 2013 fee model and value proposition

• Focus is enabling adoption of required value proposition:– Support for Data Intensive Researcher

– Support for Movement to the Cloud (R&AE)

– Support Software Driven Infrastructure

– Ecosystem View

– Reset Internet2 Economics

NGI Service Model Discussion

[ 41 ]

• Complete R&E routing table to support member to member collaboration (Required)• Included software-enabled access via the Cloud Connect Portal to the (currently

three) major cloud IaaS providers• Access to the Internet2 Peering Exchange [I2PX] & Net+ Peers (formerly TR-CPS) • On demand point to point and multipoint Layer 2 connections• On demand point to point and multipoint Layer 3 networks• Implementation of software tools that will leverage telemetry and measurement data

to improve network efficiency and performance

Highlights of the “Platform Fee” model concept

[ 42 ]

2019 Software, Automation & OrchestrationProgram Activities

[ 43 ]

• Goal: Automate internal processes & configurations– consistency, rapid delivery

• Goal: Add self-service & API features – reduce time to results – enable infrastructure sharing

• Goal: Update measurement & analytics tools

• Goal: Provide leading network security capabilities – Enable and protect science & administrative workflows

• 2019 Activity: Testbed to begin to model these activities in conjunction with Packet Network Development

NGI: Software, Systems, Automation

[ 44 ]

• Automation – In collaboration with GlobalNOC looking to jointly look at toolkits for configuration

management, policy management, etc

• Orchestration– Preliminary Market Study Complete– Packet RFI will include some

• Telemetry

• Performance Systems

NGI: Software, Systems, Automation

CoordinationFacilitate global

operational communication and

coordination between network operators

Maintain globally accessible up-to-date contact

information in common routing databases

Anti-spoofingPrevent traffic with spoofed source IP

addresses

Enable source address validation for at least single-

homed stub customer networks, their own end-users, and infrastructure

Guard Against Hijacks of Cloud ConnectionsProtect the R&E Ecosystem

Take MANRS ActionsFiltering

Prevent propagation of incorrect routing

information

Ensure the correctness of your own announcements and announcements from

your customers to adjacent networks with prefix and AS-

path granularity

Global Validation

Facilitate validation of routing information on a

global scale

Publish your data, so others can validate

[ 46 ]

46 Participating MANRS MembersMutually Agreed Norms for Routing Security

[ 47 ]

Next Generation Infrastructure High Level Program Update

Discussion?