internet2 presentation brian stengel, director of operations, kinber peter j. heverin, project...
TRANSCRIPT
“Internet2 Presentation”
Brian Stengel, Director of Operations, Kinber
Peter J. Heverin, Project Manager, KinberMike Carey, Network Engineer, Kinber
Jon Paul Herron, Director of Engineering, GlobalNOCLuke Fowler, Senior Manager, Systems Engineering, GlobalNOC
Marianne Chitwood, Director of Operations, GlobalNOC
September 20th, 2012
What is PennREN
• Capital Budget - $128,958,031– Federal Stimulus Funds - $99,660,678– Matching Funds - $29,297,353
• Outside Plant Infrastructure Constructed for PennREN – 48 Stands of NZD Fiber optic Cable– 1700± Route Miles– Outsourced Fiber Maintenance
• 13 Optical Regeneration Service Nodes
• 56 Service Distribution Access Nodes
a high-speed, state-wide, research & education serving healthcare, K-20 and the public good
Project Route
Total PennREN Fiber route is estimated at 1,613 miles.
The route consists of the following;
• 1,086 miles of new aerial construction.• 486 miles of aerial overlash.• 14 miles of new underground construction.• 27 miles of leased underground conduit.
Network Backbone Engineering is 95% complete overall.
3
Project Route
4
PennREN LocationsService Nodes
Lehigh University
401 N. Broad
Penn State Hershey Medical Center
Indiana University of Pennsylvania
Allegheny Center Mall
Slippery Rock University of PennsylvaniaPenn State Erie (Behrend)
Clarion University of Pennsylvania
Penn State Dubois
Penn State – State College
Windstream
Bucknell University
University of Scranton
Amp NodesEast Stroudsburg University of Pa
Kutztown University of Pa
West Chester University of Pa
Millersville University of Pa
Shippensburg University of Pa
Pitt Johnstown
Community College of Beaver Co
Allegheny College
Pitt Titusville
Penn State Hazelton
Access Nodes46 Locations on PennREN fiber
Project Teams
• Hardware procurement• Design• Integration• Project Management• Professional Services
• Plan/Design• Optical engineering• Field services• Professional services
• Staging• Shipping• Configuration• Testing• Turn-up• Field Services
• Plan/Design• Consulting• Engineering• Support• Training
• Fiber OSP• Building entrances• Cabinets, Installation• Splicing• Testing• Last mile solutions
• Earthlink sourcing for OOB MPLS Network
• Last mile solutions
• Service Desk, Tier 1, Tier 2• Systems and Network Tools• Network Engineering• Net, Perf, Change Management• Operations support
Node Installation Status
Segment in Production
Equipment Installed
Under Construction
Co-Location at Member Sites
Construction Diagrams
Optical Core/DWDM System
Packet Core – MPLS
Plan – Build – Integrate
Peering Points (Initial)
Internet
R&E
R&E
Internet
Network Management
External Networks
External Networks
Out-of-band access
In-band management
Performance Measurement Servers at every Service Node
PS1PS2PS3
DB
Operations - SupportGRNOC provides to KINBER:• Service Desk – 24x7x365 call center support, ticket management, technical
support coordination, and workflow support• Network Engineering – Expert network engineers work with the Service Desk
to ensure fast problem resolution, provisioning, and strategic engineering and planning
• Software and Systems – Provides support through a fully integrated system of network management, measurement, and visualization tools
The GlobalNOC at Indiana University provides carrier-grade operations, tools, and network expertise while placing a singular focus on the unique requirements of the research and education (R&E) Community
GRNOC supports 20+ R&E networks across the country
Operations - Maintenance
• Warranty• Hardware replacement
(pre-ship)• Software maintenance
• Warranty• Hardware
replacement (pre-ship)
• Software maintenance
• Fiber Maintenance• Emergency Restoral• Routine
Maintenance• OSP Records
• Host IT/Site teams provide remote eyes and hands support upon request
• KINBER Network Engineers
Foundation for Services
West DWDM Ring
East DWDM Ring
MPLS PE Switches
ASBR Router
Multi-DegreeROADMs
External Networks
External Networks
3rd Party Providers
On-Net: Delivery
Internet
R&E R&E
Internet
AccessNode
AccessNode
AccessNode
AccessNode
Access Nodes
Off-Net: Delivery
Internet
R&E R&E
Internet
MemberOff-Net
First/Last MileCommercial
Co-Lo
CXCarrier
Private
NNI
PennREN
EPC – Ethernet Port Connection Service
Customer Router
Service Node: PE Switch
Customer Subscribes to 1/10GE Ethernet Port(s)
R&E networks
VPLS (pt – pt, multipoint)
KMEX
Commodity Internet
Services• Services available to a customer with an EPC
R&E Transit
KINBER Member Exchange
VPLS – Pt-Pt, Multipoint
Commodity Internet
Member A
Member AMember C
Member B
Services• Community-wide, distributed service for members to
exchange traffic across a common network. Similar to an Internet Exchange
• Member-to-Member peering, ad-hoc R&E activities• Best-effort traffic exchange within the community
R&E Transit
KINBER Member Exchange
VPLS – Virtual Private LAN Service
(Point-to-PointMulti-Point)
Commodity Internet
• Transit service to major R&E networks such as Internet 2, ESNet, NLR…
• Provided by KINBER affiliates
• Virtual private networks with committed bandwidth can be established using VPLS instances in the PennREN network
• VPWS – Virtual Private Wire Service – Point to Point• VPLS – Virtual Private LAN Service - Multipoint
• Access to commodity Internet service is available over the PennREN network
Optical – Wave/Lambda Services
• Optical Waves 10G can be provisioned across the networkMember AMember A
Member AMember B Member B
• PerfSONAR Measurement Archives allow exchange of data with other network operators
• Regularly scheduled testing across the backbone
• User-initiated testing for applications such as problem diagnosis
• Multiple routing tables allow us to support both 1G and 10G testpoints on a single host
Performance Measurement
Performance Measurement
Deploying performance measurement servers at each service node
3 servers per site:– Active throughput measurement (1G and 10G)– Active latency measurement– local data collection / ad hoc performance measurement
Specific performance measurement tools include:
BWCTL
OWAMP
MaDDash
24
Network Measurement
• GlobalNOC tool-set updated to support all PennREN devices, including:– SNAPP – High-resolution SNMP-based network
utilization data– LLAMA – DWDM layer performance measurement
data– Central storage of other passive data like syslog,
configuration, flow, etc. for regular and ad-hoc processing & analysis.
PennREN Service Desk
• GlobalNOC Specialized Support Technician • Footprints PennREN project created – integrated to
TickMon, Operations Calendars, Trouble Tickets• Telephone number for PennREN customers integrated
into shared GlobalNOC phone queue• Email established [email protected] to receive customer
inquiries and or communications from vendors….this is monitored 24x7x365
PennREN Service Desk
• Network and Member impact guidelines defined
• Web form for customers/vendors to submit trouble tickets into Footprints
• Change management process/form created• Internal documentation developed/published
for staff training and reference
PennREN Service Desk
• Pro-active network monitoring• Support for scheduled maintenances and
changes• Vendor coordination• Customer install process• Reporting• Security• Tools and Communications
Weighing network needs
A new network is more than a construction project
• It’s a system:– Goals– Infrastructure– People– Services– Operations– Business
• Make decisions and plans early (also, there will be more decisions than you think, so watch for bottlenecks)
• The work on all of these areas starts right away
Service Definition
• Unrealistic to imagine services will be completely defined from the beginning of the design
• But, enough understanding is needed to guide the design and build plan
• Early “anchor” users help a lot!
Communications
• Communication needs:– to be high-bandwidth– Early is good– Changing staff can be disruptive, do it carefully.– Informal is good– Multi-channel is good– reliable technologies are good– Face to face is good
Documentation
• Documentation:– Have a place for documents– Keep the place for documents clean– Keep the purpose of each document clear/distinct
Operations
• Operations Preparedness:– Start early, there are a LOT of things to think
about, especially:• Turn-up/acceptance process• Expectations for facilities• How and where to keep network data
Build-outs make for lots of information
• Need to be ready to put it somewhere where it’ll be usable later
• Data entry is cheap• Entering/documenting is best when the
information and its context are fresh
The lowly management network
• Sometimes doesn’t get enough attention• It can get crazy complicated or crazy expensive• NOTE: in an SDN world, this becomes even
MORE important!