National Backbone Extension:Rolling out Multisite Gigabit connections over
Fibre Optics
Brian Boyle & Mike Norris
Topics1. NBE – in context2. Aims of the project
3. What has been achieved
4. Under the bonnet
5. The bigger picture
6. Summary
7. Next steps
8. Discussion2
NBE – in context• Building on the NBN• New requirements – hybrid networking• New paradigm of service and support• Proposal and resources• Support from stakeholders and clients
3
Bandwidth
BA
C
Users
2Mbit/s Gigabit
Bandwidth Use
HPC facility
ICHEC traffic
ICHEC latency
General Internet
Daily traffic
Yearly traffic
NBE – the project• Project plan and management• Inventory of sites• Criteria, priorities• Surveys and schedules• Procurement, multivendor scenario
7
Project organisationJohn Boland CEO Project sponsor
Steering group Victor Reijs Mike NorrisJohn Boland
Eoin KennyProject Manger
WG4Network
management/provisionCormac/Daniel
Out sourcing company Brian N
Out of Band
WG5Config tool
Cillian Sharkey
WG3Network Design
Eoin KennyDaniel LeteOwen Byrne
WG6 PoP Planning
StephenColm
WG2Fibre procurement
Owen ByrneLouis TwomeyPaul Cosgrove
WG1Client LiasonMike NorrisBrian Boyle
WG0Project management
Eoin Kenny
WG7DWDM Network
SiemensEoin
Project goals
• A network for connecting clients, with the following features
• High bandwidth• Multi-service• Resilient• Scaleable• Flexible• Secure
9
Goals – Multi-service
– Service orientated connectivity, e.g:• Private secure connection per Institute to An Chéim• Interconnectivity with the Government VPN
– Not just focused on internet reachability– Capable of delivering access to multiple services using
the same infrastructure– Advanced networking protocols supported – IPv6,
multicast
10
Goals - Resilience
• Seek to build in circuit, equipment, path, and routing diversity
• Equivalent service levels on primary and secondary• Using multiple ducts, termination points, equipment• Using a robust routing technology (BGP) • Can make use of existing connectivity via other service
providers
11
Goals - Scalability• Can deliver multiple 1 Gigabit/s circuits• Can operate at line speed end to end• Capable of offering even higher speeds subject to
requirements (10 Gbps, higher)• 10 Gbit/s access for clients being explored• Technology refresh available for 40 Gbit/s…• …and similar for 100 Gbps in development
12
Goals - Flexibility• Not subject to telco business case scenarios• Not tied by contract to any particular network
implementation technology or model• Can use a mix of fibre and traditional managed Ethernet
circuits• Fully compatible with existing networks (no big bang
changes)• Capable of delivering wavelengths to clients if the need is
proven
13
Client perspective• Client priorities are key throughout• Aim is to keep change to client sites minimal• HEAnet installs managed routers and switches in
addition to current network equipment• When ready, services can be tested and migrated
individually• No Big Bang change• No change without explicit client agreement
14
Typical installation• Research of available providers (fibre, managed circuits),
usually via Tender process• Client site survey by provider with client
Paths identified, leading to “way leave” contracts• Civil works commence• Fibres are pulled into building, and subsequently spliced• Routers and switches are delivered, configured by HEAnet• Services can be tested and used by client
15
New link services• Point-to-point Ethernet links (p2p)
– 1 Gbit/s, Ethernet framed– Ethernet Port Service– Ethernet VLAN Service– No aggregation– Automatic monitoring/graphing
16
Ethernet Port and VLAN services
Some examples• Services
- connectivity- institutional access and security- p2p
• Applications- Oireachtas- ICHEC- An Chéim- UCLP
• Client examples- DCU
18
LAN Extensions – Ryan Academy• Centralise services, both voice and data• Servers centrally located at DCU• IP Telephony Extension to Nortel Meridian• Seamless telephony integration• Security cameras and DVR remote view and monitoring• Buildings Management System• High-End Video Conferencing > 2 Meg/sec• Reliability• No onsite technical support ( low maintenance )
19
LAN Extensions – Mater Dei Institute
• Sister College of DCU• All Servers consolidated in DCU• Previous difficulties with server location in MD
( Dust and no Aircon )• Minimal on-site technical support
( mainly helpdesk 2/3 days per week )• No requirement for Sysadmin• Reliability of service
20
Under the bonnet• DWDM, CWDM• Ethernet switches• MPLS• New routing strategies• Out of band access for network management• CPE, routers• Outsourcing of L2 ops management• Provisioning tool• Contributions, SLA
21
Multiple sources
• We make use of many fibre providers to connect at the lowest cost for our requirements– E-net– Dublin City Council– ESBT– BT Ireland– NTL– Aurora
22
Fibre network end 2006 – approx 2300km of fibre
MAN design13/01/06
National Backbone(10G)
Core Ring(10G)
1490nm
Galway PoP
Clinical Science(UCHG)
1490nm 1530nm 1610nm
1610nm1590nm1570nm
1570nm
1550nm1530nm1510nm
1510nm
1470nm
1470nm
mux/demux
mux/demux
Templates
Galway Regional λ Planning { } everything between these will form part of the future MAN ring. Currently only a point-to-point between NUIG and the Clinical Science building exists. This will later form part of the proposed ring.· 1470nm will be used initially to provide a 1 Gbit/s point-to-point link between UCHG and NUIG. This
will provide a resilient connection when the MAN ring is complete.· 1490nm will be used to provide a 1 Gbit/s resilient link between GMIT and NUIG (upon ring closure)· 1510nm will be used to provide a 1 Gbit/s resilient link between GMIT and NUIG (upon ring closure)· 10G resilient link between NUIG and ESB Telecoms aggregating n * 1Gbit/s from Access Ring· 1530nm will be used to provide a 1 Gbit/s resilient link between NUIG and NUIG’s Research Institute
(upon ring closure)
1550nm
GMIT
1590nm
1490nm
3750 Switch
mux/demuxmux/demux
AdvaFSP
ESBT PoP
Cisco7606
3750 Stack
MAN Ring(n * 1Gbit/s)
10G
1Gig TX
Regional
National
CAO
3750 Switch
1510nm
NUIGResearch Institute
3750 Switch
1530nm
3750 Switch
1470nm
Transmission technologies
• We use a heterogeneous mix of technologies to achieve our goals:– CWDM in metropolitan areas gives us multiplexed fibre
services for short hops– DWDM between regions to give the same for longer hops– MPLS network to give multi-service offering– Multiple physical connections to add resiliency
25
HEAnet backbone network topology
How does it do it?• Circuits are managed using an automated
provisioning system• Based on client requirements, the system is
used to configure routers, service monitoring portal
• Aim to have client use the tool directly, with administrator later checking/approving circuit change requests
27
Network provisioning tool
The bigger picture
• NIBEST – NW link with NIRAN• UKERNA DF – SuperJANET 5 links• GÉANT – hybrid networking• Transit – preparing for the next step• INEX – peerings and more
29
INEX traffic growth
• HEAnet-INEX (IPv4)
• Total INEX
IP bandwidth
1
10
100
1,000
10,000
100,000
IP b
andw
idth
(Mbp
s)
Access b/w
External b/w
IP bandwidth
0
5,000
10,000
15,000
20,000
25,000
30,000
35,000
40,000
45,000
IP b
andw
idth
(Mbp
s)
Access b/w
External b/w
Summary of changesBefore After
•Network consists of managed links connecting regional POPs•Clients used managed links to connect to PoPs•Service monitored by HEAnet (IP) and telco providers (physical)
•More PoPs, connected by Dark Fibre•Clients used Dark Fibre and/or managed links to connect to PoPs•Service monitored by HEAnet (IP, Ethernet) and Fibre providers (physical)
Benefits
• Improved bandwidth• Guaranteed uncontended bandwidth service
levels• Reduced cost• Flexibility• Improved access to HEAnet services
35
Outline of servicesNETWORK
MANAGEMENTHOSTING
SERVICESMULTIMEDIA
APPLICATIONS
Network Operations Centre, Helpdesk
Co-location - equipment hosting
Hi-quality IP Video-conferencing
Up-to-minute traffic and performance
Web hosting, Database hosting, Email, Domains...
Video & audio streaming over
Internet
Spam Control
tools, MAPS
Anti-virus trends
Telco liaison, fault resolution and client
support
Hot Standby, failover website facility
High quality streamed content
(e.g. Oireachtas Live, IPTV)
Public Key Infrastructure
Authenticate Authorise
Accounting
Performance and quality maintenance
Mirroring, Multimedia content central
storage
Schools connectivity to backbone
SECURITY
Computer Emergency Response Team
Security Scanning Service
Phase 2 progress, 2nd Nov
Confirmed 7 Cancelled 4 In progress 17 Completed 13
Total number of links 43
37
Dark fibre statistics
• No. of DF providers 5• Aggregate length of DF pair 2,300 km• No. of primary client sites
with DF links 18 (of 40)• Total sites with DF connections 31
38
Next steps
• New model, service levels• Phase 3• Client service reviews• Review of research infrastructures• Integrated e-infrastructures
39
Questions?
Project history• In 2004, we commenced implementation of dark
fibre point to point links• Followed by additional of CWDM and DWDM
equipment to increase service possibilities• Commenced work on an automated provisioning
tool to create and manage configurations• Quickly grew into a heterogeneous network
combining a mix of technologies and providers
41
Scalable capacity• Backbone – 10 Gbit/s• Can add more 10 Gbit/s using existing
infrastructure• Client connections at 1 Gbit/s
42
What has been achieved
• High levels of service to clients- high bandwidth- resilience – path, equipment, service
• Scalability now and in the future• Underpins e-infrastructure• Complex and flexible• Rich mix of services• Secure
43
Trends since 1991
1
10
100
1,000
10,000
100,000
1,000,000
Jan-
91
Jan-
93
Jan-
95
Jan-
97
Jan-
99
Jan-
01
Jan-
03
Jan-
05
Valu
e re
lativ
e to
199
1
HEAnet traffic
Access bandwidth
External bandwidth
HEAnet staff