nanog55.talk43.christracy nanog55 100g jun2012
TRANSCRIPT
-
7/27/2019 Nanog55.Talk43.Christracy Nanog55 100g Jun2012
1/62
100G Deployment
Challenges & Lessons Learnedfrom the ANI Prototype & SC11
Chris Tracy, Network Engineer
ESnet Engineering Group
June 5, 2012
Version 1.2
-
7/27/2019 Nanog55.Talk43.Christracy Nanog55 100g Jun2012
2/62
Lawrence Berkeley National Laboratory U.S. Department of Energy | Office of Science
06/05/12
Title Misnomer
Talk title references a prototype 100G network
- This is no longer just a prototype
Rollout of a nationwide, production-quality IP/MPLSbackbone running at 100Gbps is underway
- Scheduled to transition to full production by Nov 2012
2
-
7/27/2019 Nanog55.Talk43.Christracy Nanog55 100g Jun2012
3/62
Lawrence Berkeley National Laboratory U.S. Department of Energy | Office of Science
06/05/12
Outline
ANI (100G Prototype) and ESnet5 (100G Production)
100Gbps optical transmission over the WAN
100Gigabit Ethernet Transmission
Pluggable optics
Testing, measurement, debugging, fault isolation
Interoperability
Case Study
References
3
-
7/27/2019 Nanog55.Talk43.Christracy Nanog55 100g Jun2012
4/62
Lawrence Berkeley National Laboratory U.S. Department of Energy | Office of Science
06/05/12
Advanced Networking Initiative (ANI)
4
100GigE prototype circuits
-
7/27/2019 Nanog55.Talk43.Christracy Nanog55 100g Jun2012
5/62
Lawrence Berkeley National Laboratory U.S. Department of Energy | Office of Science
06/05/12
Advanced Networking Initiative (ANI)
4
~13,000mi dark fiber
-
7/27/2019 Nanog55.Talk43.Christracy Nanog55 100g Jun2012
6/62
Lawrence Berkeley National Laboratory U.S. Department of Energy | Office of Science
06/05/12
Advanced Networking Initiative (ANI)
4
experimental testbed
-
7/27/2019 Nanog55.Talk43.Christracy Nanog55 100g Jun2012
7/62
Lawrence Berkeley National Laboratory U.S. Department of Energy | Office of Science
06/05/12
Advanced Networking Initiative (ANI)Seven 100GigE Circuits for SC11
5
optical add/drops and/or regens
* Geography is only representational
-
7/27/2019 Nanog55.Talk43.Christracy Nanog55 100g Jun2012
8/62
Lawrence Berkeley National Laboratory U.S. Department of Energy | Office of Science
06/05/12
Advanced Networking Initiative (ANI)Seven 100GigE Circuits for SC11
5
optical add/drop and router
* Geography is only representational
-
7/27/2019 Nanog55.Talk43.Christracy Nanog55 100g Jun2012
9/62
Lawrence Berkeley National Laboratory U.S. Department of Energy | Office of Science
06/05/12
ESnet5 - Optical Network
6
~13,000mi lit fiber
* Geography is only representational
-
7/27/2019 Nanog55.Talk43.Christracy Nanog55 100g Jun2012
10/62
Lawrence Berkeley National Laboratory U.S. Department of Energy | Office of Science
06/05/12
ESnet5 - Optical Network
6
wave capacity shared with Internet2
* Geography is only representational
-
7/27/2019 Nanog55.Talk43.Christracy Nanog55 100g Jun2012
11/62
Lawrence Berkeley National Laboratory U.S. Department of Energy | Office of Science
06/05/12
ESnet5 - Optical Network
6* Geography is only representational
over 200 amp sites
-
7/27/2019 Nanog55.Talk43.Christracy Nanog55 100g Jun2012
12/62
Lawrence Berkeley National Laboratory U.S. Department of Energy | Office of Science
06/05/12
ESnet5 - Optical Network
6* Geography is only representational
over 60 optical add/drop sites15 "directionless" add/drop
-
7/27/2019 Nanog55.Talk43.Christracy Nanog55 100g Jun2012
13/62
Lawrence Berkeley National Laboratory U.S. Department of Energy | Office of Science
06/05/12
ESnet5 - Optical Network
6* Geography is only representational
23 100G s (metro & wide-area)
-
7/27/2019 Nanog55.Talk43.Christracy Nanog55 100g Jun2012
14/62
Lawrence Berkeley National Laboratory U.S. Department of Energy | Office of Science
06/05/12
ESnet5 - Optical Network
6* Geography is only representational
46 100G add/drop transponders22 100G re-gens across wide-area
-
7/27/2019 Nanog55.Talk43.Christracy Nanog55 100g Jun2012
15/62
Lawrence Berkeley National Laboratory U.S. Department of Energy | Office of Science
06/05/12 7* Geography is only representational
ESnet5 - Routed NetworkScheduled for Production by November 2012
-
7/27/2019 Nanog55.Talk43.Christracy Nanog55 100g Jun2012
16/62
Lawrence Berkeley National Laboratory U.S. Department of Energy | Office of Science
06/05/12 7* Geography is only representational
ESnet5 - Routed NetworkScheduled for Production by November 2012
20 routers with 100G interfaces
-
7/27/2019 Nanog55.Talk43.Christracy Nanog55 100g Jun2012
17/62
Lawrence Berkeley National Laboratory U.S. Department of Energy | Office of Science
06/05/12 7* Geography is only representational
ESnet5 - Routed NetworkScheduled for Production by November 2012
52 layer-3 100GigE interfaces
-
7/27/2019 Nanog55.Talk43.Christracy Nanog55 100g Jun2012
18/62
Lawrence Berkeley National Laboratory U.S. Department of Energy | Office of Science
06/05/12 7* Geography is only representational
ESnet5 - Routed NetworkScheduled for Production by November 2012
customer-owned 100G routers
-
7/27/2019 Nanog55.Talk43.Christracy Nanog55 100g Jun2012
19/62
Lawrence Berkeley National Laboratory U.S. Department of Energy | Office of Science
06/05/12 7* Geography is only representational
ESnet5 - Routed NetworkScheduled for Production by November 2012
100G exchange point interconnects
-
7/27/2019 Nanog55.Talk43.Christracy Nanog55 100g Jun2012
20/62
Lawrence Berkeley National Laboratory U.S. Department of Energy | Office of Science
06/05/12
100G Optical Transmission over the WAN
System Perspective
100G Ethernet client signals - IEEE 802.3ba [2]
- on core-facing andcustomer/peer-facing edge ports
- weve standardized on 10x10-10km & 10x10-2km CFPs
10x10 MSA [5]
non-IEEE standard
100G client signals mapped into Optical Transport Unit 4 (OTU4)
- ITU-T G.709 [3] for encapsulating 100GigE into OTU4
- includes mandatory Forward Error Correction (FEC)
transported using dual polarization-quadrature phase shift keying(DP-QPSK) technology with coherent detection [4]
8
-
7/27/2019 Nanog55.Talk43.Christracy Nanog55 100g Jun2012
21/62
Lawrence Berkeley National Laboratory U.S. Department of Energy | Office of Science
06/05/12
100G Optical Transmission over the WAN
Dual polarization-quadrature phase shift keying (DP-QPSK)
DP: two independent optical signals, same frequency, orthogonal
- single transmit laser, each signal carries half of the data
- two polarizations ! lower modulation rate ! reduce optical bandwidth
- allows 100G payload (plus overhead) to fit into 50GHz of spectrum
9source: [1] and [4]
phase-amplitude constellation2 bits per symbol
Q
I
signals added together to form QPSK signal
-
7/27/2019 Nanog55.Talk43.Christracy Nanog55 100g Jun2012
22/62
Lawrence Berkeley National Laboratory U.S. Department of Energy | Office of Science
06/05/12
100G Optical Transmission over the WAN
Dual polarization-quadrature phase shift keying (DP-QPSK)
QPSK: encode data by changing the phase of the optical carrier
- compare to on-off keying (OOK), intensity modulation ! 0=off, 1=on
- further reduces the symbol rate by half, sends twice as much data
Together, DP and QPSK reduce required rate by a factor of 4
10source: [1] and [4]
phase-amplitude constellation2 bits per symbol
signals added together to form QPSK signal
Q
I
-
7/27/2019 Nanog55.Talk43.Christracy Nanog55 100g Jun2012
23/62
Lawrence Berkeley National Laboratory U.S. Department of Energy | Office of Science
06/05/12
Spectral Efficiency: 10 vs 40 vs 100Gb/s
11
0.15nm/div0.4nm! 50GHz
1551.
68
1551.
83
1551.53
1551.
98
1552.
13
1552.
28
1552.
43
1552.58
1552.73
1552.
88
1553.
03
nm
Source: [1] Roberts, Beckett, Boertjes, Berthold and Laperle (2010, July)
-
7/27/2019 Nanog55.Talk43.Christracy Nanog55 100g Jun2012
24/62
Lawrence Berkeley National Laboratory U.S. Department of Energy | Office of Science
06/05/12
Spectral Efficiency: 10 vs 40 vs 100Gb/s
11
0.15nm/div0.4nm! 50GHz
20GHz
14GBaudeach
10G single-polarization single-carrier 40G coherent dual-polarization single-carrier
100G coherent dual-polarization dual-carrier
1551.
68
1551.
83
1551.53
1551.
98
1552.
13
1552.
28
1552.
43
1552.58
1552.73
1552.
88
1553.
03
nm
Source: [1] Roberts, Beckett, Boertjes, Berthold and Laperle (2010, July)
-
7/27/2019 Nanog55.Talk43.Christracy Nanog55 100g Jun2012
25/62
Lawrence Berkeley National Laboratory U.S. Department of Energy | Office of Science
06/05/12
Spectral Efficiency: 10 vs 40 vs 100Gb/s
11
0.15nm/div0.4nm! 50GHz
20GHz
14GBaudeach
10G single-polarization single-carrier 40G coherent dual-polarization single-carrier
100G coherent dual-polarization dual-carrier
1551.
68
1551.
83
1551.53
1551.
98
1552.
13
1552.
28
1552.
43
1552.58
1552.73
1552.
88
1553.
03
nm
1551
.72
1552.
12
155
2.
52
155
2.
93
50GHz
Source: [1] Roberts, Beckett, Boertjes, Berthold and Laperle (2010, July)
-
7/27/2019 Nanog55.Talk43.Christracy Nanog55 100g Jun2012
26/62
Lawrence Berkeley National Laboratory U.S. Department of Energy | Office of Science
06/05/12
100G Optical Transmission over the WAN
Coherent Detection (on the receiver) - Breakthrough Technology
Technology originally developed circa 1980s - [1] and [7]
- advances in digital signal processing (# of gates, operatingfrequency) allowed coherent detection to emerge as disruptive
technology in optical communications Offers significant improvement in noise tolerance over conventional
direct detection schemes
Able to compensate for propagation impairments such as chromaticdispersion (CD) and polarization mode dispersion (PMD)
- dispersion compensating fibers are no longer needed in long-haulapplications
high-speed A/D samples incoming analog components
- DSP ASIC to apply numerical adaptations, recover signal
12
-
7/27/2019 Nanog55.Talk43.Christracy Nanog55 100g Jun2012
27/62
Lawrence Berkeley National Laboratory U.S. Department of Energy | Office of Science
06/05/12
100G Optical Transmission over the WAN
Coherent Detection - Another Breakthrough
works like a radio receiver - source: [1], [4], and [7]
- use a strong local oscillator tuned to the frequency of interest
has provided a breakthrough in optical filtering capabilities
- tunable optical filters exist but have never been cost-effective
important impacts of this technology:
- colorless ROADMs are [finally] becoming available by usingcoherent optical filtering - complete software reprogrammability
- no longer need fixed channel filters (arrayed waveguide gratings)
- no longer important to stay aligned to a 50 or 100 GHz ITU grid
- spectrum can be used more efficiently, system is more flexible
- wavelength selective elements would need to become flexible
13
-
7/27/2019 Nanog55.Talk43.Christracy Nanog55 100g Jun2012
28/62
Lawrence Berkeley National Laboratory U.S. Department of Energy | Office of Science
06/05/12
100G Optical Transmission over the WAN
Amp placement and fiber span length has become very important
100G transmission for a given channel of optical spectrum isapproaching Shannon limit
- going higher than 100Gb/s, at the same distance, in same amount of
spectrum, requires improvement of SNR (or just use more spectrum) Even if we just want to stay at 100Gb/s, reach is limited by OSNR
- EDFAs contribute noise - Amplified Spontaneous Emission (ASE)
- on a long segment consisting of cascaded EDFAs, a few very long(high-loss) fiber runs degrades the systems SNR
- could break up long spans and place more amplifiers for less lossbetween amps, or possibly deploy Raman, to improve SNR
- otherwise, extra 100G re-gens could be required to get the desiredreach - might not get advertised reach if amp spacing is not ideal
14
-
7/27/2019 Nanog55.Talk43.Christracy Nanog55 100g Jun2012
29/62
Lawrence Berkeley National Laboratory U.S. Department of Energy | Office of Science
06/05/12
Flexible Grid Concept
15Source: [8] Gringeri, Basch, Shukla, Egorov and Xia (2010, July)
At rates >100Gb/s, 50GHz spacing would require high SNR, will limit reach
Future-proof network by allowing channels to occupy more spectrum
-
7/27/2019 Nanog55.Talk43.Christracy Nanog55 100g Jun2012
30/62
Lawrence Berkeley National Laboratory U.S. Department of Energy | Office of Science
06/05/12
100G Ethernet Transmission(IEEE 802.3ba)
16Source: [11] Drolet and Duplessis (2010, July)
Unlike 1G or 10G, 100G has embraced parallelism throughout its design
-
7/27/2019 Nanog55.Talk43.Christracy Nanog55 100g Jun2012
31/62
Lawrence Berkeley National Laboratory U.S. Department of Energy | Office of Science
06/05/12
100G Ethernet Transmission(IEEE 802.3ba)
16Source: [11] Drolet and Duplessis (2010, July)
Unlike 1G or 10G, 100G has embraced parallelism throughout its design
physical coding sublayer (PCS)provides reordering & realignment
-
7/27/2019 Nanog55.Talk43.Christracy Nanog55 100g Jun2012
32/62
Lawrence Berkeley National Laboratory U.S. Department of Energy | Office of Science
06/05/12
100G Ethernet Transmission(specific to 100GBASE-LR4)
17Source: [11] Drolet and Duplessis (2010, July)
LR4
-
7/27/2019 Nanog55.Talk43.Christracy Nanog55 100g Jun2012
33/62
Lawrence Berkeley National Laboratory U.S. Department of Energy | Office of Science
06/05/12
100G Ethernet Transmission(specific to 100GBASE-LR4)
17Source: [11] Drolet and Duplessis (2010, July)
physical medium attachment (PMA)(often referred to as a gearbox)
LR4
-
7/27/2019 Nanog55.Talk43.Christracy Nanog55 100g Jun2012
34/62
Lawrence Berkeley National Laboratory U.S. Department of Energy | Office of Science
06/05/12
100G Ethernet Transmission(specific to 100GBASE-LR4)
17Source: [11] Drolet and Duplessis (2010, July)
physical medium attachment (PMA)(often referred to as a gearbox)
impact of gearbox on data ordering
LR4
-
7/27/2019 Nanog55.Talk43.Christracy Nanog55 100g Jun2012
35/62
Lawrence Berkeley National Laboratory U.S. Department of Energy | Office of Science
06/05/12
100G Ethernet Transmission(specific to 100GBASE-LR4)
17Source: [11] Drolet and Duplessis (2010, July)
physical medium attachment (PMA)(often referred to as a gearbox)
possibility of skew between wavelengths
impact of gearbox on data ordering
LR4
-
7/27/2019 Nanog55.Talk43.Christracy Nanog55 100g Jun2012
36/62
Lawrence Berkeley National Laboratory U.S. Department of Energy | Office of Science
06/05/12
CFP Pluggable Optics - LR4 or 10x10 ( )?
18
Interface Wavelength Cost Fiber Type Connector Reach *Link
BudgetComment
100GBASE-LR44 x 25G WDM lanes1294.53-1310.19nm $$$ SMF SC / LC 10 km 7.3 dB IEEE Standard [2]
10x10-2km10x10-10km
10 x 10G WDM lanes1521-1597nm
$$ SMF SC / LC2 km
10 km5.1 dB8.0 dB
non-IEEE standard, no 10:4gearbox, 10x10 MSA [5]
100GBASE-SR1010 x 10G parallel MMF
840-860nm$
parallelMMFs
MPO / MTP100 / 150 mOM3/OM4 MMF
8.3 dB IEEE Standard [2]
10x10 LR10 optics are gaining popularity and vendor support
Does your equipment support third-party optics?
Similar to 1G & 10G pluggables, vendors sell certified optics
3rd party may not work (or be supported) by your equipment
Digital Diagnostic Monitoring (optical power monitoring) may or may not be supported, especially with uncertified optics
if supported - may be aggregate, per-lane, or both
* reach is highly dependent on link budget. some modules may have higher-than-average launch power.
10x10LR10CFP
akaLR10
-
7/27/2019 Nanog55.Talk43.Christracy Nanog55 100g Jun2012
37/62
Lawrence Berkeley National Laboratory U.S. Department of Energy | Office of Science
06/05/12
CFP Pluggable Optics - LR4 or 10x10 ( )?
18
Interface Wavelength Cost Fiber Type Connector Reach *Link
BudgetComment
100GBASE-LR44 x 25G WDM lanes1294.53-1310.19nm $$$ SMF SC / LC 10 km 7.3 dB IEEE Standard [2]
10x10-2km10x10-10km
10 x 10G WDM lanes1521-1597nm
$$ SMF SC / LC2 km
10 km5.1 dB8.0 dB
non-IEEE standard, no 10:4gearbox, 10x10 MSA [5]
100GBASE-SR1010 x 10G parallel MMF
840-860nm$
parallelMMFs
MPO / MTP100 / 150 mOM3/OM4 MMF
8.3 dB IEEE Standard [2]
10x10 LR10 optics are gaining popularity and vendor support
Does your equipment support third-party optics?
Similar to 1G & 10G pluggables, vendors sell certified optics
3rd party may not work (or be supported) by your equipment
Digital Diagnostic Monitoring (optical power monitoring) may or may not be supported, especially with uncertified optics
if supported - may be aggregate, per-lane, or both
* reach is highly dependent on link budget. some modules may have higher-than-average launch power.
10x10LR10CFP
akaLR10
-
7/27/2019 Nanog55.Talk43.Christracy Nanog55 100g Jun2012
38/62
Lawrence Berkeley National Laboratory U.S. Department of Energy | Office of Science
06/05/12
Pluggable Optics - Optical Power
19
Interface Lanes Average Launch PowerEach Lane (min / max) Total Average Launch Power(max) Source
100GBASE-LR4 4 -4.3 to 4.5 dBm +10.5 dBm [2]
10x10-2km10x10-10km
10-6.9 to 3.0 dBm (2km)
-5.8 to 3.0 dBm (10km)+13 dBm [5]
Field Testing of 100G Transceivers
CFPs will appear to launch hot* with standard power meters
testing spectrum compliance and per-lane optical power withparallel optics on SMF requires use of an Optical Spectrum Analyzer
some power meters may be tunable to measure different "s(consider passband width, marginal for spectrum compliance)
every CFP will have a slightly different Tx lane transmit profile
- maximum reach over dark fiber may vary slightly depending onthis profile
* we are referring to optical launch power, but weve found CFPs also run very hot, temperature-wise
-
7/27/2019 Nanog55.Talk43.Christracy Nanog55 100g Jun2012
39/62
Lawrence Berkeley National Laboratory U.S. Department of Energy | Office of Science
06/05/12
Pluggable Optics - Other Considerations
20
100G Fiber Patches
10x10 LR10: can generally* be connected back-to-back without attenuation
SR10: MPO-terminated parallel MMF for short patches (we havent tried this)
- going through patch panels would be messy
- break-out using 1xMPO to 10xSC octopus cables?
CXPs
high-density, targeting MMF connections
to be interoperable with CFPs
active optical cables
* check the data sheets for your transceivers or talk to your vendor first, of course
source: [6]
-
7/27/2019 Nanog55.Talk43.Christracy Nanog55 100g Jun2012
40/62
Lawrence Berkeley National Laboratory U.S. Department of Energy | Office of Science
06/05/12
Testing and Measurement
October 3rd, 2011, 14:07 Pacific
First pings across transcontinental 100GigE between two routers
41 days left before the start of SC11
- Want to give users as much time as possible to tune their
applications
Ping is a good first step, need more testing before hand-off to users
- Transport system shows clean FEC, low BER, optical layer clean
- We want to drive links at 100% utilization and measure 0 drops
- Validate QoS, throughput, latency, loss, interoperability, etc.
How are we going to test seven 100GigE circuits?
- At this point, work on building/deploying hosts capable of sourcing>10Gbps of traffic had begun, still preliminary
- Do we need a 100GigE hardware tester?21
-
7/27/2019 Nanog55.Talk43.Christracy Nanog55 100g Jun2012
41/62
Lawrence Berkeley National Laboratory U.S. Department of Energy | Office of Science
06/05/12
Testing and Measurement - UDP flows
Could we leverage the 10Gbps systems we already have deployed?
22
SUNN-ANI
perfSONAR
SALT-ANI100G
SUNN-SDN210G
ESnet4
ANI
2x10G
-
7/27/2019 Nanog55.Talk43.Christracy Nanog55 100g Jun2012
42/62
Lawrence Berkeley National Laboratory U.S. Department of Energy | Office of Science
06/05/12
Testing and Measurement - UDP flows
Could we leverage the 10Gbps systems we already have deployed?
22
SUNN-ANI
perfSONAR
SALT-ANI100G
SUNN-SDN210G
ESnet4
ANI
2x10G
lets have some fun with routing loops
static-route 10.10.10.0/30next-hop [sunn-ani]
static-route 10.10.10.0/30
next-hop [salt-ani]
static-route 10.10.10.0/30
next-hop [sunn-ani]
-
7/27/2019 Nanog55.Talk43.Christracy Nanog55 100g Jun2012
43/62
Lawrence Berkeley National Laboratory U.S. Department of Energy | Office of Science
06/05/12
Testing and Measurement - UDP flows
Could we leverage the 10Gbps systems we already have deployed?
22
2x2Gbps UDP iperfTTL=52 to loop 50X SUNN-ANI
perfSONAR
SALT-ANI100G
SUNN-SDN210G
ESnet4
ANI
2x10G
lets have some fun with routing loops
hop 1
hop 2,4,6,...,52 hop 3,5,...,51
iperf -c 10.10.10.1 -u -b 2G -l8800 -i1 -t600 -T52
iperf -c 10.10.10.2 -u -b 2G -l8800 -i1 -t600 -T52
and carefully chosen TTLs
loop 50x
-
7/27/2019 Nanog55.Talk43.Christracy Nanog55 100g Jun2012
44/62
Lawrence Berkeley National Laboratory U.S. Department of Energy | Office of Science
06/05/12
Testing and Measurement - UDP flows
This was a quick solution to saturate the link, can we improve it?
23
ANI
SALT-ANI STAR-ANI AOFA-ANI
ANL-ANI
-
7/27/2019 Nanog55.Talk43.Christracy Nanog55 100g Jun2012
45/62
Lawrence Berkeley National Laboratory U.S. Department of Energy | Office of Science
06/05/12
Testing and Measurement - UDP flows
This was a quick solution to saturate the link, can we improve it?
23
use policy-based routing for more complex loops - still testing UDP
firewall ACLs for counting packets / bytes to measure loss
ANI
SALT-ANI STAR-ANI AOFA-ANI
ANL-ANI
ip-filter applied at ingress:
match src-ip [perfSONAR IP]
count number of packets and bytes
action forward next-hop [anl-ani]
ip-filter applied at ingress:
match src-ip [perfSONAR IP]
count number of packets and bytes
action forward next-hop [salt-ani]
test flows
-
7/27/2019 Nanog55.Talk43.Christracy Nanog55 100g Jun2012
46/62
Lawrence Berkeley National Laboratory U.S. Department of Energy | Office of Science
06/05/12
Testing and Measurement - TCP flows
Deployed systems on ANI experimental testbed - source/sink 4x10Gbps
24
48.6ms RTT, 97.9Gbps aggregate TCP throughput with 10 TCP streams
Thanks to Eric Pouyoul, Brian Tierney, and many others
nersc-ani
nersc-diskpt-1
eth2
eth3
eth4
eth5
nersc-diskpt-2
eth2
eth3
eth4
eth5
nersc-diskpt-3
eth2
eth3
eth4
eth5
anl-mempt-1
eth2
eth3
eth4
eth5
anl-mempt-2
eth2
eth3
eth4
eth5
anl-mempt-3
eth2
eth3
eth4
eth5
9/1/3
9/1/5
9/1/1
9/1/4
10/1/4
10/1/3
10/1/5
10/1/7
10/1/9
10/1/8
10/1/10
10/1/6
1/1/1
anl-ani
9/1/1
9/1/2
9/1/3
9/1/4
10/1/3
10/1/4
10/1/5
10/1/6
10/1/7
10/1/8
10/1/9
10/1/10
1/1/1
VPLS
10
VPLS
40
VPLS
20
VPLS
30
NERSC Argonne
-
7/27/2019 Nanog55.Talk43.Christracy Nanog55 100g Jun2012
47/62
Lawrence Berkeley National Laboratory U.S. Department of Energy | Office of Science
06/05/12
Testing and Measurement - TCP flows
Deployed systems on ANI experimental testbed - source/sink 4x10Gbps
24
48.6ms RTT, 97.9Gbps aggregate TCP throughput with 10 TCP streams
Thanks to Eric Pouyoul, Brian Tierney, and many others
nersc-ani
nersc-diskpt-1
eth2
eth3
eth4
eth5
nersc-diskpt-2
eth2
eth3
eth4
eth5
nersc-diskpt-3
eth2
eth3
eth4
eth5
anl-mempt-1
eth2
eth3
eth4
eth5
anl-mempt-2
eth2
eth3
eth4
eth5
anl-mempt-3
eth2
eth3
eth4
eth5
9/1/3
9/1/5
9/1/1
9/1/4
10/1/4
10/1/3
10/1/5
10/1/7
10/1/9
10/1/8
10/1/10
10/1/6
1/1/1
anl-ani
9/1/1
9/1/2
9/1/3
9/1/4
10/1/3
10/1/4
10/1/5
10/1/6
10/1/7
10/1/8
10/1/9
10/1/10
1/1/1
VPLS
10
VPLS
40
VPLS
20
VPLS
30
NERSC Argonne
-
7/27/2019 Nanog55.Talk43.Christracy Nanog55 100g Jun2012
48/62
Lawrence Berkeley National Laboratory U.S. Department of Energy | Office of Science
06/05/12
Testing and Measurement
25
Advantages Disadvantages
Hardware
Testers
stable & robust, tests many protocols 40/100Gbps interface speeds test single streams > 10Gbps
accurate measurement of loss/reordering very precise control of packet content can source/sink high bandwidth flows
consisting of small packets
TCP implementation at high-BW often notstateful (e.g., no congestion controlalgorithm), not a good indicator of how an
actual end host would perform generally cannot run user applications
that interface to the test equipment relatively expensive
PC-based
Testers
run user applications (data transfer suchas GridFTP, bbFTP, scp)
real-world TCP implementation- pluggable TCP congestion control algo. choice of various measurement and
analysis applications depending on exact configuration, packet
capture capabilities
at 100G, requires careful build and tuning host issues due to NIC driver, kernel,
interfering user/system processes
line-rate flows will have a limit onsmallest packet size
possibility of bugs in measurementapplications
accurate measurement problematic- implement counting on hardware
-
7/27/2019 Nanog55.Talk43.Christracy Nanog55 100g Jun2012
49/62
Lawrence Berkeley National Laboratory U.S. Department of Energy | Office of Science
06/05/12
Debugging and Fault Isolation
Tracking down loss
Relatively easy to track down if there are accurate counters
- and the ability to filter on some particular test traffic (e.g. 5-tuple)
Difficult when there is background traffic and equipment does not
allow packet counting via ACLs, etc.
Important to understand where allof the Drop counters are
Tracking down re-ordering
More difficult (especially if you dont have a hardware tester)
Some open-source test tools can report on sequence errors or mis-orders, but there is room for improvement
Can capture TCP packet dumps, analyze with tcptrace
26
-
7/27/2019 Nanog55.Talk43.Christracy Nanog55 100g Jun2012
50/62
Lawrence Berkeley National Laboratory U.S. Department of Energy | Office of Science
06/05/12
Debugging and Fault Isolation
Using loopbacks
Loopbacks can be very useful in certain situations
Bring up facility loops facing a layer-3 router on transport equipment
- ping broadcast address of layer-3 interface
- check for obvious signs of trouble
- helpful to verify patching
If wide-area link is down, bring up loops at termination points toverify patching between layer-3 and transport gear
- Place loops at re-gen points, check for link up/down on router
Facility loops always face fiber (client side or network side)
- e.g., client facility loop on transport layer CFP facing a router
Terminal loops face internals of client board close to client laser
27
-
7/27/2019 Nanog55.Talk43.Christracy Nanog55 100g Jun2012
51/62
Lawrence Berkeley National Laboratory U.S. Department of Energy | Office of Science
06/05/12
Interoperability
Between Layer-2/3 Gear and Optical Transport Equipment
no issues have arisen
transport layer is expected to be transparent, so far, it has been
Between Layer-2/3 Equipment and other Layer-2/3 Equipment
Remember, 100GigE is still quite new
Vendors have chosen slightly different ways of supporting 100G
- not necessarily optimizing for one single line-rate flow
- in some cases these have led to challenges in achieving desiredperformance
- re-working the way the system is logically configured or physicallycabled can potentially have a huge impact
Important to understand exactly how flows transit your equipment
- How exactly do line cards (and fabric) interface to the backplane?28
-
7/27/2019 Nanog55.Talk43.Christracy Nanog55 100g Jun2012
52/62
Lawrence Berkeley National Laboratory U.S. Department of Energy | Office of Science
06/05/12
Interoperability
Testing with both 100GBASE-LR4 and 10x10 MSA LR10 CFPs
Basic interoperability* demonstrated between:
- Juniper T1600
- Juniper MX
- Brocade MLX- Ciena OME6500
- Ciena 5410
- Alcatel-Lucent 7750 SR
Support foroptical power monitoring varied, hardware dependencies
- per-lane power, aggregate, both per-lane & aggregate, or neither Over short distances, 10x10-2km interoperates with 10x10-10km
Thanks to SCinet, StarLight, and Indiana University engineers for their test results.
29* basic interoperability meaning we could pass packets and links ran error-free, not that every feature worked
-
7/27/2019 Nanog55.Talk43.Christracy Nanog55 100g Jun2012
53/62
Lawrence Berkeley National Laboratory U.S. Department of Energy | Office of Science
06/05/12
Case study:Transport of 2x100G "s to SC11
30
Inter-domain, multi-vendor collaboration
Presented to SCinet WAN team as alienwaves at the Westin building in Seattle
100Gbps connection to ANI and Internet2
100Gbps connection to CANARIE
Supported numerous 40/100Gbpsdemonstrations on ANI and Internet2- https://my.es.net/topology/sc11/overview
#1180mi/1900km
-
7/27/2019 Nanog55.Talk43.Christracy Nanog55 100g Jun2012
54/62
Lawrence Berkeley National Laboratory U.S. Department of Energy | Office of Science
06/05/12
Case study:Transport of 2x100G "s to SC11
30
Inter-domain, multi-vendor collaboration
Presented to SCinet WAN team as alienwaves at the Westin building in Seattle
100Gbps connection to ANI and Internet2
100Gbps connection to CANARIE
Supported numerous 40/100Gbpsdemonstrations on ANI and Internet2- https://my.es.net/topology/sc11/overview
#1180mi/1900km OEO conversion
100GigE--OTU4at SC11 & SALT
-
7/27/2019 Nanog55.Talk43.Christracy Nanog55 100g Jun2012
55/62
Lawrence Berkeley National Laboratory U.S. Department of Energy | Office of Science
06/05/12
Case study:Transport of 2x100G "s to SC11
30
Inter-domain, multi-vendor collaboration
Presented to SCinet WAN team as alienwaves at the Westin building in Seattle
100Gbps connection to ANI and Internet2
100Gbps connection to CANARIE
Supported numerous 40/100Gbpsdemonstrations on ANI and Internet2- https://my.es.net/topology/sc11/overview
#1180mi/1900km regen 2X
OTU4 / DP-QPSK
-
7/27/2019 Nanog55.Talk43.Christracy Nanog55 100g Jun2012
56/62
Lawrence Berkeley National Laboratory U.S. Department of Energy | Office of Science
06/05/12
Case study:Transport of 2x100G "s to SC11
30
Inter-domain, multi-vendor collaboration
Presented to SCinet WAN team as alienwaves at the Westin building in Seattle
100Gbps connection to ANI and Internet2
100Gbps connection to CANARIE
Supported numerous 40/100Gbpsdemonstrations on ANI and Internet2- https://my.es.net/topology/sc11/overview
#1180mi/1900km all-optical
pass-through
-
7/27/2019 Nanog55.Talk43.Christracy Nanog55 100g Jun2012
57/62
Lawrence Berkeley National Laboratory U.S. Department of Energy | Office of Science
06/05/12
Case study:Transport of 2x100G "s to SC11
30
Inter-domain, multi-vendor collaboration
Presented to SCinet WAN team as alienwaves at the Westin building in Seattle
100Gbps connection to ANI and Internet2
100Gbps connection to CANARIE
Supported numerous 40/100Gbpsdemonstrations on ANI and Internet2- https://my.es.net/topology/sc11/overview managed by
SCinet WAN
team
#1180mi/1900km
-
7/27/2019 Nanog55.Talk43.Christracy Nanog55 100g Jun2012
58/62
Lawrence Berkeley National Laboratory U.S. Department of Energy | Office of Science
06/05/12
Case study:Transport of 2x100G "s to SC11
30
Inter-domain, multi-vendor collaboration
Presented to SCinet WAN team as alienwaves at the Westin building in Seattle
100Gbps connection to ANI and Internet2
100Gbps connection to CANARIE
Supported numerous 40/100Gbpsdemonstrations on ANI and Internet2- https://my.es.net/topology/sc11/overview
#1180mi/1900km hand-off as
alien wavesOTU4 / DP-QPSK
-
7/27/2019 Nanog55.Talk43.Christracy Nanog55 100g Jun2012
59/62
Lawrence Berkeley National Laboratory U.S. Department of Energy | Office of Science
06/05/12
Future Work
Near-term: Mixing 10G NRZ & 40G/100G DP-QPSK "s in metro rings
These technologies do not necessarily play nice together [9]
Kevin McGrattans talk [10] has a lot of detailed background on thistopic
Understand impact in terms of:
- reach penalty on 100G channels (how is OSNR affected?)
- maximum channel count
- colorless ROADM deployment (with non-coherent wavelengths)
Deployment of Colorless ROADM components
Long-term: Flexible Grid?
Flexible Wavelength Selectable Switch (WSS) components
Super channels at 1Tb/s which occupy >100GHz of spectrum31
-
7/27/2019 Nanog55.Talk43.Christracy Nanog55 100g Jun2012
60/62
Lawrence Berkeley National Laboratory U.S. Department of Energy | Office of Science
06/05/12
Acknowledgements
This work would have been possible without the help of the rest ofthe ESnet staff as well as our partners:
Internet2
GlobalNOC / Indiana University
Alcatel-Lucent
Ciena
Level3
SCinet Routing, WAN, and Fiber Teams
32
-
7/27/2019 Nanog55.Talk43.Christracy Nanog55 100g Jun2012
61/62
Lawrence Berkeley National Laboratory U.S. Department of Energy | Office of Science
06/05/12
References
[1] Roberts, K., Beckett, D., Boertjes, D., Berthold, J., Laperle, C. (2010, July). 100G and Beyond with Digital Coherent SignalProcessing. Communications Magazine, IEEE, 48(7), 62-69. Retrieved January 16, 2011, from IEEE Xplore database.
[2] IEEE 802.3ba. http://standards.ieee.org/getieee802/download/802.3ba-2010.pdf
[3] ITU-T G.709. http://www.itu.int/rec/T-REC-G.709/en
[4] OIF-FD-100G-DWDM-01.0 - 100G Ultra Long Haul DWDM Framework Document (June 2009). http://www.oiforum.com/public/documents/OIF-FD-100G-DWDM-01.0.pdf
[5] 10x10 MSA Revision 2.4. http://www.10x10msa.org/documents/MSA%20Technical%20Rev2-4.pdf
[6] Finisar C.wire. http://finisar.com/products/active-cables/C.wire
[7] Digital Coherent Receiver Technology for 100-Gb/s Optical Transport Systems. http://www.fujitsu.com/downloads/MAG/vol46-1/paper18.pdf
[8] Gringeri, S., Basch, B., Shukla, V., Egorov, R., Xia, T. (2010, July). Flexible Architectures for Optical Transport Nodes andNetworks. Communications Magazine, IEEE, 48(7), 40-50. Retrieved January 16, 2011, from IEEE Xplore database.
[9] Magill, P. (2010, September). 100G Coherent trials and deployments: AT&T plans and perspective. ECOC2010, 36th EuropeanConference and Exhibition on Optical Communication. Retrieved January 16, 2011, from IEEE Xplore database.
[10] McGrattan, K. Considerations in Migrating DWDM Networks to 100G. http://events.internet2.edu/2011/jt-clemson/agenda.cfm?go=session&id=10001587
[11] Drolet, P., Duplessis, L. (2010, July). 100G Ethernet and OTU4 Testing Challenges: From the Lab to the Field. CommunicationsMagazine, IEEE, 48(7), 40-50. Retrieved January 16, 2011, from IEEE Xplore database.
33
-
7/27/2019 Nanog55.Talk43.Christracy Nanog55 100g Jun2012
62/62
Lawrence Berkeley National Laboratory U.S. Department of Energy | Office of Science
06/05/12 34
Questions?
Thanks!