100 gigabit ethernet requirements & implementation fall 2006 internet2 member meeting december...
TRANSCRIPT
100 Gigabit Ethernet Requirements & ImplementationFall 2006 Internet2 Member MeetingDecember 6, 2006
Serge [email protected]
Drew [email protected]
Fall 2006 Internet2 Member Meeting, December 6, 2006 2
Internet Backbone Growth
Industry consensus indicates a sustainable growth rate of 75% to 100% per year in aggregate traffic demand
Traffic increased more than 10,000x from 1990 to 2000 Traffic projected to increase an additional 1,000x from 2000 to 2010
[1] K. G. Coffman and A. M. Odlyzko, ‘Growth of the Internet’, Optical Fiber Telecommunications IV B: Systems and Impairments, I. P. Kaminow and T. Li, eds. Academic Press, 2002, pp. 17-56.
Fall 2006 Internet2 Member Meeting, December 6, 2006 3
The Future Belongs to Tb/s Links! Carriers deployed Nx10 Gb/s networks several years ago
ECMP and LAG N now reaching hardware limit around 16 in some networks
Now evaluating deployment of (Nx) 40 Gb/s router networks Is this like putting out a 5-alarm fire with a garden hose?
Current Backbone growth rates, if sustained, will require IP link capacity to scale to > 1 Tb/s by 2010
Fall 2006 Internet2 Member Meeting, December 6, 2006 4
Proposed Requirements for Higher Speed Ethernet
Protocol Extensible for Speed Ethernet tradition has been 10x scaling But at current growth rates, 100 Gb/s will be insufficient by 2010 Desirable to standardize method of extending available speed
without re-engineering the protocol stack
Incremental Growth Most organizations deploy new technologies with a 4-5 yr lifetime Pre-deploying based on the speed requirement 5 yrs in advance is
economically burdensome Assuming 5 yr window and 100% growth per year, ability to grow
link speed incrementally over 25 = 32x without a “forklift upgrade” seems highly desirable
Fall 2006 Internet2 Member Meeting, December 6, 2006 5
Proposed Requirements (cont’d)
Hitless Growth Problematic to “take down” core router links for a substantial period
of time without customer service degradations SLAs may be compromised or require complicated temporary
workarounds if substantial down time is required for upgrade. Ideally, upgrade of the link capacity should therefore be hitless, or
at least only momentarily service-impacting.
Resiliency and Graceful Degradation Protocol should provide rapid recovery from failure of an individual
channel or component If the failure is such that full performance can not be provided,
degradation should only be proportional to the failed element(s).
Fall 2006 Internet2 Member Meeting, December 6, 2006 6
Proposed Requirements (cont’d)
Technology Reuse Highly desirable to leverage existing 10G PHYs, including
10GBASE-R, W, X, S, L, E, Z and LRM in order to foster ubiquity and avoid duplication of standards efforts
Deterministic Performance Latency/Delay Variation should be low for support of real-time
packet based services, e.g. Streaming video VOIP Gaming
Fall 2006 Internet2 Member Meeting, December 6, 2006 7
Proposed Requirements (cont’d)
WAN Manageability 100 GbE will be transported over wide area networks It should include features for low OpEx and should be:
Economical Reliable Operationally Manageable (e.g. simple fault isolation)
It should support equivalents for conventional transport network OAM mechanisms, e.g. Alarm Indication Signal (AIS) Forward Defect Indication (FDI) Backward Defect Indication (BDI) Tandem Connection Monitoring (TCM), etc.)
WAN Transportability Operation over WAN fiber optic networks Transport across regional, national and inter-continental networks The protocol should be resilient to intra-channel/intra-wavelength
propagation delay differences (skew)
Fall 2006 Internet2 Member Meeting, December 6, 2006 8
Technological Approaches to 100 Gb/s Transport
Time Division Multiplexing(ie: Baud Rate)
10 Mbps
100 Mbps
1 Gbps
10 Gbps
100 Gbps
Space Division Multiplexing(ie: Parallel Optics)
1
4
12
8
Modulation(ie: Bits per Hz)
1 (e.g. NRZ)
2 (e.g. PAM-4, (D)QPSK)
4 (e.g. QAM-16)
8 (e.g. QAM-256)
Wavelength Division Multiplexing
(i.e. s)1 2 4 6 8 10
DWDMCWDM
Fall 2006 Internet2 Member Meeting, December 6, 2006 9
Which Ethernet Application?
Ethernet is used today for many applications over different distances Distances > 100m primarily use
optical technologies
Performance for each application may be best advanced using a different approach
Telecom Application Class Translation Reach (km)
Very Short Reach (VSR) Intra-Room 0.1-0.3Short Reach 1 (SR-1) Intra-Campus 2Short Reach 2 (SR-2) Metro Access 10-15Intermediate Reach (IR) Metro Core 40Long Reach (LR) Regional 100Very Long Reach (VLR) Long-haul N x 100
Time Division Multiplexing(ie: Baud Rate)
Space Division Multiplexing(ie: Parallel Optics)
Modulation(ie: Bits per Hz)
Wavelength Division Multiplexing
(i.e. s)
Fall 2006 Internet2 Member Meeting, December 6, 2006 10
Scaling Beyond 10Gb/s: TDM
• Too many problems!• 65nm CMOS will cap out long
before 100Gb/s• 100x shorter reach due to
dispersion (modal, chromatic, PMD, etc.)
• Bandwidth of copper backplane technology
• Fundamental R&D required to develop enabling technologies for low cost
100 Gb/s TDM unlikely to be a low-cost approach for any application in near future
Time Division Multiplexing(ie: Baud Rate)
10 Mbaud
100 Mbaud
1 Gbaud
10 Gbaud
100 Gbaud
Space Division Multiplexing(ie: Parallel Optics)
1
4
12
8
Modulation(ie: Bits per Hz)
1 (e.g. NRZ)
2 (e.g. PAM-4, (D)QPSK)
4 (e.g. QAM-16)
8 (e.g. QAM-256)
Wavelength Division Multiplexing
(i.e. s)1 2 4 6 8 10
DWDMCWDM
Fall 2006 Internet2 Member Meeting, December 6, 2006 11
Scaling Beyond 10Gb/s: Modulation
Has never been applied to a high-volume optical standard and difficult for most applications of interest
Modulation(ie: Bits per Hz)
1 (e.g. NRZ)
2 (e.g. PAM-4, (D)QPSK)
4 (e.g. QAM-16)
8 (e.g. QAM-256)
Space Division Multiplexing(ie: Parallel Optics)
1
4
12
8
Wavelength Division Multiplexing
(i.e. s)1 2 4 6 8 10
DWDMCWDM
Time Division Multiplexing(ie: Baud Rate)
10 Mbps
100 Mbps
1 Gbps
10 Gbps
100 Gbps
• Digital Communication theory is well-established• Proven technology for copper technologies
1000BASE-T, DSL, Cable Modems, etc.• Limited use with optical technology• May be used in conjunction with other approaches
Fall 2006 Internet2 Member Meeting, December 6, 2006 12
Scaling Beyond 10Gb/s: SDM
• OIF standards for Parallel Optical Interfaces• 10Gb/s VSR4 and 40Gb/s VSR5
• Slow adoption due to minimal market traction• Low volumes limits economic savings• Could be extended to 100 Gbps
• 12x 10 Gbps VCSELs
Most applicable to VSR applicationsSpace Division Multiplexing
(ie: Parallel Optics)
1
4
12
8
Modulation(ie: Bits per Hz)
1 (e.g. NRZ)
2 (e.g. PAM-4, (D)QPSK)
4 (e.g. QAM-16)
8 (e.g. QAM-256)
Wavelength Division Multiplexing
(i.e. s)1 2 4 6 8 10
DWDMCWDM
Time Division Multiplexing(ie: Baud Rate)
10 Mbps
100 Mbps
1 Gbps
10 Gbps
100 Gbps
Fall 2006 Internet2 Member Meeting, December 6, 2006 13
Scaling Beyond 10Gb/s: WDM
Extensive WDM technology development in past decade• Proven deployments in all telecom networks• Focus on cost reduction: CWDM, EMLs, etc.• 10GBASE-LX4 achieved success
• 4-color CWDM• SR applications
100 Gbps
Proven approach to reach Tb/s level bandwidth for even long reach applications
Time Division Multiplexing(ie: Baud Rate)
10 Mbps
100 Mbps
1 Gbps
10 Gbps
100 Gbps
Space Division Multiplexing(ie: Parallel Optics)
1
4
12
8
Modulation(ie: Bits per Hz)
1 (e.g. NRZ)
2 (e.g. PAM-4, (D)QPSK)
4 (e.g. QAM-16)
8 (e.g. QAM-256)
Wavelength Division Multiplexing
(i.e. s)1 2 4 6 8 10
DWDMCWDM
Fall 2006 Internet2 Member Meeting, December 6, 2006 14
Drivers for a Super- (Multi-wavelength) Protocol
Per-channel bit rate growth historically and dramatically out-paced by Core Router interconnection demand growth
Requirement for WAN transportability strongly favors approach leveraging multiple wavelengths (Super-service
Fall 2006 Internet2 Member Meeting, December 6, 2006 15
Won’t 802.3ad Link Aggregation (LAG) Solve the Scaling Problem?
LAG and ECMP rely on statistical flow distribution mechanisms
Provide fixed assignment of “conversations” to channels Unacceptable performance as individual flows reach Gb/s range A single 10 Gb/s flow will exhaust one LAG member yielding 1/N
blocking probability for all other flows VPN and security technologies make all flows appear as one
True deterministic ≥ 40G link technology required today Deterministic packet/fragment/word/byte distribution mechanism
Fall 2006 Internet2 Member Meeting, December 6, 2006 16
Possible Channel Bonding Techniques
Traffic may be distributed over multiple links by a variety of techniques Bit/Octet/Word Distribution
Fixed units of the serial stream are assigned sequentially to lanes Small additional overhead allows re-alignment at the receiver Examples: 10GBASE-X, SONET/SDH/OTN Virtual Concatenation (VCAT)
Packet Distribution Sequence numbers added to packets to enable re-ordering at the receiver Large packets within the stream may induce excessive delay/delay variation to
smaller, latency-sensitive packets Examples: Multilink PPP, 802.3ah PME Aggregation Clause 61
Packet Distribution with Fragmentation Fragmentation bounds buffering requirements and delay associated with packet
size and packet size variation Overhead/link inefficiency is a function of the maximum fragment size chosen At 100 Gb/s and above, a fragment size can be chosen such that an effective
compromise between link efficiency and the QoS of individual, time-sensitive flows can be readily achieved
Examples: 802.3ah PME Aggregation, Multilink PPP
Fall 2006 Internet2 Member Meeting, December 6, 2006 17
10Gigabit Ethernet Protocol
Reconciliation
PCS
PMA
PMD
Medium
XGMII
MDI
1
MAC
LAG
PHY
(Media Access Control)
(Link Aggregation Group)
Fall 2006 Internet2 Member Meeting, December 6, 2006 18
Multilink Ethernet – N x 10G
PHY
Reconciliation
PCS
PMA
PMD
Medium
XGMII
MDI
Reconciliation
PCS
PMA
PMD
Medium
XGMII
MDI
1 N (ie: N = 10 for 100GbE)MultiLink Ethernet a.k.a. Aggregation at the Physical Layer (APL)
Reconciliation
PCS
PMA
PMD
Medium
XGMII
MDI
PHY
2
MAC
LAG
PHY
Multilink
MAC – Media Access Control
LAG – Link Aggregation Group
Fall 2006 Internet2 Member Meeting, December 6, 2006 20
Multilink Ethernet Benefits
Ensures ordered delivery
Resilient and scalable Incremental hitless growth up to 32 channels
Minimal added latency
Line code independent, preserves all existing 10G PHYs
Orthogonal to and lower level than LAG
Scales into future as individual channel speeds increase
Fall 2006 Internet2 Member Meeting, December 6, 2006 21
Multilink Ethernet Benefits (Cont.)
Concept well proven Packet fragmentation, distribution, collection and
reassembly similar to 802.3ah PME aggregation
Fits well with multi-port (4x, 5x, 10x, etc.) PHYs
Preserves existing interfaces (e.g. XGMII, XAUI)
Compatible with physical layer transport implementation over N x wavelengths
Fall 2006 Internet2 Member Meeting, December 6, 2006 22
FPGAprovided by Xilinx
100GbE MAC with packet reordering,
implemented by UCSC
10 x 10Gb/s XFP boards, provided by
Finisar
Infinera DTN, provided by
Infinera
10x10Gb/s1310nm
10x10Gb/selectrical
10x11.1Gb/s15xxnm
New internet2 networkChicago – New York
Opt
ical
loop
bac
ks
2000km
Live 100 GbE Demo - Chicago to New York
* 100 GbE first demonstrated Nov 13 at SC06 between Tampa and Houston
Fall 2006 Internet2 Member Meeting, December 6, 2006 27
Summary
100 GbE Requirements Protocol extensible for speed Hitless, incremental growth Resiliency and graceful degradation WAN transportability Technology reuse Deterministic performance Multi-channel operation
Multilink Ethernet meets the requirements
Technology proven over real networks