beyond 100ge - 2017.apricot.net

Copyright © Arista 2017. All rights reserved.Copyright © Arista 2017. All rights reserved.1

Beyond 100GE-What will be happened Ethernet?-Shishio [email protected]

Copyright © Arista 2017. All rights reserved.

Network Topology today

• East-West Traffic is increasing• Server Performance more efficient

2

10GE/40GE Uplink

1GE/10GEServer


Ethernet Server Shipment -Source: Dell’Oro Group 9/2016

• Dell`Oro group focusing 25GE ports overs 40GE in 2017• It will be reach x2.5 than 40GE by 2018

3

25Gbps 50Gbps 40Gbps


Why 25GE? Back to 100GE

• 100GE was first defined IEEE 802.3ba-2010• And 802.3bg-2011, 802.3bj-2014 and 802.3bm-

2015 standards• 802.3bm and 802.3bj were defined 4 lanes

specification for 100GE.

4

PHYASIC4 Lane x 25Gbps


What is Lanes and Clock rate?

• High-speed components are connected by SerDes• Data to be transmitted and Serialized , and the Deserialized on received.• SerDes Technology’s latest speed is 25Ghz• The Number of SerDes connection are called “lanes”

5

Interface Clock rate Lanes1GE 1.25Ghz 1

10GE 10.31Ghz 125GE 25.78Ghz 140GE 10.31Ghz 4

100GE 25.78Ghz 4


PCIe 3.0 and Network interface

• PCIe 3.0 x 4 slots is widely shipping as server platforms• PCIe 3.0 physical bandwidth 8.0Gbps/lanes • 25Gbps could smoothly upgrade from 10Gbps server platform

and it’s bandwidth efficiency is better than 10GE.

6

Interface Lane requirement Bandwidth Utilization

10GE 2 62.5%25GE 4 78.1%40GE 8 62.5%100GE 16 78.1%


Migration to 25Gbps

• 25GE uses SFP28 optics which can use same cabling infrastructure.• OPEX of the migration is minimum • Bandwidth will increase x 2.5

7

10G ToR to 10G Server

10/40G Switch

SFP+SwitchSilicon

1x10G10G ServerSFP+

25G ToR to 25G Server

100G Switch

SFP28SwitchSilicon

1x25G25G ServerSFP28

40G ToR to 10G Servers

10G ServerSFP+10/40G Switch

QSFP+SwitchSilicon

4x10G

10G ServerSFP+

10G ServerSFP+

10G ServerSFP+

100G ToR to 25G Servers

25G ServerSFP28100G Switch

QSFP28SwitchSilicon

4x25G

25G ServerSFP28

25G ServerSFP28

25G ServerSFP28


Merchant Silicon

8

How Many Lanes?How much total

bandwidth?

Vender Broadcom Broadcom Broadcom Cavium

Name XGS Trident +

XGS Trident II

XGS Tomahawk

XPAXP80

Total Bandwidth 640Gbps 1.28Tbps 3.2Tbps 3.2Tbps

Number of SerDes 64 128 128 128

Support interfaces 40GE/10GE 40GE/10GE 10GE/25GE/40GE/50GE/

100GE

10GE/25GE/40GE/50GE/

100GE


25G Consortium and IEEE802.3by• 25G Consortium- http://25gethernet.org/- draft was completed in September 2015 and use technology 802.3ba and 802.3bj

• IEEE802.3by- http://www.ieee802.org/3/by/- was formed in November 2015- June 30, 2016, the IEEE 802.3by standard was approved.

• There is some difference - Forwarding Error Detection(FEC) ≫ Cause 74(Fire Code) FEC≫ Cause 91(Reed-Solomon) FEC≫ Auto negotiation detects FC-FEC , RS-FEC and No-FEC

• Arista 7160 supports both technology IXIA Novus 100/25GE module will support IxOS 8.21EA

9


Transceiver and cable

10

Standard Distance support25GBASE-SR Up to 70m over Parallel OM3 Multi-mode fiber and 100m over parallelOM4 Multi-

mode fiber

25GBASE-LR Up to 10km over Single-mode fiber

25GBASE-AOC Up to 30m pre-terminated 25GbESFP Active optical cable

25GBASE-CR Up to 3m pre-terminated 25GbESFP twinax copper cable


Key takeaway• 25GE is cost effective technology - can use same NIC form factor( use one lane) with 10GE- can use same cabling infrastructure

• it increases x 2.5 Speed than current 10GE • ¼ resources compare with 40GE• There are two implementation IEEE802.3by and

25G Consortium.

11


Network Topology today

• What is the best speed of Spine-Leaf uplink ?

12

10GE/40GE Uplink

25GE in 2017


Yes, 100GE!

13

Source:http://www.fiber-optic-transceiver-module.com/overview-of-100g-client-side-transceivers.html*Cisco Proprietary form factorhttp://www.cisco.com/c/en/us/products/collateral/interfaces-modules/transceiver-modules/data_sheet_c78-728110.pdf

Module CFP CFP2 CXP CFP4 CPAK* QSFP28

Line rate 40/100G 40/100G 12x12G or10x12G

40/100G 40/100G 4x25-28G

Applications EthernetSONET/SDHOTN

EthernetSONET/SDHOTN

InfinibandEthernet



InfinibandEthernet

Dimensions 82x145x14 41x104x14 21x29 22x92x10 34.8x101.2x11.6 18x52x8.5

Max Power 8-32W 3-18W 6W 1.5-6W 4.5-7.5W* 3.5W

• QSFP28 most smallest/ low power standard technology

• CFP2/CFP4 would be used in long reach environment


Then what is next?

• Industry target is 400Gbps, it will use SerDes technology by 50Gbps x n • It includes 1x400Gbps,2x200Gbps, 4x100Gbps and 8x50Gbps• IEEE 802.3bs 200 Gb/s and 400 Gb/s Ethernet Task Force is targeting by End

of 2017 for IEEE standard14

402550

100

50

100

200

400

0

50

100

150

200

250

300

350

400

450

1 2 3 4 5 6 7 8

Spee

d[G

bps]

Number of Lanes

SerDes Speed growth and prediction

2013

2015

2018

x2.5 /2Years

x2 /3Years?


IEEE 802.3bs 200 Gb/s and 400 Gb/s Ethernet Task Force

15

Standard Speed Technology distance

200GBASE-DR4 200Gbps 4 lanes SMF 500m

200GBASE-FR4 200Gbps 4 WDM lanes 2km

200GBASE-LR4 200Gbps 4 lanes WDM 10km

400GBASE-SR16 400Gbps 16 lanes MMF 100m

400GBASE-DR4 400Gbps 4 lanes SMF 500m

400GBASE-FR8 400Gbps 8 WDM lanes 2km

400GBASE-LR8 400Gbps 8 WDM lanes 10km


Form Factor?

CFP8 QSFP-DD OSFPSize(HxLxT) 40x102x9.5 18.35x89.4x8.5 22.58x107.8x13.0Terminal Capacity 12-18W 7-10W 12-15WOrganization http://www.cfp-msa.org/ http://www.qsfp-dd.com/ http://osfpmsa.org/

16


What is important thing for next gen form factor?

• Don’t repeat 100GE mistakes

• Size , it should support at least 32 port per 1U-CFP8 is too wide in this requirement

• Terminal capacity, it requires 12-15W to support 12.8Tbps

17


OSFP MSA FAQ• Q: What is the OSFP (Octal Small Form Factor Pluggable)?• A: The OSFP is a new pluggable form factor with eight high speed electrical

lanes that will initially support 400 Gbps (8x50G). It is slightly wider and deeper than the QSFP but it still supports 32 OSFP ports per 1U front panel, enabling 12.8 Tbps per 1U.

•Q: Why a New Optics Module Form Factor?

• A: 400G Optics require more power than 100G Optics and many of them will not fit into the existing QSFP form factor. Depending on the specific 400G optics technology the initial power projections range from 7.5 Watt to 15 Watt.

• Q: How does the OSFP Address this Thermal Requirement?• A: The OSFP has an integrated heat-sink that greatly improves thermal

performance and enables modules with up to 15W power in a switch chassis with conventional airflow.

18

http://osfpmsa.org/


Key Takeaway• IEEE802.3bs will be standardized 400GE in 2017• It will use current SerDes Technology with

enhancement • OSFP MSA is only one solution form factor to

support high density 400GE

19


Key Takeaway• 2017- 25GE would increase at server side- IEEE will be finalized 400GE technology

• 2018- 400GE product would be expected in end of this year

20

beyond 100ge - 2017.apricot.net

Documents