Data Links OptionsInitial Outlook

Babak AbiDUNE DAQ consortia hardware group meeting

07 Dec 2017

1

Outline:• Data link roadmap current and future • Example of form factors• Components and suggestion for current standard• Costs per link and Gb/s

Data Links Options

• Main questions :1. what are the data links current and future options?

• Link speed, form factor power consumption

2. Long-haul and short-haul links from the WIB to the DAQ 1. Technologies with coper for 30m till 40Gb2. Long-haul 100m > should be optical links

3. Cost and trade-offs of each

• BUT Networking and high-speed solution and components have a very broad phase space since we are on the verge of a technological change in the landscape of data links.

• The IEEE is proposing more Ethernet standards from 2015 to 2020 (five years) than its released in the last 40. There is even realistic talk of 10 TbE towards the end of the next decade.

• Should we focus our designs on current or future standard ?• We need to review future standard since the hardware system design level would be effected if want to use future

standard

• We present the suggestion base on current standard

Ethernet Alliance Technology Roadmap 2014 paper

2

Link roadmap; speed

• EthernetAlliance_Roadmap_whitepaper_FINAL-032015-21

3

Link roadmap; speed

• TbE (Terabit Ethernet) vs. GbE (Gigabit Ethernet)• GbE link speeds to 100Gb and TbE links speeds from 100Gb

• If we want to use TbE we need to design hardware now

• CWDM and DWDM dominant transceivers are SFP and SFP+

EthernetAlliance_Roadmap_whitepaper_FINAL-032015-21

4

C form-factor pluggable (CFP) • Just as an example how much hardware interface design could be affected by data links!

• C form-factor pluggable (CFP) : The CFP supposed to be the future of SFP, but is significantly larger to support 100 Gbit/s. While the electrical connection of a CFP uses 10 × 10 Gbit/s lanes in each direction (RX, TX) the optical connection can support both 10 × 10 Gbit/s and 4 × 25 Gbit/s

• Variants:• CFP

• 148 pin electrical connection (82x13.6×144.8 mm, 24 W power consumption)• integrated digital signal processor within package• 10×10G or 4×25G lanes

• CFP2• 104 pin electrical connection (41.5×12.4x107.5mm, 12 W consumption)• no digital signal processor in package, relies on host card• 10×10G, 4×25G, 8×25G, or 8×50G lanes

• CFP4• 56 pin electrical connection 1.5× 9.5× 92 mm (6 W consumption)• no digital signal processor in package, relies on host card• 4×10G or 4×25G lanes

• CFP8• 124 pin electrical connection 40 mm × 9.5 mm × 102 mm ( 24 W )• no digital signal processor in package, relies on host card• 16×25G lanes (25.78125 or 26.5625 GB) or 8×50G lanes

5

Optical MUX

• Optical MUX• Simplex MUX, needs two fibre, cheaper • Duplex MUX, needs ONE fibre, expensive

• CWDM or DWDM? • Dense wavelength division multiplexing (DWDM) up to 80

(theoretically) channels can be multiplexed into on a single optical fiber.

• CWDM (Coarse Wavelength Division Multiplexing) Up to 18 channel spacing 20 nm, 1270 nm to 1610 nm, Cost-effective

• If we assume we fix design on SFP+ (10G)• DWDM 40 links per fiber (400Gb) and CWDM 18 links per fiber

(180Gb) • There are hacks for increase it like using qsfp+ in 1310nm line • Limitation on fibres we have only 96

DWDM 38Tb/s and CWDM 19Tb/s

6

Simplex MUX

Duplex MUX

CWDM and DWDM module • FMU 1U Rack Mount, LC/UPC, with Monitor Port, 3.0 to 6dB Typical IL

• Could add a 22dBm Output Booster DWDM EDFA C-band 24dB Gain for long-haul

• Price: • Retail price $1700 for 40 channels DWDM (4 APA if we use 10x10GbE )

• Retail price $710 for 18 channels CWDM ( almost 2 APA if we use 10x10GbE )

• BUT in mass order or FOB can be much cheaper

• Mux muddles are fairly cheap but SFP module • CWDM SFP+ starts from $80

• DWDM SFP+ starts from $270

• But prices are falling down , need extrapolation for 2024?

• Current cost for MUX end to end included TRX and MUX module but fibre :• DWDM $70 per Gb

• CWDM $25 per Gb

• DWDM 400Gb per fibre and CWDM 180Gb per fibre!

• Fibre selection: Zero-water-peak fibres might be needed (backup slide)

• Further options& considerations ?• BiDi SFP+ not widely used + we do not need downstream fibre??

• Enterprise Ethernet network switch !

(32*100GbE) 100GbE Spine/Core Layer for $8000

7

40km10km

Backup slides

8

water-peak fibres

9

Link roadmap; speed

•EthernetAlliance_Roadmap_whitepaper_FINAL-032015-21

10

References

1. "The 2016 Ethernet Roadmap" ethernetalliance.org. from Ref 1

11

12