h p n cern it pdp cern arie van praag igh erformance etworking to-day arie van praag cern it/adc...

H P N

CERN IT PDP

CERN

ARIE VAN PRAAG

igh erformance etworking

E-Mail: [email protected]

TO-D

AY

Arie Van Praag CERN IT/ADC1211 Geneva 23 SwitzerlandE-mail [email protected]

1 High Performance Networking as sign of its time.A Historical Overview of Hardware and Protocols

2 Yesterdays High Performance NetworksUltranet, HIPPI, Fibre Channel, Myrinet, Gigabit Ethernet

3 GSN ( the first 10 Gbit/s network and secure )

Physical Layer, Error Correction, ST Protocol, SCSI-ST

4 Infiniband ( the imitating 2.5 – 30 Gbit/s interconnect )

Physical Layer, Protocols, Network Management

5 SONET and some facts about DWDM, 10 Gigabit Ethernet, Physical Layers, Coupling to the WAN

H P N

CERN IT PDP

CERN

ARIE VAN PRAAG



S O N E T1972 The first Electro optical specifications and synchronization protocol are

published by an industry consortium called :Synchronous Optical NETwork or SONET. Bandwidth 3.125 Mbit/s

1985 SONET was adapted by the ANSI standards body T1 X1

as Synchronous Fibre Optics Network for Digital communications.

1986 CCITT ( now ITU ) joined the movement.

Implemented Optical Level Europe Electrical Line Rate Payload Overhead H Equivalent

ITU Level (Mbps) (Mbps) (Mbps)

1989 OC - 1 --- STS - 1 51.840 50.112 1.728 ---

1992 OC - 3 SDH1 STS - 3 155.520 150.336 5.184 STM- 1

1995 OC - 12 SDH4 STS - 12 622.080 601.344 20.736 STM- 4

1999 OC - 48 SDH16 STS - 48 2488.320 2405.376 82.944 STM- 16

2002 OC-192 SDH48 STS-192 9953.280 9621.504 331.776 STM- 64

OC-768 SDH192 STS-768 39613.120 39486.016 1327.064 STM-256

H P N

CERN IT PDP

CERN

ARIE VAN PRAAG



StandardsSONET is a multitude of standards:

It defines the Physical transfer and the protocols; ATM, POS,

It defines hardware Interfaces at different levels; Utopia, XSBI, XAUI, etc.

It defines the Bandwidth running up with a factor 4; SONET/SDH

It defines Wavelength for DWDM.

It defines switching layers as proposed by ISO;Level 2 VLAN DA SALevel 3 IP AddressesLevel 4 SessionsLevel 5-7 URLs, H.323 information etc.

It defines routing and switching strategies such that different manufacturers stay compatible without limiting design freedom.

H P N

CERN IT PDP

CERN

ARIE VAN PRAAG



SONET - ATM

CONNECTPacket Packet PacketMessage Data

LLC OUI PID Data Payload PADING UU/CPI+ Length FCS

BYTES 3 3 2 0 - 65.527 0 – 47 4 4

BYTES 48 48 48 48 48 48 48 48 48 48 48 48

ATM ATM ATM ATM ATM ATM ATM ATM ATM ATM ATM ATM

GFC = Generic Flow ControlVPI = Virtual Path IdentifierVCI = Virtual Channel IdentifierPTI = Payload Type IdentifierCLP = Cell Loss Priority

GFC VPI VCI PTI CLP Data

Bits 4 8 12 3 1

H P N

CERN IT PDP

CERN

ARIE VAN PRAAG



SONET - POS

CONNECTUser Frame User Frame User FrameMessage Data

TOHSOH Data Payload Flag

POH ADDR CNT ProtocolSAPI Data Payload FCS Flag

1 1 1 2 POS 0 - 1500 X85 0 - 1600 4 1

HDLC Frame Header

Data Space in POS mode up to 32 K – 65 KWill this be good place to put IP and Ethernet data ?

may belet’s see it with 10GE

H P N

CERN IT PDP

CERN

ARIE VAN PRAAG



DWDM

Dense Wavelength Division Multiplexing

H P N

CERN IT PDP

CERN

ARIE VAN PRAAG



What is Wavelength Multiplexing

Sending Multiple colors ( wavelength ) over a single fiber

How is it done ( simplified )Old Technology:but still much used

By heating the laser the reflecting resonance cavity is expanded which changes the frequency.

Problem: Stability

Modern Technology:

Using MEMS technology the mirror is moved by electro static forces, and as such expands the lasing cavity.The laser can be tuned with a DC voltage.

Arrayed Wavelength Gratins = ¼ Selection Platealso called “Phase Array”or “Phaser Plate”

Graded IndexReflex Plate

Or

Holographic Diffraction Plate

H P N

CERN IT PDP

CERN

ARIE VAN PRAAG



How the AWG works

DWDM Input

Field Lens

Wave-Guides Single Color

outputs

Field Lens with¼ interference

H P N

CERN IT PDP

CERN

ARIE VAN PRAAG



Problems with DWDM FibersFibers are still a medium with a lot of critical parameters, such that ongoing research

is an important factor to limit theme to a minimum by choosing materials and doping, and optimise production methods.

Some Physical Problems Are:

Solutions: Much progress is made on connectors and on the fibers itself, better glass ( plastic ) material, sophisticated doping and cladding, and stress limiting production methods have limited most of this problems to a minimum.

Second Order Polarization Mode Dispersion: A dispersion of the polarization that is occasioned by chromatic Dispersion, only if there is Polarization Mode Dispersion. It is mainly a problem with very long distances.

Polarization Mode Dispersion: Simplified; Horizontal and Vertical polarization do not travel at the same speed which occasions pulse broadening. It is an unstable stochastic process that is due to all kind of stress on the fiber. It gets more sensitive at higher bit rates .

Chromatic Dispersion: The group delay per Unit Wavelength is not equal and gives different travel speeds to the channels. Chromatic Dispersion accumulates with distance.

Optical Return Loss: Change of material and interconnecting surfaces both reflect small parts of light, As different material may have a different index ( transfer speed ) a frequency shift is not excluded that may provoke cross-talk.

Brillouin Backscattering: Backwards reflexions from connectors and strong bends create in the laser an acoustic wave that travels in the fiber and occasions optical disturbances. The Doppler effect is source of frequency shift and increases the risk of cross talk or cross modulation.

H P N

CERN IT PDP

CERN

ARIE VAN PRAAG



DWDM - EquipmentCapacity is up to 2048 and more Channels. Common in the Communication Industry is 1024

Each color with its modulation logic and its stabilization electronics is a plug in unit

Take 25 Units in a crate, and 10 crates in a rack = 4 racks Transmitters and 4 racks Receivers

CERN can solve its internal future data transport problems with bandwidth extension to 10 Gbit/s and higher using one stream per Fiber.

For external connections it is up to the Service Provider but 10GE in POS mode or 10GE in Native mode seems a good solution to communicate physics data to outside institutes and to handle GRID Data distribution.

Is DWDM technology a good solution for CERNTo ExpensiveTo ComplicatedTo Much MaintenanceTo Much Place

H P N

CERN IT PDP

CERN

ARIE VAN PRAAG



DWDM and some Simple Economic Aspects

Up to 1990 traffic was mainly some e-mail and remote maintenance.

1990 The web was born incrementing traffic. Fiber connections ware view, single stream and at best OC12/SDH4 at + 622 Mbit/s.

1995 – 2000 Enormous investments ware made in larger cables (1000 fibers and more). Cable laying ships ware build, and cables pulled all calculated at Single stream traffic.

1998 and later: bandwidth goes up OC48/SDH16 at 2.5 Gbit/s and even to OC192/SDH48 with 10 Gbit/s and DWDM and multiplies the capacity by 1000 such that a 100 fiber cable now moves 1 000 000 streams at 10 Gbit/s.Overcapacity let prices fall and communication companies go bankrupt on their enormous investments.

H P N

CERN IT PDP

CERN

ARIE VAN PRAAG



Optical Switching

LUCENT CALIENT

MIRRORS ARE POSITIONEDBY ANALOG VOLTAGES

THE LIGHT BEAM HAS TO BE TARGETED AT A SINGLE MODE FIBER OF A VIEW MICRONS.

Conclusion:EVEN IF INTEGRATED THIS WAY OF SWITCHING IS VERY SENSITIVE TO THE STEERING VOLTAGE AND TO TEMPERATURE

Can it be done more Reliable ?

H P N

CERN IT PDP

CERN

ARIE VAN PRAAG



More Stable Optical Routing Switches

Switching Speed + 12 msec.

OMM

If there is a On/Off Mechanism we are back to a Binary Function.

H P N

CERN IT PDP

CERN

ARIE VAN PRAAG



Solid State Optical SwitchIn Gallium Arsenid (GaAs) optical properties can be

influenced by an electric field.

The electrical influenced refraction index delays the passing light.

A voltage gradient in multiple waveguides turns the wavefront ( ¼ λ interference ) and with it the direction of the output beam.

The result is an “Optical Phase Array” or “Beam Deflector”

Can be produced with Semiconductor manufacturing technology, no moving parts.

Switching speed 20 to 30 nsec.

Fast enough to do IP routing of 10 Gbit/s and 40 Gbit/s networks.

Format independence gives high scalability.

Large switches possible ( 64 X 64 ) demonstrated.

128 wave-guides

* According to CHIARO Networks

H P N

CERN IT PDP

CERN

ARIE VAN PRAAG



10GE

H P N

CERN IT PDP

CERN

ARIE VAN PRAAG



10 Gigabit Ethernet

Standard IEEE 802.3ae is accepted 17 June 2002 June 2002First Commercial hardware: may be 4 Q 2002 2003

The 10 Gigabit Ethernet started in 1999 in the IEEE 802.3 working group

Study GroupFormed 802.3 Ballot80.3ae

FormedSponsor

Ballot

First Draft Final Draft 80.3aeStandard

1999 2000 2001 2002

H P N

CERN IT PDP

CERN

ARIE VAN PRAAG



Payload: 10 Gbit/s

10 GE Physical

Transfer: Full Duplex

Media: Fiber only ( for the moment at least )

PMD

PMAWIS

PCS 64B/66B PCS 8B/10BPCS 64B/66B

PMD

PMAPMD

PMA

MEDIUM10GBase-W

MEDIUM10Gbase-X

MEDIUM10GBase-R

XGMII

MDI MDI MDI

Physical Coding Sublayer

Physical Medium Attachment

Medium Dependent Interface

WAN Interface Sublayer

Physical Medium Dependent

10 Gigabit Media Independent Interface 2X 32bit data + 2X 5 bit Control

WAN compatible framing

Retime, SerDes, CDR

Optical Transceiver

Connecting medium

Fiber type etc

Upcoding 8B/10B or 64B/66B

H P N

CERN IT PDP

CERN

ARIE VAN PRAAG



10GE Optical & Fiber TypeSTACK 10 GE LAN Phy 10 GE WAN Phy

Serial WWDM Serial

MAC 10.0 Gbit/s 10.0 Gbit/s 10.0 Gbit/s

PCS 64B/66B 8B/10B64B/66B

SONET FramingScrambling ( X7 + X6 + 1 )

PMA Interface XSBI XAUI XSBI

PMD1550 nm DFB1310 nm FP

850 nm VCEL1310 nm CWDM

1550 nm DFB1310 nm FP

850 nm VCEL

Line Rate 10.3 Gbit/s 4X 3.125 Gbit/s 9.953 Gbit/s

Wavelength:

1275.7 nm, 1300.2 nm, 1324.7 nm, 1349.2 nm.

3.125 Gbit/s / channel

OC – 48/SDH16 2488.320

OC – 192/SDH48 9953.280

OC – 768SDH/192 39813.12

PMD Type of Fiber Target DistanceOptical TRC Meters

850 nm serial Multi Mode 65

1310 nm CWDM Multi Mode 300

1310 nm CWDM Single Mode 10 000

1310 serial Single Mode 10 000

1550 serial Single mode 40 000

Standard Interface foreseen For 4X INFINIBAND

H P N

CERN IT PDP

CERN

ARIE VAN PRAAG



Problems with CWDM or WDWM

Chromatic Dispersion travel speed in a fiber is not equal for all colors

Wavelength Drift Transmitter Data drifts out of the filter slot and can not be received

Wavelength Drift wavelength comes in the region where couples to other channels ( cross-talk )

Chromatic Dispersion Source of limited distance covered, and stops use of fiber amplifiers

Wavelength Drift The receiver filter drifts and does not select the correct wavelength

Conclusion 10 GE will only be popular if a cheap and reliable single fiber single wavelength solution is available

H P N

CERN IT PDP

CERN

ARIE VAN PRAAG



10 GE Networking

Protocol: TCP/IP follows IEEE 802.3 full 48 bit addressing

Frame size: 1500 Bytes Ethernet

Bandwidth: 12.5 Gbit/s

10 Gbit/s = 830 000 frames of 1500 bytes, or 1.2 s / frame.

= 2 X 830 000 Interrupts/s for transmission and for reception.

Without an Operating System Bypass it will be extremely difficult

OC-3OC-12

OC-48OC-192

OC-768

MIPS Needed forCommunication

Applications

GP MIPS Trend

1 0.7 0.5 0.35 0.25 0.18 0.13 0.1 0.07 0.05 0.03 0.02

1 000 000

100 000

10 000

1000

1000

10

Technology

MIPS

H P N

CERN IT PDP

CERN

ARIE VAN PRAAG



Syst

em C

apac

ity (M

bit/s

)

10 6

10 5

10 4

10 3

10 2

10 1

1985 1990 1995 2000 2005Year

Optical DWDM CapacityEthernetInternet Backbone

T1

T3

OC-3c

OC-12c

OC-48c

10-GE

Ethernet

Fast Ethernet

GigE

OC-192c

135 Mbit/s

565 Mbit/s

1.7 Gbit/s OC-48c

10 Gbit/s 1024

10 Gbit/s 160

10 Gbit/s 32

10 Gbit/s 16

10 Gbit/s 8 10 Gbit/s 410 Gbit/s 2

I/0 Rates=

Optical Wavelength

Capacity

OC-768c 40-GE

H P N

CERN IT PDP

CERN

ARIE VAN PRAAG



EZ-Chip a SolutionTOP = Task Operating Processor

Existing Technology for 10 Gbit networks

Ready for next generation for 40 Gbit Networks

32 TOP search engines, 64 processors total

Onboard Memory up to 5 MByte 256 to 512 bit wide with 200 MHz clock

Processes all 7 network layers:Level 2 VLAN DA SA Level 3 IP

Addresses Level 4 SessionsLevel 5-7 URLs,

H.323 information etc.

Capacity 8 X GigE or 1 X 10GE or 1 X OC192

Future versions

Ready for 8X 10GE or 1 X 40 GE or OC768

queuing

TOPmodifyengines

TOPresolveengines

TOPsearchengines

TOPparseengines

MAC

memory

memory

memory

memory

Externalmemory

NP-1

LINK/SWITCH Fabric

LINK/SWITCH Fabric

H P N

CERN IT PDP

CERN

ARIE VAN PRAAG



Does the I/O have the bandwidth

Data given by PCI-SIG

H P N

CERN IT PDP

CERN

ARIE VAN PRAAG



IP to SON

ET Header

Conversion

MAC

SNAP

PAYLOAD

DES. ADDR. 6

SRC-ADDR. 6

M LENGTH 4

DSAD 2

SSAD 2

ctl x03 1

org x00 3

ETHERTYPE 2

IP

Packet

40

GSN

SONET/SDHOC48c PPPHDLC IP FramesConversion

Hardware

Processor

PPP Prot. Field 2

Address 8

Control8

PPP IP Packet

Flag8

Flag8

FCS16 / 32

Compliant to RFC 2615IP - Internet Protocol IPv4: 020b

PPP PADDING

IP

Packet

H P N

CERN IT PDP

CERN

ARIE VAN PRAAG



POS Packing

17280 Bytes

Section

Line

TRANSPORT OVERHEAD

576 Bytes

9rows

16640 Bytes

16704 Bytes

9rows

SONET/SDH Payload

Up to 64 full Ethernet Frames

FIXEDHEADER

63 Columns

OVERHEAD

FRAME FRAMEFRAME

SONET/SDH Payload Envelope (SPE)

ATM is not efficient anymore at bandwidth over 1 GHz, and even less in safe mode.

The previous High Speed Way to move Ethernet packets over SONET/SDH used Byte Stuffing at 2.5 MByte/s for OC48/SDH16 in POS mode. It is also used in OC192/SDH48

An extension on the 10GE standard to be accepted by ANSI and UCI will transfer 10GE Packets directly over OC192/SDH48. The difference in bandwidth will be covered with by inserting IDLE’s at regular distances in the Ethernet data stream.

H P N

CERN IT PDP

CERN

ARIE VAN PRAAG



10GE direct on OC192c

PPP IP IPCONNECTData Padding PPP

CONNECTPPP IP Data IP Padding PPP

CONNECTEthernet IP Data IP Padding Eth.

CONNECTIP Data IP IP packet

Ethernet Frame

PPP packet

SONET


CONNECTIP Data IP


IP packet

Ethernet Frame

SONET

H P N

CERN IT PDP

CERN

ARIE VAN PRAAG



Economics of 10 GE

$0

$2

$4

$6

$8

$10

$12

$14

$16

2000 2001 2002 2003 2004

USD

per

Meg

a BW

OC-192 OC-48 10Gig 1Gig

SONET

Ethernet

Ethernet offers a superiorsuperior price/performanceand TCOTCO over alternative technologies(SONET/ATM)

Up to 10:1 price advantage in upfront costs

Up to 5:1 advantage in bandwidth provisioning expenses

Up to 5:1 advantage in annual maintenance

Provides bandwidth on demand without costly truck rolls

H P N

CERN IT PDP

CERN

ARIE VAN PRAAG



ProductsProducts start to arrive on the market now, mostly “Prove of Concept” commercially available are view.

Silicon: Infineon, Sierra, Agilenta, Broadband and some small development houses are all advertising NIC’s and SERDES circuits, sometimes for 10 GHz/s and for 40 GHz/s.

No general interfaces NIC’s are around and will not be before PCI-X2 is available

Switches and routers are announced. The most common are line concentrators with 10 X GE to 1X 10GE

10GE Alliance makes a large effort by organizing compatibility workshops with all manufacturers concerned.

It also does a general marketing job to show products and its compatibility efforts ( plug Fest ) on trade shows.

H P N

CERN IT PDP

CERN

ARIE VAN PRAAG



10 GigE Examples

Examples of Future Applications by Ciscoand the 10 Gigabit Ethernet Alliance

H P N

CERN IT PDP

CERN

ARIE VAN PRAAG



And the Last Mile

New VHDL technology will such allow a bandwidth good enough for 10/100 Base T Ethernet

And 802.17 introduces “Virtual Concatenation” for efficient packing of non corresponding frame sizes

Example: SONET STS 1 = 51.84 Mb/s, STS 3 = 155.52 Mb/s.Standard: With RPR

Ethernet 100 Base T needs STS 3 and wastes 55.52 Mb/s Ethernet 100 Base T can use STS1 and wastes 3.6 Mb/s

RESULT: Your future Internet connection will be cheaper for the Service provider, and such for the user, to use the same old phone line that brings 10/100 Base T Ethernet to the end user.

VHDL using standard twisted telephone wire is developing rapidly to faster and higher reliability with new technology standards ( moving from “Discreet Multitone Modulation” to “Quadrature Amplitude Modulation” )

The Metropolitan Network will move to double arbitrated ring structures ( 802.17 Resilient Packet Ring )

H P N

CERN IT PDP

CERN

ARIE VAN PRAAG



Grid and the NetworkAs Grid started the interconnections planned used

OC12 at 622 Mbit/s and OC3 at 155 Mbit/s.

with a single exception at OC48 at 2.5 Gbit/s

To move the large data files from LHC Experiments

Research and Academic Institutes•CESNET (Czech Republic)•Commissariat à l'énergie atomique (CEA) – France•Computer and Automation Research Institute, Hungarian Academy of Sciences (MTA SZTAKI)•Consiglio Nazionale delle Ricerche (Italy)•Helsinki Institute of Physics – Finland•Institut de Fisica d'Altes Energies (IFAE) - Spain•Istituto Trentino di Cultura (IRST) – Italy•Konrad-Zuse-Zentrum für Informationstechnik Berlin - Germany•Royal Netherlands Meteorological Institute (KNMI)•Ruprecht-Karls-Universität Heidelberg - Germany•Stichting Academisch Rekencentrum Amsterdam (SARA) – Netherlands•Swedish Research Council - Sweden

Industrial Partners•Datamat (Italy)•IBM-UK (UK)•CS-SI (France)

H P N

CERN IT PDP

CERN

ARIE VAN PRAAG



10 GE and the GRID

We can now foresee that in the

near Future Grid communications

will move to 10 GE Ethernet.

But this depends if there are

sufficient good quality

Single Mode Fiber available

And if Service Providers are ready

To Handle Gigabit Ethernet.

H P N

CERN IT PDP

CERN

ARIE VAN PRAAG


E-Mail: [email protected] 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 2000 01 02 03 04 05

Standards & Popularity( made in 1995 and extended 2000 )

Gigabit Ethernet

Ethernet

T base 100

Fibre Channel

ATM ( as computer interconnect )

HIPPI

HIPPI-Serial

GSN ( Gigabyte System Network )

PCI / PCI-X /

10 Gigabyte Ethernet

Infiniband

H P N

CERN IT PDP

CERN

ARIE VAN PRAAG



10 GE Conclusions10GE is Ethernet, the name everybody knows. Also the manager that decides the IT

Budget.

The Single Stream version connects the Local Network, to the Metropolitan Network and to the Wide Area Network within one protocol environment.

The small 1500 Byte frames will stay a handicap in local data handling and RDMA will only solve half of the problem. TOE engines are necessary, but some transfer latency stays

Using PPP encapsulation in POS mode every standard communication channel can be used immediately, including DWDM channels

WWDM is difficult to stabilize, and can not easily be coupled to telecommunication channels.

We will see 10GE at CERN, as backbone for the network, as backbone in the computer-center and as a data-link between LHC experiments and the central computing facilities.

Two standards for data transfer: 4 parallel streams using WWDM and a single stream.

10GE Silicon with 110 nm and 90 nm technology is able to handle the bandwidth and will come at reasonable prices. The optical parts will stay high price, even if moved to VCSEL’s

H P N

CERN IT PDP

CERN

ARIE VAN PRAAG



References:

10GEA 10 Gigabit Ethernet Alliance, http://www.10gea.org/ with many white papers and links to other cites.

Ethernet becomes king of the networking world, L. E. Frenzel, Electronic Design, December 9, 2002, p 45-52.

IEEE P802.3ae 10Gb/s Ethernet Task Force http://grouper.ieee.org/groups/802/3/ae/index.htmlThis is the IEEE working group that made the 802.3ae standard

Strategic Directions Moving the Decimal Point: An introduction to 10 Gigabit Ethernet. B Tolley, Cisco, Jan.5, 2001.http://www.cisco.com/warp/public/cc/techno/lnty/etty/ggetty/tech/10gig_wp.htm

The jump to 40GB Ethernet, P. Judge, Oct. 5, 2002 ZDNET (UK)

Guide to WDM Technology,A. Girard, et all, EXFO Electro Engineering Inc. Quebec City, Canada, 2000, ISBN 1-55342-000-4

Design Trade-offs for Arrayed Waveguide Grating DWDM MUX/DMUX Jane Lam, Ph.D. and Liang Zhao, Ph.D. Lamwhitepaper.pdf

Digital MEMS switch for planar photonic crossconnects, L. Fan, et all, OMM, Inc San Diego, 1999, OCIS codes 060.1810 http://www.omminc.com/technology/whitepapers.html. And much more interesting documentation on this site.

Optical Phased Array Technology for High-Speed Switching, http://www.chiaro.com/pdf/CHI100_OPA_1.pdfwith more interesting white papers under http://www.chiaro.com/proof_points/index.jsp

H P N

CERN IT PDP

CERN

ARIE VAN PRAAG



END LAST

PART

Thank you for your attention during this long and not always easy material,where my hope is that you learned about 10 Gbit/s problems and highlights

h p n cern it pdp cern arie van praag igh erformance etworking to-day arie van praag cern it/adc...

Documents