ethernet over sonet dr. john s. graham university of london computer centre

Ethernet over SONET

Dr. John S. Graham

University of London Computer Centre

Ethernet vs SONET

Feature Ethernet SONET

Guaranteed Delivery

No (Packet Switched)

Yes (Circuit Switched)

Guaranteed BW No Yes

Performance Monitoring

CRC Count BER

Protection None (STP?) APS

Fault Location None AIS

Access Costs Low High

SONET/SDH Model

RRMUX DXC MUX

Path Layer Connection

Line LayerLine Layer

Section Layer Section Layer Section LayerSection Layer

SONET Network Elements

SONET STS-1 Frame

Section

Line

PTR

SPE

3

5

86 columns

Path

30 59 9041

Payload Capacity = 756 Bytes = 48.384 Mb/s

Section Overhead Bytes

Data Com

D3

Data Com

D2

Data Com

D1

User

F1

Orderwire

E1

BIP-8

B1

STS-1 ID

C1Framing

A2Framing

A1

Section

Line Overhead Bytes

Line

Orderwire

E2

Line FEBE

Z2

Sync Stat

Z1

Data Com

D12

Data Com

D11

Data Com

D10

Data Com

D9

Data Com

D8

Data Com

D7

Data Com

D6

Data Com

D5

Data Com

D4

APS

K2

APS

K1

BIP-8

B2

Path Overhead Bytes

Path

Multi-frame

H4

User Channel

F2

Path Status

G1

Signal Label

C2

BIP-8

B3

Path Trace

J1

Accurate Bit-Level Timing• Line Coding

– AMI, AMI RZ– HDB3

• Block Coding– 4B/5B (FDDI)– 5B/6B (E3)– 8B/10B (Gigabit Ethernet)

• Scrambling– Frame Synchronous– Self Synchronous

8B/10B Encoding (1/2)• Ensures equal number of 1’s and 0’s for

– Clock recovery– DC Balance

• Detection of single bit transmission errors

• Predefined ‘comma’ (0011111 and 1100000) characters

8B/10B Encoding (2/2)• Data (D) and commands (K) encoded separately• Octet split into 5 MSB and 3 LSB• Code groups generated from lookup tables

Decoded

A B C D E F G H

0 1 0 0 0 0 0 0

2 0

Encoded

A B C D E i F G H j

1 0 1 1 0 1 0 1 0 0

D2.0

Control Words (‘Ordered Sets’)

Symbol Encoding Purpose

F K28.5 D21.5 LINK_NOT_AVAILABLE

C K28.5 D10.5 config_reg LINK_CONFIGURATION

I1 K28.5 D5.6 Idle

I2 K28.5 D16.2 Idle

S K27.7 SoF

T K29.7 EoF

R K23.7 CARRIER_EXTEND

H K30.7 ERROR

Scrambling in SONET/SDH

SDH STM-1 uses x7 + x6 +1 polynomial

ConcatenationSTS-3 155 Mb/s

145.152 Mb/s

STS-3c/VC-4 155 Mb/s149.76 Mb/s

STS-12 622 Mb/s580.608 Mb/s

VC-4-4 622 Mb/s599 Mb/s

STS-12c 622 Mb/s599.04 Mb/s

Provisioning Example

A

D

B

E C

F3 - 5

7

9

10 -12

1

???

1 – 12 (622 Mb/s)

Bandwidth Fragmentation

STS-3c

STS-12

Blocked!

Drop Interface (Node F)

Ring

Virtual Concatenation

STS-3c

STS-12

STS-1-3v

Drop Interface (Node F)

Ring

The Magic of VCAT

ProtocolClient Rate

Contiguous Concatenation

Virtual Concatenation

Fast Ethernet 100 STS-3c 65% STS-1-2v 98%

ESCON 160 STS-12c 26% STS-1-4v 78%

Fibre Channel 850 STS-48c 34% STS-3c-6v 92%

Gigabit Ethernet 1000 STS-48c 40% STS-3c-7v 92%

Fibre Channel 2 1700 STS-48c 66% STS-3c-12v 92%

Differential Delay

TX Ty

Source

STS-1-12v

SinkCore Network(Partial Mesh)

Multiframe IndicatorBit 1 Bit 2 Bit 3 Bit 4 Bit 5 Bit 6 Bit 7 Bit 8

MFI2 MSBs (bits 1 - 4) 0 0 0 0

MFI2 LSBs (bits 5 – 8) 0 0 0 1

CTRL 0 0 1 0

GID 0 0 1 1

0 1 0 0

0 1 0 1

CRC-8 0 1 1 0

CRC-8 0 1 1 1

MST 1 0 0 0

MST 1 0 0 1

1 0 1 0

1 0 1 1

1 1 0 0

1 1 0 1

SQ MSBs (bits 1 – 4) 1 1 1 0

SQ LSBs (bits 5 – 8) 1 1 1 1

VCAT Example

GID-a, SQ#=01

GID-a, SQ#=12

GID-a, SQ#=23

6

5

4

12

11

GID-b, SQ#=310

GID-b, SQ#=29

GID-b, SQ#=18

GID-a, SQ#=37

STS-1-4v STS-1-3v

STS-12

VCAT Puzzle• An STS-1-2v between Chicago and

London

• One STS-1 sent via geostationary satellite

• The other STS-1 sent via transatlantic fibre

• Will it work? ;-)

VCAT: Buffer RequirementsVC Path Type

Transport Signal

Number Paths

Buffer Size

STS-1VC-3

STS-12STM-4

12 18 MB

STS-1VC-3

STS-48STM-16

48 71 MB

STS-3cVC-4

STS-12STM-4

4 18 MB

STS-3cVC-4

STS-48STM-16

12 55 MB

Link Capacity Adjustment Scheme• Modify membership of a VCG

• On-demand and hitless bandwidth changes

• Automatic removal of failed VCG members

• Interworking of LCAS-enabled VCG with non-LCAS VCG

• Old-style ‘bridge and roll’ requires double bandwidth

LCAS Messaging• Member Status (MST)• Re-Sequence Acknowledge (RS-Ack)• Control (CTRL)

– Fixed– Add– Norm– EOS– Idle– DNU

• Group ID• CRC

LCAS Signalling Flows (1/3)

CTRL = ADD

SQ = 255

Source PTE STS-1 #1

CTRL = EOS

SQ = 2

CTRL = NORM

SQ = 1

MST = OK (#3)CTRL = NORM

SQ = 2

CTRL = NORM

SQ = 1

STS-1 #2 STS-1 #3

CTRL = NORM

SQ = 1

CTRL = EOS

SQ = 2

CTRL = EOS

SQ = 3

CTRL = IDLE

SQ = 255

CTRL = NORM

SQ = 1

CTRL = EOS

SQ = 2

RS-ACKCTRL = NORM

SQ = 2

CTRL = NORM

SQ = 1

CTRL = EOS

SQ = 3

MST = FAIL (#3)


CTRL = EOS

SQ = 3

Source PTE STS-1 #1

CTRL = NORM

SQ = 2

CTRL = IDLE

SQ = 255

CTRL = EOS

SQ = 3

CTRL = NORM

SQ = 2

CTRL = NORM

SQ = 1

MST = FAIL (#1)CTRL = EOS

SQ = 2

CTRL = NORM

SQ = 1

STS-1 #2 STS-1 #3

RS-ACKCTRL = EOS

SQ = 2

CTRL = NORM

SQ = 1


CTRL = IDLE

SQ = 255

Source PTE STS-1 #1

CTRL = NORM

SQ = 2

CTRL = EOS

SQ = 3

CTRL = NORM

SQ = 2

CTRL = NORM

SQ = 1

MST = FAIL (#3)CTRL = EOS

SQ = 2

CTRL = NORM

SQ = 1

STS-1 #2 STS-1 #3

RS-ACKCTRL = EOS

SQ = 2

CTRL = NORM

SQ = 1

CTRL = NORM

SQ = 1

Gigabit Ethernet Implementations

Gigabit Ethernet (802.3z)

FC-0 Interface & Media

FC-1 Encode/Decode

FC-2 Signalling

FC-3 Common Services

FC-4 Upper Layer Mapping

IEEE 802.3 PHY

IEEE 802.3 CSMA/CD

IEEE 802.2 LLC

ANSI X3T11 Fibre Channel IEEE 802.3 Ethernet IEEE 802.3z Gigabit Ethernet

MDL

IEE 802.3 MAC

IEEE 802.2 LLC

PMD

PMA

PCS

1000BASE-LX/SX

Sequence of EventsLINK_NOT_AVAILABLE LINK_NOT_AVAILABLE~ LINK_CONFIGURATION

~ LINK_CONFIGURATION IDLE ~ IDLE SOP PREAMBLE

SFD DA LLC DataSA LENGTH

PADDING FCS

Generic Framing Procedure• ITU-T G.7041 and ANSI T1.105.02• Defines mapping for many types of service

onto SONET/SDH or OTN:– Ethernet, IP/PPP– GbE, Fibre Channel (inc. DVB), FICON etc

• Excellent bandwidth utilization; efficiency tailored to suit different client types

• Simple delineation and robust error control• Extensible

Motivation for GFP (1/2)• ATM

– Cell overhead causes 10% bandwidth inflation– Adaptation functions needlessly complex

• Packet over SONET (POS)– Requires all frames to be converted to PPP

over HDLC– Byte stuffing causes non-deterministic

bandwidth inflation– QoS hard to monitor or guarantee

Motivation for GFP (2/2)• Minimal overhead

• Transport of client PDUs in native format

• Designed for optimized processing

• Easy aggregation of frames from multiple client and multiple protocols into shared bandwidth channels

• Low latency capabilities for SAN

GFP: Types of Frame

GFP

Client FramesControl Frames

Data FramesManagement

FramesIdle Frames

OA&M Frames

GFP: Functional Model

Source = IEEE Communications Magazine, May 2002

GFP: Frame Structure

Source = IEEE Communications Magazine, May 2002

Payload TypesValue of UPI Field Protocol GFP Mode

0x01 Ethernet F

0x02 PPP (IP & MPLS) F

0x03 Fibre Channel T

0x04 FICON T

0x05 ESCON T

0x06 GbE T

0x07 reserved

0x08 MAPOS F

0x09 – 0xFE reserved

0x00 & 0xFF unavailable

GFP: Client-Independent Processes

• Synchronization– Bit Level– Frame Delineation

• Scrambling

• Multiplexing– GFP Frame– Client PDU

GFP: Synchronization

Hunt Presync

SynccH

EC

Mism

atch2

nd c

HE

C M

atch

cHEC Match

No 2nd cHEC Match

cHEC Match

No CHEC Match

GFP: Frame Delineation

PayloadPLI cHEC PLI cHEC Payload

Payload PLI cHEC PayloadPLI cHEC PayloadPLI cHEC

PayloadPLI cHEC

GFP: Client PDU Multiplexing

Source Sink

STS-3c-2vGFP Framer

1

2

3

GFP Framer

1

2

3

1

2

3

GFP-F Encapsulation (1/2)

Preamble

SFD

Destination

Source

Length

Data

FCS

PLI (MSB)

PLI (LSB)

cHEC (MSB

cHEC (LSB)

Payload Header

Payload

FCS

1 8

Bit Transmission Order

Transm

ission Order

7

1

6

6

2

4 4

X = 4 to 64

0 to 65,535 - X

Ethernet MAC Frame

GFP-F Encapsulation (2/2)

Flag (0x7E)

Address

Control

PPP Type

Data

FCS

PLI (MSB)

PLI (LSB)

cHEC (MSB

cHEC (LSB)

Payload Header

Payload

FCS

1 8

Bit Transmission Order

Transm

ission Order

4

X = 4 to 64

0 to 65,535 - X

PPP/HDLC Frame

1

1

2

4

1

GFP-T: 64B/65B Payload

D1 K1 D2 D3 K2D4 D5 D6

000 001 010 011 101 110 111F

65B Sequence 1 001 C1 D2 D3 D4 D5 D61 0 101 C2 D1

100Octet Number

64B Sequence

000 001 010 011 100 101 110 111Octet Number

GFP-T Error Detection• Leading flag bit is errored

• Error affects ‘Last control-code’ indicator

• Control-code location address received in error

• Error causes 4-bit control code to be modified

GFP-T: Superblocks

1,1 1,2 … 1,7 1,8

8,1 8,2 8,7… 8,8

CRC (MSB)

CRC (LSB)

1,1

1,2

…

1,7

1,8

8,1

8,2

…

8,7

8,8

536-bit Byte-Aligned

GFP-T: Bandwidth Requirements

Client Signal

Client Signal

Bandwidth

Minimum Transport Channel

Size

Nominal Transport Channel

Bandwidth

Number Superblocks

per GFP Frame

ESCON 160 Mb/sSTS-1-4v

VC-3-4v193.536 Mb/s 1

Fibre Channel

850 Mb/sSTS-3c-4vVC-4-6v

898.56 Mb/s 13

Gigabit Ethernet

1000 Mb/sSTS-3c-7vVC-4-7v

1.04832 Gb/s 95

Pros and Cons: GFP-F

Higher bandwidth efficiency

Higher Latency

More buffer memory required

Core header fields must be calculated

Pros and Cons: GFP-T

Supports many protocols Low latency Ingress core header fields need not be

calculated

Less bandwidth efficientMore logic required

10 G Ethernet – 802.3ae

10GBASE-R

10GBASE-W

S = 850 nm MM 300 m

L = 1310 nm SM 10 km

E = 1550 nm SM 40 km

Full Duplex 802.3 MAC

64B/66B PCS

Serial PMASerial PMA

WIS

E L SE L S

XGMII

PMDPMD

10 GbE Overview• Usual 802.3 MAC and frame format

– Jumbo frames not included in standard

• Full duplex only– No shared media– No CSMA/CD

• Only optical fibre– No copper interface– LAN-PHY or WAN-PHY at various reaches

The STS-192c WIS Frame

J1

B3

C2

G1

F2

H4

Z3

Z4

Z5

Fixed S

tuff

Synchronous Payload Envelope (SPE) = 16704 Columns

63 columns

9.58464 Gbs-1 Capacity

802.3 FrameIdle Idle

802.3 FrameIdle Idle

…

…

Section & Line Overhead

A1

B1

D1

A1 … A1 A2 … A2

Section

A2

E1

D2

J0

F1

D3

Z0 … Z0

Overhead …

H1

B2

D4

D7

D10

S1

H1

B2

Z1

……

…

H1

B2

Z1

H2

K1

D5

D8

D11

Z2

…

…

H2

Z2

H2

M1

H3

K2

D6

D9

D12

E2

…H3 H3

Line Overhead …

1 2 192 193 194 384 385 386 576

Defined by WAN PHY as Fixed Value

Supported by WAN PHYUnused by WAN PHYOptional for SONET/SDH

Unused by WAN PHYUndefined for SONET/SDH

NetherLight Connectivity

UKLight – NetherLight Today

MetroDirector

UKLight

15454

NetherLight

HDXc

NetherLight

15454

CERN

CoreDirector

UKLight

MetroDirector

UKLight

UKLight – NetherLight Tomorrow?

HDXc

NetherLight

CoreDirector

UKLight

MetroDirector

UKLight

15454

CERN

Requires that MSPPs are GFP-F enabled

15454

NetherLight

Peering Using 10 G WAN-PHY

Force10

StarLight

CoreDirector

UKLight

10GBASE-SW

CoreDirector

UKLightS

TM

-6410GBASE-SW

Switch

UKLight

London

Chicago

ethernet over sonet dr. john s. graham university of london computer centre

Documents