belgrade university aleksandra smiljanić: high-capacity switching high-capacity packet switches

Aleksandra Smiljanić: High-Capacity Switching

Belgrade University

High-Capacity Packet Switches


Belgrade University

Switches with Input Buffers (Cisco)


Belgrade University

Packet Switches with Input Buffers

Switching fabric Electronic chips (Mindspeed, AMCC, Vitesse) Space-wavelength selector (NEC, Alcatel) Fast tunable lasers (Lucent) Waveguide arrays (Chiaro)

Scheduler Packets compete not only with the packets destined for

the same output but also with the packets sourced by the same input. Scheduling might become a bottleneck in a switch with hundreds of ports and gigabit line bit-rates.


Belgrade University

Optical Packet Cross-bar (NEC,Alcatel)

A 2.56 Tb/s multiwavelength and scalable switch-fabric for fast packet-switching network, PTL 1998,1999, NEC


Belgrade University

Optical Packet Cross-bar (Lucent)

A fast 100 channel wavelength tunable transmitter for optical packet switching, PTL 2001, Bell Labs


Belgrade University

Scheduling Algorithms for Packet Switches with Input Buffers

In parallel iterative matching (PIM), SLIP or dual round-robin (DRR) inputs send requests to outputs, outputs grant inputs, and inputs then grant outputs in one iteration. It was proven that PIM finds a maximal matching after log2N +4/3 steps on average.

Maximum weighted matching and maximum matching algorithm maximize the weight of the connected pairs, and achieve 100% for i.i.d. traffic but have complexities O(N3log2N) and O(N2.5).

Sequential greedy scheduling is a maximal matching algorithm that is simple to implement. Maximal matching algorithm does not leave input-output pair unmatched.


Belgrade University

PIM, SLIP and DRR

In PIM and SLIP each input sends requests to all outputs for which it has packets, and in DRR only to one chosen output. SLIP and DRR use round-robin choices.

Theorem: PIM finds a maximal matching after log2N +4/3 steps on average.

Proof: Let n inputs request output Q, and let k of these inputs receive no grants. With probability k/n all requests are resolved, and with probability 1-k/n at most k requests are unresolved. The average number of requests is at most (1-k/n)·k≤n/4. So if there are N2 requests at the beginning, the expected number of unresolved requests after I iterations is N2/4i


Belgrade University

PIM, SLIP and DRR

Proof (cont.): Let C be the last step on which the last request

is resolved. Then:

3

4log

4}iterationsafterrequests{

}iterationsafterrequests{}0{][

20

2

0 1

0 10

NN

ijjP

ijPCPCE

ii

i j

i ji


Belgrade University

Typical Central Controllers (Cisco)


Belgrade University

SGS Implementation

All inputs one after another choose outputs, SGS is a maximal matching algorithm


Belgrade University

SGS Uses Pipelining

Ii -> Tk Input i chooses output for time slot k


Belgrade University

Bandwidth ReservationsPacket Switches with Input Buffers

Anderson et al.: Time is divided into frames of F time slots. Schedule is calculated in each frame; Statistical matching algorithm.

Stiliadis and Varma: Counters are loaded per frame. Queues with positive counters are served with priority according to parallel iterative matching (PIM), their counters are then decremented by 1. DRR proposed by Chao et al. could be used as well.

Kam et al.: Counter is incremented for the negotiated bandwidth and decremented by 1 when the queue is served. Maximal weighted matching algorithm is applied.

Smiljanić: Counters are loaded per frame. Queues with positive counters are served with priority according to the maximal matching algorithm preferrably sequential greedy scheduling algorithm (SGS), where inputs sequentially choose outputs to transmit packets to.


Belgrade University

Weighted Sequential Greedy Scheduling

i=1; Input i chooses output j from Ok for which

it has packet to send; Remove i from Ik and j from Ok;

If i<N choose i=i+1 and go to the previous step;


Belgrade University

Weighted Sequential Greedy Scheduling

If k=1 mod F then cij=aij;

Ik={1,...,N}; Ok={1,...,N}; i=1; Input i chooses output j from Ok for which

it has packet to send such that cij>0; Remove i from Ik and j from Ok; cij=cij-1;

If i<N choose i=i+1 and go to the previous step;


Belgrade University

Performance of WSGS

Theorem: The WSGS protocol ensures aij time slots per frame to input-output pair (i,j), if

Proof: Note that

FaaaaRT ijm

mjm

imijji

ijim

mjjm

imim

mjjm

im aFaacc

where Ti is the number of slots reserved for input i, and Rj is the number of slots reserved for output j.


Belgrade University

Analogy with Circuit Switches

Inputs ~ Switches in the first stage

Time slots in a frame ~ Switches in the middle stage

Outputs ~ Switches in the last stage

Non-blocking condition: Fn Strictly non-blocking condition: Fn 12


Belgrade University

Admission Control for WSGSThe WSGS protocol ensures aij time slots per frame to input-output pair (i,j) if:

1 FRT jiI:

2/)1(,2/)1( FRFT jiII:

III: 2/1,2/1 ji rt

F frame lengthTi the number of slots reserved for input i, Rj the number of slots reserved for output j. ti, rj are normalized Ti, Rj.


Belgrade University

Non-blocking Nature of WSGS

Maximal matching algorithm does not leave input or output unmatched if there is a packet to be transmitted from the input to the output in question.

It can be proven that all the traffic passes through the cross-bar with the speedup of two which is run by a maximal matching algorithm, as long as the outputs are not overloaded.


Belgrade University

Rate and Delay Guranteed by WSGS

Assume a coarse synchronization on a frame by frame basis, where a frame is the policing interval comprising F cell time slots of duration Tc.

Then, the delay of D=2·F·Tc is provided for the utilization of 50%. Or, this delay and utilization of 100% are provided for the fabric with the speedup of 2.

%50

2

U

TFD c

2

%100

2

S

U

TFD c


Belgrade University

Port Congestion Due to Multicasting

1,,

lkjkl

m

lkk

ik

m

ik ppΜ

Μ

bit-rate reserved for multicast session k of input im

ikp

ikΜ multicast group k sourced by input i

Solution: Packets should be forwarded through the switch by multicast destination ports.


Belgrade University

Forwarding Multicast Traffic


Belgrade University

Adding the Port to the Multicast Tree


Belgrade University

Removing the Port from the Multicast Tree


Belgrade University

Admission Control for Modified WSGS

1 FRET jii

1FRRPT jii

where Ei is the number of forwarded packets per frame

1)1(

,

,

FFPF

FR

FT

rt

ri

ti


Belgrade University


2

1

1

1,minmax

),min(max,max

P

F

P

FFF

FEFC

ttF

rtFF

t

rt

M

2

1

P

FFF rt

for


Belgrade University


)2/()1(),2/()1( PFRPFT iiI:

II: )2/(1),2/(1 PrPt ii

Modified WSGS protocol ensures negotiated bandwidths to input-output pairs if for :

Ti the number of slots reserved for input i, Ri the number of slots reserved for output i. ti, ri are normalized Ti, Ri.

F frame length, P forwarding fan-out

Ni 1


Belgrade University

Rate and Delay Guaranteed by Modified WSGS

Assume again a coarse synchronization on a frame by frame basis.

Then, the delay of D= F·Tc is provided for the utilization of 1/(P+2), where P is the forwarding fan-out. Or, this delay and utilization of 100% are provided for the fabric speedup of P+2.

)2/(1

log

PU

TFND cP

2

%100

log

PS

U

TFND cP

NPlog


Belgrade University

Quality of Service, P=2, S=4, B=10Gb/s, Tc=50ns

N 1000 4000

F 104 5·104 104 5·104

C [Tb/s] 2.5 2.5 10 10

G [Mb/s] 1 0.2 1 0.2

D [ms] 5 25 5.5 27.5


Belgrade University

Clos Packet Switches


Belgrade University

Load Balancing in Packet Switches

J. Turner introduces load balancing of multicast sessions in Benes packet switches, INFOCOM 1993

C. Chang et al. propose load balancing in two-stage Birkhoff-von-Neumann switch, while Iyer et al. analyze the performance of the parallel plane switch (PPS) which applies load balancing.

Keslassy et al. propose the implementation of high-capacity PPS or Birkhoff-von-Neumann architecture.

Smiljanić examines rate and delay guarantees in three-stage Clos packet switches based on load balancing. These switches provide the larger number of lower speed ports.


Belgrade University

Load Balancing Algorithms

Packets are split into cells, and cells are grouped into flows.

Cells of each flow are balanced over center SEs Balancing of a flow can be implemented in the

following way: One counter is associated with each flow. When a cell of the flow arrives, it is marked to be transmitted

through the center SE whose designation equals the counter value, and then counter is incremented (decremented) modulo l, where l is the number of center SEs.


Belgrade University

Load Balancing Algorithms

A flow comprises cells determined by different rules, but that have the same input port or the input switching element (SE), and have the same output port or the output SE. Examples: SEs with input buffers

Cells sourced by the same inputCells sourced by the same input bound for the same outputCells sourced by the same input bound for the same output SE

SEs with shared buffersCells sourced by the same input SE bound for the same output Cells sourced by the same input SE bound for the same output

SE


Belgrade University

Non-Blocking Load Balancing

ii

ic l

nR

l

sR

1SE'

''

' SE'

''

3

''i k

kic

kl

nR

l

rR

Non-blocking if: , no speedup is needed 1nR

lRS c

l


Belgrade University

Let us assume the implementation with the coarse synchronization of the switching elements (SEs), i.e: the switching elements are synchronized on a frame-by-frame

basis in each frame any SE passes cells that arrived to this SE in the

previous frame The delay through a three-stage Clos network with such

coarse synchronization including packet reordering delay is:

Note that if multicasting is accomplished by the described packet forwarding, the utilization is decreased 3 times, and the delay is increased logPN times.

Rate and Delay Guarantees

cFTD 4


Belgrade University

Utilization under which the delay is guaranteed to be below D:

where S is the switching fabric speedup, Nf is the number of flows whose cells pass the internal fabric link, and Tc is the cell time slot duration.

Utilization Formula

nD

TlNSU cf

a

4


Belgrade University

Derivation of Utilization The maximum number of cells transmitted over a given

link from an input to a center SE, Fc, fulfills:

where fig is the number of cells per frame in flow g of cells from input SE i, and Fu is the maximum number of cells assigned to some port

If Nf -n flows have one cell per frame, and remaining n flows are assigned max(0,nFu-Nf+n) cells per frame

l

nFN

l

fN

l

fF u

fNg

igf

Ng

igc

ff

00

l

NNnF

l

NlFN

nl

N

l

FnnNNF

fufuf

fuffc

mod)()1(,max

,max


Belgrade University

Derivation of Utilization

So:

The same expression holds for Fc and Ua over the links from center to output SEs

Since F=D/(4Tc):

F

l

nF

lNS

F

FU

nF

lNS

l

nSFF

l

nFNFn

l

nFN

fua

fc

ufc

uf

,

nD

TlNS

F

FU cfu

a

4


Belgrade University

The delay of D is guaranteed for 100% utilization for the speedup of:

where Nf is the maximum number of flows whose cells pass any internal fabric link, and Tc is the cell time slot duration.

Speedup Formula

nD

TlNS cf

a

41


Belgrade University

We put Ua=1 in the formula for utilization, and readily obtain expression for the required speedup:

Derivation of Speedup

nD

TlNSU cf

aa

41

nD

TlNS cf

a

41


Belgrade University

Counter Synchronization

The utilization was decreased because all flows may be balanced starting for the same center SE, so this SE will not be able to deliver all the passing cells within a frame.

Higher utilization can be achieved if the counters of different flows are synchronized.

Counter of flow g sourced by input SE1i is reset at the beginning of each frame to cig =( i+g ) mod l, where l is the number of center SEs. And, counter of flow g bound for output SE3j is reset at the beginning of each frame to

cjg =( j+g ) mod l.


Belgrade University

Utilization under which the delay is guaranteed to be below D:

where S is the switching fabric speedup, Nf is the maximum number of flows whose cells pass any internal fabric link, and Tc is the cell time slot duration.

Utilization Formula when Counters are Synchronized

nS

TlND

TlN

DnSnS

TlND

nD

TlNS

Ucf

cf

cfcf

r 4

8

42

2


Belgrade University

Derivation of Utilization when Counters are Synchronized

The maximum number of cells transmitted over a given link from an input to a center SE2(l-1), Fc, fulfills:

where fig denotes the number of cells in flow g that are balanced starting from input SE1i

,22

1

mod)(mod)(

00

fufu

Ng

ig

Ng

igc

N

l

nFl

l

N

l

nF

l

lgif

l

lgifF

ff


Belgrade University


Fc is maximized for:

where yig > 0 are integers.

Fc is maximized if:

And Fc is then equal to:

igig yllgilf mod)(

n

lNl

n

NF

lgilfnF

ffu

NgNgigu

ff

22

1

)mod)((00

2fu

c

N

l

nFF


Belgrade University


If Fu<lNf /(2n), it holds that for some z<l:

In this case, Fc is maximized for

and equal to:

f

ufu

f

N

nlFz

zz

l

NnF

zz

l

N 2

2

)2()1(

2

)1(

zllgi

llgikzlklfig mod)(00

mod)(

l

NnF

l

zNF fuf

c

2


Belgrade University


From Fc nSF/l, it follows that:

nl

NF

nlF

Nn

lNF

nl

N

lN

FnS

nF

lNn

lNF

nF

lNS

F

FU

fu

f

fu

f

f

f

fu

f

ur

8

10

8

1028

10

2,

2min

222


Belgrade University


And:

nS

TlND

TlN

DnSnS

TlND

nD

TlNS

F

FU

nS

lNF

lN

FnSnS

lNF

nF

lNS

F

FU

cf

cf

cfcf

ur

f

f

ff

ur

4

8

42

2

2

2

2


Belgrade University

The delay of D is guaranteed for 100% utilization for the speedup of:

where Nf is the maximum number of flows whose cells pass any internal fabric link, and Tc is the cell time slot duration.

Speedup Formula when Counters are Synchronized

ccf

ccf

r

NTDnD

TlN

NTDnD

TlN

S2

8

22

1


Belgrade University

Derivation of Speedup when Counters are Synchronized

Speedup providing 100% utilization of the transmission capacity is derived when Fu=F in inequality Fc nSF/l:

n

lNF

nN

lN

l

nFNn

lNF

N

l

nF

FFS

n

lNF

nN

lN

l

NnFn

lNF

N

l

nF

FFS

fff

ff

cr

fu

ffu

fu

fu

cr

28

10222

28

10222


Belgrade University

Derivation of Speedup when Counters are Synchronized

Because F Nf >(10Nf)/(8N), for N 2.

n

TlND

lD

TnNn

TlND

nD

TlN

S

n

lNF

lF

nNn

lNF

nF

lN

Scfcf

cfcf

rff

ff

r 28

221

2

222

1


Belgrade University

Utilization vs. Number of Ports, Tc=50ns, n=m=l

Nf=nN

Nf=N

COUNTERS OUT OF SYNC COUNTERS IN SYNC


Belgrade University

Speedup vs. Number of Ports, Tc=50ns, n=m=l

Nf=nN

Nf=N



Belgrade University

Utilization and Speedup vs. Number of Ports,

D=3ms, Nf=m=l=640

UTIL

SPEED


1ms


Belgrade University

Conclusions: Scalable Implementations

Switches with input buffers require simple implementation: pipelining relaxes processing of output selector, central controller has a linear structure

Clos packet switches based on load balancing is even more scalable: they require neither the synchronization on a cell by cell basis across the whole fabric nor the high-capacity fabric.


Belgrade University

Conclusions: Performance Advantages

Both examined architectures provide nonblocking with moderate fabric speedups, i.e. the fabric passes all the traffic as long as outputs are not overloaded.

Rate and delay are guaranteed even to the most sensitive applications.

Due to the nonblocking nature of the fabric, the admission control can be distributed, and therefore more agile.


Belgrade University

References

T. E. Anderson, S. S. Owicki, J. B. Saxe, and C. P. Thacker, “Highspeed switch scheduling for local-area networks,” ACM Transactions on Computer Systems, vol. 11, no. 4, November 1993, pp. 319-352.

N. McKeown et al., “The Tiny Tera: A packet switch core,” IEEE Micro, vol. 17, no. 1, Jan.-Feb. 1997, pp. 26-33.

A. Smiljanić, “Flexible bandwidth allocation in high-capacity packet switches,” IEEE/ACM Transactions on Networking, April 2002, pp. 287-293.


Belgrade University

References

A. Smiljanić, “Scheduling of multicast trafc in high-capacity packet switches,” IEEE Communication Magazine, November 2002, pp. 72-77.

A. Smiljanić, “Performance of load balancing algorithms in Clos packet switches,” Proceedings of IEEE HPSR, April 2004, pp. 304-308.

J. S. Turner, “An optimal nonblocking multicast virtual circuit switch,” Proceeding of INFOCOM 1994, vol. 1, pp. 298-305.

belgrade university aleksandra smiljanić: high-capacity switching high-capacity packet switches

Documents