winter 2004ee384x handout 11 ee384x: packet switch architectures handout 1: logistics and...
TRANSCRIPT
Winter 2004 EE384x Handout 1 1
EE384x: Packet Switch Architectures
Handout 1: Logistics and Introduction
Professor Balaji [email protected]
Professor Nick [email protected]
Winter 2004 EE384x Handout 1 2
Outline
This two course sequence is about the theory and practice of designing packet switches and Internet routers.
1. Introduction: What is a packet switch? The evolution of Internet routers, their basic architectural components, and some example architectures.
2. Part I: Output Queued Switches (Emphasis on Deterministic Analysis) OQ as the simplest and ideal architecture.Output queueing and shared-memory switches. Packet arrival processes: ()-constrained arrivals, leaky buckets, Bernoulli arrivals, bursty arrivals, adversaries.Providing bandwidth and delay guarantees, scheduling, fairness, Fair-Queueing, Generalized Processor Sharing and Deficit Round Robin.Practical difficulties: When output queued switches are impractical.Memory bandwidth and capacity scaling. Some approaches: Emulating output queued switches. Parallel packet buffers as standalone shared memory, with design examples. Routers with a single stage of buffering and constraint sets, Parallel Shared Memory Routers, Distributed Shared Memory Routers, and Parallel Packet Switches. Output link scheduling in a Distributed Shared Memory router. Combined input and output queued (CIOQ) switches, stable marriage matchings.
Winter 2004 EE384x Handout 1 3
Outline
3. Part II: Input Queued Switches (Emphasis on Probabilistic Analysis). What is an input-queued (IQ) switch?Definition of IQ switch with single FCFS queue. Switching fabrics, crossbars. Head of line blocking. The balls and bins model. Proof of Karol's 2-sqrt(2) (58%) result. Virtual output queues and crossbar schedulers. Bipartite Matchings: Maximum Sized Matchings, Maximum Weight Matchings, maximal matchings. Definitions of 100% throughput. When traffic is uniform: simple RR and random matchings. When traffic matrix is known: Birkhoff- von Neuman decomposition. When traffic is not known: heuristics. PIM, iSLIP, WFA.
4. Fundamentals (Review Sessions): Introduction to probability, Poisson process, Discrete and Continuous-time Markov chains. Basic queueing theory: M/M/1, M/G/1, Little’s result, PASTA.
Winter 2004 EE384x Handout 1 4
Outline EE384y
1. Part II: Input Queued Switches (Continued).Intro to Lyapunov functions, proof that max weight matching gives 100% throughput.Some case studies: The Tiny Tera architecture. The Cisco GSR 12000.
2. Part III: Other Switch ArchitecturesBuffered crossbars. Scaling crossbars and parallelism. Multistage switches: Clos networks, 2-stage switches (random and deterministic).
3. Part IV: Other Switch FunctionsAddress Lookup: Exact matches, longest prefix matches, performance metrics, hardware and software solutions. Packet Classification: For firewalls, QoS, and policy-based routing; graphical description and examples of 2-D classification, examples of classifiers, theoretical and practical considerations.
4. Special topics.5. Project presentations.
5 EE384x Handout 1 Winter 2004
Some logisticsWeb page: http://www.stanford.edu/class/ee384x
Office hours: Balaji Prabhakar – TBA on web pageNick McKeown – Wednesday 4.00pm – 5.00pm
Course assistant:Theresa Wan – [email protected]
Gates 351; Tel: (650) 725 9077
TAs:Rui Zhang-Shen – [email protected] Keshavarzian- [email protected] hours: See class web page.
Grades:You need to sign up with “eeclass” on the EE384x web page.
Winter 2004 EE384x Handout 1 6
More LogisticsPrerequisite
EE284/CS244A and familiarity with probability.
Useful Stats 116 (or EE178, EE278) and CS161
Papers URLs to all the papers are on the eeclass web page.
Grading (40%) 5 Problem sets (10%) Several surprise quizzes (20%) In-class midterm exam (Tuesday February 17) (30%) Final exam (Wednesday March 17)
SITN Students Same schedule as in-class students Fax your assignment to us at: (702) 977 0556.
All deadlines are hard!
Winter 2004 EE384x Handout 1 7
An IntroductionThe class starts here!
Background What is a router? Why do we need faster routers? Why are they hard to build?
Architectures and techniques The evolution of router architecture. IP address lookup. Packet buffering. Switching.
Winter 2004 EE384x Handout 1 8
What is Routing?
R3
A
B
C
R1
R2
R4 D
E
FR5
R5F
R3E
R3D
Next HopDestination
D
Winter 2004 EE384x Handout 1 9
What is Routing?
R3
A
B
C
R1
R2
R4 D
E
FR5
R5F
R3E
R3D
Next HopDestination
D
16 3241
Data
Options (if any)
Destination Address
Source Address
Header ChecksumProtocolTTL
Fragment OffsetFlagsFragment ID
Total Packet LengthT.ServiceHLenVer
20
byte
s
Winter 2004 EE384x Handout 1 10
What is Routing?
A
B
C
R1
R2
R3
R4 D
E
FR5
Winter 2004 EE384x Handout 1 11
Points of Presence (POPs)
A
B
C
POP1
POP3POP2
POP4 D
E
F
POP5
POP6 POP7POP8
Winter 2004 EE384x Handout 1 12
Where High Performance Routers are Used
R10 R11
R4
R13
R9
R5
R2R1 R6
R3 R7
R12
R16R15
R14
R8
(2.5 Gb/s)
(2.5 Gb/s)(2.5 Gb/s)
(2.5 Gb/s)
Winter 2004 EE384x Handout 1 13
What a Router Looks LikeCisco GSR 12416 Juniper M160
6ft
19”
2ft
Capacity: 160Gb/sPower: 4.2kW
3ft
2.5ft
19”
Capacity: 80Gb/sPower: 2.6kW
Winter 2004 EE384x Handout 1 14
Basic Architectural Components
of an IP Router
Control Plane
Datapathper-packet processing
SwitchingForwarding
Table
Routing Table
Routing Protocols
Winter 2004 EE384x Handout 1 15
Per-packet processing in an IP Router
1. Accept packet arriving on an incoming link.
2. Lookup packet destination address in the forwarding table, to identify outgoing port(s).
3. Manipulate packet header: e.g., decrement TTL, update header checksum.
4. Send packet to the outgoing port(s).5. Buffer packet in the queue (在 output
queue).6. Transmit packet onto outgoing link.
Winter 2004 EE384x Handout 1 16
Generic Router Architecture
LookupIP Address
UpdateHeader
Header ProcessingData Hdr Data Hdr
~1M prefixesOff-chip DRAM
AddressTable
AddressTable
IP Address Next Hop
QueuePacket
BufferMemoryBuffer
Memory~1M packetsOff-chip DRAM
Winter 2004 EE384x Handout 1 17
Generic Router Architecture
LookupIP Address
UpdateHeader
Header Processing
AddressTable
AddressTable
LookupIP Address
UpdateHeader
Header Processing
AddressTable
AddressTable
LookupIP Address
UpdateHeader
Header Processing
AddressTable
AddressTable
Data Hdr
Data Hdr
Data Hdr
BufferManager
BufferMemory
BufferMemory
BufferManager
BufferMemory
BufferMemory
BufferManager
BufferMemory
BufferMemory
Data Hdr
Data Hdr
Data Hdr
Winter 2004 EE384x Handout 1 18
Why do we Need Faster Routers?
1. To prevent routers becoming the bottleneck in the Internet.
2. To increase POP capacity, and to reduce cost, size and power.
Winter 2004 EE384x Handout 1 19
0,1
1
10
100
1000
10000
1985 1990 1995 2000
Spec
95In
t CPU
resu
ltsWhy we Need Faster Routers
1: To prevent routers from being the bottleneck
0,1
1
10
100
1000
10000
1985 1990 1995 2000
Fib
er
Ca
pa
cit
y (
Gb
it/s
)
TDM DWDM
Packet processing Power Link Speed
2x / 18 months 2x / 7 months
Source: SPEC95Int & David Miller, Stanford.
Winter 2004 EE384x Handout 1 20
POP with smaller routers
Why we Need Faster Routers 2: To reduce cost, power & complexity of
POPsPOP with large routers
Ports: Price >$100k, Power > 400W. It is common for 50-60% of ports to be for interconnection.
Winter 2004 EE384x Handout 1 21
Why are Fast Routers Difficult to Make?
1. It’s hard to keep up with Moore’s Law: The bottleneck is memory speed. Memory speed is not keeping up with Moore’s Law.
Winter 2004 EE384x Handout 1 22
Why are Fast Routers Difficult to Make?
Speed of Commercial DRAM
1. It’s hard to keep up with Moore’s Law: The bottleneck is memory speed. Memory speed is not keeping up with Moore’s Law.
0.001
0.01
0.1
1
10
100
1000
1980 1983 1986 1989 1992 1995 1998 2001
Acc
ess
Tim
e (n
s)
Moore’s Law2x / 18 months
1.1x / 18 months
Winter 2004 EE384x Handout 1 23
Why are Fast Routers Difficult to Make?
1. It’s hard to keep up with Moore’s Law: The bottleneck is memory speed. Memory speed is not keeping up with Moore’s Law.
2. Moore’s Law is too slow: Routers need to improve faster than Moore’s Law.
Winter 2004 EE384x Handout 1 24
Router Performance Exceeds Moore’s Law
Growth in capacity of commercial routers: Capacity 1992 ~ 2Gb/s Capacity 1995 ~ 10Gb/s Capacity 1998 ~ 40Gb/s Capacity 2001 ~ 160Gb/s Capacity 2003 ~ 640Gb/s
Average growth rate: 2x / 18 months.
Winter 2004 EE384x Handout 1 25
Outline
Background What is a router? Why do we need faster routers? Why are they hard to build?
Architectures and techniques The evolution of router architecture. IP address lookup. Packet buffering. Switching.
Winter 2004 EE384x Handout 1 26
RouteTableCPU Buffer
Memory
LineInterface
MAC
LineInterface
MAC
LineInterface
MAC
Typically <0.5Gb/s aggregate capacity
First Generation RoutersShared Backplane
Line Interface
CPU
Memory
Winter 2004 EE384x Handout 1 27
Second Generation RoutersRouteTableCPU
LineCard
BufferMemory
LineCard
MAC
BufferMemory
LineCard
MAC
BufferMemory
FwdingCache
FwdingCache
FwdingCache
MAC
BufferMemory
Typically <5Gb/s aggregate capacity
Winter 2004 EE384x Handout 1 28
Third Generation Routers
LineCard
MAC
LocalBuffer
Memory
CPUCard
LineCard
MAC
LocalBuffer
Memory
Switched Backplane
Line Interface
CPUMem
ory FwdingTable
RoutingTable
FwdingTable
Typically <50Gb/s aggregate capacity
Winter 2004 EE384x Handout 1 29
Fourth Generation Routers/Switches
Optics inside a router for the first time
Switch Core Linecards
Optical links
100sof metres
0.3 - 10Tb/s routers in development
Winter 2004 EE384x Handout 1 30
Outline
Background What is a router? Why do we need faster routers? Why are they hard to build?
Architectures and techniques The evolution of router architecture. IP address lookup. Packet buffering. Switching.
Winter 2004 EE384x Handout 1 31
Generic Router Architecture
LookupIP Address
UpdateHeader
Header Processing
AddressTable
AddressTable
LookupIP Address
UpdateHeader
Header Processing
AddressTable
AddressTable
LookupIP Address
UpdateHeader
Header Processing
AddressTable
AddressTable
BufferManager
BufferMemory
BufferMemory
BufferManager
BufferMemory
BufferMemory
BufferManager
BufferMemory
BufferMemory
LookupIP Address
AddressTable
AddressTable
LookupIP Address
AddressTable
AddressTable
LookupIP Address
AddressTable
AddressTable
Winter 2004 EE384x Handout 1 32
IP Address Lookup
Why it’s thought to be hard:1. It’s not an exact match: it’s a longest prefix match. 2. The table is large: about 150,000 entries today,
and growing. 3. The lookup must be fast: about 30ns for a 10Gb/s
line. 批注:那么假设一个 Packet的平均大小为 300bit
Winter 2004 EE384x Handout 1 33
IP Lookups find Longest Prefixes
128.9.16.0/21128.9.172.0/21
128.9.176.0/24
0 232-1
128.9.0.0/16142.12.0.0/1965.0.0.0/8
128.9.16.14
Routing lookup: Find the longest matching prefix (aka the most specific route) among all prefixes that match the destination address.
Winter 2004 EE384x Handout 1 34
IP Address Lookup
Why it’s thought to be hard:1. It’s not an exact match: it’s a longest prefix match. 2. The table is large: about 150,000 entries today,
and growing. 3. The lookup must be fast: about 30ns for a 10Gb/s
line.
Winter 2004 EE384x Handout 1 35
Address Tables are Large
Source: http://www.cidr-report.org/
Winter 2004 EE384x Handout 1 36
IP Address Lookup
Why it’s thought to be hard:1. It’s not an exact match: it’s a longest prefix match. 2. The table is large: about 150,000 entries today,
and growing. 3. The lookup must be fast: about 30ns for a 10Gb/s
line.
Winter 2004 EE384x Handout 1 37
Lookups Must be Fast
12540Gb/s2003
31.2510Gb/s2001
7.812.5Gb/s1999
1.94622Mb/s1997
40B packets (Mpkt/s)
LineYear
Winter 2004 EE384x Handout 1 38
Outline
Background What is a router? Why do we need faster routers? Why are they hard to build?
Architectures and techniques The evolution of router architecture. IP address lookup. Packet buffering. Switching.
Winter 2004 EE384x Handout 1 39
Generic Router Architecture
LookupIP Address
UpdateHeader
Header Processing
AddressTable
AddressTable
LookupIP Address
UpdateHeader
Header Processing
AddressTable
AddressTable
LookupIP Address
UpdateHeader
Header Processing
AddressTable
AddressTable
QueuePacket
BufferMemory
BufferMemory
QueuePacket
BufferMemory
BufferMemory
QueuePacket
BufferMemory
BufferMemory
BufferManager
BufferMemory
BufferMemory
BufferManager
BufferMemory
BufferMemory
BufferManager
BufferMemory
BufferMemory
Winter 2004 EE384x Handout 1 40
Fast Packet Buffers
Example: 40Gb/s packet bufferSize = RTT*BW = 10Gb; 40 byte packets批注: SRAM速度快,但容量小; DRAM则相反
Write Rate, R
1 packetevery 8 ns
Read Rate, R
1 packetevery 8 ns
BufferManager
BufferMemory
Use SRAM?+ fast enough random access time, but
- too low density to store 10Gb of data.
Use SRAM?+ fast enough random access time, but
- too low density to store 10Gb of data.
Use DRAM?+ high density means we can store data, but- too slow (50ns random access time).
Use DRAM?+ high density means we can store data, but- too slow (50ns random access time).
Winter 2004 EE384x Handout 1 41
Outline
Background What is a router? Why do we need faster routers? Why are they hard to build?
Architectures and techniques The evolution of router architecture. IP address lookup. Packet buffering. Switching.
Winter 2004 EE384x Handout 1 42
Generic Router Architecture
LookupIP Address
UpdateHeader
Header Processing
AddressTable
AddressTable
LookupIP Address
UpdateHeader
Header Processing
AddressTable
AddressTable
LookupIP Address
UpdateHeader
Header Processing
AddressTable
AddressTable
QueuePacket
BufferMemory
BufferMemory
QueuePacket
BufferMemory
BufferMemory
QueuePacket
BufferMemory
BufferMemory
Data Hdr
Data Hdr
Data Hdr
1
2
N
1
2
N
N times line rate
N times line rate
Winter 2004 EE384x Handout 1 43
Generic Router Architecture
LookupIP Address
UpdateHeader
Header Processing
AddressTable
AddressTable
LookupIP Address
UpdateHeader
Header Processing
AddressTable
AddressTable
LookupIP Address
UpdateHeader
Header Processing
AddressTable
AddressTable
QueuePacket
BufferMemory
BufferMemory
QueuePacket
BufferMemory
BufferMemory
QueuePacket
BufferMemory
BufferMemory
Data Hdr
Data Hdr
Data Hdr
1
2
N
1
2
N
Data Hdr
Data Hdr
Data Hdr
Scheduler
Winter 2004 EE384x Handout 1 440% 20% 40% 60% 80% 100%
Load
Del
ayA Router with Input Queues
The best that any queueing system can
achieve.
Winter 2004 EE384x Handout 1 450% 20% 40% 60% 80% 100%Load
Del
ayA Router with Input Queues
Head of Line Blocking
The best that any queueing system can
achieve.
2 2 58%
Winter 2004 EE384x Handout 1 46
Head of Line Blocking
Winter 2004 EE384x Handout 1 47
Virtual Output Queues批注: VOQ消除了 IQ 的 HOL问题
Winter 2004 EE384x Handout 1 480% 20% 40% 60% 80% 100%
Load
Del
ayA Router with Virtual Output
Queues
The best that any queueing system can
achieve.
Winter 2004 EE384x Handout 1 49
Maximum Weight Matching
A1(n)
N N
LNN(n)
A1N(n)
A11(n)
L11(n)
1 1
AN(n)
ANN(n)
AN1(n)
D1(n)
DN(n)
L11(n)
LN1(n)
“ Request” Graph Bipartite Match
S*(n)
MaximumWeight Match
*
( )( ) argmax( ( ) ( ))T
S nS n L n S n
Winter 2004 EE384x Handout 1 50
Outline of Proof
( 1) ( 1 .
( )
) ( ) (
[ ( )
) ( )
]
( )
We can show that:
Hence, if is large enough, there is an expected
single-step downward drif t in occupancy.
and 100
T TE L n L n L n L n |
L n
E L n
L n c L n
%throughput is achieved.
Winter 2004 EE384x Handout 1 51
There are now many ways to achieve 100% throughput…
52
Scheduling algorithms to achieve 100% throughput
1. Basic switch model.2. When traffi c is uniform (Many algorithms…)3. When traffi c is non-unif orm, but traffi c matrix is known.
• Technique: Birkhoff -von Neumann decomposition.
4. When matrix is not known.• Technique: Lyapunov f unction.
5. When algorithm is pipelined, or inf ormation is incomplete.• Technique: Lyapunov f unction.
6. When algorithm does not complete.• Technique: Randomized algorithm.
7. When there is speedup.• Technique: Fluid model.
8. When there is no algorithm.• Technique: 2-stage load-balancing switch.• Technique: Parallel Packet Switch.
Winter 2004 EE384x Handout 1 52
The Evolution of Switching
Theory:
Practice:
InputQueueing
(IQ)
InputQueueing
(IQ)
InputQueueing
(IQ)
InputQueueing
(IQ)
58% [Karol, 1987]
IQ + VOQ,Maximum weight matching
IQ + VOQ,Maximum weight matching
IQ + VOQ,Sub-maximal size matching
e.g. PIM, iSLIP.
IQ + VOQ,Sub-maximal size matching
e.g. PIM, iSLIP.
100% [M et al., 1995]
Different weight functions,incomplete information, pipelining.
Different weight functions,incomplete information, pipelining.
Randomized algorithmsRandomized algorithms
100% [Tassiulas, 1998]
100% [Various]
Various heuristics, distributed algorithms,
and amounts of speedup
Various heuristics, distributed algorithms,
and amounts of speedup
IQ + VOQ,Maximal size matching,
Speedup of two.
IQ + VOQ,Maximal size matching,
Speedup of two.
100% [Dai & Prabhakar, 2000]
Winter 2004 EE384x Handout 1 53
Current Internet Router TechnologySummary
There are three potential bottlenecks: Address lookup, Packet buffering, and Switching. (问题:这是什么意思?与前 2 者有何不同)
Techniques exist today for: 10+Tb/s Internet routers, with 40Gb/s linecards.