scinet caltech-slac experiments
DESCRIPTION
SC2002 Baltimore, Nov 2002. SCinet Caltech-SLAC experiments. Acknowledgments. netlab.caltech.edu/FAST. Prototype C. Jin, D. Wei Theory D. Choe (Postech/Caltech), J. Doyle, S. Low, F. Paganini (UCLA), J. Wang, Z. Wang (UCLA) Experiment/facilities - PowerPoint PPT PresentationTRANSCRIPT
SCinet Caltech-SLAC experiments
netlab.caltech.edu/FAST
SC2002 Baltimore, Nov 2002
PrototypeC. Jin, D. Wei
TheoryD. Choe (Postech/Caltech), J. Doyle, S. Low, F. Paganini (UCLA), J. Wang, Z. Wang (UCLA)
Experiment/facilities Caltech: J. Bunn, C. Chapman, C. Hu (Williams/Caltech), H. Newman, J. Pool, S.
Ravot (Caltech/CERN), S. Singh CERN: O. Martin, P. Moroni Cisco: B. Aiken, V. Doraiswami, R. Sepulveda, M. Turzanski, D. Walsten, S. Yip DataTAG: E. Martelli, J. P. Martin-Flatin Internet2: G. Almes, S. Corbato Level(3): P. Fernes, R. Struble SCinet: G. Goddard, J. Patton SLAC: G. Buhrmaster, R. Les Cottrell, C. Logg, I. Mei, W. Matthews, R. Mount, J.
Navratil, J. Williams StarLight: T. deFanti, L. Winkler
Major sponsorsARO, CACR, Cisco, DataTAG, DoE, Lee Center, NSF
Acknowledgments
FAST Protocols for Ultrascale Networks
netlab.caltech.edu/FAST
Internet: distributed feedback control system TCP: adapts sending rate to congestion AQM: feeds back congestion information
Rf (s)
Rb’(s)
x
))((1
lll
l ctyc
p
)()(1)( tan)(
)()(1-2
tqtttT
wx iid
tqtxi
ii ii
ii
y
pq
TCP AQM
Theory
Calren2/Abilene
Chicago
Amsterdam
CERN
Geneva
SURFNet
StarLight
WAN in LabCaltech
research & production networks
Multi-Gbps50-200ms delay
Experiment
155Mb/s
slowstart
equilibrium
FASTrecovery
FASTretransmit
timeout
10Gb/s
Implementation
Students Choe (Postech/CIT) Hu (Williams) J. Wang (CDS) Z.Wang (UCLA) Wei (CS)
Industry Doraiswami (Cisco) Yip (Cisco)
Faculty Doyle (CDS,EE,BE) Low (CS,EE) Newman (Physics) Paganini (UCLA)
Staff/Postdoc Bunn (CACR) Jin (CS) Ravot (Physics) Singh (CACR)
Partners CERN, Internet2, CENIC, StarLight/UI, SLAC, AMPATH, Cisco
People
netlab.caltech.edu
Outline
Motivation Theory
TCP/AQM TCP/IP
Experimental results
netlab.caltech.edu
HEP high speed network
… that must change
netlab.caltech.edu
HEP Network (DataTAG)
NLNLSURFnet
GENEVA
UKUKSuperJANET4
ABILENE
ABILENE
ESNETESNET
CALREN
CALREN
ItItGARR-B
GEANT
NewYork
FrFrRenater
STAR-TAP
STARLIGHT
Wave
Triangle
2.5 Gbps Wavelength Triangle 2002 10 Gbps Triangle in 2003
Newman (Caltech)
netlab.caltech.edu
Network upgrade 2001-06
’01155
’02622
’032.5
’04 5
’05 10
netlab.caltech.edu
Projected performance
Ns-2: capacity = 155Mbps, 622Mbps, 2.5Gbps, 5Gbps, 10Gbps100 sources, 100 ms round trip propagation delay
’01155
’02622
’032.5
’04 5
’05 10
J. Wang (Caltech)
netlab.caltech.edu
Projected performance
Ns-2: capacity = 10Gbps100 sources, 100 ms round trip propagation delay
FAST TCP/RED
J. Wang (Caltech)
netlab.caltech.edu
Outline
Motivation Theory
TCP/AQM TCP/IP
Experimental results
netlab.caltech.edu
Congestion control
xi(t)
pl(t)
Example congestion measure pl(t) Loss (Reno) Queueing delay (Vegas)
netlab.caltech.edu
TCP/AQM
Congestion control is a distributed asynchronous algorithm to share bandwidth
It has two components TCP: adapts sending rate (window) to congestion AQM: adjusts & feeds back congestion information
They form a distributed feedback control system Equilibrium & stability depends on both TCP and AQM And on delay, capacity, routing, #connections
pl(t)
xi(t)TCP: Reno Vegas
AQM: DropTail RED REM/PI AVQ
netlab.caltech.edu
Network model
F1
FN
G1
GL
Rf(s)
Rb’(s)
TCP Network AQM
x y
q p
lieR lis
lif link uses source if
lieR lislib link uses source if H
netlab.caltech.edu
Vegas model
F1
FN
G1
GL
Rf(s)
Rb’(s)
TCP Network AQM
x y
q p
1)(
l
ll c
tyG
ii
ii
dtqtx
i tTF
)()(
21sgn
)(
1
netlab.caltech.edu
Methodology
Protocol (Reno, Vegas, RED, REM/PI…)
Equilibrium Performance
Throughput, loss, delay
Fairness Utility
Dynamics Local stability Cost of stabilization
))( ),(( )1(
))( ),(( )1(
txtpGtp
txtpFtx
netlab.caltech.edu
Summary: duality model
cRx
xUs
ssxs
subject to
)( max0
Flow control problem
TCP/AQM Maximize utility with different utility functions
Primal-dual algorithm
))( ),(( )1(
))( ),(( )1(
txtpGtp
txtpFtx
Reno,
VegasDropTail, RED, REM
Theorem (Low 00): (x*,p*) primal-dual optimal iff 0 ifequality with ** lll pcy
netlab.caltech.edu
Equilibrium of VegasNetwork
Link queueing delays: pl
Queue length: clpl
Sources
Throughput: xi
E2E queueing delay : qi
Packets buffered:
Utility funtion: Ui(x) = i di log x Proportional fairness
iiii dqx
netlab.caltech.edu
Persistent congestion
Vegas exploits buffer process to compute prices (queueing delays)
Persistent congestion due to Coupling of buffer & price Error in propagation delay estimation
Consequences Excessive backlog Unfairness to older sources
Theorem (Low, Peterson, Wang ’02)
A relative error of i in propagation delay estimation distorts the utility function to
iiiiiiiii xdxdxU log)1()(ˆ
netlab.caltech.edu
Validation (L. Wang, Princeton)
Source rates (pkts/ms)# src1 src2 src3 src4 src51 5.98 (6) 2 2.05 (2) 3.92 (4)3 0.96 (0.94) 1.46 (1.49) 3.54 (3.57)4 0.51 (0.50) 0.72 (0.73) 1.34 (1.35) 3.38 (3.39)5 0.29 (0.29) 0.40 (0.40) 0.68 (0.67) 1.30 (1.30) 3.28
(3.34)
# queue (pkts) baseRTT (ms)1 19.8 (20) 10.18 (10.18)2 59.0 (60) 13.36 (13.51)3 127.3 (127) 20.17 (20.28)4 237.5 (238) 31.50 (31.50)5 416.3 (416) 49.86 (49.80)
netlab.caltech.edu
Methodology
Protocol (Reno, Vegas, RED, REM/PI…)
Equilibrium Performance
Throughput, loss, delay
Fairness Utility
Dynamics Local stability Cost of stabilization
))( ),(( )1(
))( ),(( )1(
txtpGtp
txtpFtx
netlab.caltech.edu
TCP/RED stability
Small effect on queue AIMD Mice traffic Heterogeneity
Big effect on queue Stability!
netlab.caltech.edu
Stable: 20ms delay
0 1000 2000 3000 4000 5000 6000 7000 8000 9000 100000
10
20
30
40
50
60
70Window
time (ms)
Win
dow
(pk
ts)
individual window
Window
Ns-2 simulations, 50 identical FTP sources, single link 9 pkts/ms, RED marking
netlab.caltech.edu
0 1000 2000 3000 4000 5000 6000 7000 8000 9000 100000
100
200
300
400
500
600
700
800Instantaneous queue
time (ms)
Inst
anta
neou
s qu
eue
(pkt
s)
Queue
Stable: 20ms delay
0 1000 2000 3000 4000 5000 6000 7000 8000 9000 100000
10
20
30
40
50
60
70Window
time (ms)
Win
dow
(pk
ts)
individual window
0 1000 2000 3000 4000 5000 6000 7000 8000 9000 100000
10
20
30
40
50
60
70Window
time (ms)
Win
dow
(pk
ts)
individual window
average window
Window
Ns-2 simulations, 50 identical FTP sources, single link 9 pkts/ms, RED marking
netlab.caltech.edu
0 1000 2000 3000 4000 5000 6000 7000 8000 9000 100000
10
20
30
40
50
60
70Window
time (10ms)
Win
dow
(pk
ts)
individual window
Unstable: 200ms delay
Window
Ns-2 simulations, 50 identical FTP sources, single link 9 pkts/ms, RED marking
netlab.caltech.edu
0 1000 2000 3000 4000 5000 6000 7000 8000 9000 100000
10
20
30
40
50
60
70Window
time (10ms)
Win
dow
(pk
ts)
individual window
0 1000 2000 3000 4000 5000 6000 7000 8000 9000 100000
10
20
30
40
50
60
70Window
time (10ms)
Win
dow
(pk
ts)
individual window
average window
Unstable: 200ms delay
0 1000 2000 3000 4000 5000 6000 7000 8000 9000 100000
100
200
300
400
500
600
700
800Instantaneous queue
time (10ms)
Inst
anta
neou
s qu
eue
(pkt
s)
QueueWindow
Ns-2 simulations, 50 identical FTP sources, single link 9 pkts/ms, RED marking
netlab.caltech.edu
Other effects on queue
0 1000 2000 3000 4000 5000 6000 7000 8000 9000 100000
100
200
300
400
500
600
700
800Instantaneous queue
time (ms)
Inst
anta
neou
s qu
eue
(pkt
s)
0 1000 2000 3000 4000 5000 6000 7000 8000 9000 100000
100
200
300
400
500
600
700
800Instantaneous queue
time (10ms)
Inst
anta
neou
s qu
eue
(pkt
s)
20ms
200ms
0 10 20 30 40 50 60 70 80 90 1000
100
200
300
400
500
600
700
800Instantaneous queue (50% noise)
time (sec)
inst
anta
neou
s qu
eue
(pkt
s)
30% noise
0 10 20 30 40 50 60 70 80 90 1000
100
200
300
400
500
600
700
800Instantaneous queue (50% noise)
time (sec)
inst
anta
neou
s qu
eue
(pkt
s)
30% noise
0 10 20 30 40 50 60 70 80 90 1000
100
200
300
400
500
600
700
800
time (sec)
Instantaneous queue (pkts)
inst
anta
neou
s qu
eue
(pkt
s)
avg delay 16ms
0 10 20 30 40 50 60 70 80 90 1000
100
200
300
400
500
600
700
800
time (sec)
Instantaneous queue (pkts)
inst
anta
neou
s qu
eue
(pkt
s)
avg delay 208ms
netlab.caltech.edu
222
2
3
33
)1(4
)1 )(
2
-(Nc
N
c
Theorem (Low et al, Infocom’02) Reno/RED is stable if
Stability: Reno/RED
F1
FN
G1
GL
Rf(s)
Rb’(s)
TCP Network AQM
x y
q p
TCP: Small Small c Large N
RED: Small Large delay
netlab.caltech.edu
Stability: scalable control
F1
FN
G1
GL
Rf(s)
Rb’(s)
TCP Network AQM
x y
q p
lll
l ctyc
tp )(1
)()(
)(tq
mii
iii
i
extx
Theorem (Paganini, Doyle, Low, CDC’01) Provided R is full rank, feedback loop is locally stable for arbitrary delay, capacity, load and topology
netlab.caltech.edu
Stability: Vegas
ii
ii
dtqtx
i tTx
)()(
21sgn
)(
1
F1
FN
G1
GL
Rf(s)
Rb’(s)
TCP Network AQM
x y
q p
lll
l ctyc
tp )(1
)(
Theorem (Choe & Low, Infocom’03) Provided R is full rank, feedback loop is locally stable if
), ;( max 20 kMTx ii
netlab.caltech.edu
Stability: Stabilized Vegas
)()(1)( tan)(
1 )()(1-
2tqtt
tTx iid
tqtxi ii
ii
F1
FN
G1
GL
Rf(s)
Rb’(s)
TCP Network AQM
x y
q p
lll
l ctyc
tp )(1
)(
Theorem (Choe & Low, Infocom’03) Provided R is full rank, feedback loop is locally stable if
),( max aTx ii
netlab.caltech.edu
Stability: Stabilized Vegas
)()(1)( tan)(
1 )()(1-
2tqtt
tTx iid
tqtxi ii
ii
F1
FN
G1
GL
Rf(s)
Rb’(s)
TCP Network AQM
x y
q p
lll
l ctyc
tp )(1
)(
Application Stabilized TCP with current routers Queueing delay as congestion measure has right scaling Incremental deployment with ECN
netlab.caltech.edu
Fast AQM Scalable TCP
Equilibrium properties Uses end-to-end delay and loss Achieves any desired fairness, expressed by utility function Very high utilization (99% in theory)
Stability properties Stability for arbitrary delay, capacity, routing & load Robust to heterogeneity, evolution, … Good performance
Negligible queueing delay & loss (with ECN) Fast response
netlab.caltech.edu
Implementation
Sender-side kernel modification Build on
Reno, NewReno, SACK, Vegas New insights
Difficulties due to Effects ignored in theory Large window size
First demonstration in SuperComputing Conf, Nov 2002 Developers: Cheng Jin & David Wei FAST Team & Partners
netlab.caltech.edu
Outline
Motivation Theory
TCP/AQM TCP/IP
Experimental results WAN in Lab
netlab.caltech.edu
Network
(Sylvain Ravot, caltech/CERN)
netlab.caltech.edu
FAST BMPS
Internet2Land Speed
Record
FAST
1 2
1
2
7
9
10
Gen
eva-
Sunn
yval
e
Baltim
ore-S
unnyvale
#flows
FAST Standard MTU Throughput averaged over > 1hr
netlab.caltech.edu
FAST BMPS
flows BmpsPeta
ThruputMbps
Distancekm
Delayms
MTUB
Durations
TransferGB
Path
Alaska-Amsterdam
9.4.2002
1 4.92 401 12,272 - - 13 0.625 Fairbanks, AL – Amsterdam,
NL
MS-ISI29.3.2000
2 5.38 957 5,626 - 4,470 82 8.4 MS, WA – ISI, Va
Caltech-SLAC19.11.2002
1 9.28 925 10,037 180 1,500 3,600 387 CERN -Sunnyvale
Caltech-SLAC19.11.2002
2 18.03 1,797 10,037 180 1,500 3,600 753 CERN -Sunnyvale
Caltech-SLAC18.11.2002
7 24.17 6,123 3,948 85 1,500 21,600 15,396 Baltimore -Sunnyvale
Caltech-SLAC19.11.2002
9 31.35 7,940 3,948 85 1,500 4,030 3,725 Baltimore -Sunnyvale
Caltech-SLAC20.11.2002
10 33.99 8,609 3,948 85 1,500 21,600 21,647 Baltimore -Sunnyvale
Mbps = 106 b/s; GB = 230 bytes
netlab.caltech.edu
Aggregate throughput
1 flow 2 flows 7 flows 9 flows 10 flows
Average utilization
95%
92%
90%
90%
88%FAST Standard MTU Utilization averaged over > 1hr
1hr 1hr 6hr 1.1hr 6hr
SCinet Caltech-SLAC experiments
netlab.caltech.edu/FAST
SC2002 Baltimore, Nov 2002
Experiment
Sunnyvale Baltimore
Chicago
Geneva
3000km 1000km
70
00
km
C. Jin, D. Wei, S. LowFAST Team and Partners
Internet: distributed feedbacksystem Rf (s)
Rb’(s)
x
p
TCP AQM
Theory
FAST TCP Standard MTU Peak window = 14,255 pkts Throughput averaged over > 1hr 925 Mbps single flow/GE card
9.28 petabit-meter/sec 1.89 times LSR
8.6 Gbps with 10 flows 34.0 petabit-meter/sec 6.32 times LSR
21TB in 6 hours with 10 flows
Implementation Sender-side modification Delay based
Highlights
1 2
1
2
7
9
10G
enev
a-Sunnyv
ale
Baltim
ore-
Sunn
yval
eFA
ST
I2 L
SR
#flows
netlab.caltech.edu
FAST vs Linux TCP
flows BmpsPeta
ThruputMbps
Distancekm
Delayms
MTUB
Durations
TransferGB
Path
Linux TCPtxqueulen=100
1 1.86 185 10,037 180 1,500 3600 78 CERN - Sunnyvale
Linux TCPtxqueulen=10000
1 2.67 266 10,037 180 1,500 3600 111 CERN - Sunnyvale
FAST19.11.2002
1 9.28 925 10,037 180 1,500 3600 387 CERN -Sunnyvale
Linux TCPtxqueulen=100
2 3.18 317 10,037 180 1,500 3600 133 CERN - Sunnyvale
Linux TCPtxqueulen=10000
2 9.35 931 10,037 180 1,500 3600 390 CERN - Sunnyvale
FAST19.11.2002
2 18.03 1,797 10,037 180 1,500 3600 753 CERN -Sunnyvale
Mbps = 106 b/s; GB = 230 bytes; Delay = propagation delayLinux TCP expts: Jan 28-29, 2003
netlab.caltech.edu
Aggregate throughput
Linux TCP Linux TCP FAST
Average utilization
19%
27%
92%FAST Standard MTU Utilization averaged over 1hr
txq=100 txq=10000
95%
16%
48%
Linux TCP Linux TCP FAST
2G
1G
netlab.caltech.edu
Effect of MTU
(Sylvain Ravot, Caltech/CERN)
Linux TCP
SCinet Caltech-SLAC experiments
netlab.caltech.edu/FAST
SC2002 Baltimore, Nov 2002
PrototypeC. Jin, D. Wei
TheoryD. Choe (Postech/Caltech), J. Doyle, S. Low, F. Paganini (UCLA), J. Wang, Z. Wang (UCLA)
Experiment/facilities Caltech: J. Bunn, C. Chapman, C. Hu (Williams/Caltech), H. Newman, J. Pool, S.
Ravot (Caltech/CERN), S. Singh CERN: O. Martin, P. Moroni Cisco: B. Aiken, V. Doraiswami, R. Sepulveda, M. Turzanski, D. Walsten, S. Yip DataTAG: E. Martelli, J. P. Martin-Flatin Internet2: G. Almes, S. Corbato Level(3): P. Fernes, R. Struble SCinet: G. Goddard, J. Patton SLAC: G. Buhrmaster, R. Les Cottrell, C. Logg, I. Mei, W. Matthews, R. Mount, J.
Navratil, J. Williams StarLight: T. deFanti, L. Winkler
Major sponsorsARO, CACR, Cisco, DataTAG, DoE, Lee Center, NSF
Acknowledgments
netlab.caltech.edu
FAST URL’s
FAST websitehttp://netlab.caltech.edu/FAST/
Cottrell’s SLAC websitehttp://www-iepm.slac.stanford.edu/monitoring/bulk/fast
netlab.caltech.edu
Outline
Motivation Theory
TCP/AQM TCP/IP Non-adaptive sources Content distribution
Implementation WAN in Lab
1
20
1
20
fiber spool
OPM
Max path length = 10,000 kmMax one-way delay = 50ms
S
S
S
S
R
R
H
R
: server
: router
electroniccrossconnect(Cisco 15454)
S
S
S
S
R
R
EDFA EDFA
500 km
netlab.caltech.edu
Unique capabilities
WAN in Lab Capacity: 2.5 – 10 Gbps Delay: 0 – 100 ms round trip
Configurable & evolvable Topology, rate, delays, routing Always at cutting edge
Risky research MPLS, AQM, routing, …
Integral part of R&A networks Transition from theory, implementation,
demonstration, deployment Transition from lab to marketplace
Global resource
(a) Physical network
R1
R2
R10
1
3
20
2
18
19
1
3
20
2
4
19
netlab.caltech.edu
Unique capabilities
WAN in Lab Capacity: 2.5 – 10 Gbps Delay: 0 – 100 ms round trip
Configurable & evolvable Topology, rate, delays, routing Always at cutting edge
Risky research MPLS, AQM, routing, …
Integral part of R&A networks Transition from theory, implementation,
demonstration, deployment Transition from lab to marketplace
Global resource
R1 R2
R3R10
(b) Logical network
1 23
419
20
netlab.caltech.edu
WAN in Lab Capacity: 2.5 – 10 Gbps Delay: 0 – 100 ms round trip
Configurable & evolvable Topology, rate, delays, routing Always at cutting edge
Risky research MPLS, AQM, routing, …
Integral part of R&A networks Transition from theory, implementation,
demonstration, deployment Transition from lab to marketplace
Global resource
Unique capabilities
Calren2/Abilene
Chicago
Amsterdam
CERN
Geneva
SURFNet
StarLight
WAN in LabCaltech
research & production networks
Multi-Gbps50-200ms delay
Experiment
netlab.caltech.edu
Coming together …
Clear & presentNeed
Resources
netlab.caltech.edu
Clear & presentNeed
Coming together …
Resources
netlab.caltech.edu
Clear & presentNeed
Coming together …
Resources FASTProtocols
FAST Protocols for Ultrascale Networks
netlab.caltech.edu/FAST
Internet: distributed feedback control system TCP: adapts sending rate to congestion AQM: feeds back congestion information
Rf (s)
Rb’(s)
x
))((1
lll
l ctyc
p
)()(1)( tan)(
)()(1-2
tqtttT
wx iid
tqtxi
ii ii
ii
y
pq
TCP AQM
Theory
Calren2/Abilene
Chicago
Amsterdam
CERN
Geneva
SURFNet
StarLight
WAN in LabCaltech
research & production networks
Multi-Gbps50-200ms delay
Experiment
155Mb/s
slowstart
equilibrium
FASTrecovery
FASTretransmit
timeout
10Gb/s
Implementation
Students Choe (Postech/CIT) Hu (Williams) J. Wang (CDS) Z.Wang (UCLA) Wei (CS)
Industry Doraiswami (Cisco) Yip (Cisco)
Faculty Doyle (CDS,EE,BE) Low (CS,EE) Newman (Physics) Paganini (UCLA)
Staff/Postdoc Bunn (CACR) Jin (CS) Ravot (Physics) Singh (CACR)
Partners CERN, Internet2, CENIC, StarLight/UI, SLAC, AMPATH, Cisco
People
netlab.caltech.edu
Backup slides
netlab.caltech.edu
TCP Congestion States
Established
Slow Start
High Throughput
ack for syn/ack cwnd > ssthreshpacing? gamma?
netlab.caltech.edu
From Slow Start to High Throughput
Linux TCP handshake differs from the TCP specification
Is 64 KB too small for ssthresh? 1 Gbps x 100 ms = 12.5 MB !
What about pacing? Gamma parameter in Vegas
netlab.caltech.edu
TCP Congestion States
Established
Slow Start
High Throughput
FAST’sRetransmitTime-out *
3 dup acks
retransmision timer fired
netlab.caltech.edu
High Throughput
Update cwnd as follows: +1 pkts in queue < + kq’ - 1 otherwise
Packet reordering may be frequent Disabling delayed ack can generate
many dup acks Is THREE the right number for Gbps?
netlab.caltech.edu
TCP Congestion States
Established
Slow Start
High Throughput
FAST’sRecovery
FAST’sRetransmit
3 dup acks
retransmit packetrecord snd_nxt
reduce cwnd/ssthresh
snd_una > recorded snd_nxt
send packet if in_flight < cwnd
netlab.caltech.edu
When Loss Happens
Reduce cwnd/ssthresh only when loss is due to congestion
Maintain in_flight and send data when in_flight < cwnd
Do FAST’s Recovery until snd_una >= recorded snd_nxt
netlab.caltech.edu
TCP Congestion States
Established
Slow Start
High Throughput
FAST’sRecovery
FAST’sRetransmitTime-out *
3 dup acks
retransmit packetrecord snd_nxt
reduce cwnd/ssthresh
retransmision timer fired
netlab.caltech.edu
When Time-out Happens
Very bad for throughput Mark all unacknowledged pkts as lost and
do slow start Dup acks cause false retransmits since
receiver’s state is unknown Floyd has a “fix” (RFC 2582).
netlab.caltech.edu
TCP Congestion States
Established
Slow Start
High Throughput
FAST’sRecovery
FAST’s RetransmitTime-out *
ack for syn/ackcwnd > ssthresh
3 dup acks
retransmit packetrecord snd_nxt
reduce cwnd/ssthresh
snd_una > recorded snd_nxt
retransmision timer fired
netlab.caltech.edu
Individual Packet States
Birth Sending In Flight Received
Queued Dropped Buffered
Freed Delivered
queueing
out of order queueand no memoryack’d
SCinet Bandwidth Challenge
netlab.caltech.edu/FAST
SC2002 Baltimore, Nov 2002
Experiment
Sunnyvale Baltimore
Chicago
Geneva
3000km 1000km
70
00
km
C. Jin, D. Wei, S. LowFAST Team and Partners
Internet: distributed feedbacksystem Rf (s)
Rb’(s)
x
p
TCP AQM
Theory
22.8.02IPv6
9.4.021 flow
29.3.00multiple
Balt
imore
-Geneva
Baltim
ore-
Sunn
yval
eSC20021 flow
SC20022 flows
SC200210 flows
I2 LSR
Sunnyvale
-Geneva
FAST TCP Standard MTU Peak window = 14,100 pkts 940 Mbps single flow/GE card
9.4 petabit-meter/sec 1.9 times LSR
9.4 Gbps with 10 flows 37.0 petabit-meter/sec 6.9 times LSR
16TB in 6 hours with 7 flows
Implementation Sender-side modification Delay based Stabilized Vegas
Highlights
netlab.caltech.edu
Baltim
ore-
Sunn
yval
e
Sun
nyv
ale-
Gen
eva
29.3.2000multiple
22.8.2002IPv6
9.4.20021 flow
SC2002 1 flow
SC200210 flows
FAST BMPS
I2 L
SR
Bmps Thruput Duration
37.0 9.40 Gbps min
9.42 940 Mbps 19 min
5.38 1.02 Gbps 82 sec
4.93 402 Mbps 13 sec
0.03 8 Mbps 60 min
FA
ST
netlab.caltech.edu
FAST: 7 flows
Statistics Data: 2.857 TB Distance: 3,936 km Delay: 85 msAverage Duration: 60 mins Thruput: 6.35 Gbps Bmps: 24.99 petab-m/sPeak Duration: 3.0 mins Thruput: 6.58 Gbps Bmps: 25.90 petab-m/s
Network SC2002 (Baltimore) SLAC (Sunnyvale), GE , Standard MTU
18 Nov 2002 Mon
cwnd = 6,658 pkts per flow
17 Nov 2002 Sun
netlab.caltech.edu
FAST: single flow
Statistics Data: 273 GB Distance: 10,025 km Delay: 180 msAverage Duration: 43 mins Thruput: 847 Mbps Bmps: 8.49 petab-m/sPeak Duration: 19.2 mins Thruput: 940 Mbps Bmps: 9.42 petab-m/s
Network CERN (Geneva) SLAC (Sunnyvale), GE, Standard MTU
17 Nov 2002 Sun
cwnd = 14,100 pkts
SCinet Bandwidth Challenge
netlab.caltech.edu/FAST
SC2002 Baltimore, Nov 2002
PrototypeC. Jin, D. Wei
TheoryD. Choe (Postech/Caltech), J. Doyle, S. Low, F. Paganini (UCLA), J. Wang, Z. Wang (UCLA)
Experiment/facilities Caltech: J. Bunn, S. Bunn, C. Chapman, C. Hu (Williams/Caltech), H. Newman, J.
Pool, S. Ravot (Caltech/CERN), S. Singh CERN: O. Martin, P. Moroni Cisco: B. Aiken, V. Doraiswami, M. Turzanski, D. Walsten, S. Yip DataTAG: E. Martelli, J. P. Martin-Flatin Internet2: G. Almes, S. Corbato SCinet: G. Goddard, J. Patton SLAC: G. Buhrmaster, L. Cottrell, C. Logg, W. Matthews, R. Mount, J. Navratil StarLight: T. deFanti, L. Winkler
Major sponsors/partnersARO, CACR, Cisco, DataTAG, DoE, Lee Center, Level3, NSF
Acknowledgments