Produced by and Copyright, 2016Philip Hippensteel
A Look at TCP Behavior & Bufferbloat:
Understanding, Detecting and Mitigating Performance Issues
NY and New EnglandTelephone Associations: 2/1/17
Dr. Phil [email protected](717) 448-0552
2Produced by and Copyright, 2016Philip Hippensteel
Why Look at this Topic?
Bufferbloat creates inefficiency in worker performance if they use the network or the Internet. It is not a security issue (although exploits
may be possible). IT generally is not unaware of the issue. Also, your ISP will likely be clueless.
3Produced by and Copyright, 2016Philip Hippensteel
How Did This problem Develop?
Due to poor understanding of network operations, particularly the TCP protocol, we: Developed the notion that dropping packets was
always bad. Didn’t really study and understand TCP, even though it
carried 90% of the traffic across our networks. Increased link speed dramatically on both the LAN and
the WAN. Increased speed differences at network connections.
4Produced by and Copyright, 2016Philip Hippensteel
What is Bufferbloat?
The effect that oversized buffers have on network latency. It causes Slow responses to web requests. Decreases in throughput of file transfers when the
return path is congested. Decreases in quality of adaptive bit rate video
(Netflix, Hulu, etc.) Can be easily confused with the affects of
malware.
5Produced by and Copyright, 2016Philip Hippensteel
The Background
Understanding bufferbloat means understanding TCP operation The role of DNS How buffers work and are managed
Understanding it’s impact means understanding DNS HTTP operation
6Produced by and Copyright, 2016Philip Hippensteel
Normal TCP Operation(Send Policy)
Station A Station Bsyn
syn/ackack
datadata
data ack
datadata
data
data
Cwnd=2
Cwnd=4
Cwnd= 8
VIP: The TCP sequence numbers increase by one per byte transferred. i.e. The seq # measures the number of TCP bytes
delivered.
data ack
Etc.
data
7Produced by and Copyright, 2016Philip Hippensteel
TCP Exchange
* View generated with Viavi Observer Gigastor.
8Produced by and Copyright, 2016Philip Hippensteel
The Receive Policy
When packets arrive in order: Ack a packet if either: 200 ms. has transpired since last ack OR Ack second packet received but unacked.
When packet arrives out-of-order Ack previously received in-order packet
9Produced by and Copyright, 2016Philip Hippensteel
Error Free Ideal TCP Performance
TCP send action has three phases. * Assuming no jitter, uniform delay, no packet loss, etc. Steady-state is a goal that can only be reached under ideal
conditions. Slow-start assumes cwnd ≤ ssthresh (later).
time
Segments sent
slowstart
CongestionAvoidance
Steady State*
10Produced by and Copyright, 2016Philip Hippensteel
Wireshark Shows Slow Start
Note delta time
11Produced by and Copyright, 2016Philip Hippensteel
Ideal TCP Performance w/ Packet loss
time
Segments sent
slowstart
Congestion Avoidance Steady
State
Dropped packet
slowstart
12Produced by and Copyright, 2016Philip Hippensteel
More on Send/Receive Policies
Sender Policy: cwnd: the number of segments that may
currently be sent before receiving an ack. rwnd: the number of incoming receiver buffer
bytes available number of segments that can be sent:
Min(cwnd, rwnd)
13Produced by and Copyright, 2016Philip Hippensteel
Cwnd and Latency
Cwnd depends on RTT. Therefore increased latency slows the
grow of cwnd (the ability to send).
cwnd
48
163264
1 RTT 2 RTT 3 RTT 4 RTT 5 RTT
time
14Produced by and Copyright, 2016Philip Hippensteel
Error Handling (more)
When a packet arrives out-of-order, the receiver:(1) immediately ack the previous packet (sender gets dup ack)
(2)Next packet out-of-order, receiver sends same ack. (sender has second dup ack)
(3) Next packet out-of-order, receiver sends same ack (sender receives third dup ack)
(4) Sender retransmits packet requested and proceeds with packet loss policy.
(5) Typically sender cuts cwnd to half and restarts slow-start.
slowstart
CongestionAvoidanc
e
slowstart
15Produced by and Copyright, 2016Philip Hippensteel
Wireshark Shows Dup-acks
16Produced by and Copyright, 2016Philip Hippensteel
TCP Stack Variations
Tahoe TCP Reno TCP New Reno Compound – current MS implementation Cubic – current Linux distros
17Produced by and Copyright, 2016Philip Hippensteel
Real Example of Uncongested TCP:Netflix
This is a connection to Netflix. DSL: 20M down, 2 M up.
18Produced by and Copyright, 2016Philip Hippensteel
Real Example of Uncongested TCP:Ubuntu Download
Connection to a Ubuntu repository server.
DSL: 20M down, 2 M up.
19Produced by and Copyright, 2016Philip Hippensteel
Real Example of TCP:Netflix(SSL) & Web Browsing(HTTP)
DSL: 20M down, 2 M up.
20Produced by and Copyright, 2016Philip Hippensteel
Real Example of TCP:Ubuntu Download & Dropbox Upload
• DSL: 20M down, 2 M up.
• So, at t=55, why doesn’t TCP use the available total bandwidth of 22Mb/sec?
** I believe this slide should be memorized!!!
21Produced by and Copyright, 2016Philip Hippensteel
PART 2: HTTP: HYPERTEXT TRANSFER PROTOCOL
22Produced by and Copyright, 2016Philip Hippensteel
HTTP Has Been Around
HTTP versions 0.9,1.0,1.1 Essentially a client request-server fulfill
protocolClient ServerStart TCP Session
Start TCP sessionGet Request(s)
Get Response (200 OK)Response Transmitted
TCP Session CloseTCP Session Cloe
23Produced by and Copyright, 2016Philip Hippensteel
HTTP 1.1 Typical Operation
Client Browser Web ServerGet index file
Index file transmittedGet page components
Other components sent
* * A recent web development text quoted a study in 2013 that showed that the average web request involved 90 retrievals of text, images, scripts, icons, etc. Many of these will involve a DNS query/response.
24Produced by and Copyright, 2016Philip Hippensteel
PART 3: BUFFERS AND BUFFERBLOAT
25Produced by and Copyright, 2016Philip Hippensteel
Where Are Buffers
In the stack between layers. In the NIC drivers. In the edge routers (especially wireless
routers). In boundary devices between service
providers.
26Produced by and Copyright, 2016Philip Hippensteel
Purpose of Buffers
Smooth flow of packets. Allow QOS policies to be implemented. Make packet drops more unlikely. Allow for security policy implementations.
27Produced by and Copyright, 2016Philip Hippensteel
Flows Though Different BW Links
Above: height is BW, width is time used. So, area is data carried. Ack rate is dependent on slowest link.
receiver
High BWLow BW
28Produced by and Copyright, 2016Philip Hippensteel
Buffer Action
1) Group of segments arrive
2) Segments buffered3) Segments begin to exit
buffer at slower speed4) Another group of
segments arrive5) Buffer full6) Last to arrive are
dropped (tail-drop)discards
Flow direction
29Produced by and Copyright, 2016Philip Hippensteel
Buffers and File Transfer
30Produced by and Copyright, 2016Philip Hippensteel
Ubuntu Download Starts
31Produced by and Copyright, 2016Philip Hippensteel
A Buffer Overfills
32Produced by and Copyright, 2016Philip Hippensteel
When is segment Retransmitted?
Note: 413 dup acks received before retranmission!
33Produced by and Copyright, 2016Philip Hippensteel
BUFFERS, DNS, HTTP AND SLOW START
34Produced by and Copyright, 2016Philip Hippensteel
A Practical View of DNS Use
request number dns queries average response time (ms)
get vanguard 15 24 get ebay 26 28 get google 18 25 get irs.gov 6 35 get highmark 2 25 get highmark mission values 3 36 get washington.edu 22 80
35Produced by and Copyright, 2016Philip Hippensteel
Buffers and DNS
Test with wired Ethernet/DSL
36Produced by and Copyright, 2016Philip Hippensteel
Buffers and DNS with WiFi/DSL
37Produced by and Copyright, 2016Philip Hippensteel
DNS and Upload on Wi-fi
No Upload
SimultaneousUpload
38Produced by and Copyright, 2016Philip Hippensteel
Summarizing Bufferbloat Affects
Slows slow start Decreases file transfer throughput by
delaying acks Increases DNS response times, increasing
response times for loading web pages
39Produced by and Copyright, 2016Philip Hippensteel
Part 3: Mitigation
40Produced by and Copyright, 2016Philip Hippensteel
Queue Management:Commonly Deployed Drop Policies
RED: random early drop WRED: weighted RED CoDel: drop based on queue occupancy fq-codel: codel with fairness to flows.
upstream
downstream
discard
41Produced by and Copyright, 2016Philip Hippensteel
Good Queues, Bad Queues
Bad Queue
Good Queue
packets
time
42Produced by and Copyright, 2016Philip Hippensteel
CoDel Operation
The critical metric is minimum queue sojourn time. A second parameter, target, is also used. Drop Policy: if a packet sojourns have a
minimum that exceed target for more than one interval time, then randomly drop a packet. In practice, interval = 100 ms and target =
5-10% of RTT.
43Produced by and Copyright, 2016Philip Hippensteel
CoDel Illustrated
Per packet sojourn time
time
target
interval
interval
44Produced by and Copyright, 2016Philip Hippensteel
fq-codel Operation
1: Incoming Flow 2: Random assignment to up to 1024 sub queues. 3: CoDel applied per queue. 4. One MTU per queue scheduled out, round robin.
●●●
2
1
3
45Produced by and Copyright, 2016Philip Hippensteel
Summary:Big Buffer Problems
Slow ack on large file transfers. Slow DNS response resulting in decreased
web performance. Collaboration Voice/RTP video: without VLANs or QOS, it can
cause packet loss (due to jitter buffer overflow) Video conferencing: cause SVC based VC to
degrade Adaptive video: cause decrease in resolution
or playback pause.
46Produced by and Copyright, 2016Philip Hippensteel
TESTING FOR BUFFERBLOAT
47Produced by and Copyright, 2016Philip Hippensteel
ICSI Netalyzr
48Produced by and Copyright, 2016Philip Hippensteel
www.dslreports/speedtest/
49Produced by and Copyright, 2016Philip Hippensteel
Manual Test
Start a continuous ping (ping www.google.com –t) Run test such as www.speedtest.net Observe change in ping times
50Produced by and Copyright, 2016Philip Hippensteel
Another Manual Test(continuous ping with upload)
Upload starts
Upload stops
51Produced by and Copyright, 2016Philip Hippensteel
Some Final Thoughts
Does QOS mitigate bufferbloat? No. It can determine which flows are affected
and by how much they are affected. If fq-codel is a mitigation, where should it
be implemented? At the edge router or in front of it.
Can traffic shaping mitigate bufferbloat? Yes, but at the expense of some throughput.
52Produced by and Copyright, 2016Philip Hippensteel
AQM with Wireless
Source: Controlling Queue Delay, ACM Queue, Kathy Nichols and Van Jacobson, 2012.
53Produced by and Copyright, 2016Philip Hippensteel
Questions?