![Page 1: policing-paper@google.com An Internet-Wide Analysis of · An Internet-Wide Analysis of Traffic Policing Tobias Flach, Pavlos Papageorge, Andreas Terzis, Luis Pedrosa, Yuchung Cheng,](https://reader033.vdocuments.mx/reader033/viewer/2022051604/6002331e2285606e082ea4fc/html5/thumbnails/1.jpg)
An Internet-Wide Analysis of Traffic Policing
Tobias Flach, Pavlos Papageorge, Andreas Terzis, Luis Pedrosa, Yuchung Cheng, Tayeb Karim, Ethan Katz-Bassett, Ramesh Govindan
1
![Page 2: policing-paper@google.com An Internet-Wide Analysis of · An Internet-Wide Analysis of Traffic Policing Tobias Flach, Pavlos Papageorge, Andreas Terzis, Luis Pedrosa, Yuchung Cheng,](https://reader033.vdocuments.mx/reader033/viewer/2022051604/6002331e2285606e082ea4fc/html5/thumbnails/2.jpg)
Users
Internet Service Provider
(ISP)Content Providers
2
![Page 3: policing-paper@google.com An Internet-Wide Analysis of · An Internet-Wide Analysis of Traffic Policing Tobias Flach, Pavlos Papageorge, Andreas Terzis, Luis Pedrosa, Yuchung Cheng,](https://reader033.vdocuments.mx/reader033/viewer/2022051604/6002331e2285606e082ea4fc/html5/thumbnails/3.jpg)
3
Exponential growth of
video traffic
Want to accommodate multitude of services/policies
→ Traffic Engineering
Account for ~ 50% of traffic in North America
Want to maximize quality of experience (QoE) for
their users
Often need high bitrate with low tolerance for
latency and packet loss
![Page 4: policing-paper@google.com An Internet-Wide Analysis of · An Internet-Wide Analysis of Traffic Policing Tobias Flach, Pavlos Papageorge, Andreas Terzis, Luis Pedrosa, Yuchung Cheng,](https://reader033.vdocuments.mx/reader033/viewer/2022051604/6002331e2285606e082ea4fc/html5/thumbnails/4.jpg)
Focus ofthis talk
Traffic Engineering: Policing vs. Shaping
Goal: Enforce a rate limit (maximum throughput)
4
Solutions:
a. Drop packets once the limit is reached→ Traffic Policing
b. Queue packets (and send them out at the maximum rate)→ Traffic Shaping
![Page 5: policing-paper@google.com An Internet-Wide Analysis of · An Internet-Wide Analysis of Traffic Policing Tobias Flach, Pavlos Papageorge, Andreas Terzis, Luis Pedrosa, Yuchung Cheng,](https://reader033.vdocuments.mx/reader033/viewer/2022051604/6002331e2285606e082ea4fc/html5/thumbnails/5.jpg)
Contribution
Analyze the prevalence and impact oftraffic policing on a global scale,
as well as explore ways tomitigate the impact of policers.
5
![Page 6: policing-paper@google.com An Internet-Wide Analysis of · An Internet-Wide Analysis of Traffic Policing Tobias Flach, Pavlos Papageorge, Andreas Terzis, Luis Pedrosa, Yuchung Cheng,](https://reader033.vdocuments.mx/reader033/viewer/2022051604/6002331e2285606e082ea4fc/html5/thumbnails/6.jpg)
Outline
1. How Policing Works
2. Detecting the Effects of Policing in Packet Captures
3. A Global-Scale Analysis of Policing in the Internet
4. Mitigating the Impact of Policers
6
![Page 7: policing-paper@google.com An Internet-Wide Analysis of · An Internet-Wide Analysis of Traffic Policing Tobias Flach, Pavlos Papageorge, Andreas Terzis, Luis Pedrosa, Yuchung Cheng,](https://reader033.vdocuments.mx/reader033/viewer/2022051604/6002331e2285606e082ea4fc/html5/thumbnails/7.jpg)
How Policing Works
Policer?
Packet leaves if enough tokens
are available
7
Tokens refreshed at predefined policing rate
![Page 8: policing-paper@google.com An Internet-Wide Analysis of · An Internet-Wide Analysis of Traffic Policing Tobias Flach, Pavlos Papageorge, Andreas Terzis, Luis Pedrosa, Yuchung Cheng,](https://reader033.vdocuments.mx/reader033/viewer/2022051604/6002331e2285606e082ea4fc/html5/thumbnails/8.jpg)
How Policing Works
Policer?
8
Packet leaves if enough tokens
are available
Tokens refreshed at predefined policing rate
![Page 9: policing-paper@google.com An Internet-Wide Analysis of · An Internet-Wide Analysis of Traffic Policing Tobias Flach, Pavlos Papageorge, Andreas Terzis, Luis Pedrosa, Yuchung Cheng,](https://reader033.vdocuments.mx/reader033/viewer/2022051604/6002331e2285606e082ea4fc/html5/thumbnails/9.jpg)
Policing in Action
9
Throughput allowed by
policer
![Page 10: policing-paper@google.com An Internet-Wide Analysis of · An Internet-Wide Analysis of Traffic Policing Tobias Flach, Pavlos Papageorge, Andreas Terzis, Luis Pedrosa, Yuchung Cheng,](https://reader033.vdocuments.mx/reader033/viewer/2022051604/6002331e2285606e082ea4fc/html5/thumbnails/10.jpg)
Policing in Action
10
Throughput allowed by
policer
Plus: initial bursts from
saved tokens
![Page 11: policing-paper@google.com An Internet-Wide Analysis of · An Internet-Wide Analysis of Traffic Policing Tobias Flach, Pavlos Papageorge, Andreas Terzis, Luis Pedrosa, Yuchung Cheng,](https://reader033.vdocuments.mx/reader033/viewer/2022051604/6002331e2285606e082ea4fc/html5/thumbnails/11.jpg)
Policing in Action
11
Throughput allowed by
policer
Plus: initial bursts from
saved tokens
![Page 12: policing-paper@google.com An Internet-Wide Analysis of · An Internet-Wide Analysis of Traffic Policing Tobias Flach, Pavlos Papageorge, Andreas Terzis, Luis Pedrosa, Yuchung Cheng,](https://reader033.vdocuments.mx/reader033/viewer/2022051604/6002331e2285606e082ea4fc/html5/thumbnails/12.jpg)
Policing in Action
12
Throughput allowed by
policer
Plus: initial bursts from
saved tokens
Overshooting by 1 MB
![Page 13: policing-paper@google.com An Internet-Wide Analysis of · An Internet-Wide Analysis of Traffic Policing Tobias Flach, Pavlos Papageorge, Andreas Terzis, Luis Pedrosa, Yuchung Cheng,](https://reader033.vdocuments.mx/reader033/viewer/2022051604/6002331e2285606e082ea4fc/html5/thumbnails/13.jpg)
Policing in Action
13
Throughput allowed by
policer
Plus: initial bursts from
saved tokens
Overshooting by 1 MB
Multiple retransmission
rounds
![Page 14: policing-paper@google.com An Internet-Wide Analysis of · An Internet-Wide Analysis of Traffic Policing Tobias Flach, Pavlos Papageorge, Andreas Terzis, Luis Pedrosa, Yuchung Cheng,](https://reader033.vdocuments.mx/reader033/viewer/2022051604/6002331e2285606e082ea4fc/html5/thumbnails/14.jpg)
Policing in Action
14
Throughput allowed by
policer
Plus: initial bursts from
saved tokens
Overshooting by 1 MB
Multiple retransmission
rounds
![Page 15: policing-paper@google.com An Internet-Wide Analysis of · An Internet-Wide Analysis of Traffic Policing Tobias Flach, Pavlos Papageorge, Andreas Terzis, Luis Pedrosa, Yuchung Cheng,](https://reader033.vdocuments.mx/reader033/viewer/2022051604/6002331e2285606e082ea4fc/html5/thumbnails/15.jpg)
Policing in Action
15
Throughput allowed by
policer
Plus: initial bursts from
saved tokens
Overshooting by 1 MB
Transmission rate matches policing
rateMultiple
retransmission rounds
![Page 16: policing-paper@google.com An Internet-Wide Analysis of · An Internet-Wide Analysis of Traffic Policing Tobias Flach, Pavlos Papageorge, Andreas Terzis, Luis Pedrosa, Yuchung Cheng,](https://reader033.vdocuments.mx/reader033/viewer/2022051604/6002331e2285606e082ea4fc/html5/thumbnails/16.jpg)
Policing can have negative side effects for all parties
● Content providers○ Excess load on servers forced to retransmit dropped packets
(global average: 20% retransmissions vs. 2% when not policed)
● ISPs○ Transport traffic across the Internet only for it to be dropped by the policer
○ Incurs avoidable transit costs
● Users○ Can interact badly with TCP-based applications○ We measured degraded video quality of experience (QoE) → user dissatisfaction
16
![Page 17: policing-paper@google.com An Internet-Wide Analysis of · An Internet-Wide Analysis of Traffic Policing Tobias Flach, Pavlos Papageorge, Andreas Terzis, Luis Pedrosa, Yuchung Cheng,](https://reader033.vdocuments.mx/reader033/viewer/2022051604/6002331e2285606e082ea4fc/html5/thumbnails/17.jpg)
17
Analyze theprevalence and impact of policing on a global scale
Develop a mechanism to detect
policing in packet captures
Tie connection performance back to
already collected application metrics
Collect packet traces for sampled client
connections at most Google frontends
![Page 18: policing-paper@google.com An Internet-Wide Analysis of · An Internet-Wide Analysis of Traffic Policing Tobias Flach, Pavlos Papageorge, Andreas Terzis, Luis Pedrosa, Yuchung Cheng,](https://reader033.vdocuments.mx/reader033/viewer/2022051604/6002331e2285606e082ea4fc/html5/thumbnails/18.jpg)
Collect packet traces
HTTP Response
Forward samples to analysis backend Detect policing Cross-reference with
application metrics
Analysis Pipeline
18
Application metrics
![Page 19: policing-paper@google.com An Internet-Wide Analysis of · An Internet-Wide Analysis of Traffic Policing Tobias Flach, Pavlos Papageorge, Andreas Terzis, Luis Pedrosa, Yuchung Cheng,](https://reader033.vdocuments.mx/reader033/viewer/2022051604/6002331e2285606e082ea4fc/html5/thumbnails/19.jpg)
Progress
Time
Packets droppedby policer
Packets passthrough policer
Detection Algorithm Policing rate
19
Packets are always dropped when crossing the “policing rate” line
![Page 20: policing-paper@google.com An Internet-Wide Analysis of · An Internet-Wide Analysis of Traffic Policing Tobias Flach, Pavlos Papageorge, Andreas Terzis, Luis Pedrosa, Yuchung Cheng,](https://reader033.vdocuments.mx/reader033/viewer/2022051604/6002331e2285606e082ea4fc/html5/thumbnails/20.jpg)
Progress
Time
Policing rate
Detection Algorithm
20
1 Find thepolicing rate
● Use measured throughput between an early and late loss as estimate
2Match performance to expected policing
behavior
● Everything above the policing rate gets dropped
● (Almost) nothing below the policing rate gets dropped
![Page 21: policing-paper@google.com An Internet-Wide Analysis of · An Internet-Wide Analysis of Traffic Policing Tobias Flach, Pavlos Papageorge, Andreas Terzis, Luis Pedrosa, Yuchung Cheng,](https://reader033.vdocuments.mx/reader033/viewer/2022051604/6002331e2285606e082ea4fc/html5/thumbnails/21.jpg)
Progress
Time
But: Traffic below policing rate should go through
But: Traffic above policing rate should be dropped
Progress
Time
Avoiding Falsely Labeling Loss as Policing
21
![Page 22: policing-paper@google.com An Internet-Wide Analysis of · An Internet-Wide Analysis of Traffic Policing Tobias Flach, Pavlos Papageorge, Andreas Terzis, Luis Pedrosa, Yuchung Cheng,](https://reader033.vdocuments.mx/reader033/viewer/2022051604/6002331e2285606e082ea4fc/html5/thumbnails/22.jpg)
Progress
Time
Packets are usually dropped when a router’s
buffer is already full
Use inflated latency as signal that loss is not caused by a policer
Latency
Congestion Looks Similar to Policing!
22
Buffer fills → queuing delay increases
![Page 23: policing-paper@google.com An Internet-Wide Analysis of · An Internet-Wide Analysis of Traffic Policing Tobias Flach, Pavlos Papageorge, Andreas Terzis, Luis Pedrosa, Yuchung Cheng,](https://reader033.vdocuments.mx/reader033/viewer/2022051604/6002331e2285606e082ea4fc/html5/thumbnails/23.jpg)
Validation 1: Lab Setting
● Goal: Approximate the accuracy of our heuristic● Generated test traces covering common reasons for dropped packets
○ Policing (used a router with support for policing)○ Congestion○ Random loss○ Shaping
● High accuracy for almost all configurations (see paper for details)○ Policing: 93%○ All other reasons for loss: > 99%
23
![Page 24: policing-paper@google.com An Internet-Wide Analysis of · An Internet-Wide Analysis of Traffic Policing Tobias Flach, Pavlos Papageorge, Andreas Terzis, Luis Pedrosa, Yuchung Cheng,](https://reader033.vdocuments.mx/reader033/viewer/2022051604/6002331e2285606e082ea4fc/html5/thumbnails/24.jpg)
Validation 2: Live Traffic● Observed only few policing rates in
ISP deep dives○ ISPs enforce a limited set of data plans
24
● Confirmed that per ISP policing rates cluster around a few valuesacross the whole dataset
● And: Observed no consistency across flows without policing
![Page 25: policing-paper@google.com An Internet-Wide Analysis of · An Internet-Wide Analysis of Traffic Policing Tobias Flach, Pavlos Papageorge, Andreas Terzis, Luis Pedrosa, Yuchung Cheng,](https://reader033.vdocuments.mx/reader033/viewer/2022051604/6002331e2285606e082ea4fc/html5/thumbnails/25.jpg)
Outline
1. How Policing Works
2. Detecting the Effects of Policing in Packet Captures
3. A Global-Scale Analysis of Policing in the Internet
4. Mitigating the Impact of Policers
25
![Page 26: policing-paper@google.com An Internet-Wide Analysis of · An Internet-Wide Analysis of Traffic Policing Tobias Flach, Pavlos Papageorge, Andreas Terzis, Luis Pedrosa, Yuchung Cheng,](https://reader033.vdocuments.mx/reader033/viewer/2022051604/6002331e2285606e082ea4fc/html5/thumbnails/26.jpg)
Internet-Wide Analysis of Policing
● Sampled flows collected from most of Google’s CDN servers○ 7-day sampling period (in September 2015)○ 277 billion TCP packets○ 270 TB of data○ 800 million HTTP queries○ Clients in over 28,400 ASes
● To tie TCP performance to application performance, we analyzed flows at HTTP request/response (“segment”) granularity
26
![Page 27: policing-paper@google.com An Internet-Wide Analysis of · An Internet-Wide Analysis of Traffic Policing Tobias Flach, Pavlos Papageorge, Andreas Terzis, Luis Pedrosa, Yuchung Cheng,](https://reader033.vdocuments.mx/reader033/viewer/2022051604/6002331e2285606e082ea4fc/html5/thumbnails/27.jpg)
#1: Prevalence of PolicingRegion Policed
segments (overall)
Policed (among lossy)
Loss(policed)
Loss (non-policed)
Africa 1.3% 6.2% 27.5% 4.1%
Asia 1.3% 6.6% 24.9% 2.9%
Australia 0.4% 2.0% 21.0% 1.8%
Europe 0.7% 5.0% 20.4% 1.3%
N. America 0.2% 2.6% 22.5% 1.0%
S. America 0.7% 4.1% 22.8% 2.3%
27
![Page 28: policing-paper@google.com An Internet-Wide Analysis of · An Internet-Wide Analysis of Traffic Policing Tobias Flach, Pavlos Papageorge, Andreas Terzis, Luis Pedrosa, Yuchung Cheng,](https://reader033.vdocuments.mx/reader033/viewer/2022051604/6002331e2285606e082ea4fc/html5/thumbnails/28.jpg)
#1: Prevalence of PolicingRegion Policed
segments (overall)
Policed (among lossy)
Loss(policed)
Loss (non-policed)
Africa 1.3% 6.2% 27.5% 4.1%
Asia 1.3% 6.6% 24.9% 2.9%
Australia 0.4% 2.0% 21.0% 1.8%
Europe 0.7% 5.0% 20.4% 1.3%
N. America 0.2% 2.6% 22.5% 1.0%
S. America 0.7% 4.1% 22.8% 2.3%
Up to 7% of lossy segmentsare policed
28
Lossy:15 losses or more
per segment
![Page 29: policing-paper@google.com An Internet-Wide Analysis of · An Internet-Wide Analysis of Traffic Policing Tobias Flach, Pavlos Papageorge, Andreas Terzis, Luis Pedrosa, Yuchung Cheng,](https://reader033.vdocuments.mx/reader033/viewer/2022051604/6002331e2285606e082ea4fc/html5/thumbnails/29.jpg)
#2: Policer-induced LossesRegion Policed
segments (overall)
Policed (among lossy)
Loss(policed)
Loss (non-policed)
Africa 1.3% 6.2% 27.5% 4.1%
Asia 1.3% 6.6% 24.9% 2.9%
Australia 0.4% 2.0% 21.0% 1.8%
Europe 0.7% 5.0% 20.4% 1.3%
N. America 0.2% 2.6% 22.5% 1.0%
S. America 0.7% 4.1% 22.8% 2.3%
Up to 7% of lossy segmentsare policed
Average loss rate increases from 2% to over 20% when policed
Lossy:15 losses or more
per segment
29
![Page 30: policing-paper@google.com An Internet-Wide Analysis of · An Internet-Wide Analysis of Traffic Policing Tobias Flach, Pavlos Papageorge, Andreas Terzis, Luis Pedrosa, Yuchung Cheng,](https://reader033.vdocuments.mx/reader033/viewer/2022051604/6002331e2285606e082ea4fc/html5/thumbnails/30.jpg)
30
Progress
Time
Sudden Bandwidth Change Induces Heavy Loss
![Page 31: policing-paper@google.com An Internet-Wide Analysis of · An Internet-Wide Analysis of Traffic Policing Tobias Flach, Pavlos Papageorge, Andreas Terzis, Luis Pedrosa, Yuchung Cheng,](https://reader033.vdocuments.mx/reader033/viewer/2022051604/6002331e2285606e082ea4fc/html5/thumbnails/31.jpg)
Sudden Bandwidth Change Induces Heavy Loss
31
Progress
Time
Burst throughput
Policing rate
Sudden change in bandwidth
TCP does not adjust to large changes quickly enough
![Page 32: policing-paper@google.com An Internet-Wide Analysis of · An Internet-Wide Analysis of Traffic Policing Tobias Flach, Pavlos Papageorge, Andreas Terzis, Luis Pedrosa, Yuchung Cheng,](https://reader033.vdocuments.mx/reader033/viewer/2022051604/6002331e2285606e082ea4fc/html5/thumbnails/32.jpg)
#3: Burst Throughput vs. Policing Rate
Up to 7% of lossy segmentsare policed
Average loss rate increases from 2% to over 20% when policed
Policing rate often over 50% lower than burst throughput
32
90th percentile:Policing rate is 10x lower than burst throughput
![Page 33: policing-paper@google.com An Internet-Wide Analysis of · An Internet-Wide Analysis of Traffic Policing Tobias Flach, Pavlos Papageorge, Andreas Terzis, Luis Pedrosa, Yuchung Cheng,](https://reader033.vdocuments.mx/reader033/viewer/2022051604/6002331e2285606e082ea4fc/html5/thumbnails/33.jpg)
Quality of Experience Metrics
Rebuffer Time:Time that a video is paused after playback starteddue to insufficient stream data buffered
Watch Time:Fraction of the video watched by the user
Rebuffer to Watch Time Ratio:Goal is zero (no rebuffering delays after playback started).
33
![Page 34: policing-paper@google.com An Internet-Wide Analysis of · An Internet-Wide Analysis of Traffic Policing Tobias Flach, Pavlos Papageorge, Andreas Terzis, Luis Pedrosa, Yuchung Cheng,](https://reader033.vdocuments.mx/reader033/viewer/2022051604/6002331e2285606e082ea4fc/html5/thumbnails/34.jpg)
#4: Impact on Quality of Experience
Up to 7% of lossy segmentsare policed
Average loss rate increases from 2% to over 20% when policed
Policing rate often over 50% lower than burst throughput
In the tail, policed segmentscan have up to 200% higher
rebuffering times
34
(For playbacks with the same throughput)
![Page 35: policing-paper@google.com An Internet-Wide Analysis of · An Internet-Wide Analysis of Traffic Policing Tobias Flach, Pavlos Papageorge, Andreas Terzis, Luis Pedrosa, Yuchung Cheng,](https://reader033.vdocuments.mx/reader033/viewer/2022051604/6002331e2285606e082ea4fc/html5/thumbnails/35.jpg)
Mitigating Policer Impact
For content providers For policing ISPs
No access to policers andtheir configurations
But can control transmission patterns to minimize risk of hitting
an empty token bucket
Access to policers andtheir configurations
Can deploy alternative traffic management techniques
35
![Page 36: policing-paper@google.com An Internet-Wide Analysis of · An Internet-Wide Analysis of Traffic Policing Tobias Flach, Pavlos Papageorge, Andreas Terzis, Luis Pedrosa, Yuchung Cheng,](https://reader033.vdocuments.mx/reader033/viewer/2022051604/6002331e2285606e082ea4fc/html5/thumbnails/36.jpg)
Mitigating Policer Impact
For content providers For policing ISPs
Rate limiting
Pacing
Policer optimization
Shaping
36
Reducing lossesduring recovery in Linux
![Page 37: policing-paper@google.com An Internet-Wide Analysis of · An Internet-Wide Analysis of Traffic Policing Tobias Flach, Pavlos Papageorge, Andreas Terzis, Luis Pedrosa, Yuchung Cheng,](https://reader033.vdocuments.mx/reader033/viewer/2022051604/6002331e2285606e082ea4fc/html5/thumbnails/37.jpg)
Mitigating Policer Impact
37
Reducing lossesduring recovery in Linux
For content providers For policing ISPs
![Page 38: policing-paper@google.com An Internet-Wide Analysis of · An Internet-Wide Analysis of Traffic Policing Tobias Flach, Pavlos Papageorge, Andreas Terzis, Luis Pedrosa, Yuchung Cheng,](https://reader033.vdocuments.mx/reader033/viewer/2022051604/6002331e2285606e082ea4fc/html5/thumbnails/38.jpg)
Slow start during recovery
Policer
Sender transmits at twice the policing rate
Reducing Losses During Recovery in LinuxSolution:Packet conservation until ACKs indicate no further losses
● Reduces median loss rates by 10 to 20%
● Upstreamed to Linux kernel 4.2
38
Policer
Round trips(one per column)
Policer
Packets leave at
policing rate
Send only one packet per ACK
![Page 39: policing-paper@google.com An Internet-Wide Analysis of · An Internet-Wide Analysis of Traffic Policing Tobias Flach, Pavlos Papageorge, Andreas Terzis, Luis Pedrosa, Yuchung Cheng,](https://reader033.vdocuments.mx/reader033/viewer/2022051604/6002331e2285606e082ea4fc/html5/thumbnails/39.jpg)
● ISPs need ways to deal with increasing traffic demands and want to enforce plans → traffic policing is one option
● On a global scale up to 7% of lossy segments are affected by traffic policing● Policed connections see ...
○ Much higher loss rates○ Long recovery times when policers allow initial bursts○ Worse video rebuffering times (QoE)
● Negative effects can be mitigated○ Content providers: Rate limiting, pacing, prevention of loss during recovery○ ISPs: Better policing configurations, shaping
Conclusion● ISPs need ways to deal with increasing traffic demands and want to enforce
plans → traffic policing is one option● On a global scale up to 7% of lossy segments are affected by traffic policing
39
Questions? Email us: [email protected]: http://usc-nsl.github.io/policing-detection/