ac dc tcp: virtual congestion control enforcement for...
TRANSCRIPT
![Page 1: AC DC TCP: Virtual Congestion Control Enforcement for ...conferences.sigcomm.org/sigcomm/2016/files/program/sigcomm/Se… · Datacenter Network Congestion Control • Congestion is](https://reader033.vdocuments.mx/reader033/viewer/2022051915/6006cc3c5e2bbd17826c1f66/html5/thumbnails/1.jpg)
AC⚡DCTCP:VirtualCongestionControlEnforcementforDatacenterNetworks
Keqiang He,EricRozner,Kanak Agarwal,YuGu,WesFelter,JohnCarter,AdityaAkella
1
![Page 2: AC DC TCP: Virtual Congestion Control Enforcement for ...conferences.sigcomm.org/sigcomm/2016/files/program/sigcomm/Se… · Datacenter Network Congestion Control • Congestion is](https://reader033.vdocuments.mx/reader033/viewer/2022051915/6006cc3c5e2bbd17826c1f66/html5/thumbnails/2.jpg)
DatacenterNetworkCongestionControl
• Congestionisnotrareindatacenternetworks[Singh,SIGCOMM’15]
• Taillatencyishuge• 99.9th-tilelatencyisordersofmagnitudehigherthanthemedian[Mogul,HotOS’15]• Queueinglatencyisthemajorcontributor[Jang,SIGCOMM’15]
• NewdatacenterTCPcongestioncontrolschemeshavebeenproposed• E.g.,DCTCP,TIMELY,DCQCN,TCP-Bolt,ICTCP,etc
2
![Page 3: AC DC TCP: Virtual Congestion Control Enforcement for ...conferences.sigcomm.org/sigcomm/2016/files/program/sigcomm/Se… · Datacenter Network Congestion Control • Congestion is](https://reader033.vdocuments.mx/reader033/viewer/2022051915/6006cc3c5e2bbd17826c1f66/html5/thumbnails/3.jpg)
But,WeCanNotControlVMTCPStacks
• Inmulti-tenantdatacenters,adminscannotcontrolVMTCPstacks• BecauseVMsaresetupandmanagedbydifferententities
3
Virtualization
Servers
Storage
Networking
Tenant1VM
Tenant2VM
Tenant3VM
Infrastructure
TCP/IPstack TCP/IPstack TCP/IPstackTherefore,outdated,inefficient,ormisconfiguredTCPstackscanbeimplementedintheVMs.
Thisleadsto2mainproblems.
![Page 4: AC DC TCP: Virtual Congestion Control Enforcement for ...conferences.sigcomm.org/sigcomm/2016/files/program/sigcomm/Se… · Datacenter Network Congestion Control • Congestion is](https://reader033.vdocuments.mx/reader033/viewer/2022051915/6006cc3c5e2bbd17826c1f66/html5/thumbnails/4.jpg)
Problem#1:LargeQueueingLatency
4
switchqueue
sender receiver
Noqueueinglatency,TCPRTTisaround60to200microsecondsTCPRTTcanreachtensofmillisecondsbecauseofpacketqueueing.
PPPPPPPP
![Page 5: AC DC TCP: Virtual Congestion Control Enforcement for ...conferences.sigcomm.org/sigcomm/2016/files/program/sigcomm/Se… · Datacenter Network Congestion Control • Congestion is](https://reader033.vdocuments.mx/reader033/viewer/2022051915/6006cc3c5e2bbd17826c1f66/html5/thumbnails/5.jpg)
Problem#2:TCPUnfairness
• ECNandnon-ECNcoexistenceproblem[Judd,NSDI’15]• Non-ECN:e.g.,CUBIC• ECN:e.g.,DCTCP
5
![Page 6: AC DC TCP: Virtual Congestion Control Enforcement for ...conferences.sigcomm.org/sigcomm/2016/files/program/sigcomm/Se… · Datacenter Network Congestion Control • Congestion is](https://reader033.vdocuments.mx/reader033/viewer/2022051915/6006cc3c5e2bbd17826c1f66/html5/thumbnails/6.jpg)
Problem#2:TCPUnfairness(cont.)
• Differentcongestioncontrolalgorithmsleadtounfairness
65flowswithdifferentCCalgorithmscongesta10GlinkDumbbelltopology
send
ers
receivers
CC:CongestionControl
![Page 7: AC DC TCP: Virtual Congestion Control Enforcement for ...conferences.sigcomm.org/sigcomm/2016/files/program/sigcomm/Se… · Datacenter Network Congestion Control • Congestion is](https://reader033.vdocuments.mx/reader033/viewer/2022051915/6006cc3c5e2bbd17826c1f66/html5/thumbnails/7.jpg)
AC!V⚡DCTCP:AdministratorControloverDataCenterTCP
7
ImplementsTCPcongestioncontrolintheVirtualSwitch
EnsuresVMTCPstackscannotimpactthenetwork
![Page 8: AC DC TCP: Virtual Congestion Control Enforcement for ...conferences.sigcomm.org/sigcomm/2016/files/program/sigcomm/Se… · Datacenter Network Congestion Control • Congestion is](https://reader033.vdocuments.mx/reader033/viewer/2022051915/6006cc3c5e2bbd17826c1f66/html5/thumbnails/8.jpg)
AC⚡DC:HighLevelView
8
OS OS OS
AppsApps Apps
ControlplaneDatapath(AC/DC)
vNIC vNIC vNIC
DatacenterNetwork
vSwitch
VirtualMachinesAC/DC(sender)
AC/DC(receiver)
Unifo
rmper-flow
CC
Per-flowCCfeed
back
Server
Casestudy:DCTCPCCinthevSwitch
![Page 9: AC DC TCP: Virtual Congestion Control Enforcement for ...conferences.sigcomm.org/sigcomm/2016/files/program/sigcomm/Se… · Datacenter Network Congestion Control • Congestion is](https://reader033.vdocuments.mx/reader033/viewer/2022051915/6006cc3c5e2bbd17826c1f66/html5/thumbnails/9.jpg)
AC⚡DCBenefits• NomodificationstoVMsorhardware
• Lowlatencyprovidedbystate-of-the-artCCalgorithms
• ImprovedTCPfairness andsupportbothECNandnon-ECNflows
• Enforceper-flowdifferentiationviacongestioncontrol,e.g.,• East-westandnorth-southflowscanusedifferentCCs(webserver)• Givehigherpriorityto“mission-critical”traffic(backendVM)
9
![Page 10: AC DC TCP: Virtual Congestion Control Enforcement for ...conferences.sigcomm.org/sigcomm/2016/files/program/sigcomm/Se… · Datacenter Network Congestion Control • Congestion is](https://reader033.vdocuments.mx/reader033/viewer/2022051915/6006cc3c5e2bbd17826c1f66/html5/thumbnails/10.jpg)
AC⚡DCDesign• ObtainingCongestionControlState
• DCTCPCongestionControlinthevSwitch
• EnforcingCongestionControl
• Per-flowDifferentiationviaCongestionControl
10
![Page 11: AC DC TCP: Virtual Congestion Control Enforcement for ...conferences.sigcomm.org/sigcomm/2016/files/program/sigcomm/Se… · Datacenter Network Congestion Control • Congestion is](https://reader033.vdocuments.mx/reader033/viewer/2022051915/6006cc3c5e2bbd17826c1f66/html5/thumbnails/11.jpg)
ObtainingCongestionControlState
• Per-flowconnectiontracking• Alltrafficgoesthroughthevirtualswitch• WecanreconstructCCviamonitoringallthepacketsofaconnection
• Maintainper-flowcongestioncontrolvariables• E.g.,CC-relatedsequencenumbers,dupack counteretc
11
PacketFlow
classificationUpdatingCCvariables
![Page 12: AC DC TCP: Virtual Congestion Control Enforcement for ...conferences.sigcomm.org/sigcomm/2016/files/program/sigcomm/Se… · Datacenter Network Congestion Control • Congestion is](https://reader033.vdocuments.mx/reader033/viewer/2022051915/6006cc3c5e2bbd17826c1f66/html5/thumbnails/12.jpg)
DCTCPCongestionControlinthevSwitch
• UniversalECNmarking
• GetECNfeedback
12
![Page 13: AC DC TCP: Virtual Congestion Control Enforcement for ...conferences.sigcomm.org/sigcomm/2016/files/program/sigcomm/Se… · Datacenter Network Congestion Control • Congestion is](https://reader033.vdocuments.mx/reader033/viewer/2022051915/6006cc3c5e2bbd17826c1f66/html5/thumbnails/13.jpg)
UniversalECNMarking
• Why?• NotallVMsrunECN-CapableTransports(ECT)likeDCTCP
• UniversalECNMarking• AllpacketsenteringthefabricshouldbeECN-markedbythevirtualswitch• SolvestheECNandnon-ECNcoexistenceproblem
13
![Page 14: AC DC TCP: Virtual Congestion Control Enforcement for ...conferences.sigcomm.org/sigcomm/2016/files/program/sigcomm/Se… · Datacenter Network Congestion Control • Congestion is](https://reader033.vdocuments.mx/reader033/viewer/2022051915/6006cc3c5e2bbd17826c1f66/html5/thumbnails/14.jpg)
GetECNFeedback
14
congestedswitch
Needawaytocarrythecongestioninformationback.
CongestionExperienced(CE)marked
Senderside
Receiverside
![Page 15: AC DC TCP: Virtual Congestion Control Enforcement for ...conferences.sigcomm.org/sigcomm/2016/files/program/sigcomm/Se… · Datacenter Network Congestion Control • Congestion is](https://reader033.vdocuments.mx/reader033/viewer/2022051915/6006cc3c5e2bbd17826c1f66/html5/thumbnails/15.jpg)
GetECNFeedback
15
congestedswitch
AC/DCAC/DC
sender receiver
Congestionfeedbackisencodedas8bytes:{ECN_bytes,Total_bytes}.
PiggybackedonanexistingTCPACK(PACK).
CongestionExperienced(CE)marked
![Page 16: AC DC TCP: Virtual Congestion Control Enforcement for ...conferences.sigcomm.org/sigcomm/2016/files/program/sigcomm/Se… · Datacenter Network Congestion Control • Congestion is](https://reader033.vdocuments.mx/reader033/viewer/2022051915/6006cc3c5e2bbd17826c1f66/html5/thumbnails/16.jpg)
DCTCPCongestionControlinthevSwitch
16
ExtractCCinfoifitisPACK;
IncomingACK
Updateconnectiontrackingvariables;Update⍺ onceeveryRTT;
Congestion?
tcp_cong_avoid();
No
Loss?Yes
⍺=max_alpha;
Yes
No
wnd=wnd*(1- ⍺/2);
AC/DCenforcesCContheflow;SendACKtoVM;
Cutwnd inlastRTT?
Yes
No
DCTCPCongestionControlLaw
![Page 17: AC DC TCP: Virtual Congestion Control Enforcement for ...conferences.sigcomm.org/sigcomm/2016/files/program/sigcomm/Se… · Datacenter Network Congestion Control • Congestion is](https://reader033.vdocuments.mx/reader033/viewer/2022051915/6006cc3c5e2bbd17826c1f66/html5/thumbnails/17.jpg)
EnforcingCongestionControl
• TCPsendsmin(CWND,RWND)• CWNDiscongestioncontrolwindow(congestioncontrol)• RWNDisreceiver’sadvertisedwindow(flowcontrol)
• AC⚡DCreusesRWNDforcongestioncontrolpurpose• VMswithunalteredTCPstackswillnaturallyfollowourenforcement
• Non-conformingflowscanbepolicedbydroppinganyexcesspacketsnotallowedbythecalculatedcongestionwindow• Losshastoberecoverede2e,thisincentivizestenantstorespectstandards
17
![Page 18: AC DC TCP: Virtual Congestion Control Enforcement for ...conferences.sigcomm.org/sigcomm/2016/files/program/sigcomm/Se… · Datacenter Network Congestion Control • Congestion is](https://reader033.vdocuments.mx/reader033/viewer/2022051915/6006cc3c5e2bbd17826c1f66/html5/thumbnails/18.jpg)
ControlLawforPer-flowDifferentiation
18
𝑅𝑊𝑁𝐷 = 𝑅𝑊𝑁𝐷 ∗ (1 −𝛼2)
𝑅𝑊𝑁𝐷 = 𝑅𝑊𝑁𝐷 ∗ (1 − (𝛼 −𝛼𝛽2 ))
When𝛽 iscloseto1,itbecomesDCTCP.When𝛽 iscloseto0,itbacks-offaggressively.
Larger𝛽 forhigherprioritytraffic.
DCTCP:
AC⚡DCTCP:
![Page 19: AC DC TCP: Virtual Congestion Control Enforcement for ...conferences.sigcomm.org/sigcomm/2016/files/program/sigcomm/Se… · Datacenter Network Congestion Control • Congestion is](https://reader033.vdocuments.mx/reader033/viewer/2022051915/6006cc3c5e2bbd17826c1f66/html5/thumbnails/19.jpg)
Implementation• PrototypeimplementationinOpenvSwitchkerneldatapath• ~1200LoCadded
• Ourdesignleveragesavailabletechniquestoimproveperformance• RCU-enabledhashtablestoperformconnectiontracking• AC⚡DCmanipulatesTCPsegments,insteadofMTU-sizedpackets• AC⚡DCleveragesNICchecksummingsotheTCPchecksumdoesnothavetoberecomputedafterheaderfieldsaremodified
19NIC
HypervisorTCP/IP
AC⚡DC
VM1Stack
VM2Stack
ManipulatesTCPsegments
NICrecalculatesTCPchecksum
TCPsegment
P P P P
TSO
![Page 20: AC DC TCP: Virtual Congestion Control Enforcement for ...conferences.sigcomm.org/sigcomm/2016/files/program/sigcomm/Se… · Datacenter Network Congestion Control • Congestion is](https://reader033.vdocuments.mx/reader033/viewer/2022051915/6006cc3c5e2bbd17826c1f66/html5/thumbnails/20.jpg)
Evaluation
• Testbed:17servers(6-core,60GBmemory),6 10Gbpsswitches• Microbenchmark topologies
20send
ers
receiver
Incast topologyDumbbelltopology
send
ers
receivers
![Page 21: AC DC TCP: Virtual Congestion Control Enforcement for ...conferences.sigcomm.org/sigcomm/2016/files/program/sigcomm/Se… · Datacenter Network Congestion Control • Congestion is](https://reader033.vdocuments.mx/reader033/viewer/2022051915/6006cc3c5e2bbd17826c1f66/html5/thumbnails/21.jpg)
Evaluation• Macrobechmark topology
• Metrics:TCPRTT,lossrate,FlowCompletionTime(FCT)21
17serversattachedtoa10Gswitch.
![Page 22: AC DC TCP: Virtual Congestion Control Enforcement for ...conferences.sigcomm.org/sigcomm/2016/files/program/sigcomm/Se… · Datacenter Network Congestion Control • Congestion is](https://reader033.vdocuments.mx/reader033/viewer/2022051915/6006cc3c5e2bbd17826c1f66/html5/thumbnails/22.jpg)
ExperimentSetting(compared3schemes)• CUBIC• CUBICstackontopofstandardOVS
• DCTCP• DCTCPstackontopofstandardOVS
• AC⚡DC• CUBIC/Reno/Vegas/HighSpeed/IllinoisstacksontopofAC⚡DC
22
VM VM VM
CUBIC DCTCP Any
CUBIC DCTCP AC⚡DC
OVS OVS AC⚡DC
VMs
Hypervisor
![Page 23: AC DC TCP: Virtual Congestion Control Enforcement for ...conferences.sigcomm.org/sigcomm/2016/files/program/sigcomm/Se… · Datacenter Network Congestion Control • Congestion is](https://reader033.vdocuments.mx/reader033/viewer/2022051915/6006cc3c5e2bbd17826c1f66/html5/thumbnails/23.jpg)
TrackingWindowSize
23
RunningDCTCPstackontopofAC⚡DC,onlyoutputscalculatedRWNDwithoutenforcement.AC⚡DCcloselytracksthewindowsizeofDCTCP.
Dumbbelltopology
send
ers
receivers
![Page 24: AC DC TCP: Virtual Congestion Control Enforcement for ...conferences.sigcomm.org/sigcomm/2016/files/program/sigcomm/Se… · Datacenter Network Congestion Control • Congestion is](https://reader033.vdocuments.mx/reader033/viewer/2022051915/6006cc3c5e2bbd17826c1f66/html5/thumbnails/24.jpg)
Convergence
24
CUBIC DCTCP AC/DC
AC/DChascomparableconvergencepropertiesasDCTCPandisbetterthanCUBIC.
Dumbbelltopology
send
ers
receivers
![Page 25: AC DC TCP: Virtual Congestion Control Enforcement for ...conferences.sigcomm.org/sigcomm/2016/files/program/sigcomm/Se… · Datacenter Network Congestion Control • Congestion is](https://reader033.vdocuments.mx/reader033/viewer/2022051915/6006cc3c5e2bbd17826c1f66/html5/thumbnails/25.jpg)
AC⚡DCimprovesfairnesswhenVMsusedifferentCCs
25StandardOVS AC⚡DC
Dumbbelltopology
send
ers
receivers
![Page 26: AC DC TCP: Virtual Congestion Control Enforcement for ...conferences.sigcomm.org/sigcomm/2016/files/program/sigcomm/Se… · Datacenter Network Congestion Control • Congestion is](https://reader033.vdocuments.mx/reader033/viewer/2022051915/6006cc3c5e2bbd17826c1f66/html5/thumbnails/26.jpg)
Overhead(CPUandMemory)
26
Senderside
Lessthan1%additionalCPUoverheadcomparedwiththebaseline.Eachconnectionuses320bytestomaintainCCvariables(10kconnectionsuse3.2MB).
Dumbbelltopology
send
ers
receivers
![Page 27: AC DC TCP: Virtual Congestion Control Enforcement for ...conferences.sigcomm.org/sigcomm/2016/files/program/sigcomm/Se… · Datacenter Network Congestion Control • Congestion is](https://reader033.vdocuments.mx/reader033/viewer/2022051915/6006cc3c5e2bbd17826c1f66/html5/thumbnails/27.jpg)
TCPIncast RTT&droprate
27
50th percentileRTT 99.9th percentileRTT Packetdroprate
AC⚡DCtrackstheperformanceofDCTCPclosely.
send
ers
receiver
Incast topology
![Page 28: AC DC TCP: Virtual Congestion Control Enforcement for ...conferences.sigcomm.org/sigcomm/2016/files/program/sigcomm/Se… · Datacenter Network Congestion Control • Congestion is](https://reader033.vdocuments.mx/reader033/viewer/2022051915/6006cc3c5e2bbd17826c1f66/html5/thumbnails/28.jpg)
Flowcompletiontimewithtrace-drivenworkloads
28
Web-searchingworkload(DCTCP) Data-miningworkload(CONGA)
AC⚡DCobtainssameperformanceasDCTCP.AC⚡DCcanreduceFCTby36%-76%comparedwithdefaultCUBIC.
17serversattachedtoa10Gswitch.
![Page 29: AC DC TCP: Virtual Congestion Control Enforcement for ...conferences.sigcomm.org/sigcomm/2016/files/program/sigcomm/Se… · Datacenter Network Congestion Control • Congestion is](https://reader033.vdocuments.mx/reader033/viewer/2022051915/6006cc3c5e2bbd17826c1f66/html5/thumbnails/29.jpg)
Summary
• AC⚡DCallowsadministratorstoregaincontroloverarbitrarytenantTCPstacksbyenforcingcongestioncontrolinthevirtualswitch
• AC⚡DCrequiresnochangestoVMsornetworkhardware
• AC⚡DCisscalable,light-weight(<1%CPUoverhead)andflexible
29
![Page 30: AC DC TCP: Virtual Congestion Control Enforcement for ...conferences.sigcomm.org/sigcomm/2016/files/program/sigcomm/Se… · Datacenter Network Congestion Control • Congestion is](https://reader033.vdocuments.mx/reader033/viewer/2022051915/6006cc3c5e2bbd17826c1f66/html5/thumbnails/30.jpg)
Thanks!
30
![Page 31: AC DC TCP: Virtual Congestion Control Enforcement for ...conferences.sigcomm.org/sigcomm/2016/files/program/sigcomm/Se… · Datacenter Network Congestion Control • Congestion is](https://reader033.vdocuments.mx/reader033/viewer/2022051915/6006cc3c5e2bbd17826c1f66/html5/thumbnails/31.jpg)
BackupSlides
31
![Page 32: AC DC TCP: Virtual Congestion Control Enforcement for ...conferences.sigcomm.org/sigcomm/2016/files/program/sigcomm/Se… · Datacenter Network Congestion Control • Congestion is](https://reader033.vdocuments.mx/reader033/viewer/2022051915/6006cc3c5e2bbd17826c1f66/html5/thumbnails/32.jpg)
RelatedWork
• DCTCP• ECN-basedcongestioncontrolforDCNs
• TIMELY• Latency-basedcongestioncontrolforDCNs• AccuratelatencymeasurementprovidedbyaccurateNICtimestamps
• vCC• vCCandAC⚡DCarecloselyrelatedworksbytwoindependentteamsJ
32
![Page 33: AC DC TCP: Virtual Congestion Control Enforcement for ...conferences.sigcomm.org/sigcomm/2016/files/program/sigcomm/Se… · Datacenter Network Congestion Control • Congestion is](https://reader033.vdocuments.mx/reader033/viewer/2022051915/6006cc3c5e2bbd17826c1f66/html5/thumbnails/33.jpg)
ECNandnon-ECNCoexistence
33
SwitchconfiguredwithWRED/ECN
ECN
Non-ECN
Whenqueueoccupancyislargerthanmarkingthreshold,non-ECNpacketsaredropped
![Page 34: AC DC TCP: Virtual Congestion Control Enforcement for ...conferences.sigcomm.org/sigcomm/2016/files/program/sigcomm/Se… · Datacenter Network Congestion Control • Congestion is](https://reader033.vdocuments.mx/reader033/viewer/2022051915/6006cc3c5e2bbd17826c1f66/html5/thumbnails/34.jpg)
IPSec
• AC⚡DCisnotabletoinspecttheTCPheadersforIPSectraffic
• Mayperformapproximatingratelimitingbasedoncongestionfeedbackinformation.
34