internet-scale virtual networking - nanog archive · mobility recap •data-plane driven cache...
TRANSCRIPT
![Page 1: Internet-scale Virtual Networking - NANOG Archive · Mobility recap •Data-plane driven cache invalidation •ILA routers provide fallback on cache invalidation Deployment](https://reader035.vdocuments.mx/reader035/viewer/2022070902/5f5ae0c9da58dc08e00b35c1/html5/thumbnails/1.jpg)
Internet-scale Virtual Networking
Petr LapukhovNetwork Engineer
Using Identifier-Locator Addressing
![Page 2: Internet-scale Virtual Networking - NANOG Archive · Mobility recap •Data-plane driven cache invalidation •ILA routers provide fallback on cache invalidation Deployment](https://reader035.vdocuments.mx/reader035/viewer/2022070902/5f5ae0c9da58dc08e00b35c1/html5/thumbnails/2.jpg)
Virtual networking is confusing!
![Page 3: Internet-scale Virtual Networking - NANOG Archive · Mobility recap •Data-plane driven cache invalidation •ILA routers provide fallback on cache invalidation Deployment](https://reader035.vdocuments.mx/reader035/viewer/2022070902/5f5ae0c9da58dc08e00b35c1/html5/thumbnails/3.jpg)
What problem FB is trying to solve?
![Page 4: Internet-scale Virtual Networking - NANOG Archive · Mobility recap •Data-plane driven cache invalidation •ILA routers provide fallback on cache invalidation Deployment](https://reader035.vdocuments.mx/reader035/viewer/2022070902/5f5ae0c9da58dc08e00b35c1/html5/thumbnails/4.jpg)
Linux application containers
Simpler and more lightweight than
![Page 5: Internet-scale Virtual Networking - NANOG Archive · Mobility recap •Data-plane driven cache invalidation •ILA routers provide fallback on cache invalidation Deployment](https://reader035.vdocuments.mx/reader035/viewer/2022070902/5f5ae0c9da58dc08e00b35c1/html5/thumbnails/5.jpg)
Container networking: challenges•Many containers per host: address sharing•Containers can move: address would change
![Page 6: Internet-scale Virtual Networking - NANOG Archive · Mobility recap •Data-plane driven cache invalidation •ILA routers provide fallback on cache invalidation Deployment](https://reader035.vdocuments.mx/reader035/viewer/2022070902/5f5ae0c9da58dc08e00b35c1/html5/thumbnails/6.jpg)
Container networking: two goals…•IPv6 address per process•Address mobility <>
![Page 7: Internet-scale Virtual Networking - NANOG Archive · Mobility recap •Data-plane driven cache invalidation •ILA routers provide fallback on cache invalidation Deployment](https://reader035.vdocuments.mx/reader035/viewer/2022070902/5f5ae0c9da58dc08e00b35c1/html5/thumbnails/7.jpg)
Identifier Locator Addressing (ILA)
![Page 8: Internet-scale Virtual Networking - NANOG Archive · Mobility recap •Data-plane driven cache invalidation •ILA routers provide fallback on cache invalidation Deployment](https://reader035.vdocuments.mx/reader035/viewer/2022070902/5f5ae0c9da58dc08e00b35c1/html5/thumbnails/8.jpg)
Identifier / Locator split
![Page 9: Internet-scale Virtual Networking - NANOG Archive · Mobility recap •Data-plane driven cache invalidation •ILA routers provide fallback on cache invalidation Deployment](https://reader035.vdocuments.mx/reader035/viewer/2022070902/5f5ae0c9da58dc08e00b35c1/html5/thumbnails/9.jpg)
Predecessors: ILNP/GSE/8+8…
![Page 10: Internet-scale Virtual Networking - NANOG Archive · Mobility recap •Data-plane driven cache invalidation •ILA routers provide fallback on cache invalidation Deployment](https://reader035.vdocuments.mx/reader035/viewer/2022070902/5f5ae0c9da58dc08e00b35c1/html5/thumbnails/10.jpg)
64 bit: Locator 64 bit: Identifier
IPv6 Address
Used for routingImmutable name
128 bit
![Page 11: Internet-scale Virtual Networking - NANOG Archive · Mobility recap •Data-plane driven cache invalidation •ILA routers provide fallback on cache invalidation Deployment](https://reader035.vdocuments.mx/reader035/viewer/2022070902/5f5ae0c9da58dc08e00b35c1/html5/thumbnails/11.jpg)
Mobility with Locator/ID split•Every host gets /64 prefix - locator (!)•Processes migrate between machines•Identifier remains the same, locator changes •Mutable locator require transport stack modification <>
![Page 12: Internet-scale Virtual Networking - NANOG Archive · Mobility recap •Data-plane driven cache invalidation •ILA routers provide fallback on cache invalidation Deployment](https://reader035.vdocuments.mx/reader035/viewer/2022070902/5f5ae0c9da58dc08e00b35c1/html5/thumbnails/12.jpg)
ILA specifics•Hides locator changes from transport layer•Transport always sees one fixed locator (/64 prefix)•Stateless rewrites (NAT) below transport layer <>
![Page 13: Internet-scale Virtual Networking - NANOG Archive · Mobility recap •Data-plane driven cache invalidation •ILA routers provide fallback on cache invalidation Deployment](https://reader035.vdocuments.mx/reader035/viewer/2022070902/5f5ae0c9da58dc08e00b35c1/html5/thumbnails/13.jpg)
ILA Host•Every host needs a routable locator: IPv6 /64 prefix•Hosts need to maintain ILA mapping cache•Non-ILA hosts talk to ILA hosts via ILA routers <>
![Page 14: Internet-scale Virtual Networking - NANOG Archive · Mobility recap •Data-plane driven cache invalidation •ILA routers provide fallback on cache invalidation Deployment](https://reader035.vdocuments.mx/reader035/viewer/2022070902/5f5ae0c9da58dc08e00b35c1/html5/thumbnails/14.jpg)
Process 1
ILA Addressface:b00c::1234
ILA Addressface:b00c::5678
Process 2
Locatorfec0:cafe::/64
Host 1 Host 2
Locatorfec0:beef::/64
NA NA
face:b00c::
face:b00c::
fec0:cafe::1234
fec0:beef::
Before NAT
On Wire (after 1st NAT)
SIR Prefixface:b00c::/64
face:b00c::
face:b00c:: After 2nd
![Page 15: Internet-scale Virtual Networking - NANOG Archive · Mobility recap •Data-plane driven cache invalidation •ILA routers provide fallback on cache invalidation Deployment](https://reader035.vdocuments.mx/reader035/viewer/2022070902/5f5ae0c9da58dc08e00b35c1/html5/thumbnails/15.jpg)
SIR Prefix•SIR = “Standard Identifier Representation”•SIR Prefix = 64 bit “fixed-locator” seen by transport•Injected into network by all ILA Routers (anycast) <>
![Page 16: Internet-scale Virtual Networking - NANOG Archive · Mobility recap •Data-plane driven cache invalidation •ILA routers provide fallback on cache invalidation Deployment](https://reader035.vdocuments.mx/reader035/viewer/2022070902/5f5ae0c9da58dc08e00b35c1/html5/thumbnails/16.jpg)
ILA networkface:b00c::/64
Process 1face:b00c::1234
Process 3face:b00c::5678
Process 2face:b00c::abc
ILA Addresses: one “virtual” /64 subnet
Non-ILAnetworks
ILARouter
![Page 17: Internet-scale Virtual Networking - NANOG Archive · Mobility recap •Data-plane driven cache invalidation •ILA routers provide fallback on cache invalidation Deployment](https://reader035.vdocuments.mx/reader035/viewer/2022070902/5f5ae0c9da58dc08e00b35c1/html5/thumbnails/17.jpg)
ILA Router•Knows of all active mappings•Injects /64 SIR prefix into IPv6 network•“Mediates” between ILA and non-ILA hosts•May also mediate between ILA-hosts •Acts like an IPv6 router on “virtual” /64 segment <>
![Page 18: Internet-scale Virtual Networking - NANOG Archive · Mobility recap •Data-plane driven cache invalidation •ILA routers provide fallback on cache invalidation Deployment](https://reader035.vdocuments.mx/reader035/viewer/2022070902/5f5ae0c9da58dc08e00b35c1/html5/thumbnails/18.jpg)
ILA Hosts
Non-ILA Host
Injects SIR prefix
face:b00c::
face:b00c::
face:b00c::ab
Talks toILA Router
Translates
ILA Host responds directly to non-ILA
ILA Router and non-ILA hosts
IPv6
![Page 19: Internet-scale Virtual Networking - NANOG Archive · Mobility recap •Data-plane driven cache invalidation •ILA routers provide fallback on cache invalidation Deployment](https://reader035.vdocuments.mx/reader035/viewer/2022070902/5f5ae0c9da58dc08e00b35c1/html5/thumbnails/19.jpg)
ILA Hosts
Injects SIR prefix
ID:1234
ID: 5678
ILA Router
Sends redirect
Translates androutes to
Using ILA Router to b/w ILA hosts
No locator for face:b00c::1234
send to ILA router
Route using
Talks to
1234
![Page 20: Internet-scale Virtual Networking - NANOG Archive · Mobility recap •Data-plane driven cache invalidation •ILA routers provide fallback on cache invalidation Deployment](https://reader035.vdocuments.mx/reader035/viewer/2022070902/5f5ae0c9da58dc08e00b35c1/html5/thumbnails/20.jpg)
What about control plane?
![Page 21: Internet-scale Virtual Networking - NANOG Archive · Mobility recap •Data-plane driven cache invalidation •ILA routers provide fallback on cache invalidation Deployment](https://reader035.vdocuments.mx/reader035/viewer/2022070902/5f5ae0c9da58dc08e00b35c1/html5/thumbnails/21.jpg)
Goal: disseminate ILA mappings
![Page 22: Internet-scale Virtual Networking - NANOG Archive · Mobility recap •Data-plane driven cache invalidation •ILA routers provide fallback on cache invalidation Deployment](https://reader035.vdocuments.mx/reader035/viewer/2022070902/5f5ae0c9da58dc08e00b35c1/html5/thumbnails/22.jpg)
Good news: there is no standard!
![Page 23: Internet-scale Virtual Networking - NANOG Archive · Mobility recap •Data-plane driven cache invalidation •ILA routers provide fallback on cache invalidation Deployment](https://reader035.vdocuments.mx/reader035/viewer/2022070902/5f5ae0c9da58dc08e00b35c1/html5/thumbnails/23.jpg)
ILA specifics•ILA routers know of all mappings•ILA hosts always publish into mapping system <>
![Page 24: Internet-scale Virtual Networking - NANOG Archive · Mobility recap •Data-plane driven cache invalidation •ILA routers provide fallback on cache invalidation Deployment](https://reader035.vdocuments.mx/reader035/viewer/2022070902/5f5ae0c9da58dc08e00b35c1/html5/thumbnails/24.jpg)
ILA: Data-plane assistance•ILA routers may send redirect messages•Hosts may send stale mapping messages •Similar to ICMPv6 messages <>
![Page 25: Internet-scale Virtual Networking - NANOG Archive · Mobility recap •Data-plane driven cache invalidation •ILA routers provide fallback on cache invalidation Deployment](https://reader035.vdocuments.mx/reader035/viewer/2022070902/5f5ae0c9da58dc08e00b35c1/html5/thumbnails/25.jpg)
Now the fun: identifier mobility
![Page 26: Internet-scale Virtual Networking - NANOG Archive · Mobility recap •Data-plane driven cache invalidation •ILA routers provide fallback on cache invalidation Deployment](https://reader035.vdocuments.mx/reader035/viewer/2022070902/5f5ae0c9da58dc08e00b35c1/html5/thumbnails/26.jpg)
A
ILA Hosts
C
ID: 1234
ID: 5678
Translates
ILA Routers
Container moves b/w hosts
B
Invalid
Fallback to
Redirect
New Locator for 1234
Flow
Flow
Forwar
Scheduler removes
![Page 27: Internet-scale Virtual Networking - NANOG Archive · Mobility recap •Data-plane driven cache invalidation •ILA routers provide fallback on cache invalidation Deployment](https://reader035.vdocuments.mx/reader035/viewer/2022070902/5f5ae0c9da58dc08e00b35c1/html5/thumbnails/27.jpg)
Mobility recap•Data-plane driven cache invalidation•ILA routers provide fallback on cache invalidation <>
![Page 28: Internet-scale Virtual Networking - NANOG Archive · Mobility recap •Data-plane driven cache invalidation •ILA routers provide fallback on cache invalidation Deployment](https://reader035.vdocuments.mx/reader035/viewer/2022070902/5f5ae0c9da58dc08e00b35c1/html5/thumbnails/28.jpg)
Deployment @ FB
![Page 29: Internet-scale Virtual Networking - NANOG Archive · Mobility recap •Data-plane driven cache invalidation •ILA routers provide fallback on cache invalidation Deployment](https://reader035.vdocuments.mx/reader035/viewer/2022070902/5f5ae0c9da58dc08e00b35c1/html5/thumbnails/29.jpg)
Network Setup•Every server gets /64 route•Summarized to /54 on rack switch•Summarized to /46 on pod switch•Sums up to /32•Can fit 32 data-centers per /32 <>
Rack = /54
Pod = /46
Spine
DC Hierarchy
![Page 30: Internet-scale Virtual Networking - NANOG Archive · Mobility recap •Data-plane driven cache invalidation •ILA routers provide fallback on cache invalidation Deployment](https://reader035.vdocuments.mx/reader035/viewer/2022070902/5f5ae0c9da58dc08e00b35c1/html5/thumbnails/30.jpg)
Host Configuration•New /64 per host - every machine @FB•Part of host bootstrap info•Applied by Chef recipe
$ ip -6 a ls1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 inet6 ::1/128 scope host valid_lft forever preferred_lft forever2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qlen 1000 inet6 2803:6082:18e0:e825::1/64 scope global deprecated valid_lft forever preferred_lft forever inet6 2401:db00:11:d03a:face:0:25:0/64 scope global valid_lft forever preferred_lft forever inet6 fe80::f652:14ff:febe:fe54/64 scope link valid_lft forever preferred_lft forever
Locator
![Page 31: Internet-scale Virtual Networking - NANOG Archive · Mobility recap •Data-plane driven cache invalidation •ILA routers provide fallback on cache invalidation Deployment](https://reader035.vdocuments.mx/reader035/viewer/2022070902/5f5ae0c9da58dc08e00b35c1/html5/thumbnails/31.jpg)
Unique IPv6 per process!•Random 64bit ID allocated on container start•UUID64 - timestamp + host name + some magic <>
![Page 32: Internet-scale Virtual Networking - NANOG Archive · Mobility recap •Data-plane driven cache invalidation •ILA routers provide fallback on cache invalidation Deployment](https://reader035.vdocuments.mx/reader035/viewer/2022070902/5f5ae0c9da58dc08e00b35c1/html5/thumbnails/32.jpg)
How can process use IPv6?•Passed explicitly as environment variable•…Could be enforced via LD_PRELOAD•Namespaces/ipvlan currently experimental <>
![Page 33: Internet-scale Virtual Networking - NANOG Archive · Mobility recap •Data-plane driven cache invalidation •ILA routers provide fallback on cache invalidation Deployment](https://reader035.vdocuments.mx/reader035/viewer/2022070902/5f5ae0c9da58dc08e00b35c1/html5/thumbnails/33.jpg)
DNS Support•DNS name per container•E.g. ‘tsp-prn.netsystems.test-task.0.tw.local’•Both AAAA and PTR created simultaneously•ZippyDB as backing store <>
![Page 34: Internet-scale Virtual Networking - NANOG Archive · Mobility recap •Data-plane driven cache invalidation •ILA routers provide fallback on cache invalidation Deployment](https://reader035.vdocuments.mx/reader035/viewer/2022070902/5f5ae0c9da58dc08e00b35c1/html5/thumbnails/34.jpg)
Host support: Kernel 4.x+•ILA rewrites: Light-weight tunnels (LWT)•Linux route lookup + rewrite action•Programmable via netlink API <>
![Page 35: Internet-scale Virtual Networking - NANOG Archive · Mobility recap •Data-plane driven cache invalidation •ILA routers provide fallback on cache invalidation Deployment](https://reader035.vdocuments.mx/reader035/viewer/2022070902/5f5ae0c9da58dc08e00b35c1/html5/thumbnails/35.jpg)
Host support: ip route primer
# Set local SIR address ip -6 addr add face:b00c:0:0:2555:0:1:0/128 dev lo
Remote Locator
My Locator
My ID
Remote ID
SIR Prefix
# Add peer with ILA translation ip -6 route add face:b00c:0:0:2555:0:2:0/128 encap ila 2803:6080:8960:4473 via 2401:db00:20:4001::a
# Add local prefix translation ip -6 route add table local local 2803:6082:1950:401:2555:0:1:0/128 encap ila face:b00c:0:0 dev lo
modprobe ila
SIR Prefix
![Page 36: Internet-scale Virtual Networking - NANOG Archive · Mobility recap •Data-plane driven cache invalidation •ILA routers provide fallback on cache invalidation Deployment](https://reader035.vdocuments.mx/reader035/viewer/2022070902/5f5ae0c9da58dc08e00b35c1/html5/thumbnails/36.jpg)
ILA Routers @ FB•Linux machine with IPv6 forwarding enabled•Regular routing with LWT “ila” rules•Currently: all hosts are ILA routers <>
![Page 37: Internet-scale Virtual Networking - NANOG Archive · Mobility recap •Data-plane driven cache invalidation •ILA routers provide fallback on cache invalidation Deployment](https://reader035.vdocuments.mx/reader035/viewer/2022070902/5f5ae0c9da58dc08e00b35c1/html5/thumbnails/37.jpg)
Control plane hack
ZippyDB
ILA Hosts ==
Containerstarts
& address get
Publish Mapping
+ Downloadmappingsevery 5s
ZippyDBAsyncreplication
ILA cachessynchronized
![Page 38: Internet-scale Virtual Networking - NANOG Archive · Mobility recap •Data-plane driven cache invalidation •ILA routers provide fallback on cache invalidation Deployment](https://reader035.vdocuments.mx/reader035/viewer/2022070902/5f5ae0c9da58dc08e00b35c1/html5/thumbnails/38.jpg)
Control plane recap•ZippyDB to push & pull mappings•Runs on ~ 10k+ hosts•Low number of mobile tasks (100s)•Very easy to experiment with <>
![Page 39: Internet-scale Virtual Networking - NANOG Archive · Mobility recap •Data-plane driven cache invalidation •ILA routers provide fallback on cache invalidation Deployment](https://reader035.vdocuments.mx/reader035/viewer/2022070902/5f5ae0c9da58dc08e00b35c1/html5/thumbnails/39.jpg)
Operational implications•ICMP: TTL expired, unreachable (traceroute, PMTUD)•Contain “translated” SRC/DST addresses•Need fix in kernel to translate back <>
![Page 40: Internet-scale Virtual Networking - NANOG Archive · Mobility recap •Data-plane driven cache invalidation •ILA routers provide fallback on cache invalidation Deployment](https://reader035.vdocuments.mx/reader035/viewer/2022070902/5f5ae0c9da58dc08e00b35c1/html5/thumbnails/40.jpg)
What’s next?
![Page 41: Internet-scale Virtual Networking - NANOG Archive · Mobility recap •Data-plane driven cache invalidation •ILA routers provide fallback on cache invalidation Deployment](https://reader035.vdocuments.mx/reader035/viewer/2022070902/5f5ae0c9da58dc08e00b35c1/html5/thumbnails/41.jpg)
eBPF
![Page 42: Internet-scale Virtual Networking - NANOG Archive · Mobility recap •Data-plane driven cache invalidation •ILA routers provide fallback on cache invalidation Deployment](https://reader035.vdocuments.mx/reader035/viewer/2022070902/5f5ae0c9da58dc08e00b35c1/html5/thumbnails/42.jpg)
eBPF•BPF (Berkeley Packet Filter) - stuff you use in tcpdump•eBPF - extended BPF•JIT-compiled BPF with richer instruction set•Virtual machine in Linux kernel! <>
![Page 43: Internet-scale Virtual Networking - NANOG Archive · Mobility recap •Data-plane driven cache invalidation •ILA routers provide fallback on cache invalidation Deployment](https://reader035.vdocuments.mx/reader035/viewer/2022070902/5f5ae0c9da58dc08e00b35c1/html5/thumbnails/43.jpg)
Why it’s a big deal?•eBPF allows extending kernel functions•…From user-space. On the fly.•Multiple points of code injection in kernel•We built the ILA router code in eBPF <>
![Page 44: Internet-scale Virtual Networking - NANOG Archive · Mobility recap •Data-plane driven cache invalidation •ILA routers provide fallback on cache invalidation Deployment](https://reader035.vdocuments.mx/reader035/viewer/2022070902/5f5ae0c9da58dc08e00b35c1/html5/thumbnails/44.jpg)
XDP
![Page 45: Internet-scale Virtual Networking - NANOG Archive · Mobility recap •Data-plane driven cache invalidation •ILA routers provide fallback on cache invalidation Deployment](https://reader035.vdocuments.mx/reader035/viewer/2022070902/5f5ae0c9da58dc08e00b35c1/html5/thumbnails/45.jpg)
eXpress Data Path•XDP == Linux kernel bypass inside kernel!•Fast in-kernel networking•Packet processing pre-network-stack via eBPF•E.g. lookup and address rewrite•Punt to network stack if needed <>
![Page 46: Internet-scale Virtual Networking - NANOG Archive · Mobility recap •Data-plane driven cache invalidation •ILA routers provide fallback on cache invalidation Deployment](https://reader035.vdocuments.mx/reader035/viewer/2022070902/5f5ae0c9da58dc08e00b35c1/html5/thumbnails/46.jpg)
The finale
![Page 47: Internet-scale Virtual Networking - NANOG Archive · Mobility recap •Data-plane driven cache invalidation •ILA routers provide fallback on cache invalidation Deployment](https://reader035.vdocuments.mx/reader035/viewer/2022070902/5f5ae0c9da58dc08e00b35c1/html5/thumbnails/47.jpg)
ILA is…
![Page 48: Internet-scale Virtual Networking - NANOG Archive · Mobility recap •Data-plane driven cache invalidation •ILA routers provide fallback on cache invalidation Deployment](https://reader035.vdocuments.mx/reader035/viewer/2022070902/5f5ae0c9da58dc08e00b35c1/html5/thumbnails/48.jpg)
IPv6 Address per process
![Page 49: Internet-scale Virtual Networking - NANOG Archive · Mobility recap •Data-plane driven cache invalidation •ILA routers provide fallback on cache invalidation Deployment](https://reader035.vdocuments.mx/reader035/viewer/2022070902/5f5ae0c9da58dc08e00b35c1/html5/thumbnails/49.jpg)
Location independence
![Page 50: Internet-scale Virtual Networking - NANOG Archive · Mobility recap •Data-plane driven cache invalidation •ILA routers provide fallback on cache invalidation Deployment](https://reader035.vdocuments.mx/reader035/viewer/2022070902/5f5ae0c9da58dc08e00b35c1/html5/thumbnails/50.jpg)
Builds on XDP + eBPF
![Page 51: Internet-scale Virtual Networking - NANOG Archive · Mobility recap •Data-plane driven cache invalidation •ILA routers provide fallback on cache invalidation Deployment](https://reader035.vdocuments.mx/reader035/viewer/2022070902/5f5ae0c9da58dc08e00b35c1/html5/thumbnails/51.jpg)
Thank you<>