15744 - Fall 2004 Lecture 3 1
Inter-Domain Routing: BGP, Overlay Routing,
Multihoming
Vyas Sekar
Based on slides from:
Srini Seshan, Tim Griffin
15744 - Fall 2004 Lecture 3 2
Readings
• Assigned– [Mit notes] – Overview of BGP/ Interdomain routing– [Gao01] – Inferrring AS relationships– [Lab00] – BGP Convergence– [S+99] -- Suboptimality in Internet routing
• Optional– [Griffin01] – BGP Tutorial– [SARK01] – Characterizing the Internet Hierarchy from multiple
vantage points– [APMS+04] – A Comparison of Overlay routing and multihoming– [AWBKM] – Resilient Overlay Networks– [GW02] – iBGP configuration
15744 - Fall 2004 Lecture 3 3
Outline
• Need for hierarchical routing• BGP
– ASes, Policies– BGP Attributes– BGP Path Selection– iBGP – Inferring AS relationships
• Problems with BGP– Convergence– Sub optimal routing
• Overlay Routing and Multihoming• Summary
15744 - Fall 2004 Lecture 3 4
Routing Hierarchies• Flat routing doesn’t scale
– Each node cannot be expected to have routes to every destination (or destination network)
• Key observation– Need less information with increasing distance
to destination
• Two radically different approaches for routing– The area hierarchy?– The landmark hierarchy (discuss in routing
alternatives)?
15744 - Fall 2004 Lecture 3 5
Areas
• Divide network into areas– Areas can have nested
sub-areas– Constraint: no path
between two sub-areas of an area can exit that area
• Hierarchically address nodes in a network– Sequentially number top-
level areas– Sub-areas of area are
labeled relative to that area– Nodes are numbered
relative to the smallest containing area
1 2
3
1.1
1.2
2.12.2
3.1 3.2
2.2.1
2.2.2
1.2.1
1.2.2
15744 - Fall 2004 Lecture 3 6
Routing
• Within area– Each node has routes to every other node
• Outside area– Each node has routes for other top-level
areas only– Inter-area packets are routed to nearest
appropriate border router
• Can result in sub-optimal paths
15744 - Fall 2004 Lecture 3 7
Path Sub-optimality
1 2
3
1.11.2
2.1 2.2
3.1 3.2
2.2.1
3 hop red pathvs.2 hop green path
startend
3.2.1
1.2.1
15744 - Fall 2004 Lecture 3 8
A Logical View of the Internet
Tier 1 Tier 1
Tier 2
Tier 2
Tier 2
Tier 3
• National (Tier 1 ISP)– “Default-free” with
global reachability info
Eg: AT & T, UUNET, Sprint
• Regional (Tier 2 ISP)– Regional or country-
wide
Eg: Pacific Bell
• Local (Tier 3 ISP)Eg: Telerama DSL
Customer
Provider
15744 - Fall 2004 Lecture 3 9
Landmark Routing: Basic Idea
Source wants to reach LM0[a], whose address is c.b.a:•Source can see LM2[c], so sends packet towards c•Entering LM1[b] area, first router diverts packet to b•Entering LM0[a] area, packet delivered to a
- Not shortest path- Packet does not necessarily follow specified landmarks- No policy routing- Not source routing-Why
-Small routing tables-Dynamic, self configuring algorithms
15744 - Fall 2004 Lecture 3 10
Outline
• Need for hierarchical routing• BGP
– ASes, Policies– BGP Attributes– BGP Path Selection– iBGP – Inferring AS relationships
• Problems with BGP– Convergence– Sub optimal routing
• Overlay Routing and Multihoming• Summary
15744 - Fall 2004 Lecture 3 11
Autonomous Systems (ASes)
• Autonomous Routing Domain– Glued together by a common administration, policies etc
• Autonomous system – is a specific case of an ARD – ARD is a concept vs AS is an actual entity that participates in
routing– Has an unique 16 bit ASN assigned to it and typically
participates in inter-domain routing• Examples:
– MIT: 3, CMU: 9– AT&T: 7018, 6341, 5074, … – UUNET: 701, 702, 284, 12199, …– Sprint: 1239, 1240, 6211, 6242, …
• How do ASes interconnect to provide global connectivity • How does routing information get exchanged
15744 - Fall 2004 Lecture 3 12
Nontransit vs. Transit ASes
ISP 1ISP 2
Nontransit ASmight be a corporateor campus network.Could be a “content provider”
NET ATraffic NEVER flows from ISP 1through NET A to ISP 2(At least not intentionally!)
IP traffic
15744 - Fall 2004 Lecture 3 13
Customers and Providers
Customer pays provider for access to the Internet
provider
customer
IP trafficprovider customer
15744 - Fall 2004 Lecture 3 14
The Peering Relationship
peer peer
customerprovider
Peers provide transit between their respective customers
Peers do not provide transit between peers
Peers (often) do not exchange $$$trafficallowed
traffic NOTallowed
AB
C
15744 - Fall 2004 Lecture 3 15
Peering Wars
• Reduces upstream transit costs
• Can increase end-to-end performance
• May be the only way to connect your customers to some part of the Internet (“Tier 1”)
• You would rather have customers
• Peers are usually your competition
• Peering relationships may require periodic renegotiation
Peering struggles are by far the most contentious issues in the ISP world!
Peering agreements are often confidential.
Peer Don’t Peer
15744 - Fall 2004 Lecture 3 16
Routing in the Internet
• Link state or distance vector?– No universal metric – policy decisions
• Problems with distance-vector:– Bellman-Ford algorithm may not converge
• Problems with link state:– Metric used by routers not the same – loops– LS database too large – entire Internet– May expose policies to other AS’s
15744 - Fall 2004 Lecture 3 17
Solution: Distance Vector with Path
• Each routing update carries the entire path
• Loops are detected as follows:– When AS gets route check if AS already in
path• If yes, reject route• If no, add self and (possibly) advertise route further
• Advantage:– Metrics are local - AS chooses path, protocol
ensures no loops
15744 - Fall 2004 Lecture 3 18
BGP-4• BGP = Border Gateway Protocol
• Is a Policy-Based routing protocol
• Is the EGP of today’s global Internet
• Relatively simple protocol, but configuration is complex and the
entire world can see, and be impacted by, your mistakes.
• 1989 : BGP-1 [RFC 1105]– Replacement for EGP (1984, RFC 904)
• 1990 : BGP-2 [RFC 1163]
• 1991 : BGP-3 [RFC 1267]
• 1995 : BGP-4 [RFC 1771] – Support for Classless Interdomain Routing (CIDR)
15744 - Fall 2004 Lecture 3 19
BGP Operations (Simplified)
Establish session on TCP port 179
Exchange all active routes
Exchange incremental updates
AS1
AS2
While connection is ALIVE exchangeroute UPDATE messages
BGP session
15744 - Fall 2004 Lecture 3 20
Interconnecting BGP Peers
• BGP uses TCP to connect peers• Advantages:
– Simplifies BGP– No need for periodic refresh - routes are valid until
withdrawn, or the connection is lost– Incremental updates
• Disadvantages– Congestion control on a routing protocol?– Inherits TCP vulnerabilities!– Poor interaction during high load
15744 - Fall 2004 Lecture 3 21
Four Types of BGP Messages
• Open : Establish a peering session.
• Keep Alive : Handshake at regular intervals.
• Notification : Shuts down a peering session.
• Update : Announcing new routes or withdrawing
previously announced routes.
announcement = prefix + attributes values
15744 - Fall 2004 Lecture 3 22
Policy with BGP
• BGP provides capability for enforcing various policies• Policies are not part of BGP: they are provided to BGP
as configuration information• BGP enforces policies by choosing paths from multiple
alternatives and controlling advertisement to other AS’s• Import policy
– What to do with routes learned from neighbors?– Selecting best path
• Export policy– What routes to announce to neighbors?– Depends on relationship with neighbor
15744 - Fall 2004 Lecture 3 23
Examples of BGP Policies
• A multi-homed AS refuses to act as transit– Limit path advertisement
• A multi-homed AS can become transit for some AS’s– Only advertise paths to some AS’s– Eg: A Tier-2 provider multi-homed to Tier-1
providers
• An AS can favor or disfavor certain AS’s for traffic transit from itself
15744 - Fall 2004 Lecture 3 24
Export Policy
• An AS exports only best paths to its neighbors– Guarantees that once the route is announced the AS
is willing to transit traffic on that route• To Customers
– Announce all routes learned from peers, providers and customers, and self-origin routes
• To Providers– Announce routes learned from customers and self-
origin routes• To Peers
– Announce routes learned from customers and self-origin routes
15744 - Fall 2004 Lecture 3 25
Import Routes
Frompeer
Frompeer
Fromprovider
Fromprovider
From customer
From customer
provider route customer routepeer route ISP route
15744 - Fall 2004 Lecture 3 26
Export Routes
Topeer
Topeer
Tocustomer
Tocustomer
Toprovider
From provider
provider route customer routepeer route ISP route
filtersblock
15744 - Fall 2004 Lecture 3 27
BGP Route Processing
Best Route Selection
Apply Import Policies
Best Route Table
Apply Export Policies
Install forwardingEntries for bestRoutes.
ReceiveBGPUpdates
BestRoutes
TransmitBGP Updates
Apply Policy =filter routes & tweak attributes
Based onAttributeValues
IP Forwarding Table
Apply Policy =filter routes & tweak attributes
Open ended programming.Constrained only by vendor configuration language
15744 - Fall 2004 Lecture 3 28
BGP UPDATE Message
• List of withdrawn routes
• Network layer reachability information– List of reachable prefixes
• Path attributes– Origin– Path– Metrics
• All prefixes advertised in message have same path attributes
15744 - Fall 2004 Lecture 3 29
Path Selection Criteria
• Information based on path attributes
• Attributes + external (policy) information
• Examples:– Hop count– Policy considerations
• Preference for AS• Presence or absence of certain AS
– Path origin– Link dynamics
15744 - Fall 2004 Lecture 3 30
Important BGP Attributes
• Local Preference
• AS-Path
• MED
• Next hop
15744 - Fall 2004 Lecture 3 31
LOCAL PREF
• Local (within an AS) mechanism to provide relative priority among BGP routers
R1 R2
R3 R4I-BGP
AS 256
AS 300
Local Pref = 500 Local Pref =800
AS 100
R5
AS 200
15744 - Fall 2004 Lecture 3 32
LOCAL PREF – Common Uses
• Handle routes advertised to multi-homed transit customers– Should use direct connection (multihoming
typically has a primary/backup arrangement)
• Peering vs. transit– Prefer to use peering connection, why?
• In general, customer > peer > provider– Use LOCAL PREF to ensure this
15744 - Fall 2004 Lecture 3 33
AS_PATH• List of traversed AS’s
• Useful for loop checking and for path-based route selection (length, regexp)
AS 500
AS 300
AS 200 AS 100
180.10.0.0/16 300 200 100170.10.0.0/16 300 200
170.10.0.0/16 180.10.0.0/16
15744 - Fall 2004 Lecture 3 34
Multi-Exit Discriminator (MED)
• Hint to external neighbors about the preferred path into an AS – Non-transitive attribute – Different AS choose different scales
• Used when two AS’s connect to each other in more than one place
15744 - Fall 2004 Lecture 3 35
MED• Typically used when two ASes peer at multiple locations• Hint to R1 to use R3 over R4 link• Cannot compare AS40’s values to AS30’s
R1 R2
R3 R4
AS 30
AS 40
180.10.0.0MED = 120
180.10.0.0MED = 200
AS 10
180.10.0.0MED = 50
15744 - Fall 2004 Lecture 3 36
MED• MED is typically used in provider/subscriber scenarios• It can lead to unfairness if used between ISP because it
may force one ISP to carry more traffic:
SF
NY
• ISP1 ignores MED from ISP2• ISP2 obeys MED from ISP1• ISP2 ends up carrying traffic most of the way
ISP1
ISP2
15744 - Fall 2004 Lecture 3 37
Other Attributes
• ORIGIN– Source of route (IGP, EGP, other)
• NEXT_HOP– Address of next hop router to use
• Check out http://www.cisco.com for full explanation
• Question: Too many choices/ attributes how to select routes !
15744 - Fall 2004 Lecture 3 38
Route Selection Process
Highest Local Preference
Shortest ASPATH
Lowest MED
i-BGP < e-BGP
Lowest IGP cost to BGP egress
Lowest router ID
traffic engineering
Enforce relationships
Throw up hands andbreak ties
15744 - Fall 2004 Lecture 3 39
Internal vs. External BGP
R3 R4R1
R2
E-BGP
•BGP can be used by R3 and R4 to learn routes•How do R1 and R2 learn routes?•Option 1: Inject routes in IGP
•Only works for small routing tables•Option 2: Use I-BGP
AS1 AS2
15744 - Fall 2004 Lecture 3 40
Internal BGP (I-BGP)
• Same messages as E-BGP• Different rules about re-advertising prefixes:
– Prefix learned from E-BGP can be advertised to I-BGP neighbor and vice-versa, but
– Prefix learned from one I-BGP neighbor cannot be advertised to another I-BGP neighbor
– Reason: no AS PATH within the same AS and thus danger of looping.
15744 - Fall 2004 Lecture 3 41
Internal BGP (I-BGP)
R3 R4
R1
R2
E-BGP
I-BGP
• R3 can tell R1 and R2 prefixes from R4• R3 can tell R4 prefixes from R1 and R2• R3 cannot tell R2 prefixes from R1
R2 can only find these prefixes through a direct connection to R1Result: I-BGP routers must be fully connected (via TCP)!
• contrast with E-BGP sessions that map to physical links
AS1 AS2
15744 - Fall 2004 Lecture 3 42
Route ReflectoreBGP update
iBGP updates
Mesh does not scale
RR RR
RR
Each RR passes only best routes, no longer N^2 scaling problem
15744 - Fall 2004 Lecture 3 43
Policy Impact
• Different relationships – Transit, Peering
• Export policies selective export
• “Valley-free” routing– Number links as (+1, 0, -1) for customer-to-
provider, peer and provider-to-customer– In any path should only see sequence of +1,
followed by at most one 0, followed by sequence of -1
15744 - Fall 2004 Lecture 3 44
How to infer AS relationships?
• Can we infer relationship from the AS graph– From routing information– From size of ASes /AS topology graph– From multiple views and route announcements
• [Gao01]– Three-pass heuristic – Data from University of Oregon RouteViews
• [SARK01]– Data from multiple vantage points– Formulate TOR problem– Heuristic for solving the relationship assignment
15744 - Fall 2004 Lecture 3 45
[Gao00] Basic Algorithm
• Phase 1: Identify the degrees of the ASes from the tables
• Phase 2: Annotate edges with “transit” relation– AS u transits traffic for AS v if it provides its
provider/peer routes to v.
• Phase 3: Identify P2C, C2P, Sibling edges– P2C -> If and only if u transits for v, and v does not,
Sibling otherwise– Peering relationship ?
• Refined Algorithm : Another parameter L
15744 - Fall 2004 Lecture 3 46
How does Phase 2 work?
• Notion of Valley free routing– Each AS path can be
• Uphill• Downhill• Uphill – Downhill• Uphill – P2P• P2P -- Downhill• Uphill – P2P – Downhill
• How to identify Uphill/Downhill– Heuristic: Identify the highest degree AS to be the end
of the uphill path (path starts from source)
15744 - Fall 2004 Lecture 3 47
Observations from [Gao00]
• Heuristic to identify top provider does not work• Algorithm inferences verified from sources within
AT & T.• Majority are P2C, few peering, few sibling
– Peering is few because of the dataset used?– Sibling relationships are becoming more common
• Mergers, takeovers, backup relationships
• AS relationships are often complex – inferred relationships are “dominant commercial relationships”
15744 - Fall 2004 Lecture 3 48
Questions..
• Is inter-AS relationship– Prefix based– Customer based– Independent?
• Is the degree based heuristic valid?• Are peering relationships underestimated?• How useful is inferring the relationships
– Policy violations– Anomaly detection?– Is this information revealed against the providers will?
15744 - Fall 2004 Lecture 3 49
BGP Inefficiencies,Overlays andMultihoming
Aditya Akella
15744 - Fall 2004 Lecture 3 50
BGP Complexity
• BGP is a very complicated protocol– Too many knobs– Need to accommodate (sub-optimal)
ISP policies– Requires complex, human
configuration
• For all its complexity, BGP offers no guarantees– Performance??– Reliability??– Correctness??– Reachability??
• All of BGPs complexity begets…
Headache!
15744 - Fall 2004 Lecture 3 51
BGP Pitfalls and Problems
• Pitfalls and problems– Misconfiguration– Convergence– Performance– Reliability– Stability– Security– And the list goes on…
15744 - Fall 2004 Lecture 3 52
Favorite Scapegoat!
BGP
Networkingcommunity
15744 - Fall 2004 Lecture 3 53
Misconfiguration [Mahajan02Sigcomm]
• Origin misconfiguration: accidentally inject routes for prefixes into global BGP tables
Old Route New Route
Self deaggregation
(failure to aggregate)
a.b.0.0/16 X Y Z a.b.c.0/24 X Y Z
Related origin
(likely connected to the network– human error)
a.b.0.0/16 X Y Z a.b.0.0/16 X Y
a.b.0.0/16 X Y Z O
a.b.c.0/24 X Y
a.b.c.0/24 X Y Z O
Foreign origin
(address space hijack!)
a.b.0.0/16 X Y Z a.b.0.0/16 X Y O
a.b.c.0/24 X Y O
e.f.g.h/i X Y O
15744 - Fall 2004 Lecture 3 54
Misconfiguration
• Export misconfiguration: export route to a peer in violation of policy
Export Policy Violation
Provider AS Provider Route exported to provider was imported from a provider
Provider AS Peer Route exported to peer was imported from a provider
Peer AS Provider Route exported to provider was imported from a peer
Peer AS Peer Route exported to peer was imported from a peer
15744 - Fall 2004 Lecture 3 55
Interesting Observations
• Origin misconfig– 72% of new routes may be misconfig– 11-13% of misconfig incidents affect connectivity
• Pings and e-mail checks
– Self de-aggregation is the main cause
• Export misconfig– Upto 500 misconfiguration incidents per day– All forms are prevalent, although provider-AS-provider
is more likely
15744 - Fall 2004 Lecture 3 56
Effects and Causes
• Effects– Routing load– Connectivity disruption– Extra traffic– Policy violation
• Causes (Origin misconfig)– Router vendor software bugs:
announce and withdraw routes on reboot
– Reliance on upstream filtering– New configuration not saved to
stable storage (separate command and no autosave!)
– Hijacks of address spaces– Forgotten to install filter– Human operators and poor
interface
P1 P2
A
C
• Intended policy: Provide transit to C through link A-C
• Configured policy: Export all routes originated by C to P1 and P2
• Correct policy: export only when AS path is “C”
Export Misconfig
15744 - Fall 2004 Lecture 3 57
BGP Convergence [Labovitz00Sigcomm]
• Conventional beliefs– Path vector converges faster than traditional DV (eliminates the
count to infinity problem)– Internet path restoration takes order of 10s of seconds
• Convergence– Recovery after a fault may take as much as ten minutes– Single routing fault could result in multiple announcements and
withdrawals– Loss and RTT around times of faults are much worse
• Upon route withdrawal, explore paths of increasing length– In the worst case, could explore n! paths– Depends which messages are processed and when
• Limit between update message could reduce messages– Forces all outstanding messages to be processed
15744 - Fall 2004 Lecture 3 58
IBGP Problems [Griffin02Sigcomm]
• Route reflectors could impose signaling and forwarding anamolies instability!
Ri is a reflector for Ci (updates sent between Ri and Ci, i=1, 2)
Ri is a BGProuter announcingPi into the network
Ci will only know about Pi and it as best path
But Ci---Pi shortest path is Ci Ci+1Ri and this causes a forwarding loop!
15744 - Fall 2004 Lecture 3 59
End-to-End Routing Behavior [Paxson96Sigcomm]
• Large scale routing behavior as seen by end-hosts, based on analysis of traceroutes
• Pathologies: persistent routing loops, routing failures and long connectivity outages
• Stability: 9% or routes changed every 10s of minutes, 30% about ~6hrs and 68% took a few days
• Symmetry: more than half of paths probed were asymmetric at router level
15744 - Fall 2004 Lecture 3 60
Inefficiencies in BGP &Internet Routing
• Route convergence and oscillations• Poor reliability
– No way to exploit redundancy in Internet paths
• Inefficiency: sub-optimal RTTs and throughputs– What are some of the causes?
• Policies in routing: Inter-domain and Intra-domain• Lack of direct routes, “sparseness” of the Internet
graph
15744 - Fall 2004 Lecture 3 61
Inefficiency of Routes [Spring03Sigcomm]
• Three classes of reasons for poor performance (“inflation”)– Intra-domain topology and policy
• Topology: no direct link between all cities• Routing policy: “shortest paths” may be avoided due to
engineering
– ISP Peering• Peeering topology: limited peering between ISPs• Peering policy: hot-potato routing or early-exit routing
– Inter-domain• Topology: AS graph is sparse• Inter-domain policies: policies are policies
15744 - Fall 2004 Lecture 3 62
Path Inflation Summary
15744 - Fall 2004 Lecture 3 63
Performance: End-to-End Perspective
• From an end-to-end view…– Is there a way of extracting better performance?
• Is there scope?
• How do we realize this?
• Scope: Savage99, Akella03, Akella04
• Reality: UW’s “Detour” system, MIT’s RON, Akamai’s SureRoute, CMU’s Route Control
15744 - Fall 2004 Lecture 3 64
Quantifying Performance Loss [Savage99Sigcomm]
• Measure round trip time (RTT) and loss rate between pairs of hosts
• Alternate path characteristics– 30-55% of hosts had lower latency– 10% of alternate routes have 50% lower
latency– 75-85% have lower loss rates
15744 - Fall 2004 Lecture 3 65
Bandwidth Estimation
• RTT & loss for multi-hop path– RTT by addition– Loss either worst or combine of hops – why?
• Large number of flows combination of probabilities• Small number of flows worst hop
• Bandwidth calculation– TCP bandwidth is based primarily on loss and
RTT
• 70-80% paths have better bandwidth• 10-20% of paths have 3x improvement
15744 - Fall 2004 Lecture 3 66
Possible Sources of Alternate Paths
• A few really good or bad AS’s – No, benefit of top ten hosts not great
• Better congestion or better propagation delay?– How to measure?
• Propagation = 5th percentile of delays
– Both contribute to improvement of performance
15744 - Fall 2004 Lecture 3 67
Overlay Networks
• Basic idea:– Treat multiple hops through IP network as one hop in overlay network– Run routing protocol on overlay nodes
• Why?– For performance – like the Savage 99 paper showed– For efficiency – can make core routers very simple
• E.g. CSFQ, • Also aid deployment. E.g. Active networks
– For functionality – can provide new features such as multicast, active processing
15744 - Fall 2004 Lecture 3 68
Future of Overlay
• Application specific overlays– Why should overlay nodes only do routing?
• Caching– Intercept requests and create responses
• Transcoding– Changing content of packets to match available
bandwidth
• Peer-to-peer applications
15744 - Fall 2004 Lecture 3 69
Overlay Challenges
• “Routers” no longer have complete knowledge about link they are responsible for
• How do you build efficient overlay– Probably don’t want all N2 links – which links to create?– Without direct knowledge of underlying topology how to know what’s
nearby and what is efficient?
• Do we need overlays for performance?
15744 - Fall 2004 Lecture 3 70
Number of Route Choices
• Flexible control of end-to-end path many route choices
Multiple candidatepaths
Single path
Multiple BGPpaths
• BGP: one path via each ISP choices linked to #ISPs
Few more route choices…?
15744 - Fall 2004 Lecture 3 71
Route Selection Mechanism
• BGP: simple, coarse metrics such as least AS hops, policy
Best performingpath
Least AS hopsPolicy compliant
Current best performingBGP path
• Overlays: complex, performance-oriented selection
Sophisticated selection among multiple BGP routes
Smartselection
“Multihoming route control”
15744 - Fall 2004 Lecture 3 72
Overlay Routing vs Multihoming Route Control[Akella04Sigcomm]
>
>
~
~
1-multihoming
k-multihoming
k-multihoming
1-overlays
1-overlays
k-overlays
~
15744 - Fall 2004 Lecture 3 73
Overlay Routing vs. Multihoming Route Control
Cost
Operational issues
Route Control Overlay Routing
Sprint$$
Genuity$$
ATT$$
Overlay provider$$
ATT$$
Overlay nodeforces inter-mediate ISP to provide transit
/18 netblock
Announce/20 sub-blocksto ISPs
If all multihomed ends do this
Routing table expansion Bad interactions with policies
Connectivity fees Connectivity fees + overlay fee
15744 - Fall 2004 Lecture 3 74
Summary
• Route control similar to overlay routing for most practical purposes
• Overlays very useful for deploying functionality– Multicast, VPNs, QoS, security
• But overlays may be overrated for end-to-end performance and resilience
• Don’t abandon BGP – there’s still hope of extracting good performance and availability
