Evolution 0.9:The Evolution of the U.S.
Peering Ecosystem
Gigabit Peering Forum VIIHerndon, VA September 9, 2003
William B. Norton
Co-Founder & Chief Technical Liaison
Equinix, Inc.
Internet Researcher
• Four Years of Research• Document Internet Operations Practices –
Interconnect/Peering/IXes/etc.• Research Process – Focus with Community and
– Write White Paper version M.m– Walk throughs & Feedback– Fix/Update Paper
• 200 Walkthroughs later – a document reflecting community Internet Operations knowledge
So far 6 White Papers
Internet Operations White Papers
1) “Interconnection Strategies for ISPs”2) “Internet Service Providers and Peering”3) “A Business Case for Peering”4) “The Art of Peering: The Peering
Playbook”5) “The Peering Simulation Game”6) “Do ATM-based Internet Exchanges
Make Sense Anymore?” Freely available. Send e-mail to [email protected]
New white Paper: The Evolution of the U.S. Peering Ecosystem, Agenda…
Agenda
The Evolution of the Internet Peering Ecosystem– The Internet Peering Ecosystem Environment– Each Player, their Motivations, their Behavior
• Four Major Events, led to …
• Three Major Evolutions in the Peering Ecosystem
The Evolution of the U.S. Peering Ecosystem
Act I: 1998-2000
The Internet Peering Ecosystem
• The Environment
• The Players– Their role in the Peering Ecosystem– Their behavior in the Peering Ecosystem– Their relationship to others in the Peering
Ecosystem
• The Evolution of the Peering Ecosystem
The Internet Peering Ecosystem• Many “Internet Regions” (usually defined
by geopolitical boundaries)• Each with its own Peering Ecosystem
consisting of– Tier 1 ISPs– Tier 2 ISPs– Content Playersand Enterprises
Content / Enterprise Companies
Tier 1 ISPs
Tier 2 ISPs
Internet Region
Interconnect Regions• Large Internet Regions have multiple
geographical areas where ISPs peerInternet Region
Interconnect Region
Interconnect Region
Interconnect Region
Interconnect Region
Interconnect Region
Player: Tier 1 ISPs
Transit (Downstream)Connections
Backbone Backhaul
Peering Interconnections
Tier 1 ISPTier 1 ISP
Transit Customers
Def: A Tier 1 ISP is an ISP that has access to the entire RegionalInternet routing table solely through Peering Relationships (I.e. doesn’t buy transit from anyone).
Tier 1 Peering• Full Mesh Peering among Tier 1 ISPs
within each Interconnect Region
Tier 1 PeeringTier 1 PeeringTier 1 ISPTier 1 ISP
Tier 1 ISPTier 1 ISP
Tier 1 ISPTier 1 ISP
Tier 1 ISPTier 1 ISP
Tier 1 ISPTier 1 ISP
Tier 1 ISPTier 1 ISP
Some Tier 1 ISPs in the U.S.
Sprint
AT&T
UUNet
C&W
Level3
Qwest
Tier 1 Peering Across the U.S.
Sprint
AT&T
UUNet
C&W
Level3
Qwest
P P
P P
P
P PPP
P
PP
P
Sprint
AT&T
UUNet
C&W
Level3
Qwest
P P
P P
P
P PPP
P
PP
P
Sprint
AT&T
UUNet
C&W
Level3
Qwest
P P
P P
P
P PPP
P
PP
P
Sprint
AT&T
UUNet
C&W
Level3
Qwest
P P
P P
P
P PPP
P
PP
P
Sprint
AT&T
UUNet
C&W
Level3
Qwest
P P
P P
P
P PPP
P
PP
P
Sprint
AT&T
UUNet
C&W
Level3
Qwest
P P
P P
P
P PPP
P
PP
P
Sprint
AT&T
UUNet
C&W
Level3
Qwest
P P
P P
P
P PPP
P
PP
P
Sprint
AT&T
UUNet
C&W
Level3
Qwest
P P
P P
P
P PPP
P
PP
P
8 Interconnect Regions
Tier 1 Peering Motivations
• Generally not interested to peer with anyone else
1) They already get the traffic for free
2) They may be foregoing revenue (they could sell transit)
3) They don’t want to give away that which differentiates them
Player: Tier 2 ISPDef: A Tier 2 ISP is an ISP that purchases
(and therefore resells) transit within an Internet Region.
Transit (Downstream)Connections
Backbone Backhaul(optional)
Peering Interconnections(optional)
Tier 2 ISPTier 2 ISP
Transit Customers
UpstreamTransitConnection(s)
Tier 2 ISP Peering Motivations
1) Reduce transit costs
2) Improve Performance
3) Greater control over routing
Tier 2 Peering• Sparse Mesh Peering in each Interconnect
Region
Tier 2 PeeringTier 2 PeeringTier 2 ISPTier 2 ISP
Tier 2 ISPTier 2 ISP
Tier 2 ISPTier 2 ISP
Tier 2 ISPTier 2 ISP
Tier 2 ISPTier 2 ISP
Tier 2 ISPTier 2 ISP
Tier 2 ISPs
• Widely varying characteristics and behaviors
• Scope: Global, National, Regional, Local• Underfunded to well funded• Food fights to solid relationships within the
community• Big Peoples Table and Little Peoples Table
perhaps a bad generalization
The Content Provider and Enterprise Player
• Always buy transit
• Focus on Content
• Focus on Service
• Few Network Engineers
• SLAs valuable
Content Provider (and Enterprise)
Transit Provider Interconnections (Upstream(s))
The Internet Peering Ecosystem
Content / Enterprise Companies
Tier 1 ISPs
Tier 2 ISPs
Internet Region
Peering Inclinations and Peering Policies
• Def: A Peering Inclination is a predilection towards or against peering as demonstrated by Peering behavior in a Peering Ecosystem
• Def: A Peering Policy documents and defines the prerequisites to peering.
Four types of Peering Inclination
• Open Peering – Anyone, Anywhere
• Selective Peering – Some Requirements
• Restrictive Peering – No
• No Peering – No, prefer buying transit
The Evolution of the U.S. Peering Ecosystem
Act II: 2000-2003
Evolution of the U.S. Peering Ecosystem
Four key events leading to drastic disruption in the Peering Ecosystem:
1) The 1999/2000 Telecom Collapse2) The growth of the Used Equipment Market3) The Upstream Provider for the Cable Companies
(@Home) went bankrupt4) Peer-to-Peer file sharing systems (like Kazaa)
grow in popularity, traffic grows exponentially between Access Providers (1.5MB MP3 700MB AVI files)
Leads to 3 Major Evolutions
Evolution #1 – Cable Companies are Peering
• 40% of traffic Kazaa• 3-5 Gbps of transit
traffic-> Gbps of peering potential !
Significant because1) Volume2) Open Peering3) Kazaa Effect
(Optional)Backbone Backhaul
Cable Company
Transit Provider Interconnections (Upstream(s))
Peering
U.S. Cable Companies Eyeballs
MSO Country YE02 Change in 02Comcast USA 3,620,300 1,199,100Time Warner USA 2,613,000 1,027,000Cox USA 1,407,900 524,400Charter USA 1,180,000 572,300CableVision USA 770,100 263,500Shaw (Big Pipe) Canada 750,000 ?Rogers Canada 650,000 ?Adelphia USA 610,000 305,900Bright House USA 490,000 159,000Mediacom USA 191,000 76,000RCN USA 160,400 49,800Insight USA 144,800 56,700Cable One USA 78,100 45,200Total 12,665,600 4,278,900
Cable Modem Subscribers
Source: The Bridge & Leichtman ResearchSource: I$P HO$TING ReportMay 2003, Volume VII, Number 5, Page 8Article "Dialup Bakeoff - Is it really so bad at EarthLink? Or so good at United?"Source: Bill Norton conversations with Rogers and Shaw 2002 (numbers may be understated)
Cable Company
Cable Company
Cable Company
Cable CompanyCable Company
Cable Company
Cable Company
AOL
Content
1998-2000
Tier 2 ISPs
Tier 1 ISPs
Transit$
Thickness representsIn-Group Peering
Content
2000-2003 Evolution #1 – Cable Companies Peering
Tier 2 ISPs
CableCos
Tier 1 ISPs
Peering
Evolution #2 – Network Savvy Large Scale Content Companies
get into Peering• To reduce transit costs • To improve the end user experience• They need to move out of bankrupt colo anyway
Significant because
1) Volume of traffic is huge
2) Content Providers have Open Peering
3) Leading players pave the way
Content
1998-2000
Tier 2 ISPs
Tier 1 ISPs
Transit$
Thickness representsIn-Group Peering
Content
2000-2003 Evolution #2 – Large Scale Network Savvy Content
Peers
Tier 2 ISPs
LSNSC
Tier 1 ISPs
Other Broadband Players
• Significant, but…
• Not Open Peers so disincentive to peer
• So less volume and less disruptive
Evolution #3 – Cable Companies Freely Peer with Content
Companies• Content literally on the Cable Company
Network
• Lowest possible latency from content to eyeballs
• Internet Gaming, Broadband Streaming, etc.
Content
1998-2000
Tier 2 ISPs
Tier 1 ISPs
Transit$
Thickness representsIn-Group Peering
Content
2000-2003 Evolution #1 – Cable Companies Peering
Tier 2 ISPs
CableCos
Tier 1 ISPs
Content
2000-2003 Evolution #2 – Large Scale Network Savvy Content
Peers
Tier 2 ISPs
LSNSC
Tier 1 ISPs
Content
2000-2003 Evolution #3 – Cable Peers with Large Scale Network
Savvy Content
Tier 2 ISPs
LSNSC
Tier 1 ISPs
CableCos
International Dynamics – Separate White Paper
Content
Tier 2 ISPs
LSNSC
Tier 1 ISPs
CableCos
Content
Tier 2 ISPs
Tier 1 ISPs
NTT
KDDI
IIJ
JTCableCos
DSL
JP U.S.
U.S. Tier 1 ISPs are Tier 2 ISPs in Japan Internet RegionJapan Tier 1 ISPs are Tier 2 ISPs in the U.S. Internet Region
Summary 2003 Peering Ecosystem Evolving
NoPeering Open Selective Restrictive
Lg.Content
Tier2 ISPs
CableCos
RBOCs
Tier 1 ISPs
Foreign ISPs
Excite@Home