7/11/19

1

1

ISP Essentials Workshop - IXP

Manila, Philippines8-12 July 2019

22

Content• Internet Peering

• IXP Design Considerations• How to Build an IXP

• Connecting to an IXP

• IXP Best Common Practices

7/11/19

2

3

INTERNET PEERINGModule 1

44

Customer’s Expectation

7/11/19

3

55

Or maybe this

66

IP Transit• Provide access to “The Internet”• Requires a circuit to an “upstream” ISP

– Could be local (domestic) or international– Submarine circuits are fixed capacity, not tied to usage

• Also requires service from the “upstream” ISP– Billing is based on usage, typically 95th percentile– Or based on the speed of the connection (rate-limited or not)

• Repeat to get the level of redundancy required– Two circuits to the same “upstream” ISP– Circuits to two, or more, “upstream” ISPs

7/11/19

4

77

But it’s really just…

88

Until this happens

7/11/19

5

99

Or this2 core1-vl400.vcc.kidanet.com.fj (113.20.64.49) 327.221 ms 204.824 ms 12.070 ms

3 202.170.41.85 (202.170.41.85) 1.564 ms 2.537 ms 1.383 ms

4 202.170.33.2 (202.170.33.2) 2.826 ms 2.738 ms 1.563 ms

5 gi0-2-1-4.rcr21.b001848-1.sjc01.atlas.cogentco.com (38.122.92.249) 123.644 ms 123.736 ms 123.017 ms

6 be2063.ccr21.sjc01.atlas.cogentco.com (154.54.1.161) 124.323 ms

be2095.ccr22.sjc01.atlas.cogentco.com (154.54.3.137) 124.578 ms

be2063.ccr21.sjc01.atlas.cogentco.com (154.54.1.161) 123.759 ms

7 be3144.ccr41.sjc03.atlas.cogentco.com (154.54.5.102) 124.418 ms 124.695 ms

be3142.ccr41.sjc03.atlas.cogentco.com (154.54.1.194) 123.785 ms

8 zayo.sjc03.atlas.cogentco.com (154.54.10.194) 126.692 ms 125.425 ms 124.132 ms

9 ae16.cr2.sjc2.us.zip.zayo.com (64.125.31.14) 126.694 ms 123.851 ms 124.828 ms

10 ae27.cs2.sjc2.us.eth.zayo.com (64.125.30.232) 142.824 ms 142.947 ms 142.736 ms

11 ae3.cs2.sea1.us.eth.zayo.com (64.125.29.41) 142.369 ms 142.763 ms 142.015 ms

12 ae28.mpr1.sea1.us.zip.zayo.com (64.125.29.105) 142.880 ms 144.592 ms 142.519 ms

13 64.125.193.130.i223.above.net (64.125.193.130) 162.471 ms 163.139 ms 162.358 ms

14 xe-1-0-1.pe2.brwy.nsw.aarnet.net.au (202.158.194.120) 163.443 ms 162.016 ms 163.059 ms

15 ae9.bb1.a.syd.aarnet.net.au (113.197.15.57) 162.210 ms 163.574 ms 162.243 ms

16 ge-1-1-0.bb1.a.suv.aarnet.net.au (202.158.194.226) 198.100 ms 197.932 ms

1010

What’s wrong with this picture?• Fintel customer in Suva• Accessing content at the

University of the South Pacific in Suva

• Packet travels > 25,000km• Physical distance < 10km• Adding latency• Possibly jitter too• Using expensive submarine

capacity

7/11/19

6

1111

Interconnection (aka Peering)• Connection to a “peer” network

– Exchange of traffic to customers of each peer

• Requires a circuit to the peer (or to an Internet Exchange)– Fixed cost based on capacity of the link– May also require a cross connect in a data centre– Could be fixed cost or more likely monthly recurring fee

• Traffic is settlement free mostly• Cost is the same if zero bytes exchanged or link saturated

– Don’t saturate the link, customers will be grumpy J

1212

We compete, why interconnect?• International Connections…

– If satellite, RTT is around 550ms per hop– Compared to local traffic < 10ms round trip

• International bandwidth– Costs significantly more than domestic bandwidth– Don’t congest it with local traffic

• Wastes money– Harms overall performance (end-user experience)

• Lose-lose if not interconnect locally

1

7/11/19

7

1313

Private Interconnect

1

ISP A

ISP B

Autonomous System 99

Autonomous System 334

border border

1414

Interconnection (aka Peering)• Local (loop) connections

• Not in a customer/transit relationship• Sharing customer & infrastructure routes only

– Routes that generate revenue for you

• Share costs– Two circuits, pay for one each

7/11/19

8

1515

Results of Peering• Both save money

• Local traffic stays local• Better performance, better QoS, …

• Expensive international bandwidth available for actual international traffic

• Everyone is happy (except submarine cable and satellite owners)

• It is win-win

1616

Scaling peering• What happens when new ISPs enter the equation?

– Just repeat the process?

• Private peering means that each ISP has to buy circuits to every other peer (perhaps 2 for redundancy)– For (n) peers in total, each peer needs (n-1) half circuits

• Eg 10 peers in total => 9 half circuits for each

7/11/19

9

1717

Why an Internet eXchange Point (IXP)• Private peering relies on just the two parties making best use of

the circuit– by building dedicated circuits to each peer

• n peers in total => n(n-1)/2 circuits in total

• With an IXP:– Every participant has to buy just one whole circuit

• From their premises to the IXP fabric– Improve latency performance between peers where traffic volume

wouldn’t justify a dedicated circuit– Maximizes the opportunity to fill the circuit

• Peak traffic may not be the same across all peers

1818

Internet eXchange Point (IXP)• Need a location or facility that ISPs can access and can

connect to each other over a common shared media– Eg: Ethernet switch

• Should be a NEUTRAL venue

• Needs to have multiple telco circuit providers and/or allow any licenced provider to install services

• Needs controlled environment & access

7/11/19

10

1919

Internet eXchange Points• Variety of shapes and sizes

– Commercial– Community– Tbps to Mbps– Single location or Metropolitan Area scoped– Purely a traffic exchange– Value added services

• Layer 2 exchange point– Ethernet Switches (100Gbps/10Gbps/1Gbps/100Mbps)

2020

Internet eXchange Point• Border routers in different Autonomous Systems

IXP

ISP1

ISP2

ISP3

ISP4

ISP5

ISP6

7/11/19

11

2121

Layer 2 Exchange

ISP 1 ISP 2ISP 3

IXP ManagementNetwork

ISP 6 ISP 5 ISP 4

Ethernet Switch

IXP Services:Root & TLD DNS,Routing Registry

Looking Glass, etc

2222

Layer 2 Exchange

ISP 1 ISP 2ISP 3

IXP ManagementNetwork

ISP 6 ISP 5 ISP 4

Ethernet Switches

IXP Services:Root & TLD DNS,Routing Registry

Looking Glass, etc

7/11/19

12

2323

Layer 2 Exchange• Two switches for redundancy

• ISPs use dual routers for redundancy or load-sharing• Offer services for the “common good”

– Internet portals and search engines– DNS Root & TLDs, NTP servers– Routing Registry and Looking Glass

2424

Layer 2 Exchange• Requires neutral IXP management

– Usually funded equally by IXP participants– 24x7 cover, support, value add services

• Secure and neutral location

• Configuration– Private address space if non-transit and no value add services– Otherwise public IPv4 (/24) and IPv6 (/48, /56, /64)– ISPs require ASN, basic IXP does not

• Route Servers need ASN

7/11/19

13

2525

Layer 2 Exchange• Network Security Considerations

– LAN switch needs to be securely configured– Management routers require AAA authentication, vty security– IXP services must be behind router(s) with strong filters

2626

Types of Peering• Private Peering

• Bi-lateral Peering• Multi-lateral Peering

7/11/19

14

2727

Private Peering• Dedicated circuit between two peers

– Can use a cross connect within a data centre– Or via dark fibre, telco circuit, microwave, …

• Used where traffic levels high between two peers

• Expensive, cost shared between only two parties– Often in pairs; each peer pays for one

• But ultimate in control

2828

Bi-lateral Peering• Uses an Ethernet switch at an Internet Exchange• Single cross connect to the switch

– Peer can be remote (e.g. using Metro-Ethernet)

• Dedicated BGP peering between two peers• Relies on the IXP to manage the switch• Bandwidth shared by multiple peering relationships• But direct relationship between the two peers

– More control (granularity)– If bad things happen can turn down BGP on one peer

7/11/19

15

2929

Multi-lateral Peering (MLPA)• Uses an Ethernet switch at an Internet Exchange• Single cross connect to the switch• Single BGP peering session to a “route server”• Easiest to setup, only one session

– Automatically peer with everyone else

• Reliant on IXP for both switch and route server• Relationship is with the IXP• Lesser control (granularity)

– If a peer has a problem less options to workaround

3030

Types of Peering Policy• Open

• Selective• Restrictive

7/11/19

16

3131

Open Peering• “Have a pulse peering”

• Will peer with anyone– Typically bi-lateral or multi-lateral at an existing facility– Negligible additional cost so why not?

• Typically content providers have open peering policy

3232

Selective Peering• Conditional peering

– Ex: at an IXP, will ONLY peer bilaterally and NOT with the RS

• Some negotiation may be necessary• May have some rules that peers must fulfil

– volumes, ratios, number of multiple connects

• May only peer outside of primary market

7/11/19

17

3333

Restrictive Peering• Rules!

• Has a (written) policy that defines if they will peer– Often with rules, which are set so that they don’t peer

• Often involves a minimum level of traffic– Could require a test peering to check conformance

• Also can include a “ratio” in/out traffic levels

3434

Cost tied to circuit size (not byte count)• Peering is typically settlement free

– No charge for the traffic exchanged

• Cost to peer– Router interface– Circuit to the peering fabric– Charges imposed by the IXP– All fixed, either capital expenditure or monthly recurring fee

7/11/19

18

3535

Choosing a IXP• Some markets have more than one

• Even if there is only one IXP it might appear in multiple locations– E.g. LINX is built on two rings through multiple data centres across

London

• Best location might be dictated by availability of IPLC, transit, or other factors

3636

Which IXP?• How many routes are available?

– How many other operators/providers are at the IX?– What is the traffic to and from these destinations, and how much will it reduce the

transit cost?

• What is the cost of co-lo space?– Availability of power, type of cabinet, …

• What is the cost of a circuit to the location?– If similar to transit costs are you getting a benefit?

• What is the cost of remote-hands?– For maintenance purposes to avoid serious outages

7/11/19

19

3737

Remote locations• If building to a remote location

• Make sure remote hands work at times when it’s important to you– Their 9-5 is not normally your office hours

• Check the skill set of the remote hands– Maybe engage a local consultant to help

3838

Worked Example• Single International Transit versus Local IXP + Regional

IXP + Transit

7/11/19

20

3939

Worked Example• ISP A is local access provider

– Some business customers (around 200 fixed links)– Some co-located content provision (datacentre with 100 servers)– Some consumers on broadband (5000 DSL/Cable/Wireless)

• They have a single transit provider– Connect with a 16Mbps international leased link to their transit’s PoP– Transit link is highly congested

4040

Worked Example (2)• There are two other ISPs serving the same locality

– There is no interconnection between any of the three ISPs– Local traffic (between all 3 ISPs) is traversing International

connections

• Course of action for our ISP:– Work to establish local IXP– Establish presence at overseas co-location

• First Step– Assess local versus international traffic ratio– Use NetFlow on border router connecting to transit provider

7/11/19

21

4141

Worked Example (3)• Local/Non-local traffic ratio

– Local = traffic going to other two ISPs– Non-local = traffic going elsewhere

• Example: balance is 30:70– Of 16Mbps, that means 5Mbps could stay in country and not congest

International circuit– 16Mbps transit costs $50 per Mbps per month

• local traffic charges = $250 per month, or $3K per year for local traffic

– Circuit costs $100K per year => $30K is spent on local traffic

• Total is $33K per year for local traffic

4242

Worked Example (4)• IXP cost:

– Simple 8 port 10/100 managed switch plus co-lo space over 3 years could be around US$30K total => $3K per year per ISP

– One router to handle 5Mbps would be around $9K, good for 3 years => $3K per year

– One local 10Mbps circuit from ISP location to IXP location would be around $5K per year, no traffic charges

– Per ISP total: $11K– Somewhat cheaper than $33K– Business case for local peering is straightforward - $22K saving per

annum

7/11/19

22

4343

Worked Example (5)• After IXP establishment

– 5Mbps removed from International link– Leaving 5Mbps for more International traffic – and that fills the link

within weeks of the local traffic being removed

• Next step is to assess transit charges and optimise costs– ISPs visits several major regional IXPs– Assess routes available– Compares routes available with traffic generated by those routes

from its NetFlow data– Discovers that 30% of traffic would transfer to one IXP via peering

4444

Example: South Asian ISP @ LINX• Date: May 2013

• Data:– Route Server plus bilateral peering offers 70K prefixes– IXP traffic averages 247Mbps/45Mbps– Transit traffic averages 44Mbps/4Mbps

• Analysis:– 85% of inbound traffic comes from 70K prefixes available by peering– 15% of inbound traffic comes from remaining 380K prefixes from

transit provider

7/11/19

23

4545

Example: South Asian ISP @ HKIX• Date: May 2013

• Data:– Route Server plus bilateral peering offers 67K prefixes– IXP traffic is 159Mbps/20Mbps– Transit traffic is 108Mbps/50Mbps

• Analysis:– 60% of inbound traffic comes from 67K prefixes available by peering– 40% of inbound traffic comes from remaining 383K prefixes from

transit provider

4646

Example: South Asian ISP• Summary:

– Traffic by Peering: 406Mbps/65Mbps– Traffic by Transit: 152Mbps/54Mbps

– 73% of incoming traffic is by peering– 55% of outbound traffic is by peering

7/11/19

24

4747

Example: South Asian ISP• Router at remote co-lo

– Benefits: can select peers, easy to swap transit providers– Costs: co-lo space and remote hands

• Overall advantage:– Can control what goes on the expensive connectivity “back to home”

4848

Value propositions• Peering at a local IXP

– Reduces latency & transit costs for local traffic– Improves Internet quality perception

• Participating at a Regional IXP– A means of offsetting transit costs

• Managing connection back to home network

• Improving Internet Quality perception for customers

7/11/19

25

49

IXP DESIGN CONSIDERATIONSModule 2

5050

IXP Capital Expenses• Managed Ethernet Switch(es) – mandatory

• Route Server(s) – desirable• Router(s) and server(s) to support other IXP services –

optional

7/11/19

26

5151

IXP Operating Expenses• Data Center space, including

– Rack Space– Appropriate Electricity (AC or DC, with UPS/genset)– Cooling– CCTV Camera and other physical security measures– Cross-connects

• Administration and Technical support

• Equipment maintenance• Internet transit for other IXP services – optional

5252

IXP Organization Model• Possible Models include:

– Donation / sponsorship– Cost sharing by participants

• Fixed fee, Port speed based fee, …– Membership based organization– Volunteers vs Outsourcing vs Staff

• Or any combinations of above• Neutrality is the most important, even at the beginning• Long term sustainability is the hard part so sustainable

financial model has to be established in due course

7/11/19

27

5353

Neutral Location as Starting Point• May choose one of the followings as starting point:

– University– Technology Park– Carrier Neutral Data Center – Government Data Center– Submarine Cable Station

• Having multiple carrier options is the most important• Should maintain neutrality continuously• Expansion to multiple sites can be done gradually, coupled

with growth

5454

Requirements of IXP Site • Proximity to the networks of the potential members /

participants

• Options, availability, capacity and reliability of fiber carriers

• Support for additional fiber carriers

• 24x7 access for IXP authorized support personnel

7/11/19

28

5555

Requirements of IXP Site • Availability and stability of electricity supply, including UPS

and backup power generator– Do you need DC power?

• Sufficient cooling facilities

• Good physical security – 24x7 surveillance and access control

• Availability of additional rack space for future growth

5656

General Guidelines – Governance• Organisation-wise, multi-stakeholder bottom-up approach is

the preferred approach for maximum acceptance of the community while government support is critical

• IXP should be as inclusive as possible in order to provide maximum benefits to the whole community which it serves

• Should be fair and consistent to every participant

• Should be open and transparent as much as possible

7/11/19

29

5757

General Guidelines - Geography• IXP should be within a metro area

– to avoid competing with participants and to maintain better neutrality

• Should start with the city with the largest concentration of ISPs first and gradually set up separate infrastructure in other cities if needed

5858

General Guidelines - Policies• AUPs

– Acceptable Use Policy– Minimal rules for connections

• IXP should just provide the platform to facilate participants to do peering– Peering agreements left to participants instead of mandated by IXP– Route server can be set up to facilitate easy multilateral peering as

an option – But participants should always bear in mind the end goal of “Keeping

Local Traffic Local”

7/11/19

30

5959

General Guidance - Fees• Rely on donations

• Cost recovery– Fixed membership fees– Per port fees

• One-off fee or recurring

• Commercial operations

6060

Services to Offer• Services offered should avoid competing with particpants

– e.g. web hosting at an IXP is a bad idea unless all members agree to it

• IXP operations should make performance and throughput statistics available to members– Use tools such as LibreNMS and IXP Manager to produce IX

throughput graphs for member (or public) information

7/11/19

31

6161

Services to Offer• ccTLD DNS

– the country IXP could host the country’s top level DNS– e.g. “.SE” TLD is hosted at Netnod IXes in Sweden– Offer back up of other country ccTLD DNS

• Root server– Anycast instances of root servers (I.root-servers.net, F.root-

servers.net etc are present at many IXes)

• gTLD DNS– .com & .net are provided by Verisign at many IXes

6262

Services to Offer• Route Server

– Helps provide easy multilateral peering with simple BGP configuration for participants

– Technical details to be covered later on

• Looking Glass– One way of making the Route Server routes available for global view

(e.g. www.traceroute.org)– Public or members-only access

7/11/19

32

6363

Services to Offer• Content Redistribution/Caching

– For example, Akamised update distribution service

• Network Time Protocol– Locate a stratum 1 time source (GPS receiver, atomic clock, etc) at

IXP

64

HOW TO BUILD AN IXPModule 3

7/11/19

33

6565

How to setup an IXP?• The IXP core is an Ethernet switch

– Managed switch with reasonable security features including Port Security

– Large enough buffer size with no shared buffer

• Has superseded all other types of network devices for an IXP– From the smallest 12 or 24 port 1G/10G switch– To the largest switches supporting 40G/100G interfaces– Link Aggregation support is preferable

6666

How to setup an IXP?• Each ISP participating in the IXP brings a router to the IXP

location– NOTE: ISPs may connect directly to the IXP (depending on

availability of fibre connections) instead of a dedicated router at the IXP

• Router needs:– One Ethernet port to connect to IXP switch– One WAN port to connect to the WAN media leading back to the ISP

backbone– To be able to run BGP

7/11/19

34

6767

How to setup an IXP?• IXP switch located in one dedicated equipment rack

– Also includes other IXP operational equipment

• Routers from participant ISPs located in adjacent rack(s)– May NOT be needed if remote Ethernet circuits are used by all

participants

• Using SM fibres for 1/10/40/100Gbps is recommended

6868

Peering• Each participant needs to run BGP

– They need their own AS number– Public ASN, NOT private ASN

• Each participant configures external BGP directly with the other participants in the IXP– Peering with all participants– OR– Peering with a subset of participants

• AND/OR– Joining multilateral peering via the route server

7/11/19

35

6969

Routing Advice• ISP border routers at the IXP should NOT advertise default

route or the full Internet routing table

– Carrying default or full table means that this router and the ISP network is open to abuse by other IXP members

– Recommended configuration is only to carry routes offered to IXP peers on the IXP peering router

• NOTE: Some ISPs offer transit across IX fabrics– They do so at their own risk – see above

7070

Routing (more)• ISP border routers at the IXP should not be configured to

carry the IXP LAN network within the IGP or iBGP– Use next-hop-self BGP concept

• Don’t generate ISP prefix aggregates on IXP peering router– If connection from backbone to IXP router goes down, normal BGP

failover will then be successful

7/11/19

36

7171

Address Space• Smaller IXPs may use private addresses for the IX LAN

– Public address space means IXP network could be leaked to Internet which may be undesirable

– Because most ISPs filter RFC1918 address space, this avoids the problem

– But the address space used may be in conflict with the address space used internally by some ISPs

• Bigger IXPs use public addresses for the IX LAN– Address space available from the RIRs– IXP terms of participation often forbid the IX LAN to be leaked to Internet

so special care has to be taken– Does produce documentation for traceroute– Still RECOMMENDED

7272

APNIC Policy on IXP Address Space• The End-User Assignments policy caters for IXP’s Public

Address space under IXP Address Assignment – https://www.apnic.net/get-ip/faqs/ixp-address-assignment/

• It requires that IXP have minimum 3 ISPs connected and have clear and open policy for joining

• The minimum IXP Assignment is /24 of IPv4 and /48 for IPv6

7/11/19

37

7373

Hardware• Ethernet switch needs to be managed

– Unmanaged switch means an unmanaged IXP

• Insist that IXP participants bring their own router– moves buffering problem off the IXP– Avoid spanning tree and other L2 security issues– Run port-security (MAC filtering) to protect the IX– security of the ISP connection is responsibility of the ISP, not the IXP

7474

How to set up an IXP? • The hard part with establishing an IXP is NOT the technical

part, but for relevant stakeholders to come together to build a creditable governance structure for the IXP with which everyone is happy (TRUST)

7/11/19

38

75

CONNECTING TO AN IXPModule 4

7676

IX Etiquette and Hygiene• Connect using a layer 3 device• Don’t proxy ARP• No CDP, RIP, EIGRP, OSPF, ISIS

– https://ams-ix.net/technical/specifications-descriptions/config-guide

• Don’t steal default• Don’t leak the IX prefix to the Internet• Do use consistent announcements• Do register prefixes in an Internet Routing Registry

7/11/19

39

7777

Filtering Announcements• Only send infrastructure and customer routes

– Can use community tagging to easily identify them

• Filter what you accept– Route filters (use Routing Registry data)– AS path filters– Maximum prefix count

• Minimum prefix size– Typically a /24 for IPv4 (/48 for IPv6)– May special case host routes for blackhole

7878

PeeringDB• https://www.peeringdb.com

• Identifies your AS number• Provides contacts for NOC

– Very useful when using a MLPA and need to contact peer

• Shows which facilities you use for peering– IP numbers in use at those facilities

• Brief description of who you are, how to contact you, your traffic levels, type of customers, your peering policy

7/11/19

40

7979

PeeringDB for Internet eXchange• Location and contact information

• Who is there, both on fabric and for cross connect– useful for planning when building out or searching for peers

• IXP-DB under development which will sync up with PeeringDB

8080

Tools to create router configuration• Typically use Internet Routing Registry (IRR) data

– Mostly communicate with RADB, which mirrors other registry data– APNIC members can use APNIC RR-DB

• IRRToolSet– The first public tool set, current development unclear but probably

stable

• BGPQ3– Newer tool, currently actively developed– Creates filters so you will need to script using it

7/11/19

41

8181

IRRToolSet• https://github.com/irrtoolset/irrtoolset

• Can create router configurations from policy defined in Routing Policy Specification Language (RPSL)

• Uses autnum, as-set and route objects

• rtconfig – creates configuration file

• peval – queries IRR data• Example policy in autnum objects – AS2764 & AS7575

8282

BGPQ3• https://github.com/snar/bgpq3

• Creates AS path or route filters based on IRR data• Supports a variety of formats

– IOS (both “classic” and XR), JUNOS, JSON, BIRD – Can also DIY format

7/11/19

42

8383

BGPQ3 Example (IOS-XR)

% bgpq3 -PXl prefixset-as38442 AS38442P- prefix listX- IOS-XRl- name of generated entry

no prefix-set prefixset-as38442prefix-set prefixset-as3844227.123.128.0/18,43.245.56.0/22,103.244.228.0/22,183.81.128.0/20

end-set

8484

Using communities for filters• Set a community when you import a route from a customer

or create a static (aggregate) route

• Use that community to control export to peers & transit

• Don’t allow peers or transits to set it though

• Now when you add a prefix on a router it will automatically get exported on other routers without updating their prefix lists

7/11/19

43

8585

Peering is not just technical• A personal relationship helps• Support your local NOG (Network Operator Group) • If you expand to other markets try to attend their NOG or Peering

Forums• For TL networks, this might include:

– APRICOT – Asia Pacific– Peering Asia – Asia Pacific– MyNOG – Malaysia– SGNOG – Singapore– IDNOG – Indonesia– PhNOG – Philippines

86

ROUTE COLLECTORSModule 4.1

7/11/19

44

8787

What is a Route Collector?• Usually a router or Unix system running BGP

• Gathers routing information from service provider routers at an IXP– Peers with each ISP using BGP

• Does not forward packets

• Does not announce any prefixes to ISPs

87

8888

Purpose of a Route Collector• To provide a public view of the Routing Information

available at the IXP– Useful for existing members to check functionality of BGP filters– Useful for prospective members to check value of joining the IXP– Useful for the Internet Operations community for troubleshooting

purposes• E.g. www.traceroute.org

88

7/11/19

45

89

Route Collector at an IXP

Route Collector

R1

R3

R5SWITCH

R2 R4

89

9090

Route Collector Requirements• Router or Unix system running BGP

– Minimal memory requirements – only holds IXP routes– Minimal packet forwarding requirements – doesn’t forward any packets

• Peers eBGP with every IXP member– Accepts everything; Gives nothing– Uses a private ASN– Connects to IXP VLAN

• “Back-end” connection– Second Ethernet globally routed– Connection to IXP Website for public access

90

7/11/19

46

9191

Route Collector Implementation• Most IXPs now implement some form of Route Collector

• Benefits already mentioned• Great public relations tool

• Unsophisticated requirements– Just runs BGP

91

92

ROUTE SERVERSModule 4.2

7/11/19

47

9393

What is a Route Server?• Has all the features of a Route Collector

• But also:– Announces routes to participating IXP members according to their

routing policy definitions

• Implemented using the same specification as for a Route Collector

93

9494

Features of a Route Server• Helps scale routing for large IXPs

• Simplifies Routing Processes on ISP Routers• Optional participation

– Provided as service, is NOT mandatory

• Optionally uses Policy registered in IRR

94

7/11/19

48

9595

Diagram of N-squared Peering Mesh

• For large IXPs (dozens for participants) maintaining a larger peering mesh becomes cumbersome and often too hard

95

9696

Peering Mesh with Route Servers

• ISP routers peer with the Route Servers– Only need to have two eBGP sessions rather than N

RS RS

96

7/11/19

49

97

RS based Exchange Point Routing Flow

TRAFFIC FLOWROUTING INFORMATION FLOW

RS

97

9898

Advantages of Using a Route Server• Helps scale Routing for very large IXPs

• Separation of Routing and Forwarding• Simplify Routing Configuration Management on ISPs

routers

98

7/11/19

50

9999

Disadvantages of using a Route Server• ISPs can lose direct policy control

– If RS is only peer, ISPs have no control over who their prefixes are distributed to• Some IXPs provide community based filtering option

• Completely dependent on 3rd party– Configuration, troubleshooting, etc…

99

100100

Typical usage of a Route Server• Route Servers may be provided as an OPTIONAL service

– Most common at large IXPs (>50 participants)– Examples: LINX, HKIX, AMS-IX, etc

• ISPs peer:– Directly with significant peers– With Route Server for the rest

100

7/11/19

51

101101

Things to think about...• Would using a route server benefit you?

– Avoids having to maintain a large number of eBGP peers– But can you afford to lose policy control? (An ISP not in control of

their routing policy is what?)

101

102

IXP BEST PRACTICESModule 5

7/11/19

52

103103

Concept• Some Service Providers attempt to cash in on the

reputation of IXPs

• Market Internet transit services as “Internet Exchange Point”– “We are exchanging packets with other ISPs, so we are an Internet

Exchange Point!”– So-called Layer-3 Exchanges — really Internet Transit Providers– Router used rather than a Switch– Most famous example: SingTel-IX

104104

Competition• Too many exchange points in one locale

– Competing exchanges defeats the purpose– Los Angeles and Tokyo have multiple but it’s a rarity

• Becomes expensive for ISPs to connect to all of them

7/11/19

53

105105

Rules and Restrictions• IXPs try to compete with their membership

– Offering services that ISPs would/do offer their customers

• IXPs run as a closed privileged club– E.g. Restrictive membership criteria (closed shop)

• IXPs providing access to end users rather than just Service Providers

• IXPs interfering with ISP business decisions e.g. Mandatory Multi-Lateral Peering

106106

Technical Design Issues• Interconnected IXPs

– IXP in one location believes it should connect directly to the IXP in another location

– Who pays for the interconnect?– How is traffic metered?– Competes with the ISPs who already provide transit between the two

locations (who then refuse to join IX, harming the viability of the IX)– IXP spanning multiple data centres in a city work ok (e.g. LINX)

7/11/19

54

107107

Technical Design Issues• ISPs bridge the IXP LAN back to their offices

– “We are poor, we can’t afford a router”– Financial benefits of connecting to an IXP far outweigh the cost of a

router– In reality it allows the ISP to connect any devices to the IXP LAN

• with disastrous consequences for the security, integrity and reliability of the IXP

108108

Routing Design Issues• Route Server Mandated

– Mandatory multilateral peering is generally not welcomed– ISPs have no incentive to learn BGP– Therefore have no incentive to understand peering relationships, peering

policies,– Entirely dependent on operator of RS for troubleshooting, configuration,

reliability• RS can’t be run by committee!

• Route Server is mainly to help easy peering at IXPs but should not just rely on it– Should do bilateral peering as well

7/11/19

55

109109

eXchange Point Examples• AMS-IX, DE-CIX and LINX in Europe• Equinix IX, in every Equinix Data Centre• SIX in Seattle, Washington, USA• SGIX in Singapore• MyIX in Kuala Lumpur, Malaysia• BBIX, JPIX and JPNAP in Tokyo, Japan• HKIX in Hong Kong• IX Australia in Perth, Sydney, Melbourne, Brisbane• ………• All use Ethernet Switches

110110

HKIX• Started and owned by CUHK since 1995

– Neutral to various service providers

• Gradually opened up to all networks to connect– Served licensed ISPs only initially

• 8 full-time staff– Run by part-time staff initially

• Expanded to 4 different commercial data centers in 2016-17 while still maintaining neutrality

• 290+ networks connected• Peak traffic at 1.1+Tbps • Top 3 IXP in Asia Pacific now in terms of traffic volume• See: http://www.hkix.net

7/11/19

56

111111

SGIX• Not-for-profit / Independent legal entity / Membership-based • Set up with the support of Singapore Government• With full-time staff from beginning• Went through a painful start-up process when potential

participants preferring to take wait-and-see approach– Persistency helped them get through the difficult period

• 100+ networks now• Peak traffic at 180+Gbps• Has presence in 7 commercial data centers• See: https://www.sgix.sg

112112

MyIX• Non-profit / Independent Legal Entity / Membership-based• Set up with the support of Malaysian Government• Full-time staff• Governance by elected representatives of local Malaysian ISPs• Multiple nodes in multiple cities• 40 Ordinary Members (NSP licensees; with voting right) / 30+

Associate Members (without voting right) / 20+ Members by Invitation (without voting right)

• See: http://myix.my

7/11/19

57

113113

APIX • A forum for IXPs to exchange experiences.

• 26 IXPs from 17 economies• See: http://apix.asia

114114

More info about IXPs• https://www.pch.net/resources/wiki/

– Another excellent resource of IXP locations, papers, IXP statistics, etc

• https://www.internetexchangemap.com/– Tele Geography: A collection of IXPs and interconnect points for ISPs

• https://www.peeringdb.com/– Searchable database of Exchange Points, Networks & Facilities