Route Server Automation and ROV

Nick Pratleynick@ix.asn.auLife Under Lockdown: how to stop heists, hijacks, and hostages

Timeline

When

Metrics and Reporting to Members NOW

RS1 Upgrades 17th – 18th August

RS2 Upgrades 24th – 25th August

Route Server Automation (yes, including daily AS-SET updates) 1st September

Drop Invalid Routes 1st September

Metrics and Reporting• What are we going to drop when we enable ROV?

• No RPKI support on current software, no easy & fast way to get and validate rib

• Emails to members with invalid routes, do you know about them, can we help? Hopefully after tonight everyone is just going to go and do it? ;-)

• Hacked some python scripts:• SSH to route servers & get rib: “ssh -c ’sudo birdc show route table master’ rs1.*.ix.asn.au”• Validate routes using Cloudflare's rpki.json (https://rpki.cloudflare.com/rpki.json) with a

Radix tree in python for searches• Export data to influxdb as a telegraf input running hourly• Graph with Grafana

• https://metrics.ix.asn.au/d/58WdNHGMk/ix-rpki

So – what will we drop?AS10AS10214 10214 121.200.32.0/24AS10214 10214 121.200.33.0/24AS132405 132405 2001:df0:2c7:100::/64AS132405 132405 2001:df0:2c7:1::/64AS132405 132405 2001:df0:2c7:200::/64AS132405 132405 2001:df0:2c7:3001::/64AS132405 132405 43.250.92.0/24AS132405 132405 43.250.93.0/24AS132405 132405 43.250.94.0/24AS132405 132405 43.250.95.0/24AS13335 13335 103.21.244.0/24AS13335 13335 2606:4700:7000::/48AS134090 134090 103.106.90.0/24AS134090 134090 103.106.91.0/24AS134090 59256 2401:9cc0:200::/48AS134090 59256 2401:9cc0:300::/48AS135513 17741 114.31.103.0/24AS136001 136001 202.179.134.0/24AS139609 45891 202.1.160.0/20AS139609 45891 202.1.176.0/20AS23838 23838 116.90.135.0/24AS23838 23838 2401:f000:2:200::/56AS23838 23838 2401:f000:2::/56AS23838 23838 2401:f000:32:101::/64AS23838 23838 2401:f000:32:103::/64AS23838 23838 2401:f000:32:13::/64AS23838 23838 2401:f000:32:16::/64AS23838 23838 2401:f000:32:18::/64AS23838 23838 2401:f000:32:2::/64AS23838 23838 2401:f000:32:27::/64AS23838 23838 2401:f000:32:28::/64AS23838 23838 2401:f000:32:29::/64AS23838 23838 2401:f000:32:37::/64AS23838 23838 2401:f000:32:4::/64AS23838 23838 2401:f000:32:5::/64AS23838 23838 2401:f000:32:6::/64

AS23838 23838 2401:f000:32:86::/64AS23838 23838 2401:f000:32:87::/64AS23838 23838 2401:f000:32:92::/64AS23838 23838 2401:f000:32:99::/64AS23838 23838 2401:f000:32:c000::1/128AS23838 23838 2401:f000:32:c000::2/128AS23838 23838 2401:f000:32:c000::/64AS23838 23838 2401:f000:32:c001::105/128AS23838 23838 2401:f000:32:c001::11/128AS23838 23838 2401:f000:32:c002::2/128AS23838 23838 2401:f000:32:c002::3/128AS23838 23838 2401:f000:32:c003::1/128AS23838 23838 2401:f000:32:c005::/64AS23838 23838 2401:f000:5:1::/64AS23838 23838 2401:f000:5::/64AS23838 23838 2402:1c00:0:1::/64AS23838 23838 2402:1c00:1000:1::/64AS23838 23838 2402:1c00:10::/126AS23838 23838 2402:1c00:2000:1001::/64AS23838 23838 2402:1c00::/64AS23838 23838 2402:1c00:dead:beef::/64AS23838 23838 2402:1c00:fffe::/127AS23838 23838 2402:1c00:fffe::4/127AS23838 23838 2402:1c00:ffff:ffff:ffff:ffff:0:a0/124AS23838 23838 2402:1c00:ffff:ffff:ffff:ffff:0:c0/124AS23838 23838 2402:1c00:ffff:ffff:ffff:ffff:0:c2/127AS23838 23838 2402:1c00:ffff:ffff:ffff:ffff:0:c6/127AS23838 23838 2402:1c00:ffff:ffff:ffff:ffff:0:d0/124AS23838 23838 2402:1c00:ffff:ffff:ffff:ffff:0:e0/124AS23838 23838 2402:1c00:ffff:ffff:ffff:ffff:0:f0/127AS23838 23838 2402:1c00:ffff:ffff:ffff:ffff:0:f2/127AS3356 132199 120.28.146.0/24AS3356 132199 120.28.165.0/24AS3356 132199 120.28.252.0/22AS3356 132199 180.190.84.0/24AS3356 132199 180.191.192.0/23

AS3356 132199 180.191.194.0/23AS3356 132199 180.191.196.0/22AS3356 132199 180.191.224.0/23AS3356 132199 180.191.227.0/24AS3356 132199 180.191.228.0/22AS3356 132199 180.191.232.0/22AS3356 132199 180.191.236.0/22AS3356 132199 180.191.240.0/21AS3356 132199 180.191.248.0/22AS3356 132199 180.191.252.0/22AS3356 132199 180.191.32.0/22AS3356 132199 180.191.36.0/22AS3356 132199 180.191.40.0/22AS3356 132199 180.191.44.0/22AS3356 132199 180.191.48.0/24AS3356 132199 180.191.49.0/24AS3356 132199 180.191.50.0/23AS3356 132199 180.191.52.0/22AS3356 132199 180.191.58.0/23AS3356 132199 180.191.60.0/22AS3356 132199 222.127.196.0/24AS3356 138197 103.126.150.0/24AS3356 18190 120.28.15.0/24AS3356 27281 2620:116:800e::/48AS3356 45731 103.20.190.0/24AS36351 36351 185.147.58.0/24AS36351 36351 185.147.59.0/24AS36351 7489 27.100.39.0/24AS38195 134409 2407:c280:ffff::/48AS38220 38220 2403:cc00:4000::/36AS38561 38561 2402:f00::/32AS45177 38220 2403:cc00:4000::/36AS45177 45177 150.107.32.0/23AS45177 45177 150.107.34.0/23AS45280 45280 2402:7e00:0:102::/64AS45280 45280 2402:7e00:10:100::/56

So – what will we drop?AS4826 10214 121.200.32.0/24AS4826 10214 121.200.33.0/24AS4826 133075 2407:f100:4::/48AS4826 13335 2606:4700:7000::/48AS4826 17918 122.252.150.0/24AS4826 17918 122.252.151.0/24AS4826 56030 45.118.190.0/24AS4826 56068 131.203.76.0/23AS4826 56304 131.203.63.0/24AS4826 59256 2401:9cc0:200::/48AS4826 59256 2401:9cc0:300::/48AS4826 9503 2402:6000:106::/48AS4826 9503 2402:6000:109::/48AS4826 9503 2402:6000:10d::/48AS4826 9503 2402:6000:10e::/48AS4826 9503 2402:6000:201::/48AS4826 9503 2402:6000:202::/48AS4826 9790 2404:4400:1000::/36AS4826 9790 2404:4400:2000::/36AS4826 9790 2404:4408:8::/48AS56030 56030 45.118.190.0/24AS58511 134409 2407:c280:ffff::/48AS64098 38220 2403:cc00:4000::/36AS64098 64098 2403:780:f::/48AS7175 7175 2402:c00:2:a00::/56AS7545 137079 103.107.247.0/24AS7545 18405 122.200.160.0/20AS7575 7575 2001:388:70d2::/48AS7600 9297 103.74.188.0/24AS7600 9297 103.74.189.0/24AS9297 9297 103.74.188.0/24AS9297 9297 103.74.189.0/24AS9500 9500 2407:7000:f300::/48AS9790 56068 131.203.76.0/23AS9790 56304 131.203.63.0/24AS9790 56304 2400:3d80:2000:100::/56

AS9790 56304 2400:3d80:2000:200::/56AS9790 56304 2400:3d80:2000:400::/56AS9790 56304 2400:6900:1030:200::/56AS9790 56304 2400:6900:1030:3::/64AS9790 56304 2400:6900:1030:462::/64AS9790 56304 2400:6900:1030:464::/64AS9790 56304 2400:6900:1030:500::/56AS9790 56304 2400:6900:1030:5::/64AS9790 56304 2400:6900:1030:600::/64AS9790 56304 2400:6900:1030:6::/64AS9790 56304 2400:6900:1030:700::/56AS9790 56304 2400:6900:1030:e00::/56AS9790 56304 2400:6900:2000:1000::/56AS9790 56304 2400:6900:2000:1200::/56AS9790 56304 2400:6900:2000:1500::/56AS9790 56304 2400:6900:2000:1800::/56AS9790 56304 2400:6900:2000:1900::/56AS9790 56304 2400:6900:2000:1b00::/56AS9790 56304 2400:6900:2000:1c00::/56AS9790 56304 2400:6900:2000:1d00::/56AS9790 56304 2400:6900:2000:1e00::/56AS9790 56304 2400:6900:2000:1f00::/56AS9790 56304 2400:6900:2000:2300::/56AS9790 56304 2400:6900:2000:300::/56AS9790 56304 2400:6900:2000:3800::/56AS9790 56304 2400:6900:2000:3900::/56AS9790 56304 2400:6900:2000:3a00::/56AS9790 56304 2400:6900:2000:400::/56AS9790 56304 2400:6900:2000:4e00::/56AS9790 56304 2400:6900:2000:500::/56AS9790 56304 2400:6900:2000:600::/56AS9790 56304 2400:6900:2000:800::/56AS9790 56304 2400:6900:2000:900::/56AS9790 56304 2400:6900:2000:a00::/56AS9790 56304 2400:6900:2000:c00::/56AS9790 56304 2400:6900:2000:e00::/56

AS9790 56304 2400:6900:2000:f00::/56AS9790 56304 2400:6900:2030:9::/64AS9790 56304 2400:6900:2030:e::/64AS9790 56304 2400:6900:2:2::/64AS9790 56304 2400:6900:2:3::/64AS9790 56304 2400:6900:2::/64AS9790 56304 2400:6900:2:ff00::/64AS9790 56304 2400:6900:3010:200::/56AS9790 56304 2400:6900:3010:4::/64AS9790 56304 2400:6900:ffff:100::/56AS9790 56304 2400:6900:ffff:10::/60AS9790 56304 2400:6900:ffff:1::/64AS9790 56304 2400:6900:ffff:200::/56AS9790 56304 2400:6900:ffff:2::/64AS9790 56304 2400:6900:ffff:3::/64AS9790 56304 2400:6900:ffff:a::/64AS9790 56304 2400:6900:ffff:b::/64AS9790 56304 2400:6900:ffff:f300::/56AS9790 56304 2400:6900:ffff:f500::/56AS9790 56304 2400:6900:ffff:f::/64AS9790 56304 2400:6900:ffff:fe00::/56AS9790 56304 2400:6900:ffff:fffd::/64AS9790 56304 2400:6900:ffff:fffe::/64AS9790 56304 2401:9480:300:10d::/64AS9790 9503 202.53.182.0/24

Route Server Software Upgrades

• Current Route Server Software• Ubuntu 16.04 LTS – End of Support in April 2021• BIRD 1.6.2 - Released September 2016

• No RPKI / ROV support (without mangling prefix lists – ew)

• New Software• Ubuntu 20.04 LTS – Released April 2020• BIRD 2.0.7 – Released October 2019

• RTR support

• Plan• There are two route servers in each exchange (are you peering with both)?• Pick a time window, disable rs1, upgrade Ubuntu, upgrade bird, deploy automated BIRD config• 2 hours per route server – apt full-upgrade takes ageeeeeees, actual downtime of around 10 minutes for

reboot and software ugprade• Rinse and repeat for second RS• Rinse and repeat for each exchange

Route Server Automation – Current State

• We have an operations portal that is the source of truth• It has a restful json API to query data

• Slack chatops to generate all network and RS configs• errbot• More hacky python scripts• Jinja2 templates

• AS-SET updates are manual still• Tried early alpha automation (cronjob re-generating

some tagged services), lead to some interesting edge cases of broken configs

• Disabled that, and started working on version 2

Route Server Automation – Version 2

• ARouteServer (https://github.com/pierky/arouteserver)• Codify Route Server Config

• YAML clients built from Portal API• Community config defined in git

repo

• Takes ~15 minutes to deploy globally

• Rundeck• Less moving parts than Stackstorm• Runs workflow at 3pm AEST daily• Generates config for all exchanges

using ARouteServer• Test and Deploy Route Server config

via SSH

Route Origin Validation - Tooling

• OctoRPKI (https://github.com/cloudflare/cfrpki#octorpki)• Running on 5 servers, one in each exchange• Generates a json list of Route Object Authorization (ROA) • Performs all crypto validation• Exports ROA as signed curated json list to GoRTR instances

• GoRTR (https://github.com/cloudflare/gortr)• Run locally on each Route Server• Uses json from OctoRPKI sources and streams into special BIRD tables for

validation• Basically no logic and no overhead required, thus running on route server is

acceptable

Route Origin Validation – BIRD Test• BIRD 2.0.7 supports RTR protocol

• Connect to local GoRTR instance & instance running on partner route server

• Validate routes from peers, before importing

• Accept valid and unknown routes

• Drop invalids before importing into master table

Route Origin Validation – Example Config

Route Server Automation and ROV

Nick Pratleynick@ix.asn.auLife Under Lockdown: how to stop heists, hijacks, and hostages