ipv6 topology mapping - cmand · what is the topology of the ipv6 internet? motivation:...

IPv6 Topology Mapping

Robert Beverly*, Ram Durairajan†, Justin Rohrer*, David Plonka‡

∗Naval Postgraduate School†University of Oregon

‡Akamai

April 20, 2018

R. Beverly et al. (NPS/UOregon/Akamai) IPv6 Topology Mapping April 20, 2018 1 / 31

Background

Context

This talk:Really a bunch of slides for informal discussiomPlanned IMC 2018 submissionLooking for feedback/suggestions, especially on:

MotivationAnalysis/metrics/result interpretation

R. Beverly et al. (NPS/UOregon/Akamai) IPv6 Topology Mapping April 20, 2018 2 / 31

Background

Problem

What is the topology of the IPv6 Internet?

Motivation:Understanding Internet topology is important (CDNs, security,resilience, diagnostics, research)IPv6 increasingly important (e.g., by metrics of traffic, enabledsites, edge deployment)But, do we have a good understanding of the IPv6 topology?

R. Beverly et al. (NPS/UOregon/Akamai) IPv6 Topology Mapping April 20, 2018 3 / 31

Background

Prior Work

State-of-the-art:CAIDA has been collecting IPv6 topology maps for 10 yearsEssentially replicates their IPv4 probing methodologyFor every IPv6 prefix in the global BGP table*:

Use scamper (active measurement using ICMP6-Paris)Traceroute to ::1 in prefixTraceroute to random host in prefixTakes approximately 9 hours for < 100k targets (!)

*March 5, 2018: probed 98,120 unique destinations; 1,782destinations probed more than once (one probed six times)

Coverage/Completeness/Efficiency of state-of-art IPv6 topology mapshas not been evaluated

R. Beverly et al. (NPS/UOregon/Akamai) IPv6 Topology Mapping April 20, 2018 4 / 31

Background

Prior Work

State-of-the-art:CAIDA has been collecting IPv6 topology maps for 10 yearsEssentially replicates their IPv4 probing methodologyFor every IPv6 prefix in the global BGP table*:

Use scamper (active measurement using ICMP6-Paris)Traceroute to ::1 in prefixTraceroute to random host in prefixTakes approximately 9 hours for < 100k targets (!)

*March 5, 2018: probed 98,120 unique destinations; 1,782destinations probed more than once (one probed six times)

Coverage/Completeness/Efficiency of state-of-art IPv6 topology mapshas not been evaluated

R. Beverly et al. (NPS/UOregon/Akamai) IPv6 Topology Mapping April 20, 2018 4 / 31

Background

A different approach for IPv6?

Strategies for increasing coverage:Probe more destinationsProbe fasterSelect better destinations

Probing faster:

RFC4443, §2.1.1: “an IPv6 node MUST limit the rate of ICMPv6 errormessages it originates”

Assertion:Existing tools/techniques ill-suited to IPv6 active topology mapping

R. Beverly et al. (NPS/UOregon/Akamai) IPv6 Topology Mapping April 20, 2018 5 / 31

Background

A different approach for IPv6?

Strategies for increasing coverage:Probe more destinationsProbe fasterSelect better destinations

Probing faster:

RFC4443, §2.1.1: “an IPv6 node MUST limit the rate of ICMPv6 errormessages it originates”

Assertion:Existing tools/techniques ill-suited to IPv6 active topology mapping

R. Beverly et al. (NPS/UOregon/Akamai) IPv6 Topology Mapping April 20, 2018 5 / 31

Background

A different approach for IPv6?

Our approach:1 Explore use of recent randomized, high-speed active topology

probing techniques (Yarrp)2 Develop IPv6 version of Yarrp3 Evaluate efficacy of various hitlists, targets, and protocols4 Goal: Produce the most complete IPv6 topology currently

available

R. Beverly et al. (NPS/UOregon/Akamai) IPv6 Topology Mapping April 20, 2018 6 / 31

Methodology

Yarrp

Yarrp: “Yelling at Random Routers Progressively” (IMC2016)

https://www.cmand.org/yarrp/

Uses a block cipher to randomly permute the 〈IP,TTL〉 domainIs stateless, recovering necessary information from repliesBy randomly spreading probes, permits fast Internet-scale activetopology probing

(Runs > 300kpps, discovers > 0.5M ints in < 10min from singleVP)

Hypothesis: Yarrp-mapping of the IPv6 Internet will suffer lessrate-limiting, even at higher probing rates

R. Beverly et al. (NPS/UOregon/Akamai) IPv6 Topology Mapping April 20, 2018 7 / 31

Methodology

Yarrp

Yarrp: “Yelling at Random Routers Progressively” (IMC2016)

https://www.cmand.org/yarrp/

Uses a block cipher to randomly permute the 〈IP,TTL〉 domainIs stateless, recovering necessary information from repliesBy randomly spreading probes, permits fast Internet-scale activetopology probing

(Runs > 300kpps, discovers > 0.5M ints in < 10min from singleVP)

Hypothesis: Yarrp-mapping of the IPv6 Internet will suffer lessrate-limiting, even at higher probing rates

R. Beverly et al. (NPS/UOregon/Akamai) IPv6 Topology Mapping April 20, 2018 7 / 31

Methodology Methodology

Yarrp Features Update

Lots of development since the 2016 IMC yarrp-0.1.

yarrp-0.4:

TCP (SYN or ACK), UDP, or ICMP probesIPv4/IPv6Linux and BSD“Fill mode”Decoupled probing / receivingBiased probingBetter usability / documentation

R. Beverly et al. (NPS/UOregon/Akamai) IPv6 Topology Mapping April 20, 2018 8 / 31

Methodology Methodology

IPv6: Encoding State (new)

RFC4443, §3.3: ICMP6 TTL exceeded quotation includes “As much ofinvoking packet as possible without the ICMPv6 packet exceeding the

minimum IPv6 MTU”

In contrast to IPv4 which only guarantees 28B quoteSignificantly simplifies encoding – place in payload!

R. Beverly et al. (NPS/UOregon/Akamai) IPv6 Topology Mapping April 20, 2018 9 / 31

Methodology Methodology

Yarrp6

IPv6: Encoding State

Ver

Len TTLP=x

Destination IP = target

3216

Send TTL

Send ElapsedTime (ms)

Target IP

Class Flow label

Magic

Elapsed Time

fudge

Source IP = prober

TTL

Fix cksumsrc_port = cksum(target)

Maintain constant transport (TCP/UDP/ICMP) header fieldsEncode TTL, elapsed time in transport payloadUse 2B of “fudge” to correct payload so that checksum is correct

R. Beverly et al. (NPS/UOregon/Akamai) IPv6 Topology Mapping April 20, 2018 10 / 31

Methodology Methodology

Fill Mode

Yarrp is statelessMust select TTL range (maxTTL) (potentially missing hops)Don’t know when to stop probing (potentially wasting probes)(can be beneficial, as we discover hops beyond gap limit)

Fill modeFor response to a probe with TTL=h, probe with TTL=h + 1 iff

h ≥ maxTTL. (not random, but uncommon and at path tail)

Win/win efficiency gain: Allows us to lower the maxTTL (less wastedprobing), without missing hops.

R. Beverly et al. (NPS/UOregon/Akamai) IPv6 Topology Mapping April 20, 2018 11 / 31

Methodology Methodology

SPECK

SPECKUse SPECK lightweight block cipherFaster, and supports intermediate block lengths (e.g., 48 and 96bits – useful for IPv6-wide scanning)

R. Beverly et al. (NPS/UOregon/Akamai) IPv6 Topology Mapping April 20, 2018 12 / 31

Methodology Methodology

Target Selection

Must select IPv6 target addressesFor 128-bit IPv6 address:

How to choose prefix? (Upper 64 bits)How to host identifier? (Lower 64 bits)

R. Beverly et al. (NPS/UOregon/Akamai) IPv6 Topology Mapping April 20, 2018 13 / 31

Methodology Choosing prefixes

Choosing prefixes

Basic strategies

Uniform: probe all possible prefixes (generally infeasible)Random: chose prefixes at random (IPv6 space is too sparse)BGP: Select prefixes in global BGP table (does not capturesubnetting)Hitlists: Select prefixes based on hitlists (utility/comparison ofhitlists has not been previously explored in literature)Generative: Build a model of how prefixes are allocated usingseeds of known addresses, generate new candidate prefixes (hasnot been previously explored in literature)

R. Beverly et al. (NPS/UOregon/Akamai) IPv6 Topology Mapping April 20, 2018 14 / 31

Methodology Choosing prefixes

Hitlists

Name Size MethodCAIDA 93,894 BGP-derivedFiebig 97,746 rDNSRapid7 196,012 fDNS

CDN Clients 372,930 Anonymous aggregatesDNS-DB Varies Farsight

6gen Varies Generative

Lots of recent work on IPv6 hitlist/target generationNo work to compare/understand these hitlists!Note, many hitlists have many addresses within same /64

We reduce these to unique /48s(unlikely to produce interesting topology results)

R. Beverly et al. (NPS/UOregon/Akamai) IPv6 Topology Mapping April 20, 2018 15 / 31

Methodology Choosing prefixes

Hitlists

Fiebig

Intuition: leverage rDNS, i.e., IPv6 PTR recordsRecall, IPv6 PTR records are in the ip6.arpa namespace. Texthex nibbles separated by periods.e.g., 0.0.1.0.0.0.0.0.0.0.0.0.1.0.0.0.1.2.c.0.7.0.f.1.0.7.4.0.1.0.0.2.ip6.arpa.

3600 IN PTR ralph.rbeverly.net.

Trick: recent DNS standard specifies that resolvers respond topartial PTR queries with:

NXDOMAIN: no record and nothing under in the treeNOERROR: children in the tree

Permits (efficient) enumeration of the namespaceAssumes providers populate the portion of the ip6.arpa hierarchythey are authoritative for

R. Beverly et al. (NPS/UOregon/Akamai) IPv6 Topology Mapping April 20, 2018 16 / 31

Methodology Choosing prefixes

Hitlists

Rapid7scans.io performs regular internet-wide IPv4 service scans (e.g.,web servers)Hitlist generated from AAAA queries for all names discovered

R. Beverly et al. (NPS/UOregon/Akamai) IPv6 Topology Mapping April 20, 2018 17 / 31

Methodology Choosing prefixes

Hitlists

CDN clientsObtained in cooperation with AkamaiContains “anonymous aggregates” – prefixes based on the IPv6addresses of clients that query the Akamai CDN platformGrouped in prefix aggregates such that the prefix is more specificthan the BGP prefix, but has enough clients within it so that theyremain anonymous

R. Beverly et al. (NPS/UOregon/Akamai) IPv6 Topology Mapping April 20, 2018 18 / 31

Methodology Choosing prefixes

Hitlists

6gen

Murdock et al. , IMC 2017: “Target Generation for Internet-wideIPv6 Scanning”Takes a set of input addresses (seed), attempts to learn theaddressing structureGenerates candidate addresses

R. Beverly et al. (NPS/UOregon/Akamai) IPv6 Topology Mapping April 20, 2018 19 / 31

Methodology Choosing prefixes

Hitlists

Coverage distribution comparison

0 5000 10000 15000 20000 25000 30000 35000Routed prefix index

0.0

0.2

0.4

0.6

0.8

1.0

Cum

ulat

ive

Frac

tion

of T

arge

ts

CAIDAFiebigRapid7-48Rapid7-64CDN-256CDN-32

Figure: Cumulative targets per routed prefix

R. Beverly et al. (NPS/UOregon/Akamai) IPv6 Topology Mapping April 20, 2018 20 / 31

Methodology Choosing prefixes

Hitlists

Coverage distribution comparison

0 100000 200000 300000 400000 500000/48 subnet index

0.0

0.2

0.4

0.6

0.8

1.0

Cum

ulat

ive

Frac

tion

of T

arge

ts

CAIDAFiebigRapid7-48Rapid7-64CDN-256CDN-32

Figure: Cumulative targets per routed /48

R. Beverly et al. (NPS/UOregon/Akamai) IPv6 Topology Mapping April 20, 2018 20 / 31

Methodology Choosing prefixes

Hitlists

Coverage distribution comparison

0 500000 1000000 1500000 2000000 2500000 3000000/64 subnet index

0.0

0.2

0.4

0.6

0.8

1.0

Cum

ulat

ive

Frac

tion

of T

arge

ts

CAIDAFiebigRapid7-48Rapid7-64CDN-256CDN-32

Figure: Cumulative targets per routed /64

R. Beverly et al. (NPS/UOregon/Akamai) IPv6 Topology Mapping April 20, 2018 20 / 31

Methodology Choosing hosts

Host identifier

Explored two strategiesProbe ::1

Probe 1234:5678:1234:5678

R. Beverly et al. (NPS/UOregon/Akamai) IPv6 Topology Mapping April 20, 2018 21 / 31

Methodology Initial Results

Initial Active IPv6 Probing Results

Vantage Points:

NPSUOregonSwiss IXP

Probed:Different speedsDifferent hitlistsDifferent prefixes/hostsDifferent protocols

R. Beverly et al. (NPS/UOregon/Akamai) IPv6 Topology Mapping April 20, 2018 22 / 31

Methodology Initial Results

Benefit of randomization

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

Fraction

Responsive

Hop

Probing rate (pps)scamper 20pps

yarrp 20ppsscamper 1000pps

yarrp 1000ppsscamper 2000pps

yarrp 2000pps

NPS

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

Fraction

Responsive

Hop

Probing rate (pps)scamper 20pps

yarrp 20ppsscamper 1000pps

yarrp 1000ppsscamper 2000pps

yarrp 2000pps

UOregonSame targets, same vantage pointYarrp outperforms scamper, especially near sourceClearly some hops exhibit different rate-limiting behavior(lax-agg6-lax-hpr3-100g.cenic.net. and(3.be-1.uonet9-gw.uoregon.edu.))

R. Beverly et al. (NPS/UOregon/Akamai) IPv6 Topology Mapping April 20, 2018 23 / 31

Methodology Initial Results

Different Router BehaviorsClearly some hops exhibit different rate-limiting behavior3.be-1.uonet9-gw.uoregon.edu.

Queried owner, is a Cisco ASR9000“It’s going to be replaced with a Juniper MX10003 some time inthe next month or so. We’re not doing anything special beyondACLs and Netflow sampling on those router interfaces”

R. Beverly et al. (NPS/UOregon/Akamai) IPv6 Topology Mapping April 20, 2018 24 / 31

Methodology Initial Results

Probe Type

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

Fraction

Responsive

Hop

Probing rate (pps)ICMP 20ppsUDP 20pps

TCP_ACK 20pps

NPS

0

0.2

0.4

0.6

0.8

1

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

Fraction

Responsive

Hop

Probing rate (pps)ICMP 20ppsUDP 20pps

TCP_ACK 20pps

UOregon

Take-away:Marginal difference; ICMP/UDP most innocuous

R. Beverly et al. (NPS/UOregon/Akamai) IPv6 Topology Mapping April 20, 2018 25 / 31

Methodology Initial Results

Host identifier

ICMP6 response types

CDNtype/code ::1 1234:5678 Fiebig

Hop lim 3598838 (98.1%) 3530329 (98.1%) 380089 (95.8%)No route 26728 (0.7%) 26039 (0.7%) 2454 (0.6%)

Adm prohib 23595 (0.6%) 21080 (0.6%) 1696 (0.4%)Addr unrch 12158 (0.3%) 14781 (0.4%) 2631 (0.7%)Port unrch 5250 (0.1%) 1045 (0.0%) 9084 (2.3%)Reject rte 2577 (0.1%) 6279 (0.2%) 818 (0.2%)

R. Beverly et al. (NPS/UOregon/Akamai) IPv6 Topology Mapping April 20, 2018 26 / 31

Methodology Initial Results

Hitlist evaluation

40254 41282395

45962

631731673373

rapid7

fiebig

caida

IPv6 Hitlist Topo Comparison (Ints)

38613 60972191

53246

436130231908

rapid7

fiebig

caida

IPv6 Hitlist Topo Comparison (Edges)

Take-away:Missing topology in state-of-the-artHitlists are complementary

R. Beverly et al. (NPS/UOregon/Akamai) IPv6 Topology Mapping April 20, 2018 27 / 31

Methodology Initial Results

Hitlist Power

104 105 106 107 108

Probes

0

10000

20000

30000

40000

50000

60000U

niq

ue Inte

rface

sFiebigRapid7CDN.k2566gen.48RandomCAIDA

Again, very complementary behavior

R. Beverly et al. (NPS/UOregon/Akamai) IPv6 Topology Mapping April 20, 2018 28 / 31

Methodology Initial Results

Subnetting

0

0.2

0.4

0.6

0.8

1

1 10 100

CDF

of Prefxes

Unique paths discovered

CAIDARapid7FiebigCDN

How many distinct paths to different targets within same routedprefix?Use as a basic proxy for amount of subnetting

R. Beverly et al. (NPS/UOregon/Akamai) IPv6 Topology Mapping April 20, 2018 29 / 31

Methodology Initial Results

Outstanding questions/work:

How to best evaluate hitlists (counts, graph metrics, subnetting,etc)?Characterize discovered topology (ASes, edge/core, etc)Resolve aliases, determine how many new routers discoveredCombining hitlistsCreating the definitive “topology-of-record” for IPv6

R. Beverly et al. (NPS/UOregon/Akamai) IPv6 Topology Mapping April 20, 2018 30 / 31

Methodology Initial Results

R. Beverly et al. (NPS/UOregon/Akamai) IPv6 Topology Mapping April 20, 2018 31 / 31

Backup Slides

Path diameters

Distribution of interfaces over all VPs

1 2 3 4 5 6 7 8 9 10 11

12

13

14 15

16 17

18

19

20 21

22

23

24 25

26 27

28

29

30 31

32

Trace TTL

100

101

102

103

104

105

Uniq

ue Inte

rface

s

IPv41 2 3 4 5 6 7 8 9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

29

30

31

32

Trace Hop Limit

100

101

102

103

104

Uniq

ue Inte

rface

sIPv6

Interesting skew toward smaller diameters in IPv6Likely due to tunneling (e.g., Hurricane Electric)And relative size of big IPv6 ASes

R. Beverly et al. (NPS/UOregon/Akamai) IPv6 Topology Mapping April 20, 2018 32 / 31

Backup Slides

Biased Probability1 2 3 4 5 6 7 8 9 10 11

12

13

14 15

16 17

18

19

20 21

22

23

24 25

26 27

28

29

30 31

32

Trace TTL

100

101

102

103

104

105

Uniq

ue Inte

rface

s

Given the distribution of routers vs. TTLUse different probability distributions to bias the search

R. Beverly et al. (NPS/UOregon/Akamai) IPv6 Topology Mapping April 20, 2018 33 / 31

Backup Slides

Current Option Summary

OPTIONS:-i, --input Input target file-o, --output Output file (default: output.yrp)-c, --count Probes to issue (default: unlimited)-t, --type Type: ICMP, TCP_SYN, TCP_ACK, UDP, ICMP6, UDP6, TCP6_SYN, TCP6_ACK (default: TCP_ACK)-r, --rate Scan rate in pps (default: 10)-m, --maxttl Maximum TTL (for ip input list only)-v, --verbose verbose (default: off)-F, --fillmode Fill mode maxttl (default: 0)-s, --sequential Scan sequentially (default: random)-n, --neighborhood Neighborhood TTL (default: 0)-b, --bgp BGP table (default: none)-S, --seed Seed (default: random)-p, --port Transport dst port (default: 80)-T, --test Don’t send probes (default: off)-Q, --entire Entire IPv4/IPv6 Internet (default: off)-I, --interface Network interface (required for IPv6)-G, --dstmac MAC of gateway router (default: auto)-M, --srcmac MAC of probing host (default: auto)

R. Beverly et al. (NPS/UOregon/Akamai) IPv6 Topology Mapping April 20, 2018 34 / 31

Backup Slides

Internet-Wide Probing

Forming an IPv6 target4b 44b

/48

32b

2 cipher :: f(prefix)

Use 48-bit Speck block cipherFirst 4-bit nibble fixed (IANA allocation of 0x2)Add 44 bits of cipher to form /48 prefix to probeLower 32 bits a deterministic function of /48 prefixRemaining 4 bits of cipher determine TTL (1-16)

R. Beverly et al. (NPS/UOregon/Akamai) IPv6 Topology Mapping April 20, 2018 35 / 31