comparing ripe atlas and caida ark data using bigquery · stephen strowes | mat wg, ripe 79 |...
TRANSCRIPT
![Page 1: Comparing RIPE Atlas and CAIDA Ark Data using BigQuery · Stephen Strowes | MAT WG, RIPE 79 | 2019-10-17 Comparing RIPE Atlas and CAIDA Ark Data using BigQuery](https://reader035.vdocuments.mx/reader035/viewer/2022070811/5f0a95427e708231d42c5859/html5/thumbnails/1.jpg)
Stephen Strowes <[email protected]> | MAT WG, RIPE 79 | 2019-10-17
Comparing RIPE Atlasand CAIDA Ark Datausing BigQuery
![Page 2: Comparing RIPE Atlas and CAIDA Ark Data using BigQuery · Stephen Strowes | MAT WG, RIPE 79 | 2019-10-17 Comparing RIPE Atlas and CAIDA Ark Data using BigQuery](https://reader035.vdocuments.mx/reader035/viewer/2022070811/5f0a95427e708231d42c5859/html5/thumbnails/2.jpg)
Stephen Strowes <[email protected]> | MAT WG, RIPE79 | 2019-10-17 2
RIPE Atlas• Collect a lot of measurement data
• We provide no general means to filter or analyse that data
![Page 3: Comparing RIPE Atlas and CAIDA Ark Data using BigQuery · Stephen Strowes | MAT WG, RIPE 79 | 2019-10-17 Comparing RIPE Atlas and CAIDA Ark Data using BigQuery](https://reader035.vdocuments.mx/reader035/viewer/2022070811/5f0a95427e708231d42c5859/html5/thumbnails/3.jpg)
Stephen Strowes <[email protected]> | MAT WG, RIPE79 | 2019-10-17 3
CAIDA Ark• https://www.caida.org/projects/ark/
• Goal: topology discovery
![Page 4: Comparing RIPE Atlas and CAIDA Ark Data using BigQuery · Stephen Strowes | MAT WG, RIPE 79 | 2019-10-17 Comparing RIPE Atlas and CAIDA Ark Data using BigQuery](https://reader035.vdocuments.mx/reader035/viewer/2022070811/5f0a95427e708231d42c5859/html5/thumbnails/4.jpg)
Stephen Strowes <[email protected]> | MAT WG, RIPE79 | 2019-10-17 4
Google BigQuery• General platform for data warehousing and analysis
• Feedback loop: “is this useful?”
![Page 5: Comparing RIPE Atlas and CAIDA Ark Data using BigQuery · Stephen Strowes | MAT WG, RIPE 79 | 2019-10-17 Comparing RIPE Atlas and CAIDA Ark Data using BigQuery](https://reader035.vdocuments.mx/reader035/viewer/2022070811/5f0a95427e708231d42c5859/html5/thumbnails/5.jpg)
Stephen Strowes <[email protected]> | MAT WG, RIPE79 | 2019-10-17 5
Google BigQuery• Can manage this much data without flinching
• Few hardware limitations
• Queries are fast, but not real-time- think: O(seconds — minutes), not O(milliseconds)
- responsive enough to iterate during data exploration
• Allows exploration of the full dataset in new ways- ask for results related to ANY measurement
- and you can combine with any other dataset you have access to
![Page 6: Comparing RIPE Atlas and CAIDA Ark Data using BigQuery · Stephen Strowes | MAT WG, RIPE 79 | 2019-10-17 Comparing RIPE Atlas and CAIDA Ark Data using BigQuery](https://reader035.vdocuments.mx/reader035/viewer/2022070811/5f0a95427e708231d42c5859/html5/thumbnails/6.jpg)
Stephen Strowes <[email protected]> | MAT WG, RIPE79 | 2019-10-17 6
Using BigQuery: Ark + Atlas• Two quick views of the data in this talk
- How long it takes each dataset to reveal responsive ASNs
- How long it takes each dataset to reveal AS adjacencies
![Page 12: Comparing RIPE Atlas and CAIDA Ark Data using BigQuery · Stephen Strowes | MAT WG, RIPE 79 | 2019-10-17 Comparing RIPE Atlas and CAIDA Ark Data using BigQuery](https://reader035.vdocuments.mx/reader035/viewer/2022070811/5f0a95427e708231d42c5859/html5/thumbnails/12.jpg)
Stephen Strowes <[email protected]> | MAT WG, RIPE79 | 2019-10-17 12
Looking at Adjacencies• First: build tables of IPn — IP(n+1) adjacencies
• Then: build tables of ASn — AN(n+1) adjacencies
![Page 13: Comparing RIPE Atlas and CAIDA Ark Data using BigQuery · Stephen Strowes | MAT WG, RIPE 79 | 2019-10-17 Comparing RIPE Atlas and CAIDA Ark Data using BigQuery](https://reader035.vdocuments.mx/reader035/viewer/2022070811/5f0a95427e708231d42c5859/html5/thumbnails/13.jpg)
Stephen Strowes <[email protected]> | MAT WG, RIPE79 | 2019-10-17 13
Looking at Adjacencies• This is one place where SQL doesn’t quite cut it
• But, BigQuery permits arbitrary code
![Page 17: Comparing RIPE Atlas and CAIDA Ark Data using BigQuery · Stephen Strowes | MAT WG, RIPE 79 | 2019-10-17 Comparing RIPE Atlas and CAIDA Ark Data using BigQuery](https://reader035.vdocuments.mx/reader035/viewer/2022070811/5f0a95427e708231d42c5859/html5/thumbnails/17.jpg)
Stephen Strowes <[email protected]> | MAT WG, RIPE79 | 2019-10-17 17
Direction of Travel• BigQuery doesn’t solve all problems but it’s extremely powerful
• Ultimately: open up access to (some of?) our data via BigQuery
• We’d like something that looks like the following:- Storage cost, data availability: the NCC’s responsibility
- Computation cost and further storage: on anybody who wants to query the data
![Page 18: Comparing RIPE Atlas and CAIDA Ark Data using BigQuery · Stephen Strowes | MAT WG, RIPE 79 | 2019-10-17 Comparing RIPE Atlas and CAIDA Ark Data using BigQuery](https://reader035.vdocuments.mx/reader035/viewer/2022070811/5f0a95427e708231d42c5859/html5/thumbnails/18.jpg)
Stephen Strowes <[email protected]> | MAT WG, RIPE79 | 2019-10-17 18
Upcoming• Articles on:
- the work required in the BigQuery platform to query this data
- the distinctions between the two datasets