pinger: navigating the web site and mining the data les cottrell slac workshop at the university of...
TRANSCRIPT
![Page 1: PingER: Navigating the web site and mining the data Les Cottrell SLAC Workshop at the University of Malaya (UM), Kuala Lumpur, June 24-25, 2013](https://reader036.vdocuments.mx/reader036/viewer/2022081516/56649ea05503460f94ba2a5a/html5/thumbnails/1.jpg)
PingER: Navigating the web site and mining the data
Les CottrellSLAC
Workshop at the University of Malaya (UM), Kuala Lumpur, June 24-25, 2013
![Page 2: PingER: Navigating the web site and mining the data Les Cottrell SLAC Workshop at the University of Malaya (UM), Kuala Lumpur, June 24-25, 2013](https://reader036.vdocuments.mx/reader036/viewer/2022081516/56649ea05503460f94ba2a5a/html5/thumbnails/2.jpg)
Agenda• Navigating the PingER web site
• Accessing and viewing analyzed data
• Downloading and mining with Excel etc.
• Managing the system
![Page 3: PingER: Navigating the web site and mining the data Les Cottrell SLAC Workshop at the University of Malaya (UM), Kuala Lumpur, June 24-25, 2013](https://reader036.vdocuments.mx/reader036/viewer/2022081516/56649ea05503460f94ba2a5a/html5/thumbnails/3.jpg)
PingER home site• Set bookmark at
http://www-wanmon.slac.stanford.edu/cgi-wrap/pingtable.p
• Prefer Chrome or Firefox, mostly works with MSIE
• Public face ofPingER
![Page 4: PingER: Navigating the web site and mining the data Les Cottrell SLAC Workshop at the University of Malaya (UM), Kuala Lumpur, June 24-25, 2013](https://reader036.vdocuments.mx/reader036/viewer/2022081516/56649ea05503460f94ba2a5a/html5/thumbnails/4.jpg)
Home site menu bar: Introduction• Provides:
– Upper level information– An IEEE publication – Technical details– Etc.
![Page 5: PingER: Navigating the web site and mining the data Les Cottrell SLAC Workshop at the University of Malaya (UM), Kuala Lumpur, June 24-25, 2013](https://reader036.vdocuments.mx/reader036/viewer/2022081516/56649ea05503460f94ba2a5a/html5/thumbnails/5.jpg)
Home page: Visualizations• Click on African
Internet Weather
• Congestion wave
• Monitoring sites/ participants
• Hosts per country by region
![Page 6: PingER: Navigating the web site and mining the data Les Cottrell SLAC Workshop at the University of Malaya (UM), Kuala Lumpur, June 24-25, 2013](https://reader036.vdocuments.mx/reader036/viewer/2022081516/56649ea05503460f94ba2a5a/html5/thumbnails/6.jpg)
Access to PingER site map• Provides site map &
• Geographical deployment maps
PingER site map
![Page 8: PingER: Navigating the web site and mining the data Les Cottrell SLAC Workshop at the University of Malaya (UM), Kuala Lumpur, June 24-25, 2013](https://reader036.vdocuments.mx/reader036/viewer/2022081516/56649ea05503460f94ba2a5a/html5/thumbnails/8.jpg)
8
Home page Icons access
• Pingtable: workhorse of PingER analysis
• Summary table of country to country connections
• Google map of PingER hosts
• Google Explorer
• Motion Metrics Bubble chart
![Page 9: PingER: Navigating the web site and mining the data Les Cottrell SLAC Workshop at the University of Malaya (UM), Kuala Lumpur, June 24-25, 2013](https://reader036.vdocuments.mx/reader036/viewer/2022081516/56649ea05503460f94ba2a5a/html5/thumbnails/9.jpg)
9
Pingtable• This is the workhorse• www.slac.stanford.edu/cgi-wrap/pingtable.pl
UNIMASWorkshop, Sarawak, Dec 2012
HelpTable Form
![Page 10: PingER: Navigating the web site and mining the data Les Cottrell SLAC Workshop at the University of Malaya (UM), Kuala Lumpur, June 24-25, 2013](https://reader036.vdocuments.mx/reader036/viewer/2022081516/56649ea05503460f94ba2a5a/html5/thumbnails/10.jpg)
10
Pingtable: Help
UNIMASWorkshop, Sarawak, Dec 2012
Technical assistance on metrics meanings
etc,
Help on table content etc.,
Coloring by thresholds
What to click on
![Page 11: PingER: Navigating the web site and mining the data Les Cottrell SLAC Workshop at the University of Malaya (UM), Kuala Lumpur, June 24-25, 2013](https://reader036.vdocuments.mx/reader036/viewer/2022081516/56649ea05503460f94ba2a5a/html5/thumbnails/11.jpg)
11
Host labeling
• Host labeled <country>.<site>.. (most important 1st). – Assist in sorting etc.
UNIMASWorkshop, Sarawak, Dec 2012
![Page 12: PingER: Navigating the web site and mining the data Les Cottrell SLAC Workshop at the University of Malaya (UM), Kuala Lumpur, June 24-25, 2013](https://reader036.vdocuments.mx/reader036/viewer/2022081516/56649ea05503460f94ba2a5a/html5/thumbnails/12.jpg)
12
Pingtable: Scroll right for host names etc.
• Get Internet host name, TLD and region for each host pair
UNIMASWorkshop, Sarawak, Dec 2012
![Page 13: PingER: Navigating the web site and mining the data Les Cottrell SLAC Workshop at the University of Malaya (UM), Kuala Lumpur, June 24-25, 2013](https://reader036.vdocuments.mx/reader036/viewer/2022081516/56649ea05503460f94ba2a5a/html5/thumbnails/13.jpg)
13
Pingtable: Form selection
eGY Africa 2012Workshop, Nairobi Oct 2012
![Page 14: PingER: Navigating the web site and mining the data Les Cottrell SLAC Workshop at the University of Malaya (UM), Kuala Lumpur, June 24-25, 2013](https://reader036.vdocuments.mx/reader036/viewer/2022081516/56649ea05503460f94ba2a5a/html5/thumbnails/14.jpg)
14
Pingtable: Metric selection
• Pull down to select from Avg RTT, jitter (ipdv), loss, throughput, MOS etc.
UNIMASWorkshop, Sarawak, Dec 2012
![Page 15: PingER: Navigating the web site and mining the data Les Cottrell SLAC Workshop at the University of Malaya (UM), Kuala Lumpur, June 24-25, 2013](https://reader036.vdocuments.mx/reader036/viewer/2022081516/56649ea05503460f94ba2a5a/html5/thumbnails/15.jpg)
15
Pingtable: Select tick (time window)
UNIMASWorkshop, Sarawak, Dec 2012
• Pull down to select: hourly, daily, monthly, yearly etc.
![Page 16: PingER: Navigating the web site and mining the data Les Cottrell SLAC Workshop at the University of Malaya (UM), Kuala Lumpur, June 24-25, 2013](https://reader036.vdocuments.mx/reader036/viewer/2022081516/56649ea05503460f94ba2a5a/html5/thumbnails/16.jpg)
16
Pingtable: Select From (Monitor(s))
UNIMASWorkshop, Sarawak, Dec 2012
• Pull down to select monitoring node(s): single monitoring host or all monitors in a country or region etc.
![Page 17: PingER: Navigating the web site and mining the data Les Cottrell SLAC Workshop at the University of Malaya (UM), Kuala Lumpur, June 24-25, 2013](https://reader036.vdocuments.mx/reader036/viewer/2022081516/56649ea05503460f94ba2a5a/html5/thumbnails/17.jpg)
17
Pingtable: Select To (monitored host or site)
UNIMASWorkshop, Sarawak, Dec 2012
• Pull down to select To node(s): single monitored host/site or all hosts/sites in a country or region etc.
![Page 18: PingER: Navigating the web site and mining the data Les Cottrell SLAC Workshop at the University of Malaya (UM), Kuala Lumpur, June 24-25, 2013](https://reader036.vdocuments.mx/reader036/viewer/2022081516/56649ea05503460f94ba2a5a/html5/thumbnails/18.jpg)
18
Aggregate all hosts in a site
UNIMASWorkshop, Sarawak, Dec 2012
• Pull down to select whether to aggregate all hosts in a site into one measurement
![Page 19: PingER: Navigating the web site and mining the data Les Cottrell SLAC Workshop at the University of Malaya (UM), Kuala Lumpur, June 24-25, 2013](https://reader036.vdocuments.mx/reader036/viewer/2022081516/56649ea05503460f94ba2a5a/html5/thumbnails/19.jpg)
19
Select packet size
UNIMASWorkshop, Sarawak, Dec 2012
![Page 20: PingER: Navigating the web site and mining the data Les Cottrell SLAC Workshop at the University of Malaya (UM), Kuala Lumpur, June 24-25, 2013](https://reader036.vdocuments.mx/reader036/viewer/2022081516/56649ea05503460f94ba2a5a/html5/thumbnails/20.jpg)
20
Pingtable: Select host type
UNIMASWorkshop, Sarawak, Dec 2012
• Pull down to select type of hosts to display: All, Monitoring, Beacons, Remote
![Page 21: PingER: Navigating the web site and mining the data Les Cottrell SLAC Workshop at the University of Malaya (UM), Kuala Lumpur, June 24-25, 2013](https://reader036.vdocuments.mx/reader036/viewer/2022081516/56649ea05503460f94ba2a5a/html5/thumbnails/21.jpg)
21
Pingtable: other choices
• Change dataset: no longer any other choices
• Data points: removes pairs with little data
• Filter: Removes known anomalous data
UNIMASWorkshop, Sarawak, Dec 2012
![Page 22: PingER: Navigating the web site and mining the data Les Cottrell SLAC Workshop at the University of Malaya (UM), Kuala Lumpur, June 24-25, 2013](https://reader036.vdocuments.mx/reader036/viewer/2022081516/56649ea05503460f94ba2a5a/html5/thumbnails/22.jpg)
22
Pingtable: Load selected data
UNIMASWorkshop, Sarawak, Dec 2012
• Click on Load into table and wait while data loads and display is rendered
![Page 23: PingER: Navigating the web site and mining the data Les Cottrell SLAC Workshop at the University of Malaya (UM), Kuala Lumpur, June 24-25, 2013](https://reader036.vdocuments.mx/reader036/viewer/2022081516/56649ea05503460f94ba2a5a/html5/thumbnails/23.jpg)
23
Pingtable: Active links
UNIMASWorkshop, Sarawak, Dec 2012
![Page 24: PingER: Navigating the web site and mining the data Les Cottrell SLAC Workshop at the University of Malaya (UM), Kuala Lumpur, June 24-25, 2013](https://reader036.vdocuments.mx/reader036/viewer/2022081516/56649ea05503460f94ba2a5a/html5/thumbnails/24.jpg)
24
Pingtable: Sorting
• Click on column heading:– Sorts by host name, host type (? Monitor|Beacon|
remote), metric value, TLD, Region
UNIMASWorkshop, Sarawak, Dec 2012
![Page 25: PingER: Navigating the web site and mining the data Les Cottrell SLAC Workshop at the University of Malaya (UM), Kuala Lumpur, June 24-25, 2013](https://reader036.vdocuments.mx/reader036/viewer/2022081516/56649ea05503460f94ba2a5a/html5/thumbnails/25.jpg)
25
Pingtable ?• Click ? for more info
– Details on hosts, – Latest raw data,
• Format (see http://www-iepm.slac.stanford.edu/pinger/tools/retrievedata.html#rawdataformat)
– Summarized data
UNIMASWorkshop, Sarawak, Dec 2012
![Page 26: PingER: Navigating the web site and mining the data Les Cottrell SLAC Workshop at the University of Malaya (UM), Kuala Lumpur, June 24-25, 2013](https://reader036.vdocuments.mx/reader036/viewer/2022081516/56649ea05503460f94ba2a5a/html5/thumbnails/26.jpg)
26
Pingtable (?): More information on pair • Select ‘From’ as EDU.SLAC.STANFORD.N3 &
Load into Table
• Click on ?
• Provides info from database on hosts, e.g.:– Lat/lon, location, web site, comments etc.
UNIMASWorkshop, Sarawak, Dec 2012
![Page 27: PingER: Navigating the web site and mining the data Les Cottrell SLAC Workshop at the University of Malaya (UM), Kuala Lumpur, June 24-25, 2013](https://reader036.vdocuments.mx/reader036/viewer/2022081516/56649ea05503460f94ba2a5a/html5/thumbnails/27.jpg)
27
Pingtable: Rawdata
• Useful for seeing exactly when a host disappears or is reachable again by PingER , e.g. in case of events such as loss of Internet connectivity due to earthquakes, cable cuts, uprisings
UNIMASWorkshop, Sarawak, Dec 2012
![Page 28: PingER: Navigating the web site and mining the data Les Cottrell SLAC Workshop at the University of Malaya (UM), Kuala Lumpur, June 24-25, 2013](https://reader036.vdocuments.mx/reader036/viewer/2022081516/56649ea05503460f94ba2a5a/html5/thumbnails/28.jpg)
28
Pingtable(?): Summarized data
• Provides summary statistics (min, avg, median max) for all chosen remote hosts seen from monitor, for 100 &1000 Byte pings
• Plus the slope of Bytes vs. RTT
UNIMASWorkshop, Sarawak, Dec 2012
Latest summarized data from (pinger.slac.stanford.edu): click here
![Page 29: PingER: Navigating the web site and mining the data Les Cottrell SLAC Workshop at the University of Malaya (UM), Kuala Lumpur, June 24-25, 2013](https://reader036.vdocuments.mx/reader036/viewer/2022081516/56649ea05503460f94ba2a5a/html5/thumbnails/29.jpg)
29
Graphs of usage (via ?)
• Time series plot of RTT & Loss last 7 days including today so far from MY.UNIMAS.N4(pinger.unimas.my) to MY.UMT.EDU.N1(www.umt.edu.my):
click here, add csv file.
UNIMASWorkshop, Sarawak, Dec 2012
100% loss =unreachable
![Page 30: PingER: Navigating the web site and mining the data Les Cottrell SLAC Workshop at the University of Malaya (UM), Kuala Lumpur, June 24-25, 2013](https://reader036.vdocuments.mx/reader036/viewer/2022081516/56649ea05503460f94ba2a5a/html5/thumbnails/30.jpg)
30
Graphs of RTT• Valuable if do not have Smokeping graphs, e.g.
for non SLAC monitors
• One day I will fix the tick marks.
• Also add csv file and cut and paste into Excel for more elegant plots
UNIMASWorkshop, Sarawak, Dec 2012
![Page 31: PingER: Navigating the web site and mining the data Les Cottrell SLAC Workshop at the University of Malaya (UM), Kuala Lumpur, June 24-25, 2013](https://reader036.vdocuments.mx/reader036/viewer/2022081516/56649ea05503460f94ba2a5a/html5/thumbnails/31.jpg)
31
Pingtable: (?) Dynamic on demand traceoute
• Uses traceroute server available at most PingER monitors
• Click on
UNIMASWorkshop, Sarawak, Dec 2012
Current traceroute using http://www-wanmon.slac.stanford.edu/cgi-bin/nph-traceroute.pl?target=www.uda.ad
![Page 32: PingER: Navigating the web site and mining the data Les Cottrell SLAC Workshop at the University of Malaya (UM), Kuala Lumpur, June 24-25, 2013](https://reader036.vdocuments.mx/reader036/viewer/2022081516/56649ea05503460f94ba2a5a/html5/thumbnails/32.jpg)
32
Traceroutes• Measured once per day
UNIMASWorkshop, Sarawak, Dec 2012
![Page 33: PingER: Navigating the web site and mining the data Les Cottrell SLAC Workshop at the University of Malaya (UM), Kuala Lumpur, June 24-25, 2013](https://reader036.vdocuments.mx/reader036/viewer/2022081516/56649ea05503460f94ba2a5a/html5/thumbnails/33.jpg)
33
Traceroutes: Form
• Can select 3 days to view concurrently to look for important changes• Also can select the remote host• Currently only available for SLAC monitoring host
UNIMASWorkshop, Sarawak, Dec 2012
![Page 34: PingER: Navigating the web site and mining the data Les Cottrell SLAC Workshop at the University of Malaya (UM), Kuala Lumpur, June 24-25, 2013](https://reader036.vdocuments.mx/reader036/viewer/2022081516/56649ea05503460f94ba2a5a/html5/thumbnails/34.jpg)
34
Traceroute: format
UNIMASWorkshop, Sarawak, Dec 2012
Traceroute for First selected datewww-wanmon.slac.stanford.edu_to_www.uda.ad_2012_11_30 1 rtr-servcore1-serv01-webserv.slac.stanford.edu (134.79.197.130) 0.452 ms2 rtr-core2-p2p-servcore1.slac.stanford.edu (134.79.252.162) 0.336 ms3 rtr-border1-p2p-core2.slac.stanford.edu (134.79.252.137) 0.388 ms4 slac-mr2-p2p-rtr-border1.slac.stanford.edu (192.68.191.245) 0.284 ms5 sunnsdn2-ip-slacmr2.es.net (134.55.217.2) 0.622 ms6 sunncr1-sunnsdn2.es.net (134.55.209.98) 0.714 ms7 eqxsjrt1-te-sunncr1.es.net (134.55.38.146) 1.138 ms8 *9 xe-4-1-0.par72.ip4.tinet.net (89.149.181.146) 151.131 ms10 interoute-gw.ip4.tinet.net (77.67.75.238) 209.653 ms11 ae1-0.mad-001-score-1-re1.interoute.net (89.202.161.21) 178.515 ms12 ae0-0.mad-001-score-2-re0.interoute.net (89.202.161.18) 178.086 ms13 ae1-0.bcn-002-score-1-re1.interoute.net (217.118.118.237) 178.303 ms14 (195.81.193.146) 176.546 ms15 *16 *
Traceroute for First selected datewww-wanmon.slac.stanford.edu_to_www.uda.ad_2012_11_30 1 rtr-servcore1-serv01-webserv.slac.stanford.edu (134.79.197.130) 0.452 ms2 rtr-core2-p2p-servcore1.slac.stanford.edu (134.79.252.162) 0.336 ms3 rtr-border1-p2p-core2.slac.stanford.edu (134.79.252.137) 0.388 ms4 slac-mr2-p2p-rtr-border1.slac.stanford.edu (192.68.191.245) 0.284 ms5 sunnsdn2-ip-slacmr2.es.net (134.55.217.2) 0.622 ms6 sunncr1-sunnsdn2.es.net (134.55.209.98) 0.714 ms7 eqxsjrt1-te-sunncr1.es.net (134.55.38.146) 1.138 ms8 *9 xe-4-1-0.par72.ip4.tinet.net (89.149.181.146) 151.131 ms10 interoute-gw.ip4.tinet.net (77.67.75.238) 209.653 ms11 ae1-0.mad-001-score-1-re1.interoute.net (89.202.161.21) 178.515 ms12 ae0-0.mad-001-score-2-re0.interoute.net (89.202.161.18) 178.086 ms13 ae1-0.bcn-002-score-1-re1.interoute.net (217.118.118.237) 178.303 ms14 (195.81.193.146) 176.546 ms15 *16 *
![Page 35: PingER: Navigating the web site and mining the data Les Cottrell SLAC Workshop at the University of Malaya (UM), Kuala Lumpur, June 24-25, 2013](https://reader036.vdocuments.mx/reader036/viewer/2022081516/56649ea05503460f94ba2a5a/html5/thumbnails/35.jpg)
35
More Traceroutes
• Background information on traceroute servers– http://www.slac.stanford.edu/comp/net/wan-mon/
traceroute-srv.html
• List of servers by country:– http://www.traceroute.org/
UNIMASWorkshop, Sarawak, Dec 2012
![Page 36: PingER: Navigating the web site and mining the data Les Cottrell SLAC Workshop at the University of Malaya (UM), Kuala Lumpur, June 24-25, 2013](https://reader036.vdocuments.mx/reader036/viewer/2022081516/56649ea05503460f94ba2a5a/html5/thumbnails/36.jpg)
36
PingER traceroute servers API• Most PingER installations have traceroute
servers.
• Perl API and web accessible xml file for database can be found by accessing file http://www-iepm.slac.stanford.edu/pinger/pingerworld/nodes.cf, see comment at start– This can be used to find out about PingER hosts &
in particular PingER traceroute servers
![Page 37: PingER: Navigating the web site and mining the data Les Cottrell SLAC Workshop at the University of Malaya (UM), Kuala Lumpur, June 24-25, 2013](https://reader036.vdocuments.mx/reader036/viewer/2022081516/56649ea05503460f94ba2a5a/html5/thumbnails/37.jpg)
37
Pingtable: Smokeping• Click on the SLAC monitoring host in a line
– Not available for other monitors
UNIMASWorkshop, Sarawak, Dec 2012
![Page 38: PingER: Navigating the web site and mining the data Les Cottrell SLAC Workshop at the University of Malaya (UM), Kuala Lumpur, June 24-25, 2013](https://reader036.vdocuments.mx/reader036/viewer/2022081516/56649ea05503460f94ba2a5a/html5/thumbnails/38.jpg)
38
Pingtable: Smokeping graph • Background is colored for loss (see legend)
– Black not reachable– Red 50%-90% loss– Magenta 20%-40% loss– Orange 5%-10% loss– Light yellow 0%-5% loss– Cyan 0% loss
UNIMASWorkshop, Sarawak, Dec 2012
![Page 39: PingER: Navigating the web site and mining the data Les Cottrell SLAC Workshop at the University of Malaya (UM), Kuala Lumpur, June 24-25, 2013](https://reader036.vdocuments.mx/reader036/viewer/2022081516/56649ea05503460f94ba2a5a/html5/thumbnails/39.jpg)
39
Pingtable: Smokeping
UNIMASWorkshop, Sarawak, Dec 2012
Blue line average RTT, Smoke=jitterX axis in GMTY axis in seconds (m = msec)Start and end times given in legendSummary statistics given in legend
![Page 40: PingER: Navigating the web site and mining the data Les Cottrell SLAC Workshop at the University of Malaya (UM), Kuala Lumpur, June 24-25, 2013](https://reader036.vdocuments.mx/reader036/viewer/2022081516/56649ea05503460f94ba2a5a/html5/thumbnails/40.jpg)
40
Pingtable: Smokeping graph Form
• Allows time window selection (also icons)• Size of graph, manual scaling• Colors for losses
UNIMASWorkshop, Sarawak, Dec 2012
![Page 41: PingER: Navigating the web site and mining the data Les Cottrell SLAC Workshop at the University of Malaya (UM), Kuala Lumpur, June 24-25, 2013](https://reader036.vdocuments.mx/reader036/viewer/2022081516/56649ea05503460f94ba2a5a/html5/thumbnails/41.jpg)
41
Pingtable:Summary table• At the bottom provides:
– Summary stats (min, median, 95%, # pairs, etc.) for each time interval for chosen metric
– Can click on columns heading to download statistic
UNIMASWorkshop, Sarawak, Dec 2012
![Page 42: PingER: Navigating the web site and mining the data Les Cottrell SLAC Workshop at the University of Malaya (UM), Kuala Lumpur, June 24-25, 2013](https://reader036.vdocuments.mx/reader036/viewer/2022081516/56649ea05503460f94ba2a5a/html5/thumbnails/42.jpg)
42
Pingtable.pl extract summary• Summary table at end (e.g. access via wget)
- Buried in the output from pingtable.pl is a comment line containing the summary data, forma (… = more of same)t:–<!--?+Oct2007+...+Nov2005,+0.141+...+737.923,,-->–<!--?+1998+...+2010,.+17.1429+...+35.7714+32.9469,-->–Each line of data is separated by a comma (,) –within a line each token is separated by a +. –The lines in turn are those from the 'Summary for Sites' table at the bottom of pingtable.pl output.–The first line gives the date, the second the min, the 3rd the 25%, the 4th the average, the 5th (i.e. \$lines[4]) line the median etc.–If there is no data for a token then there is an adjacent ++.
UNIMASWorkshop, Sarawak, Dec 2012
![Page 43: PingER: Navigating the web site and mining the data Les Cottrell SLAC Workshop at the University of Malaya (UM), Kuala Lumpur, June 24-25, 2013](https://reader036.vdocuments.mx/reader036/viewer/2022081516/56649ea05503460f94ba2a5a/html5/thumbnails/43.jpg)
43
Download Pingtable data
• First choose some simple, interesting data:
• Select From = Malaysia
• Select To = Malaysia
• Metric = Average RTT
• Tick-type = last 60 days
UNIMASWorkshop, Sarawak, Dec 2012
![Page 44: PingER: Navigating the web site and mining the data Les Cottrell SLAC Workshop at the University of Malaya (UM), Kuala Lumpur, June 24-25, 2013](https://reader036.vdocuments.mx/reader036/viewer/2022081516/56649ea05503460f94ba2a5a/html5/thumbnails/44.jpg)
44
Pingtable: Downloading data & further analysis
• Click here
UNIMASWorkshop, Sarawak, Dec 2012
This report is also provided in downloadable tab-separated-value (.tsv) format for use with Excel.
Firefox: box pops up, choose save. It should be in Downloads. Can try Start and Search for it
MSIE: box pops up, choose ‘save as’ and save on the desktop
Chrome: Appears at bottom left of browser window. Drag it to desktop
![Page 45: PingER: Navigating the web site and mining the data Les Cottrell SLAC Workshop at the University of Malaya (UM), Kuala Lumpur, June 24-25, 2013](https://reader036.vdocuments.mx/reader036/viewer/2022081516/56649ea05503460f94ba2a5a/html5/thumbnails/45.jpg)
45
Open downloaded file with Excel
• Right click the pingtable.pl icon (probably on desktop).
• Choose “Open with” => “Choose Microsoft Excel”.
• Open the file
UNIMASWorkshop, Sarawak, Dec 2012
Dots = no data
![Page 46: PingER: Navigating the web site and mining the data Les Cottrell SLAC Workshop at the University of Malaya (UM), Kuala Lumpur, June 24-25, 2013](https://reader036.vdocuments.mx/reader036/viewer/2022081516/56649ea05503460f94ba2a5a/html5/thumbnails/46.jpg)
46
Preamble to Excel analysis
• Insert the URL where in a newline at the start
• Delete columns A & C
• Select all (e.g. click on top left hand corner of table)
• Click on Replace in Home menu
• Replace all cells with only a period with a null
• Fix the date: take the Left most add in Excel format, the take the cell in next column to the right set =left one -1 and replicate right
UNIMASWorkshop, Sarawak, Dec 2012
![Page 47: PingER: Navigating the web site and mining the data Les Cottrell SLAC Workshop at the University of Malaya (UM), Kuala Lumpur, June 24-25, 2013](https://reader036.vdocuments.mx/reader036/viewer/2022081516/56649ea05503460f94ba2a5a/html5/thumbnails/47.jpg)
47
Draw chart
• Select relevant data (e.g. a2:bf13)
• Click on Scatter chart under Insert Tab
• Now it’s a matter of improving the format to better understand the data
UNIMASWorkshop, Sarawak, Dec 2012
![Page 48: PingER: Navigating the web site and mining the data Les Cottrell SLAC Workshop at the University of Malaya (UM), Kuala Lumpur, June 24-25, 2013](https://reader036.vdocuments.mx/reader036/viewer/2022081516/56649ea05503460f94ba2a5a/html5/thumbnails/48.jpg)
48
Icons access
• Table of country to country connections
• Maps of metric performance by Country
• Maps of number of PingER hosts by country
• Google map of PingER hosts
• Motion Metrics Bubble chartUNIMAS
Workshop, Sarawak, Dec 2012
![Page 49: PingER: Navigating the web site and mining the data Les Cottrell SLAC Workshop at the University of Malaya (UM), Kuala Lumpur, June 24-25, 2013](https://reader036.vdocuments.mx/reader036/viewer/2022081516/56649ea05503460f94ba2a5a/html5/thumbnails/49.jpg)
49
PingER Summary Table• URL= www.slac.stanford.edu/cgi-wrap/table.pl
• Patience takes a long time
UNIMASWorkshop, Sarawak, Dec 2012
Help
Choose area aggregation
Metric
Month
![Page 50: PingER: Navigating the web site and mining the data Les Cottrell SLAC Workshop at the University of Malaya (UM), Kuala Lumpur, June 24-25, 2013](https://reader036.vdocuments.mx/reader036/viewer/2022081516/56649ea05503460f94ba2a5a/html5/thumbnails/50.jpg)
50
Drilling down• Colors represent quality
• Click on headings & values to drill down to pingtable
UNIMASWorkshop, Sarawak, Dec 2012
![Page 51: PingER: Navigating the web site and mining the data Les Cottrell SLAC Workshop at the University of Malaya (UM), Kuala Lumpur, June 24-25, 2013](https://reader036.vdocuments.mx/reader036/viewer/2022081516/56649ea05503460f94ba2a5a/html5/thumbnails/51.jpg)
51
Pinger Host per country
• URL: www-iepm.slac.stanford.edu/pinger/geomaps/pinger-node-count-geomap.html
• Scroll down for other regions• Mouseover country for node count
UNIMASWorkshop, Sarawak, Dec 2012
Can choose monitors or all nodes
![Page 52: PingER: Navigating the web site and mining the data Les Cottrell SLAC Workshop at the University of Malaya (UM), Kuala Lumpur, June 24-25, 2013](https://reader036.vdocuments.mx/reader036/viewer/2022081516/56649ea05503460f94ba2a5a/html5/thumbnails/52.jpg)
52
Map
• Map– See monitors, beacons, remotes– Select metric, src, dst, draw colored lines
connecting, colors = metric value– Graphs of metric and pinger hosts per country
• Does not work in MSIE
UNIMASWorkshop, Sarawak, Dec 2012
![Page 53: PingER: Navigating the web site and mining the data Les Cottrell SLAC Workshop at the University of Malaya (UM), Kuala Lumpur, June 24-25, 2013](https://reader036.vdocuments.mx/reader036/viewer/2022081516/56649ea05503460f94ba2a5a/html5/thumbnails/53.jpg)
53
Map• URL:
www.slac.stanford.edu/wan-mon/viper/pinger-coverage-gmap.html
UNIMASWorkshop, Sarawak, Dec 2012
Help
More information
![Page 54: PingER: Navigating the web site and mining the data Les Cottrell SLAC Workshop at the University of Malaya (UM), Kuala Lumpur, June 24-25, 2013](https://reader036.vdocuments.mx/reader036/viewer/2022081516/56649ea05503460f94ba2a5a/html5/thumbnails/54.jpg)
54
Map help
UNIMASWorkshop, Sarawak, Dec 2012
![Page 55: PingER: Navigating the web site and mining the data Les Cottrell SLAC Workshop at the University of Malaya (UM), Kuala Lumpur, June 24-25, 2013](https://reader036.vdocuments.mx/reader036/viewer/2022081516/56649ea05503460f94ba2a5a/html5/thumbnails/55.jpg)
55
Map More information• Mainly for developers
UNIMASWorkshop, Sarawak, Dec 2012
![Page 56: PingER: Navigating the web site and mining the data Les Cottrell SLAC Workshop at the University of Malaya (UM), Kuala Lumpur, June 24-25, 2013](https://reader036.vdocuments.mx/reader036/viewer/2022081516/56649ea05503460f94ba2a5a/html5/thumbnails/56.jpg)
56
Map: choose Monitors
UNIMASWorkshop, Sarawak, Dec 2012
Selected only Monitors
![Page 57: PingER: Navigating the web site and mining the data Les Cottrell SLAC Workshop at the University of Malaya (UM), Kuala Lumpur, June 24-25, 2013](https://reader036.vdocuments.mx/reader036/viewer/2022081516/56649ea05503460f94ba2a5a/html5/thumbnails/57.jpg)
57
Map: Beacons
• Beacons are remote hosts monitored by all monitors
UNIMASWorkshop, Sarawak, Dec 2012
Selected only Monitors
![Page 58: PingER: Navigating the web site and mining the data Les Cottrell SLAC Workshop at the University of Malaya (UM), Kuala Lumpur, June 24-25, 2013](https://reader036.vdocuments.mx/reader036/viewer/2022081516/56649ea05503460f94ba2a5a/html5/thumbnails/58.jpg)
58
Map: Remote hosts
UNIMASWorkshop, Sarawak, Dec 2012
Selected only remotes
![Page 59: PingER: Navigating the web site and mining the data Les Cottrell SLAC Workshop at the University of Malaya (UM), Kuala Lumpur, June 24-25, 2013](https://reader036.vdocuments.mx/reader036/viewer/2022081516/56649ea05503460f94ba2a5a/html5/thumbnails/59.jpg)
59
Map: Controls
UNIMASWorkshop, Sarawak, Dec 2012
![Page 60: PingER: Navigating the web site and mining the data Les Cottrell SLAC Workshop at the University of Malaya (UM), Kuala Lumpur, June 24-25, 2013](https://reader036.vdocuments.mx/reader036/viewer/2022081516/56649ea05503460f94ba2a5a/html5/thumbnails/60.jpg)
60
Map: Locate Pakistan
UNIMASWorkshop, Sarawak, Dec 2012
![Page 61: PingER: Navigating the web site and mining the data Les Cottrell SLAC Workshop at the University of Malaya (UM), Kuala Lumpur, June 24-25, 2013](https://reader036.vdocuments.mx/reader036/viewer/2022081516/56649ea05503460f94ba2a5a/html5/thumbnails/61.jpg)
61
Map: Locate Kota Samarahan
UNIMASWorkshop, Sarawak, Dec 2012
UMIMAS
![Page 62: PingER: Navigating the web site and mining the data Les Cottrell SLAC Workshop at the University of Malaya (UM), Kuala Lumpur, June 24-25, 2013](https://reader036.vdocuments.mx/reader036/viewer/2022081516/56649ea05503460f94ba2a5a/html5/thumbnails/62.jpg)
62
Map: Locate UNIMAS
UNIMASWorkshop, Sarawak, Dec 2012
UNIMAS
![Page 63: PingER: Navigating the web site and mining the data Les Cottrell SLAC Workshop at the University of Malaya (UM), Kuala Lumpur, June 24-25, 2013](https://reader036.vdocuments.mx/reader036/viewer/2022081516/56649ea05503460f94ba2a5a/html5/thumbnails/63.jpg)
63
Map: plot monitored links
UNIMASWorkshop, Sarawak, Dec 2012
![Page 64: PingER: Navigating the web site and mining the data Les Cottrell SLAC Workshop at the University of Malaya (UM), Kuala Lumpur, June 24-25, 2013](https://reader036.vdocuments.mx/reader036/viewer/2022081516/56649ea05503460f94ba2a5a/html5/thumbnails/64.jpg)
64
Map plot time series last 365 days
UNIMASWorkshop, Sarawak, Dec 2012
Mouseover points to
get coordinates
Click on graph
![Page 65: PingER: Navigating the web site and mining the data Les Cottrell SLAC Workshop at the University of Malaya (UM), Kuala Lumpur, June 24-25, 2013](https://reader036.vdocuments.mx/reader036/viewer/2022081516/56649ea05503460f94ba2a5a/html5/thumbnails/65.jpg)
65
Demo and Video• Interactive demonstrations of the data mining
capabilities of public data sources provided by organizations such as the UN and ITU coupled with monitoring data from PingER
• http://www-iepm.slac.stanford.edu/pinger/explorer.html
Summer Joint TechsStanford, July 2012
![Page 66: PingER: Navigating the web site and mining the data Les Cottrell SLAC Workshop at the University of Malaya (UM), Kuala Lumpur, June 24-25, 2013](https://reader036.vdocuments.mx/reader036/viewer/2022081516/56649ea05503460f94ba2a5a/html5/thumbnails/66.jpg)
66
Video of motion metric bubble charts
• URL: – www-iepm.slac.stanford.edu/pinger/motion chart/
• 5.5 minutes
UNIMASWorkshop, Sarawak, Dec 2012
![Page 67: PingER: Navigating the web site and mining the data Les Cottrell SLAC Workshop at the University of Malaya (UM), Kuala Lumpur, June 24-25, 2013](https://reader036.vdocuments.mx/reader036/viewer/2022081516/56649ea05503460f94ba2a5a/html5/thumbnails/67.jpg)
67
Managing• Site map
– Introductions
• Meta database– Oracle database => Perl require script and
pinger.xml files
• Data gathering status• Measurements:
– Run from cronjob each 30 mins
• Spotting anomalies, hosts with same IP addr, hosts with missing information (country, lat/lon
• Finding hosts• Finding location
eGY Africa 2012Workshop, Nairobi Oct 2012
![Page 68: PingER: Navigating the web site and mining the data Les Cottrell SLAC Workshop at the University of Malaya (UM), Kuala Lumpur, June 24-25, 2013](https://reader036.vdocuments.mx/reader036/viewer/2022081516/56649ea05503460f94ba2a5a/html5/thumbnails/68.jpg)
68
Problems• Hosts change IP address
– Maybe v frequent, e.g. a cluster or slowly as host with same name is replaced with a new host with new IP address.
• Hosts not where you think they are
• No packet loss how to calculate Throughput ~ 1460(bytes)*8(bits)/(RTT*sqrt(loss))
• Throughput approx. only good for TCP Reno– OS’ such as Windows, Linux now allow other
congestion control algorithms
eGY Africa 2012Workshop, Nairobi Oct 2012
![Page 69: PingER: Navigating the web site and mining the data Les Cottrell SLAC Workshop at the University of Malaya (UM), Kuala Lumpur, June 24-25, 2013](https://reader036.vdocuments.mx/reader036/viewer/2022081516/56649ea05503460f94ba2a5a/html5/thumbnails/69.jpg)
69
Joining• We need several monitoring hosts in Malaysia
and S.E. Asia
• Remove any UNIMAScaused anomalies
• Detailed study of MalayE2E Internet performance
• Trilateration researchplatform
• From PingER home pagepull down joining and get started
UNIMASWorkshop, Sarawak, Dec 2012
![Page 70: PingER: Navigating the web site and mining the data Les Cottrell SLAC Workshop at the University of Malaya (UM), Kuala Lumpur, June 24-25, 2013](https://reader036.vdocuments.mx/reader036/viewer/2022081516/56649ea05503460f94ba2a5a/html5/thumbnails/70.jpg)
70
Joining – Remote Hosts • Only monitored
• No need to do anything
• Host has to respond to pings & be reliable– Lightly or consistently loaded
• Finding hosts – Contacts– Developed HostSearcher
eGY Africa 2012Workshop, Nairobi Oct 2012
![Page 71: PingER: Navigating the web site and mining the data Les Cottrell SLAC Workshop at the University of Malaya (UM), Kuala Lumpur, June 24-25, 2013](https://reader036.vdocuments.mx/reader036/viewer/2022081516/56649ea05503460f94ba2a5a/html5/thumbnails/71.jpg)
71
HostSearcher.pl• Gets list of <1000 hosts in TLD from Google
– Options to only select www or .edu etc.
• Checks if pingable
• Uses GeoIPTools (www.geoiptools.com) to get location
• Manually go to web site – Look for street address, city etc., (contacts)
• If well known site try latitude & longitude finder– http://www.satsig.net/maps/lat-long-finder.htm
• Choose based on how representative it is – Proximity to other hosts etc. UNIMAS
Workshop, Sarawak, Dec 2012
![Page 72: PingER: Navigating the web site and mining the data Les Cottrell SLAC Workshop at the University of Malaya (UM), Kuala Lumpur, June 24-25, 2013](https://reader036.vdocuments.mx/reader036/viewer/2022081516/56649ea05503460f94ba2a5a/html5/thumbnails/72.jpg)
72
Monitoring Hosts
• 80 working monitors worldwide
• Invitation letter for monitoring sites– www-iepm.slac.stanford.edu/pinger/letters/invite-monitor.doc– Describes PingER & its goals– Gives information on public access to results– Provides requirements of monitoring host– Tells how to get started (download and install)
UNIMASWorkshop, Sarawak, Dec 2012
![Page 73: PingER: Navigating the web site and mining the data Les Cottrell SLAC Workshop at the University of Malaya (UM), Kuala Lumpur, June 24-25, 2013](https://reader036.vdocuments.mx/reader036/viewer/2022081516/56649ea05503460f94ba2a5a/html5/thumbnails/73.jpg)
73
Monitoring Host requirements• Responds to pings, known location, contact
• In DNS
• Linux and usual utilities (dig, mail, make Perl +XML::Simple
• Web server (e.g. Apache)
• Lightly loaded
UNIMASWorkshop, Sarawak, Dec 2012
![Page 74: PingER: Navigating the web site and mining the data Les Cottrell SLAC Workshop at the University of Malaya (UM), Kuala Lumpur, June 24-25, 2013](https://reader036.vdocuments.mx/reader036/viewer/2022081516/56649ea05503460f94ba2a5a/html5/thumbnails/74.jpg)
74
First install Reverse Traceroute server
UNIMASWorkshop, Sarawak, Dec 2012
• Self contained Perl CGI for web server – Supports IPv4 & IPv6
• Instructions:– www.slac.stanford.edu/comp/net/wan-mon/Pinger2/
install2.html– Appears at: http://ip_name/cgi-bin/traceroute.pl
• E.g. http://pinger.cern.ch/cgi-bin/traceroute.pl
• Responds to requests for traceroutes from traceroute server to a target host
• Also a ping and tracepath server• Installed at most PingER monitor sites
– www-wanmon.slac.stanford.edu/cgi-wrap/reflector.cgi?PE=set&function=landmarks
![Page 75: PingER: Navigating the web site and mining the data Les Cottrell SLAC Workshop at the University of Malaya (UM), Kuala Lumpur, June 24-25, 2013](https://reader036.vdocuments.mx/reader036/viewer/2022081516/56649ea05503460f94ba2a5a/html5/thumbnails/75.jpg)
75
Also is a ping server
• icfamon.dl.ac.uk/cgi-bin/traceroute.pl?target=www.cern.ch&function=ping
• Used by TULIP trilateration research project– Locate a target by pinging from multiple landmarks– Use the RTT to estimate distance– Use trilateration to locate target– See
www.slac.stanford.edu/comp/net/wan-mon/tulip/
UNIMASWorkshop, Sarawak, Dec 2012
![Page 76: PingER: Navigating the web site and mining the data Les Cottrell SLAC Workshop at the University of Malaya (UM), Kuala Lumpur, June 24-25, 2013](https://reader036.vdocuments.mx/reader036/viewer/2022081516/56649ea05503460f94ba2a5a/html5/thumbnails/76.jpg)
76
Install PingER2 measurement engine
UNIMASWorkshop, Sarawak, Dec 2012
• Install: www.slac.stanford.edu/comp/net/wan-mon/Pinger2/
Joint Techs: I2 & ESnet,Stanford
Internet10 ping request packets each 30 mins
RemoteHost(typicallyweb server)
>ping host
Ping response packets
Measure Round Trip Time & Loss
Uses ubiquitous ping
Monitor Host Caches data
On
ce a Day
Repositories
NUST
![Page 77: PingER: Navigating the web site and mining the data Les Cottrell SLAC Workshop at the University of Malaya (UM), Kuala Lumpur, June 24-25, 2013](https://reader036.vdocuments.mx/reader036/viewer/2022081516/56649ea05503460f94ba2a5a/html5/thumbnails/77.jpg)
77
Status of Monitoring hosts
• See http://www-iepm.slac.stanford.edu/monitoring/checkdata/Data successfully
gathered
Data partially gathered
Host not pingable
today
No data gathered
![Page 78: PingER: Navigating the web site and mining the data Les Cottrell SLAC Workshop at the University of Malaya (UM), Kuala Lumpur, June 24-25, 2013](https://reader036.vdocuments.mx/reader036/viewer/2022081516/56649ea05503460f94ba2a5a/html5/thumbnails/78.jpg)
78
Trilateration for User Location of an IP Host (TULIP)
• Want to verify where a IP node is located– Tools such as GeoIPTools work reasonably well for
end host (names and IP addresses)– Usually (www.satsig.net/maps/lat-long-finder.htm)
usually works for well known sites– For tracing routes however unless the name of a
router gives it away they are usually identified (e.g. by GeoIPtools) as being at the corporate HQ of the router owner (e.g. Qwest, Level3, the NREN etc.)
UNIMASWorkshop, Sarawak, Dec 2012
![Page 79: PingER: Navigating the web site and mining the data Les Cottrell SLAC Workshop at the University of Malaya (UM), Kuala Lumpur, June 24-25, 2013](https://reader036.vdocuments.mx/reader036/viewer/2022081516/56649ea05503460f94ba2a5a/html5/thumbnails/79.jpg)
79
TULIP
• Based on using ping to measure RTT from a landmark (e.g. a PingER monitor) to target
• Today most of the RTT is based on how far the signal has to go over the backbone
• Distance = d(km)=RTT(ms)*100km/ms– Speed of light in vacuum = c=300km/sec– In fibre or copper cable=(2/3)c= 200km/sec– For round trip =(1/2)*(2/3)c= 100km/sec
UNIMASWorkshop, Sarawak, Dec 2012
![Page 80: PingER: Navigating the web site and mining the data Les Cottrell SLAC Workshop at the University of Malaya (UM), Kuala Lumpur, June 24-25, 2013](https://reader036.vdocuments.mx/reader036/viewer/2022081516/56649ea05503460f94ba2a5a/html5/thumbnails/80.jpg)
80
• Can use these distance measured from multiple nearby landmarks to use trilateration to estimate the location of the target.
• Challenges:– The fibre path is not direct
• Globally or locally
– There are no nearby landmarks
UNIMASWorkshop, Sarawak, Dec 2012