Google Maps to Graphsa howto on large-scale collaboration using

Google Maps and Dropbox

Marat Zhanikeev2013/08/28

The Objective

1. Take a list of locations ex: Family Mart convenience stores in some area

2. Use Google Maps API to get the route for each pair of locations

if A→B = B→A, then roughly 0.5 * ( N^2) in code: for ( a = 0; a < n – 1; a++) { for ( b = a + 1; b < n; b++) { … }}

3. Split data in blocks and write each block to Dropbox• all clients write to the same Dropbox folder (yours)• use chunk_upload to split data in batches (otherwise too many

files)• make it work for a large number of concurrent clients

4. Ask people for help (limited by single-client daily quota)

The Reason for Crowdsourcing• 100 locations need ~5000 Map API requests• Daily quota of Maps API is 2500/day/IP• You figure out the math• Reality: even a small crowd can create a

dataset in no time• After all, it is all about creating datasets on

demand

Step 1: Build Dropbox API for JS

• Dropbox does not have a JS API … yet• https://www.dropbox.com/developers/core• Dropbox vs Google Drive– Only Dropbox is suitable– 24h valid upload_id for chunk_upload– Makes it possible to aggregates data into batches

• Status: DONE (URL on the last page)

Step 2: Learn Google Maps API

• Piece of cake!• DirectionsService for routes (car, walking,…)• Easy – you can feed it address strings, names

of places, anything…• Status: DONE (URL on the last page)

Step 3: Create a Serverless App

• Serverless Webapp: a web application which does not require a web server

• Both GoogleMapsAPI and DropboxAPI use JS• You can pack it all into one HTML file…• … put it into your Dropbox folder ….• … and share it!• Status: DONE (URL on the last page)

Step 4: Make It Work for Crowds

• Concurrent access to data by many users• Write of data are OK ( base64( JSON) = text)• Need status file to be shared by all– Need it for clients to know which routes to request– Overlaps are inevitable, but random selection of

tasks will have to avoid too much overlap• Use a smart logic ( 1 req / 5s, safeguards,

smarts, …)• Status: DONE (URL on the last page)

Step 5: Send it Out into the Wild

• Test run• 300 Family Marts in Fukuoka, Japan• ~35K requests

https://t.co/F5zPCeqhQw• Deploy Status: DONE (that t.co URL)• Crowds Status: WAITING

• failed to get my tweets show up in #googlemapsapi #googlemaps #gis #datasets

• miserably failed to have it noticed anywhere else• Don’t have that many friends

• Deep Thoughts: is on-demand online collaboration actually possible?

It looks like this…• Fully

automatic, you can just let it run

• Otherwise, if you have to run right now, just press COMMIT and go.