cartodb inside out
TRANSCRIPT
![Page 1: CartoDB Inside Out](https://reader034.vdocuments.mx/reader034/viewer/2022050614/588535141a28ab26518b5bc9/html5/thumbnails/1.jpg)
CartoDBInside Out
Jorge Sanz · [email protected] · @xurxosanzSASIG · OSGeo.pt · Lisbon · 2015
A technical overview of a web mapping platform
![Page 2: CartoDB Inside Out](https://reader034.vdocuments.mx/reader034/viewer/2022050614/588535141a28ab26518b5bc9/html5/thumbnails/2.jpg)
Agenda● Why
● CartoDB nowadays
● Architecture of the platform
○ Components
○ Setup
● How to serve dynamic data at scale
No maps in this talk sorry
![Page 3: CartoDB Inside Out](https://reader034.vdocuments.mx/reader034/viewer/2022050614/588535141a28ab26518b5bc9/html5/thumbnails/3.jpg)
https://congosto.cartodb.com/viz/e5da12e2-9fe7-11e4-bc43-0e853d047bba/embed_map
![Page 4: CartoDB Inside Out](https://reader034.vdocuments.mx/reader034/viewer/2022050614/588535141a28ab26518b5bc9/html5/thumbnails/4.jpg)
http://elpais.com/elpais/2015/08/20/media/1440070260_752772.html
![Page 5: CartoDB Inside Out](https://reader034.vdocuments.mx/reader034/viewer/2022050614/588535141a28ab26518b5bc9/html5/thumbnails/5.jpg)
Why● @jatorre and @saleiva start Vizzuality in April 2008
● Company focused on visualization projects
● Every geospatial project had the same stack:
○ Database + Map Server + APIs + client code
● CartoDB: internal product to deploy faster this geo-stack
![Page 6: CartoDB Inside Out](https://reader034.vdocuments.mx/reader034/viewer/2022050614/588535141a28ab26518b5bc9/html5/thumbnails/6.jpg)
Mission
With CartoDB everybody can analyse,visualise and share insights on location data.We are democratising location intelligence
![Page 7: CartoDB Inside Out](https://reader034.vdocuments.mx/reader034/viewer/2022050614/588535141a28ab26518b5bc9/html5/thumbnails/7.jpg)
CartoDB nowadays● A VC founded company of ~70 people (NY + Madrid)
● A leading mapping service with +130.000 users
● An Open Source product that can be deployed
on the cloud or on just one server (on-premises)
● Full stack solution from storage to client side libraries
● It’s a mapping editor supported by a platform (APIs)
![Page 8: CartoDB Inside Out](https://reader034.vdocuments.mx/reader034/viewer/2022050614/588535141a28ab26518b5bc9/html5/thumbnails/8.jpg)
CartoDB architecture
![Page 9: CartoDB Inside Out](https://reader034.vdocuments.mx/reader034/viewer/2022050614/588535141a28ab26518b5bc9/html5/thumbnails/9.jpg)
http://www.di.unito.it/~schifane/smellymaps/explorer.html
![Page 10: CartoDB Inside Out](https://reader034.vdocuments.mx/reader034/viewer/2022050614/588535141a28ab26518b5bc9/html5/thumbnails/10.jpg)
https://team.cartodb.com/u/piensaenpixel/viz/8a6530e0-5239-11e5-9cc4-0e0c41326911/public_map
![Page 11: CartoDB Inside Out](https://reader034.vdocuments.mx/reader034/viewer/2022050614/588535141a28ab26518b5bc9/html5/thumbnails/11.jpg)
The CartoDB platform● Spatial database for users● NoSQL database for
configurations and cache● Rendering, imports,
database and other services● Cache and Content
Delivery Network● JavaScript Library● Map Editor
Platform
On-premisesor in the Cloud
Data servicesYour data
Editor
![Page 12: CartoDB Inside Out](https://reader034.vdocuments.mx/reader034/viewer/2022050614/588535141a28ab26518b5bc9/html5/thumbnails/12.jpg)
The CartoDB platform
CartoDB.js Odyssey.jsTorque.js Leaflet /Gmaps
PostgreSQL
PostGIS
SQL API
Redis
Maps API
TorqueMapnik
Varnish Caching
Fastly CDN
Car
toD
B o
n A
maz
on
el
asti
c R
ecip
es
Car
toD
B o
n
Am
azo
n e
last
ic
Rec
ipes
PostgreSQL
PostGIS
CartoDB Server API
Rest API Imports API
CartoDB Editor
Browser
![Page 13: CartoDB Inside Out](https://reader034.vdocuments.mx/reader034/viewer/2022050614/588535141a28ab26518b5bc9/html5/thumbnails/13.jpg)
Cloud setup● 1 - load balancer (nginx)
● 1 - caching (varnish)
● N - UI servers (CartoDB UI)
● N - database servers (user data)
● 1 - Metadata server (CartoDB metadata)
● 1 - redis server (+slave)
● N - Maps API + API SQL servers
![Page 14: CartoDB Inside Out](https://reader034.vdocuments.mx/reader034/viewer/2022050614/588535141a28ab26518b5bc9/html5/thumbnails/14.jpg)
![Page 15: CartoDB Inside Out](https://reader034.vdocuments.mx/reader034/viewer/2022050614/588535141a28ab26518b5bc9/html5/thumbnails/15.jpg)
Postgres + Postgis● Geodata, users and editor metadata storage
● Tiny Well Known Binary
● CartoDB - Postgres extension
Redis● key-value highly performant storage
● configuration for APIs
● cache
![Page 16: CartoDB Inside Out](https://reader034.vdocuments.mx/reader034/viewer/2022050614/588535141a28ab26518b5bc9/html5/thumbnails/16.jpg)
Backend services● Ruby on Rails
● Import API (ogr2ogr)
● Background management tasks
Invalidation service● Node.js + HA Proxy + Consul
● Invalidates caches by request
● Multi-agent, replicated
![Page 17: CartoDB Inside Out](https://reader034.vdocuments.mx/reader034/viewer/2022050614/588535141a28ab26518b5bc9/html5/thumbnails/17.jpg)
SQL API● node.js
● Exposes an entry point for running queries
● Acts as an export API
Maps API · Windshaft● node.js + mapnik
● renders tiles from CartoCSS + SQL
● SQL API client
![Page 18: CartoDB Inside Out](https://reader034.vdocuments.mx/reader034/viewer/2022050614/588535141a28ab26518b5bc9/html5/thumbnails/18.jpg)
nginx● web server
● load balancer
Varnish + Fastly/GCP● web cache
● Content Delivery Network
![Page 19: CartoDB Inside Out](https://reader034.vdocuments.mx/reader034/viewer/2022050614/588535141a28ab26518b5bc9/html5/thumbnails/19.jpg)
CartoDBMap Editor
the easiest web mapping tool
ever
![Page 20: CartoDB Inside Out](https://reader034.vdocuments.mx/reader034/viewer/2022050614/588535141a28ab26518b5bc9/html5/thumbnails/20.jpg)
Map Editor● Ruby on Rails + JavaScript
● Consumes the Imports, Maps, SQL API and its own API
● Highly focused on UX-UI
● Manage datasets (schema, privacy, etc)
● Manage maps (basemap, infowindows,
other visual elements, privacy, etc)
● Define layer data source with filters and SQL editor
● Define symbology through wizards and CartoCSS editor
● Publish maps as embeds, links or with CartoDB.js
![Page 21: CartoDB Inside Out](https://reader034.vdocuments.mx/reader034/viewer/2022050614/588535141a28ab26518b5bc9/html5/thumbnails/21.jpg)
CartoDB.js● Based on jQuery + Backbone + Underscore
● Leverages Leaflet or Google Maps API
● Easy interaction with the SQL and Maps API
● Manages transparently utf-grids for interactivity
● Allows easy customizations on infowindows,
layer SQL and CartoCSS definitions, etc
![Page 22: CartoDB Inside Out](https://reader034.vdocuments.mx/reader034/viewer/2022050614/588535141a28ab26518b5bc9/html5/thumbnails/22.jpg)
Serving data at scale
![Page 23: CartoDB Inside Out](https://reader034.vdocuments.mx/reader034/viewer/2022050614/588535141a28ab26518b5bc9/html5/thumbnails/23.jpg)
https://scordivano.cartodb.com/viz/f55a3268-0c29-11e3-827e-5fd91394b353/embed_map
![Page 24: CartoDB Inside Out](https://reader034.vdocuments.mx/reader034/viewer/2022050614/588535141a28ab26518b5bc9/html5/thumbnails/24.jpg)
http://mappingkat.github.io/indian-removal/
![Page 25: CartoDB Inside Out](https://reader034.vdocuments.mx/reader034/viewer/2022050614/588535141a28ab26518b5bc9/html5/thumbnails/25.jpg)
Story of a tile: first request
![Page 26: CartoDB Inside Out](https://reader034.vdocuments.mx/reader034/viewer/2022050614/588535141a28ab26518b5bc9/html5/thumbnails/26.jpg)
Story of a tile: next request
![Page 27: CartoDB Inside Out](https://reader034.vdocuments.mx/reader034/viewer/2022050614/588535141a28ab26518b5bc9/html5/thumbnails/27.jpg)
Story of a tile: insert & next request
![Page 28: CartoDB Inside Out](https://reader034.vdocuments.mx/reader034/viewer/2022050614/588535141a28ab26518b5bc9/html5/thumbnails/28.jpg)
Cache invalidation
Write queries fire a trigger that updates
cdb_tablemetadata and starts the invalidation
process for the tiles that use that table
![Page 29: CartoDB Inside Out](https://reader034.vdocuments.mx/reader034/viewer/2022050614/588535141a28ab26518b5bc9/html5/thumbnails/29.jpg)
Summary● CartoDB is an Open Source, full stack mapping product
● Runs on a single server or distributed
● Delivers to thousands of users the superpowers of Postgis
and Mapnik, not just pretty maps but also new insights
● Provides tools for users and developers
● It’s designed to scale, but we are always improving it
![Page 31: CartoDB Inside Out](https://reader034.vdocuments.mx/reader034/viewer/2022050614/588535141a28ab26518b5bc9/html5/thumbnails/31.jpg)
CartoDBInside Out
Jorge Sanz · [email protected] · @xurxosanzSASIG · OSGeo.pt · Lisbon · 2015http://bit.ly/cartodb-sasig2015
Muito obrigado! · Questions?
![Page 32: CartoDB Inside Out](https://reader034.vdocuments.mx/reader034/viewer/2022050614/588535141a28ab26518b5bc9/html5/thumbnails/32.jpg)
Links● https://cartodb.com
● https://github.com/CartoDB/cartodb
● https://github.com/CartoDB/cartodb.js
● https://github.com/CartoDB/cartodb-postgresql
● https://github.com/CartoDB/Windshaft-cartodb
● https://github.com/CartoDB/CartoDB-SQL-API
● http://docs.cartodb.com
● http://cartodb.readthedocs.org