gdal & webgl: european commission, joint research center, ispra

Research & Development with WebGL, Closure, GDAL & MapRank

Joint Research Centre - European Commision, November 22, Ispra, Italy.

Petr Pridal <[email protected]>

mailto:[email protected]


KLOKAN TECHNOLOGIES - Museo Sans 500

KLOKAN TECHNOLOGIES - Museo Sans 500 ItalicKLOKAN TECHNOLOGIES - Museo Sans 700

KLOKAN TECHNOLOGIES - Museo 300

KLOKAN TECHNOLOGIES - Museo 500KLOKAN TECHNOLOGIES - Museo 700

CMYK100 75 10 0

RGB0 81 151

#005096

Closure ToolsWebGL Earth

MapTiler ClusterGDAL2Tiles on steroids. GeoSearch with ranking.

3D digital globe. JavaScript Toolset.

WebGL EarthFree and open source 3D digital globe

for web and mobile devices

!

beta

HTML5 & WebGL

• HTML5 <canvas> WebGL extension

• Based on OpenGL ES

• Supported by NVidia, ATI, Intel on desktop

• HW acceleration on mobile devices

• Firefox 4+, Chrome 9+, Opera, Safari

• Internet Explorer: with Chrome Frame





CMYK100 75 10 0

RGB0 81 151

#005096

WebGL: Shaders in GLSL





CMYK100 75 10 0

RGB0 81 151

#005096





CMYK100 75 10 0

RGB0 81 151

#005096





CMYK100 75 10 0

RGB0 81 151

#005096Inspired by OpenLayers.Layer.GoogleNG(done by Andreas Hocevar)

Compliant to Google TOS?

Try the demo application:

www.webglearth.comKLOKAN TECHNOLOGIES - Museo Sans 500




CMYK100 75 10 0

RGB0 81 151

#005096

Mission of the project

• User friendly (dragging, zooming, ...)

• Keep it simple & maximally web oriented

• Raster data from prerendered tiles, WMS, existing tile APIs and TileCache, GeoServer...

• “NoServer” = o!ine usage & cloud

• Calculations from CPU to GPU. Preprocess!

• Open-Source (GPLv3), Community powered





CMYK100 75 10 0

RGB0 81 151

#005096

Open Source project

• Simple JavaScript API

• Embed <script src=“...../api.js”>

• Mashup with your own tiles, custom WMS, overlays, markers and popups

• The full codebase + demo app sources on GitHub, object oriented, documented code, manual describing internals available

• Contributors: KlokanTech, CampToCamp, AI2 Spain, BIMServer.org, NOAA





CMYK100 75 10 0

RGB0 81 151

#005096

JavaScript API: Demo





CMYK100 75 10 0

RGB0 81 151

#005096

JavaScript API





CMYK100 75 10 0

RGB0 81 151

#005096

• Embedding of the 3D globe

• Custom base maps and overlays with adjustable transparency (tiles + WMS)

• Camera API, flyTo, bbox

• User interaction and events

• Markers & popups

• <canvas> is part of HTML DOM

Codebase: we, weapp

• Demo application (www.webglearth.com) completely open-source, API exports too

• Compiled with Closure Compiler with types: http://blog.klokantech.com/2010/12/closure-compiler-for-openlayers-3x.html

• Documented (google style guideline)

• Technically, anything is possible :-)

• Manual describing the implementation details: http://data.webglearth.com/doc/





CMYK100 75 10 0

RGB0 81 151

#005096

http://www.webglearth.com


http://blog.klokantech.com/2010/12/closure-compiler-for-openlayers-3x.html


http://data.webglearth.com/doc/

http://data.webglearth.com/doc/





CMYK100 75 10 0

RGB0 81 151

#005096

Object hierarchy

“Keep it simple” design

• Sphere with diameter 1.0

• WGS84 coordinates used without ellipsoid transformation (a la Google Maps, OSM, etc.)

• Textures in Spherical Mercator (EPSG:3857, EPSG:900913)

• Clip stack for megatextures with lookup

• Segmented plane as the model





CMYK100 75 10 0

RGB0 81 151

#005096

Segmented plane





CMYK100 75 10 0

RGB0 81 151

#005096

Terrain





CMYK100 75 10 0

RGB0 81 151

#005096





CMYK100 75 10 0

RGB0 81 151

#005096

Vector data (direct rendering)

• Now: rasterized only (WMS, tiles with Mapnik, ...). Raster tiles must be there for aerial data.

• Future: direct rendering (from GeoJSON, KML, WFS, ...)

• Sharing code for loading data formats with OpenLayers 3, Leaflet, Polymaps?

• Vector Tiles - Google MapsGL





CMYK100 75 10 0

RGB0 81 151

#005096

Future: Google MapsGL





CMYK100 75 10 0

RGB0 81 151

#005096

Future: Mobile devices





CMYK100 75 10 0

RGB0 81 151

#005096

Problems: drivers + CORS





CMYK100 75 10 0

RGB0 81 151

#005096

• Long list of blacklisted graphic cards

• June 2011: Introduced restrictions on cross-domain textures for WebGL :-(

• CORS HTTP Header: Access-Control-Allow-Origin: *

• http://enable-cors.org/ or chrome --disable-web-security

• Support only in Chrome 15+ (beta), Firefox 8 (11/2011)

• Workaround for now: HTTP proxy

• Tiles hosted on your own server runs always!

http://enable-cors.org

http://enable-cors.org

Try it: 10 minut cookbook

• JavaScript API: check examples

• Prepare data (one of):

• WMS / TMS / WMTS in EPSG:900913

• Convert any GeoTIFF with GDAL2Tiles / MapTiler

• Copy everything on your web server

• Display in any modern web browser





CMYK100 75 10 0

RGB0 81 151

#005096





CMYK100 75 10 0

RGB0 81 151

#005096

Support, open-source contracted development:Klokan Technologies: http://www.klokantech.com/

Try it at:http://www.webglearth.com/

Project website, mailing-list, documentation:http://www.webglearth.org/

Plans for integration with OpenLayers 3.0

http://www.klokantech.com

http://www.klokantech.com



http://www.webglearth.org

http://www.webglearth.org

Closure ToolsOpenSource JavaScript Toolset

developed by Google Inc.

"

"

Compiler LibraryTemplates Linter Stylesheets JS Test

http://code.google.com/closure/

(Coding Style Guidelines)



It is all about compiler...

1. Compiles readable JavaScript into compressed machine-readable JavaScript: dead code removal, function inlining, etc.

2. JSDoc Tags: typos, wrong use of a @constructor, wrong type of a variable, misuse of a field annotated with @private and @protected, etc.

3. A reusable JavaScript library, such as OpenLayers, exports public API - and the compiler optimizes the internal code.





CMYK100 75 10 0

RGB0 81 151

#005096

http://goo.gl/H6twe

http://goo.gl/H6twe

http://goo.gl/H6twe

It is all about compiler...

4. Unused parts of the library stripped. Application dependencies solved during compilation.

5. Constants to remove unwanted functionality: Mobile WebKit only, no Quirks mode support, compilation without support of IE6

6. Debugging with FireBug possible even for the compiled version

7. Dynamically loadable modules, additional functionality loaded only when required.





CMYK100 75 10 0

RGB0 81 151

#005096

http://goo.gl/H6twe

http://goo.gl/H6twe

http://goo.gl/H6twe

Hello world





CMYK100 75 10 0

RGB0 81 151

#005096

goog.require('goog.dom');

function sayHi() { var newHeader = goog.dom.createDom('h1', {'style': 'background-color:#EEE'}, 'Hello world!'); goog.dom.appendChild(document.body, newHeader);}

goog.exportSymbol('sayHi', sayHi);

How to start?

• Easiest is to get plovr - where all the tools are packed into one .jar application.

• Run java -jar plovr.jar serve config.json and use it with HTML as <script src="http://localhost:9810/compile?id=hello"></script>

• Plovr: Hello world + skeletons at GitHub: https://github.com/klokantech/closure-library-plovr-hello-world-skeletons/

• Closure Library websites: http://code.google.com/closure/library/





CMYK100 75 10 0

RGB0 81 151

#005096

http://localhost:9810/compile?id=hello






http://code.google.com/closure/library/docs/overview.html

http://code.google.com/closure/library/docs/overview.html

Where it excels?

• Complex online applications - any serious work done in JS

• More developers working on the same code

• High peformance app + mobile apps

• Libraries and APIs

• :-( no fast copy & paste hacking

• :-( steep learning curve, smaller community





CMYK100 75 10 0

RGB0 81 151

#005096





CMYK100 75 10 0

RGB0 81 151

#005096

Read more at:http://code.google.com/closure/

Plovr.jar - the “packed” toolset:http://plovr.com/

The book: Closure Library, The Definitive Guide

Blog post: http://goo.gl/H6twe



http://plovr.com/

http://plovr.com/

http://goo.gl/H6twe

http://goo.gl/H6twe

#

GDAL2Tiles on steroids.

MapTiler Cluster

• MapTiler is an application for rapid online map publishing

• Input: raster geodata (GeoTIFF, JPEG2000, MrSID, ECW, Erdas HFA, BSB, ...). Output: jpeg/png tiles

• Support for existing georeference (World Files, WKT, EPSG,...). Transformation (warping) of projections possible.

• Standards: OSGEO Tile Map Service Specification - TMS, comming OGC WMTS:

• Web viewers (OpenLayers, Google Maps API) + applications

• KML files (Google Earth)

• iPhone + iPad + Android native applications

• ArcGIS - with extensions (ArcBruTile). 10.1+ native WMTS support

MAPTILER - WHAT IS IT?

GOOGLE MAPS OVERLAY DEMO

OPENLAYERS DEMO

GOOGLE EARTH DEMO

MAP TILER GENERATES TILES:

NECESSARY TRANSFORMATIONS AND WARPING

GEODETIC COORDINATESLATITUDE LONGITUDEWGS 84 (EPSG:4326)

SPHERICAL MERCATORMETERS

EPSG:3785 / EPSG:900913

PYRAMID COORDINATESXYZ PIXELS / ZOOM

WEB VIEWERS

TILE INDEXXYZ TILE / ZOOM

TILE MAP SERVICE

SPHERICAL MERCATOR:TILE PROJECTION AND COORDINATES

NOT ONLY IN GOOGLE MAPS!

• Popular on the web: Google Maps, Microsoft Bing Maps, Yahoo Maps, OpenStreetMap, ...

• Lat/Long WGS84 (GPS devices) datum coordinates always displayed to users - KML

• Map rasters are projected by a global Mercator projection: “Spherical Mercator”. EPSG:900913 / EPSG:3857 (meters)

• Conformal, cylindrical. Cropped to square (maximal latitude ~ 85.05 degrees), WGS84 ellipsoid is sphere!

OPEN-SOURCE ALTERNATIVES: PREGENERATING X CACHING TILES

• Generating from raster input:

• MapTiler & GDAL2Tiles

• MSR MapCruncher

• Generating from features (vectors)

• Mapnik

• GMapCreator,

• ...

• Caching tiles from WMS servers (MapServer, GeoServer):

• TileCache.org (Python)

• GeoWebCache.org (Java Servlet)

• Commercial alternatives

• License per server (ArcGIS Server)

EXTREMELY FAST ONLINE MAPSFOR THOUSANDS OF PEOPLE

• Easy hosting: only static files / or raster blobs in a database

• Extremely fast delivery, highest-quality output possible (you have more time for rendering), SLA for up to 99.99% availability

Our software is used ...

• ... in: web applications, mashups, native mobile applications (iPhone/iPad/Android), hardware appliances and 3D visualizations

• ... for: standard base maps, aerial imagery, coverage maps, thematic maps, marine navigation, by pilots in airplains, on the ground during disaster events, ...

• ... by: Google, NOAA, NASA, CERN, Vodafone, T-Mobile, Orange, Three, United Nations, and thousands of developers





CMYK100 75 10 0

RGB0 81 151

#005096

PARALLELIZED RENDERING:MAPTILER CLUSTER

• Parallelized rendering on Multi-Core processors or even on computer clusters (Amazon EC2 supported)

• Speed-up the rendering on modern computers

• Rendering on a computer cluster (Amazon EC2) - price: 0.1 USD / hour / core

• Publishing of tiles on a world-wide CDNs (Amazon S3/CloudFront, Google App Engine, Azure, SimpleCDN) - fast multi-threaded upload for tiles possible

•Merging of huge raster datasets into a seamless map a la Google Maps on a computer cluster:

• Fast parallelized rendering and merging of large data collections

• Cluster parallelization based on MPI

• Test of MapReduce approach (via Nokia labs Disco project) for cl

ients

MapTiler Cluster vs GDAL2Tiles

• Fast rendering: multi-core CPU & MPI cluster

• Direct merging of several input files

• Excellent visual quality

• Heavily optimized PNGs/JPEGs tiles to speed up the delivery and save disk space

• Precise "data over-zooming"

• Various coordinate systems supported

for cl

ients

Runs under Windows, Linux and Mac OS X + GDAL2Tiles

You can try it on the FOSS4G LiveDVD!

Community: MapTiler User Group

[email protected]

MapTiler Cluster, commercial support, integration, customization

http://www.maptiler.com/

WWW . MAPTILER . ORG



http://www.maptiler.com

http://www.maptiler.com

http://www.maptiler.org


$

Geosearch with ranking.

• Research, scientific articles, work on Ph.D. thesis (Petr Přidal), Computer Science + Geodesy and Cartography

• Alternative projects tested in OldMapsOnline.org (R&D project in MZK):

• Lucene with GeoTemporal Ext. (LGTE) from DigMap.eu

• Alexandria Digital Library

• TimeMap.net (ECAI.org)

• Google GeoSearch API

• Knowledge of traditional GIS search methods (PostGIS, ...)

GEOGRAPHICAL SEARCHING





CMYK100 75 10 0

RGB0 81 151

#005096

Result: Development of a new geographical search engine for map collections.

• Map-driven interface: Simple. Intuitive. Web 2.0. Users love it!

• Combined queries: Where, When, Who & What

• Results sorted geographically with an advanced spatial ranking algorithm.

• Instant search. Immediate delivery of the results by moving the map or typing.

• It is fast. Even for large databases (million+ records).

CORE FEATURES





CMYK100 75 10 0

RGB0 81 151

#005096

• Indexing metadata in MARC, DublinCore, ISO 19139 formats, ...

• Batch import or harvesting: OAI-PMH or OGC CWS

• Numerical geographical boundaries necessary! (MARC 034, DublinCore DCMI) - enrichment with Geoparser or Georeferencer tools

• Possible customization of the user interface, integration into products or websites

• For developers: REST API (JSON format), queries following OpenSearch.org

METADATA & CUSTOMIZATION





CMYK100 75 10 0

RGB0 81 151

#005096

Try it at: http://demo.mapranksearch.com/

http://demo.mapranksearch.com


Try it at: http://rumsey.mapranksearch.com/



Try it at: http://mapy.mzk.cz/



TOUCH INTERFACE





CMYK100 75 10 0

RGB0 81 151

#005096

• MapRank ranking: an advanced algorithm calculating similarity of geographical area of query and area covered by every document and combining it with map scale, time range, predefined categories and traditional full text search.

• Data indexing implemented specifically to provide ranked real-time results even for large databases. Traditional R-Tree indexing or Lucene are failing.

• Server component: FastCGI, C++

• Default user interface developed with Closure JavaScript Library + Closure Compiler.

MAPRANK RANKING & SCALABILITY





CMYK100 75 10 0

RGB0 81 151

#005096

Geoparser

Text parsing and semi-automatic assignment of bounding boxes to map records with detectable geonames (gazetteer), known physical size and map scale.

Georeferencer

Crowdsourced georeferencing. Higher accuracy, more outputs beside geometadata such as visualization, 3D view, comparing of maps, OGC WMS for use in GIS.

ENRICHMENT OF METADATA





CMYK100 75 10 0

RGB0 81 151

#005096

WWW.MAPRANKSEARCH . COM







CMYK100 75 10 0

RGB0 81 151

#005096

Petr Pridal <[email protected]>

Thank you!Questions, ideas?

Cooperation on OpenSource and R&D?Coding week for OpenLayers3 & WebGL Earth?Interest in applications of MapTiler / MapRank?



gdal & webgl: european commission, joint research center, ispra

Technology