high performance osm data manipulation with osmium - state of the map 2013
DESCRIPTION
*** Presented by Jochen Topf at State of the Map 2013 *** For the video of this presentation please see http://lanyrd.com/2013/sotm/scpkqw/ *** Full schedule available at http://wiki.openstreetmap.org/wiki/State_Of_The_Map_2013 Osmium is a highly flexible and performant C++ library for working with OpenStreetMap data. It helps with reading, writing, and filtering OSM data, OSM history data and OSM change files in XML or PBF format. Osmium assembles way and (multi)polygon geometries and converts them into many GIS formats such as shapefiles. It also contains code to store OSM data efficiently and much more. All of this in an easy to use C++ header-only library. Some functions are also available from Javascript to make it's use even easier. The talks shows what we have learned over the years about high- performance OSM data manipulation, the current state of Osmium and what the future might bring.TRANSCRIPT
![Page 1: High Performance OSM Data Manipulation With Osmium - State of the Map 2013](https://reader033.vdocuments.mx/reader033/viewer/2022060117/5587b073d8b42a58108b46b8/html5/thumbnails/1.jpg)
High Performance OSM Data Manipulation With Osmium
Jochen Topf
![Page 2: High Performance OSM Data Manipulation With Osmium - State of the Map 2013](https://reader033.vdocuments.mx/reader033/viewer/2022060117/5587b073d8b42a58108b46b8/html5/thumbnails/2.jpg)
![Page 3: High Performance OSM Data Manipulation With Osmium - State of the Map 2013](https://reader033.vdocuments.mx/reader033/viewer/2022060117/5587b073d8b42a58108b46b8/html5/thumbnails/3.jpg)
![Page 4: High Performance OSM Data Manipulation With Osmium - State of the Map 2013](https://reader033.vdocuments.mx/reader033/viewer/2022060117/5587b073d8b42a58108b46b8/html5/thumbnails/4.jpg)
CC-BY http://www.flickr.com/photos/x1brett/4562610437/
![Page 5: High Performance OSM Data Manipulation With Osmium - State of the Map 2013](https://reader033.vdocuments.mx/reader033/viewer/2022060117/5587b073d8b42a58108b46b8/html5/thumbnails/5.jpg)
Typical Problems
Slow.
Needs a lot of memory/disk space.
Doesn't work with entire planet.
![Page 6: High Performance OSM Data Manipulation With Osmium - State of the Map 2013](https://reader033.vdocuments.mx/reader033/viewer/2022060117/5587b073d8b42a58108b46b8/html5/thumbnails/6.jpg)
OSM Data
There isn't all that much data(current planet PBF: 23 GB)
But we need to store it efficiently!
![Page 7: High Performance OSM Data Manipulation With Osmium - State of the Map 2013](https://reader033.vdocuments.mx/reader033/viewer/2022060117/5587b073d8b42a58108b46b8/html5/thumbnails/7.jpg)
OSM Data
Often we can work on the datapiece by piece
Streaming
![Page 8: High Performance OSM Data Manipulation With Osmium - State of the Map 2013](https://reader033.vdocuments.mx/reader033/viewer/2022060117/5587b073d8b42a58108b46b8/html5/thumbnails/8.jpg)
C++
![Page 9: High Performance OSM Data Manipulation With Osmium - State of the Map 2013](https://reader033.vdocuments.mx/reader033/viewer/2022060117/5587b073d8b42a58108b46b8/html5/thumbnails/9.jpg)
Osmium
A fast and flexible C++ libraryfor working with OSM data
![Page 10: High Performance OSM Data Manipulation With Osmium - State of the Map 2013](https://reader033.vdocuments.mx/reader033/viewer/2022060117/5587b073d8b42a58108b46b8/html5/thumbnails/10.jpg)
CC-BY http://www.flickr.com/photos/jronaldlee/4479381576/
Modular
![Page 11: High Performance OSM Data Manipulation With Osmium - State of the Map 2013](https://reader033.vdocuments.mx/reader033/viewer/2022060117/5587b073d8b42a58108b46b8/html5/thumbnails/11.jpg)
Has to work withdata of entire
planet!
...or asmall extract!
![Page 12: High Performance OSM Data Manipulation With Osmium - State of the Map 2013](https://reader033.vdocuments.mx/reader033/viewer/2022060117/5587b073d8b42a58108b46b8/html5/thumbnails/12.jpg)
Features
Basic OSM objects:Nodes, ways, relations, tags, ...
And operations on them.
Tag filtering
![Page 13: High Performance OSM Data Manipulation With Osmium - State of the Map 2013](https://reader033.vdocuments.mx/reader033/viewer/2022060117/5587b073d8b42a58108b46b8/html5/thumbnails/13.jpg)
Input/Output
Read from: file, stdin or URL.Write to: file or stdout.
XML or PBF.Compressed or uncompressed.
OSM data (.osm) or changes (.osc).With or without history.
![Page 14: High Performance OSM Data Manipulation With Osmium - State of the Map 2013](https://reader033.vdocuments.mx/reader033/viewer/2022060117/5587b073d8b42a58108b46b8/html5/thumbnails/14.jpg)
Geometry
Add node locations to ways
Assemble Multipolygons
Convert geometries to WKT, WKB, OGR, GEOS
Line length (haversine)
![Page 15: High Performance OSM Data Manipulation With Osmium - State of the Map 2013](https://reader033.vdocuments.mx/reader033/viewer/2022060117/5587b073d8b42a58108b46b8/html5/thumbnails/15.jpg)
Handler
OSMfile
OSMfile Handler Writer Reader
Temp.Storage
For converter and filter
![Page 16: High Performance OSM Data Manipulation With Osmium - State of the Map 2013](https://reader033.vdocuments.mx/reader033/viewer/2022060117/5587b073d8b42a58108b46b8/html5/thumbnails/16.jpg)
Example: main
#include <osmium/io/any_input.hpp>
int main(int argc, char* argv[]) {osmium::io::Reader reader(argv[1]);
NamesHandler handler;
reader.open();reader.push(handler);
}
![Page 17: High Performance OSM Data Manipulation With Osmium - State of the Map 2013](https://reader033.vdocuments.mx/reader033/viewer/2022060117/5587b073d8b42a58108b46b8/html5/thumbnails/17.jpg)
Example: handler
#include <iostream>#include <osmium/handler.hpp>
struct NamesHandler : publicosmium::handler::Handler<NamesHandler> {
void node(const osmium::Node& node) {auto n = node.tags().get_value_by_key("name");
if (n) std::cout << n << std::endl; }
};
![Page 18: High Performance OSM Data Manipulation With Osmium - State of the Map 2013](https://reader033.vdocuments.mx/reader033/viewer/2022060117/5587b073d8b42a58108b46b8/html5/thumbnails/18.jpg)
Statistics for61 million different tags
on 2.2 billion objects.
Runs for about two hours every day.
Needs less than 8 GB RAM.
taginfo.openstreetmap.org
![Page 19: High Performance OSM Data Manipulation With Osmium - State of the Map 2013](https://reader033.vdocuments.mx/reader033/viewer/2022060117/5587b073d8b42a58108b46b8/html5/thumbnails/19.jpg)
Linux
Mac OS X
Windows
![Page 20: High Performance OSM Data Manipulation With Osmium - State of the Map 2013](https://reader033.vdocuments.mx/reader033/viewer/2022060117/5587b073d8b42a58108b46b8/html5/thumbnails/20.jpg)
Osmium History
Development started October 2010
Recently started „New Osmium“
![Page 21: High Performance OSM Data Manipulation With Osmium - State of the Map 2013](https://reader033.vdocuments.mx/reader033/viewer/2022060117/5587b073d8b42a58108b46b8/html5/thumbnails/21.jpg)
The New Osmium
Object Storage/Transport
Indexes
Multithreading
(no multipolygon support yet)
The New Osmium
![Page 22: High Performance OSM Data Manipulation With Osmium - State of the Map 2013](https://reader033.vdocuments.mx/reader033/viewer/2022060117/5587b073d8b42a58108b46b8/html5/thumbnails/22.jpg)
C++11
Modern C++
Official ISO standard
Works with GCC 4.7.3, clang 3.2
Easier to write, more efficient, cleaner code
The New Osmium
![Page 23: High Performance OSM Data Manipulation With Osmium - State of the Map 2013](https://reader033.vdocuments.mx/reader033/viewer/2022060117/5587b073d8b42a58108b46b8/html5/thumbnails/23.jpg)
Multithreading
Better design to take advantage of multithreading
Dynamic memory allocation is even worse than with single thread
The New Osmium
![Page 24: High Performance OSM Data Manipulation With Osmium - State of the Map 2013](https://reader033.vdocuments.mx/reader033/viewer/2022060117/5587b073d8b42a58108b46b8/html5/thumbnails/24.jpg)
osmcode.org
Osmiumand
Osmium-basedsoftware
github.com/osmcode
The New Osmium
![Page 25: High Performance OSM Data Manipulation With Osmium - State of the Map 2013](https://reader033.vdocuments.mx/reader033/viewer/2022060117/5587b073d8b42a58108b46b8/html5/thumbnails/25.jpg)
Javascript
Old Osmium: osmjs
New Osmium: Working on NodeJS module
![Page 26: High Performance OSM Data Manipulation With Osmium - State of the Map 2013](https://reader033.vdocuments.mx/reader033/viewer/2022060117/5587b073d8b42a58108b46b8/html5/thumbnails/26.jpg)
Status
Old Osmium: Tried and tested,In production for >2 years
New Osmium: New and untested,Not production ready yet
![Page 27: High Performance OSM Data Manipulation With Osmium - State of the Map 2013](https://reader033.vdocuments.mx/reader033/viewer/2022060117/5587b073d8b42a58108b46b8/html5/thumbnails/27.jpg)
Thanks!
![Page 28: High Performance OSM Data Manipulation With Osmium - State of the Map 2013](https://reader033.vdocuments.mx/reader033/viewer/2022060117/5587b073d8b42a58108b46b8/html5/thumbnails/28.jpg)
Thanks!
wiki.osm.org/wiki/Osmiumgithub.com/joto/osmium
osmcode.orggithub.com/osmcode/libosmium
Jochen [email protected]
Hackdaytomorrow!