distributed processing of raster geodata

RASTER DATA PARTITIONING FOR SUPPORTING DISTRIBUTED GIS PROCESSING

Binh Nguyen Thai - Angéla Olasz

GeoBigData - ISPRS Geospatial Week

La Grande Motte, France

Sep. 28 - Oct. 2. 2015

Institute of Geodesy, Cartography and Remote Sensing (FÖMI)

Directorate of Geoinformation

Our goal is to find a solution for processing of big geospatial data in a distributed ecosystem providing an environment to run algorithms, services, processing modules without any limitations on implementation programming language as well as data partitioning strategies and distribution among computational nodes. As a first step we focus on

(i) data decomposition and

(ii) distributed processing.

The challenges associated with each focus area, related methodology and first results are analyzed and discussed in the paper.

The data decomposition and the NDVI calculation were tested using Landsat 8 imagery for the territory of Hungary with a ground resolution of 30 m.

2

Summary

Raster data partitioning • ISPRS Geospatial WeekSep. 28 - Oct. 2. 2015, La Grande Motte, France

1. Executing existing programs on distributed environment.

2. Implementing new GIS processing services on any language

and execute them on distributed environment.

3. Full control over data partitioning and distribution.

4. Free from Map-Reduce programming model.

Choosing the „right” programming language

Before jumping into implementation, a study has been made on which programming language should be used based on the following criteria:• Platform dependencies.• Support on GIS libraries and data types.• Ease-of-use and descriptive syntax.• Well documented and community support.

3

Technical goals


4

Methods


Results


5

Comparison of time results on distributed processing:

Decomposition time for NxN regular grids are increasing substantially after N > 4.

Currently we have decomposed original data into NxN regular grids and distribute them over M processing nodes, where:N >> M

6

Conclusion


Prototype have confirmed that distributed processing with pre-decomposed data will give us good runtime results as well as the flexibility over running different processing algorithms on the same dataset.

However this prototype system should evolve with future development features like:

1. Data catalog

2. Service catalog

3. Service deployer

Our goal is to design and implement a framework using DevOps methodology.

Institute of Geodesy, Cartography and Remote Sensing (FÖMI)

Directorate of Geoinformation

5. Bosnyák sqr. BUDAPEST, HUNGARY 1149 www.fomi.hu

Thank you for your attention!

Angéla [email protected]

7

Binh Nguyen Thai

[email protected]

Please find our Poster and Article in ISPRS Archives:

http://www.fomi.hu/

mailto:[email protected]



distributed processing of raster geodata

Documents