raster data partitioning for supporting distributed gis processing

RASTER DATA PARTITIONING FOR SUPPORTING DISTRIBUTED GIS PROCESSING

Binh Nguyen Thai - Angéla Olasz

GeoBigData ISPRS Geospatial WeekGeoBigData - ISPRS Geospatial WeekLa Grande Motte, FranceSep. 28 - Oct. 2. 2015 p

Institute of Geodesy, Cartography and Remote Sensing (FÖMI)

Directorate of Geoinformation

SummarySummaryOur goal is to find a solution for processing of big geospatial data in a

distributed ecosystem providing an environment to run algorithmsdistributed ecosystem providing an environment to run algorithms,services, processing modules without any limitations onimplementation programming language as well as datapartitioning strategies and distribution among computationalpartitioning strategies and distribution among computationalnodes. As a first step we focus on

(i) data decomposition and(ii) distributed processing.

Th h ll i t d ith h f l t d th d lThe challenges associated with each focus area, related methodologyand first results are analyzed and discussed in the paper.

The data decomposition and the NDVI calculation were tested usingp gLandsat 8 imagery for the territory of Hungary with a groundresolution of 30 m.

2Raster data partitioning • ISPRS Geospatial WeekSep. 28 - Oct. 2. 2015, La Grande Motte, France

Technical goalsTechnical goals1. Executing existing programs on distributed environment.2. Implementing new GIS processing services on any languageand execute them on distributed environment.3. Full control over data partitioning and distribution.

f4. Free from Map-Reduce programming model.Choosing the „right” programming language

Before jumping into implementation, a study has been made on which programming language should be used based on the following criteria:• Platform dependencies.• Support on GIS libraries and data types.• Ease of use and descriptive syntax• Ease-of-use and descriptive syntax.• Well documented and community support.


MethodsMethods


ResultsResults

Decomposition time for NxN regular grids areCurrently we have decomposed original data into NxN regular Decomposition time for NxN regular grids are increasing substantially after N > 4.

Currently we have decomposed original data into NxN regular grids and distribute them over M processing nodes, where:N >> M

Comparison of time results on distributed processing:

Raster data partitioning • ISPRS Geospatial WeekSep. 28 - Oct. 2. 2015, La Grande Motte, France

5

ConclusionConclusionPrototype have confirmed that distributed processing with pre-

decomposed data will give us good runtime results as well asthe flexibility over running different processing algorithms on thesame dataset.

However this prototype system should evolve with future developmentfeatures like:

1 D l1. Data catalog2. Service catalog3 Service deployer3. Service deployer

Our goal is to design and implement a framework using DevOpsmethodology.


Thank you for your attention!Thank you for your attention!Please find our Poster and Article in ISPRS Archives:

Ö

Angéla [email protected]

Binh Nguyen [email protected]

Institute of Geodesy, Cartography and Remote Sensing (FÖMI)

Directorate of Geoinformation

5. Bosnyák sqr. BUDAPEST, HUNGARY 1149 www.fomi.hu7

raster data partitioning for supporting distributed gis processing

Science