![Page 1: Processing data in GeoServer with WPS and SQL views](https://reader035.vdocuments.mx/reader035/viewer/2022081716/53fea2c68d7f72965c8b45a3/html5/thumbnails/1.jpg)
Processing Data In GeoServer With WPS And SQL Views
Ing. Andrea Aime, GeoSolutions
Ing. Alessio Fabiani, GeoSolutions
FOSS4G-Europe 2014, Bremen 14th-17th July 2014
![Page 2: Processing data in GeoServer with WPS and SQL views](https://reader035.vdocuments.mx/reader035/viewer/2022081716/53fea2c68d7f72965c8b45a3/html5/thumbnails/2.jpg)
FOSS4G Europe 2014, Bremen 14th-17th July 2014
Some historical perspective
Year 2008:
Publish maps on the net
Nice styling, maybe time/elevation based selector, some little extra filtering
Maybe some editing, some PDF printing
Year 2013:
All of the above, but…
We can hardly make a new application without some data processing in it
![Page 3: Processing data in GeoServer with WPS and SQL views](https://reader035.vdocuments.mx/reader035/viewer/2022081716/53fea2c68d7f72965c8b45a3/html5/thumbnails/3.jpg)
WPS: some quick reminders
![Page 4: Processing data in GeoServer with WPS and SQL views](https://reader035.vdocuments.mx/reader035/viewer/2022081716/53fea2c68d7f72965c8b45a3/html5/thumbnails/4.jpg)
FOSS4G Europe 2014, Bremen 14th-17th July 2014
Web Processing Service
Wikipedia introduces OGC WPS as:
[A service] designed to standardize the way that GIS calculations are made available to the Internet.
WPS can describe any calculation including all of its inputs and outputs, and trigger its execution
The specific processes served up by a WPS implementation are defined by the owner of that implementation.
Although WPS was designed to work with spatially referenced data, it can be used with any kind of data.
![Page 5: Processing data in GeoServer with WPS and SQL views](https://reader035.vdocuments.mx/reader035/viewer/2022081716/53fea2c68d7f72965c8b45a3/html5/thumbnails/5.jpg)
FOSS4G Europe 2014, Bremen 14th-17th July 2014
An Example
Buffer a L shaped geometry with distance “2”
Get the result back as GML
![Page 6: Processing data in GeoServer with WPS and SQL views](https://reader035.vdocuments.mx/reader035/viewer/2022081716/53fea2c68d7f72965c8b45a3/html5/thumbnails/6.jpg)
FOSS4G Europe 2014, Bremen 14th-17th July 2014
GeoServer WPS integration
WPS
Remote WCS
Remote WFS
HTTP server
WPS client
All GeoServer Layers
WMS client WMS
GeoServer UI
![Page 7: Processing data in GeoServer with WPS and SQL views](https://reader035.vdocuments.mx/reader035/viewer/2022081716/53fea2c68d7f72965c8b45a3/html5/thumbnails/7.jpg)
FOSS4G Europe 2014, Bremen 14th-17th July 2014
Rendering transformations
On-the-fly data transformations inside rendering
chain
Calling WPS processes from SLD docs
Optimized for performance
![Page 8: Processing data in GeoServer with WPS and SQL views](https://reader035.vdocuments.mx/reader035/viewer/2022081716/53fea2c68d7f72965c8b45a3/html5/thumbnails/8.jpg)
FOSS4G Europe 2014, Bremen 14th-17th July 2014
• GeoServer ships with lots of built-in processes
• Basic geometry manipulation
• Aggregation options missing from WFS
• Raster and vector clipping
• And more!
• Ability to chain processes to build more complex functionality
• Open API to manage inputs and outputs
• See all of it at http://demo.geo-solutions.it/share/foss4g2011/wps_aaime_foss4g2011.pdf
More On GeoServer WPS
![Page 9: Processing data in GeoServer with WPS and SQL views](https://reader035.vdocuments.mx/reader035/viewer/2022081716/53fea2c68d7f72965c8b45a3/html5/thumbnails/9.jpg)
When the WPS does not cut it…
![Page 10: Processing data in GeoServer with WPS and SQL views](https://reader035.vdocuments.mx/reader035/viewer/2022081716/53fea2c68d7f72965c8b45a3/html5/thumbnails/10.jpg)
FOSS4G Europe 2014, Bremen 14th-17th July 2014
• Never under-estimate the processing power of your BDMS:
• Designed to efficiently juggle large quantities of data
• Efficient spatial primitives (at least, in PostGIS)
• Doesn’t get more local to your data than this!
• Passing params down?
• Parametric SQL views!
Spatial DBMS!
![Page 11: Processing data in GeoServer with WPS and SQL views](https://reader035.vdocuments.mx/reader035/viewer/2022081716/53fea2c68d7f72965c8b45a3/html5/thumbnails/11.jpg)
FOSS4G Europe 2014, Bremen 14th-17th July 2014
Parametric SQL views
WMS/WFS client
GeoServer
Spatial database
&viewparams=low:2000000;high:5000000
Expanded Query
![Page 12: Processing data in GeoServer with WPS and SQL views](https://reader035.vdocuments.mx/reader035/viewer/2022081716/53fea2c68d7f72965c8b45a3/html5/thumbnails/12.jpg)
Enough theory Let’s get down to business!
![Page 13: Processing data in GeoServer with WPS and SQL views](https://reader035.vdocuments.mx/reader035/viewer/2022081716/53fea2c68d7f72965c8b45a3/html5/thumbnails/13.jpg)
FAO Tuna Atlas
![Page 14: Processing data in GeoServer with WPS and SQL views](https://reader035.vdocuments.mx/reader035/viewer/2022081716/53fea2c68d7f72965c8b45a3/html5/thumbnails/14.jpg)
FOSS4G Europe 2014, Bremen 14th-17th July 2014
Mapping Tuna Catches
• Multiple Filtering
• Aggregation
• Joining quartely stats against the grid
![Page 15: Processing data in GeoServer with WPS and SQL views](https://reader035.vdocuments.mx/reader035/viewer/2022081716/53fea2c68d7f72965c8b45a3/html5/thumbnails/15.jpg)
FOSS4G Europe 2014, Bremen 14th-17th July 2014
Filtering, joining and aggregation
Some example control regexps: • Y_INTERV:
• Default: 1
• Regex: ^(\d)+$
• OP
• Default: sum
• Regex: ^[avg|sum]$
SELECT (T.TS_VALUE / %Y_INTERV%) AS TS_VALUE, T.CD_TA_OCEANAREA, G.GEOMETRY FROM (SELECT CD_TA_OCEANAREA, OP%(TS_VALUE) AS TS_VALUE FROM FIGIS.TS_FI_TA WHERE FIC_ITEM IN (%FIC_ITEM%) AND CD_GEAR IN (%CD_GEAR%) AND YR_TA IN (%YR_TA%) AND QTR_TA IN (%QTR_TA%) GROUP BY CD_TA_OCEANAREA ) t LEFT OUTER JOIN FIGIS_GIS.GRID_G5 g ON T.CD_TA_OCEANAREA = g.CD_OAREA ORDER BY T.CD_TA_OCEANAREA
![Page 16: Processing data in GeoServer with WPS and SQL views](https://reader035.vdocuments.mx/reader035/viewer/2022081716/53fea2c68d7f72965c8b45a3/html5/thumbnails/16.jpg)
FOSS4G Europe 2014, Bremen 14th-17th July 2014
• Animator tool
• GetMap + variying parameters + frame control => animated GIF!
• http://docs.geoserver.org/stable/en/user/tutorials/animreflector.html
Animation
![Page 17: Processing data in GeoServer with WPS and SQL views](https://reader035.vdocuments.mx/reader035/viewer/2022081716/53fea2c68d7f72965c8b45a3/html5/thumbnails/17.jpg)
FOSS4G Europe 2014, Bremen 14th-17th July 2014
• …/geoserver/wms/animate?request=GetMap&… &format=image/gif;subtype=animated &aparam=viewparams:YR_TA &avalues=2000,2001,2002,2003,…
Animator
![Page 18: Processing data in GeoServer with WPS and SQL views](https://reader035.vdocuments.mx/reader035/viewer/2022081716/53fea2c68d7f72965c8b45a3/html5/thumbnails/18.jpg)
What’s the wheater like today?
![Page 19: Processing data in GeoServer with WPS and SQL views](https://reader035.vdocuments.mx/reader035/viewer/2022081716/53fea2c68d7f72965c8b45a3/html5/thumbnails/19.jpg)
FOSS4G Europe 2014, Bremen 14th-17th July 2014
The Situation
• MetOc data is extremely dynamic
• In-Situ sensors acquire data in near real-time
• Meteorological and Oceanographic model runs multiple times a day
• Remote Sensing data is acquired at fast pace
• Meteo Radar data is acquired at fast pace
• Large amount of data is produced to model fast changing environmental variables
• We must shorten to the small possible extent the time and resources for preprocessing!
![Page 20: Processing data in GeoServer with WPS and SQL views](https://reader035.vdocuments.mx/reader035/viewer/2022081716/53fea2c68d7f72965c8b45a3/html5/thumbnails/20.jpg)
FOSS4G Europe 2014, Bremen 14th-17th July 2014
Solutions: On-the-fly contouring
![Page 21: Processing data in GeoServer with WPS and SQL views](https://reader035.vdocuments.mx/reader035/viewer/2022081716/53fea2c68d7f72965c8b45a3/html5/thumbnails/21.jpg)
FOSS4G Europe 2014, Bremen 14th-17th July 2014
Solutions: On-the-fly Wind-Barbs
![Page 22: Processing data in GeoServer with WPS and SQL views](https://reader035.vdocuments.mx/reader035/viewer/2022081716/53fea2c68d7f72965c8b45a3/html5/thumbnails/22.jpg)
FOSS4G Europe 2014, Bremen 14th-17th July 2014
Solutions: On-the-fly Currents
![Page 23: Processing data in GeoServer with WPS and SQL views](https://reader035.vdocuments.mx/reader035/viewer/2022081716/53fea2c68d7f72965c8b45a3/html5/thumbnails/23.jpg)
FOSS4G Europe 2014, Bremen 14th-17th July 2014
Solutions: On-the-fly Storm Tracks
![Page 24: Processing data in GeoServer with WPS and SQL views](https://reader035.vdocuments.mx/reader035/viewer/2022081716/53fea2c68d7f72965c8b45a3/html5/thumbnails/24.jpg)
Download services
![Page 25: Processing data in GeoServer with WPS and SQL views](https://reader035.vdocuments.mx/reader035/viewer/2022081716/53fea2c68d7f72965c8b45a3/html5/thumbnails/25.jpg)
FOSS4G Europe 2014, Bremen 14th-17th July 2014
• New addition to the MapStore project
• Requirements
• Download large amounts of data
• Generic data filtering
• Clip on polygon/bbox/circle, both vector and raster
• Reproject to target CRS
• Mail notification when download is ready (or fails)
• Work in a cluster
• Solution: new WPS processes, asynch WPS calls and WPS customization for state sharing
Advanced Clip and Ship
![Page 26: Processing data in GeoServer with WPS and SQL views](https://reader035.vdocuments.mx/reader035/viewer/2022081716/53fea2c68d7f72965c8b45a3/html5/thumbnails/26.jpg)
FOSS4G Europe 2014, Bremen 14th-17th July 2014
MapStore GUI
Buffer process called synchronously when buffer size changes
![Page 27: Processing data in GeoServer with WPS and SQL views](https://reader035.vdocuments.mx/reader035/viewer/2022081716/53fea2c68d7f72965c8b45a3/html5/thumbnails/27.jpg)
FOSS4G Europe 2014, Bremen 14th-17th July 2014
MapStore GUI
Tracking download status (asynch WPS)
![Page 28: Processing data in GeoServer with WPS and SQL views](https://reader035.vdocuments.mx/reader035/viewer/2022081716/53fea2c68d7f72965c8b45a3/html5/thumbnails/28.jpg)
FOSS4G Europe 2014, Bremen 14th-17th July 2014
Download service architecture
WPS
All GeoServer Layers
Ma
pS
tore
WMS
GetCapabilities List of layers
Buffer
DownloadEstimator
Download
GetStatus
Fetch data
Status database
Store and retrieve status
Mail server
Custom process manager
Start/ Complete/
Failed
notifications
![Page 29: Processing data in GeoServer with WPS and SQL views](https://reader035.vdocuments.mx/reader035/viewer/2022081716/53fea2c68d7f72965c8b45a3/html5/thumbnails/29.jpg)
The CMRE-IDA Project
![Page 30: Processing data in GeoServer with WPS and SQL views](https://reader035.vdocuments.mx/reader035/viewer/2022081716/53fea2c68d7f72965c8b45a3/html5/thumbnails/30.jpg)
FOSS4G Europe 2014, Bremen 14th-17th July 2014
• The Integrated Decision Aid (IDA) has been developed at the Centre for Maritime Research and Experimentation (CMRE) in order to provide naval exercise planners with an effective tool to assess risk to marine mammals from acoustic transmissions.
• Assessment of potential hazard is performed through sound propagation modeling available in the interface.
Intro
Sound Propagation Model
![Page 31: Processing data in GeoServer with WPS and SQL views](https://reader035.vdocuments.mx/reader035/viewer/2022081716/53fea2c68d7f72965c8b45a3/html5/thumbnails/31.jpg)
FOSS4G Europe 2014, Bremen 14th-17th July 2014
Sound Propagation Model
• External Octave Matlab Process
• Logs the asynchronous process status to the DB as a Feature
Octave Matlab Script for SPM
WFS Follows the process status through the WFS WFSLog
Process allows to insert and update the status of SPM into the DB
Dynamically builds an external Octave command line using the input parameters through a FreeMarker Template (FTL)
![Page 32: Processing data in GeoServer with WPS and SQL views](https://reader035.vdocuments.mx/reader035/viewer/2022081716/53fea2c68d7f72965c8b45a3/html5/thumbnails/32.jpg)
FOSS4G Europe 2014, Bremen 14th-17th July 2014
• The list of model runs is available on the bottom Data Grid.
• It is possible to get information on the input parameters and execution from the WFS
Sound Propagation Model
![Page 33: Processing data in GeoServer with WPS and SQL views](https://reader035.vdocuments.mx/reader035/viewer/2022081716/53fea2c68d7f72965c8b45a3/html5/thumbnails/33.jpg)
FOSS4G Europe 2014, Bremen 14th-17th July 2014
• Advanced Raster Algebra using Jiffle
• Wraps Jiffle (http://code.google.com/p/jiffle/) in order to execute complex Raster Algebra scripts against the input layers
Raster Algebra Processes
Jiffle is a scripting language for creating and analysing raster images. Rather than having to write and test lots of JAI or Java AWT boiler-plate code to access and manipulate images, Jiffle lets you concentrate on the interesting bit: your algorithm. Jiffle is being developed as part of the JAITools project. Update 16 January 2013: Jiffle sources have moved to GitHub
![Page 34: Processing data in GeoServer with WPS and SQL views](https://reader035.vdocuments.mx/reader035/viewer/2022081716/53fea2c68d7f72965c8b45a3/html5/thumbnails/34.jpg)
FOSS4G Europe 2014, Bremen 14th-17th July 2014
• Advanced Raster Algebra using Jiffle
• Gets an OGC Filter containing logical operations between raster layers and produces a binary light layer
Raster Algebra Processes
![Page 35: Processing data in GeoServer with WPS and SQL views](https://reader035.vdocuments.mx/reader035/viewer/2022081716/53fea2c68d7f72965c8b45a3/html5/thumbnails/35.jpg)
FOSS4G Europe 2014, Bremen 14th-17th July 2014
• Raster Algebra detailed info
• Input parameters and outcomes are logged into WFS
• WPS Raster Stats Process provides statistics on the coverages
Raster Algebra Processes
Raster Statistics and Area in sq Km
Raster Algebra Process user inputs
![Page 36: Processing data in GeoServer with WPS and SQL views](https://reader035.vdocuments.mx/reader035/viewer/2022081716/53fea2c68d7f72965c8b45a3/html5/thumbnails/36.jpg)
Script me plenty An example using GeoScript
![Page 37: Processing data in GeoServer with WPS and SQL views](https://reader035.vdocuments.mx/reader035/viewer/2022081716/53fea2c68d7f72965c8b45a3/html5/thumbnails/37.jpg)
FOSS4G Europe 2014, Bremen 14th-17th July 2014
Sample scripting application
http://localhost:8080/geoserver/www/wps.html
![Page 38: Processing data in GeoServer with WPS and SQL views](https://reader035.vdocuments.mx/reader035/viewer/2022081716/53fea2c68d7f72965c8b45a3/html5/thumbnails/38.jpg)
FOSS4G Europe 2014, Bremen 14th-17th July 2014
Land property distrib. in the polygon
![Page 39: Processing data in GeoServer with WPS and SQL views](https://reader035.vdocuments.mx/reader035/viewer/2022081716/53fea2c68d7f72965c8b45a3/html5/thumbnails/39.jpg)
FOSS4G Europe 2014, Bremen 14th-17th July 2014
A look at the data dir
1. Data
2. WPS process scripted in Jython
3. The GUI
![Page 40: Processing data in GeoServer with WPS and SQL views](https://reader035.vdocuments.mx/reader035/viewer/2022081716/53fea2c68d7f72965c8b45a3/html5/thumbnails/40.jpg)
FOSS4G Europe 2014, Bremen 14th-17th July 2014
The Jython script
http://geoscript.org/py/index.html
![Page 41: Processing data in GeoServer with WPS and SQL views](https://reader035.vdocuments.mx/reader035/viewer/2022081716/53fea2c68d7f72965c8b45a3/html5/thumbnails/41.jpg)
The Destination Project
![Page 42: Processing data in GeoServer with WPS and SQL views](https://reader035.vdocuments.mx/reader035/viewer/2022081716/53fea2c68d7f72965c8b45a3/html5/thumbnails/42.jpg)
FOSS4G Europe 2014, Bremen 14th-17th July 2014
• The Destination project computes the risk of accidents involving dangerous goods (chemicals, petrol, gases and so on)
Intro
Road segments and stats about car accidents
Human and environmental «targets»
Involved area, depending on type of good and amount of damage
![Page 43: Processing data in GeoServer with WPS and SQL views](https://reader035.vdocuments.mx/reader035/viewer/2022081716/53fea2c68d7f72965c8b45a3/html5/thumbnails/43.jpg)
FOSS4G Europe 2014, Bremen 14th-17th July 2014
• Road network of good part of northern Italy
• Road divided into segments
• 100m portions (500k of them)
• 500m aggregation (120k of them)
• 1 km square cells (few hundreds)
• 51 buffer distances (depending on good, scenario, level of damage)
• Several types of targets: schools, malls, hospitals, populated areas, superficial and underground acquifers, crops, woods, ….
Large Data Volume
![Page 44: Processing data in GeoServer with WPS and SQL views](https://reader035.vdocuments.mx/reader035/viewer/2022081716/53fea2c68d7f72965c8b45a3/html5/thumbnails/44.jpg)
FOSS4G Europe 2014, Bremen 14th-17th July 2014
The road arc risk formula (s)
• Adding togheter the risk caused by the different
• Arc own propension to accidentds
• Types of goods
• Human and enviromental targets
• The system allows to compute partial views of the formula, either by selection of targets/goods or by computing portions of it
• Has a number of coefficients that can be hand-tuned by the caller
![Page 45: Processing data in GeoServer with WPS and SQL views](https://reader035.vdocuments.mx/reader035/viewer/2022081716/53fea2c68d7f72965c8b45a3/html5/thumbnails/45.jpg)
FOSS4G Europe 2014, Bremen 14th-17th July 2014
The results, visually
• Rendering transformation
• Read the arcs/polys from the DB, compute their risk based on the chosen formula, scenario, targets, and coefficients
![Page 46: Processing data in GeoServer with WPS and SQL views](https://reader035.vdocuments.mx/reader035/viewer/2022081716/53fea2c68d7f72965c8b45a3/html5/thumbnails/46.jpg)
FOSS4G Europe 2014, Bremen 14th-17th July 2014
How to compute it efficiently?
• Using SQL Views? No, the possible aggregations variants are too many
• Using a pure Java process? No, too much data to transfer from the DBMS
• Fully on the fly? No, too much data involved
• Pre-compute all buffers and locate all involved targets before hand (pre-cooked per buffer risk)
• Use a process that builds a final aggregation query on the fly (dynamic sql views)
![Page 47: Processing data in GeoServer with WPS and SQL views](https://reader035.vdocuments.mx/reader035/viewer/2022081716/53fea2c68d7f72965c8b45a3/html5/thumbnails/47.jpg)
FOSS4G Europe 2014, Bremen 14th-17th July 2014
Efficient rendering tx
Risk process
Queries database
Arcs/Buffer areas db
Map renderer
Build overall query, replace params
Compute risk for a batch of arcs
Raw arcs
Arcs + risk
• Compute risk on the fly in the viewing area
• Batch requests to the BDMS to minimize round-trip overhead
![Page 48: Processing data in GeoServer with WPS and SQL views](https://reader035.vdocuments.mx/reader035/viewer/2022081716/53fea2c68d7f72965c8b45a3/html5/thumbnails/48.jpg)
FOSS4G Europe 2014, Bremen 14th-17th July 2014
Efficient cross layer filtering
• Show only targets involved in the scenario under study, e.g., the ones crossing the buffer areas where there is significant risk
• Limit query to the current bbox
![Page 49: Processing data in GeoServer with WPS and SQL views](https://reader035.vdocuments.mx/reader035/viewer/2022081716/53fea2c68d7f72965c8b45a3/html5/thumbnails/49.jpg)
FOSS4G Europe 2014, Bremen 14th-17th July 2014
Efficient cross layer filtering
SELECT v_geo_popolazione_residente_pl.*
FROM v_geo_popolazione_residente_pl
WHERE v_geo_popolazione_residente_pl.fk_bersaglio_umano_pl in
(
SELECT distinct bersaglio.fk_bersaglio_umano_pl
FROM v_geo_popolazione_residente_pl bersaglio
join siig_geo_ln_arco_1 on
st_dwithin(bersaglio.geometria,
siig_geo_ln_arco_1.geometria,
%distanzaumano%) WHERE siig_geo_ln_arco_1.geometria &&
st_makeenvelope(%bounds%, 32632) )
• This is a job for a parametric sql view