State ofGeoServer 2016

Victoria GeoGeeks

Andrea AimeTechnical Leadandrea.aime@geo-solutions.it@geowolf

GeoSolutionsInnovative, robust and cost-effective solutions leveraging best-of-breed Open Source products.

WelcomeOpen Source ProjectsGeoToolsGeoServerImageIO-ExtJAI-ExtJAI Tools

OSGeo FoundationCharter Member

Jody GarnettCommunity Leadjgarnett@boundlessgeo.com@jodygarnett

BoundlessProvides geospatial tools and services for managing data and building applications.

WelcomeOpen Source ProjectsGeoToolsGeoServeruDig

OSGeo FoundationBoard MemberOSGeo Incubation ChairGeoTools Project Officer

Eclipse FoundationLocationTech Steering CommitteeLocationTech Technology Project

GeoServer 2016

GeoServer at a GlanceJava Web Application to share and edit geospatial data.

Publish data from any major spatial data source using open standards.

Core Protocols

WMS – mapsWFS – vector WFS-T – editing WCS – coverageWPS – process

Tile Protocols

WMTS – tilesTileService – tilesWMS-C – tiles

28 → 37 members of team geoserver624 → 619 members on geoserver-devel500+ → 525 pull requestsNew contributors being added(others stepping back)Smaller base of active contributors this year.Pull requests still going strong, very active codebase.Simone replaces Andrea as OSGeo Officer.13 releases this year

Health CheckupHow about team user-list?

1983 → 2005 geoserver-users

Also geoserver-italia, espanol and opengeo.cn mailing lists

Release Schedule

stable maintenance

} delay

August 2.10.5

July

June 2.10.4

May

April 2.10.3

March 2.9.5

February 2.10.2

January 2.9.4

December 2.10.1

November 2.9.3

October 2.10.0

September 2.8.5 2.9.2

Take care to report the issue in a responsible fashion:

● Keep exploit details out of issue report● Mark the issue as a vulnerability.● Be prepared to work with Project Steering Committee (PSC) on a solution● Keep in mind PSC members are volunteers

and an extensive fix may require fundraising / resources

If you are not in position to communicate in public please consider commercial support, contacting a PSC member, or reaching us via the Open Source Geospatial Foundation at info@osgeo.org.

Security Vulnerabilities - Responsible Disclosure

Security VulnerabilitiesFixed for GeoServer 2.9.1 (backported to 2.8.5)

● Remote Execution (RE) Vulnerability● Limited to REST API administration

Fixed for GeoServer 2.10 (backported to 2.9.2)

● XML External Entity (XXE) Processing● Please update now!

Q: “Update the headers” was the most common feedback on incoming pull requests. Was it needed?

A: Checked with OSGeo legal (thanks board):

- headers are an historical artifact from before the Berne Convention

- no longer needed, now just informative.

Header Maintenance

Maintenance &Technical Debt

● Relying on volunteer time to keep an eye on the codebase was not working out

○ team busy on customer work?○ backlog of bugs piling up

● Trying an experiment○ Once a month meeting○ Distributed, on-line code sprint

● When:○ Join us on November 18th!

Monthly bug fixing code sprints

Monthly bug fixing code sprints● Actions

○ Vet existing tickets○ close invalid/non reproducible ones○ close feature requests that failed to gather

resourcing in a long time...○ Fix bugs

● Nice side effects:○ Gets more developers familiar with the

contents of the issue tracker○ Opportunity to get proficient with new parts

of the code base○ Predictable, scheduled activity

Wicket 7Held sprint in victoria to clear up this long standing technical debt!

Thanks to OSGeo, Boundless, Vivid Solutions, How 2 Map, San Jose Water Company, Transient, GeoBeyond, Scitus, GeoSolutions, CCRi, Astun Technology, and Voyager Search for sponsoring this work.

2.9 2.10Victoria Code Sprint

Java 8GeoServer now requires Java 8.

Why? What changed ...

- Traditionally GeoServer takes forever to support the latest Java- Oracle now charges for Java 7 security updates

2.9 2.10Joint EffortCommunity

Required to upgrade the “Spring Framework” used to wire GeoServer together:

- We needed a newer version of “Spring” for Java 8 compatibility

- Upgraded to Spring 4 → 2 month delay to GeoServer 2.9 release

Check that your application server supports:

- Servlet 3.0 was released in 2009 - Supported by Tomcat 7 and Tomcat 8

Spring 4

2.9 2.10Joint-EffortCommunity

Vector data sources

GeoPackage module moving to supported land● Approved “Community module” → “Official plugin” on October 26th

● Increasing test coverage● More compatibility testing with OGR and friends● OGC compliance tests

2.10Ian TurtonGeoSolutions

Swedish University of Agricultural Sciences

WFS cascading updates● The original WFS client is being retired● WFS-NG will be taking over● Compatibility testing done with MapServer, TinyOWS, ….

2.10Scitus DevelopmentBoundless

TinyOWS

ArcGIS WFSWFS/WMS/WMTS

WFS

Raster data sources

Improved masking support● Raster masking available for both image mosaic and GDAL data sources● Both in vector (polygon footprint) and raster (binary mask) for

2.8 2.9 2.10GeoSolutionsLiguria digitale

Heterogeneous mosaics - color models● Support for mosaics with sources in multiple color models

(RGB, gray, indexed), since 2.8.x

2.8 2.9 2.10GeoSolutions

Heterogeneous mosaics - resolution / projections

2.10BoundlessSOCOM

● Support for mosaics in multiple coordinate reference systems(Coming in 2.10.x)

Excess granule removal ● Optimization for deep z-order mosaic stacks● E.g., time series of scattered data● Do not load imagery that is not contributing to the output

2.9 2.10GeoSolutionsBEV

In development!

Multiple coverages for mosaic and pyramid (2.10)● Already available for multidimensional data (NetCDF, Grib)● Now available also for “flat” data sources● Meaningful if the sources are normally managed and used together (e.g.,

multispectral satellite data and its derived products)

2.9 2.10ImplementorBEV

Coverage view optimizations (2.10)● A coverage view allows to bind together bands from different coverages

coming from the same source (e.g., NetCDF)● Optimization to avoid reading all sources if the style is only using a subset of

the bands

A

B

C

CombinerSLD Style

selecting only A

Map

Hint about which bands are actually going to be

used

GeoSolutionsBEV 2.9 2.10

GeoServer support for rotated pole projections

2.9 2.10

Rotated pole projection support for GRIB2/NetCDF/ImageMosaic data stores and WCS NetCDF output.

Publish the native GRIB2 file format of NOAA RAPv3 North American weather forecast model: http://rapidrefresh.noaa.gov/

Upgraded to NetCDF Java 4.6.6 (Ben contributed a fix in it)

TransientBoundless/NOAA

Styling

Perpendicular Offset for Lines and Polygons

<LineSymbolizer>

<Stroke>

<CssParameter name="stroke">#AAAAAA</CssParameter>

<CssParameter name="stroke-width">3</CssParameter>

</Stroke>

<PerpendicularOffset>-2</PerpendicularOffset>

</LineSymbolizer>

<LineSymbolizer>

<Stroke>

<CssParameter name="stroke">#FF0000</CssParameter>

<CssParameter name="stroke-width">3</CssParameter>

<CssParameter name="stroke-dasharray">5 2</CssParameter>

</Stroke>

<PerpendicularOffset>3</PerpendicularOffset>

</LineSymbolizer>

2.8 2.9 2.10GeoSolutionsDLR

ncWMS like extensions to GetMap (community)● A new styling language designed for dynamic colormaps● Applies to all rasters based on their statistics● Use can control min/max, log scale, animation

2.10GeoSolutions

Style Page

2.10BoundlessCommunity

Refactored CSS Style Page into the main application.

Works withSLD, CSS, YSLD.

Improved experience for everybody!

Live Demo Next

New features for CSS (2.10)● Rendering transformations supported (feature parity with SLD!)● Rule nesting, helps with selector duplication and makes styles more readable

/* @title Levels */* { transform: ras:Contour(levels: 1100 1200 1300 1400 1500 1600 1700); stroke: black; z-index: 0;}

/* @title Values */[@scale < 3000] { transform: ras:RasterAsPointCollection(); mark: symbol('square'); :mark { size: 2; fill: black; } label: [GRAY_INDEX]; label-anchor: 0 0.5; label-offset: 5 0; font-family: Arial; font-fill: black; z-index: 1;}

Extract contours at the desired levels

When zoomed-in enough, extract point values, display center with a mark, and value on top of it

2.10Andrea AimeCommunity

YSLD community module● Style Layer Descriptor represented in

YAML○ Uses indentation rather than XML○ Easier to read○ More compact○ More flexible syntax○ Contains variables for reusable code○ Compatible with SLD

● REST API can convert formats○ Preview your existing styles as YSLD

using REST API○ View the SLD representation of YSLD

2.10Implementor

symbolizers:- polygon: stroke-color: 'blue' stroke-width: 1 fill-color: '#7EB5D3'- text: label: ${name} fill-color: 'black' anchor: [0.5, 0.5] x-maxDisplacement: 40 x-autoWrap: 70

Community

Custom Legend GraphicTake control of the legend graphic(Advertised in WMS GetCapapabilities).

Legend shown in compatible clients when listing your layer.

2.9 2.10BoundlessBoundless

Improved QGIS Compatibility● Additional custom “Well Known” marks● Support for SLD 1.1 External Marks● SVG Parameters 1.0 spec

<PointSymbolizer> <Graphic> <ExternalGraphic> <OnlineResource xlink:type="simple" xlink:href="firestation.svg?fill=#FF0000" /> <Format>image/svg</Format> </ExternalGraphic> <size>128</size> </Graphic> </PointSymbolizer>

2.10Andrea AimeCommunity

WMS

image/vnd.jpeg-png● Smart format, chooses

between JPEG and PNG depending on image contents

● JPEG for max compression on fully opaque images

● PNG when transparency is needed

● Supported both in WMS and WMTS

PNG PNG PNG

PNG JPEG JPEG

2.9 2.10GeoSolutions

UTFGrid (format=application/json;type=utfgrid)

● MapBox style UTFGrid output for WMS and WMTS

● More flexibility○ Non square tiles○ Available in all projections

● Of course you can limit yourself to simple tiles

2.9 2.10GeoSolutionsCianad

Vector ExtensionCommunity module for publishing vector tiles

- Great solution for modern High Definition Screens

- Use for GeoWebCache for “MapBox without MapBox”

- (see talk tomorrow)

2.10ImplementorCustomer

Tiling

MBTiles like storage● Store tiles in MBTiles database● Possibility to use many databases, following a pattern, e.g.,

○ /path/to/{grid}/{dim}/{tileset}/{z}/{x}-{y}.sqlite (x and y are actually groups of values)○ Helps with scalability, especially if not fully seeded

● Allows to store non Google Mercator projections (extension)● Allows to store formats other than png or jpeg (extension)● Stores in lookup table creation date for expiration support● REST API to replace single files (automate migration from seeding

machine to serving ones)● Not compatible with disk quota!

https://github.com/GeoWebCache/geowebcache/wiki/MBTiles-BlobStore

2.9 2.10GeoSolutionsCapgemini

WMTS config page● Allows WMTS to have

its own service metadata configured from the UI

● Adds a point where the INSPIRE module can plug into for extra configuration

2.9 2.10GeoSolutionsBEV

WMTS workspace specific and INSPIRE compatibility● WMTS now works also as a

workspace specific service (finally tile caching for multitenant setups)

● Allows to setup INSPIRE harmonized layer names

● INSPIRE module extended to inject INSPIRE minimum metadata in WMTS GetCapabilities output

Workspace1

Workspace 2

INSPIREcompliant

Generic tiles

2.9 2.10GeoSolutionsBEV

WMS/WMTS ND discovery extensions

● Data with N dimensions (time, elevation, custom ones)● Dimensions can be related (forecasts, time and run time), the data can be

scattered (remote sensing), hard to locate data in the ND cube● Add extra calls to query the domains● Check the current specification and give us feedback

2.9 2.10GeoSolutionsBEV

In development!http://demo.geo-solutions.it/share/wmts-multidim/wmts_multidim_geosolutions.html

WPS

Improved WPS resource control● Processes are run in thread pools, they

can wait for their execution slot in queue● Separate control for total time, and pure

execution time

2.10utas.edu.au

Download process improvements (community)

● Community module for“clip and ship” support

● Asynch requests allow for large extractions

● Can have maximum extraction limits configured

● Added ability to also select bands, and improved output size estimation

WPS

2.9 2.10GeoSolutionsBEV

Configuration and management

Signed “OSGeo” Distributions AvailableRequired for macOS 10.12 Sierra Required for Windows 10

2.9 2.10BoundlessOSGeo

● Allows GeoServer to store resources(icons, fonts, config files) in either file system or database “blobs”.

● Run without a “Data Directory”(Helpful in clustered/cloud environment)

● All instances of file access were converted to resource api

● Backwards compatible - file will be unpacked on disk if required

● Manage via REST API● GUI available as community module

Resource API

2.9 2.10Scitus DevelopmentBoundless/NGA

Web Services

Configuration

DataDirectory

ResourceCatalog

Web Services

Configuration

JDBCStore

ResourceCatalog

Web interface in Tools / Resource Browser

● Manage and edit resources

● Data Directory or JDBCStore

● Community module

● Packaged plugin coming soon

Resource Browser (community)

2.10Scitus DevelopmentBoundless/NGA

Module “status” REST API (and GUI)● Automated testing of GeoServer is a

challenge● REST API access to the “status page”● Easy to add additional entries● Used to check environment for difficult to

configure modules that use native code

2.9 2.10BoundlessBoundless QA

Module name: Rendering Engine● Module : jvm● Component : java2d● Version : 1.8.0_66● Enabled : true● Available : true● Message : Java 2D configured with

DuctusRenderingEngine.● Provider: OracleJDK

geoserver/rest/about/status

Parametric configuration● Switching data directories between

integration/beta/production environments can be a pain

● GeoServer 2.10 allows to stick parameters in the configuration, and read them from the environment

● Look ma, switching environments without grep/sed on the data dir :-)

● Store params in property file, system or environment variables

2.9 2.10GeoSolutionsEumetsatWorld Bank

Backup/restore (community)● Save the configuration to a zip file (not the

data!)● Restore later, in the same or different

environment● UI to run backup and restore● Asynch operations

○ During backup the configuration is forced read only○ During restore it’s fully locked

2.9 2.10GeoSolutionsEumetsatWorld Bank

Backup/restore (community)● Can do a “dry run”

to check if there is any error prior to restore

2.9 2.10GeoSolutionsEumetsatWorld Bank

Backup/restore (community)● REST API to run, control,

eventually cancel backups and restores

2.9 2.10GeoSolutionsEumetsatWorld Bank

Security

LDAP User Group Service● We previously only had an “authentication

provider”, meaning, it used the username/pw provided and tried to open a LDAP connection with them

● We now also have a “user group service”,it opens a LDAP connection using fixed credentials, and searches the user in the LDAP database, matching the password

● E.g., similar to storing users in a DBMS

2.10Scitus development?

Research and Development

GeoGig (community)Manage repositories and publish via WFS and WFS-T for web client access.

2.9 2.10BoundlessGeoGig

GeoGig (community)

curl -X PUT -H "Content-Type: application/json" -d '{ "parentDirectory": "/opt/geogig/repos", "authorName": "geogig", "authorEmail": "geogig@geogig.org" }' "http://localhost:8080/geoserver/geogig/repos/myrepo/init"

REST API for managing repositories, when combined with GeoServer REST API a wide range of scripts are possible.

2.9 2.10BoundlessGeoGig

Looking Ahead

Java 9 CompatibilityJava 9 has been delayed until 2017, but we would like to ensure GeoServer will function. Oracle will stop shipping free security updates to Java 8 on

We have a couple incompatibilities with the GeoTools “service provider interface” plugin system that we will look at resolving during the code sprint!

The Java Advanced Imaging library enables GeoServer to do all kinds of great image processing! The JAI-EXT project extends this library with replacement no-data / footprint aware operations.

Sadly Oracle has not maintained the project, and has not released it as open source.

(see Replace-JAI on GeoTools wiki)

Raster Processing EngineJoint OSGeo / LocationTech game plan:

● Raster Processing Engine APIdefinition of raster-processing-api

● Raster Processing Enginereference pure-java implementation

● Raster Processing Wrapperbenchmark jai-wrapper implementation

● GeoTools Replace JAImigrate to raster processing engine api and coordinate transition affected projects (GeoServer, GeoNode, uDig, GeoScript, etc...)

Thanks!