Download - GDAL 2.0 overview - FOSS4G NA 2015 2.0... · GDAL 2.0 overview About me Contributor to the GDAL/OGR project since 2007 and Project Steering Commitee member since 2008. Chair of the

GDAL 2.0 overview

Even RouaultSPATIALYS

March 12th 2015

GDAL 2.0 overview

Plan

● Introduction to GDAL/OGR● Community ● GDAL 1.11 summary● GDAL 2.0 features● Future directions● Discussion

GDAL 2.0 overview

About me

● Contributor to the GDAL/OGR project since 2007 and Project Steering Commitee member since 2008.

● Chair of the PSC since 2015.● Contributor to MapServer● Co-maintainer: libtiff, libgeotiff, PROJ.4● Funder of Spatialys, consulting company around

geospatial open source technologies, with a strong focus on GDAL and MapServer

GDAL 2.0 overview

GDAL/OGR Introduction

● Geospatial Data Abstraction Library● Raster (GDAL) and Vector (OGR)● Read/write access to more than 200 geospatial formats and

protocols● Widely used (FOSS & proprietary): GRASS, MapServer,

Mapnik, QGIS, PostGIS, OTB, SAGA, FME, ArcGIS, Google Earth…

(93 referenced at http://trac.osgeo.org/gdal/wiki/SoftwareUsingGdal)

● Started in 1998 by Frank Warmerdam● A project of OSGeo since 2008● MIT/X Open Source license (permissive)● 1.2 M physical lines of code (175 kLOC from external

libraries) for library, utilities and language bindings● 166 kLOC for Python autotest suite. 61% line coverage

http://trac.osgeo.org/gdal/wiki/SoftwareUsingGdal

GDAL 2.0 overview

Main features

● Support datasets of arbitrary size with limited resources● C++ library with C API● Multi OS: Linux, Windows, MacOSX/iOS, Android, ...● Language bindings: Python, Perl, C#, Java,...● OGC WKT coordinate systems, and PROJ.4 as

reprojection engine● Format support through drivers implemented a common

interface● Utilities for translation,reprojection, subsetting,

mosaicing, tiling…● Can file with local, remote (/vsicurl), compressed

(/vsizip/, /vsigzip/, /vsitar), in-memory (/vsimem) files

GDAL 2.0 overview

Raster Features

● Efficient support for large images (tiling, overviews)● Block caching● Warping engine● Various algorithms: rasterization, vectorization

(polygonization / contour), interpolation of void cells, sieve filter

● Utilities for translation & subsetting (gdal_translate), warping/reprojection (gdalwarp), pyramid building (gdaladdo), mosaicing (gdalwarp/gdalbuildvrt), indexing (gdaltindex), tiling (gdal_retile, gdal2tiles), DEM processing (gdaldem), grid interpolation, ...

GDAL 2.0 overview

Raster Formats

● Plain Raster (images): JPEG, PNG, GIF, WebP, BPG ...● Geospatial Files: GeoTIFF, .img, NITF, ...● Wavelet: JPEG 2000, ECW, MrSID, ...● RDBMS: Oracle Raster, PostGIS Raster, Rasdaman● Portable database: Rasterlite, MBTiles, GeoPackage● Web Service: WMS, WCS● Radar: CEOS, Envisat● Elevation: DTED, USGS DEM, SRTM HGT● Containers: HDF4, HDF5, NetCDF● Other: Geospatial PDF● GDAL specific: in-memory, VRT

⇒ 137 Formats

GDAL 2.0 overview

Vector Features

● OGC/ISO geometry model● Commandline utilities for translation (ogr2ogr), query

(ogrinfo), indexing (ogrtindex)● Spatial filter● GEOS for geometry operations● SQL:

○ OGR SQL or SQLite engine for all formats○ Passthrough SQL for RDBMS

GDAL 2.0 overview

Vector Formats

● GIS: Shapefile, MapInfo, ESRI Personal/File Geodatabase● CAD: DXF, DWG, DGN (pre-V7)● RDBMS: PostGIS, Oracle, MySQL, Ingres, MSSQL, ODBC● Portable databases: SQLite/Spatialite, GeoPackage● Exchange: KML (GDAL 1.11:reference implementation)

GML, GeoJSON● Web Service: WFS, Fusion Tables, CartoDB, CouchDB,

Cloudant, GME● National: SDTS, NAS, NTF, TIGER/Line, Interlis, VFK,

Edigeo, SOSI, SXF, MTK GML, RUIAN GML, INSPIRE Cadastral GML

● Non spatial : CSV, XLS, XLSX, ODS● GDAL specific: in-memory, VRT

⇒ 81 formats

GDAL 2.0 overview

Community

● 57 SVN committers registered○ 21 active in last 12 months + 42 non-committer

contributors○ https://www.openhub.net/p/gdal

● 2100 gdal-dev list members. 2900 messages in last 12 months, 759 discussion threads, 74% with at least one answer

● About 500 tickets filed in last 12 months (5850 over history). ~580 opened, after ~200 closed as a winter cleanup (special thanks to Jukka Rahkonen!)

https://www.openhub.net/p/gdal

https://www.openhub.net/p/gdal

GDAL 2.0 overview

GDAL/OGR 1.11.0 Release

● Released April 2014● 2 bugfixes releases with ~140 tickets fixed:

GDAL 1.11.2 released in February 2015.● 8 new drivers :

○ raster: KRO○ vector: CartoDB, GME, GeoPackage, OpenFileGDB,

SXF, WALK, WaSP● 5 new RFCs including:

○ Enhanced Raster Attribute Table support (RFC 40)○ Multiple geometry fields per feature (RFC 41)○ Raster access as virtual memory mapping (RFC 45)

GDAL 2.0 overview

… and soon GDAL 2.0 !

● 8 RFCs implemented over GDAL 2.0 cycle○ Trade-off: new features vs backward compatibility○ Application code and out-of-tree drivers will need updates, mostly on

the vector side. See MIGRATION_GUIDE.TXT● 7 new drivers :

○ BPG (Better Portable Graphics)○ GeoPackage Raster○ KEA○ ROI_PAC (NASA JPL format)○ Cloudant○ JML (OpenJUMP)○ Selaphin (Telemac hydraulic model)

GDAL 2.0 overview

RFC 46: GDAL/OGR unification

● A long-time dream come true!● Raster and vector drivers unified :

○ handled by the same registrar/manager○ vector drivers can have metadata○ efficient driver identification

● Raster and vector datasets unified :○ a dataset can have both raster bands and vector layers.○ vector datasources&layers can have metadata

● Unified drivers: PCIDSK, PDF, GeoPackage

GDAL 2.0 overview

RFC 46: GDAL/OGR unification

● Open options (to avoid global configuration options)● C API backward compatibility preserved.● C++ API impacted. e.g. OGRSFDriverRegistrar::Open()

replaced by GDALOpenEx()

More at https://trac.osgeo.org/gdal/wiki/rfc46_gdal_ogr_unification

https://trac.osgeo.org/gdal/wiki/rfc46_gdal_ogr_unification

GDAL 2.0 overview

RFC 46: GDAL/OGR unification$ ogrinfo --format postgresqlFormat Details:

Short Name: PostgreSQL

Long Name: PostgreSQL/PostGIS

Supports: Vector

Help Topic: drv_pg.html

Supports: Open() - Open existing dataset.

Supports: Create() - Create writeable dataset.

Creation Field Datatypes: Integer Integer64 Real String Date DateTime Time IntegerList Integer64List RealList StringList Binary

Supports: Creating fields with NOT NULL constraint.

Supports: Creating fields with DEFAULT values.

Supports: Creating geometry fields with NOT NULL constraint.

<CreationOptionList />

<OpenOptionList>

<Option name="LIST_ALL_TABLES" type="boolean" description="Whether all tables, including non-spatial ones, should be listed" default="NO" />

</OpenOptionList>

GDAL 2.0 overview

RFC 46: GDAL/OGR unification<LayerCreationOptionList>

<Option name="GEOM_TYPE" type="string-select" description="Format of geometry columns" default="geometry">

<Value>geometry</Value>

<Value>geography</Value>

</Option>

<Option name="OVERWRITE" type="boolean" description="Whether to overwrite an existing table with the layer name to be created" default="NO" />

<Option name="LAUNDER" type="boolean" description="Whether layer and field names will be laundered" default="YES" />

<Option name="PRECISION" type="boolean" description="Whether fields created should keep the width and precision" default="YES" />

<Option name="DIM" type="integer" description="Set to 2 to force the geometries to be 2D, or 3 to be 2.5D" />

<Option name="GEOMETRY_NAME" type="string" description="Name of geometry column. Defaults to wkb_geometry for GEOM_TYPE=geometry or the_geog for GEOM_TYPE=geography" />

<Option name="SCHEMA" type="string" description="Name of schema into which to create the new table" />

<Option name="SPATIAL_INDEX" type="boolean" description="Whether to create a spatial index" default="YES" />

<Option name="TEMPORARY" type="boolean" description="Whether to a temporary table instead of a permanent one" default="NO" />

<Option name="NONE_AS_UNKNOWN" type="boolean" description="Whether to force non-spatial layers to be created as spatial tables" default="NO" />

<Option name="FID" type="string" description="Name of the FID column to create" default="ogc_fid" />

<Option name="FID64" type="boolean" description="Whether to create the FID column with BIGSERIAL type to handle 64bit wide ids" default="NO" />

<Option name="EXTRACT_SCHEMA_FROM_LAYER_NAME" type="boolean" description="Whether a dot in a layer name should be considered as the separator for the schema and table name" default="YES" />

</LayerCreationOptionList>

GDAL 2.0 overview

RFC 46: GDAL/OGR unification$ gdalinfo --format gpkgFormat Details:

Short Name: GPKG

Long Name: GeoPackage

Supports: Raster Supports: Vector Extension: gpkg

Help Topic: drv_geopackage.html

Supports: Subdatasets

Supports: Open() - Open existing dataset.

Supports: Create() - Create writeable dataset.

Supports: CreateCopy() - Create dataset by copying another.

Supports: Virtual IO - eg. /vsimem/

Creation Datatypes: Byte

Creation Field Datatypes: Integer Integer64 Real String Date DateTime Binary

Supports: Creating fields with NOT NULL constraint.

Supports: Creating fields with DEFAULT values.

Supports: Creating geometry fields with NOT NULL constraint.

GDAL 2.0 overview

RFC 46: GDAL/OGR unification<CreationOptionList>

<Option name="RASTER_IDENTIFIER" type="string" description="Human-readable identifier (e.g. short name)" />

<Option name="RASTER_DESCRIPTION" type="string" description="Human-readable description" />

<Option name="BLOCKSIZE" type="int" description="Block size in pixels" default="256" max="4096" />

<Option name="TILE_FORMAT" type="string-select" description="Format to use to create tiles" default="PNG_JPEG">

<Value>PNG_JPEG</Value>

<Value>PNG</Value>

<Value>PNG8</Value>

<Value>JPEG</Value>

<Value>WEBP</Value>

[...]

</Option>

<LayerCreationOptionList>

<Option name="GEOMETRY_NAME" type="string" description="Name of geometry column." default="geom" deprecated_alias="GEOMETRY_COLUMN" />

<Option name="GEOMETRY_NULLABLE" type="boolean" description="Whether the values of the geometry column can be NULL" default="YES" />

<Option name="PRECISION" type="boolean" description="Whether text fields created should keep the width" default="YES" />

<Option name="SPATIAL_INDEX" type="boolean" description="Whether to create a spatial index" default="YES" />

[...]

</LayerCreationOptionList>

GDAL 2.0 overview

RFC 51: RasterIO improvements

● GDAL raster band or dataset RasterIO methods can now accept a resampling method:

■ Nearest neighbour (default, fastest but not pretty)■ Bilinear■ Bicubic (Catmull Rom)■ Cubic-spline■ Lanczos■ mode (majority)■ Gauss

gdal_translate -r, GDALRasterIOEx(), GDALDatasetRasterIOEx()● Intel SSE-2 acceleration of most of those methods available● Progress/cancel callback available● 64-bit spacings in user buffer possibleMore at https://trac.osgeo.org/gdal/wiki/rfc51_rasterio_resampling_progress

https://trac.osgeo.org/gdal/wiki/rfc51_rasterio_resampling_progress

GDAL 2.0 overview

RFC 51: RasterIO improvements● 2% reduction of a 21600x21600 uncompressed tiled GeoTIFF BMNG

Nearest neighbour, 45s total (5s CPU) Bicubic, 1m5s total (15s CPU)

GDAL 2.0 overview

RFC 51: RasterIO improvements● 75% reduction of a OpenStreetMap extract

Nearest neighbour, ~100ms Lanczos, ~100ms

GDAL 2.0 overview

RFC 41: Multiple geometry fields

● Available in GDAL 1.11● Proper solution to hacks: multiple layers,

GEOMETRYCOLLECTION● Addition of OGRGeomFieldDefn object, integrated with

OGRFeature/OGRFeatureDefn similarly to OGRFieldDefn○ field name○ geometry type: Point, LineString, Polygon, …○ spatial reference system○ not null constraint (GDAL 2.0)

● OGRFeature::GetGeomFieldRef(idx) / SetGeomField(idx, geom)

● ogrinfo / ogr2ogr updated● Driver support: PostGIS, GML, SQLite/Spatialite (GDAL 2.0),

Interlis, CSV, Memory, VRTMore at https://trac.osgeo.org/gdal/wiki/rfc41_multiple_geometry_fields

https://trac.osgeo.org/gdal/wiki/rfc41_multiple_geometry_fields

GDAL 2.0 overview

RFC 41: Multiple geometry fieldsLayer name: Bodenbedeckung__BoFlaechen

Geometry (Form): Polygon

Geometry (Form__Point): Point

Feature Count: 3

Extent (Form): (113.000000, 149.940000) - (194.799782, 228.970000)

Extent (Form__Point): (133.950000, 170.850000) - (168.270000, 206.060000)

SRS WKT (Form):(unknown)

SRS WKT (Form__Point):(unknown)

Geometry Column 1 = Form

Geometry Column 2 = Form__Point

_TID: String (0.0)

Art: Integer (0.0)

Form__Point_0: Real (0.0)

Form__Point_1: Real (0.0)

OGRFeature(Bodenbedeckung__BoFlaechen):10

_TID (String) = 10

Art (Integer) = 0

Form__Point_0 (Real) = 148.2

Form__Point_1 (Real) = 183.48

Form = POLYGON ((146.92 174.98 0,138.68 187.51 0,147.04 193.0 0,149.79 188.82 0,158.15 194.31 0,163.64 185.96 0,146.92 174.98 0))

Form__Point = POINT (148.2 183.48)

GDAL 2.0 overview

RFC 49: Curve geometries

● GDAL 1.X : OGC Simple Feature Access Part 1 v1.1.0● GDAL 2.0: ISO SQL/MM Part 3 / OGC SFA v1.2.1

⇒ 5 new geometry types : CircularString, CompoundCurve, MultiCurve, CurvePolygon, MultiSurface

GDAL 2.0 overview


Exemple: MULTISURFACE(

CURVEPOLYGON(COMPOUNDCURVE( CIRCULARSTRING(5 0,0 5,5 10),

(5 10,15 10),CIRCULARSTRING(15 10,20 5,15 0),(15 0, 5 0)),

CIRCULARSTRING(10 1,10 9,10 1)),CURVEPOLYGON(CIRCULARSTRING(10 3,10 7,10 3)))

GDAL 2.0 overview


● What about spatial operations ?

g1 = ogr.CreateGeometryFromWkt('CURVEPOLYGON(COMPOUNDCURVE(CIRCULARSTRING (0 0,1 1,2 0),(2 0,0 0)))')g2 = ogr.CreateGeometryFromWkt('CURVEPOLYGON(COMPOUNDCURVE(CIRCULARSTRING (0 0,1 -1,2 0),(2 0,0 0)))')g3 = g1.Union(g2)

g3 = ???????????????

GDAL 2.0 overview

RFC 49: Curve geometriesPOLYGON ((0 0,0.002435949740176 0.069756473744125,0.009731931258429 0.139173100960061,0.021852399266194 0.20791169081776,0.038738304061681 0.275637355817011,0.060307379214091 0.342020143325669,0.086454542357401 0.406736643075803,0.117052407141074 0.469471562785898,0.151951903843575 0.529919264233229,0.190983005625057 0.587785252292491,0.233955556881021 0.642787609686564,0.280660199661355 0.694658370459024,0.330869393641151 0.743144825477401,0.384338524674348 0.788010753606727,0.440807096529255 0.829037572555052,0.5 0.866025403784448,0.561628853210948 0.898794046299173,0.625393406584095 0.927183854566806,0.690983005625071 0.951056516295154,0.75807810440034 0.970295726276021,0.826351822333095 0.984807753012234,0.895471536732373 0.99452189536828,0.965100503297521 0.999390827019113,1.034899496702508 0.999390827019113,1.104528463267684 0.99452189536828,1.173648177666962 0.984807753012234,1.241921895599716 0.970295726276021,1.309016994374985 0.951056516295182,1.374606593415933 0.927183854566806,1.438371146789109 0.898794046299173,1.5 0.866025403784448,1.559192903470773 0.829037572555052,1.61566147532568 0.788010753606727,1.669130606358862 0.743144825477401,1.719339800338673 0.694658370459024,1.766044443119028 0.642787609686564,1.809016994374985 0.587785252292491,1.848048096156446 0.529919264233229,1.882947592858954 0.469471562785898,1.913545457642613 0.406736643075803,1.939692620785934 0.342020143325669,1.961261695938333 0.275637355817011,1.978147600733848 0.20791169081776,1.990268068741613 0.139173100960068,1.997564050259825 0.069756473744128,2 0,1.997564050259825 -0.069756473744128,1.990268068741613 -0.139173100960068,1.978147600733848 -0.20791169081776,1.961261695938333 -0.275637355817011,1.939692620785934 -0.342020143325669,1.913545457642613 -0.406736643075803,1.882947592858954 -0.469471562785898,1.848048096156446 -0.529919264233229,1.809016994374985 -0.587785252292491,1.766044443119028 -0.642787609686564,1.719339800338673 -0.694658370459024,1.669130606358862 -0.743144825477401,1.61566147532568 -0.788010753606727,1.559192903470773 -0.829037572555052,1.5 -0.866025403784448,1.438371146789109 -0.898794046299173,1.374606593415933 -0.927183854566806,1.309016994374985 -0.951056516295182,1.241921895599716 -0.970295726276021,1.173648177666962 -0.984807753012234,1.104528463267684 -0.99452189536828,1.034899496702508 -0.999390827019113,0.965100503297521 -0.999390827019113,0.895471536732373 -0.99452189536828,0.826351822333095 -0.984807753012234,0.75807810440034 -0.970295726276021,0.690983005625071 -0.951056516295154,0.625393406584095 -0.927183854566806,0.561628853210948 -0.898794046299173,0.5 -0.866025403784448,0.440807096529255 -0.829037572555052,0.384338524674348 -0.788010753606727,0.330869393641151 -0.743144825477401,0.280660199661355 -0.694658370459024,0.233955556881021 -0.642787609686564,0.190983005625057 -0.587785252292491,0.151951903843575 -0.529919264233229,0.117052407141074 -0.469471562785898,0.086454542357401 -0.406736643075803,0.060307379214091 -0.342020143325669,0.038738304061681 -0.275637355817011,0.021852399266194 -0.20791169081776,0.009731931258429 -0.139173100960061,0.002435949740176 -0.069756473744125,0 0))

GDAL 2.0 overview


● What about spatial operations ?


g3 = ???????????????Try again….

GDAL 2.0 overview


● Conversion methods between non-curve and curve geometry types, explicit or implicit (e.g. GEOS based operations)


CURVEPOLYGON (CIRCULARSTRING (0 0,1 1,2 0,1 -1,0 0))

GDAL 2.0 overview


● Driver support: GML, PostGIS, Interlis, GeoPackage, SQLite, MEM, VRT

● Methods to import/export ISO WKB/WKT for 2.5D● Application code must be ready to deal with the new types● Still no M-dimension support though or TIN /

POLYHEDRALSURFACE

More at https://trac.osgeo.org/gdal/wiki/rfc49_curve_geometries See also: https://github.com/mhugent/QGIS-Enhancement-Proposals/blob/master/QEP-8-geometry_redesign.rst

https://trac.osgeo.org/gdal/wiki/rfc49_curve_geometries

https://github.com/mhugent/QGIS-Enhancement-Proposals/blob/master/QEP-8-geometry_redesign.rst

GDAL 2.0 overview

RFC 31: 64-bit feature IDs and fields

● Fix issues with big integer values being truncated / overflowed to 32bit values

● GetFID() / SetFID() now return a 64bit integer on all platforms● GetFeature(), DeleteFeature(), SetNextByIndex() now take a

64bit integer argument.● GetFeatureCount() now returns a 64bit integer. ● OFTInteger64 and OFTInteger64List field data types● GetFieldAsInteger64(), GetFieldAsInteger64List(), SetField

(xx, 64bit integer)● Driver support: Shapefile, PostGIS, GeoJSON, CSV, SQLite,

GeoPackage, MySQL, Oracle, MEM, VRT, GML, WFS, OSM● Application code must be ready to deal with the new types

and the interface changesMore at https://trac.osgeo.org/gdal/wiki/rfc31_ogr_64

https://trac.osgeo.org/gdal/wiki/rfc31_ogr_64

GDAL 2.0 overview

RFC 50: OGR field subtypes

● A subtype is a hint that over-specifies a base field type● Boolean, 16 bit integers or 32 bit floating point values● Supported in GeoJSON, GML, CSV, PostgreSQL,

GeoPackage, SQLite, FileGDB/OpenFileGDB, VRT● OGR SQL dialect extended:

○ The field list of a SELECT can now accept boolean expressions, such as "SELECT x IS NULL, x >= 5 FROM foo"

○ The WHERE clause of a SELECT can now accept boolean fields, such as "SELECT * FROM foo WHERE a_boolean_field"

● Transparent for application code that doesn’t care about it

More at https://trac.osgeo.org/gdal/wiki/rfc50_ogr_field_subtype

https://trac.osgeo.org/gdal/wiki/rfc50_ogr_field_subtype

GDAL 2.0 overview

RFC 52: Stricter OGR SQL quoting

● Conformance with SQL 92 standard● String literals must be quoted with single(‘) quote characters● Identifiers may be quoted, with double(“) quote characters● Example :

■ OK: SELECT “a_column’, another_one FROM “a_table” WHERE “a_column” = ‘some_value’

■ Wrong: SELECT ‘a_column’ FROM a_table WHERE ‘a_column’= “some_value”

● Application code must be audited. Some requests that used to work with GDAL 1.X might fail, or return different results

More at https://trac.osgeo.org/gdal/wiki/rfc52_strict_sql_quoting

https://trac.osgeo.org/gdal/wiki/rfc52_strict_sql_quoting

GDAL 2.0 overview

RFC 53: Not-null constraints and default values

● Get/set methods for not-null constraints and default values of OGR fields

● Example: Geometry Column 1 NOT NULL = WKT

Geometry Column 2 NOT NULL = geom2id: Integer (0.0) NOT NULL DEFAULT 1234567dbl: Real (0.0) NOT NULL DEFAULT 1.456str: String (0.0) NOT NULL DEFAULT 'a'd: Date (0.0) NOT NULL DEFAULT CURRENT_DATEt: Time (0.0) NOT NULL DEFAULT CURRENT_TIMEdt: DateTime (0.0) NOT NULL DEFAULT CURRENT_TIMESTAMPdt2: DateTime (0.0) NOT NULL DEFAULT '2013/12/11 01:23:45'

● Automatic propagation with ogr2ogr● Driver support: GML, PostGIS, CartoDB, GeoPackage, SQLite,

MEM, VRT, Oracle, MySQL, WFS, FileGDB/OpenFileGDBMore at https://trac.osgeo.org/gdal/wiki/rfc53_ogr_notnull_default

https://trac.osgeo.org/gdal/wiki/rfc53_ogr_notnull_default

GDAL 2.0 overview

RFC 54: Dataset transactions

● RDBMS support BEGIN/COMMIT/ROLLBACK per connection

● GDAL 1.X: available at OGRLayer level ⇒ not the appropriate scope

● GDAL 2.0: available as GDALDataset methods● Driver support: PostGIS, GeoPackage, SQLite● Generic support for emulated transaction for drivers not

naturally transactional. Implemented in FileGDB for now

More at https://trac.osgeo.org/gdal/wiki/rfc54_dataset_transactions

https://trac.osgeo.org/gdal/wiki/rfc54_dataset_transactions

GDAL 2.0 overview

GeoPackage support

● SQLite based container, OGC standard since 2014.● Implements core spec in read/write/update + most

standardized extensions● Vector:

○ Spatial index (R-Tree)○ Curve geometries○ A-spatial tables (GDAL specific extension)

● Raster:○ PNG, JPEG, WEBP, PNG+JPEG, PNG8 tiles○ Tiling schemes: dataset specific or global tiling

schemes (GoogleMercator, GlobalGeodetic)○ Overviews/pyramids: power of 2 or arbitrary values

● Metadata (flattened)More at http://gdal.org/drv_geopackage.html and http://gdal.org/drv_geopackage_raster.html

http://gdal.org/drv_geopackage.html

http://gdal.org/drv_geopackage_raster.html

GDAL 2.0 overview

Other raster enhancements

● Warping:○ can use overviews of source dataset○ correctness fixes when subsampling with cubic, …○ speed improvements for most resampling methods

(apply to gdaladdo as well)● Better scalability of raster block cache due to reduced

lock contention. Ex. with i7-4700 HQ (8 vCPUs):○ GDAL 1.11: time testblockcache: 19 s○ GDAL 2.0: time testblockcache: 7.6 s○ GDAL 2.0: with GDAL_RB_LOCK_TYPE=SPIN: 2.7s

● Memory driver: most RasterIO() operations no longer involve the block cache at all

GDAL 2.0 overview

Other raster enhancements

● GeoTIFF driver:○ Default: time ./testblockcache -ondisk: 7.5s○ GTIFF_DIRECT_IO=YES: short circuit the block cache&libtiff for most

RasterIO() operations (restricted to uncompressed stripped GeoTIFF).time ./testblockcache -ondisk: 2s

○ GTIFF_VIRTUAL_MEM_IO=YES: same as above, with tiled GeoTIFF as well. Uses memory-mapped file access. Linux only for now, 64bit recommended (could be extended to other platforms possible).time ./testblockcache -ondisk: 0.3s

● New KEA driver :○ Developed by Landcare Research/P.Bunting/S.Gillingham○ Based on HDF5 container○ Designed to support all GDAL concepts○ Full read/write/update capabilities

● Improvements in JPEG 2000 OpenJPEG driver. Conformity with INSPIRE Orthoimagery technical guidelines

GDAL 2.0 overview

Other vector enhancements

● PostGIS:○ Bulk insertions in COPY mode by default○ Defered table creation○ Curve geometries support○ Dataset transactions○ Interleaved layer reading○ 64-bit integer identifiers and fields values

● MapInfo .TAB: update support● CSV: autodetection of field types

GDAL 2.0 overview

When can I get it ?

● Expected schedule :○ Beta1: end April 2015○ Release: end May 2015

● Dev version already deployed in production on a few customer sites. Grab&test it!○ https://svn.osgeo.org/gdal/trunk○ https://github.com/OSGeo/gdal ○ http://download.gisinternals.com/development.php

https://svn.osgeo.org/gdal/trunk

https://svn.osgeo.org/gdal/trunk

https://github.com/OSGeo/gdal

https://github.com/OSGeo/gdal

http://download.gisinternals.com/development.php

http://download.gisinternals.com/development.php

GDAL 2.0 overview

Potential project directions

● CMake build system○ Aim: out-of-tree builds, correct header dependency.○ When completed and stable, unified build system for Unix&Windows○ Ongoing effort lead by Dmitriy Baryshnikov:

https://github.com/nextgis/gdal_svn/tree/rsmd-reader-dev

● Per-dataset block cache:○ To reach zero lock contention○ https://trac.osgeo.org/gdal/wiki/rfc47_dataset_caching (Blake

Thompson)



https://trac.osgeo.org/gdal/wiki/rfc47_dataset_caching

GDAL 2.0 overview


● Integration of Geography Network Model:○ Creating network topology over spatial data manually or automatically○ Adding/removing spatial layers/features to the network○ Defining business logic of the networks (way of assigning graph costs)○ Several methods of network analysis.○ Generic support for OGR formats with write capabilities○ Potentially link with dedicated interfaces (PgRouting, Spatialite

network, GML...)○ Developed by Mikhail Gusev for GSoC 2014

https://trac.osgeo.org/gdal/wiki/rfc48_geographical_networks_support



GDAL 2.0 overview


● Planar topology:○ New abstraction based on related ISO SQL/MM Part3 modeling○ Topology primitives: nodes, edges, faces○ TopoGeometry build on primitives / hierarchical TopoGeometry○ Building of topology from geometries○ Geometry ←→ TopoGeometry conversions○ Interface with PostGIS, GRASS, Oracle, GML, Spatialite, TopoJSON○ Conversion : topo2topo

GDAL 2.0 overview


● OpenFileGDB write support● MongoDB driver● SRS handling improvements: guessing EPSG codes,

proposing appropriate datum shifts according to location, …

● Alternative geometry library: Boost::Geometry● Complex features???● More drivers, performance improvements, ...

GDAL 2.0 overview

Questions?

See Also: http://www.gdal.org/ https://trac.osgeo.org/gdal/wiki/RfcList

Contact: [email protected]

http://www.gdal.org/

https://trac.osgeo.org/gdal/wiki/RfcList

mailto:[email protected]

GDAL 2.0 overview

= GDAL/OGR 2.0 Release Notes (to r28500) =

***** Work in progress *****

== In a nutshell... ==

* New GDAL drivers:

- BPG: read-only driver for Better Portable Graphics format (experimental, no build support)

- GPKG: read/write/update capabilities in the unified raster/vector driver

- KEA: read/write driver for KEA format

- ROI_PAC: read/write driver for image formats of JPL's ROI_PAC project (#5776)

* New OGR drivers:

- Cloudant: read/write driver for Cloudant service

- JML: read/write driver for OpenJUMP .jml format

- Selaphin: read/write driver for the Selaphin/Seraphin format (#5442)

* Significantly improved drivers: CSV, GPKG, GTiff, MapInfo file, PG, SQLite

* RFC 31: OGR 64bit Integer Fields and FIDs (trac.osgeo.org/gdal/wiki/rfc31_ogr_64)

In OGR core, OGR SQL, Shapefile, PG, PGDump, GeoJSON, CSV, GPKG, SQLite, MySQL,

OCI, MEM, VRT, JML, GML, WFS, CartoDB, XLSX, ODS, MSSQLSpatial, OSM, LIBKML, MITAB

* RFC 46: GDAL/OGR unification ( http://trac.osgeo.org/gdal/wiki/rfc46_gdal_ogr_unification)

- GDAL and OGR PDF drivers are unified into a single one

- GDAL and OGR PCIDSK drivers are unified into a single one

* RFC 49: Add support for curve geometries (http://trac.osgeo.org/gdal/wiki/rfc49_curve_geometries)

In OGR core, and GML, NAS, PostgreSQL, PGDUMP, GPKG, SQLite, VFK, VRT drivers

* RFC 50: Add support for OGR field subtypes (http://trac.osgeo.org/gdal/wiki/rfc50_ogr_field_subtype)

In OGR core, OGR SQL, swig bindings, CSV, FileGDB, GeoJSON, GML, GPKG, OpenFileGDB, PG, PGDump, SQLite, VRT

* RFC 51: RasterIO() improvements : resampling and progress callback (http://trac.osgeo.org/gdal/wiki/rfc51_rasterio_resampling_progress)

* RFC 52: Stricter SQL quoting (http://trac.osgeo.org/gdal/wiki/rfc52_strict_sql_quoting)

* RFC 53: OGR not-null constraints and default values (http://trac.osgeo.org/gdal/wiki/rfc53_ogr_notnull_default)

In OGR core, OGR SQL, PG, PGDump, CartoDB, GPKG, SQLite, MySQL, OCI, VRT, GML, WFS, FileGDB, OpenFileGDB

* Upgrade to EPSG v8.5 database

* Fix locale related issues when formatting or reading floating point numbers (#5731)

== New installed files ==

== Backward compatibility issues ==

See MIGRATION_GUIDE.TXT

== GDAL/OGR 2.0 - General Changes ==

Build(Unix):

* Fix for cpl_recode_iconv.cpp compilation error on freebsd 10 (#5452)

* Fix pthread detection for Android

* Fix in Armadillo detection test (#5455)

* Fix detection of OCI by changing linking order to please modern GCC (#5550)

* Fix test to accept MariaDB 10.X as valid MySQL (#5722)

* Make sure $(GDAL_INCLUDE) is first to avoid being confused by GDAL headers of a previous version elsewhere in the include path (#5664)

* Always use stat rather than stat64 for Mac OSX in AC_UNIX_STDIO_64. (#5780, #5414).

* Add support for ECW SDK 5.1 (#5390)

Build(Windows):

* PDF: fix compilation issue with Visual Studio 2012 (#5744)

* PDF: Add support to compile the pdf driver as plugin (#5813)

* Add support for MrSID 9.1 SDK (#5814)

* when building netCDF, HDF4, HDF5 as plugins, call registration of 'sub-drivers' GMT, HDF4Image and HDF5Image (#5802)

Build(all):

* Fix compilation errors with json-c 0.12 (#5449)

* Fix compilation error in alg/gdalgrid.cpp when AVX is available, but not SSE (#5566)

Changelog: page 1/50

v0 of this presentation :-)

GDAL 2.0 overview

== GDAL 2.0 - Overview of Changes ==

Port:

* Introduce a more generic lock API (recursive mutex, adaptive mutex, spinlock)

* Add types for CPLMutex, CPLCond and CPLJoinableThread (only enforced in -DDEBUG mode)

* Add CPLGetPhysicalRAM() and CPLGetUsablePhysicalRAM()

* CPLSpawn() on Windows: quote arguments with spaces in them (#5469)

* /vsigzip/: avoid infinite loop when reading broken .gml.gz file (#5486)

* /vsizip/ : fix bug that caused premature end of file condition with some read patterns (#5530)

* /vsizip/ on >4GB zips: accept .zip declare 0 disks (#5615)

* /vsistdout/: add VSIStdoutSetRedirection() for compatibility with MapServer FCGI (https://github.com/mapserver/mapserver/pull/4858)

* /vsimem/: update st_mtime and return it with Stat()

* /vsicurl/: avoid reading after end-of-file and fix failure when reading more than 16MB in a single time (#5786)

* VSIWin32Handle::Flush(): no-op implementation is sufficient to offer same guarantee as POSIX fflush() (#5556)

* Unix VSIL: reset eof in all cases in Seek()

* Windows plugins: complementary fix to #5211 to avoid error dialog box when there are dependency problems (#5525)

* Fix VSIReadDirRecursive() recursing on the parent or current directory (#5535)

* cpl_error: obfuscate password

* HTTP: set CURLOPT_NOSIGNAL if available (#5568)

* Add COOKIE option to CPLHTTPFetch() (#5824)

* Fix stack corruption upon thread termination with CPLSetThreadLocalConfigOption on Windows 32 bit (#5590)

* cpl_csv: Stop probing for csv/horiz_cs.csv. (#5698)

* vsipreload: implement clearerr() and readdir64() (#5742)

* CPLsetlocale(): return a string that is thread-locale storage to avoid potential race in CPLLocaleC::CPLLocaleC() (#5747)

* CPLHexToBinary(): faster implementation (#5812)

Core:

* Reduce lock contention on the global cache mutex and make it possible to use spin lock instead with GDAL_RB_LOCK_TYPE=SPIN

* EXIF reader: fix memleak in error code path

* EXIF reader: add missing validation for some data types (#3078)

* Fix crash in GDALPamRasterBand::SerializeToXML() when saving an empty RAT (#5451)

* ComputeStatistics(): use Welford algorithm to avoid numerical precision issues when computing standard deviation (#5483)

* Fix crashing issue with TLS finalization on Unix (#5509)

* GDALJP2Metadata::CreateGMLJP2(): use EPSGTreatsAsLatLong() and EPSGTreatsAsNorthingEasting() to determine if axis swapping is needed (#2131)

* GMLJP2: be robust when parsing GMLJP2 content that has nul character instead of \n (#5760)

* GDALMultiDomainMetadata::XMLInit(): when importing XML metadata, erase the existing document to replace it with the new one

* Metadata: fix correct sorting of StringList / metadata (#5540, #5557)

* Make GetMaskBand() work with GDT_UInt16 alpha bands (#5692)

* Fix 32bit overflow in GDALRasterBand::IRasterIO() and GDALDataset::BlockBasedRasterIO() (#5713)

* RasterIO: small optimization in generic RasterIO() implementation to avoid loading partial tiles at right and/or bottom edges of the raster when they are going to be completely written

* Fix crash when calling GetTiledVirtualMem() on non-Linux platform (#5728)

* Add GDAL_OF_INTERNAL flag to avoid dataset to be registered in the global list of open datasets

* GDALDriver::CreateCopy(): accept _INTERNAL_DATASET=YES as creation option, so as to avoid the returned dataset to be registered in the global list of open datasets

* Implement GDALColorTable::IsSame()

* GDALPamDataset: do not serialize dataset metadata unless it has been set through GDALDataset::SetMetadata() or GDALDataset::SetMetadataItem()

* GDALLoadTabFile: add TAB_APPROX_GEOTRANSFORM=YES/NO configuration option to decide if an approximate geotransform is OK (#5809)


GDAL 2.0 overview

Algorithms:

* RPC transformer: fix near interpolation in RPC DEM (#5553)

* RPC transformer: take into account nodata in RPC DEM (#5680)

* RPC transform: add RPC_DEM_MISSING_VALUE transformer option to avoid failure when there's no DEM at the transformed point (#5730)

* TPS transformer: fix crash if the forward or backward transform cannot be computed (#5588)

* OpenCL warper: remove unused variable in bilinear resampling that can cause compilation error (#5518)

* OpenCL wraper: fix code compilation with NVIDIA OpenCL (#5772)

* Overview: Fix and speed-up cubic resampling in overview computation to take into account scaling factor (#5685)

* Overview: ignore alpha=0 values when compute an average overview of an alpha band; and also avoid memory errors when calling GetMaskBand()/GetMaskFlags() after overview computation if GetMaskXXX() has been called before (#5640)

* Overview: avoid crash when computing overview with a X dimension much smaller than Y dimension (#5794)

* GDALRegenerateOverviewsMultiBand(): fix stride calculation error with certain raster dimensions (#5653)

* Warper: various speed optimizations

* Warper: fix Cubic and Bilinear resampling to work correctly with downsizing (#3740)

* Warper: fix and optimize CubicSpline

* Warper: regardless of the warping memory limit, add heuristics to determine if we must split the target window in case the 'fill ratio' of the source dataset is too low (#3120)

* Warper: accept warping options METHOD=NO_GEOTRANSFORM and DST_METHOD=NO_GEOTRANSFORM to run gdalwarp on ungeoreferenced images

* Warper: fix GDALSuggestedWarpOutput() wrong extent in some circumstances (e.g. dataset of big dimension with world coordinates) (#5693)

* Warper: fix integer overflow when reprojecting into an area with (part of) bounds completely outside of the source projection (#5789)

* GDALReprojectImage(): takes into account nodata values set on destination dataset

* Median cut and dithering: optimizations and enhancements to deal with 8-bit precision (only if using internal interface for now)

* rasterfill: add option to specify driver to use for temporary files

Utilities:

* gdalinfo: display extra metadata domains attached to band, and refactor code a bit (#5542)

* gdalinfo: add -oo option per RFC 46

* gdaladdo: add -oo option per RFC 46

* gdaladdo: add warning when subsampling factor 1 specified

* gdal_translate: add -oo option per RFC 46

* gdal_translate: add -r and -tr options per RFC 51

* gdal_translate: avoid preserving statistics when changing data type in situations where clamping can occur

* gdal_translate: don't recopy band units if rescaling or unscaling is involved (#3085)

* gdal_translate: increase GDAL_MAX_DATASET_POOL_SIZE default value to 450. (#5828)

* gdalwarp: add -oo option per RFC 46

* gdalwarp: initialize destination dataset to no_data value when automatically propagating source nodata (#5675)

* gdalwarp: add a -ovr option to select which overview level to use, and default to AUTO. Also add a generic OVERVIEW_LEVEL=level open option, and make it available in standard VRT (#5688)

* gdalwarp: only apply INIT_DEST when processing the first input dataset (#5387)

* gdalwarp: increase GDAL_MAX_DATASET_POOL_SIZE default value to 450. (#5828)

* gdaldem: avoid too large files to be produced when using -co COMPRESS=xxxx -co TILED=YES (#5678)

* gdal_grid: use nodata= parameter in the algorithm string to determine the nodata value to set on the band (#5605)

* gdal_edit.py: add -unsetstats option (and fix -a_nodata to run on all bands, ant not just first one)

* gdal_edit.py: add -stats and -approx_stats flags (patch by mwtoews, #5805)

* gdal_edit.py: change -mo add metdata to existing one; add new option -unsetmd to clean existing metadata

* gdal_retile.py: fix to make it work with input images of different resolutions (#5749)

* gdal_retile.py: implement progress bar (#5750)

* gdal_merge.py: add timing information in verbose output

* gdal2tiles.py: fix inverted long/lat in BoundingBox and Origin elements of tilemapresource.xml (#5336)

* pct2rgb.py: make it work with color tables with less than 256 entries (#5555)

* gdal_fillnodata.py: FillNodata: copy no data value to destination band when creating a dataset (if available) (#4625)

* gdalcompare.py: add options to supress selected comparisons

* gdalcompare.py: takes into account differences in overview bands

* gdalcompare.py: compute difference on float to avoid integer underflow

* epsg_tr.py: change to make it possible to export GEOCCS and COMPD_CS to proj.4 epsg and PostGIS spatial_ref_sys.sql files


GDAL 2.0 overview

…

GDAL 2.0 overview

The end !

SXF driver:

* Fix SXF file version check (#5456)

* Fix wrong Miller Cylindrical projection string

* Fix encoding issues (#5647)

* Fix extract z value to OGRGeometry

VFK driver:

* recode also header values

* process DKATUZE from header properly

* handle also duplicated records

* check attribute 'parametry_spojeni'

* speed-up GetFeatureCount()

* fix reading properties. Escape characters for SQL

* change SRS from EPSG 2065 to 5514

* fix reading SBP datablock (fix mismatch when reading from file and db)

VRT driver:

* do not propagate ignoring of x and y cols of a PointFromColumns to the source layer (#5777)

Tiger driver:

* Fix potential buffer underflow when providing /vsistdin/ to Tiger driver (#5567)

WAsP driver:

* added options and changed output precision to match WAsP Map Editor

* improve TestCapability(ODsCCreateLayer)

WFS driver:

* Add COOKIE option (#5824)

* when parsing a layer schema without geometry from the GML .xsd, do not expose a geometry field at the WFS layer level (#5834)

XLSX driver:

* fix column numbering when there are more than 26 columns (#5774)

== SWIG Language Bindings ==

All bindings:

* bind GDALGetBandDataset() as Band.GetDataset()

* add Feature.GetFieldAsBinary()

CSharp bindings:

Java bindings:

* Pass eRWFlag to allow both reading or writing. Write was broken in DatasetRasterIO().

Perl bindings:

* NoDataValue: set max float if undef is given.

* Unit: set '' if undef is given.

* Also other changes that will also remove some "use of uninitialized value in subroutine entry" warnings.

* Automatic handling of SQL result layers.

* Switch to using doxygen-filter-perl from CPAN for the documentation.

* Fix issue with index attribute for field meta data (schema) (#5662)

Python bindings:

* add optional buf_xsize, buf_ysize and buf_type parameters to Dataset.ReadAsArray() and Dataset.LoadFile(), and use dataset RasterIO for better efficiency

* avoid generating Python exception when PyString_FromStringAndSize() fails and GDAL errors as Python exceptions are disabled

* Band.ReadRaster() and Dataset.ReadRaster(): clear the buffer in case there are holes in it due to odd spacings specified by the user

* Fix hang of Python in case of repeated call to gdal/ogr.UseExceptions() and CE_Warning emitted (#5704)

* for Python 2, accept unicode string as argument of Feature.SetField(idx_or_name, value) (#4608)

* for Python 2, accept Unicode strings to be passed as key and/or value of the dictionary passed to SetMetadata() (#5833)

* fix processing error of ogr_python.i with SWIG 3 (#5795)

* NUMPY driver: avoid returning CE_None in GetGeoTransform() when there's no geotransform set (#5801)

* Make GetFieldAsBinary() work with OFTString fields

* For Python3 compat, make Feature.GetField() use GetFieldAsBinary() if GetFieldAsString() fails (#5811)


GDAL 2.0 overview

RFC 45: Raster access as virtual memory mapping

● Purpose: to be able to work with a GDAL dataset transparently has a simple&stupid 2D array without doing API calls

● Using virtual memory tricks (only implemented on Linux for now) and page fault handlers to populate the memory mapping on demand

● Works with any GDAL driver. A few ones have optimized implementations that use OS memory file mapping mechanisms.

● You can get a NumPy array of several terabytes with a few hundred megabytes of RAM and access arbitrary indices: array[y][x]

More at http://trac.osgeo.org/gdal/wiki/rfc45_virtualmem

http://trac.osgeo.org/gdal/wiki/rfc45_virtualmem

Download - GDAL 2.0 overview - FOSS4G NA 2015 2.0... · GDAL 2.0 overview About me Contributor to the GDAL/OGR project since 2007 and Project Steering Commitee member since 2008. Chair of the

Top Related