storage and management of large scale point …pointclouds.nl/docs/workshop_dubai_cut.pdforacle...

Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |

Storage and Management of Large Scale Point Cloud DataOracle Workshop9th International 3DGeoInfo 2014

Hans Viehmann1, Mike Horhammer1 ,Peter van Oosterom2 and Oscar Martinez Rubi31 Oracle 2 TU Delft 3 Netherlands eScience CenterNovember 11, 2014

Image by Fugro


Program Agenda

1

2

3

4

5

Introduction

File formats and tested datasets

Point cloud data management systems

Loading

Querying

Research and others

Conclusions

6

7


Introduction2-year Netherlands eScience research project on Massive Point Clouds

• TU Delft:

1. GIS technology

2. TU Delft Library

3. 3TU.Datacentrum

4. TU Delft Shared Service Center ICT

5. Cooperation with CWI/MonetDB

• NLeSC (Netherlands eScience Center)

• Oracle Corporation (NEDC)

• Rijkswaterstaat

• Fugro

Imag

e co

urt

esy

of:

PD

OK

, NL


Introduction

• Collection of point cloud data grows rapidly

–Many new applications now economically viable

• Relevant use cases in GIS:

– Digital elevation model of terrains

– 3D models of urban environment

– Flood risk management

– Dike monitoring (NL)

– Forest mapping

• Requiring new techniques for

–Managing massive datasets

– Integrating various kinds of data

• Other potential applications:– 3D models of manufactured parts

– Visualization, animation, rendering

–Medical imaging

– Industrial metrology


IntroductionEvaluation of user requirements

• Based on structured interviews with:

– Government community: RWS (Ministry)

– Commercial community: Fugro (company)

– Scientific community: TU Delft Library

• Report at MPC public website http://pointclouds.nl

• Basis for conceptual benchmark:

– Tests for functionality

– Classified by importance

• Testing single-user environments to avoid load-generator complexities

http://pointclouds.nl/


IntroductionPoint cloud data management/processing systems

7


Why use a database here?Requires a spatially enabled database

• Data integration with other sources

–Online availability

– Geo-referenced imagery, existing 3D structures, attributes,...

• Fast access to arbitrary part of data set– For processing or visualization

• General benefits of mature DBMS

– Information lifecycle management – data administration, tuning

– Scaleability – multi-user environment, multi-processor support, clustering, ...

– Executing data-intense logic where the data resides

8


Possible 3D processing flows in Oracle 12c

LiDAR Files

LiDAR loader

SDO_PC tables

or flat table

Query and Clip

Convert to

Geometries

Flat files

Load point

cloud

Point Tables

TIN tablesGenerate TIN

Query and Clip

Convert to

Geometries

Georaster

Generate DEM

Query and Clip

Calculate

Contour Lines


ÖBB-Infrastruktur AG, R&DOracle Spatial and Graph: Railway Network Management

Objectives

Optimize railway planning, construction and maintenance

Enable efficient routing of railway lines

Solution

Stores and processes more than 8 billion points of objects along railway tracks

Enables LiDAR data to be viewed with existing infrastructure vector data

Provides comprehensive metadata about railway tracks through CSW

Delivers data through open WebServices

“[This technology…] is indispensable to process geospatial data with high efficiency at low cost,”Dr. Michaela Haberler-WeberÖBB-Infrastruktur AG, R & D

Image courtesy of: IQsoft, Austria


Program Agenda

1

2

3

4

5

Introduction



Loading

Querying

Research and others

Conclusions

6

7


File formats and tested datasetsLAS

• De facto standard

• Current version 1.4

– http://www.asprs.org/a/society/committees/standards/LAS_1_4_r13.pdf

– Header 375 bytes

– 12+ attributes

– 11 Record (point) formats

• Binary (20+ bytes per point)

• Limited to 4.2B pts (< 1.4)

• LIDAR data schema evolution

– LAS 1.0 (2003)

• Format 0: X, Y, Z, Intensity, Return #, …, classification, …

• Format 1: …, GPS time

– LAS 1.1 (2005)

• Formats 0, 1: similar

– LAS 1.2 (2008)

• Format 0, 1

• Format 2, 3: …, R, G, B

– LAS 1.3 (2009)• Format 0 – 3

• Format 4, 5: (1, 3) …, waveform packet info

– LAS 1.4 (2011)• Formats 6 – 10: …, near infrared channel, 64-Bit

Value Meaning

0 Created, Never Classified

1 Unclassified 1

2 Ground

3 Low Vegetation

4 Medium Vegetation

5 High Vegetation

6 Building

7 Low Point (noise)

8 Model Key Point (mass point)

9 Water

10 Reserved for ASPRS Def

11 Reserved for ASPRS Def

12 Overlap Points 2

13-31 Reserved for ASPRS Def

ASPRS Standard LIDAR Point Classes (Point Data Record Format 0-5)

http://www.asprs.org/a/society/committees/standards/LAS_1_4_r13.pdf


File formats and tested datasetsLAS: compression and in-file indexing

• Highly compressible

– LAZ (LAStools)

– zLAS (ESRI)

– Compression rate 10-15

• Indexable

– LAX (LAStools)

– LASX (ESRI)


File formats and tested datasetsActual Height Model of the Netherlands (AHN2)

• Covering surface of the entire country

• 6 -10 pts/m2 640 billion pts

• 60,185 LAZ files, 987 GB/11.64 TB

• (X, Y, Z) only

• “Future”

– AHN3 higher resolution

– Cyclorama-based photogrammetric datasets (50x AHN2, and with RGB)


File formats and tested datasetsTested datasets

Name Points LAS files Disk size [GB] Area [km2] Description

20M 20,165,862 1 0.4 1.25 TU Delft campus

210M 210,631,597 16 4.0 11.25 Major part of Delft city

2201M 2,201,135,689 153 42.0 125 City of Delft and surroundings

23090M 23,090,482,455 1,492 440.4 2,000 Major part of Zuid-Holland province

639478M 639,478,217,460 60,185 11,644.4 40,000 The Netherlands


File formats and tested datasetsBenchmarks

• “Mini-benchmark”

– 1 dataset: 20M

– X, Y, Z

–Management systems:

• PointCloud (blocks) solutions in Oracle and PostgreSQL

• Flat tables in Oracle and PostgreSQL

• LAStools (not only X,Y,Z)

– 7 queries



• Medium benchmark

– Four datasets: 20M, 210M, 2201M, 23090M

– X, Y, Z

–Management system:

• Same as mini-benchmark

• Use compression

• Add MonetDB column-store (flat table)

• Add Exadata (diff. Hardware)

– 20 queries


File formats and tested datasetsBenchmarks queries



• Full benchmark

– AHN2 (640bn points)

– X, Y, Z

–Management systems:

• Oracle EXADATA

• Oracle blocks

• PostgreSQL blocks

• LAStools

• MonetDB

– 30 queries


File formats and tested datasetsHardware & Software

• Server HP Proliant DL380p Generation8 (2U rack)

– 2 x 8-core Intel Xeon processors E5-2690, 2.9 GHz

– 128 GB RAM (DDR3)

– Disk storage:• 400 GB SATA SD device

• 300 GB SAS 10K rpm in RAID-1 configuration (internal, 2 disks)

• 6 TB SAS 15K rpm in RAID-5 configuration (internal, 22 disks)

• 2 x 41 TB SATA 7200 rpm in RAID-5 configuration (external in 4U rack 'Yotta-III' box, 24 disks)

– Network: 4 x 1 Gbit Ethernet

– Software:• Operating system: Red Hat Enterprise Linux, version 6.5

• DBMS: Oracle, PostgreSQL/PostGIS, MonetDB

• 'Geo' tools and libraries: CGAL, GDAL, GEOS, LAStools, PROJ.4, GeoTIFF, NetCDF, etc., etc.


File formats and tested datasetsHardware & Software

• Oracle Exadata X4-2: Oracle SUN hardware for Oracle database software

“DBMS counterpart of GPU for graphics”


Characteristics of Oracle Exadata Database Machine

Standard Database Servers

– 8x 2-socket servers 192 cores, 2TB DRAMor

– 2x 8-socket servers 160 cores, 4TB DRAM

Unified Ultra-Fast Network– 40 Gb InfiniBand internal connectivity

– 10 Gb or 1 Gb Ethernet data center connectivity

Scale-out Intelligent Storage Servers

– 14x 2-socket servers 168 cores in storage

– 168 SAS disk drives 672 TB HC or 200 TB HP

– 56 Flash PCI cards 44 TB Flash + compressionFully Redundant


Oracle Exadata – Key innovations

Database offload in storage –Data intensive query operations offloaded to

storage CPUs

–100 GB/sec SQL data throughput

–Storage Index data skipping

Database optimized compression–Hybrid Columnar for 10x DB size reduction

and faster analytics

Database optimized PCI Flash–Smart caching of database data

–2.66 Million Database IOs/sec

–Smart Flash log speeds transactions

Database optimized QoS–End-to-end prioritization from

application to DB and storage

Database optimized availability–Fastest recovery of failed database,

server, storage or switch

–Fastest backup. Incremental offload

–Exachk top-to-bottom validation of hardware, software, settings

Database optimized messaging–SQL optimized InfiniBand protocol for

high throughput low latency SQL


Oracle Exadata for Spatial Workloads

• Oracle Spatial and Graph is particularly suitable for the Exadata architecture

– CPU intensive geospatial calculations

– excellent use of parallelism

– load distribution between server nodes and storage heads

– index operations benefiting from parallelism and large memory

• Breaks new boundaries for ingesting spatial data

– using the fast I/O and high network bandwidth

• Specific optimizations for point cloud data

– Excellent use of hybrid columnar compression (HCC)


Scalable from Eighth-Rack to Multi-Rack

Field Upgradeable - Supports Multiple Generations of Hardware

HalfFull

Multi-Rack

EighthQuarter


Additional work packageStandardization

• ISO/OGC spatial data:

– at abstract/generic level, 2 types of spatial representations: features and coverages

– at next level (ADT level), 2 types: vector and raster, but perhaps points clouds should be added

– at implementation/ encoding level, many different formats (for all three data types)

• nD point cloud:

– points in nD space and not per se limited to x,y,z(n ordinates of point which may also have m attributes)

– make fit in future ISO 19107 (as ISO 19107 is under revision).

– note: nD point clouds are very generic; e.g. also cover moving object point data: x,y,z,t (id) series.


Program Agenda

1

2

3

4

5

Introduction



Loading

Querying

Research and others

Conclusions

6

7


Point cloud data management systems (PC DMS)

• Oracle – Blocks

– Flat table

• PostgreSQL– Blocks

– Flat table

• LASTools

• MonetDB

Large scale point cloud data


PC DMS – Oracle Blocks


Point cloud data management systemsOracle blocks

• Loading:

– Global loading:

– Incremental loading:

• Blocking methods:

– R-Tree

– Hilbert R-Tree

• Querying:

– CLIP_PC function

– SDO_PC_NN (12.1.0.1)

– CLIP_PC_GET_MIN_HEIGHT (12.1.0.1)

– CREATE_CONTOUR_GEOMETRIES (12.1.0.1)


Point cloud data management systemsOracle blocks – Schema

Table Column:

SDO_POINT_CLOUD

POINTS:

BLOB

BLK_EXTENT:

SDO_GEOMETRY

BLK_IDOBJ_ID

TABLE of SDO_PC_BLK type

• PC BLOB:

– n points

• (x, y, z, …): 8-byte IEEE doubles (<= 8)

• Blk_id: 4-byte big-endian integer

• Pt_id: 4-byte big-endian integer

• Storage:

– Up to 30 Mpts/GB

– Compressed LOB: 100 Mpts/GB


Point cloud data management systemsOracle blocks


Point cloud data management systemsOracle blocks – Querying

Query


Point cloud data management systemsOracle blocks – Overlapping Blocks, Example Hilbert R-tree

Query


PC DMS – PostgreSQL Blocks


Point cloud data management systemsPostgreSQL blocks

• Group points in blocks

– Defined in 2D (x,y) space

– Block type: pcpatch

• PointCloud extension by P. Ramsey

https://github.com/pramsey/pointcloud

• Requires PostGIS

• Storage

– Compression: 100 Mpts/GB

– Compression+: 200 Mpts/GB

• Loading:

– Incremental loading

• Require good input data spatial organization

• Ordered loading (avoid de-clustering)

– External tool PDAL (pipeline)

http://www.pdal.io

– No parallel (doable with scripting, e.g. Python)

• GIST (R-Tree) index for the blocks

• Querying:

– PC_Intersects and PC_Intersection

– No native parallel

https://github.com/pramsey/pointcloud

http://www.pdal.io/


Point cloud data management systemsPostgreSQL blocks: PDAL chipper blocks

Images from http://www.pdal.io/stages/filters.chipper.html


PC DMS – LAStools


Point cloud data management systemsLAStools

• File-based solution by M. Isenburg

http://rapidlasso.com/lastools/

• Partly commercial:

– Some tools free open source

– Some tools require license (Windows)

– No license: noise, limit parallel

• Directly on LAS/LAZ

• Storage:

– LAS: 53 Mpts/GB

– LAZ: 540 Mpts/GB

• No data loading, but data preparation

– lastile, lassort, lasindex

• Require good input data spatial organization

• Multi-core support (not really in queries)

• Wine in Linux for Windows-only tools

• Hybrid solutions (DB) for massive PCs

• Querying:

– lasmerge for rectangles and circles

– lasclip for polygons

– Polygon holes ignored

http://rapidlasso.com/lastools/


PC DMS – Oracle Flat


Point cloud data management systemsOracle flat

• Pro

– No blocking required fast creation

– Dynamic updates fast updates

– Flexible schema

• Con

– Queries less scalable on non-Exadata(< 1 billion pts)

• Storage: 23 Mpts/GB (200 Mpts/GB on Exadata)

• Indexing

– None on Exadata

– B-tree on (x, y)

– IOT works well

• Querying:

– Plain old SQL

• Range query for rectangles

• Distance method for circles

– SDO_PointInPolygon for polygons


PC DMS – PostgreSQL Flat


Point cloud data management systemsPostgreSQL flat

• Points in flat table

– Row per point

– Data type: Double precission

• Storage: 13 Mpts/GB

• Loading:

– Las2txt + “COPY FROM”

– Faster NLeSC binary loader (convert LAS to PostgreSQL binary dump format)

– No parallel (doable with scripting, e.g. Python)

• B-Tree index on X,Y

– GIST too expensive

– No parallel

• Querying:

– No native parallel

– Range query for rectangles

– Distance method for circles

– _ST_Contains,ST_MakePoint for polygons

• Bad scaling: Only use for small datasets (< 0.5 . 109)


PC DMS – MonetDB


Point cloud data management systemsMonetDB

• Column store

• Points in flat table

– Row per point

– Data type: Double precission

• Storage: 40 Mpts/GB

• Loading:

– Las2txt + “COPY FROM”

– Faster NLeSC binary loader (convert LAS to MonetDB columnar format)

– Parallel

• Imprints indexing

• Querying:

– Parallel

• Active development

• Not released yet

• Bad scaling: Only use for small datasets (< 1 . 109)


Point cloud data management systemsStorage models

Storage model Pro Con

DB blocks • Storage (compression)• Scaling• Indexing• DB functionalities• Complex queries

• Loading (make blocks)• Block overhead in queries (noticeable in simple queries)

DB flat • Faster loading/updating• DB functionalities• Dynamic schema• Simple queries

• Storage (except Exadata)• Not scaling (except Exadata)• Indexing (except Exadata)

File-based • Storage (LAZ)• Data preparation• Simple queries (if not LAZ)

• Limited functionalities• Fixed schema (LAS)• Scaling requires DB help


Program Agenda

1

2

3

4

5

Introduction



Loading

Querying

Research and others

Conclusions

6

7


Loading – Oracle Blocks


LoadingOracle Blocks – Loading Process

LAZ

LAS

CSV

Flat table

SDO_PC

LASzip CREATE_PC (1)

SqlLoader & preprocessor (4)



LAZ

LAS

CSV

Flat table

SDO_PC

LASzip CREATE_PC

SqlLoader & preprocessor


LoadingOracle Blocks (Hilbert blocking) – Create tables

CREATE TABLE AHN_BLCK

TABLESPACE USERS

PCTFREE 0

NOLOGGING

LOB(POINTS)

STORE AS SECUREFILE (TABLESPACE USERS COMPRESS HIGH) AS

SELECT * FROM MDSYS.SDO_PC_BLK_TABLE WHERE 0 = 1;

CREATE TABLE AHN_BASE (

id number,

PC SDO_PC)

TABLESPACE USERS

PCTFREE 0

NOLOGGING;


LoadingOracle Blocks (Hilbert blocking) – Create PC

declare

pc sdo_pc;

begin

pc :=

sdo_pc_pkg.init(

basetable => 'PCS',

basecol => 'PC',

blktable => 'BLOCKS',

ptn_params => 'blk_capacity=10000',

pc_extent => mdsys.sdo_geometry(2003, null, null,

mdsys.sdo_elem_info_array(1, 1003, 3),

mdsys.sdo_ordinate_array(289020.90, 4320942.61,

290106.02, 4323641.57)),

pc_tol => 0.05,

pc_tot_dimensions => 3,

pc_domain => null,

pc_val_attr_tables => null,

pc_other_attrs => xmltype(

'<opc:sdoPcObjectMetadata

xmlns:opc="http://xmlns.oracle.com/spatial/vis3d/2011/sdovis3d.xsd"

xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"

blockingMethod="Hilbert R-tree">

</opc:sdoPcObjectMetadata>'));

sdo_pc_pkg.create_pc(

inp => pc,

inptable => 'INPTAB',

clstpcdatatbl => null);

insert into pcs values (1, pc);

end;

/


LoadingOracle Blocks (Hilbert blocking) – Metadata, Index

• Update the SRID of the blocks, drop the staging table and add the blocks index (SQL).UPDATE AHN_BLCK B SET B.BLK_EXTENT.SDO_SRID = 28992;

DROP TABLE AHN_STAGING;

INSERT INTO USER_SDO_GEOM_METADATA VALUES (

’AHN_BLCK’,

’BLK_EXTENT’,

SDO_DIM_ARRAY(

SDO_DIM_ELEMENT(’X’, [min. X], [max. X],0.0001),

SDO_DIM_ELEMENT(’Y’, [min. Y], [max. Y],0.0001)),

28992);

CREATE INDEX AHN_BLCK_SIDX

ON AHN_BLCK (BLK_EXTENT)

INDEXTYPE IS MDSYS.SPATIAL_INDEX

PARAMETERS (’TABLESPACE=INDX WORK_TABLESPACE=PCWORKLAYER_GTYPE=POLYGON SDO_INDX_DIMS=2

SDO_RTR_PCTFREE=0’)

PARALLEL 16;



LAZ

LAS

CSV

Flat table

SDO_PC

LASzip CREATE_PC



LoadingOracle Blocks (Hilbert blocking) – Incremental Loading

$JDKHOME/bin/java -Xms2048m -classpath /…/ojdbc6.jar:/…/sdoutl.jar oracle.spatial.util.Las2SqlLdrIndep 1 BLOCKS BLOCK_ID_SEQ

/…/ahn_bench023090_01.las 10000 jdbc:oracle:thin:@//host:port/sid largepc largepc 100000000 (10K block capacity, 100M pts in memory)

Connecting to jdbc:oracle:thin:@//host:port/sid...

Reading 100,000,000 points... 23.324 s. Sorting: 65.362 s. Writing: 164.608 s




















Reading 139,854 points... 0.110 s. Sorting: 0.097 s. Writing: 0.375 s

------------------------------------------------------------------------------------------

Read 2,000,000,000 points... 1,078.165 s. Sorting: 2,854.376 s. Writing: 4,752.362 s (~⅔ sequential I/O – networked drive)

Time elapsed......................... 8,708.131 s (~ 2½h)


Loading – PostgreSQL Blocks


LoadingPostgreSQL blocks

• Load extensions and create blocks table (SQL):CREATE EXTENSION postgis;

CREATE EXTENSION pointcloud;

CREATE EXTENSION pointcloud_postgis;

CREATE TABLE patches (id SERIAL PRIMARY KEY, pa PCPATCH);

• Requires XML with data format inserted in pointcloud_formats

– Attributes

– Scales and offsets

– Compression: “none” or dimensional

– Multiple formats possible

INSERT INTO pointcloud_formats (pcid, srid, schema) VALUES (1,

SRID,’

<?xml version="1.0" encoding="UTF-8"?>

<pc:PointCloudSchema xmlns:pc=

"http://pointcloud.org/schemas/PC/1.1"

xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">

<pc:dimension>

<pc:position>1</pc:position>

<pc:size>4</pc:size>

<pc:description>X coordinate </pc:description>

<pc:name>X</pc:name>

<pc:interpretation>int32_t </pc:interpretation>

<pc:scale> 0.01 </pc:scale>

<pc:offset> 0 </pc:offset>

</pc:dimension>

<pc:dimension>



<pc:description>Y coordinate</pc:description>

<pc:name>Y</pc:name>

<pc:interpretation>int32_t</pc:interpretation>



</pc:dimension>

<pc:dimension>



<pc:description>Z coordinate</pc:description>

<pc:name>Z</pc:name>

<pc:interpretation>int32_t</pc:interpretation>



</pc:dimension>

<pc:metadata>

<Metadata name="compression">dimensional</Metadata>

</pc:metadata>

</pc:PointCloudSchema>’);


LoadingPostgreSQL blocks

• PDAL (Command-line) with configuration XML:

– pcid as in pointclouds_format

– Block size: Chipper filter capacity (smaller than Oracle)

– Selector filter to select attributes

– Multiple PDAL processes possible

pdal pipeline [xml File]

• Create a PostGIS GIST index (SQL):CREATE INDEX pa_gix ON blocks USING GIST(geometry(pa));

• VACUUM and ANALYZE (SQL)

– optional

VACUUM FULL ANALYZE blocks;

<?xml version="1.0" encoding="utf-8"?>

<Pipeline version="1.0">

<Writer type="drivers.pgpointcloud.writer">

<Option name="connection"> host=’dbhost’ dbname=’dbname’ password=’dbpass’

user=’dbuser’ </Option>

<Option name="table">blocks</Option>

<Option name="srid">SRID</Option>

<Option name="overwrite">false</Option>

<Option name="pcid">1</Option>

<Filter type="filters.chipper">

<Option name="capacity">BlockSize</Option>

<Filter type="filters.cache">

<Option name="max_cache_blocks">1</Option>

<Filter type="filters.selector">

<Option name="keep">

<Options>

<Option name="dimension">X</Option>

<Option name="dimension">Y</Option>

<Option name="dimension">Z</Option>

</Options>

</Option>

<Option name="overwrite_existing_dimensions">false</Option>

<Reader type="drivers.las.reader">

<Option name="filename"> [input LAS file]</Option>

<Option name="spatialreference"> EPSG:SRID</Option>

</Reader>

</Filter>

</Filter>

</Filter>

</Writer>

</Pipeline>


Loading – LAStools


LoadingLAStools

• Optional tiling (if bad spatial organization)lastile –i *.las –o tile.las

• Resort and index (Command-line):lassort.exe *las

lasindex *las

• Multi-core:

– option “cores”

lassort.exe *las –cores 8

lasindex *las –cores 8

– Scripting

lassort.exe -i [input LASfile] -o [output LASfile]

lasindex -i [output LASfile]


LoadingLAStools (Hybrid: PostgreSQL-PostGIS)

• Create DB and table for LAS files extents (SQL).CREATE EXTENSION postgis;

CREATE TABLE lasindex (filepath text, geom public.geometry(Geometry,28992));

• Extract header information (lasinfo) and insert in DB:

– Multiple processes possiblelasinfo [LAS file] -nc -nv -nco

INSERT INTO lasindex (filepath,geom) VALUES (’[LAS file]’, ST_MakeEnvelope(minx,miny,maxx,maxy, SRID));

• Create the GIST index on the extents (SQL).CREATE INDEX ON lasindex USING GIST (geom);

• VACUUM and ANALYZE (SQL)

– optional

VACUUM FULL ANALYZE lasindex;


Loading – Oracle Flat (Exadata)


LoadingOracle flat (Exadata) – Loading Process

LAZ

LAS

CSV

Flat table

SDO_PC

LASzip CREATE_PC



LoadingOracle flat (Exadata) – SqlLoader from CSV / ASCIIconnect sys as sysdba

create or replace directory las_dir as

'/scratch/…';

CREATE TABLE lidar_ext (

val_d1 NUMBER,

val_d2 NUMBER,

val_d3 NUMBER,

d NUMBER)

ORGANIZATION EXTERNAL (

TYPE ORACLE_LOADER

DEFAULT DIRECTORY las_dir

ACCESS PARAMETERS (

RECORDS DELIMITED BY NEWLINE

FIELDS (

val_d1 CHAR(8),

val_d2 CHAR(8),

val_d3 CHAR(8),

d CHAR(8)))

LOCATION (

'lidar1.txt',

'lidar2.txt',

'lidar3.txt'))

PARALLEL;

create table lidar as select * from

lidar_ext;

more lidar*txt

::::::::::::::

lidar1.txt

::::::::::::::

1 1 1 123

2 2 2 124

3 3 3 125

::::::::::::::

lidar2.txt

::::::::::::::

4 4 4 123

5 5 5 124

6 6 6 125

::::::::::::::

lidar3.txt

::::::::::::::

7 7 7 123

8 8 8 124

9 9 9 125


LoadingOracle flat (Exadata) – Loading Process

LAZ

LAS

CSV

Flat table

SDO_PC

LASzip CREATE_PC



LoadingOracle flat (Exadata) – SqlLoader from LAScreate directory las_dir as ‘…';

create directory exe_dir as ‘…';

create directory log_dir as ‘…';

grant read on directory las_dir to mdsys;

grant read, execute on directory exe_dir to

mdsys;

grant write on directory log_dir to mdsys;

create table inptab_ext(

val_d1 number,

val_d2 number,

val_d3 number,

d number)

organization external (

type oracle_loader

default directory las_dir

access parameters (

records delimited by newline

preprocessor exe_dir:'preprocessor.sh'

BADFILE log_dir:'preprocess.bad'

LOGFILE log_dir:'preprocess.log'

fields terminated by ",")

location (

'Serpent_Mound_Model_LAS_Data.las',

'Serpent_Mound_Model_LAS_Data2.las',

'Serpent_Mound_Model_LAS_Data3.las',

'Serpent_Mound_Model_LAS_Data4.las'))

parallel 16

reject limit unlimited;

preprocessor.sh

#!/bin/bash

$JDKHOME/bin/java -classpath /scratch/mhorhamm/sdoutl.jar

oracle.spatial.util.Las2SqlLdr $1


LoadingOracle flat (Exadata) – Materialize table & Create Index (no for Exadata)CREATE TABLE AHN_FLAT

TABLESPACE USERS

PCTFREE 0

NOLOGGING

PARALLEL 16 AS

SELECT

VAL_D1,

VAL_D2,

VAL_D3

FROM

INPTAB_EXT;

CREATE INDEX AHN_FLAT_IDX

ON AHN_FLAT (

VAL_D1,

VAL_D2)

TABLESPACE INDX

PCTFREE 0

NOLOGGING

PARALLEL 16;


LoadingTuning

• Tune DB server parameters (working memory, …)

• Use different file-system for DB and input data (faster IO)

• If available use SSD disks for indexes (different table space)

• > 40bn points consider partitions

• Block sizes Oracle > PostgreSQL (also depend on size typical query regions)

• Input data not spatial overlap Incremental loading

• Compression: Better storage, slower queries and loading

• Not native parallel easy do-it-your-own (be careful with de-clustering)


LoadingMedium benchmark comparison

PostgreSQL flat: Binary loader, not native parallel 16 by script, index not parallel

Oracle flat: SQL loader from LAS (ext. table, pre-processor script) , parallel 16

MonetDB flat: Binary loader, parallel

PostgreSQL blocks: Incremental loading with PDAL, no native parallel 12 by script, blocksize 3000

Oracle blocks: Global loading, SQL loader from LAS, Hilbert blocking, parallel 16, blocksize 10000

LAStools: (unlicensed), LAS files, no native parallel 16 by script , hybrid DB solution (PostgreSQL)

Oracle EXADATA X4-2 half rack: 96 cores, 5 compression levels, SQL loader from LAS, flat table, no index


LoadingMedium benchmark comparison: Loading (inc. indexing) and storage of 23090M points

Loading [s] Loading [Mpts/s]

Storage [GB]

Storage [Mpts/GB]

Oracle blocks global 70000,00 0,33 700,00 32,99

Oracle blocks global compress 192612,07 0,12 220,10 104,91

Oracle blocks increm. 14000,00 1,65 700,00 32,99

Oracle blocks increm. compress 45000,00 0,51 220,10 104,91PostgreSQL blocks increm. * 7000,00 3,30 200,00 115,45

PostgreSQL blocks increm. compress* 9079,57 2,54 106,80 216,20

Oracle flat 15825,53 1,46 997,10 23,16PostgreSQL flat 95014,05 0,24 1780,90 12,97

MonetDB 8490,90 2,72 529,40 43,62

LAStools** 1639,30 14,09 440,50 52,42LAStools LAZ** 2377,30 9,71 35,67 647,28Oracle Exadata (oe-qh) 928,84 22,71 94,24 223,80

* Ordered loading** Licensed version


LoadingMedium benchmark comparison: Loading (inc. indexing) of 23090M points

0,33 0,12

1,65

0,51

3,302,54

1,46

0,24

2,72

14,09

9,71

22,71

0,00

5,00

10,00

15,00

20,00

25,00

Oracle blocksglobal

Oracle blocksglobal compress

Oracle blocksincrem.

Oracle blocksincrem. compress

PostgreSQL blocksincrem.

PostgreSQL blocksincrem. compress

Oracle flat PostgreSQL flat MonetDB LAStools LAStools LAZ Oracle Exadata

Loading [Mpts/s]


LoadingMedium benchmark comparison: Loading (inc. indexing) of 23090M points

0,33

0,12

1,65

0,51

3,30

2,54

1,46

0,24

2,72

0,00

0,50

1,00

1,50

2,00

2,50

3,00

3,50

4,00

Oracle blocksglobal







Loading [Mpts/s]


LoadingMedium benchmark comparison: Storage of 23090M points

32,99

104,91

32,99

104,91 115,45

216,20

23,16 12,97

43,62 52,42

647,28

223,80

0,00

100,00

200,00

300,00

400,00

500,00

600,00

700,00

Oracle blocksglobal







Storage [Mpts/GB]


Program Agenda

1

2

3

4

5

Introduction



Loading

Querying

Research and others

Conclusions

6

7


Querying – Oracle Blocks


QueryingOracle blocks – CLIP_PC (return blocks)

SQL> select

blk_id,

num_points

from

table(

sdo_pc_pkg.clip_pc(

inp => (select pc from pcs where id = 1),

ind_dim_qry => sdo_geometry(

2003,

null,

null,

sdo_elem_info_array(1, 1003, 3),

sdo_ordinate_array(10, 10, 14, 14)),

other_dim_qry => null,

qry_min_res => 1,

qry_max_res => 1));

BLK_ID NUM_POINTS

---------- ----------

53 25


QueryingOracle blocks – CLIP_PC (return individual point records)

select

query_points.x,

query_points.y,

query_points.z

from

table(

sdo_pc_pkg.clip_pc(



2003, null, null,




qry_min_res => 1,

qry_max_res => 1)) query_blocks,

table(

sdo_util.getvertices(

geometry => sdo_pc_pkg.to_geometry(

pts => query_blocks.points,

num_pts => query_blocks.num_points,

pc_tot_dim => 3,

srid => null))) query_points;


QueryingOracle blocks – CLIP_PC (query by non-indexed dim)

select

query_points.x,

query_points.y,

query_points.z,

query_points.w Intensity -- followed by v5 - v11

from

table(

sdo_pc_pkg.clip_pc(



2003, null, null,



other_dim_qry => sdo_mbr(

lower_left => sdo_vpoint_type(123, 1),

upper_right => sdo_vpoint_type(123, 1000)),

qry_min_res => 1,


table(





pc_tot_dim => 4,


X Y Z INTENSITY

---------- ---------- ---------- ----------

10 13 123 456

…


QueryingOracle blocks – CLIP_PC (support non-blocked dimensions)select

query_points.x,

query_points.y,

query_points.z,

query_points.w Intensity, -- followed by v5 - v11

out.non_blocked_dim

from

table(

sdo_pc_pkg.clip_pc(


ind_dim_qry => sdo_geometry(2003, null, null, sdo_elem_info_array(1, 1003, 3), sdo_ordinate_array(10, 10, 14, 14)),

other_dim_qry => sdo_mbr(

lower_left => sdo_vpoint_type(123, 1),

upper_right => sdo_vpoint_type(123, 1000)),

qry_min_res => 1,


table(





pc_tot_dim => 4,

srid => null ,

get_ids => 1))) query_points ,

pcs_out out

where

out.ptn_id = query_points.v5 and -- v5 is blk_id, v6 is pt_id

out.point_id = query_points.v6;

X Y Z INTENSITY NON_BLOCKED_DIM

---------- ---------- ---------- ---------- --------------------------------

11 10 123 456 # 21

…


QueryingOracle blocks – SDO_PC_NN

SQL> select

2 rownum pt_pos,

3 sdo_geometry(

4 3001,

5 null,

6 sdo_point_type(x, y, z),

10 null,

11 null) pts

12 from

13 table(

14 sdo_util.getvertices(

15 geometry => sdo_pc_pkg.to_geometry(

16 pts => sdo_pc_pkg.sdo_pc_nn(

17 pc => (select pc from pcs where pc_id = 1),

18 center => sdo_geometry(3001, null, sdo_point_type(15, 15, 30), null, null),

24 n => 3200),

25 num_pts => 3200,

26 pc_tot_dim => 3,

27 srid => null,

28 blk_domain => null,

29 get_ids => 1)));

PT_POS PTS(SDO_GTYPE, SDO_SRID, SDO_POINT(X, Y, Z), SDO_ELEM_INFO, SDO_ORDINATES)

------------------------------------------------------------------------------------------

1 SDO_GEOMETRY(3001, NULL, SDO_POINT_TYPE(15, 15, 30), NULL, NULL)


…



Querying – PostgreSQL Blocks


QueryingPostgreSQL blocks – PC_Intersects / PC_Intersection (return blocks)

SELECT

PC_Intersection(pa,geom)

FROM patches, query_polygons

WHERE PC_Intersects(pa,geom) and query_polygons.id = 1;

CREATE TABLE query_polygons (id integer, geom public.geometry(Geometry, SRID));

INSERT INTO query_polygons VALUES (1,ST_GeomFromEWKT(WKT));


QueryingPostgreSQL blocks – PC_Intersects / PC_Intersection (return individual point records)

SELECT

PC_Get(qpoint, 'x') AS x,PC_Get(qpoint, 'y') AS y, PC_Get(qpoint, 'z') AS z

FROM

(

SELECT

PC_Explode(PC_Intersection(pa,geom)) AS qpoint

FROM patches, query_polygons

WHERE PC_Intersects(pa,geom) and query_polygons.id = 1

) AS qtable;


QueryingPostgreSQL blocks – Nearest Neighbor

SELECT

PC_Get(qpoint, 'x') as x, PC_Get(qpoint, 'y') as y, PC_Get(qpoint, 'z') as z

FROM

(SELECT

PC_explode(pa) as qpoint

FROM

(SELECT

pa

FROM patches ORDER BY geometry(pa) <#> ST_SetSRID(ST_MakePoint(px, py), SRID)

LIMIT 9

) as A

) as B

ORDER BY ((PC_Get(qpoint, 'x') - px)^2 +(PC_Get(qpoint, 'y') - py)^2)

LIMIT 1000;


Querying – LAStools


QueryingLAStools – lasmerge

• If hybrid approach with PostgreSQL-PostGIS:

CREATE TABLE query_polygons (id integer, geom public.geometry(Geometry, SRID));

INSERT INTO query_polygons VALUES (1,ST_GeomFromEWKT(WKT));

psql dbname -U dbuser -h dbhost -p dbport -t -A -c \

“SELECT l.filepath FROM lasindex l,query_polygons qp WHERE ST_Intersects(qp.geom, l.geom) AND qp.id=1"

\

> input1.list

• Rectangle:

lasmerge -lof input.list -inside minx miny maxx maxy -o output.las

• Circle:

lasmerge -lof input.list -inside_circle cx cy r -o output.las


QueryingLAStools – lasclip (query region in ShapeFile)

pgsql2shp -f query5.shp -h dbhost -p dbport -u dbuser -P dbpass dbname \

“select ST_SetSRID(geom, 28992) from query_polygons where id = 5;”

lasclip.exe -lof input5.list -poly query5.shp -o output5.las -merged

• If hybrid approach with PostgreSQL-PostGIS:

psql dbname -U dbuser -h dbhost -p dbport -t -A -c \

“SELECT l.filepath FROM lasindex l,query_polygons qp WHERE

ST_Intersects(qp.geom, l.geom) AND qp.id=5" \

> input5.list


QueryingLAStools

• Tune LAZ chunksize (default 50000) depending on your type of queries

• lassort of large files done with sub-tiling

• lasclip/lasmerge option io_buffer (default 262144 bytes) for granular accesses

• Ask in LAStools forum


Querying – Oracle Flat


QueryingOracle flat – Plain old SQL

• Rectangle:

SELECT

*

FROM

FLAT

WHERE

(VAL_D1 BETWEEN minx AND maxx) AND

(VAL_D2 BETWEEN miny AND maxy);

• Circle:

SELECT

*

FROM

(

SELECT

*

FROM

FLAT

WHERE

(VAL_D1 BETWEEN cx - r AND cx + r) AND

(VAL_D2 BETWEEN cy - r AND cy + r)

) B

WHERE

POWER((VAL_D1 - cx), 2) +

POWER((VAL_D2 - cy), 2) <= POWER(r, 2);


QueryingOracle flat – SDO_PointInPolygon

• Other regions (example circle):SQL> SELECT

x,y,z

FROM

TABLE(

mdsys.sdo_PointInPolygon(

CURSOR(

select

val_d1 x,

val_d2 y,

val_d3 z

from

flat),

MDSYS.SDO_GEOMETRY(2003, NULL, NULL,

MDSYS.SDO_ELEM_INFO_ARRAY(1, 1003, 4), -- circle with

MBR(10,10, 14,14)

MDSYS.SDO_ORDINATE_ARRAY(10, 12, 12, 10, 14, 12)),

0.05));

X Y Z

---------- ---------- ----------

10 12 123

…

13 rows selected.

Elapsed: 00:00:13.16

• Other regions (with pre-filtering):SQL> SELECT

x, y, z

FROM

TABLE(

mdsys.sdo_PointInPolygon(

CURSOR(

select

val_d1 x,

val_d2 y,

val_d3 z

from

flat

where

val_d1 between 10 and 14 and

val_d2 between 10 and 14),

MDSYS.SDO_GEOMETRY(2003, NULL, NULL,

MDSYS.SDO_ELEM_INFO_ARRAY(1, 1003, 4), -- circle with MBR

(10, 10, 14, 14)

MDSYS.SDO_ORDINATE_ARRAY(10, 12, 12, 10, 14, 12)), 0.05));

X Y Z

---------- ---------- ----------

10 12 123

…

13 rows selected.

Elapsed: 00:00:00.00


QueryingOracle flat

• Nearest neighbor - Approximation:

SELECT

*

FROM

(

SELECT

*

FROM

flat

WHERE

(

(x BETWEEN (px – r) AND (px + r)) AND

(y BETWEEN (py – r) AND (py + r))

)

ORDER BY

POWER(x - px, 2) + POWER(y - py, 2)

) a

WHERE

ROWNUM <= 1000;


QueryingMedium benchmark comparison

PostgreSQL flat: parallel querying with PQGR algorithm

Oracle flat: native parallel

MonetDB flat: native parallel

PostgreSQL blocks: : parallel querying with PQGR algorithm

Oracle blocks: : parallel querying with PQGR algorithm

LAStools: single-process

Oracle EXADATA X4-2 half rack: only 23090M


QueryingMedium benchmark comparison: 20M and 23090M; Queries #1 and #6

Query 1 20M

Query 1 23090M

Query 6 20M

Query 6 23090M

Oracle blocks global* 0,35 0,35 1,20 1,20

Oracle blocks global compress* 0,41 0,41 1,38 1,40

Oracle blocks increm.* 0,35 0,35 1,20 1,20

Oracle blocks increm. compress* 0,41 0,41 1,38 1,40PostgreSQL blocks increm.* 0,25 0,25 1,50 1,50PostgreSQL blocks increm. compress* 0,32 0,32 1,72 1,68Oracle flat 0,24 1,25 1,32 27,18PostgreSQL flat* 0,35 5,17 1,72 15,46

MonetDB 0,06 7,21 187,16 198,51

LAStools** 0,02 0,03 1,46 1,48LAStools LAZ** 0,14 0,12 1,90 1,91Oracle Exadata 0,18 0,67 * Parallelized with

algorithmn (not native)** Single process


QueryingMedium benchmark comparison: Queries #1 Rectangle 51x53 m, 0.0027 km2, ~74872 points

0,00

0,05

0,10

0,15

0,20

0,25

0,30

0,35

0,40

0,45

0,50

Oracle blocksglobal




PostgreSQLblocks increm.


compress


Tim

e [s

]

Query #1 response time

Query 1 20MQuery 1 23090M


QueryingMedium benchmark comparison: Queries #6

0,00

0,20

0,40

0,60

0,80

1,00

1,20

1,40

1,60

1,80

2,00

Oracle blocksglobal




compress



compress


Tim

e [s

]

Query #6 response time

Query 6 20M

Query 6 23090M

Complex Polygon, 792 pts, 0.025 km2, 1 hole, ~387135 points


Program Agenda

1

2

3

4

5

Introduction



Loading

Querying

Research and others

Conclusions

6

7


Create PC with pyramid

declare

pc sdo_pc;

begin

pc :=

sdo_pc_pkg.init(

basetable => 'PCS', --(base table)

basecol => 'PC', --(base column)

blktable => 'BLOCKS', --(data table)

ptn_params => 'blk_capacity=10000', --(block capacity, tablespace)

pc_extent => mdsys.sdo_geometry( --(spatial extent)

2003,

null,

null,

mdsys.sdo_elem_info_array(1, 1003, 3),

mdsys.sdo_ordinate_array(1, 1, power(2, 13), power(2, 13))),

pc_tol => 0.5, --(tolerance)

pc_tot_dimensions => 3, --(total dimensions)

pc_domain => null,

pc_val_attr_tables => null,

pc_other_attrs => xmltype( --(object metadata – pyramiding)

'<opc:sdoPcObjectMetadata

xmlns:opc="http://xmlns.oracle.com/spatial/vis3d/2011/sdovis3d.xsd"

xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">

<opc:sdoPcPyramid preserveLevel1="true"/>

</opc:sdoPcObjectMetadata>'));

sdo_pc_pkg.create_pc(

pc,

'INPTAB'); --(input table)

insert into pcs values (1, pc);

end;

/

Research and othersLevel of detail

• Currently available in Oracle SDO_PCR-tree, SDO_TIN, not yet SDO_PCHilbert

• Makes visualization of point cloud scale very well

Resulting blocks in pyramid

SQL> select

2 pcblk_min_res "Pyramid Level",

3 sum(num_points) "# Points",

4 count(*) "# Blocks"

5 from

6 blocks

7 group by

8 pcblk_min_res

9 order by

10 pcblk_min_res;

Pyramid Level # Points # Blocks

------------- ---------- ----------

1 1048576 105

2 524288 41

…

8 27166 1

8 rows selected.


Query with LoD

select

query_points.x,

query_points.y,

query_points.z

from

table(

sdo_pc_pkg.clip_pc(



2003,

null,

null,




qry_min_res => 1,


table(





pc_tot_dim => 3,



• Currently available in Oracle SDO_PCR-tree, SDO_TIN, not yet SDO_PCHilbert

– Thinning of points• Currently single simple approach

• Others planned, maybe based on slope, category, intensity, ...

• Maybe customizable with Java function...

1

2

1

3

1

2

1

5

4

1

2

1

3

1

2

1 1

2

1

3

1

2

1

4

1

2

1

3

1

2

1

6

1

1 2 1 3 1


Research and othersParallel querying algorithms

• Parallel Querying with Candidate blocks (PQCB)

• Parallel Querying with Gridded Region (PQGR)


Research and othersParallel Querying with Candidate blocks: Example Oracle blocks (Python cx_Oracle)

-- Get IDs of blocks “touching” the query region

SELECT BLK_ID

FROM blocks, query_polygons

WHERE SDO_FILTER(BLK_EXTENT,GEOM) = 'TRUE' AND id = 1;

-- Split the list of IDs in 8 chunks (in python)

-- Create table to store the results

CREATE TABLE query_results_1 (x NUMERIC, y NUMERIC, z NUMERIC);

-- The 8 processes insert data in parallel (example for process 1)

INSERT INTO query_results_1

SELECT x, y, z

FROM table (

sdo_PointInPolygon (

cursor (

SELECT pnt.x x, pnt.y y, pnt.z z

FROM (

SELECT points, num_points

FROM blocks

WHERE (BLK_ID = 1 OR BLK_ID = 2 …)

) pcblob,

TABLE (

sdo_util.getvertices( sdo_pc_pkg.to_geometry(pcblob.points,

pcblob.num_points, 3,NULL))

) pnt

),

(select geom from query_polygons where id = 1), 0.0001, NULL));


Research and othersParallel Querying with Gridded Region: Example PostgreSQL blocks (Python psycopg2)

-- Create table with 8 sub-regions of query region

CREATE TABLE query_grid_1 AS

WITH A AS (

SELECT geom, st_xmin(geom) as minx, st_xmax(geom) as maxx,

st_ymin(geom) as miny, st_ymax(geom) as maxy

FROM query_polygons where id = 1)

SELECT B.row, B.col,

ST_Intersection(A.geom, ST_SetSRID(B.geom, ’28992’)) as geom

FROM

A,

(SELECT F.row, F.col, F.geom

FROM A,

ST_CreateFishnet(2, 4, (maxx-minx) / 4,

(maxy-miny) / 2, minx, miny) F

) AS B ;

CREATE INDEX query_grid_1_rowcol ON query_grid_1 ( row, col );

-- Create table to store the results

CREATE TABLE query_results_1 (x NUMERIC, y NUMERIC, z NUMERIC);

-- The 8 processes insert data in parallel (example for process 1-1)

INSERT INTO

query_results_1 ( x,y,z )

SELECT PC_Get(qpoint, ’x’) as x,

PC_Get(qpoint, ’y’) as y,

PC_Get(qpoint, ’z’) as z

FROM

(SELECT PC_explode(pc_intersection(pa,geom)) as qpoint

FROM patches, query_grid_1

WHERE pc_intersects(pa,geom) AND row = 1 AND col = 1) as D;

-- Remove possible duplicated points

CREATE TABLE cleaned_query_results_1 AS SELECT DISTINCT * FROM query_results_1;

http://trac.osgeo.org/postgis/wiki/UsersWikiCreateFishnet


Research and othersFaster flat tables: Query comparison

0

1

2

3

4

5

6

7

8

20M 210M 2201M 23090M

Tim

e[s

]

Query #1 (rectangle 51 x 53 m, ~74872 points)

PostgreSQL flat

Oracle flat

MonetDB flat

PostgreSQL blocks

Oracle blocks

LAStools


Research and othersFaster flat tables

• Flat tables fast when small dataset but not scaling

• Blocks not so fast but “perfectly” scaling (better storage but need “blocks” processing in queries)

• What if we want DBMS with speed of flat tables but good scaling? (we do not care about storage)


Research and othersFaster flat tables: spatial clustering

• Reorder data based on spatial dimensions

– Efficiency in storage and queries

– Already used in:

• LAStools (lassort)

• Oracle blocks (Data preparation in Hilbert R-Tree blocking)

• Space filling curves: Hilbert/Morton

– Useful for flat model directly

– X, Y -> Code (position in the curve) No need to store X,Y!

– Can also be used in cases where point dimension is not per se spatial (x, y, z), but of different nature (t or vario-LoD/imp)

– Queries need to be re-written to use spatial clustering and be more efficient


Research and othersFaster flat tables: Current approach flat table

0 1 2 3 y

x

3

2

1

0

x y pos

0 0 0

0 1 2

0 2 3

0 3 9

1 0 1

1 1 4

…

index

0

15

Example input data(sorted as acquired)

x y

0 0

1 0

0 1

0 2

1 1

....

Flat table B-Tree XY


Research and othersFaster flat tables: Current approach flat table with clustering (on B-Tree)

0 1 2 3 y

x

3

2

1

0

0

15x y pos

0 0 0

0 1 1

0 2 2

0 3 3

1 0 4

1 1 5

…

clustering

0 1 2 3 y

x

3

2

1

0

x y pos

0 0 0

0 1 2

0 2 3

0 3 9

1 0 1

1 1 4

…

0

15 x y

0 0

1 0

0 1

0 2

1 1

....

Flat table B-Tree XY Flat table


Research and othersFaster flat tables: Space filling curves

0 1 2 3 y

x

3

2

1

0

0

15

0 1 2 3 y

x

3

2

1

0

0

15

0 1 2 3 y

0 15

x

3

2

1

0

(first y, then x) Morton / PeanoHilbert

Bad spatial clustering Better spatial clustering


Research and othersFaster flat tables: Constructing Morton curve

0

00 01

row

01

00

3

0 1 2 3 col

row

3

2

1

0

0

15

Peano or Morton or N-order (or Z-order)

Recursively replace each vertex of basic curve with the previous order curve

Alternative: bitwise interleaving

Also works in 3D (or nD)

0 1 2 3

3

2

1

0

0

4 5 6 7 col

row

7

6

5

4

63


Research and othersFaster flat tables: 3D Morton curve

Illustrations from http://asgerhoedt.dk

2x2x2 4x4x4 8x8x8

http://asgerhoedt.dk/


Research and othersFaster flat tables: Constructing Hilbert curve

Hilbert

rotate previous order curve at vertex 0 (-) and vertex 3 (+)

Also in 3D

00 01

row

01

000 3

0 1 2 3 col

row

3

2

1

0

0 150 63


Research and othersFaster flat tables: 3D Hilbert curve

Illustrations from Wikimedia Commons

2x2x2 4x4x4 8x8x8


Research and othersFaster flat tables: Average number of clusters for all possible range queries

N=8, number of Clusters for a given range query:

0 63

Peano (3 ranges) Hilbert (2 ranges)

N*NGRID

HILBERT PEANO l

2*2 1.11 1.22 0.11

4*4 1.64 2.16 0.52

8*8 2.93 4.41 1.48

16*16 5.60 9.29 3.69

Faloutsos and Roseman, 1989



• LAStools lassort+lasindex

• Oracle Hilbert blocking

• PostgreSQL Morton/QuadTree blocking (NLeSC/TU Delft test)

• SFC flat table model (NLeSC/TU Delft test)

– compute Hilbert / Morton codes for all points

– create b-tree index on Hilbert / Morton code (position in the curve)

– cluster flat table on Hilbert / Morton curve

– re-write queries to select ranges of codes



• Better SFC flat table model:

– Not use the default heap-table, but an indexed organized table (not PostgreSQL)

– No separate index structure needed -> more compact, faster (not PostgreSQL)

– No need to store encoded dimensions (if 2D: x, y not necessary -> only code, z)


Research and othersFaster flat tables: SQL DDL (2D morton codes, still using X, Y)

• Oracle

• PostgreSQL

• MonetDB

CREATE TABLE flat (x,y,z,morton);

[… BINARY LOAD …]

CREATE INDEX morton_index ON flat (morton);

CLUSTER flat using morton_index;

CREATE TABLE flat_temp (x,y,z,morton);

[… BINARY LOAD …]

CREATE TABLE flat (x,y,z,morton);

INSERT INTO flat SELECT * FROM flat_temp ORDER BY morton;

DROP TABLE flat_temp;

CREATE TABLE flat_temp (x,y,z,morton);

[… LOAD …]

CREATE TABLE flat (x,y,z,morton), CONSTRAINT PRIMARY KEY (morton, z) ORGANIZATION INDEX;

DROP TABLE flat_temp;


Research and othersFaster flat tables: Computing Morton code

000 001 010 011 100 101 110 111

0 1 2 3 4 5 6 7

X

111

7

110

6

101

5

100

4

011

3

010

2

001

1

000

0

Y

0

1

8

75

6

4

3

2

61

62

63

• Bitwise interleaving x-y coordinates

• Also works in higher dimensions (nD)

• Two example of Morton code:

x= 110, y=111 xy= 111101 (decimal 61)

x= 001, y=010 xy= 000110 (decimal 6)


Research and othersFaster flat tables: “Morton queries”

• Based on concepts of Region Quadtree & Quadcodes

• Works for any type of query geometry (point, polyline, polygon)

• Also works in 3D (Octree) and higher dimensions

Quadcode 0: Morton range 0-15




(Morton code gaps resp. 0, 4, 20)

query_geometry, polygon

Note : SW=0, NW=1, SE=2, NE=3

111

7

110

6

101

5

100

4

011

3

010

2

001

1

000

0

Y

0

12300

10

000 001 010 011 100 101 110 111

0 1 2 3 4 5 6 7

X



CREATE TABLE query_results_1 AS ( SELECT x,y,z FROM (

SELECT x,y,z FROM ahn_flat WHERE ((morton2D between 1341720113446912 and 1341720117641215)

OR (morton2D between 1341720126029824 and 1341720134418431) OR (morton2D between 1341720310579200 and 1341720314773503)OR (morton2D between 1341720474157056 and 1341720478351359) OR (morton2D between 1341720482545664 and 1341720503517183) OR (morton2D between 1341720671289344 and 1341720675483647) OR (morton2D between 1341720679677952 and 1341720683872255))

) a WHERE (x between 85670.0 and 85721.0) and (y between 446416.0 and 446469.0));



0

1

2

3

4

5

6

20M 210M 2201M 23090M

Tim

e[s

]

Query #1 (rectangle 51 x 53 m, 74872 points)

PostgreSQL

PostgreSQL Morton

0

1

2

3

4

5

6

20M 210M 2201M 23090M

Tim

e[s

]

MonetDB

MonetDB Morton


Research and othersFull AHN2 benchmark: loading

system Total load

time [hours]

Total

size [TB]

#points

LAStools unlic. 22:54 12.181 638,609,393,087

LAStools lic 16:47 11.617 638,609,393,101

LAStools lic LAZ 15:48 0.973 638,609,393,101

Oracle Exadata 4:39 2.240 639,478,217,460

MonetDB* 17:21 15.00 639,478,217,460

* Run in system with 256 GB RAM (instead of 128 GB)


Research and othersFull AHN2 benchmark: querying

#points Time[s]

QUERY LAStools unlic. LAStools Exadata MonetDB LAStools unlic. LAS DB LAStools unlic. LAS No DB LAStools LAS DB LAStools LAS No DB LAStools LAZ DB LAStools LAZ No DB Exadata MonetDB01_S_RECT 74861 74863 74863 74855 0,07 0,79 0,06 0,77 0,2 0,84 0,48 108,9902_L_RECT 718057 718070 718070 718132 0,16 0,85 0,16 0,78 0,49 1,09 0,79 111,3303_S_CIRC 34700 34699 34675 34699 0,07 0,78 0,06 0,7 0,27 0,89 1,22 109,3604_L_CIRC 563119 563105 563082 563105 0,16 0,85 0,12 0,78 0,44 1,15 1,69 112,910_S_RECT_MAXZ 2413 1734 2434 2434 0,08 0,82 0,08 0,69 0,43 1,02 0,4 112,4311_S_RECT_MINZ 591 591 591 591 0,05 0,84 0,05 0,72 0,13 0,73 0,44 111,3112_L_RECT_MINZ 343168 343168 343171 343171 0,26 1 0,26 0,98 1,32 1,95 0,6 114,9816_XL_RECT_EMPTY 0 0 0 0 0,04 0,75 0,04 0,66 0,03 0,69 0 113,5118_NN_1000 1000 112,919_NN_5000 5000 107,9420_NN_1000_river 0 109,205_S_SIMP_POLY 182871 182875 182875 182875 0,7 31,05 0,39 28,2 0,63 29,46 1,43 110,4206_L_COMP_POLY_HOLE 460140 460158 387201 387193 1,52 32 1,57 30,2 1,97 30,31 1,29 112,1707_DIAG_RECT 45831 45813 45815 45815 0,55 31,43 0,55 28,76 0,99 29,23 1,71 112,1408_XL_POLYGON_2_HOLES 2365925 2365896 2273469 2273380 3,72 34,72 3,87 33,88 5,67 35,88 2,86 120,9309_S_POLYLINE_BUFFER 620568 620527 620719 620529 2,34 33,44 2,27 32,4 3,94 33,43 1,58 118,7115_XL_RECT 3992330 3992377 3992290 3992525 0,49 1,37 0,57 1,23 1,94 2,63 2,23 120,1817_XL_CIRC 2203066 2203055 2201280 2203055 0,32 1,16 0,32 1,01 1,1 1,71 2,51 114,3121_L_NARROW_RECT 382395 382335 382335 386278 2,23 4,43 2,38 4,11 16,94 18,81 0,95 109,227_STREET_DIAG_RECT 27443 27453 27459 27456 1,61 914,44 0,25 29,18 0,35 29,06 1,23 107,9213_L_POLYLINE_BUFFER 897042 896927 897359 897126 412,29 804,29 266,13 958,88 317,91 1039,09 23,34 723,5814_DIAG_POLYLINE_BUFFER 765989 766039 766029 766029 102,19 419,78 104,09 428,51 151,45 600,93 15,05 559,7322_L_NARROW_DIAG_RECT 12148049 12147863 121478022246.9 154,97 1115,2 141,95 1800,41 284,58 2478,47 113,37 2246,923_XL_NARROW_DIAG_RECT 691422551 6,91E+08 6,91E+08151.16 379,2 886,35 318,93 1051,76 552,01 1154,7 330,85 151,1624_L_NARROW_DIAG_RECT_2 2468239 2468268 2468367 2468367 251,13 4106,54 294,99 13454 551,13 16510,9 393,21 6308,5325_PROV_DIAG_RECT 2,03E+09 3,53E+1412228.76 1193,1 12228,7626_MUNI_RECT 2124162497 2,12E+09 2,12E+09622.73 417,28 1127,56 214,98 405,26 536,1 560,95 25,79 622,7328_VAALS 809097723 8,09E+08 8,67E+081469.47 1635,3 1883,19 2240,1 2525,98 2426,43 2718,26 67,67 1469,4729_MONTFERLAND 1509662615 1,51E+09 1,51E+095577.69 3579,73 5746,61 4809,9 6419,05 3855,93 6396,54 120,02 5577,6930_WESTERVELD 2,9E+09 1,34E+1427683.62 3569,54 27683,62



0

0,1

0,2

0,3

0,4

0,5

0,6

0,7

0,8

0,9

1

Tim

e [

s]

Easy queries

LAStools LAS DB

LAStools LAS No DB

LAStools LAZ DB

LAStools LAZ No DB

Exadata

MonetDB



0

0,5

1

1,5

2

2,5

3

3,5

4

4,5

5

Tim

e [

s]

Medium queries

LAStools LAS DB

LAStools LAS No DB

LAStools LAZ DB

LAStools LAZ No DB

Exadata

MonetDB



0

500

1000

1500

2000

2500

3000

3500

4000

Tim

e [

s]

Difficult queries

LAStools LAS DB

LAStools LAS No DB

LAStools LAZ DB

LAStools LAZ No DB

Exadata

MonetDB


Program Agenda

1

2

3

4

5

Introduction



Loading

Querying

Research and others

Conclusions

6

7


Conclusions

• For simple queries LAStools is the fastest but has not DBMS functionalities.

• Careful with multi-user environments

• LAStools with LAZ offers best storage (but slower queries)

• For complex queries DBs with blocks offer good performance

• For small datasets (million points) flat tables are suitable

– For Exadata, a lot more

• Blocks require packing/unpacking which adds overhead (noticeable in small queries)

• DB loading should be improved

• Parallel algorithms improve query response times

– Parallelism can be pushed further

• SFC-based flat tables improve scaling issue and faster queries (being investigated)

• Oracle, MonetDB and LAStools actively improving PointCloud support

storage and management of large scale point …pointclouds.nl/docs/workshop_dubai_cut.pdforacle...

Documents