deep learning for point cloud analysis and...

Deep Learning for Point Cloud Analysis and Classification

EuroSDR 2nd International Workshop on Point Cloud Processing

Sören Discher

Copyright © 2019 Point Cloud Technology GmbH Deep Learning for Point Cloud Analysis and Classification | EuroSDR 2nd International Workshop on Point Cloud Processing 2

Background

q Spin-Off of Hasso Plattner Institute, Potsdam, Germany

q IT solutions for the management, computational use, and visualization of large-scale, highly detailed 3D point clouds

q www.pointcloudtechnology.com

Copyright © 2019 Point Cloud Technology GmbH

http://www.pointcloudtechnology.com/


Typical Data Sources

q Airborne LiDAR

q Rotterdam, The Netherlands

q 14 - 140 pts/m²

q Overall size: 13 billion points, 450 GB



Typical Data Sources

q Aerial Photogrammetry

q Frankfurt, Germany

q 61 - 93 pts/m²

q Overall size: 7.5 billion points, 435 GB



Datasets

q Hamburg, Germany, 668 GB, XYZ + LiDAR Intensity

q Essen, Germany, 226 GB, XYZ + LiDAR Intensity

Goals:

q Detailed asset identificationv Ground, Vegetation, Building Walls, …v Poles, Street Signs, Street Lights, …

q Automated export of fitting cadastral data

Challenges:

q Massive amount of data

q Large variety of semantic classes

q Varying point density

q Shadowing, noise

Deep Learning for Point Cloud Analysis and Classification | EuroSDR 2nd International Workshop on Point Cloud Processing 5

Classifying Mobile Mapping Data: A Real-World Example


Datasets



Goals:



Challenges:




q Shadowing, noise




Idea:q Compute or extract predefined metrics for individual points or point

groups

q Use metrics’ values to identify segments and most probable semanticclass per point segment

Discussion:q Manually defined, rigid, and increasingly complex rule setq Frequent manual adjustments required

v Changing point cloud characteristics (w.r.t. density, 2.5D/3D, …)v Additional semantic classes


Metric-based Classification


Idea:q Rule set is learned evolutionary from already classified sample dataq Based on deep neural networks, applied to...

v Point cloud structure itselfv Images of parts of the point cloud

Ø Hot research area featuring many recently introduced approachesv PointNet (PointNet++)v EdgeConv

Ø Typical workflow: v Trainingv Preprocessingv Predictingv Postprocessing (e.g., export to varying formats)


Deep Learning for 3D Point Clouds


Deep Learning for 3D Point Clouds: Training Data

q Large and fitting training sets are usually not freely available

Deep Learning for Point Cloud Analysis and Classification | EuroSDR 2nd International Workshop on Point Cloud Processing 10Copyright © 2019 Point Cloud Technology GmbH




q Classification with metric-based approach





q Classification with metric-based approach

q Classification based on existing cadastral data

q Manual classification



Preprocessing: Density Reduction

q Problem: Varying point density, plenty redundant information for certain classes (e.g., roads)





q Static reduction: Aggregate point cloud using a 3D grid





q Static reduction: Aggregate point cloud using a 3D grid

q Adaptive reduction: Take surface curvature into account to keep “interesting” details

q Mapping of results to removed points via majority voting in k-neighborhood



Preprocessing: Sampling

q Problem: Point cloud too large to be handled at once by GPU





q Tiling based on 2D Grid: Problematic for irregular data with many empty areas





q Tiling based on 2D Grid: Problematic for irregular data with many empty areas

q Idea: Randomly select seeds (out of so far unused points), then choose k-nearest neighbors

q Points might get classified several times, aggregate results



Point-Based Deep Learning: PointNet

q Idea: Identify critical points defining the shape of a point cloud

q Invariant to affine transformations and input permutations


Point-Based Deep Learning: EdgeConv

q Idea: Exploit local geometric structures via local neighborhood graphs and “edge convolutions”

q Iterative process, computation intense


Point-Based Deep Learning: Hybrid

q Idea: Execute only a single EdgeConv layer instead of several

q Reduces computational effort while preserving (some) information about local geometry


Comparison of Results

q IntensityDeep Learning for Point Cloud Analysis and Classification | EuroSDR 2nd International Workshop on Point Cloud Processing 26Copyright © 2019 Point Cloud Technology GmbH


q EdgeConv




q Hybrid





Deep Learning for Point Cloud Analysis and Classification | EuroSDR 2nd International Workshop on Point Cloud ProcessingCopyright © 2019 Point Cloud Technology GmbH

q PointNet q EdgeConv q Hybrid


q PointNet




q EdgeConv




q Hybrid




Technology Stack:

q Deep Learning: CUDA, PyTorch, NumPy, Pandas, SciPy

q Visualization: C++, OpenGL, QTCopyright © 2019 Point Cloud Technology GmbH


Hardwareq Workstation:

v Intel Xeon E5-2630 (6 cores, 2.60 GHz)v 128 GB RAMv 1x Nvidia RTX 2080 Tiv Windows 10

q Oracle Cloud Infrastructure Shape:v BM.GPU 2.2 + 4TB Block Storagev OCPU with 56 coresv 192 GB RAMv 2x Nvidia P100v Ubuntu 18.04


Evaluation: Setup


Training Data:

q 6 representative areas of Hamburg + Essen data sets

q 156 million points (XYZ+ LiDAR intensity)

q 50 epochs each


Evaluation: Training

Network best epoch (loss) best epoch (accuracy) best epoch (mAP) best epoch (mIoU) best loss best accuracy best mAP best mIoU

PointNet 12 48 50 47 87 83.40 34.85 23.89

Hybrid 17 46 25 50 63 88.42 42.44 32.97

EdgeConv 19 19 23 45 64 87.75 39.64 32.42

Network Time trained (Workstation)

Time trained (Oracle Cloud)

PointNet 169.047s 39.314s

Hybrid 322.796s 62.802s

EdgeConv 656.175s 100.794s


Predicted Data:

q One sector of Hamburg data set

q 1.08 billion points (XYZ+ LiDAR intensity)

q Preprocessing includes: v Adaptive density reduction with factor 0.8 (i.e., 20% of points removed)v k-NN-based sampling of fixed batches of size 64


Evaluation: Preprocessing and Predicting

Network Preprocessing Prediction Remapping

PointNet 501.95s 3136.39s 1158.93s

Hybrid 501.95s 3531,52s 1158.93s

EdgeConv 501.95s 8874,09s 1158.93s

Network Preprocessing Prediction Remapping

PointNet 220.15s 778.26s 591.29s

Hybrid 220.15s 834.94s 591.29s

EdgeConv 220.15s 1684.39s 591.29s

Workstation Oracle Cloud


q Processing time in seconds w.r.t. density reduction (Oracle Cloud, EdgeConv)q Preprocessing (blue), Predicting (red), Remapping (yellow)


Evaluation: Density Reduction


Evaluation: Density Reduction

q Effects of density reduction on IoU for common classes


Outlook: Road Assets Exportq Goal: Post-processing step to detect

and classify 2D road assetsv Manhole coversv Road markings

q Idea: Image-based deep learningv Render orthogonal images of detected road

segments v Use established neural networks for object

detection + classification in imagesv Derive generalized shapes from results

q Challengesv Rendering position, direction, and style:

Occlusion vs sharpness vs empty spacev Post processing of rendered imagesv Performance + accuracy



q State-of-the-art neural networks for 3D point clouds allow to classify massive real-world data sets with high accuracy

q Extremely efficient once properly trained (which may take a while), benefits from massive parallelization and GPU architectures

q Transfer of already learned network configurations to other data sets often difficult due to varying properties of the point cloudv 2.5D vs 3Dv Value ranges of per-point attributes (e.g., LiDAR intensity)v Density

q Depending on classification goal: Image-based deep learning equally or more helpful, e.g., when color/intensity is the main aspect


Conclusions


pointcloudtechnology.com/demos/


Special thanks to:

deep learning for point cloud analysis and...

Documents