classification of indoor point clouds using multiviews€¦ · vladeta stojanovic, matthias trapp,...
TRANSCRIPT
Classification of Indoor Point Clouds using Multiviews
Vladeta Stojanovic, Matthias Trapp, Rico Richter and Jürgen DöllnerComputer Graphics Systems Group, Hasso Plattner Institute, University of Potsdam, Germany
Web’3D 2019, 19-21 July Los Angeles, USA
Introduction
● Point clouds present a “snapshot” of the physical
environment.
● This makes them useful for base-data for Digital
Twin representations or as-is Building Information
Models (BIMs).
● However, captured point clouds do not contain any
semantics by default.
● In the realm of Real Estate 4.0, the classification of
base-data such as point clouds is an important task.
2Vladeta Stojanovic – www.hpi3d.de Web3D 2019, Los Angeles, USA, 19-21 July 2019.
PhD Research Topics
• Focus on automated semantic
enrichment of point cloud data and
generation of basis data for as-is BIM
and Digital Twin representations
• Implementation of these technologies
as a web-based application using a
services-oriented methodology
• Integration with current FM and
Industry 4.0 practices and standards
3Vladeta Stojanovic – www.hpi3d.de Web3D 2019, Los Angeles, USA, 19-21 July 2019.
Motivation and Research Contribution
● 3D point clouds can represent the physical features of indoor
environments, such as typical office environments.
● Point clouds may feature either RGB or intensity color-mapped
representations, but semantics are not explicitly contained in the
data.
● Manually segmenting 3D point clouds and assigning semantics
tends to be a time consuming, error prone, and costly process.
● Our approach classifies 3D point clouds using a retrained CNN
based on image analysis for entropy-selected 3D views that
allows for object segmentation and adding of semantics.
4Vladeta Stojanovic – www.hpi3d.de Web3D 2019, Los Angeles, USA, 19-21 July 2019.
Approach Overview
● We have implemented the presented cluster, viewpoint entropy selection, and point
cloud visualization methods using Web3D technologies.
● A retrained CNN used to classify entropy-selected multiview images of point cluster.
● These multiview images are selected using an entropy function.
● We classify cluster and classify the point cloud automatically using a service-oriented
implementation (client/server model).
5Vladeta Stojanovic – www.hpi3d.de Web3D 2019, Los Angeles, USA, 19-21 July 2019.
Approach Overview - Process Flowchart
6Vladeta Stojanovic – www.hpi3d.de Web3D 2019, Los Angeles, USA, 19-21 July 2019.
Clustering Methods● Prior to entropy computation and multiview
classification, we must cluster the given point
cloud.
● The point cloud first has any homogeneous
structural features (e.g., walls, floor and
ceilings), removed using a region growing
algorithm. Normals are also computed for each
point.
● We use either the k-Means or DBSCAN
clustering methods to further segment the
furniture clusters.
7Vladeta Stojanovic – www.hpi3d.de Web3D 2019, Los Angeles, USA, 19-21 July 2019.
Multiview Selection and Image Generation Algorithm
● Randomly sample a number of points around a
bounding sphere of each cluster.
● From each random vertex on the bounding
sphere, a ray is projected inwards towards the
point cloud cluster.
● For each point normal in cluster, we check if the
normal vector is parallel within a threshold to
the bounding sphere vertex vector.
● If true, the given bounding sphere vertex is used
as a multiview camera position and direction.
8Vladeta Stojanovic – www.hpi3d.de Web3D 2019, Los Angeles, USA, 19-21 July 2019.
Multiview Selection and Image Generation
9
Bounding sphere vertices are selected as multiview camera positions and directions, based on the parallelism entropy measure between the inverted bounding vertex normal and the point cluster normal vectors.
Vladeta Stojanovic – www.hpi3d.de Web3D 2019, Los Angeles, USA, 19-21 July 2019.
Classification Implementation
● Classification of multiview images is accomplished using two retrained 2D CNNs (Inception V3
using Tensorflow for Python).
● The first CNN is re-trained using three categories featuring photographs of common office
furniture (chairs, sofas and tables), while the second is re-trained using only photos of chairs
and tables.
● A total of 9759 RGB images for three different categories was used. Each image was resized to
300 x 300 pixels while maintaining the aspect ratio of the image.
● The CNNs were re-trained using 4000 learning steps, with a learning rate of 0.01, with a
predicted classification accuracy of 92.9% and 94.5%.
10Vladeta Stojanovic – www.hpi3d.de Web3D 2019, Los Angeles, USA, 19-21 July 2019.
Service Oriented System Design
● The clustering, entropy viewpoint selection and classification methods are all
implemented as software components within a service-oriented system design.
● We make use of Node.js to create a echo-style server to handle the processing of
multiview images and classification server-side.
● We perform the clustering, 3D point cloud and result visualization client-side (using
JavaScript, with Three.js as the main visualization framework component).
11Vladeta Stojanovic – www.hpi3d.de Web3D 2019, Los Angeles, USA, 19-21 July 2019.
Service-Oriented System Design
12Vladeta Stojanovic – www.hpi3d.de Web3D 2019, Los Angeles, USA, 19-21 July 2019.
Testing Procedure
● 64 different test sets. 32 tests using the Stanford dataset [Armeni et al.
2017] with 16 different scenes, and 32 tests based on 16 synthetic scenes
that we constructed.
● Additional synthetic scenes of varying complexity from furniture point
clusters captured using a mobile phone scanning device, and using the
RGB-D Object Dataset from University of Washington [Lai et al. 2011].
● The Stanford dataset was chosen as it features a high number of
cluttered items in the desired classification categories (e.g., chairs, tables
and sofas).
● For each of the two datasets, we tested each of the 16 scenes using
either k-Means or DBSCAN clustering.
13Vladeta Stojanovic – www.hpi3d.de Web3D 2019, Los Angeles, USA, 19-21 July 2019.
Experimental Results
● We present experimental results using the entropy-based multiview selection and
classification approach.
● We tested our approach using both k-Means and DBSCAN clustering methods.
● Test data included both real-life and synthetically generated datasets.
● Red clusters indicate chair object, blue clusters indicate sofa object, and green
clusters indicate table object classifications.
14Vladeta Stojanovic – www.hpi3d.de Web3D 2019, Los Angeles, USA, 19-21 July 2019.
Experimental Results
15
(a) Original (b) k-Means Clustering (c) DBSCAN Clustering
(d) Original (e) k-Means Clustering (f) DBSCAN Clustering
Vladeta Stojanovic – www.hpi3d.de Web3D 2019, Los Angeles, USA, 19-21 July 2019.
Experimental Results
16
(a) Original (b) k-Means Clustering (c) DBSCAN Clustering
(d) Original (e) k-Means Clustering (f) DBSCAN Clustering
Vladeta Stojanovic – www.hpi3d.de Web3D 2019, Los Angeles, USA, 19-21 July 2019.
Experimental Results - Previous Approach
17
(a) [Stojanovic et al. 2018] Octree-based multiview classification, (b) [Stojanovic et al. 2019] k-Means Clustering and (c) [Stojanovic et al. 2019] DBSCAN Clustering.
(a) (b) (c)
Vladeta Stojanovic – www.hpi3d.de Web3D 2019, Los Angeles, USA, 19-21 July 2019.
Discussion - Approach
● The entropy-based viewpoint selection method generates multiview images featuring perspective projection
of 3D point clusters.
● Main drawback: using only highly parallel normal vectors as an entropy measure is that for flat surfaces only
directly top-down views will mostly be included in the entropy measure.
● We generate an average of 11 images for a given point cluster using the default parameters.
● This allows us to classify a scene in a few minutes on average using the implement client-server
web-application model.
● The CNN was re-trained using actual photographs of furniture items (chairs, tables, and sofas), thus it is more
general.
18Vladeta Stojanovic – www.hpi3d.de Web3D 2019, Los Angeles, USA, 19-21 July 2019.
Discussion - Results
● The highest probability of correct classification is achieved if the DBSCAN clustering method is used for
clustering.
● We observed that the DBSCAN algorithm generally provides better clustering results, if the point set has
uniform density.
● The DBSCAN algorithm is also faster at computing clusters than the k-Means algorithm.
● The k-Means algorithm is better suited for clustering scenes with lots of clutter and lack of spatial divisions.
● The probability of correct classification largely varies on the types of scenes and the visual style of the 3D
point clusters.
● Method provides a better results than the previous Octree-based approach [Stojanovic et al. 2018].
19Vladeta Stojanovic – www.hpi3d.de Web3D 2019, Los Angeles, USA, 19-21 July 2019.
Conclusions and Outlook
● The experimental results show that our approach is mostly accurate and able to quickly classify more than
half of the objects in the tests scenes correctly.
● We have implemented our approach as a service-based software component, which can simplify its
integration in complex and larger service-oriented enterprise systems.
● For future work, we plan to evaluate spatial inference methods to better evaluate the classification results
(e.g., bounding volume comparisons), increase the classification categories (e.g. different types of chairs,
sofas and tables), and to evaluate the OPTICS clustering algorithm.
20Vladeta Stojanovic – www.hpi3d.de Web3D 2019, Los Angeles, USA, 19-21 July 2019.
Key ReferencesVladeta Stojanovic, Matthias Trapp, Rico Richter, and Jürgen Döllner. 2018. A Service-Oriented Approach for Classifying 3D Points Clouds by Example of Office Furniture
Classification. In Proceedings of the 23rd International ACM Conference on 3D Web Technology. ACM, 2.
Vladeta Stojanovic, Matthias Trapp, Rico Richter, Benjamin Hagedorn, and Jürgen Döllner. 2018. Towards The Generation of Digital Twins for Facility Management based on 3D
Point Clouds. In Proceeding of the 34th Annual ARCOM Conference. 270–279.
Iro Armeni, Ozan Sener, Amir R Zamir, Helen Jiang, Ioannis Brilakis, Martin Fischer, and Silvio Savarese. 2016. 3D Semantic Parsing of Large-Scale Indoor Spaces. In Proceedings of
the IEEE Conference on Computer Vision and Pattern Recognition. 1534–1543.
Helin Dutagaci, Chun Pan Cheung, and Afzal Godil. 2010. A Benchmark for Best View Selection of 3D Objects. In Proceedings of the ACM Workshop on 3D Object Retrieval. ACM,
45–50.
Martin Ester, Hans-Peter Kriegel, Jörg Sander, Xiaowei Xu, et al.1996. A Density-Based Algorithm for Discovering Clusters in Large Spatial Databases with Noise. In Kdd, Vol. 96.
226–231.
Hang Su, Subhransu Maji, Evangelos Kalogerakis, and Erik Learned-Miller. 2015. Multi-view convolutional Neural Networks for 3D Shape Recognition. In Proceedings of theIEEE
international conference on computer vision. 945–953.
Pere-Pau Vázquez, Miquel Feixas, Mateu Sbert, and Wolfgang Heidrich. 2001. View-point Selection using Viewpoint Entropy. In VMV, Vol. 1. 273–280.
Kevin Lai, Liefeng Bo, Xiaofeng Ren, and Dieter Fox. 2011. A Large-Scale Hierarchical Multi-view RGB-D Object Dataset. In Robotics and Automation (ICRA), 2011 IEEEInternational
Conference on. IEEE, 1817–1824.
21Vladeta Stojanovic – www.hpi3d.de Web3D 2019, Los Angeles, USA, 19-21 July 2019.
Computer Graphics System Group | Hasso Plattner Institute | University of Potsdam
Vladeta Stojanovic | [email protected]
Dr. Matthias Trapp | [email protected]
Dr. Rico Richter | [email protected]
Prof. Jürgen Döllner | [email protected]
22Vladeta Stojanovic – www.hpi3d.de Web3D 2019, Los Angeles, USA, 19-21 July 2019.