efficient volumetric fusion of airborne and street -side ...€¦ · stixels (2.5d) octrees (more...
TRANSCRIPT
![Page 1: Efficient Volumetric Fusion of Airborne and Street -Side ...€¦ · stixels (2.5D) octrees (more complicated) cell complex (poor scalability, needs planes) ICPR 2016. Andras Bodis-Szomoru](https://reader035.vdocuments.mx/reader035/viewer/2022070915/5fb5fca9f8ac8529f92eee44/html5/thumbnails/1.jpg)
Efficient Volumetric Fusion of Airborne and Street-Side Data for Urban Reconstruction
András Bódis-Szomorú, Hayko Riemenschneider, Luc Van Gool
varcity.eu
ICPR 2016, Cancún, Mexico
(Extended slides)
![Page 2: Efficient Volumetric Fusion of Airborne and Street -Side ...€¦ · stixels (2.5D) octrees (more complicated) cell complex (poor scalability, needs planes) ICPR 2016. Andras Bodis-Szomoru](https://reader035.vdocuments.mx/reader035/viewer/2022070915/5fb5fca9f8ac8529f92eee44/html5/thumbnails/2.jpg)
Andras Bodis-SzomoruICPR 2016 2
INCOMPLETE MAP COVERAGE
no upper floors / roofs uncovered pedestrian-only areas undriven districts no courtyards occlusions (trees, parking cars, fences) often no ground
Street-side mobile mappingDENSE 3D POINT CLOUD FROM IMAGERY
![Page 3: Efficient Volumetric Fusion of Airborne and Street -Side ...€¦ · stixels (2.5D) octrees (more complicated) cell complex (poor scalability, needs planes) ICPR 2016. Andras Bodis-Szomoru](https://reader035.vdocuments.mx/reader035/viewer/2022070915/5fb5fca9f8ac8529f92eee44/html5/thumbnails/3.jpg)
Andras Bodis-SzomoruICPR 2016
Airborne 3D acquisition
3
MULTI-VIEW STEREO FROM15-CENTIMETER NADIR IMAGERY CAPTURED AT 3 KM
no detail on street-side walls not visible shadows smoothed (blurry) walls defects (holes)
TYPICAL DEFECTS DUE TO MISSING DATA
![Page 4: Efficient Volumetric Fusion of Airborne and Street -Side ...€¦ · stixels (2.5D) octrees (more complicated) cell complex (poor scalability, needs planes) ICPR 2016. Andras Bodis-Szomoru](https://reader035.vdocuments.mx/reader035/viewer/2022070915/5fb5fca9f8ac8529f92eee44/html5/thumbnails/4.jpg)
Andras Bodis-SzomoruICPR 2016
Idea of airborne/street-side fusion
4
COMPLETENESS
DETAIL
AIRBORNE NADIR 3DSTREET-SIDE 3D
(A MOBILE MAPPING EXAMPLE)
ROOFSHIGH FLOORSCOURTYARDS
MISSING STREETS
Requirements: efficient + watertight + large-scale
![Page 5: Efficient Volumetric Fusion of Airborne and Street -Side ...€¦ · stixels (2.5D) octrees (more complicated) cell complex (poor scalability, needs planes) ICPR 2016. Andras Bodis-Szomoru](https://reader035.vdocuments.mx/reader035/viewer/2022070915/5fb5fca9f8ac8529f92eee44/html5/thumbnails/5.jpg)
Andras Bodis-SzomoruICPR 2016
This paper: efficient volumetric airborne/streetside fusion
5
INPUT AIRBORNE POINT CLOUD INPUT STREET-SIDE POINT CLOUD(SPARSE SFM OR DENSE MVS)LINES OF SIGHT
LINES OF SIGHTAssumptions: visibility data (lines of sight) given point clouds are accurately geo-registered (via precise GPS/IMU or GCP’s)
![Page 6: Efficient Volumetric Fusion of Airborne and Street -Side ...€¦ · stixels (2.5D) octrees (more complicated) cell complex (poor scalability, needs planes) ICPR 2016. Andras Bodis-Szomoru](https://reader035.vdocuments.mx/reader035/viewer/2022070915/5fb5fca9f8ac8529f92eee44/html5/thumbnails/6.jpg)
Andras Bodis-SzomoruICPR 2016
This paper: efficient volumetric airborne/streetside fusion
6
AIRBORNE ONLY (FOR COMPARISON) OUR FUSION RESULT
![Page 7: Efficient Volumetric Fusion of Airborne and Street -Side ...€¦ · stixels (2.5D) octrees (more complicated) cell complex (poor scalability, needs planes) ICPR 2016. Andras Bodis-Szomoru](https://reader035.vdocuments.mx/reader035/viewer/2022070915/5fb5fca9f8ac8529f92eee44/html5/thumbnails/7.jpg)
Andras Bodis-SzomoruICPR 2016
This paper: efficient volumetric airborne/streetside fusion
7
AIRBORNE ONLY (FOR COMPARISON)
OUR FUSION RESULT
![Page 8: Efficient Volumetric Fusion of Airborne and Street -Side ...€¦ · stixels (2.5D) octrees (more complicated) cell complex (poor scalability, needs planes) ICPR 2016. Andras Bodis-Szomoru](https://reader035.vdocuments.mx/reader035/viewer/2022070915/5fb5fca9f8ac8529f92eee44/html5/thumbnails/8.jpg)
Andras Bodis-SzomoruICPR 2016
Related work
8
General surface reconstruction explicit methods, e.g. zippering depth map integration via TSDF, e.g. VRIP, KinectFusion (Curless & Levoy SG‘96, Izadi et al. ’06) s/t cut over voxel grid (unsigned, UDF) (Hornung & Kobbelt EG’06, Lempitsky & Boykov CVPR’07) Poisson Surface Reconstruction (octree) (Kazhdan EG’06) convex variational (voxels or height map), e.g. TV-L1, TGV-fusion cell complex (Chauve CVPR’10) 3DT approaches (Labatut et al. ICCV’07, Lafarge & Alliez EG’13)
Street-side & airborne data superpixel meshes (Bodis et al. CVIU’16) most address geo-localization / registration DSM + street-side LiDAR (Fruh & Zakhor CGA’03) Octree & Dual Contouring (Fiocco et al. 3DIM’05) Poisson Surface Reconstruction (Shan et al. 3DPVT’13)
Superpixel meshes (Bodis et al. CVIU’16)
![Page 9: Efficient Volumetric Fusion of Airborne and Street -Side ...€¦ · stixels (2.5D) octrees (more complicated) cell complex (poor scalability, needs planes) ICPR 2016. Andras Bodis-Szomoru](https://reader035.vdocuments.mx/reader035/viewer/2022070915/5fb5fca9f8ac8529f92eee44/html5/thumbnails/9.jpg)
Andras Bodis-SzomoruICPR 2016
Contributions
9
1st to propose 3DT-fusion for airborne/street-side
Point cloud blending against gross ray conflicts
Techniques to reduce workload (large urban scenes)
Many experiments on detail vs. workload (sparse & dense input)
![Page 10: Efficient Volumetric Fusion of Airborne and Street -Side ...€¦ · stixels (2.5D) octrees (more complicated) cell complex (poor scalability, needs planes) ICPR 2016. Andras Bodis-Szomoru](https://reader035.vdocuments.mx/reader035/viewer/2022070915/5fb5fca9f8ac8529f92eee44/html5/thumbnails/10.jpg)
Andras Bodis-SzomoruICPR 2016
Volumetric surface reconstruction via 3DT
10
UNKNOWN SURFACE TO RECONSTRUCT
![Page 11: Efficient Volumetric Fusion of Airborne and Street -Side ...€¦ · stixels (2.5D) octrees (more complicated) cell complex (poor scalability, needs planes) ICPR 2016. Andras Bodis-Szomoru](https://reader035.vdocuments.mx/reader035/viewer/2022070915/5fb5fca9f8ac8529f92eee44/html5/thumbnails/11.jpg)
Andras Bodis-SzomoruICPR 2016
Volumetric surface reconstruction via 3DT
11
NOISE-FREEINPUT SAMPLES
![Page 12: Efficient Volumetric Fusion of Airborne and Street -Side ...€¦ · stixels (2.5D) octrees (more complicated) cell complex (poor scalability, needs planes) ICPR 2016. Andras Bodis-Szomoru](https://reader035.vdocuments.mx/reader035/viewer/2022070915/5fb5fca9f8ac8529f92eee44/html5/thumbnails/12.jpg)
Andras Bodis-SzomoruICPR 2016
Volumetric surface reconstruction via 3DT
12
3D DELAUNAY TRIANGULATION
Why 3DT? simple adaptive / scalable efficient detail-preserving
Alternatives voxels (poor scalability) stixels (2.5D) octrees (more complicated) cell complex
(poor scalability, needs planes)
![Page 13: Efficient Volumetric Fusion of Airborne and Street -Side ...€¦ · stixels (2.5D) octrees (more complicated) cell complex (poor scalability, needs planes) ICPR 2016. Andras Bodis-Szomoru](https://reader035.vdocuments.mx/reader035/viewer/2022070915/5fb5fca9f8ac8529f92eee44/html5/thumbnails/13.jpg)
Andras Bodis-SzomoruICPR 2016
Volumetric surface reconstruction via 3DT
13
RAY SHOOTING
SENSOR
![Page 14: Efficient Volumetric Fusion of Airborne and Street -Side ...€¦ · stixels (2.5D) octrees (more complicated) cell complex (poor scalability, needs planes) ICPR 2016. Andras Bodis-Szomoru](https://reader035.vdocuments.mx/reader035/viewer/2022070915/5fb5fca9f8ac8529f92eee44/html5/thumbnails/14.jpg)
Andras Bodis-SzomoruICPR 2016
Volumetric surface reconstruction via 3DT
14
INSIDE/OUTSIDE VOLUMETRIC
CLASSIFICATION
![Page 15: Efficient Volumetric Fusion of Airborne and Street -Side ...€¦ · stixels (2.5D) octrees (more complicated) cell complex (poor scalability, needs planes) ICPR 2016. Andras Bodis-Szomoru](https://reader035.vdocuments.mx/reader035/viewer/2022070915/5fb5fca9f8ac8529f92eee44/html5/thumbnails/15.jpg)
Andras Bodis-SzomoruICPR 2016
Volumetric surface reconstruction via 3DT
15
REAL DATA (NOISY)
Post processing Denoising: simple smoothing Remove floating components
Expensive alternative Mesh tuning for photoconsistency
(see Vu et al. PAMI 2012)
![Page 16: Efficient Volumetric Fusion of Airborne and Street -Side ...€¦ · stixels (2.5D) octrees (more complicated) cell complex (poor scalability, needs planes) ICPR 2016. Andras Bodis-Szomoru](https://reader035.vdocuments.mx/reader035/viewer/2022070915/5fb5fca9f8ac8529f92eee44/html5/thumbnails/16.jpg)
Andras Bodis-SzomoruICPR 2016
Raycasting and voting scheme
16
![Page 17: Efficient Volumetric Fusion of Airborne and Street -Side ...€¦ · stixels (2.5D) octrees (more complicated) cell complex (poor scalability, needs planes) ICPR 2016. Andras Bodis-Szomoru](https://reader035.vdocuments.mx/reader035/viewer/2022070915/5fb5fca9f8ac8529f92eee44/html5/thumbnails/17.jpg)
Andras Bodis-SzomoruICPR 2016
Raycasting and voting scheme
18
inside and outside score per ray per tetrahedron
![Page 18: Efficient Volumetric Fusion of Airborne and Street -Side ...€¦ · stixels (2.5D) octrees (more complicated) cell complex (poor scalability, needs planes) ICPR 2016. Andras Bodis-Szomoru](https://reader035.vdocuments.mx/reader035/viewer/2022070915/5fb5fca9f8ac8529f92eee44/html5/thumbnails/18.jpg)
Andras Bodis-SzomoruICPR 2016
Volumetric optimization
19
binary labels
tetrahedraunary preference for in/out per tetrahedron
penalty for a face to be part of the surface Optimization: globally optimal GCO
![Page 19: Efficient Volumetric Fusion of Airborne and Street -Side ...€¦ · stixels (2.5D) octrees (more complicated) cell complex (poor scalability, needs planes) ICPR 2016. Andras Bodis-Szomoru](https://reader035.vdocuments.mx/reader035/viewer/2022070915/5fb5fca9f8ac8529f92eee44/html5/thumbnails/19.jpg)
Andras Bodis-SzomoruICPR 2016
Volumetric optimization
20
binary labels
tetrahedraunary preference for in/out per tetrahedron
penalty for a face to be part of the surface
sum over all rays
normalization
Optimization: globally optimal GCO
![Page 20: Efficient Volumetric Fusion of Airborne and Street -Side ...€¦ · stixels (2.5D) octrees (more complicated) cell complex (poor scalability, needs planes) ICPR 2016. Andras Bodis-Szomoru](https://reader035.vdocuments.mx/reader035/viewer/2022070915/5fb5fca9f8ac8529f92eee44/html5/thumbnails/20.jpg)
Andras Bodis-SzomoruICPR 2016
Volumetric optimization
21
binary labels
tetrahedraunary preference for in/out per tetrahedron
penalty for a face to be part of the surface
sum over all rays
normalization
area penalty(worked best with our unary)
Other examples tried triangle elongatedness long edge-triangles beta-skeleton (Labatut et al., CGF 2009)
Optimization: globally optimal GCO
![Page 21: Efficient Volumetric Fusion of Airborne and Street -Side ...€¦ · stixels (2.5D) octrees (more complicated) cell complex (poor scalability, needs planes) ICPR 2016. Andras Bodis-Szomoru](https://reader035.vdocuments.mx/reader035/viewer/2022070915/5fb5fca9f8ac8529f92eee44/html5/thumbnails/21.jpg)
Andras Bodis-SzomoruICPR 2016
Airborne / street-side surface fusion
22
AIRBORNE
TRUE SURFACE
![Page 22: Efficient Volumetric Fusion of Airborne and Street -Side ...€¦ · stixels (2.5D) octrees (more complicated) cell complex (poor scalability, needs planes) ICPR 2016. Andras Bodis-Szomoru](https://reader035.vdocuments.mx/reader035/viewer/2022070915/5fb5fca9f8ac8529f92eee44/html5/thumbnails/22.jpg)
Andras Bodis-SzomoruICPR 2016
Airborne / street-side surface fusion
23
AIRBORNE
TRUE SURFACE
STREET-SIDE
![Page 23: Efficient Volumetric Fusion of Airborne and Street -Side ...€¦ · stixels (2.5D) octrees (more complicated) cell complex (poor scalability, needs planes) ICPR 2016. Andras Bodis-Szomoru](https://reader035.vdocuments.mx/reader035/viewer/2022070915/5fb5fca9f8ac8529f92eee44/html5/thumbnails/23.jpg)
Andras Bodis-SzomoruICPR 2016
Airborne / street-side surface fusion
24
AIRBORNE
STREET-SIDE
TRUE SURFACE
FUSED SURFACE
![Page 24: Efficient Volumetric Fusion of Airborne and Street -Side ...€¦ · stixels (2.5D) octrees (more complicated) cell complex (poor scalability, needs planes) ICPR 2016. Andras Bodis-Szomoru](https://reader035.vdocuments.mx/reader035/viewer/2022070915/5fb5fca9f8ac8529f92eee44/html5/thumbnails/24.jpg)
Andras Bodis-SzomoruICPR 2016
Airborne / street-side ray conflicts
25
INSIDE VOTES(BEHIND RAYS)
OUTSIDE VOTES(CROSSED BY RAYS)
rays to low-quality airborne surface ruin the solution
![Page 25: Efficient Volumetric Fusion of Airborne and Street -Side ...€¦ · stixels (2.5D) octrees (more complicated) cell complex (poor scalability, needs planes) ICPR 2016. Andras Bodis-Szomoru](https://reader035.vdocuments.mx/reader035/viewer/2022070915/5fb5fca9f8ac8529f92eee44/html5/thumbnails/25.jpg)
Andras Bodis-SzomoruICPR 2016
Airborne / street-side ray conflicts
26
INSIDE VOTES(BEHIND RAYS)
OUTSIDE VOTES(CROSSED BY RAYS)
rays to low-quality airborne surface ruin the solution
![Page 26: Efficient Volumetric Fusion of Airborne and Street -Side ...€¦ · stixels (2.5D) octrees (more complicated) cell complex (poor scalability, needs planes) ICPR 2016. Andras Bodis-Szomoru](https://reader035.vdocuments.mx/reader035/viewer/2022070915/5fb5fca9f8ac8529f92eee44/html5/thumbnails/26.jpg)
Andras Bodis-SzomoruICPR 2016
Airborne / street-side point cloud blending
27
Observations airborne covers map airborne has much less detail not practical to do weighting
instead of blending
![Page 27: Efficient Volumetric Fusion of Airborne and Street -Side ...€¦ · stixels (2.5D) octrees (more complicated) cell complex (poor scalability, needs planes) ICPR 2016. Andras Bodis-Szomoru](https://reader035.vdocuments.mx/reader035/viewer/2022070915/5fb5fca9f8ac8529f92eee44/html5/thumbnails/27.jpg)
Andras Bodis-SzomoruICPR 2016
Airborne / street-side point cloud blending
28
![Page 28: Efficient Volumetric Fusion of Airborne and Street -Side ...€¦ · stixels (2.5D) octrees (more complicated) cell complex (poor scalability, needs planes) ICPR 2016. Andras Bodis-Szomoru](https://reader035.vdocuments.mx/reader035/viewer/2022070915/5fb5fca9f8ac8529f92eee44/html5/thumbnails/28.jpg)
Andras Bodis-SzomoruICPR 2016
Airborne / street-side point cloud blending
29
![Page 29: Efficient Volumetric Fusion of Airborne and Street -Side ...€¦ · stixels (2.5D) octrees (more complicated) cell complex (poor scalability, needs planes) ICPR 2016. Andras Bodis-Szomoru](https://reader035.vdocuments.mx/reader035/viewer/2022070915/5fb5fca9f8ac8529f92eee44/html5/thumbnails/29.jpg)
Andras Bodis-SzomoruICPR 2016
Airborne / street-side point cloud blending
30
substitute likelhood
distance term normal term
![Page 30: Efficient Volumetric Fusion of Airborne and Street -Side ...€¦ · stixels (2.5D) octrees (more complicated) cell complex (poor scalability, needs planes) ICPR 2016. Andras Bodis-Szomoru](https://reader035.vdocuments.mx/reader035/viewer/2022070915/5fb5fca9f8ac8529f92eee44/html5/thumbnails/30.jpg)
Andras Bodis-SzomoruICPR 2016
Airborne / street-side point cloud blending
31
blending energy
likelihood smoothness
smoothness term
![Page 31: Efficient Volumetric Fusion of Airborne and Street -Side ...€¦ · stixels (2.5D) octrees (more complicated) cell complex (poor scalability, needs planes) ICPR 2016. Andras Bodis-Szomoru](https://reader035.vdocuments.mx/reader035/viewer/2022070915/5fb5fca9f8ac8529f92eee44/html5/thumbnails/31.jpg)
Andras Bodis-SzomoruICPR 2016
Airborne / street-side point cloud blending
32
![Page 32: Efficient Volumetric Fusion of Airborne and Street -Side ...€¦ · stixels (2.5D) octrees (more complicated) cell complex (poor scalability, needs planes) ICPR 2016. Andras Bodis-Szomoru](https://reader035.vdocuments.mx/reader035/viewer/2022070915/5fb5fca9f8ac8529f92eee44/html5/thumbnails/32.jpg)
Andras Bodis-SzomoruICPR 2016
Airborne / street-side point cloud blending
33
3DT-BASED SURFACERECONSTRUCTION
![Page 33: Efficient Volumetric Fusion of Airborne and Street -Side ...€¦ · stixels (2.5D) octrees (more complicated) cell complex (poor scalability, needs planes) ICPR 2016. Andras Bodis-Szomoru](https://reader035.vdocuments.mx/reader035/viewer/2022070915/5fb5fca9f8ac8529f92eee44/html5/thumbnails/33.jpg)
Andras Bodis-SzomoruICPR 2016
Data reduction / speed boost for large datasets
34
O(107) points (street / aerial)O(108) raysO(km2) area
(illustration: mesh from airborne-only input)
![Page 34: Efficient Volumetric Fusion of Airborne and Street -Side ...€¦ · stixels (2.5D) octrees (more complicated) cell complex (poor scalability, needs planes) ICPR 2016. Andras Bodis-Szomoru](https://reader035.vdocuments.mx/reader035/viewer/2022070915/5fb5fca9f8ac8529f92eee44/html5/thumbnails/34.jpg)
Andras Bodis-SzomoruICPR 2016
Data reduction / speed boost for large datasets
35
(1) Point decimation (2) Ray decimation (3) Ray truncation
![Page 35: Efficient Volumetric Fusion of Airborne and Street -Side ...€¦ · stixels (2.5D) octrees (more complicated) cell complex (poor scalability, needs planes) ICPR 2016. Andras Bodis-Szomoru](https://reader035.vdocuments.mx/reader035/viewer/2022070915/5fb5fca9f8ac8529f92eee44/html5/thumbnails/35.jpg)
Andras Bodis-SzomoruICPR 2016
Data reduction / speed boost for large datasets
36
voxel grid (with hashing) point clustering centroid prototype merge rays parameter: voxel size
(1) Point decimation (2) Ray decimation (3) Ray truncation
![Page 36: Efficient Volumetric Fusion of Airborne and Street -Side ...€¦ · stixels (2.5D) octrees (more complicated) cell complex (poor scalability, needs planes) ICPR 2016. Andras Bodis-Szomoru](https://reader035.vdocuments.mx/reader035/viewer/2022070915/5fb5fca9f8ac8529f92eee44/html5/thumbnails/36.jpg)
Andras Bodis-SzomoruICPR 2016
Data reduction / speed boost for large datasets
37
voxel grid (with hashing) point clustering centroid prototype merge rays parameter: voxel size
(1) Point decimation (2) Ray decimation (3) Ray truncation
![Page 37: Efficient Volumetric Fusion of Airborne and Street -Side ...€¦ · stixels (2.5D) octrees (more complicated) cell complex (poor scalability, needs planes) ICPR 2016. Andras Bodis-Szomoru](https://reader035.vdocuments.mx/reader035/viewer/2022070915/5fb5fca9f8ac8529f92eee44/html5/thumbnails/37.jpg)
Andras Bodis-SzomoruICPR 2016
Data reduction / speed boost for large datasets
38
voxel grid (with hashing) point clustering centroid prototype merge rays parameter: voxel size
(1) Point decimation (2) Ray decimation (3) Ray truncation
![Page 38: Efficient Volumetric Fusion of Airborne and Street -Side ...€¦ · stixels (2.5D) octrees (more complicated) cell complex (poor scalability, needs planes) ICPR 2016. Andras Bodis-Szomoru](https://reader035.vdocuments.mx/reader035/viewer/2022070915/5fb5fca9f8ac8529f92eee44/html5/thumbnails/38.jpg)
Andras Bodis-SzomoruICPR 2016
Data reduction / speed boost for large datasets
39
voxel grid (with hashing) point clustering centroid prototype merge rays parameter: voxel size
(1) Point decimation (2) Ray decimation (3) Ray truncation
rays are dense keep one per point most perpendicular
to reduce mistakes
![Page 39: Efficient Volumetric Fusion of Airborne and Street -Side ...€¦ · stixels (2.5D) octrees (more complicated) cell complex (poor scalability, needs planes) ICPR 2016. Andras Bodis-Szomoru](https://reader035.vdocuments.mx/reader035/viewer/2022070915/5fb5fca9f8ac8529f92eee44/html5/thumbnails/39.jpg)
Andras Bodis-SzomoruICPR 2016
Data reduction / speed boost for large datasets
40
voxel grid (with hashing) point clustering centroid prototype merge rays parameter: voxel size
(1) Point decimation (2) Ray decimation (3) Ray truncation
rays are dense keep one per point most perpendicular
to reduce mistakes
![Page 40: Efficient Volumetric Fusion of Airborne and Street -Side ...€¦ · stixels (2.5D) octrees (more complicated) cell complex (poor scalability, needs planes) ICPR 2016. Andras Bodis-Szomoru](https://reader035.vdocuments.mx/reader035/viewer/2022070915/5fb5fca9f8ac8529f92eee44/html5/thumbnails/40.jpg)
Andras Bodis-SzomoruICPR 2016
Data reduction / speed boost for large datasets
41
voxel grid (with hashing) point clustering centroid prototype merge rays parameter: voxel size
rays are dense keep one per point most perpendicular
to reduce mistakes
similar to limiting penetration depth
truncate rays at points less tetrahedra crossed
(1) Point decimation (2) Ray decimation (3) Ray truncation
![Page 41: Efficient Volumetric Fusion of Airborne and Street -Side ...€¦ · stixels (2.5D) octrees (more complicated) cell complex (poor scalability, needs planes) ICPR 2016. Andras Bodis-Szomoru](https://reader035.vdocuments.mx/reader035/viewer/2022070915/5fb5fca9f8ac8529f92eee44/html5/thumbnails/41.jpg)
Andras Bodis-SzomoruICPR 2016
Data reduction / speed boost for large datasets
42
voxel grid (with hashing) point clustering centroid prototype merge rays parameter: voxel size
similar to limiting penetration depth
truncate rays at points less tetrahedra crossed
(1) Point decimation (2) Ray decimation (3) Ray truncation
rays are dense keep one per point most perpendicular
to reduce mistakes
![Page 42: Efficient Volumetric Fusion of Airborne and Street -Side ...€¦ · stixels (2.5D) octrees (more complicated) cell complex (poor scalability, needs planes) ICPR 2016. Andras Bodis-Szomoru](https://reader035.vdocuments.mx/reader035/viewer/2022070915/5fb5fca9f8ac8529f92eee44/html5/thumbnails/42.jpg)
Andras Bodis-SzomoruICPR 2016
Pipeline summary
43
POINT CLOUD BLENDING
VOLUMETRIC SURFACE RECONSTRUCTION(INTERPOLATORY)
AERIAL POINTS & RAYS
STREET-SIDE POINTS & RAYS
PREPROCESSING(NORMALS & DATA REDUCTION)
POST-PROCESSING(DE-NOISING)
Implementation 3DT partitioning (Matlab ~ CGAL) Graph cuts (GCoptimization library) 3DT adjacencies & ray shooting (C++)
![Page 43: Efficient Volumetric Fusion of Airborne and Street -Side ...€¦ · stixels (2.5D) octrees (more complicated) cell complex (poor scalability, needs planes) ICPR 2016. Andras Bodis-Szomoru](https://reader035.vdocuments.mx/reader035/viewer/2022070915/5fb5fca9f8ac8529f92eee44/html5/thumbnails/43.jpg)
Andras Bodis-SzomoruICPR 2016
Parameters
44
fixed in our experiments
![Page 44: Efficient Volumetric Fusion of Airborne and Street -Side ...€¦ · stixels (2.5D) octrees (more complicated) cell complex (poor scalability, needs planes) ICPR 2016. Andras Bodis-Szomoru](https://reader035.vdocuments.mx/reader035/viewer/2022070915/5fb5fca9f8ac8529f92eee44/html5/thumbnails/44.jpg)
Andras Bodis-SzomoruICPR 2016
Datasets
45
Airborne Nadir 15 cm GSD CapturingReality 50 GCPs
Street-side Hand held camera VisualSfM (sparse) / PMVS (dense) registered to the airborne data
(via IMU/GPS in industrial mobile mapping)
Munsterhof (140x160 m2) Limmatquai (400x400 m2)
629 street-side images 847 street-side images
![Page 45: Efficient Volumetric Fusion of Airborne and Street -Side ...€¦ · stixels (2.5D) octrees (more complicated) cell complex (poor scalability, needs planes) ICPR 2016. Andras Bodis-Szomoru](https://reader035.vdocuments.mx/reader035/viewer/2022070915/5fb5fca9f8ac8529f92eee44/html5/thumbnails/45.jpg)
Andras Bodis-SzomoruICPR 2016
Registration accuracy in our experiments
46
Munsterhof Limmatquai
registration errors between airborne and street-side point clouds at overlaps(distances in normal direction between mutual NNs)
![Page 46: Efficient Volumetric Fusion of Airborne and Street -Side ...€¦ · stixels (2.5D) octrees (more complicated) cell complex (poor scalability, needs planes) ICPR 2016. Andras Bodis-Szomoru](https://reader035.vdocuments.mx/reader035/viewer/2022070915/5fb5fca9f8ac8529f92eee44/html5/thumbnails/46.jpg)
Andras Bodis-SzomoruICPR 2016
Results: Process on Munsterhof
47
Airborne input (272k points) Street-side input (1.5M points, PMVS) Blending unary energies
![Page 47: Efficient Volumetric Fusion of Airborne and Street -Side ...€¦ · stixels (2.5D) octrees (more complicated) cell complex (poor scalability, needs planes) ICPR 2016. Andras Bodis-Szomoru](https://reader035.vdocuments.mx/reader035/viewer/2022070915/5fb5fca9f8ac8529f92eee44/html5/thumbnails/47.jpg)
Andras Bodis-SzomoruICPR 2016
Results: Process on Munsterhof
48
Point cloud segmentation Blended point clouds Surface reconstruction
![Page 48: Efficient Volumetric Fusion of Airborne and Street -Side ...€¦ · stixels (2.5D) octrees (more complicated) cell complex (poor scalability, needs planes) ICPR 2016. Andras Bodis-Szomoru](https://reader035.vdocuments.mx/reader035/viewer/2022070915/5fb5fca9f8ac8529f92eee44/html5/thumbnails/48.jpg)
Andras Bodis-SzomoruICPR 2016
Munsterhof Results
49
m2 areaM pointsM tetrahedraM rayspoints/m2 aerialpoints/m2 streetmin @ 3.4 GHz 1coremin with decim / trunc
140x1601.811
13.28
1085
<1
![Page 49: Efficient Volumetric Fusion of Airborne and Street -Side ...€¦ · stixels (2.5D) octrees (more complicated) cell complex (poor scalability, needs planes) ICPR 2016. Andras Bodis-Szomoru](https://reader035.vdocuments.mx/reader035/viewer/2022070915/5fb5fca9f8ac8529f92eee44/html5/thumbnails/49.jpg)
Andras Bodis-SzomoruICPR 2016
Limmatquai Results
50
m2 areaM pointsM tetrahedraM rayspoints/m2 aerialpoints/m2 streetcm vox decimation)GB memorymin @ 3.4 GHz 1coremin with decim / trunc
400x4002.6
16.767.8
726
(201221<5
![Page 50: Efficient Volumetric Fusion of Airborne and Street -Side ...€¦ · stixels (2.5D) octrees (more complicated) cell complex (poor scalability, needs planes) ICPR 2016. Andras Bodis-Szomoru](https://reader035.vdocuments.mx/reader035/viewer/2022070915/5fb5fca9f8ac8529f92eee44/html5/thumbnails/50.jpg)
Andras Bodis-SzomoruICPR 2016
Limmatquai Results
51
m2 areaM pointsM tetrahedraM rayspoints/m2 aerialpoints/m2 streetcm vox decimation)GB memorymin @ 3.4 GHz 1coremin with decim / trunc
400x4002.6
16.767.8
726
(201221<5
![Page 51: Efficient Volumetric Fusion of Airborne and Street -Side ...€¦ · stixels (2.5D) octrees (more complicated) cell complex (poor scalability, needs planes) ICPR 2016. Andras Bodis-Szomoru](https://reader035.vdocuments.mx/reader035/viewer/2022070915/5fb5fca9f8ac8529f92eee44/html5/thumbnails/51.jpg)
Andras Bodis-SzomoruICPR 2016
Results on different input types
52
Airborne onlyFusion aerial + VisualSfM
~8 points/m2~19 points/m2
Fusion aerial + PMVS
~108 points/m2
![Page 52: Efficient Volumetric Fusion of Airborne and Street -Side ...€¦ · stixels (2.5D) octrees (more complicated) cell complex (poor scalability, needs planes) ICPR 2016. Andras Bodis-Szomoru](https://reader035.vdocuments.mx/reader035/viewer/2022070915/5fb5fca9f8ac8529f92eee44/html5/thumbnails/52.jpg)
Andras Bodis-SzomoruICPR 2016
Cumulated histogram of mesh errors (vs runtimes)
53
our highest-density fused mesh (Munsterhof)on a single 3.4 GHz CPU core
PMVSray truncation
PMVS, ray decim(with/out trunc)
PMVS, point decim(10 / 20 / 30 cm voxels)
VisualSfMwith/out reductions
aerial-only inputwith/out reductions
good quality
poor quality
52s16s
29s
47s33s
70s23s46s82s
125s137s182s285s - reference
reference:timings:
![Page 53: Efficient Volumetric Fusion of Airborne and Street -Side ...€¦ · stixels (2.5D) octrees (more complicated) cell complex (poor scalability, needs planes) ICPR 2016. Andras Bodis-Szomoru](https://reader035.vdocuments.mx/reader035/viewer/2022070915/5fb5fca9f8ac8529f92eee44/html5/thumbnails/53.jpg)
Andras Bodis-SzomoruICPR 2016
Numerical details (see paper)
54
Parameter variations (others fixed) Accuracy measures Timings
Detailed statisticsDatasets
![Page 54: Efficient Volumetric Fusion of Airborne and Street -Side ...€¦ · stixels (2.5D) octrees (more complicated) cell complex (poor scalability, needs planes) ICPR 2016. Andras Bodis-Szomoru](https://reader035.vdocuments.mx/reader035/viewer/2022070915/5fb5fca9f8ac8529f92eee44/html5/thumbnails/54.jpg)
Andras Bodis-SzomoruICPR 2016
Qualitative effect of prior blending with normals
55
WithBlending
WithoutBlending
Without Normals(Distance Only)
With Normals
![Page 55: Efficient Volumetric Fusion of Airborne and Street -Side ...€¦ · stixels (2.5D) octrees (more complicated) cell complex (poor scalability, needs planes) ICPR 2016. Andras Bodis-Szomoru](https://reader035.vdocuments.mx/reader035/viewer/2022070915/5fb5fca9f8ac8529f92eee44/html5/thumbnails/55.jpg)
Andras Bodis-SzomoruICPR 2016
Qualitative effect of ray decimation and truncation
56
PM
VS
SfM
No reductions
![Page 56: Efficient Volumetric Fusion of Airborne and Street -Side ...€¦ · stixels (2.5D) octrees (more complicated) cell complex (poor scalability, needs planes) ICPR 2016. Andras Bodis-Szomoru](https://reader035.vdocuments.mx/reader035/viewer/2022070915/5fb5fca9f8ac8529f92eee44/html5/thumbnails/56.jpg)
Andras Bodis-SzomoruICPR 2016
Qualitative effect of ray decimation and truncation
57
PM
VS
SfM
Ray decimation
![Page 57: Efficient Volumetric Fusion of Airborne and Street -Side ...€¦ · stixels (2.5D) octrees (more complicated) cell complex (poor scalability, needs planes) ICPR 2016. Andras Bodis-Szomoru](https://reader035.vdocuments.mx/reader035/viewer/2022070915/5fb5fca9f8ac8529f92eee44/html5/thumbnails/57.jpg)
Andras Bodis-SzomoruICPR 2016
Qualitative effect of ray decimation and truncation
58
PM
VS
SfM
Ray decimation and truncation
![Page 58: Efficient Volumetric Fusion of Airborne and Street -Side ...€¦ · stixels (2.5D) octrees (more complicated) cell complex (poor scalability, needs planes) ICPR 2016. Andras Bodis-Szomoru](https://reader035.vdocuments.mx/reader035/viewer/2022070915/5fb5fca9f8ac8529f92eee44/html5/thumbnails/58.jpg)
Andras Bodis-SzomoruICPR 2016
Quick Comparison to Poisson Surface Reconstruction
59
Ours PMVS+aerial after blending full rays
![Page 59: Efficient Volumetric Fusion of Airborne and Street -Side ...€¦ · stixels (2.5D) octrees (more complicated) cell complex (poor scalability, needs planes) ICPR 2016. Andras Bodis-Szomoru](https://reader035.vdocuments.mx/reader035/viewer/2022070915/5fb5fca9f8ac8529f92eee44/html5/thumbnails/59.jpg)
Andras Bodis-SzomoruICPR 2016
Quick Comparison to Poisson Surface Reconstruction
60
Poisson PMVS+aerial after blending Depth 12, Div 8 no rays used Inflated / smoother ground collapses narrow struct lost aerial artifacts remain
![Page 60: Efficient Volumetric Fusion of Airborne and Street -Side ...€¦ · stixels (2.5D) octrees (more complicated) cell complex (poor scalability, needs planes) ICPR 2016. Andras Bodis-Szomoru](https://reader035.vdocuments.mx/reader035/viewer/2022070915/5fb5fca9f8ac8529f92eee44/html5/thumbnails/60.jpg)
Andras Bodis-SzomoruICPR 2016
Conclusions
61
3DT-fusion of airborne & street-side data
Point cloud blending against gross ray conflicts
Reduction techniques for large urban scenes
Detailed runtime vs. quality experimentation
Complete & detailed (LoD-3) models in minutes / km2
varcity.eu
![Page 61: Efficient Volumetric Fusion of Airborne and Street -Side ...€¦ · stixels (2.5D) octrees (more complicated) cell complex (poor scalability, needs planes) ICPR 2016. Andras Bodis-Szomoru](https://reader035.vdocuments.mx/reader035/viewer/2022070915/5fb5fca9f8ac8529f92eee44/html5/thumbnails/61.jpg)
Efficient Volumetric Fusion of Airborne and Street-Side Data for Urban Reconstruction
András Bódis-Szomorú, Hayko Riemenschneider, Luc Van Gool
varcity.eu
Thank you!