Markus Schuetz, 2016.07.26

MASSIVE TIME-LAPSE POINT CLOUD RENDERING IN VIRTUAL REALITY

2

AGENDA

• Why?

• Performance and Rendering Techniques

• Rendering Quality

• Interaction in Virtual Reality

3

NVIDIAS NEW HEADQUARTERCURRENTLY UNDER CONSTRUCTION

4

DRONE SCANS

• ~Daily

• Point cloud created from drone images

• ~decimeter resolution

• 20 to 60 million points per time-slice

• 200 time-slices within first year

• Exterior only

5

LASER SCANS

• ~Monthly

• Terrestrial laser scanning

• ~millimeter/centimeter resolution

• ~800 million points per time-slice

• 10 time-slices within first year

• Interior & Ground Level Scans

6

VIRTUAL REALITYTRADITIONAL

VIEWER PERFORMANCE REQUIREMENTS

• 30-60 FPS

• ~2 Million Pixel

• Anti-Aliasing nice to have

• 90 FPS

• Render scene twice, once for each eye

• >2 Million Pixel per Eye

• Anti-Aliasing must-have!(especially for point clouds)

7

MEETING PERFORMANCE REQUIREMENTS

• Too much data. Out-Of-Core structures necessary

• Multi-Resolution OctreeSource: “Domitilla Catacomb Walkthrough – Dealing with more than 1 Billion Points”, Claus Scheiblauer

• Load and render only visible parts up to desired Level of Detail

source: “Potree: Rendering Large Point Clouds in Web Browsers”, Markus Schuetz

8

ADAPTIVE POINT SIZES

• Noticeable difference in point-density and holes where LOD changes

• Adjust point size to level of detail

• Nodes with different level overlap-> LOD != node level

• LOD = local leaf-node level

• Find local leaf-node level through octree-traversal in vertex-shader

9

EYE-DOME-LIGHTING

• Most point clouds do not contain surface normals. Sometimes no colors, either.

• Colors may suffer from overexposure

• EDL does not require normals!

• Creates Illumination & Outlines

• Conceptually close to SSAO

• See:“Interactive Scientific Visualisation of Large Datasets: Towards a Perception-based Approach”, Christian Boucheny

10

POINT-INTERPOLATION

• Points usually rendered as squares or circles

• Occlusions can reduce readability

• Render as paraboloids instead

• By altering depth in fragment shader

• Disables early-z, recover some speed with:“layout(depth_greater) out float gl_FragDepth;”

• Results in nearest-neighbor-like interpolation between points -> produces Voronoi Diagrams

“High-Quality Point-Based Rendering Using Fast Single-Pass Interpolation ”,

Schütz M., Wimmer M.

11

QUALITY

• Strong aliasing inherent to Point Cloud Rendering

• Surfaces made up of overlapping points that occlude each other. Closest to camera wins.

• Aliasing more noticeable in VR due to constant motion and low resolution

• Perceived as “sparkling”

12

SILHOUETTESLEVEL OF DETAIL

SOURCES OF ALIASING

Object Silhouettes

Point Sprite Silhouettes

Building Multi-Resolution Octree, only considering point coordinates

Like Nearest-Neighbor

OCCLUSIONS

Surface Patches made up of overlapping points

Points fighting for visibility

source: “Potree: Rendering Large Point Clouds in Web Browsers”, Markus Schuetz

13

POINT CLOUD MIP-MAPS

• Additionally store averaged colors in lower Levels-Of-Detail

• Like Mip-Mapping for point clouds

• Averaged colors partially reduce occlusion-aliasing

14

MSAA

• Multisample Anti-Aliasing

• Different sample sizes for quality vs. speed

• Reduces impact of noise

• Helps with inhomogeneous colors from merging multiple scan locations

• Reduces “sparkling” during motion!

• Partially reduces occlusion-aliasing

15

ALIASING FROM OCCLUSIONS

• Largely solved through combination of adaptive point sizes, Mip-Maps and MSAA.

• Adaptive Sizes make points as big as necessary but not bigger

• Mip-Maps let otherwise unintentionally occluded points affect the result by contributing to the average

• MSAA lets multiple points affect the same pixel

16

ANTI-ALIASED POINT CLOUDS

17

POINT CLOUDS IN VR

• Point clouds often not dense enough for real-world scale

• Can’t just do arbitrary locomotion.

• Tracked area restricted to a few meters

• Movements in VR that are counter to what the body feels and expects can easily make users dizzy

Interaction Challenges

18

POINT CLOUDS IN VR

• User stuck in a small room but arbitrary exploration possible through squeezing/stretching/rotating/dragging the model

• Drag & Drop using a single controller

• Pinch-To-Zoom like gesture to scale & rotate

• Predefined views to choose from

Interaction Challenges

19