fondamentals of virtual and augmented reality click to edit … · 2019-10-02 · 3 “virtual and...

Click to edit Master title style

“Virtual and Augmented Reality”

Interaction speciality – Computer Science Master - University Paris-Saclay

Fondamentals of Virtual and Augmented Reality

AR: an introduction

Jeanne Vezien

Vezien@limsi.fr

“Virtual and Augmented Reality” course Interaction speciality – Computer Science Master - University Paris-Saclay

Plan of the lecture

1. Augmented reality:

what and why

2. User tracking

3. Real world reconstruction

4. 3D Graphics

5. Augmentation

Wikipedia says…

Augmented reality (AR) is a term for a live

direct or indirect view of a physical, real-world

environment whose elements are augmented by

computer-generated sound, video, graphics,

haptic or GPS data.

Augmentation is conventionally in real-time and

in semantic context with environmental

elements, such as sports scores on TV during a

match.

With the help of advanced AR technology

(computer vision and object recognition) the

information about the surrounding world

becomes interactive, e.g. artificial information

about the environment can be overlaid.

Augmented reality was coined by Thomas

Caudell, working at Boeing, in 1990. ARToolkit (Kato & Billinghurst, 2001)

InteractiveReal-time

Contextualized

The dinosaur

is rendered:

- Interactively

- In real-time

- In context

AR example

2018: the competition began

• Hololens (Microsoft, 2016)

• Commercial in 2017

• 50 000 units sold.

• V2 in 2019

• Magic Leap (Magic Leap Inc, 2018)

• Released august 2018

• Maximum Hype

• No business (yet)

2019: the competition is on

Hololens 2

Real time hand tracking

(grab, pinch, push,

slide…)

Gaze tracking

Magic Leap 1

2.295 $

Real/virtual Continuum (Milgram 1994)

AR: examples

Not a new idea….

• User position and gaze provides context ,

sometimes with help (markers)

• Audio is nice (non-obstrusive), can be

stopped anytime.

• No computer involved

• No sensing involved : limited interaction

Now associated with head-tracking,

localization…

Started in 1957 (Roosevelt home)

Principio system

(2007)

AR: examples

See-through augmented vision: « classic AR »

Tourism: Archeoguide (2002)

AR: examples

Assemply / Maintenance / Repair

Maintenance: help technician with

contextualized content

BMW, 2010

Matris project, 2007

Fiducial Text

ActionGraphics

AR: examples

Way-findingIphone app: NY nearest subway (and many cities, airports have their own app)

https://www.youtube.com/watch?v=ps49T0iJwVg

Augmented Car Finder

• Data is collected off-line

• Access to database is native on mobile phones

(http protocol)

AR: examples

Safe landing

Hunter system: Landing guiding system by AR

tracking techniques (computer vision based)

www.novemberkiloecho.com

War "games" (1/2)

Track target w.r.t. weapon

Coupled with (off-line) Geographical Information

system

… and strategic realtime info (GPS, detectors)

gun-mounted display

War "games" (2/2)

Tanagram.com concept (closed)

Heads Up Displays

TopOwl helmet (helicopter)

In operation since 2008

Thalesgroup.com

Eye tracking

Symbology

Gun pointing

Night VisionHeavy and $$$ !

Heads Up Displays (2)

Skully web helmet (2015)

(+ rear cameras)

Intuitive Aerial (Sweden, 2013)

MR: examples

Games and promotional contents

EyePet for PSP (2009)

Topps 3D baseball cards (2009)

PSVita (2012)

Father.io (2016): Massive Multiplayer Laser Tag

Pokémon GO !

The Pokemon Company (Nintendo) + Niantic (Ingress)

Since July 2016

Game brought in more than $1.2 Billion !

2018: 104M$ / month !

Success = the right mix of game + social

Possible only with AR (or connected VR ?)

Other examples

Medical: Surgery planning, nurse and student training…

University of North Carolina

Augmented virtuality (presented on screen)

Track hand-held device or body

Coupled with (off-line) anatomic data

DaVinci Robot

Intuitive Surgical

3 000 ex.

1 M.$ per unit30 problems over 17000

interventions, 1 death (bad

manip.)

Less stress

More precision

Comfortable

Complete asepsis

No tremor

abdomen

surgery

• Militaires: « future soldier », BARS

• Médical : assistance chirurgicale

• Tourisme

• Customization (mode)

Virtual try-ons and Customize

Augmented virtuality or

Track users anatomy

and motion (better still)

Coupled with external

data on-line

Augment.com (2017)Ikea (2014)

ForeverMark (2011)

MR: the SACARI example

Mixed virtuality: user not moving with camera,

indirect view of real world

Tele-immersion : provide sense of presence of remote environment

Internship possibilities in VENISE team !

N. Khenak et al. 2019

Blurred boundaries: augmented movies

Is this real ? Augmented ? Virtual ?

On-line ? Off-line ?

Avatar (2009)Zoe Zaldana

Blurred boundaries

Is this still AR ?

AR will be everywhere

when efficient, cheap see-

through displays will

become available

Google moto: The World's

Information in Context

Street View is close to

Augmented Virtuality

Google glass 2 (2019)

The Ingredients for AR

Capture real world

Capture virtual world

(Computer Graphics)

Capture real world

Capture virtual world

(Computer Graphics)

Present to user

(Augmentation)

Real 3D world capture

Two ingredients necessary for

successful AR:

Real-time 3D tracking of user

viewpoint w.r.t. world

3D scene analysis realistic

augmentation WorldViz tracking

Structure sensor

(Occipital, 2019) https://structure.io/

Real 3D world

successful AR:

Real-time 3D tracking of

user viewpoint w.r.t. world

3D scene analysis

realistic augmentation

Real-Time tracking

Many means to compute positioning information:

Geolocalization

Electro-magnetic

Acoustic

Inertial

Vision-based (active or passive)

Covered in « stereo and tracking » course

Image-based 3D tracking

It is a special case of adaptive

rendering ( for HMDs)

Environment control ++

Markers are still needed

(often)

Special case of

structure/motion estimation

AR meets movie industry:

the user is the director !

markersKnown as motion estimation / match moving

From [3]

= user

From [3]

Inside-out vs. Outside-in

= user

Inside-out: user wears camera Outside-in: camera observes user

• Inside-out better at estimating relative rotation

• Outside-in much more convenient : don’t have to wear a camera

• Special case: console gaming

• Inside-out easier for wearable AR: mobility, ego-centered

reference frame

• Needed for nomadic apps

• Cameras are getting small and commodity items

Projection is necessary to map 3D points on 2D pixels:

P(X,Y,Z)

p(x,y)

Focal point

Focal plane

3D tracking: equations

Objective: find camera parameters and position

110001

333231

232221

131211

Tracker

Camera

Perspective equation (extrinsic parameters):

Cam. Position,

Intrinsic parameters (scale, offset):

Pixels

M(X,Y,Z)

3D tracking: equations

TtTtTt

Projection equation:

image 3D worldm = P . M

motion structure

The Maths

TtTtTt

motion structure

The Maths

World camera =

Camera position

TtTtTt

motion structure

The Maths

3D 2D projection

TtTtTt

motion structure

The Maths

Sensor image (pixels)

TtTtTt

motion structure

The Maths

(i,j,k,Tx,Ty,Tz) can be computed if m and M are known.

Not linear in the motion parameters !!

Dementhon-Davis (1992)

Several methods to recover calibration exist (Tsai, Lowe…)

" Model-Based object pose in 25 lines of code" is a simple

yet elegant method to obtain pose rapidly if a rigid 3D model

is provided.

),( 00 yx Tz

Projection of T(Tx,Ty,Tz) Relative depth around Z0=Tz

Idea consists in linearizing xi and yi to compute iJI ,,

ziz TTZ

i PJyy

Geometrical interpretation:

Image plane

Reference plane

Orthographic

projection

Perspective

projection

Linear approximation of perspective

POSIT algorithm

Summarize:

jPJyyyy

iPIxxxx

=Jwith.)1(

=Iwith.)1(

If εi is known then I,J,Tz,X0=x0Tz,Y0=y0Tz can be computed

If I,J,Tz k is known

Iterate starting with

Converges rapidly:

3D tracking: off-the-shelf solution

• Kato & Billinghurst (HIT lab, univ. of Washington) introduce

in 1999 a tool for teleconferencing:

"Marker Tracking and HMD Calibration for a video-

based Augmented Reality Conferencing System.“

Soon to become ARToolkit

ARToolkit: main characteristics

• Fast and cheap 6D marker tracking.

• Distributed with complete source code.

OpenSource with GPL license for

noncommercial usage.

• Multiplatform (Linux, MacOS and Windows) .

• Multiple input sources (USB, Firewire) ,

multiple format (RGB, YUV) supported.

• Multiple camera tracking supported.

• GUI initializing interface.

• Easy calibration routine.

• Fast rendering based on OpenGL.

• 3D VRML support.

• Simple and modular API (in C and C++).

• Complete set of samples and utilities.

• Supports both video and optical see-through

ARToolkit: main processing loop

Since v4, Pose estimation is performed using the

Iterative Closest Point (ICP) algorithm (Besl, 1992)

Important details

Virtual objects appear only when complete markers are visible:

Movement

Orientation

Lighting conditions

Pattern Size (cm.) Usable Range (cm.)

6.98 40.64

8.89 63.5

10.79 86.36

In practice: close range only !

3D tracking: off-the-shelf solution

www.layar.com

web-based, per page service

www.wikitude.com: AR SDK

Vuforia (qualcomm) multiplatform SDK

Including Unity

Inglobe AR: AR solutions, ARPlayer+plugins (40.000 registered users)

2490€

focus on printed text

Markerless 3D tracking and AR dev: battle of

the SDKs

Spatial anchors

Cloud anchors

1.2ARKit 2

ARWorldMap

Real 3D world

successful AR:

Real-time 3d tracking of user

viewpoint w.r.t. world

3D scene analysis

realistic augmentation

2 components: Light and Geometry

Lighting coherency Motion coherency

Geometric coherency

Light probe

3D real world analysis

Aim: Capture light coming from light sources in real world

10,000:1 is the (static) human eye ratio between brightest and

darkest shade: cannot be represented on 8 bits

Stored in High Dynamic Range images (.hdr image format)

From all directions at a given point

Light Probe

Half-life 2: normal and HDR

Note: advanced graphics techniques can

use hdr images (Unreal engine)

1,1),( vu

v1tan 22 vu

A direction vector in the world (Dx, Dy, Dz)

u = r. Dx and v = r.Dy with

unit vector pointing in the direction (u,v) is obtained by rotating

(0,0,-1) by:

1) degrees around the y (up) axis

2) degrees around the -z (forward) axis.

1 )(cos1

Light Probe (2)

World content

Geometry: 3D reconstruction : what is

where ?

Semantic : augmentation of context

object recognition

interpretation of content

Comprendre l’environnement 3D en termes : http://www.truevisionsys.com

Markers cannot always be present !

If structure is known: a priori model tracking (see before):

an object is a (collection of) marker

Remember : projection combines structure and motion (m

= P.M)

Idea: recover both simultaneouly !

+ gives tracking and reconstruction at the same time

- highly non-linear : requires iterations + data filtering (remove

outliers )

Geometry: 3D reconstruction

Factorization method introduced by Tomasi and Kanade (1992)

mij = Pj. Mi

Factorization method

2x1 = 2x3 3x1

jth frame

ith point

• Compact representation:

• Consider P points projecting on F frames:

• There is an of possible (P,M) pairs!

• Solved via SVD decomposition (but not uniquely)

Carlo Tomasi and Takeo Kanade. (November 1992)

"Shape and motion from image streams under orthography: a factorization method."

International Journal of Computer Vision, 9 (2): 137–154.

Images !

m is big, but rank 3 ! m= P.M

Modern Approach: SfM

• Structure from Motion (or SLAM)

Inertial-Aided Sequential 3D Metric Surface

Reconstruction from Monocular Image Streams

Aufderheide et al. (2013).

• Initial pose estimation with factorization with a few keyframes

• Non-linear (costly) optimization (bundle adjustment)

• Real-time solutions are starting to emerge

S. Christy et R. Horaud : "Euclidean Shape and Motion from Multiple Perspective Views by Affine Iterations"

IEEE Transactions on Pattern Analysis and Machine Intelligence, Volume 18, Number 11, Pages 1098--1104 - November 1996

Factorization: results

Hanno Ackermann, University of Hanovre (2008)

Computer Vision is good at locating features:

points

regions

textures

contours

… at different scales !

Image-based 3D reconstruction

Harris (1988)

SURF detector (OpenCV), Bay (2006)

2D region detection (region-growing)

Problems :

Robustness of detection

Matching (space/time)

Is becoming robust enough

For now: limited use , constraints

(model-based)

Moving fast with new AR devices

Markerless tracking

Wang & Popovic, MIT, 2010

Boujou: http://vicon.com/boujou/

Commercial solutions

Pricey (3500€)

New real-time sensors: depth cameras (Zcam)

Infer depth for ALL pixels

Early : laser, stereoPerceptron

LIDAR , 1995

Devernay, 1994

Now: structured lighting : Zcam for 100 € !

Kinect, 2010

Augmented Reality Magic

Mirror using the Kinect

Tobias Blum

Solves the "where" but not the "what":

objects must still be identified (segmented)

Hololens (2016, 3000$)

Augmented reality « see-through »

glasses

Markerless 3D scanner with :

depth camera (Kinect-like) Spatial

Mapping

4 environment cameras

Inertial Measurement UnitHead tracking

SLAM = Simultaneous Localization and Mapping

= markerless tracking !

Magic Leap (2018, 2295$)

Augmented reality « see-through »

glasses

2 focus: near/far !

Markerless 3D scanner with :

depth camera (Kinect-like) Spatial

Mapping

4 environment cameras

Inertial Measurement Unit

Magnetic tracker

Head tracking

SLAM based on Deep Neural Network

}DeTone & Malisiewicz

Towards Geometriv Deep SLAM

https://arxiv.org/pdf/1707.07410.pdf

Stereo for AR: augmentation

Object location (table…)

Object occlusion

Shadows and light

interactions

In real-time !!

Will soon happen in

movie industry ($$$)

3D reconstruction is necessary for realistic CG blending:

Source: X3d consortium

3D analysis of

real images

(camera +

reconstruction)

Augmentation: CG blending

CG graphics+3D analysis of

real images

(camera +

reconstruction)

CG graphics

Compositing masks

(shadows, occlusions…)

+3D analysis of

real images

(camera +

reconstruction)

match move

CG graphics

Compositing masks

(shadows, occlusions…)

Augmented image

+3D analysis of

real images

(camera +

reconstruction)

match move

Matching constraints:

• geometry (epipolar)

• photometry

H. Jin, P. Favaro, and S. Soatto.

A Semi-direct Approach to Structure From Motion. The Visual Computer, 19(6): 377-394, October 2003.

Example of augmentation pipeline (1)

image analysis

Region matching (1992)

3D reconstruction

3D reconstruction of regions based on planar equations:

Hypothesis: world is piece-wise planar

Prior information:

explicit 3D model

motion constraints

3D registration by ICP = Iterative

Closest Point (Besl 92):

always converges

model points do not coincide with

reconstruction points

3D reconstruction

3D virtual content

Two steps:

Geometric modeling

(sometimes based on 3D

scans): Maya, Unity,

Blender, Sketchup, etc.

Photo-realistic rendering:

textures, lights, reflectance,

shadows, etc.

Note: Light probes

environment mapping

Light interaction: a must

Geometric coherency is good

but… light is essential !

Shadows in real-time =

geometry + light

Standard technique for CG renderers:

Ground (real)

(virtual)PSP

Shadow projection techniques

Many different techniques !

Plane Projected

Shadows

Projected

Shadows

Depth Shadow

MappingVertex Projection

Shadow

Volumes

Quick, not much

calculations

Quick, almost no

calculations

Very quick, not

much calculations

Slow with high-res

meshes

Slow, lots of

calculations

High detail Detail depends

on texture

Detail depends

on textureHigh Detail High detail

No self-

shadowingNo self-

shadowingSelf-shadowing

No self-shadowing Self-shadowing

No shadow

receivers

Shadow

receivers

Shadow

receivers

No shadow

receivers

Shadow

receivers

Shadow mapping

M. Haller, S. Drab, and W. Hartmann, 2003.

"A real-time shadow approach for an augmented reality application using shadow

volumes," in VRST 03: Proceedings of the ACM symposium on Virtual reality software and

technology, New York, NY, USA, 2003, pp. 56-65

Shadow volume with stencil drawing

animations

Two steps:

Virtual motions must be

coherent with real ones :

match-moving

Motion is camera-centric

Virtual camera must be

identical to real one (off-line

calibration is necessary)

Accommodate zoom !

Occlusion mask

Virtual object rendering

(alone)

Step by step rendering

Real objects: reference white

(albedo)

"black" virtual objects:

Shadow computation

Final composition =

real image *(1-mask) * attenuation

Virtual objects * mask

+ reflexions * (1-mask)

Final compositing

Other example (1)

Objets déformables + occultations réel/virtuel

Other example (2)

Reference white

Shadows of

virtual objects on

the real scene

Other example (3)

Final result

Conclusion

• Augmented reality is a reality

• Is a complicated process, requires

a lot of expertise and hard/software

• In-depth coherency of geometry

and photometry

• Real-time challenges

Elaborate rendering (GPU)

Match moving (markerless)

3D reconstruction (occlusions)

User interaction: hand and

body tracking

THE END !

Denno Coil, 2007

fondamentals of virtual and augmented reality click to edit … · 2019-10-02 · 3 “virtual and...

Documents

virtual commerce using augmented reality

august 2017 | virtual reality & augmented reality

when is reality not real enough? augmented and virtual...

virtual & augmented reality

augmented reality vs virtual reality

ip rights in augmented reality and virtual...

virtual & augmented reality · virtual reality augmented...

inscale - virtual reality & augmented reality services

enable your virtual reality & augmented reality technology...

present & future, augmented reality & virtual reality

state of augmented reality, virtual reality and mixed...

hci : virtual reality (augmented & mixed reality)

virtual reality and augmented reality in education

virtual mob - augmented reality solutions

vintechdreamlab - augmented reality, virtual reality

immersive mobile technologies: virtual/augmented reality...

efektivitas media virtual reality dan augmented reality

augmented reality and virtual reality scaffoldings in

an augmented reality and virtual reality pillar for ... ·...

virtual- and augmented reality in education · •augmented...